The cost of moving optimally: kinematic path selection

Abstract

It is currently unclear whether the brain plans movement kinematics explicitly or whether movement paths arise implicitly through optimization of a cost function that takes into account control and/or dynamic variables. Several cost functions are proposed in the literature that are very different in nature (e.g., control effort, torque change, and jerk), yet each can predict common movement characteristics. We set out to disentangle predictions of the different variables using a combination of modeling and empirical studies. Subjects performed goal-directed arm movements in a force field (FF) in combination with visual perturbations of seen hand position. This FF was designed to have distinct optimal movements for muscle-input and dynamic costs while leaving kinematic cost unchanged. Visual perturbations in turn changed the kinematic cost but left the dynamic and muscle-input costs unchanged. An optimally controlled, physiologically realistic arm model was used to predict movements under the various cost variables. Experimental results were not consistent with a cost function containing any of the control and dynamic costs investigated. Movement patterns of all experimental conditions were adequately predicted by a kinematic cost function comprising both visually and somatosensory perceived jerk. The present study provides clear behavioral evidence that the brain solves kinematic and mechanical redundancy in separate steps: in a first step, movement kinematics are planned; and in a second, separate step, muscle activation patterns are generated.

it is suggested in the literature that the brain computes muscle activation patterns that minimize a task-relevant cost function (e.g., Hogan 1984; Hasan 1986; Todorov 2004). The most important cost variables put forward in the literature can be categorized in three distinct levels: a control (muscle input) level, a dynamic level, and a kinematic level. At the control level there is, for example, control effort (i.e., the sum of squared muscle activation; Fagg et al. 2002; Todorov 2002; van Bolhuis and Gielen 1999), which has been theorized to minimize end-point variance due to signal-dependent noise (Diedrichsen et al. 2010; Harris and Wolpert 1998). Two influential variables at the dynamic level are torque change (Uno et al. 1989) and energy expenditure (Alexander 1997; Kistemaker et al. 2010). At the kinematic levels there is one important variable, jerk (i.e., the time derivative of acceleration), proposed to capture common invariant kinematic features observed experimentally (Hogan 1984). All of these cost variables have been successful at predicting common kinematics of human movements in free space (in this work taken as movements performed in an inertial reference frame without additional forces applied to the moving limb), even though they are very different in nature. As a result it is experimentally impractical to discern if and what unique cost function may be used by the brain.

Evidence exists in the literature that (visual) kinematics play an important role in movement planning. For example, Thoroughman et al. (2007) showed human arm movements performed in a robot-induced force field after learning were consistent with minimization of kinematic jerk, but not with minimizing either end-point variance or minimal torque change. In addition, Wolpert et al. (1995) used a simple, yet very clever experimental setup to artificially increase the curvature of the seen hand trajectory. Even though subjects did not need to change their muscle activation patterns to reach the targets, it was found that participants moved to reduce the visual perturbation. These results clearly suggest that kinematic variables play an important role in kinematic planning. However, it does not exclude the possibility that dynamic and/or control variables may also play a role. Furthermore, Todorov and Jordan (2002) theorized that due to the systematic discrepancy introduced between “expected and received feedback,” the internal model that generates motor commands undergoes changes, which may lead to adaptation in kinematic planning.

The goal of the present study was to investigate the role that control, dynamic, and kinematic variables play in movement planning. To do so, we designed a combined virtual and mechanical environment that allowed us to independently manipulate cost variables at a control, dynamic, and kinematic level. Subjects moved in a force field (FF) induced by a robotic manipulandum at different strengths while being visually perturbed. We used optimal control (see Optimization) to find optimal muscle activations for a detailed musculoskeletal model of the arm that included, among others, nonlinear activation dynamics accounting for the electromechanical delay and muscle activation level-dependent optimum lengths, nonlinear elastic tendon forces, and nonlinear force-length-velocity relationships of muscle fibers (Kistemaker et al. 2006, 2010). This model was used to predict arm kinematics that minimized various cost variables at the level of muscle activation (e.g., sum of squared muscle activation and min/max), at the level of dynamics (e.g., muscle torque change, muscle fatigue), and at the level of kinematics (jerk).

The experimental and model results of this study strongly suggest that neither control variables nor dynamical variables play an important role in kinematic planning of human arm movements. The paths taken by the subjects are only consistent with a cost function that comprises solely kinematic variables.

METHODS

Ethics statement.

All subjects reported no history of visual, neurological, or musculoskeletal disorder. Written informed consent was obtained from each subject prior to participation. All procedures were approved by the University of Western Ontario Research Ethics Board.

Experimental setup.

Subjects made movements while grasping the handle of an InMotion robotic manipulandum (Interactive Motion Technologies, Cambridge, MA; see Fig. 1A). Commanded forces to the robot were adjusted to compensate for position dependency of the robot arm's inertia to create an isotropic inertial characteristic with a mass of 1 kg. This was done using an inverse model of the inertial properties of the robot to calculate the force applied to the hand such that the total handle force that a subject experiences equals that of accelerating an object with a mass of 1 kg. The right arm was supported by a custom-made air sled, which expelled compressed air beneath the sled to minimize surface friction. The subject's arm and the manipulandum were beneath a semi-silvered mirror, which reflected images projected by a computer-controlled liquid crystal display (LCD) screen. Visual targets (diameter of 2 cm) were projected which appeared to lie in the same plane as the hand. Positional and force data were sampled at 600 Hz. All movements were made in the dark, and only reflected images of the LCD were visible to the subjects.

Fig. 1.

A: schematic drawing of the experimental setup. Targets were projected onto a semi-silvered mirror by using a liquid crystal display monitor suspended 15 cm above the mirror (not shown). Subjects moved from the red start position to the green target position. Two seconds after the target was reached, start and target position were swapped and subjects initiated a new movement. B: illustration of the visual perturbation using 2 hypothetical trajectories of the hand (black) and the perturbed hand trajectory (blue) with width w set to −30 mm. If a subject were to move straight to the target, the depicted movement would be an arc from the start position (green circle) to the target position (red circle; left). The visual distortion was 0 at the start and target position and was maximal halfway the 2 positions. To move visually straight, participants needed to make a counter arc to the right (right). C: schematic drawing of the musculoskeletal model. The implemented Hill-type muscle-tendon complex model consisted of a contractile element (CE), a series elastic element (SE), and a parallel elastic element (PE) and included activation dynamics modeling the CE length-dependent Ca²⁺ dynamics and active state-dependent force-length-velocity relationship. D: flowchart of the musculoskeletal modeling procedure. See text for definitions.

Force field.

The FF used in this study was similar to that used in Kistemaker et al. (2010). We had designed a new FF for several reasons. Most importantly, we wanted to make the FF such that there is a mechanical advantage (less muscle force/torque) to make movements curved markedly to the right, instead of the typical slightly leftward path taken in free space. In this study we used this FF to test variables that might be used by the brain to select a kinematic path in a novel mechanical environment. The force applied at the hand in the y-direction (fore-aft; F_y) was

$$F_y=b\left(-\dot{y}+\dot{x}\right)\cdot \mid y_{\text{target}}-y\mid .$$ (1)

In contrast with our previous study in which we used a value of either 0 or 150 N·s/m², we have set b to 0, 150, or 225 N·s/m². y_target, y, ẏ, and ẋ are, respectively, the y-position of the target, the y-position of the hand, the y-velocity of the hand, and the leftward/rightward (x) velocity of the hand. Note that the force in the x-direction (F_x) was always zero. For a detailed description of the FF, please see Kistemaker et al. (2010).

Visual perturbation.

Visual warp was similar to that used in Wolpert et al. (1995) and only warped the depicted hand position along the x-axis. The depicted x-direction of the hand was the actual x-position plus a function of the y-position that defined an arc from the start position to the target position with a maximal lateral distance halfway along the y-position between the start and target. The maximal lateral distance, w, was set to 0 (no warp), 20, or 30 mm to the right or −30 mm to the left (see Fig. 1B). Note that this warp results in no visual distortion of the hand position at both the start and target position. Therefore, no correction in movement path was required to adequately reach the target. To further explain the warp, we have plotted a sample trajectory of the hand in solid lines and the accompanying visually warped hand position in dashed lines for the case that the warp was set to −30 mm (Fig. 1B). This means that to make the hand go visually straight, subjects need to make a counter arc of 30 mm to the right.

To implement the visual perturbation, we calculated the perturbed visual hand position in the x-direction (c_x) as a function of the actual hand position in the x-direction (x_h) and the (unperturbed) hand position in the y-direction (c_y ≡ y_h). First, radius R of the circle was calculated, leading to an apex halfway along the movement with the desired width w and movement distance D:

$$R=\frac{{\left(0.5\cdot D\right)}^2-w^2}{2w}+w.$$

c_x was calculated as follows:

$$c_x=R\cdot \cos \left[\arcsin \left(\frac{y_h}{R}\right)\right]+R-w+x.$$ (2)

In a pilot study we investigated the maximal level of warp (gradually introduced) that was not detected by any of the participants. Several subjects reported that “something was wrong” for visual perturbations of 50 mm and greater, and data analysis showed that participants ignored visual feedback altogether. Only for distortions as small as 30 mm did none of the subjects become aware of the visual manipulation. Furthermore, after the experiments, subjects were asked if they had noticed the visual perturbation. All subjects responded in the negative and were surprised that the depicted and real hand positions did not coincide. None of the subjects in this pilot study participated in the experiment.

Experimental protocol.

To ensure that the FF and associated motor costs were salient, we used FF strength levels and desired movement times that resulted in considerable forces applied to the hand. To have a uniform group of participants that experienced similar relative forces applied to their hand, we restricted our selection criterion to right-handed males. Forty-nine participants performed point-to-point movements to visual targets while grasping the handle of the robotic linkage with their right hand. Movements (30 cm) were made toward and away from the body in a horizontal plane along the surface of a desk, at shoulder height (see Fig. 1A). The subjects' view of their arm was occluded by the semi-silvered mirror. Visual targets and a small dot representing the position of the hand were displayed on the mirror using an LCD monitor. When the target circle was reached, the target changed color to provide feedback indicating that the movement was either well timed (between 300 and 500 ms), too slow, or too fast. To avoid biasing subjects to move along a particular hand path, apart from the timing aspect, no instructions were given as to how the target was to be reached. After 2 s, start and target position were swapped, and subjects initiated a new movement toward the original start position. In contrast with our previous study (Kistemaker et al. 2010), we provided continuous feedback of hand position by plotting a small white circle (diameter 1 cm) that matched the position of the robot handle (or was warped; see above).

The subjects were randomly assigned to seven different experimental groups on the basis of the strength of the FF (0, 150, or 225 N·s/m²) and level of warp (−30, 0, 20, or 30 mm). The different experimental groups are shown in Table 1.

Table 1.

Overview of experimental groups

	FF0w2	FF1w0	FF1w2	FF0w3	FF2w0	FF2w3	FF2w-3
b, N·s/m2	0	150	150	0	225	225	225
w, mm	20	0	20	30	0	30	−30

Experimental groups based on strength of the forcefield (FF; b = 0, 150, or 225 N·s/m², corresponding to FF0, FF1, or FF2, respectively) and level of warp (w = 0, 20, 30, or −30 mm; corresponding to w0, w2, w3, or w-3, respectively).

All experimental groups started first with block 1 in which subjects made 100 movements in free space (i.e., force field turned off, FF0) and no visual warp (w0). After that, in block 2 either 100 or 150 movements were made while the perturbations (both visual warp and FF) were gradually introduced according to experimental group. These differences in number of movements in the second block were such that the subjects had an equal gradual increase in visual warp and forces applied to the hand across the different experimental groups. Subjects that either had a FF strength of b = 150 N·s/m² and/or had a visual warp level of −20 mm had 100 movements in block 2. Subjects that either had a FF strength of b = 225 N·s/m² and/or had a visual warp level of ± 30 mm had 150 movements in block 2. In blocks 3 and 4, subjects always made 100 movements in the condition according to their experimental group. In block 5, all subjects made 100 movements in FF0 and with normal visual feedback of their hand position.

Data analysis.

Positional and force data were sampled at 600 Hz and then digitally filtered using a fourth-order bidirectional Butterworth filter with a cutoff frequency of 15 Hz. Successful trials were selected on the basis of the following conditions: 1) at movement onset, hand position was within the start circle (diameter of 2 cm), 2) initial velocity was lower than 0.01 m/s [in some trials subjects anticipated the appearance of the target circle (“go-cue”) before it actually appeared, leading to a movement start before the robot motors were turned on], and 3) the time between leaving the start circle and entering the target circle was between 300 and 500 ms. All successful trials per subject per block were analyzed, and movements were omitted for which the absolute maximum lateral deviation was greater than the mean plus three times the standard deviation for that subject in that block. Kinematics were analyzed on basis of the last 10 successful inward and the last 10 successful outward movements in block 1 (baseline) and those of block 4.

Musculoskeletal model.

The musculoskeletal model of the arm consisted of three rigid segments interconnected by two hinge joints representing the glenohumeral joint and elbow joint and has been described in full detail elsewhere (Kistemaker et al. 2006, 2010; see Fig. 1C). In short, the arm was actuated by six Hill-type muscle units (2 monoarticular shoulder and elbow muscles and 2 biarticular muscles; see Fig. 2). The implemented Hill-type muscle model consisted of a contractile element (CE), a series elastic element (SE), and a parallel elastic element (PE). Figure 1D shows schematically the musculoskeletal modeling procedure. Activation dynamics describe the nonlinear dynamic relationship between muscle stimulation (STIM, the only independent input to the musculoskeletal system) and active state (q, the relative amount of Ca²⁺ bound to troponin C; see Kistemaker et al. 2005). Activation dynamics were furthermore dependent on the length of the CE to account for the experimentally observed shifts in optimum CE length as a function of the activation of a muscle (Kistemaker et al. 2005). Contraction dynamics relates q to the muscle torques (M) using force-length-velocity relationship and joint angles. The lengths of the muscle-tendon complexes and moment arms were functions of the joint angles. All skeletal and muscle parameters (such as maximal force, maximal contraction velocity, tendon stiffness, etc.) were based on human cadaver studies and in vivo and in vitro (human and animal) experimental data (see Kistemaker et al. 2005, 2006, 2007). Except for one parameter introducing discontinuity of the force-velocity relationship (see below), none of the parameters were changed or attuned to this study. Please see the Appendix of Kistemaker et al. (2006) for a sensitivity analysis of the used muscle parameters. The robot forces were modeled as a force applied to the hand using Eq. 1.

Fig. 2.

Overview of model predictions of kinematic paths under the various cost functions and experimental data. Please note the equal scaling of the x- and y-axes. Model predictions shown are the outward optimal movements for all cost functions investigated at control, dynamic, and kinematic levels. Note that the predicted kinematic paths for minimizing jerk are identical for all experimental conditions. Note also that the predicted kinematic paths do not change for different force field (FF) strengths, both for jerk and weighted jerk. Thus, for example, the red and gray lines overlap completely. The predicted paths are depicted for the conditions indicated. Experimental data furthermore show the average hand paths of outward and inward movements of the last 10 trials per condition. FF0, FF1, and FF2 refer to the strength of the FF (b = 0, 150, or 225 N·s/m², respectively) and w0, w2, w3, and w-3 to the visual perturbation (w = 0, 2, 3, and −3 cm, respectively). Note that the red lines almost completely overlay the gray lines.

The musculoskeletal model was reformulated to ensure that all functions were continuous up to the first derivatives and useable for direct collocation (part of the optimization procedure; see Optimization). This included the change in a single parameter that defined the difference in concentric and eccentric slope of the force-velocity relationship at zero CE contraction velocity. This was originally set such that the slope of the eccentric part is twice that of the concentric part (see also van Soest and Bobbert 1993). Such discontinuity in the derivative of the force-velocity relationship was found to be problematic for the optimal control solver and was set such that the slopes were equal. We carried out forward simulations with both slope factors for the movements investigated in this study, and they showed no notable differences. Also, we have reformulated the force-length relationship of the tendon and parallel elastic components (without changing their behavior) such that it is continuously differentiable. This was done by multiplying the force-length relationship of these elastic components by a sigmoid function. In the original formulation, tendon force (F_SE) was modeled as a quadratic spring (see e.g., Kistemaker et al. 2006):

$$F_{SE}=k_{SE}\cdot \max {\left[0,\left(l_{SE}-l_{SE\_0}\right)\right]}^2,$$

where l_SE is the tendon length and l_{SE_0} is the tendon slack length. In the original formulation, the term max[0,x] ensures that the force of the tendon is always zero if the length of the tendon is less than the slack length (i.e., the tendon is not allowed to push), yet is not continuously differentiable. To address this, we calculated F_SE by

$$F_{SE}=\frac{1}{1+e^{-\beta \left(l_{SE}-l_{SE\_0}\right)}}\cdot k_{SE}\cdot {\left(l_{SE}-l_{SE\_0}\right)}^2.$$

The sigmoid function goes from 0 to 1, is 0.5 at l_{SE_0}, and has an arbitrary steep slope, set by β. In words, the sigmoidal function is zero when l_SE is under its slack length and one when l_SE is above its slack length (apart from a small region around slack length). We have similarly changed the formulation for the parallel elastic component.

Cost functions: control level.

The most influential cost variable at the level of control is the sum of the squared muscle stimulations, also known as control effort (Diedrichsen et al. 2010; Fagg et al. 2002; van Bolhuis and Gielen 1999). Control effort was calculated by

$$J={\displaystyle \sum _{n=1}^6{\displaystyle \underset{0}{\overset{T}{\int }}STI{M}_n{\left(t\right)}^{2}dt.}}$$

The cost function is denoted as J, T is the movement duration, STIM is the muscle activation, and n denotes each muscle.

It has been suggested in the literature that fatigue is related to the total amount of muscle fibers activated (Crowninshield and Brand 1981) and was implemented by weighting the control effort cost by their relative maximal force (maximal force F_max is assumed here to have a fixed relationship with physiological cross-sectional area).

$$J={\displaystyle \underset{}{\overset{6}{\sum _{n=1}^6\frac{F_{\max ,n}}{\underset{}{\overset{}{\sum _{n=1}^6}}{F}_{\max ,n}}}}{\displaystyle \underset{0}{\overset{T}{\int }}STI{M}_n{\left(t\right)}^{2}dt.}}$$

In the remainder of this article we refer to this cost variable as control fatigue.

The last control variable investigated is called the MinMax and minimizes the maximal muscle activation (Rasmussen et al. 2001). For numerical reasons, an approximation for MinMax was used (see Ackermann and van den Bogert 2010):

$$J={\displaystyle \sum _{n=1}^{6}k{\displaystyle \underset{0}{\overset{T}{\int }}STI{M}_n{\left(t\right)}^{10}dt.}}$$

Because STIM is the relative muscle stimulation frequency (between 0 and 1), setting the exponent to 10 causes the cost for MinMax to be extremely low (<10⁻¹⁰). Rather than lowering the threshold of optimality conditions on the cost function, we used an arbitrary large number, k, to rescale the cost. For k > 1 × 10⁵, no differences in solutions were found.

Cost functions: dynamic level.

At the dynamic level, three cost variables were used. First, we implemented muscle torque change (Uno et al. 1989) that minimizes the sum of squared individual muscle torques (M) differentiated with respect to time:

$$J={{\displaystyle \sum _{n=1}^6{\displaystyle \underset{0}{\overset{T}{\int }}\left[\frac{d{M}_n\left(t\right)}{dt}\right]}}}^{2}dt.$$

Second, muscle effort (or load sharing) typically assumes that the sum of squared forces produced by the individual muscles is minimal (e.g., An et al. 1984),

$$J={\displaystyle \sum _{n=1}^6{\displaystyle \underset{0}{\overset{T}{\int }}{F}_{CE,n}{\left(t\right)}^2}}dt.$$

or, alternatively, they can be scaled for the maximal isometric force (F_max) of the individual muscles (F_CE) to minimize muscle fatigue (e.g., Crowninshield and Brand 1981):

$$J={\displaystyle \sum _{n=1}^6\frac{F_{\max ,n}}{{\displaystyle \sum _{n=1}^6{F}_{\max ,n}}}{\displaystyle \underset{0}{\overset{T}{\int }}{F}_{CE,n}{\left(t\right)}^2}}dt.$$

Cost functions: kinematic level.

At the level of kinematic cost variables, there is only one cost variable referred to in the literature, termed jerk (Hogan 1984),

$$J={\displaystyle \sum _{n=1}^2{\displaystyle \underset{0}{\overset{T}{\int }}{\stackrel{⃛}{\phi }}_n^{2}dt,}}$$

in which ${\stackrel{⃛}{\phi }}_n$ refers to joint space jerk, the third time derivative of the shoulder and elbow joint angle.

As mentioned above, in this study we used visual perturbations that influenced the curvature of the perceived kinematics. Based on the idea that the brain may integrate visual and somatosensory information to estimate hand position (e.g., Smeets et al. 2006; van Beers et al. 1999), and hence jerk, we also investigated a cost function of weighted jerk:

$$J=\alpha {\displaystyle \sum _{n=1}^2{\displaystyle {\displaystyle \underset{0}{\overset{T}{\int }}{\stackrel{⃛}{\phi }}_n^{2}dt}+\left(1-\alpha \right){\displaystyle \sum _{k=x,y}{\displaystyle \underset{0}{\overset{T}{\int }}{\stackrel{⃛}{c}}_k^{2}dt,}}}}$$

where α is the relative weight factor and ${\stackrel{⃛}{c}}_x$ and ${\stackrel{⃛}{c}}_y$ denote the visual Cartesian hand jerk. ${\stackrel{⃛}{c}}_x$ is calculated by (see also Eq. 2)

$${\stackrel{⃛}{c}}_x=\frac{d^3}{d{t}^3}\left(-R\cdot \cos \left[\arcsin \left(\frac{c_y}{R}\right)\right]+x_h\right).$$

Note that c_y is unaffected by the visual perturbation, and, as stated before, x_h is the actual hand position in the x-direction. Note also that the two terms in the cost function have different dimensions and are nonlinearly related, and thus the exact value of the relative weight is noninformative. As a first-order approximation of equal weighting of the visual and somatosensory information, we first identified the α for which in the unperturbed condition the two costs were equal (α = 0.11). The optimal path for (unperturbed) visual jerk of the hand is a straight line from the start to the end, and as such the optimal path for this value of α is about half of the curve that is typical for minimizing joint-based jerk. This value was used to find the optimal paths in the experimental conditions.

There is, however, no a priori need for equal contributions. For example, one could use Bayesian optimal cue combination and allow for different visual and somatosensory contributions. This may improve the quality of the predictions. Such a weighting requires knowledge about how the brain may combine two signals with different coordinate frames and about the relative variance of the signals (van Beers et al. 1999, 2002). Experimentally estimating the relative variance goes beyond the scope of this article, and we have chosen to find the optimal weighting through numerical optimization. A simple golden section search method (fminbnd.m) implemented in MATLAB (The MathWorks) was used to find the optimal value for α (0.128) that minimized the root mean squared difference between the experimentally observed maximal lateral x-position of the hand and that of the optimal path of the model (see also results, Model predictions vs. experimental data) found using the optimal control techniques described below.

Optimization.

For each cost function, optimal activations were found for the six muscle-tendon units of the 2-degrees-of-freedom musculoskeletal model for a fast arm movement, similar to the experimental task. In the experiments, successful trials were those with a “movement time” of 400 ± 100 ms, defined as the time between leaving the start circle and entering the target circle. Therefore, the actual movement time was a bit longer, and we have heuristically set the constraint movement time of the model to 425 ms. Boundary constraints matched the experiments and were the start positions of the shoulder and elbow joints (35° and 172°) and desired end positions (64° and 127°), with zero angular velocity and acceleration. Importantly, initial CE length and tendon length were set such that the tendons are at their slack length (the optimal control solver is otherwise “intelligent” enough to choose very short CE lengths at movement onset, therefore “spring-loading” the tendons that release energy). All other initial states were set to zero. The dynamic equations of the musculoskeletal model were translated into dynamic constraint functions and discretized on several temporal nodes termed collocation points (direct collocation method). Important in this process was the use of an implicit formulation of musculoskeletal dynamics (see van den Bogert et al. 2011). To identify the minimal required collocation points leading to accurate solutions of the musculoskeletal model, we carried out forward simulations of the model using a variable step-size ordinary differential equation (ODE) solver embedded in MATLAB, with absolute and relative tolerance set to 1e−8, and using the optimal control STIM patterns obtained using different numbers of collocation points. The dynamic constraints, together with the task and boundary constraints and the cost function, were solved simultaneously using a sparse nonlinear optimal controller (SNOPT; TOMLAB Optimization, Pullman, WA). The derivatives and second derivatives of the constraint and cost function were computed analytically using PROPT (TOMLAB Optimization). It was found that the states obtained from a forward simulation and those obtained from SNOPT were nearly identical for 45 collocation points (on average about 1 collocation point per 10 ms).

To reduce computation time, we ran optimizations for each cost function with increasing numbers of collocation points starting from 15 to the desired amount of 45. For each cost function, the optimizer was run several times using different initial guesses. First, we used an initial guess in which all states and inputs were set to 0. In a second set of optimization runs we used the optimal outcome of a particular cost function and used those as an initial guess for all other cost functions. Reassuringly, optimal solutions were found to be independent of initial guesses tested. Two exceptions were found. First, minimizing jerk led to different optimal STIM patterns, yet with similar kinematics. This result is, however, to be expected because minimizing jerk only solves the kinematic redundancy and does not lead to a unique STIM solution (i.e., there is an infinite number of solutions for STIM yielding identical kinematics). Second, the MinMax criterion showed local minima depending on the initial guess. Investigating the costs of several optimization runs showed that the costs were very similar, indicating a flat cost landscape. This can be understood since this criterion is not sensitive to changes in the activation of muscles that have a low activation. The optimal kinematic paths were, however, robust against initial guesses, and therefore we simply selected per condition the optimal solution that had the lowest cost and discarded the rest as local minima.

RESULTS

Model predictions.

We first examined the predictions with the optimal control arm model for the different cost functions for the different levels of FF. The FF was constructed such that participants required less force to reach the target using a rightward curved movement, instead of the leftward curved movements observed experimentally in free space (see Kistemaker et al. 2010 for a detailed explaination of the force field). The FF was either turned off (FF0), set to a medium level (FF1; similar to that in Kistemaker et al. 2010), or set to a high level (FF2). To gain more insight in the relationship between predicted paths and the FF used, we also have included two intermediate FF strengths (0.33 and 0.66, corresponding to b = 50 and 100 N·s/m²). Inward and outward movements were very similar, and we only show here the optimal kinematic paths for the outward movements.

Figure 2 shows an overview of the predicted movements for the various cost functions at the control, dynamic, and kinematic levels. A first interesting result is that the predicted movements in the FF0 conditions were very similar to each other: a slightly leftward curved hand path for right arm movements (gray lines). This finding is in agreement with the literature showing that several cost variables adequately predict the common characteristics of human arm movements in free space (e.g., Alexander 1997; Hogan 1984; Kistemaker et al. 2010; Uno et al. 1989), even though they are very different in nature. However, importantly and as argued in the Introduction, these results indicate that it is experimentally very difficult to distinguish the different cost functions from each other for movements made in free space (see also discussion).

The predicted optimal movements in the FFs were different. Kinematics obtained by minimization of cost variables at the control level and the dynamic level were in general greatly influenced by the FF and were markedly different for the different levels of FF strength (dashed lines refer to intermediate levels of FF strengths, solid lines were identical to those used experimentally). This is because the FF was designed such that the robot opposes the movement less for movements curved rightward. This can readily be seen in the optimal solutions for control variables that include muscle force. Also, in general less muscle activation is required for smaller forces, and hence control effort cost is smaller when a rightward movement is made. These effects are stronger with increasing FF strength: the rightward bend becomes increasingly greater with the FF strength. Only for minimizing muscle torque change were changes in optimal paths less pronounced. This is discussed separately (see discussion).

In contrast, minimum jerk hand paths are not at all influenced by the FF (gray and red lines in Fig. 2 overlap completely). This can be readily understood, because minimization of jerk is by definition purely minimizing a kinematic variable, and so the cost function does not depend on how much muscle activation or muscle force is required. Note, however, that even though kinematic paths are identical, the optimal STIM pattern changed substantially for the different strengths of the FF to compensate for the forces applied to the hand.

Figure 2 also shows the kinematic paths predicted by the cost function combining visual (perturbed) hand jerk and somatosensory (unperturbed) joint jerk. Optimizations were run for all experimental conditions. The movements for no visual warp were similar to that of minimizing joint jerk alone, yet were straighter. This too can be readily understood, because hand-based jerk, in the absence of a visual perturbation, would predict zero jerk in x-direction and thus a straight line from start to target. The diminished curvature is thus due to the added cost on visual (Cartesian) jerk. The optimal relative weighting factor α (see methods) and the value for approximately equal relative weighting of visual and somatosensory jerk were very close (0.13 vs. 0.11, respectively) and showed very similar results. In the remainder of this article we show only the results for α set to 0.13. As expected, the identified optimal hand paths were not dependent on the presence or strength of the FF. The optimal movements minimizing weighted jerk were affected by the presence of the visual perturbation: movements were such that they countered the perturbation.

Experimental data.

Figure 2 also shows the average of the last 10 trials in FF1 and FF2, with and without visual perturbation, for both the outward and inward movements. For reference, also plotted are the averages of the last 10 baseline movements in the first block of movements with FF0w0 (recall that all subjects started with this in block 1). One subject (in the FF2w−3 group) was removed from the data set because he was not able to meet the minimal amount of 20 successful movements per block (even in the easiest block FF0w0, for which the group average success rate was ∼85%).

By visual inspection of experimental data in Fig. 2 it can be appreciated that subjects changed their kinematics when visually perturbed to counteract the perturbation; the FF0w2 movements (blue lines) are more to the left than the baseline movements (FF0w0, gray lines; note that the red lines almost entirely overlay the gray lines). This effect was larger for a greater visual perturbation (c.f. FF0w2 and FF0w3 in left and right experimental data, respectively). Neither the presence nor the magnitude of the FF influenced the kinematic path taken by the subjects. As mentioned before, the FF1w0 and FF2w0 (red lines) are very similar to the baseline movements (gray lines). Only in combination with the visual perturbation did the subjects in the FF change their kinematics to counter the visual perturbation. Subjects in the FF visually perturbed to the right moved more to the left (FF1w2 and FF2w3, black lines) and when visually perturbed to the left moved more to the right (FF2w−3, green lines).

The effects appear rather subtle, but as it turns out they are very consistent. The effects were in general small due to the small visual perturbation used and especially relative to the distance travelled in the y-direction. To look into more detail, we have computed the change in maximal x-deviation of the kinematic paths taken by the subjects (from now on referred to as Δx-dev). This was done by calculating per subject the maximal x-deviation subtracted by the average value for the last 10 × 2 movements in their baseline measurement (block 1). Figure 3A shows the Δx-dev of successful trials during blocks 3 and 4 binned in 10 × 2 (inward and outward) successful trials. The successful trials were binned starting from the last 10 × 2 successful trials in block 4 (bin 20) to the first successful trials in block 3. Bins 1 and 2 were omitted due to a lack of successful trials (indicating an average hit rate across blocks 3 and 4 of around 80%). Data points are the mean values across subjects, and the error bars indicate the standard error of the mean. Figure 3 shows that changes in x-deviation occur early in block 3 and remain rather constant until the end of block 4.

Fig. 3.

A: average changes in maximal x-deviation (Δx-dev) of the hand trajectories relative to baseline (last 10 successful inward and 10 outward movements in block 1) during blocks 3 and 4 for all 7 experimental conditions. Movements were binned in 20 successful (10 inward and 10 outward) movements, starting from the last 20 successful trials. The first 2 bins were omitted due to a lack of successful movements. Data points are the mean values, and error bars indicate standard error of the mean. B: close-up of Δx-dev in the last 10 inward and outward movements of block 4. Note that the error bars denote the 95% confidence interval. *P ≤ 0.05; **P ≤ 0.01; ***P ≤ 0.001 denote means significantly different from zero. The arrows represent the predicted changes in maximal x-deviation for the cost function using weighted visual and somatosensory jerk.

To test for reliable differences in kinematics among the groups, we performed a split plot (mixed design) ANOVA on the last 10 × 2 successful movements in blocks 1 and 4 (see Fig. 3B). The ANOVA indicated a significant interaction [F(6,41) = 14.8, P < 0.0001]. Post hoc paired t-tests showed that subjects moved on average significantly 4.0 ± 3.3 mm (P = 0.019) and 6.8 ± 6.5 mm (P = 0.032) more to the left when visually perturbed to the right by 20 and 30 mm, respectively (see Fig. 3B; note that these error bars indicate 95% confidence intervals). The subjects did not significantly change their maximal x-deviation when moving in the FF (P = 0.86 for FF1w0 and P = 0.23 for FF2w0). When moving in the FF while being visually perturbed, subjects changed their kinematics independently of the FF, but only to counter the visual perturbation. They moved significantly more to the left when visually perturbed to the right (−6.7 ± 2.7 mm, P < 0.001 for FF1w2 and −9.5 ± 3.1 mm, P < 0.001 for FF2w3) and moved significantly more to the right when visually perturbed to left (9.0 ± 4.8 mm, P < 0.01). Thus, although curvature was entirely independent of the FF level, subjects changed their kinematics significantly to counteract the visual perturbation. However, subjects did not fully compensate for the distorting effect of the warped visual feedback, on average, 28%. These results are in agreement with a previous study using a similar visual distortion (Wolpert et al. 1995). Results are summarized in Table 2.

Table 2.

Overview of statistical results for change in maximal x-deviation

	Δx-dev, mm	SD, mm	P Value
FF0w2	−4.0	3.3	0.019
FF1w0	−0.2	3.4	0.856
FF1w2	−6.7	2.7	0.001
FF0w3	−6.8	6.5	0.032
FF2w0	−1.5	2.9	0.230
FF2w3	−9.5	3.1	0.000
FF2w-3	9.0	4.8	0.006

Values are mean and SD of change in maximal x-deviation of the kinematic paths taken by the subjects (Δx-dev).

Model predictions vs. experimental data.

Table 3 shows a comparison between the experimentally observed Δx-dev and those predicted for all cost functions. The optimal kinematics based on cost variables from the control and dynamic levels did not match those observed experimentally; all optimal paths showed rightward curved movements that increased with the strength of FF set, whereas none of the kinematic paths of the subjects did. Only the paths predicted by the kinematic variables (jerk and weighted jerk) were in agreement with those observed experimentally.

Table 3.

Comparison of predicted and experimentally observed change in maximal x-deviation

		Control Level			Dynamic Level			Kinematic Level
	Experimental Data	CE	CF	MM	TC	ME	MF	J	WJ
FF0w2	−4.0	0	0	0	0	0	0	0	−5.5
FF1w0	−0.2	35.4	95.4	51.7	7.7	98.0	111.4	0	0
FF1w2	−6.7	35.4	95.4	51.7	7.7	98.0	111.4	0	−5.5
FF0w3	−6.8	0	0	0	0	0	0	0	−8.2
FF2w0	−1.5	47.4	122.0	71.7	22.1	124.4	134.1	0	0
FF2w3	−9.5	47.4	122.0	71.7	22.1	124.4	134.1	0	−8.2
FF2w-3	9.0	47.4	122.0	71.7	22.1	124.4	134.1	0	7.6

Values are Δx-dev in mm for all cost functions: CE, control effort; CF, control fatique; MM, MinMax; TC, torque change; ME, muscle effort; MF, muscle fatique; J, jerk; and WJ, weighted jerk.

The subjects readily changed their kinematics when visually perturbed. Irrespective of FF magnitude, they moved to counteract visually perturbed movement curvature. These changes in kinematics cannot be explained by minimizing control or dynamics cost variables. When visually perturbed to the right, subjects changed their hand paths to the left, and therewith increasing the cost at the control and dynamic levels. Furthermore, the visual perturbation was such that it was zero at both the start and the target position, and thus muscle activation patterns do not need to be adjusted to adequately reach the target when visually perturbed (see also Wolpert et al. 1995). Also, the cost for the variables at the control level and dynamic level are not influenced by the (perceived) changes in kinematics. On the other hand, a cost function that combined visually perceived jerk and somatosensory perceived jerk was capable of adequately predicting the kinematic hand paths taken in all experimental conditions. The arrows in Fig. 3B show the predicted changes in maximal x-deviation predicted using weighted jerk (α = 0.128; see methods).

After showing that a cost function based on weighted jerk is capable of adequately predicting the spatial aspects of the kinematic paths, we investigated the temporal aspects by comparing the velocity profiles of the model and the subjects. In Fig. 4, the grand mean of the y-velocity profiles of the hand of all subjects in the last 10 movements in all experimental conditions is plotted, together with the standard deviation. The subject averages per experimental condition were aligned on instant of peak velocity to calculate the grand mean. Shown in black, the velocity profile is plotted for the optimal path for the weighted visual and somatosensory jerk. The optimal y-velocity profiles for the different experimental conditions did not change notably. The experimentally observed maximal velocity was 1.37 ± 0.13 m/s, and that for the model was 1.38 ± 0.00 m/s. The experimentally observed time to peak velocity (time between 1% of peak velocity and reaching peak velocity) was 0.189 ± 0.018 s, and that for the model was 0.199 ± 0.003 s. Together with Fig. 3B, these results show the excellent fit between experimental data and the optimal kinematic paths for weighted jerk in all conditions tested experimentally.

Fig. 4.

Average velocity profile of the hand in the y-direction (fore-aft) of all subjects of the last 10 movements in all experimental conditions (gray line; the gray shading indicates standard deviation). The optimal velocity profile predicted for the minimization of combined visual and somatosensory perceived jerk is plotted in black.

DISCUSSION

In this study we tested a number of hypotheses about how the brain controls voluntary arm movement. Specifically, we investigated to what extent control, dynamic, and kinematic variables play a role in movement path selection. To do so, we had seven groups of subjects move in a novel FF at different strength levels while being visually perturbed. The visual perturbation changed the visual curvature of their hand paths. We used a detailed nonlinear musculoskeletal model of the human arm in combination with direct collocation and a sparse nonlinear optimal control solver to predict optimal movement paths using various cost variables. The first important finding was that, in line with the literature, all cost variables investigated (be it at the control, dynamic, or kinematic level), were capable of adequately predicting the kinematic paths taken by subjects in free space, at least for the two-dimensional planar tasks studied here. These results clearly show that it is problematic to discern different cost functions on the basis of movements performed in free space. The predicted movements in the FFs did show a clear change: optimal movement paths based on all control and dynamic variables were different depending on the strength of the FF (see Fig. 2 and Table 3). In general, the stronger the FF, the more movements curved rightward. The predicted kinematics based on the kinematic cost variable (jerk; Hogan 1984) were not affected by the FFs. The experimental results showed that none of the subjects' kinematics were influenced by the FF; subjects moved as they did in free space. However, when feedback was perturbed such to change the visual curvature of the hand, subjects moved more to the left when visually perturbed to the right, and vice versa. They also did so irrespective of FF magnitude. The experimental results were found to be consistent only with a cost function that solely comprises cost variables on the kinematic level (see Fig. 3 and Table 3).

It is unlikely that the lack of adaptation of the subjects to the FF is caused by flaws in the experimental setup and/or protocol. The strength of the FFs was set such that it is implausible that the change in control and dynamic costs was below the participants' threshold for detection. The two levels of the FFs led to an average maximal force in the y-direction of about 35 and 46 N, respectively, and subjects also reported getting tired from making movements in the FF. Furthermore, if the different paths in the FF lead only to changes in cost below their threshold, they would do so when subjects were moving in free space and hence would not play a substantial role. It is unlikely that providing visual feedback of the hand played a role in not adjusting the kinematics of the subjects. The FF1w0 condition in this study was, apart from the provided visual feedback, identical to that in a previous study that showed similar results in the FF (Kistemaker et al. 2010). It is also unlikely that subjects did not converge to stable behavior due to lack of training. The subjects were gradually introduced to the perturbations of their condition over 100–150 movements (block 3; see methods) and then twice made 100 movements with the full perturbation (blocks 3 and 4), of which only the last 10 inward and last 10 outward movements of block 4 were used for statistical data analyses. Moreover, as can be appreciated by studying Fig. 3A, the changes in x-deviation did not change much from early exposure to the perturbations in block 3 to that at the end of block 4, providing further evidence that subjects' behavior is not due to a lack of training. Also, in a previous study with a similar FF (yet without visual feedback and visual perturbations), movements did not change over hours of training (Kistemaker et al. 2010). Furthermore, the subjects had a high percentage of successful movements in block 4 (∼80 vs. ∼85% in the FF0w0 condition). Last, the standard errors of the mean of maximal x-deviation (i.e., the error bars in Fig. 3A) indicated no difference in difficulty when another path was taken than they did in FF0. One might argue that control difficulties in the FF impede the motor system's ability to adapt movements. The presence of visual perturbation data dispels this concern. When visually perturbed to the left during FF learning, subjects readily changed their hand paths to the right, resulting in lower cost at the control and dynamic levels. When visually perturbed to the right, subjects readily moved more to the left and therewith increased the control and dynamic costs. However, subjects did not change their path in the FF when not visually perturbed. Last, the presence of local minima could have hampered subjects to adapt. However, the optimal trajectories for several intermediate strength levels of the FF show a gradual increase in curvature for the control and dynamic variables and as such do not indicate local minima. On the contrary, this result indicates a continuous negative gradient in cost that, from a pure minimization perspective, would simplify finding the optimal paths.

Our data clearly suggest that neither control cost variables nor dynamic variables play an important role in kinematic path selection. On the basis of these results, one may argue that a possible exception would be muscle torque change: the kinematics predicted were much less affected by the FF than the other dynamic variables. The effect of the FF is less pronounced for the movements minimizing this cost, because it does not penalize the magnitude of the muscle torques, but rather the rate of change of the produced muscle torque. Be that as it may, muscle torque change is unlikely to play an important role in kinematic path selection. First, the optimal solutions of the control variables and dynamic variables, including torque change, are not affected by the visual perturbation (see also Wolpert et al. 1995). This is because 1) at the start and target position, visual warp was zero, and as such the control signals do not need to be adjusted to adequately arrive at the target, and 2) control cost and dynamic costs do not change on the basis of (perceived) kinematics. However, subjects showed clear and significant changes in kinematics while being visually perturbed. Second, although the predicted effects of the FFs on torque change may be small compared with those obtained when other cost variables are minimized, they are actually very large compared with the experimental data: the predicted changes in x-deviation are about 8 mm to the right for FF1 and over 22 mm for FF2 vs. −0.2 and −1.5 mm to the left (not significantly different from 0, P > 0.8 and P > 0.2; see Fig. 3B and Table 3) observed experimentally. Clearly, these predictions based on minimization of muscle torque change are not supported by the experimental data.

The behavioral results of this study are consistent with minimization of only kinematic cost variables. Yet, if path selection does not depend on control and/or dynamic variables, why do subjects change their kinematics when visually perturbed? There are two major sources of information informing the brain about movement kinematics: vision and proprioception. It is likely that the two sources of information are combined to generate an estimate of the actual limb kinematics, for example, through Bayesian maximal likelihood estimation (Kording and Wolpert 2004). In the current study, we have perturbed the visual information and hence may indeed have caused a change in perceived kinematics that is partway between the visually and somatosensory sensed limb kinematics. This explanation is in line with our experimental results. Subjects clearly and significantly responded to the visual perturbation of perceived hand curvature yet did not fully compensate: on average, about 28% (which is in agreement with Wolpert et al. 1995). This was independent of the direction of the visual perturbation and the presence of the FF. To test the idea of cue combinations, we implemented a cost function that combined visually perceived (perturbed) and somatosensory perceived (actual) kinematics. This cost function was not only capable of qualitatively describing the spatial hand paths taken by the subjects in all experimental conditions but also adequately predicted the temporal aspects such as maximal velocity and time to peak velocity.

The current study provides evidence that selection of movement kinematics is based purely on kinematic variables. In this study we have used only one kinematic variable: jerk, the third time derivative of (angular) position (Hogan 1984). Here we note that our results would likely be consistent with any other kinematic variable that adequately predicts movements in free space. Furthermore, even though we have used optimal control to find the optimal paths for weighted jerk and have shown that they were in agreement with the experimental data, this does not mean that the human brain necessarily uses optimal control; obviously, any process based on kinematic variables that would yield the same kinematic planning would be consistent with our finding.

A consequence of explicit movement selection is that the brain needs to generate the required muscle activation in a separate step. Such a hierarchical view of motor control has been suggested in work starting from the late 1900s (John Hughlings-Jackson; see York and Steinberg 2006) to more recent studies (e.g., Hollerbach 1982; Kawato et al. 1987; Rosenbaum 1983; Saltzman 1979) and is consistent with a body of neurophysiological data (e.g., Dum and Strick 2002; Rao et al. 1993).

We stress here that if control and dynamic costs do not play a role in movement planning, this does not mean that movements in free space are performed inefficiently. In fact, using a detailed optimally controlled musculoskeletal model, we have shown that the minimal jerk trajectory is very similar to those obtained for several control and dynamic variables, such as effort, fatigue, and energy in free space. Furthermore, muscle activation patterns leading to the planed movement trajectory may in fact be selected for on the basis of effort, fatigue, or energy. Such a view is consistent with recent behavioral data and oxygen consumption measured during FF learning. Subjects rapidly learned how to move in a FF as they do in free space, while oxygen consumption decreased over a longer time scale (Huang et al. 2012).

GRANTS

D. A. Kistemaker is supported by European Commission Seventh Framework Programme CORDIS FP7-PEOPLE-2011-CIG-303849. J. D. Wong is supported by Canadian Institutes of Health Research (CIHR). P. L. Gribble is supported by CIHR, National Sciences and Engineering Research Council (Canada), and the National Institutes of Health (USA).

DISCLOSURES

No conflicts of interest, financial or otherwise, are declared by the authors.

AUTHOR CONTRIBUTIONS

D.A.K., J.D.W., and P.L.G. conception and design of research; D.A.K. performed experiments; D.A.K. analyzed data; D.A.K., J.D.W., and P.L.G. interpreted results of experiments; D.A.K. prepared figures; D.A.K. drafted manuscript; D.A.K., J.D.W., and P.L.G. edited and revised manuscript; D.A.K., J.D.W., and P.L.G. approved final version of manuscript.