Learning of Action Through Adaptive Combination of Motor Primitives

Abstract

Understanding how the brain constructs movements remains a fundamental challenge in neuroscience. The brain may control complex movements through flexible combination of motor primitives¹, where each primitive is an element of computation in the sensorimotor map that transforms desired limb trajectories into motor commands. Theoretical studies have shown that a system’s ability to learn actions depends on the shape of its primitives². Using a time-series analysis of error patterns, here we find evidence that humans learn dynamics of reaching movements through flexible combination of primitives that have Gaussian-like tuning functions encoding hand velocity. The wide tuning of the inferred primitives predicts limitations on the brain’s ability to represent viscous dynamics. We find close agreement between the predicted limitations and subjects’ adaptation to novel force fields. The mathematical properties of the derived primitives resemble the tuning curves of Purkinje cells in the cerebellum. Activity of these cells may encode primitives that underlie learning of dynamics.

Studies of reaching movements have demonstrated that humans construct motor commands based on a prediction of forces that will be experienced in the upcoming movement³. When novel forces are imposed on the arm, the prediction is in error and the arm does not follow the desired trajectory^3-4. With practice the motor commands are modified⁵ and the trajectory approximates the desired path. The learning of dynamics, however, affects movements outside the region of training^3,6-8, suggesting that the brain builds a state-dependent approximation of external forces⁹, called an internal model (IM). Occasional movements with unexpectedly altered dynamics, termed catch trials, have been used to quantify how the IM generalizes^3,4. Catch trials, however, not only test the IM for a given movement but cause errors that in turn change the IM and affect future movements. We demonstrate that the effect of errors experienced in a given movement on subsequent movements can reveal characteristics of primitives with which motor commands are generated.

We consider the IM to be a sensorimotor map transforming desired arm trajectories into muscle forces^10-12 through a flexible combination of a set of primitives:

$$\widehat{\mathbf{f}}=W^T\mathbf{g}({\mathbf{x}}^{\ast },{\stackrel{.}{\mathbf{x}}}^{\ast },{\ddot{\mathbf{x}}}^{\ast })$$ (1)

where T is the transpose operator, $\widehat{\mathbf{f}}$ is a vector approximation of forces f to be produced by muscles to compensate for task dynamics, and g is a vector of scalar valued primitives [g₁, ···, g_j]^T. Although g generally can depend on desired position, velocity, and acceleration $({\mathbf{x}}^{\ast },{\stackrel{.}{\mathbf{x}}}^{\ast },{\ddot{\mathbf{x}}}^{\ast })$, here we investigated learning of viscous forces and therefore considered a simpler subset of primitive functions that depended only on desired velocity. The IM is learned through experience dependent modification of the weight matrix W. Assuming a learning rule that minimizes ${\tilde{\mathbf{f}}}^2\equiv {\mid \mathbf{f}-\widehat{\mathbf{f}}\mid }^2$ is adjusted after a movement (indexed 1) according to:

$$\Delta W_1=-\eta \mathbf{g}\left({\stackrel{.}{\mathbf{x}}}_1^{\ast }\right){{\tilde{\mathbf{f}}}_1}^T$$ (2)

where η is a constant learning step. This adaptation changes the IM output in the subsequent movement (indexed 2):

$$\begin{array}{cc}\hfill {\widehat{\mathbf{f}}}_2(W+\Delta W_1)-{\widehat{\mathbf{f}}}_2\left(W\right)& ={(W+\Delta W_1)}^T\mathbf{g}\left({\stackrel{.}{\mathbf{x}}}_2^{\ast }\right)-W^T\mathbf{g}\left({\stackrel{.}{\mathbf{x}}}_2^{\ast }\right)\hfill \\ \hfill & =-\eta {\mathbf{g}}^T\left({\stackrel{.}{\mathbf{x}}}_1^{\ast }\right)\mathbf{g}\left({\stackrel{.}{\mathbf{x}}}_2^{\ast }\right){\tilde{\mathbf{f}}}_1.\hfill \end{array}$$ (3)

The change in the IM output depends on experienced error and the mutual projection between evaluations of the primitives, but does not depend on the weight matrix. Since the primitives depend on desired velocity, when the two movements have the same desired trajectory (e.g., toward the same target), the change should be proportional to the error experienced. When the two movements are toward different targets, the change will also depend upon the breadth of the receptive fields of the primitives.

We first tested whether an error experienced in a given movement causes a proportional change in the IM for the next movement to the same target. We asked subjects to make reaching movements while holding a manipulandum¹³ which produced viscous forces $\mathbf{f}=B\stackrel{.}{\mathbf{x}}$, where B={0, 13;-13, 0} N·s/m. Catch trials, movements during which f=0, were randomly interspersed among the targets. Our proxy for error was hand displacement perpendicular to target direction (perpendicular displacement; p.d.) measured 250 msec into the movement. The first movement in the field (1st in Figure 1A) had significant error (p.d.=2.38 cm), but with training (ct-1 in Figure 1A) became less disturbed (p.d.=0.45 cm). In the next movement toward this direction (90°), a catch trial (ct), there was a large error in the direction opposite the initial error, suggesting formation of an IM¹³. In the subsequent movement to 90° (ct+1), during which the force field was present, p.d. was substantially greater (p.d.=1.22 cm) than in ct-1, indicating partial unlearning of the IM as predicted by Equation 2. In agreement with Equation 3, there was a significant correlation between magnitude of movement errors in ct and the unlearning observed in ct+1 (Figure 1B, r=0.65). The physiological correlate of this unlearning was evident in the spatial tuning of movement-initiating muscle activations. The computational construct of an IM predicted that spatial tuning of EMG would undergo a specific rotation with training⁵. During training, the preferred direction of this tuning gradually rotated. However, between ct-1 and ct+1 the preferred direction rotated back toward the initial orientation (Figure 1C), indicating unlearning of the IM. This unlearning washed out by movement ct+3 (Figure 1D).

Figure 1.

Catch trials induced short-term unlearning. A: Hand trajectories of single movements, averaged across all 40 subjects, during the first (labeled 1st), 38th (ct-1), 39th (ct), and 40th (ct+1) movements toward 90° (0° is at 3 o’clock). Intersection of error bars indicates parallel and perpendicular displacement (p.d.) 250 ms into the movement. B: Jumps in p.d. between before (ct-1) and after (ct+1) catch trials versus p.d. in catch trials (ct), averaged within target directions across subjects. C: Orientation of preferred direction of movement-initiating EMG in ct-1, ct, and ct+1. Asterisks indicate significant (p < 0.05) rotation of EMG preferred direction between ct-1 and ct+1. D: Errors in force field movements following ct. In all figures, error bars indicate 95% confidence intervals of the mean (CIM) across subjects.

We next investigated the shape of primitives underlying IM formation by quantifying, independent of the model in Equation 3, the temporal dynamics of movement errors first within and then across directions. In a sequence of random target directions, the time series of movement errors for a given direction was fitted to the following system of equations:

$$\begin{array}{cc}\hfill z_{n+1}& =a{z}_n+b{u}_n\hfill \\ \hfill y_n& =z_n+d{u}_n.\hfill \end{array}$$ (4)

Here y represented error in the IM as quantified by p.d.; n was movement number, u indicated whether the force field was present (u=-1) or turned off (u=1). The hidden state of the system, z, represented the amount of movement error generated by the IM; actual error (y) also depended on whether the force field was applied. The implicit assumption in this initial model was that errors experienced in one target direction did not affect the IM for generating movements toward other targets. The best-fit model correlated to actual errors reasonably well (Figure 2A, black line; across directions, mean r=0.60). The fit mimicked subjects’ recovery from initial error, their large error in ct, and their jump in error from ct-1 to ct+1. Whereas this initial model smoothly decayed after jumps in error, subjects often generated a nonmonotonic change in error between catch trials. We hypothesized that this was because errors experienced in one target direction changed the IM for other directions.

Figure 2.

Sensitivity to movement error across target directions. A: Errors in subjects’ reaches (·) (p.d. at 250 msec) toward 90°, averaged across subjects (n=40). Catch trials are the large negative spikes (one labeled ct). Lines are best fits of scalar (black) and vector (red) state-space models to the data (y in Equation 4). B: Average sensitivity of the IM to errors experienced in previous movements (b in Equation 4) as a function of angular distance ϕ. Error bars (95% CIM) calculated through bootstrapping. C: Average sensitivity b(ϕ) for simulated adaptive controllers that constructed an IM with Gaussians of width σ. b(ϕ) in simulations and subjects were averaged across target directions.

To investigate whether locally experienced errors affected other directions of movement, we expanded both u and b in Equation 4 to eight-dimensional vectors. Each element of the input vector u flagged recently experienced dynamics in a particular target direction i: whether, since the last movement in the modeled target direction, a force field movement (u(i)=-1) or a catch trial (u(i)=1) had been most recently experienced, or if no movement had occurred in direction i (u(i)=0). Each element of b, b(i), quantified the sensitivity to errors experienced in direction i. The expanded model now accounted for subtle changes in actual movement error (Figure 2A, red line; mean r=0.81). Confidence intervals on b suggested that there was a significant, non-zero influence of local errors on subsequent control in other directions. To calculate sensitivity across target directions, the elements of b were re-indexed by the angular distance between the direction in which errors were experienced and the modeled movement direction. This distance was represented by ϕ. Averaging b(ϕ) across movement directions (Figure 2B) demonstrated that errors experienced in a given movement maximally influenced the IM for that direction. This influence decayed in neighboring directions. Surprisingly, sensitivity became significantly negative when angular distances were larger than 90°. This indicated that when two force field movements were separated by angle ϕ, if ϕ was small, then errors experienced in the first movement improved the IM for the second movement. If ϕ was large, then errors in the first movement destructively interfered with the IM used to generate the second movement.

To explain this result, note that sensitivity of the IM to experienced errors, b(ϕ), was quantified in terms of p.d. Both the output and the error signal of the IM, however, are in terms of force (Equations 1 and 2). Because the force field is linear in velocity, the direction of force error corresponding to positive (clockwise) p.d. toward one target opposes the direction of force error corresponding to positive p.d. toward the opposite target. Interpreting the sensitivity of subjects’ IM (Figure 2B) through the adaptation rule (Equation 3) suggests that both the positive values of b for -45° < ϕ < 45° and the negative values of b for large ϕ correspond to the same direction of force compensation. From this we deduced that the mutual projection ${\mathbf{g}}^T\left({\stackrel{.}{\mathbf{x}}}_i^{\ast }\right){\mathbf{g}}^T\left({\stackrel{.}{\mathbf{x}}}_{i+1}^{\ast }\right)$ declines but always remains positive as the angular distance between two movements increases. This result rules out bases that encode velocity space linearly. Furthermore, because information experienced in each direction most strongly affects that direction and less so its neighbor, basis functions that have specific regions of preferred activity are more likely to underlie learning than global representations of dynamics.

We therefore investigated what conditions on $\mathbf{g}\left({\stackrel{.}{\mathbf{x}}}^{\ast }\right)$ were sufficient to generate the generalization function b(ϕ). A salient property of cells in the motor system is their directional tuning¹⁴ and modulation with hand speed¹⁵. In the cerebellum, a region which lesion^16-19 and functional imaging studies^20,21 have linked to learning and control of arm dynamics, many Purkinje cells simultaneously encode the direction and speed components of velocity²². These cells broadly encode hand velocity during planar reaching, firing maximally at preferred velocities (PVs) distributed in velocity space. This encoding precedes in time the actual movement, suggesting that these cells encode desired velocity. The behavior of each cell k could therefore be represented as a Gaussian with a center located at position c_k in desired velocity space. We simulated a controller attached to a biomechanical model of the arm that learned an IM with basis functions:

$$g_k\left({\stackrel{.}{\mathbf{x}}}^{\ast }\right)=\exp ({\mid {\stackrel{.}{\mathbf{x}}}^{\ast }-{\mathbf{c}}_k\mid }^2∕2{\sigma }^2)$$ (5)

where σ is the standard deviation of the Gaussian. To accommodate the possibility that the exact shape of b(ϕ) depended on the training paradigm, we trained subjects and the simulated controller with the identical set of targets and catch trials. A crucial component of the simulations was σ, the width of the primitives. When the Gaussians were narrow, the time series of errors generated by the simulation showed a spike after each catch trial and a smooth decay afterwards, similar to the scalar-input state-space model fit but unlike the performance of our subjects. Simulations driven by broad Gaussians, however, produced nonmonotonic changes that mimicked subjects’ actual patterns of adaptation. Simulation results were fitted with Equation 4 to produce the generalization function b(ϕ) (Figure 2C). The b(ϕ) generated with narrow Gaussians rapidly dropped to zero as ϕ changed from zero. Learning with wide Gaussians, however, exhibited a generalization that was very similar to actual subject performance, including negative sensitivity for large ϕ. The correlation between b(ϕ) in the simulations and the subject data was strongest for σ=0.12 m/s.

We next used the model to predict behavior beyond the data set with which the primitives were estimated. We noted that Gaussian width influences how force estimation generalizes across both directions and speeds. Simulations predicted that when learning relied upon wide Gaussians, reaching movements would not monotonically converge onto a straight line desired trajectory but would become S-shaped (Figure 3A). While the force field was linear in velocity, wide Gaussians produced an approximation that overestimated forces at low speeds and underestimated forces at high speeds (Figure 3B). Overestimation of the forces resulted in overcompensation of the field early in the movement; the magnitude of overcompensation depended on the Gaussian width (Figure 3C). With narrow Gaussians, the simulated IM did not overcompensate, but with wide Gaussians movements became S-shaped. To test this prediction, we trained 24 subjects in target sets without catch trials. Movements of subjects were S-shaped (Figure 3E-1), similar to movements made by simulations that learned with wide Gaussians (Figure 3A).

Figure 3.

Movement characteristics of systems that learn an IM with velocity encoding Gaussians. A: Simulated adaptive controllers trained in a target set without catch trials. The 11th (left) and 21st (right) movements toward 90° are shown for each controller. Learning with wide Gaussians produced S-shaped movements. B: Adaptive controllers’ estimation of the force field after training for 100 movements in a target set without catch trials. Peak velocity of typical movements is 0.35 m/s. C: Time course of error (p.d. at 200 msec) for simulated controllers with various Gaussian widths, smoothed across a 13-movement window. Overcompensation (generation of S-shaped hand trajectories) occurs when p.d. becomes negative. D: Error (p.d.) averaged over the subset of movements 65-128 during which the force field was on. With narrow Gaussians, overcompensation never occurred, regardless of catch trial probability. With wide Gaussians, movements became S-shaped below critical probabilities (near 17%). E: Parallel and perpendicular displacements averaged across movements and subjects. E1: Subjects (n=24) trained in target sets without catch trials, resulting in S-shaped movements. E2: Subjects (n=40) trained in target sets with 17% catch trial probability and did not produce S-shaped movements. F: A controller relying upon wide Gaussians (σ=0.12 m/s) trained in target sets with 17% catch trial probability did not produce S-shaped trajectories.

Simulations further predicted that the probability of catch trials influenced whether movements would become S-shaped (Figure 3D). If catch trials occurred with 17% probability, then even with Gaussians of σ=0.12 m/s there should be sufficient unlearning caused by each catch trial such that hand trajectories would converge toward a straight line, without overcompensation (Figure 3F). We tested this prediction by training subjects in target sets with 17% catch trial probability. As predicted, subjects did not show overcompensation (Figure 3E-2).

Because approximation of a high frequency signal with low frequency bases generally results in poor representation, we next explored the limitations of a system that learns with wide Gaussians. We simulated learning of nonlinear force fields:

$$\begin{array}{c}\mathbf{f}=-13\sqrt{{\stackrel{.}{x}}^2+{\stackrel{.}{y}}^2}\left[\begin{array}{c}\hfill -\sin \left(m\theta \right)\hfill \\ \hfill \cos \left(m\theta \right)\hfill \end{array}\right]\hfill \\ \theta =\arctan (\stackrel{.}{y}∕\stackrel{.}{x})\hfill \end{array}$$ (6)

where $\stackrel{.}{x}$ and $\stackrel{.}{y}$ were components of hand velocity in Cartesian space. When m=1, the field was the curl pattern¹³ learned by subjects above. As m increased, the field’s spatial frequency increased. For various σ, we simulated movements to 50 targets and correlated the IM learned by the simulation to the actual force field (Figure 4A). As the field’s spatial frequency increased, the accuracy of the IM decreased for all basis function widths. However, the learning capability of wider bases collapsed at lower frequencies. This agrees with the recent finding that humans demonstrated a lesser ability to adapt in higher spatial frequency force fields²³.

Figure 4.

Learning with wide Gaussians imposes limits on adaptation. A: Correlation between actual force field and IM acquired after 50 targets by adaptive controllers with various width Gaussians, versus the frequency of the nonlinear force field (m in Equation 6). B: A high spatial frequency field (m=4) in which subjects and a simulated controller trained. C: The IM learned by the controller with σ=0.12 m/s Gaussians after training for 50 movements. Note that for directions 22.5° and 157.5°, the controller predicts the field to be resistive, whereas actual forces are assistive. D: Hand velocities parallel to target direction of three subjects trained in the same high spatial frequency field in the same target set as the simulated controller. Plotted are velocities in baseline null field movements (mean ± standard deviation) and during catch trials in targets 51 and 58 toward 22.5° and 157.5°. In catch trials, the hand is moving significantly faster, consistent with an IM that expects a resistive field.

To illustrate this deficiency, we trained an adaptive controller (σ=0.12 m/s) for 50 movements in a high spatial frequency field (m=4, Figure 4B). Because approximation is performed with wide bases, the IM learned by the controller (Figure 4C) cannot represent faithfully the rapidly changing forces. In particular, the simulation predicts that in movements toward 22.5° and 157.5° the IM expects resistive forces where the force field is assistive. We tested for this prediction in three subjects by training each with the same random pattern of targets presented to the simulation, then presenting two catch trials toward 22.5° and 157.5°. Subjects behaved as though they were expecting a resistive field in those directions, as illustrated by their hand velocities in the catch trials (Figure 4D).

Errors in learning of arm movements suggest that the brain composes motor commands with computational elements that are broadly tuned in arm velocity. When expressed in polar coordinates, the Gaussians exhibit a preferred direction of movement, much like the cosine tuning curves typically associated with cells in the motor system. However, our inferred bases have on average a half-width at half-height value of about 40°, significantly less than the 90° value required of cosines. Recent results²⁴ have demonstrated that tuning curves in monkey motor cortex have a median width of 56°, also much narrower than cosines. Motor cortical cells, however, have been reported to encode hand speed linearly¹⁵. Learning a linear force field with bases that have cosine directional tuning and linear speed tuning results in an IM that does not produce the S-shaped movements observed in our subjects. Wide Gaussians predict this curious behavior. They also explain why humans generalize sublinearly to fast movements after training in a linear force field at slow speeds⁸. The nonlinear encoding of speed inherent to Gaussians resemble tuning properties of Purkinje cells that encode arm movements in the cerebellum²². While several investigators have proposed a major role for the cerebellum in learning of IMs^25-27, our results suggest a link between patterns of generalization and firing properties of cells in this area. Since the output of the cerebellum partially affects the motor cortex²⁹, the finding that the preferred direction of motor cortical cells rotates during learning of force fields²⁸ may be a consequence of changing input from the cerebellum³⁰.

Methods

Three groups of right-handed normal human subjects were trained to make horizontal movements while holding the handle of a lightweight robotic arm. All movements were toward a pseudorandomly chosen target, then back to a center target. Targets appeared at 0°, 45°, ... 315°, at 10 cm displacement. Desired movement duration was 500±50 ms. Timing feedback was provided by changing the target color. All subjects initially practiced the task without any perturbing force (the null field). The first group of subjects (n=40) trained in target sets of 192 movements during which the force field B={0, 13;-13, 0} N·s/m was applied in 83% of the targets. The remaining 17% of targets (pseudorandomly selected) were catch trials during which the force field was turned off. In 13 of these subjects, EMG from anterior and posterior deltoid, biceps, and triceps were measured with surface electrodes, amplified, filtered, RMS rectified, averaged from -50 to 100 ms into the movement, multiplied by unit vectors pointing toward movement direction, and summed across movement directions to produce a preferred direction for each muscle⁵. The second group of subjects (n=24) trained in the same target sets as above, but with the force field always on (no catch trials). A final group of subjects (n=3) trained for 58 movements in a higher frequency force field (Equation 6; m=4), receiving catch trials only on targets 51 and 58 in directions 22.5° and 157.5°.

A simulated anthropomorphic controller¹³ made movements to the same sequence of targets experienced by subjects. The controller’s IM learned to map desired hand velocities into forces. Desired hand trajectories were minimum jerk, 0.5 sec in duration, and 10 cm long. The IM approximated the imposed force field (Equation 1) with Gaussian bases (Equation 5; controllers were differentiated by σ). Gaussian centers spanned a range of desired velocities (-0.5 to 0.5 m/s in the x- and y-direction) spaced one σ apart. Sensitivity analyses suggested there were no significant changes in the results when the density of the bases was increased by an order of magnitude. Weights were initially randomized, then updated using Equation 2 with η=0.0025. To compare the output of Gaussian basis functions with cosine tuning curves, unweighted output of each Gaussian was averaged during movement time. The collection of averaged outputs across movement directions formed a tuning curve for each Gaussian primitive. Tuning curves were aligned to the preferred direction of each Gaussian and averaged across all Gaussians, from which the half-width at half-height value was calculated.