Motor expertise facilitates the accuracy of state extrapolation in perception

Abstract

Predicting the behavior of objects in the environment is an important requirement to overcome latencies in the sensorimotor system and realize precise actions in rapid situations. Internal forward models that were acquired during motor training might not only be used for efficiently controlling fast motor behavior but also to facilitate extrapolation performance in purely perceptual tasks. In this study, we investigated whether preceding virtual cart-pole balancing training facilitates the ability to extrapolate the virtual pole motion. Specifically, subjects had to report the expected pole orientation after an occlusion of the pole of 900ms duration. We compared a group of 10 subjects, proficient in performing the virtual cart-pole balancing task, to 10 naïve subjects without motor experience in cart-pole balancing task. Our results demonstrate that preceding motor training increases the accuracy of pole movement extrapolation, although extrapolation is not trained explicitly. Additionally, we modelled subjects’ behaviors and show that the difference in extrapolation performance can be explained by individual differences in the accuracy of internal forward models. When subjects are provided with feedback about the true orientation of the pole after the occlusion in a second phase of the experiment, both groups improve rapidly. The results indicate that the perceptual capability to extrapolate the state of the cart-pole system accurately is implicitly trained during motor learning. We discuss these results in the context of shared representations and action-perception transfer.

Introduction

Expert tennis players are able to extrapolate the motion of a tennis ball and return it skillfully in order to score a point solely based on their present percept. The current consensus is that internal forward models, which predict the dynamic behavior of the body and objects in the environment (such as the ball and tennis racket) support the control of movements [1–3]. Especially in fast situations internal forward models seem to be exploited to overcome the delay of sensory input [4,5] and to predict events [6–9]. Since motor control is inherently related to the prediction of sensory consequences in order to act optimally, the question arises whether motor expertise facilitates the process of perceptual state extrapolation when asked to explicitly report the state as accurate as possible and how motor expertise determines the accuracy of such extrapolations.

In past studies, subjects’ general ability to predict the behavior of objects in order to act purposefully has been investigated using diverse paradigms. For example, La Scaleia et al. [10] have examined subjects’ ability to intercept partially occluded ball trajectories. Despite the occlusion, subjects were able to intercept the ball with high accuracy. In another paradigm subjects predictively control objects with internal degrees of freedom [11], such as mass-spring-damper objects [12], a virtual cup of coffee [13] or the cart-pole system [14]. Predicting the behavior of these objects is non-trivial, because all exhibit a high degree of complexity. In this context, Mehta and Schaal [14] examined subjects while controlling the cart-pole system [15] with the goal to balance the pole (cart-pole balancing task) under different conditions of visual feedback. They compared subjects’ actions under full visual feedback to actions during short occlusions of up to 550 milliseconds. Despite missing visual feedback, subjects’ actions were indistinguishable from those under full vision. In correspondence with the above-mentioned studies they concluded, that actions during the occlusion are performed based on an extrapolated state, which replaces the missing visual feedback. In line with this conclusion, we recently showed in a virtual reality implementation of the cart-pole balancing task that subjects gradually learn to perform their actions predictively [16], suggesting that subject implicitly extrapolate the systems behavior to plan and time the actions in advance. Moreover, if the dynamics of the system were changed, subjects needed to adapt the action timing to the changes just as it is known from sensorimotor adaptation paradigms [17]. This suggests that an internal forward model, which mimics the cart-pole dynamics, is adapted and, furthermore, that it is used to extrapolate the state during occlusion of the pole in Metha and Schaal’s experiment [14]. Altogether, these studies indicate that motor training alters not only motor control but also affects mechanisms of sensory processing. However, these studies only provide evidence in favor of improved state extrapolation capabilities for the purpose of motor control.

Psychophysical studies suggest that internal simulations are also exploited during different tasks of perceptual motion extrapolation. Graf et al. [18], for example, showed subjects human actions as point-light movies and asked them to judge whether a static posture, shown after an occlusion of varying duration, is a plausible succession of the action. The static postures either matched with the duration of the occlusion or not. They found, that subjects’ responses match with the internal simulation hypothesis, which predicts that subjects’ error rates depend on the mismatch of the shown posture and the duration of the occlusion. Since we are all acquainted with human motion, comparing different levels of motor expertise was not the goal of the study but the study shows that humans can be very precise in spatial-temporal extrapolation. Aglioti et al. [19], in comparison, examined how the expertise in playing basketball influences the ability to judge the success of a free shot. They found that elite basketball players are more accurate in this task than subjects with similar visual but with much less motor experience (coaches and sports journalists). While these studies show that internal simulation and motor expertise influence non-motor related discriminative decisions, it is still an open question whether motor expertise improves the accuracy in extrapolating the state of the controlled system. Predicting the exact state of a complex dynamical system, such as the cart-pole system, over a certain duration of visual occlusion and reporting it explicitly, is presumably a considerably more challenging task than, for example, only judging the success of a free shot.

In this study, we examined whether prior motor experience facilitates the accuracy in predicting the state of the cart-pole system not only implicitly for the purpose of control but also explicitly for reporting the state. We hypothesized that subjects who are able to balance the cart-pole system are more accurate in extrapolating the pole angle dynamics than subjects without motor expertise who are at most visually familiar with the cart-pole system. Furthermore, we hypothesized that the reason for the enhanced ability to predict the pole angle dynamics accurately is that motor experienced subjects possess a more accurate internal model of the cart-pole system. Specifically, we hypothesized that the time horizon, over which accurate predictions can be performed, is larger for subjects with motor expertise and fitted subjects’ responses with a corresponding model. Lastly, we show that subjects without motor expertise are able to improve in extrapolation performance when provided with feedback about the true pole angle (correct response). Our results are in line with the hypothesis, that motor training facilitates perceptual capabilities and, furthermore, suggest that these perceptual capabilities are not only implicitly accessible for the purpose of control but also to perform explicit and accurate state extrapolation when asked to report the state in a perceptual task.

Methods

Subjects

Twenty healthy young participants (age range 18–33 years, mean age 26.1) participated in the main experiment (Fig 1). All participants gave informed written consent prior to participation. The study had been approved by the ethical review board of the medical faculty of the Eberhard-Karls-University and University Clinics in Tübingen, Germany (AZ 409/2014BO2). Of the twenty participants, ten subjects participated in a previous experiment in our laboratory, in which they acquired the cart-pole balancing skill during a training session of 90 minutes, and were thereby able to control the cart-pole system (skill acquisition). Consequently, these subjects were assigned to the motor control familiar (MF) group. The average time between the skill acquisition and the present experiment was 335 days (range: 190–570 days, sd: 125 days). During this time, they did not participate in any other study involving the cart-pole system. Thus, they have 90 minutes of experience with the cart-pole system. The remaining ten subjects (visual familiar group, VF) were naïve regarding the control of the cart-pole system (without motor expertise). All participants were naïve regarding the goal of this study. Gender and age have been balanced between groups (mean ±sd: MF 26.3 ±3.2 years, VF 25.9 ±3.5 years). All participants had normal or corrected to normal vision. Subjects were paid 8 Euros per hour independent of their performance.

Fig 1. Cart-pole extrapolation task.

(A) Sequence of the blocks and tasks. Subjects in the group VF followed the upper (orange) path, while subjects in the group MF followed the lower (blue) path. In block E, the extrapolation task was explained. In block B, the baseline performance was measured. The subsequent four blocks were preceded by a visual familiarization block (orange) or motor control block (blue). Feedback about the correct response was only provided during the last five blocks (dark gray, F1-F5, see below). (B) Phases of each trial during the cart-pole extrapolation task. During the observation phase, a red arrow indicated the direction and magnitude of the force, which was applied during balancing. During the no input phase, the force applied to the cart was set to zero (see main text) while the pole was still visible. In the prediction phase, the pole was occluded but the dynamical system was simulated further with the force set to zero. Thus, the cart and (invisible) pole kept moving. Subsequently the time bar turned red and the subject gave a response (keypress) representing the expected orientation of the pole. The feedback phase indicated the correct pole angle and was only provided during the last five blocks (F1-F5, see A). RT: response time. (C) Schematic illustration of the prediction error regarding the pole angle. Notice that an underestimation of the pole movement (blue) corresponds to a positive error irrespective of the side.

Experimental protocol

We examined two groups representing different degrees of motor expertise. In the first group (motor familiar, MF), subjects had learned to balance the pole on the cart in a previous study and were therefore familiar with the cart-pole system (with motor expertise). Subjects in the second group (visual familiar, VF) had no prior exposure to the cart-pole system (without motor expertise). We familiarized the subjects with the cart-pole system by asking them to watch and rate balancing attempts of other subjects (visual familiarization).

The experimental protocol consisted overall of three tasks based on the cart-pole system: (1) extrapolation task, (2) motor control task and (3) visual familiarization task (see following sections for details). The main task of the experiment was the extrapolation task, in which the subjects had to extrapolate the motion of the cart-pole system’s pole over 900ms and report the final orientation (see below). Subjects of both groups performed overall 11 blocks, each consisting of 40 trials, of the cart-pole extrapolation task (Fig 1A). The very first block (block E) was used to ensure task understanding. The baseline extrapolation performance was assessed during the second block (block B). Each of the subsequent four blocks (T1-T4) was preceded by a 5 minutes long block consisting either of the motor control or visual familiarization task (Fig 1A) depending on the group affiliation. Subjects in the group MF only performed motor control blocks, in which they had to perform the cart-pole balancing task, while subjects in the group VF only performed visual familiarization blocks, in which they rated balancing attempts of other subjects. Subsequent to these blocks (T1-T4), participants performed the extrapolation task for another five blocks (F1-F5). Only during these five blocks (F1-F5) feedback about the true final pole orientation (correct response) was provided after the subject had responded (feedback phase, Fig 1B, see below for details). The whole experiment lasted about 90 minutes.

Experimental setup

Visual feedback was provided on a 17 inch monitor (1920x1200px) using the Psychtoolbox [20–22] for MATLAB (The Mathworks, Inc.) at a refresh rate of 60 Hz (Fig 1 and Fig 2). Subjects’ heads were supported using a chin-rest 60 cm away from the screen. The pole of the virtual cart-pole system was rendered as a 160 pixels long line, corresponding to about 3cm on the screen or 2.9° in visual space. The cart had a width of 160 pixels. A keyboard for recording subjects’ responses during the extrapolation and visual familiarization blocks was positioned between the subject and the monitor. For subjects in the group MF, a SpaceMouse® Pro (3Dconnexion) was additionally placed next to the keyboard for controlling the cart-pole system in the motor control blocks. The device’s knob can be displaced by ±1.5mm in lateral direction and exerts a force of 7.4N at full lateral displacement back to the rest position in the center. The cart-pole dynamics [15] are described by the pole mass (0.08 kg), pole length (1 m), cart mass (0.4 kg) and gravitational constant (3.5 m/s²). We did not simulate friction. The knob’s lateral displacement was linearly mapped to a virtual force of up to 4N pushing against the cart, which is enough force to lift the pole from the maximum simulated angle of ±60 degrees. Like in our previous study [16], we implemented the simulation in MATLAB (The MathWorks, Inc.) using the 4^th-order Runge-Kutta method.

Fig 2. Motor control and visual familiarization task.

(A) Visualization of the cart-pole system for both tasks. In the visual familiarization blocks the force was indicated as red arrow (like in the extrapolation task). The reward (blue number) that we presented during the motor control blocks (MF subjects) was not shown during the visual familiarization (VF subjects). We did not show the arrow during the motor control blocks. (B) The input device, which was used for controlling the system during the motor control blocks. The knob of the input device can be shifted left and right, which was used to control the virtual force that is applied to the cart from either side. (C) Phases of each trial during the visual familiarization. During the observation phase, a balancing attempt was shown that lasted up to 30 seconds. Afterwards, subjects rated the attempt on a scale from one (very bad) to five (very good). BD: balancing duration, RT: response time.

Cart-pole extrapolation task

In the extrapolation task, participants were asked to indicate the expected angle of the pole after a short occlusion (900ms) during the simulation of the cart-pole system (Fig 1B). Every trial began with a short sequence (4.5sec) extracted from previous balancing attempts of other subjects (Fig 1B). The force (magnitude and direction) that was applied by the subject to balance the pole on the cart was indicated as red arrow. Afterwards the system was simulated over one second with the input force set to zero. Within this second, the pole was occluded after 100ms for the remaining time of 900ms. Since the input force was zero during this time, there were neither any discontinuities nor any external sources influencing the systems behavior even 100ms before the occlusion, making a deterministic extrapolation of the pole dynamics without additional information possible. To indicate the expected pole angle, subjects had to choose one of thirteen response options (Fig 1B and 1C), which were evenly distributed over the pole angle range [-65°, 65°] and identified by nearby letters. Each letter corresponds to the key subjects had to press for choosing the respective response option. Subjects were instructed to always choose the response option that is closest to the expected pole angle. A time bar (Fig 1B) indicated the time progression within the trial. Subsequent to observing the system’s movement for overall 5.5 seconds, the time bar turned red which was the sign for the subjects to respond. In the blocks F1-F5 (Fig 1A), subjects received feedback after having responded, which consisted of the presentation of the true pole orientation. In every block of the extrapolation task, we presented the same 40 stimuli, corresponding to the 40 trials in each block, in pseudorandom order and recorded subjects’ responses. The stimuli were selected such that the correct responses were balanced between the left and right side, while additionally covering a variety of pole angles and angular velocities at the time of occlusion onset.

Familiarization tasks

During the motor control blocks, participants had to balance the virtual cart-pole system (Fig 2A, for details see also [16]). Specifically, they were asked to balance the pole on the cart for a maximum of 30 seconds without letting the pole fall out of the green arc (±60 degrees) and without driving off the track (±5 m). They therefore had to apply lateral virtual forces to the cart, which in turn accelerated the cart. In order to control the force, they used an input device (SpaceMouse® Pro, 3Dconnexion) with a lateral degree of freedom in displacement (see Experimental Setup and Fig 2B). Pushing the knob of this device to the side was translated into a proportional force into the same direction. Because the device’s position was aligned with the monitor, the left-right knob movement was in correspondence with the direction of the virtual force and cart movement on the monitor. Hence, a rightward knob movement caused a force that pushed the cart to the right. Subjects quickly remembered how to balance the pole on the cart and therefore rapidly reached high performance (see S1 Appendix).

In the visual familiarization blocks (Fig 2C), participants had to observe balancing attempts of previously recorded subjects instead of controlling the system themselves. Like in the extrapolation task, a red arrow indicated the magnitude and direction of the force, which had been applied during actual balancing. Apart from these arrows, visual information was identical in both conditions (see S2 Appendix). In order to ensure that participants paid attention to the balancing performance and to the dynamic behavior of the cart-pole system, they had to rate every balancing attempt on a scale from one (very bad) to five (very good). Participants were instructed that the task of the individuals, whose balancing performance was shown, was to balance the pole on the cart for a maximum of 30 seconds and, that the persons were rewarded (points) depending on the duration and proficiency of the balancing attempt. Importantly, the presented attempts during these blocks were different from those shown during the extrapolation task. Additionally, instead of only showing short sequences as in the extrapolation task, the entire balancing attempts were shown, which each lasted up to 30 seconds (duration range: 1.3–30 seconds, 12% of the balancing attempts lasted 30 seconds).

Control experiment

In order to examine whether subjects improved in extrapolation accuracy if the visual familiarization exhibits a similar overall duration as the preceding motor control training of the subjects in the group MF, we performed a control experiment (group extVF). In this condition, each visual familiarization block (orange blocks in Fig 1A) was 20 minutes long (instead of 5 minutes), providing the participant with an overall interaction time of 80 minutes with the system. This is comparable to the duration, which subjects in the group MF needed during the initial motor training to master the cart-pole balancing task. It turned out that this procedure was quite cumbersome and annoying for the subjects, not only because of the overall duration but also because subjects did not improve without receiving feedback (ANOVA which examined the effect of the blocks B and T1-T4 on the prediction error, within-subject factor block: p = 0.92). Hence, we only recorded three subjects in this control experiment. Although a statistical comparison between groups is not possible, subjects in this group exhibited a similar average prediction error (blocks B and T1-T4) as subjects in the group VF (mean ±sd: extVF -14.84° ±1.92, VF -14.84° ±2.94). We therefore concluded for the further analysis that there is no difference between repetitively observing and rating balancing attempts in blocks of 5 or 20 minutes duration.

Data analysis

Data preprocessing has be performed in MATLAB (The Mathworks, Inc.). Due to a recording error, one of the subjects in the group MF had to be excluded, leaving overall 9 subjects in the group MF and 10 in the group VF. All statistical analyses have been performed in R (v3.3.2) using the packages lme4 (v1.1), lmerTest (v2.0), phia (v0.2) and nnet (v7.3). Measures have been examined regarding normality using visual inspection of quartile-quartile plots.

Rating of balancing attempts during visual familiarization

Based on participants’ responses during the visual familiarization blocks, we investigated how the two parameters balancing duration (BD) and mean absolute pole angle (maPA) influenced the rating. While the balancing duration is an obvious measure of balancing performance, the mean absolute pole angle represents how easeful the balancing was performed. The lower the mean absolute pole angle, the smoother the system was controlled and the more vertical the pole was held throughout the trial. Notice that the two parameters are correlated in the examined data. However, there are also successful trials with long balancing duration that exhibit a high mean absolute pole angle. Similarly, there are short trials with low balancing duration that exhibit a low mean absolute pole angle. First, we examined the effect of the rating on the two parameters using a mixed-effects ANOVA. This provides information about the difference between the rating classes in these parameters. Secondly, we fitted a multinomial log-linear model that mapped the two parameters (BD, maPA) and their interaction to the rating. Using model comparison (likelihood ratio test), we determined the significance of each of the two parameters (and their interaction) for the rating. Thereby, we revealed that both parameters and their interaction are influencing the rating of the balancing attempts significantly (see Results).

Pole angle prediction error

The main measure for the accuracy in predicting the pole angle is the prediction error (Fig 1C). In the extrapolation task, subjects chose in every trial one of 13 response options corresponding to the orientation where they expected the pole to be after the occlusion. We defined the prediction error (PE) as the angular difference between the true pole angle and subject’s response multiplied by the sign of the true pole angle (Eq 1, Fig 1C).

$$PE=\left({\theta }_{resp}-{\theta }_{true}\right)\mathrm{sign}\left({\theta }_{true}\right)$$ (1)

Thus, negative errors correspond on both sides to an underestimation of the pole’s downwards motion, while positive errors correspond to an overestimation. Statistical examination of the factors influencing the prediction error was performed using linear mixed-effect ANOVAs. In all models, we accounted for individual performance and learning rates by introducing random effects per subject.

We first examined whether the ability to extrapolate the pole angle improves during the familiarization phase (B, T1-T4). Since we did not find any improvement (see Results), we then compared the prediction error between the groups in the blocks T4 (no feedback, but familiar) and F5 (feedback). These blocks represent different knowledge levels: (T4) subjects are familiar with the system, but did not receive feedback yet, and (F5) subjects had time to utilize the feedback for improvement. We conducted a mixed-effect ANOVA that examined the effect of group and feedback on the prediction error in these blocks.

Models of inaccurate state predictions

In attempt to identify the cause for subjects’ inability to predict the pole angle accurately, we investigated whether an inaccuracy of the subjects’ individual internal forward models could account for the difference in observed prediction errors. Specifically, if subjects had a perfect forward model (PERF) of the pole dynamics, the prediction error would be roughly zero (neglecting the discretization error introduced by the finite number of response options). Simplification of the dynamics, such as assuming constant pole acceleration (cACC, S3 Appendix) or constant pole velocity (cVEL, S3 Appendix), as it might be appropriate for falling objects depending on whether air-resistance is considered, introduces a considerable error in the extrapolated pole angle because the actual pole acceleration depends on the angle. Subjects may also be able to predict the pole movement accurately only over the first few milliseconds, before they have to switch to a heuristics because of their limited computational capabilities. As outlined in the introduction, our hypothesis is that motor expertise facilitates these capabilities, leading to an increased duration of the interval for which accurate extrapolation is possible (extrapolation horizon). We modelled this behavior by a class of partly heuristic response models, which are described in more detail in the appendix (lh_cVEL, S3 Appendix). The models simulate the pole dynamics accurately over the first h milliseconds (extrapolation horizon) and then assume constant pole velocity for the remaining time of the occlusion (900ms—h). We investigated 61 values of the parameter h, corresponding to the 61 frames (Δt = 1/60s) from 100ms (h = -100ms) before to 900ms (h = 900ms) after the pole occlusion. Notice, that the models for h = 0ms and h = 900ms coincide with the previously specified models cVEL and PERF. The average prediction error of the models decreases monotonically with increasing h (S3 Appendix). Our interpretation of negative extrapolation horizons is that the respective subject solely relies on the constant velocity assumption to extrapolate the pole angle and additionally uses a deprecated estimate of the pole velocity (e.g. derived from 50ms before occlusion), instead of the actual velocity immediately before the occlusion.

We fitted the parameter h for each subject separately using the individual responses in each trial of block T4 (see S3 Appendix for details). Notice, that this procedure is different from fitting the model parameter based on the mean prediction errors. After having determined the best fitting model (h*) for each subject, we compared the groups also based on the parameter h*. In analogy to the model class lh_cVEL, we also explored the class lh_cACC, which corresponds to assuming constant pole acceleration after first extrapolating the pole angle for h milliseconds accurately. However, the model class lh_cACC was not able to explain subjects’ behavior, because subjects made larger errors than the model could explain for any value of h between minus 100ms and plus 900ms.

Results

Rating of balancing attempts is influenced by the balancing duration and mean absolute pole angle

During visual familiarization, every subject in the group VF rated overall 100 balancing attempts on a scale from one (very bad) to five (very good). In order to verify that subjects did not respond randomly, but paid attention to the balancing attempts and thereby observed the cart-pole dynamics, we examined whether the ratings of the subjects were meaningful. Two salient parameters for rating the balancing attempts are the balancing duration (BD) and the mean absolute pole angle (maPA).

First, a mixed-effects ANOVA was conducted, for subjects in the group VF, that examined the effect of rating for the two parameters, BD and maPA (Fig 3A and 3B). We found a significant effect of the factor rating for each of the parameters (p<0.001), suggesting that the rating separated each parameter into distinct classes. Post-hoc test revealed significant pairwise differences between all rating classes for BD (p<0.001, Holm corrected). For the maPA, all but one (3 vs 4) pairwise difference were significant (p<0.01, Holm corrected). Hence, subjects used the rating to classify the two performance measures, balancing duration and mean average pole angle, into ordered, significantly different, and thereby meaningful classes.

Fig 3. Analysis of subjects’ ratings during visual familiarization.

(A) Balancing duration and (B) average absolute pole angle as function of subjects’ ratings. Data shown are the averages (bar) ± 1 S.E.M. expressing the between-subject variability. (C) Predicted rating of the fitted multinomial log-linear model as function of balancing duration and average absolute pole angle fitted to subjects’ responses. The model prediction was sampled uniformly in both dimensions for visualization. Opacity indicates whether training samples are close. Notice, that the rating (color) depends on both parameters, balancing duration and average absolute pole angle. *** p<0.001, n.s. p>0.1.

The influence of the two parameters (BD, maPA) on the rating was examined using a multinomial log-linear model (see Data Processing and Analysis). Both parameters and their interaction contributed significantly to the classification (all p<0.01). Fig 3C shows the predicted rating of the fitted model as function of the two parameters. Notice that the classification depends on both parameters, which is evident by the diagonal boundaries of the classes. While the balancing duration can easily be interfered at the end of the presented balancing attempt, inferring the mean absolute pole angle requires a more attentive observation of the trial. Thus, our results suggest that subjects in the group VF paid attention to the presentation of each balancing attempt and adhered to a non-trivial set of rules, based on the trial length and mean absolute pole angle, for classifying the balancing attempts into meaningful classes.

Accuracy in extrapolation does not improve without feedback

For the pole extrapolation task, we first inspected the prediction error across blocks visually. Subjects of both groups show in average negative errors, which corresponds to a systematic underestimation of the pole movement (Fig 4A). This suggests that extrapolating the state-dependent acceleration of the pole is even for motor familiar subjects difficult. Furthermore, we noticed, that the variability between subjects in the group MF seems to be higher than for group VF. A possible reason could be a correlation between the extrapolation performance and the balancing proficiency, which varied between subjects although all motor familiar subjects were able to balance the cart-pole system (see S1 Appendix). However, we did not find any significant correlation between the average balancing duration in the motor control blocks and the prediction error in the subsequent extrapolation task block (for all blocks p>0.14, Spearman’s rank correlation). We did not find any significant correlation between the time elapsed since initial skill acquisition and the prediction error (for all blocks p>0.4, Spearman’s rank correlation). Furthermore, the between-subject variability was similar for both groups over the last blocks (F1-F5).

Fig 4. Average prediction error for both groups.

(A) Average prediction error of the motor familiar (MF) and visually familiar (VF) group across all examined blocks. Negative errors correspond to an underestimation of the pole downwards movement. Notice that neither of the two groups improves significantly over the blocks B and T1-T4. Feedback is only provided during the blocks F1-F5. (B) Prediction error of both groups in the blocks T4 (light gray) and F5 (dark gray). Within each group, subjects improve significantly due to feedback. There is a significant difference between the two groups before feedback was provided (T4). The prediction error is however not significantly different between groups after feedback was provided (F5). ** p<0.01, *** p<0.001. Error bars indicate ±1 S.E.M. expressing the between-subject variability.

Before comparing groups, we examined whether there is any improvement in extrapolating the pole angle across the blocks without feedback (Fig 4A), which would suggest that subjects improve in pole extrapolation due to the visual familiarization (group VF) or motor control (group MF) task. Specifically, we conducted for each group a mixed-effects ANVOA that examined the effect of block on the prediction error over the blocks B and T1-T4. There was no significant effect of block in any of the groups (MF: p = 0.87, VF: p = 0.95). Hence, the ability to predict the pole angle does not improve without task specific feedback.

Motor familiar subjects predict the pole angle more accurately

In order to test our hypothesis, that motor familiar subjects show a smaller prediction error than subjects without motor expertise, we compared the prediction error between the groups just before any task specific feedback was provided (T4) and at the end of all feedback blocks (F5). To this end, a mixed-effect ANOVA was conducted that examined the effect of feedback (see Data Analysis) and group on the prediction error (Fig 4B). Both effects reached significance (group: p<0.02, feedback: p<0.001). The interaction did not reach significance (p = 0.07). Post-hoc tests revealed a significant difference between groups before feedback was provided (p<0.01, Holm corrected, mean MF: -9.6°, VF: -14.9°) but not after (p = 0.22, Holm corrected, mean MF: -5.2°, VF: -7.5°). Post-hoc comparison of the two blocks (T4: no feedback vs. F5: feedback) revealed a significant improvement within groups (both p<0.001). In summary, this analysis revealed a significant difference between the groups MF and VF before task-specific feedback was provided. Thus, motor familiar subjects can predict the pole movement more accurately than subjects can, who have no motor expertise. Subjects in both groups are however able to utilize the feedback about the correct pole orientation for improvement, resulting in statistically indistinguishable extrapolation performance in the last block (F5).

Larger extrapolation horizon accounts for better extrapolation performance

We also investigated the average prediction error of different response models for extrapolating the pole angle during the occlusion (see Methods). Visual inspection reveals that, on average, both groups performed worse than a response model that accurately predicts the pole movement (PERF) or than assuming constant acceleration (cACC, Fig 5). For a more detailed analysis, we fitted subjects’ individual responses for every trial in the block T4 using our model class lh_cVEL, which extrapolates the state over the first h milliseconds (extrapolation horizon) accurately and then assumes constant pole velocity for the remaining time of the occlusion (see Methods and S3 Appendix). This model mimics the potentially limited capability to predict accurately over longer periods. Notice that the average prediction error of the model with h set to the average (or median) h* of one group deviates from the average prediction error of the corresponding group. This is because individual responses are fitted and not the group average (or median). In comparison to the subjects in the group VF, the behavior of the subjects in the group MF was described by a significantly larger extrapolation horizon (parameter h*, Fig 6, p = 0.026, Wilcoxon rank sum test, median MF: 183.33ms, VF: 16.66ms). Thus, the subjects in the group MF seem to possess a more accurate, although not perfect, internal representation of the pole dynamics, which accounts better for the angle-depended acceleration of the pole.

Fig 5. Average prediction errors for the investigated response models.

In addition to the average prediction error for each model, the average prediction errors of subjects in the groups MF (light blue) and VF (orange) are shown for visual comparison. The three highlighted models (red: PERF, magenta: cACC, blue: cVEL) correspond to common assumptions in model-based extrapolation of motion (perfect model, constant acceleration and constant velocity). The average prediction error of each model in the class of partly heuristic forward models (lh_cVEL) is plotted as gray line, where h denotes the extrapolation horizon of the model. Notice that h = 0ms and h = 900ms coincide with the models cVEL and PERF. The model cACC is not in the class lh_cVEL.

Fig 6. Comparison of the two groups based on the model lh_cVEL and parameter h*.

Inter-quartile range (box) and median (dot) of the fitted h* for the motor familiar (MF) and visually familiar (VF) subjects in block T4. The median h* for subjects in the group MF is significantly higher and therefore closer to the perfect model (h = 900ms).

Discussion

We compared human’s ability to extrapolate the pole angle of the cart-pole system between subjects with and without motor expertise in the cart-pole balancing task. Subjects without motor expertise rated balancing attempts of other subjects (visual familiarization) in dedicated blocks during the experiment, while motor familiar subjects instead performed again the balancing task during this time as they have done previously during skill acquisition. During the extrapolation task, subjects observed balancing attempts and indicated for each attempt where they expected the pole to be after the pole was occluded for 900ms. Our analysis has revealed that neither of the two groups improves without receiving feedback about the true pole angle (Fig 4). Subjects with motor expertise were, however, even before receiving feedback significantly more accurate in extrapolating the pole angle, although both groups received similar visual information during the visual familiarization and motor control blocks. We examined and explained the higher accuracy of motor familiar subjects in terms of exploiting a more accurate model of the pole dynamics, especially regarding the influence of the angle dependent gravitational acceleration of the pole and the ability to extrapolate this influence over a long duration accurately (extrapolation horizon, Fig 5). Subjects of both groups improved significantly in terms of minimizing the prediction error by using the feedback about the true pole angle in the last phase of the experiment (F1-F5). Finally, at the end of the experiment (F5), the extrapolation performance of the two groups was statistically indistinguishable.

La Scaleia et al. [10] argued that subjects incorporate prior experiences about gravity and air drag when intercepting ball trajectories in the form of model-based control. Since we are all used to the motion and behavior of objects during free fall, we already possess the knowledge that is necessary to catch falling objects. Similarly, we are used to human motion in the sense that we are able to interpret and extrapolate it [18]. It is important to note, that we usually acquire this ability and the corresponding knowledge implicitly and only rarely use it explicitly, meaning we use it for performing actions instead of expressing our expectations verbally. Nevertheless, subjects were able to discriminate plausible from implausible future body postures and to report their decisions [18]. Although the way subjects responded in the two studies differed substantially (actual catching vs. binary discrimination), the concept of internal simulation of the motion plays a central role for both studies in explaining subjects’ performances. These internal simulations of motion and the corresponding neural representations are arguably formed already during our childhood as result of throwing and catching balls, observing other people move as well as moving ourselves. During the training of specific tasks, such as playing basketball, these internal simulations are refined and specialized, which enables, for example, elite basketball players to outperform less experienced individuals in predicting the binary outcome of free shots [19]. Similarly, subjects with motor expertise in the virtual cart-pole balancing task were in our experiment more accurate in predicting the pole angle than subjects without motor expertise. However, in comparison to Aglioti et al.’s experiment, the most important difference is that the cart-pole extrapolation task required subjects to indicate the expected pole angle as accurately as possible instead of making only binary decisions (outcome of free shot). Thus, the underlying internal simulation has to fulfil substantially more restrictive accuracy requirements. In other words, the extrapolation of the pole angle has to be very accurate (and precise) in order to be able to discriminate between the diverse response options reliably. According to the internal simulation hypothesis [23], the accuracy of the extrapolation is determined by the accuracy of the internal forward model that represents the cart-pole system. We found that the responses of subjects with motor expertise were more similar to those of a perfect model than the responses of subjects without motor expertise (larger extrapolation horizon, Fig 6). Thus, motor familiar subjects seem to have used a more accurate model for extrapolation. However, we also found that even motor familiar subjects were not able to extrapolate the motion without flaws. In fact, our model-based analysis suggested that subjects only extrapolated for about 183ms accurately (group MF in block T4, Fig 6) before they switched to the constant velocity assumption. Similarly, Mehta and Schaal [14] reported that subjects show an average visuomotor delay of 269ms during virtual pole balancing and that trained subjects can tolerate occlusions of 500-600ms duration without losing balance. Regarding the last point, it should however be noted that balancing the pole might be possible without being able to extrapolate the pole movement perfectly over the whole duration of the occlusion. In fact, since small mistakes during balancing can, to a certain degree, be corrected for later, subjects might acquiesce imperfections in order to reduce the effort for extrapolating more accurately. Nevertheless, when being confronted with the error in the extrapolation task by receiving feedback about the true pole angle, subjects used the additional task specific information to improve in the extrapolation task.

Intriguingly, motor familiar subjects could not only use the forward model, which they acquired during motor training, for performing actions predictively [16] but also to extrapolate and report the state of the system accurately, suggesting that information between action and perception is shared. Hecht et al. [24] have investigated the transfer of information and the relation between acting and perceiving in more detail. In their experiment subjects either had to rate (perceptual task) or perform (motor task) two subsequent movements that exhibited a certain relative timing. By permuting the sequence of tasks across groups, they investigated the transfer of knowledge from action to perception (action-perception transfer, APT) and vice versa (perception-action transfer, PAT). In comparison to a control group, both test groups were better in the second task, suggesting that information is transferred in both directions. Another interpretation is that for the perceptual and motor tasks the same representation is used and thereby the information is shared [25,26].

Recent research in the field of sensorimotor control addressed the topic of shared representations for action and perception from a slightly different perspective. Instead of acquiring a new motor skill, researchers investigated how the adaptation of motor behavior influences the perception. Sensorimotor adaptation is commonly explained by the adaptation of internal forward models that map actions to their sensory consequences. By minimizing the sensory prediction error (difference between predicted and actual outcome), predictions about the sensory consequences of actions are maintained accurate [27]. When the sensory consequences are artificially biased, for example due to a visuomotor rotation in virtual reality [27,28], the sensorimotor map is perturbed. Intriguingly, after having adapted to such perturbation as result of the minimization of the sensory prediction error, not only the subsequent motor execution is biased but also the perceived sensory consequences are biased in comparison to reality. Indeed, for classical visuomotor rotation paradigms, it has been shown that healthy subjects perceive their hand at a different position after sensorimotor adaptation and that the perceived hand position corresponds to the visuomotor rotation [29]. These studies suggest that extrapolation mechanisms for motor control and perception are not only tightly related but might also share the same internal forward models. Regarding the neural correlates of this behavioral finding, Synofzik et al. [29] have shown that not only the adaptation of motor behavior [17] but that also the shift in perception depends on the integrity of the cerebellum. Especially the posterior-lateral cerebellum seems to be important not only for the planning of motor actions [17,30] but also for the perception of action [31,32] and sensory prediction [5–9,33,34]. It seems therefore plausible that the cerebellum implements mechanisms, which are equally used for perception and action. Whether the cerebellum and other brain areas, such as the parietal lobe [35–39], are involved in the cart-pole balancing and extrapolation tasks, and whether there are different neural correlates depending on the preceding type of training (visual vs. motor), is an interesting direction for future research.

Limitations of the study

Finally, we would like to address some limitations of this study and elaborate why these limitations do not contradict our conclusions. First, motor familiar subjects interacted overall for a longer duration with the cart-pole system due to the prior skill acquisition that lasted 90 minutes. For acquiring the cart-pole balancing skill, vision is essential. Thus, motor familiar subjects were presumably also visually more familiar with the dynamics of the cart-pole system. This consideration limits the interpretability of our data in terms of the difference between visual and motor familiarity. However, our main hypothesis was that subjects with motor expertise (in terms of being able to balance the pole on the cart) could extrapolate the pole motion more accurately than subjects without motor expertise. The visual familiarization that we provided could only have reduced (if at all) the found contrast between the motor experienced and unexperienced group.

Second, our visual familiarization task consisted of observing and rating balancing attempts of others. Observing and rating balancing attempts might be a less engaging task in terms of acquiring detailed knowledge about the system dynamics than actively balancing the cart-pole system. However, the blocks of visual familiarization and extrapolation tasks were alternated (T1-T4, see Fig 1A), which provided the subjects with the opportunity to extract specific information about the system dynamics from a visual familiarization block and use it in the next extrapolation block. Since subjects were asked to perform in the extrapolation task as good as possible, they also had the incentive to exploit the order of the tasks in this way and to be more engaged during the visual familiarization blocks. Thus, it is hard to judge whether there was a difference in engagement to learn about the dynamics between the two groups. Nevertheless, our conclusion that motor experienced subjects can extrapolate the pole motion more accurately than motor unexperienced subjects is not contradicted or limited by this consideration.

Third, we investigated the difference in pole extrapolation accuracy between motor experienced and unexperienced subjects only in a virtual reality task. There is a considerable difference between performing a task in virtual reality or real-life, especially when the task requires to control, observe or predict the behavior of a dynamic system that is simulated with limited temporal resolution and represents only a model of a real system. Thus, although we motivated the topic more generally in the introduction, we can only draw conclusions about human behavior in this virtual reality task.

In summary, none of these limitations invalidates or contradicts our main hypothesis, which is that motor expertise in performing the cart-pole balancing task facilitates the accuracy of extrapolating the pole movement in virtual reality. Furthermore, our results are in line with other studies, which revealed a general behavioral difference between visual or motor familiar subjects [19,24]. We add to these reports by showing that the differences in extrapolation performance between motor experienced and unexperienced subjects are very specific. Our model-driven analysis additionally characterizes the individual extrapolation performances and explains the resulting difference between the groups in terms of a limited extrapolation horizon.

Conclusions and outlook

In conclusion, we have shown that motor training does not only improve motor control but also specific task-relevant perceptual abilities in a virtual reality task. Being able to extrapolate the state of the virtual cart-pole system into the future is an important ability for easeful balancing performance. Correspondingly, subjects who had previously learned to control the virtual cart-pole system were more accurate in extrapolating the pole movement than subjects who had no motor expertise in the cart-pole balancing task. Our results suggest that motor training yields an accurate, although not perfect, internal forward model of the controlled dynamics, which can be used for both, controlling and accurately extrapolating the dynamic behavior of the cart-pole system. Similar extrapolation accuracy can, however, also be achieved without motor training by minimizing the extrapolation error when task-specific feedback is provided.

Our study motivates multiple future research directions. First, since our model explained the difference in extrapolation performance based on the duration of accurate internal simulation, a future study could test this prediction explicitly by varying the duration of the occlusion. Secondly, with the acquired data we could only investigate the accuracy of the extrapolation mechanism. Evaluating the variability of subjects’ responses within the blocks would provide valuable insights about the confidence and precision of the extrapolation. Another interesting question, inspired by Hecht et al. [24], is to investigate whether the ability to extrapolate the pole movement (even more) accurately after performing the pole extrapolation task, improves the cart-pole balancing skill or facilitates its acquisition (perception-action transfer). Moreover, investigating the transfer of acquired knowledge in the virtual cart-pole balancing task to a corresponding real life task is another interesting topic not only for basic neuroscience research but also for the training of professional skills (see also [40]). Finally, identifying and comparing the neural correlates corresponding to virtual cart-pole balancing and extrapolating the pole movement might foster the understanding of knowledge representations in the brain.

Supporting information

S1 File. Behavioral data.

(ZIP)

S1 Appendix. Cart-Pole balancing performance of motor familiar subjects.

(PDF)

S2 Appendix. Comparison of visual information between the motor and visual groups.

(PDF)

S3 Appendix. Response models with limited extrapolation horizon.

(PDF)

Data Availability

All relevant data are within the paper and its Supporting Information files.

Funding Statement

NL would like to thank the German National Academic Foundation for granting a doctoral student fellowship. Financial support for this study has been received from the German Research Foundation (DFG GZ: KA 1258/15‐1), European Union Seventh Framework Programme (CogIMon H2020 ICT‐ 644727) and the Human Frontiers Science Program (HFSP RGP0036/2016). We acknowledge the support by the Deutsche Forschungsgemeinschaft and Open Access Publishing Fund of the University of Tübingen. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.