The role of feedback in the production of skilled finger sequences

Abstract

Actions involving fine control of the hand, for example, grasping an object, rely heavily on sensory information from the fingertips. Although the integration of feedback during the execution of individual movements is well understood, less is known about the use of sensory feedback in the control of skilled movement sequences. To address this gap, we trained participants to produce sequences of finger movements on a keyboard-like device over a 4-day training period. Participants received haptic, visual, and auditory feedback indicating the occurrence of each finger press. We then either transiently delayed or advanced the feedback for a single press by a small amount of time (30 or 60 ms). We observed that participants rapidly adjusted their ongoing finger press by either accelerating or prolonging the ongoing press, in accordance with the direction of the perturbation. Furthermore, we could show that this rapid behavioral modulation was driven by haptic feedback. Although these feedback-driven adjustments reduced in size with practice, they were still clearly present at the end of training. In contrast to the directionally specific effect we observed on the perturbed press, a feedback perturbation resulted in a delayed onset of the subsequent presses irrespective of perturbation direction or feedback modality. This observation is consistent with a hierarchical organization of even very skilled and fast movement sequences, with different levels reacting distinctly to sensory perturbations.

NEW & NOTEWORTHY Sensory feedback is important during the execution of a movement. However, little is known about how sensory feedback is used during the production of movement sequences. Here, we show two distinct feedback processes in the execution of fast finger movement sequences. By transiently delaying or advancing the feedback of a single press within a sequence, we observed a directionally specific effect on the perturbed press and a directionally non-specific effect on the subsequent presses.

INTRODUCTION

Most motor behaviors strongly depend on feedback. When we grasp a full cup and feel a sudden slip, we can swiftly adjust our grip force to avoid the cup slipping from our hand. This correction can occur in less than 100 ms (1–3). Feedback from other senses such as vision (4, 5) and audition (6, 7) is also used for the control of an ongoing movement, albeit at slightly slower speeds (at 90–260 ms and 100–200 ms, respectively). For most movements, the integration of sensory feedback is essential for accurate movement production (8).

Although much is known about the rapid sensory feedback integration during the execution of individual movements (for reviews see Refs. 9–11), little is known about the integration of sensory feedback during the execution of sequences of finger movements. Previous studies investigating this topic have primarily focused on tasks in which participants were asked to synchronize their movements with an external pacing tone (12–17). However, studies investigating the role of sensory feedback in tasks in which participants execute finger movements as fast as possible are scarce (18, 19). Moreover, the majority of studies investigating this topic have focused on perturbing the slower visual or auditory feedback channels. Hence, these studies were unable to examine the full range of rapid feedback adjustments that are possible during a finger press.

It is commonly thought that longer movement sequences are controlled in a hierarchical manner (20–23), with one or more abstract layers representing the sequence information, and a lower-level layer generating the detailed muscle commands. At least early in training, it is thought that each finger press is controlled and triggered separately. With training, however, finger movements start to overlap each other in time. This process is often interpreted as the formation of movement chunks (24–27), small groups of finger movements that are represented as a single unit, and therefore, only need to be triggered once by a single higher-level controller.

How sensory feedback is used across these different hierarchical layers, however, is unknown (28). Feedback from the finger should first have an effect on the immediate control of that finger movement (Fig. 1A, dashed line). The same sensory information may also have a separate effect on the higher-level sequence controller (Fig. 1A, dotted line) and therefore will either delay or accelerate the next key press. In contrast, if with training, overlapping finger presses are being controlled as a single movement chunk, the entire chunk should be affected exactly in the same way as the ongoing press.

Figure 1.

Two hypothetical representations of sensory feedback in fast motor sequences. A: a hierarchical controller that represents the movement sequence across multiple interacting layers. The sequence controller represents the sequences of finger presses and activates the corresponding finger controllers, which are then responsible for generating the muscle commands for moving the finger. Sensory feedback (dashed line) goes back both to the relevant finger controller and to the overall sequence controller. B: after training, neighboring finger presses may be represented as a movement chunk, which generates the muscle commands for multiple, overlapping finger presses. Sensory feedback would lead to slowing or acceleration of the execution of the entire movement chunk.

To explore these possibilities, we trained participants to execute fast finger sequences for 4 days. Each day, we probed how sensory feedback is used during sequence execution, by either delaying or advancing the sensory feedback (haptic, visual, and auditory together) on a single press within a sequence. We then studied how this perturbation impacted the control of the ongoing finger press, as well as the execution of subsequent presses.

METHODS

Participants

Twenty-six participants were recruited for this study [11 males; ages 18–44 yr; mean age 25.5 (±7.25)]. All participants were right-handed (self-declared) and completed informed consent. On average, participants had received 6.44 (±7.25) year of musical training based on their longest played instrument, with 57% having at least 1 year of piano playing experience. The study protocol was approved by the ethics board of the University of Western Ontario, and all participants gave their signed consent before starting the study.

Apparatus

To test participants, we used a custom-built five-finger keyboard (Fig. 2A). The keys were not depressible but a force transducer (FSG-15N1A, Sensing and Control, Honeywell) was mounted underneath each key measuring isometric force production with a repeatability of <0.02 N and a dynamic range of 16 N (29–31). The digital sampling rate of the measured force was 200 Hz. In addition, each key was equipped with a linear resonant actuator (LRA, LVM061930B-L20, Jinlong Machinery & Electronics, Inc.) that provided haptic feedback during the experiment. LRAs vibrate at a frequency between 200 and 250 Hz. In our application, a haptic controller creates a specific waveform to elicit the click sensation. The haptic stimulation was produced by a haptic motor controller (DRV2605L, Adafruit Industries LLC) that produces a computer-controlled click/vibratory sensation that feels similar to the sensation experienced from smartphone keys or trackpads on laptops (see the DRV2605L data set for more information regarding the specific waveform).

Figure 2.

Apparatus and achieved time advancements of feedback. A: isometric keyboard-like device. Each key was associated with a number (these numbers were not shown to the participants but verbally explained). B: histogram of the time intervals between feedback presentation and the actual press onset for the two advancement conditions. Vertical doted lines indicate −30 and −60 ms. The delay conditions could always be achieved accurately.

Discrete Sequence Production Task

Participants performed a discrete sequence production task (DSP), executing sequences of 11 keypresses as fast and as accurately as possible. Participants were instructed to move as fast as possible while maintaining an error rate of under 15% for each block of trials. Each finger was associated with a number (thumb = 1, index = 2, middle = 3, ring = 4, and little = 5). Each trial began with the presentation of a sequence of numbers on a computer screen (white font). A trial was completed after 11 finger presses were executed. The numbers stayed on the screen throughout execution. Participants were trained on three fixed sequences and none of the sequences had directly repeating numbers (i.e., 33 or 44). The same three sequences were used for all participants; however, the presentation order was randomized across trials separately for each participant. Each block consisted of 39 trials and each sequence was presented 13 times during a block.

The force magnitude applied to each key by the participant was displayed as five lines on an LCD monitor, where each line height indicated the amount of force applied to the corresponding key. When the force on a key exceeded 1.5 N, the keypress was registered and the feedback was triggered. Some coarticulation between fingers emerged as the next key could be pressed before the previous key was released.

When participants pressed the correct key, the visual cue on the screen turned green, a short pleasant auditory sound could be heard (each key was assigned a specific tone that was different from the rest), and a small click could be felt on the finger, which was generated by the resonant actuator. All three types of sensory feedback were presented concurrently to the participant. We used the following note for each key: thumb = A, index = C, middle = D, ring = E, and little = G. If an incorrect key was pressed, the visual cue changed to red, a lower-pitch sound could be heard (same across keys), and a click (same for accurate and incorrect press) could be felt.

For each completed trial, participants received points based on their performance. If the participant pressed all keys correctly and their median movement speed (MT – the time between the first press and last release) was within 95% to 110% of the current speed threshold (MT threshold), they received one point. If they correctly executed the sequence and their median movement speed was faster than 95% of the current MT threshold they received three points. If they pressed one or multiple keys incorrectly or their median speed was slower than 110% of their MT threshold, they received zero points. At the end of a block, we provided participants with feedback regarding their error rate, median speed (MT), points obtained for the current block, and total points obtained across the session. To motivate participants to improve their performance throughout the sessions, we first set the MT threshold at 10 s at the beginning of each session and then adjusted it by lowering it to the median MT of a given block if the participant had a lower median MT compared with the current MT threshold and if their error rate was below 15%.

Feedback Manipulation

The first three blocks in each session were completely unperturbed, meaning no feedback perturbation was presented. In each of the following blocks, we perturbed 24 trials out of the 39 trials. Participants completed a total of 74 blocks over the four days of training. On these perturbation trials, we either advanced or delayed the feedback by 30 or 60 ms on one of the 11 key presses. For each perturbation, haptic, visual, and auditory feedback were shifted by the same amount. To generalize our findings across fingers and press location within the sequence, we chose two fixed positions within each sequence where feedback perturbations were given. This also reduced the potential predictability of the perturbation location in each sequence. In sequence 1, we gave the feedback perturbation either at position 6 (finger 5) or position 9 (finger 4), in sequence 2 at positions 4 (finger 2) or position 7 (finger 1), and in sequence 3 at positions 5 (finger 4) or position 8 (finger 3). In total, we presented the perturbation at six different sequence positions across all sequences.

For the advanced feedback conditions, we used a real-time algorithm that predicted when the press onset (the time at which the force on the key exceeded 1.5 N) of the critical finger press would occur. This prediction was updated in real-time every 2 ms during trial execution and was used to deliver the feedback either 30 or 60 ms before the actual press. The prediction was based on three factors: the current force, the current force change (numerical derivative based on three time points), and the time since the last press onset. We separately trained this predictive model for each subject, sequence position, and delay condition (−30 or −60 ms) using a logistic regression. This was done twice in each session. The first time, we fit the model on the data from the first three blocks, using the unperturbed trials as training data. To account for speed changes during the session, we repeated the estimation in the middle of the session based on the unperturbed trials of all previous blocks (excluding the three initial blocks and at least 6 blocks of trials). The predicted outcome variable was zero if it was too early to present feedback and one if it was too late. Feedback was provided once the predicted probability exceeded 0.5. This approach led to an average time advancement of 29.3 ms (SD: 11.4 ms) for the –30 ms advancement condition and an average of 57.9 ms (SD: 23.3 ms) advancement for the −60 ms condition (see Fig. 2B).

On the advanced trials, participants could press the next key as soon as the feedback was presented on the current press, meaning they were allowed to press the next key before reaching the press threshold for the perturbed press. This led to an average of 2.36% (SD: 1.55%) of the advanced trials not reaching the press threshold. We excluded these trials from our analyses.

In the delay conditions, feedback was withheld upon reaching the press threshold, and instead presented 30 or 60 ms after press onset. However, in the delay conditions, participants were not required to wait for the feedback to be presented before moving on to press the next press. This was important as participants did not have to take the feedback perturbation into account and could potentially perform the sequences just as fast as when no perturbation was present.

Experimental Procedure

Participants completed four sessions that lasted ∼1.5 h each depending on how fast the participant was able to complete the required blocks of trials. Participants completed one session per day, and the four sessions were scheduled over a time span of approximately 2 wk. We encouraged participants to take breaks between blocks, as necessary, and offered a longer break in the middle of the experimental session. The participants were told that the goal was to perform the sequences as accurately and fast as possible. At the end of the four sessions, we asked participants several questions about their experience that became more and more specific (see appendix). This questionnaire was used to determine whether participants were conscious of the experimental manipulation. Only two participants expressed clear conscious knowledge of the experimental manipulation, whereas the rest of the participants did not notice the manipulation. The performance of these two participants was similar to the performance of the other participants and therefore were not excluded from the analyses. Overall, the majority of participants were not consciously aware of our experimental manipulation, and hence we believe that they did not change their behavior consciously.

Statistical Analysis

Our analyses focused on the intervals between the onset of the perturbed press and specific press time points on the perturbed and subsequent finger presses. The five landmarks (Fig. 3A) were: early onset (EO; when force first was great or equal to 0.75 N), onset (O; when force first was equal or exceeded 1.5 N), peak (P; time at highest force – between onset and late release), release (R; when the force first fell under 1.5 N after peak), and late release (LR; when force first fell under 0.75 N after onset). All analyses were done relative to the onset of the perturbed press (or for unperturbed trials, the matching unperturbed press in the same sequence). We analyzed the relative timing of the landmarks on the perturbed press (+0), and the two presses after the perturbed press (+1 and +2). In addition, we calculated the overall movement speed (movement time/MT) between the onset of the first press (first time it reached the press threshold) and the release of the last press (force fell below 1 N).

Figure 3.

Calculation of feedback differences across presses and landmarks. A: for our analyses we calculated time intervals between the onset of the perturbed press (blue onset dot in the figure) and different force landmarks (green dots) on the perturbed press as well as on subsequent presses (indicated with +1). We chose five specific time landmarks on each press: early onset (≥0.75 N), onset (≥1.5 N), peak (maximum force between onset and release), release (first time <1.5 N after onset), and late release (first time <0.75 after onset). B: for the calculation of the offset, we choose a single time point (onset of +1 press) and found the corresponding force level on the perturbed press for unperturbed trials (black line). For perturbed trials (red line), we then located the same landmarks. The offset is defined as the difference between the estimated delay on the next press (interval 1) and the estimated simultaneous delay on the perturbed press (interval 2).

All analyses were performed using custom-written code in MATLAB (The MathWorks) and the dataframe toolbox (github.com/jdiedrichsen/dataframe). We excluded any error trials from our analyses, as well as trials in which the press was delayed by more than 100 ms after the advanced feedback was given, as we believe that this could either suggest conscious awareness or an incorrect estimation from our algorithm that predicts when feedback should be given. We analyzed the data using paired one- and two-sample t tests that were based on clear a priori predictions, and we chose a probability threshold of P < 0.05 for the rejection of the null hypothesis.

To estimate how quickly participants reacted to the delayed feedback by adjusting the perturbed press, we conducted a change point analysis, separately for each day. We first calculated the difference between the average force curves for the delayed trials (+30 ms or +60 ms) and unperturbed trials from 20 ms before press onset and 240 ms after onset. Using the data before the occurrence of the peak difference between the two curves, we estimated the time point when the difference started to emerge. We modeled the difference as a piece-wise linear function with a change point of b₀ between the two segments:

$$\widehat{\text{y}}(\text{t})=\{\begin{array}{l}0,\text{t}<{\text{b}}_0\\ (\text{t}-{\text{b}}_0){\text{b}}_{1,},\text{t}\ge {\text{b}}_0\end{array},$$

where ŷ(t) is the predicted force values for time t, b₀ is the chosen change point, and b₁ is the slope of the function. Using the function fminsearch in MATLAB, we found the values for b₀ and b₁ that minimized the sum of squares of the difference between observed and fitted data.

If overlapping presses are controlled as a single unit (Fig. 1B), the perturbed press (+0) and the press following the perturbed press (+1) should be delayed or accelerated (relative to an average unperturbed press) by the same amount (Fig. 3B). That is, if the release of the perturbed press is delayed, the simultaneously occurring onset of the next press should be delayed by the same amount. To test this idea, we used the onset of the +1 press and identified on the force level that the +0 press had reached on the average unperturbed trial at that moment. We then identified this matching landmark on the perturbed trials. If the two presses were controlled as a single unit, both landmarks should be delayed by the same amount of time. In contrast, a difference in delay (positive or negative) between the +1 press and the +0 press would indicate that the effect of the perturbed feedback was not the same for the two presses.

Control Experiment

In a separate experiment, we probed to what degree the modality of the sensory feedback (auditory, haptic, and visual) had differential effects on participants’ performance. We recruited 48 participants for this experiment. They were assigned to one of the three feedback groups (auditory, haptic, or visual) at the beginning of training based on an algorithm that matched participants’ speed, calculated as the time between the onset of the first press to the release of the last press (MT). This was done to ensure that the groups had similar average speeds at the start of the experiment. Participants only received one type of feedback throughout the study (how each feedback was given was the same as described in the Experimental procedure earlier). When an incorrect finger press occurred, all groups saw the visual cue on the screen turn red to make it easier for them to know where they made the error in the sequence. Participants practiced four different sequences (three were the same as in the main experiment) for 5 days on the same keyboard-like device. Press threshold was 1 N. Because of the difference in press threshold, we adjusted our landmark criteria for this experiment: early onset (EO; when force first was great or equal to 0.6 N), onset (O; when force first was equal or exceeded 1 N), peak (P; time at highest force – between onset and late release), release (R; when the force first fell under 1 N after peak), and late release (LR; when force first fell under 0.6 N after onset). Feedback perturbations were given on a single press within the sequence at two possible locations (similar to the main experiment but the locations were not identical). In this experiment, we only perturbed participants’ feedback by delaying it by 80 ms. The rest of the experimental design was identical to the main experiment (point system, threshold change, etc.). As in the main experiment, most participants were unaware of the perturbation when asked about it using a questionnaire at the end of the sessions.

RESULTS

Feedback Perturbations Cause Directionally Specific Behavioral Adjustments to the Perturbed Finger Press

To investigate how sensory feedback is used during the execution of fast finger sequences, we used transient perturbations of the sensory feedback that indicated the successful pressing of a key. The perturbation was only applied to a single press within a sequence. Participants practiced three different sequences over 4 days. If sensory feedback is used to control the near-isometric keypress, the delay and advancements of feedback should prolong or shorten the ongoing press, respectively.

The group average force traces (Fig. 4A) indicated that even though each finger press was completed within ∼300 ms, participants indeed reacted to the feedback perturbation by extending or shortening the ongoing press. To quantify this effect, we calculated the time interval between the onset (first time ≥1.5 N is reached) and the peak of the perturbed press (Fig. 4B, onset-peak), as well as the interval between the onset and the release (first time <1.5 N after onset; Fig. 4B, onset-release). On day 1, both the +30 ms (t₂₅ = 11.189, P = 1.59e-11) and the +60 ms delay condition (t₂₅ =4.969, P = 2.02e-05) resulted in a longer onset-peak intervals. Similar effects can also be seen on the onset-release interval (+30 ms: t₂₅ = 6.630, P = 3.01e-07, +60 ms: t₂₅ = 5.963, P = 1.58e-06). For the advanced feedback conditions, the onset-release intervals on day 1 were shortened in response to perturbations (onset-release −30 ms: t₂₅ = 5.308, P = 8.42e-06; −60 ms: t₂₅ = 4.291, P = 3.78e-10). These results suggest participants used sensory feedback indicating the successful pressing of a key to finely control the duration of the force production.

Figure 4.

Effects of perturbation on perturbed and subsequent press. A and C: average force traces for day 1 and day 4 and the following press interpolated and standardized to the average time of each condition. Dotted line indicates press onset, for which the sensory feedback was shifted in time. Error bars represent the mean ± standard error across subjects. B and D: differences between the onset-to-peak and onset-to-release intervals of perturbed and unperturbed trials for day 1 and day 4.

Perturbation Effects Diminish but Do Not Disappear with Training

Does feedback control still play a role in movement execution when the finger movements are fast and extensively practiced? Our results indicated that this was indeed the case (Fig. 4A vs. Fig. 4C). On day 4 (Fig. 4D), both delay conditions showed larger onset-peak intervals (+30 ms: t₂₅ = 5.963, P = 1.17e-04; +60 ms: t₂₅ =6.420, P = 5.05e-07) and onset-release intervals (+30 ms: t₂₅ = 6.143, P = 1.01e-06, +60 ms: t₂₅ = 5.082, P = 1.51e-05) compared with the unperturbed condition. Similarly, shorter onset-release intervals were observed for the advancement conditions (onset-release −30 ms: t₂₅ = 3.774, P = 4.46e-04, −60 ms: t₂₅ = 4.785, P = 3.26e-05). The finding of a clear adjustment of the perturbed press at the end training suggests that even finger presses performed in the context of well-practiced sequences are controlled using sensory feedback.

While the overall effect was clearly present across all days, the effect caused by the large perturbations reduced by ∼40%. Specifically, the difference between perturbed and unperturbed onset-release interval reduced from day 1 to day 4 for the +60 ms (−38%, t₂₅ = 2.502, P = 0.019) and the −60ms condition (−40%; t₂₅ = −3.859, P = 7.106e-04). Although the overall effect also reduced for the smaller perturbations, these changes were not significant (+30 ms: −29%, t₂₅ = 1.848, P = 0.076; −30 ms: −35%, t₂₅ = −1.639, P = 0.113). This suggests that some transition from feedback to feed-forward control took place in our task with practice.

Perturbations Lead to Reactions within 80 ms

To determine the speed with which sensory feedback is used to control an ongoing finger press, we calculated a difference curve between the average force traces of the delayed perturbation conditions and the unperturbed condition for each participant. We then used a change point analysis (see methods for details) to estimate the time at which the difference curve was impacted by the feedback delay. On day 1 in the +60 ms delay condition, it took an average of 106.4 ms (95% confidence interval [97, 115]) after press onset (and absence of the expected feedback) for participants to show a divergence between the two force traces. For the +30 ms delay condition, the difference started at 77.3 ms [65, 90]. For day 4, our estimate of adjustment onset for the +60 ms condition was 92.5 ms [83, 102], faster than day 1 (t₂₅ = 2.085, P = 0.047). The estimate for the +30 ms condition was comparable with day 1 (67.5 ms [46, 89]; t₂₅ = 0.738, P = 0.467).

For the advancement conditions, we estimated the onset of the adjustment relative to the time of the (unexpected) feedback. This analysis is slightly more difficult as the time between feedback and the onset of the press varies on a trial-by-trial basis (Fig. 2B). Nonetheless, we found similar adjustment onset times: For day 1, the adjustment occurred at 88.3 ms [80, 96] for the −60 and at 116.6 ms [109, 124] for the −30 condition. On day 4, it occurred at 112.6 ms [108, 117] for the −60 and at 92.2 ms [88, 97] for the −30 condition. Overall, the reaction to both the unexpected absence (in the delay conditions) and to the unexpected occurrence (in the advancement conditions) of sensory feedback was consistently very fast.

Subsequent Presses Are Delayed Irrespective of Perturbation Direction

So far, we have established that sensory feedback about the keypress is used to control the finger that produces the press, even during fast performance after extended training. Next, we investigated how the subsequent presses are impacted by the perturbation. This provides us with an opportunity to investigate to what degree the feedback control is implemented in a hierarchical manner (Fig. 1A) or in form of a single unit or chunk (Fig. 1B).

To visualize how the perturbation influences both the current and subsequent presses, we plotted the timing of five events (early onset, onset, peak, release, and late release, see methods) for the perturbed and the two subsequent presses across the four sessions (Fig. 5). As the independent variable (i.e., x axis), we plotted the group-averaged time estimates of these landmarks for the nonperturbed trials relative to the onset of the perturbed press (0 ms). As the dependent variable (i.e., y axis), we plotted the change in the average time interval relative to the unperturbed condition. Each press is indicated by a line that connects the five corresponding landmarks.

Figure 5.

Effects of feedback perturbation on the perturbed press (press 0) and subsequent finger presses (Press +1, Press +2) across feedback conditions and training days. Five landmarks (EO: early onset, O: onset, P: peak, R: release, LR: later release) are plotted per press (see methods). The x axis shows the average time of occurrence of the landmark on unperturbed trials relative to the onset of the first press. The y axis shows the time difference of the landmarks between the perturbation conditions and the unperturbed condition. Landmarks belonging to a finger press are connected by a line. Positive differences indicate that the perturbation resulted in a delay, whereas negative differences indicate a speed-up. The four panels show results for the different training sessions (i.e. days). Day 4 shows how we tested the offset between presses, with an example of the 2nd to 3rd press for the +60 ms condition. Error bars represent the means ± standard error across participants.

The feedback perturbations impacted the execution not only of the current press but also of subsequent presses. On the first day of training, both the +30ms perturbation (t₂₅ = 6.055, P = 2.51e-06) and the +60 ms perturbation (t₂₅ =9.078, P = 2.177e-09) delayed the onset (interval onset-onset + 1) of the next press relative to when no perturbation was present (i.e., red lines vs. gray line at zero). Moreover, the delay of feedback impacted even the onset of the press two positions after the perturbation (+60 ms: t₂₅ = 7.172, P = 8.11e-08). In contrast, time advancements did not alter the timing of subsequent presses relative to the unperturbed trials (onset-onset + 1: −30 ms: t₂₅ = −0.904; P = 0.375; −60 ms: t₂₅ = −1.488, P = 0.149).

On the last day of training we observed clear coarticulation between presses. Here, the onset of the +1 press roughly occurred at the same time as the release of the perturbed press (Fig. 5). This overlap in press execution raises the possibility that the presses are controlled together as a single movement chunk. If this was the case, we would predict that any speed-up or slow-down on the perturbed press would impact the following press in the same way (Fig. 1B). To test this prediction, we compared the effect of the perturbation on the onset of the next press (onset + 1 Fig. 5) to the effect on the release of the perturbed press, i.e., the force level that occurred normally at the same time (see methods for detail). For the +60 condition, we found a significantly longer delay for the subsequent press in comparison with the perturbed press (t₂₅ = 2.522, P = 0.018). This effect can be seen as an offset between the end of the line for the perturbed press and the overlapping line of the subsequent press (Fig. 5, day 4, offset). Surprisingly, a similar offset was also present between the second and third press after the perturbation (t₂₅ = 3.429, P = 0.002). These additional delays across presses resulted in an overall slower execution speed for the entire sequence (MT; day 4: +60 ms: t₂₅ = 5.828, P = 4.456e-06).

Similarly to what we have observed for the +60 ms delay condition, an offset between the different presses was also observed for time-advancement of the feedback by −60 ms (dark blue in Fig. 5), although this effect did not reach significance (t₂₅ = 2.043, P = 0.052). Nevertheless, the offset was significant when comparing the second and third press after the perturbation (t₂₅ = 3.877, P = 6.799e-04). In the −60 ms perturbation condition, these additional offsets did not result in a significant slowdown of the overall sequence speed (day 4: t₂₅ = −0.858, P = 0.399), suggesting that the additional delays of subsequent presses were cancelled out by the speed-up on the perturbed press. In contrast to the ±60 ms feedback perturbations, no clear offset was present for the ±30 ms perturbation condition (Fig. 5; light blue and light red). Indeed, the comparison did not reach statistical significance for either time delay (+30 ms: t₂₅ = 0.882, P = 0.193) or advancement perturbation (−30 ms: t₂₅ = 0.589, P = 0.281).

In summary, for larger perturbations, participants delayed subsequent presses after the occurrence of a perturbation, irrespective of whether the sensory feedback was advanced or delayed. This slowdown impacted the onset of the next press, but not the simultaneously occurring release of the ongoing press. Thus, these findings indicate that subsequent finger presses are controlled in a hierarchical manner (Fig. 1A) rather than in terms of a single movement chunk (Fig. 1B), even in the trained state when they are highly overlapping.

One counter-argument against this interpretation is that our sequences were too long to be organized into a single chunk (see Ref. 32). Thus, it is likely that the transitions between the perturbed and subsequent finger presses consisted of a mixture of between-chunk and within-chunk intervals. The offset may only be observed for the longer between-chunk intervals, but not for the short within-chunk intervals. We, therefore, conducted an additional analysis for the 4th day. For each participant, sequence, and perturbation location, we calculated the time difference between the release of press 0 and the onset of press +1 during unperturbed performance (Fig. 6, x axis). Note that even a 0 ms time difference indicates a considerable overlap between presses (Fig. 3). Only presses with a >100 ms difference were actually separated by a gap in the force production, as would be expected if they were encoded in different chunks.

Figure 6.

Dependence of offset and overlap between presses on day 4. Dots show the data for each individual participant for each of the six specific presses that could be perturbed. The x axis shows the time from the release of the perturbed to the onset of the next press. Smaller values indicate more overlap. The y axis shows the offset measured in the +60 ms perturbation condition (see Fig. 5).

We plotted this overlap measure against the offset between presses (dashed line in Fig. 5, see method) that was observed for the same transition in the +60 perturbation condition (Fig. 6). Although the largest offsets were observed for presses that had a larger separation during unperturbed performance, this relationship was not significant when testing the slope of the regression for each individual participant against zero, t₂₅ = 0.555, P = 0.583. Most importantly, however, the intercepts of the regression, i.e., the expected offset for presses with considerable overlap (0 ms), were significantly larger than zero, t₂₅ = 9.486, P = 9.19e-10.

Overall, our findings therefore suggest two qualitatively different ways in which sensory feedback controls the execution of a sequence of finger movements. First, the timing of the feedback directionally either lengthens or shortens the perturbed key press. Second, the occurrence of a perturbation slowed down the onset of future presses in a directionally nonspecific manner. This effect was mainly present for larger (60 ms) compared with smaller (30 ms) perturbations but did not depend on the direction of the temporal shift. Most importantly, the slowing down only affected the timing between presses (the offset), but not the speed of the execution of the presses themselves. This finding provides evidence that sensory information is used on multiple levels of a hierarchical control scheme (Fig. 1A).

Rapid Behavioral Adjustments Are Caused by Haptic Feedback

Finally, we investigated to what degree the effects observed in the main experiment were due to the perturbation of haptic, visual, or auditory feedback. To test this, we conducted a control experiment, in which a separate set of participants was assigned to one of three experimental groups, with each group receiving only one of the three types of feedback (auditory, visual, or haptic). As in the main experiment, we delayed the feedback on selected finger presses within the sequence. In this case, we only chose a single perturbation condition (delay +80 ms) and participants practiced the task for 5 days. Examining the effect of the delay on the perturbed press (see Fig. 7), we found that only the haptic group demonstrated a significantly longer onset-peak interval following the perturbation both in the beginning (day 1: t₁₅ = 2.980, P = 0.009) and toward the end of training (day 4: t₁₅ = 3.579, P = 0.003). Neither the visual (day 4: t₁₅ = 0.901, P = 0.382) nor the auditory group (day 4: t₁₅ = 1.060, P = 0.306) showed a significant effect of the feedback perturbation on the onset-peak interval. These results clearly show that the rapid adjustments of the ongoing press were driven by haptic feedback from the fingertip.

Figure 7.

Effect of haptic, visual, and auditory feedback perturbation in control experiment across training days. As in Fig. 5, five landmarks per press (connected by a line) are plotted. The control experiment only had +80 ms perturbations, but each group of subjects received only one type of feedback. The different panels indicate the different training sessions (i.e. days). The error bars represent the means ± standard error across participants for each group.

Delay of Subsequent Presses Arises from All Three Feedback Modalities

In contrast, the delay of subsequent presses was observed for all three feedback modality groups. Consistent with the effect on the perturbed press, the delay of the onset of the press following the perturbation (+1, averaged across days 2–5) was largest in the haptic group (69 ms, t₁₅ = 6.890, P = 5.146e-06). However, both the auditory group (35 ms, t₁₅ = 4.888, P = 1.971e-04), as well as the visual group (19 ms, t₁₅ = 4.828, P = 2.214e-04), showed a clear delay in the onset of the subsequent press, even though no such effect was observed on the perturbed press (Fig. 7). This result suggests that the delay we observed on the subsequent presses in our main experiment could be induced by the perturbations in each of the three feedback modalities. Although the slowing effect observed after the first press appeared to be larger in the auditory as compared with the visual condition, we could not dissociate directional from nondirectional effects cleanly, as this experiment only used a delay condition.

DISCUSSION

In this study, we used small transient feedback perturbations to probe how sensory feedback is used in the control of finger movement sequences. Specifically, we examined how sensory feedback modulates the execution of skilled finger movements across 4 days of training, and how feedback differentially affects the execution of the ongoing press and subsequent presses.

Sensory Feedback Rapidly Modulates Movement Execution of the Perturbed Press

Throughout training, we found clear evidence of rapid behavioral adjustments of the finger press for which the sensory feedback was perturbed. This result illustrates the continuous use of sensory feedback even when controlling fast finger movements. Participants adjusted their ongoing behavior even though our task was designed so that it could be accomplished without considering the feedback. The keypresses were isometric and participants simply needed to exceed a specific force threshold. Nonetheless, participants adjusted their behavior based on the perturbation.

Furthermore, we found that the effects of the perturbation were directionally specific: the delay in sensory feedback resulted in a lengthening of the perturbed press, whereas a time advancement resulted in a shortening. Previous studies have primarily investigated feedback delays (17, 33–35) but have rarely advanced participants’ feedback (36, 37). By including both feedback delays and advancements, we provided evidence of the directional nature of sensory feedback integration in fast finger movements.

The reaction to the delay of haptic feedback was very fast and occurred within 60–90 ms after the expected time of the feedback. This finding is consistent with previous reports that demonstrate responses between 65–110 ms following a haptic input (38–40). In contrast, auditory and visual feedback alone did not elicit a strong reaction on the perturbed press, consistent with the fact that the fastest reactions to changes in these two modalities are noticeably slower (4–7, 41, 42). Therefore, by including a haptic feedback condition, we were able to show the very rapid integration of sensory feedback in the execution of a finger press within the context of a fast movement sequence. Although we did find differences in regards to the size of the perturbation effect across the feedback groups, we are unable to make a strong argument as to whether this difference is also primarily driven by the differences in feedback integration speeds or whether this difference could be related to genuine differences in how feedback is used across finger presses in the task. Given that we did not test a time-advancement condition in this experiment, we cannot distinguish between these two possibilities.

Shift from Feedback to Feed-Forward Control with Learning

Although the feedback perturbation still impacted the execution of the perturbed press on the last day of practice, we did find that the effect reduced by ∼40% with training. This observation is in line with previous research that observed a shift from feedback to feed-forward control with training (43, 44). It has been suggested that feedback plays an important role in the initial phases of acquiring a novel motor skill, but that its importance decreases, and potentially even disappears altogether, with prolonged training (43–47). The main theoretical idea is that the skill learner acquires an accurate internal representation of the instructed movements, such that sensory feedback becomes less necessary for execution (44, 46, 48). Alternatively, participants potentially learned that the large deviations of the sensory feedback were irrelevant for overall performance and therefore could be ignored (49). This interpretation is consistent with the observation that the reduction with training was most pronounced in the +60 ms perturbation condition.

Feedback Processing within a Hierarchical Sequence Controller

By studying the influence of a small feedback perturbation on subsequent finger presses in a sequence, we were able to investigate how feedback is processed throughout the control hierarchy. Research studying time delays and advancements of an external pacing signal in synchronization paradigms (16, 33, 37) has provided evidence for a central timing mechanism that can bring the generation of finger presses back into synchronization with the metronome. In our paradigm, there was no external timing goal, but rather participants were instructed to produce the sequence as fast as possible. We found that the feedback perturbation on a single press also affected the execution of subsequent presses, both at the beginning and at the end of training. Importantly, the reaction to the feedback perturbation was different for the perturbed and subsequent presses. The feedback perturbation led to a directionally specific effect on the perturbed press, either accelerating or delaying the execution of the presses based on the perceived moment of the button press. In contrast, subsequent presses were delayed, irrespective of the direction of the feedback perturbation. The most likely interpretation for this effect is that the sequence controller reacts to an unexpected advancement or delay of the sensory feedback with a cautionary adjustment, delaying the onset of the next finger press. Consistent with this idea, we found that the delay was larger when the absolute size of the perturbation was larger (+60 and −60 ms).

This delay on subsequent presses was also observed on the last day of training when groups of finger presses were produced in a highly overlapping manner. Based on this overlap, we would hypothesize that these movements were “chunked” (24–27). One theoretical idea of a movement chunk is that the muscle commands for all chunked movements are produced by a single controller that only needs to be triggered once by the sequence controller (Fig. 1B). Our data, however, provide clear evidence against this hypothesis. The cautionary response delayed the onset of the subsequent press, but not the simultaneous execution of the ongoing press, such that the entire pattern co-articulation between the fingers changed. This suggests that even after 4 days of training and considerable overlap, top-down processes still impact the triggering of each finger press, even within a “chunk” of finger presses.

The slowing in response to an unexpected sensory perturbation bears some similarities to the phenomenon of posterror slowing (50–52). Although the classical posterror slowing effect is elicited by an actual task-relevant error, the effect observed here seems to rely only on the mismatch between the predicted and observed sensory consequences of an action. Nonetheless, it is possible that the two phenomena rely on similar neural systems. Importantly, however, we show that the observed slowing effect influences the onset asynchrony between presses, rather than the execution of the presses themselves, providing evidence for differences in the use of feedback across two hierarchical levels of sequence control (Fig. 1A).

Conclusion

In this study, we demonstrated that sensory feedback is continuously used to adjust the production of fast and well-practiced finger sequences. Haptic feedback is used very rapidly to adjust the length of the ongoing press in a directionally specific manner. Perturbations of haptic, visual, and/or auditory feedback also led to a slowing of the initiation of subsequent presses, but that effect was only visible in the relative onset of the presses, not in the execution of the presses themselves. This effect was present even after extensive training, showing that even sequences of finger presses that overlap considerably are controlled in a hierarchical manner.

GRANTS

This work is supported by a Canada First Research Excellence Fund (BrainsCAN) to Western University, a Natural Sciences and Engineering Council of Canada (NSERC) Discovery Grant RGPIN-2016-04890 (to J.D.), a NSERC Grant RGPIN 238338, a Canadian Institutes of Health Research Grant PJT-153447, and a National Institute of Child Health and Human Development Grant R01 HD075740 (to P.L.G).

DISCLOSURES

No conflicts of interest, financial or otherwise, are declared by the authors.

AUTHOR CONTRIBUTIONS

N.J.P., C.R.H.-C., P.L.G., and J.D. conceived and designed research; N.J.P. performed experiments; N.J.P. analyzed data; N.J.P., P.L.G., and J.D. interpreted results of experiments; N.P. prepared figures; N.P. drafted manuscript; N.J.P., C.R.H.-C., P.L.G., and J.D. edited and revised manuscript; N.J.P., C.R.H.-C, P.L.G., and J.D. approved final version of manuscript.

APPENDIX

Subj: Study: Sequence Integration 4

Questionnaire about Experience:

Did you notice anything during the experiment?

We manipulated an aspect of the task during the experiment what was it?

Which of these manipulations did we implement (chose any that apply)?

■ Change the frequency of the tones that were presented when a key was pressed

■ Delay the feedback of a press

■ Provide false feedback on a press (if you were correct it would show as incorrect)

■ Interleave the 3 trained sequences with random sequences

■ Change the frequency of the vibration when a key was pressed

■ Advance the feedback of a press

■ Switched a single press within the sequence (switch which number is presented)

■ Randomize the points you received for each trial rather than making them dependent on performance

■ Omit the feedback of a press

■ Give you false feedback regarding your average speed at the end of a block (higher or lower than you actually were)