Proprioception contributes to the sense of agency during visual observation of hand movements: evidence from temporal judgments of action

Abstract

The ability to recognize one's own movement visually is important for motor control and, through attribution of agency, for social interactions. Agency of actions may be decided by comparisons of visual feedback, efferent signals and proprioceptive inputs. Because the ability to identify own visual feedback from passive movements is decreased relative to active movements, or in some cases is even absent, the role of proprioception in self-recognition has been questioned. Proprioception during passive and active movements may, however, differ and so to address any role for proprioception in the sense of agency the active movement condition must be examined. Here we tested a chronically deafferented man (IW) and an age-matched group of 6 healthy controls in a task requiring judgement of the timing of action. Subjects performed finger movements and watched a visual cursor that moved either synchronously or asynchronously with a random delay, and reported whether or not they felt they controlled the cursor. Movement accuracy was matched between groups. In the absence of proprioception, IW was less able than the control group to discriminate self-from computer-produced cursor movement based on the timing of movement. In a control visual discrimination task with concurrent similar finger movements but no agency detection, IW was unimpaired, suggesting that this effect was task specific. We conclude that proprioception does contribute to the visual identification of ownership during active movements and thus to the sense of agency.

Introduction

Agency is the sense that one is causing an effect (Gallagher. 2000). Disturbances in the sense of agency occur, for instance, during auditory hallucinations or delusions of control when a patient's own movements or thoughts are misattributed to an external source (Frith, Blakemore, & Wolpert. 2000).

During movements it is reasonable to assume that the sense of agency relies on the spatial and temporal congruence of movement intention, motor command and the sensory feedback from the execution of action. A theoretical model which has received extensive experimental support suggests that the motor command can be used to predict the consequences of an action (Wolpert, Ghahramani, & Jordan. 1995) and can change the perception of the sensory feedback, binding the action and its consequences. Thus, self-produced sensory stimuli feel less intense (Blakemore, Wolpert, & Frith. 1998; Shergill, Bays, Frith, & Wolpert. 2003) and also appear to occur earlier than externally produced sensory events (Haggard, Clark, & Kalogeras. 2002). In this way externally produced, unpredictable sensation is distinguished from self-produced feedback (Frith. 1996). According to this model, a sense of agency from visual feedback depends on whether the visual stimulus matches the motor command in place and time.

Because of proprioception is personal and related to movement, one might imagine that it, too, would provide a cue for agency. However, the contribution of proprioception to self–recognition remains unclear. Proprioceptive feedback in the absence of motor command, for instance during passive hand movements, clearly does not elicit a sense of agency. Moreover, it has been noted that the ability to identify the self from visual feedback during passive movement is at a near-chance level (Tsakiris, Haggard, Franck, Mainy, & Sirigu. 2005). This, however, does not rule out a contribution from proprioception to self-recognition during active movement. Compared with passive movements, active movements elicit a stronger signal from proprioceptive receptors (Jones, Wessberg, & Vallbo. 2001) and allow for tuning of the sensitivity of muscle spindles via gamma innervation (Prochazka. 1999; Hulliger, Nordh, & Vallbo. 1982). Active movement might thus allow more salient comparisons between proprioception and visual feedback in making the judgement of self.

A role for proprioception in self-recognition of active movement was recently suggested by a study that compared the ability to identify self-produced visual feedback of the arm position at the end of a movement in a patient (GL) with chronic loss of proprioception and a group of age-matched healthy controls (Farrer, Franck, Paillard, & Jeannerod. 2003). This study found that the deafferented subject was less accurate than the controls in recognizing her own movements. Subjects used a joystick to move a virtual hand from a central position to a screen target. All movements were executed with the eyes closed, so that subjects saw only the initial and the final position of the virtual hand. The final position either replicated the position of the joystick exactly or was deviated from it by an angle of 40-90 degrees. The subjects judged if the change in the position of the virtual hand corresponded with the movement they had performed. Though this study offered support for a role for peripheral feedback in agency from information about mismatch between intended and observed movement position, the study does have two problems which might weaken their conclusion about the proprioceptive contribution to a sense of agency.

First, because in deafferented patients movement trajectory without vision is variable and inaccurate (Ghez, Gordon, & Ghilardi. 1995), an observed trajectory deviation could have reflected an error in movement execution as well as the externally imposed perturbation. We assume that control subjects, whose trajectories are normally on target, can readily recognize that a trajectory with an angular rotation over 40 degrees is externally produced, and therefore, they could outperform GL in deciding movement ownership in the task of Farrer et al. (2003). Thus, because trajectory errors were not taken into account, the task could not separate whether GL's inability to detect a trajectory rotation was due to her proprioceptive deficit or her reduced directional accuracy.

Secondly, it is known that deafferented subjects need to allocate more attentional resources to their movements than controls (Ingram, van Donkelaar, Cole, Vercher, Gauthier, & Miall. 2000), which may result in lower performance on a simultaneous discrimination task. GL's impairment on the recognition task may have reflected more general attentional factors.

The present study was designed to address these methodological concerns. By examining the temporal rather than spatial dimension of agency discrimination, it both complements and extends Farrer et al's study. Subjects performed simple finger movements along a track that prevented any trajectory deviation and saw a cursor move either synchronously or asynchronously with their finger movement. They had to judge whether they felt they controlled the cursor or not on the basis of temporal mismatch alone between finger and cursor movement. Only trials with a correct movement amplitude were included in the analysis. Post hoc analyses of the reaction time and movement duration confirmed that the controls and deafferented subject were also matched with respect to movement timing. We also included an attentional control task that required participants to execute finger movements while performing an independent visual discrimination task. If proprioception contributes to the sense of agency for movements under visual feedback then we expected the deafferented subject to be impaired compared with controls in the first task; the absence of an impairment in the control task would exclude a purely attentional effect.

Methods

Subjects

IW, a 53 year old, left-handed man who had suffered a peripheral deafferentation 34 years previously (for a clinical description see (Cole & Sedgwick. 1992)) and 6 healthy controls (all male, 1 left-handed, median age 57, range 50-58) gave written informed consent to participate in the study. All subjects had normal, or corrected to normal, vision and used some form of computer mouse daily (IW) or at least once a week (control participants). The study was approved by the Ethics Committee at the University of Birmingham.

Study design

Both the proprioceptively deafferented man and the control group completed two tasks: detection of ownership of cursor movement and detection of the direction of target jump. In the first task the subjects saw cursor movement that was either synchronous or asynchronous with their finger movement and were asked to tell whether they felt they controlled the cursor or not.

In the second task the subjects saw a brief cursor jump in either left or right direction and were asked to detect the direction of cursor jump. The two target directions were equiprobable and presented in random order. In both tasks the participants performed identical finger movements.

Set up

The subjects had the index finger of their dominant hand on a sliding mouse (FELIX Pointing Device, Altra, Rawlins, WY, Figure 1) that moved on a rectangular active area of the size 24×30 mm surrounded by a frame. The finger was fixed to the mobile piece of the mouse with surgical tape. The position of the mouse was recorded every 20 ms. A screen prevented the subjects from seeing their hand and forearm. Because the movements were restricted to the finger and wrist joints the subjects had no direct vision of their movements. For each trial, the subjects moved the finger back or forth, along the (parasaggital) y-axis of the mouse, in a straight line between the proximal and distal edge of the mouse area (24 mm). A plastic restraint attached to the mouse prevented movement along the x-axis. The subjects were instructed to start their movement at the onset of a visual cue, move at high speed, and stop when they reached the opposite edge of the sliding mouse area. During both tasks the subjects listened to music in headphones to prevent them from hearing a click when the mouse hit the frame at the edge of the active area (and thus from solving the task by audio-visual matching).

Figure 1. The setup for finger movements.

The subjects had their index finger on a sliding mouse that moved on a rectangular active area of size 24×30 mm surrounded by a frame. The finger was fixed to the mouse using surgical tape. For each trial, the subjects moved the finger back or forth, along the y-axis of the mouse, in a straight line between the proximal and distal edge of the mouse area (24 mm). A plastic support attached to the mouse prevented movement along the x axis.

Visual cursor movement was presented on a computer screen (300×220 mm, 640×480 pixels, subtending approximately 33 × 25 degrees visual angle), placed at 50 cm in front of them. The cursor (size 30 × 30 pixels, visual angle = 1.5 × 1.5 degrees) was a filled white square which, at the start of each trial, appeared at the centre of the screen, cuing the subjects to start the movement. The reaction time for the onset of finger movement was calculated by subtracting the time for cue onset from the time of movement onset. The cursor was presented for 2 s then hidden until the beginning of the next trial.

The subjects indicated their decisions verbally. They were instructed to respond after each trial once both the finger and cursor had stopped moving. For control subjects, the experimenter started the next trial immediately after they made a response. For IW, to maintain a good hand posture, he was allowed to view his hand on the mouse between trials, before verbally confirming that he was ready.

A diagram of the two tasks is presented in Figure 2.

Figure 2. Diagram of the two tasks.

At the start of each trial, a cursor was presented at the centre of the computer screen and the subject moved a sliding computer mouse back or forth, from edge to edge (the solid black lines in the right images). A plastic sheet prevented direct vision of the forearm and hand. During the ownership task (A), the cursor moved along a horizontal line either to the left or right, at random, and stopped at the lateral edge of the screen. Cursor and finger could either move synchronously or asynchronously, with a random delay. Because there was no spatial correspondence between cursor and mouse, identification of ownership relied entirely on timing signals. During the cursor jump task (B) the cursor jumped briefly to the side - left or right, at random - and back. The arrows show the direction of movement in a sample trial. The coordinate system show x and y axes in register with the set-up presented in figure 1.

Task 1. Ownership of cursor movement

To avoid ceiling effects in performance we increased the difficulty of the feedback identification task by showing transverse cursor movements, randomly moving to left or right, in response to parasagittal mouse movement. The screen cursor appeared at the centre of the screen, then in response to mouse action, moved along a horizontal line in either left or right direction, randomly selected, and finally stopped at the lateral edge of the screen. The amplitude of cursor movement across the computer screen was 15 cm in response to full-range deflection of the mouse of 24mm. This corresponds to 16.5 degrees visual angle at the normal viewing distance of approximately 50 cm. The cursor could either move synchronously with the finger, as it is the case during normal mouse movement, or asynchronously, with the cursor starting before or after the movement of the mouse.

In the synchronous condition there was no perceptible delay between the start of the movement as detected by visual and proprioceptive feedback. In the asynchronous trials, cursor movement was determined from the mouse movement recorded in one of the previous synchronous trials. Cursor movement was then started before or after the onset of this mouse movement, with a random onset asynchrony. To present cursor movement before the onset of mouse movement, the reaction time for starting a mouse movement was estimated from the reaction time of the previous trial. Then the stored cursor movement was replayed, starting at a selected time before the estimated movement onset.

Cursor-mouse movement onset asynchrony was calculated post hoc for each trial by subtracting the actual onset of finger movement from the onset of cursor movement. For each subject, the onset asynchrony was averaged over trials, separately for positive (cursor after finger) and negative (finger after cursor) onset values. The average negative and positive asynchronies were −193 ms and 129 ms in IW, and −165 ms and 123 ms in the control group, with no significant difference between groups (one-sample t-tests, p>0.05) . The range was from −400 to −20 and from 20 to 250 ms.

The subjects responded ‘me’ if they felt they controlled the cursor and ‘not me’ otherwise.

Task 2. Direction of cursor jump

To control for IW's possible difficulty in attending to the visual feedback cursor while making an active finger movement, we tested all subjects in a task requiring detection of a jump in cursor position, independent of movement feedback. The screen cursor appeared statically at the centre of the screen, while the subject performed the same finger movement task as above, without visual feedback. After a random time interval from the start of the trial, the cursor jumped 1 pixel (0.05 degrees visual angle) to the side - left or right - and back. The jump took two screen refresh cycles to complete (33ms) and could occur at any point during the 2s-trial. This jump was completely unrelated to the subject's mouse movement and could occur before, during or after it. The subjects responded by saying ’left’ or ’right’.

The experimental session consisted of two trial blocks with 72 trials of the ownership task (presented first) and 72 trials of the cursor jump task (presented last), and took approximately 20 minutes to complete.

Practice

All subjects completed a practice session with the same number of trials and blocks of trials as the experimental session. The purpose of the practice session was to familiarize all subjects with the paradigm.

Data analysis

Each subject's sensitivity to the difference between conditions was indexed by d-prime (d′). D′ is a measure from signal detection theory (Macmillan & Creelman. 1991), which represents the discrepancy between a hit rate and a false-positive rate. The hit rate was defined as the correct ‘me’ responses divided by the number of synchronous trials. The false positive rate was defined as the false ‘me’ responses divided by the number of asynchronous trials. The higher the hit rate and the lower the false positive rate, the higher the value of d′. The ceiling value for 100% hit rate and 0% false positive is d′= 4.65. D′ was compared across tasks and groups using one-sample t-tests.

Results

Movement execution during the ownership task

The reaction time for the onset of finger movement, calculated relative to the onset of the visual cue was 356 ms in IW and between 334 –559 ms (average 438) in the control group. The standard deviation of the reaction time was 162 ms in IW and between 47 –211 ms (average 124 ms) in the control group. Movement duration was 220 ms in IW and between 181 and 361 ms (average 228 ms) in the control group. The standard deviation for movement duration was 210 ms in IW and between 58-361 ms (average 135) in the control group. There was no significant difference between groups in any of these variables (one sample t-tests, p>0.1).

Sample trajectories for correctly and incorrectly answered asynchronous trials are shown in Figure 3.

Figure 3. Sample trajectories from synchronous (dashed line) and asynchronous (continuous line) trials.

The plots show the cursor trajectory across the screen against time, with the asynchronous trials using cursor trajectories replayed from previous trials (see Methods). A: typical trials from subject IW. B: Typical trials from a control subject. The arrows indicate the onset of mouse movement.

All control participants performed finger movement of the required amplitude, from one edge to the other of the frame surrounding the active area of the sliding mouse. Post hoc analysis of the trajectories showed that IW's movements failed to reach the end stop in 40% of the trials. The trials where the amplitude of the mouse or cursor movement was shorter than required were excluded from the analysis. in order to make sure that the difficulty of the ownership task was matched between groups. Judging ownership of cursor movement is more difficult for incorrect compared with correct trajectories because an incorrect trajectory may reflect both an error in execution and an externally produced movement.

Task accuracy

The number of valid and correctly answered trials, the hit and false positive rate and d′ for the ownership task are presented in Table 1.

Table 1.

Individual accuracy for the ownership task.

	No. correct trials/valid trials		Hit rate (%)	False positive rate (%)
Subject	Sync	Async			d-prime
IW	19/23	11/15	83	27	1.59
control1	36/36	27/36	100	25	3
control2	31/36	29/36	86	19	1.96
control3	35/36	22/36	97	39	2.19
control4	35/36	28/36	97	22	2.65
control5	36/36	20/36	100	44	2.48
control6	36/36	19/36	100	47	2.4

Key: Sync- synchronous condition, Async – asynchronous condition, Hit rate – % correct “me” responses for synchronous trials, False positive rate – % false “me” responses in asynchronous trials. To avoid infinite z-scores when calculating d-prime, control subject hit rates of 100% were corrected according to Macmillan and Creeling (1991) by subtracting 1-1/(2N), where N=36, to a value of 98.6%.

Analysis of d′ showed that IW was less sensitive than controls to the difference between self- and computer generated cursor movement (one-sample t-test, p = 0.002, Figure 4). His accuracy was 2.36 standard deviations below the group mean. For the cursor jump task, IW was significantly more accurate than the control group (2.4 standard deviations above group mean, one-sample t-test p=0.002, Figure 4).

Figure 4. Accuracy of detecting ownership of cursor movement and direction of cursor jump.

The mean data for the proprioceptively deafferented subject IW (line) and a boxplot (median, inter-quartile interval and range) for 6 healthy controls are shown.

We have also compared the hit rates and false positive rates across groups (Table 1). The hit rate in IW was significantly lower than in the control group (single sample t-test, p=0.001). The similarity of the false positive rates across groups (p>0.2) rules out that the difference in hit rates merely reflects a change in response bias. This confirms the results from the statistical analysis of d′.

The asynchrony value was random, and in the case of “cursor before mouse” asynchronous trials, it could only be calculated post-hoc. It is possible that IW was, by chance, exposed to asynchrony values that were smaller than those in the control groups, in which case his task would have been more difficult and hence his false positive rate would have been an overestimation of his true performance. To make sure this was not the case we compared the asynchrony value for false positive trials post-hoc in the two groups, and showed that the average delay between cursor and mouse for the trials where the subjects misattributed a computer generated movement to themselves was similar or larger in IW as compared with the control group: −114 ms (cursor preceding movement) and 115ms (movement preceding cursor) as opposed to −82 ms (one sample t-test, p=0.045) and 103 ms (one sample t-test, p= 0.09, not significant). So IW achieved a level of errors similar to control subjects at larger ansynchronies.

The probability of a false ‘me’ response as a function of the finger-cursor movement onset asynchrony was similar in the two groups (Figure 5).

Figure 5. Cumulative probability of a false ‘me’ response as a function of asynchrony.

The scatterplot shows all valid asynchronous trials in IW (●) and for all six control subjects (○).

Discussion

In accord with the study by Farrer et al. (2003) we have found a decreased ability to recognize agency from visual feedback during movement in the absence of proprioception. Further to that study, we can rule out that this impairment in feedback recognition reflects either a decrease in movement accuracy, by testing the temporal rather than spatial domain, or a more general impairment in attention, with a control for attention. We conclude therefore that proprioception contributes to self-identification during active hand movements as determined by temporal mismatch between movements made and seen.

Error or variability in movement execution might, theoretically, have reduced the ability to discriminate whether or not a visual movement is self-produced. To avoid lateral trajectory variability, a plastic restraint on the mouse prevented angular deviations from a predefined trajectory and trials with incorrect movement amplitude were excluded from the analysis. There was no statistically significant difference between IW and the controls in the mean and standard deviation of the reaction time for movement onset or for movement duration. This suggests that the groups were well matched with respect to movement execution and rules out error or variability as an explanation of the IW's impairment.

In our visuomotor control task that required similar movements and visual discrimination, but no detection of agency, IW was unimpaired. Thus his decrease in performance was task-specific and not due to the need to share attention between the visual discrimination task and performance of hand movements. From the observation that deafferented subjects need to allocate more attention to their movements than controls (Ingram, van Donkelaar, Cole, Vercher, Gauthier, & Miall. 2000) we expected his performance on the simultaneous visual discrimination task to be reduced. Surprisingly, IW performed this task significantly better than the control group. His generally heightened level of visual attention has previously been remarked upon (Nougier, Rossi, Bard, et al. 1994). Perhaps, because he has learnt to rely entirely on visual supervision for movement control, visual signals are processed with an increased accuracy. It has previously been noted that the lack of sensory information in one modality results in a compensatory increased accuracy in other modalities - e.g. better tactile accuracy in blind (Van Boven, Hamilton, Kauffman, Keenan, & Pascual-Leone. 2000) or blindfolded (Facchini & Aglioti. 2003) subjects. That IW was worse than controls at detecting the agency for visual feedback despite his heightened levels of visual attention further strengthens the conclusion that proprioception is important for the temporal judgement of the sense of agency.

An alternative explanation of our findings would be a deficit in time estimation per se in IW, such that he is less able to temporally match action and cursor movement. IW, however, shows a similar dependence between the magnitude of movement onset asynchrony and the probability of a false ‘me’ response, which is evidence against him having a deficit in temporal perception in relation to the test (Figure 5).

Deprived of proprioception, IW was still able to perform the task well above chance levels. This underscores the importance of interaction between the brain's predictive models and visual feedback for self-identification, confirming a previously proposed mechanism (Frith. 1996). IW has been through an extensive rehabilitation over the years and is now able to function relatively normally when allowed visual supervision. It is likely that in his case the brain's predictive models that estimate the outcome of movement function at their highest possible capacity. Therefore, his reduced ability to recognize when the feedback was self-produced suggests that for this task, the brain's internal models, even when calibrated with visual feedback, cannot fully compensate for the lack of proprioceptively derived temporal signals.

It is not known how motor intention, command and its proprioceptive and visual consequences interact in order to provide a sense of agency. One possibility would be that the brain estimates the state of the hand from both proprioceptive and motor command signals, as it is the case during hand movements performed in the dark (Wolpert, Ghahramani, & Jordan. 1995) and then compares this estimate against the visual feedback to assess agency. Another possibility is that both the comparison between the incoming visual and proprioceptive sensory streams and between the motor command and its sensory consequences occur simultaneously and complement each other to provide a sense of agency. Although the present work cannot distinguish between these two possibilities, it does emphasise that proprioception is important in such judgements.