Visual information gleaned by observing grasping movement in allocentric and egocentric perspectives

Abstract

One of the major functions of vision is to allow for an efficient and active interaction with the environment. In this study, we investigate the capacity of human observers to extract visual information from observation of their own actions, and those of others, from different viewpoints. Subjects discriminated the size of objects by observing a point-light movie of a hand reaching for an invisible object. We recorded real reach-and-grasp actions in three-dimensional space towards objects of different shape and size, to produce two-dimensional ‘point-light display’ movies, which were used to measure size discrimination for reach-and-grasp motion sequences, release-and-withdraw sequences and still frames, all in egocentric and allocentric perspectives. Visual size discrimination from action was significantly better in egocentric than in allocentric view, but only for reach-and-grasp motion sequences: release-and-withdraw sequences or still frames derived no advantage from egocentric viewing. The results suggest that the system may have access to an internal model of action that contributes to calibrate visual sense of size for an accurate grasp.

1. Introduction

Vision is known to be tightly linked to action. There is ample evidence for the existence of specialized and independent visual neural processing linked with action [1–3], the so-called vision-for-action pathways: in particular conditions, we can act accurately on objects that are not perceived consciously, indicating that some visual information reaches the motor but not the perceptual system. The existence of the complementary loop that feeds back the motor information on the visual brain is less certain. Is the information of the vision-for-action pathways then relayed back to the visual system to help visual processing? And if so, is it used for basic visual discriminations or only for high cognitive visual functions? Only recently is some evidence emerging that the motor system can indeed influence basic visual processing. It is known for example that active exploration by rotation of three-dimensional objects by joystick can help later visual recognition of the object [3] and, importantly, the benefit is not present when the rotation is not controlled by the subject. It is also known that performing a simultaneous weight-lifting action biases visual weight estimation from the observed action [4]. Given that in all these experiments, the visual signals in the active and passive condition are well matched, the modulation effects suggest that performing an action may have a direct influence on basic visual analysis. Evidence is also emerging about the neuronal circuitry mediating the influence. For example, it has been demonstrated that performing open-loop pointing in total darkness can modulate BOLD signals of the visual cortex, particularly the lateral occipital (LO) cortex, the subregion specialized for the recognition of body parts named EBA [5,6]. It is also possible that observing an action may influence basic visual discrimination, given that both self-generated actions and observed actions of others activate overlapping neural networks, the so-called mirror-neuron system [7–9]. Facilitation by observation may be particularly useful for visual size discrimination of objects that we have to act upon. Human performance in visually discriminating the three-dimensional shape of simple objects can be very poor, with biases as large as 25 per cent [10,11]. Interestingly, shaping of the hand during a reach-and-grasp movement is quite precise, with maximum grip aperture accurately matched to the shape and size of the grasped object [12–14]. In addition, grip aperture is not subject to Weber's Law, with just-noticeable difference thresholds constant with fourfold variation of object size [15]. It would be functionally very useful if the visual system could use the additional information provided by the observation of the grasping action or from the motor system itself to disambiguate the size of three-dimensional objects and to perform accurate size estimation. However, at present, it is not known whether observing grasping kinematics provides information useful for judging the visual size of the object goal of the grasp.

Here, we address this question using biological motion movies of grasping action. In constructing the movie, we eliminate most of the visual cues that could mediate size discrimination by synthesizing point-light displays [16] of biological motion of the grasping action. The human visual system is efficient in detecting biological motion [17,18], even when the global visual motion information is highly impoverished or corrupted by dynamic noise. The visual system can retrieve from point-light biological motion sequences subtle details about the agent or goal of an action, such as the weight of a lifted object (for review, see [19,20]). The most informative period for weight discrimination is during the early stage of lifting action, indicating that observation of the kinematics of the reach–and-grasp action may not influence the discrimination [21]. Nevertheless, these results do not exclude the possibility that information about kinematics during the reach phase may also be used to reduce the object size visual error, as postulated by Lee et al. [10]. Both natural and point-light biological motion selectively activate cortical circuits of the superior temporal sulcus of the monkey specific for the full-body or hand movements [22–24]. Interestingly, some of these neurons are viewpoint dependent [25] and most are poly-sensory, receiving visual, somatosensory and visual–motor feedback in monkeys and humans [26–28].

To demonstrate the ability of the human visual system to retrieve the size of a grasped object by passive observation, we synthesized two-dimensional ‘biological motion’ movies of a human hand grasping objects of variable size and measured size discrimination performance. The results show that the kinematic information of the reach-and-grasp movement has a strong facilitation for the size discrimination when the action is observed from a view consistent with the observer performing the action.

2. Methods

The experiment consisted of a motion recording phase and a subsequent visual object identification task.

Thirty subjects took part in the experiment, 10 of them (actor group) also performing the recordings of the three-dimensional action movies. All participants showed right-handed dominance and had normal or corrected-to-normal vision. All subjects were naive to the purpose of the experiment and gave informed consent to participate in the study.

Subjects in the actor group were recorded grasping four objects with an array of cameras positioned to capture the action in the three-dimensional space, using 23 markers placed on the centre of the nails, joints of all digits, the dorsal aspect of the hand and the radial and ulnar styloid process (see movie 1 and full explanation in the electronic supplementary material). The objects were a sphere, cube, cylinder and trunk different in size, the diameter of the sphere was 8 cm, side of the cube was 6.5 cm, the cylinder was 6.5 cm high and 4 cm wide, the trunk was 7 cm high, the major and minor basis 3.8 and 2 cm, respectively. Actors were required to grasp each object both from the side and from the top in four separate grasping actions. From each three-dimensional recorded trajectories, two separate two-dimensional biological motion movies were constructed of the grasping movement (no object visible), one from an egocentric (observing self-action) and one from an allocentric point of view (observing others action). To change the perspective of the grasping movement, the three-dimensional motion was rotated around the azimuth by 180°. The three-dimensional movies, in the two perspectives, were projected on a plane orthogonal to the parasagittal and forming a 45° angle with the horizontal plane (see examples in movie 2, 3 and 4 in the electronic supplementary material). The motion was always presented in the centre of the screen, starting from either the bottom or the top of the display for the egocentric and allocentric perspectives, respectively. These same movies were also presented in backward motion, from the last frame of contact point to the initial hand position (backward movies). Additionally, the two-dimensional egocentric movies were also presented after a rotation of 180°, simulating an action in allocentric perspective. In these cases, the allocentric and the egocentric perspective movies contained the same motion signals (apart from a rotation) and the same spatial cues of the hand shape. To simulate left-hand grasping, the two-dimensional movies in both perspectives were flipped left to right.

The biological motion stimuli of the schematic hand marked with black dots and lines, or with black dots only, were generated with Psychophysics Toolbox [29,30] and displayed on a 22 inch LCD monitor at a refresh rate of 75 Hz for 1.35 ± 0.25 s (s.d.) (figure 2). The hand subtended about 13 × 15° (see the electronic supplementary material for full details). Stimuli were viewed binocularly from a distance of 57 cm and in a dimly lit room.

Figure 2.

Size identification performance (d′) for each subject for dynamic two-dimensional biological motion movies plotted versus static stimulus condition (last frame of the movie presented for 250 ms), for stimuli in egocentric (c) and allocentric viewpoint (d). Circles represent the actor group, black symbols observations of their own action, red for others' action and squares subjects who were not the actors of the biological motion movies. Arrows on the axes report the mean values for each group. Black diamonds report means and s.d. for the entire dataset. The panels on the left show three frames of the movie in (a) the egocentric and (b) allocentric perspective, grey levels used to indicate order of appearance during the movie presentation. For example of the full movie sequences, see the electronic supplementary material.

Twenty subjects were required to indicate (with a mouse-click on a response page containing the images of the four objects) the object that was the goal of the reach-and-grasp movement. Each subject performed 10 training trials before data acquisition, two sessions of 50 trials plus one session of 60 trials on object discrimination. Feedback on the single-trial discrimination was never given. The number of movements associated with each object (p = 0.25) and the two perspectives (p = 0.50) were balanced for trials of each individual subject. In addition, each subject in the actor group observed the biological motion movie reproducing their own reach-and-grasp movement for half the trials. The actors received the same instructions as the non-actor group and they were unaware that some movies replayed their own actions. A subset of the non-actor subjects performed also discrimination for the other types (rotate, left hand and release-and-withdraw) of movies. In separate experimental sessions, 10 of the subjects also performed the same four-alternative forced-choice task, observing the last frame of the movies presented for 250 ms. A different subgroup of subjects (n = 10) performed the discrimination for the same static frame for 500 ms exposure and the discrimination for the frame corresponding to the maximum grip aperture, again with an exposure of 500 ms (see the electronic supplementary material for full description). Discrimination performance was measured in d′ corresponding to the difference between the means of the signal and of the signal plus noise distributions, normalized to the standard deviation of the noise distribution [31], d′ is calculated as the difference between the z-scores of the hits and of the false alarms and, for two-alternative forced-choice design assumes a value of 1 for 76 per cent correct responses (threshold value) and of 0 for 50 per cent correct responses (chance level).

3. Results

Before investigating what information the visual system could extract about the object from the observation of grasping, it was necessary to assess that object size information was indeed present in the motion stimuli. To this end, we analysed the kinematics of the real-life grasping movement in three-dimensional space and assessed that it contained the object information (electronic supplementary material). In agreement with previous studies [12,14], we found that maximum grip aperture (F_1,118 = 60.49, p < 0.001), peak velocity of finger aperture (F_1,118 = 16.26, p < 0.001) and percentage of time to maximal finger aperture (F_1,118 = 11.04, p < 0.01) significantly increased when subjects grasped the two large objects (sphere and cube) with respect to the two small objects (trunk and cylinder). No size effect was found on arm peak velocity (F_1,118 = 0.002, p > 0.05), on percentage of deceleration time (F_1,118 = 0.03, p > 0.05) and on reach phase duration (F_1,118 = 0.61, p > 0.05). None of these parameters were significantly different for grasping movements directed to different shaped objects of the same size (square versus sphere or cylinder versus trunk; see table of the electronic supplementary material), indicating that the discrimination of object shape might not be possible. In a three-dimensional view, all these parameters are equal between allocentric and egocentric perspectives. However, the geometrical projection of the three-dimensional movie on the plane (see example sequence in figure 2a,b) induces changes that depend upon perspective. In the egocentric perspective, the maximum grip aperture (F_1,118 = 41.07, p < 0.001) as well as its maximum velocity (F_1,118 = 6.87, p < 0.01) are significantly different for the small and large objects, while the difference is annulled in the allocentric view.

Figure 1 reports the mean probability of discriminating the shape of the object goal of the reach-to-grasp movement for the actor group. The performance in identifying the trunk from the cylinder and the cube from the sphere was nearly at chance, not surprising as the basic kinematic parameters were not different between grasps to objects of the same size. There was also no bias of the response categories, the correct answer for the large- and small-object classifications differing by less than 3 per cent. Similarly, no significant bias was observed for the confusion matrix calculated independently for the two different perspectives and for the agent. Given these results, we analysed the data only for size discrimination, calculating the performance for discriminating sphere or cube from cylinder or trunk.

Figure 1.

Plot of the confusion matrix of the 4-AFC discrimination performance. The colour map indicates the identification performance (% correct) averaged across subjects for each individual object; the spread of each coloured blob is proportional to the standard deviation of the mean.

Figure 2 shows the subjects' performance for the size discrimination task observing the full movement kinematics (abscissa) or the last frame (ordinates, static condition) in either egocentric (figure 2c) or allocentric perspective (figure 2d). The comparison between the static and the full movie yields the contribution of the kinematics towards the size discrimination performance. We used the last frame of the video (presented for 250 ms, static condition) because it contains the maximum information about object shape and size as demonstrated by Santello & Soechting [32].

In the egocentric view (figure 2c), the majority of data falls below the unity line, indicating a greater ability to discriminate object size when all the movement kinematic information is available. The average values shown by the arrows indicate that the subjects performed well above threshold (defined as d′ = 1) for the dynamic stimuli, and at threshold for the static stimuli. When the stimuli were delivered in the allocentric view, however, there was no advantage for the full dynamic movie (figure 2d). A two-way repeated-measures ANOVA shows that the performance for static and dynamic stimuli is significantly different (F_1,72 = 18.1, p < 0.001) as is the interaction with the perspective factor (F_1,74 = 5.01, p < 0.05). The performance in discriminating static stimuli in allocentric and egocentric perspectives is not significantly different, both when the exposure was 250 ms and when it was prolonged at 500 ms and when the maximum grip aperture frame was used instead of the final frame of the movies (figure 4b). The performances of these various classes of static stimuli vary between a d′ of 0.4 and 2, but a perspective facilitation was never observed (two-way ANOVA shows that perspective (p = 0.65) and the interaction between perspective and stimulus type (p = 0.88) were not significant). On the other hand, the facilitatory effect of perspective is strong and robust and the increase in performance for the egocentric view is present in all groups, as shown in figure 3, which reports the average between subjects. Importantly, it is present also in the subjects who never performed the grasping task (non-actor group), indicating that the effect is not due to prior information about spatial constraints in performing the grasping movement or to a cognitive strategy associated with the movement. It is also interesting to note that the performance is better for the group of subjects who were also the actors of the original grasping movie when they observed their own movements (actor own movement).

Figure 4.

(a) Scatter plot of the size identification performance (d′) in egocentric and allocentric perspective for dot biological motion movie (blue triangles) and for the stick–dot hand movie (green squares). The stick–dot hand movies in the egocentric perspective are the same as in figures 1 and 2, and in the allocentric perspective are obtained by rotating by 180° the egocentric movies. The arrows on the axes indicate the mean size identification performance for each stimulus type. (b) Solid bars: mean performance in egocentric (grey columns) and allocentric (orange columns) viewpoint for the static stimuli corresponding to the last frame of the movie displayed for 500 (first two bars) and 250 ms (second pair of bars) and for the frame displaying maximum grip aperture for 500 ms (third pair of bars). Hatched bars: performance for the same movies of figure 2 displayed backwards, from the putdown frame to the initial starting point. The dashed lines show the average performance for the dynamic movies of figures 1 and 2.

Figure 3.

Mean advantage of dynamic versus static stimuli for actor and non-actor groups, in egocentric (grey columns) and allocentric (orange columns) viewpoint. The actor group is subdivided for observing their own and others' grasping action.

The analysis performed on the two-dimensional movies shows that grip aperture is different between allocentric and egocentric perspectives and this is probably true also for other hand conformation cues (see movies in the electronic supplementary material and table S1). In principle, it would be possible that in the egocentric view the dot-line movie would be easier to recognize and to interpret, explaining the facilitatory effect. To test this possibility, we performed a series of control experiments. We measured the number of occurrences of occluding signals (cross-over of the line and dots) in the two perspectives for all the frames and in the last frame of the two-dimensional movie and found they were not significantly different for the two different points of view (mean occlusion rates were 238 ± 72 (s.d.) and 265 ± 75 pixels for all frames and 173 ± 87 (s.d.) and 155 ± 106 pixels for the last frame for the egocentric and allocentric views, respectively).We performed three further controls. Firstly, we used movies in which the lines were deleted, leaving only the dots. In these movies, the number of occlusions is very small and individual still frames cannot carry useful information given that the hand shape cannot be easily recognized. Also in this condition, subjects show a preference for the egocentric perspective (figure 4a, blue triangles). Secondly, we measured performance for two movies that contained identical form information, apart from an overall rotation of 180°. Nevertheless, the size discrimination performance was higher in the egocentric view (figure 4a, green squares), confirming the data of figure 2. A mixed multifactorial ANOVA calculated for the three different types of movies (dot-stick, dot and rotated movie) and the two perspectives revealed that the only significant factor was perspective (F_1,66 = 23.52, p < 0.001). Thirdly, we measured performance for release-and-withdraw actions (figure 4b, hatched bars), where the visual motion is locally temporally reversed, but the hand posture of each frame is unaffected. Size discrimination performance for the backward movie was greatly impaired, and became equal in the two perspectives. Overall, the effect is specific to reach-and-grasp actions and independent of the specific hand conformational visual cues.

To control for left- or right-hand specificity, we measured object size discrimination in egocentric and allocentric views for the left hand (obtained by mirror-inverting the two-dimensional movie). The performances were not significantly different (egocentric view: average left-hand performance 1.82 ± 0.44 (s.d.; n = 8), right-hand performance 1.61 ± 0.45 (n = 20), F_1,18 = 1.2, p > 0.05; allocentric view: average left-hand performance 1.56 ± 0.47 (s.d.; n = 8), right-hand performance 1.33 ± 0.58 (n = 20), F_1,18 = 1.61, p > 0.05). ANOVA, performed on the discrimination data of the 20 subjects (figure 2) for the right hand and eight for the left hand, revealed that the perspective effect is significant, but not the factor hand (p = 0.14). Also, the interaction between hand and perspective was not significant.

4. Discussion

The present results show that the visual system can infer correctly the size of an object to be grasped by observing the hand posture and the reaching movement. In addition, they show that the kinematic information induces an improvement in discrimination when presented in a viewpoint consistent with the subject's own movement (egocentric perspective). The control experiments demonstrate that the perspective effect is consistent across the hand used for the grasping, type of reaching dynamics (the rotate movies experiment), but not for release-and-withdraw motion.

The visual information contained in the egocentric and allocentric perspective two-dimensional stimuli is not identical, mainly because in the egocentric view, the fingers are shown in transparency behind the palm and this may generate an additional difficulty in the size discrimination task. However, it is unlikely that this is the cause for viewpoint selectivity for several reasons. Firstly, performance for the static stimuli is equal in the two viewpoints, and these stimuli suffer from the transparency problem as do the dynamic stimuli. Secondly, the perspective effect is absent when subjects observed movies in backward motion, although they contain the same hand conformational shape cues. Thirdly, the effect is retained in movies that are identical except for a rotation. Importantly, these movies are still perceived as a reach-and-grasp action in egocentric and allocentric perspectives. Furthermore, the viewpoint selectivity cannot be ascribed to a differential sensitivity to visual biological motion. Even for visual signals that we do not experience in real life, like our-own body ambulation movements, visual recognition is invariant with perspective [33], suggesting that it is not the discrimination of biological motion of the reach-and-grasp action that it is impaired in the allocentric view, but rather that the relevant information is not used for object size discrimination.

Shmuelof & Zohary [34] have shown that the BOLD response of the left parietal cortex (superior parietal lobule) increases preferentially when subjects observe grasping actions performed with the right hand in the egocentric perspective and with the left hand in the allocentric perspective. All subjects in the present study were right-handed and presumably have a stronger representation of the grasping motor programme in the left hemisphere, which is engaged by the right hand in the egocentric perspective. Given that an egocentric facilitation was also measured during observation of the left-hand grasp, the viewpoint selectivity of the present results does not directly reflect the stronger motor representation of the left hemisphere, but rather is general and present for all biological motions that are consistent with the observer performing the grasping action.

Some studies have demonstrated that performing an action can change the visual perception of its consequence, such as the judgement of visual time [35], size [36] or three-dimensional object recognition [3,4]. Recently, evidence for a direct influence of motor signals in visual areas is emerging. For example, it has been demonstrated that performing a pointing action with an unseen hand elicits a response in the visual LO cortex, which represents body parts (EBA area [5,6]). LO is a major associative cortical area comprising several subregions highly specialized [37] for shape and object recognition. Although at this stage speculative, it is possible that cortical activity modulation observed for the pointing might also be present when performing a grasp, in this case not for the EBA region that encodes body parts, but possibly for the region of LO that encodes solid objects. These modulatory signals could be the mechanisms needed to mediate the priming influence of the motor signal on visual discrimination.

In the present study, we did not require our subjects to perform actions, but to passively observe. Nevertheless, the selectivity for viewpoint shows that the effect cannot be solely visual, but implicates the action of the motor system. Observing a reach-and-grasp action can strongly facilitate object size estimation, provided that the viewpoint is congruent with the egocentric perspective. These results are consistent with much evidence showing a cortical activation selective for viewpoint of observation [34,38]. Egocentric view generates a greater response in the sensory–motor area located in the anterior section of the superior parietal lobule and a stronger facilitation of motor-evoked potentials [39,40]. In addition, when the egocentric view is also associated with an illusory attribution of the action (rubber-hand illusion), the facilitation of the motor system is reduced, indicating that the effect is strongly interlinked with the online execution of the motor programme [41].

The subjects who were the actors for the grasping movies showed a trend of performing better in size discrimination than the other subjects. It is unlikely that the effect is due to easily recognizable grasping strategies peculiar for some of the actors and object sizes. The effect of the agent has already been described in the literature. Usually, recognition is better for detecting biological movement on one's self [33], and learning new motor skills (based on verbal or haptic, but not visual, feedback) improves selectively visual recognition of the learned kinematics [42]. Interestingly, athletes infer the outcome of an action more accurately of the sport discipline that they practise [43], than do individuals with comparable visual experience. Expert dancers activate strongly premotor, parietal and cerebellar areas when dancers viewed moves from their own motor repertoire, compared with opposite-gender moves [44]. In all these cases, as for the present results, the ability to extract relevant information from observing movement kinematics is based not only on visual recognition processes but also on one's own motor repertoire, indicating an agent's selectivity and the involvement of the motor and mirror systems.

The present results are consistent with the direct matching hypothesis [9], that observing an action engages the same motor programme used to generate it. However, at first sight, the perspective selectivity for size discrimination might be interpreted as evidence contrary to the mirror-neuron system mediating the modulation of allocentrically observed actions. The mirror system was proposed after descriptions in monkey ventral premotor cortex of neurons that respond selectively both to their own physical movements and to observation of the same actions made by other individuals [45]. This system has been interpreted as a way to match observation and execution of actions [9]. Interestingly, recent data (A. Casile 2009, personal communication) show that many mirror neurons are selective for the egocentric rather than the allocentric viewpoint, resolving the apparent discrepancy between the present data and the presumed function of the mirror-neuron system. In addition, mirror neurons selective for peri-personal space have been reported [46], and these may well prefer the egocentric viewpoint.

Visual estimation of the three-dimensional object size is usually poor and subject to distortion. It has been suggested that haptic information may be used to calibrate size estimation [47,48]. However, important kinematic parameters of grasping, such as maximum or terminal grip aperture, are not influenced by distorted haptic feedback [10], suggesting that haptic signals might not be the only ones able to subserve visual calibration of size. When available, the visual information of the reach provides useful feedback to adjust grip aperture [10]. The good performance measured here for static frames of the terminal grip aperture further support that biological motion signal of the grasp can also be used for object size judgments. However, and more importantly, the viewpoint selectivity shown only for the dynamic grasp indicates that also motor signals might be involved in providing information to calibrate the visual sense of size. Calibration of visual size is particularly important when performing a grasping action, and much less when observing the action of others, so it may be functionally important that the motor system could directly participate in the calibration process. In the present experiment, the motor signal implicated in the effect might have been the action planning that accompanies the observation of grasping [7,9], given that no real movement was performed. It remains for future research to assess whether the action planning and the related intention-to-move signal can exert a similar visual calibration in more natural conditions during real grasping.