Neural Integration of Information Specifying Human Structure from Form, Motion, and Depth

Abstract

Recent computational models of biological motion perception operate on ambiguous two-dimensional representations of the body (e.g., snapshots, posture templates) and contain no explicit means for disambiguating the three-dimensional orientation of a perceived human figure. Are there neural mechanisms in the visual system that represent a moving human figure's orientation in three dimensions? To isolate and characterize the neural mechanisms mediating perception of biological motion, we used an adaptation paradigm together with bistable point-light (PL) animations whose perceived direction of heading fluctuates over time. After exposure to a PL walker with a particular stereoscopically defined heading direction, observers experienced a consistent aftereffect: a bistable PL walker, which could be perceived in the adapted orientation or reversed in depth, was perceived predominantly reversed in depth. A phase-scrambled adaptor produced no aftereffect, yet when adapting and test walkers differed in size or appeared on opposite sides of fixation aftereffects did occur. Thus, this heading direction aftereffect cannot be explained by local, disparity-specific motion adaptation, and the properties of scale and position invariance imply higher-level origins of neural adaptation. Nor is disparity essential for producing adaptation: when suspended on top of a stereoscopically defined, rotating globe, a context-disambiguated “globetrotter” was sufficient to bias the bistable walker's direction, as were full-body adaptors. In sum, these results imply that the neural signals supporting biomotion perception integrate information on the form, motion, and three-dimensional depth orientation of the moving human figure. Models of biomotion perception should incorporate mechanisms to disambiguate depth ambiguities in two-dimensional body representations.

Introduction

Humans engage in myriad activities, each comprising complex, hierarchical, pendular motions of the limbs of the body. These diverse and highly complex motions would seem to pose a serious computational challenge for the visual system to register and identify. Yet this complexity is belied by the apparent ease with which we actually perceive human actions, even when the hierarchical, pendular motions characterizing these actions are conveyed by disembodied human representations portrayed only by dots attached to a dozen or so positions on the body. Termed biological motion perception (Johansson, 1973), this capacity is present in young infants (Fox and McDaniel, 1982) and develops during life to the extent that features such as an individual's gender (Mather and Murdoch, 1994) and identity (Troje et al., 2005) can be easily distinguished on the basis of motion cues alone.

Despite the recent surge of interest in this topic (Blake and Shiffrar, 2007), the fundamental neural mechanisms underlying perception of biological motion remain a matter of debate. Functional neuroimaging studies have found regions in both dorsal and ventral visual streams in the human brain responsive to biological motion. These areas include occipito-temporal and fusiform areas, motion-sensitive hMT/V5, and the posterior superior temporal sulcus (pSTS) (Vaina et al., 2001; Grossman and Blake, 2002; Michels et al., 2005; Peuskens et al., 2005; Peelen et al., 2006; Jastorff and Orban, 2009). Although researchers disagree about the exact roles played by these brain regions in biomotion recognition, a common view is that, at some stage of processing, activity in cells responsive to static two-dimensional representations of the body (e.g., snapshots, posture templates) must be integrated over time to form the neural representation of a moving human figure (Giese and Poggio, 2003; Lange and Lappe, 2006).

In this paper, we investigate whether biological motion stimuli engage neural mechanisms that unambiguously represent the three-dimensional orientation of the perceived human figure. This question has received little attention in the literature on biological motion perception [for exceptions, see Proffitt et al. (1984) or Vanrie and Verfaillie (2006)], and extant models contain no explicit means for disambiguating three-dimensional orientation from two-dimensional postural representations. To pursue this question, we exploited an important, heretofore unrecognized aspect of point-light (PL) biomotion stimuli. Specifically, when devoid of all depth cues, conventional PL sequences are ambiguous with respect to depth ordering of the limbs (Vanrie et al., 2004; Jackson et al., 2008). For some animations, this ambiguity is perceptually evident: observers experience alternations over time between two conspicuously different directions of heading of a PL walker (Fig. 1; supplemental Movie 1, available at www.jneurosci.org as supplemental material), with reversal dynamics similar to those produced by other ambiguous structure from motion (SFM) stimuli (Jackson et al., 2008). In experiment 1, we show that, after adaptation to a PL figure with an unambiguous three-dimensional orientation, perception of bistable versions of the same PL figure is biased away from the adapted heading direction. Experiments 2–6 investigate the adaptation conditions necessary to produce this heading direction aftereffect, thereby revealing key properties of the neural mechanisms supporting three-dimensional biomotion perception.

Figure 1.

Bistable biological motion. The schematic illustrates how a standard PL walker can be perceived with either of two heading directions; in the experiments reported in this paper, observers viewed a bistable PL walker that could be perceived as facing 30° left (frontward-facing) or 150° left (backward-facing) of the observer's line of sight.

Materials and Methods

Participants

Eighteen observers, including both authors, took part in one or more of the experiments (experiments 1a and 1b, n = 5; experiment 2, n = 4; experiment 3, n = 7; experiment 4, n = 9; experiment 5, n = 5; experiment 6, n = 5). All observers had normal or corrected-to-normal vision, and all observers in the stereo experiments had excellent stereoacuity. Observers completed one or two practice sessions monitoring reversals with the ambiguous walker before completing any of the adaptation experiments. Experiments were approved by the Institutional Review Board of Vanderbilt University and all observers gave written, informed consent before participating. A small number of test sessions was completed while the first and second authors were located at University College Dublin and University of Sydney, respectively. Test setup for all sessions was kept as similar as possible.

Stimuli

Ambiguous and unambiguous PL walkers were created using a modified version of a publicly available motion capture recording of treadmill walking (Vanrie and Verfaillie, 2004), resampled at 120 Hz and smoothed using a second-order Butterworth filter. The basic stimulus [3.5 degrees of visual angle (dva) tall] consisted of 13 black dots placed on the various joints of the walker. When the walker's 1 s step cycle is played on loop, smooth continuous walking motion is apparent. The ambiguous walker was plotted at an angle facing 30° to the observer's left (as though facing over the observer's left shoulder, orthographic projection), such that the two alternative percepts were (1) 30° left (frontward-facing) and (2) 150° left (backward-facing).

To disambiguate the PL adaptor for experiments 1–4, the walker was presented separately to each eye with a slight “orientation” disparity, so as to mimic the effects of varying viewpoint onto the walker from each eye (Fig. 2; supplemental Movie 2, available at www.jneurosci.org as supplemental material). One image sequence displayed the walker orientated at 28°; in the other, the walker was orientated at 32°. By varying which eye received which image sequence, an unambiguous frontward-facing (30° left) or backward-facing (150° left) walker could be created. During the test period, the same ambiguous walker sequence was presented to both eyes. For the phase-scrambling experiment (experiment 1b), the adaptor walker was scrambled by offsetting the phase of individual dots. Within each limb (3 dots), one dot was offset to start at a random point within each third of the frame sequence (frames 0–40, 41–80, 81–120). The offsets were chosen separately for each limb, and which dot was assigned which level of offset was randomly chosen. Phase-scrambling the walker in this deterministic manner limited the possibility that, by chance, any one limb would remain mostly intact. To test the effects of adaptation at different implied distances (experiment 2), two versions of the walker were used; the “near” walker (3.5 dva tall approximately; same as in experiment 1) and the “far” walker (1.75 dva approximately). To test the retinal specificity of the adaptation effect (experiment 3), adapt/test walkers, who subtended 1.75 or 3° in the respective eccentricity conditions (1/2.5° eccentricity), were presented on the same or opposite sides of fixation in different trials. The fusion frame was widened for 2.5° eccentricity testing. To test the orientation tuning of the aftereffect (experiment 4), the stereo adaptor was presented in one of three directions in separate blocks: facing 30° left of frontward (30°L), facing directly frontward (0°), or facing 30° right of frontward (30°R). The test walker was always the same bistable PL walker (perceived as facing 30° left/150° left). Experiment 5 tested the effects of a context-disambiguated adaptor on perception of the bistable walker. The PL adaptor was disambiguated by presenting it as though walking on top of a rotating, stereo-defined SFM globe (“globetrotter” configuration) (supplemental Movie 3, available at www.jneurosci.org as supplemental material). Both the walker and globe subtended 1.33 dva in height and were presented as white dots on a gray background. The walker's gait and the rotation of the globe were matched for speed. The walker was presented without disparity throughout. To disambiguate the rotation of the globe, separate image sequences were presented to the left and right eyes as appropriate. To test the transfer of the aftereffect across form (experiment 6), a full-body adapting walker was used (supplemental Movie 4, available at www.jneurosci.org as supplemental material). The full-body adaptor and PL test walker were matched for size and gait speed. The full-body animations were designed with the help of members of the Department of Computer Science, School of Engineering, at Vanderbilt University.

Figure 2.

Stereo presentation of a PL walker. To disambiguate the PL walker's direction for the adaptation experiments, observers viewed image sequences generated separately for the left and right eyes. These were presented by means of a mirror stereoscope. Each image sequence depicted identical walking motions, although with a slight orientation disparity between the eyes, mimicking the fact that the left and right eyes have slightly rotated viewpoints onto an object in the real world. By cross-fusing the images in this figure, readers can perceive a frontward-facing figure rotated 30° left (i.e., as though facing beyond the reader's left shoulder). In the actual experiments, walkers were presented as black dots on a gray background, inside patterned fusion frames. Here, sticks are presented on the PL figure for illustration only.

Procedure

Stimulus presentation.

Observers viewed image frames presented separately to the left eye and right eye through a mirror stereoscope. The half-images were presented on each side of a 21 inch cathode ray tube (CRT). Display resolution was 1024 × 768 (120 Hz), with a viewing distance of 100 cm. A few test sessions were performed using a 19 inch CRT (60 Hz), with viewing distance scaled to maintain the same angular sizes across experiments. All stimuli were presented on a gray background. A square, patterned fusion frame (5.7 dva wide) surrounded each half-image, thereby serving to promote stable binocular alignment of the two half-images. In all experiments except for experiment 5, observers stared at a fixation cross placed over the walker's torso. In experiment 5, observers fixated at or just above the point of contact between walker and globe, such that they could discriminate the walker's direction while maintaining good stereo perception of the globe.

Testing sessions.

In experiment 1a, observers completed eight trials, blocked by adapting direction (four trials per direction). Each trial began with a 90 s adaptation period. After this, the area inside the fusion lock was blanked for 1 s, followed by the presentation of a 75 s test period with the ambiguous walker. Observers monitored the walker's perceived facing direction throughout the entire trial using keyboard responses. Setup and task were identical for experiment 1b, except that observers were not required to monitor the perceived “facing” direction of the scrambled adaptor. Before testing the effects of varying walker size from adaptation to test (experiment 2), observers completed sessions monitoring reversals with the far walker, confirming that approximately comparable reversal rates are experienced with this smaller stimulus. For experiment 2, observers completed eight trials per condition (adapt near/test far, adapt far/test near, adapt far/test far), four each per adapting direction (frontward/backward). Adapt near/test near data were taken from participants' corresponding data from experiment 1a. Trials were completed over two sessions, and adapt and test periods again lasted 90 and 75 s, respectively.

In testing the effects of placing adapt/test stimuli on either side of fixation (experiment 3), adapt/test periods were shortened to 45/30 s, respectively. Four sets of trials were run for each eccentricity condition (1 and 2.5°). Each set consisted of eight test trials (2 adapt sides × 2 test sides × 2 adapt directions), two additional adaptation trials at fixation, and eight 30 s tracking trials with an ambiguous walker placed eccentrically (four per side). For the orientation tuning experiment (experiment 4), observers completed four test trials per condition, with adapt/test periods set at 90/75 s, and four baseline tracking trials with the ambiguous walker (75 s). In the globetrotter experiment (experiment 5), observers first completed a tracking session consisting of 60 s trials with the walker on top of a static globe (eight trials) and walker and rotating stereo globe together (eight trials). The separate adaptation session consisted of 60 s adaptation periods followed by 30 s test. For both the globetrotter (eight trials) and globe alone (eight trials) conditions, the two globe rotation directions were presented an equal number of times, in random order. For the full-body adaptor experiment (experiment 6), observers completed four sets of nine test trials (2 adapt directions × 4 timing sequences plus 1 control trial with a fixation cross as adaptor).

Results

Adaptation to a stereo-defined walker biases an ambiguous walker's perceived facing direction

Extended viewing of an unambiguous stimulus biases subsequent perception of ambiguous versions of the same stimulus away from the adapted percept (Virsu, 1975; Harris, 1980; von Grünau et al., 1984; Nawrot and Blake, 1989, 1991). In experiment 1a, we investigated whether this effect occurs with bistable biological motion by adapting observers to biomotion figures that walked with an unambiguous heading direction, and then testing them with bistable versions of the same figures. Unambiguous walkers were created by presenting the same movement sample to each eye, but with a slight horizontal disparity between the two eyes' views, mimicking the slightly different viewpoint perspectives created when binocularly viewing a three-dimensional object (Fig. 2) (see Materials and Methods) (supplemental Movie 2, available at www.jneurosci.org as supplemental material). In separate trials, we created unambiguous, frontward-facing (30° left) or backward-facing (150° left) walkers by varying which eye received which image sequence. On each trial, observers viewed the disambiguated walker for 90 s. Immediately after this adaptation period, an ambiguous version of the very same walker was presented (i.e., no horizontal disparity between the eyes), and observers tracked perceived direction of this bistable walker for 75 s. Observers completed four blocked trials per adapting direction, with at least 1 min rest between trials.

A very strong adaptation aftereffect was experienced by all observers tested (Fig. 3A). Regardless of the unambiguous walker's heading direction, observers perceived the subsequent ambiguous figure to face in the opposite direction for the majority of the 75 s test period; indeed, on some trials observers perceived only the opposite walking direction during the entire 75 s test. The strength of this aftereffect is striking, considering that in preadaptation sessions the same observers experienced reversals with the ambiguous walker typically every 3–10 s (range of observer means). The fall-off over time in the aftereffect is displayed in Figure 3B. For many trials, recovery to a state in which either percept was equally probable did not occur during the 75 s test period; additional trials using a longer test period found that recovery could take up to 90 s or more. To appreciate the strength of this aftereffect, consider that aftereffects after 90 s adaptation to a kinetic depth globe have previously been found to last ∼45 s before returning to a state of equal dominance (Nawrot and Blake, 1991).

Figure 3.

Adaptation aftereffects with bistable biomotion. A, Plots present the mean number of seconds of the 75 s test period during which the perceived heading of the PL walker was opposite to that experienced during adaptation (n = 5). The bars represent the mean of four trials per adapting direction. Results from frontward adaptor (top) and backward adaptor (bottom) trials are given separately. The aftereffect was equally strong for both adaptor types (t₍₄₎ = −0.034; p = 0.97). B, Proportion dominance of the opposite percept during consecutive 15 s segments of the 75 s test period. The curves present the mean of eight trials for each observer, collapsed across adapting direction. C, Mean total duration of the opposite percept (collapsed across directions) after adaptation to the stereo-defined walker and to the phase-scrambled walker. For all observers, the aftereffect disappeared when phase-scrambling was introduced in the adaptor (t₍₄₎ = 10.02; p < 0.001). The horizontal dashed lines indicate an equal total duration (or equal proportion) for same/opposite percepts. Error bars represent ±1 SE throughout.

Is local, disparity-specific, motion adaptation responsible for this aftereffect?

It is tempting to assume that the neural site of adaptation is some brain region selective for high-level information associated with these PL animations, such as the figure's global form or general direction of heading. It is possible, however, that this seemingly high-level adaptation, in fact, arises from local, disparity-selective motion adaptation. According to this alternative explanation, each of the walker “dots” adapts its own, unique, stereo-defined trajectory, such that for any dot location, a dot subsequently presented in the same location but without disparity would be biased away from the adapted trajectory in depth.

To test this possibility, we performed a second experiment (experiment 1b) identical with the first, except that the dots comprising the stereo-defined adaptor were now phase-scrambled in time: each of the 13 dots followed the same trajectories in depth at the same locations as they did in the coherent walker, but the temporal coherence of dot motions was randomized so that the walker appeared globally incoherent (see Materials and Methods). If the observed aftereffect in experiment 1a were the result of adaptation to individual, local trajectories in depth, this phase-scrambled stimulus should produce a comparable degree of adaptation. Adaptation to the phase-scrambled walker, however, had no effect on subsequent perception of the ambiguous walker, with each percept having close to equal probability immediately after 90 s of adaptation to the phase-scrambled, disparity-defined dots (Fig. 3C). This finding rules out adaptation of local dot trajectories as the explanation for the adaptation effect of the first experiment.

Having confirmed that this adaptation aftereffect indeed reflects the operation of neural mechanisms involved in registering biological motion, we performed several additional experiments to study properties of those mechanisms. Results from those experiments are described in the following sections.

Scale invariance of the aftereffect

An interesting feature of neural coding in object-selective brain regions is the scale-invariant manner in which cells respond to their preferred object (Ito et al., 1995; Logothetis et al., 1995). This characteristic has been reported in cells in the anterior part of the macaque superior temporal polysensory area (STPa) that respond to bodies in motion (Bruce et al., 1981; Perrett et al., 1985). (Biomotion-selective cells in the macaque are not limited to STPa, located in the upper bank region of STS, but are also found in the lower bank and in the fundus region, a point we shall consider in subsequent sections of this paper.) Presumably, scale invariance provides an efficient means of representing objects that are regularly seen at different distances from the viewer, as well as representing object categories whose members naturally vary in size. The results from our first experiment led us to conclude that adaptation is occurring within a high-level neural representation of the human figure portrayed by the PL stimuli. In light of the scale-invariant property of object-selective neurons, this conclusion implies that adaptation to an unambiguous PL walker should generalize across changes in walker size. This implication was tested in experiment 2 by presenting trials that were identical with previous sessions, although this time adapt and test walkers could subtend one of two possible visual angles (Fig. 4A)—a near walker identical in size with the walker in the first experiment (∼3.5 dva tall) and a far walker that subtended one-half this angular size (∼1.75 dva tall).

Figure 4.

Scale invariance of the aftereffect. A, Illustration of the relative size and implied distance of stimuli used in the scale invariance experiment; note that the actual experimental stimuli were PL walkers. In different trials, observers adapted to a near walker (3.5 dva tall) and tested with an ambiguous far walker (1.75 dva tall); in others trials, the opposite occurred. B, Mean total duration of the opposite percept, collapsed across adapting directions. Results are also presented for adapt near/test near and adapt far/test far trials for comparison. The bars represent the mean of eight trials per condition. Comparing conditions in which the test stimuli were kept identical, no significant differences were found between adapt near and adapt far conditions (test near: t₍₃₎ = 0.56, p = 0.62; test far: t₍₃₎ = 1.97, p = 0.14). The horizontal dashed lines indicate an equal total duration for same/opposite percepts. Error bars represent ±1 SE.

We tested three different combinations of adapt size/test size: near/far, far/near, and far/far. Results from these conditions are plotted in Figure 4B, together with results for each observer tested on the near/near condition from the previous experiment. All observers experienced a robust aftereffect, whether or not the test and adapt figures were the same size. Indeed, there was no evidence that the size relationships between adapt and test mattered at all, as evidenced by the nonsignificant statistical differences between conditions (Fig. 4, legend). Thus, this global adaptation effect is indeed scale invariant, at least within the range of sizes tested here.

Position invariance of the aftereffect

The scale-invariant nature of this aftereffect implies that absolute dot positions can vary from adapt to test without weakening aftereffect transfer. This property suggests that the adaptation effect may show some degree of position invariance. There are also other reasons to believe that biomotion-related aftereffects of the sort investigated here might transfer to locations beyond the region exposed to the adaptation stimulus. For one thing, brain imaging in humans reveals that PL animations imaged in one visual field can produce bilateral activations spanning the vertical midline (Grossman et al., 2000). For another, neurophysiological evidence shows that single cells in both ventral and dorsal pathways can have large receptive fields that extend into the ipsilateral visual field; for example, several classic studies cite receptive field sizes of 10 × 10° in extent and greater (Gross et al., 1969; Bruce et al., 1981; Motter et al., 1987; Ito et al., 1995). Cells sensitive to biological stimuli are known to be present in both the upper and lower banks of monkey STS, namely the STPa region (Bruce et al., 1981; Oram and Perrett, 1996) and an area on the lower bank referred to as the LST (lower superior temporal) region (Nelissen et al., 2006; Vangeneugden et al., 2009). We do appreciate, however, that discussion of position invariance must be treated in relative terms. Cells or groups of cells that respond to the same object in different parts of the visual field may do so in a graded manner, thus retaining some ability to code for position (Op de Beeck and Vogels, 2000). Still, we felt that any transfer of the aftereffect from one region of the visual field to another would be revealing, for it would set this aftereffect apart from other motion-related aftereffects that do not transfer across the vertical midline (Nawrot and Blake, 1991; Meng et al., 2006).

Experiment 3 assessed the position specificity of the biomotion aftereffect by testing observers (n = 5, including both authors) on trials in which adapt and test walkers (1.75 dva tall) appeared in different locations symmetrically placed each side of a foveally viewed fixation mark (Fig. 5A). In one set of trials, those two locations were 1° to either side of fixation (i.e., adapt and test figures were separated by 2°), and in another set of trials the locations were at 2.5° eccentricity (i.e., 5° separation); in this latter condition, the walker's size was scaled up to 3 dva. For neither of these conditions involving eccentric stimulation did the screen positions of any of the walker dots ever overlap with the fixation point. For both eccentricities, comparison trials were administered in which adapt and test stimuli appeared either both to the left or both to the right of fixation, as well as trials in which both stimuli appeared at fixation. Given this large number of conditions and the fact that aftereffects had been reliably observed thus far, we shortened adapt/test durations to 45/30 s, respectively.

Figure 5.

Position invariance of the aftereffect. A, Schematic illustrating the eccentric placement of adapt/test walkers (see Materials and Methods). B, Opposite percept ratio averaged across observers at 1° (left; n = 5) and 2.5° (right; n = 5) eccentricities. Three observers took part in both conditions. Ratio measures were calculated by dividing the duration of the opposite percept after adaptation by the mean duration of that percept when presented in tracking trials at the eccentric location (i.e., separate denominators for left-side and right-side test locations). The horizontal dashed lines indicate a postadapt/control ratio of 1 (i.e., no adaptation). Error bars represent ±1 SE computed across observer means. At 1°, no main effect was found for test location relative to adapt location (i.e., the aftereffect transferred across hemifield) (F_(1,4) = 3.02; p = 0.16). No other main effects or interactions were significant. Although aftereffects were equivalent for same/opposite hemifield presentation at 1° (and equivalent in magnitude to trials in which both stimuli appeared at fixation) (data not shown), on average aftereffects transferred less strongly across hemifield in the 2.5° eccentricity condition. The main effect for same/opposite hemifield, however, was not significant (F_(1,4) = 2.36; p = 0.2), although this lack of significance may have resulted from the presence of one outlier dataset. Analyses revealed a significant interaction between adaptor direction and same/opposite hemifield presentation (F_(1,4) = 11.8; p < 0.05) and a trend toward interaction between adapt side (left/right) and same/opposite hemifield presentation (F_(1,4) = 7.13; p = 0.056). See Results for discussion.

In some contexts, walker orientation and hemifield of presentation may interact in biological motion perception (de Lussanet et al., 2008); to control for this possibility in our task and to ensure that we obtained a pure measure of adaptation, we compared the percept dominance proportions after adaptation to corresponding proportions taken from tracking trials with an ambiguous walker placed at the eccentric positions (Fig. 5, legend). Ratio scores >1 indicate an aftereffect of adaptation, with the opposite, nonadapted percept experienced for more time after adaptation than during control tracking periods. Analysis of the control tracking data at 1° eccentricity did reveal a small but significant difference in the average proportions of time spent perceiving the alternative percepts—the frontward-facing figure was experienced more often left of fixation than right of fixation (mean proportion left, 0.545; mean proportion right, 0.44; t₍₄₎ = 3.73, p < 0.05). This effect was also clearly present at 2.5° eccentricity (mean proportion left, 0.606; mean proportion right, 0.437; t₍₄₎ = 3.6, p < 0.05). We have no ready explanation for this visual field asymmetry and its relationship to other findings (de Lussanet et al., 2008). This asymmetry, however, does not invalidate our comparison of control and adaptation trials; indeed, the asymmetry underscores the importance of using ratio measures as an index of aftereffect strength for the eccentricity experiments.

At 1° eccentricity, aftereffects transferred robustly across the vertical midline for all observers (Fig. 5B). Thus, for example, adaptation to a frontward-facing walker placed 1° left of fixation caused a subsequently viewed bistable walker imaged 1° to the right of fixation to appear to walk in the backward-facing direction. Note that this result contrasts with what occurs with rigid SFM (Nawrot and Blake, 1991) and reversible perspective figures (von Grünau et al., 1984; Toppino and Long, 1987) for which the effects of adaptation are confined to the visual field location of the adapting stimulus. Of course, one could argue that 2° of separation between adapt and test stimuli (i.e., the separation produced by the 1° eccentricity condition) is simply too small to guarantee that adapt and test stimulation never overlapped. Even though we ensured that the screen positions of the walker dots never encroached on the fixation area, we cannot rule out the possibility that observers might occasionally have made small, inadvertent eye movements that brought a walker stimulus closer to the vertical midline. We did stress the importance of maintaining strict fixation, but we did not record eye movements.

Testing with 5° separation of adapt and test stimuli (i.e., 2.5° eccentricity), we still found evidence for transfer of adaptation across the vertical midline, as indicated by a nonsignificant main effect for test location relative to adapt location (Fig. 5B, legend). For four of the five observers, however, aftereffects that transferred from one hemifield to the other were somewhat weaker than when adapt and test stimuli appeared on the same side of fixation. We also noted a significant interaction between adaptor direction and same/opposite hemifield presentation, and a trend toward interaction between adapt side (left/right) and same/opposite hemifield presentation (Fig. 5B, legend).

For several reasons, we feel it is premature to draw conclusions about the exact “site” of adaptation based on these eccentricity results. First of all, the interactions observed in the 2.5° eccentricity condition suggest a complex interplay among several factors. For example, although our control tracking trials were designed to factor out potential hemifield biases with the bistable walker, one could envisage a situation in which some residual effect remains in the measurement of the aftereffect. If this were the case, then a backward-facing adaptor might be most successful at producing an aftereffect when the test stimulus appears on the left side of fixation, regardless of the location of the adaptor relative to test (reflecting the consistent tendency for more frontward-facing percept left of fixation). Conversely, a frontward-facing adaptor might be more successful at producing an aftereffect when the test walker appears on the right of fixation (again reflecting the consistent biases we found in control tracking). In addition, the possibility that inherent biases in attentional allocation relative to fixation play a part in the eccentric experiments cannot be ruled out. This could be a particularly important factor in the 2.5° eccentricity condition, which most observers found quite challenging. It should be stressed that tracking the bistable walker, even at these relatively small eccentricities, is an attentionally demanding task, entirely different from the typical left/right direction discrimination tasks used in other biomotion studies that use nonreversing stimuli. Finally, from physiological studies of biomotion-selective brain regions in macaque (Bruce et al., 1981; Oram and Perrett, 1996; Nelissen et al., 2006; Vangeneugden et al., 2009), we know that cells selective to biomotion exist in both upper and lower superior temporal regions. The issue of homology between these regions and biomotion-selective regions in humans will be touched on in Discussion (for an insightful review of this issue, see Jastorff and Orban, 2009). However, we can say that the relatively robust transfer across hemifield at 1° and the finding of at least some transfer at 2.5° again implies that this is indeed a high-level visual aftereffect.

To test more critically the exact nature of this aftereffect, we performed three additional experiments. First, we tested what should be a critical property of this aftereffect, “tuning in depth” (i.e., tuning to the heading orientation of the adaptor). We then performed two additional experiments, one questioning the necessity of disparity information in the adaptor, and the second investigating the role of adaptor form.

Is the aftereffect selective for heading orientation?

Is the aftereffect reliant on a strict mapping between the orientation in three-dimensional space in which the adapt walker is heading and the heading orientation of the test walker? Previous psychophysical and neurophysiological results would suggest an affirmative answer to this question (Verfaillie, 1993; Verfaillie et al., 1994; Oram and Perrett, 1996; Daems and Verfaillie, 1999). We investigated this question (experiment 4) by adapting observers (n = 9) to one of three frontward-facing walkers in different blocks (facing 30° left, 0° straight ahead, facing 30° right) while holding the bistable PL test walker constant across blocks (perceived as facing 30° left or 150° left) (Fig. 6A). If the aftereffect is tuned around the heading direction of the adaptor, we would expect the greatest aftereffect for the 30° left condition (i.e., observers should perceive the test walker as “backward-facing” for a considerably longer duration in this condition than in either of the other two conditions, as the 30° left adaptor maps directly onto one of the two bistable walker percepts).

Figure 6.

Tuning of the aftereffect to heading orientation. A, Orientation tuning of the aftereffect was studied by adapting observers (n = 9) to one of three frontward-facing walkers in separate blocked trials (see Materials and Methods). The test stimulus was always the same bistable PL walker (30° left/150° left). B, Backward percept ratio measures for each observer. The horizontal dashed line indicates a postadapt/control ratio of 1 (i.e., no adaptation). Analyses revealed a strong effect of adaptor type (F_(2,16) = 94.6; p < 10⁻⁹). Post hoc tests confirmed that the aftereffect fell off significantly from the 30°L to 0° adaptors (t₍₈₎ = 7.4; p < 0.0001), and from 0° to 30°R adaptors (t₍₈₎ = 5.9; p < 0.0005).

The results confirm this prediction conclusively—aftereffects were overwhelmingly strongest for the 30° left adaptor and fell off considerably for the 0 and 30° right adaptors, implying a definite view dependence to the adaptation effect (Fig. 6B). This result and our finding of position invariance in the transfer of the aftereffect agree nicely with other results. For example, in a transaccadic integration study (Verfaillie et al., 1994), spatial translations of a PL walker in the image plane went mostly unnoticed when those translations occurred during saccades, but changes to the figure's in-depth orientation were readily detected by observers. Those findings imply that the representation carried across saccades is position invariant but tuned to orientation. In addition, view dependence has also recently been reported for perception of static human bodies (Lawson et al., 2009).

During the course of experiment 4, we noted another conspicuous aftereffect, this having to do with the perceived direction of heading of the ambiguous walker viewed during test periods. Specifically, after adaptation to the 0° adaptor (in which the stereo-defined figure appeared to walk directly toward the observer), the ambiguous figure's apparent direction of walking, when perceived in the front-facing direction, nonetheless deviated noticeably away from the 30° heading specified by the stimulus. Similar observations have been described by Jiang and He (2008). Repulsion aftereffects of this sort are characteristic of other forms of visual adaptation as well, including adaptation to curved lines (Gibson, 1933), to translational motion (Levinson and Sekuler, 1976), to spatial frequency (Blakemore and Sutton, 1969), to stereoscopically defined surfaces (Blakemore and Julesz, 1971), and to faces (Leopold et al., 2005). These kinds of repulsion aftereffects are generally interpreted as evidence for some form of population coding in which the perceived quality of a visual stimulus along some dimension (e.g., orientation) is specified by the pattern of activity among neurons tuned to different values of that dimension. In Discussion, we shall return to this idea in the case of biological motion.

Is disparity-defined global motion necessary for the aftereffect?

In the experiments described so far, disparity was used to define the depth relationships among the dots defining the limbs of the walker and, thereby, to disambiguate the direction of walking of this figure. Is disparity essential for generating this aftereffect? To answer this question requires creating a disambiguated walker without using disparity. Ideally, we would like to do this without altering other physical characteristics of the stimulus (e.g., looming dots), so as to keep adapt and test stimuli as physically similar as possible. How can that be done?

To achieve this aim in experiment 5, we took advantage of a previous result demonstrating stabilization of ambiguous SFM when presented in a context suggesting frictional interactions with a separate unambiguous object (Gilroy and Blake, 2004). By placing the bistable PL walker on top of a stereo-defined rotating globe that rotated in one of two directions (Fig. 7A) (see Materials and Methods) (supplemental Movie 3, available at www.jneurosci.org as supplemental material), we were able to stabilize the ambiguous walker's perceived heading direction for considerable periods of time. For example, when the front surface of the globe moved from bottom to top, with the horizontal axis of rotation offset 30° (Fig. 7A, left), the bistable walker appeared to walk predominantly in the forward facing direction, as though the walker's footsteps were providing the thrust for the rotation of the globe. Reversing the rotation direction of the globe and offsetting its rotational axis appropriately produced the opposite result—the walker could be stabilized predominantly in backward-facing mode (Fig. 7A, right). For all observers, the proportion of total time perceiving the friction-compatible interpretation was significantly greater when viewing the potentially ambiguous PL figure walking on top of the rotating, stereo-defined globe (globetrotter stimulus) than it was when viewing the walker in matched control trials with a static stereo globe (Fig. 7B). Moreover, the degree of stabilization varied considerably among observers. These differences in the strength of stabilization allowed us to pose a simple question with a potentially revealing answer: is the magnitude of the adaptation aftereffect related to the incidence of perceived stabilization experienced with this globetrotter configuration? If disparity is not critical for producing adaptation, then the strength of the aftereffect should be positively correlated with the degree of stabilization during adaptation.

Figure 7.

Context-disambiguation and adaptation. A, Locating the bistable PL walker on top of a stereo-defined, rotating globe created conditions that promoted stabilization of the bistable walker's perceived direction toward the friction-compatible percept. In the actual experiment, observers viewed the disparity-free walker and stereo globe through a mirror stereoscope; stimuli were presented as white dots on a gray background. B, Mean friction-compatible durations for globetrotter trials and randomly matched control trials (static globe) in the tracking session (see Materials and Methods). The bars represent the mean of eight 60 s tracking trials. With the moving stereo globe, observers (n = 5) experienced the friction-compatible percept for significantly longer durations than in control trials (t₍₄₎ = 4.5; p < 0.05). The horizontal dashed line indicates an equal total duration for friction-compatible/incompatible percepts. Error bars represent ±1 SE. C, In the adaptation session, observers completed 60 s tracking periods, which were immediately followed by 30 s tracking of the bistable walker alone. Control trials consisted of adaptation to the stereo globe alone, followed by 30 s test with the bistable walker. For each trial, the x-axis depicts the total time during the adapt period (60 s) in which the bistable walker was perceived in the friction-compatible mode; on the y-axis, a ratio measure is given, calculated as the total time in which the opposite percept was perceived during the 30 s test period divided by the corresponding measure from the control trials. This ratio measure controls for the possibility that adaptation resulted from stereo signals provided by the globe, as well as ensuring a pure measure of adaptation free of individual observer biases (e.g., observer DB experienced prolonged dominance of frontward walking in many trials). The vertical dashed line indicates an equal total duration for friction-compatible/incompatible percepts during the 60 s adapt period; the horizontal dashed line indicates a postadapt/control ratio of 1 for the test period (i.e., no adaptation).

Figure 7C plots results on each trial (n = 8) for each observer (n = 5; denoted by different symbols). Clear individual differences are seen in the degree of stabilization, with several observers experiencing essentially complete stabilization during adaptation. The strength of the aftereffect after exposure to the globetrotter is significantly correlated with the degree of stabilization experienced during test (r = 0.62; p = 0.00002). Control trials in which observers adapted to the stereo globe alone, followed by the test walker, confirmed that the aftereffect was not attributable to depth signals provided by the globe. Evidently, then, it was the implied, frictional interaction between a rigid object and its nonrigid controller that was paramount. It is worth noting that for several observers who experienced very strong stabilization, the resulting aftereffect was comparable with their previous results obtained with the stereo-defined PL walker.

Does the aftereffect transfer across stimulus type?

The findings presented so far point to the involvement of temporary adaptation within a population of neurons that represent direction of heading in a manner that is not specific to a particular stimulus size or location. In addition, adaptation could be produced by a two-dimensional PL walker whose direction was biased by contextual cues, implying that PL walkers portrayed without explicit disparity engage neuronal representations that operate in the three-dimensional orientation space activated by disparity-defined figures. Our results suggest, in other words, that standard PL animations containing ambiguous two-dimensional motion signals can nevertheless activate a “three-dimensional” representation of a moving human figure. This idea has received little emphasis in the literature on biological motion perception [for exceptions, see Proffitt et al. (1984) or Vanrie and Verfaillie (2006)]. More importantly, current models of biomotion perception contain no explicit means for disambiguating the in-depth orientation of the human figure (Giese and Poggio, 2003; Lange and Lappe, 2006).

If neural representations of direction of heading are indeed a general feature of biomotion perception, then it should be possible to disambiguate temporarily an ambiguous PL walker by previous exposure to an animation portraying a full-body figure walking unambiguously in a given direction. To test this prediction (experiment 6), we replaced our disparity-defined PL adaptation sequence with a full-body adapting walker devoid of detailed surface features or gender-related information (Fig. 8A; supplemental Movie 4, available at www.jneurosci.org as supplemental material). After adaptation to one of these figures, observers tracked the bistable PL walker as usual. Using full-body stimuli also allowed us to investigate the possibility that form-only stimuli (i.e., representations of static body posture) promote adaptation. Thus, in addition to trials in which observers adapted to the moving full-body figure (form-motion), we also included trials on which observers adapted to single static frames or sequences of frames chosen randomly from the walking sequence (form-only) (Fig. 8, legend) (see Materials and Methods).

Figure 8.

Transfer of adaptation across stimulus type. A, Observers completed trials adapting (45 s) to a full-body walker (form-motion) or to sequences of random, static postures taken from the same walking sequence (form-only). During form-only trials, the random posture was replaced with another static posture every 2.5 s, 15 s, or not at all (45 s condition). After adaptation, observers tracked (30 s) the perceived direction of a bistable PL walker as usual. Observers completed control trials in which the adaptation period consisted of fixating only the fixation cross over a blank background. B, Mean opposite percept ratio measures across observers (n = 5), calculated by dividing the duration of the opposite percept after adaptation by the mean duration of that percept in control trials. A repeated-measures ANOVA revealed a strong effect of adaptor type (F_(3,12) = 30.03; p < 10⁻⁶). For the form-motion and 2.5 s conditions, observers had on average larger ratio measures for frontward-facing adaptor trials leading to a small interaction (F_(3,12) = 3.94; p < 0.05). This was most likely related to the tendency in observers tested for smaller backward-percept denominators. Data are collapsed across adapting direction as a similar trend was apparent for both frontward and backward adaptor trials. Post hoc tests confirmed that the aftereffect was significantly stronger for the form-motion condition than for the form-only (2.5 s) adaptor (t₍₄₎ = 5.3; p = 0.0059), or for either of the mostly static adaptors (both p < 0.005). A modest aftereffect in the PL walker's direction was experienced when observers adapted to static postures that alternated every 2.5 s, although with Bonferroni's correction for multiple comparisons this was only marginally significant (compared with 45 s condition: t₍₄₎ = 3.5, p = 0.025; compared with 15 s condition: t₍₄₎ = 4.6, p = 0.0098). The horizontal dashed line indicates a postadapt/control ratio of 1 (i.e., no adaptation). Error bars represent ±1 SE computed across observer means.

The results were unequivocal: adaptation to a full-body figure walking in place smoothly in a given, unambiguous heading direction, produced a robust change in perception of the otherwise ambiguous PL walker (Fig. 8B). These results definitively show that the presentation format of the adaptor's motion is not important: both full-body and PL figures produce robust adaptation when those figures are seen to walk predominantly in one direction.

Does this robust biomotion aftereffect require the presence of a moving human figure during adaptation or, alternatively, can robust adaptation be induced through prolonged viewing of static images of the human body oriented in a given direction? The present results reveal that adaptation to a static human posture chosen randomly from the walking sequence has no influence on perception of the ambiguous PL walker; rather, the walker was seen to alternate in heading direction just as it does without previous adaptation (Fig. 8B, 15 and 45 s conditions). It is interesting to note that an aftereffect did occur when observers viewed successive images of different, static postures presented every 2.5 s; however, this form-only aftereffect was significantly weaker than that produced by adaptation to the full-body figure that walked smoothly in place (Fig. 8B). So why does the 2.5 s condition produce an aftereffect at all? One possibility is that the regular, 2.5 s update of the static posture adapts a larger number of form templates/snapshots than when postures are updated less frequently (15 and 45 s conditions). Alternatively, the 2.5 s presentation regimen could have created multiple, chance instances of apparent motion during the adaptation period, a possibility much less likely with the 15 s presentation regimen and impossible with the 45 s condition. For our purposes, it does not matter why the 2.5 s regimen produces a weak aftereffect. What does matter is that disambiguation of an ordinarily ambiguous PL walker can be generated by full-body animations as well as by PL walkers disambiguated by disparity.

Discussion

Previous adaptation experiments with biomotion have focused on the perceived gender of a PL walker (Jordan et al., 2006; Troje et al., 2006) or on the neurophysiological correlates of brief periods of adaptation to PL displays (Hirai and Kakigi, 2008). The experiments presented here are the first to investigate the bistable nature of PL stimuli using an adaptation method, and the results highlight an important feature required of models seeking to explain biological motion perception, namely, the explicit representation of the three-dimensional configuration of the human body. In a variety of experiments, we found strong adaptation aftereffects when the perceived direction of a bistable walker was judged after prolonged exposure to an unambiguous walker. These aftereffects transferred across scale, position, and stimulus type, and did not require stereo depth information in the adapting walker.

One current model of biological motion perception emphasizes the initial processing of local motion and orientation information in early visual cortex, the separation of this processing into motion/form streams in extrastriate cortex, and the eventual reintegration of these signals in pSTS (Giese and Poggio, 2003). A similarly well developed model, in contrast, focuses directly on the temporal integration of information relating to the postures of the human figure (Lange and Lappe, 2006). The intriguing implication of our results is that either at the stage at which snapshot/template cells become activated or at the later stage at which dynamic form (Lange and Lappe, 2006) or motion pattern processing (Giese and Poggio, 2003) is thought to occur, a three-dimensional representation of the human figure is available. A future goal for these models, therefore, is to incorporate mechanisms that perform this disambiguation. To put it in other words, the endpoint of the local integration (Giese and Poggio, 2003) or template matching/integration (Lange and Lappe, 2006) processes should not be simply a two-dimensional representation or “winning” template (both of which are inherently bistable representations) that solves, for instance, leftward/rightward direction discrimination tasks. Instead, the computational route must incorporate a component relating to the figure's orientation in depth, a point we consider in more detail in the following section.

Neural mechanisms supporting biomotion perception

Consider the responses after adaptation within a population of biomotion-selective cells, each tuned to a particular heading direction, and whose combined pattern of activity represents the perceived heading direction of the moving human figure being viewed. There is considerable evidence, both from psychophysics and neurophysiology, for a multichannel organization in the neural mechanisms supporting biomotion and human body perception (Perrett et al., 1985; Verfaillie, 1993; Oram and Perrett, 1994, 1996; Verfaillie et al., 1994; Daems and Verfaillie, 1999; Lawson et al., 2009). The aftereffect we observed displayed a very definite tuning to orientation, with the strongest adaptation occurring when the adaptor's orientation mapped directly onto one of the two possible perceptual interpretations afforded by the test walker. A recent report of heading direction aftereffects using a contrasting PL setup points to similar conclusions (Jiang and He, 2008). Thus, a reasonable conclusion is that the aftereffect studied here results from adaptation of a subset of biomotion-selective cells whose preferred heading orientations are centered at or near the heading of the adaptor figure. This account provides a straightforward interpretation for our results—after adaptation, reductions in response gain at and around the adapted orientation would lead to the relative strengthening of the neural representation of all other orientations. Thus, during the subsequent test period, the nonadapted heading should temporarily but reliably dominate perception.

Where in the visual cortex might such mechanisms be found? Recent physiological findings shed light on the question of where neural information specifying biological shape and kinematics is represented in macaque visual cortex (Nelissen et al., 2006; Vangeneugden et al., 2009), and parallel searches using brain imaging are closing in on homologs of these regions in human visual cortex (Peelen et al., 2006; Jastorff and Orban, 2009). Studies by Orban and coworkers (Nelissen et al., 2006; Jastorff and Orban, 2009) are particularly relevant here. Stimuli depicting action are known to elicit responses from neurons in both upper and lower banks of macaque superior temporal sulcus (Oram and Perrett, 1994, 1996; Barraclough et al., 2006; Nelissen et al., 2006; Vangeneugden et al., 2009). The lower bank regions appear to be more sensitive to intact shape than to scrambled versions of the same stimuli (Nelissen et al., 2006) and respond to both hand actions and static snapshots from those actions (Vangeneugden et al., 2009). In contrast, neurons found in the upper bank respond less to static snapshots than to the actual motion sequences themselves (Vangeneugden et al., 2009). Comparing these results with biomotion-selective regions found in human visual cortex, Jastorff and Orban (2009) have proposed that the upper and lower banks of macaque STS (middle/rostral parts) correspond functionally to the human pSTS and ventral temporal areas, respectively.

With these results as a backdrop, can our psychophysical findings provide clues about brain areas containing neural representations of the three-dimensional orientation of the human figure? Our results implicate a mechanism that is tightly tuned to the figure's orientation (i.e., underlying form), while remaining invariant to the detailed surface features or size of that biological figure. In addition, without motion information, we found that the aftereffect was essentially abolished. Thus, biomotion-selective areas that respond to both the shape/form of the figure and its related motion are likely to be involved in representing the three-dimensional orientation of the human figure. (By three-dimensional orientation, we mean that the figure's orientation is unambiguously defined in a three-dimensional space, and not just stereoscopically.) So where do form and motion pathways interact in biomotion-selective processing? Interactions between shape and motion cues have recently been found in two biomotion-selective occipito-temporal regions, the extrastriate body area (EBA) and fusiform body area (FBA), somewhat earlier in the visual cortex than where these information sources were previously thought to combine (Jastorff and Orban, 2009). It has been argued that these areas link the form of the actor (i.e., body shape) to the action being performed (i.e., kinematics), with EBA emphasizing the processing of kinematics and FBA more involved in the processing of configural information (Jastorff and Orban, 2009). It will be interesting to learn whether neurons in either the EBA or FBA exhibit adaptation effects that parallel our psychophysical findings.

In conclusion, our present results, together with other recent advances in understanding the information extracted from biomotion (Hiris, 2007; Garcia and Grossman, 2008; Chang and Troje, 2009; Neri, 2009), highlight the need for a more comprehensive model of biomotion perception. Toward this end, several questions still need to be answered. For example, do fast-acting mechanisms that detect the acceleration in an animate figure's feet (Chang and Troje, 2009) subsequently trigger more detailed feedforward pathway processes (Giese and Poggio, 2003), or do these mechanisms provide a bypass signal directly to a global form analysis stage (Lange and Lappe, 2006)? It is now widely accepted that task and attention-related factors influence biological motion perception (Vaina et al., 2001; Thornton et al., 2002; Battelli et al., 2003; Jastorff and Orban, 2009; Thirkettle et al., 2009), but can these influences be modeled in a principled manner? And what role do such factors play in the buildup of the aftereffect studied here? The approach taken here to studying biomotion perception, orthogonal to more standard methods, offers a novel means for answering these kinds of questions.