Spatial selectivity in adaptation to gaze direction

Abstract

A person's focus of attention is conveyed by the direction of their eyes and face, providing a simple visual cue fundamental to social interaction. A growing body of research examines the visual mechanisms that encode the direction of another person's gaze as we observe them. Here we investigate the spatial receptive field properties of these mechanisms, by testing the spatial selectivity of sensory adaptation to gaze direction. Human observers were adapted to faces with averted gaze presented in one visual hemifield, then tested in their perception of gaze direction for faces presented in the same or opposite hemifield. Adaptation caused strong, repulsive perceptual aftereffects, but only for faces presented in the same hemifield as the adapter. This occurred even though adapting and test stimuli were in the same external location across saccades. Hence, there was clear evidence for retinotopic adaptation and a relative lack of either spatiotopic or spatially invariant adaptation. These results indicate that adaptable representations of gaze direction in the human visual system have retinotopic spatial receptive fields. This strategy of coding others' direction of gaze with positional specificity relative to one's own eye position may facilitate key functions of gaze perception, such as socially cued shifts in visual attention.

1. Introduction

The ability to perceive another creature's focus of attention facilitates adaptive behaviour across animal species. Gaze direction can signal the attention of a potential predator or social competitor, help to coordinate visual attention across social groups, and link fear responses of conspecifics to the features of the environment that prompt those responses [1,2]. In humans, our ability to monitor another person's direction of attention helps to initiate and regulate conversation and provides critical input to social cognition (e.g. perspective-taking and theory of mind; [3,4]). To determine the direction of another person's gaze, we rely on visual cues provided by their eye region, namely the pattern of contrast produced by the position of the pupil and iris relative to the (lighter) sclera, together with visual cues to head and body orientation [5,6]. Pioneering electrophysiological experiments in macaque monkeys identified face-selective cells in the temporal cortex that are tuned to the gaze direction of a viewed face [7,8] and neuroimaging studies in humans similarly implicate temporal pathways in the extraction and perceptual representation of others' gaze direction (e.g. anterior superior temporal sulcus, aSTS; [9]). Computational modelling of psychophysical data supports the existence of at least three sensory populations in the human visual system with distinct tuning to gaze direction (direct versus leftwards versus rightwards gaze), the responses of which are flexibly combined to encode the specific direction of gaze that we perceive in a given moment when we look at another person [10–12].

A fundamental property of our visual world is its spatial organization. The spatial layout of sensory inputs in the retina is replicated at multiple stages of visual processing, such that retinotopic organization of neural responses can be used to define functional regions throughout the visual cortex [13]. There is also behavioural and neural evidence for spatiotopic representations of the external environment, whereby information about a visual object is maintained in external coordinates independent of the viewer's eye position [14]. Some form of spatial representation seems necessary for key functions of gaze perception, to enable the visual system to map between the position of a person's face, their direction of gaze, and the location of other objects and events in the environment. However, to our knowledge, the spatial receptive field properties of neural mechanisms that encode perceived gaze direction are unknown. More generally, individual cells in the macaque STS that respond to faces can have large spatial receptive fields that extend across visual hemifields, including cells tuned to a specific view of the face (e.g. [15–17]). Behavioural studies in humans provide varying evidence for retinotopic, spatiotopic and spatially invariant components of face processing, which may depend partly on the facial characteristic being processed (e.g. identity versus gender; [18]). In the current study, we investigate these components of spatial representation in gaze processing.

A key method for investigating spatial representation in the visual system is sensory adaptation [19]. Repeated or prolonged exposure to a stimulus tends to cause temporary habituation of sensory mechanisms that respond to the stimulus, influencing perception of stimuli presented subsequently within the spatial receptive field of those mechanisms while leaving unaffected the perception of stimuli presented elsewhere. The spatial generalization of perceptual aftereffects is therefore revealing about the receptive fields of neural mechanisms that respond to the adapting stimulus. Retinotopic, spatiotopic, and spatially invariant components of adaptation can be dissociated following an eye movement, by comparing the perception of stimuli presented in the same retinal location of the adapting stimulus, the same external location of the adapting stimulus, or a novel location [18]. Perceptual adaptation is an important complement to single-cell electrophysiology not only because it can be applied in humans, but also because it taps into the sensory coding that underlies perception. For example, a cell in the temporal cortex that responds to eye contact may in principle reflect a post-perceptual response to the stimulus (e.g. an affective response to being looked at), and for that reason might lack sensitivity to the location of the stimulus. In contrast, the effects of adaptation on perceived gaze direction can be more directly revealing about the visual coding of gaze direction.

Adaptation to faces with a specific direction of gaze causes a repulsive perceptual aftereffect; for example, adaptation to faces with leftwards gaze can result in subsequently viewed faces appearing to gaze more rightwards than they really are [10,20]. There is evidence that adaptation to gaze direction reflects changes occurring at face-selective levels of visual processing, rather than simply being inherited from adaptation of low-level visual responses (e.g. [21]; reviewed in §4.c). The direction-dependent effects of adaptation can be accounted for by a population-coding model of perceived gaze direction, in which adaptation produces a selective reduction in gain on gaze-selective sensory populations proportional to their response to the adapting stimulus [10]. The present study was designed to test the spatial selectivity of adaptation to gaze direction. Human observers were adapted to faces that displayed averted gaze, presented in one visual hemifield. The effects of adaptation were compared between faces presented in the same visual hemifield as the adapter and faces presented in the opposite visual hemifield. Across this comparison, participants performed eye movements that allowed us to dissociate any retinally specific component of adaptation from spatiotopic or spatially invariant components of adaptation, providing insight into the spatial receptive field properties of mechanisms that encode perceived gaze direction in the human visual system.

2. Methods

(a) . Participants

Participants were six adults, including three women and three men. This included the two authors and four participants naive to the purposes of the study. We tested experienced psychophysical observers due to a demand for careful visual fixation during the experiment (described in §2.d). In previous work, we found that perceptual aftereffects following adaptation to gaze direction occur robustly in samples of this size, particularly when collecting a large amount of data per subject and quantifying aftereffects using a model of gaze perception fit at the individual level (e.g. [11,22]). All participants provided informed consent, and the study was approved by the UNSW ethics committee (HREAP C: Behavioural Sciences, reference 3129).

(b) . Design

Participants completed a psychophysical task designed to measure the effects of sensory adaptation on perceived gaze direction. Adapter and test faces were displayed in the centre of a computer monitor, while the participants' point of fixation was varied to determine whether the faces fell within the left or right side of their visual field (figure 1a). The experiment comprised a repeated-measures 2 × 2 × 2 design. The factors were visual field adapted (left versus right), visual field tested (left versus right) and adapter gaze direction (left versus right). Each participant completed all eight conditions, across four sessions conducted on separate days. The side of the visual field that adapters were presented in, as well as the gaze direction of the adapter, was varied across sessions. Within each session, the perception of gaze direction was tested for faces presented in both the left and right side of the visual field, both before and after adaptation. Thus, we could compare how adaptation to faces presented in one hemifield affected perception of faces in the same hemifield and the opposite hemifield. The order of sessions was varied across participants.

Figure 1.

Dissociating retinotopic from spatiotopic representations of gaze direction. (a) The test and adapting stimuli were always presented in approximately the same spatiotopic location—the centre of the screen. Participants fixated on a cross that varied in location between the left and right side of the screen, determining the retinotopic location of the stimuli. To test retinotopic selectivity of adaptation, participants were adapted to faces presented in one side of their visual field, and tested on faces presented in either the same or opposite side of their visual field. For a full illustration of the trial cycle, see electronic supplementary material, figure S1. (b) The gaze direction of the test faces varied along the horizontal dimension. A subset of horizontal gaze angles is shown here for a single identity. A stimulus gaze direction of 0° indicates that the face was looking straight-ahead. Positive angles indicate rightwards gaze direction, and negative angles indicate leftwards gaze direction. (c) Participants judged whether the test faces were looking direct, left or right, while the gaze direction of the faces varied across trials. A psychophysical model of gaze perception was fit to the data that quantifies the position and width of category boundaries (blue and red dashed lines) and the midpoint of perceived direct gaze (black dashed line). The effect of adaptation on perceived gaze direction was quantified in terms of the change in the midpoint of perceived direct gaze following adaptation compared to baseline. This figure shows the model fit to baseline response data from a single subject and condition. (Online version in colour.)

The logic of the design was as follows: if the visual system represents other people's direction of gaze in retinotopic coordinates, perceptual aftereffects should be observed more strongly for faces presented on the same side of fixation as the adapter faces. By contrast, if gaze direction is represented in spatiotopic coordinates, perceptual aftereffects should be observed regardless of whether the participant is fixating on the left or right side of the screen, because the adapter and test faces were always presented in the same real-world location. Similarly, if gaze direction is represented without spatial selectivity (either spatiotopic or retinotopic), varying the retinotopic location of the test and adapting faces should not affect the strength of perceptual aftereffects. Hence, the design isolates any retinally specific component of adaptation. To foreshadow our results, this retinally specific component accounted for the effects of adaptation to gaze direction, such that a further experiment to disambiguate spatiotopic and spatially invariant components was not necessary.

Adapter gaze direction was included as a factor in the design to confirm that any aftereffects observed were related systematically to the gaze direction of the adapting stimulus. Perceptual aftereffects that occur following adaptation to gaze direction are stimulus-dependent: adaptation to faces with leftwards gaze direction tends to produce a shift in perceived gaze direction towards the right, while adaptation to faces with rightwards gaze direction tends to produce a shift in perceived gaze direction towards the left. This direction-dependence of perceptual aftereffects can be well explained by a population-coding model of perceived gaze direction [10] and provides a useful way of testing whether perceptual aftereffects reflect a systematic change in the sensory coding of gaze direction, rather than a general worsening of task performance, for example.

(c) . Stimuli

The stimuli were computer-generated faces with varying directions of gaze (figure 1b). Textured 3D models of faces were created in FaceGen Modeler (v. 3.50) and further manipulated in Blender (v. 2.70). In Blender, the eyes were modelled separately from the rest of the face, such that the gaze direction of the stimulus could be controlled precisely by setting the rotation of each eye to fixate a specific point within the simulated three-dimensional environment. Images were rendered with a frontal view of the face, with gaze direction varying along the horizontal axis. Face images used as adapter stimuli had gaze direction 25° left or right. Face images used as test stimuli had gaze direction that varied in 5° intervals between 15° left and 15° right. The fixation distance of the face stimuli was approximately 50 cm, signalled by the convergence of the two eyes, corresponding approximately to the viewing distance of the participant during the adaptation task. Images were generated for six facial identities, with three used as test images and three as adapter images. To control for any incidental left–right asymmetries in the appearance of the face, we presented a duplicate of each test stimulus that was flipped around the vertical axis, with the horizontal gaze direction of the stimulus relabelled accordingly.

Participants viewed the face images on a Cambridge Research Systems Display ++ LCD monitor (1920 × 1080 pixel resolution; 28 pixels per cm; 120 Hz refresh rate). The stimulus presentation was controlled using MATLAB Psychtoolbox. The test images were approximately life-sized on screen (approx. 14° width and 25° height in visual angle). The adapter images were presented at 75% of the size of the test images. This size difference was introduced to minimize systematic effects of (retinotopic) adaptation to low-level visual properties of the image, following the approach of previous studies of adaptation to gaze direction (e.g. [20]).

(d) . Adaptation task

The experimental task closely resembled those used previously in the literature to measure the effects of sensory adaptation on perceived gaze direction (reviewed in 10), but modified to test the spatial selectivity of perceptual aftereffects. This was achieved by varying the fixation point of the participant relative to the face images presented on-screen. The task consisted of a baseline test of perceived gaze direction, an adaptation period, and a post-adaptation test of perceived gaze direction, repeated in each of the four testing sessions.

In the baseline period, participants were presented with a series of faces that varied in their horizontal gaze direction between 15° left and 15° right (described in §2.c). Participants reported whether each face was looking leftwards, rightwards or direct, using the keyboard. Participants were instructed to carefully maintain their fixation on a cross that was presented on-screen throughout the task. The location of the fixation cross varied across trials to control whether the test image was presented in the participant's left or right visual hemifield. Specifically, the fixation cross appeared in the centre of the screen at the beginning of each trial (500 ms), then shifted to either the left or right side of the screen (500 ms), after which the test image appeared in the centre of the screen (500 ms). When the fixation cross was on the left side of the screen, the test stimulus was presented in the participant's right visual hemifield, whereas when the fixation cross was on the right side of the screen, the test stimulus was presented in the participant's left visual hemifield. The eccentricity of the fixation cross during presentation of the test images was approximately 9° of visual angle. The location of the test image for each trial was jittered randomly within a range of 50 pixels (approx. 1.8° of visual angle) in each of the horizontal and vertical dimensions. During the 500 ms presentation of the test stimulus, the contrast of the stimulus was ramped up and down using a raised-cosine function. There was a fixed response period of 1500 ms following each test image, during which only the fixation cross was presented and the participant could make their response. Trials in which the participant did not make a response within this period were repeated at the end of the block. The fixed duration of the response period ensured that participants were exposed to the stimuli at a consistent rate, regardless of any individual differences in response times (which was relevant to maintaining consistent exposure to the adapting stimuli presented in the post-adaptation period, described below). Participants completed 12 trials for each of the seven stimulus gaze directions and each hemifield, corresponding to 168 trials in total during the baseline period. These trials were completed intermixed in two blocks, with trial order randomized within each block. See electronic supplementary material figure S2 for an illustration of individual baseline response data collected within a session.

After the baseline test of perceived gaze direction, participants underwent a short adaptation period, in which they viewed a series of faces with gaze direction 25° left or right, presented in the centre of the screen. Each face was presented for 4 s, separated by a 200 ms blank interval. Thirty images were presented in a random order, varying between the three identities used as adapter stimuli, for a total duration of approximately 2 min. The fixation cross remained in a fixed position throughout this period, either on the left or right side of the screen, such that all the adapter images were presented in either the right or left visual hemifield. The gaze direction of the adapter faces, and the hemifield in which they were presented, varied across conditions conducted on separate days.

The post-adaptation test of perceived gaze direction was similar to the baseline test of perceived gaze direction, except that each trial began with the presentation of one adapter image for 4 s. Specifically, each trial began with the fixation cross located centrally (500 ms), then located on the left or the right side of the screen (500 ms), then the adapter image appeared in the centre of the screen while the participant maintained their fixation to the left or the right (figure 1a). The side of fixation during presentation of the adapter stimulus matched that of the adaptation period. After presentation of the adapter image, the fixation cross remained on the side of the screen for a further 500 ms, then the remainder of the trial proceeded as per the baseline test of perceived gaze direction, with the participant's fixation being directed first to the centre of the screen and then directed to either the left or the right side of the screen before presentation of the test image. Thus, there were two saccades required before the presentation of each test image. The full trial cycle for post-adaptation trials is shown in electronic supplementary material, figure S1. Importantly, the number of shifts in fixation that occurred in each trial was matched between trials in which the test stimulus appeared in the same hemifield as the adapter and trials in which the test stimulus appeared in the opposite hemifield to the adapter. This controlled for any modulation of perception or adaptation that might occur following an eye movement, as has been reported previously for orientation-contingent colour adaptation, for example [23]. The identical test stimuli were presented in the baseline and post-adaptation periods of the task. Participants completed two adaptation blocks in each session, each consisting of the same adaptation period and post-adaptation test of perceived gaze direction. Trial order was randomized within each block. See electronic supplementary material, figure S2 for an illustration of individual post-adaptation response data collected within a session.

As described, the approach of the task was to present all face stimuli centrally, varying their visual field location by changing the subject's point of fixation between the left and right side of the monitor. An advantage of this approach is that a face stimulus that was looking straight-ahead (labelled as 0° gaze direction) was always looking directly at the subject. This avoids a situation that can occur with peripheral presentation where a face stimulus that is looking ‘straight-ahead’ is not actually looking directly at the viewer if presented off-centre relative to the viewer's head position.

(e) . Analysis

The proportion of trials in which the participant perceived the face as looking either direct, left or right was calculated for each test gaze direction. A psychophysical model of perceived gaze direction was fit to these data to quantify the shift in perceived gaze direction that occurred following adaptation. This model is described in detail by Mareschal and colleagues [24, p. 5]. The model describes the categorization of a given stimulus gaze direction as looking direct, left or right, using three free parameters (figure 1c). These parameters are the midpoint between category boundaries (i.e. between the boundaries of left/direct and direct/right gaze), the distance between category boundaries (equivalent to the range of gaze directions categorized as looking direct), and the uncertainty associated with sensory representations of gaze direction. In this model, the midpoint between category boundaries represents the peak in ‘direct’ responses, or the stimulus gaze direction most likely to be perceived as looking directly at the observer. The shift in perceived gaze direction induced by adaptation was quantified in terms of how the midpoint between category boundaries differed between the baseline and adapted conditions. To illustrate: if adaptation to faces with leftwards gaze direction shifts the perceived angle of gaze for test faces toward the right (i.e. a repulsive perceptual aftereffect), test faces with leftwards gaze direction would be more likely to be judged as looking direct following adaptation, and hence the range of gaze directions judged as looking direct would be shifted towards the left across the stimulus space. We report the aftereffects as baseline – adapted, such that positive values represent a shift in category boundaries towards the left following adaptation, consistent with a shift in the perceived angle of gaze toward the right.

The model was fit separately to the data collected before and after adaptation, separately for test stimuli presented in the left and right sides of the visual field, and separately for each condition that varied in the visual field location and gaze direction of the adapting stimulus. The model was fit to the data by minimizing the sum of squared errors using the MATLAB function fminsearch, allowing the three parameters described above to vary. The model fit the data very well, accounting for 97.9% of the variance on average across conditions and subjects (range = 88.5–99.9%). Perceptual aftereffects were calculated separately for test stimuli presented in the left visual field and right visual field, and for each condition that varied in the visual field location of the adapting stimulus and the gaze direction of the adapting stimulus. Note that aftereffects were always calculated by comparing responses to test stimuli presented in the same visual hemifield during the baseline and post-adaptation periods of the task. This controlled for any differences in the perceived gaze direction of test stimuli that depended on their visual field location (independent of adaptation). See electronic supplementary material for a comparison of baseline data between visual hemifields.

3. Results

The mean changes in perceived gaze direction that occurred following adaptation are shown in figure 2. The effect of adaptation depended on the gaze direction of the stimulus to which the participant was adapted, such that test stimuli were perceived as looking more rightwards following adaptation to leftwards-looking faces, but perceived as looking more leftwards following adaptation to rightwards-looking faces. Importantly, the magnitude of this effect depended on whether the test stimuli were presented in the same or opposite visual hemifield to the adapter. In particular, perceptual aftereffects occurred robustly for test stimuli presented in the same visual hemifield as the adapter, but appeared absent for test stimuli presented in the opposite visual hemifield to the adapter. This pattern occurred in all subjects, illustrated in figure 3.

Figure 2.

Perceptual aftereffects averaged across the sample. The data are plotted as a function of the visual field location of the adapting and test stimuli. Aftereffects are shown here as the difference in the perceived gaze direction of test stimuli before and after adaptation. Positive aftereffects indicate a shift in the perceived angle of gaze toward the right following adaptation, and negative aftereffects indicate a shift in the perceived angle of gaze toward the left. Markers show the mean ± 1 s.e., and colour indicates whether the adapting stimulus had leftwards or rightwards gaze direction. The results show that perceived gaze direction was biased away from the gaze direction of the adapting stimulus, but only when the adapting and test stimuli were presented in the same side of the visual field. (Online version in colour.)

Figure 3.

Perceptual aftereffects for individual subjects. Aftereffects are shown here as a shift in the perceived gaze direction of test stimuli away from the gaze direction of the adapting stimulus (i.e. a repulsive perceptual aftereffect). Aftereffects are averaged across conditions in which the adapting and test stimuli were each presented in the same visual hemifield versus the opposite visual hemifield. Markers show the mean ± 1 s.e. There was a consistent pattern across subjects of stronger repulsive aftereffects for test stimuli presented in the same visual hemifield as the adapting stimulus compared to test stimuli presented in the opposite visual hemifield. Across this comparison, the adapting and test stimuli were matched in their spatiotopic position (but not their retinotopic position).

To test these observations statistically, we compared perceptual aftereffects across conditions with a 2 × 2 × 2 repeated-measures ANOVA. The factors were visual field adapted (left versus right), visual field tested (left versus right) and adapter gaze direction (left versus right). There was a significant main effect of adapter gaze direction, F_1,5 = 31.8, p < 0.005, ${\eta }_p^2$ = 0.86. This reflected the direction-dependence of perceptual aftereffects: there was a shift in perceived gaze direction toward the right following adaptation to faces with leftwards gaze direction (mean shift = 1.8°, s.e. = 0.3°) and a shift in perceived gaze direction towards the left following adaptation to faces with rightwards gaze direction (mean shift = −1.6°, s.e. = 0.3°). The three-way interaction between the factors was also statistically significant, F_1,5 = 51.8, p < 0.001, ${\eta }_p^2$ = 0.91. All other main effects and interaction terms were non-significant (p > 0.05).

To follow up on the interaction effect, we calculated the shift in the perceived gaze direction of test stimuli away from the gaze direction of the adapting stimulus (i.e. the repulsive perceptual aftereffect), and averaged this for each participant across conditions in which the adapting and test stimuli were presented in the same visual hemifield versus the opposite visual hemifield (figure 3). A paired-samples t-test confirmed that repulsive perceptual aftereffects were stronger for test stimuli presented in the same visual hemifield as the adapter (M = 3.2°, s.e. = 0.5°) compared to when presented in the opposite visual hemifield to the adapter (M = 0.2°, s.e. = 0.3°), t₅ = 7.2, p < 0.001, Cohen's d = 2.9. Moreover, the magnitude of repulsive perceptual aftereffects was significantly greater than zero when test stimuli were presented in the same visual hemifield as the adapter [t₅ = 7.1, p < 0.001, Cohen's d = 2.9], but not when presented in the opposite visual hemifield to the adapter [t₅ = 0.8, p = 0.46, Cohen's d = 0.3].

The average magnitude of repulsive aftereffects for test stimuli presented in the opposite visual field location as the adapter was also computed as a percentage of the magnitude for test stimuli presented in the same visual field location as the adapter. This was very small on average (mean = 3.4%, range = −14 to +29% across subjects) and not significantly different from zero, t₅ = 0.48, p = 0.65, Cohen's d = 0.2. Hence, there was little transfer of adaptation across visual hemifields apparent in the data.

4. Discussion

The current study tested the spatial selectivity of adaptation to gaze direction. Human observers were adapted to faces with averted gaze, presented in one visual hemifield, then tested in their perception of gaze direction for faces presented in either the same or opposite visual hemifield. We saw clear evidence for positional specificity, whereby adaptation caused strong, repulsive perceptual aftereffects that depended on the gaze direction of the adapting stimulus, but only for faces presented in the same visual hemifield as the adapter. In other words, the state of altered visual function induced by repeated stimulus exposure was present in one visual hemifield while being simultaneously absent in the other visual hemifield. This pattern of results was apparent in all subjects tested. These results indicate that mechanisms in the human visual system that represent gaze direction have retinotopic spatial receptive fields. By contrast, there was little transfer of adaptation across visual hemifields, even though the adapting and test stimuli were presented in the same external coordinates, indicating a relative lack of either spatiotopic or spatially invariant adaptation to gaze direction.

(a) . Visual mechanisms encoding gaze direction

A growing body of research examines the visual mechanisms that detect and encode gaze direction. Perceived gaze direction depends on visual integration of cues to head rotation and eye direction [5,6]. These cues feed into a system that appears to code gaze direction comparably to how lower-level stimulus properties, such as orientation and motion direction, are coded across neuronal populations in the early visual cortex. In particular, there is evidence for a population-coding mechanism specific to gaze direction, whereby the perceived direction of gaze associated with a face is coded in terms of the relative activity that the stimulus elicits across a population of neurons that vary in their tuning across the full range of possible gaze directions [10,12,20,25]. The perceptual effects of adaptation, in which the perceived gaze direction following adaptation is biased systematically depending on the gaze direction of the adapting stimulus, can be modelled computationally in terms of a selective reduction in the gain on gaze-selective sensory populations proportional to their response to the adapting stimulus. More specifically, the tuning of perceptual aftereffects provides evidence for at least three broadly tuned sensory channels that jointly encode the horizontal gaze direction of a face relative to the viewer, and are subject to gain control including the divisive normalization of sensory responses [11,22,26]. The results of the current study add to this picture by indicating the existence of multiple, independent population codes in the brain, carrying information about the gaze direction of faces present in different regions of the visual field.

Where in the brain are sensory populations coding gaze direction located? There is evidence from single-cell recording studies in non-human primates for face-selective cells in the STS that show tuning to the direction of attention signalled by eye direction and head orientation [7,27], and corresponding evidence from functional neuroimaging in humans that relatively fine-grained information about horizontal gaze direction is carried in responses in anterior STS when viewing faces [9,28]. A handful of studies have measured neural responses in humans following adaptation to gaze direction; these studies have observed feature-selective habituation of responses in anterior STS and inferior parietal lobule measured with fMRI [25] and modulation of late occipitotemporal components of event-related potentials measured with EEG (∼250–350 ms; [29,30]). Together, these results suggest that neural population codes carrying information about gaze direction may be localized in part to anterior regions of the STS. In principle, gaze-selective neurons with inputs originating in each visual hemifield might be located in opposite hemispheres of the brain, consistent with the initial lateralization of neural responses in the visual cortex to stimuli presented in each visual hemifield, or might be clustered to a region of the same hemisphere. Several of the human neuroimaging studies described above report a degree of lateralization of gaze-dependent responses to right temporal regions when viewing faces presented centrally [25,30,31] while other results suggest bilateral coding of gaze direction [28]. Interestingly, two patients with surgical section of the corpus callosum were found to exhibit reflexive attentional cueing in response to gaze cues presented on one side of visual fixation but not the other, consistent with hemispheric lateralization of gaze processing [32]. Future work might aim to localize neural population codes with different spatial receptive fields by combining fMRI-adaptation [25] with a design similar to the current study that varies correspondence between the position of the adapting and test stimulus.

(b) . Comparison to other stimulus properties

The positional specificity of adaptation has been examined previously for a range of stimulus types, including lower-level visual properties (e.g. orientation, motion; [33,34]) and higher-level properties (e.g. faces, biological motion; [35,36]). For face stimuli, the peak perceptual effects of adaptation are commonly found when the adapting and test stimuli are presented in the same retinal location, though face aftereffects can also show a degree of spatial invariance (e.g. transferring significantly, but sometimes more weakly, across visual hemifields; [37–40]). There is contrasting evidence regarding whether a spatiotopic component of adaptation to faces occurs, which has been examined by varying the observer's point of fixation between presentation of the adapting and test faces (reviewed in [18]). For example, Melcher [35] reports evidence for both spatially invariant and spatiotopic components of adaptation on identity-discrimination performance. By contrast, Afraz and Cavanagh [41] report evidence for retinal-specificity of adaptation to face gender, and a lack of any spatiotopic component of the effect. Face processing encompasses multiple perceptual characteristics with dissociable neural substrates, and the spatial receptive fields of mechanisms that extract different social cues from a face may vary in their nature. The results of the current study contrast to these previous findings in that we observe strongly retinotopic effects of adaptation to gaze direction together with an apparent absence of either transfer across visual hemifields or a spatiotopic effect.

Why might information about gaze direction be coded retinotopically? To derive the focus of another person's attention, it seems necessary to represent the angle of their gaze together with the position of their face within a spatial reference frame that can be mapped to other objects or events in the environment. We can speculate that the particular strategy of coding gaze direction with retinotopic specificity may facilitate key functions in gaze perception, including cued shifts in spatial attention (i.e. gaze following or gaze-cueing of attention; [42]) and detecting eye contact [43]. Positional specificity of the face is clearly important to these functions of gaze perception—for example, a face looking slightly to the right positioned in your left visual field may be looking toward you, while the same face positioned in your right visual field is looking away from you. Moreover, retinotopic representations may be more efficient than spatiotopic representations for certain functions. For example, it is the other person's direction of gaze relative to the position of your own eyes that is important to determine the saccade necessary to shift your central vision in the direction that the other person is looking. This information is at least partly built into a retinotopic representation of gaze direction.

(c) . Level of visual processing

A sceptical view of high-level perceptual aftereffects is that they might reflect a cognitive bias, such as a shift in decision criterion, rather than a change in visual processing per se. (For related discussions, see [44–46].) However, as retinotopy is a hallmark of the functional anatomy of the visual system, the retinal specificity of gaze aftereffects observed here strongly implicates visual processing in the phenomenon.

At what stage of visual processing does adaptation to gaze direction occur? As discussed in §4.a, there is evidence that adaptation to gaze direction is associated with changes in neural responses in higher-level visual pathways in the temporal cortex [25,30], consistent with neuroimaging and single-cell data that links such regions to gaze-selective processing more generally [9]. In principle, however, these changes in higher-level visual responses following adaptation could be inherited from more direct effects of adaptation occurring earlier in the visual system that are not specific to the processing of faces or gaze direction (e.g. low-level adaptation to the luminance pattern of the eye). At first glance, the retinotopic specificity of gaze aftereffects that we report in the current study may appear consistent with the latter hypothesis, given that early visual responses tend to be organized with more precise retinotopic specificity, while later visual processing incorporates neurons with broader spatial receptive fields. However, there are several reasons to believe that adaptation to gaze direction cannot be explained by changes in low-level visual processing alone. First, the perceptual aftereffects observed in the current study occurred despite small differences in the scale and position of the adapting and test stimuli. In particular, the adapter images were presented at 75% of size of the test images, and location of the test image for each trial was jittered randomly within a range of approximately 1.8° of visual angle horizontally and vertically. This was to minimize systematic effects of (retinotopic) adaptation to low-level visual properties of the image, following the design of previous studies (e.g. [20]). Hence, the retinotopic specificity of perceptual aftereffects that we observed was not dependent on direct retinal overlap of face features in the adapting and test images, but relates to positional specificity relative to the visual field more broadly. Second, distinct effects of adaptation have been observed when participants are adapted to faces consisting of identical eye features, but which evoke a different sense of gaze direction due to the conjunction of head and eye features, indicating that adaptation to gaze direction occurs at a stage of visual processing that follows holistic processing of the face [21]. Similarly, other studies have found that gaze aftereffects persist despite considerable changes between the adapting and test faces in their low-level visual features, consistent with adaptation occurring at a level of visual processing in which information about gaze direction is abstracted from the low-level visual features of the image [22,47].

(d) . Conclusions and future directions

To conclude, we found that adaptation to gaze direction exhibits strong spatial selectivity: changes in perception that occur following adaptation depend on the correspondence between the visual field location of the adapting and test faces. By contrast, there was no evidence of either spatiotopic or spatially invariant adaptation. These results indicate that adaptable representations of gaze direction in the human visual system have retinotopic spatial receptive fields.

A key organizing principle of the human visual system is the local ordering of neurons based on the retinal position of their receptive fields, forming the basis of functionally defined retinotopic maps at multiple stages of visual processing [13]. The present results raise questions about the spatial organization of neurons in cortical areas representing gaze direction—for example, can retinotopic maps for gaze direction be defined in the higher-level visual cortex? In principle, there may be a systematic spatial organization to gaze-selective neurons based on their spatial receptive fields, feature specificity (e.g. tuning to angles of gaze varying from left to right), or both—similar to the columnar organization of functionally related neurons described for simple visual features in the early visual cortex [48] and for more complex features in the temporal cortex [49,50]. Future work might explore the spatial organization of gaze-selective processing with the use of fMRI-adaptation and/or retinotopic mapping in humans, as well as single-cell recordings in non-human primates.

The present results also raise questions regarding the functional role retinotopic specificity plays in the ‘downstream’ effects of gaze perception. We have speculated that retinotopic coding of gaze direction may facilitate some key functions of gaze perception, such as triggering socially cued saccades (i.e. gaze following). However, there is also the question of how information about gaze direction is mapped to a broader representation of the physical environment. To determine the precise focus of another person's gaze, for example, would seem to require integrating information about the location of their face, their angle of gaze, and the position of other objects and events in three-dimensional space. The first two of these might be achieved in part by retinally specific representations of gaze direction, as revealed in the current study. While it is not yet clear how information about gaze direction is related to the position of other objects and events in the environment, this might rely on interactions between temporal and parietal mechanisms coding information about social attention and spatial position, respectively [9,50,51]. Future work might also profitably investigate the extension of the current paradigm to naturalistic environments, such as whether the form of spatial selectivity to adaptation observed in the present study persists in visual environments that contain richer cues to the spatial relations between objects and engage higher-level functions of gaze perception, such as joint attention.

Ethics

The study was approved by the UNSW ethics committee (HREAP C: Behavioural Sciences, reference 3129). All participants provided informed consent.

Data accessibility

The data and analysis code are provided in the electronic supplementary material [52].

Authors' contributions

C.J.P.: formal analysis, investigation, methodology, software, visualization, writing—original draft; C.W.G.C.: conceptualization, formal analysis, funding acquisition, investigation, methodology, resources, software, writing—review and editing.

All authors gave final approval for publication and agreed to be held accountable for the work performed therein.

Conflict of interest declaration

We declare we have no competing interests.

Funding

This research was funded by Australian Research Council Discovery Project no. DP190100491. C.J.P. is supported by an Australian Research Council Discovery Early Career Researcher Award no. (DE190100459).