The effects of short-term and long-term learning on the responses of lateral intraparietal neurons to visually presented objects

Abstract

The lateral intraparietal area (LIP) of the dorsal visual stream is thought to play an important role in visually directed orienting, or the guidance of where to look and pay attention. LIP can also respond selectively to differently shaped objects. We sought to understand how and to what extent short-term and long-term experience with visual orienting can determine the nature of responses of LIP neurons to objects of different shapes. We taught monkeys to arbitrarily associate centrally presented objects of various shapes with orienting either toward or away from a preferred peripheral spatial location of a neuron. For some objects the training lasted for less than a single day, while for other objects the training lasted for several months. We found that neural responses to visual objects are affected both by such short-term and long-term experience, but that the length of the learning period determines exactly how this neural plasticity manifests itself. Short-term learning over the course of a single training session affects neural responses to objects, but these effects are only seen relatively late after visual onset; at this time, the neural responses to newly learned objects start to resemble those of familiar over-learned objects that share their meaning or arbitrary association. Long-term learning, on the other hand, affects the earliest and apparently bottom-up responses to visual objects. These responses tend to be greater for objects that have repeatedly been associated with looking toward, rather than away from, LIP neurons’ preferred spatial locations. Responses to objects can nonetheless be distinct even though the objects have both been similarly acted on in the past and will lead to the same orienting behavior in the future. Our results therefore also indicate that a complete experience-driven override of LIP object responses is difficult or impossible.

Introduction

The ability to use visual information to predict where important things will be in the near future has obvious survival value. Visual changes over space and time (such as a red berry in a green bush, or a bee that suddenly enters your visual field) are likely to be important and accordingly attract attention both quickly and seemingly automatically (see e.g. Eriksen and Hoffman 1972; Franconeri et al 2005; Jonides 1980; Kristjánsson et al 2001; LaBerge 1983; Nakayama and Mackeben 1989; Posner and Cohen 1984 ------ Cavanagh & Chase, 1971; Egeth, Virzi & Garbart, 1984; Julesz, 1984; Smith, 1962; Treisman & Gelade, 1980; see e.g. Wolfe, 1998). Visual attention can also be deliberately directed and maintained (see e.g. Alvarez and Scholl 2005; Bashinski and Bacharach 1980; Colegate et al 1973; Engel 1971). These two ways of visual orienting have been reported to follow different time courses, the former of which has a transient effect on performance with a rapid rise and fall, while the latter takes more time to have its effects (e.g. Cheal and Lyon 1991; Nakayama and Mackeben 1989).

Transient attention has mainly been thought to be captured by visual information in the periphery so that attention is oriented to the location of the visual objects or events that also initiate the attentional shift [cite]. Sometimes objects can nonetheless give important clues about where other things or events will be in the near future. Take a street sign with a leftward-pointing arrow and the words “look left” that prompts people to check for approaching cars in a particular direction. This, at least at a first glance, seems to be an indirect and symbolic way of representing space that would require the slow and deliberate visual orienting of sustained attention. However, it is now increasingly recognized that visual objects can both swiftly and automatically guide orienting away from themselves because of the way that they are shaped (Driver et al., 1999; Fischer, Castel, Dodd, & Pratt, 2003; Friesen & Kingstone, 1998; Hommel, Pratt, Colzato, & Godijn, 2001; Kuhn & Kingstone, 2009; Tipples, 2002, 2008; Sigurdardottir, Michalak, & Sheinberg, 2014). For example, even centrally presented novel objects can guide people’s eyes and attention in a particular direction despite the fact that doing so is task-irrelevant or even detrimental to task performance (Sigurdardottir, Michalak, & Sheinberg, 2014). These orienting effects are hard or impossible to fully overcome, and their time course resembles that of transient visual attention (Sigurdardottir, Michalak, & Sheinberg, 2014).

While objects that people have never seen before can guide attention, the orienting effects of some familiar objects such as arrows do appear to be particularly robust (Hommel et al., 2001; Kuhn & Kingstone, 2009; Sigurdardottir, Michalak, & Sheinberg, 2014; Tipples, 2002, 2008). This might, at least in part, be due to the fact that arrows generally point to something important; a distant target has repeatedly been associated with this shape over the course of an entire lifetime. The deliberate training of an arbitrary association between a central visual stimulus and a target found in a particular peripheral location also leads to a seemingly obligatory attentional shift toward that location (Dodd & Wilson, 2009; Van der Stigchel, Mills, & Dodd, 2010). The attentional effects of such arbitrary associations appear to be quite weak and slow after a short training period but might increase in magnitude and speed with a longer training session (Dodd & Wilson, 2009; Van der Stigchel et al., 2010).

Visual attention and cueing effects have been less extensively studied in monkeys than in humans, but the general effects and the time course of spatial precueing appear to be comparable in the different species (Lee & McPeek 2013). Humans and monkeys are also believed to have several homologous visually responsive brain regions, including a parietal region known as the lateral intraparietal area (LIP) in the monkey [cite]. The function of LIP is still a subject of debate, but the region has mainly been implicated in the visual guidance of spatial attention and saccadic eye movements (Andersen & Buneo, 2002; Bisley & Goldberg, 2003, 2010; Colby & Goldberg, 1999; Gottlieb, Kusunoki, & Goldberg, 1998; Snyder, Batista, & Andersen, 1997), collectively known as visual orienting (Posner, 1980). LIP is structurally connected to various visual areas (Felleman & Van Essen, 1991; Lewis & Van Essen, 2000) and to several oculomotor structures (Ferraina, Pare, & Wurtz, 2002; Field, Johnston, Gati, Menon, & Everling, 2008; Lewis & Van Essen, 2000; Prevosto, Graf, & Ugolini, 2010; Stanton, Bruce, & Goldberg, 1995), making it perfectly situated to gather and combine various sources of visual information with the objective of guiding visual orienting.

LIP has been found to respond selectively to differently shaped visual objects (Durand et al., 2007; Janssen, Srivastava, Ombelet, & Orban, 2008; Konen & Kastner, 2008; Lehky & Sereno, 2007; A. B. Sereno & Amador, 2006; A. B. Sereno & Maunsell, 1998; M. E. Sereno, Trinath, Augath, & Logothetis, 2002). This is akin to many regions within the ventral visual stream (Goodale & Milner, 1992; Logothetis & Sheinberg, 1996; Milner & Goodale, 1995; Palmeri & Gauthier, 2004; Ungerleider & Mishkin, 1982) although the responses of LIP to visual objects are far less studied and understood. The fact that LIP is selective for the shape of objects and plays a role in visual orienting makes this parietal region a primary candidate for carrying out the necessary computations for extracting an orienting bias from an object, such as a leftward-pointing arrow or the words “look left”, that might have come about through the association of such objects with an important target in a distant location.

The role of the parietal cortex in arbitrary associations is nonetheless somewhat controversial (for reviews on arbitrary visuomotor associations, see e.g. Brasted & Wise, 2004; Graybiel, 2005; Hadj-Bouziane, Meunier, & Boussaoud, 2003; Murray, Bussey, & Wise, 2000; Passingham, Toni, & Rushworth, 2000; Seger, 2009; Wise & Murray, 2000). Functional neuroimaging studies have reported that neural activity in the parietal cortex can be affected by associative learning that happens over the course of a single session (Deiber et al., 1997; Eliassen, Souza, & Sanes, 2003), and that the parietal cortex can become progressively more involved as the arbitrary associations become increasingly automatic and overtrained (Eliassen et al., 2003; Grol, de Lange, Verstraten, Passingham, & Toni, 2006). Removing parts of the parietal cortex, however, does not seem to affect the learning of new associations or the retention of familiar ones (Pisella et al., 2000; Rushworth, Nixon, & Passingham, 1997). As an example, LIP neurons can become sensitive to colors if they have been arbitrarily associated with certain behaviors (Toth & Assad, 2002), but monkeys nonetheless do not have particular problems with relearning a similar task after parietal lesions, including a lesion of LIP (Rushworth et al., 1997).

We sought to understand to which extent and how LIP responses change through learning of arbitrary associations, or, more specifically, how experience with visual orienting – which is most likely the primary task supported by LIP – can determine the nature of responses of LIP neurons to visual objects. We set out to answer three main questions: 1) How are the neural responses to visually presented objects affected by short-term learning of arbitrary associations between objects and orienting? 2) How are these responses affected by long-term experience with such arbitrary associations? 3) Can experience with arbitrary associations ever completely override the responses of LIP neurons to visual objects, or are these responses resistant to such experience?

Materials and Methods

Surgery, MRI, and Recordings

Two male macaca mulatta monkeys (monkey J: 10.5 kg, monkey R: 9.5 kg) were implanted with titanium head posts for restraining head movements during the training and recording sessions. The monkeys had two separate surgeries under isoflurane anesthesia. During the first surgery we implanted a recording chamber of diameter 16 mm at approximately 5P and 12L over the right hemisphere. The chamber was made of MRI compatible plastic material (PEEK, polyetheretherketone). The craniotomy was performed during a second surgery after structural MRI had verified that the chambers were located above the lateral intraparietal area (LIP).

The structural MRI was also used to properly position a metal guide tube so that an electrode would reach LIP. During each recording session, a tungsten microelectrode (1.5 MΩ, Alpha Omega) was lowered using a hydraulic microdrive (David Kopf Instruments), and the neural signals were filtered and amplified (BAK Electronics). Eye movements were monitored and recorded using EyeLinkII (SR Research) with a 500 Hz sampling rate and streamed to a disk at 200 Hz. Experimental protocols were in accordance with animal care guidelines of the National Institutes of Health (Council, 2011) and Brown University’s Institutional Animal Care and Use Committee.

Tasks and Stimuli

Stimulus generation and presentation

All shapes, familiar and novel, were originally generated by randomly selecting and overlapping four black strokes or pieces out of a set of 64, and then scaling them so that all composite shapes had the same area. Example shapes can be seen in figure 1. All stimuli were shown on a 1024 × 768 resolution screen with a refresh rate of 100 Hz, and the experiments were controlled using in-house software running on Windows XP (Microsoft) and QNX RTOS (QNX Software Systems).

Figure 1. Example shapes.

The shapes are not shown to scale.

Tasks

The monkeys were trained on three tasks, run consecutively in each session (figure 2). The first two tasks were mainly used for stimulus and unit selection. We will briefly report some results from these tasks, but our focus here will be on the main task, an active shape-saccade association task (see below).

Figure 2. The three consecutive behavioral tasks.

The approximate position of gaze is marked with a green dotted circle. Possible locations of upcoming stimuli are indicated by question marks. Neither the green dotted circle nor the question marks were actually present. A, Location selectivity mapping of eight peripheral locations equidistant from the center used to determine preferred (PREF) and anti-preferred (ANTI) locations. B, Passive shape-mapping task probing responses to visually presented shapes in the PREF, CENTER, and ANTI locations. C, Active shape-saccade association task where a centrally presented shape serves as a cue for saccading to the PREF or ANTI location.

Location selectivity mapping task

In the location selectivity mapping task, a yellow fixation square (side length 0.3°) appeared in the center of a light gray background. After the monkey acquired fixation, a dark gray target disk (radius 1°) was flashed for 110 ms at 7° eccentricity. In each trial the disks were randomly chosen to appear in one of eight possible radial directions from the center (22.5°, 67.5°, 112.5°, 157.5°, 202.5°, 247.5°, 292.5°, or 337.5°; 0° corresponds to a location at the right of fixation; numbers increase in counterclock direction). After a brief delay of 60 ms, the fixation square jumped to that same peripheral location, and the monkey was given juice for saccading to it. We recorded from LIP neurons which, based on online spike data, were deemed to respond to one or more locations. The preferred (PREF) location was defined as the location which evoked the highest mean responses over a 250 ms window (from 40 ms to 290 ms after visual onset of the peripheral disk). The anti-preferred (ANTI) location was a location of the same eccentricity but in the opposite radial direction, in other words 180 degrees away from the PREF.

Passive shape-mapping task

Each trial of the passive shape-mapping task consisted of four rapid serial visual presentations (RSVPs) where black shapes (diameter approximately 3°; for examples, see figure 1) were flashed on a light gray background for 150 ms, with an inter-stimulus interval of 130 ms, in three possible locations: PREF (7° eccentricity), CENTER, and ANTI (7° eccentricity), as determined by online spike recordings from the location selectivity mapping task. The monkeys’ only requirement was that they kept their eyes within 4° of the center of the screen, so the trials aborted if the monkeys looked to the PREF or ANTI locations. A yellow fixation square (side length 0.3°) was shown in the center of the screen. 60 ms after the disappearance of the last shape, the fixation square jumped to one of four possible locations randomly picked to be up, down, right, or left of the center (6° eccentricity). These target locations never overlapped with the PREF or ANTI locations. The monkeys were rewarded for making a saccade to the new location of the square.

A shape and a location were chosen pseudorandomly for each presentation, so that all shapes were shown equally often in all three locations. A total of eight different shapes were shown during a recording session. In most recording sessions, each shape appeared 20 times in each location. Four of the shapes were highly familiar to each monkey because he had been previously trained over the course of months to associate them with particular locations during the active shape-saccade association task, as explained in more detail below. In each session we also showed four novel shapes which the monkey had never seen before and therefore had no particular associations.

Active shape-saccade association task

The final and main task was an active shape-saccade association task where the eight shapes previously seen in the passive shape-mapping task now served as central precues, cueing the monkey to saccade either to the PREF or the ANTI location after a brief delay. Two of the novel shapes were randomly chosen to cue the PREF location and the other two cued the ANTI location. The same was true for the four familiar shapes, except that their associations were randomly chosen at the start of the monkey’s training and this initial assignment to a location was maintained for the entire training and recording period. Each monkey trained on four sets of familiar shapes with four shapes in each, so they were highly familiar with 16 shapes which were different for the two monkeys. In each recording session, a set of familiar shapes was chosen to match the PREF and ANTI locations of the neuron being recorded from.

The active shape-saccade association task was run in blocks of 96 trials each. The first block had equal numbers of novel and familiar shapes. Provided that the monkeys were willing to complete further trials, this first mixed block was in most cases followed by three blocks of trials where only novel shapes were shown in order to provide more experience with novel shapes. These novel blocks were followed by mixed blocks, again provided that the monkeys were willing to work. Our analysis will mostly focus on the first block of trials.

A trial started with the appearance of a central fixation square (side length 0.3°). After the monkey acquired fixation, one of the shapes from the previous passive shape-mapping task was randomly chosen to appear in the center of the screen and was visible for the remainder of the trial. The monkey was required to keep fixating within 1° of the screen center for 500 ms (shape period), after which the fixation square disappeared and two identical gray choice disks (radius 1°) appeared, one in the PREF location and the other in the ANTI location (choice period). The monkey was then free to break fixation by saccading to one of the two choice disks. The shape served as a 100% valid central precue so it determined what was considered the correct choice. The monkey received visual and auditory feedback for his choice; a correct choice was followed by a low-pitched tone and the chosen disk was substituted by a black diamond, while an incorrect choice was followed by a high-pitched tone and the disappearance of the chosen disk.

Cell Recording and Selection

Recorded action potentials were sorted offline using the WaveClus spike clustering algorithm (Quiroga, Nadasdy, & Ben-Shaul, 2004). From this we identified a total of 117 units, 82 of which were suitable for further analysis (monkey J: N=44, monkey R: N=38). We included cells that met the following criteria: a) the assumed PREF and ANTI locations, as determined by online spikes from the location selectivity mapping task, corresponded to the actual PREF and ANTI locations, as determined by offline analysis, b) firing rate for centrally presented shapes remained high enough for the maximum depth of selectivity index to be calculated (see Results), c) the monkeys completed all tasks, i.e. the location selectivity mapping task, the passive shape-mapping task, and more than one block of the shape-saccade association task.

Data Analysis

Unless otherwise noted, all statistical tests were two-sided and had an alpha level of 0.05. For repeated measures ANOVAs, results were Greenhouse-Geisser corrected if Mauchly’s test of sphericity was significant.

When analyzing data from the passive shape-mapping and active shape-saccade association tasks, we aligned neural responses in every trial to the visual onset of each shape and counted the number of action potentials within a 50 ms window centered on the time of onset. We repeated this process for windows spaced 10 ms apart. Unless otherwise stated, any reference to timing in the following text indicates the center time of such a window.

Results

Location selectivity mapping task

As previously described, data from the location selectivity mapping task was used to define a preferred location of each LIP unit. Most units had a preferred location in the contralateral hemifield (67 CONTRA units) while a minority preferred a location in the ipsilateral hemifield (15 IPSI units). The circular mean direction of the preferred location was 192°, and the circular standard deviation was 59.7°.

Passive shape-mapping task

In the passive shape-mapping task, more CONTRA units also showed greater responses for shapes presented contralaterally than ispilaterally, and vice versa for IPSI units. This gave us some reassurance that the neurons’ spatial preference would not change dramatically across tasks and stimuli. One-sample binomial tests at times 0 to 190 ms after visual onset of shapes in the passive task confirmed that more units kept their location preference (contralateral versus ipsilateral) across the two tasks (location selectivity mapping task and passive shape-mapping task) than would be expected by chance (CONTRA units: significant at all times from 30 to 190 ms after shape onset, all significant ps < 0.001; IPSI units: significant from times 30 to 80 ms after shape onset, first significant p = 0.035, all other significant ps < 0.008). Figure 3 shows an example unit’s responses to shapes shown in different locations.

Figure 3. Example neural responses in the passive shape shape-mapping task.

These spike density functions show one unit’s responses to familiar and novel shapes shown in the preferred (PREF) location, the center of the screen (CENTER), and the anti-preferred (ANTI) location. In the following active shape-saccade association task, the FUTURE-ANTI shapes served as 100% valid central precues to the anti-preferred location, and the FUTURE-PREF served as such cues to the preferred location. The neuron’s responses are not necessarily representative as the selectivity of LIP neurons varied greatly from unit to unit.

In the main task (the active shape-saccade association task), the visual shapes were always presented in the center of the screen. We therefore briefly describe the neural responses to visually presented central shapes in the passive shape-mapping task. In this task, LIP neurons showed varying degrees of responses to visually presented central shapes. Some neurons did not seem to respond much to the shapes at all, while others responded to the shapes, sometimes selectively so. We wanted to try to quantify the selectivity of each neuron’s responses to visually presented central shapes, and compare the selectivity for novel and familiar shapes. LIP responses were often brief and dynamic so selectivity and preference often seemed to change over a short period of time. We therefore calculated a depth of selectivity (DoS) index (Rainer & Miller, 2000) for each time window centered on 40 ms after shape onset to 190 ms after shape onset (or, equivalently, 40 ms after a shape’s visual offset), and used the maximum DoS index as a measure of the neuron’s selectivity to centrally presented shapes. DoS can range from 0 (cell responds equally to all shapes) to 1 (cell responds only to one shape). We did this separately for familiar and novel shapes.

$$\mathit{DoS}=\frac{n-\frac{\mathrm{\sum }{R}_i}{R_{\mathit{max}}}}{n-1}$$

The selectivity of responses to centrally presented shapes varied greatly between neurons (M_familiar =0.59, SD_familiar =0.191; M_novel = 0.56, SD_novel = 0.179). The selectivity of responses to centrally presented familiar and novel shapes was significantly correlated (r(80) = 0.746, p < 1.0 × 10⁻⁶). The selectivity of responses to central familiar shapes was nonetheless significantly greater than to central novel shapes (see figure 4; paired samples t-test: t(81) =2.183, p = 0.032). The selectivity of responses to the familiar, behaviorally relevant shapes therefore appears to be enhanced. The differences between the selectivity of responses to novel and familiar shapes were nonetheless slight and should be interpreted with some caution given the fact that eye position was not tightly controlled in this secondary task. We turn now to the main task, the active shape-saccade association task.

Figure 4. Relationship between the selectivity for familiar and novel shapes shown in the CENTER location.

Each marker corresponds to one LIP unit. The line shows the linear least squares fit.

Active shape-saccade association task

Changes in neural responses after short-term and long-term learning

We wanted to see if and how the responses to novel shapes in the active shape-saccade association task changed with short-term learning, or learning over the course of a single session. We also wanted to contrast these shorter-term learning effects with the effects of long-term learning over the course of days, weeks or months.

The monkeys’ performance for familiar shapes in the first block of the active shape-saccade association task was in general very good or 92% on average in 73 sessions. Mean performance for novel shapes in the first block was only 54%. While their performance was significantly better than expected based on chance alone (t(72) = 3.355, p = 0.001, one-sample t-test), it was still low, indicating that the monkeys in general did not know much about the meaning of the novel shapes in the first block of trials. Their performance, however, generally improved for the novel shapes over the course of the training session.

In order to look for changes in neural responses related to shorter-term learning within a single day, we examined the responses to novel shapes in two blocks of the active shape-saccade association task: the first block in a session and the block with the best behavioral performance for novel shape cues in the same session, provided that the block included at least 30 novel shape trials and that the monkey showed any behavioral improvements after the first block (77 out of 82 units). Note that we refer to shape cues as novel as long as they have not been seen in previous sessions, and will use the terms early and late novel shape trials. To examine long-term learning effects, we used the first block of trials to compare neural responses to familiar shapes that in the past had been associated multiple times with either orienting to the PREF or ANTI location of the neuron recorded from.

We compared the distributions of each neuron’s firing rates for shapes which cued the PREF location, and shapes which cued the ANTI location. This was done separately for familiar shape trials, early novel shape trials, and late novel shape trials. More specifically, for every time window between 0 and 1500 ms after shape onset, we calculated the area under the receiver operating characteristic curve (AUC) comparing these two distributions (Green & Swets, 1966). We then scaled the AUC scores so that they could theoretically range from −100 to 100. In the rest of the paper we will refer to the scaled score as congruency:

$$\mathit{congruency}=200\times ({\text{AUC}}_{\text{FUTURE}-\text{PREF}\text{over}\text{FUTURE}-\text{ANTI}}-0.5)$$

A positive congruency score implies that in general the neural responses to central shapes that cued the PREF location were higher than to shapes that cued the ANTI location, i.e. the responses were congruent. The reverse is true for a negative score; it represents incongruent activation. The greater the absolute value of a score, the greater was the separation between the neural response distributions of PREF and ANTI shape cues.

Only correct trials of the active task were included, so in all cases the monkey eventually made an eye movement to the location cued by a central shape, regardless of whether it was novel or familiar.

As can be seen in figure 5, the average congruency of the neural population changed across a trial of the active shape-saccade association task. We are mainly interested in the shape period, or the 500 ms time period between the visual onset of shapes and choice disks. To first briefly describe the neural dynamics in the following choice period, the responses of the neural population became congruent almost immediately after the visual onset of the choice disks, sharply became incongruent around 160 ms later, and finally leveled off. This was most apparent for familiar shape trials and least apparent for early novel shape trials even though the monkeys’ overt behavioral choices were equated. Within a single session of learning where to orient in response to seeing a novel shape, the population response dynamics during the choice period started to resemble those of the familiar over-learned shape trials.

Figure 5. Congruency of the neural population responses across a trial of the active shape-saccade association task.

The mean congruency scores at each time after the visual onset of shape are shown for early novel (blue), late novel (gray), and familiar (brown) shape trials. A positive congruency score signifies that the firing rate was in general higher during PREF shape trials than ANTI shape trials, while a negative score indicates the opposite.

We wanted to know whether and then how shape responses were affected by short- and long-term learning of associating shapes with orienting to either a neurons’ preferred or anti-preferred location. We started by looking at the response dynamics within the shape period of the active shape-saccade association task for early novel, late novel, and familiar shape trials. We first ran three separate repeated measures ANOVAs with time as the single factor. The time variable consisted of congruency scores for 10 non-overlapping 50 ms time bins centered on 50 to 500 ms after stimulus onset. Congruency did not significantly vary over time for early (F(6.164,468.435) = 0.702, p = 0.652) or late (F(3.645, 277.008) = 0.533, p = 0.851) novel shapes. Congruency did, however, vary over time for familiar shapes (F(3.019, 229.414) = 5.345, p = 0.001). Responses to familiar shapes were congruent right after visual onset but became increasingly incongruent as the start of the choice period drew nearer.

We followed up with single sample t-tests at each time point and for each type of cue (early novel, late novel, and familiar) where we looked at whether the congruency scores were significantly different from zero (30 tests in total). Using conventional significance levels, the early novel responses were significantly congruent 450 ms after shape onset (p = 0.032; all other ps > 0.074), late novel responses were never significantly congruent or incongruent (all ps > 0.055), and responses to familiar shapes were significantly congruent 50 ms (p = 0.0005) and 100 ms (p = 0.035) after shape onset, and significantly incongruent 400 ms after shape onset (p = 0.036). Early neural responses (i.e. 50 ms after shape onset, M = 6.8, SD = 16.8) to familiar shapes were significantly congruent even when a stringent correction for multiple comparisons was applied (threshold of significance with Bonferroni correction: 0.0017). These early responses to familiar shapes were also significantly more congruent than those to both early (paired samples t-test, t(81)=2.518, p=0.014) and late (paired samples t-test, t(76)=2.541, p=0.013) novel shapes. Unlike for the novel shapes, the neurons showed characteristic response differences between familiar PREF and ANTI shape cues extremely early after the visual onset of the shapes. This difference was modest but robust; around twice as many neurons favored familiar shapes that cued their preferred rather than their anti-preferred location.

These early responses to familiar shapes tended to be congruent regardless of whether a unit’s preferred location was contralateral or ipsilateral (congruency at 50 ms after shape onset for units with contralateral preference: M=6.9, SD=17.37; ipsilateral preference: M=5.9, SD=14.49; independent samples t-test for differences in means, t(80)=0.214, p=0.831). Units with contralateral preference would be exposed to the same visual stimuli as units with ipsilateral preference, but the shapes’ meaning would differ; a shape associated with a contralateral unit’s preferred location would be associated with an ipsilateral unit’s anti-preferred location and vice versa. Early congruent responses are therefore unlikely to stem from some accidental properties of the shapes themselves, but instead reflect the learned task-related relationship between a shape and a neuron’s spatial selectivity.

At a first glance, LIP responses to centrally presented shapes might seem little affected by the learning that took place within a single session. However, a neural population whose mean responses are neither congruent nor incongruent might nonetheless have gone through experience-dependent changes that are not reflected in the average congruency scores. In addition to looking at population averages, we therefore looked at whether congruency scores for late novel shapes could be predicted based on congruency scores for familiar shapes over and above the prediction based on congruency scores for early novel shapes alone.

Specifically, we performed a hierarchical regression at each of the 10 time points in the shape period of the active shape-saccade association task. Congruency scores for late novel shapes were treated as a dependent variable. Congruency scores for early novel shapes and familiar shapes were entered as predictor variables in consecutive steps. We then looked at whether a model that included congruency scores of both early novel and familiar shapes predicted congruency scores for late novel shapes significantly better than a model where the congruency scores of early novel shapes were used as the sole predictor variable.

Congruency scores of early novel shapes alone significantly predicted congruency scores of late novel shapes at all time points in the shape period (minimum R² = 0.081, maximum R² = 0.307; all ps < 0.011). Adding congruency scores of the familiar shapes as a second independent variable significantly improved the predictive power of the statistical model at time 150 ms after shape onset (R² change = 0.042, p = 0.032) and then again at time 300 ms after shape onset and at all times from thereon (minimum R² change = 0.095, maximum R² change = 0.225; all ps < 0.004; threshold for significance after Bonferroni correction: 0.005). While the congruency scores of the late novel shapes kept some similarity to the congruency scores of the early novel shapes throughout the shape period, they increasingly resembled the congruency scores of the familiar shapes as the shape period progressed.

Persistent distinctive shape-related activity after long-term learning

We have shown that repeatedly associating visual shapes with orienting to particular locations can change how LIP neurons respond when those shapes are seen, so that even the earliest neural responses can reflect the orienting behavior to which the shapes have been linked. The question remains, then, to what extent the responses are overwritten by experience. Do responses to familiar shapes that cue the same location still retain some individual characteristics, even though the monkeys have been extensively trained, over the course of several months, on reacting to them in the same way?

Our experiment was set up so that two centrally presented familiar objects of different shapes cued each possible target location. We compared the neural responses of such same-meaning familiar shapes to see if responses to objects repeatedly linked to the same behavior were still distinct from one another. We did this by sliding a 50 ms window in 10 ms steps from 0 ms to 1500 ms after the visual shape onset in the first block of the active shape-saccade association task, counting the number of spikes within each window, and comparing the distribution of the number of spikes evoked by same-meaning shapes by calculating the area under the receiver operating characteristic curve (AUC) comparing these two distributions (Green & Swets, 1966). Since we had no specific predictions about which of any two same-meaning shapes would evoke higher neural activity, we took the absolute value of the scaled AUC for each shape pair. For each time point we therefore found two such scores for each neuron, one comparing the response distributions of the two familiar PREF shapes and the other comparing the two familiar ANTI shapes, and defined the distinction score at each time point as the combination of the two scores:

$$\mathit{distinction}=100\times (\mathit{abs}({\text{AUC}}_{\text{familiar}\text{PREF}\text{shape}\text{pair}}-0.5)+\mathit{abs}({\text{AUC}}_{\text{familiar}\text{ANTI}\text{shape}\text{pair}}-0.5))$$

This gave us a vector of distinction scores for each neuron that signified how well it differentiated among same-meaning familiar shapes at each time point after the visual onset of shape. The population means of the distinction scores at each time point in the active shape-saccade association task can be seen in figure 6.

Figure 6. Distinction scores of familiar shapes in the active shape-saccade association task.

Familiar shapes that have repeatedly been associated with the same arbitrary orienting action can still evoke differentiable neural responses. A permutation test showed that the mean distinction scores were significantly greater than expected by chance throughout almost the entire shape period (from 40 ms after shape onset and onward, excluding the time window centered at 50 ms after visual onset of shape). 95% of the permuted distinction scores fell within the dark gray band, and 99% within the light gray band.

The distinction scores can theoretically range from 0 to 100, where 0 indicates that the neural responses to same-meaning familiar over-learned shapes are indistinguishable, while 100 signifies that they are completely separable. In reality, the neural responses to two same-meaning shapes will almost always be somewhat different by chance alone. In light of our previous results, we wanted to know whether there were differences in the early shape responses to familiar shapes even though they had repeatedly been associated with the same orienting action.

We performed a one-sided permutation test to see whether distinction scores for same-meaning familiar shapes were significantly greater than expected by chance alone. For each shape pair, we shuffled the labels (shape A or B) of the responses in all familiar shape trials and calculated a vector of distinction scores based on the shuffled labels. We repeated the process 1000 times. Figure 6 shows the distribution of shuffled distinction scores at each time point after shape onset in the active shape-saccade association task. The graph depicts how much LIP neurons tended to differentiate between same-meaning shapes at any given time.

The average distinction score in the active shape-saccade association task first became significant 40 ms after visual onset (p=0.026), marginally missed the significance level at 50 ms after visual onset (p=0.056) and stayed significantly greater than expected by chance throughout the rest of the shape period (i.e. until 500 ms after visual onset, highest p=0.044, lowest p<0.001) and beyond. The barely missed significance level at 50 ms in the active task is probably due to the fact that early responses to familiar shapes start to reflect the learned associations, as described above.

Discussion

In this paper, we have documented and compared the experience-dependent changes of LIP responses to visual objects after longer and shorter learning periods of arbitrary pairings between objects and orienting.

The effects of short-term learning

Experience-dependent changes in LIP responses to visual objects start to unfold over a short period of learning but these changes are seen relatively late after the visual onset of these objects. The effects of short-term learning do not manifest themselves as an overall increase or decrease in LIP responses to objects that have been paired with orienting toward or away from the neurons’ preferred locations. Instead, the responses of LIP neurons to novel objects increasingly resemble activity seen for familiar objects that share their meaning (i.e. cue the same location).

This late information is therefore not related in any obvious way to the responses evoked by the presentation of visual stimuli in the preferred or anti-preferred locations of LIP neurons; it might be independent of the neurons’ spatial selectivity and could be considered akin to the categorical information that has been previously reported to exist in LIP (Fitzgerald, Freedman, & Assad, 2011; Freedman & Assad, 2006, 2009). Late categorical information, task or rule selective activity (Stoet & Snyder, 2004) could be relayed to LIP from prefrontal regions (Asaad, Rainer, & Miller, 2000) like the dorsolateral prefrontal cortex with which it is structurally connected (Blatt et al., 2004). Through top-down control, the prefrontal cortex might be able to modulate information in more posterior regions according to task demands (Chao & Knight, 1998; Miller & D’Esposito, 2005; see Miller & D’Esposito, 2005 for a discussion on top-down control signals originating in the prefrontal cortex; see Pan & Sakagami, 2012 for a review of categorical representations in the prefrontal cortex; see Seger & Miller, 2010 for a review of the neuroscience of categorical learning).

The effects of long-term learning

Our results indicate that the earliest, apparently visual responses of LIP neurons can carry information about well-established yet still arbitrary associations of objects with particular orienting actions. These bottom-up visual responses tend to be greater for objects that have repeatedly been associated with looking toward, rather than away from, LIP neurons’ preferred spatial locations. After a long training period, LIP neurons reflect an arbitrary association by responding to familiar central shape cues as if a weak visual stimulus was shown in the associated empty peripheral location.

Orienting-related responses to visual shapes appear so early after visual onset that it is highly unlikely that they are the result of motor commands fed back from other brain regions. It is also implausible that these early responses to visual shapes are inherited from ventral visual regions; while LIP is known to be interconnected with shape selective ventral areas (Blatt, Andersen, & Stoner, 2004; Ungerleider, Galkin, Desimone, & Gattass, 2008; Webster, Bachevalier, & Ungerleider, 1994), the visual onset latencies of neurons in those regions tend to be long (Baylis, Rolls, & Leonard, 1987; Kiani, Esteky, & Tanaka, 2005; Schmolesky et al., 1998; Tamura & Tanaka, 2001). It is therefore doubtful that object or shape information reaches LIP solely through a circuitous route through the ventral visual stream, although we do consider it likely that LIP eventually receives some information about visual objects from the temporal cortex and other ventral regions.

Instead, these responses appear to be generated from the initial bottom-up wave of visual signals that reach LIP, presumably created de novo in the parietal cortex from yet unknown inputs. When the association between a visual stimulus and an orienting response is highly overlearned, LIP might become able to support extremely rapid arbitrary visuomotor transformations independent of top-down feedback from regions such as the prefrontal cortex (Swaminathan & Freedman, 2012). Without parietal cortex, the associations could still be remembered, but the associated behavior might be slower and less automatic.

Experience-resistant responses to visual objects

LIP responses to visual objects can be modified and overridden but not completely overwritten by experience. During the visual presentation of an object, before any overt behavioral response is allowed, LIP neurons do carry information about the orienting action associated with the object and which the monkey is going to perform. During this same period, the responses to the visual objects can nonetheless be distinct even though they were similarly acted on in the past and will lead to the same orienting behavior in the future. Neural responses to such objects can be separable and resistant to a complete experience-dependent overhaul despite the fact that the monkeys were trained over the course of many months to treat the objects as equivalent.

Relations to behavior

Orienting guided by central cues is often described as endogenous, voluntary, or controlled, as opposed to the exogenous, reflexive, and automatic effects of peripheral cues (Müller & Rabbitt, 1989; Posner, 1980). Indeed, learning the meaning of novel central cues only has a measurable effect on neural responses in LIP relatively late after cue onset, presumably through top-down feedback, and these responses are nothing like the responses to peripheral visual stimuli in the locations cued by the central objects.

Still, our results are in alignment with the cumulating behavioral evidence that a former endogenous visual cue might be said to become exogenous with enough training (Dehaene, Bossini, & Giraux, 1993; Dodd & Wilson, 2009; Fischer et al., 2003; Fischer, Warlop, Hill, & Fias, 2004; Van der Stigchel, Mills, & Dodd, 2010; Shaki & Fischer, 2008). LIP neurons can respond to familiar central shape cues as if a weak stimulus is actually being presented in the empty peripheral location cued by the central object. The time course of these neural effects closely follows that of transient visual attention; the facilitatory behavioral effects of an uninformative peripheral cue is greatest for a target shown in that location around 50 ms after cue onset, and this facilitation gives way to inhibition around 300 ms after cue onset (Castel, Chasteen, Scialfa, & Jay Pratt, 2003; Nakayama and Mackeben 1989; Posner & Cohen, 1984; Posner, Rafal, Choate, & Vaughn, 1985); our familiar central shape cues also evoked responses that were maximally congruent at 50 ms after cue onset and these responses became incongruent around 300 ms after cue onset. Our interpretation of these effects is that the familiar shape cues gained the ability to rapidly bias spatial attention to a particular peripheral location away from the central objects themselves. Since the target did not appear in this location until 500 ms after cue onset, and because it was maladaptive for the monkeys to actually direct their gaze to that location until a target appeared, initial congruent responses gave way to incongruent responses until the target appeared and had to be acted on.

With enough training, an object of any shape might acquire the ability to bias orienting to a particular location. However, while our results show that experience can affect the responses of LIP neurons to visual objects, these neurons can nonetheless respond differently to two objects that cue the same location despite a lengthy training period that encourages the monkeys to treat the two objects as equivalent. We speculate that persistent response differences to same-meaning objects reflect their inherent shape-derived orienting biases. Our own behavioral work (Sigurdardottir et al., 2014) shows that information derived from the shape of objects – even never-before-seen novel ones – can rapidly and automatically bias orienting to particular spatial locations. These links between shape and space, which can be thought of as initial hypotheses on where to look and pay attention, might be hard or impossible to fully overcome (Sigurdardottir et al., 2014).

The activity of LIP might best be understood as competing orienting biases or affordances (Cisek, 2007; Cisek & Kalaska, 2010; Gibson, 1979), or the relative merit of the possible sources of information worth exploring with the eyes and attention. We propose that the shape of objects, because of intrinsic properties and previous experience, systematically biases orienting (Egly, Driver, & Rafal, 1994; He & Kowler, 1991; Melcher & Kowler, 1999; Ristic & Kingstone, 2006; Sigurdardottir, Michalak, & Sheinberg, 2014; Theeuwes, Mathôt, & Kingstone, 2010; Tipples, 2002; Vishwanath, Kowler, & Feldman, 2000). We hypothesize that LIP plays a crucial role in extracting such a shape-induced orienting bias and that this bias contributes to the brain region’s selective responses to visual objects of different shapes.

Thinking of LIP shape selectivity as serving the purpose of orienting might help to make sense of the puzzling finding that LIP and its putative human homologue can be relatively tolerant to image transformations like scaling and translation (Janssen et al., 2008; Konen & Kastner, 2008; A. B. Sereno & Maunsell, 1998). Such invariance has most often been thought to be a hallmark of the ventral visual pathway (Booth & Rolls, 1998; Robert Desimone, Albright, Gross, & Bruce, 1984; Gross, 1973; Ito, Tamura, Fujita, & Tanaka, 1995; Logothetis & Sheinberg, 1996; Tanaka, 1996). Visual stimuli can however also show invariance of the orienting bias they evoke, such as when seeing a face tilted 90 degrees evokes orienting shifts to the side to which the person’s eyes would have been looking had the face been in its canonical upright position (Bayliss & Tipper, 2006; Bayliss, di Pellegrino, & Tipper, 2004). We expect LIP neurons to be tolerant to changes in a visual stimulus that preserve not its identity or form but its inherent or acquired orienting bias.