Saccades trigger predictive updating of attentional topography in area V4

Summary

During natural behavior saccades and attention act together to allocate limited neural resources. Attention is generally mediated by retinotopic visual neurons, therefore specific neurons representing attended features change with each saccade. We investigated the neural mechanisms that allow attentional targeting in the face of saccades. Specifically, we looked for predictive changes in attentional modulation state or receptive field position that could stabilize attentional representations across saccades in area V4, known to be necessary for attention-dependent behavior. We recorded from neurons in monkeys performing a novel spatiotopic attention task where performance depended on accurate saccade compensation. Measurements of attentional modulation revealed a predictive attentional “hand-off” corresponding to a presaccadic transfer of attentional state from neurons inside the attentional focus before the saccade to those that will be inside the focus after the saccade. The predictive nature of the hand-off ensures that attentional brain maps are properly configured immediately after each saccade.

ETOC Blurb

Using a novel behavioral task, the authors report that the attentional modulation state of neurons in extrastriate area V4 is updated before saccade onset. This attentional hand-off is independent of changes in receptive field position and represents a new type of perisaccadic updating.

Introduction

One hallmark of natural vision is the spatiotemporally rich pattern of stimulation generated by the simultaneous appearance of multiple objects in the visual field. Humans and monkeys prioritize proce ssing of features in complex scenes using a combination of saccadic eye movements and visual attention. Saccades are fast eye movements that guide the fovea towards behaviorally significant features to facilitate processing. At the same time, attentional mechanisms can enhance processing of attended locations and features throughout the visual field by facilitating neural responses (Moran and Desimone, 1985, Reynolds et al., 2000, Motter, 1993, Luck et al., 1997, Treue and Maunsell, 1996). Oculomotor and attentional control are mediated by an overlapping set of cortical and subcortical brain structures (Corbetta et al., 1998, Corbetta, 1998) that work together during natural vision to facilitate efficient use of limited neural resources. Saccades, however, are a potential problem for neurons in retinotopic visual areas that are critical for attention-guided behavior (Gallant et al., 2000) – in retinotopically organized areas, the specific neurons representing an attended feature can change with each saccade. Without some compensatory mechanism, eye movements would lead to attentional facilitation of the wrong visual feature.

Behavioral studies have shown that the attentional control system in humans can compensate for saccades, allowing us to sustain attention at a fixed environmental or “spatiotopic” location across saccades (Golomb et al., 2008, Marino and Mazer, 2016). While saccade-related activity, in the absence of attention, and attentional modulation, in the absence of eye movements, both have been extensively studied in humans and monkeys (reviewed in Carrasco, 2011), little is known about the neural mechanisms that provide saccade compensation in attentional priority maps to allow spatiotopic attentional targeting.

Two neural models for attentional saccade compensation have been suggested (Marino and Mazer, 2016). In one, the spatial selectivity of attentionally modulated neurons in visual cortex, and therefore the retinotopic organization of attentional maps, remains fixed across saccades, while the spatial pattern of attentional modulation across maps changes. Alternatively, the attentional state of any given neuron in one of these maps could remain constant, while the spatial selectivity, i.e., the RF, of the neuron shifts or “remaps.” Studies of attentional targeting and spatial selectivity around the time of saccade initiation have provided conflicting support for both models.

Behavioral studies of perisaccadic spatiotopic attention (Golomb et al., 2008, Rolfs et al., 2011, Mathôt and Theeuwes, 2010, Jonikaitis et al., 2013, Posner and Cohen, 1984, Jiang and Swallow, 2013, Szinte et al., 2015) have shown transient, but obligatory, post-saccadic attentional benefits at a behaviorally irrelevant location corresponding to the retinotopic location occupied by the attentional locus before the saccade (Posner and Cohen, 1984, Golomb et al., 2008, Jonikaitis et al., 2013, Jiang and Swallow, 2013). Consistent with these findings, a combined fMRI/ERP study found that in retinotopic visual cortex, presaccadic attentional facilitation persists after the saccade at visual field locations made task irrelevant by the saccade (Golomb et al., 2010). These findings support a retinotopic model of spatiotopic attention, where attention is targeted in retinotopic coordinates and actively transferred or handed off between neurons in retinotopic cortex based on oculomotor plans. Behavioral studies have also shown enhanced presaccadic visual sensitivity at visual field locations represented by neurons with RFs that will occupy the attentional focus after the saccade (Rolfs et al., 2011, Szinte et al., 2015, Jonikaitis et al., 2013), suggesting that transfers of attentional state are predictive and start before saccade initiation. Predictive updating of attentional maps could minimize the impact of saccades on behavior by ensuring attention is directed towards appropriate environmental targets as soon as the eye movement is complete.

However, there is also substantial evidence that in many brain regions modulated by attention, neuronal RFs are not static, but dynamically shift or “remap” in response to saccade plans. RF remapping was first described in the lateral intraparietal area (LIP) by Duhamel and colleagues (1992) who reported that nearly half of LIP neurons responded to stimuli appearing at the post-saccadic RF location (termed the “future field”) during the interval between saccade target onset and saccade initiation. Subsequent reports have shown that presaccadic RF remapping occurs throughout the primate visual system, including intermediate visual areas V2, V3, and V3a (Nakamura and Colby, 2002), V4 (Neupane et al., 2016, Tolias et al., 2001), the frontal eye fields (Umeno and Goldberg, 1997, Umeno and Goldberg, 2001, Zirnsak et al., 2014), MT (Yao et al., 2016, although see Ong and Bisley, 2011), MST (Inaba and Kawano, 2014), and the superior colliculus (Churan et al., 2012, Walker et al., 1995) as well as LIP.

At first glance, remapping seems like an obvious candidate mechanism for attentional stabilization (see Marino and Mazer, 2016 for review). However, to date only a few neurophysiological studies have examined how attentional topography in visual cortex changes in preparation for or in response to a saccade. Those studies that have investigated remapping in the context of attention (e.g., Joiner et al, 2011, Mirpour and Bisley, 2012) have generally focused on a specific question: Is attentional modulation state remapped along with spatial selectivity in neuronal populations that are known to remap? While informative, this has led to an exclusive focus on remapping neurons and does not directly address whether remapping functions to stabilize attentional representations in the brain or whether attentional representations can compensate for saccades without spatial remapping.

To address these questions, we recorded from single neurons in area V4 of two macaque monkeys performing a novel “go/no-go” spatiotopic attention task. The task required animals to execute a guided saccade while simultaneously searching for and detecting a target stimulus that could appear before, during or after the saccade at a separate, cued spatiotopic location on the display. We analyzed neural responses to a continuous stream of dense, task-irrelevant mapping probes presented continuously during each trial to precisely measure RF position and sensitivity throughout the trial with high spatiotemporal resolution and found evidence of a pre-saccadic, predictive hand-off of attentional state distinct from RF remapping. This hand-off reflects a transfer of attentional state between neurons that encode stimuli inside the attentional focus before the saccade and neurons that will encode the same stimuli after the saccade. Our results suggest that spatiotopic attention is mediated by an active transfer of attentional modulation state from one set of visual neurons to another, and not by classical remapping.

Results

Behavioral evidence of spatiotopic attention

Two macaque monkeys were trained to perform a novel spatiotopic attention task that required them to attend to a fixed location on a computer display while executing a single, accurate saccade to a different location. In brief, animals executed a guided saccade between two fixation spots (FP1 and FP2) selected at random on each trial while simultaneously performing a target detection task at a cued location. When targets appeared at the cued location, they had to release a touch bar within 500 ms. If the target appeared at any one of five possible uncued (and therefore unattended) locations, they were required to continue holding the bar. Targets were oriented bars filled with a binary white noise texture presented for 200–300 ms at a uniformly distributed random time ±300 ms relative to predicted saccade onset (based on a running average of saccadic reaction times). Animals received a liquid reward upon making both a correct manual target response (go or no-go) and, on trials where the saccade target appeared, executing a single accurate saccade to FP2. Trials were organized into blocks of 40–75 correct trials. The first 3–7 correct trials in each block were instruction trials, where a black rectangle, visible for the entire trial, cued the attended spatiotopic location. After the instruction period, the rectangle was removed and animals performed the task without an explicit cue for the remainder of the block. On each trial, FP1 was selected from one of three possible locations symmetrically arranged along an imaginary horizontal line; FP2 was always constrained to be adjacent to FP1, resulting in four equiprobable saccade vectors originating from three different initial fixation positions. Once stable fixation at FP1 was achieved, a dynamic stream of task irrelevant mapping probes appeared and flickered continuously for the duration of the trial. Probes were low contrast bars matching the cell’s preferred orientation and RF size and were of the same mean luminance as the target, but were uniformly light or dark gray. Probes appeared along two imaginary horizontal lines, one passing through the RF center, the other through the RF’s reflection across the horizontal meridian (see STAR Methods). Animals were extensively trained to ignore mapping probes while monitoring the attended location for targets. Task structure and timing are illustrated in Figure 1. All neurophysiological results reported here are based on data from uninstructed trials where the monkey executed both a correct manual response and, if instructed, an accurate saccade.

Figure 1.

Behavioral task. (a) Schematic illustration of trial structure. Animals had to fixate at FP1 and grasp a touch bar to fully initiate each trial. After a randomized interval, FP1 disappeared, FP2 appeared and animals had to make a saccade to FP2. On non-catch trials a target stimulus appeared at a randomized time at either an attended or unattended location. The target could appear before, during or after the saccade. Animals had to release the bar when the target appeared at the attended location and hold the bar if it appeared anywhere else. Animals were rewarded on trials following a correct manual response and an accurate saccade. The black outline rectangle visible in the cue period indicated the attended location and was visible only on instructed trials. (b) The appearance of the probes and target, as well as the task geometry, were adjusted for each cell studied based the on the cell’s RF size (σ) and position. Targets could appear at 6 possible locations; only the two central locations were ever cued. Saccade lengths (‘a’) were adjusted to be 1–2.25 σ, spacing between mapping probes (‘b’) was set to 0.3–0.5 σ and the exact position of the stimulus array set to locate the lower central target location at the center of the RF. (c) Black traces indicate the timing of a typical trial. The times at which the target and FP2 appeared were drawn from uniform distributions (see STAR Methods); the red dashed lines indicate the range of those distributions.

To verify that monkeys, like humans (Golomb et al., 2008, Posner and Cohen, 1984), can sustain a spatiotopic attentional locus across saccades, we calculated hit rates on go trials (where the target appeared at the attended location and monkeys had to respond) and false alarm rates on no-go trials (where the target appeared at one of the five possible unattended locations and monkeys had to withhold response). If monkeys, like humans, can sustain a spatiotopic locus of attention we would expect high hit rates and low false alarm rates both before and after the saccade. If saccade execution prevented or interfered with maintaining an attentional locus, we would expect a reduction in hit rate and/or an increase in false alarm rate around the time of the saccade. However, this is not what we observed; the mean hit rate (combining data from both animals) before the saccade was 64.3% [64.27, 64.33%] ([ ] indicates 95% confidence intervals estimated from 1000 bootstraps, throughout) and the false alarm rate was 22.0% [21.07, 22.03%]. After the saccade, the mean hit rate was 66.8% [66.73, 66.83%] and false alarm rate was 22.7% [22.67, 22.73%] (Figure 2a). Hit and false alarm rates were used to compute a continuous measure of target discriminability,

Figure 2.

Behavioral performance summary. (a) Hit and false alarm rates are plotted for attended and unattended targets appearing during the presaccadic and post-saccadic intervals. Dashed line indicates chance performance. Circles and solid lines indicate combined performance of both animals; squares and diamonds indicate individual performance of monkeys B and M, respectively. Small black vertical lines indicate 95% CIs (b) Upper panel shows a continuous estimate of hit and false alarm rates as a function of time relative to the saccade (both monkeys); lower panel shows the corresponding visual sensitivity index. Shaded regions indicate 95% CIs.

$$d^{\prime }=Z(HR)-Z(FA),$$ (1)

where HR and FA are hit and false alarm rates, respectively, and Z() is the inverse normal cumulative distribution; d′ was computed as a function of target onset time relative to saccade onset (Figure 2b). d′ was nearly constant and well above chance across all task epochs, confirming that monkeys, like humans, can robustly target and sustain attention using a spatiotopic reference frame and that saccades have little effect on sustained attention. Although saccades did not affect d′, there were small dips in both hit and false alarm rates around the time of the saccade (Figure 2b, upper panel). This simultaneous reduction in hit and false alarm rates is consistent with a non-specific change in target sensitivity without a change in discriminability and is consistent with the established effects of saccadic suppression on visual contrast sensitivity (Duffy & Lombroso, 1968).

Attentional Modulation of Neuronal Responses at Fixation

We recorded neuronal activity at 174 sites in dorsal V4 of two monkeys (102 single neurons and 72 multi-unit sites). At each site, spatial RFs were estimated by reverse correlation of responses to a sparse noise stimulus while monkeys performed a simple fixation task (Jones and Palmer, 1987, Mazer et al., 2002). RFs were fit with a circularly symmetric 2D Gaussian to determine size (σ) and location(x₀, y₀). Responses to sparse noise were also used to define the temporal analysis window (response latency and duration) used in the analyses described below (see STAR Methods). Preferred orientations were estimated by reverse correlation of responses to dynamic sinusoidal grating sequences (Mazer et al., 2002, Ringach, Hawken, & Shapley, 1997). In the spatiotopic attention task, target and mapping probe orientations were matched to the preferred orientation of the cell, and RF size and location were used to determine the size, spacing and positions of the probes and targets (see STAR Methods and Figure 1b).

To determine whether target-evoked visual responses were modulated by attention, we initially compared responses to targets appearing in the RF on go trials (where attention was directed inside the RF) to responses on no-go trials (where a target appeared in the RF, but attention was directed elsewhere), excluding responses to targets appearing within 100ms of a saccade, by computing a target modulation index,

$$\mathit{TMI}=\frac{T_{\text{in}}-T_{\mathit{out}}}{T_{in}+T_{\mathit{out}}},$$ (2)

where T_in and T_out correspond to the average target-evoked response for attend-in go trials and attend-out no-go trials respectively. Responses to target stimuli were significantly modulated by attention at 45% (78/174; two-tailed t-test, p<0.05) of all V4 recording sites. Facilitation was observed at 86% (67/78) and suppression at 14% (11/78) of the modulated sites (see Figure 3a). The population average TMI was significantly greater than zero (0.059 ± 0.010; mean ± SEM, unless otherwise noted; t₍₁₇₃=5.69; p<10⁻⁷, unpaired t-test), indicating a net attentional facilitation of target responses, consistent with previous V4 studies (Roe et al., 2012). We found no systematic differences between single- and multi-unit recording sites (t₍₁₇₂₎=0.65; p=.52; unpaired two-tailed t-test), so data were combined in subsequent analyses.

Figure 3.

Attentional modulation of V4 neuronal responses to targets and probes during fixation. (a) Scatterplot compares visual responses evoked by attended and unattended target stimuli during the presaccadic interval (>100 ms before saccade onset). Purple indicates neurons where target responses were significantly facilitated by attention (p<0.05), red indicates those where target responses were significantly suppressed and black indicates those with no significant target modulation. Closed and open circles indicate neurons with or without significantly attentional modulation of probe responses, respectively, regardless of sign. Inset histogram shows the distribution of deviations from unity line (arrow indicates mean deviation). (b) Spatiotemporal RF (STRF) modulation for two example cells. Inset figures show complete STRFs; the white boxes indicate boundaries of the spatial and temporal analysis windows derived from the Gaussian and gamma fits, respectively (see text and STAR Methods). Main figures indicate spatial response profiles at peak response latency derived from the full Poisson model in the attend-in and attend out conditions for the same cells as depicted in the insets (shading indicates 95% CIs). In the first example cell, probe responses were significantly modulated by attentional state while in the second example cell, they were not. (c) Summary of the relationship between target and probe modulation. Modulation indices (TMI and PMI) for the two stimulus classes were significantly correlated across the population of V4 neurons studied (r=0.52, p<0.0001, n=174).

To assess the effects of attention on RF position and sensitivity, we used generalized linear modeling (GLM) to model responses to probe stimuli as a Poisson function of retinotopic probe location and attentional focus (either inside or outside RF),

$$log(y_t)=\alpha A_t+\sum _{i=1}^N{\beta }_i{A}_t{S}_{t,i}+\delta (1-A_t)+\sum _{i=1}^N{\gamma }_i(1-A_t)S_{t,i},$$ (3)

where S_t,i is a binary matrix encoding the visibility and retinotopic position of the N mapping probes in the stimulus array along the line passing through the RF, A_t is a binary vector of length N representing the location of the attentional focus relative to the RF (1 for inside, 0 for outside), and y_t is the neuronal firing rate at time t. Firing rates were taken as the mean rate in a temporal analysis window determined from responses to the sparse noise stimuli used for the initial RF measurements: y_t is average firing rate on interval [t+lat, t+lat+dur], where lat and dur are response latency and duration (see STAR Methods). Regressors and response vectors were aligned to saccade onset and binned at the video frame rate (60/75 Hz) before regressing. As for the TMI analysis above, we excluded bins within 100 ms of a saccade from this analysis.

Scalar model parameters α and δ reflect the stimulus-independent firing rate modulation during the attend-in and attend-out portions of each trial, respectively, and vector parameters β_i and γ_i represent responses to probes at each position in the attend-in and -out conditions, respectively. Fits were used to predict the response to isolated probe stimuli at each position in the attend-in and attention-out conditions to construct attend-in and attend-out RF profiles (and therefore to simultaneously represent the stimulus-nonspecific and specific terms of the model). Attend-in and –out RF profiles were then used to quantify attention-related position and shape changes (see Figure 3b).

To confirm the model fit a significant portion of the attention-related response variance, we compared log likelihood (LL) values of the models fit from the actual data to a null distribution of LL values for models fit from randomized datasets generated by shuffling attentional labels (A_t). During shuffling, attentional state was either flipped or conserved uniformly within each trial based on a single fair coin toss to disrupt the relationship between attentional state and neuronal response without altering the temporal dynamics of A_t. In 75.3% (131/174) of neurons studied, probe responses were significantly modulated by attentional state, that is, the LL of the true model was significantly greater than the null distribution (p<0.05, 1000 shuffles; see Table S1), confirming that model fits capture attention-related RF changes.

To compare probe and target modulation, we computed a probe modulation index (PMI), analogous to the TMI above, based on the model’s predicted average response to probe stimuli inside the RF (RF center ±1.5σ),

$$\mathit{PMI}=\frac{P_{in}-P_{\mathit{out}}}{P_{in}+P_{\mathit{out}}},$$ (4)

where P_in and P_out correspond to the average predicted response to RF probes in the attend-in and -out conditions, respectively. Across all recording sites the average PMI was significantly greater than zero (0.037 ± 0.0075, t₍₁₇₃₎=4.84; p<10⁻⁵, unpaired two-tailed t-test), and modulation of target (TMI) and probe (PMI) responses were significantly correlated (r=0.52, p<0.0001; Figure 3c). Eighty-six percent (67/78) of neurons with significant target modulation also exhibited significant probe modulation. The converse, however, was not true: only 51% (67/131) of sites with significant probe modulation also exhibited target modulation (see Table S1). This indicates that although modulation of responses to probes, which each appeared 3.4 times per trial on average, were smaller in magnitude, modulation of responses to probe stimuli may be a more sensitive measure of attentional modulation state than modulation of responses to targets, which appeared only once per trial.

Perisaccadic attentional updating

Next, we investigated the effects of saccade plans on attentional topography. We first classified each trial as: AU, RF moved from an attended to an unattended location, UA, RF moved from an unattended to an attended location or UU, RF was at an unattended location for the entire trial (see Figure 4a). For each trial type we fit a simplified Poisson model in a sliding window:

$$log(y_t)=\alpha +\sum _{i=1}^N{\beta }_i{S}_{i,t},$$ (5)

where the α,β_i, S_i,t and y_t are the same as in Eq. 3 and t indicates the sliding window’s center (relative to saccade onset). We fit the model to data in a 100 ms window stepped from −300 to 300 ms in 25 ms steps. Again, we used model fits to predict the average response to generate RF profiles at each time, for each trial type (Figure 4a, upper panel). Finally, to eliminate modulations unrelated to attentional state but caused by the saccade itself, e.g., any non-specific effects of saccadic suppression, we computed a running visual sensitivity index (VSI),

Figure 4.

Evidence of presaccadic attentional hand-off. (a,b) Single cell (a) and population (b) Estimates of neuronal sensitivity to probes appearing in the RF at different times relative to the saccade (see text for details). Traces correspond to trials where the saccade translated the RF between attended and unattended (AU, red), unattended and attended (UA, purple) or unattended and unattended (UU, black) locations. All three trajectories are illustrated in the upper panels; in the lower panel, the AU and UA trajectories are normalized by the UU trajectory to eliminate the effects of eye movements not related to attentional state. Trajectories are aligned to saccade onset (black vertical line). The temporal pattern of modulation shows a predictive hand-off of attentional state, with RFs initially inside the attentional focus becoming less responsive to probe stimuli appearing well before the saccade when the saccade will bring the RF out of the focus. Conversely, when the RF is initially outside the attentional focus, but the saccade will place the RF inside the focus, cells show and increase in sensitivity to stimuli that appear well before the saccade is actually initiated. Crossing points in bottom panels reflect a conservative estimate of when the attentional hand off occurs (112 ms before the saccade for the sample cell and 45 ms before saccade for the population). Red and purple arrows indicate the earliest time at which attentional state changes significantly. (c,d) The data from (a) and (c) are replotted after adjusting for neuronal response latency to indicate the time at which the attentionally modulated neuronal responses (vs. attentionally modulated stimuli) occur relative to the saccade. Dashed black line in (d) indicates the mean response latency across the population (123.4 ± 1.4ms, center of the temporal analysis window). (e) Histogram shows the distribution of attentional hand-off times (AU/UA VSI trajectory crossing points) for individual neurons. The mean (−33.4 ± 11.7 ms, indicated by arrow) was significantly less than zero (t₍₄₈₎=−2.85; p=0.0032 unpaired one-tailed t-test).

$${\mathit{VSI}}_t=\frac{y_t-{UU}_t}{y_t+{UU}_t},$$ (6)

where y_t corresponds to the probe response temporal profile for either AU or UA trials and UU_t is the profile for UU trials (Figure 4a, lower panel). Figure 4b (lower panel) shows the population VSI trajectories for AU and UA trials, which cross 45 ms [−59, −33 ms] before saccade onset, indicating that attentional state in V4 is updated predictively, before saccade onset. The crossing point represents a conservative, but robust, estimate of when attentional state starts to change. The VSI trajectories actually start changing well before the crossing point, so we looked for the earliest detectable changes in slope for each trajectory (see STAR Methods). At the population level, attentional modulation begins to turn off 85 ms [−129, −83 ms] before the saccade on AU trials (Figure 4b, red arrow) and begins to turn on 161 ms [−207, −108 ms] before the saccade on UA trials (Figure 4b, purple arrow).

We next determined crossing times for the 49 neurons with VSI trajectories crossing exactly once on the interval ±300 ms relative to saccade onset (see Figure 4e). On average, attentional updating occurred 33.4 ± 11.7 ms, significantly before the saccade (t₍₄₈₎=−2.85; p=0.0032, unpaired one-tailed t-test). As noted above, this reflects a conservative estimate for the time of the attentional hand-off; however, VSI trajectories from individual neurons were too variable to apply the slope analysis used to characterize the population response.

The hand-off reflects pre-saccadic state changes that alter responses evoked by stimuli that appear well before saccade onset. However, because V4 response latencies can exceed 100 ms (Schmolesky et al., 1998), responses evoked by these pre-saccadic stimuli can occur after the saccade. Figures 4a and b indicate the response to stimuli appearing at different times relative to the saccade. The same data are replotted in Figures 4c and d to indicate the time at which the actual attentionally-modulated stimulus-evoked responses occur. When plotted this way, the AU and UA trajectories cross 90 ms [78, 105 ms] after saccade onset, and AU and UA shoulder points are at 36 ms [25, 47 ms] after and 20ms [−87, 89 ms] before saccade onset, respectively. This means that the attentional hand-off affected action potentials occurring after the saccade, but those action potentials were evoked by stimuli that appeared before the saccade, consistent with the idea that the hand-off is, in fact, predictive.

Spatial Remapping

Next, we looked for evidence of classical RF remapping, i.e., changes in the retinotopic position of the RF, around the time of the saccade (Neupane et al., 2016, Tolias et al., 2001). We first focused on the subset of neurons (49/174) exhibiting a clear attentional hand-off (exactly one VSI trajectory crossing point within 300 ms of the saccade). To facilitate pooling of data across recording sites and trial types, we normalized mapping probe positions by reflecting probe stimuli around the RF center on leftward saccade trials and then fit responses using the simplified Poisson model (Eq. 5) in a sliding window (150 ms window size, 50ms steps). In a preliminary analysis we found similar patterns of remapping in AU, UA, and UU trials (data not shown), so we fit the combined data from all three trial types. RF profiles, generated as described above, were normalized by RF size and then averaged (n=49) to generate a population perisaccadic spatiotemporal RF for cells that show a clear attentional hand-off (Figure 5a). The trajectory of the RF center of mass over time reveals a small but significant shift in the pro-saccade direction in the window centered at 25ms after saccade onset. The shift is small relative to RF size (0.064 RF σ, t₍₄₈₎=3.21; p=0.002, one-sample two-tailed t-tests for individual time windows with Bonferroni correction, α=0.0033) and the timing is such that this shift overlaps the saccade where measurement errors due to retinal motion would most likely appear as shifts. We observed little evidence of the large amplitude shifts that would be expected from remapping to either the future field location or the saccade endpoint – RFs remained close to their retinotopic positions at all times. We also performed the same analysis on those cells that did not show a clear attentional hand-off (n=125; Figure 5b) and found the same pattern: a small but significant shift in RF center near the time of the saccade (window centered at −75ms relative to saccade onset: 0.0575 RF σ, t₍₁₂₄₎=3.96, p=0.0001; centered at −25ms: 0.0671 RF σ, t₍₁₂₄₎=4.27, p=0.00004; centered at 25ms: 0.0576 RF σ, t₍₁₂₄₎=3.99, p=0.0001, one-sample two-tailed t-tests for individual time windows with Bonferroni correction, α=0.0033). Finally, we directly compared the trajectories of the RF center position for the hand-off and non-hand-off cells (Figure 5c) and found no significant differences in RF position between the two neuronal populations (two-sample two-tailed t-tests for individual time windows with Bonferroni correction, α=0.0033).

Figure 5.

Relationship between spatial remapping to presaccadic attentional hand-off. (a) Population summary of normalized RF profile before, during and after saccades for the 49 neurons that exhibited attentional modulation and showed exactly one attentional crossing point. Each row represents a horizontal slice through the V4 population RF at a different time relative to saccade onset (t=0). Black trace indicates the average center of mass of population RF at each time point; * indicates center of mass differs significantly from zero, p<0.05 with Bonferroni correction. (b) Identical to (a), except represents the 125 neurons that did not exhibit a clear attentional handoff. (c) The mean population receptive field center of mass (± SEM) is plotted as a function of time relative to saccade onset for neurons showing attentional updating (red) and neurons not showing attentional updating (black). No time points showed significantly different RF centers of mass between these two populations, p<0.05 with Bonferroni correction. (d) The time of the attentional hand-off is plotted as a function of receptive field remapping latency for the 31 cells with at least one time point exhibiting significant spatial remapping. Remapping latency and attentional hand-off times were not significantly correlated (r=−.0077; p=0.9673). (e) Scatterplots illustrate the relationship between the time of the attentional hand-off and the extent of presaccadic RF remapping observed across the time window 300–150ms before saccade onset and the time window 150-0ms before saccade onset. Normalization, plotting symbols, and arrows as in (f). 11/49 cells show individually significant RF shifts although the population as a whole did not show a significant shift in the pro-saccade direction (scaled by RF size: 0.0125 ± 0.012, t₍₄₈₎ = 1.03, p=0.3093, unpaired two-tailed t-test; scaled by saccade length: 0.0051 ± 0.0048, t₍₄₈₎ = 1.07, p=0.2901, unpaired t-test). (d) Scatterplots illustrate the relationship between the time of the attentional hand-off and the extent of remapping observed in the 150 ms window just before and just after the attentional handoff for each of the 49 neurons used to compute the population surface shown in (a). Upper plots are normalized by RF size, lower plots by saccade length. Filled circles/bars indicate individual cells showing significant RF remapping. Right histograms show the distribution of RF remapping magnitudes across the same population of cells. Arrows indicate the mean of each distribution. 16/49 cells show individually significant RF remapping and the population shows a small, but significant RF shift in the pro-saccade direction (scaled by RF σ: 0.039 ± 0.014, t₍₄₈₎ = 2.68, p=0.0102; scaled by saccade length: 0.012 ± 0.005, t₍₄₈₎ = 2.20, p=0.0327, unpaired two-tailed t-test).

We also compared the timing of the attentional hand-off to the timing of the observed RF remapping effects in our population. We determined the remapping latency for each neuron showing an attentional hand-off by comparing RF centers from RFs computed as described above from pairs of adjacent sliding windows (2×50 ms, 25 ms steps). We defined remapping latency as the earliest time point with a significant change in RF position between the two windows based on a randomization test (p<0.05, see STAR Methods). We detected significant shifts in 63.3% (31/49) of the neurons studied for at least one time point. However, we found no correlation between the time of the attentional hand-off and remapping latencies (r=−0.0077; p=0.9673, n=31; see Figure 5d), suggesting that the attentional hand-off is mechanistically distinct from spatial RF remapping.

We also applied a paired-window comparison of RF center location at several hand-selected latencies, based on remapping latencies reported in the literature (Neupane et al., 2016, Tolias et al., 2001, Zirnsak et al., 2014), as well as straddling the exact time of the attentional hand-off. Here we used larger paired windows (150 ms) to increase our statistical power. At the reported point of maximal spatial remapping (150 ms before saccade onset) 22.4% of neurons (11/49) remapped, though at the population level, this shift was not significant (scaled by RF size: 0.0125 ± 0.012, t₍₄₈₎ = 1.03, p=0.3093, unpaired two-tailed t-test; scaled by saccade length: 0.0051 ± 0.0048, t₍₄₈₎ = 1.07, p=0.2901, unpaired t-test), as shown in Figure 5e. Remapping at the time of the attentional hand-off was significant for 32.7% of neurons (16/49) and the population as a whole showed a small, but significant shift in the pro-saccade direction (scaled by RF size: 0.039 ± 0.014, t₍₄₈₎ = 2.68, p=0.0102; scaled by saccade length: 0.012 ± 0.005, t₍₄₈₎ = 2.20, p=0.0327, unpaired two-tailed t-test), as shown in Figure 5f. Although some of the observed shifts were statistically significant, they were all substantially smaller than published V4 remapping effects (0.21° on average vs. 10–15° at similar eccentricities; Neupane et al., 2016, Tolias et al., 2001). Taken together, these data indicate remapping is unlikely to be the substrate of the attentional hand-off, since the timing and extent of remapping is the same in hand-off and non-hand-off cells, the magnitude of remapping is not sufficient to explain the attentional hand-off, and the temporal properties of the hand-off are uncorrelated with those of remapping.

Discussion

We used a novel spatiotopic attention task to characterize the temporal dynamics of perisaccadic attentional updating in V4 neurons. The behavioral results from this study show that monkeys, like humans (Golomb et al., 2008), can in fact target and sustain attention in a spatiotopic reference frame. This confirms that monkeys can robustly attend to a fixed environmental location even when saccades change the specific neurons representing that location in retinotopically organized visual areas.

At the neuronal level, we found attentional modulation of responses to both behaviorally relevant targets and behaviorally irrelevant mapping probe stimuli. Responses to probe stimuli revealed significant changes in attentional topography in V4 around the time of saccades. These changes affect the visual responses to stimuli that appear well before the saccade and reflect a hand-off of attentional state between neurons whose RFs currently occupy the attentional focus and neurons whose RFs will occupy the focus after the saccade has been executed. This attentional hand-off could help to stabilize attentional representations in the brain during visually guided behavior by ensuring that the spatial pattern of attentional facilitation in retinotopic visual cortex is behaviorally appropriate as soon as the eyes stop moving. The observed neuronal hand-off is consistent with predictive attentional benefits first described by Rolfs and colleagues (2011) where visual sensitivity develops at nominally unattended visual field locations that will become the target of attention as a direct consequence of saccade execution.

We observed attentional modulation of probe responses at substantially more recording sites than modulation of target responses (75% vs 45%). This is likely due in part to the fact that each of the ~10 probe stimuli appeared multiple times per trial, while the target appeared only once, giving analysis of probe responses more statistical power to detect modulation. Our results demonstrate that evoked responses to task-irrelevant probe stimuli can be effectively used to continuously assess attentional state in single neurons. Because it is difficult to assess attentional dynamics based on responses to single stimuli, this approach is particularly useful in conjunction with behavioral paradigms where only one target (or distractor) appears on each trial, as is the case here.

Although a smaller number of neurons exhibited attentional modulation of target responses, target modulations were consistently larger than probe modulations. There are several possible explanations for this. First, target stimuli were defined by a combination of location and appearance (texture), so it is likely target modulation reflects a combination of spatial and feature-based attention, while the modulation of responses to probes, which matched the location, but not the appearance, of the remembered target, reflects only spatial attention. It is also possible that target responses in V4 reflect additional non-attentional, non-visual high level signals like expected value and reward expectation (Baruni et al., 2015, Mazer, 2011, Maunsell, 2004).

Although this is not the first behavioral demonstration that monkeys can perform a spatiotopic task (Rawley and Constantinidis, 2010), it is the first to isolate and explore the temporal dynamics of perisaccadic attentional updating and the first to describe a presaccadic hand-off of attentional state consistent with an active transfer of attentional resources between neurons inside the attentional locus before and after saccades. This predictive transfer ensures that V4 has an appropriate attentional posture once saccade execution is complete. Although we observed RF remapping in a subset of V4 neurons, remapping latencies were not correlated with the attentional hand-off time, and the small amplitude of the remapping effects, typically only a fraction of the RF diameter, were too small to account for the attentional hand-off. Therefore, we must conclude that the attentional hand-off is not mediated by classical spatial RF remapping (Cavanagh et al., 2010) and that the hand-off perhaps represents a novel form of remapping, one that remaps attentional gain without altering spatial selectivity.

To date, neurophysiological studies of remapping have focused primarily on RF changes in the absence of attention. It is possible that some of the neural circuits involved in spatial remapping could also be participate in the attentional hand-off, however, the remapping circuits have not yet been studied sufficiently using attention-demanding tasks to determine the degree of overlap. Regardless, spatial remapping in V4 appears complex. For example, the term “remapping” has generally been applied to any perisaccadic change in the RF’s spatial properties (e.g., position or shape), but there is some controversy about the direction of remapping in V4. Several groups reported remapping shifts the RF towards the saccade endpoint in V4 (Tolias et al., 2001, Zirnsak et al., 2014), while others have described shifts towards the post-saccadic or future field location (Duhamel et al., 1992, Walker et al., 1995, Umeno and Goldberg, 2001, Nakamura and Colby, 2002). Recently, Neupane and colleagues (2016) reported that both types of remapping can occur in the same V4 neuron at different response latencies. The latencies and modest amplitude of the remapping effects observed in this study are more consistent with remapping towards saccade endpoints as initially described by Tolias and colleagues (2001) than the LIP-style remapping parallel to the saccade vector described by Neupane et al. (2016). However, it is possible that differences in remapping between this study and the previous studies are in part due to differences in the visual stimuli. This study used a dense array of low contrast mapping probes to measure RF position, so multiple probes were often present simultaneously in the RF, while most prior studies have relied on responses to sparse, high contrast probes presented sequentially or once per trial. Churan and colleagues (2011) found spatial remapping in the superior colliculus to be stimulus dependent – remapping effects measured with sparse stimuli vanished when probed with dense stimuli. It could be that remapping in V4 is also sensitive to stimulus density. It is important to note that the use of a dense stimulus configuration in this study was intended to emulate the conditions of natural vision, where multiple stimuli are often concurrently present in the visual field and attention is required to prioritize processing. So, our finding of attentional updating in the absence of spatial remapping indicates that remapping is likely not required to maintain a stable attentional topography during natural vision.

While only a limited number of physiological remapping studies have considered or even controlled attentional state, there are indications that attention remaps along with spatial selectivity in both parietal and frontal cortex (Mirpour and Bisley, 2012, Yao et al., 2016, Joiner et al., 2011). However, it is not clear that simultaneous remapping of attention and spatial selectivity is required to obtain the predictive attentional shifts observed psychophysically by Rolfs and colleagues (2011). In fact, the attentional hand-off alone is sufficient to explain the behavioral findings. The timing and extent of the hand-off are similar to that observed in the Rolfs et al (2011) study, and this renders functional significance of remapping attentional state along with the spatial selectivity of the RF uncertain.

The results of this study, then, are inconsistent with theories that posit either the primary function of spatial remapping is to stabilize attentional topography or that spatial remapping is the proximal cause of attentional updating (Cavanagh et al., 2010). Our findings argue for a dissociation between the transfer of attentional state between neuronal populations and spatial remapping at the single neuron level in V4 and support the idea that our ability to sustain a spatiotopic locus of attention is mediated by a predictive hand-off of modulation state in V4, and perhaps other retinotopically organized visual areas.

STAR Methods

Contact for Reagent and Resource Sharing

Dr. James Mazer (james.mazer@montana.edu) is the Lead Contact for resource sharing. All resources will be shared on an unrestricted basis; resource requests should be directed to the lead author.

Experimental Model and Subject Details

Two male adult rhesus monkeys (B and M, 8–16 kg, 7–14 years old) participated in these studies. All experimental procedures were in accordance with the NIH Guide for the Care and Use of Laboratory Animals and approved by the Institutional Animal Care and Use Committee at Yale University. Monkey M participated in neurophysiological experiments before the onset of these experiments and had been previously trained to perform a feature-based attention task in addition to the basic fixation task described below. Monkey B was experimentally naive at the start of these experiments.

Method Details

Neurophysiological Recording

In two separate sterile surgical procedures performed under isoflurane anesthesia animals were implanted with titanium headposts for head stabilization and recording chambers to allow electrode access through craniotomies overlying area V4 (Mazer et al., 2003, Touryan & Mazer, 2015). Headposts and recording chambers were anchored to the bone using cortical bone screws and acrylic cement. Access was provided by a 5 or 12mm diameter craniotomy performed following training. V4 was targeted stereotaxically using anatomical MRI scans and confirmed based on RF properties of recorded neurons (i.e., response latency, relationship between visual field eccentricity and receptive field size, retinotopic progression within the craniotomy, and stimulus selectivity; see Figure S2).

Visual stimuli, behavior and data collection were controlled by custom software (https://github.com/mazerj/pype3). Stimuli were presented on a gamma corrected LCD display (Dell P2210 LCD) with a 60/75Hz frame rate at a viewing distance of 41.5/40.0 cm (monkey B/M, convention followed for subsequently reported parameters). Eye movements were recorded digitally at 1000 Hz using an infrared eye tracker (EyeLink 1000, SR Research, Missisauga, Canada), and neuronal activity was recorded with high impedance (nominally 1 MΩ at 1 kHZ) glass coated tungsten microelectrodes (250–350 μm diameter, Alpha Omega Engineering, Alpharetta, GA). Electrodes were inserted transdurally each day using a custom built motorized micromanipulator. Recorded signals were digitized and filtered (0.5–10 kHz) with a DSP-based system (RZ5, Tucker-Davis Technologies, Alachua, FL). Action potentials were detected and timestamped (1 ms precision) on-line using a digital time-amplitude window. In some cases, additional spike sorting was performed off-line using custom template-matching software to further refine neuronal isolation or to isolate additional neurons.

Fixation Task

Animals were initially trained to perform a basic fixation task. The onset of fixation trials was signaled by a high contrast white fixation target (0.1–0.3° square) appearing on a uniform gray background (26 cd/m²). Monkeys grasped a capacitive touch bar and fixated the target. During fixation stimuli were presented in the visual periphery. After a random (2–3 s) interval, the contrast of the fixation target was reduced by 30% and monkeys had release the bar within 500ms to receive a liquid reward. Errors (fixation breaks, misses and false alarms) were signaled by visual and auditory error signals, followed by a brief timeout period.

Isolated neurons were initially characterized using a range of experimenter-controlled stimuli, including color- and luminance-defined oriented bars and sinusoidal gratings presented during periods of fixation. Initial RF estimates were quantitatively refined by reverse correlation of responses to sparse, low contrast optimally oriented black and white probe stimuli flashed at a 1–50 Hz on a regular grid (Connor et al., 1997, Jones & Palmer, 1987, Mazer et al., 2002). Probe locations were randomized and presented with a 50% duty cycle. Reponses to at least five complete sets of probes (all positions and both luminance polarities) were used to generate a two-dimensional RF map (Touryan & Mazer, 2015), which was fit with a circularly symmetric Gaussian,

$$\mathrm{y}=\mathrm{A}exp\left[-\left(\frac{{(\mathrm{x}-{\mathrm{x}}_{\mathrm{o}})}^2}{2{\mathrm{\sigma }}^2}+\frac{{(\mathrm{y}-{\mathrm{y}}_{\mathrm{o}})}^2}{2{\mathrm{\sigma }}^2}\right)\right],$$ (6)

to determine the size (σ) and location (x₀, y₀) of each neuron’s RF.

Spatiotopic Attention Task

In the spatiotopic attention task (see Figure 1) animals were required to sustain attention at a cued spatiotopic location while simultaneously executing a guided horizontal saccade. Targets stimuli could appear before, during or after saccades, and animals had to make a manual response when the target appeared at the attended location or withhold the response if the target appeared elsewhere. Trials were initiated when monkeys both acquired a cyan fixation target at FP1 and grasped a touch bar. On instruction trials the attended location was then immediately cued by the appearance of a high contrast black rectangular outline visible for either 600 ms (monkey B) or the entire trial (monkey M). After 3–7 correct instruction trials, the cue was turned off for the rest of the 40–75 trial block, and the task was performed from working memory. For monkey B, additional instruction trials were scattered randomly throughout each block at low (0–20%) frequency to re-cue the attended location. For both monkeys, a reminder spatial cue was also shown during the timeout period following each error. Data from instruction and error trials were not used for the analyses reported here. During attend-in blocks the cue appeared at the location occupied by the RF when the monkey fixated the central fixation target. On these trials, the RF either started inside the focus of attention and moved out or started outside and moved in as a result of the saccade. On attend-out trials attention was directed to the same location reflected across the horizontal meridian, so was always outside the attentional focus. The geometry of the stimulus display was adjusted for each neuron based on the size and retinotopic location of the RF (see Figure 1b).

FP1 was selected at random from three possible positions: left, center and right, with probabilities of 25, 50 and 25% respectively, separated by 1–2.5 σ. FP2 was always adjacent to FP1, and these parameters defined four equiprobable saccade vectors. Two events occurred on each correct trial: (1) the fixation point was displaced horizontally from FP1 to FP2 after 800–1500 ms (exponential distribution, mean=1125 ms), signaling monkeys to execute a short-latency (<700 ms), accurate saccade from FP1 to FP2 and without deviating from the saccade path by more than 2.5/2.9° (Monkey B/M), and (2) a target stimulus appeared at one of six possible locations on the display. If the target appeared at the attended location (go trial), monkeys had to release the bar within 500 ms. If the target appeared at one of the five unattended location, the monkey had to continue to hold the bar. The five possible unattended locations consisted of two locations flanking the attended location and the same flanking locations reflected cross the horizontal meridian, as well as the reflected attended location. The frequency at which targets appeared at each location were based on the following constraints: on 50/53% of trials targets appeared at the attended location, and on the remainder of trials targets were distributed among the five unattended locations such that on 90–95/85% of trials the target appeared at the unattended location corresponding to the reflected attended location. These distributions were designed to prevent animals from simply responding to all central locations, which would have been an effective strategy if all unattended locations were equally likely. Finally, there was a 2% chance that any trial could be a catch trial where the target did not actually appear. Holding the bar at least 500ms past a randomly selected imaginary target onset time was considered a correct response on catch trials. Catch trials were used to ensure animals were not using a timing strategy to perform the task.

Bar releases in response to targets appearing at unattended locations (i.e., false alarms), anticipatory manual or saccadic responses, fixation breaks, inaccurate or late saccades and missed targets were counted as errors and signaled with both visual and auditory displays followed by a 4–12 second timeout during which the attended location was re-cued. Target onset times were uniformly randomly distributed ±300 ms relative to predicted saccade onset, based on a running average estimate of saccadic reaction time, and targets were visible for 200/300ms. A complete list of target locations and saccade vectors was generated and shuffled at the start of each block. Following errors, trial parameters were returned to the queue at a random location, ensuring that animals executed at least one correct trial for every configuration before the block terminated.

A dynamic sequence of mapping probes started 50–150ms after fixation at FP1. Probes were 6–18% contrast bars arranged in two horizontal bands, as indicated in Figure 1. Probes were the same size, shape and orientation as the target stimuli, but uniformly dark or light gray (depending on the cell’s preference). Probes and targets had the same mean luminance. One probe was placed at the center of the RF and the others distributed uniformly such that at least three probes fell inside the RF (0.3–0.5 σ separation between probes). Initially 10–20% of probe positions were turned on at random and assigned a uniformly distributed random lifetime (20–100 ms). At the end of each probe’s life it was turned off and a new probe, selected at random with replacement and with a new lifetime, was turned on.

A progressive reward schedule was used with 1 drop of juice for the 1^st correct response, 2 drops for the 2^nd, etc. up to a maximum of 4 drops. After each error, the reward size was reset to 1 drop.

Due to differences in training procedures used with each monkey there were small differences in the spatial configuration of the stimuli between animals. For animal B, probe stimuli could appear at the cued location, while for animal M, probes never appeared at the cued location (see Figure S1). This meant animal M could have used the absence of probes at the target location as a supplemental attentional cue. We used post hoc behavioral testing to ensure this was not the case. Animal M was tested daily in the original missing probe condition until asymptotic, stable performance levels were obtained. After eight consecutive testing sessions at asymptotic performance, the missing probe was turned on without warning in the middle of a block of trials (i.e., without additional instruction trials). If animal M was using the missing probe as a supplemental attentional cue, we expected to observe a drop in behavioral performance immediately after the switch. However, we observed no significant changes in either accuracy or reaction time after the switch (see Figure S1b–d), confirming both animals consistently used the remembered cue location to perform the task.

Quantification and Statistical Analysis

Saccade Detection

Saccades were detected off-line by low pass filtering the eye position signal (0–20 Hz) and then differentiating to compute eye velocity. Saccade onsets were defined as the first time velocity exceeded 150 deg/s. Saccades taking the eye off screen or <100ms after another saccade (under- or overshoots) were excluded from analysis, as were trials where the monkey made more than one saccade to reach FP2. The time at which the eye entered the fixation window surrounding FP2 was used on-line to update a running estimate of saccadic reaction time, which was in turn used to dynamically adjust the distribution of target onset times.

Spatiotemporal RF estimation and analysis windows

To estimate the spatiotemporal response to low contrast probes, probe locations were binned in space (with bin size adapted to ensure uniform sampling centered on the RF center), probe onsets within bins were found and responses to probe onsets were extracted and binned in time (0–300 ms in 10 ms bins). The responses to probe onsets in each bin were averaged to form a peristimulus time histogram (PSTH) for each probe position bin, and PSTHs were formed into a matrix to generate a spatiotemporal receptive field (STRF), which was then smoothed using a 2D Gaussian (σ_space =0.4 probe bins, σ_time =2 ms). The spatial analysis window was determined by fitting a spatial slice through the STRF peak with a 1D Gaussian (μ ± 1.5σ). In the rare cases where the spatial slice was not well fit by the Gaussian (13/174 cells), width at half height was used to define the spatial analysis window. The temporal window was defined by fitting a temporal slice through the peak with a gamma function and computing the range exceeding 20% of the peak height. The temporal impulse response function was well fit by a gamma function in all 174 recordings.

Attentional modulation of target responses

To characterize attentional modulation of responses to target stimuli we computed average neuronal responses to target onsets for targets appearing within the RF in the attended and unattended conditions. To minimize the effects of eye position, we restricted analysis to central fixation trials where targets appeared either >100 ms before or >100 ms after the saccade. This ensured responses were not contaminated by either saccadic suppression (Duffy & Lombroso, 1968) or gaze angle modulations (Bremmer, 2000). Mean firing rates in the temporal analysis window (see above) were used to calculate the target modulation index (TMI) as described in Results.

Attentional modulation of probe responses

We used GLM to fit a Poisson model that describes how attentional state modulates the response of each neuron (see Eq. 3). The model separately quantifies changes in baseline firing rate and neuronal gain based on responses to mapping probes. We first constructed a regressor matrix, S_t,i, for each cell based on a binary representation of the stimulus sequence in the region of the RF. The regressor matrix was binned in time at the video frame rate (60 or 75 Hz). A_t and (1-A_t) in the model are binary vectors that indicate whether each stimulus frame corresponds to an attend-in configuration (i.e., RF inside the focus of attention) or an attend-out configuration. The response vector, y_t, consisted of the firing rate elicited by the stimulus at the mean bin time t in the temporal analysis window corresponding to time t. The model was fit only with data from correct, uninstructed trials.

To determine whether the probe responses of individual cells were modulated by attention, we randomized the attention labels, A_t, on a trial-by-trial basis to disrupt the relationship between A_t and actual attentional state and calculated the log likelihood fit to the shuffled data 1000 times to obtain a null distribution of the log likelihood (LL) values ins the absence of attention, and then used this distribution to calculate a p value associated with the LL of the fit to the actual data. Cells where the true model’s LL was greater than 95% of the null distribution were considered significantly modulated by attention.

To directly compare target and probe modulations, we used the full model (Eq. 3) to predict responses to isolated probe stimuli within the RF (±1.5σ) in attend-in and -out conditions and then used predictions to compute a probe modulation index (Eq. 4) analogous to the target modulation index (Eq. 1).

Perisaccadic attentional dynamics

To understand how attentional state was dynamically modulated around the time of the saccade uninstructed correct trials were sorted into AU, UA, and UU conditions for separate analysis, as described in Results (see Figure 4), using a sliding 100 ms window moved in 1 ms steps from 300ms before to 300 ms after the saccade. At each step we constructed a regressor matrix S_t,i, as described above and fit to the responses within the temporal analysis window with a simplified Poisson model (Eq. 4). The simplified model did not include an explicit attentional term, since a separate fit was computed for each attentional condition. Mean responses to probe stimuli in the RF (±1.5σ) at each time step were predicted using the model and used to construct AU, UA and UU trial trajectories, which were then smoothed (Gaussian, σ=10ms). To dissociate changes in attentional state from saccadic suppression, we calculated a running visual sensitivity index (VSI, Eq. 5). The time of the attentional hand-off for individual cells was calculated from the crossing point for AU and UA trial VSI curves (Figure 4e).

Population AU, UA, and UU trajectories were computed by normalizing individual trajectories in all three conditions by the maximum predicted firing rate and averaging across neurons (Figure 4b and d). Population VSI trajectories were computed by applying Eq. 5 to the population AU, UA and UU trajectories. Shoulder points were determined by comparing adjacent 25 ms sliding windows from the population AU and UA curves. We started with windows straddling the crossing point and moved them together either backwards and forwards in time in 1 ms increments until we found three consecutive times where the two windows were no longer significantly different (p>0.05) to estimate the beginning and end of the attentional hand-off period, respectively. Firing rates in each window were compared using a two-tailed unpaired t-test. Bootstrapped 95% confidence intervals for the population crossing points and shoulder points were calculated by selecting with replacement from the 174 experimental recordings. Trajectories, as well as VSI crossing and shoulder points, were initially computed aligned to stimulus onset (Figure 4a and b), i.e., responses were plotted at the time corresponding to the onset of the stimulus that evoked them. In Figures 4c and d, the same data are replotted to reflect the time of the neural response, shifting the trajectories for each cell by that cell’s measured response latency prior to computing the population average.

Spatial Remapping

To characterize spatial remapping, we used GLM to fit a reduced model (Eq. 5) in a sliding window to the combined data from all trial types for the 49 neurons that exhibited attentional modulation and showed exactly one attentional crossing point. The stimulus matrix S_t,i was adjusted before fitting by reflecting probe positions on leftward saccade trials around the RF center so the stimuli were normalized with respect to saccade direction. Models were fit in 150 ms windows moved in 50 ms steps from 400 ms before to 400 ms after saccade onset. Model fits were used to generate a series of RF spatial profiles, i.e., 1D horizontal slices through the 2D RF, at each window position, for each cell. To assess whether the population center of mass was significantly different from 0 at any time point, the center of mass was calculated for the population of 49 cells and compared to 0 using a one-sample t-test with Bonferroni correction (adjusted α=0.0033). For the purposes of display, individual RF profiles were normalized to the range 0–1 and then averaged across the population of 49 cells. Each time point in the population average was normalized to 0–1 to compensate for non-specific changes in neuronal sensitivity due to saccadic suppression and emphasize the size and shape of the RF (Figure 5a). We also characterized spatial remapping in the 125 cells that did not show a clear attentional hand-off using an identical analysis (Figure 5b). To compare remapping between the population of cells that showed an attentional hand-off and the population of cells that did not, we used a Bonferroni-corrected t-test to compare the center of mass distributions between these two populations in each time interval (Figure 5c).

To estimate RF remapping latencies for individual neurons we fit the simplified GLM model (Eq. 5) in a pair of flanking sliding windows (2×50ms windows, 25ms steps starting 250 ms before saccade onset) and used each fit to estimate RF center of mass for each window. We calculated a simple shift index, defined as the scalar difference between the two centers. Statistical significance of the index was assessed using a null distribution obtained by randomizing the label (left/right) of each time window (1000 permutations). The remapping latency of cells showing statistically significant remapping was defined as the earliest window position (time between the left and right windows) with a significantly non-zero shift index.

We also examined remapping between pairs of windows selected to either maximize remapping effects in V4 based on published reports (0:−150ms and −150:−300ms relative to saccade onset, Figure 5c) (Neupane et al., 2016, Tolias et al., 2001) or to bracket the attentional hand-off (−150:0 ms and 0:150 ms relative to hand-off, Figure 5d) or. A single shift index was calculated for each pair of time windows and statistical significance (p<0.05) was determined as described above. To facilitate comparison between neurons with different RF sizes we normalized the shift index by either RF size (σ) or saccade length.

Highlights.

• Spatiotopic attention is mediated by attentional shifts to compensate for saccades

• In V4, this “hand-off” of attention between neurons occurs before the saccade

• The hand-off, and therefore spatiotopic attention, does not depend on RF remapping