Unmasking saccadic uncrowding

Abstract

Stimuli that are briefly presented around the time of saccades are often perceived with spatiotemporal distortions. These distortions do not always have deleterious effects on the visibility and identification of a stimulus. Recent studies reported that when a stimulus is the target of an intended saccade, it is released from both masking (De Pisapia, Kaunitz, & Melcher, 2010) and crowding (Harrison, Mattingley, & Remington, 2013). Here, we investigated pre-saccadic changes in single and crowded letter recognition performance in the absence (Experiment 1) and the presence (Experiment 2) of backward masks to determine the extent to which saccadic “uncrowding” and “unmasking” mechanisms are similar. Our results show that pre-saccadic improvements in letter recognition performance are mostly due to the presence of masks and/or stimulus transients which occur after the target is presented. More importantly, we did not find any decrease in crowding strength before impending saccades. A simplified version of a dual-channel neural model, originally proposed to explain masking phenomena, with several saccadic add-on mechanisms, could account for our results in Experiment 1. However, this model falls short in explaining how saccades drastically reduced the effect of backward masking (Experiment 2). The addition of a remapping mechanism that alters the relative spatial positions of stimuli was needed to fully account for the improvements observed when backward masks followed the letter stimuli. Taken together, our results (i) are inconsistent with saccadic uncrowding, (ii) strongly support saccadic unmasking, and (iii) suggest that pre-saccadic letter recognition is modulated by multiple perisaccadic mechanisms with different time courses.

Introduction

Objects that fall on the fovea can be easily recognized whereas they may be difficult to recognize in the periphery due to factors such as the reduced photoreceptor density of the peripheral retina. Recognition is even more difficult when objects are closely surrounded by other objects, a phenomenon called crowding (Bouma, 1970; Chung, Levi, & Legge, 2001). The crowding strength is generally defined as a reduction in recognition performance due to the presence of flanking objects (Levi, 2008; Whitney & Levi, 2011). There are many accounts of crowding, but one account suggests that crowding results from the obligatory integration of features within a spatial window. The extent of this window, the critical distance, scales with eccentricity (Bouma, 1970; Chung et al., 2001). The crowding strength and the critical distance depend on temporal properties of the stimuli (Chung & Patel, 2011; Chung, 2016; Lev, Yehezkel, & Polat, 2014). Many other factors, such as attention (Freeman & Pelli, 2007; Grubb et al., 2013; Yeshurun & Rashal, 2010) and perceptual grouping (Manassi, Sayim, & Herzog, 2012, 2013) also affect crowding. Crowding is a major bottleneck for object recognition and a fundamental component of conscious spatial vision (Levi, 2008; Pelli & Tillman, 2008; Whitney & Levi, 2011).

While crowding impairs the recognition of an object, it leaves the detectability of its features unaffected (e.g., Levi, Hariharan, & Klein, 2002). Masking refers to the reduction in visibility of a (target) stimulus when it is presented in close spatiotemporal proximity to another (mask) stimulus (Breitmeyer & Ogmen, 2006). Masking and crowding are affected similarly by certain spatiotemporal properties of the stimuli such as stimulus onset asynchrony and duration (Chung et al., 2001; Lev & Polat, 2015); however, they also have distinct characteristics (Chung et al., 2001; Pelli, Palomares, & Majaj, 2004).

In this paper, we define flankers (i.e., stimuli crowding a target stimulus) as stimuli of the same object category (e.g., letters) as the target, although we note that flankers that belong to different object categories can also induce crowding (Chanceaux, Mathot, & Grainger, 2014; He & Tjan, 2004). Masks are defined as stimuli that do not have structures or features as those of the target (e.g., noise masks). Crowding can occur in the absence of masking, and vice versa. When flankers and masks are presented in combination, the net effect may not be equal to the sum of the individual effects. Certain types of masks reduce flankers’ visibility, which in turn reduces crowding (Chakravarthi & Cavanagh, 2009). When a target is crowded and weakly masked, the combined deleterious effect is larger than the sum of their individual effects, known as “super-crowding” (Vickery, Shim, Chakravarthi, Jiang, & Luedeman, 2009). Crowding the “crowders” (i.e., flankers) or masking the masks can restore the recognizability or the visibility of a target (Manassi et al., 2013; Ogmen, Breitmeyer, Todd, & Mardon, 2006). The evidence reviewed here suggests that, although they can be dissociated in some settings, masking and crowding might share common neural mechanisms under certain conditions. Moreover, most stimuli used to study crowding and masking activate both mechanisms, therefore, it is difficult to disentangle their individual contributions to the final percept.

A powerful paradigm to study these processes is the pre-saccadic “uncrowding” and “unmasking”, in which the presentation of the stimuli (i.e., target-mask or target-flanker displays) prior to an impending saccade reduces the impairments due to crowding and masking (De Pisapia et al., 2010; Harrison, Mattingley, et al., 2013). Saccades constitute a fundamental aspect of normal vision; hence, it is essential to understand how visual processes operate under pre-saccadic conditions. As a consequence of saccades, retinal images are frequently displaced, yet we perceive a stable and coherent world. How does the visual system achieve perceptual stability? Among many proposals, much attention is received by those based on updating a highly detailed retinotopic map via shifts of neuronal receptive fields (RF) or attentional resources (Burr, Ross, Binda, & Morrone, 2010; Cavanagh, Hunt, Afraz, & Rolfs, 2010; Melcher & Colby, 2008; Wurtz, 2008). Neurons which shift their RFs in parallel to the direction of an impending saccade toward the retinal locations that they will occupy (“future field”) after a saccade, commonly referred to as “remapping”, have been found in several cortical and subcortical structures (Duhamel, Colby, & Goldberg, 1992; Nakamura & Colby, 2002; Sommer & Wurtz, 2006; Umeno & Goldberg, 1997; Walker, FitzGibbon, & Goldberg, 1995). However, recent studies indicate that these RF shifts in FEF and V4 occur toward the saccade target rather than the future field (Tolias et al., 2001; Zirnsak, Steinmetz, Noudoost, Xu, & Moore, 2014), leaving open the question of whether there is only one type of remapping or whether different areas show different patterns of remapping. In fact, a more recent study revealed both types of perisaccadic RF changes in monkey V4 cells, with remapping parallel to the saccade vector occurring earlier than convergence of RFs (Neupane, Guitton, & Pack, 2016).

Pre-saccadic modulations in crowding and masking have been associated with remapping (Harrison, Retell, Remington, & Mattingley, 2013; Hunt & Cavanagh, 2011). Admittedly, behavioral studies without explicit measurement of RFs cannot speak for or against this association, however, they provide a new avenue for inference about the underlying mechanisms and may potentially inform the theories based on neurophysiological data. Therefore, pre-saccadic uncrowding/unmasking paradigm is also important since it informs us whether or not, and to what extent RF modulations play a role in these phenomena. Here, we investigated how crowding, masking, saccade-related processes, including shifting RFs, contribute to pre-saccadic object recognition. In order to determine whether or not, and how saccadic eye movements modulate object recognition at the saccade target location, we carried out two experiments where observers reported the identity of a target letter presented in the peripheral retina. In separate and randomly interleaved blocks of trials, observers performed the task either during fixation (with the letters presented in the peripheral retina), or after a saccadic eye movement following the offset of a cue located at the location of the target letter. To quantify the changes in crowding strength due to impending saccades, the target letter was presented either alone or with two horizontally flanking letters. In Experiment 1, we specifically tested the hypotheses that (i) saccade targets are released from crowding, and (ii) perisaccadic mechanisms interact with the temporal order of the stimuli. In Experiment 2, we specifically tested the hypothesis that impending saccades reduce masking. By comparing the results from both experiments, we also tested the hypothesis that remapping (in the form of perceptual displacements) affects pre-saccadic masking but not crowding.

Saccades have been shown to result in increased detection thresholds (known as saccadic suppression) and enhanced discrimination performance (e.g., Bridgeman, Hendry, & Stark, 1975; Deubel & Schneider, 1996). In order to understand how mechanisms with such opposite effects might play a role in pre-saccadic object recognition, we used a simplified version of a dual-channel neural model of visual masking to account for the data in both experiments. This model could account for letter recognition performance during fixation, suggesting that masking and crowding might share common low-level mechanisms. Finally, by adding several independent saccadic mechanisms to this model, we sought to tease apart different components of pre-saccadic modulations in letter recognition.

Methods

Overview

In Experiment 1, we aimed to determine whether or not, and how pre-saccadic letter recognition is affected by the temporal order of presentation of a target and its flankers. The target letter was presented either alone (unflanked) or accompanied by two horizontally flanking letters with a varying flanker-target onset asynchrony (FTOA). A negative FTOA means that the flanker letters were presented before the target letter, a positive FTOA means that the target letter was presented first, and zero FTOA corresponds to the simultaneous presentation of the target and flanker letters.

In Experiment 2, we investigated whether or not, and how letter judgments are affected by saccades in the presence of backward noise-masks. Within a block of trials, a target letter was presented either alone or with two horizontally flanking letters. Each letter was always followed by spatially overlapping backward noise mask. By comparing the results between the flanked and unflanked conditions, we were able to determine whether crowding and masking interact. Moreover, by comparing the results of Experiment 2 and the zero FTOA condition in Experiment 1, we were able to determine whether or not saccadic uncrowding is just a manifestation of saccadic unmasking. In Experiment 2, the target and flanker letters were always presented simultaneously (i.e., with zero FTOA) but all letters were followed by noise-masks at their respective locations. In both experiments, timing of events was adjusted for each observer and block of trials such that in the majority of trials, target presentation was completed before saccade onset. This manipulation allowed us to investigate the time course of letter recognition performance before saccades. Note that our fixation and saccade conditions were identical in terms of retinal locations of the target and flanker letters.

Participants

Six observers (two males, four females) participated in the study. One of the observers was the first author (MNA), and the remaining observers were unaware of the purpose of the study. Four observers participated in Experiment 1, and three observers participated in Experiment 2. All observers had normal or corrected-to-normal vision (20/20 or better in each eye) and had prior experience in other psychophysical studies. Each observer gave written informed consent after the procedures of the experiment were explained. The experimental protocol was approved by the IRB at the University of California, Berkeley. All procedures were in accordance with the Declaration of Helsinki.

Stimuli and Procedures

Visual stimuli were presented on a 21 inch Sony CRT (GDM-F500R) monitor at a resolution of 1240 × 1120, and a frame rate of 85 Hz. Participants sat at a distance of 47 cm from the display with their heads stabilized with a chin rest. Eye movements were recorded at 500 Hz with an EyeLink II (SR Research) infrared eye-tracker after a 9-point calibration at the beginning of each block of trials. Stimulus presentation and eye-movement recording were programmed in MATLAB (v8.3.0.532) (MathWorks, MA) by using the Psychophysics Toolbox 3 (Brainard, 1997; Pelli, 1997) and its Eyelink Toolbox extensions (Cornelissen, Peters, & Palmer, 2002). Observers’ responses were obtained via a keyboard.

Figure 1A illustrates the stimuli and procedures in Experiment 1. Each trial began with a black (2.5 cd/m²) fixation cross (0.6 deg) at the center of the display on a uniform gray (55.6 cd/m²) background. Observers pressed a key on the keyboard to confirm fixation and initiate a drift correction. During any trial, if the drift error exceeded 2 deg, the trial was aborted and the calibration procedure was repeated. Following drift correction, a bright (150 cd/m²) disk (with a diameter of 0.16 deg) was presented at 8 deg eccentricity in the right visual-field for a variable duration of 400-1200 ms (randomly drawn from a uniform distribution) as a cue to indicate the target location. After the cue offset, only the fixation cross was shown for another variable delay (0-250 ms), which was followed by the presentation of the target and flanker letters. The luminance of letters was also 150 cd/m². In roughly 25% of the trials, the target letter was presented alone (the unflanked condition), and in the remaining trials, flanker letters were presented with the target letter (the flanked condition) with three different temporal separations (FTOA) between them (−60, 0, and 60 ms, see Figure 1B). The duration of both the target and flanker letters were ~35 ms (3 frames). The target and flanker letters were selected randomly from different sets of uppercase letters rendered in Courier New font type. The target letter could be L, H, T, or N, and the flanking letters could be selected without replacement from the set of K, M, R, E, F, Z, and I. Having different set of letters for target and flankers allowed us to avoid mislocation errors, where one of the flankers is reported as the target.

Figure 1.

A. Visual stimuli and procedures used in Experiment 1. Each trial started with a drift correction while observers maintained fixation at the center of the display. A trial commenced with the presentation of a cue presented at 8 deg on the right of the fixation cross. After a random delay, the placeholder was turned off, which was followed by the presentation of the target and flanker letter sequence. Once the stimulus presentation was over, observers reported the target letter via a keyboard. In fixation blocks, observers had to maintain fixation throughout the trials whereas in saccade blocks, the offset of the placeholder served as a go signal for observers to make a saccadic eye movement towards this location. B. The time course of events in Experiment 1. The stimulus timings were adjusted for each observer to maximize the number of trials where the target letter presentation was completed before saccade onset. There were four stimulus configurations: (i) unflanked, (ii) flankers preceded the target by 60 ms, (iii) flankers and target appeared simultaneously, and (iv) flankers followed the target by 60 ms. C. Stimuli and procedures used in Experiment 2. The target and flankers were always presented together and followed by a set of noise masks. D. The time course of events in Experiment 2.

Crowding strength is conventionally defined as the performance difference between the unflanked and flanked conditions, and it has been shown to decrease with large asynchronous presentation of target and flankers (Chung & Patel, 2011; Chung, 2016). Therefore, letter sizes and spacing should be selected such that (i) substantial crowding is obtained in all FTOA values (i.e., performance in flanked condition is significantly lower than that in unflanked case) and (ii) performances in all flanked conditions are above chance level (i.e., 25%), to be able to investigate any potential pre-saccadic changes in crowding. In pilot experiments, letter size of 1.0 deg and target-flanker spacing of 1.3 deg were found to satisfy these requirements for all observers, and were thus used in Experiment 1.

In separate blocks of trials, observers were asked to make a saccade to the location of the cue immediately after its offset, or to maintain fixation throughout each trial. Runs of saccade and fixation blocks were randomly interleaved, however, many more saccade blocks were run in order to get sufficient amount of trials for different time-bins before saccade onset. In order to maximize the number of trials within 200 ms before saccade onset, we used a method similar to that of Hunt and Cavanagh (2011) and Harrison et al. (2013). We computed the median saccadic latency (the time between cue offset and saccade onset) after each saccade block and subtracted 50, 100, or 150 ms from this value to adjust the variable delay between cue offset and target-flanker sequence in the next block. In the first block, variable delay was randomly selected from 50-150 ms. In fixation blocks, we used delays from the immediate previous saccade block to ensure that timings in saccade and fixation blocks were closely matched. There was a delay of 750 ms after stimulus presentation and before observers could indicate the target letter (L, T, N, or H; four-alternative forced-choice task). There were 50 or 100 trials in each block depending on the observer, but the total number of trials each observer completed was similar (3000-3500). All observers practiced the task for 2-3 blocks of trials before the actual data collection. Practice trials were excluded from further analyses.

In Experiment 2, the stimuli and procedures were identical to Experiment 1 with the following exceptions (Figure 1C-1D). In order to avoid the floor effect from the combined effects of the masks and the flankers, we increased the letter size and spacing to 1.5 deg and 3.0 deg, respectively (based on pilot data). As soon as the trigrams disappeared from the screen, square-shaped static noise patches (1.5 × 1.5 deg²) were presented at the letter locations. Each pixel value for patches was selected randomly from a normal distribution (mean = 128, sd = 50). The duration of the masks was 100 ms. Each participant ran 250 trials (in five blocks) in the fixation condition, and 500 trials (in ten blocks) in the saccade condition.

Statistical Analyses

All analyses were performed offline. Saccades were detected using a velocity criterion of 35 deg/sec and an acceleration criterion of 9500 deg/sec². Trials were included for further analyses if (i) gaze during fixation (or before the saccade cue disappeared in saccade conditions) did not drift outside a 2×2 deg² virtual square; (ii) no saccade was made within 200 ms of target presentation in fixation conditions; (iii) a saccade was made in saccade conditions; (iv) the target letter was presented later than 175 ms, or earlier than 35 ms before saccade onset; (v) observers did not blink during stimulus presentation; (vi) the saccade landing-point was within a 4 deg window around the cue location; (vii) saccade duration was no longer than 100 ms. Roughly 70% of trials satisfied all of these criteria in both experiments. In Experiment 1, 2683 trials (71%) from MNA, 2735 trials (65%) from ML, 2141 trials (70%) from LL, 2711 trials (70%) from SA, and in total, 10270 trials (69%) were included for further analyses. In Experiment 2, 634 trials (85%) from MNA, 529 trials (70%) from DW, 481 trials (64%) from TF, and in total, 1644 trials (73%) were included for further analyses.

To quantify changes in letter-recognition performance across TSO, letter responses in saccade trials were first sorted in ascending order of TSO. A moving Gaussian window with a standard deviation of 25 ms and width of 50 ms centered at increments of 10 ms TSOs from 150 ms to 30 ms before saccade onset was used to calculate percent correct over time. At each increment, trials falling within this Gaussian window were bootstrapped to estimate the mean and the standard error for each observer separately as well as for the entire data combined. Bootstrapping was performed by randomly sampling data from a time bin (with replacement). This procedure was repeated 1000 times to obtain a distribution of bootstrapped means, standard errors, and 95% confidence intervals. Confidence intervals were used to pinpoint significant changes between fixation and saccade conditions. Note that bootstrapping analysis outperforms conventional t-tests in terms of statistical power, and does not require data to conform to several assumptions such as normality and homoscedasticity (Ahad, Abdullah, & Lai, 2012; Efron & Robert, 1994).

Results

Psychophysics

Figure 2A and 2C show the number of trials used to compute recognition performance for each observer in Experiment 1 and Experiment 2, respectively. Each bar represents the number of trials within a moving kernel (with a width of 50 ms) centered at a specific target-to-saccade onset (TSO) with 10 ms increments. Note that trials where the offset of the target letter occurred after saccade onset (i.e., TSO > −35 ms) were removed from further analyses. The average gaze position at saccade onsets and the average saccade landing positions in Experiment 1 and Experiment 2 are shown in Figure 2B and 2D, respectively.

Figure 2.

The distribution of (valid) trials that were used for further analyses, as a function of target-to-saccade onset (TSO) in Experiment 1 (A), and in Experiment 2 (C). Note that each bar represents the number of trials within a moving kernel with a width of 50 ms centered at a certain TSO. The average saccade starting and landing positions in two dimensional space are shown for Experiment 1 (B) and Experiment 2 (D). In all plots, different colors represent different observers. Error bars represent ± 1 SD.

Experiment 1

Letter recognition performance as a function of TSO, obtained for the fixation and saccade conditions in Experiment 1, is plotted for the four observers in Figure 3. Despite some individual differences in the data, all observers demonstrated the classical crowding effect, viz., letter performance was best when the target letter was presented alone (black: unflanked conditions) and worst when the target and flankers were presented simultaneously (e.g., orange: FTOA = 0). To illustrate the other effects more clearly, in Figure 4, we present the letter recognition performance that was averaged across the four observers. Because percent correct is not a linear metric of performance, for instance, the amount of crowding needed to reduce performance from 90% to 70% is different from that needed for the same amount of reduction from 50%; in order to quantify the crowding strength properly, we converted letter recognition performances to z scores. The difference between z scores in the unflanked and flanked conditions was used to represent the crowding strength (bottom panels in Figure 4).

Figure 3.

Letter recognition performance (percent correct) is plotted as a function of time-to-saccade onset (TSO) for the four observers, for results obtained in Experiment 1. Horizontal solid lines represent performance in the fixation conditions, markers represent the saccade conditions. The shaded regions and all error bars represent ± SEM obtained from bootstrapping.

Figure 4.

Results of Experiment 1 pooled across observers. (A) The fixation condition. (B) The saccade condition. In the top row of (A), letter recognition performance is shown as a function of flanker-target onset asynchrony (FTOA). “Unf” in the upper left panel represents the unflanked condition. In the bottom row of (A), crowding strength, defined as the difference between z-scores corresponding to performances in unflanked and flanked conditions, is plotted as a function of FTOA. Markers represent empirical data and the solid lines represent the model fit to the group data (see the Modeling section). Horizontal dotted-lines in the top row represent the chance level. All error bars represent ± 1 SEM obtained from bootstrapping the data combined from all observers. To facilitate comparisons, significant differences (when 95% confidence intervals do not overlap) between performances in the saccade and fixation conditions are indicated by filled symbols in the upper right panel.

Consider first the fixation condition (Figure 4A). When the target letter was presented alone (black inverted triangles: unflanked), observers recognized it almost perfectly. However, in the presence of flanking letters, recognition performance dropped substantially. This is the classic crowding effect. Crowding was strongest when the target and flankers were presented simultaneously (i.e., FTOA = 0), which was the condition that was the most conventionally used to study spatial crowding in the literature (e.g. Bouma, 1970; Chung et al., 2001; Pelli et al., 2004).

However, the spatio-temporal interactions on crowding have also been studied in the past. Consistent with previous findings in the literature (Chung, 2016; Kooi et al., 1994; Ng & Westheimer, 2002; Tripathy et al., 2014; Tripathy & Cavanagh, 2002), breaking the temporal synchrony affected crowding: when the flankers preceded or followed the target (i.e., FTOA ≠ 0) within a certain temporal window (~150 ms), performance improved (crowding was reduced) compared with the simultaneous presentation but it was still below the unflanked recognition rate.

When observers were asked to make a saccade toward the target letter before responding, letter recognition performance differed significantly from those in the fixation condition (Figure 4B). When the flankers preceded the target (purple circles: FTOA = −60), there was a significant improvement in recognition performance (indicated by filled symbols in Figure 4B), which started between 150 ms and 100 ms before saccade onset and remained fairly constant. When the flankers and the target were presented simultaneously (orange squares: FTOA = 0), there was a modest (<8%) but statistically significant increase in recognition performance only between 150 ms and 120 ms before saccade onset, but not for shorter TSOs. Finally, when the flankers followed the target (green diamonds: FTOA = 60), we found drastic improvements in letter recognition performance, which started around 80 ms before saccade onset and persisted for about 50 ms. A one-way ANOVA yielded a significant main effect of FTOA (F_2,6=19.784; p=0.002). Bonferroni corrected post-hoc comparisons showed that FTOA −60 and FTOA 60 resulted in significantly higher recognition performances than FTOA 0 (−60 vs. 0: t=6.183, p=0.002; 60 vs. 0: t=4.093, p=0.019), whereas the difference between FTOA −60 and FTOA 60 conditions was not significant (t=2.090; p=0.245).

In a double-step saccadic adaptation paradigm, Awater et al. (2005) measured localization of briefly flashed vertical bars, and found that the bars are mislocalized toward the saccade landing points rather than the saccade target. This finding suggests that, if the release in crowding is purely mediated by eye movement commands, the saccade landing-point might have influenced letter judgements. In order to determine the effect of saccadic accuracy on recognition performance, we compared the performance in trials where saccades landed within 0.75 deg of the target location with those where saccades deviated from the target location by more than 0.75 deg. This classification resulted in roughly equal number of trials (5411 trials within 0.75 deg radius from target, 5375 trials outside this virtual circle). We did not find any statistical difference between the two (see Figure S1); hence our data do not support the aforementioned hypothesis.

Experiment 2

In a recent study, Harrison, Mattingley, and Remington (2013) showed that crowding can be fully eliminated within 50 ms before saccadic eye-movements. These authors presented a tilted Gabor patch (amount of tilt adjusted for each observer to yield 75% discrimination performance during fixation) for about 24 ms with or without flanking vertical Gabors, and measured tilt discrimination performance during steady fixation as well as at various times before saccade-onset. The size of the patches was 1.0 deg and the center-to-center distance between the target and flankers were 1.3 deg. The target was presented at 7-7.7 deg eccentricity. Performance showed an increasing trend as a function of target-to-saccade onset, which reached (and exceeded in some cases) the unflanked recognition rates during fixation. In contrast, we did not find any reduction in crowding in the zero FTOA condition. This might be due to several important differences between the two studies. First, they used Gabor patches as the target and flankers and observers had to discriminate the tilt of the target Gabor. We used letters. Letter recognition is carried over conjunctions of features, and therefore, necessitates the extraction of multiple features and attributing them to correct identities, whereas the tilt-discrimination task operates on a single feature dimension (i.e., orientation). Second, in Harrison et al.'s study, targets and flankers were preceded and followed in time by dynamic noise patches. In other words, the target was crowded and both forward and backward masked, creating super-crowding (Vickery et al., 2009). In Experiment 1, we did not use any masks. This begs the question of whether the reduction in crowding in Harrison et al.'s study was due to “saccadic unmasking” as raised in a critique by van Koningsbruggen and Buonocore (2013) (also see Melcher et al. 2015). This is also consistent with improvements in letter judgments when the flankers followed the target in time (i.e., FTOA = 60 ms). Here, the assumption is that onset transients of the flanker letters might have a masking effect on the target. As the target-flanker display is presented closer to the onset of an impending saccade, the flankers might be suppressed more and more due to saccadic suppression while the relative distance between the target and flankers might also be changing (due to saccadic distortions in space).

In Experiment 2, we directly tested whether or not a mask that immediately followed a target is essential in yielding a release of crowding right before a saccade by presenting noise masks following the letter stimuli. Flanking letters, if present, were presented simultaneously with the target letter (FTOA = 0). As in Experiment 1, observers performed the letter recognition task during steady fixation and with saccadic eye movements.

Letter recognition performance as a function of TSO, for each individual observer, is given in Figure 5. The group-averaged data are shown in Figure 6. Our results show that letter recognition performance significantly improved before saccades (Figure 6, the orange square symbols). Further, we found similar improvements for single letter recognition (Figure 6, the black triangle symbols). Crowding strength in the saccade condition showed a non-monotonic trend as a function of TSO but it never fell below the level obtained in the fixation condition (Figure 6B). In other words, the improvement in letter recognition performance observed in the saccade condition was not specific to the flanked condition, but also present in the single letter recognition. In short, our results demonstrate “saccadic unmasking” but do not support “saccadic uncrowding”.

Figure 5.

Letter recognition performance (percent correct) is plotted as a function of time-to-saccade onset (TSO) for the four observers, for results obtained in Experiment 2. Horizontal solid lines represent performance in the fixation conditions, markers represent the saccade conditions. The shaded regions and all error bars represent ± SEM obtained from bootstrapping.

Figure 6.

Group-averaged results of Experiment 2. Letter recognition performance (top) and crowding strength (bottom) are shown for (A) the fixation conditions, and (B) the saccade conditions. Markers represent the empirical data whereas the solid lines represent the model predictions (see the Modeling section). The black and orange colors represent the unflanked and flanked conditions in (A). Error bars represent ± 1 SEM obtained from bootstrapping the pooled data. To facilitate comparisons, significant differences (when 95% confidence intervals do not overlap) between performances in the saccade and fixation conditions are indicated by filled symbols in the upper right panel.

Modeling

Introduction

In order to investigate the underlying mechanisms of pre-saccadic changes in letter recognition, we used a simple neural network model that consists of only seven units (or “cells”). The starting point of the model was the dual-channel model of visual masking (Breitmeyer & Ganz, 1976; Breitmeyer & Ogmen, 2000; Ogmen, 1993). According to this model, a visual stimulus gives rise to a fast transient, and a slow sustained activity in separate pathways (i.e., channels). Inter-channel and intra-channel inhibition of target-related activities by mask-related activities results in performance reduction. Spatiotemporal characteristics of the two stimuli affect the way these interactions occur and masking strengths in various ways (Bachmann, 1994; Breitmeyer & Ogmen, 2006). The motivation of using this model as a starting point was that it can explain a wide range of masking data, both qualitatively and quantitatively (Breitmeyer & Ogmen, 2006), and hence, it is a good candidate for investigating masking effects in a crowding experiment. In addition, previous investigations of crowding demonstrated that stimulus duration and temporal asynchrony between target and flankers strongly modulate crowding effects (Chung & Mansfield, 2009; Chung & Patel, 2011; Chung, 2016; Kooi et al., 1994; Ng & Westheimer, 2002; Tripathy et al., 2014; Tripathy & Cavanagh, 2002). Here, we used a simplified dual-channel model of masking to account for crowding in the fixation condition (with various FTOAs), and we extended this model to explain pre-saccadic changes in recognition performance.

Methods

A schematic of the model is given in Figure 7. The model assumes that crowding impairs the “what” information about the target which is carried within the sustained channel, and that the net sustained activity left due to the target in the sustained channel is directly correlated with behavioral performance. Although there were two horizontally flanking letters in the experiments, for simplicity we used only one pair of cells to represent activity due to flankers. The sustained activity due to the target letter (S_T) projects to, and sends excitatory signals to another sustained cell (denoted by S_C in Figure 7) downstream. Both the sustained (S_F) and transient (T_F) activities due to flankers make inhibitory projections (with varying weights) to S_C. In other words, both intra-channel and inter-channel inhibition take place in the model. The activity of S_C is then fed to a stage where its activity is integrated and passed through a sigmoidal nonlinearity to map its activity to letter identification performance (i.e., percent correct). Note that in the simplified model we have not included separate stages for feature representations (e.g., orientation-tuned mechanisms), feature detection, feature integration, feature grouping, nor for internal templates for letter matching and recognition. Rather, the activity of S_C represents all these stages in a combined manner. Hence, this simplified model is not developed to distinguish between crowding/masking effects that might occur at various levels such as feature detection, feature integration, grouping, and recognition (e.g., template matching). Instead, since stimuli are relatively brief (hence dynamic) and saccades are associated with dynamic processes, the model is focused on the temporal aspects of interactions in order to capture the highly dynamic nature of pre-saccadic stimulus visibility and recognition.^‡

Figure 7.

A modified dual-channel model. Each stimulus creates a fast transient and a slow sustained response in two separate pathways. For simplicity, the transient activity due to the target is omitted from the model since it is assumed to have no role in letter identification. Moreover, again for simplicity, the flanker activity is represented by only a pair of sustained-transient cells even though there were two flanking letters in the actual experiments. The target cell (S_T) and the flanker cells (S_F, T_F) make excitatory and inhibitory projections, respectively, to another sustained cell (S_C) downstream. Therefore, the model consists of four cells (three cells with sustained response and one cell with transient response) and three (not necessarily distinct) perisaccadic mechanisms: (i) enhancement (E), (ii) suppression (S), (iii) release (R). The output of S_C is integrated within a finite temporal window and fed to a sigmoidal nonlinearity to obtain performance in percent-correct format.

The dynamics of all “cells” in the model was expressed by a Hodgkin-Huxley type shunting differential-equation of the form:

$$\frac{dx}{dt}=-Ax+(B-x)(R+I_e)-(x-D)I_i,$$ (1)

where x represents the membrane potential, A represents passive decay constant, B and D represent upper and lower limits (Nernst or reversal potentials) for x, respectively, R represents background activity. I_e and I_i represent the net excitatory and inhibitory inputs to the cell. The membrane potential was then converted to a “firing rate” by a nonlinear function given in Equation (2).

$$FR\left(x\right)=\{\begin{array}{cc}0,\hfill & x<\theta \hfill \\ cx,\hfill & x\ge \theta \hfill \end{array}\phantom{\}}$$ (2)

where θ represents a threshold for firing, and c is a constant.

Each perisaccadic modulation mechanism was modeled as a separate unit whose activity was a function of saccade-onset and was unaffected by stimuli. Equation (3) describes the temporal dynamics of each of these mechanisms. In order to explain the dynamics of changes in performance before saccades, we added three separate mechanisms: (i) enhancement (E) which improves the gains of all synaptic connections, (ii) saccadic suppression (S) which reduces all synaptic weights, and (iii) a release mechanism (R) solely devoted to the enhancement of target-related activity. Admittedly, there need not be three distinct mechanisms to account for our results. Any pair of the aforementioned mechanisms or even all three of them might stem from a single neural mechanism. Behavioral and neural correlates of these mechanisms are outlined in the Discussion section.

$$\frac{dz}{dt}=-\alpha z+(\beta -z)\gamma \mathrm{I},$$ (3)

where α represents passive decay rate, β represents upper limit, γ represents input gain, and I represents eye-movement related input signal (a rectangle function). Model simulations revealed that exact values of α and β are not crucial, and hence identical values were used for all E, S, and R processes. What distinguishes these mechanisms is the input gain, γ, and the time course of I (i.e., its onset and offset relative to saccade onset). The final stage of the model integrates the sustained activity of S_C within a finite interval and passes through a cumulative Gaussian whose mean and standard deviation was determined by fitting integrated activity to data in the fixation condition. The mean and standard deviation obtained from fixation data were used to make predictions for the saccade condition.

Masking strongly depends on the degree of spatial proximity and/or overlap between the target and masks (Breitmeyer & Ogmen, 2006). A target and mask pair, separated in time, but otherwise spatially overlapping, might appear spatially misaligned due to remapping during saccades. To determine whether or not remapping is needed to account for our results, we added a remapping mechanism to our model. This mechanism basically increases the spatial separation between the two temporally non-overlapping stimuli. For simplicity, the model assumes a small horizontal shift in the direction of saccades. This assumption is motivated and supported by previous studies (e.g., De Pisapia et al., 2010) as well as by informal reports by the observers during our experiments who indicated that they often perceived the target and mask as being spatially misaligned during saccadic conditions. Similar observations were reported by De Pisapia et al. (2010) in their perisaccadic masking experiment. The horizontal shift results in a different spacing between the target letter and each flanker in Experiment 1, and between the letters and the masks in Experiment 2. The misalignment between the letters and the masks was simulated by a gradual reduction of mask input to S_T and a slight increase in mask input to S_F and T_F. Equations (4a) and (4b) describe the mask input modulations as a function of TSO.

$$M_{TM}=1∕(1+{\lambda }_{TM}\mathrm{N}(TSO,{\mu }_{disp},{\sigma }_{disp}))$$ (4a)

$$M_{FM}=1∕(1-{\lambda }_{FM}\mathrm{N}(TSO,{\mu }_{disp},{\sigma }_{disp})),$$ (4b)

where M_TM and M_FM represent the mask-input modulation-factors for the target and flanker masks, respectively. λ_TM and λ_FM represent the amplitude of the modulation, and N(.) represents a Gaussian as a function of TSO. The parameters μ_disp and σ_disp together control the temporal window of remapping. All parameter values used in model simulations are tabulated in Table A1 in Appendix. The model has 22 parameters. However, not all of these parameters were used to “fit” the data. Fitting was performed only for the standard deviation and mean of the cumulative Gaussian in the final stage, where arbitrary neural activity is related to performance. Each saccadic mechanism with varying time courses and gains (Is and γs in Table AI) was added individually and in combination to fully capture the trends in the data. All other parameters were identical to those used for the data in the fixation condition. For the parameter values which provided the best fit (in terms of R², or explained variance) to behavioral performances in the saccade conditions, we varied only the parameters of the perisaccadic mechanisms (gain, offset and onset times). In this sense, the model for fixation performance has only two free parameters, and each saccadic mechanism has three free parameters.

Results

Our empirical data in Experiment 1 clearly showed that saccadic modulation of performance is different for different FTOAs. This is at odds with a single mechanism uniquely devoted to releasing crowding, and suggests that multiple mechanisms are involved. In order to understand the underlying mechanism(s) of pre-saccadic letter recognition, we first used the dual-channel model of visual masking to account for crowding in the fixation condition. The solid lines in Figure 4A show the model fits to the group data in the fixation condition. Next, we added several saccadic mechanisms which modulate the strength of inter-channel and intra-channel projections. In short, a global enhancement mechanism, a suppression mechanism, and a target-specific facilitation mechanism together can account for the dynamic changes in performance during saccades. The solid lines in Figure 4B show the model predictions. Although very primitive, this model can predict letter recognition performance considerably well: all predicted values fell within 95% confidence intervals of the empirical data. In order to demonstrate the individual contributions of each perisaccadic mechanism in the model, we simulated our model with one of the perisaccadic mechanisms at a time, and compared model performances by using Akaike Information Criterion (AIC), which penalizes models with more free parameters. Results are shown in Figure S2. In short, even after penalizing for the number of free parameters, the combined model outperforms all other models with only one saccadic mechanism. We will discuss how these mechanisms might map onto the neural substrate in Discussion.

Next, we simulated our model for the stimuli in Experiment 2 to determine whether or not it can account for the results. Dashed lines in Figure 6 represent the model predictions (“Model 1”). Although the model predicts equivalent improvements in both unflanked and flanked conditions, the magnitude of improvement does not match the actual data. This suggests that an additional mechanism is needed to account for the drastic improvements in Experiment 2. De Pisapia et al. (2010) showed that backward masking is almost completely eliminated before saccades. They further reported that the magnitude of reduction in masking was larger when observers perceived a misalignment between the target and the mask, but was still evident when the target and mask were perceived as perfectly aligned. These findings can be explained by a remapping mechanism that alters the relative spatial-positions of stimuli. We added an additional mechanism to our model which simulates remapping of activity generated by the letters and the masks. Our assumption was that the perceived misalignment reduces the effectiveness of the mask, which is presented at the target letter location (since the spatial overlap of the mask with the target becomes reduced due to the spatial misalignment), and mildly increases the effectiveness of the flanker masks (since flanker masks become closer to the target). Here, the effectiveness is defined as the ability to suppress the target letter activity, even for the flanker masks. The solid lines in Figure 6 represent the prediction of this updated model (“Model 2”). In short, it can account for the improvements in both single-letter and flanked-letter recognition. It should be noted, however, that the updated model predicts a monotonic increase in crowding strength as a function of TSO, which is not fully consistent with the data since there is a slight increase in crowding strength between 150 ms to 100 ms before saccade onset (Figure 6B, the bottom panel).

It may be that the number of trials in certain TSOs might affect observers’ responses. In Figure 2, we plotted the distribution of number of trials as function of TSO for each observer in both experiments. Although they look qualitatively similar, we sought to quantitatively assess the relationship between number of trials and performance. To this end, we normalized each individual histogram so that the area under the curve equals unity. Then, we computed the correlation between letter recognition performance and relative frequency of trials in certain TSOs. Figure S3 illustrates the results of this analysis for both experiments. In short, there is no correlation between the two.

In summary, these results suggest that letter recognition is substantially modulated by an impending saccadic eye-movement. The magnitude as well as the time course of this modulation depend on the temporal order of presentation and the presence of well-localized masks.

Discussion

Saccadic eye movements impose a great challenge on the visual system to preserve a coherent and stable percept of the environment despite an unstable stream of retinal images. At the behavioral level, saccades cause an increased uncertainty in perceived space and time (Binda, Bruno, Burr, & Morrone, 2007; Kumar & Stevenson, 2007), large mislocalization errors (Honda, 1989; Ross, Morrone, & Burr, 1997), failure to detect sudden changes of stimulus attributes due to increased threshold contrasts (Bridgeman et al., 1975; Burr, Holt, Johnstone, & Ross, 1982; Zuber & Stark, 1966) around the time of the saccades. At the neurophysiological level, drastic perisaccadic changes in the selectivity of cells in the Superior Colliculus (Li & Basso, 2008), and functional connectivity between cells in FEF and V4 (Noudoost, Clark, & Moore, 2014) have been reported. Moreover, RF profiles of the cells in FEF and V4 shrink and converge towards the saccade target location (Tolias et al., 2001; Zirnsak et al., 2014). Although it is unclear whether or not saccadic modulations in percepts and neural responses are byproducts of a stabilization mechanism, they provide us with a great means to understand how conscious perception of objects are established and maintained.

The spatiotemporal fluctuations mentioned so far are expected to interact with the visual processes related to the visibility of the stimuli, low-level topographical representations and chronological order of the stimuli. For instance, perisaccadic mislocalization is modulated by contrast and visibility (Michels & Lappe, 2004). Consistent with the convergence of RFs before saccades, perisaccadic transfer of the tilt aftereffect is directed toward the saccade target (Zirnsak, Gerhards, Kiani, Lappe, & Hamker, 2011). Since visual masking is sensitive to the spatiotemporal distance as well as the temporal order of the target and its mask, the presentation of a target-mask pair just before a saccadic eye-movement should modulate the strength of masking. Indeed, De Pisapia and colleagues (2010) showed that backward masking can be completely eliminated before saccades partly due to perceived mislocalization and partly due to remapping (or transfer) of target and mask related neural activity to non-overlapping (or less overlapping) populations. Crowding is another phenomenon which depends on the (perceived) spatiotemporal distance between the stimuli (Chung & Patel, 2011; Chung, 2016; Dakin, Greenwood, Carlson, & Bex, 2011; Maus, Fischer, & Whitney, 2011).

Masking reduces visibility, and by consequence impairs recognition. Crowding directly disrupts recognition without affecting visibility. Here, we investigated how object recognition is influenced by saccadic eye movements in the presence of crowding and masking. Now, we look at the hypotheses listed in the Introduction in light of our results. In Experiment 1, for the saccade conditions, we observed a substantial improvement in letter judgements compared to fixation when flankers followed the target in time. The improvement was smaller when flankers preceded the target in time. There was no clear improvement when they were presented simultaneously. Therefore, our results reject the hypothesis that saccade targets are released from crowding in general. Moreover, crowding was reduced only when flankers were presented after the target. This is most probably due to a ceiling effect in the unflanked condition; since unflanked performance reached the ceiling, potential pre-saccadic benefits might be lost. We addressed this point in Experiment 2, where both the ceiling and floor effects were avoided. Nevertheless, different pattern of changes as a function of TSO with different FTOAs suggests that perisaccadic mechanisms of object recognition interact with the temporal order. This is a particularly interesting finding since it allows us to understand what kind of mechanism(s) might be playing a role in pre-saccadic object recognition. If a mechanism is specific to the target object (such as target-specific enhancement), different FTOAs will result in different patterns of performances as a function of TSO. Alternatively, if a mechanism is time-locked to saccades such as saccadic suppression, then any potential effect on behavioral performance should interact with FTOA. On the other hand, if a mechanism operates only on the surround inhibition, then the time course of its effect on performance will be the same regardless of FTOA (see Figure S2).

In Experiment 2, when noise masks were presented following trigrams, we found that recognition performance improved as TSO approached saccade onset. Letter recognition without flankers was similarly affected by saccades, consistent with De Pisapia et al.'s findings. Crowding strength showed a non-monotonic trend as a function of TSO but it never fell below that for the fixation condition. In line with our findings, Melcher et al. (2015) reported different pattern of perisaccadic changes in tilt discrimination performance in the presence and absence of backward masks. Furthermore, Rolfs and Carrasco (2012) used backward masks in an orientation discrimination task before saccades and showed strong improvements in performance just before saccade onset. These authors also found that the performance increase before saccades is far less pronounced without a mask (Rolfs, personal communication). Therefore, these findings support the hypothesis that masking can be reduced or rendered completely ineffective by impending saccades. Furthermore, a comparison of the results from Experiments 1 and 2 suggests that “saccadic uncrowding” is not a genuine phenomenon; because pre-saccadic improvements in performance are not specific to crowded displays (i.e., letter trigrams) but also occur for unflanked single letter recognition.

Our interpretation is in stark contrast with that of Harrison et al. (2013) although both studies report pre-saccadic improvements in performance. However, we believe that this discrepancy could be due to two critical issues in the methodology of Harrison et al.'s study. First, they did not measure the unflanked discrimination performance in the saccade condition. Without this baseline measurement, it is difficult, if not impossible, to draw any conclusion about pre-saccadic changes in crowding strength. Second, Harrison et al. (2013) presented well-localized forward and backward masks in their experiments. Therefore, “saccadic uncrowding” reported in Harrison et al. (2013) was likely to be simply a manifestation of saccadic unmasking. In a recent study, Yildirim and Cornelissen (2015) also did not find any evidence for saccadic modulation of crowding strength, lending further support to our findings. Furthermore, in a separate study, we performed a series of experiments where we investigated the effect of contrast similarity/grouping on pre-saccadic crowding (Agaoglu & Chung, under review). One of the conditions examined in that study was almost identical to the stimuli in Harrison et al.'s study (i.e., Gabor stimuli) with one exception. Our stimuli did not have backward noise masks. With a set of seven (of which six were new and naïve observers), we did not find any sign of improvement in performance before saccade onset. In fact, if any, we found decrease in performance just before saccade onset.

It is important to note that the retinal locations of the stimuli were identical in the saccade and fixation conditions. In both conditions, observers knew the location of the target. The flankers were selected from a different set of letters to avoid mislocation errors and to restrict crowding only to feature-level interactions. Therefore, accounts of crowding based on retinal locations of the target and flankers, as well as uncertainty about their locations cannot explain our results (Pelli, 2008; van den Berg, Johnson, Anton, Schepers, & Cornelissen, 2012).

What are the underlying mechanisms of the pre-saccadic improvements in recognition performance we observed? The change in physical contrast of the proximal stimuli due to saccades might account for our data in the saccade conditions. Alternatively, convergence of RFs in several cortical areas can be considered as a potential account. According to this account, the target and flankers are processed with higher spatial resolution compared to fixation, which increases the cortical distance between the target and flanker representations, and hence, reduces crowding. However, it is difficult to derive predictions about different temporal configurations of the stimuli, partly due to the presence of multiple saccadic mechanisms at play. An alternative way to investigate the modulation of performance due to saccades is through modeling. One can add or remove any combination of enhancement or suppression mechanisms and test their ability to account for the results, singly or in combination. Here, we used a simplified dual-channel model of visual masking to account for the data in the fixation conditions, and we added saccadic mechanisms as needed to account for the results. A simple model based on contrast reduction due to eye movements could explain performance in FTOA 60ms condition in Experiment 1 very well, however, it fell short in accounting for the changes in performance in other conditions (see Supplementary Results). Our simulation results point to saccadic mechanisms with varying time courses: (i) a global enhancement mechanism, which increases the weight of all synaptic projections and operates dominantly about 100ms before saccade onset, (ii) a saccadic suppression mechanism, which decreases all synaptic weights and is most effective midway through saccades up to 100ms after saccade onset, and (iii) a target-specific enhancement mechanism, which increases only the synaptic weights of the cells representing the target information and contributes mostly from ~25ms before and after saccade onset.

De Pisapia and colleagues (2010) reported that the pre-saccadic unmasking was larger when observers perceived a spatial misalignment between the target and the mask, but it was still evident even when both stimuli appeared perfectly aligned. Consistently, our simulations revealed that although the above-mentioned mechanisms predict some improvements in letter recognition, perceived misalignment of the letters and the masks is likely to be the primary factor for pre-saccadic improvements in letter recognition in the presence of backward masks.

Each mechanism has empirical and theoretical bases. Shifts of attention to the saccade target location has been documented, even when observers were specifically told to attend elsewhere (Deubel & Schneider, 1996). Attention and saccade target-selection mechanisms have been claimed to at least partially overlap (Carrasco, 2011). Changes in the visual stimuli that occur within a short temporal interval around saccades go unnoticed due to saccadic suppression. Saccadic suppression is manifested in various forms, such as elevated contrast thresholds or the failure to detect small displacements that could easily be detected during fixation (review: Krock and Moore, 2014). Moreover, facilitatory effects of attention can be target-specific, justifying a target-specific enhancement or equivalently, a flanker-specific inhibition mechanism (Cepeda, Cave, Bichot, & Kim, 1998; Foley & Schwarz, 1998; Somers, Dale, Seiffert, & Tootell, 1999).

Each saccadic mechanism had a different time course in our simulations. This can be explained either by different eye-movement related signals mediating each mechanism, or a single eye-movement signal mediating all mechanisms but with different pathways, resulting in different time courses. A recent study showed that FEF cells exhibit perisaccadic reductions in sensitivity in their future RFs (where the RF of the cell will be after the saccade), a neurophysiological correlate of saccadic suppression (Joiner, Cavanaugh, & Wurtz, 2013). Similar to the perceptual compression of visual space during saccades, FEF cells also undergo temporal compression. More importantly, these authors found different time courses for saccadic suppression and temporal compression.

Premotor theories of attention claim that covert shifts of attention and saccade generation share common oculomotor circuits (Kowler, Anderson, Dosher, & Blaser, 1995; Rizzolatti, Riggio, Dascola, & Umiltá, 1987). Suppressive effects have been identified during covert shifts of attention during fixation (Zénon, Corneil, Alamia, Filali-Sadouk, & Olivier, 2014). Saccades and covert attention can produce similar modulations in perceptual and neural responses, including shrinkage and convergence of RFs towards the focus of attention (Connor, Preddie, Gallant, & Van Essen, 1997; Moore, Armstrong, & Fallah, 2003; Tolias et al., 2001). Perceptual distortions (such as compression of space) similar to those reported in early studies on saccadic eye-movements can also be obtained by using visual masks during fixation (Born, Zimmermann, & Cavanagh, 2015; Suzuki & Cavanagh, 1997; Zimmermann, Born, Fink, & Cavanagh, 2014). Temporal crowding can be well accounted for by a dual-channel model of masking, as we have also shown here. These findings suggest that there is a substantial overlap or interplay between mechanisms underlying saccades, covert attention, masking, and crowding, especially when the stimulus of interest is presented briefly at the periphery. Therefore, we note that it is difficult to pinpoint precisely the origin of pre-saccadic modulations we found. However, we can at least reach several general conclusions. First, our results are consistent with the idea that information about an impending eye-movement modulates object recognition. This modulation was evident only when the temporal synchrony of the stimuli was broken. When the target and flankers were presented simultaneously, crowding was not affected by an intended eye movement. Moreover, recognition of a masked object drastically improved before saccades, regardless of whether it was flanked or not. Therefore, our results suggest that saccadic uncrowding reported previously is only a manifestation of saccadic unmasking. However, this is not to say that there is no form of “saccadic uncrowding.” Our conclusion applies to only the case where observers are asked to report a certain aspect (e.g., orientation, or identity) of the saccade target. What happens to the rest of the visual field in terms of crowding is still an empirical and outstanding question.

Conclusion

The visual system has severe spatiotemporal limitations, which are manifested in numerous perceptual effects. Crowding is a major bottleneck for object recognition. Masking is important in establishing sharp and clear percepts of the environment. Saccades frequently modify the spatio-temporal representations in the brain, which often result in misperceptions. How the mechanisms underlying crowding, masking, and saccade-related processes interact, and to what extent they overlap have been hotly debated. Previous studies showed that masking and crowding are both released at the time of saccades. Here, we clarify the individual, as well as the combined, contributions of these processes to pre-saccadic object recognition. Our results reveal important methodological issues and missing crucial control experiments in previous studies which reported saccadic uncrowding. Our main finding is that impending saccades improve letter recognition (compared to performance during fixation) only when there are spatially overlapping backward masks or onset transients of other spatiotemporally proximal stimuli. Therefore, our results unmask saccadic uncrowding. Our modeling results also suggest that temporal crowding and temporal masking might share similar or common neural mechanisms. Finally, we identified several perisaccadic mechanisms that modulate object recognition with varying timings and strengths: global enhancement, target-specific enhancement (i.e., release of target from inhibition), saccadic suppression, and remapping. Future studies with full-field masks and local masks are needed to elucidate whether strong unmasking reported here originates from perceptual distortions of space, or allocation of resources to increase both spatial and temporal resolution of perisaccadic processing.

Appendix

Model Parameters

Table A1.

Parameter values used in modeling.

Parameter	Description	Value
A	Passive decay constant	100 ST, 20 for the rest
B	Upper level for membrane potential	1
D	Lower level for membrane potential	−1
R	Excitatory background activity	0.1 for SC, 0 for the rest
wst	Weight of the target sustained activity	1
wsf	Weight of the flanker sustained activity	0.6
wtf	Weight of the flanker transient activity	0.3
c	Threshold for firing	0
α	Passive decay constant of saccadic mechanisms	100
β	Upper level of activity for saccadic mechanisms	1
γE	Gain of the enhancement mechanism	30
γS	Gain of the suppression mechanism	20
γR	Gain of the release mechanism	12
IE	Input to the enhancement mechanism	u(t+0.025)-u(t-0.025)†
IS	Input to the suppression mechanism	u(t-0.025)-u(t-0.100)
IR	Input to the release mechanism	u(t+0.090)-u(t+0.065)
μ	Mean of the cumulative Gaussian	281.3
σ	Standard deviation of the cumulative Gaussian	105.9
λTM	Amplitude of modulation for target mask	0.5
λFM	Amplitude of modulation for flanker masks	0.0625
μdisp	Time at which maximum misalignment occurs	0.075
σdisp	Temporal span of misalignment in seconds	0.040

^†u(t) is a Heaviside function where t=0 represents saccade onset.