Between the Scenes

Abstract

Abstract. We constantly move our eyes to new information while inspecting a scene, but these patterns of eye movements change based on the task and goals of the observer. Inhibition of return (IOR) may facilitate visual search by reducing the likelihood of revisiting previously attended locations. However, IOR may present in any visual task, or it may be search-specific. We investigated the presence of IOR in foraging, memorization, change detection, and two versions of visual search. One version of search used a static search array that remained stable throughout the trial, but the second used a scene flickering paradigm similar to the change detection task. IOR was observed in both versions of visual search, memorization, and foraging, but not in change detection. Visual search and change detection both had temporal nonscene components, and we observed that IOR could be maintained despite the scene removal but only for search. Although IOR is maintained in scene coordinates, short disruptions to this scene are insufficient to completely remove the inhibitory tags. Finally, we compare return saccades in trials without a probe and observe fewer return saccades in tasks for which IOR was observed, providing further evidence that IOR might serve as a novelty drive.

Despite a subjective experience of visual stability while perceiving the world, our eyes move about with fixations occurring about 3–4 times per second (MacInnes et al., 2018). The distribution of fixations across an image can be guided by many factors. Bottom-up factors are related to the physical properties of the stimulus, and top-down factors are related to internal factors of the observer’s mental state, such as prior experience, goals, etc. (Wolfe & Horowitz, 2017). An early demonstration of the importance of the observer’s goal was the work by Yarbus (1965) who showed that task instructions dramatically influenced fixation locations. More recent works have also supported the conclusion that the task influences fixation locations and other eye-movement properties (Castelhano et al., 2009; Dodd et al., 2009; MacInnes et al., 2018).

Another factor that has been proposed to influence eye movements is inhibition of return (Posner & Cohen, 1984; Klein, 2000 for an overview). In a typical paradigm, a stimulus is presented after a spatial cue with a various time delay. When the cue-target interval is short, reaction times are faster at the previously cued location, but when the delay is longer than 300 ms, responses to the previously cued location are slowed. IOR may serve an additional purpose in visual search by inhibiting previously attended locations and reducing the likelihood of refixation at previously attended locations (Klein, 1988; Posner & Cohen, 1984). It can be encoded in scene coordinates (Malevich et al., 2020; Maylor & Hockey, 1985), it is long lasting (Samuel & Kat, 2003), and it is observed in serial search (Klein, 1988). IOR was thus chosen as the key mechanism for preventing refixations in many computational models of salience (Itti & Koch, 2000; Krasovskaya & MacInnes, 2019 for an overview).

IOR’s role as a foraging facilitator was tested directly in the context of visual search with pictures from the Where’s Waldo? book series (Klein & MacInnes, 1999). During the search, a target probe was presented at either a previous fixation location or a novel one. Saccadic reaction times (SRT) to probes were slower at previously attended locations, but importantly, preprobe saccades also showed a bias away from previously fixated locations. The researchers proposed that IOR was acting as a foraging facilitator by promoting exploring unvisited locations. IOR was not observed when the scene was removed for probe presentation, suggesting that IOR was encoded in scene properties (Müller & Mühlenen, 2000; Takeda & Yagi, 2000; Redden, 2017). While IOR as an influence on probe reaction times has been well replicated (Wang & Klein, 2010 for a review), it’s role in biasing saccades is still debated with evidence against (Hooge et al., 2005; Smith & Henderson, 2011) and for the proposal (Bays & Husain, 2012; MacInnes et al., 2014; MacInnes & Klein, 2003).

While IOR in search has been consistently observed, it is less clear as a strategy in other visual tasks. It has been observed while looking for something interesting (MacInnes & Klein, 2003), searching in a virtual environment (Thomas et al., 2006), and foraging multiple items by nonhuman primates (Torbaghan et al., 2012), but not in free viewing (Dodd et al., 2009). Memorization of a scene and rating a scene for pleasantness have shown mixed results with some showing IOR (Luke et al., 2014) and others showing an opposite facilitation of return (Dodd et al., 2009; Smith & Henderson, 2009). Return saccades are reliably fewer than forward saccades but typically more frequent than oblique saccades (Hooge et al, 2005; MacInnes et al, 2014). This has led to the question of what the base rate of return fixations might be if IOR were not present. Bays and Husain (2012) analyzed saccades in natural scenes and determined that return saccades were less likely than should be expected based on scene properties alone. Similarly, testing IOR in multiple tasks where IOR might be eliminated or reduced may help establish an appropriate base rate by comparing IOR and return rates for the different tasks (Dodd et al., 2009).

Although IOR has been called a foraging facilitator and both do share some common traits (Kristjánsson et al., 2020), foraging and search are not the same task (Gilchrist et al., 2001). Visual search is typically defined by a target singleton among multiple distractors with the task completed when the target is found or confirmed absent. Foraging, however, has multiple targets and multiple distractors with trial completion after all target items are found and selected. IOR has been shown in foraging with nonhuman primates (Torbaghan et al., 2012), but the task only had a reward attached to one of the potential targets.

We also know that IOR is not observed if the scene is removed (Klein & MacInnes, 1999; Redden et al., 2017) but that it can survive if the task is interrupted (Höfler et al., 2011). Change detection (CD) is an example of a task that includes regular and frequent scene removals but also continues the original task across those low-level changes. It has been shown that IOR can be induced in a CD task with spatial cueing, and it makes the change more difficult to detect (Smith & Schenk, 2010). In that study however, IOR was generated using a subliminal cue so it remains uncertain whether this would be the same type of IOR as caused by an unforced previous fixation at a location (Redden et al, 2021; Hilchey 2016).

There may be reason to believe that scene removal in change detection may not interfere with IOR since the high-level task extends beyond the low-level scene disruption. But in paradigms where IOR is measured with responses to an unexpected probe, there is a chance on every trial that the probe might be presented during the scene off phase of the flicker, and these trials would closely resemble the scene removed condition that has been shown to interfere with IOR (Klein & MacInnes, 1999). Thomas and Lleras (2009), for example, demonstrated that IOR could be measured beyond a scene removal when the previous visual search scene was frequently removed and reinstated. The off period used in their design was 900 ms, which even exceeded the flicker-off duration typically used in change detection tasks.

Change detection, however, not only hosts perceptual differences from other complex visual tasks but also fundamental differences that may impact a person’s attentional set when approaching the task. IOR as a foraging facilitator requires not only a spatiotopic coordinate system to be reliable across saccades (see Klein, 2000), but the unstated assumption is that the spatiotopic world be stable. CD breaks that assumption of scene stability since at least one item, by design, lacks object permanence. It is possible that this loss of assumed scene stability may interfere with the expression of IOR even if the short flicker of scene blanking does not. IOR might be able to survive short, frequent scene removals in a visual search task, though perhaps not in a perceptually similar CD task.

Finally, there has been an understandable focus on realistic scenes when testing IOR in complex visual behavior but that shift has come at a cost of some control of stimuli properties that may be important on the expression of IOR. For example, when presenting realistic scenes across multiple tasks, it is not possible to use the same image more than once per subject since previous exposure to the scene will certainly impact performance on the primary task of search or memorization. In addition, it is difficult to control perfectly for scene complexity and number of items to remember. On the other hand, simple shapes may offer a high degree of control but may lack the external validity to extend to realistic and complex visual behavior. For this study, we will use the alien road sign stimuli (Chetverikov et al., 2018) that offer interesting and complex combinations of recognizable objects while maintaining a high degree of control over stimuli placement and intertrial randomization. Further, the use of empty space between stimuli in the array should help maximize the visibility of the probe and increase the amount of usable probe reaction time data.

Proposal

In summary, while it has been fairly well established that different tasks can change our ability to measure IOR, we tried to choose tasks that allowed us to determine which task features might be important for IOR’s detection. A flicker manipulation from the CD task allowed us to test a degree of scene stability that would need to exist for IOR to remain coded in spatiotopic coordinates; the differing memory load of multiple tasks allows us to explore whether memory load weakened IOR; CD and memorization tasks, likewise, might have encouraged a beneficial strategy of return fixations.

The controlled stimuli provided us the ability to show difficult yet extremely similar stimuli arrays for all subjects and all conditions without the problem of showing the repeated scenes to subjects for different tasks. The memory load of the stimuli itself was also only influenced by the task and not by other differences in the scenes. We do not claim that controlled stimuli are better than experiments with realistic scenes but merely complementary. Finally, our design allowed us to test the idea that tasks with greater IOR might also be characterized by fewer return saccades. We include trials where the task is allowed to continue uninterrupted, and if any of these tasks do not elicit IOR to probes, they may serve as a baseline for return saccades in tasks that do elicit IOR.

Methods

Participants performed five different visual tasks to examine whether IOR would differ based on task instructions. The tasks were divided into two experiments: (1) search, foraging, and memorization used a stable-search array with all stimuli visible throughout the primary task and (2) change detection and search flicker each contained stimuli that were interleaved with a blank screen to produce stimuli with an on/off flicker effect typical of change detection tasks. The stable search and the flicker search used identical tasks and instructions and differed only in the stability of the presentation. The method for all five tasks was adopted from Klein & MacInnes, 1999, and we predicted that IOR, when present, would result in slower SRT to probes appearing at previously fixated locations as compared to novel locations.

An earlier pilot experiment suggested that participants were able to partition the display into the central area where stimuli were presented and the outer area without stimuli, and further, this division was important for the expression of IOR. We therefore only analyze data from probes presented within this central area. For power, our primary hypothesis relied on the task by IOR interaction in Dodd et al. (2009) who reported a partial η² of .22. This was a large effect and resulted in a minimum of 16 participants needed for .95 power (1 − β). The critical t-test (180° vs. 0° within search task) showed an effect size of 1.18 with a required sample size of 19, and we rounded this larger number up to 20 (webPower; Zhang & Yuan, 2018).

Participants

The five tasks were divided into two experiments with 20 participants (18.74 ± 1.15 years old, 18 females) in the stable set (search, memorization, and foraging) and 20 participants (20.05 ± 5.05 years old, 16 females) in the flickering set (change detection and search with flickering). Five students participated in both groups. Order was counterbalanced within flickering and stable sets. Participants had normal or corrected-to-normal vision and were naive to the experimental hypotheses. The study was approved by the HSE University ethics committee; all participants provided written consent and were compensated 200 rubles for their time.

Apparatus

Eye movements were tracked using EyeLink1000 + Tower Mount with 1,000 Hz sampling rate. An automatic algorithm detected saccades using minimum velocity and acceleration criteria of 30°/s and 8,000°/s², respectively. For all participants, the dominant eye was recorded. Nine-point calibration was conducted before the experiment and was defined successful if the average error in the computation of gaze position was less than 0.5° and less than 1° for individual points.

The experiments were programmed using Open Sesame (Mathôt et al., 2012), Python version 2.7, and conducted on an liquid-crystal display (LCD) with a 1920 × 1,080 graphics display resolution running at 144 Hz. The distance between the monitor and eyes was held constant at 62.5 cm with the use of a chin rest. Participants were instructed to respond by pressing a button on the keyboard for the primary tasks (described below) or by fixating the probe when it appears (saccadic response for probe trials).

Stimuli and Procedure

The experiment display included 32 icons in an eight column by four row grid. Icons were the alien road sign stimuli (Chetverikov et al., 2018). The full stimulus set includes 64 unique symbols within the signs (Figure 1), and each symbol was placed in white relief on a square (1.64 ° × 1.64 °) or a diamond (square rotated 45°) with a gray (RGB: red = 117, green = 117, blue = 117; tone = 160, contrast = 0, brightness = 110) background. The distance between each icon was 1.60°. The choice of symbols and orientation of all stimuli were chosen randomly from the full set of 128 (64 symbols × 2 shapes) for each trial with each symbol presented only once per trial (except the foraging task). The duration of each experiment, including instructions, calibration, and (2 or 3) tasks, was approximately 30–50 min, depending on the speed of participants’ responses. Each task consisted of 99 trials with each trial having two possible outcomes. The primary instructed task (search, foraging, memorization, change detection) continued uninterrupted on 33% of trials. In the remaining two thirds of the trials, the primary task was interrupted with the appearance of a probe. Participants were told that when this probe appeared, they should move their eyes to the probe location as quickly and accurately as possible. Probe trials were random within a given block, and participants did not know if a given trial would be a probe trial until it was presented.

Figure 1. Example of the stimulus matrix (not to scale). The array of stimuli was presented in an 8 × 4 grid in the middle of the monitor. An empty band of monitor space around the grid did not contain any alien road sign stimuli but could be a possible location for probes on some trials depending on the location of the penultimate fixation.

Participants initiated trials by looking at a central fixation and pressing the keyboard space bar, which initiated a drift correction procedure by the eye tracker. There was a primary goal for each of the five tasks, which are described in detail below (see Table 1 for a summary). For trials in which the primary task was interrupted, the probe timing and location was gaze-contingent and presented based on the following rules. A critical fixation was chosen randomly from the 6th to 10th fixation after the start of the trial, and the onset of this fixation triggered the probe display. The probe was a red (RGB: red = 255, green = 0, blue = 0; tone = 0, contrast = 240, brightness = 120) circle of 0.70° of visual angle and co-occurred with a 50-ms beep to alert the participants. It could appear at one of four locations (Figure 1) relative to the current fixation. The possible probe locations were on a circumference with its origin at the current fixation point and its radius equal to the distance between the current fixation location and the 2-back fixation location. The angular deviation of the probe was 0° (the location of the 2-back fixation), 90°, 180°, or 270°.

Table 1. Summary of basic design differences for the five tasks.

Experiment	Task	Flicker	Target preview	Target postview	Multiple targets	2af choice
E1	Search (stable)	No	Yes	No	No	Present/absent
E1	Foraging	No	Yes	No	Yes	< 5 <
E1	Memorization	No	No	Yes	No	Present/absent
E2	Change detection	Yes	No	Yes	No	Present/absent
E2	Flicker search	Yes	Yes	No	No	Present/absent

Trials containing a probe automatically finished when the participant fixated the probe. If a probe location was calculated to be presented outside the screen, if the amplitude of the penultimate saccade was less than 1°, or if the participant completed the primary task before the probe onset, then the probe did not appear on the screen and the trial was treated as a nonprobe trial.

Primary Tasks: Stable Experiment Set

Visual Search

Search trials started with the fixation point replaced by a symbol for 1.5 s. The participant was to remember this symbol as the target of their search for the coming trial. Search then began with the onset of an array of 32 unique symbols within a 4 × 8 matrix for up to 8 s, among which could be the target symbol. The participant’s task was to press the left button if the target symbol was in the array and the right button if it was not.

Foraging

Similar to search, at the beginning of each trial, a symbol was presented to replace the fixation point for 1.5 s. The participant was to remember this symbol as the target for the coming trial. The primary task began with the onset of the matrix presented for up to 8 s, which included multiple instances (3, 4, 6, or 7) of the target symbol. The participants’ task was to press the left button if there were more than five instances of the target symbol in the array and the right button if there were fewer. Although this was a nonstandard variant of the foraging task, we wanted to maintain response equivalence of a two-alternative forced choice across experiments.

Memorization

The primary task for memorization began immediately after drift correction with the onset of the matrix. The stimulus array remained on the screen for 8 s, and the participant was to remember as much of the array as possible in order to complete the task. After 8 s on nonprobe trials, the grid was replaced with a single symbol in the center of the screen, and a participant was to press the right button if that symbol was among those previously presented in the array and the left button if it was not.

Primary Tasks: Flickering Experiment Set

In the two flickering tasks, the same array of 32 symbols was used with the same arrangement as in the stable tasks. However, the entire scene array was presented alternately with a white screen at intervals of 750 ms (stimulus array duration) and 250 ms (blank screen duration). The screen only alternated during the primary task but froze in its most recent display state upon a probe onset (when applicable). Due to the increased difficulty of the flicker manipulation and the CD task, the maximum time in nonprobe trials was increased to 20 s.

Change Detection

The primary task for change detection began immediately after drift correction with the onset of the grid. The stimulus array started as visible before alternating with the white screen as described above. On every second iteration of the visible array, a single stimulus in the array changed appearance to a randomly selected alternate, and the participants were instructed to find this change. It was also possible for the item to change shape (e.g., square to diamond) to reduce the task difficulty slightly. After a keyboard response or up to 20 s, a symbol appeared in the center of the screen, and the participant had to press the left button if this symbol was the one that had changed during the trial and the right button, if it was not.

Flicker Search

Visual search with flickering was analogous to the task, instruction, and stimuli of the stable visual search procedure, except the display used the same perceptual scene flickering presentation and had the longer presentation time (up to 20 s) of the change detection task. Unlike the change detection, the target stimuli did not change identity on alternate presentations for flicker search.

Task Summary

These tasks differed in a number of features in addition to their instructions. For example, the search and foraging tasks presented previews of the target prior to the trial, whereas the change detection and memorization presented the target after the trial. These feature differences are summarized in Table 1. We used tasks with a variety of feature combinations to best allow us to differentiate whether any observed differences in IOR were due to task features or top-down instructions.

Analyses

SRT to the probe was measured as the primary dependent variable. The results were subjected to a linear mixed-effects model using the glmer function (R package lme4 (Bates et al., 2015), using the gamma distribution as the best fit for SRT data (Lo & Andrews, 2015).

Fixed effects were included for both slope and intercept as long as they improved the fit of the model as determined by a χ² test using the CAR package (Fox & Weisberg, 2019). Pairwise comparisons and plots of key interactions were tested using expected marginal means of the model with the EMMeans package. Trials with blinks did not affect the presence of IOR in search (Dodd et al., 2009), so they were not excluded from our analysis to maximize the amount of available data. For removal of outliers (Leys et al., 2013), we used median absolute deviation (MAD) and values outside the interval median ± 2.5 MAD were excluded from the data set. Subject was added as a random effect in all analyses. Effect sizes were calculated using the effect size R package and used Cohen’s w for χ² test and Cohen’s d for pairwise comparisons. All data and stimuli are available online (MacInnes, 2022).

Probes Position Outside

The central stimuli matrix did not cover the entire monitor screen, and the area of empty space around the matrix was not related to the tasks and was typically not visited by participants unless the probe appeared there. Based on our pilot results, we observed that probes in this empty space behaved differently for IOR that was coded in scene or object coordinates in a manner similar to removal of the search array (Klein & MacInnes, 1999). Probes presented in the empty area outside of the search array were also more likely to be 180° trials (46.8%) as compared to 90° (35.2%) or 0° (18.0%) and as such were removed from the main analyses.

Results

Experiment 1: Stable Tasks

The tasks for the stable-array experiment included search, memorization, and foraging. The model’s fixed effects included two variables: task (search, memorization, and foraging) and angular deviation (0°, 90°, 180°). Trials were removed if the participant failed to fixate the probe in a single saccade, if the distance between the probe and first fixation point was larger than 2°, or if the subjects were not able to fixate the probe (total exclusions for these three reasons were 25% for search, 2.2% for memorization, 17.8% for foraging). Outlier detection of 2.5 MAD resulted in 9.66% of trials excluded in visual search, 7.7% in memorization, and 7.89% in foraging. As a result, the final generalized linear mixed-effect (GLME) analysis included 67.69% of trials from search, 90.3% from memorization, and 75.75% from foraging. Mean times to complete the primary task for nonprobe trials were 3.3 s for search, 3.8 s for foraging, and 8.0 s for memorization. Accuracies for the primary tasks were 74.1%, 72.5%, and 61.0%, respectively.

For saccadic reaction time to the probes, we observed a main effect of Angle (χ²(2) = 54.3, p < .001, w = 1.65) and a main effect of Task (χ²(2) = 429, p < .001, w = 4.6; Figure 2) as well as a significant interaction of Angle and Task (χ²(2) = 11.6, p = .020, w = .74). All three tasks (search, foraging, and memorization) showed a significant slowing of probe SRT between 0° and 180° locations. Namely, there were significant differences in pairwise comparisons between 0° and 180° for visual search (z = 5.82, p < .001, d = 1.34), foraging (z = 5.1, p < .001, d = 1.14), and memorization (z = 3.80, p < .001, d = .85). The interaction of Task and Angle was largely caused at 90° with comparisons between 0° and 90° only significant for Search (z = 3.70, p < .001, d = .47) and Foraging (z = 3.40, p < .001, d = .76) but not Memorization (z = 2.1, p = .081, d = .21). Differences between 90° and 180° were not significant for any of the three tasks (search: z = 1.4; memorization: z = 1.7; foraging: z = 1.9). Thus, the IOR effect was larger in search and foraging than in memorization, and generally, the 0°–90° difference was greater than the 90°–180° difference. Importantly, the primary contrast of 0° versus 180° suggested significant IOR in all three tasks inside the stimulus grid.

Figure 2. Saccadic reaction time to the probe in the stable experiment set: visual search, memorization, and foraging. Saccades to previous (0°) locations were slower than 180° locations for probes inside the search grid, suggesting IOR in all three tasks.

Experiment 2: Flickering Tasks

The tasks for the flickering experiment set included change detection and flicker search with the latter being similar to the stable-search task but with a flicker dynamic. The fixed effect model included the three variables: task (change detection, flicker search), angular deviation (0°, 90°, 180°), and screen flicker state (on-off). For the flickering tasks, a probe could be presented while the scene was visible or in the flicker-off state. Although probes in the flicker-off state were less likely, we believe it an important test for IOR for probes presented when the scene was not visible. Specifically, the flicker-off condition matches the timing of the scene removal condition in studies by Klein and MacInnes (1999). Trials were removed from the analysis in the same way as in nonflickering tasks. As a result, in the generalized linear mixed-effect (GLME) analysis, 80.52% of trials were used from flicker search and 86.53% from change detection. Mean times to complete the primary task for nonprobe trials were 3.9 s for flicker search and 8.3 s for CD. Accuracies for the primary tasks were 75.9% and 58.0%, respectively.

There was a main effect of Task (χ²(1) = 259, p < .001, w = 3.6), Angle (χ²(2) = 16.70, p < .001, w = .91), and Screen (χ²(1) = 25.1, p < .001, w = 1.1). We found significant interactions between Task and Angle (χ²(2) = 34.7, p < .001, w = 1.3) and Task and Screen (χ²(2) = 11.0, p < .001, w = .74).¹ Importantly, the main effect of Angle was qualified by the interaction between Angle and Task suggesting a difference in IOR as observed in Figure 3.

Figure 3. Saccadic reaction time to the probe in the flickering experiment set: flicker search, change detection. Saccades to previous (0°) locations were slower than 180° locations but only for the flicker-search task.

In particular, pairwise comparisons showed a significant difference between 0° and 180 ° for visual search (z = 6.4, p < .001, d = 1.4) suggesting IOR, but did not show a similar result for change detection (z = .6, p = .80, d = .13). Contrary to the stable-search results, the flicker search showed a larger 90°–180° difference as compared to the 0°–90° difference.

Return Saccades

Given the idea that IOR may be a foraging facilitator in visual tasks, we would expect that the rate of return saccades would be higher in tasks that do not elicit IOR. We looked at saccades in trials without a probe and rated the relative angle as compared to the vector of the previous saccades. Only the first 8 s were used to fairly compare all tasks and minimize the chance that a target was already found. To define relative angle, any saccade in the same vector as the previous saccade was considered to have a relative angle of 180°, and any saccade with the opposite vector would have a relative angle of 0°. We grouped saccades in bins of 10° resulting in 18 bins from 0° to 170° (Smith & Henderson, 2011). Overall results show a familiar pattern (Figure 4a) with forward (180°) saccades being the most frequent (14.9%, SE = .17) and return (0°) saccades (9.4%, SE = .17) also being more frequent than many of the oblique angles (90° 5.4%, SE = .17). Our primary interest was in the rate of return saccades, so the rate of return was calculated for each participant and task (from both experiments) and subjected to an GLME with the rate of return saccades as dependent variable, task as a fixed effect and subject as random factor. There was a main effect of task (χ²(1) = 59.2, p < .001, w = 1.9) with different return rates across the multiple tasks (Figure 4b).

Figure 4. (A) The rate of saccades during the primary tasks for all bins of 10°. (B) Focusing only on return saccades (the 0° bin), we show the rates of return saccades separately for all five tasks. Values are expected marginal means from the model with SE and 95% confidence intervals.

Figure 5. Correlation between the probability of return fixations and the magnitude of IOR for the five tasks. Although correlation is not causation, this is the pattern of results we would expect to see if IOR acts as a novelty facilitator in complex visual tasks.

Bonferroni-corrected comparisons showed change detection with the highest rate of return saccades and significantly more than stable search (t(95) = 5.7, p < .001, d = 2.5), memorization (t(95) = 2.8 , p = .009, d = 1.3), foraging (t(95) = 6.6, p < .001, d = 2.7), and flicker search (t(62.5) = 3.9, p = .003, d = 1.3). Memorization also showed more return saccades than stable search (t(62.5) = 3.8, p = .008, d = 1.2) and foraging (t(62.5) = 3.5, p = .007, d = 1.2). Flicker search had a greater rate than foraging (t(95) = 3.5, p = .007) and stable search (t(95) = 2.6, p = .120), but the latter difference did not reach significance. Effectively, CD > Memorization ∼ Flicker Search > Foraging ∼ Stable Search.

Discussion

The purpose of the study was to investigate IOR as a general characteristic of complex visual behavior across two experiments and five tasks. In previous research, IOR was suggested to be a search-specific strategy, with conflicting results for other tasks (Dodd et al., 2009; Luke et al., 2014; Smith & Henderson, 2011; Thomas et al., 2006). Our study included tasks of search, memorization, foraging, and change detection and tested how IOR is coded in scene coordinates by testing changes in the scene grid both temporally and spatially. We observed IOR in the stable search, flicker search, memorization, and foraging tasks with responses to probes at previous locations slower than equidistant controls. The change detection task was the only exception with no IOR observed for that task during any portion of the display. In support of the idea that IOR may promote novelty seeking behavior, we also observe reduced likelihood of return saccades in tasks in which IOR is observed as compared to those for which it was not.

Our study provides evidence that IOR is a characteristic of broader visual behavior and is not specific to only search. Although IOR has been claimed a foraging facilitator (Klein & MacInnes, 1999), it had only been tested previously in a task with multiple targets with nonhuman primates (Torbaghan et al., 2012). Here, we are able to compare IOR in foraging and search with similar stimuli and responses. The change detection task did not show IOR; however, it did have several low-level differences from the more typical search task. For example, the target was presented after the scene inspection, and the scene was frequently removed in a flicker paradigm to make the change more difficult to spot.

These features were shared with other tasks, however, since memorization also showed the target postscene and our flicker search task sharing the same scene removals as the change detection task. We therefore believe it is likely that the task of change detection itself leads to an attentional state where inhibitory tags at previous locations were not needed or may even have been harmful to the task performance. Successful change detection may require longer fixations or more frequent revisits at an attended location since a comparison of preflicker and postflicker objects may require more than a single, short fixation. Change detection also breaks the assumption that the search array will remain stable across multiple fixations leading to a state in which inhibitory tags are not maintained. IOR as a facilitator in search is possible because it is typically coded in scene/environmental coordinates (Malevich et al., 2020; Maylor & Hockey, 1985; Posner & Cohen, 1984), and scene removal has been shown to eliminate IOR (Klein & MacInnes, 1999; Müller & Mühlenen, 2000; Takeda & Yagi, 2000).

Our flickering manipulation was a test of whether these tags could survive short, repeated removal given that the task was maintained and an expectation that the scene would resume. The flicker-search task showed that IOR can survive scene removal in these circumstances. This is consistent with other results for which the continuation of IOR tags is dependent on task demands. For example, Thomas and Lleras (2009) were able to observe IOR despite frequent scene removals in a visual search task in a design similar to our flicker search. They frequently removed the search array for long periods (900 ms) with much shorter (100 ms) scene visible periods. Similar to our flicker-search condition, probes could be presented during the scene off phase, but unlike ours, the probes were only presented once the primary search task was completed. Similarly, Höfler et al. (2011) had participants perform back to back searches for different targets on the same stimuli. They tested for IOR during the second search at locations that were fixated during the first search and found IOR as long as the first search was not completed. We believe these results, taken together, suggest that IOR may be maintained given an attentional set that is primarily dependent on task demands.

Our two search tasks (flicker and stable) did show similar magnitude of IOR (0°–180°) but did differ somewhat at the orthogonal (90°) location. Harris et al. (2015) suggested that larger 0°–90° differences such as we observe in the stable search are likely to represent IOR whereas the larger 90°–180° difference as observed in our flicker search is more likely to be an opposite facilitation effect or saccadic momentum. However, they based their predictions on a vector model of opposite facilitation effect that facilitates responses that continue the most recent saccade, but we believe this model is less likely to influence the 2-back probe location of our results.

Our results seem clear with no IOR observed in the change detection task but raised the question of what made this task different from other complex visual behaviors. It is possible that change detection loads spatial working memory more than other tasks, and this may be interfering with IOR (Castel et al., 2003). However, spatial working memory may have less impact on saccadic IOR, which is presumably the type we observe here. For example, memory load may not impact IOR at all (Zhang & Zhang, 2011) or it may reduce, but not eliminate IOR under high load (Shen et al., 2021). The change detection task certainly loaded memory with likely strategies of loading one or more stimuli into memory for a time span that allowed comparison with a subsequent on phase of the array. But our memorization task also required a significant memory load and still did show significant IOR. Memorization required loading a large subset of stimuli into memory to successfully do the task, but CD required a smaller subset that was frequently replaced. These tasks do load memory in different ways, but given that we did observe (reduced) IOR under memory load for the memorization task, we believe it was unlikely to be the only reason for the absence of IOR in CD. The memorization and CD task also had the lowest accuracy on the primary task, perhaps reflecting that increased memory load. However, they also had the highest rates of acquiring the probe in a single saccade when presented, so perhaps this could also reflect a shift in focus away from the primary task for these two conditions.

Alternatively, the change detection task may simply require longer or repeated fixations to a location in order to spot a change that happens over the course of a 1,000 ms cycle. If IOR is adaptable to the attentional set of the task demands, then inhibitory tags may lose their advantage as a foraging facilitator in this situation. It may also be a combination of low-level scene changes in addition to the attentional set, since there was no guarantee of object permanence during scene removal for change detection as there was in the flicker search condition. Scene removal in the change detection task may have a similar impact as previous research that observed IOR’s removal, and in this case, the current result for which IOR was maintained across the flicker search condition becomes all the more striking. Although we did observe IOR in our memorization task, this is not always true in the literature. We did observe reduced IOR as compared to search, but Danziger, Kingstone, and Snyder (1998) did not observe IOR at all when participants memorized real scenes. Given the differences in stimuli (icons vs. scenes), it is likely that participants developed different strategies or attentional sets for these two memorization tasks. Different tasks do produce different eye-movement placement (Castelhano et al., 2009; Mills et al, 2011; Yarbus, 1967), but these differences were shown in realistic scenes where semantic knowledge of relationships between objects could guide selection. Our stimuli, however, required the icons to be memorized independently. It is possible that memory for a scene and memory for a spatial array encourage different attentional sets. With regard to IOR, the attentional set we observed in this study may have allowed inhibitory tags to be more easily associated with unique objects. Alternatively, scene memorization may rely on general viewing goals (Mills et al, 2011) whereas icon memorization, such as search, focuses on specific known objects. Although our participants did not know the memory object to be tested in advance, the discretization of the scene may have resulted in an object-based strategy that shifted the task closer to that of visual search. A more direct comparison of stimulus arrays versus scenes with the same task instructions is needed to test this idea, however.

Finally, for IOR to be a foraging facilitator in vision, the temporal slowing of responses to probes should not only be shown in the task but the task should also show a reduced likelihood of refixations at previously attended locations (Wang & Klein, 2010). Since IOR in complex visual tasks is likely to combine with the attraction of salient items (Itti & Koch, 2000) and novelty seeking strategies (Najemnik & Geisler, 2005) to determine future attended locations, it becomes problematic predicting how much of a reduction IOR might account for. Bays and Husain (2012), for example, controlled for the salience of previous locations and determined that the rate of repeat fixations was indeed lower than the salience-controlled baseline. In our results, we were able to compare return fixations in tasks with and without IOR, and consistent with Dodd et. al (2009), we observed a lower rate of return fixations in tasks for which IOR was present. In fact, the magnitude of IOR (0–180° probe SRT) in our five tasks was negatively correlated (r² = .35)² with the probability of return fixations in the same task. Specifically, tasks with significant IOR resulted in a reduction of between 19% (memorization) and 39% (foraging) return saccades as compared to change detection. IOR, in any task in which it was observed, seems to have coincided with fewer repeat fixations.

There are limitations in the current experiments that we hope to address in future research. For example, while we tried to cover key combinations of task features in our five tasks (see Table 1), not all possible combinations were tested. Although the flicker manipulation in itself did not interrupt IOR, it is possible that it might have interacted with other task features such as task memory load. Similarly, we only tested one flicker rate commonly used in CD tasks, but it is possible that a longer off scene duration might serve to disrupt the scene stability and produce a less suitable environment for inhibitory tags. Finally, our current results used controlled stimuli to make cross-task comparisons easier for a within-subject design, but a complementary replication of these tasks would certainly help extend these results for greater real-world validity.

Publication Ethics: The study was approved by HSE University ethics committee; all participants provided written consent and were compensated 200 rubles for their time.

Open Data To the best of my ability and knowledge, we have provided all original materials and clear references to all other materials via a stable online repository. All data and stimuli are available for download at the Open Science Framework: https://osf.io/9jxuh/