The Functional Visual Field(s) in simple visual search

Abstract

During a visual search for a target among distractors, observers do not fixate every location in the search array. Rather processing is thought to occur within a Functional Visual Field (FVF) surrounding each fixation. We argue that there are three questions that can be asked at each fixation and that these imply three different senses of the FVF. 1) Can I identify what is at location XY? This defines a resolution FVF. 2) To what shall I attend during this fixation? This defines an Attentional FVF. 3) Where should I fixate next? This defines an Exploratory FVF. We examine FVFs 2&3 using eye movements in visual search. In three Experiments, we collected eye movements during visual search for the target letter T among distractor letter Ls (Exps 1 and 3) or for a color X orientation conjunction (Exp 2). Saccades that do not go to the target can be used to define the Exploratory FVF. The saccade that goes to the target can be used to define the Attentional FVF since the target was probably covertly detected during the prior fixation. The Exploratory FVF is larger than the Attentional FVF for all three experiments. Interestingly, the probability that the next saccade would go to the target was always well below 1.0, even when the current fixation was close to the target and well within any reasonable estimate of the FVF. Measuring search-based Exploratory and Attentional FVFs sheds light on how we can miss clearly visible targets.

Introduction

During visual search for a target among distractors, people will normally move their eyes from one place to another in order to use the fovea with its higher acuity to gather information that would be degraded or unavailable in the peripheral visual field. Indeed, in most real-world search tasks, the decline of acuity and the rise of crowding in the periphery will require eye movements (Whitney & Levi, 2011). The portion of the scene that can be processed around the current fixation is known as the “Useful Field of View” or, to use Sanders’ (1970) term, the “functional visual field” (FVF; see also Ikeda & Takeuchi, 1975). Sanders divided the observer’s visual field into three attentional areas: the stationary field, where people can process information without making eye movements; the eye field, where people would make eye movements but not head movements to collect the next sample of information; and the head field, where the head movements are also required (Sanders & Houtmans, 1985). What stimuli can be processed with the eyes fixated in one spot? It is intuitively obvious that the answer is task-dependent. Sanders found that, in a simple target detection task, observers barely made any eye movements when the target was presented within 30 degrees of fixation. Of course, if you were asked to search for something like the letter T among Ls, your FVF for that task would not have a 30 deg radius. Looking to medical image perception as an example, Kundel, Nodine, Thickman and Toto (1987) found that, when radiologists were asked to search for low contrast lung nodules in lung x-rays, the target detection was most effective when the nodule was within a radius of 3.5 degrees of visual angle. In another medical image study, Carmody, Nodine, and Kundel (1980) also asked radiologists to look for a nodule in chest x-ray films. The images were presented only for 300 msec in order to simulate a single fixation. They found that detection rates dropped by one-half when the nodules were presented at 5 degrees from the fixation. In a more recent addition to the challenges of computing the FVF in radiology, consider the situation of 3D volumetric images, such as lung computed tomography (CT) in which a reader looks for targets in a stack of images represented a volume of image data. Ebner et al. (2017) showed that the nodule detection rates on chest CT were highly correlated not only with the size of the nodule and local lung complexity but also with the size of an individual’s FVF.

Understanding the FVF is critical to understand what it means when we say that we have “looked at” an image. When can a radiologist stop searching an image for some specific finding? Theoretically, a radiologist could look directly at every pixel in the image in order to confirm that a target is absent, but this would not be a sensible behavior. On the other hand, simply scrolling through a stack of CT images so that each slice is fixated at least once is not likely to be adequate even if, in some sense, the reader did look at everything (Drew, Vo, Olwal, Jacobson, Seltzer & Wolfe, 2013; also see Venjakob & Mello-Thoms, 2016 and Williams & Drew, 2019 for review). Images like lung CTs are complicated, as are other real-world search scenes such as kitchen drawers or crowded beaches. To understand how such complex images are searched, the role of the FVF in search should not be neglected. As a prelude to tackling such problems, the goal of the present paper is to provide a richer description of the FVF in simple visual search.

In this paper, we would like to argue that, for any given visual search task and stimulus set, it is not adequate to talk about “The FVF”. In a visual search for a target among distractors, one can speak of the FVF in three different senses of the term. We do not wish to argue that these need to be three distinct entities in, say, a physiological sense. These three FVFs can be thought of as answers to three questions that can be asked during a fixation.

1. Can I identify what is at a specific location XY? This defines a resolution FVF.

2. To what shall I covertly attend during this fixation? This defines an Attentional FVF.

3. Where should I fixate next? This defines an Exploratory FVF.

   These different FVFs can be defined in more detail:

   1. The Resolution FVF - This is the extent of the field within which a specific target can be identified when the target location is known and attention is directed to that location. Acuity and crowding constraints limit the Resolution FVF. Detection or discrimination isopters can be thought of as measure of a Resolution FVF for a specific stimulus (e.g. Abrams, Nizam, & Carrasco, 2012).
   2. The Attentional FVF - For a given point of fixation, the Attentional FVF covers the set of items that might be covertly attended and processed, at least, to some extent, while the eyes are fixated at one location. Though it is likely that the deployment of covert attention is constrained by the resolution FVF, they can be decoupled (see Figure 1). In a search, attention could, in principle, be deployed to a plausible target location outside the Resolution FVF even if the observer cannot recognize that item without an eye movement. Alternatively, attention in a search might be restricted to items quite close to fixation even if a more eccentric target could have been identified. In our work, we are predisposed to think of the attentional FVF as defined by the distribution of a set of serial, covert deployments of attention to items. However, it would be also possible to think of the attentional FVF as describing a region within which all items are processed in parallel in some fashion (Hulleman & Olivers, 2017; Motter & Simoni, 2008). Deciding between serial and parallel accounts is an interesting (and longstanding) issue. The data presented here will underline why it is so difficult to decide between these accounts.
   3. The Exploratory FVF - If the target was not found during the current fixation, the eyes will move to a new location to continue the search. The set of locations where the eyes might go next defines the Exploratory FVF. Note that this FVF is not the same as the physical limits on possible eye movements. It is a probabilistic map of where the eyes are deployed in a specific task. The Exploratory FVF is similar to Sander’s eye field when observers’ head was constrained and is similar to Sander’s head field when observers’ head is free to move.

Figure 1:

Search for a duck. From fixation at X, the chick (1) can be identified as not duck because it is inside the resolution (Green) and attentional (blue) FVFs. The rhino (2) and duck (3) can be attended and considered as possible target items, but proper identification would require refixation. They are in the attentional FVF but outside the resolution FVF. The buffalo (4) is outside the attentional FVF but a saccade might be directed there in order to determine if there were ducks in the neighborhood. This, it is in the exploratory FVF.

In this paper, we are specifically interested in the role of the FVF in visual search for a target among distractor items. The interaction between the different senses of the FVF is illustrated in Figure 1.

Let us suppose that the observer is currently fixated at X and that the task is to find a duck. While at X, the observer will spend 200–300 msec attending to items before moving fixation to a new spot. The current attentional FVF is defined by the set of all items that could be attended from fixation on X. If the observer, fixates on X and attends to “1”, he can “process” the item to the point of identifying a chick. The chick falls inside the resolution FVF (Green) for this fixation on this task. Fixated at X, the observer can direct attention to “2”. In processing the item at 2, he may not recognize a rhino, but it may be possible to reject the item as not a duck. The rhino is inside the attentional FVF (Blue) but outside the resolution FVF for this search task. Item 3 is also inside the attentional field. That item might be attended, identified as a possible target, and made the destination of the next fixation. Being outside of the resolution field, the duck cannot be identified but it can be attended and processed to some extent. Location 4 could be the destination for an exploratory saccade. There is something out there making it a sensible spot to explore, even if nothing can be more extensively processed from the current point of fixation ¹.

This example should make it clear that the Resolution, Attentional, and Exploratory FVFs are not independent of each other in visual search, but neither are they identical to each other. One can legitimately attempt to measure and manipulate each of these aspects of the FVF. The Resolution FVF can be measured by standard visual field measures (perimetry). In a visual search, if the eyes were fixated at one point, the Resolution FVF would constrain where a target could be found and properly identified, even if covert attention could, in principle, be deployed to candidate targets beyond the Resolution FVF (see Motter & Simoni, 2008). We will not measure the Resolution fields in the experiments described here, focusing, instead on the Exploratory and Attentional FVFs. Those FVFs can be estimated using eye movement methods as cartooned in Figure 2.

Figure 2.

Eye movement signatures of the Exploratory and Attentional FVFs. Saccades are numbered backward from the saccade that goes to the target (the “targeting saccade”). Distribution of purple saccades define the Exploratory FVF. The distribution of the green saccades that land inside a region of interest (ROI) around the target, defines the Attentional FVF. Small (red) corrective saccades need to be filtered out. These corrective saccades categorized as “post-targeting saccades” might be used to reduce the target eccentricity after the target was initially fixated.

The figure shows a hypothetical scan path when searching for a letter “T” among letter “L”s. Saccades are numbered backward from the final “targeting” saccade (#1, shown in green). That saccade brings the eyes within some region of interest (ROI) around the target. If we assume that this eye movement was intended to fixate the target, it follows that the target must have been attended to and identified from the fixation location at the start of that saccade; the previous fixation point. Thus, the target must lie within the Attentional FVF for this situation. The distribution of all these “targeting” saccades can define the Attentional FVF. Presumably, the target was not found when the eyes were fixated at the starting points of saccades 2, 3, 4, or 5 (shown in purple). Each of these “search” saccades moves to a point within the Exploratory FVF around the previous fixation. The distribution of these saccades can be used to define the Exploratory FVF for this task.

As we will discuss in the Results section, there are often some very short saccades, executed after the saccade that takes the eyes to the near vicinity of the target (shown in red). These post-targeting saccades are not of particular relevance to the FVF question, though we will describe their distribution as well.

Several simple predictions fall out of this account of the eye movements during search; at least, for a simple search for something like finding a “T” among “L”s. In a relatively homogeneous display, like those used here (see Figure 3), it seems likely that a search saccade would go to a location, relatively rich in unattended items (c.f. Najemnik & Geisler, 2005). It follows that the Exploratory FVF, based on those search saccades, should be larger on average than the targeting saccades that define the Attentional FVF. Second, if the search for a T among Ls is serial & self-terminating (Bergen & Julesz, 1983), then observers will need to search through (N+1)/2 items on average before stumbling on the target (Sternberg, 1966). Thus, in the context of a serial model, the size of the Attentional FVF should be consistent with 50% of items being attended on a successful target present search (Visiting an average of 50% of items is consistent with other accounts of search as well, e.g. Young and Hulleman, 2013). If observers are not keeping track of rejected distractors (Horowitz et al., 2006), they would still need to search through (N+1)/2 items though some of those items might be visited more than once.

Figure 3.

The stimuli and procedure used in the foraging task of Experiment 1. Observers were asked to click 500 Ts during the experiments. Once a target was clicked, another target immediately appeared at a randomly selected location. The red circles marking target and prior target locations are shown for illustrative purposes only and were not part of the experimental stimuli.

In the three experiments presented here, we provide evidence in favor of the hypothesis that the Exploratory FVF is larger than the Attentional FVF in the search task. Our data will argue against the hypothesis that every item inside the Attentional FVF is being processed during a fixation (e.g. Mackworth, 1976, Anderson et al, 2011). The FVF data constrain models of visual search, even if they do not resolve the classic debates about the search process.

Experiment 1 : Basic T among L search

Method

Participants

Twenty-two observers participated in Experiment 1. All participants were recruited from the Brigham and Women’s Hospital Visual Attention Lab volunteer pool. All had normal or corrected-to-normal vision and passed the Ishihara color screen. Participants gave informed consent as approved by the Brigham and Women’s Hospital IRB and were paid $11/hour. Participants ranged in age from 20 – 53 years.

Apparatus:

Eye movements were recorded by a SMI RED250 mobile eye tracker with sample rate 250 Hz. Stimuli were presented on a 15” HP laptop monitor with a screen resolution of 1920 × 1080. The visual display subtended 40.2° of visual angle horizontally and 22.6° vertically at a viewing distance of 47 cm. Both eyes were tracked during the experiment. The experiments were written in MATLAB 8.3 with Psychtoolbox version 3.0.12 (Brainard, 1997; Kleiner et al., 2007; Pelli, 1997).

Stimuli and procedure:

There were two parts of the experiment: A foraging task followed by a search task. In the foraging task, the stimulus display consisted of one letter T and seventy-nine letter Ls as shown in Figure 3. To avoid variations in crowding across the display which might bias observers’ search strategy, the letters were uniformly distributed in an 8×10 array. The size of each letter was 1 deg × 1 deg and the center to center separation between any two adjacent letters was 2.5 deg. Only one letter T appeared at any given moment during the experiment. Observers were asked to use the mouse to collect the letter T by ‘clicking’ on it. Once a letter was clicked, a new target letter T immediately appeared at a new random location. To avoid any pop-out effect due to the onset of a new target, all letters were randomly rotated when a new letter, T, appeared. Observers were asked to collect 500 Ts to complete this foraging task. This is the equivalent of 500-trials of a T among L search task but with a single, essentially stable scene.

After the foraging task, observers conducted a visual search task using a similar stimulus display in order to collect standard reaction time x set size data for these particular stimuli. Half of the trials contained a target letter T in the set of Ls and the other trials only contained Ls. Two different set sizes (set size = 80 & 42) with the same letter size and separation between any two adjacent letters were tested. Since density was fixed, the 80 element sets covered a greater area than the 42 element sets. The 80 element sets were positioned in an 8×10 arrays and the 42 element sets were positioned in a 6×7 arrays. All displays were centered on the middle of the screen during the experiment. There were 20 practice trials and 100 experimental trials for each of the two set sizes. Two set sizes were intermixed across trials. Observers used a keyboard to respond whether there was or was not a letter T in the display. Observers’ eye movements were recorded in both foraging and search tasks.

Data Analysis

Eye movements events were detected by the SMI BeGaze built-in event detector where saccade peak velocity threshold was set as 40°/sec. Observers’ tracking ratios (time tracked vs time on task) were monitored to check how well gaze was tracked during the recording. Based on these results eye tracking data from 4 observers were excluded due to a low tracking ratio (< 70%) meaning that more than 30% of time their gaze tracking was lost. This left 18 observers whose results are reported below. The experiment was preregistered on the Open Science Framework (Foster & Deardorff, 2017) (https://osf.io/vzg28/) where the data files are uploaded as well.

Results

As discussed above and shown in Figure 2, there are three types of saccade of interest:

• Search saccades, made as the observer forages through the Ls, look for a T.

• Targeting saccades, made when the observer finds the T and plans to respond to it.

• Post-targeting saccades, made after the observer has found the T but before they click on the T that ends that trial.

The identity of the targeting saccade needs to be inferred since it is not necessarily the first fixation on or near the target or the last fixation before the target is clicked. We used the following rules. Since the separation between any two adjacent items was 2.5 deg, a fixation could be considered to have landed on the target item if it fell within 1.25 deg from the target center. Because there is some error in the eye tracking, we set the expanded the region of interest around the target to 1.5 deg from the center of the target item. In addition, to eliminate saccades that landed on the target but did not lead to target identification, the targeting saccade had to fall within five fixations of the end of the trial. This five-saccade window allowed for observers to fixate on a target and then think about it for a moment before responding and/or to find and move the cursor, often making a few other saccades in the process. In fact, in Experiment 1, over 70% of targeting saccades were the final saccade for that target. Modest variations in the rules (e.g. using a 1.25 deg radius around the target rather than 1.5 deg) do not markedly change the pattern of results. These rules produce targeting saccades on about 90% of the 500 trials per observer. The other 10% are lost due to tracking errors and, perhaps, due to rare cases where observers responded without having the eyes within 1.5 deg of the target.

We defined all saccades prior to the targeting saccade to be search saccades. The distribution of search saccade lengths is very similar for all saccade positions relative to the targeting saccade. That is, for example, there is no evidence that observers make saccades that get smaller and smaller as they approach the target. For post-target saccades, we used all saccades that occurred after the designated targeting saccade.

Figure 4 shows saccade length histograms using bins that are 1 deg wide. Results are presented as proportions for each type of saccades separately so as to allow functions to be compared against each other (For each type of saccade, the proportion was calculated by the number of saccades at each saccade length divided by the total number of saccades). In raw numbers, there are, of course, many more search saccades than targeting saccades since you can only have one targeting saccade per trial. Distributions were created for each of the 18 observers and averaged. Dashed lines show the histograms for individual observers. Thicker, solid lines show average data. The distributions for different types of saccade are significantly but not dramatically different. Average saccade length is shorter for targeting saccades (3.72 deg) than searching saccades (4.41 deg, t(17) = 9.3, p < .0001). Using the Kolmogorov-Smirnov test, the search saccade and targeting saccade distributions are significantly different (p = .0002, Kolmogorov-Smirnov D = 0.91).

Figure 4:

The proportion of saccades as a function of saccade length (in degree) in Experiment One. Dashed lines show functions for each observer. Solid lines show the average function. Post-target saccades are all saccades that occurred after targeting saccade.

We can visualize the spatial layouts of the FVFs by placing the endpoints of each saccade at the origin to see where the saccades come from. That is, where did the eyes move from to reach the current point of fixation? In order to visualize the FVFs, we counted the number of saccades landing in each 0.25 deg × 0.25 deg square in a 10 deg window surrounding fixation. Thus, Figure 5 shows a 2D version of the histograms of Figure 5. The FVFs do not have hard borders, beyond which there are no saccades. To show a somewhat arbitrary but illustrative FVF border, we can display the smallest set of the 0.25 deg × 0.25 deg regions that contains N% of all fixations. In Figure 5, we pool data from all 18 observers and use a 75% threshold. A region is colored red if it contains at least 2.5% of all saccades. Colors move linearly through yellow and green to blue as the percentage of saccades decreases.

Figure 5.

2D histogram showing proportion of saccades originating at each location in the field relative to the saccade endpoint. Color codes the proportion on the same scale in all figures. Red denotes the spots with the greatest proportion of saccades. The set of all colored pixels show the smallest area that includes 75% of all saccades. Thus, for example, it can be seen that search saccades, defining the Exploratory FVF, come from a somewhat wider area around the current point of fixation than targeting saccades, defining the Attentional FVF.

A number of features of these visualizations of the FVFs are worth noting. First, these FVFs are elongated along the horizontal axis. Second, the area containing 75% of saccades is 1.45 times larger for the Exploratory (Search) FVF than for the Attentional (Targeting) FVF. Third, the Post-target saccades have a strong representation near the original point of fixation. That is, they are mostly small, re-fixating saccades. These are accompanied by a scatter of much longer saccades. We suspect that, in some cases, observers fixated the target, but before firmly concluding that the item was the target, they produced what was, in effect, a search saccade, before realizing that they needed to go right back to where they had just been. A similar scenario has been reported in previous studies (Godwin et al. 2017; Steinberg & Logothetis, 2001). Fourth, the ‘hot spots’ in the search and target distributions correspond to the locations of the items nearest to the current fixation in the regular search array used here. It is interesting that the search saccades favor those locations and not more remote items. The oval shapes of the FVFs in Figure 5 are reminiscent of the shape of the visual field and of acuity isopters. However, this deviation from circularity appears to be driven by the shape of the stimulus field, not by the properties of the visual system. We suspect that the search and targeting saccade distributions would be different with a differently shaped and/or less regular search display was used. Indeed, in a similar study of FVFs involving radiologists viewing mammograms that are taller than they are wide, the resulting FVF estimates are vertically elongated (Wolfe, Wu, Li, & Suresh, 2021).

Histograms of the angular distributions of pairs of saccades give further insight into observers’ behavior. Figure 6 shows the distributions of those angles calculated as the unsigned angles formed by the pair of saccades. Saccades of 0 deg mean that the second saccade goes back in the direction that the previous saccade had come from. Saccades of 180 deg mean the second saccade continues in the direction of the first. If the second saccade turns left or right, these data preserve only the angle, not the direction. As shown in Figure 6, search and targeting saccades are biased in the forward direction, consistent with ‘saccadic momentum’ (Wilming, Harst, Schmidt, & Konig, 2013). Post-targeting saccades are biased toward return saccades (0 deg). As noted, we would presume that the target was found but the observer only realized that it was the target after moving away from that location. A similar result was also reported in Horstmann et al. (2017).

Figure 6.

Histograms of the angles formed by successive saccades. Angles of 180 indicate saccades moving in the same direction. Angles of zero indicate saccades returning toward the N-1 fixation point. Bar height shows the proportion of total saccade angles in each bin.

Can the FVFs explain search performance?

In the visual search version of the experiment, the same 18 observers had an average error rate of 11% on target present trials (misses, 13% in set size 80 and 9% in set size 42) and 4% on target absent (where 2% were false alarms, and 2% exceeded an 8 second time limit). The slopes of the reaction time x set size function were 20 msec/item for target present trials and 39 msec/item for target absent trials. This is quite typical of a standard serial, self-terminating search as seen in prior data. Here, our interest is in the relationship of the FVF results to the search data. What size of an Attentional FVF is consistent with the search data? Specifically, we assume that observers need to search, on average through (N+1)/2 items before finding the T. Observers find the T on 87% of trials in the search task. What size FVF would be needed to produce that 87% average HIT rate in the set size of 80 that we see in the data.

To estimate this size, it is first important to realize that the Attentional FVF is not homogeneous. When the target is within the attentional FVF, it is not guaranteed that this target will be found and fixated by the next saccade nor is the probability of finding the target uniform across the field. Using the eye tracking data from the foraging experiment, we computed the probability that the next saccade would fixate on the target as a function of the distance from the fixation to the target. (the probability was calculated by the number of saccades going to the target next divided by the total numbers of saccades at the same distance from the target). These data are shown in Figure 7B. Note that, even for a target 2 deg away from the current fixation, there is less than a 50% probability that the next fixation will go to that target. Figure 7C shows that the probability rises only to 70% when we ask if any of the next 3 saccades goes to the target. Note that the use of three saccades here allows for the possibility of a corrective saccade. That is, it is possible that saccade 1 was programed away from the true target location before the target was identified. Thus, it would take additional 1–2 saccades to get back to that identified target.

Figure 7.

A) The simulation process. B) The probability that the next saccade goes to the target as a function of eccentricity in the foraging task. C) The probability that one of the next three saccades would land on the target as a function of eccentricity in the foraging task. Notice that the target was defined as fixated when the eyes landed within 1.5 deg from the target. Thus, the function is plotted from 2 to 10 deg.

This probability density function is, itself, an estimate of the Attentional FVF. However, some unknown percentage of saccades to the target (especially from far away) will hit the target by luck. In the simulation cartooned in Figure 7A, we truncate the PDF at different distances from fixation and ask how what radius of FVF is needed to account for the 87% HIT percentage.

Referring to the numbers in Figure 7A, the simulation ran as follows.

1. The scan path is taken from a true negative trial in the search task data having a set size of 80.

2. A target present stimulus with set size of 80 is randomly generated and the scan path from that true negative trial is run over this target present display.

3. Each fixation creates a surrounding Attentional FVF. Items inside the Attentional FVF are examined according to the PDF in Fig 7C and if the target is “found”, the trial is declared to be a simulated HIT. Thus, if the target is 3 deg from fixation, based on Fig 7C, there would be ~60% chance that the target would be found.

4. If no fixation lands on the target before all fixations in the current scan path are used, the trial would be a simulated MISS.

5. If the target is outside the attentional FVF, it would not be attended. The function shown in Fig 7C is a continuous function that extends far into the periphery. As noted, some fraction of those saccades, landing on the target, will occur by chance. In order to gain an estimate of the effective range of the attentional FVF, we calculate the probability of finding the target with the 7C function truncated at different eccentricities from 2 to 10 deg. The goal of this exercise is to determine the FVF radius that would produce a HIT rate similar to the HIT rate for set size 80 in the visual search task. Note, however, that the probability for the trial increases if the target lands in the FVF multiple times. This, of course, becomes more likely as the simulated FVF gets bigger.

Figure 8 shows the model’s HIT rate as a function of FVF sizes. This result suggests that the FVF with 8 deg radius would produce a HIT rate similar to that produced by the observers. This FVF may seem quite large, but it is important to remember that only a very small proportion of the saccades from 7 or 8 degrees will be directed to the target. Most targeting saccades will come from closer to the target as shown in Figures 7B&C. This is reflected in Figure 5A where we see some, but only a few saccades coming to the target from 8 degrees away (at least on the horizontal axis). It is also worth noting that a fairly substantial ‘error bar’ should be placed around that 8 deg value. Looking at Figure 8, we might be better saying that the Attentional FVF has a radius in the 5–10 deg range. Much smaller would be inconsistent with the error data.

Figure 8.

The HIT rate of the model as a function of FVF radius.

Using the estimated 8 deg FVF, we can calculate the number of fixations that the model requires for average HIT and MISS trials (Figure 9A). Again, we superimpose a target absent scanpath on a target present stimulus and calculate how many fixations are required before the model has “found” the target by getting that target inside the Attentional FVF and then detecting it using the probabilistic function in Figure 7C. As Figure 9A shows, the simulated results are very similar to the real data (7.36 fixations for the model and 7.44 for observers on HIT trials; 15.11 fixations for the model and 14.37 for Observers on the Misses trials).

Figure 9.

A. The number of fixations used for the model and observers in search with set size 80. B. The coverage of display during the search. Error bars are ± SEM.

We can also calculate how much the display is covered by the FVF. That is, how many items in the display would be processed on HIT, MISS, and True Negative trials. To determine this, we turn again to the function in Figure 7C. For each fixation, there is some probability that each item will be processed. That probability declines with distance from fixation and is zero outside of the FVF. The probability for a specific item goes up each time that item falls in the current FVF. We assume each fixation was independent from each other for simplicity of calculation. Thus, if one fixation produced a probability of 0.4 that information of an item would be processed, and if another fixation produced a probability of 0.3 for the same item, the final probability of that item being processed would be 1-(1-.4)*(1-.3) = 0.58. If this is repeated for all fixations in a scan path in both real search and simulation, then the coverage can be calculated as the average probability of the 80 items. This should be about 50% for target present trials and it is (Figure 9B). The model produces a slightly lower coverage than Observers in the HIT trials (0.51 for the model and 0.56 for Observers in set size 80, t(34) = 2.73, p < .01). For MISS trials, the model and humans produce similar results (0.85 for the model and 0.82 for observers, t(33) = 1.16, p = 0.26).

Arguably, the most interesting aspects of these data are the functions shown in Figures 7B & C. Even when the target is just two deg away from the current fixation, there is only about a 50% chance that the target will be fixated after the next saccade (7B). The chance rises only to 70% over the next three saccades (7C). This strongly argues against any unlimited capacity parallel processing model in which every item in the FVF would be fully processed. We would argue that the Attentional FVF describes a region within which items are sampled during a fixation (Wolfe, 2021). It is also possible to argue for parallel processing of all items within the FVF (Hulleman & Olivers, 2017) as long as one allows for incomplete processing. For instance, Hulleman & Olivers (2017) argue for a form of parallel ensemble processing of all items in the current FVF. These serial and parallel accounts are not as different as they sound. In our data, the serial account proposes that an item four deg from fixation would have a 20% chance of been selected and identified during a fixation. The parallel account would propose that the item would receive enough attention to permit identification 20% of the time. Moreover, there are intermediate possibilities between the serial and parallel accounts (Liesefeld & Mueller, 2020). Regardless of the precise account, the results make clear that the observer may fixate at a location where target detection is perfectly possible but they just don’t detect it. This can be considered a form of inattentional blindness (Mack & Rock, 1998; Most, Scholl, Clifford, & Simons, 2005; Simons & Chabris, 1999). The target is visible, but it does not generate a response.

In the medical image perception literature (Kundel, Nodine, & Carmody, 1978; Nodine & Mello-Thoms, 2010), the errors where a visible target was not reported have been divided into three categories on the basis of eye movement records. “Search” errors are those where the fixation point never landed within some region of interest around the target. In “Recognition” errors, the eyes visited the target for less than 500 msec. Finally, “Decision” errors are said to occur when the eyes spend more than 500 msec fixed on or near the target but where the scrutinized item is still not categorized as a target. In the present experiment, our target was an uncomplicated letter T. Therefore, any miss errors during search would likely be search or recognition errors, rather than decision errors.

Is there any pattern to the targets that are found versus those that are not found in a single fixation? To answer this, we take all fixations that fall within some distance to the target. Here we will use a range from 1.5 to 4.5 deg. A distance less than 1.5 deg would be considered a fixation with the ROI around the target. An outer limit of 4.5 deg fits the distribution of the bulk of the targeting saccades shown in Fig 5. That set of fixations can be divided into two groups: Those where the next fixation goes to the target and those where it does not. For each of those groups, we can measure the distribution of angles from fixation to the target and we can ask if the groups differ.

As can be seen in Figure 10, observers are more likely to find the target if it falls to the left or right of the current fixation. This probably reflects a tendency to ‘read’ these displays along a row. Note, however, that the green distributions of other fixations show only a slight bias toward the horizontal. Observers moved their eyes in all directions, they were apparently more like to deploy their covert attentional resources to the left or right to find a target. The differences between the two distributions were significant by a chi-sq test comparing the 12-value red and green functions for every observer (Chi-sq(23) > 33, p < .001 for all cases).

Figure 10.

Radial distribution of the direction to the target from fixation. Red lines show the distribution of those fixations immediately preceding a target fixation. Green lines show the distribution of other fixations where the next saccade did not go to the target even though they fell within the same range of distances from the target (1.5 to 4.5 deg).

Experiments 2 & 3 - Replication with different search tasks

The results in Experiment 1 show that the FVF can be defined in several ways. Even for a simple task, the FVF is not a simple measure. Search saccades define an Exploratory FVF that is larger than the Attentional FVF defined by the targeting saccades, while basic visual constraints define a resolution FVF. The Exploratory and Attentional FVFs are probabilistic, not regions with sharp boundaries. Moreover, while a target that falls within the attentional FVF can be found, in principle; there is a less than 50% chance, in this task, that it will be found on the next fixation. To assess the generality of these results and claims, we replicated the basic study with a “guided” conjunction search (Exp 2) and a harder version of the search for a T among Ls (Exp 3).

Method

Fourteen observers (10 females) were tested in Experiment 2 (conjunction) and another sixteen observers (10 females) were tested in Experiment 3 (hard T vs L). Each observer only participated one experiment (Experiment 1, 2 or 3) in the current study. Observers in Experiment 2 ranged in age from 22 – 55 years and observers in Experiment 3 ranged in age from 19 – 55 years. All observers had normal or corrected-to-normal vision and passed the Ishihara color screen. They gave informed consent approved by the Brigham and Women’s Hospital IRB and were paid $11/hour. Two observers in Experiment 2 and four observers in Experiment 3 were excluded due to the low eye movement tracking ratio. We report data from twelve observers in each experiment.

Sample stimulus displays used in Experiments 2 & 3 are shown in Figure 11. Experiment 2 used a conjunction search display in which the target was a red vertical bar embedded amongst red horizontal and green vertical bars (Fig 11, left). Experiment 3 used a more difficult version of the T vs L stimuli of Experiment 1. In Experiment 3 each letter was made by offsetting the two strokes of the Ts and Ls so that the targets and distractors became more similar to each other (Fig 11, right). For both experiments, the size of each item and the separation between any two adjacent items were identical to Experiment 1. The experimental procedures were identical to Experiment 1. First, in the foraging phase of the experiment, observers were asked to collect 500 targets (red verticals in Exp 2, hard Ts in Exp 3). After that, they completed 200 trials of a visual search task with set sizes of 42 or 80 and 50% were target present trials.

Figure 11. Stimuli for Experiment 2 (Conjunction) and 3 (Hard T among Ls).

Result and Discussion

Figure 12A shows saccade length distributions in Experiment 2. Figure 12B shows the probability that the next saccade goes to the target as a function of the target distance from the current fixation. Figure 12C shows the 2D distributions of starting points of saccades, relative to a common endpoint.

Figure 12.

Results of Experiment 2: A) saccade length distributions. Dashed lines show individual observer data. B) probability that the next saccade goes to the target as a function of the distance to the target from the current fixation. C) 2D distributions of starting points of saccades, relative to a common endpoint.

Results are broadly similar to the results for Experiment 1 with some interesting differences. The search saccades are longer on average (5.05 deg) than the targeting saccades (4.07 deg, t(11) = 13.83, p < .0001). The search and targeting distributions differ significantly (Kolmogorov-Smirnov D = 0.83, p = 0.013). Both search saccades and targeting saccades in the conjunction task of Experiment 2 are longer that those in the T vs L task in Experiment 1 (Tukey’s multiple comparisons test: Search saccades: q(39) = 5.95, p(corrected) = .0004, Targeting saccades: q(39) = 5.15, p(corrected) = .0022). Turning to the probability that the next saccade goes to the target (Figure 12B), notice that, when fixation is 2 deg away, the next saccade goes to the target on 2/3 of the cases, as opposed to less than 50% of cases in Experiment 1. An ANOVA with Experiment (TvL vs Conjunction) and Distance to Target as factors shows a large effect of Experiment (F(1,280) = 187.4, p < .0001). Of course, there is a large effect of Distance to Target (F(9,280) = 85.4, p < .0001). A significant interaction shows that difference between the functions from the two experiments is not additive (F(9,280) = 7.9, p < .0001).

The differences between the results can be understood as a result of the task difference. The conjunction task in Experiment 2 is a ‘guided search’ task (Wolfe, 1994, 2007). While all the items within the FVF could be targets in the TvL task, attention will be guided by color and orientation to prioritize some items in the conjunction task. Indeed, if there was no noise in the guidance process, attention would be guided directly to the red vertical target as the only item with both “red” and “vertical” features. In the context of our probabilistic model as shown in Figure 7, attention should be guided to a promising subset of the items near fixation. This could have and does have two effects. The probability that the next saccade will go to the target is higher in Experiment 2 (Figure 12B vs Figure 7B) and observers can forage further afield leading to a somewhat larger FVF. It is interesting that observers do not devote all their effort to improving detection within a smaller FVF.

Turning to Experiment 3, where the TvL task is made more difficult, the results go in the opposite direction from Experiment 2 when comparing both to Experiment 1.

As can be seen in Figure 13A, the search saccades are, again, longer on average (4.25 deg) than the targeting saccades (3.99 deg), though the effect is not strong (t(11) = 2.52, p < .05). The search and targeting distributions differ significantly (Kolmogorov-Smirnov D = 0.83, p < .0001). The search saccades in the Hard T vs L task of Experiment 3 are shorter on average than those in the T vs L task in Experiment 1, but this is not statistically significant (Tukey’s multiple comparisons test: search saccades: q(39) = 2.19, p(corrected) = .2796, targeting saccades: q(39) = 2.92, p(corrected) = .1114). Turning to the probability that the next saccade goes to the target (Figure 13B). In Experiment 3, when fixation is 2 deg away, the next saccade goes to the target on only 1/3 of the cases. An ANOVA with Experiment (TvL vs Hard TvL) and Distance to Target as factors shows a large effect of Experiment (F(1,280) = 15.5, p < .0001). Again, there is a large effect of Distance to Target (F(9,280) = 130.7, p < .0001). A significant interaction shows that Experiments 1 & 3 produce functions differ by something other than a constant term (F(9,280) = 2.9, p < .0028).

Figure 13.

Results of Experiment 3: A) saccade length distributions. Dashed lines show individual observer data. B) probability that the next saccade goes to the target as a function of the distance to the target from the current fixation. C) 2D distributions of starting points of saccades, relative to a common endpoint.

In Experiment 3, it probably takes somewhat longer to determine if an item is a T or an L than in Experiment 1, but their average fixation durations were similar (247.57 msec in Exp 1; 249.6 msec in Exp 3). As a result, fewer items can be attended to at each fixation. As a consequence, the chance that an item, inside the FVF, will be identified as the target goes down.

Our results show that both the Exploratory FVF and Attentional FVF, as measured by the length of saccades, were slightly larger in conjunction search task (Figure 12C), but they did not change much when a more difficult in T vs L task was used (Figure 5 vs. Figure 13C).

As in Experiment 1, we can see a bias toward attending to items lying to the left and right of the current fixation. Figure 14 shows the distributions of directions to the target from the current fixation for fixations where the next saccade went to the target (red) and fixations where the next saccade went elsewhere (green).

Figure 14.

Radial distribution of the direction to the target from the current fixation. Red lines show the distributions of those fixations immediately preceding a target fixation. Green lines show the distributions of other fixations where the next saccade did not go to the target even though they fell within the same range of distances from the target (1.5 to 6.5 deg in Exp 2 and 1.5 to 4.5 degin Exp 3).

For Experiment 2, we used the set of fixations between 1.5 and 6.5 degrees from fixation, reflecting the larger FVFs in that condition. Using an outer limit of 6.5 degrees allowed us to incorporate roughly the same proportion of fixations as used in Experiment 1 (Figure 10). In Figure 14A, the average distributions for saccades going towards the target (red) or elsewhere (green) are significantly different (Chi-sq(23) = 170.2, p < .001). Though this effect is somewhat weaker than in Experiment 1. Only 3 of 12 Observers show distributions that differ (p < .05). The result provides marginal evidence that the horizontal bias is not limited to letter stimuli. For Experiment 3 (figure 14B), we used the set of fixations between 1.5 and 4.5 degrees from the target. Again, the average distributions are significantly different (Chi-sq(23) = 177.9, p < .001. Here 9 of 12 observers show functions that differ (p < .05). As before, the conclusion to be drawn from this analysis is that simply falling inside of the FVF does not guarantee that a target will be successfully detected. In this task with its regular rows of stimuli, there appears to be a bias to select or process more fully items on the left and right sides of the fixation than items lying above or below the fixation even though those other stimuli fall in the FVF and could be selected and identified.

Discussion

The basic idea of a Functional Visual Field (or Useful Field of View - UFOV) is straight-forward enough. We cannot fully process all of the visual field at once. One way of capturing this limit is to propose a FVF around the point of fixation within which some sort of processing occurs and outside of which, it does not. For instance, B. Wolfe et al. (2017) quote Andersen as saying, “Any information that falls within the UFOV is processed whereas any information that falls outside of this region is not processed.” (Andersen, Ni, Bian, & Kang, 2011). The results of the present experiments make it clear that the FVF is more complicated than this. First, we see and process stimuli across the visual field outside of the Attentional FVF (e.g. Bronfman, Brezis, Jacobson, & Usher, 2014; Larson & Loschky, 2009; Wolfe et al., 2017). Second, it has always been clear that any FVF must depend on the nature of the task and stimulus. The present experiments illustrate this point with basic visual search tasks. The effect of task on the FVF is more apparent and more important in real-world search tasks like those in medical imaging (Wolfe et al., 2021). For example, the FVF will be different if you are looking for relatively large lesions in a relatively uncluttered liver CT or small calcifications in a noisy mammogram (Lago, Sechopoulos, Bochud, & Eckstein, 2020).

In this paper, we make the point that the definition of the FVF in visual search depends on what aspect of the search process is under discussion. For example, the Lago et al. (2020) paper is concerned with the Resolution FVF. If the observer is fixated at one location, how far away can a target be identified, given that the observer knows where to attend? That is related to, but not the same as an Attentional FVF, the FVF that describes the region within which a target can be found when the observer does not know where to attend. In this paper, we have based the definition of the Attentional FVF on saccades that go from the current fixation, directly to the target. Though the eyes could move to the target by chance, it is more likely that a saccade to the target means that the target was provisionally identified from the location of the current fixation. Under some circumstances, that provisional identification may be quite incomplete. In a search for a specific child on the playground, your attention could go to a child-like item, well outside the Resolution FVF. Once fixated, that ‘item’ might or might not prove to be the right child. The Attentional FVFs, based on that reasoning, are shown in Figures 5, 12C,& 13C.

If the target is not identified from the location of the current fixation, the next eye movement moves the eyes and the FVF somewhere else to sample new items. These search saccades define an Exploratory FVF (shown in the same figures). In the current experiments, those search saccades are somewhat larger on average than the targeting saccades that define the Attentional FVF. We do not propose to artificially reify these three types of FVF. For example, we would not want to propose that each FVF has a specific brain locus. They are three, logically distinct ways of thinking about what processing occurs during search.

All of the varieties of FVF discussed here are probabilistic in nature. It is a mistake to think of the FVF as having a hard boundary where some processing occurs inside the boundary and ceases immediately outside that boundary. Even detection data, defining the Resolution FVF would show a transition from perfect performance to guessing as a function of distance from fixation. Statements about an FVF having a diameter of X degrees should be thought of as a threshold on a psychometric function. Figures 7, 12, and 13 show that this function can be quite shallow.

The use of different types of saccades to define the Attentional and Exploratory FVFs becomes less convincing when the stimuli become more complex. In a parallel study of radiologists’ eye movements while searching for cancerous masses in mammograms, we found that a significant percentage of targeting saccades were larger than would be expected on the basis of the story given above (Wolfe et al., 2021). Experts were making large eye movements to small regions of interest around a target from a distance from which they could not have covertly attended and identified that target. That is, the target would have been outside the Resolution and Attentional FVFs. To see the solution to this problem, consider a search for apples in an orchard. From a considerable distance, you might not know if there are good apples on a specific tree, but you might move yourself to that tree, knowing that, if there are apples, that would be the best place to look. In an inhomogeneous stimulus like a mammogram, experts made saccades to places where masses might be expected. If they were wrong, that saccade would be a search saccade. If they happened to be correct, that long saccade could turn out to be classified as a targeting saccade to a target that could not actually have been resolved or even specifically attended as an item from the starting place of that saccade.

We can rephrase this problem by noting that a saccade to a target could have gotten there because the target had been covertly identified or because the observer made a lucky saccade to a plausible location. In a homogeneous, regular display like those used in the present experiments, those lucky saccades would be based on no expertise beyond knowing the difference between empty space and an item in the search array. A different way to look at the Attentional FVF is to ask about the probability that the eyes will go to the target, given that they are at some specific distance away from the target. Figures 7B, 12B, and 13B show this analysis. The striking result is that, even when the target is quite close to the current fixation, the probability is nowhere near 100%. This result shows that falling inside the FVF is not a guarantee that a target will be found. We would argue that, in the tasks studied here, the finding reflects observer sampling a subset of items within the FVF rather than processing every item. For example, even using a 4.5-degree radius (as suggested in some of the radiology literature), the FVF would contain about 12 items in our stimulus. The slope of the reaction time x set size function in Exp 1 suggests that items are being processed at a rate of 40–50 msec/item.

At this rate, given the average of 250 msec fixation duration in the current study, the observer could process fewer than half of the available items during the fixation. The precise numbers are not critical here. The point becomes more dramatic if one considers all the items inside our estimated 8 deg probabilistic FVF. What matters is that no plausible serial model would allow processing of all the items covered by a plausible FVF. Observers find the target because their scan paths can place items inside the FVF multiple times during a search. Note that we are not arguing that one item takes 40–50 msec to be fully identified. Evidence suggests that identification takes longer. We would argue that items are selected into a pipeline in which several items can undergoing identification at the same time (Wolfe, 2003, 2021).

Other models would argue that all the items in the FVF are being processed in parallel. (Alwis & Haberman, 2019; Hulleman & Olivers, 2017; Palmer, Verghese, & Pavel, 2000; Young & Hulleman, 2013). That would be consistent with the data if those models do not require complete processing of all items in the FVF. As mentioned above, the serial and parallel accounts are not as different as they may appear. In the serial selection account, observers have time to serially select only N of the items in the FVF. In a plausible parallel account, observers would have time to complete processing of N of the items in the FVF. Hybrid possibilities exist. Perhaps observers select N of the items in a single clump. These might then be processed fully in parallel (Liesefeld & Mueller, 2020). Another alternative is that observers may take all items within FVF and compute a summary or ensemble statistic as the basis of a present/absent decision (Hulleman & Oliver, 2017). The important point is that a target of search can be missed even when the eyes are pointed at or near the location of that target (Comparable neurophysiological evidence is found in Sheinberg & Logothetis (2001)). This is not a complete account of phenomena like inattentional blindness as observers are looking for the specified target (Castel, Vendetti, & Holyoak, 2012; Cohen, Cavanagh, Chun, & Nakayama, 2012; Mack & Rock, 1998; Wolfe, 1999), but it does help to explain how observers can miss clearly visible stimuli.

If the observers are not processing everything in the FVF, is there any pattern to what they process? Clearly, the answer is ‘yes’. Observers are more likely to find a target that is close to the current fixation. Moreover, as shown in Figures 10 and 14, the observers are more likely to find the target in these experiments if it lies to the left or right of fixation than if it falls in some other directions. This finding could be related to the orderly, grid-like structure of these stimuli which might have encouraged observers to search along rows as if ‘reading’ the display (Bertera and Rayner, 2000; Rayner, 2009). However, a horizontal bias to saccades can be also found in search tasks with items in more random arrays (Kamienkowski, Ison, Quiroga, & Sigman, 2012). We did not find the same bias when we performed this analysis on the data from radiologists’ search in mammograms even though, like the observers here, radiologists could have their eyes positioned very close to the target without, apparently, detecting it during that fixation (Wolfe, Wu, Li & Suresh, 2021). It is possible that the interaction of specific search tasks with specific searchers produces idiosyncratic patterns of covert attention during fixation, but evidence for such a hypothesis would require more extensive data collection.

In sum, the present results illustrate the multifaceted nature of the seemingly simple idea of a functional visual field. These results suggest that there could be three different probabilistic ways to think about the FVFs involved rather than a single fixed FVF during the search. The results also shed light on how observers miss clearly visible targets. Targets can be missed even when fixation is nearby because falling inside the FVF does not guarantee complete processing of an item, just as falling outside the FVF does not mean that the item was completely invisible.