Specifying the Precision of Guiding Features for Visual Search

Abstract

Visual search is the task of finding things with uncertain locations. Despite decades of research, the features that guide visual search remain poorly specified, especially in realistic contexts. This study tested the role of two features—shape and orientation—both in the presence and absence of hue information. We conducted five experiments to describe “preview-target mismatch” effects, decreases in performance caused by differences between the image of the target as it appears in the preview and as it appears in the actual search display. These mismatch effects provide direct measures of feature importance, with larger performance decrements expected for more important features. Contrary to previous conclusions, our data suggest that shape and orientation only guide visual search when color is not available. By varying the probability of mismatch in each feature dimension, we also show that these patterns of feature guidance do not change with the probability that the previewed feature will be invalid. We conclude that the target representations used to guide visual search are much less precise than previously believed, with participants encoding and using color and little else.

Characterizing the feature dimensions involved in guidance—the non-random direction of your attention and gaze to locations likely to contain visual search targets—has been a topic of interest in the attention literature for decades (e.g. Williams, 1966). Attempts to identify the features of the target template—the mental representation of the features of the search target—have relied on a variety of criteria. Unfortunately, difficulties in interpreting patterns of data and a lack of consensus in the criteria that should be used has led to the conclusion that only four particularly well-studied features (color, motion, orientation, and size) are at best “probable” sources of search guidance (Wolfe & Horowitz, 2004; Wolfe & Horowitz, 2017). From among the potentially infinite features to test, based on converging evidence the field has some assurance that the features considered in the present study are effective in guiding search.

From Reaction Times to Eye Movements

Several serious challenges contributed to this tight focus on only a few guiding features. For one, previous efforts in characterizing the guidance of attention primarily relied on inference and indirect measures, most commonly search slopes. Such approaches—although informative—proved unable to distinguish between “parallel” and “serial” patterns of performance as clearly as researchers originally hoped (Palmer, 1995; Townsend, 1990; Wolfe, 1998), due in large part to there being no single mechanism that completely determines reaction time (Zelinsky & Sheinberg, 1997). Relatedly, the conservative criteria for choosing guiding features were introduced largely to exclude the many non-guidance factors that can drive performance in visual search tasks, such as: experimental artifacts (Donk, 1999, 2001), guessing strategies (Johnston & Pashler, 1990), feature misperceptions (Ashby, Prinzmetal, Ivry, & Maddox, 1996), and post-perceptual factors (Navon & Ehrlich, 1995).

In response to the ambiguities of RT as a dependent measure, researchers began measuring eye movements during search (Zelinsky & Sheinberg, 1997). While it is possible for attention to be directed to spatial locations without executing an eye movement (Klein & Farrell, 1989; Murthy, Thompson, & Schall, 2001), shifts of attention necessarily precede eye movements once those movements are executed (Deubel & Schneider, 1996; Peterson, Kramer, & Irwin, 2004; see Findlay & Gilchrist, 2003, for additional discussion). This relationship between gaze and attentional shifts allows for direct tests of guidance. For example, if the time taken by participants to fixate on a target depends on the presence or absence of a given feature in the visual periphery, this demonstrates that gaze (and attention) were guided to the target’s location using that feature. These kinds of eye movement measures help to minimize the many sources of variability embedded within RT measures: A feature may be said to guide search if participants direct their gaze using that feature—regardless of how long it takes them to make the button press response after the target is fixated (Alexander, Schmidt, & Zelinsky, 2014; Alexander & Zelinsky, 2018; Castelhano, Pollatsek, & Cave, 2008).

Realistic Contexts

While the field was once dominated by the use of simple patterns as stimuli, the search literature has since adopted more natural contexts and visually-complex stimuli such as arrays of common objects and computer-generated or fully realistic scenes. In these more natural contexts, the features that guide search are even less understood. It was therefore important to show that core findings from earlier search experiments generalize beyond simple displays. This was largely true for some classic visual similarity effects, which were found to replicate in the context of real-world stimuli (Alexander et al., 2014; Alexander & Zelinsky, 2011, 2012; Yang, Chen, & Zelinsky, 2009; Zelinsky, 2008). Still, the extent that basic features guide visual search in realistic contexts (which may be processed differently than simple stimuli) has not been systematically explored.

The fact that some feature dimensions can guide search does not mean that those features actually do guide search. Most work exploring potential guiding features has tested whether those features were available early enough after the onset of the search display that they could be used to guide search (Wolfe & Horowitz, 2004; Wolfe & Horowitz, 2017). This approach is important, in identifying those features that are incapable of guiding search, but this approach cannot identify what features are generally used even when the task could be performed without them. Even if a feature dimension is identified successfully as one that is detected and processed early enough to direct attention, and researchers demonstrate that the feature is available to search processes, that does not mean that people actually use that feature in everyday search tasks. When two or more features are available to search—as with realistic contexts—it is still largely unknown what features are used for guidance to common object categories.

The Current Experiments

The current study provides both a direct test of guidance and a means of testing feature importance while having multiple features available. This is accomplished by systematically varying the differences between the preview of the target and the search target as it appears in the actual search array (see also Alexander & Zelinsky, 2018; Bravo & Farid, 2009; Hout & Goldinger, 2015; Vickery, King, & Jiang, 2005). In this “preview-target mismatch” paradigm, mismatch is created along a single feature dimension, enabling the parametric study of how much that dimension guides search. Using a common object with few features as an example, if you are searching for a red apple when only a green apple is present, poor performance would indicate that you are using color information. It also follows that if the color, shape, or orientation of the apple changes but performance stays the same, then that information is not being used. Unlike other paradigms where conclusions require that distractors be identical to targets in every way except for the feature being tested, this paradigm is more naturalistic and leaves distractors free to vary along many different dimensions: The apple can be among bananas and pears. Moreover, if the only way to identify the apple is the way that it is tilted (as in other paradigms where targets are defined by a single dimension), you would likely use orientation information, which you probably do not normally use when searching for apples.

Our preview-target mismatch approach is inspired by previous work exploring mismatch effects for a variety of feature dimensions, including differences in 2d and 3d orientation, reflection across the vertical axis, and size (Bravo & Farid, 2009; Hout & Goldinger, 2015; Vickery et al., 2005). Using a similar paradigm as in the current study, this previous work demonstrated that mismatch affects performance, with greater mismatch resulting in longer RTs. However, several aspects of these previous studies make it difficult to draw conclusions about the relative importance of features in guiding search. First, previous studies either evaluated mismatch without evaluating the contribution of individual feature dimensions (Bravo & Farid, 2009; Hout & Goldinger, 2015) or only explored different degrees of mismatch for orientation—not other features—and only with a few large (30°) increments of mismatch (Vickery et al., 2005). Second, because neither Vickery et al. (2005) nor Bravo et al. (2009) used eye-tracking, it is unclear whether any of these effects might be explained—at least in part—by processes that occurred after the target was fixated, rather than reflecting true differences in guidance. Third, both Bravo and Farid (2009) and Vickery et al. (2005) used tasks that may have overestimated the impact of mismatch in the tested feature dimensions by preventing or limiting the use of color to guide search: Vickery et al. (2005) used only grayscale stimuli, and Bravo and Farid (2009) used very colorful background images (coral reef) that likely created a degree of target-background similarity to their target images (sea animals). Hout and Goldinger (2015) used color stimuli but did not test the impact of individual feature dimensions: Stimuli were grouped instead on psychological distance in similarity space along multiple dimensions. This study extends this previous work by manipulating preview-target mismatch along single dimensions (such as orientation) in grayscale and color displays where color is—and is not—available, and by assessing the impact of these mismatch manipulations using eye movement measures.

In Experiment 1, we examine mismatch in the orientation dimension. We decided to test orientation (rather than some other dimension) for several reasons: First, orientation has been well-established as a dimension that can guide search, meeting all of the criteria previously used to identify guiding features (Wolfe & Horowitz, 2004). Second, we believed that orientation might not actually be used as much as previous evidence has suggested (e.g. Vickery et al, 2005), mainly because of search guidance likely being dominated by color (Hannus, van den Berg, Bekkering, Roerdink, & Cornelissen, 2006; Hwang, Higgins, & Pomplun, 2009; Rutishauser & Koch, 2007; Williams, 1966). Previous mismatch studies presented stimuli in either grayscale or with very colorful backgrounds, thereby potentially overestimating the importance of orientation in guiding search. Our Experiment 1 examines orientation mismatch both when color information is available to guide search and in a grayscale condition where color is not available. We predict finding orientation mismatch effects using grayscale stimuli, consistent with Vickery et al. (2005), and that these orientation mismatch effects will be less pronounced (or even non-existent) in a color condition.

Experiment 2 extends this work to testing mismatch on a previously-unexplored feature dimension: Aspect ratio, manipulated as a surrogate for altering the shape of real-world objects. Changes in aspect ratio alter many local shape features, such as the curvature and the length and the width of individual parts of objects, making it likely that we will find an effect of aspect ratio mismatch if shape is used to guide search. We again evaluated feature mismatch in both color and grayscale contexts to test our hypothesis that aspect ratio might be affect guidance in a grayscale condition but not in a color condition due to the dominate role that color plays in guiding search.

In each of these experiments, we also extend previous methodologies for studying target guidance, such as the use of sensitive eyetracking measures to determine the features and contexts in which mismatch affects guidance. Experiments 1 and 2 explore a wide range of mismatch differences to better characterize these effects. Where previous work either did not test whether the degree of mismatch affected performance (Bravo & Farid, 2009), or tested only large (30°) increments of orientation (Vickery et al., 2005), the current experiments test 25 different increments of mismatch. For example, for orientation we explore the full range of rotation—from 0–360°—allowing the maximum effect of orientation mismatch on guidance to be determined. Such a systematic evaluation is important for finding the weightings assigned to each feature dimension during search and for providing parameters for search models.

In a final experiment, we manipulated preview-target mismatch on more than one feature dimension within the same block of trials so as to study how feature information is used to guide search when multiple sources are available. If participants preferentially rely on features which are more likely to be correct, we should find larger mismatch effects in Experiment 3—where those dimensions are invalid on fewer trials. If, however, participants rely on shape or orientation in an automatic fashion, then performance will be equivalent between Experiment 1 (where they know it will be invalid most of the time) and Experiment 3 (where orientation is invalid on fewer trials). We explore this possibility by manipulating both shape and orientation within the same experimental session, but in different trials. As in Hout & Goldinger (2015), we compare the results from this experiment to those from conditions where the same images varied on only a single dimension (Experiments 1–2). When only one dimension changes throughout a block of trials, participants can predict the feature that will change on each trial. To the extent that participants can flexibly bias their use of different features, they should therefore rely less on that mismatched dimension—avoiding the use of that feature because it is highly likely to be invalid. However, to the extent that encoding and use of different features is automatic, hard-coded and obligatory, this change in context should not alter the degree to which participants rely on either dimension. In this eventuality we would predict equally pronounced mismatch effects when participants know which feature dimension will change on a given trial (Exp 1–2) and when the feature mismatch is uncertain (Exp 3). Finding this pattern would suggest that the use of features—other than hue—may be hard-wired and resistant to change; if the features used by search are obligatory in nature, then a given feature cannot be avoided even when it is known to be invalid.

General Methods

Participants

Demographic information is provided in the participants section of each individual experiment. We recruited participants from the undergraduate Psychology participant pool at Stony Brook University. Participants all reported normal or corrected-to-normal vision and normal color vision.

Equipment/Apparatus

Gaze position was recorded using an SR Research EyeLink 1000 eye tracking system with default saccade detection settings. This video-based eye-tracker has a sampling rate of 1000 Hz and is typically accurate within 0.25–0.5°. Initial calibrations were not accepted unless the average spatial error was less than 0.45° and the maximum error was less than 0.90°. Head position and viewing distance were fixed at 70 cm from the screen with a chin rest. The search stimuli were displayed on a flat-screen CRT monitor operating at a resolution of 1280 × 960 pixels and a refresh rate of 60 Hz. Manual responses were made using a GamePad controller.

Stimuli generation

We hand-selected a wide variety of target and distractor images from the Hemera Photo-Objects collection (Gatineau, Quebec, Canada). Several constraints were placed on image choice. To prevent extreme or inaccurate changes to the image that would compromise the aspect ratio manipulation, images that appeared oriented diagonally in the image canvas or ones that were excessively tall or wide (those having a width more than 150% of the height or a height more than 150% of the width) were excluded. Round images (such as balls, pizzas, and plates viewed from above) were excluded, as those images do not have a global orientation. Images with little to no hue information (objects that are mostly black, white, or gray, such as steel kitchen utensils or soccer balls) prevent a meaningful exploration of differences between grayscale and color displays and were therefore also excluded. We therefore selected images with 80% or more of their pixels differing by at least 10 units in color depth on either the red, blue, or green color channel, effectively enforcing color variability in our stimuli. For example, a pixel having RGB values of 244, 250, 255 would meet our color variability requirement, but a pixel with values of 249, 250, 255 would not. Note that these constraints were applied to both target and distractor images.

Feature changes were implemented using MATLAB (7.0.4). Orientation was manipulated by rotating the matrix of pixel values, without interpolation. Orientation mismatch levels in Experiment 1—further described below—were: 0° (“no change”), 5°, 15°, 30°, 45°, 60°, 75°, 90°, 105°, 120°, 135°, 150°, 165°, 180°, 195°, 210°, 225°, 240°, 255°, 270°, 285°, 300°, 315°, 330°, and 345° of change. See Figure 1A for representative examples of stimuli at each level of the orientation manipulation. Angles of rotation are expressed in this manuscript in the counter-clockwise direction, with 0° as the starting value.

Figure 1.

A) Examples of pig and traffic light stimuli at all 25 levels of orientation mismatch used in Experiment 1, and (B) all 25 levels of aspect ratio mismatch used in Experiment 2. Experiment 3 selectively used the “no change” condition; 30°, 60°, 90°, 120°, 150°, 180°, 210°, and 240° orientation-change levels; and 110%, 120%, 130%, 140%, 150%, 160%, 170%, and 180% aspect-ratio-change levels.

Images were rescaled so that each had as close as possible to 6000 non-white pixels in total, with each resulting image ~1.77° of visual angle in size. This was done to normalize for size, while leaving aspect ratio to vary. Aspect ratio was manipulated by downsampling the height of each image before rescaling the images. The result was that the width of aspect-mismatch images increased, and the height decreased. The mismatch levels in Experiment 2—further described below—were: 100% normal shape (“no change”), 102.5%, 105%, 110%, 115%, 120%, 125%, 130%, 135%, 140%, 145%, 150%, 155%, 160%, 165%, 170%, 175%, 180%, 185%, 190%, 195%, 200%, 205%, 210%, and 215% increase in width (with a corresponding decrease in height). These values were selected to prevent the objects from overlapping or appearing too near an outer edge or the center of the display. See Figure 1B for representative examples of stimuli at each level of aspect ratio manipulation.

Target previews were never altered (images were presented as they were photographed), meaning that feature mismatches were restricted to images of objects in the search displays. For each trial, one target and seven distractors were selected randomly from the stimulus set. Targets were always present in the search displays, and their mismatch was relative to the target’s appearance at preview (and the search target at the “no change” level). Feature manipulations performed on the targets differed by experiment: targets in Experiment 1 were altered in orientation, while target aspect ratio was altered in Experiment 2.

Feature manipulations were also performed on each distractor object. This manipulation discouraged participants from strategically searching for objects having unusual feature values. The amount of change applied to each distractor was randomly determined, and ranged from 0–360° for orientation and 0–215% for aspect ratio. The resulting search displays were therefore heterogeneous with respect to the feature manipulation, a conservative design decision that would be expected to work against us finding an effect of our feature manipulations. The same search displays were used for each experiment (same target-distractor pairings, same display positions of objects, etc.), with the only difference being the feature manipulation performed on the targets. Each search display was counterbalanced across all 25 mismatch levels, such that each trial appeared at a different level for each participant. Stimuli repeated five times throughout the experiment, but each time in a different context (different target/distractor combinations and different mismatch levels).

General procedure

Figure 2A shows a graphic depiction of the procedure. Participants fixated a central dot and pressed a button on the controller to start the trial. A preview of the target then appeared for 200 ms in the center of the screen. This preview was replaced with the search display, which consisted of eight evenly spaced objects (the target and seven distractors) arranged on an imaginary circle having a radius of 275 pixels (6.3°) relative to the center of the screen (see Figure 2B&2C for example search displays). Participants were instructed to find and look at the target, and upon doing so to press a button on a hand-held controller with the index finger of their dominant hand. Participants were told that the target would always be present, but might look different from how it appeared during preview. Note that feature-manipulated targets were still considered to be targets: There was no option for a “target absent” response.

Figure 2.

A) Example trial sequence for an orientation-change trial (Experiment 1). Example search displays from the (B) color and (C) grayscale conditions.

A fixation falling within 100 pixels (2.3°) of the center point of an object in the search display was considered a fixation on that object. This size—smaller than the ~2.7° objects—was selected after piloting. If participants pressed the response button while fixating on the target, the trial was counted as correct and terminated without feedback. If participants pressed the button while fixating a distractor object, an error sound was produced and “INCORRECT” was flashed on the screen for 500 msec. If participants pressed the response button while not fixating any object, this was also counted as incorrect. Trials timed out after four seconds. Participants performed one practice trial and twenty-five experimental trials for each mismatch level, for a total of 650 experimental trials. Preview-target mismatch levels were interleaved throughout, and there was a break halfway through the experiment. Participants were debriefed upon completion of the session.

Experiment 1: Orientation Mismatch

We chose to explore orientation because orientation meets all of the criteria previously used to identify guiding features (Wolfe & Horowitz, 2004). Search can be efficient to orientation-defined targets, and oriented targets can “effortlessly” pop-out (Foster & Ward, 1991) and be segmented from a texture (Nothdurft, 1991). Search asymmetries have also been found for orientation, with easier search for tilted line segments among vertical distractors than for the reverse (Foster & Ward, 1991; Poirier & Gurnsey, 1998; Treisman & Souther, 1985). Eye-tracking studies have confirmed these findings, revealing orientation asymmetries in the direction of initial saccades during search tasks with simple line stimuli (Foster, Savage, Mannan, & Ruddock, 2000), and in showing that orientation information correlates with gaze patterns in realistic scenes (Pomplun, 2006).

In Experiment 1, we varied the orientation of the target preview and the target in the search display so as to systematically evaluate the effect of preview-target orientation mismatch on search. Our experiment therefore replicates and extends the work of Vickery et al. (2005) and Bravo and Farid (2009) to include more increments of mismatch and a color/grayscale manipulation. Experiment 1 included twenty-five levels of mismatch, where Vickery et al. (2005) used less than ten levels and Bravo and Farid (2009) did not analyze the effects of different increments of orientation mismatch. These added levels allow for a fine-grained parametric analysis of orientation mismatch across the full range of the feature dimension.

The color/grayscale manipulation tested the effects of feature mismatch with and without the availability of hue information. Color was identified early on as a feature that guides search (Williams, 1966), and has shown to be useful in guiding search in complex displays: Guidance correlates more strongly with color features than with other features in scenes (Hwang et al., 2009), and search is faster in color displays than grayscale displays, at least for some very colorful classes of targets (Hayakawa, Kawai, & Masataka, 2011). We hypothesized that the observed mismatch effects would depend on the presence—or absence—of color in the displays, with participants relying more on orientation in the grayscale condition than in the color condition.

If preview-target orientation mismatch effects are not found in the color condition, this would suggest that target templates may in practice be often orientation-invariant, given that color is typically available for incorporation and use in a target template. This position stands in contrast to previous conclusions in the literature that the target template uses a relatively specific and detailed representation of orientation (Vickery et al., 2005). Finding this data pattern would also suggest that orientation mismatch effects may be less pronounced than what the existing search literature has led us to believe. Orientation, for example, might only be included in the target template when color information is unavailable. There are several reasons why this may be the case. First, visual working memory limitations may restrict the amount of information used by the target template (Oh & Kim, 2004; Olivers, Meijer, & Theeuwes, 2006; Schmidt, MacNamara, Proudfit, & Zelinsky, 2014; Woodman & Luck, 2004). Second, orientation information may not be used because hue information is more successful in guiding search to a target and adding orientation would therefore produce no additional benefit. Third, using too many features to guide search, particularly when a feature is not very informative, is likely undesirable, as each feature dimension will have some noise. Hout and Goldinger (2015) provided a clear demonstration that the addition of extra features can degrade search: when participants were shown a pair of objects at preview, guidance was poorer when searching simultaneously for two similar items than when searching for two dissimilar items (which were presumed to have fewer overlapping features). Considering only dimensions having large signal-to-noise ratios, which are correlated with stronger search guidance (Wolfe, 1994; Zelinsky, 2008), would therefore be reasonable. Although distinguishing between these possibilities is not the goal of this study, any of them (or some combination thereof) might result in the neglect of orientation features in the target representations and the lack of orientation mismatch effects.

In the grayscale condition, where hue information is unavailable to guide search, we hypothesize that participants will rely more on orientation information, resulting in larger orientation mismatch effects than in the color condition. This pattern would suggest that previous results using grayscale stimuli (Vickery et al. 2005) or very colorful backgrounds (Bravo and Farid, 2009) may have overestimated the size of mismatch effects by preventing or limiting the use of hue information, and that the representations used to guide search are generally less precise than what is suggested by the current literature.

Finding orientation mismatch effects in the color condition would suggest that search guidance used a representation of orientation in the target template. This template would likely also include color (which might be evidenced by a speeding of search in the color condition relative to the grayscale condition), but an orientation-mismatch effect would mean that orientation information was included as well even though that information is known to be unreliable. We may also find that mismatch effects are equivalent between the color and grayscale conditions. This would suggest that different feature dimensions are weighted equally in the target template when used to guide search, as predicted by most image-computable models of search guidance (e.g. Adeli, Vitu, & Zelinsky, 2017; Pomplun, 2006; Zelinsky, 2008; but see Hwang et al., 2009).

We also explore the representation of target orientation during search, which is an understudied topic. This is because most previous work has relied on stimuli that physically cannot be manipulated through a full 360° range of rotation. For example, line segments rotated 180° are identical to non-rotated line segments. Work that has addressed this issue concluded that search can be guided over 180° of orientation manipulation, but not more (Wolfe, Klempen, & Shulman, 1999). Specifically, search with orientation cues was efficient, except when these cues resulted in a 180° change in polarity from the target. That is, search was guided to upside-down stimuli as strongly as right-side-up stimuli, and guidance to stimuli rotated 45° was equivalent to those rotated 315°. The mismatch effects in Vickery et al. (2005) were consistent with this pattern, with an effect on RTs that increased up to 90° in their task: the pattern for 90–180° was noisy and did not reveal a clear effect across that range. Because these previous studies did not use eye movement measures, it is unclear whether their results were due to guidance effects or due to differences in other processes. For example, participants may have mentally rotated the images after fixating them. As such, it is possible that orientation guidance may have occurred over 360°, but participants relied on a restricted 180° range after the target object was located and needed to be identified. Further, both Vickery et al. (2005) and Wolfe et al. (1999) used simple stimuli, and Wolfe et al. noted that their data pattern may not hold with real-world objects, which typically have an unambiguously defined orientation (due to cues such as lighting direction). This means that a 360° orientation range might be used to guide search in the current study context—and other more realistic scenarios—even if 180° is the range used when searching for simple stimuli (Wolfe et al., 1999).

This experiment tests whether search in the context of realistic stimuli uses the full 360° of orientation, and clarifies—using eye movement measures—whether this pattern exists in guidance or other processes. If the literature is correct in suggesting that search is limited to a 180° reference frame, the current data should show an increasing mismatch effect up to 90°, followed by a decrease as mismatch increases from 90° to 180°. In contrast, a 360° reference frame would be evidenced by increasing mismatch effects from 0–180° of mismatch.

Method

We created 25 within-participant orientation mismatch levels, with 15° increments of mismatch from 0° to 360°. A 5° mismatch level was also included to explore a finer-grained degree of mismatch. The levels were: 0° (“no change”), 5°, 15°, 30°, 45°, 60°, 75°, 90°, 105°, 120°, 135°, 150°, 165°, 180°, 195°, 210°, 225°, 240°, 255°, 270°, 285°, 300°, 315°, 330°, and 345° of change. The procedure was otherwise identical to that reported in the general procedure.

Participants

Fifty participants took part in Experiment 1 (25 female, age range = 17–26). Participants were evenly divided into grayscale and color conditions, and no one took part in more than one experiment. The number of participants was based on counterbalance, with each search display image counterbalanced across 25 different increments of feature mismatch. To confirm that this sample size would provide sufficient power, we conducted a power analysis (using G*Power 3.1.9.2) based on data from the set size 10 condition of Vickery et al. (2005)’s Experiment 5B, in which preview-target mismatch was created by rotating realistic objects up to 180° in 2D orientation. We determined the effect size in that previous study (f = ~.26), and based on this analysis (and using α = 05), we estimated that at least 12 participants would be needed to obtain statistical power at the .80 level for the comparison across mismatch levels in our study (when collapsed across clockwise and counterclockwise directions, which best matches the context tested by Vickery et al.).

Results and Discussion

We predicted that orientation mismatch effects would be present in grayscale contexts, consistent with Vickery et al. (2005), but that the dominance of color in the target template (Hwang et al., 2009; Williams, 1966) would lessen or eliminate entirely mismatch effects in the color condition.

Turning first to RT, we found no reliable effect of orientation mismatch on RT in the color condition, F(24,576) = 1.17, p = .32, η² =.05. In the grayscale condition, however, RTs were significantly affected by orientation mismatch, F(24,576) = 4.83, p < .001, η² =.17, the pattern that one would expect if participants were coding and using orientation. As shown in Figure 3A, this relationship generally followed an inverted u-shaped pattern due to the circular nature of orientation, suggesting both the use of orientation and a 360° reference frame (further discussed below). RTs increased as the stimuli were rotated to around 180°, with a maximum cost (relative to 0°) of 83 ms occurring at 240°. Except for a dip at 210°, which we attribute to noise in the data, further rotation past 180° decreased the mismatch between the preview and target and resulted in a lessening of the RT cost. Note also that RTs were overall faster in the color condition than the grayscale condition—F(1,49) =7.96, p <.05, η² =.14, confirming a previously-observed advantage for search with color stimuli (Hayakawa et al., 2011). Together these results suggest that orientation was used in the grayscale condition but not in the color condition. Figure 3B showed orientation collapsed across clockwise and counterclockwise directions to visualize the effect of mismatch more clearly, without regard to direction of rotation. Importantly, orientation mismatch did not reliably affect accuracy—F(24,576) ≤ 0.99, p ≥0.45, η²≤.04—suggesting that the effect in RTs was not due to a speed-accuracy trade-off. Accuracy was 97.1% or higher at all mismatch levels. Error trials were removed from all other analyses (including the RT analysis).

Figure 3.

A) Experiment 1 RT varied across orientation mismatch levels in the grayscale condition but not in the color condition. B) Experiment 1 RT collapsed over clockwise and counterclockwise orientation mismatch. Best-fit lines indicate a linear increase in RTs with increasing mismatch in the grayscale condition (0.41msec/°), but not in the color condition (0.09msec/°). Error bars indicate one standard error of the mean (SEM).

Effects of mismatch on RTs were previously reported, and so were expected. However, factors other than search guidance can affect manual RTs, calling into question inferences about mismatch effects based solely on the RT dependent measure. We therefore used eye movement measures to break the coarser RT measure into two finer-grained components: Time-to-target and verification time (Alexander et al., 2014; Alexander & Zelinsky, 2012, 2018; Castelhano et al., 2008; Maxfield & Zelinsky, 2012). Time-to-target, the time from display onset to when a participant first fixates on the target, is a dependent measure that primarily reflects a target guidance process. Verification time, the time it takes a participant to make a target-present/absent decision after fixating the target, primarily reflects a recognition process.

No reliable effect of orientation mismatch was found on time-to-target in the color condition, F(24,576) = 1.23, p = 0.27, η² =.05, indicating that participants were not using orientation to guide their search. In the grayscale condition, however, time-to-target varied with orientation mismatch, F(24,576) = 2.90, p < .01, η² =.11, with a maximum cost of 50 ms occurring at 255°. For grayscale stimuli, orientation was therefore encoded by participants at preview and used to direct gaze: As orientation mismatch increased, search was guided less efficiently to the target—see Figure 4. This finding is consistent with prior work indicating that orientation can guide search (Bravo & Farid, 2009; Pomplun, 2006; Vickery et al., 2005; Wolfe & Horowitz, 2004). However, as in overall RTs, this effect only appeared in the absence of hue information.

Figure 4.

A) Experiment 1 target guidance, as measured by time-to-target, was affected by orientation mismatch only in the absence of hue information. B) Time-to-target collapsed over clockwise and counterclockwise orientation mismatch, more clearly showing an effect of orientation mismatch only in the grayscale condition. The best-fit lines in panel B indicate a linear increase in time-to-target with increasing mismatch in the grayscale condition (0.22msec/°) but not for color (0.01msec/°). Error bars indicate one SEM.

Turning to verification times, in the color condition verification times were not reliably affected by orientation mismatch, F(24,576) = 1.06, p = 0.39, η² =.04. In the grayscale condition, however, verification times were significantly affected by orientation mismatch, F(24,576) = 3.30, p < .001, η² =.12, with a maximum cost of 37 ms occurring at 165°. As in overall RTs—and consistent with the circularity of the orientation feature space—verification times increased up to 180° of mismatch and then decreased with increasing mismatch—see Figure 5.

Figure 5.

A) Verification time varied across orientation mismatch levels in the Experiment 1 grayscale condition, but not the color condition. B) Verification data collapsed across clockwise and counterclockwise orientation mismatch, which shows a linear relationship between verification time and orientation mismatch most clearly in the grayscale condition (grayscale best-fit line = 0.19msec/°; color best-fit line = 0.08msec/°). Error bars indicate one SEM.

We next examined the reference frame used by the guidance process, focusing this analysis on the grayscale condition where guidance based on orientation was observed. Previous work has suggested that—for simple stimuli—the guidance system uses a 180° frame of reference: The implication of this would be that shapes having a polarity and appearing upside-down when foveated cannot be detected preattentively among right-side-up versions of the same stimuli (Wolfe et al., 1999). In the alternative 360° frame of reference, a 180°-rotated object would be treated as “upside-down”, rather than identical to an un-rotated 0° object, and this would cause a mismatch.

If participants used a 180° frame of reference we should have found equal guidance between the 0° and 180° levels. Consistent with Vickery et al. (2005), we did not, t(24)=−3.81, p<.001—see Figure 4A. Instead, and contrary to Wolfe et al. (1999), our results suggest that guidance may rely on a 360° reference frame. There was a linear increase in mismatch from 0–180° in RT, time-to-target, and verification time, (see Figure 3B, 4B, and 5B), consistent with a 360° reference frame (and not a 180° reference frame). The one notable exception to this general pattern is the decrease in performance observed at 210°. Given that there is only this one anomalous data point, which has no matching decrease in performance at 150°, and because we found no evidence for the increasing/decreasing patterns of mismatch effects that were expected from a 180° reference frame (increases up to 90°, decreases to 180°, increases to 270°, then finally decreases to 360°), we attribute this dip at 210° to noise in the data rather than evidence of a 180° reference frame.¹ The findings of the current study are consistent with Wolfe et al.’s (1999) predictions that additional cues may be used by the guidance system in photorealistic stimuli, supporting a 360° reference frame in these contexts.

In summary, orientation mismatch had no reliable effect in the color condition for any measure we analyzed, all ps > .22, see Figures 3B and 4B. This pattern suggests that the target template is not as detailed or specific as previously believed, and that previewed features are not automatically integrated into the template. Instead, our data suggest that when hue is present orientation is not used to guide search nor to identify the target. The prior work reporting contradictory results used stimuli that minimized the usefulness of hue information: Bravo and Farid (2009) used highly colorful backgrounds that were likely to share at least some hue values with the target, and Vickery et al. (2005) used grayscale stimuli. While mismatch effects were found in those tasks, it may be that orientation is only used when guidance processes cannot rely on hue information. Our finding also partly contradicts prior work suggesting that orientation information can be used to direct eye movements, although the results from these studies were correlational, and they also found that color information is dominant in guiding eye movements (Hwang et al., 2009; Rutishauser & Koch, 2007). Still other previous work has concluded that search is predominantly driven by color (Pomplun, 2006; Williams, 1966), although the present data suggest a stronger—that in some contexts orientation is not only limited, but not used at all. As for our grayscale condition, we interpret these results as suggesting that orientation can be used to guide search, but only in contexts where hue information is either less useful or non-existent. Since most real-world objects have hue information, this would suggest that orientation does not normally contribute significantly to search in the real world.

Experiment 2: Shape Mismatch

Shape has also met previous criteria used to identify guiding features (Wolfe & Horowitz, 2004). Search can be efficient to targets with at least some unique shape features (Levin, Takarae, Miner, & Keil, 2001), as evidenced by some shape features being easily segmented from texture backgrounds (Simmons & Foster, 1992), and some producing search asymmetries (Kristjánsson & Tse, 2001; Treisman & Gormican, 1988; Wolfe, Yee, & Friedman-Hill, 1992). Guidance is also worse towards objects when the shape features in the search display do not match the preview (Alexander et al., 2014).

The term “shape” includes many image properties: curvature, length, width, global shape (square, triangle, etc.), the number of corners or edges, and so on. Efficient search based on shape appears to rely primarily on local shape features, such as line terminators, T-junctions, curvature, and convexity, and less on overall form, as in the attributes of a shape’s contour (Wolfe & Bennett, 1997). However, at least some global form information seems to be processed preattentively (Alexander et al., 2014), since shapes are preattentively completed behind occluders during visual search (Rensink & Enns, 1998; Wolfe et al., 2011; but see Alexander & Zelinsky, in press). There is also limited evidence supporting preattentive processing for the global shape feature we chose to manipulate in the current study: Aspect ratio.

We focused on aspect ratio for several reasons. First, the process of altering the aspect ratio of real-world objects is very simple and thus easily replicable: Aspect ratio can be manipulated with most image editing software, while systematically altering the edge-based information in an image can be far less straightforward. Second, there is some suggestion that aspect ratio may be preattentively available to guide search, thereby elevating the importance of this feature (Pramod & Arun, 2014; Treisman & Gormican, 1988, but see Kristjánsson & Tse, 2001). Although search tasks with ellipses placed among circles have not supported the preattentive coding of aspect ratio (the ellipses do not “pop out”), suggestive patterns of data from early aspect ratio studies led some researchers to speculate that evidence for guidance using aspect ratio would have been found if more extreme deviations in aspect ratio had been tested (Treisman & Gormican, 1988). To date, the question of whether or not aspect ratio can guide search remains open (Wolfe & Horowitz, 2004), in large part due to the potentially many features that vary when aspect ratio is manipulated. For example, a variety of other non-aspect features differ between ellipses and circles, such as differences in curvature (Levin et al., 2001; Treisman & Gormican, 1988; Wolfe et al., 1992). What is clear, however, is that there is some feature that varies with aspect ratio that plays a role in guiding search.

The goal of the present experiment was not to identify a specific shape feature, but rather to determine how shape in general guides search (Alexander et al., 2014; Chen & Zelinsky, in press). Our third reason for choosing to manipulate aspect ratio was therefore to exploit the fact that the manipulation of aspect ratio also changes other simpler shape features, such as length, width, curvature, and the angles of any corners. Given our goal, simultaneously varying multiple shape features in an aspect ratio manipulation is desirable because it allows a stronger test of shape than a manipulation of only a single shape feature: Any effect is an indication that shape is important. This also allows the use of a wide range of stimuli, because any real-world object with an aspect ratio could have its aspect altered, but many of those objects would likely be excluded if a simpler feature was used.

Experiment 2 varied preview-target mismatch in terms of shape, again in the context of a search task using real-world stimuli. As with orientation, knowledge that shape information in the preview will typically mismatch the target might lead to a de-weighting of this feature dimension. Note that the nature of our shape manipulation was dramatic: Many shape features were changed simultaneously, and the changes we included (up to a 215% increase in width) were particularly salient. Because we manipulated shape to such a large degree, our results are biased in two respects. First, we are more likely to find an effect of shape mismatch. To the extent that shape is encoded and included in the target template—despite the consistently invalid previews and the accessibility of other more valid features—we would expect to find preview-target mismatch effects in this task. However, if we find that shape mismatch does not affect search guidance even at our largest levels of mismatch, we can be confident that participants are not using shape in our task. Second, the size of this manipulation is large enough that—if participants de-weight uncertain dimensions because of frequently-invalid previews—we might find no effect of shape mismatch. We explore this de-weighting possibility further in Experiment 3. As in Experiment 1, we included grayscale and color conditions to test whether shape (like orientation) is used differently in the presence and in the absence of hue information.

Method

We created 25 shape mismatch levels by stretching images in 5% increments, from 100% to 215%. As in the case of orientation, we also added a finer-grained level (102.5% mismatch). These mismatches always involved increasing width (though the decision to increase width, rather than height, was arbitrary). The levels were: 100% (“no change”), 102.5%, 105%, 110%, 115%, 120%, 125%, 130%, 135%, 140%, 145%, 150%, 155%, 160%, 165%, 170%, 175%, 180%, 185%, 190%, 195%, 200%, 205%, 210%, and 215% increase in width (with a corresponding decrease in height). The procedure was the same as the general procedure in all other respects.

Participants

Fifty participants took part in Experiment 2 (23 female, age range = 18–22). As in Experiment 1, this sample size was determined by counterbalance (25 participants per condition). To confirm that this sample size was adequate, we conducted a power analysis using the observed effect size of the mismatch from the grayscale condition of Exp 1 (η² =.17 for RT). With an α of .05, 25 participants per condition provided power above the .95 level (1 – β = .97).

Results

As in Experiment 1, we predicted that we wound find mismatch effects in the grayscale condition. In Experiment 2, the presence of mismatch effects would show the use of shape to guide search, even though participants know shape to be unreliable (the shape of the target in the search display mismatches the preview on 24/25ths of the trials). We also predicted the absence of shape mismatch effects in the color condition of Experiment 2, in which color could once again dominate the target template and limit the use of shape information.

In both the color and in the grayscale condition, accuracy was 96.3% or higher at all levels, and was not reliably affected by shape mismatch—F(24,576) ≤ 1.32, p ≥ 0.22, η²≤.05 — suggesting that any effects observed in RT are not due to a speed-accuracy tradeoff. Error trials were removed from all other analyses.

In the color condition, RT did not reliably vary with shape mismatch, F(24,576) = 1.44, p = 0.16, η² =.06, indicating that participants were not using shape information when hue was available—see Figure 6. In the grayscale condition, however, participants relied on shape to perform this task, as evidenced by a significant effect of shape mismatch on RTs, F(24,576)=4.73, p < .001, η² =.17, with a maximum cost of 104 ms occurring at 205%. RTs were also again faster in the color condition than in the grayscale condition—F(1,49) =23.63, p <.001, η² =.33.

Figure 6.

Manual RT increased with increasing shape mismatch in the grayscale condition of Experiment 2, but did not vary with shape mismatch in the color condition. The best-fit line for the grayscale data had a slope of 0.24msec/%, but this slope was only 0.01msec/% for the color data. Error bars indicate one SEM. Unlike orientation, aspect ratio is not a circular feature space and therefore the mismatch levels are not presented in collapsed form.

Analysis of time-to-target revealed a pattern similar to what was found for RT. In the color condition, time-to-target was not reliably affected by shape mismatch, F(24,576) = 0.76, p = 0.68, η² =.03, again indicating that search was not guided using shape information. In the grayscale condition, however, time-to-target significantly varied with shape mismatch. This pattern suggests that shape is used to guide search to the target when hue is not available—F(24,576)=2.58, p < .01, η² =.10, with a maximum cost of 50 ms occurring at 200%—Figure 7.

Figure 7.

Time-to-target varied across shape mismatch levels in the grayscale condition of Experiment 2, but not the color condition. Best-fit lines indicate an increase of 0.10 msec/% across mismatch levels in the grayscale condition, but only a 0.02 msec/% increase across levels in the color condition. Error bars indicate one SEM.

These consistent data patterns extended into verification times. In the color condition, verification times did not reliably vary with shape mismatch, F(24,576) = 1.45, p = 0.16, η² =.06, indicating that participants did not use shape information to make a target decision when hue was available. In the grayscale condition, however, verification times significantly varied with shape mismatch—F(24,576)=3.99, p < .001, η² =.14, with a maximum cost of 67 ms occurring at 195%. Figure 8 illustrates this color-dependent use of shape in this search task.

Figure 8.

Experiment 2 verification times did not change with mismatch in the color condition (decreasing by 0.01 msec/%, by best fit line), but significantly increased with mismatch in the grayscale condition (0.14 msec/%). Error bars indicate one SEM.

Experiment 3: Interleaved feature dimension mismatches

Experiments 1 and 2 tested whether search guidance to real-world target objects relies on an obligatory coding of orientation and shape information into the target template. In the context of our feature-mismatch paradigm, orientation or shape information was uncertain at best and invalid at worst, with features mismatching most of the time. One approach taken by models of search is to assume that color and orientation are weighted equally (Zelinsky, 2008). Another approach is to weight the multiple dimensions in some way depending on the target definition. For example, Guided Search (Wolfe, 1994) predicts that searchers will use their knowledge of the target to weight feature dimensions. In the context of Experiments 1–2, participants might therefore have assigned less weight to orientation and shape because they knew that those dimensions would not match the target 24/25ths of the time. Such a de-weighting of the manipulated dimension, if it occurred, would have resulted in an underestimation of the orientation and shape mismatch effects and, therefore, the importance of these features in guiding search. This possibility is in line with suggestions in the search literature that the target template can be altered to meet task demands (Bravo & Farid, 2009; Machizawa, Goh, & Driver, 2012).

Data from Hout and Goldinger (2015) argue for an opposing possibility: That when participants are presented with picture previews of targets, these are encoded with “surprising” levels of detail and specificity (p. 25), resulting in sensitivity to relatively small amounts of mismatch. If participants encode and use the same features without regard to the trial context, there should be no effect of manipulating the proportion of trials in which the target mismatches the preview. Hout and Goldinger found precisely this pattern, with no reliable difference between conditions where targets mismatched the previews 20%, 53%, or 80% of the time, along a wide variety of dimensions. In the current Experiments 1–2, however, where participants knew which feature would vary (orientation or aspect ratio), we found no evidence of mismatch effects in the presence of hue information. This pattern suggests that encoding may flexibly vary with knowledge of the reliable feature dimensions. If, as our data suggest, encoding is not automatic when participants know the feature dimensions that are likely to vary, then varying the likelihood of mismatch on a given dimension should affect which feature dimensions are encoded (and to what degree). If, however, participants automatically encode all the features that are available at preview, varying these likelihoods should not affect how feature dimensions are weighted at preview.

Experiment 3 tested both for effects of mismatch uncertainty by manipulating orientation and shape within the same set of grayscale experimental trials. Our decision to use only grayscale stimuli was informed by the results of Experiments 1 and 2, which showed that hue information dominates the target template when hue is available and likely overwhelms any signal from other features. Whereas orientation mismatched 24/25^th of the time in Experiment 1 and shape mismatched 24/25^th of the time in Experiment 2, Experiment 3 was designed such that each feature dimension was substantially less likely to mismatch (12/25^th of the trials). This means that on more than 50% of the trials, the shape of the target perfectly matched the preview (compared to 1/25^th of trials in Experiment 1), with the same being true for orientation. Also unlike Experiments 1–2, participants did not know which feature dimension would change on a given trial, because the two dimension mismatches were interleaved. Note that this interleaved testing of both orientation and aspect ratio stands in contrast to the design used by Vickery et al. (2005), which manipulated only a single feature dimension (orientation). While Bravo and Farid (2009) did vary multiple dimensions, they did not include the contrasting condition in which only one feature mismatched, and so could not assess any effects of interleaving these changes.

If uncertainty/expectation in feature mismatch affects which feature dimensions participants use to guide their search, then we expect to find larger mismatch effects in Experiment 3 compared to those observed from the grayscale conditions of Experiments 1 and 2. This hypothesis follows from the fact that, when target feature values on a given feature dimension are uncertain, using that dimension to guide search would be maladaptive. So long as all of the feature dimensions from the preview are not used obligatorily by the guidance process, it seems likely that participants would either not encode feature values from the manipulated dimension or de-weight an uncertain feature dimension to the extent they are able to do so. If participants can adjust their target templates to give less weight to features that are expected to mismatch, preview-target mismatch effects should therefore be significantly greater when trials having mismatches on different dimensions are interleaved (Experiment 3) compared to when only a single dimension is manipulated throughout the course of the experiment (Experiments 1 and 2). Alternatively, the act of attending to the target preview might result in the obligatory encoding and use of the previewed features for guidance, without any differential weighting of those features. This hypothesis predicts that the magnitude of the mismatch effects observed in Experiment 3 should be comparable to those observed in Experiments 1 and 2.

Method

To avoid a prohibitive number of Experiment 3 trials, we sub-sampled the 49 levels from Experiments 1 and 2 to obtain 17 within-participant levels: One with no mismatch on any dimension, eight orientation mismatch levels (consisting of 30° increments), and eight shape mismatch levels (consisting of 10% increments). Specifically, these levels were: 0° “no mismatch”; orientation mismatches of 30°, 60°, 90°, 120°, 150°, 180°, 210°, and 240°; and shape mismatches of 110%, 120%, 130%, 140%, 150%, 160%, 170%, and 180%. We randomly interleaved these levels, which means that roughly half of the trials were followed by another trial in which the same dimension mismatched. Note that these levels all have an equivalent in either Experiment 1 or 2, allowing us to compare results across these experiments. Twenty-five experimental trials were included for each level, totaling 425 experimental trials. The procedure was otherwise the same as in the general procedure.

Participants

Twenty-five participants took part in Experiment 3 (19 female, age range = 17–24). To confirm that this sample size was adequate, we again conducted a power analysis using the observed effect size of the mismatch from the grayscale condition of Exp 1 (η² =.17 for RT). With an α of .05, this sample size provided power at the .95 level for the within-dimension comparisons.

Results

As in the grayscale conditions from Experiments 1 and 2, RTs varied with mismatch—F(8,192) = 4.83, p < .001—Figure 9. Additionally, there was a main effect of feature dimension was also found—F(8,192) = 15.50, p = .001—such that RTs were slower on shape mismatch trials compared to orientation mismatch trials. Mismatch and feature dimension did not interact, F(8,192)=1.43, p = .21. Mismatch also did not reliably affect accuracy—F(8,192)=1.07, p = .38—nor was there a main effect or interaction between orientation and shape (ps > .37), This analysis, combined with overall high levels of accuracy (greater than 95.8% for all conditions), suggests that the mismatch effects in RTs were not due to a speed-accuracy trade-off. Error trials were removed from all other analyses.

Figure 9.

Experiment 3 RTs increased with increasing mismatch for both the orientation and shape manipulations. Error bars indicate one SEM.

Time-to-target differed between orientation and aspect ratio trials within Experiment 3, F(8,192)=13.69, p = .001, with generally slower times in orientation-change trials. Unlike the grayscale conditions of Experiments 1 and 2, however, there was only a marginal mismatch effect in time-to-target—F(8,192)=2.26, p = .05—see Figure 10. This marginal effect, in contrast to the significant effects in Experiments 1–2, likely does not reflect a difference in how features were weighted between the experiments, as the magnitude of the effect is comparable. The interaction of feature dimension and degree of mismatch did not reliably affect time-to-target—F(8,192)=.90, p = .49.

Figure 10.

Experiment 3 time-to-target increased with increasing mismatch for both the orientation and shape manipulations. Error bars indicate one SEM.

As shown in Figure 11 verification times were significantly affected by amount of mismatch—F(8,192)=4.42, p < .001—indicating that orientation and shape were used to make a target present/absent decision in this task. Feature dimension did not reliably affect verification time, F(1,24)=2.92, p = .10, nor did feature dimension interact with degree of mismatch, F(8,192)=1.73, p = .14. Moreover, the u-shaped pattern expected from a 360° space was not observed in orientation mismatch: Mismatch at 30° had relatively minimal effect, and increasing mismatch beyond 180° continued to increase the magnitude of the mismatch effect.

Figure 11.

Experiment 3 verification time increased with increasing mismatch for both the orientation and aspect ratio manipulations. Error bars indicate one SEM.

To the extent that participants can effectively de-weight a feature dimension, the mismatch effects in Experiment 3 should have a larger magnitude than in Experiments 1–2: Participants should give relatively more weight to both orientation and shape than in the earlier experiments, because—unlike in Experiments 1 and 2—those features are validly previewed on at least half of the trials of Experiment 3. If, however, no difference in the magnitude of mismatch effects is found between experiments, this would suggest that top-down weighting is relatively inflexible, or perhaps even applied preattentively. To test this, we conducted 2 (experiment) × 9 (mismatch level) mixed ANOVAs. Shape mismatch trials were excluded when comparing Experiment 3 data to Experiment 1, and orientation mismatch trials were excluded when comparing Experiment 3 data to Experiment 2. Only levels with an equivalent in both experiments were included in the comparisons. Significant interactions between experiment and mismatch level would demonstrate an effect of uncertainty in participants knowing the feature dimension that would change from trial-to-trial.

To compare the mismatch effects on guidance in Experiments 1 and 2 to those in Experiment 3, we focused our analysis on the time-to-target measure. There was no reliable difference in orientation mismatch effects between the grayscale conditions of Experiment 1 and Experiment 3, F(1,48)=.04, p = .84. There was a marginal interaction with the level of mismatch, F(8,384)=1.96, p = .06, with time-to-target in Experiment 3 being longer than in Experiment 2 at the higher mismatch levels (Figure 12A). For the shape manipulation, there was similarly no reliable difference between Experiment 2 and 3, F(1,48)=.44, p = .51, nor was there an interaction with the level of mismatch, F(8,384)=.89, p = .52 (Figure 12B). Taken together, these patterns suggest that uncertainty in shape or orientation mismatch did not cause participants to differentially weight these feature dimensions in their target templates. Our participants appear to have given shape and orientation some weight in the guidance process (in the grayscale conditions of the experiments). Changing the probability that one of these features would mismatch did not result in meaningful changes to these feature weightings.

Figure 12.

A) Time-to-target varied across orientation mismatch levels in Experiment 1 (grayscale condition) and Experiment 3, and (B) across shape mismatch levels in Experiment 2 (grayscale condition) and Experiment 3. Error bars indicate one SEM.

General Discussion

Most previous work exploring the target template used simple stimuli and focused on determining what features can be used to guide search because of their preattentive availability to the visual system. Importantly, however, the features that can guide search are not necessarily those that are used to guide search. If, for example, participants are capable of guiding search efficiently to angry face targets among neutral face distractors, this could suggest that emotional expressions are available (at least to some degree) preattentively and used to guide search (Palermo & Rhodes, 2007). However, it seems extremely unlikely that facial expressions are a basic feature in the sense that it is included in the target template for most search tasks. Facial expression features are useless, for example, when searching for a pen on your desk or your car in a parking lot.

In the present experiments we replicated findings describing orientation mismatch effects—in which differences in orientation between a preview and the search target cause a decrease in performance—and we extended these findings to a feature dimension not previously explored: Aspect ratio. Because we used real-world objects, the mismatch paradigm that we used allowed the targets to differ from distractors along many dimensions, a more natural context than tasks where targets differ from distractors only along a single dimension. Importantly, participants could use any (or all) of the features that distinguish the target from the distractor items to perform the task. The present experiments tested whether orientation and aspect ratio are among the basic features, those that are used in the search for a wide variety of objects. We also examined whether these features are weighted depending on whether participants expected those features to mismatch or not. Collectively, these data are important for modeling search, providing a set of empirically-established parameters to inform the top-down component of target guidance.

Because eye movement measures were not included in previous studies of mismatch (Bravo & Farid, 2009; Vickery et al., 2005), it was possible that previously-observed mismatch effects were driven purely by verification processes. In fact, Vickery et al. (2005) did not find any reliable interaction between set size and 2d orientation mismatch, which might suggest an effect resulting from verification alone and specifically not guidance. Our results not only confirm the presence of mismatch effects in search guidance, but we show that these effects exist in both guidance and verification—although only in the grayscale conditions of our experiments.

When hue was available, we found little-to-no effect of orientation or shape mismatch on search efficiency. We speculate that this reliance on color is because of targets being more easily discriminable from distractors in the color dimension compared to the other non-color features tested (Alexander & Zelinsky, 2012). However, we were surprised by the lack of reliance on orientation and shape features, and believe that this observation requires a qualification of conclusions from previous orientation–mismatch studies stating that detailed visual information is represented in target templates and used to guide search (Bravo & Farid, 2009; Vickery et al., 2005). Orientation information appears to be used for guidance only when hue is either not available (in grayscale experiments; e.g. Vickery et al., 2005) or when hue is made less useful by the experimental context, as in the Bravo and Farid (2009) study where the background consisted of brightly-colored coral reefs. Participants may also have decreased their reliance on hue information in the Hout and Goldinger (2015) study, where hue was one of the unconstrained feature dimensions in which mismatch sometimes existed. This sometimes-inaccurate hue information may have been strategically deweighted during search. When hue is available and unaltered—as in our color experiment and in most real-world tasks for people with normal color vision—the target template is dominated by hue and largely invariant to changes in orientation and shape.

Our observed invariance to orientation and shape mismatch is inconsistent with current theories of visual search that assume that attention and gaze are guided by target templates using—among other features—orientation and shape (Wolfe & Horowitz, 2017). Moreover, most image-based models that are designed to accommodate visually-complex targets typically assume that many features from a preview are simultaneously extracted and used in equal weighting to guide search (e.g. Adeli et al., 2017; Pomplun, 2006; Zelinsky, 2008; but see Hwang et al., 2009). The current data show that many of the non-hue features available from the preview went unused, suggesting a need to modify these search theories.

Why might participants use relatively imprecise target representations when information is available to construct more detailed guiding templates? One explanation might be due to a limitation on visual working memory: If only a finite number of feature values can be encoded into the target template (e.g. Olivers, Peters, Houtkamp, & Roelfsema, 2011), and hue information is more useful than other feature dimensions, orientation and shape may be selectively excluded from this limited set of guiding features. Another possibility is that precise visual details from a target preview may be relied upon less commonly than what has been believed; a possibility suggested by recent work using a categorical search task (Maxfield, Stadler, & Zelinsky, 2013; Schmidt & Zelinsky, 2009; Yang & Zelinsky, 2009; Yu, Maxfield, & Zelinsky, 2016). If participants construct a categorical representation of the target to guide their search, when previewed with a red sports car they might search for any red sports car rather than the specific car shown at preview. In the context of the present experiments, such a categorical representation, perhaps elaborated with some visual detail, would allow search to be effectively guided to the target without relying on very precise hue, orientation, or shape information. Note that because such a red-sports-car template would be more precise than a simple text label (“car”), this explanation would still be consistent with previous research showing that image previews are superior to text labels (Bravo & Farid, 2009; Castelhano et al., 2008; Malcolm & Henderson, 2009; Maxfield & Zelinsky, 2012; Vickery et al., 2005; Wolfe, Horowitz, Kenner, Hyle, & Vasan, 2004; Yang & Zelinsky, 2009; see Schmidt & Zelinsky, 2009, for evidence that more precise text labels benefit guidance). The relationship between categorical search and previewed exemplar search remains unclear, with the only firm distinction made in the literature being that categorical search likely relies on long-term memory, while previewed visual search likely relies on visual working memory (Yang & Zelinsky, 2009). If it is the case that previewed visual search also uses representative visual features that have been learned for the target category (Yu et al., 2016), albeit to a lesser degree than when conducting a purely categorical search, the imprecision arising from these categorical features may be enough to explain the lack of mismatch effects reported in this study. Supporting this view is evidence that categorical search operates in a qualitatively similar way to the previewed search for a particular target exemplar (Alexander & Zelinsky, 2011), lending weight to the possibility that categorical templates are involved in both forms of search tasks.

A final explanation is that our evidence for orientation and aspect ratio invariance reflects a bias specific to the preview-target mismatch paradigm. The use of precise features in this paradigm is in a sense undesirable, as the features of the target are likely to mismatch the features appearing in the search display. We tested this possibility that subjects intentionally de-weighted these features by varying the probability across experiments in which this mismatch occurred on a given dimension. Orientation mismatched most of the time in Experiment 1, but less than half the time in Experiment 3, so differences in the magnitude of the mismatch effect would be expected if participants were avoiding the use of features that were known to mismatch. What we found was that mismatch effects were not reliably affected by uncertainty in the target feature dimension that changed from trial to trial (Figure 12). This suggests that, in the absence of hue information, participants use non-color features without consideration for the relative validity of these features. It may be the case that features from the preview are encoded automatically and obligatorily, but our results qualify this relationship by suggesting that hue information is so important to search that this dimension may override the contributions of other feature dimensions. However, when hue information is absent any feature that might help to compensate for the lack of hue is included in the target template.

We were surprised that participants did not show more evidence for using shape features to guide their search, even in the color conditions of our experiments. Shape features are known to dominate over color features in object recognition tasks (see Wichmann, Sharpe, & Gegenfurtner, 2002 for a discussion). In fact, object recognition research has traditionally stressed the role of shape over that of color, relying primarily on shape features such as “geons” instead (Biederman, 1987; Biederman & Bar, 1999). The fact that verification times were not highly affected by shape mismatch suggests that object recognition and target detection follow different rules (but see Zelinsky, Peng, Berg, & Samaras, 2013). Target detection, the task of visual search, may be an example of a task in which an object decision can be made with sufficient confidence based on color information over shape (Sanocki, Bowyer, Heath, & Sarkar, 1998). Different rules may also apply to target guidance; the features that are most successful in guiding attention to a target for the purpose of fixation need not be the same features that are used to recognize the target during fixation. An alternative explanation for the lack of shape effects in verification time may be that a subtle effect of shape might be masked by noise in the verification time measure, perhaps due to some participants deciding to delay their button press response until after careful fixation of the target so as to ensure that the trials would be treated as correct. Even so, the present data indicate that shape is used less for target verification than either orientation or color in this task.

Another possible explanation for the observed guidance and verification effects is that the representation used to guide search—in our data a target template based primarily on hue—might bias the following verification stage of the search task. Maxfield and Zelinsky (2012) suggested that the need to first guide search to the target object “locks in” or biases a set of guiding features such that they are then more likely to be used for object verification, even though a different set of target features might be more ideal (see also Navalpakkam & Itti, 2005). The similarity between patterns of mismatch effects in guidance and verification measures in the present study is consistent with this possibility. If the features used for target guidance and verification are more similar than what has typically been believed (we speculate that it is common opinion that target detection is thought to involve more than just the few basic features used to guide search), then the features used in verification (and perhaps recognition more generally) may be highly dependent on whether guidance was engaged prior to classifying the object as a target. If guidance is driven by hue and the target template carries over to the verification epoch, then verification will also be driven by hue. Conversely, in tasks where guidance is unnecessary or not critical, hue may be less represented among an object’s features, resulting in evidence for other feature dimensions playing a greater role.

There are also concerns specific to the shape and orientation features that we tested in our study. The shape features we manipulated might simply not be those used to perform the task. Shape features not captured by the aspect ratio manipulation may certainly have been engaged, and we cannot rule out this possibility. While manipulating aspect ratio did alter a wide variety of shape features (e.g. height, width, and curvature) there are many features (e.g. area, the number of corners, and the number of line terminators) that were not assessed in the current work and might potentially yield much larger mismatch effects. Orientation mismatch effects may be larger in other contexts than the one explored here. For example, different features may be used to guide search through realistic scenes, where mismatch would change the object’s relationship to the surrounding environment (e.g., Neider & Zelinsky, 2006; Nuthmann & Malcolm, 2016; Pomplun, 2006; Wu, Wick, & Pomplun, 2014). Our data therefore do not conclusively demonstrate that color dominates over all shape features in search—or over all features in all contexts—and therefore caution should be taken when generalizing the present results to claims about features in general.

What our data does demonstrate is a relative dominance of color over other features that are typically available to guide search, confirming previous results demonstrating the usefulness of color in visual search (e.g., Hayakawa et al., 2011; Hwang et al., 2009). Specifically, orientation and shape mismatch effects were only present under conditions where hue was not available. This was true despite shape and orientation being sufficiently discriminable and available for use, as evidence by mismatch effects existing in the grayscale conditions of the experiments at the same eccentricities where they were not found in the color conditions. Participants therefore could have used orientation or shape, but relied on hue instead. Note that our findings are consistent with correlational studies demonstrating that fixations are more likely to land on patterns sharing the target color than on patterns sharing the target shape (e.g. Williams, 1966) or patterns having target-similar orientations or spatial frequencies (Hwang et al., 2009; Rutishauser & Koch, 2007). This dominance of color in guidance has important implications both for color-blind participants and for searches performed under dark-adapted conditions (for example, at night). While it is possible that non-hue feature dimensions might play a larger role with greater degrees of experience or training (as would be the case with color-blind searchers), it is likely that searchers with color vision abnormalities are at a large disadvantage compared to those with normal color vision (see also (Cole & Lian, 2006; O’Brien, Cole, Maddocks, & Forbes, 2002).

In conclusion, researchers are still debating one of the most basic of search questions: Which features are used to guide attention to search targets. In contrast to the many measures traditionally used to infer that a feature is capable of guiding search in a task, the mismatch paradigm used here provides a single, direct means of assessing guidance: If the mismatching feature is used for guidance, then gaze should be guided less efficiently to the target. We believe that future work using the preview-target mismatch paradigm—paired with eyetracking measures—will finally answer a key question about the nature of search guidance: From among the features that are preattentively available and can be used for guidance, which ones actually are used to guide search?

Public Significance Statement:

Many real-world tasks involve visual search, including critical safety tasks such as luggage screening and finding tumors in x-rays. This study demonstrates that the mental representations used to direct eye movements and attention to targets in such tasks are much less precise than previously believed: when shown a picture preview of the search target, participants used little of the information from the preview. Whenever color could be used to find a target, little other than color information was used—even when the color information was likely to be invalid, and when other information could be used effectively.