The Spatial Dynamics of Mouse-Tracking Reveal that Attention Capture is Stimulus-Driven Rather than Contingent upon Top-Down Goals

Abstract

There has been a long-standing debate over why stimuli capture attention. Some argue that capture is driven by stimulus salience, while others believe that capture only occurs when the features of a stimulus match what we are searching for. This debate has recently focused on attentional disengagement, with the stimulus-driven camp claiming that all salient stimuli capture attention but attention is quickly disengaged from items dissimilar from our target, producing little cost in terms of response time. We used mouse-tracking to examine the spatial effect of cues that either matched or mismatched an observer’s target. Experiment 1 demonstrated that a cue mismatching the feature defining the target initially produced a spatial effect that was rapidly resolved, consistent with quick disengagement. Experiment 2 was a pre-registered replication with double the sample size that replicated the results of Experiment 1. Overall, computer mouse-tracking provided a direct observation of attentional disengagement, supporting stimulus-driven capture.

Introduction

If I am searching for my red coffee cup, what happens if there is a bright yellow highlighter on my desk? Does the highlighter capture my attention because it is unique? Or does it fail to capture my attention because it is not what I am looking for? For decades there has been a fundamental disagreement over the nature of attention capture. The bottom-up view holds that when an object captures attention, this attention shift is driven solely by stimulus salience (e.g., Franconeri, Simons, & Junge, 2004; Theeuwes, 1994; Yantis, 1993). For example, Theeuwes (1994) reported that when observers searched for an item of a unique shape, a uniquely colored item captured their attention, as evidenced by slower response times when the colored item was present compared to when it was absent. Proponents of this view argue that when a scene is first viewed, the initial sweep of attention is automatically directed to the most salient item in the scene. In stark contrast, the top down view holds that only stimuli consistent with an observer’s current goals capture attention and that salience plays no role. For example, in search and cuing paradigms, an observer looking for a red target will only have their attention captured by irrelevant red items, and not other salient but irrelevant items (abrupt onsets, uniquely green items; e.g., Folk, Remington, & Johnston, 1992; Folk, Remington, & Wright, 1994; Leber & Egeth, 2006).

Currently, the pivotal difference between the two views is disagreement over when an observer’s goals influence visual attention. While the top-down view argues that goals influence initial selection of visual information (only objects consistent with one’s goals capture attention), the bottom-up view argues that all salient stimuli capture attention and goals only alter the subsequent speed of disengagement from that stimulus (e.g., Belopolsky, Schreij, & Theewues, 2010). According to the bottom-up view, if a stimulus shares the feature defining the target, attention will linger before disengaging and rejecting it. However, if a stimulus does not share the feature defining the search target, attention is very quickly disengaged; in some cases this disengagement is so quick that there is little behavioral evidence that an attention shift ever occurred (Theeuwes, 2010). Crucially, the question of whether top-down control alters the initial selection of visual information or subsequent disengagement is spatial in nature. Thus, differentiating between these competing views requires spatial measures.

In the current study, we reasoned that computer mouse-tracking would allow us to directly observe the spatial influence of attention, including whether attention is captured, and when the influence of a distractor is overcome (i.e., disengagement). Like previous research using reaching movements (e.g., Song, 2017), computer mouse-tracking is ideally positioned to distinguish between top-down and bottom-up accounts of attention capture because it yields a continuous real-time measure of online processing across space and time as attentional processes resolve (Dale, Kehoe, & Spivey, 2007; Freeman & Ambady, 2010; Kieslich & Henninger, 2017). As the observer is in the process of making their response, higher level cognitive mechanisms continuously communicate with and influence the motor system (Cisek & Kalaska, 2010; see also Freeman, Ambady, Midgley, & Holcomb, 2011). By examining the spatial dimension of response trajectories, we are able to discern whether attention was captured by distractors that mismatched a participant’s goals and, if so, whether attention was quickly or slowly disengaged.

Experiment 1: Modified Contingent Capture

Our experiments were a modified contingent capture paradigm (similar to Folk, Remington, & Johnston, 1992; Folk, Remington, & Wright, 1994) amenable to mouse-tracking. Before searching for a target, participants were cued with a flash of color that either matched or mismatched their top-down attention set. Thus, the critical comparison dissociating top-down from bottom-up approaches is what happens to trajectories when a mismatching cue precedes the target. If the top-down view is correct, when an observer sees a green cue but is searching for a red target, trajectories should not deviate from the path to the red target. This pattern of results would demonstrate that top-down mechanisms affect the initial selection of information. However, if the bottom-up view is correct, when an observer is searching for a red target, both green and red cues will attract their trajectories, but green cues will be relatively easy to disengage from. This pattern of results would demonstrate that top-down mechanisms do not affect initial selection but rather come later and affect our ability to disengage. Schematic examples of these predictions are illustrated in Figure 1.

Figure 1.

Schematic trajectories illustrating the dissociable predictions for the bottom-up approach (panel A) and the top-down approach (panel B) for Experiments 1 and 2. On match trials, the target and cue were the same color. On mismatch trials, the target and cue were different colors. The critical comparison is what happens on mismatch trials: “do trajectories deviate and then correct?” as predicted by the bottom-up approach or “do they never deviate to begin with?” as predicted by the top-down approach. SS = same side cue. DS = different side cue.

Method

Participants

Participants were undergraduate students (N = 20, 13 females, M_age= 18.85 years) who completed the study in exchange for partial course credit. This initial exploratory experiment was followed by a preregistered replication (Experiment 2). For our primary analysis, our sample of 20 participants is powered to detect an effect size of dz = .66, assuming a β of .8 and an α level of .05. All effect sizes reported below were Cohen’s dz calculated using the lsr package (Navarro, 2015) in R (R Core Team, 2016) using the “paired” argument which uses standard deviation of mean differences for its calculations.

Procedure

Each screen contained four response boxes. From left to right, we refer to the boxes as B1, B2, B3, and B4. The boxes on the outside (B1 and B4) appeared halfway up the center of the screen. The boxes in the center (B2, B3) appeared at the top. Stimuli were colored Xs that appeared inside the boxes.

Each trial had the following sequence (see Figure 2). First, a screen with a start button and four response boxes appeared. Once participants clicked the “Start” button, there was a 500ms delay where the four response boxes were shown. This was followed by a screen where the outline of one of the four boxes changed color for 75ms (referred to as the cue). The cue matched the target’s color on 50% of trials and mismatched on 50%. This was followed by a screen showing the four response boxes with the white outlines again for 50ms. To discourage anticipatory movements, prior to tracking the trajectory of the mouse, the cursor was reset to the bottom center of the screen on each trial. Finally, a screen appeared with Xs inside each of the four boxes. The Xs were colored red, green, yellow, and blue and they appeared at a random location. For half the participants, the target was green and the incongruent color cue was red and vice versa for the other half. Participants had to click on the box containing the X in the target color within the response deadline (2000ms). They were explicitly warned to ignore the flash as this was intended to distract them and compromise their performance. Thirty-two practice trials were followed by five blocks of 96 trials each. Target and cue location were randomized and counterbalanced within each block.

Figure 2.

Procedure for Experiments 1 and 2. Participants searched for either a green or red X. After clicking the start button, there was a 500ms delay, and then one of the four boxes was cued with a flash of color for 75ms. The color either matched or mismatched the target. The cue could appear in any of the four locations, with two of the locations being on the same side as the target and two being on the opposite, different side. Then there was a 50ms delay where the boxes were shown, followed by a screen where colored Xs were displayed inside the boxes. Participants clicked on the box with their target.

The experiment was programmed in OpenSesame 3.1.9 (Mathôt, Schreij, & Theeuwes, 2012) using the mousetrap package (Kieslich & Henninger, 2017). Data were preprocessed in R (R Core Team, 2016) using a number of different packages (Kieslich & Henninger, 2016; Kieslich, Wulff, Henninger, Haslbeck, & Schulte-Mecklenbeck, in press). See the supplementary materials for more information.

Data Preprocessing

Prior to analyses, we excluded practice trials and incorrect trials. As determined a priori, only trials where the target appeared at the center two boxes (B2 and B3) were analyzed in order to prevent obfuscation of trajectories that might occur by averaging at two distinct locations. If the target was B2 and the cue appeared at B1 or B2, this was counted as a “same side” trial. However, if the target was B2 and the cue appeared at B3 or B4, this was counted as a “different side” trial. The converse is true for when the target appeared at B3. Finally, trajectories were remapped and normalized. The supplementary materials contain a fuller discussion of all preprocessing.

Our dependent measures of interest were area under the curve (AUC), maximum absolute deviation (MAD), and sample entropy. AUC is a measure of the pixels between an idealized straight path to the response and the actual trajectory and is thought to be a measure of response competition/attraction (Hehman, Stolier, & Freeman, 2015). MAD is similar to AUC except it conveys the greatest point of deviation from the idealized straight line (Hehman, Stolier, & Freeman, 2015). Sample entropy is a measure of spatial disorder (Hehman, Stolier, & Freeman, 2015) and indexes the complexity of the trajectories. The supplementary materials contain additional analyses on initiation time and response time.

Results and Discussion

The purpose of this experiment was to explicitly test the bottom-up account of attention capture against the top-down account. Thus, our primary analysis of interest is what occurs for mismatching color cue trials. The critical comparison here is whether cues on the same side differ from cues on the opposite side on mismatching trials. To test this, we conducted a repeated measures ANOVA with the within-subjects factor of cue side (same side, different side) on mismatching trials for AUC, MAD, and entropy.

Visual inspection of Figure 3 suggests attention was captured on mismatching trials. This was confirmed by our analyses (see Figure 4). Our ANOVA revealed that different side cues elicited greater AUC than same side cues, F(1, 19) = 5.98, p = .024, η_p² = .24, dz = .55. Similarly, different side cues elicited greater MAD than same side cues, F(1, 19) = 7.94, p = .011, η_p² = 29, dz = .63. These analyses confirm that capture occurred on mismatch trials: when the cue appeared on the opposite side, trajectories were attracted to it. Analyses of sample entropy did not reveal any significant differences between same side and different side cues (see Table 1 for details), suggesting that there was no qualitative difference in the complexity of trajectories.

Figure 3.

Averaged time-normalized trajectories for Experiment 1. Trajectories show that mismatching cues on the opposite different side of the screen attracted attention compared to mismatching cues on the same side. SS = same side cue. DS = different side cue.

Figure 4.

Area under the curve plot for Experiment 1. Error bars are 95% confidence intervals.

Table 1.

Primary Analyses for Experiments 1 and 2 Comparing Same Side to Different Side Cue Conditions; Note these Analyses are Limited to ANOVAs on Mismatching Trials

	Dependent Variables
	AUC			MAD			SE
	p	ηp2	dz	p	ηp2	dz	p	ηp2	dz
EXP 1	.024	.24	.55	.011	.29	.63	.747	.01	.07
EXP 2	< .001	.39	.79	< .001	.41	.82	.783	<.01	.04

p = significance value; η_p² = partial eta squared; dz = within subjects effect size; AUC = area under the curve; MAD = maximum absolute deviation; SE = sample entropy.

Secondary analyses were conducted to compare whether capture effects differed in size between matching and mismatching conditions. Analyses did in fact find that capture was larger in the matching condition than the mismatching condition. A fuller discussion of these analyses can be found in the supplementary discussion.

The results of Experiment 1 were consistent with the bottom-up view. When the cue mismatched the target, movement trajectories deviated towards the cue before being quickly corrected. This is consistent with the bottom-up notion that top-down effects on attention capture come after the initial sweep of perceptual information and affect only our ability to disengage from salient perceptual information (Theeuwes, 2010; Belopolsky, Schreij, & Theeuwes, 2010).

Experiment 2: Contingent Capture Replication

Experiment 2 was an exact replication of Experiment 1 except it was preregistered and the sample size was doubled. All of our analyses, data processing, and hypotheses were preregistered with the Open Science Framework (https://osf.io/rq3zg/). Raw data along with scripts are also available (https://osf.io/b8v4t/).

Methods

To detect an effect size equal to that of our first experiment (dz = .55) with a β of .9 and an α level of .05, we would need a sample size of 38 participants for our primary analysis. Forty undergraduate students (30 females, M_age= 19.6 years) completed the study in exchange for partial course credit. All procedures, equipment, data processing, etc. were exactly the same as Experiment 1.

Results

As in Experiment 1, our primary analysis of interest is what occurs for mismatching color cue trials. Thus, we conducted a repeated measures ANOVA with the within-subjects factor of cue side (same side, different side) on mismatching trials on AUC, MAD, and SE. Our pattern of results replicate those of Experiment 1. Different side cues elicited greater AUC than same side cues, F(1, 39) = 25.50, p < .001, η_p² = .39, dz = .79. Similarly, different side cues elicited greater MAD than same side cues, F(1, 39) = 26.90, p < .001, η_p² = .41, dz = .82. As above, sample entropy did not differ between same side and different side cues (see Table 1 for details), suggesting that there was no qualitative difference in the complexity of trajectories. Collectively, these analyses confirm what is visually apparent from inspection of Figure 5. Namely, capture occurred on mismatch trials such that when the cue appeared on the opposite side, trajectories were attracted to it.

Figure 5.

Averaged time-normalized trajectories for Experiment 2. Trajectories show that mismatching cues on the opposite different side of the screen attracted attention compared to mismatching cues on the same side. SS = same side cue. DS = different side cue.

As above, secondary analyses found that capture was larger in the matching condition than the mismatching condition. Overall, these analyses replicated the pattern of results in Experiment 1. A fuller discussion of these analyses is presented in the supplementary materials.

General Discussion

The nature of attention capture has been debated for several decades. Our results suggest that progress can be made by moving beyond measures of capture in which spatial processing must be inferred. Although a popular method, response times only provide an indirect measure of spatial attention and its behavioral consequences. As such, capture has been inferred on the basis of comparing the response times across conditions. In the current paper, we used computer mouse-tracking in the hopes that it would allow us to more directly observe the real-time allocation of spatial attention.

Across both experiments, movement trajectories were attracted to both matching and mismatching cues, suggesting that attention was involuntarily captured by the flashes of color. This is evidence that the initial selection of perceptual information is driven by the local salience of the stimulus; in other words, selection is bottom-up. Across both experiments we also found that the degree of capture—i.e., deviation in movement trajectory—was greater for matching cues than mismatching cues. This is evidence that top-down forces modulate our ability to disengage from stimuli that capture attention.

This work builds on and acknowledges similar research examining saccade curvature, latency, and direction effects in eye movement tasks. Mulckhuyse, van Zoest, and Theeuwes (2008) compared the effect of target-similar and dissimilar distractors and observed that saccade parameters (latency, direction) were influenced initially by stimulus salience, and only later did top-down goals have an effect, similar to effects reported here. Additionally, saccade curvature differences have been observed based on target and distractor similarity in a simple eye movement task (Mulckhuyse, Van der Stigchel, & Theeuwes, 2009). However, suggesting potentially important differences in paradigm and/or mechanism, movements in these cases were generally curved away from distractors, not toward. It should also be noted, in both of these cases, distractors already provided a strong capture signal by appearing as abrupt onsets. Our work is better positioned to link movement trajectory effects to classic findings interpreted as top-down capture, as our experiments better reflect the timing and task parameters of these original studies, and our studies focus on detecting the spatial influence of distractors specifically on mismatch trials.

Our results are a crucial step forward in resolving the top-down versus bottom-up debate of attention capture. Consistent with the bottom-up stimulus driven account, our evidence suggests salient cues seem to always attract attention. Top-down effects seem to occur after the initial capture of attention and alter the ease with which attention is disengaged. In other words, attention is captured in a bottom-up fashion but how long it is captured for seems to be modulated in a top-down manner.

Figure 6.

Area under the curve plots for Experiment 2. Error bars are 95% confidence intervals.

Public Significance.

There are times where we want our attention captured, such as when the check engine light comes on, and there are other times where we don’t want our attention captured, such as by billboards which may take our eyes off the road. The results of our current paper suggest that tracking an individual’s reaching movements (via the movement of a computer mouse) will help us understand the degree to which attention is under our conscious control.