Effector-independent representations guide sequential target selection biases in action

Abstract

Previous work shows that automatic attention biases toward recently selected target features transfer across action and perception, and even across different effectors such as the eyes and hands on a trial-by-trial basis. While these findings suggest a common neural representation of selection history across effectors, the extent to which information about recently selected target features is encoded in overlapping versus distinct brain regions is unknown. Using fMRI and a priming of popout task where participants selected unpredictable, uniquely colored targets among homogeneous distractors via reach or saccade, we show that color priming is driven by shared, effector-independent underlying representations of recent selection history. Consistent with previous work, we found that the intraparietal sulcus (IPS) was commonly activated on trials where target colors were switched relative to those where the colors were repeated; however, the dorsal anterior insula exhibited effector-specific activation related to color priming. Via multivoxel cross-classification analyses, we further demonstrate that fine-grained patterns of activity in both IPS and the medial temporal lobes (MTL) encode information about selection history in an effector-independent manner, such that ROI-specific models trained on activity patterns during reach selection could predict whether a color was repeated or switched on the current trial during saccade selection and vice versa. Remarkably, model generalization performance in IPS and MTL also tracked individual differences in behavioral priming sensitivity across both types of action. These results represent a first step to clarify the neural substrates of experience-driven selection biases in contexts that require the coordination of multiple actions.

Introduction

Everyday perception-action interactions often require selection of a single goal from multiple possibilities. Historically, effects that influenced this attentional selection process were separated into a binary between top-down influences such as current behavioral goals, and bottom-up factors such as perceptual salience (Tipper, 1985; Theeuwes, 1992; Itti & Koch 2000; 2001; Connor, Egeth, & Yantis, 2004). However, the influence of what has been recently attended does not fit neatly into either of those two categories, and thus in recent years, the term “selection history” has gained popularity as a term to describe the ways in which attentional selection in the recent past can influence attentional selection in the current moment (Maljkovic, 1994; 1996; Awh, Belopsky, & Theeuwes, 2012; Anderson et al., 2021).

Recent experience can have a profound impact on the way humans perceive and interact with the environment (Schneider & Shiffrin, 1977; Egeth & Sager, 1977; Tipper et al., 1985; Kristjánsson, Wang, & Nakayama, 2002; Wolfe et al., 2003; Awh et al., 2012). One widely used context in which researchers have studied the effects of recent experience on behavior is that of visual search, where participants are asked to select and respond to a target object among several distractors. For example, Maljkovic and Nakayama (1994) asked participants to identify a uniquely colored diamond among similarly shaped distractors, and then press a key corresponding to the direction of a notch in the target diamond. Critically, although the response itself was orthogonal to the color of the target on a given trial in their study, the authors observed a strong reaction time benefit for discriminations where the target color from the previous trial was repeated rather than switched – an effect that has since been referred to as “priming of popout” (PoP). PoP suggests that peoples’ attention may be automatically biased to select targets that share features with those they have recently selected (Maljkovic & Nakayama, 1994; 1996; for review, see Kristjánsson & Campana, 2010). PoP fits neatly into the category of selection history because behavior on the current trial depends on the extent to which the current target matches the target that was selected on the previous trial. This is distinct from simple stimulus history without selection, as prior research has shown that PoP does not occur if a response to the repeated (or switched) feature was not required on the previous trial (e.g., Fecteau, 2007; Kristjánsson, Saevarsson, & Driver, 2013).

The influence of selection history has more recently been shown to extend beyond perceptual discrimination contexts that require keypress responses to actions such as eye and hand movements (McPeek, Maljkovic, & Nakayama, 1999; Song & Nakayama, 2006; Song & McPeek, 2009; Moher & Song, 2014; 2016). In real-world scenarios, object selection is typically achieved through the seamless coordination of saccades and reaching movements to both identify and appropriately act on items associated with our current goals (e.g., find a ripe tomato in the produce section, grasp it, and place it in the grocery basket). Moreover, targets for action are often surrounded by distractors – task-irrelevant objects that share features with a target object (e.g., the red bell peppers across the aisle from the tomatoes). Indeed, research suggests that selection history effects such as PoP are similarly robust in action contexts; the repetition of target features across consecutive trials has been shown to increase both the speed and accuracy of eye movements (McPeek, Maljkovic, & Nakayama, 1999; Becker, 2008) and reach movements (Song & Nakayama, 2006; Song, Takahashi, & McPeek, 2007; Moher & Song, 2014) relative to trials in which the target features are switched.

The finding that attention effects such as PoP generalize across tasks that require reach or saccade selection is consistent with mounting evidence for the role of a shared, effector-independent priority map that guides target selection (Song et al., 2007; Zehetleitner, Hengenloh, & Müller, 2011; Song, Rafal, & Mcpeek, 2011; Moher & Song, 2014; 2016; Makwana et al., 2023). A number of recent neuroimaging studies suggest that the cognitive processes that support goal-directed actions are instantiated in a more widespread, distributed network of brain regions than previously thought, and that in many cases these regions directly overlap with those traditionally associated with visual attentional selection (for review, see Gallivan & Culham, 2015). For example, the frontal eye fields (FEF), intraparietal sulcus (IPS), and superior colliculus – regions often implicated with effector-specific oculomotor target selection – all appear to play crucial roles in target selection for upcoming reach movements (Cisek & Kalaska, 2010; Gallivan et al., 2011; Song & McPeek, 2011; Song, Rafal & McPeek, 2011). Additional research examining the neural substrates of sensorimotor control suggests that even motor-specific brain regions often encode information in an abstract, effector-independent manner (Haar et al., 2015; 2017; Liu et al., 2020; 2021; Gallivan et al., 2013). For example, Haar and colleagues (2017) used fMRI to decode the direction of reach movements from primary motor cortex (M1) in both the left and right hands. Despite the principal role of M1 in controlling the contralateral side of the body, they showed that fine-grained activation patterns in M1 could reliably predict the direction of movement in both the contralateral and ipsilateral arms (Haar et al., 2015; 2017). Likewise, goal-directed action intentions are associated with similar neural representational topographies in dorsal premotor cortex (PMd) and reach-related regions of posterior parietal cortex regardless of whether a reach is executed with the left or right hand (Gallivan et al., 2013). Collectively, these results open the possibility that selection history effects such as PoP might be driven by common underlying neural mechanisms not only when comparing perception and action-based selection, but also across independent effectors during target selection for action.

Two recent studies by Moher and Song (2014; 2016) provided behavioral evidence in support of the neurobiologically-informed hypothesis that PoP relies upon a shared, effector-independent representation of selection history. Using adaptations of the color singleton task employed originally by Maljkovic and Nakayama (1994), the authors found that automatic biases toward recently selected color singletons transferred across both key-press and action-based selection (Moher & Song, 2014) and across hand and eye effectors from trial-to-trial (Moher & Song, 2016). For example, by randomly intermixing trials that required target selection via reach or saccade, repeating the target and distractor colors led to more efficient responses regardless of whether the effector switched or was repeated from the previous trial. These results persisted even when participants were required to maintain central fixation on reach trials. Beyond showing that classic PoP effects extend to more dynamic selection that involves a mixture of actions, these results provide direct behavioral evidence that color priming relies on effector-independent representations. Moreover, recent experience related to inhibitory control also appears to transfer readily across effectors; anti-saccade training has been shown to benefit performance on a subsequent keypress-based Simon task (Verghese et al., 2018), and previewing distractor colors on a previous trial requiring target selection in one modality (e.g., hand) facilitates distractor suppression on the following trial in another modality (e.g., eyes; Moher & Song, 2016).

Critically, several recent imaging studies have examined the neural correlates of PoP in visual perception (Kristjánsson et al., 2007; Rorden et al., 2011; Brinkhuis et al., 2020), which both provide a strong basis for comparison and establish several candidate regions that may similarly represent selection history during target selection for action. In a first study, Kristjánsson and colleagues (2007) used fMRI to test whether color singleton priming was associated with repetition suppression – a decrease in neural activity associated with a presented stimulus following consecutive presentations (Grill-Spector et al., 2006). By contrasting the BOLD activity evoked when target colors were repeated versus switched, their study revealed several regions belonging to a broader “frontoparietal attention network” – including IPS and FEF (Corbetta & Shulman, 2002; Yantis & Serences, 2003; Scolari et al., 2015) – that exhibited activation profiles consistent with repetition suppression. However, it is important to note that the relative decrease in neural activity observed for repetition trials could also be interpreted as a relative increase in activity associated with switch trials, and this latter interpretation has since been supported empirically. Specifically, Rorden and colleagues (2011) contrasted the activity evoked for repeat and switch trials with a third, novel color condition. They found that both repeated and novel targets led to similar engagement of frontoparietal areas, while target switches were associated with significantly greater activation in these areas when compared to novel targets. Regardless, based on the few neuroimaging studies that have directly examined PoP, it is clear that IPS and FEF – visual attention regions that have also been associated with target selection for action – are sensitive to color priming.

Despite strong converging evidence that PoP may depend on a motor-unspecific priority map, the extent to which this information is encoded in overlapping or distinct brain regions across different response modalities is currently unknown. For example, behavioral results demonstrating a transfer of feature priming across different types of actions could also be explained by parallel, motor-specific representations of target selection history that share reciprocal connections to guide behavior at some later stage of processing. Here, we aimed to explicitly test these possibilities by combining fMRI with both reach- and saccade-based color priming tasks. We reasoned that regions involved in the effector-independent representation of PoP should be engaged in both hand and eye selection tasks, as revealed by standard univariate and conjunction analyses, and moreover, should contain information about recently selected targets that generalizes across eye and hand selection, which we test directly via multivoxel cross-classification analysis. Regions meeting these criteria may enable PoP to be a robust effect in complex, real-world scenarios that involve multiple coordinated actions.

Materials & Methods

Participants

Twenty healthy adult volunteers (14 male, 6 female; aged 20 – 34 years) were recruited from the Brown University community to participate in the study for $20/hr. The desired sample size of 20 was determined based on large behavioral effect sizes for color priming with reach and saccade selection with a sample size of 18 in Moher and Song (2016; Exp. 1), and the sample sizes used in recent fMRI studies testing for neural correlates of PoP which ranged from 6 – 21 (Kristjánsson et al., 2007; Rorden et al., 2011; Brinkhuis et al., 2020). All participants were neurologically healthy, had normal or corrected-to-normal vision and normal color vision, and were free of neuromuscular impairment. Two participants were excluded from analysis due to insufficient data (completing 4 or fewer functional runs), and 2 additional participants were excluded due to excessive head motion leading to registration failures, resulting in a final analyzed sample size of 16. All protocols were approved by the Brown University Institutional Review Board, and participants provided written informed consent in accordance with the Declaration of Helsinki.

Stimuli and Procedure

Visual stimuli were created using MATLAB and presented via Psychophysics Toolbox (Kleiner et al., 2007) on a Mac. Stimuli were presented at 60 Hz and projected onto a mirror in the scanner bore positioned above the participant’s head. Trials began with a black fixation cross presented at the center of the display on a white background for 500 ms (Fig. 1). Subsequently, four diamonds (subtending approximately 1.6° of visual angle) were presented for 1 s. The diamonds were arranged equidistant from each other at 1:30, 4:30. 7:30, and 10:30 on an imaginary clockface that surrounded fixation at a radius of approximately 4.5° of visual angle. The diamonds were rendered in either red (RGB: 220, 60, 10) or green (RGB: 0 134 0). The target was defined as the diamond that was differently colored from the remaining three diamonds. The target color and location were pseudo-randomly selected on each trial using fourth-order counterbalancing (Brooks, 2012). There were an equal number of red and green targets, and equal number of targets appearing at each of the four locations, and an equal number of target color repetitions and switches in each run. Each trial was followed by an intertrial interval (ITI) during which a blank white screen was presented. The duration of the ITI was either 2.5 or 4 seconds, pseudo-randomly distributed.

Figure 1.

Schematic illustration of the target selection task. Participants were prompted to select the uniquely colored targets via saccade (EYE) or reach (HAND) at the beginning of each scanning run. The top row depicts a trial sequence where the target and distractor colors are repeated on trial n relative to trial n – 1, while the bottom row depicts a trial sequence where the target and distractor colors are switched.

Participants completed between 6-8 functional runs. Each run consisted of 68 trials (with an equal number of repeat and switch trials) and lasted 346.5 seconds. Participants were given both verbal and written instructions prior to each block indicating whether it was an EYE or HAND block. On EYE blocks, participants were instructed to move their eyes to the target on each trial while keeping their finger at the center of the touchpad. If it was a HAND block, participants were instructed to move the cursor to the target on each trial while keeping their eyes at the center. The order of EYE and HAND blocks occurred pseudo-randomly within subjects. The participant controlled the cursor with a touchscreen device situated above their midsection. The experimenter monitored eye and hand movements in real time and gave verbal feedback between blocks if the participant was not complying with instructions. At the end of each run, a fixation cross was displayed alone for 8 seconds to capture the entire hemodynamic response from the last trial of the run.

During a pre-scanning behavioral session, each participant completed a training run lasting approximately 30 minutes. The training run used the same primary task of making either a hand or eye movement to the singleton color target, but in a set-up outside the scanner with the participant sitting upright. During the training session, participants were given auditory feedback to indicate whether they responded correctly and were given an auditory warning tone when they moved their eyes during a hand trial or vice versa.

Eye and Reach Tracking

Eye position was recorded during the task using a MR-compatible an IVIEW X MRI-LR system (SMI, Tetlow, Germany) sampling the right eye at 60 Hz for the first 4 participants and an EyeLink 1000 infrared eye tracker (SR Research, Ontario, Canada) sampling the right eye at 500 Hz for all subsequent participants. Importantly, we found that the patterns of saccade data collected with both eye trackers was qualitatively matched across measures with the exception of curvature; accordingly, we excluded these 4 participants from the saccade curvature analysis. Hand position was recorded via a MR-compatible digitizing tablet (Magic Touch; Tyco Touch, Inc.) measuring 17 x 12.5 cm, which was placed horizontally on the participant’s midsection and aligned with their midline and the center of the projector screen. Both the eye and reach tracker were calibrated at the beginning of each run using default software settings, and the eye tracker was validated with <1° average cartesian prediction error.

Two-dimensional reach and eye movement data were analyzed offline using custom MATLAB scripts. For reach movements, speed scalars were created for each trial using a differentiation procedure and then smoothed using a 2nd order, low-pass Butterworth filter with a cutoff of 10 Hz. We defined movement onset as the first time point on each trial after stimulus onset at which reach movement speed exceeded 25.4 cm/s (consistent with previous research in our lab; e.g., Moher & Song, 2013; 2014; 2016). The first subsequent measurement on each trial when speed decreased to below 25.4 cm/s determined movement offset. Initiation latency was defined as the time elapsed between stimulus onset and movement onset. Movement time was defined as the time elapsed between movement onset and movement offset. Likewise, parameters for initiation latency and movement time were extracted for eye movement data with a speed threshold for defining the beginning and end of a movement set at 35°/s (Godijn & Theeuwes, 2004; Ludwig & Gilchrist, 2002; Moher & Song, 2016) using a 40 Hz filter cutoff for the Butterworth filter. For both effector conditions, curvature was calculated as the maximum unsigned deviation of the movement in cm relative to a straight line between the start and end points of the movement, divided by the length of the line.

All behavioral measures not including accuracy were restricted to correct trials only. For both eye and hand movements, individual trials were visually inspected to detect any cases where the default threshold clearly missed part of the movement or included substantial movement back to the starting point (Song & Nakayama, 2006; Moher & Song, 2013; 2014; 2016).

Image Acquisition

Imaging data were acquired on a 3.0 T Siemens Prisma MRI scanner at the Brown University MRI Research Facility. Structural images were acquired in the sagittal plane using MPRAGE whole-brain anatomical scans (TR = 1.98 s; TE = 3.02 ms; θ = 9°; matrix = 256 × 256 mm; slice thickness = 1.0 mm, slices = 160). Functional images were acquired using a single-shot T2*-weighted gradient echo EPI sequence (TR = 2.25 s; TE = 30 ms; θ = 90°; °; matrix = 96 × 96; slice thickness = 2 mm with 2 mm gap).

fMRI Analysis and Preprocessing

Functional data were preprocessed and analyzed using AFNI and anatomical preprocessing was performed using Freesurfer. Images were skull stripped, motion corrected, de-obliqued, and visually inspected. For all univariate analyses, data were spatially smoothed using an 8 mm FWHM Gaussian kernel. No spatial smoothing was applied to the single-trial regression data used for multivoxel pattern analysis. First-level statistical maps obtained via AFNI’s 3dREMLfit were registered to the Montreal Neurological Institute (MNI)-N27 template using an affine transformation to align the functional image to the structural image and to subsequently align the structural image to the MNI-N27 template.

The four critical regressors in the first-level analysis included the onsets of correct color repeat hand trials, color switch hand trials, color repeat eye trials, and color switch eye trials. Nuisance regressors included trials in which subjects responded incorrectly or failed to make a response, realignment parameters from motion correction, their temporal derivatives, and volume-wise indicator variables for censoring volumes that exceeded a framewise displacement of 0.9 mm (Siegel et al., 2014). Task-based regressors were convolved with AFNI’s standard hemodynamic response function. Group analyses to contrast the relative activation evoked for target repeat versus target switch trials were performed using 3dANOVA3 and 3dttest++ where appropriate. Final statistical maps for the univariate analysis used a voxel significance threshold of p < .001 and were corrected for multiple comparisons at p < .05 using AFNI’s nonparametric 3dClusterSim.

Exploratory Conjunction Analysis and ROI Selection

Our primary research questions surround the neural basis of effector-independent selection history effects. However, because target repeat and switch trials share the same visual displays and response requirements on average, this could present a challenge for standard univariate fMRI analyses that test for spatially distributed differences in mean neural engagement across conditions. We predicted that some regions associated with selection history representation may exhibit small activation biases that are difficult to detect using more conservative (e.g., standard) thresholds via univariate analyses.

To both account for this possibility and enable further ROI-based analyses, we conducted an exploratory conjunction analysis (Friston et al., 1999) as an additional step to reveal whether and where selection history may be encoded in an effector-general manner, consistent with previous behavioral evidence (Moher & Song, 2014; 2016). For this analysis, we first performed two whole brain contrasts at the group level for color repeat > switch restricted to hand and eye runs, respectively, using an uncorrected voxel threshold of p < .05. The resulting statistical maps were binarized and combined using AFNI’s 3dCalc to create two output files corresponding to 1) above-threshold voxels weighted for target repetitions across both effectors and 2) above-threshold voxels weighted for target switches across both effectors. Critically, anatomical regions containing significant overlapping voxels for both eye and hand runs were then selected as ROIs for subsequent cross-classification analyses (Fig. 2) and brain-behavior correlations.

Figure 2.

Selection history cross-classification approach. We first performed conjunction analyses to identify anatomical regions containing voxels that were sensitive to selection history in both effector conditions. Next, for each identified ROI, we trained an SVM classifier to distinguish color repeat/switch trials based on multivoxel activation patterns associated with one effector. Finally, each model was cross-validated using independent scanning data from the other effector. This example depicts a model trained on reach runs and tested on saccade runs.

Multivoxel Cross-classification Analysis

After identifying effector-general ROIs associated with selection history, we sought to test whether a model trained on multivoxel activation patterns from one effector (e.g., eye runs) could accurately predict whether target colors were repeated or switched in the other effector (e.g., hand runs) in each of the identified ROIs (see Fig. 2). Successful cross-classification of selection history in a given ROI would suggest that beyond showing more relative activation on repeat or switch trials, the ROI contains relevant information about selection history that can be used to guide behavior independent of the physical mode of selection.

To this end, we obtained train-by-trial estimates of the hemodynamic response in AFNI for stimulus onsets on correct target repeat and target switch trials within each respective effector. These estimates were restricted to anatomical ROIs corresponding to overlapping regions identified via conjunction analyses projected into subjects’ native space. We then performed subject-specific voxel selection by further restricting the analysis to the top 5% of beta values (two-tailed) in the training data set, such that voxel selection for a given participant was fully independent of their test data.

Following voxel selection, we trained a support vector machine (SVM) exclusively on a participant’s eye or hand runs and iteratively tested its generalization accuracy to predict color repeat versus color switch trials in the other effector. To maximize the training data contributing to the model, we opted to train the SVM on all runs from one effector and iteratively test the model’s accuracy on each run from the other effector, resulting in 6-8 unique predictions per ROI per subject that were subsequently averaged. The mean accuracy estimates for each ROI across all subjects were then evaluated against chance performance and subjected to further statistical analysis.

Results

Behavioral Results

We first sought to verify that the expected repetition priming effects were present for both reach and eye movements, marked by a facilitation of target selection following target color repetitions. To this end, initiation latency on correct trials was used as the primary behavioral measure, given that it is both sensitive to intertrial priming effects and yields comparable estimates for hand and eye movements (Fig. 2; Moher & Song, 2014; 2016). A 2x2 mixed ANOVA with factors for effector (eye/hand), color repetition (repeat/switch), and participant as a random effect revealed significant main effects of color repetition, F (1, 30) = 36.4, p < .001, and effector, F (1, 15) = 39.6, p < .001. The effector x repetition interaction effect was not significant, F (1, 30) = .37, p = .54, suggesting that the magnitude of color priming effects was matched across eye and hand selection conditions despite relatively slower initiation latencies for reach (M = 477 ms) relative to saccade (M = 364 ms) selection, t (15) = 6.29, p < .001, d = 1.57. As expected, post-hoc t-tests revealed significant priming effects in both effector conditions, such that target-directed movements were initiated more quickly when the target color was repeated versus switched relative to the previous trial for both eye selection, M_diff = 16.5 ms, t (15) = 3.76, p = .002, d = .94, and hand selection, M_diff = 20.3 ms, t (15) = 4.80, p < .001, d = 1.20. We found no significant effects of target color repetition on behavior in accuracy, movement time, or curvature, however, these results are summarized in Table 1.

Table 1.

Mean behavioral effects across task conditions associated with selection history.

Dependent variable	Color repeat	Color switch
Saccade
Initiation Latency	355 ± 92 ms	372 ± 91 ms
Movement Time	155 ± 186 ms	136 ± 140 ms
Curvature	.16 ± .05	.21 ± .08
Accuracy	89 ± 17%	87 ± 20%
Reach
Initiation Latency	467 ± 67 ms	487 ± 71 ms
Movement Time	199 ± 24 ms	198 ± 22 ms
Curvature	.09 ± .09	.06 ± .03
Accuracy	98 ± 2%	99 ± 3%

Consistent with prior research (Bichot & Schall, 2002; Fectau, 2007; Makwana et al., 2023), the effects of location repetition on behavior were notably weaker than those associated with color repetition. Although subjects were numerically faster to respond when locations were repeated, we did not observe a significant main effect of location repetition on RT, F (1, 30) = 2.68, p = .11, nor an effector x location repetition interaction, F (1, 30) = 0.49, p = .49, in contrast to color repetition which had a strong effect on RT. However, we note that because the goal of the study was to study priming effects for the target-defining feature (color), location repetitions occurred relatively infrequently in the current design (25% of trials).

To summarize, we confirmed that our target selection tasks produced automatic attention biases toward recently selected target features (Maljkovic & Nakayama, 1994). Moreover, our behavioral results establish the feasibility of studying priming effects for hand and eye movements in a scanning environment using an MRI-compatible touch surface and simultaneous eye-tracking.

Univariate fMRI Results

After verifying the presence of reliable priming effects in both the reach and saccade tasks, we performed univariate fMRI analyses to test the overarching hypothesis that target selection history may be encoded via common, effector-general neural mechanisms. Notably, previous studies using keypress responses identified several regions of the frontoparietal attention network associated with intertrial priming effects, showing that bilateral IPS and FEF were more activated on color switch relative to repeat trials (Kristjánsson et al., 2007; Rorden et al., 2011). The current analysis enables us to build on these findings in two important ways. First, are frontoparietal structures similarly recruited on switch trials during target selection for action? And second, are PoP effects in reaches and saccades associated with common, or distinct patterns of neural activity?

To address these questions, we conducted a 2x2 ANOVA to test for main effects of effector (eye > hand), color repetition (repeat > switch), and their interaction on the BOLD response, restricted to correct trials only (Fig. 4; Table 2). Critically, a main effect of color repetition would suggest effector-general involvement of the identified region(s); the presence of an interaction effect would point to region(s) that are associated with selection history in an effector-specific manner.

Figure 4.

Univariate imaging results. (a) Effector-independent activation in right IPS associated with color switch relative to repeat trials (p < .05, corrected). (b) Interaction effect in left dAIC (p < .05, corrected). Uncorrected pairwise comparisons revealed that dAIC was more activated on color repeat trials for hand runs but was more activated on color switch trials for eye runs.

Table 2.

Significant cluster and peak information for the univariate imaging results in Figure 4.

Contrast	Region (mni-glasser)	Peak t-value	Rad coordinates (x, y, z)	Number of voxels	Cluster p
Switch > Repeat
	Right medial intraparietal area	5.82	−28, 66, 54	171	p < .001
Interaction: Color Repetition x Effector
	Left middle insular area	5.15	38, −4, 6	54	p = .04

As expected, we observed several significant activations for the main effect of eye > hand, including supplementary motor area (SMA), dorsal PFC, cerebellum, and left-lateralized motor cortex for hand selection; conversely, visual cortex showed more relative activation for eye selection. Given that our primary hypotheses surround the neural representation of PoP, we do not interpret these motor-specific activations further.

For the main effect of color repetition, we found that right IPS showed significantly greater activation for color switches compared to repetitions (p < .05, cluster-corrected; Fig. 4a). No regions showed greater activation for repeat relative to switch trials. Notably, the increased activation of right IPS for switch trials both replicates and extends previous findings on the neural substrates of repetition priming in keypress tasks (Kristjánsson et al., 2007; Rorden et al., 2011), suggesting that this region is also sensitive to intertrial priming in action contexts. Moreover, the consistent activation of right IPS across reach, saccade, and keypress tasks provides converging evidence that this region represents target selection history in an effector-general manner, possibly through its strong association with both spatial- and feature-based attention (Scolari et al., 2015).

Interestingly, we also found a significant effector x color repetition interaction effect in the left dorsal anterior insular cortex (dAIC), suggesting that this region exhibited distinct activation patterns in response to different repetition conditions for eye versus hand runs (p < .05, cluster-corrected; Fig. 4b). Uncorrected pairwise comparisons revealed that the interaction effect was driven by greater activation in dAIC on repeat trials during hand runs, and greater activation of dAIC on color switch trials during eye runs. These results provide new evidence that the left dAIC is associated with intertrial priming in different directions depending on the effector: Reach selection was accompanied by more activation of dAIC when targets were repeated, with the opposite being true of saccade selection. Previous research suggests that the dAIC plays an important role both in the capture of focal attention (Nelson et al., 2010) and intentional action selection (Brass & Haggard, 2010; Droutman et al., 2015). We further address the significance and interpretation of these effector-specific selection history effects in dAIC in the discussion.

Conjunction Analysis Results

In the standard univariate analysis detailed above, we show that right IPS was more engaged for color switch relative to repeat trials on average, suggesting that this region may play a role in the effector-general representation of selection history. However, we suspected that additional brain regions carrying information about selection history may exhibit subtle, but reliable activation biases that are difficult to detect via standard univariate analyses. In the color singleton task used here, color repeat and switch trials share the same response demands and visual displays, differing only with respect to their relationship to the previous trial. Thus, differences in the relative engagement of regions or functional networks related to selection history are likely to be less pronounced than when contrasting activation between different stimuli or tasks.

We therefore performed an exploratory conjunction analysis to identify candidate regions that may carry information about recent target selection history from trial to trial (including, but not limited to IPS). Using an uncorrected voxel threshold of p < .05, this more liberal analysis enabled us to identify regions with small activation biases in the same direction across both effector conditions (e.g., repeat > switch for eye and hand effectors), and eventually test whether the identified ROIs contain information about repetition conditions that can be used to cross-classify and track participants’ behavior.

The results of the conjunction analysis are depicted in Fig. 5. The top panel shows voxels with greater relative activation on color switch trials, and the bottom panel shows voxels with greater relative activation on color repeat trials. In both panels, the red regions correspond to significant overlapping voxels activated in the indicated direction across both effectors; orange regions correspond to those only activated during hand runs, and blue regions correspond to those only activated during eye runs. As anticipated, this analysis revealed several effector-general clusters that were not detected via ANOVA, including two clusters containing overlapping voxels that were more activated for repeat > switch trials. For the switch > repeat contrast, we found mutually significant voxels for eye and hand selection in both right and left IPS, consistent with our expectations based on the initial univariate analysis. We also identified three new regions containing similar patterns of activation for switch trials across eye and hand runs: right lateral occipitotemporal cortex (LOC), cerebellum, and left dorsolateral prefrontal cortex (DLPFC). Notably, this DLPFC activation corresponded to an area of the middle frontal gyrus that is adjacent to, but more rostrally located than FEF as it is traditionally defined.

Figure 5.

Effector-independent activations revealed by conjunction analyses. Labels correspond to anatomical regions containing overlapping significant voxels (red; p < .05, uncorrected) subsequently used to select ROIs for cross-effector selection history classification. Faded blue (eye) and orange (hand) regions met an uncorrected voxel threshold of p < .05 for a single effector but did not pass the conjunction analysis.

For repeat > switch, we found clusters with voxels passing the conjunction threshold in the medial temporal lobes (MTL) including bilateral hippocampus, amygdala, and parahippocampal gyrus, along with another cluster containing overlapping voxels in the left temporal pole and superior temporal gyrus (STG). Given established roles of both MTL and STG in memory encoding and retrieval, it is possible that these regions contribute to effector-general memory processes related to target identification and recent selection history.

While each of the regions identified in this analysis necessarily exhibited sensitivity to color priming, here their activation is expressed as greater mean engagement on one trial type relative to the other. Thus, we note that these differences could also be correctly interpreted as deactivation or suppression in the condition they are computed relative to (e.g., right IPS is relatively deactivated when target colors are repeated).

Cross-classification Results

In the conjunction analysis detailed above, we identified six ROIs that could serve an important role in the effector-general representation of target selection history: MTL and left STG, which contained mutually significant voxels on color repeat relative to switch trials, and IPS, right LOC, left DLPFC, and cerebellum, containing mutually significant voxels that were more activated for color switches. These anatomical ROIs were then used to spatially restrict subsequent SVM-based cross-classification analyses. For MTL and IPS, which showed bilateral activations in the conjunction analysis, we used only ROIs from the hemisphere that exhibited the highest activation peak at the group level for cross-classification (left MTL; right IPS).

Following ROI selection, we trained the models exclusively on data from scanning runs from one effector (e.g., hand), and iteratively tested model generalization for target selection history on runs of the other effector (e.g., eye). The target repeat/switch test prediction accuracy from each model iteration was then averaged within each ROI for each participant. One participant was excluded from classification analyses due to average model performance more than 3 standard deviations below the mean across all tested ROIs.

First, we tested the mean cross-classification performance for each ROI against chance performance (Fig. 6). We identified two ROIs that successfully predicted target repetitions versus switches in runs of the opposite effector at a rate greater than expected by chance: left MTL, t (15) = 3.08, p = .008, d = .80, and right IPS, t (15) = 2.81, p = .014, d = 0.73, suggesting that these ROIs contained enough effector-general information about selection history to predict outcomes in orthogonal scanning runs that required a different physical response for target selection. Cerebellum, DLPFC, STG, and LOC failed to significantly differentiate repeat versus switch trials across effectors (all ps > .10). In summary, beyond showing sensitivity in mean engagement across different selection history conditions, the successful cross-classification of target selection histories in MTL and IPS suggest that these regions contain effector-general information that could account for the transfer of priming effects from eye to hand selection (and vice versa) on a trial-to-trial basis.

Figure 6.

Group cross-classification results for each ROI and their associations with behavioral priming effects. In the left panel, the dashed line corresponds to decoding accuracy at chance (50%). Error bars depict between-subject SEM. Decoding accuracies in left MTL and right IPS were both significantly above chance and were positively associated with the magnitude of subjects’ behavioral priming effects in initiation latency, averaged across eye and hand runs. *p < .05; **p < .01.

Given the reliable cross-effector selection history decoding effects in MTL and IPS, a critical question is whether the model’s decoding performance at the subject level is linked to an individual’s behavior for a given ROI. Specifically, decoding accuracy should be highest among participants who exhibited the largest priming effects across hand and eye runs; alternatively, participants who were relatively insensitive to color priming would not be expected to have reliable neural signatures of the effect apparent in decoding accuracy. To address this question, we averaged subjects’ priming effects in initiation latency across both hand and eye runs to create a measure of global priming sensitivity, and then performed Pearson correlations to test the association between priming sensitivity and cross-effector decoding accuracy (Fig. 6). Remarkably, participants’ average behavioral priming effects were significantly associated with their cross-effector decoding accuracies in MTL, r = .57, 95% CI [.09, .84], t (13) = 2.53, p = .03, and IPS, r = .53, 95% CI [.02, .82], t (13) = 2.25, p = .04, suggesting that these regions not only carry relevant effector-independent information about selection history, but that individuals who are more sensitive to intertrial priming exhibit stronger neural signatures of selection history in MTL/IPS.

These results provide the first direct demonstration that effector-general neural representations guide PoP. Notably, none of the subject decoding accuracies in ROIs that failed to exceed chance decoding performance correlated with behavioral priming effects (all ps > .10), providing clear support for the validity of our cross-classification approach. These results also open the possibility that dissociable regions are responsible for attentional facilitation (MTL on repeat trials) and interference (IPS on switch trials) effects in behavior.

Discussion

In the present study, we tested the hypothesis that target selection for action across different effectors is guided by shared underlying neural representations of selection history. We identified two regions – IPS and MTL – that met the criteria for serving such a role. Specifically, we found that both IPS and MTL exhibited effector-independent sensitivity to whether the features of an action target were repeated or switched, and that fine-grained patterns of activity in these regions could be used to reliably decode selection history across different physical modes of selection. Moreover, these activity patterns strongly tracked individual differences in behavioral sensitivity to color priming regardless of whether target selection was performed with the eyes or hands. Finally, we provide new evidence that the anterior insular cortex may play a role in the effector-specific representation of selection history.

To our knowledge, this is the first study to examine the neural substrates of PoP in target selection for action. Our MRI-adapted reach- and saccade-based target selection tasks yielded clear behavioral evidence of automatic attention biases toward recently selected target features of a similar magnitude to previous behavioral studies (McPeek, Maljkovic, & Nakayama, 1999; Becker, 2008; Song & Nakayama, 2006; Song, Takahashi, & McPeek, 2007; Moher & Song, 2014; 2016), establishing the feasibility of studying the effects of selection history on different forms of action-based selection with fMRI.

Selection history modulates IPS activation in both perception and action-based selection

Our univariate neuroimaging results provide initial evidence that the brain regions associated with recent selection history in PoP are similar across action and perception contexts, consistent with the prediction that an effector-independent priority map encodes information about recently selected targets (Song et al., 2007; Zehetleitner, Hengenloh, & Müller, 2011; Song, Rafal, & Mcpeek, 2011; Moher & Song, 2014; 2016). Several studies using fMRI to investigate the neural correlates of PoP in keypress-based perception tasks revealed that activation in a network of frontoparietal structures including IPS and FEF covaries with target selection history (Kristjánsson et al., 2007; Rorden et al., 2011; Brinkhuis et al., 2020). Namely, activity in these regions has been consistently shown to decrease on trials where target features and target/distractor configurations are repeated from a previous trial, and conversely, increase when these features are switched. By contrasting the activity evoked for target color repetition versus switch trials, here we show that right IPS in particular exhibits this same pattern of selection history-related activation across two independent action-based selection tasks. As part of a broader frontoparietal attention network, the IPS has been strongly implicated in supporting endogenous visual attention processes (Corbetta & Shulman, 2002; Yantis & Serences, 2003; Scolari et al., 2015), and has been more recently identified as a region that encodes effector-independent information about upcoming action plans (Gallivan et al., 2011; 2013). Thus, in contrast to early investigations on the neural correlates of PoP that suggested an association with neural repetition suppression (Kristjánsson et al., 2007), we predict that the increased activity of IPS on target switch relative to repeat trials likely reflects the increased attentional demands required to identify and select new target features and/or target-distractor configurations (Rorden et al., 2011; Brinkhuis et al., 2020).

Selection history is associated with effector-specific activation of dAIC

Our task design also enabled us to test for regions that may be instead associated with an effector-specific representation of selection history. These analyses revealed that the left dorsal-anterior insular cortex (dAIC) was significantly modulated by color repetition, but that its activation patterns were action-dependent: color repetitions were associated with more relative engagement of dAIC during reach runs, and less engagement of dAIC during saccade runs. The insula exhibits a high relative base rate of activation across fMRI studies (Duncan & Owen, 2000; Yarkoni et al., 2011; Chang et al., 2013) and has been associated with a wide array of sensory and cognitive functions, including but not limited to interoception, emotional processing, executive control, and goal-directed attention (e.g., Damasio et al., 2000; Dosenbach et al., 2006; Craig, 2009; Nelson et al., 2010). While the posterior and ventral-anterior portions of insula are most strongly linked to somatosensory and emotional processing, respectively, the dAIC is specifically implicated in cognitive control, attention, and goal-directed behavior (Crobetta et al., 2002; Dosenbach et al., 2006; Nelson et al., 2010; Chang et al., 2013).

Although its diverse contributions to cognition make it difficult to infer its precise role in the current task, dAIC is part of a connectivity-based “salience network” that drives attentional control when salient or behaviorally relevant stimuli are encountered (Seeley et al., 2007; Sridharan et al., 2008; Menon & Uddin, 2010; Uddin, 2015). Thus, it is possible that the increased activation of dAIC for repeated targets in reach relative to saccade selection reflects underlying differences in saliency processing across different modes of selection. Activation of dAIC may also reflect the degree to which behavioral responses are driven by intentional, goal-directed processes versus external stimuli (Jenkins et al., 2000; Brass & Haggard, 2010). From this perspective, the effector by color repetition interaction observed for dAIC may reflect a tradeoff where selecting a recently-encountered target feature is driven more so by internally-guided processes during reach selection, and more likely to be automatically generated based on the stimulus properties in saccade selection. However, more research is needed to shed light on these potential explanations.

Multivoxel patterns in IPS and MTL encode effector-independent representations of selection history

One challenging aspect of using fMRI to test for regions that may encode selection history-related information is that both explicit target selection demands and visual stimulus properties are closely matched across repeat and switch conditions. We therefore predicted that additional brain regions beyond IPS and dAIC may contain behaviorally relevant representations related to recent target features, but that such regions may not reliably differ in mean engagement across repeat and switch trials. To test this prediction, we first employed voxelwise conjunction analyses across effector conditions to identify candidate regions. We accordingly found several ROIs with overlapping activity in response to target color repetition across both eye and hand selection runs, including IPS, LOC, DLPFC, STG, cerebellum, and MTL. For each identified ROI, we subsequently built SVM models designed to cross-classify repeat and switch trials from one effector to another across independent scanning runs. This approach enabled us to 1) establish whether there is predictive information in a given ROI that differentiates repeat from switch trials, and 2) if so, test whether the information that distinguishes trial types explicitly tracks individual differences in behavioral sensitivity to intertrial priming. We ultimately found two regions that met these criteria: MTL and IPS.

Given that our univariate analyses combined across eye and hand runs revealed significant activation in right IPS on trials where target and distractor colors were switched, it is unsurprising that this region also contained information that can reliably distinguish color repetition conditions across effectors. However, beyond suggesting the gross involvement of IPS in target selection mechanisms guided by selection history (e.g., endogenous attention to select new target features), our MVPA results show that IPS contains similar underlying representations related to recent experience regardless of whether selection is performed via reach or saccade. These results are consistent with the proposed role of IPS and related structures in maintaining an effector-independent priority map (Cisek & Kalaska, 2010; Gallivan et al., 2011; Song & McPeek, 2011; Song, Rafal & McPeek, 2011) which enables selection history information to flexibly guide behavior across different types of action. Whether IPS encodes a memory representation of recently selected targets per se or whether it reflects the downstream attentional effects of a feature match/mismatch which tracks behavior remains an open question.

Our cross-classification analysis also provides new evidence that MTL structures including the hippocampus may be critically involved in maintaining abstract representations of target selection history that guide behavior. Restricting our MVPA analysis to this region, we found that fine-grained activation patterns in MTL reliably decoded target color repetition conditions across effectors. Perhaps more importantly, the strength of effector-independent selection history information in MTL was strongly linked to subjects’ behavioral priming effects.

The MTL and hippocampus in particular serves as the anatomical foundation for declarative memory, marked by the flexible storage and retrieval of facts and experiences (Squire & Zola-Morgan, 1991; Squire, 1992; Cohen & Eichenbaum, 1993). However, the role of MTL in cognition is not restricted to long-term memory alone – rather, a number of studies suggest that MTL encodes implicit representations of recent experiences across trials, consistent with a role in visual statistical learning (Schendan et al., 2003; Turk-Browne et al., 2009; 2010; Theeuwes et al., 2022). Theeuwes and colleagues (2022) recently proposed that the hippocampus could serve as a neural substrate for location-based target selection biases driven by selection history. They hypothesized that the hippocampus encodes a spatial priority map that is biased by spatial regularities (e.g., selection history or target location probabilities), and that this spatial priority map may bias representations in a visual attentional priority map instantiated in structures such as FEF, IPS, and superior colliculus. Although our study focused on feature-based priming for colors, the findings that both MTL and IPS are involved in the effector-independent representation of target color repetition history align well with the predictions made by Theeuwes et al. regarding the neural bases of visual statistical learning in target selection.

Implications for Mechanistic Theories of PoP

The finding that effector-independent representations in IPS and MTL contribute to PoP in action-based selection could help to clarify existing mechanistic accounts of repetition priming in target selection tasks more broadly. Maljkovic and Nakayama (1994; 1996) originally proposed that PoP is a byproduct of implicit memory for recently selected target-defining features, a prediction supported by the finding that explicit knowledge of upcoming target-distractor configurations did not extinguish the effect. However, several alternative accounts have posited a more direct role for episodic or declarative memory in PoP. For example, Hommel (2004; 2005) suggested that the perceptual, contextual, and response-related characteristics on a given target selection trial could be bound into discrete event files. Thus, a subsequent trial matching the characteristics of a stored event file leads to its retrieval, and ultimately an efficient replication of the action that was previously executed. At face value, our finding that MTL distinguishes target selection history is compatible with this explanation, given the strong link between MTL and declarative memory processes. However, because recent neuroimaging evidence also implicates MTL in the implicit encoding of task regularities (Schendan et al., 2003; Turk-Browne et al., 2009; 2010), our current results cannot determine whether MTL’s role in repetition priming is strictly explicit, implicit, or reflects some combination of these two processes. Instead, our finding that brain regions linked to both attentional control (IPS) and memory (MTL) track selection history is consistent with the broader prediction that priming is likely supported by multiple anatomically- and functionally-distinct structures which contribute in a task-dependent manner (Kristjánsson & Campana, 2010).

It is notable that selection history-related activity in MTL as indicated by conjunction analyses was weighted for target repetition trials relative to switch trials, in contrast to IPS which showed the opposite activation bias. These results open the possibility that dissociable brain regions are responsible for different history-dependent cognitive effects including those associated with attentional facilitation on repetition trials, and those associated with inhibition and/or interference on switch trials. For example, it is possible that MTL plays a unique role in the encoding and retrieval of target features across trials to facilitate the localization of goal-relevant features, whereas IPS could contribute by inhibiting or facilitating prepotent motor responses for these attended features. To test these possibilities, future imaging studies would be well-served to leverage modified PoP task designs (e.g., employing target-absent trials, partial feature repetitions, or additional colors not associated with previous targets/distractors; Kristjánsson & Driver, 2008; Lamy et al., 2008; Rorden et al., 2011; Moher & Song, 2016; Makwana et al., 2023) to further clarify whether the contributions of MTL and IPS to color priming are based on distinct or similar underlying attentional mechanisms.

One such direction for future research that could further dissociate the neural substrates involved in action priming would be to directly contrast trials requiring target selection for action with standard perceptual target selection (e.g., keypress). While our results here suggest that different actions share similar underlying history representations, this approach could reveal whether engaging in action itself drives different patterns of motor-unspecific neural activity. This is a distinct possibility, given previous work demonstrating that executing or preparing for an action can modulate perceptual discrimination performance (Guo & Song, 2019), distractor suppression (Moher, Anderson, & Song, 2015), and visual stimulus representations in sensory cortex (Gutteling et al., 2015).

Another approach could involve leveraging the distractor previewing effect (DPE) to isolate the repetition effects of purely inhibitory components (e.g., Ariga & Kawhara, 2004; Goolsby, Grabowecky, & Suzuki, 2005; Lleras, Kawahara, Wan & Ariga, 2008). In this paradigm, participants perform a similar oddity target search task. However, in some trials, all objects were homogenously colored, requiring no response. If the target color matches the homogeneous items from the previous trial, responses are slower, indicating that the previewed feature is inhibited. On the other hand, if the distractors match the previewed color, responses are quicker. This effect has been extended to saccades and reach target selection and can transfer across different actions (e.g., from an eye movement to a hand movement; Caddigan & Lleras, 2010; Moher & Song, 2016). The brain mechanisms involved in tying the effector to the lack of a target remain unknown and could shed light on intertrial action effects.

PoP is just one example of selection history influencing behavior. Additional research will be needed to understand which of the brain mechanisms discussed here might generalize to other forms of selection history. Furthermore, a number of cognitive processes contribute to the intertrial effects observed in PoP. For example, reward can impact the magnitude of PoP effects (Kristjánsson, Sigurjónsdóttir, & Driver, 2010). Working memory, too, is implicated in PoP effects in various ways (Ahn, Patel, Buetti, & Lleras, 2017; Carlisle & Kristjánsson, 2017). We are hopeful that future research can shed further light on which sub-processes that comprise selection history’s influence on behavior are supported by which distinct brain mechanisms.

Conclusions

In conclusion, our results build on converging behavioral and neural evidence that shared underlying representations of attentional priority guide target selection in a manner that is largely independent of the physical response (Gallivan et al., 2011; 2013; Moher & Song, 2014; 2016; Haar et al., 2015; 2017; Liu et al., 2022; Verghese et al., 2018). Consistent with previous studies using key-press based responses (Kristjánsson et al., 2007; Rorden et al., 2011; Brinkhuis et al., 2020), we show that IPS also tracks target color repetition in target selection for action, marked by more relative engagement when target-distractor configurations are switched between trials. Conversely, we found that dAIC activity was modulated by target color repetition in an effector-dependent manner. Finally, multivoxel analyses revealed that both IPS and MTL share a common representational topography of selection history across effectors which directly guides behavior. Together, these results represent an important step toward understanding the neural basis of experience-driven selection biases in ecologically valid contexts that require the coordination of multiple actions.

Figure 3.

Behavioral effects of target color repetition and switch on saccade (left) and reach (right) initiation latency. Error bars depict between-subject SEM. **p < .01.

Data Availability Statement

Data, materials, and analysis code related to this study will be freely available upon publication at the Open Science Framework: https://osf.io/9uqjb/