Dissociating the impact of attention and expectation on early sensory processing

Abstract

Most studies that focus on understanding how top-down knowledge influences behavior attempt to manipulate either ‘attention’ or ‘expectation’ and often use the terms interchangeably. However, having expectations about statistical regularities in the environment and the act of willfully allocating attention to a subset of relevant sensory inputs are logically distinct processes that could, in principle, rely on similar neural mechanisms and influence information processing at the same stages. In support of this framework, several recent studies attempted to isolate expectation from attention, and advanced the idea that expectation and attention both modulate early sensory processing. Here we argue that there is currently insufficient empirical evidence to support this conclusion, because previous studies have not fully isolated the effects of expectation and attention. Instead, most prior studies manipulated the relevance of different sensory features, and as a result, few existing findings speak directly to the potentially separable influences of expectation and attention on early sensory processing. Indeed, recent studies that attempt to more strictly isolate expectation and attention suggest that expectation has little influence on early sensory responses and primarily influences later ‘decisional’ stages of information processing.

Attention, expectation and perceptual inference

Over the past 40–50 years, a tremendous amount of effort has been spent trying to understand how prior knowledge shapes human information processing from the earliest stages of sensory analysis to decision-making to the execution of motor responses. Prior knowledge is a ‘top-down’ modulatory factor to the extent that priors reflect internal states and neural representations that could influence perception and behavior¹. One important ‘top-down’ factor relates to knowledge about the probability that certain stimuli will occur in a specific context: a traffic light is likely to turn red after it turns yellow, a toaster is likely to be on top of a counter instead of under the kitchen sink, and so forth^2,3. These top-down priors also code for more complex statistical regularities about stimulus identity and component features: a building is likely to have structures that are composed of straight lines rather than curvatures. Thus, expectations based on fore-knowledge can exert a powerful influence on object identification and scene understanding^2,3, and a growing body of research focuses specifically on the impact of expectations on early sensory processing^4–6.

Another type of top-down knowledge pertains to the relevance of specific stimuli in the context of current behavioral goals: when looking for your car in the parking lot, knowledge of its color, shape and size can be exploited to improve search efficiency by reducing the set of stimuli that must be interrogated. Critically, expectations about statistical regularities and knowledge about relevant features could have dissociable influences on information processing, as the probability that a stimulus will be encountered in a given context is not necessarily linked to its behavioral relevance^4,5. Thus, following Summerfield and de Lange, we define expectation as the mechanism that operates based on the probability of stimulus occurrence, and we define attention as the mechanism that operates based on the behavioral relevance of different stimuli^4,5.

The classic Posner cueing paradigm highlights the difficulties associated with dissociating the effects of expectation from the effects of attention. The task manipulates the probability that a target stimulus will appear on the left or the right of fixation, and participants have to press a button when they detect the onset of the peripheral light⁷. This manipulation alters the probability that the target stimulus will appear in one spatial location, which in turn leads to faster response times and more accurate responses. This seminal result, which has given rise to thousands of subsequent studies using variants of this basic paradigm, was originally interpreted as evidence for more efficient early sensory processing related to the selective deployment of spatial attention. However, later work demonstrated that these results, as well as results from more complex visual search tasks, can often be explained via an increase in the willingness of participants to indicate that they saw a target at the cued location, irrespective of how much sensory evidence was present to support a ‘yes’ response (i.e. the cue led to a change in decisional factors)^8–11.

This debate about how to interpret what is perhaps the most widely used paradigm in the field of ‘selective attention’ illustrates two important points. First, this simple variant of a cueing paradigm conflates the theoretically distinct notions of expectation (where a stimulus is likely to appear) and attention to relevant features in the environment (which spatial position is likely to contain the task-relevant information). As a result, any influence of the cue on information processing is difficult to attribute to either factor or to some combination of the two. Second, the behavioral results can be explained either by a change in the sensitivity of early sensory processing or by a change in decisional factors. Importantly, similar issues arise in many other studies within the literature, as experimenters typically manipulate either the probability that a known target stimulus will appear or they manipulate information about which stimulus is most task-relevant. As a result, the field lacks a coherent framework that respects the potentially distinct influence of different types of top-down knowledge on sensory processing. In turn, the lack of a clear framework has important implications for canonical models of information processing such as the notion of perception as inference^12,13 [see Box 1], as well as for long-standing debates about the cognitive penetrability of perception^14–16.

Box 1 – Perception as Bayesian Inference.

In the domain of visual perception, Bayesian models frame inference as the product of the prior probability of a stimulus [denoted p(stimulus), or p(s)] and the probability of a pattern of neural responses (r) given that stimulus [referred to as a likelihood function, denoted p(r|s)]. The prior is a probability distribution over a stimulus space such as orientation or motion direction, and reflects the initial degree of belief in the current state of the world. On the other hand, the likelihood function reflects the probability that a given outcome – for example a pattern of responses over a population of feature-selective sensory neurons – will be observed for each possible stimulus value. The prior and the likelihood function are then combined to form a posterior distribution [denoted p(s|r)]. The peak of the posterior provides an estimate of the most likely stimulus, and the uncertainty associated with the posterior is determined by the precision of the prior and the likelihood functions.

Typically, the prior is thought to encode current expectations held by an observer, and these expectations can be based on a variety of factors such as previous experience in a given context or statistical regularities that are observed in natural scenes⁷¹. In contrast, other factors – such as attention induced neural gain²⁷ – can increase the fidelity of a pattern of neural responses and bias the shape of the likelihood function. In this context, better understanding how expectation and attention operate on both early sensory and later decision-related processing will inform questions about how priors and likelihoods are implemented during perception.

The effects of attention and expectation on cortical information processing

Very few studies have independently manipulated expectation and attention to assess the impact of each factor on sensory processing. However, studies that attempt to focus on either expectation or attention have claimed that both factors modulate pre-stimulus neural responses^17,18, stimulus-evoked responses^19–27, and the efficiency of sensory read-out by putative decision mechanisms in parietal and frontal cortex^28–31. For the sake of brevity, we focus here on response modulations in early sensory cortices, both before and after a stimulus has been presented. We first briefly review studies about the effects of selective attention on these responses, and then review recent studies that attempt to experimentally dissociate attention and expectation to assess the separability of their effects on early sensory processing.

The impact of attention to relevant features on early sensory processing

Many single-unit physiology³² and fMRI¹⁷ studies demonstrate that attending to relevant locations modulates neural responses in early visual cortex, even before a stimulus is presented^33–35. Manipulating the relevance of spatial positions or low-level visual features also modulates the SNR and feature-selectivity of sensory-evoked responses that are associated with attended stimuli^{24,27,33,36–41}. For example, work by Treue and colleagues demonstrated that attention increases the precision of motion-selective population response profiles in MT, and more recent fMRI work shows that these increases in feature-selectivity can occur even in the absence of an overall increase in the BOLD response^19,20 (Figure 1A–B). Critically, at least some of these studies cued a behaviorally relevant feature, such as a location or a direction of motion, without inducing any expectation about the probability of the likely target feature^19,36,39. Thus, according to the operational definitions of attention and expectation outlined above, both pre- and post-stimulus modulations appear to occur due to manipulations of behavioral relevance, independent of changes in event probabilities.

Figure 1.

Increased gain modulation of visual responses by attention and expectation. A. Schematic of the experiment design used in¹⁹. Participants fixated on the central cue, while attending to either the orientation or contrast of the gratings in alternating blocks of trials. The orientation of one grating always closely matched the oriented cue line presented at fixation, while the orientation of the remaining grating either matched or mismatched the orientation of the first grating by a small CW or CCW offset. Similarly, the contrast of the second grating either matched or mismatched the contrast of the first grating by a small contrast change. On attend-orientation blocks, the participants had to indicate whether the two gratings were rendered at the same orientation (match trials) or at different orientations (mismatch trials). On attend-contrast blocks, the participants had to ignore differences in orientation and to report whether the contrasts of the two gratings matched or did not match. Additionally, on orientation-mismatch trials, the central cue was presented in green or red to indicate either a CW or CCW rotational offset between the two gratings. B. The orientation selectivity of population responses in V1^69,70, as measured with fMRI, as participants were performing the orientation discrimination task (i.e., attend-orientation) or the contrast discrimination task (i.e., attend-contrast). Data shown here were shifted such that the 0° channel indicates the cued orientation and positive values on the x-axis indicate responses in orientation channels that were offset in the cued direction, whereas negative values indicate responses in orientation channels offset in the uncued direction. Despite similar overall amplitude of responses in attend-orientation and attend-contrast condition, attention shifts the orientation tuning towards the cued offset when participants attend to the orientation of a grating instead of to the contrast of the grating¹⁹. In contrast, responses in neural populations away from the attended feature are relatively muted. C. Schematic of the experiment design used in²³. Each trial began with an auditory cue, which indicated (with 75% validity) the overall orientation of the subsequent gratings (~45° or ~135°). Following the cue, participants saw two consecutive gratings which differed slightly in terms of orientation, contrast, and spatial frequency. In separate blocks, participants judged whether the second grating rotated CW or CCW with respect to the first (i.e., orientation task); or whether the second grating had higher or lower contrast than the first (i.e., contrast task).

D. Expected orientations evoke less overall activity in V1 relative to unexpected orientations as measured with the BOLD response (bars). However, MVPA orientation classification accuracy of the grating orientation in V1 was higher for expected relative to unexpected orientations (line plots)²³ (with permission from the authors).

The impact of expectation on early sensory processing

Initial reports regarding the impact of expectation on sensory-evoked responses demonstrated that large-scale cortical responses measured with fMRI were smaller than responses associated with unexpected stimuli^21,22,25. This finding is consistent with generative models that frame perceptual inference as the iterative combination of priors with sensory evidence, because sensory evidence that is consistent with priors can support a rapid perceptual inference without the need for extensive processing. In turn, total cortical activity, as measured using methods such as fMRI, should be lower compared to situations where disparate priors and sensory evidence must be reconciled. In addition to attenuated BOLD responses, studies also suggest that expected stimuli evoke a more precise feature-selective pattern of responses in early visual cortex compared to response patterns associated with unexpected stimuli, similar to the modulations observed with feature-based attention^19,42,43. Again, this observation is in line with the idea that consistent priors and sensory evidence should lead to a precise inference, even though overall cortical activity is reduced.

In one study, Kok et al.²³ used fMRI and a task that cued participants on a trial-by-trial basis that an impending target was either going to be a 45° or a 135° oriented grating. The authors analyzed the pattern of responses across voxels in primary visual cortex (V1) using multivariate pattern classification analysis (MVPA) and demonstrated that expectation increased the separability between response patterns associated with each grating, even before stimulus onset (Figure 1C–D). MRI studies have also shown that expectation for a particular object category can bias pre-stimulus activation in face-selective regions of IT cortex^44,45. Finally, spontaneous fluctuations in pre-stimulus fMRI signals in sub-regions of visual cortex predict the probability that a particular feature or object will be reported when viewing an ambiguous or weak sensory stimulus^42,46. These spontaneous fluctuations may reflect endogenously mediated shifts in expectation, and they highlight the Bayesian notion that small shifts in expectation can have a large impact on perceptual inference when sensory evidence is weak or ambiguous⁵

Reconciling the effects of attention and expectation on early neural modulations in sensory cortices

Despite the apparent similarity of the early neural modulations attributed to selective attention and to changing expectations, studies that manipulate expectation typically have done so by explicitly providing prior information about the identity of an upcoming stimulus (e.g. a 45° or 135° grating, as in^18,23). As a result, participants not only knew what target feature to expect, but they also knew what target feature was relevant to performing the behavioral task on each trial. A similar argument can be made about several other studies^18,47–54, and based on the operational definitions of attention and expectation articulated in Summerfield and de Lange, the expectation cue can be expected to induce a shift of attention to the cued (expected) stimulus feature⁵. Given this consideration, any changes in behavior or associated modulations in early visual cortex were likely influenced to an unknown degree by both expectation and selective attention as opposed to expectation alone.

Recently, several studies have tried to more directly compare the effects of expectation and attention on behavior and on neural responses in visual cortex. One behavioral study used cues to manipulate the probability that a faint stimulus would be presented. These expectation cues increased both hits and false-alarm rates, whereas manipulating stimulus relevance (attention) improved the precision of sensory processing by selectively lowering false-alarm rates⁵⁴. Using the reverse-correlation method and modelling, this study further suggested that the differential effects of attention and expectation could be accounted for by the fact that attention suppressed internal noise and thus increased precision while expectation biased the baseline activity of sensory processing in favor of the cued stimulus. In addition, a fMRI study found that attention increased the separability of response patterns associated with expected and unexpected stimuli in IT cortex⁵¹. However, even in these studies, the cueing scheme is set up so that expectation was manipulated by cueing relevant stimuli over a longer time frame whereas attention was cued on a trial-by-trial basis. So, while this manipulation leads to separate sources of top-down information that operate on different time scales, it is not entirely clear that one type of cue solely modulated expectation and the other attention as both cues provided information about what to expect and what features were more likely to be behaviorally relevant.

One way to isolate the effects of expectation from attention on sensory processing is to design an experiment where stimulus regularities are manipulated without using an explicit cue. For example, Rungratsameetaweemana et al. used a variant of an orientation discrimination task, where targets were either coherently oriented red or blue bars at 0° (horizontal) or 90° (vertical)⁵⁵. This gave rise to four possible target types: red horizontal, red vertical, blue horizontal, and blue vertical. Each response button was associated with a specific conjunction of color and orientation. The probability that a specific color or orientation was a target feature was independently manipulated on a block-by-block basis such that within each block, targets were presented more frequently in one color (e.g., red; color expectation) or one orientation (e.g., vertical; orientation expectation). Thus, expectations about these sensory features (i.e., color and orientation) were induced through stimulus history without an explicit cue. By not using an explicit probability cue, this study minimized the possibility that participants shifted their attention to the expected stimulus features and thus the results are less likely to be influenced by selective attention. That said, it is possible that an implicitly induced expectation about a target feature could lead participants to allocate more attention towards the feature that is most likely to be presented⁵⁶. However, even if participants noticed the expectation manipulation, knowledge about the most likely sensory feature would not provide information about the relevant behavioral response because targets were defined by the conjunction of color and orientation.

Using this behavioral paradigm allowed for a manipulation of expectation about two low-level sensory features (color and orientation) while measuring EEG markers that index early sensory processing and the accumulation of sensory evidence during decision-making (the visual negative potential, or VN, and the centroparietal positive potential or CPP, respectively). Importantly, the paradigm also included an independent manipulation of sensory evidence to validate these markers of sensory processing and to provide a point of comparison for any expectation-related modulations. The behavioral results revealed that expectations about likely sensory features improved the speed and accuracy of decision-making in a manner analogous to increasing the amount of available sensory evidence. However, while manipulations of sensory evidence increased the amplitude of the VN and the amplitude and slope of the CPP, expectations about sensory features had no impact on either of these components despite the robust effect of expectations on behavior (Figure 2A–B). Instead, expectation modulated the amplitude of posterior alpha and frontal theta oscillations, signals thought to index overall time-on-task and cognitive conflict. Together, these findings suggest that expectations about low-level sensory features, even when the expectations do not provide information about the behavioral relevance of sensory stimuli, primarily operate at post-perceptual stages of information processing.

Figure 2.

Recent studies that isolate the effects of expectation from attention on sensory processing. A. The CPP is used as an EEG marker of the accumulation of sensory evidence during decision-making and its pre-peak amplitude is shown to be sensitive to manipulations that increase the amount of sensory evidence in the stimulus display. B. Despite the CPP being sensitive to increases in sensory evidence, expectation does not impact the pre-peak CPP amplitude. Instead, violations of expectation modulate the post-peak CPP amplitude which could be associated with later stages of processing after early sensory processing⁵⁵. C. Behaviorally, predictive (left and right) cues led to criterion shift towards to the cued direction both when presented before and after the gratings. Critically, direct comparisons of the effects of pre- and post-cues showed that expectation induced via post-cues had a stronger effect on participants’ performance, which must be due to a shift in the decision criterion because the cue was presented after sensory processing of the stimulus was complete⁵⁷. D. Both pre- and post-cues have comparable influence on stimulus sensitivity (d’)⁵⁷. E. A reverse correlation analysis was performed to investigate whether pre- or post-cues affected participants’ information usage at any time throughout the 30 frames of stimulus presentation. Higher beta values indicate that participants placed more weight on the information provided by a particular stimulus frame. Temporal information usage for predictive (left and right) and neutral cues did not differ by cue time (pre- or post-cues), showing that expectation induced via pre- and post-cues had similar effects on temporal information usage throughout each trial. Note that noisier plots of neutral-cue condition are due to a smaller number of trials⁵⁷. F. Feature information usage for predictive (left and right) and neutral cues also did not differ by cue time (pre- or post-cues), suggesting that pre- and post- cues have the same effect on feature-based information usage (reprinted from⁵⁷ with permission from the authors).

Another recent study by Bang and Rahnev also converges on the idea that expectations do not impact early sensory processing but instead modulate decision criteria⁵⁷. Participants performed a discrimination task where they judged whether the overall orientation bias in a series of gratings was tilted left (clockwise) or right (counterclockwise) from vertical. The grating stimuli were either preceded or followed by a predictive cue (i.e., pre-stimulus cue or a post-stimulus cue, respectively) indicating with 66.67% validity whether the overall orientation was more likely to be left or right of vertical. An additional condition was also included where neutral (uninformative) cues were presented. A pre-stimulus cue could impact both sensory signals and later decision processes, whereas a post-stimulus cue could only influence decision processes. By comparing the behavioral effects of pre-stimulus cues and post-stimulus cues, the study could assess the impact of expectation on early sensory processing and on decision-related criterion shifts.

Direct comparisons of pre- and post-cues demonstrated similar effects of both cue types on stimulus sensitivity (d’). However, post-cues induced a greater shift in decision criterion (c) compared to pre-cues (Figure 2C–D). To further examine how participants used cue-based-information in both the temporal and feature domains, the authors employed a reverse correlation method in which they compared the impact of predictive and neutral pre- and post-cues. The results demonstrated that pre-cueing and post-cueing exerted a similar influence on the use of temporal information and feature-specific information provided by predictive and neutral cues. Since the post-cues could only influence later decisional processes but not early sensory signals, the comparable effects of pre- and post-cues suggest that expectations primarily impact decision criteria rather than directly modulating the efficiency of sensory processing (Figure 2E–F). Together with the study by Rungratsameetaweemana et al, these results are more in line with classic theoretical frameworks such as signal detection theory (SDT) and suggest that knowledge about statistical regularities of the sensory environment primarily influence later cognitive operations related to response selection and execution^58–62.

Conclusions

While we argue here that it is premature to assert that expectations about statistical regularities impact early sensory processing, there is substantial evidence that manipulations of expectation have a profound impact on behavior and on responses in higher-order parietal and frontal regions that are thought to be more directly involved in regulating decision-making and behavioral responses (i.e. saccades, reaching movements^63,64). Saccade-selective neurons in frontal cortex show a pre-stimulus response bias as a function of target probability⁶⁵, stimulus-evoked responses in the superior colliculus are mediated based on the certainty associated with a planned saccade^66,67, and disrupting saccade-selective regions in human frontal cortex attenuates the impact of target probability on behavioral performance⁶⁸. This evidence is consistent with the hypothesis that expectations can mediate priors to influence response selection. These findings are also in line with the idea that expected stimuli might exert a larger impact on sensorimotor decision mechanisms via changes in the ‘read-out’ of sensory-evoked responses rather than affecting the perceptual processing of the sensory signal itself. Moreover, as articulated in Summerfield and de Lange⁵, observers should exploit information about both statistical regularities and behavioral relevance to guide optimal decision making, as both sources of information should support the efficient processing of information to guide behavior. Future studies are needed to more thoroughly explore when and where expectation impacts information processing, and to orthogonally manipulate expectation and attention within the same paradigm to test for differences in temporal dynamics, modulations in different cortical areas, and influences on behavior.