Serial dependence across perception, attention, and memory

Abstract

Information that has been recently perceived or remembered can bias current processing. This has been viewed as both a corrupting (e.g., proactive interference in short-term memory) and stabilizing (e.g., serial dependence in perception) phenomenon. We hypothesize that this bias is a generally adaptive aspect of brain function that leads to occasionally maladaptive outcomes.

Previous memory encroaches on current memory

One of the most remarkable aspects of our visual experience is that we perceive a stable environment despite the constantly changing image on the retina (e.g., from eye movements, blinks, occlusions). A core function of visual working memory is to temporarily maintain representations that bridge disruptions and facilitate that stability. However, working memory is a limited system that sometimes fails to properly segregate recently activated (but now irrelevant) representations from those that should guide current behavior. For example, memory performance for lists of words is slower and more error-prone if a probe item belonged to a previous memory set [1] (Figure 1A). Such “proactive interference” also occurs during change detection for arrays of visual stimuli, wherein a probe item will more likely be incorrectly identified as part of the current array if it had appeared in the previous array [2] (Figure 1B). In a spatial delayed-response task, moreover, monkeys’ eye movements are biased toward the remembered location from the previous trial [3] (Figure 1C). That is, previously relevant information can linger and impact what we currently remember over the short-term. The bulk of proactive interference research has focused on control mechanisms to overcome that impact, because it is considered a fundamental obstacle to cognition [1]. Recent research in visual perception, however, may provide a window into the generally adaptive nature of the processes underlying proactive interference.

Figure 1. Example task schematics and representative results demonstrating serial dependence.

(A) In a recent-probes recognition test for verbal material, error rate is higher for non-match items that were presented in the previous trial memory set (trial n-1). (B) In a change-detection test for visual features, error rate is higher for non-match items that were presented on the previous trial, especially when the non-match item was previously presented at the same spatial location as the current test probe. (C) In an oculomotor delayed-response test for spatial locations, eye movement responses are pulled toward the previous trial’s remembered location. This bias is modulated by the distance between the current and previous location and by the length of the current trial’s inter-stimulus delay interval (ISI). In the example shown here, the location on the previous trial was 60° clockwise from the current location, and the eye movement is thus biased in the clockwise direction by 15° (on long delay trials, purple dot) or 5° (on short delay trials, green dot). (D) In a continuous-report test for orientations, orientation judgments are biased toward the orientation presented on the previous trial. This bias is modulated by the distance between the current and previous orientation and only occurs for attended (versus unattended) orientations. In the example shown here, the orientation on the previous trial was 30° clockwise from the current orientation, and the orientation judgement is thus biased in the clockwise direction by 10° (orange dot). (E) In a continuous-report test for face identity, face judgements are biased towards the identity of the face presented on the previous trial. This bias is modulated by the number of morph steps (in face-morph space) between the current and previous face.

Previous perception integrates with current perception

The perception of visual stimuli often follows a pattern that could be described as proactive interference: new stimuli appear more similar to just-seen stimuli than they truly are. In an orientation judgement task, for instance, perceived orientations are biased toward the angle of previously-attended stimuli [4] (Figure 1D). This “serial dependence” occurs for a range of features, including numerosity [5] and complex stimuli like faces [6, 7] (Figure 1E), which suggests that it is a general visual processing principle. This principle may extend beyond visual perception, moreover, and encapsulate the effect of proactive interference in working memory. In fact, many tasks that are used to investigate serial dependence in perception are structured identically to canonical working memory tasks: participants maintain a stimulus over a short delay and then report on its features (Figure 1). Therefore, serial dependence in perception and proactive interference in memory may stem from similar memory traces that persist across trials.

Serial dependence in perception describes a recency-bias much like the proactive interference effect in working memory. Unlike proactive interference, however, serial dependence has been framed as an adaptive process, integrating successive stimuli to improve signal-to-noise in the service of perceptual stability [4–7]. Because the recent visual past is typically a good predictor of the future, this carry-over may bias neural signals to enhance perception and decision-making for incoming stimuli (i.e., predictive coding [8]). However, such temporal smoothing could become maladaptive when it sustains representations that are irrelevant to the current situation (i.e., proactive interference). An intriguing hypothesis, therefore, is that proactive interference is a maladaptive form of serial dependence, which arises from a mostly advantageous processing system. That is, the very feature that allows working memory to confer such impressive coherence is also the root of one of its most notorious handicaps.

What is the value of temporal integration in perception and memory?

Intrusions by concurrent [9] and recent [2] memories have been considered a significant limitation on visual working memory capacity and quality, because they consume a finite representational space that would otherwise be used to maintain more relevant information [1, 2]. This framing assumes that recent representations should be discarded, but neglects to consider the potential advantages of using related old information to scaffold new representations. Expectations can pre-activate sensory representations and facilitate processing of predicted inputs [8], and active working memory content can improve detection and attentional processing of similar visual stimuli [10]. Likewise, the lingering activation of previously relevant stimuli may increase sensitivity for related information in the environment, and thereby enhance encoding into working memory. In the same way that our visual environment is predictable from moment-to-moment—rendering serial dependence adaptive in everyday cognition—our mental workspace is often characterized by a sequential narrative (i.e., conversation, problem solving) or by a continuous progression of sensory events. In these structured environments, it would be adaptive to integrate recent stimuli that inform current encoding and retention demands, to increase the cohesion of working memory episodes. The drawback of such an integration process is that it would hamper differentiation between successive stimuli if the priority were to detect changes in the environment.

How might we recognize when past information is likely to remain useful? Serial dependencies are tuned by the similarities between previously and currently relevant stimuli. The extent of influence from previous stimuli depends on their distance from current stimuli in time [4, 6, 11], space [2, 4], and feature similarity [3, 4, 6], while the bias is virtually non- existent once that difference exceeds a certain magnitude. This range across which serial dependence occurs is referred to as the “continuity field” [4]. Serial dependence is also limited to stimuli that were previously attended [4, 11] or actively maintained [2], and absent for stimuli that were merely passively viewed. Moreover, the temporal gradient of serial dependence may differ depending on the “changeability” or autocorrelation of particular features in the natural world [7]. Stimulus attributes with low temporal autocorrelation can even promote repulsion (instead of attraction), which would maximize sensitivity to change [7]. These modulatory influences reinforce the adaptive nature of serial dependence in a constantly changing but statistically predictable environment—it only occurs when stimulus characteristics signal that the previous information remains potentially relevant.

Neural sources of serial dependence in perception and working memory

Despite the potential utility of serial dependence for smoothing perception and memory, we know very little about how the brain produces this behavior. One possibility is that it manifests at the earliest stages of cortical sensory processing, via increased sensitivity—perhaps due to short-term synaptic plasticity [12]—among neuronal ensembles that are responsive to the feature value of the previously attended stimulus [4]. Distributed fMRI patterns across early visual cortex are shifted in the direction of the attended orientation from the previous trial [11], but the underlying source of this bias is unknown. The observation that serial dependence relies on attention [2, 4, 11] suggests that a signal may originate from higher cortical areas to produce the bias; the attentional signal from recent experience may persist in the form of active firing or a residual synaptic trace. The vast body of knowledge on attentional control and stimulus representation in working memory [12, 13] may therefore illuminate the neural mechanisms underlying serial dependence, especially if we examine the bias in the context of similar trial history effects that have been studied from the working memory perspective (i.e., proactive interference).

Like “sensory recruitment” in working memory [13], serial dependence also seems to manifest in cortical regions that represent the particular task content: proactive interference from verbal material is reflected in fMRI activity patterns in temporal cortex [14], whereas serial dependence for visual material is evident in early visual cortex [11], and residual previous trial information in an oculomotor delayed-response task manifests in the frontal eye fields of nonhuman primates [3]. Moreover, when the delay between stimulus and response is lengthened, the magnitude of attraction toward the previously remembered stimulus grows [3, 15]. This implies that the effect of integration with previous stimuli can continue to evolve while the current stimulus is maintained in working memory, which may reflect the contribution of post-perceptual processes. In fact, judgments that rely more on perceptual comparisons—and are less influenced by post-perceptual and attentional processes—can be repelled away from preceding stimuli (i.e., tilt aftereffects; [15]), and repulsive adaptation may also increase with the duration of exposure to the previous stimulus [4]. The ultimate consequence of serial dependence—i.e., whether it segregates to promote change detection or integrates to promote stability—likely depends on the balance of perceptual and memory demands.

Concluding Remarks

Serial dependencies in perception and working memory have been studied separately, but the parallels between them highlight the value of synthesizing these research areas moving forward [10]. We can harness the careful psychophysics and modeling data arising from perceptual serial dependence research to illuminate principles of stability in working memory. Conversely, we can exploit many decades of working memory research to inform the study of stimulus representations and adaptive control in serial dependence. Several outstanding questions and testable hypotheses stem from the integration of these concepts (Box 1). If we appreciate proactive interference as a consequence of a system that typically supports cohesive internal representations across time, this perspective should clarify the principles of efficient temporal dependence in brain function, and why they sometimes lead us astray.

Box 1: Predictions for a unified principle of serial dependence in perception, attention, and memory.

Although serial dependence may adaptively support visual stability over time, it fundamentally involves a biased representation of the current stimulus. Thus, it should be downregulated in certain situations (i.e., those that demand differentiation [7]). This regulation may rely on the same mechanisms that detect and dampen proactive interference in working memory. If this is the case, cues for differentiation (like a context shift) should initiate control processes to segregate representations and curtail the impact of previous information. Conversely, if proactive interference arises from the same adaptive processes as perceptual serial dependence, the impact of the previous information on current memory should be amplified when context cues (like spatial or semantic similarity) signal its continued relevance for the current circumstances. While there is a compelling rationale for the adaptive role of serial dependence, the argument is mostly theoretical. Strong empirical evidence for the value of serial dependence would emerge from manipulations of signal-to- noise ratio in a predictable visual environment, wherein endogenous representations of previous stimuli should actually boost perception and encoding of incoming stimuli to improve performance. Such investigations would clarify the context-dependent flexibility of serial dependence and confirm its utility in multiple domains of processing.