Serial dependence in visual perception: A review

Abstract

How does the visual system represent continuity in the constantly changing visual input? A recent proposal is that vision is serially dependent: Stimuli seen a moment ago influence what we perceive in the present. In line with this, recent frameworks suggest that the visual system anticipates whether an object seen at one moment is the same as the one seen a moment ago, binding visual representations across consecutive perceptual episodes. A growing body of work supports this view, revealing signatures of serial dependence in many diverse visual tasks. Yet, the variety of disparate findings and interpretations calls for a more general picture. Here, we survey the main paradigms and results over the past decade. We also focus on the challenge of finding a relationship between serial dependence and the concept of “object identity,” taking centuries-long history of research into account. Among the seemingly contrasting findings on serial dependence, we highlight common patterns that may elucidate the nature of this phenomenon and attempt to identify questions that are unanswered.

Introduction

While walking down a street, the information that light brings to our eyes is constantly changing. Every time we move and shift our gaze around, every time that cars and people go by, the details of the scene change. Yet, we maintain a sense of continuity in what we perceive: The tree beside the street is perceived as the same tree seen a moment ago, despite constant interruptions and changes in our view. This is a remarkable ability, and the best technology and engineering are yet to design a machine that can do this as well as the human brain can.

How does the brain maintain continuity in a world of change? Numerous recent results suggest that the visual system uses the recent past to inform perception in the present, exploiting the autocorrelation of events in the visual world (Burr & Cicchini, 2014; Fischer & Whitney, 2014; Kiyonaga et al., 2017; Kristjánsson, 2022; Pascucci & Plomp, 2021). Central to this idea is serial dependence (see Figure 1 and Box 1), a behavioral bias where current visual decisions are biased by the history of stimuli seen before (Fischer & Whitney, 2014).

Figure 1.

Serial dependence in visual perception and decision-making. (A) Our everyday environment is made of relatively stable and temporally correlated visual features: As we take a walk in the park, the objects around us (e.g., the leaves and trees) tend to remain the same, despite changes in luminance patterns and viewpoints. (B) To exploit such temporal continuity, the representation of a visual object (here illustrated as a probability distribution over stimulus space) can propagate from one moment to the next, biasing visual representations toward the recent past. This leads to systematic errors in perceptual decisions, which tend to be pushed toward the direction of the previous stimulus. (C–E) Three possible scenarios illustrating different accounts of the nature of representations involved in serial dependence; two different objects—the leaves and the tree—are shown inside green and brown circles, respectively. (C) Serial dependence can occur at the level of low-level visual features (e.g., orientation, motion, color) and independently of object-level representations. (D) Serial dependence can occur only for visual features of the same object. (E) Serial dependence can occur at more abstract levels of representation, where features of objects are extrapolated and reduced to elementary representations required by the task (e.g., both the tilt of the tree and of the leave can be represented as a tilted line). Note that, as in the “low-level” scenario, high-level serial dependence can be object independent. (The picture in A is from Parc de Milan, Lausanne, Switzerland.)

Over the past decade, the study of serial dependence has increased considerably, lining up with well-established research on the role of the recent past in perception and cognition. Recent findings show that perceptual decisions are systematically attracted or repelled by the history of prior stimuli, revealing effects that are pervasive and evident in seemingly any sort of visual task. This has fostered frameworks where such serial dependence is assumed to mediate our experience of object continuity.

What is the nature of serial dependence? How are attractive and repulsive biases involved in the temporal integration and segregation of visual input? At what level of processing and representation do these effects occur? And how does the visual system determine what matches what from one moment to the next? In addressing these questions, first, we survey the literature on serial dependence, focusing on the main paradigms and findings (“Serial dependence paradigms”) and the factors that seem to play a role (“Key factors determining serial dependence”). Next, we discuss challenges to establishing a direct link between serial dependence and object continuity (“The problem of object continuity”). Finally, we present a historical overview of the main frameworks and computational approaches dedicated to serial dependence over the past century (“A brief history of research on history biases”). The goal is to provide a comprehensive background for future research, reviving centuries-old questions and raising new ones.

Box 1. Box 1. Serial dependence

Serial dependence has a long history in psychology and psychophysics, starting from the adoption of scaling methods and the finding of systematic effects in psychophysical reports that extend over trials. Two such effects were termed assimilation—when perceptual reports were positively correlated with previous stimuli and responses—and contrastive effects—when the sign of this serial correlation was negative (see “A brief history of research on history biases”).

More recently, the term serial dependence has been used to describe, typically, a positive bias to judge present stimuli as more similar to recent stimuli than they are (i.e., a form of assimilation). In particular, serial dependence involves a systematic bias in perceptual decisions during psychophysical tasks, which are attracted toward aspects of the stimulus presented one or a few trials before. For instance, the decision about the orientation of a visual stimulus is biased toward the orientation of the stimulus seen before (Figures 2 and 3).

Originally, however, the term comes from time-series analysis and refers to statistical dependencies between events that are close in time (Huitema, 1986). Compared to serial and autocorrelation, serial dependence implies any linear or nonlinear relationship that can be used to predict future events from the past. It describes a unidirectional effect (the past affects the present and not the other way around) but no causality—a third unknown variable may be the true source mediating the effect. From this perspective, serial dependence is neither a mechanism nor a phenomenon exclusive to vision and cognition but a property of a time series (e.g., of behavioral and neural data).

Serial dependence paradigms

Serial dependence has been investigated with a variety of paradigms, involving different types of stimuli, responses, and analysis. This has led to disparate, often seemingly contradictory, findings. Here we focus on work with behavioral methods (see Box 2 for imaging research) highlighting common threads and missing links between the findings. The aim of this section is not simply to provide an overview of the methodology but to summarize the results and insights gained by using different methods.

Box 2. Box 2. Neural correlates of serial dependence

Functional magnetic resonance imaging (fMRI)

Compared to the vast amount of behavioral work, only a few studies have been performed on the neural underpinnings of serial dependence. In one of the first, Schwiedrzik and colleagues (2014) used functional magnetic resonance imaging (fMRI) in a task with multistable stimuli to disentangle the effects of perceptual hysteresis (e.g., positive serial dependence) from those of adaptation. They found that these two opposite biases, revealed in many other following works, as we discuss in this review, reflect distinct neural circuits: Hysteresis involves a distributed network of higher visual areas and frontoparietal areas—particularly the right dorsomedial prefrontal cortex, which is involved in predictions and memory—whereas adaptation is restricted to activity changes in early sensory areas.

In a later study, John-Saaltink and colleagues (2016) demonstrated that prior stimuli modulate activity in early sensory cortex, with an effect highly specific to V1. Notably, the effects were not due to the prior stimulus itself but to the reported stimulus even on error trials, where the reported stimulus did not match the physical one.

Within the Bayesian framework, van Bergen and Jehee (2019) used fMRI decoding techniques to decode representations of sensory uncertainty in probabilistic distributions derived from population-level activity. They found that the uncertainty decoded from the activity in the early visual cortex can be related to the strength of serial dependence, showing that, at least when comparing the relative uncertainty between two trials (e.g., high to low and low to high), the behavioral bias reflected optimal cue combination (larger serial dependence when the sensory uncertainty decoded on the previous trial was lower).

Electroencephalography (EEG) and transcranial magnetic stimulation (TMS)

Leveraging the high temporal resolution of electroencephalography (EEG), other works have investigated the electrophysiological correlates and temporal dynamics of serial dependence. Several studies have shown that prior stimulus information can be decoded from evoked EEG scalp activity in the current trial. These decoding results were obtained in tasks where the previous stimulus was either relevant or irrelevant and induced attractive or repulsive biases (Bae & Luck, 2019; Fornaciai & Park, 2020a). An important finding comes from the work of Barbosa and colleagues, which demonstrated that latent traces of prior stimuli can be reactivated by the onset of a new event (e.g., the start of a new trial), likely causing interference with the representation of a new stimulus and serial dependence effects (Barbosa et al., 2020; Stein et al., 2020). Using transcranial magnetic stimulation (TMS), these authors also provided evidence for a causal link between the reactivation of memory traces in the prefrontal cortex and serial dependence in behavior (Barbosa et al., 2020; see also de Azevedo Neto and Bartels, 2021).

Adjustment tasks

In the method of adjustment, a brief stimulus is followed by a response tool that resembles one of the stimulus’ features (e.g., the orientation, size). Participants adjust the tool to match their perception of the feature. This method is also known as a “delayed reproduction task” or simply “reproduction task.”

Typically, adjustment errors are biased by features on the preceding trials. In Fischer and Whitney (2014), the stimulus was an oriented Gabor presented for 500 ms followed by a noise mask for 1,000 ms. After 250 ms from the offset of the mask, participants rotated a line to reproduce the orientation of the Gabor (see Figure 2A). On average, adjustment errors (reported minus true orientation) were positively related to the difference in orientation (Δ) between the previous and the present stimulus, showing a bias toward the previous orientation of about 10° for Δ near 20°—at the peak of the bias, perceptual reports were halfway between the present and prior orientation.

Figure 2.

Typical paradigms and findings in serial dependence research. (A) In a standard orientation adjustment task, an oriented stimulus (e.g., Gabor) is briefly presented and followed by a noise mask. After the mask, observers reproduce the perceived orientation of the Gabor by rotating a response tool (here a simple line). Serial dependence is seen in the adjustment errors (gray dots in the bottom plot) as a function of the difference in orientation between consecutive trials (Δ: previous minus present orientation). Typically, errors deviate toward the direction of the previous stimulus orientation (attractive bias, highlighted in the green regions of the plot), when Δ is small, following the shape of the first derivative of a Gaussian function (black curve). The dots in the bottom plots are from a simulated observer. (B) An example of a sequential dual task in which an adjustment task is followed by a forced-choice task. One of two stimuli (the inducer) is cued in the adjustment task and observers reproduce its orientation. In the “test” display, two other stimuli are presented, one at the same location as the inducer. In the forced-choice task, observers make a perceptual judgment (e.g., comparison or equality judgment) about the two test stimuli. Serial dependence is evident in the forced-choice task, as a shift in the perceived orientation of the stimulus at the inducer location, compared to the stimulus at the other “unbiased” location. The plot shows a simulated pattern of data resembling the findings of Fritsche and colleagues (2017), where the inducer caused repulsive biases: To be perceived as identical to the unbiased stimulus, the test stimulus at the inducer location had to be slightly tilted in the opposite direction of the inducer (positive Δs indicate when the unbiased stimulus was more clockwise). (C) The typical structure of a postcueing paradigm investigating serial dependence in motion direction between trials containing two clouds of moving dots. A postcue indicates the color of the cloud to report. The plot shows a simulated pattern of results based on the findings from Fischer and colleagues (2020): Serial dependence is influenced by the congruency between the feature cued on present and previous trials.

The pattern in Fischer and Whitney (2014) has been widely replicated, even though the magnitude of the effect can vary (e.g., from 1° to 10°; Fritsche et al., 2017). This pattern reveals key aspects of serial dependence. First, the effect of stimulus similarity: The bias is larger when the previous and present stimuli are similar (e.g., when Δ is small) and fades out as their difference increases, forming the typical S-shape that can be approximated by the fit of the first derivative of a Gaussian function (see Figures 2A, 3 and Box 3). Second, the bias is still evident from stimuli up to three trials back (10–15 s).

Figure 3.

Contrasting forces in serial dependence. (A) Serial dependence is typically positive when the difference between the previous and present stimulus is small and slightly negative when the difference is large (Δ, here indicating the difference in orientation between the previous and present stimulus, data are from 48 subjects performing an orientation adjustment task with low spatial frequency Gabors; Ceylan et al., 2021). (B) The coexistence of these two opposite biases can be modeled as the additive effect of a weighting function narrowly tuned toward the recent past (the green distribution) and one more broadly tuned away (the brown negative distribution). If observers weigh the previous stimulus according to the two distributions, the resulting pattern resembles A. (C) Interindividual variability in the dominance of the positive and negative components of serial dependence. Typically, the positive bias dominates; some subjects, however, show no bias or only negative serial dependence. (D) Two curves with a difference of Gaussian fit depicting the pattern for the top five observers showing positive serial dependence (green, corresponding to the green square in C) and the top five observers showing only negative biases (brown, corresponding to the brown square in C). The proportion of subjects showing positive serial dependence in this data set is 70% with an effect size of d′ = 0.63 (Cohen's d).

Box 3. Box 3. Analysis methods for serial dependence

The analysis of serial dependence involves quantifying systematic relationships between the decision on the current trial and the history of events on preceding trials. The exact pipeline depends on the type of task and data (e.g., adjustment responses or forced choices). We here provide a brief description of standard approaches used in different contexts.

Adjustment tasks

For simplicity, we will use orientation adjustment tasks as an example, but the analyses described here equally apply to color, direction, or any other continuous stimulus.

Data preparation. Adjustment errors are computed as the angular (or linear) difference between the reported and the actual orientation in each trial. Typically, adjustment errors are widely scattered. To correct for outlier errors and unlikely reports, a correction is often performed by removing trials according to an arbitrary cutoff (e.g., 30°; Cicchini et al., 2018), standard deviation–based thresholds (e.g., 3 SD; Fritsche et al., 2017), or other parametric approaches (e.g., Grubb's test; Pascucci et al., 2019). Adjustment times are also used to clean the data, for instance, by removing trials with responses faster than 500 ms or slower than 10 s (Ceylan et al., 2021). In some cases, the trial following an outlier (e.g., a guess) is also removed, since no meaningful serial effects can be expected following a trial in which the observer had a lapse of attention or guessed the response. An additional procedure is to demean the cleaned errors to remove systematic tendencies to reproduce the stimulus in one direction or another (e.g., CW or CCW). Because adjustment errors are prone to other sources of biases, including, for instance, anisotropies and category boundary effects in orientation estimation, some authors have usually corrected for nonlinearities in the stimulus space (Manassi et al., 2018; Pascucci et al., 2019; van Bergen & Jehee, 2019). This has been done with different methods, from fitting sinusoidal to polynomial functions on the errors over stimulus space, to cleaning the data of biases, such as the cardinal or orientation bias and category biases in color space. Standard and validated approaches for this step have yet to be reached.

A largely unaddressed issue concerns the influence of swap errors on serial dependence. Swap errors refer to an observer reporting the wrong item in a working memory task. In serial dependence, an observer might report the previous instead of the current target, simply because of a “swap” rather than a combination of the prior and current percept. Almeida, Barbosa, and Compte (2015) reported that in a typical working memory task, swaps and attractive bias co-occur. As swap errors appear as a 100% bias, even a few swaps could create the appearance of spurious serial dependence.

Model fitting. The magnitude of the serial dependence in adjustment errors is computed by fitting the first derivative of a Gaussian function (abbreviated as DoG) to the errors as a function of Δ (the difference between the previous stimulus feature and the current one). The typical form of this function is

$$\begin{array}{c}\hfill y=\Delta awc{e}^{-{\left(w\Delta \right)}^2}\end{array}$$

where y is the adjustment error (single trial or averaged over trials and smoothed over ∆); α is the amplitude of the DoG curve multiplied by the constant $c=\sqrt{2}/e^{-0.5}$, which scales the amplitude to the curve peak in y units (e.g., degrees); and w is the inverse of the curve width. Note that, besides approximating the main pattern, the form of this function also reflects two important aspects of serial dependence, which can be decomposed into a linear component Δa, which implies a systematic relationship between errors and previous stimuli, and a Gaussian weighting component, centered on 0, which accounts for the fading of the relationship as Δ increases.

The most used parameter to determine the magnitude of serial dependence is the amplitude or half-amplitude of the DoG function (α). While this is usually estimated on the aggregate data of many subjects, to avoid the pitfalls of aggregated data, the DoG fitting procedure can be expanded to allow for variability between individual participants by fitting a mixed-effects model (Pascucci et al., 2019). Another modeling approach is to fit a hierarchical Bayesian model to the data (Sadil et al., 2021). In this model, the authors accounted for individual differences and rotational biases as well.

As mentioned throughout the article, serial dependence patterns also contain a combination of attractive and repulsive components, an aspect that simple DoG curves fail to capture. Alternative functions have been proposed to overcome this limitation (Bliss et al., 2017). Because of the putative nature of these opposite biases, a plausible approach would be to use the difference between two Gaussian functions, one accounting for the positive and one for the negative component (see Figures 3A,B)—that is, the classic “Mexican-hat” profile. This, however, would come at the expense of model complexity.

Model-free approaches. An alternative approach to model fitting is to compute the average error in a range of Δ values, typically close to zero. For example, subtracting the average error for Δ in the 1° to 25° range from the average error in the corresponding negative range (Samaha et al., 2019) quantifies the amount of systematic deviation of errors from zero—this deviation or “bias” is positive for attractive serial dependence and negative for repulsion. This approach is a straightforward way of quantifying serial dependence with few assumptions, particularly useful when limited data points are available, which is often the case in analyses for a single observer. Restricting the analysis to values of Δ close to zero is reasonable when serial dependence effects are expected in this range. A disadvantage of this approach, which also applies to the DoG fitting procedure described above, is that it does not allow capturing more complex patterns across the entire Δ range. While attractive biases are typically present for small Δ, repulsive biases appear for larger Δ. One way to address this problem would be to use more than a single bin, discretizing Δ as a function of several distances between the previous and present stimulus (e.g., three to four bins from “close” to “far”), then analyzing the data with standard repeated-measures approaches.

Statistical analysis

Statistical testing is commonly performed using a permutation approach on the parameter of interest, typically the half-amplitude α of the DoG function (Fischer & Whitney, 2014; Fritsche et al., 2017). A null distribution of α is obtained by fitting the DoG function to a large number of randomized data sets generated by randomly flipping the sign of errors in each trial (or shuffling the correspondence between single-trial Δ and errors). The proportion of such data sets that is more extreme than the parameters from the DoG function fitted to actual data denotes p, which can be interpreted in the same way as the p-value in standard frequentist statistics. Comparisons between conditions can be performed using a similar permutation approach by shuffling the labels of the conditions for each randomized data set. This approach has been often used for aggregated data from multiple subjects. In the framework of nonlinear mixed-model analysis, the final estimate of the parameters and their uncertainty can be used to quantify statistical significance (Pascucci et al., 2019).

Forced choice

When the task involves a choice between two alternatives, results are typically analyzed using a psychometric function, in which stimulus value predicts response. Serial dependence can be quantified as a shift in the threshold or point of subjective equality (PSE) of the psychometric function. For example, in comparing which one of two stimuli is tilted more clockwise, the PSE corresponds to the point of “no difference.” A shift in the PSE can be informative of whether the perception of the current stimulus has been affected by a preceding inducer, with attractive and repulsive biases that depend on the direction of the shift (Fritsche et al., 2017). Beyond PSE, more sophisticated statistical models have been also proposed to quantify or account for serial biases in psychophysical forced-choice tasks (Gekas et al., 2019).

 

Some systematic properties, like the effect of stimulus similarity and the temporal decay of serial dependence, seem to hold for a variety of adjustment tasks involving features such as shape (Collins, 2022a; Manassi et al., 2019), numerosity (Cicchini et al., 2014; Corbett et al., 2011; Fornaciai & Park, 2018b), facial identity (Liberman et al., 2014), color estimation (Barbosa & Compte, 2020), gaze direction (Alais et al., 2018), emotional expressions (Liberman et al., 2018), aesthetic judgments (Kim et al., 2019), spatial position (Manassi et al., 2018), and even for stimulus ensembles (Manassi et al., 2017; Pascucci et al., 2019).

What role does the adjustment tool play in serial dependence? Because the response tool usually resembles the relevant feature of the stimulus, the bias could partly be due to the response itself and the tendency to adjust the tool similarly to the previous trial (hysteresis in motor responses). Fischer and Whitney (2014, Experiment 2) removed the response tool on 25% of trials, finding that serial dependence is largely unaffected by the presence and use of a response tool. Similarly, serial dependence occurred without responses in Manassi et al. (2018), although the bias was significantly larger when a response was required. Cicchini, Mikellidou, and Burr (2017) decorrelated the effect of the stimulus and the response by asking participants, on half of trials, to reproduce the orthogonal instead of the actual orientation (90° away from the true one). They demonstrated that the attractive bias is maximal when the stimulus and the response match but absent when the response is orthogonal to the stimulus, corroborating the idea that the response tool itself plays no role.

While the response tool might not be necessary, how the stimulus is reproduced reflects the last instance of how the stimulus was perceived and remembered (Cicchini et al., 2014). Indeed, most of the variance in adjustment errors is explained by the feature reproduced on the previous trial, rather than the physically presented one (Pascucci et al., 2019). This latter observation is in line with studies demonstrating that, in adjustment tasks, the strength of serial dependence is also a function of postperceptual or postencoding factors, such as the confidence in the last decision (Samaha et al., 2019; Suárez-Pinilla et al., 2018) and the presence of an active task (Bae & Luck, 2020; Pascucci et al., 2019; Pascucci & Plomp, 2021).

Adjustment tasks also involve visual working memory, as the stimulus is reproduced “offline”—when no longer on screen—and needs to be briefly held in memory so the bias may partly be due to interference between the previous and current memory trace of the stimulus. While Fischer and Whitney (2014) have shown that serial dependence is independent of any explicit memory of prior stimuli (i.e., it occurs even though participants are unable to recall the previous stimulus from working memory), other studies have reported an increase in the bias toward prior stimuli with increased retention time of the present stimulus before the adjustment response, suggesting that visual working memory may still contribute to and modulate the strength of serial dependence (Bliss et al., 2017; Fritsche et al., 2017; Mei et al., 2019).

Beyond standard adjustment tasks, other studies have used variants of this method to target more specific aspects of serial dependence. In the sequential no-report paradigm, stimuli are presented in a rapid stream with only occasional adjustment responses, leading to independent history of stimuli and reports (i.e., the previous stimulus does not necessarily correspond to the last report) (Pascucci et al., 2019; Pascucci & Plomp, 2021). In this paradigm, prior stimuli and prior reports have opposite effects: The bias toward prior stimuli is repulsive, or attractive but weaker than the one toward prior reports, indicating the coexistence of two independent forms of serial dependence. In Kim, Burr, Cicchini, and Alais (2020), an adjustment task was combined with binocular rivalry, and serial dependence occurred only when the previous stimulus was consciously perceived but not suppressed by binocular rivalry. Similarly, Cicchini, Benedetto, and Burr (2021) combined the surround tilt illusion with an orientation adjustment task. In the tilt illusion, a surrounding configuration of oriented Gabors biases the perceived orientation of a central target. When this illusion was induced on the preceding trial, adjustment errors were biased toward the illusory orientation, rather than the actual one; however, the effects of the previous (illusory) orientation occurred before the illusion arose on the current trial. This led the authors to conclude that serial dependence originates at a late processing stage (i.e., after spatial context effects) but affects early processing stages.

In sum, these findings have exploited the high resolution of adjustment tasks, in which the stimulus and responses are sampled in quasi-continuous space, to reveal fundamental and consistent aspects of serial dependence. These include the effect of the similarity between current and prior stimuli, the coexistence of multiple and even opposing biases, the temporal decay of serial dependence, and, importantly, the dependence of the bias on how the previous stimulus is perceived, remembered, and eventually reported.

Forced-choice tasks

Disentangling whether serial dependence occurs in perception or at some later, decisional or memory stage is challenging. Adjustment tasks, for instance, involve both perceptual and postperceptual aspects (i.e., the stimulus must be held in memory for the time required to adjust the response and the reproduction response requires a postperceptual decision about the stimulus feature). A more direct test of perception would be to use forced-choice tasks where influences of postperceptual factors, such as the post-stimulus retention time, are minimized. As an example: An inducer stimulus might be presented at one location, and subsequently, two stimuli appear, one at the location of the inducer, the other at a neutral location. Asking observers to compare a feature of the two stimuli can reveal whether the inducer has changed the perceived features of the stimulus (see Figure 2B).

In Fischer and Whitney (2014), the inducer was one of two simultaneously presented Gabors. Observers were cued to reproduce the orientation of this Gabor in a typical adjustment task. On each trial, the adjustment response was followed by the presentation of two additional Gabors, and observers were forced to choose the Gabor that was oriented more counterclockwise. The orientation of the adjusted stimulus altered the perceived orientation of the subsequently presented Gabor at the same location, with a significant shift in the point of subjective equality—a result consistent with an attractive effect of the previous stimulus (Fischer & Whitney, 2014). Using a very similar paradigm, however, Fritsche and colleagues (2017) found that the stimulus at the previous location caused a repulsive bias upon the perceived orientation of the subsequent Gabor, in tasks where forced choices involved both comparative (e.g., which one is more clockwise?) and equality judgments (e.g., is their orientation the same?). Cicchini and colleagues (2017) failed to replicate this repulsive effect but found an attractive bias when the inducer was oriented 5° away from the current stimulus (instead of 20° as in Fritsche et al., 2017).

As with adjustment tasks, forced-choice tasks have been used to study the influence of prior stimuli and responses. Pascucci and colleagues (2019) have shown that even a choice made in the absence of any sensory stimulus (e.g., pure noise) can alter the sensitivity in a future perceptual task. In Feigin and colleagues (2021), prior choices about the location of a stimulus biased current location judgments, independently of changes in the stimulus color or the response keys. Conversely, prior choices about the stimulus color did not affect current location judgments, even if the two responses involved the same response keys. Feigin and colleagues (2021) concluded that serial dependence is strongly influenced by the task relevance of a visual feature and driven by prior choices, rather than the stimulus itself. Similarly, Zhang and Alais (2020) showed that the stimulus percept, rather than the physical stimulus per se, attracts subsequent perceptual choices, whereas the motor response might even produce a repulsive effect. The authors also focused on the source of interindividual variability in serial dependence, an aspect evident in adjustment tasks as well (Bliss et al., 2017; see also Figure 2C), and found that some subjects rely more on prior stimuli but others on prior responses (Zhang & Alais, 2020). While, collectively, these studies indicate a clear involvement of prior choices and postperceptual decisions in serial dependence, there is also evidence of a pure stimulus effect, independent of choice and response, as shown by Fornaciai and Park (2018a) in a forced-choice numerosity task where the inducer stimulus required no task.

Forced-choice paradigms have also revealed how different aspects of stimulus history can lead to opposite biases. Alais, Leung, and Van der Burg (2017) tested forced choice of motion direction, finding positive serial dependence for motion and a simultaneous repulsive effect due to orientation signals arising from motion streaks. Taubert and colleagues (Taubert, Alais, et al., 2016) reported strong positive serial dependence for gender but repulsive effects for facial expression, arguing that perception integrates temporally stable attributes (gender) but segregates changeable ones (expression).

In contrast to adjustment tasks, forced-choice tasks involve binary responses with a reduced number of stimulus categories, and there seems to be little one-to-one correspondence between the effects reported with one and the other method, particularly in the domain of visual orientation. For instance, in forced-choice tasks, serial dependence is maximal for Δs of 5° to 10°, while it is absent or repulsive for Δs of 20°; in adjustment tasks, the peak bias toward the previous stimulus is usually at around 20° and much smaller for 5° (Cicchini et al., 2017; Fischer & Whitney, 2014; Murai & Whitney, 2021). In other studies, however, serial dependence in forced-choice tasks occurs even for Δs larger than 30° and beyond the range found in adjustment tasks (Zhang & Alais, 2020). These discrepancies may simply reflect different tasks and paradigms, but they also raise the important question of whether the effects reported with one method versus another truly and unequivocally reflect the same cause. A potential method to resolve these discrepancies could be to design adjustment and forced choice tasks that are fully comparable within the framework of signal detection theory (see, e.g., Tomassini et al., 2010). It seems likely that serial dependence is shaped by the task type, whether the task involves perceptual or postperceptual processes and whether the critical dimensions are categorical or continuous. One potential explanation is that the influence of prior events depends on the “narrative” imposed by the task: When there are only two categories, the bias is due to the stimulus category (or choice) on the preceding trial; when stimuli vary along a continuum, the bias becomes more fine-tuned to gradual differences between stimuli.

Classification

Another family of paradigms can be broadly pooled under the term “classification” tasks. Classification tasks are extensions of forced-choice methods, where more than two stimulus categories are available for choice or considered in the analysis. Manassi, Kristjánsson, and Whitney (2019) used a three-alternative image classification task involving a simulated visual search task reminiscent of medical image search. They created 48 morphs between three random shapes and found that shape classification was strongly influenced by recent visual stimuli, with a 7% increase in errors toward the previous image. Murai and Whitney (2021) used a reverse correlation technique with classification images. A high-contrast clockwise or counterclockwise inducer Gabor was presented before the appearance of a low-contrast Gabor embedded in noise or a noise image alone. By recovering the internal template used for deciding whether a Gabor was present or absent, Murai and Whitney found a bias in the template toward the orientation of the preceding, irrelevant high-contrast Gabor. These findings indicate that serial dependence adapts to the paradigm at hand, revealing potentially maladaptive biases (e.g., a bias in search tasks akin to medical image search; but see Beckstead et al., 2017) and biased internal decision templates.

Postcueing paradigms

A defining feature of serial dependence is the direction and temporal order of events: A stimulus presented in the past exerts an effect on the present, not the other way around (see Box 1). However, attractive and repulsive biases can also be found when the order of events is inverted: A later stimulus influences decisions on a preceding one. In Fornaciai and Park (2020b), multiple stimuli were memorized on one trial and a postcue indicated the relevant one. They showed that a future stimulus can cause systematic biases in the judgment of a preceding stimulus, very similar to serial dependence effects. This may suggest that serial dependence is simply a form of memory interference and that event order is not crucial. However, a key difference is that standard serial dependence paradigms do not require active working memory maintenance, and the “inducer” stimulus, always presented in the past, becomes irrelevant in the present.

In a study by Czoschke and colleagues (2019), participants memorized two consecutively presented clouds of drifting dots and were subsequently cued to report the drift direction of the first or second cloud. The biases were repulsive from the drift direction memorized within trials but attractive from those memorized on preceding trials. This argued for positive serial dependence across trials and repulsive serial dependence within trials, compatible with repulsive interference between simultaneous working memory representations. Fischer and colleagues (2020) then demonstrated that across-trial serial dependence can be affected by task-relevant contextual features, such as the color or serial position indicated by a postcue (see also “Feature similarity”).

In general, while postcueing paradigms deviate from the typical approach of measuring serial dependence with a single stimulus on each trial, they are powerful tools to understand how stimulus information is bound together within and across memory episodes.

Key factors determining serial dependence

Stimulus parameters

“Serial dependence paradigms” demonstrates that serial dependence can seemingly occur in virtually any visual task, with different stimuli and processing demands. But a few key parameters can be considered necessary and sufficient: (a) The stimulus on the present trial should be weak, uncertain, and briefly presented; (b) stimulus duration and strength on the preceding trial must be controlled to minimize visual adaptation and negative aftereffects; (c) visible trails or after-images of the previous stimulus must be avoided (e.g., with backward masking); (d) serial dependence is typically stronger when the previous stimulus is attended to and task relevant (but see Fornaciai & Park, 2018a); and (e) the distance between previous and present stimuli on the dimension of interest (e.g., orientation) must be relatively small.

Uncertainty

Serial dependence is strongly influenced by uncertainty in the task. Uncertainty is inversely related to the stimulus duration, contrast, spatial frequency, and visibility (Ceylan et al., 2021; Cicchini et al., 2018; Manassi et al., 2018; Pascucci et al., 2019), but can also be modulated by the internal states of the observer, independently of the stimulus, such as performance confidence (Samaha et al., 2019; Suárez-Pinilla et al., 2018) and the focusing of attention (Fischer & Whitney, 2014; Fritsche & de Lange, 2019; Rafiei, Hansmann-Roth, et al., 2021).

It is generally agreed that serial dependence, particularly the positive form, increases when uncertainty is high. However, exactly how uncertainty modulates the effects of prior stimuli is still debated. For instance, a few models of serial dependence based on Bayesian and ideal observer principles predict that the effect of uncertainty depends on the previous trial—when the preceding stimulus is reliable and the present one is not, the bias toward the past is stronger, following classic models of optimal cue integration and Bayesian principles (Cicchini et al., 2018; Knill, 2007; Körding & Wolpert, 2006; van Bergen & Jehee, 2019). But studies testing the effect of relative uncertainty have reported increased serial dependence for both spatial frequency and noise (Ceylan et al., 2021; Cicchini et al., 2018; Gallagher & Benton, 2022) when the present stimulus is uncertain, independently of the uncertainty of the previous one.

Several potential explanations for this still await to be fully explored (Ceylan et al., 2021; Gallagher & Benton, 2022). For example, the uncertainty in the previous stimulus might deteriorate with time, leading to broadly tuned priors or “decisional templates” that are uninformative about previous uncertainty. When these broad priors are combined with the current stimulus, the effects depend mostly on the uncertainty in the current stimulus.

Attention

Another important aspect to consider is top-down processing. While some studies have revealed serial dependence with virtually no task, by simple exposure to a series of stimuli (Fornaciai & Park, 2018a; Murai & Whitney, 2021), it is nevertheless clear that attention plays an important role.

Initial evidence for this comes from Experiment 4 in Fischer and Whitney (2014), where eight Gabors were presented on a circle and one was precued for attention. When the attended location was constant between trials, there was positive serial dependence but not when the cued location changed, and the effects decreased as the distance between attended locations increased. Crucially, the direction of serial dependence was reversed for nonattended locations—that is, a repulsive bias (Fischer & Whitney, 2014).

For feature-based attention, Fritsche and De Lange (2019) demonstrated that serial dependence in orientation is drastically reduced when attention is directed to the size and not the orientation of the stimulus. Attentional modulations may also partly explain results from “sequential no-report” paradigms, where sequences of nonreported stimuli may be given less attention, leading to repulsive rather than attractive serial dependence (Pascucci et al., 2019; Pascucci & Plomp, 2021).

A complementary question is how serial dependence affects attention. This has been addressed with visual search tasks. In Collins (2020), the detection of an orientation singleton was affected by whether serial dependence pushed the representation of the singleton toward or away from the distractors presented in the same display. Rafiei and colleagues (Rafiei, Hansmann-Roth, et al., 2021) assessed serial dependence due to recently ignored distractor stimuli, showing that prior nonattended items lead to a repulsive bias upon current visual targets. Conversely, attended items led to attractive biases. Notably, this was not only seen for visual search targets but also in the representation of a single stimulus presented after the search had been performed (Rafiei, Chetverikov, et al., 2021).

Taken together, studies on the role of attention suggest that positive serial dependence requires top-down attentional processing and that attention may “gate” whether serial dependence is positive (for previously attended items) or negative (for previously unattended items).

Processing stages

A contentious issue has been whether serial dependence reflects effects upon perception or whether they are higher level, related to decisional processes or working memory. While Fischer and Whitney (2014) initially suggested that positive serial dependence occurs as early as primary visual cortex, several findings indicate that the bias might originate at later processing stages, which involve visual awareness (Kim et al., 2020), attention (Fischer & Whitney, 2014; Fritsche & de Lange, 2019), decisions (Feigin et al., 2021; Pascucci et al., 2019), or working memory (Bliss et al., 2017; Fritsche et al., 2017). While these latter studies point to the involvement of top-down aspects related to the task at hand, there is also evidence of serial dependence with no explicit task (Fornaciai & Park, 2018a; Murai & Whitney, 2021), which suggests that, under some circumstances, the effect can be ascribed to purely “bottom-up” perceptual history.

Notably, even if serial dependence originates at late processing stages, it could still affect early processing stages (e.g., by altering the phenomenological appearance of stimuli). Several results are consistent with this. Cicchini and colleagues (2021) showed that serial dependence may act before spatial context effects (e.g., the spatial tilt illusion), which are typically assumed to arise relatively early in visual processing. In line with this, many top-down effects have been claimed to penetrate perceptual processing (Firestone & Scholl, 2014), making such late-to-early history effects feasible. However, as the majority of studies have used adjustment tasks that are prone to many sources of biases (see “Adjustment tasks”), any consensus on the phenomenological consequences of serial dependence, particularly in the context of adjustment tasks, has yet to be reached.

A related question is to what extent prior and present representations at multiple levels interact (Trapp et al., 2021). For example, Ceylan, Herzog and Pascucci (2021) have shown that serial dependence can occur for elementary visual features with completely distinct stimuli (e.g., the scenario in Figure 1E), suggesting that the bias may also involve relatively abstract representations, independently of the continuity of stimulus identity (i.e., orientation might be commonly represented as a tilted line, even if it belongs to different visual objects) (Ceylan et al., 2021; Kwak & Curtis, 2022). Crucially, when serial dependence is observed for low-level features belonging to different visual objects, there are two candidate explanations: This might reflect perceptual history effects at low-level processing stages, independently of the object (Goettker & Stewart, 2022), or effects of high-level representations that reflect how information is maintained in working memory (Kwak & Curtis, 2022). In other words, the orientation of a current stimulus may be biased toward the orientation of a perceptually distinct previous stimulus, because the bias occurs at early orientation processing levels (e.g., V1) or because “orientation” is the relevant feature and can be represented as a single line. Disentangling these two possibilities or understanding their reciprocal contribution to serial dependence is a key future step.

More generally, the variety of existing paradigms and findings indicate that serial dependence is a multifactorial phenomenon, where prior perception, memory, decision-making, and the specific requirements of the task conjointly influence decisions about current stimulus features.

Retinotopy

Different stages of visual processing have been related to different levels of retinotopy. For example, many adaptation and negative aftereffects are retinotopic so that the adapter and test stimulus must be at very similar locations (Boi et al., 2011). Similarly, feature integration has been long assumed to occur in retinotopically organized feature maps (Treisman & Gelade, 1980; but see Boi et al., 2011).

Several studies have reported a broad tuning of serial dependence in retinal coordinates, spanning more than 15° of the visual field. That is, the previous stimulus continues to influence current decisions even when the current stimulus is 15° away, but the effects decrease with distance. Collins (2019) presented a Gabor stimulus at various locations and manipulated the spatiotopic and retinotopic reference frames. In the key manipulation, observers had to saccade toward the new location of a fixation spot presented before the stimulus. In one condition, the Gabor was presented at the same location as on the previous trial, but the retinal location changed because of the saccade (spatiotopic). In another condition, both the saccade and the stimulus landed in a new position that matched the previous Gabor location in retinotopic but not spatiotopic coordinates (retinotopic). Collins (2019) found that serial dependence was stronger in the retinotopic condition, consistent with the broad tuning of the “continuity field” proposed by Fischer and Whitney (2014), and mirroring receptive fields in higher-level visual areas, such as the inferior temporal cortex. Mikellidou and colleagues (2021) asked observers to perform an orientation reproduction task under two conditions, either with their head fixed (egocentric condition) or tilting their head by 40° before stimulus appearance (allocentric, e.g., world-centered condition). When the egocentric and allocentric coordinates were in conflict, serial dependence was mostly allocentric. So, while these two studies are seemingly in conflict, they support the initial findings of Fischer and Whitney, involving both retinotopic and allocentric tuning. A potential explanation is that whether serial dependence is retinotopic or spatiotopic might depend on the fast time scale of spatial remapping during saccades compared to the slow remapping during yaw rotations (Mikellidou et al., 2021). Another possibility is that feature integration shifts toward nonretinotopic processing for integration over relatively long time scales (Wutz et al., 2016). Collins (2019) also argued that the spatial tuning window of serial dependence might be broader than initially claimed (e.g., 22°), which may explain the lack of any spatial tuning in studies presenting the stimuli 10° apart (Fritsche et al., 2017).

Opposite directions

The studies reviewed above have revealed both repulsive and attractive serial dependence. These opposite biases occur both from single stimuli in the past and from stimuli at multiple time scales.

For effects of a single stimulus in the immediate past, there are reports of repulsive biases only (Bae & Luck, 2019, 2020; Pascucci & Plomp, 2021), attractive biases only (Cicchini et al., 2017; Fischer & Whitney, 2014; Manassi et al., 2018; Pascucci et al., 2019), and a mixture of the two (Bliss et al., 2017; Fritsche & de Lange, 2019; Rafiei, Chetverikov, et al., 2021; Rafiei, Hansmann-Roth, et al., 2021; Stein et al., 2020).

Some studies suggest that negative biases dominate when previous stimuli are not attended (Fischer & Whitney, 2014) or task irrelevant (Pascucci et al., 2019; Pascucci & Plomp, 2021), and when increased duration and strength promote negative adaptation-like aftereffects (Bliss et al., 2017; Fischer & Whitney, 2014; Manassi et al., 2018). Other work indicates that direction may depend on proximity in feature space, with positive serial dependence when the stimuli are similar and negative biases when they differ (Fritsche & de Lange, 2019; Rafiei, Chetverikov, et al., 2021; Rafiei, Hansmann-Roth, et al., 2021).

Recent models of serial dependence argue that the two biases reflect effects at different processing stages (Fritsche et al., 2020; Pascucci et al., 2019), broadly described as lower level (repulsion) and higher level (attraction). These stages involve different temporal dynamics: Perceptual decisions are attracted toward the most recent history (e.g., a few seconds in the past), whereas perception is systematically repelled away from more remote history (e.g., minutes in the past) (Fritsche et al., 2020; Gekas et al., 2019; Pascucci et al., 2019). Ultimately, these opposing biases interact, leading to classic patterns, where attraction for small stimulus differences is accompanied by repulsion for larger differences (Figure 3). Repulsive biases may reflect long-lasting visual adaptation, while attractive biases may reflect higher-level processes, such as decision inertia or working memory interference (Fischer et al., 2020; Fritsche et al., 2017, 2020; Pascucci et al., 2019). While this is supported by the higher spatial tuning and task independence of negative biases, in line with typical adaptation effects (Fritsche et al., 2020; Pascucci & Plomp, 2021), the exact nature of these biases and their involvement in different tasks are still debated. Alternative explanations involve repulsive biases that may arise from the active removal from working memory of information that is no longer relevant (Shan & Postle, 2022) and the use of prior stimuli as a reference for current decisions (DeCarlo & Cross, 1990; Stewart et al., 2005).

The problem of object continuity

Object continuity

If serial dependence is involved in maintaining perceptual continuity, there must be a way of solving the problem of “what matches what” from one view to the next: Do feature A at time t and feature B at time t +1 belong to the same object?

Spatiotemporal correlations among multiple features are crucial for object continuity (Treisman & Gelade, 1980): Without binding features that covary in time and space, we would clumsily fail in any object discrimination and recognition tasks (Kahneman et al., 1992). Many studies have shown that serial dependence is larger when two visual features are similar and close in space and time. This has fostered the notion that serial dependence is strictly related to object processing and the spatiotemporal feature correlations that define object continuity. However, a key question is whether serial dependence is causal to or contingent upon the experience of object continuity.

Collins (2022b) proposed two possible scenarios. First, object features may be bound in visual working memory, and serial dependence operates on “object-level” representations and is therefore object selective (see Figure 1D). A second option is that serial dependence operates directly on elementary features. In this scenario, the visual system assumes that similar features in close spatial and temporal proximity belong to the same object. Both scenarios overlap with the definition of the continuity field (Fischer & Whitney, 2014) as a mechanism that promotes the continuity of object representations by smoothing over spurious changes in object features—changes that generally do not reflect an object change. Yet, the first possibility implies no direct causal role of this mechanism in object continuity: To operate on “object-level” representations, object identity must first be established, at each time point (i.e., serial dependence is contingent on object continuity). The second possibility instead advocates a causal role, as the experience of object continuity depends on how the visual system combines elementary features and infers their common sources over time. The paradox, however, is that in this latter scenario, serial dependence does not need to be contingent on object continuity and may occur for similar visual features that belong to different objects (see Figures 1C,E).

How features are integrated into object representations is a fundamental question in perception. The link between serial dependence and object continuity may provide new insights into this relationship. But this involves a chicken–egg problem since it is difficult to separate the definition of a “visual object” from the definition of serial dependence: If objects are temporally continuous in terms of locations and features, how can we distinguish whether serial dependence operates on object-level representations or on spatially and temporally similar visual features, which tend to be an object in the first place?

In the following sections, we describe several approaches to this question by testing the role of feature similarity, feature conjunctions, and object identity.

Feature similarity

Work on serial dependence has tended to focus on how prior visual features influence the judgment of current ones. What we mean by “feature” is any property of a stimulus that can be quantified along a single dimension. This includes elementary features typically used in vision research, such as orientation, motion, or size, but also more complex and abstract properties such as facial emotions and attractiveness. In all these cases, features—and related judgments—can vary in quasi-continuous space (e.g., the full circle for circular features, a linear continuum for size). This quasi-continuous space provides a high-resolution sampling of perceptual biases, which can reveal mixtures of attractive and repulsive biases and their relation to proximity in feature space.

Serial dependence in judgments about visual features is well established (“Serial dependence paradigms”). The effects show systematic tuning in feature space: Errors tend to be more biased toward prior features when the difference between the previous and current feature is small—the effect of feature similarity or proximity. Several studies, particularly using orientation, have also shown that, as the distance between features increases, the bias reverses, becoming repulsive. For example, Fritsche and de Lange (2019) reported that typical positive serial dependence for small orientation differences between inducer and test is accompanied by a smaller repulsive bias for orientation differences larger than 45° to 50° (see also Bliss et al., 2017, and Figure 2). Similar patterns have been reported in studies manipulating task relevance and attentional roles of inducers (Rafiei, Chetverikov, et al., 2021; Rafiei, Hansmann-Roth, et al., 2021).

The presence of both attractive and repulsive biases can have at least two explanations. One involves mechanisms that simultaneously push decisions toward prior similar features and away from different ones—a behavior reminiscent of excitatory/inhibitory activity in working memory circuits (Stein et al., 2020); another, the coexistence of positive serial dependence and adaptation-like negative aftereffects (Fritsche et al., 2017, 2020; Pascucci et al., 2019; Pascucci & Plomp, 2021). Note that, to generate typical serial dependence patterns (Figures 2A,B), the repulsive component should be more broadly tuned than the attractive one, and we believe that this aspect is an interesting topic for future research. Another explanation is that these feature space effects depend on the “objecthood” assigned to a given feature change (Liberman et al., 2016; Pascucci & Plomp, 2021): When two features presented at consecutive moments are similar enough, the perceptual system integrates them into an object. Conversely, when the change is sufficiently large, they might be differentiated as separate objects.

Feature similarity effects may reveal the resolution with which the perceptual system can distinguish changes in a visual feature. However, a clear understanding of the mechanisms that lead to this mixture of positive and negative biases and their relation to object processing is still lacking. Manipulating a single feature cannot provide a conclusive answer, as effects of feature similarity can also be found when objects change (Ceylan et al., 2021). Future research should look at whether feature tuning is affected, for instance, by the interval between trials, since the expected feature variability of an object might depend on time.

Feature conjunctions

Another approach to addressing the relation between serial dependence and object processing is to manipulate conjunctions of features: Objects are, by definition, made up of multiple features that covary in time and space.

A straightforward way to test the role of feature conjunctions is to include changes in a “context” feature—in tasks that involve the judgment of another feature (the “content” feature). So, in tasks requiring the judgment of visual orientation or direction of motion (content feature), the stimuli may change color (context feature). Several studies have shown that when context changes are task irrelevant, serial dependence is unaffected by the context feature: Decisions on the current orientation are biased toward prior stimuli, independently of color changes in the stimulus. Assuming that irrelevant features still define object identity in the natural world, these results indicate that serial dependence occurs at a more abstract level of representation than the one where features are integrated (Figures 1C,E).

Studies using postcueing have, however, revealed that context features can modulate serial dependence, depending on task relevance (see “Postcueing paradigms”). In Fischer and colleagues (2020), two clouds of moving dots with different colors were presented on each trial, one after the other. A postcue indicated the color (or the serial position) of the cloud for which participants should reproduce the motion direction. Between trials, serial dependence was largest for stimuli with a matching context feature (e.g., same color, same serial position), but, crucially, only if the context feature was task relevant (if color was the cued feature to indicate the relevant dot cloud). In other experiments, changes in spatial and, to a lesser extent, serial position affected serial dependence even if these changes were task irrelevant, suggesting that spatial and serial position might be more automatically integrated into object representations in working memory. Recent work by Houborg and colleagues demonstrates, however, that unless required by attention set and task context, serial dependence is independent of the integration of elementary visual features into objects, or content–context integration (Houborg et al., 2022). In this study, serial dependence in orientation occurred irrespective of changes in the color of stimuli and even when participants were explicitly asked to discriminate the stimuli based on their color.

Collins (2022b) found that serial dependence in shape judgments occurred despite changes in the orientation of the object but was larger if the orientation of the object was constant. The same pattern was found when facial emotional expressions were the content feature and identity was the context feature: Serial dependence occurred regardless of identity but was larger when identity on the current and previous trials matched. Consistently, Taubert, Van der Burg, and Alais (2016) tested a paradigm reminiscent of online dating applications finding that serial dependence in attractiveness judgments of faces was larger when the orientation (upright or inverted) of the face was the same on consecutive trials. Also, Liberman and colleagues (2018) suggested that serial dependence in emotional expression judgments was selective for gender but not ethnicity, suggesting that gender differences create larger dissimilarity between faces than ethnicity (at least for their face stimuli).

Manipulations of feature conjunctions have yielded a wide spectrum of results, suggesting that, while not being a prerequisite, effective continuity in feature conjunctions can modulate serial dependence. Serial dependence may therefore occur both at the level of individual features and for objects with integrated features, and the particular task demands might determine which pattern occurs (Collins, 2022b). Furthermore, serial dependence for feature conjunctions may also depend on whether the context feature is completely task irrelevant or can interact with the processing and the relevance of the content feature.

These results suggest that serial dependence is not necessarily contingent on object continuity since it occurs even when the secondary features of an object change. But they also suggest a potential causal role: Serial dependence reflects the binding of sequential instances of the same feature in close spatiotemporal proximity. While this might be directly related to the spatiotemporal correlations that characterize the features of the same object in the natural world, it might lead to idiosyncratic biases in empirical settings where serial dependence occurs despite changes in context features.

Object identity

In the studies above, object identity is generally defined by stimulus similarity (whether the consecutive stimuli share features). Other studies have investigated the effects of object identity without necessarily tying it to stimulus similarity.

For instance, the coherent behavior of objects passing behind occluders creates a strong sense of continuity (Carey & Xu, 2001; Spelke et al., 1995), a technique widely used in the film industry (called the “cowboy switch”). A stunt double performs a difficult action and then disappears behind an occluder, before being substituted by the original actor. This trick leads to the impression that the same actor did the whole scene. Leveraging this phenomenon, Liberman, Zhang, and Whitney (2016) presented a sequence of Gabor stimuli on two sides of a rectangle. In one condition, the Gabor moved across the screen, passing behind the rectangular occluder with either a coherent or an incoherent trajectory. In another condition, the same Gabors were presented statically, one after the other, on either side of the rectangle. Perceptual decisions on the orientation of the final Gabor were influenced by the orientation of the preceding Gabor only when the motion of the Gabor was coherent so that it seemed to pass behind an occluder. This shows how object identity and continuity can be manipulated by exploiting the dynamics of real-world objects. This also shows that when object representations are established with real-world dynamics, serial dependence is contingent on object representations, rather than simple visual features.

As in the example above, testing the role of object identity goes beyond manipulating single features and feature conjunctions but requires the representation of an object and its spatiotemporal properties. The studies described in “Feature conjunctions,” where complex stimuli like faces were manipulated, can also assess the effects of object identity. Indeed, what we call “context change” is often a change in the face shown from one trial to another. Since faces are not easily reduced to low-level features and require holistic processing (McKone et al., 2001; Sigurdardottir et al., 2015), face changes can be considered changes in visual objects. As mentioned, these studies show that serial dependence occurs when objects are constant and the features change (e.g., for changes in the viewing angle of a face; Liberman et al., 2014), and even when there are major changes in the object itself (Ceylan et al., 2021).

Another potential “litmus” test for the role of object identity is to keep the content feature constant while changing all other object features. For instance, Ceylan and colleagues (2021) found positive serial dependence for orientation judgments even when the stimulus changed in spatial frequency or changed completely (e.g., a Gabor stimulus turning into a symmetric dot pattern). Similarly, using naturalistic stimuli in virtual reality, Tanrikulu, Pascucci, and Kristjánsson (2021) showed that serial dependence in orientation judgments was the same whether an object (e.g., a regular toothbrush) was constant or switched to another object within the same category (an electric toothbrush) or even to a different object from a different category (i.e., a sword). Goettker and Stewart (2022) also demonstrated that serial dependence in oculomotor responses to dynamic stimuli occurs between completely different objects (moving car vs. blob). Finally, serial dependence can be found between sequential (i.e., number of events over time) and simultaneous (i.e., number of items in space) numerosity representations and between semantic (presenting a numeral) and simultaneous representations of numerosity magnitudes (Fornaciai & Park, 2022).

The results reviewed earlier indicate that serial dependence occurs at many levels, from features independently of object identity, to objects, which can then affect features. However, the findings do not point to an unequivocal functional role for serial dependence in object processing and show that serial dependence might be modulated by, but is not necessarily contingent upon, object identity and continuity. As proposed by Houborg and colleagues (2022), at least when observers have to perform a task, the perceptual system may operate by parsimony, summoning feature conjunctions and holistic processing only when strictly required by the task, while otherwise reducing the complexity of its representations to the lowest-dimensional format possible (e.g., representing features like orientation and motion direction as “tilted lines” and discarding other features; Kwak & Curtis, 2022). This entails the prediction that the complexity of processing on the current trial determines the complexity of the information that propagates to the next trial.

Our review of the paradigms, findings, and theoretical challenges of serial dependence has so far revealed a complex and multifaceted phenomenon. This can also be seen in the multitude of models that have been proposed in nearly a century of research. In the following, concluding part of this review, we provide a general overview of the main modeling frameworks of serial dependence.

A brief history of research on history biases

The study of serial dependence has been accompanied by numerous conceptual and modeling accounts to explain the behavioral patterns and establish the inherent computational principles. Notably, old and new challenges echo one another. Some of the recurring questions involve the nature of repulsive and attractive effects, the relation of these biases to perception and working memory, the role of uncertainty, and the putative functional role in everyday vision. In this final section, we provide a brief overview of a century of models and frameworks of serial dependence, intending to look back to move forward.

Historical models

The classic laws of psychophysics have historically relied on a static relationship between stimulus and sensation (e.g., Weber's law). Some recognized the importance of sequential effects—the judgment of a sensation depends not only on the present stimulus but also on preceding events. Hence, alternative forms of psychophysical functions incorporated a dynamic component that could account for sequential effects in traditional magnitude scaling tasks. The approach involved finding a term that could complete a given psychophysical law (e.g., Stevens’s power law; Stevens, 1957) with a temporal component (DeCarlo & Cross, 1990). Two effects were mostly considered: assimilation (a form of positive serial dependence) and contrast (a form of negative serial dependence). Their nature represented a point of divergence for different theoretical frameworks.

In the perceptual/memory model of sequential effects (Cross, 1973; DeCarlo & Cross, 1990), information about immediately preceding stimuli was incorporated in Stevens’s power law, with a parameter determining the direction, attractive or repulsive, of the effect. The resulting equation could reflect either systematic influences of previous stimulus intensity on current perception (a purely proactive effect) or interference between current and prior perception in memory (Lockhead & King, 1983).

The models above sought to make psychophysical functions dynamic, echoing the intuition behind Helson's adaptation-level theory (Helson, 1948, 1964): Helson (1948) claimed that “any momentary state of the system [represents] a quasi-stationary process in dynamic equilibrium” (p. 298), and “for every excitation-response configuration there is assumed a stimulus which represents the pooled effect of all the stimuli and to which the organism may be said to be tuned or adapted” (Helson, 1947, p. 2).

Beyond systematic effects from previous stimuli, other models have focused on response heuristics in perceptual judgments. These models assumed that observers in psychophysical experiments tend to repeat previous responses under uncertainty (Garner, 1953; Ward & Lockhead, 1971). Forms of heuristics also include psychological phenomena where prior events are implicitly or explicitly used to predict the next outcome (Budescu, 1985; Kahneman & Tversky, 1972). Heuristics of this kind can induce biases where present stimuli are judged as similar to previous ones but different from more remote ones, under well-known fallacies (gambler's or hot-hand fallacies), where a structure is expected in random sequences (i.e., that after a few repetitions, a change is more likely). Response heuristics and perceptual/memory effects have been considered complementary components of sequential dependencies (DeCarlo & Cross, 1990).

Conceptually distinct accounts are based on the relativity of perceptual judgments (Stewart, 2007; Stewart et al., 2005). Judgments are always made about both immediate and long-term context: Perceptual decisions are relative and rely on comparing successive stimuli. Trial history, therefore, affects the perceived distance between the present and the prior stimulus, leading to assimilative (attractive) or contrastive (repulsive) effects depending on stimulus similarity (Hsu, 2021).

Other frameworks, inspired by Thurstonian models, involve criterion-setting accounts. The assumption is that responses cause a momentary shift in the location of the response criterion along the stimulus continuum (Treisman & Williams, 1984). According to an influential model of this sort, criterion shifts are mediated by two opposite mechanisms that operate under adaptive principles based on the nonrandomness of real-world events. The tracking mechanism relies on the fact that prior perceptual judgments reflect the best guess about the current state of the world. Consequently, prior responses shift the criteria to facilitate the repetition of prior observations, leading to positive serial dependence. The stabilization mechanism relies on the fact that the nonrandomness of external events is typically short-lasting, and as the interval increases, the present stimulus becomes less likely to be similar to the past one, causing negative serial dependence (Hsu, 2021; Treisman & Williams, 1984).

Despite clear differences between these models, they face the same challenge of dealing with (a) two potentially simultaneous but opposing mechanisms causing attraction/assimilation and repulsion/contrast; (b) multiple stages, from early perceptual effects to higher-level effects due to memory and decisions, an aspect usually reflected, but not fully captured, in the classic dichotomy between stimulus and response; and (c) the functional role of these opposite biases.

Recent models

The recent interest in serial dependence has seen the rise of new models grounded on more contemporary theoretical and computational frameworks, focusing mostly on modeling results from adjustment tasks with circular features, like orientation or motion. The modeling, therefore, differs from studies with magnitude scaling, absolute judgments, identification, and categorization. Additionally, the mechanisms behind sequential biases may differ between linear or circular stimulus spaces and prothetic or metathetic continua. Nevertheless, important commonalities have emerged. These recent models can be broadly distinguished by whether they rely on mechanistic principles and hardwired components, cognitive/perceptual architectures, or rational principles.

Mechanistic models

The labeled-line population coding model of serial dependence originally proposed by Fischer and Whitney (2014) belongs to the first category. In this model, positive serial dependence is implemented as a change in the response gain or shift in the tuning function of a population of low-level visual neurons encoding orientation information (i.e., a mechanism hardwired in early visual processing). A labeled-line orientation model is composed of a set of neurons “voting” for their preferred orientation and generating a population response profile whose mean or maximum represents the “perceived” orientation. The gain model involves a phasic gain increase in the response of previously stimulated neurons that bias their individual and population response to the present stimulus. This model expresses the core idea behind the “continuity field”: a mechanism that binds information based on feature similarity, spatial proximity, and (relatively long) time intervals by temporarily increasing the receptivity of visual neurons responding to specific features and objects (Fischer & Whitney, 2014).

Other examples of mechanistic models come from the perspective of serial dependence as a visual memory effect, suggesting that hysteresis in perceptual decisions arises from the interplay between activity-based and activity-silent neural states in working memory. According to the bump-reactivation hypothesis (Barbosa et al., 2020; Stein et al., 2020), neuronal activity from preceding trials remains imprinted in synaptic connections as activity silent states in the prefrontal cortex. In close analogy with the principle behind sonar and echolocation (Wolff et al., 2017), a new sensory impulse reignites these hidden states, causing an “echo” of past activity patterns in the processing of new sensory information (Bae & Luck, 2019).

Cognitive/perceptual architectures

Models based on cognitive/perceptual architectures have been proposed to account for the coexistence of attractive and repulsive serial dependence. These models typically employ a simple two-stage architecture of encoding and decoding. The two-process model represents the first example where a simple two-layer neural network generated data consistent with simultaneous negative and positive serial dependence (Pascucci et al., 2019). A low-level input layer resembling a neural population selective to basic visual features is followed by a readout layer decoding the input. Serial dependence emerges from short-term plasticity and slowly decaying weights in synaptic connections among the layers, introducing bias and inertia in the decoding of sequential sensory signals (Ceylan et al., 2021; Pascucci et al., 2019). This model is agnostic regarding the computational goals of this bias but assumes a form of neuronal adaptation causing repulsive biases at lower stages.

Rational models

Rational models of serial dependence embed explanatory and generative models with the computational goal of optimality and Bayesian principles. Cicchini and colleagues (2018) proposed an ideal observer model where serial dependence follows the rules of optimal cue integration: Previous and present stimuli are combined according to their relative uncertainty, with reliable stimuli weighted more highly than unreliable ones. A related approach is the probabilistic model of van Bergen and Jehee (2019), which incorporates an internal model of the transition distribution of natural orientation changes in a Bayesian model where prior and present stimuli are optimally combined. Similarly, Kalm and Norris (2018) proposed a Bayesian filter where current decisions combine information from present stimuli and a mixture of prior states, taking into account more than one trial in the past. This line of modeling makes the computational goal of serial dependence explicit: to use prior sensory input to optimize perception, reduce uncertainty, and minimize noise. A simplified version of these models, where only the uncertainty associated with the present (but not the previous) stimuli is taken into account, has recently been proposed to explain the lack of evident “optimal” combination between consecutive stimuli (Ceylan et al., 2021).

Building on the models above, Fritsche and colleagues (2020) combined a cognitive/perceptual architecture with rational and normative principles, formalizing a Bayesian and efficient observer model of serial dependence. As in previous attempts, this model embeds an encoding and a decoding stage. Efficient coding schemes at the encoding stage promote long-term repulsive effects where the encoding of a new stimulus feature is shifted away from the prevailing stimulus statistics over minutes in the past. An ideal Bayesian observer decodes the input from the encoding stage using the most recent stimuli as priors. Hence, both short-term attraction and long-term repulsion, originating at different stages of processing, contribute to the final decision. More recent work within Bayesian frameworks suggests that interindividual variability in positive serial dependence might be due to different beliefs about the temporal statistics of the environment (Glasauer & Shi, 2022).

Discussion

Our aim was to provide a “state-of-the-art” overview of serial dependence in vision, trying to highlight the main themes and identify critical future questions. A major challenge is that the findings do not paint a single clear picture, and small tweaks to paradigms can cause serial dependence patterns to differ largely. For example, the peak biases and whether they are attractive or repulsive seem to differ by whether the tasks involve adjustment, forced choice, or classification. A key challenge is to try and determine what conditions cause what sort of serial dependence. Our hope is that our summary will help with identifying issues for further investigation.

We also highlight some theoretical problems inherent in the literature—serial dependence is assumed by many accounts to promote object continuity, but the level of representation involved, whether it is about features, objects, or abstract information, is still unclear. Eventually, perceptual decisions often deal with complex and manifold events, incorporating many levels, from physical aspects of the stimulus to abstract descriptors and categories. One possibility is that serial dependence operates opportunistically, depending on the circumstances and the requirements of the task. Hence, serial dependence may involve different levels of processing and may lead to biases with diverse characteristics in different tasks. Whether the bias occurs at the level of features, integrated objects, or more abstract representations could depend on the most parsimonious representation that the brain can use to perform a given task.

Future directions

In our review, we have laid emphasis on several open or unresolved questions. As a final step, we list some key issues that can be fundamental fuel for future research:

• 1.What is the role of the “task” in serial dependence? Several studies have shown serial dependence without any explicit task while others support a clear role of attention and top-down, task-related factors (“Processing stages”). One direction for future research could be to understand what the absence of a task implies and under which conditions a stimulus is completely unattended to and not processed further. Whether serial dependence occurs for information that is irrelevant in our daily vision or only for attended features is a crucial question. Hence, the role of the “task” must be clarified.
• 2.When do “objects” become objects in serial dependence? As one of the core sections highlighted (“The problem of object continuity”), it is still unclear under which conditions serial dependence keeps track of objects, as they appear to us in real life. Future research might focus on more realistic stimuli and environments (e.g., virtual reality) to use familiar objects that we usually interact with every day.
• 3.What is the potential connection between serial dependence and other temporal biases such as attentional priming (Kristjánsson & Ásgeirsson, 2019; Pascucci et al., 2012), distractor learning and habituation (Geng et al., 2019; Turatto et al., 2018, 2019; Turatto & Pascucci, 2016), feature distribution learning (Chetverikov et al., 2020), and the persistence of event and object files (Pascucci & Plomp, 2021; Scholl, 2007)?
• 4.A fundamental future step will be to develop and test models of serial dependence that can conjointly account for neural and behavioral results, adhering to the rule of parsimony. Many existing models have an increasingly complex structure and number of parameters, undermining their ability to generalize and account for peculiar aspects of serial dependence. What is the minimum structure of a flexible model that can account for repulsive and attractive biases? What are the missing ingredients?
• 5.Outside the laboratory, what is the purpose of serial dependence? Are the effects in real life larger or smaller, depending on the actual properties of real stimuli (e.g., their weight, size, physical distance, intrinsic dynamics and so on)?
• 6.To what extent are the effects of temporal context (e.g., serial dependence) analogous to those of spatial context? Any overarching and sufficiently flexible theoretical framework of serial dependence should take into account the relationship between spatial and temporal context and whether a single model can account for both.