Serial dependence in vision: merely encoding the previous-trial target is not enough

Abstract

The reported perception of a visual stimulus on one trial can be biased by the stimulus that was presented on the previous trial. The present study asked whether encoding the previous-trial stimulus is sufficient to produce this serial dependence effect or whether this effect also depends on post-encoding processes. To distinguish between these possibilities, we designed a task in which participants reported either the color or the direction of a set of colored moving dots on each trial. The to-be-reported dimension was indicated by a post-cue after stimulus offset, so participants were required to encode both features of every stimulus. We assessed serial dependence for motion perception as a function of which feature dimension was reported on the previous trial. In Experiment 1, we found a serial dependence effect for motion only when participants had reported the direction of motion on the previous trial, and not when they had encoded the direction of motion but reported the color of the stimulus. Experiment 2 confirmed that this pattern of results was not driven by the difficulty of the color task. When we used the same response modality for both motion and color report in Experiment 3, we found a significant serial dependence effect following both color-report and motion-report trials, but it was significantly weaker following color-report trials. Together, these findings indicate that post-perceptual processes play a critical role in serial dependency and that the mere encoding of the previous-trial target is not sufficient to produce the serial dependency effect.

Introduction

Our current thoughts and behaviors are influenced by our past experiences. In the case of visual perception, studies of serial dependence have shown that perceptual decisions for various types of visual stimuli—including simple visual features (e.g., orientation and spatial location) and more complex objects (e.g., faces)—are systematically biased by recent stimuli even when those stimuli are no longer task-relevant (Bae & Luck, 2019; Fischer & Whitney, 2014; Liberman, Fischer, & Whitney, 2014; Papadimitriou, Ferdoash, & Snyder, 2015). Despite the ubiquity of such effects, little is known about the mechanism by which past visual experiences influence the processing of new visual inputs. In particular, although researchers assume that serial dependence is driven by the perception of the previous stimuli, the operation of postperceptual processes may also be essential.

Most previous studies of serial dependence required participants to report a single target feature dimension on every trial, which confounds the perception of this feature with the report of the feature. Some studies, however, included a condition where no explicit response was required on some trials, which can potentially unconfound the perception and the report of the target feature (Czoschke, Fischer, Beitner, Kaiser, & Bledowski, 2019; Fischer & Whitney, 2014; Suárez-Pinilla, Seth, & Roseboom, 2018). These studies found that serial dependence was present even when the target feature was not reported on the previous trial, suggesting that serial dependence can be produced by the mere encoding of the previous-trial stimulus. However, because participants were frequently required to report the target feature, it is possible that participants prepared a response even if it was not required on that trial and that some aspect of response preparation produced the serial dependency effect.

A recent study used a stimulus with two feature dimensions and tested whether serial dependence was present for one feature dimension when the other feature dimension was reported on the previous trial (Suárez-Pinilla et al., 2018). Participants in this study saw a random dot kinematogram (RDK) and reported either the mean or the variance of the dot directions on a given trial. A core finding was that, even though participants saw both the variance and the direction, serial dependence in visual variance occurred only when participants reported visual variance on the previous trial. However, it should be noted that the study used a pre-cue paradigm where the relevant feature dimension on a given trial was known to participants prior to stimulus onset. Consequently, feature-based attention might have attenuated the perceptual or memory encoding of the unattended dimension. That is, the lack of serial dependence may have reflected a reduction in the encoding of the other feature dimension rather than a lack of the report of this dimension.

In the present study, participants had to perceive and remember both features of a two-dimensional stimulus on every trial, but were asked to report only one of the two features on a given trial. Specifically, participants saw an RDK with colored dots for a short period of time and were then cued to report either the direction or the color of the dots (Figure 1a). Crucially, color-report and direction-report trials were randomly intermixed, and the relevant dimension was post-cued after stimulus offset. Consequently, participants had to perceive and remember both the direction and the color of the dots on each trial to perform the task accurately. This made it possible to test whether stimulus information that was encoded¹ but was unused on the previous trial would still bias the processing of the next stimulus.

Figure 1.

Experiment 1 task and results. a. Sequence of stimuli. On each trial, participants saw a random dot kinematogram (RDK) drawn in colored dots, with a random direction of motion and a random color. The line segments shown in the figure represent the direction of motion and were not visible. After the RDK, a post-cue indicated which feature dimension – direction of motion or color – should be reported. When the direction cue (D) was presented, participants continuously adjusted orientation of a white response line using a mouse to report the direction of motion of the RDK. When the color cue (C) was presented, two test colors were presented and the participant pressed a key to report the color of the RDK. b. Mean response error for the direction report as a function of the absolute direction difference between the current and previous trial, plotted separately for trials preceded by a direction report and for trials preceded by a color report. Positive values indicate that the current-trial direction report was biased away from the direction of the previous trial, and negative values indicate that the current-trial direction report was biased toward the direction of the previous trial. c. Mean response error for the direction report, averaged across small direction differences (22.5°~67.5°), plotted separately for trials preceded by a direction report and trials preceded by a color report. Error bars indicate ±1 SEM. *** = p <.001

If encoding the stimulus is sufficient to produce serial dependence, then serial dependence for motion direction should be observed even when participants reported color on the previous trial. However, if additional report-related processes are necessary for serial dependence to occur, then serial dependence for motion direction should be reduced or eliminated when color was reported on the previous trial. In a series of three experiments, we found that serial dependence for motion direction was largely absent following color-report trials. These results demonstrate that encoding a feature is not sufficient for serial dependence and that post-encoding processes play a key role.

Experiment 1

Method

Participants

Twenty-four college students between the ages of 18 and 30 participated (13 female, 11 male). The sample size was determined a priori on the basis of published studies of serial dependence (Bae & Luck, 2019; Fischer & Whitney, 2014). The study was approved by the UC Davis Institutional Review Board, and participants gave informed consent.

Stimuli & Tasks

Stimuli were generated in Matlab (The Mathworks, Inc.) using Psychtoolbox (Brainard, 1997; Pelli, 1997) and were presented at 60Hz on a LCD monitor (Dell U2412M) with a white background (87.6 cd/m²) at viewing distance of 70 cm. A white fixation dot (87.6 cd/m²) was continuously visible on a black disk (5° diameter, < .1 cd/m²) at the center of the screen except during the intertrial interval (see Figure 1a).

We used a standard RDK algorithm to generate the motion stimulus (Roitman & Shadlen, 2002). The stimuli consisted of three groups of dots (dot diameter = 0.3°) which were randomly distributed within the black disk (16.7 dots per square degree per second). We used a larger-than-typical dot size to increase the discriminability of the dot colors. Each group of dots was presented for one video frame (16.67 ms/group) and the dots were replotted in new locations after 2-frame delay. Thus, the coherent motion was made by the correspondence between dots in frame N and in frame N+3. When the new location of a given dot was outside of the black disk, that dot was replotted at a random location on the circumference of the black disk to maintain dot density. The coherence level was set to 100% and the speed of motion was set to 6°/s.

As illustrated in Figure 1a, each trial began with the appearance of the fixation dot for 500 ms on the black disk. The RDK was then presented for 500 ms (30 frames). The direction of motion on a given trial was one of 16 discrete values equally spaced around the 360° of possible directions (from 11.25° to 348.75°, in steps of 22.5°). The color of the dots on a given trial was chosen from a set of 180 colors that formed a circle in the CIE L*a*b* color space (L* = 70, a* = 0, b* = 0, radius = 39, sampled in steps of 2°). Participants were asked to remember both the exact direction of motion and the exact color of the dots on every trial.

After the RDK, participants reported either the direction of motion or the color of the dots. The to-be-reported feature was indicated by a white post-cue letter (“D” for direction or “C” for color; 87.6 cd/m²; 0.8° x 1.0° for width x height) presented at the center of the black disk for 500 ms. When a direction report was indicated, a mouse cursor appeared at the center of the aperture. As soon as participants moved the mouse cursor, a white line connecting the center of the disk and a point on a circumference of the disk appeared. The orientation of the white line was continuously updated depending on the orientation created by the cursor and the center of the screen, allowing observers to adjust the orientation of the white line to the perceived motion direction. The observer finalized the direction report by clicking a mouse button. When a color report was indicated, two colored circles (diameter 1. 5°) were presented on 4° to the left and to the right of the fixation dot. The color of one of the circles was always identical to the color of the dots in the RDK from that trial. The color of the other circle was 10, 20 or 40 steps away (in either direction, random across trials) from the true dot color. Observers pressed either the left or right arrow key on a computer keyboard to report which of the two colors was identical to the color of the dots. The correct color was equally likely to appear on the left and right sides. We used this simple 2AFC color task to avoid any possible interference from the color task to the motion task. The post-cue remained visible until the response was completed. The next trial began after 1000-ms intertrial interval during which only the black disk was visible.

After 16 practice trials, each participant completed 480 experimental trials, with exactly 30 trials for each of the 16 directions of motion. This included 240 direction-report trials intermixed with 240 color-report trials. The order of trials was randomly shuffled. Participants took a short break after each block of 48 trials.

Data Analysis

The main question of the present study was whether the reported direction on a given trial was influenced by the direction of motion on the previous trial. To answer this question, we computed the direction difference between a given trial and its preceding trial. The first trial in each block was necessarily excluded (total of 10 trials per participant). We used 16 discrete motion directions, and there were also16 discrete direction differences. We used the absolute direction difference to increase the number of trials in each cell of the design, resulting in 9 different absolute direction differences (0°, 22.5°, 45.0°, 67.5°, 90.0°, 112.5°, 135.0°, 157.5°, 180.0°).

Our main dependent variable was the response error, defined as difference between the reported direction and the true motion direction for the current trial. We coded the sign of this difference so that the response error reflected the bias relative to the direction of motion in the previous trial. Positive values indicated that the direction report was biased away from the previous-trial direction, and negative values indicated that the direction report was biased toward the previous-trial direction. Recoded response errors were then averaged separately for each of the 9 direction differences. We excluded trials with a direction difference of 0° or 180° because response bias relative to the previous trial is undefined for those trials. We also excluded trials on which the current-trial response error was larger than 60° (0.38% of the direction-report trials), which were likely to reflect lapses of attention.

Following previous research (Bae & Luck, 2017, 2019), we expected that the reported direction on the current trial would be repelled away from the previous-trial direction, especially when the two directions were similar (e.g., < 90°). We therefore assessed the presence of a serial dependence effect by averaging together the direction error values for the small direction difference trials (i.e., 22.5°, 45.0°, and 67.5°) and comparing this value to zero using a one-sample, two-tailed t test².

Results and Discussion

Mean accuracy for the color task was 87.6% (SEM = 0.7), demonstrating that participants successfully encoded color information into working memory on a large proportion of trials.

Figure 1b shows the mean response error for the motion-report trials, separately for trials preceded by a direction report and for trials preceded by a color report. When the direction difference between the previous trial and the current trial was smaller than 90°, the reported direction on a given trial was biased away from the direction in the previous trial, but only when the preceding trial was a direction-report trial. There was little or no evidence of serial dependence on motion-report trials when the preceding trial was a color-report trial, even though participants necessarily encoded both direction and color on every trial.

Following previous research (Bae & Luck, 2017), we quantified the serial position effect by averaging the motion report errors across the trials with direction differences of less than 90° (i.e., direction differences of 22.5°, 45.0°, and 67.5°). Figure 1c shows this averaged value for trials preceded by a direction report and for trials preceded by a color report. When we compared these values to zero with one-sample t tests, we found that trials preceded by a direction report produced a significant repulsion bias (t(23) = 5.150, p = 3.219e-05, Cohen’s d = 1.486), whereas trials preceded by a color report showed a hint of attraction bias that did not reach significance (t(23) = −1.855, p = .076, Cohen’s d = −.535). In addition, a paired t test showed that the serial position effect was significantly greater for trials preceded by a direction report than for trials preceded by a color report (t(23) = 6.011, p = 3.947e-06, Cohen’s d =1.227).

These findings provide clear evidence that, at least in the case of motion direction, encoding a stimulus dimension on one trial is not sufficient to bias the reported value of that dimension on the next trial. Instead, some post-encoding process is necessary for the perceived value on one trial to bias performance on the next trial.

Experiment 2

The color-report task in Experiment 1 was quite difficult, and this may have produced interference with the motion representation on that trial, preventing serial dependence on the next trial. Experiment 2 therefore used an easier color task (Figure 2a).

Figure 2.

Experiment 2 task and results. a. Sequence of stimuli. The dots were always either pink or green, and the test options were also pink and green. b. Mean response error for the direction report as a function of the absolute direction difference between the current and previous trial, plotted separately for trials preceded by a direction report and trials preceded by a color report. c. Mean response error for the direction report, averaged across small direction differences (22.5°~67.5°), plotted separately for trials preceded by a direction report and trials preceded by a color report. Error bars indicate ± 1 SEM. *** = p <.001

Method

The methods of Experiment 2 were identical to those of Experiment 1 except as noted. A new group of 24 college students participated (21 female, 3 male).

Instead of being drawn from a set of 180 possible colors, the dots in Experiment 2 were always either green (RGB = [68, 190, 170]) or pink (RGB = [236, 143, 173]; see Figure 2a). The two options at the time of test on color-report trials were also green and pink. Each participant completed 448 experimental trials (16 directions x 28 trials). This included 224 direction-report trials intermixed with 224 color-report trials. Trials with a response error larger than 60° (2.41% of the direction-report trials) were excluded.

Results and Discussion

Mean accuracy for the color task was 96.9% (SEM = 0.3), which was significantly greater than the mean accuracy for the color task in Experiment 1 (87.7%; t(46) = −10.84, p = 2.934e-14).

As shown in Figure 2b, the reported direction on a given trial was biased away from the direction on the previous trial (for direction differences < 90°), but only when the preceding trial was a direction-report trial. Figure 2c shows the response error averaged across trials with direction differences of less than 90°. One-sample t tests against zero indicated that trials preceded by a direction report exhibited a significant repulsion bias (t(23) = 4.417, p = .0001, Cohen’s d = 1.275), whereas trials preceded by a color report exhibited a small and nonsignificant attraction bias (t(23) = −.545, p = .590, Cohen’s d = −.157). The serial position effect was significantly greater for trials preceded by a direction report than for trials preceded by a color report (t(23) = 3.691, p = .001, Cohen’s d = 0.753).

These findings demonstrate that the main finding of Experiment 1—a serial dependence for direction of motion only when the direction of motion was actually reported on the previous trial—is replicable and does not require a difficult color task.

Experiment 3

In Experiments 1 and 2, direction of motion was reported using a continuous estimation procedure whereas color was reported with a 2AFC procedure. Thus, color-report and direction-report trials differed in the mode of the report as well as the content of the report. This raises the possibility that the lack of serial dependence in motion direction when the preceding trial was a color-response trial was a result of the mode of the response on the preceding trial. Experiment3 tested this possibility by using a delayed estimation procedure for both the color and motion reports (see Figure 3a).

Figure 3.

Experiment 3 task and results. a. Sequence of stimuli. The stimuli and task were identical to those of Experiment 1 except that participants reported the color (on color-report trials) by clicking on the color of the dots using a color wheel. b. Mean response error for the direction report as a function of the absolute direction difference between the current and previous trial, plotted separately for trials preceded by direction report and trials preceded by color report. c. Mean response error for the direction report, averaged across small direction differences (22.5°~67.5°), plotted separately for trials preceded by a direction report and trials preceded by a color report. Error bars indicate ± 1 SEM. *** = p <.001, ** = p <.01

Method

The methods of Experiment 3 were identical to those of Experiment 2 except as noted. A new group of 24 college students participated (14 female, 10 male).

To parallel the use of 16 directions of motion, the colors of the dots in Experiment 3 were selected from a set of 16 colors (in steps of ~22° on the color wheel), drawn from the same color space used in Experiment 1. On color-report trials, a color wheel composed of 180 colors from this color space was presented around the black disk (Figure 3a). Participants reported the perceived color of the dots from that trial by clicking the corresponding color on the color wheel with a mouse. The color wheel was randomly rotated every trial.

Results and Discussion

The mean absolute error for the color report was 15.88° (SEM = .64). Because the present study was designed to examine serial dependence in the motion report, our main analysis focused only on motion-report trials. However, as will be discussed later, our exploratory analysis on color-report trials found no clear evidence of serial dependence in the color report.

Figure 3b shows the mean response error for the motion-report trials. When motion was reported on the previous trial, the same repulsion pattern found in Experiments 1 and 2 was again observed, but this serial dependency was reduced when color was reported on the preceding trial. Figure 3c shows the response error averaged across trials with direction differences of less than 90°. One-sample t tests against zero indicated that a significant repulsion bias was present in both the trials preceded by a direction report (t(23) = 7.081, p = 3.252e-07, Cohen’s d = 2.044) and the trials preceded by a color report (t(23) = 3.204, p = .003, Cohen’s d = 924). However, a paired t test indicated that the effect was weaker for trials preceded by a color report (M=1.17°) compared to trials preceded by a direction report (M=3.46°), (t(23) = 3.717, p = .001, Cohen’s d = 0.759). Thus, although a small serial dependence effect in the perception of motion direction could be observed when color rather than motion was reported on the previous trial, the serial dependence effect was more than twice as large when the direction rather than the color of the stimuli was reported on the preceding trial.

The presence of a small repulsion effect when color was reported on the previous trial in this experiment may reflect an automatic preparation of the motion direction response on a subset of the color-report trials. That is, because the response involved moving the mouse to a spatial location on both color-report and motion-report trials, participants may have prepared the motion report even when cued to make a color report. Alternatively, it may be driven by the more precise encoding of the color and/or the difficulty of the color task relative to the tasks used in Experiment 1 and 2. Future research is necessary to investigate the source of the small serial dependence effect following color-report trials.

We also assessed whether the motion direction report on the current trial was influenced by the location of the mouse-click when color was reported on the previous trial. We found a hint of attraction toward the location of the previous-trial report (a mean bias of −0.86° when averaged across the 22.5°~67.5° direction differences), but this effect was not significantly different from zero (t(23) = −1.866, p = .075).

Discussion

The present study investigated whether encoding the previous-trial stimulus is sufficient to produce serial dependence in motion perception or whether serial dependence depends on post-encoding processes. We consistently found that serial dependence was reduced or eliminated when the direction information was not used on the previous trial even though the task required the direction of motion to be perceived and stored in memory on every trial. Although it will be important to determine whether this pattern of results generalizes to other visual features, these results indicate that encoding a stimulus is not sufficient to produce serial dependence on the next trial, and that serial dependence can be modulated by processes associated with the use of the relevant feature information.

We are not concluding that reporting the relevant information on the previous trial is always necessary to induce serial dependence effect. It may not be the report per se that produces serial dependence, but some other processes that accompanies the report (e.g., decision processes and/or response preparation). If the manipulation of the previous-trial report does not completely eliminate this process, then some serial dependency will still be observed. Thus, the finding that the serial dependence effect was not completely eliminated in Experiment 3 following color-report trials does not substantially weaken our main conclusion, which is that the mere encoding of the previous-trial stimulus is not sufficient for serial dependence to occur.

Additional research will be necessary to determine the specific mechanisms by which post-encoding processes in the previous trial modulate serial dependence, but there are at least two general classes of possibilities. One is that making the response on one trial enhances or transforms the memory representation of the reported feature so that it has a stronger impact on the encoding or response on the next trial. Another possibility is that responding to one feature of the stimulus (e.g., color) disrupts the memory representation of the other dimension (e.g., direction of motion) so that it can no longer influence processing on the next trial (Anderson, Bjork, & Bjork, 1994).

A recent study found that serial dependence can also be observed for color (Barbosa & Compte, 2018). Although the present study was not designed to investigate serial dependence in color perception, we explored whether color reports exhibited serial dependence in Experiment 3. We did not find clear evidence for a color serial dependence effect, irrespective of whether color or motion was reported on the previous trial. However, given the results reported by Barbosa and Compte (2018), we are not concluding that serial dependence is generally absent for color. The specific task used in the present study may have been not optimal for inducing color serial dependence. Future research is necessary to investigate this possibility.