Serial Dependence in the Perception of Faces

Summary

From moment to moment, we perceive objects in the world as continuous despite fluctuations in their image properties due to factors like occlusion, visual noise, and eye movements. The mechanism by which the visual system accomplishes this object continuity remains elusive. Recent results have demonstrated that the perception of low-level stimulus features such as orientation and numerosity is systematically biased (i.e., pulled) toward visual input from the recent past [1, 2]. The spatial region over which current orientations are pulled by previous orientations is known as the continuity field, which is temporally tuned for the past 10–15 s [1]. This perceptual pull could contribute to the visual stability of low-level features over short time periods, but it does not address how visual stability occurs at the level of object identity. Here, we tested whether the visual system facilitates stable perception by biasing current perception of a face, a complex and behaviorally relevant object, toward recently seen faces. We found that perception of face identity is systematically biased toward identities seen up to several seconds prior, even across changes in viewpoint. This effect did not depend on subjects’ prior responses or on the method used to measure identity perception. Although this bias in perceived identity manifests as a misperception, it is adaptive: visual processing echoes the stability of objects in the world to create perceptual continuity. The serial dependence of identity perception promotes object identity invariance over time and provides the clearest evidence for the existence of an object-selective perceptual continuity field.

Results

Experiment 1A: Serial Dependence of Perceived Face Identity

We presented a random series of faces drawn from an identity morph continuum (see Figure S1A available online) and measured the perceived identity of each face using a method of adjustment (MOA) task. On each trial, a random target face was presented followed by a test screen containing a random adjustment face, which subjects matched to the target face using a continuous identity morph wheel (Figures 1A and S1A). “Target face” denotes the face that subjects tried to match, “adjustment face” denotes the randomly selected starting point for matching the target, and “match face” denotes the face that subjects selected as most similar to the target face.

Figure 1. Experiment 1 Trial Sequence and Results.

(A) Trial sequence for the method of adjustment (MOA) task in experiment 1. On each trial, a randomly selected target face was presented for 750 ms, followed by a 1,000 ms noise mask of black and white pixels to reduce afterimages and a 250 ms fixation cross. Subjects then saw a test screen containing a random adjustment face, which they modified by scrolling through the continuous identity wheel to match the target face (see Figure S1A).

(B) Example data from subject 4, with each data point showing performance on one trial. In units of face morph steps, the×axis is the shortest distance along the morph wheel between the current and one-back target face (one-back target face − current target face), and the y axis is the shortest distance along the morph wheel between the selected match face and target face (match face – current target face). Positive×axis values indicate that the one-back target face was clockwise on the face morph wheel relative to the current target face, and positive y axis values indicate that the current match face was also clockwise relative to the current target face (Figure S1A). The running average (dashed line) reveals a clear trend in the data, which followed a derivative-of-Gaussian (DoG) shape (model fit depicted as solid line).

(C) Half-amplitude of the serial dependence for each subject in experiment 1A. Error bars are bootstrapped 95% confidence intervals, and p value is based on group permuted null distribution. Additional experiments show that memory confusions cannot fully explain our pattern of results (see Figure S2).

(D) Half-amplitude of the serial dependence for each subject in experiment 1B for trials with no one-back response. Sequential effects have been known to influence subjects’ responses by introducing dependencies between current and previous trial decisions [3–6]. However, these results are not due entirely to sequential decision biases, since we observed serially dependent perception without a one-back response. Error bars are bootstrapped 95% confidence intervals, and p value is based on group permuted null distribution.

Perceptual error was calculated as the shortest distance along the morph wheel between the match and target faces. Each subject’s error on the current trial was compared to the difference in target face identities between the current and previous trial (Figure 1B). We fit a simplified derivative-of- Gaussian (DoG) to each subject’s data and calculated p values using permutation analysis (Figure 1B; see Supplemental Experimental Procedures). All subjects displayed a positive DoG half-amplitude, indicating that perceived identity on a given trial was significantly pulled in the direction of identities presented in the previous trial (p < 0.01, n = 5, group permuted null) (Figure 1C). The largest attraction of perceived identity occurred when the one-back target face was, on average, ±24.5 morph frames away from the current target face, which resulted in an average perceptual pull toward the one-back face of ±3.5 face morph frames. The full amplitude of the effect was therefore 7 face morph steps, indicating that the current face appeared pulled toward the previous face by over 1.5 times the just-noticeable difference (see Supplemental Experimental Procedures). No subject showed a significant influence of faces seen two trials back, which may reflect a narrow temporal window over which identity serial dependence occurs. Average response time (RT) across subjects was 4,250 ± 2,168 ms; the one-back face occurred on average ~7,500 ms prior to the current trial face. Perceived face identity was therefore strongly attracted toward the identity of a random target face seen more than 7 s prior.

Experiment 1B: Serial Dependence without Previous Responses

To control for potential biases caused by prior responses or adjustment stimuli, four subjects completed a variation of experiment 1A in which they did not respond on a randomly selected 50% of trials. During these surprise “no response” trials, subjects saw the target face followed by a 2,000 ms blank screen before beginning the next trial. We analyzed the subset of trials in experiment 1B with no response on the one-back trial and fit a DoG to each subject’s data. All subjects had a positive DoG half-amplitude (p < 0.01, n = 4, group permuted null) (Figure 1D), indicating that subject responses per se are not necessary for the serial dependence of perceived identity. Nonetheless, the process of responding and attending to the stimulus may play an important role in serial dependence. Two participants showed a two-back serial dependence effect (p < 0.05, n = 2, permuted null), potentially due to the shorter time between current and one-back target faces with no response. The one-back target face occurred 3,250 ms prior to the current target face when no response was required on the one-back trial.

Experiment 2: Serial Dependence of Face Perception Using Constant Stimuli

Experiments 1A and 1B demonstrated that perception of face identity is pulled toward recently seen identities, a misperception that could facilitate stable face identity perception over time. In experiment 2, we used a two-interval forced choice (2IFC) design to determine whether serial dependence altered perception independent of response method. This experiment also had the benefit of reducing response time and the number of intervening stimuli seen during the response period.

The faces used in this experiment were a subset of those used in experiment 1, including original face A (#1), original face B (#50), and the 48 face morphs in between (Figure S1A). Before the experiment, subjects were trained to recognize faces A and B (see Supplemental Experimental Procedures). Immediately after training, participants were shown sequences of two faces per trial and judged which of the two faces looked more similar to face A (Figure 2A). The initial face presented in each trial, “first face,” was presented for 1,000 ms. The following face, “second face,” was presented for 500 ms and differed randomly from the first face by ±12, ±6, or 0 face morph steps (Figure S1B). Since subjects saw the first face for twice as long as the second face, we expected the one-back first face to have a stronger pull on the perception of the subsequent trial’s first face.

Figure 2. Experiment 2 Trial Sequence and Results.

(A) Trial sequence for 2IFC task in experiment 2. For each trial, the first face was presented for 1,000 ms, followed by a 1,000 ms noise mask and 250 ms fixation cross. Subjects saw the second face for 500 ms and judged whether the first face (press “1”) or second face (press “2”) looked more like original face A. Trial type was determined by comparing the position in the morph continuum of the current trial first face to that of the one-back trial first face. If the one-back first face was closer to original face A along the morph continuum, the trial was labeled an “A-previous” trial. Faces are shown here without added noise.

(B) Example psychometric functions for subject 3. The abscissa shows the difference in the identity of the first face relative to the second face in the current trial. Trials that fell in bins −12 and −6 had a first face that was more B-like relative to the second face, trials in the 0 bin had identical first and second faces, and trials in the +6 and +12 bin had a first face that was more A-like relative to the second face. The ordinate shows the proportion of first faces on the current trial that were chosen as being more A-like. The black curve consists of all trials with one-back first faces that were more “A”-like, and the gray dashed curve consists of all trials with one-back first faces that were more “B”-like. If the one-back first face positively pulled subjects’ perception of face identity, then there should be a leftward horizontal displacement of the black curve relative to the gray dashed curve, which is what we found for all subjects (see Figure S3A).

(C) Point of subject equality (PSE) difference between the black and gray dashed curve for each subject. Error bars are bootstrapped 95% confidence intervals, and p value is based on group permuted null distribution.

Trials for which the one-back first face was comparatively closer to face A along the morph continuum were labeled “A-previous,” and trials for which the one-back first face was comparatively more similar to face B were labeled “B-previous.” Each subject saw an equal number of A-previous and B-previous trials, but presentation order was shuffled. We fit separate logistic functions to A-previous and B-previous trials and calculated the slope and point of subject equality (PSE) for each logistic curve fit (Figures 2B and S3A; see also Supplemental Experimental Procedures). We also fit several lagged logistic regression models to the data to sequentially examine the influence of each previously seen face.

If the one-back first face pulled perception of the current trial’s first face more than the second face, then there should be a leftward displacement of the A-previous logistic curve relative to the B-previous curve (a significant difference in the PSE (b) parameter). A leftward shift in PSE would indicate that the presentation of a relatively more A-like one-back first face altered subjects’ perception such that the current first face actually appeared more A-like. However, if subjects’ identity perception was repelled or not influenced by previously viewed faces, we would observe a rightward displacement of the A-previous curve relative to the B-previous curve, or no displacement at all. We assessed the significance of each subject’s PSE shift using a permutation test to calculate a null distribution of PSE differences (see Supplemental Experimental Procedures).

We found a significant leftward shift of the A-previous curve (p < 0.001, permutated null distribution, n = 6) (Figure 2C), with 4 of the 5 subjects showing a significant shift (each p < 0.001, permuted null distribution; see Supplemental Experimental Procedures) (Figure S3A). For A-previous trials, subjects were more likely to perceive the current trial first face than the second face as A-like since the current and one-back first face were closer together in time and presented for twice as long. Average response time across all subjects was 435 ± 205 ms (n = 6); the one-back first face occurred ~5,685 ms prior to the current first face. We found no consistent slope differences between A-previous and B-previous trials, indicating no difference in sensitivity (permutation test; see Supplemental Experimental Procedures).

To determine whether the one-back second face also influenced subjects’ perception, we fit several lagged logistic regression models to each subject’s data and determined which model best predicted responses. Each successive model tested whether considering another face further back in the past explained significantly more variance in responses compared to a model without that face (see Supplemental Experimental Procedures). For 4 of the 5 subjects, the best-fitting lagged logistic regression model included both the one-back first and second face (least significant subject: F_2,816 = 11.682, p < 0.0001), indicating that both faces presented on the previous trial (more than 5 s ago) significantly biased perception in the present trial.

Experiment 3: Serial Dependence Occurs across Face Viewpoints

In experiment 3, our goal was to determine whether perceived identity is serially dependent across different face viewing angles, where basic image features change but identity remains stable. The procedure for experiment 3 was identical to that of experiment 2, except subjects were trained on two original neutral male face identities, face A and face B, within each of three possible viewpoints (frontal, left, right) (Figure S1C). Importantly, no two sequential trials contained the same viewpoint, but the first face and second face within a trial were always viewed from the same angle (Figure 3A).

Figure 3. Experiment 3 Trial Sequence and Results.

A) Trial sequence for the 2IFC task in experiment 3. We used grayscale image morphs based on two original neutral male faces across three different viewpoints (frontal, left, right), cropped by an oval to remove hairline (see Figure S1C). The trial sequence was identical to that of experiment 2, except both one-back trial faces were always of a different viewpoint relative to current trial faces. Faces in the figure are shown without added noise.

B) Example data from subject 3. The black curve consists of all trials with one-back first faces that were more “A”-like, and the gray dashed curve consists of all trials with one-back first faces that were more “B”-like. If the one-back first face positively pulls subjects’ perception of face identity, then there should be a leftward horizontal displacement of the black curve relative to the gray dashed curve, which is what we found for all subjects (see Figure S3B).

C) PSE difference for each subject. Error bars are bootstrapped 95% confidence intervals, and p value is based on group permuted null distribution.

Even across different viewpoints, subjects were more likely to perceive the current target face as A-like if the one-back target face was more A-like (Figures 3B and S3B). All subjects showed a leftward displacement of the A-previous curve relative to the B-previous curve (p < 0.001, permutated null distribution, n = 6), with 4 of the 6 subjects showing a significant shift (each p < 0.05, permuted null distribution) (Figures 3C and S3B). The average RT across subjects was 327 ± 174 ms; the one-back target face was seen ~5,577 ms prior to the current target face. There were no significant slope differences between A-previous and B-previous psychometric functions. We also simulated the effect of response hysteresis (responding the same way on successive trials) using the exact trial sequences presented and determined that response hysteresis could not cause this serial dependence (Figure S3C). These results show that serial dependence can operate on high-level identity representations rather than simply on low-level features.

Experiment 4: Serial Dependence across Face Viewpoints: Method of Adjustment

To extend the results of experiment 3, we presented sequential faces from different viewpoints and measured perceived face identity using an MOA task identical to that in experiment 1.We used a new set of female faces with two possible viewpoints (right- or left-facing profiles), drawn from a circular identity continuum to avoid any edge effects that might have been present in the one-dimensional stimulus arrays in experiment 3 (Figure S1D). Within a trial, the target and adjustment faces were always shown in the same viewpoint, but the viewpoint differed from one trial to the next (Figure 4A; see Supplemental Experimental Procedures).

Figure 4. Experiment 4 Trial Sequence and Results.

(A) Trial sequence for the MOA task in experiment 4. A circular morphed continuum of female face identities was created with two possible viewpoints (right- or left-facing profiles) (see Figure S1D). The identities shown here are similar to those in the experiment, with permission obtained for reproduction purposes. Randomly selected target faces were presented for 750 ms, followed by a 1,000 ms mask of black and white pixels. Subjects responded by matching the adjustment face to the target face. The one-back trial target face was always from a different viewpoint relative to the current trial target face, but the target and adjustment face had the same viewpoint.

(B) Example data from subject 2, with each dot showing performance on one trial. In units of face morph steps, the×axis is the shortest distance along the morph wheel between the current and one-back target face, and the y axis is the shortest distance along the morph wheel between the selected match face and target face. The DoG model fit is depicted as a solid line, and the running average is depicted as a dashed line.

(C) Half-amplitude of the serial dependence for each subject in experiment 4. Error bars are bootstrapped 95% confidence intervals, and p value is based on group permuted null distribution.

All subjects displayed a positive DoG half-amplitude, indicating that perceived face identity was significantly pulled toward faces presented one trial ago, even though the one-back face was always from a different viewpoint (p < 0.02, n = 5, group permuted null) (Figure 4C). The largest attraction of perceived identity occurred when the one-back target face was, on average, ±23.1 morph frames away from the current target face, which resulted in an average perceptual pull toward the one-back face of ±2.34 morph frames. Average RT across subjects was 4,508 ± 1,928 ms; the one-back face occurred ~7,758 ms prior to the current trial face. There was no significant difference in serial dependence amplitude between experiment 4 and experiment 1. The cross-viewpoint effects in experiment 4 therefore indicate that serial dependence occurs at the level of object-centered perceptual representations and does not depend on low-level stimulus features.

Discussion

Our experiments demonstrated that perceived face identity was pulled by identities encountered 5 or more seconds ago. This effect did not depend on subjects responding on the previous trial, and there was no perceptual pull on the current face if the previously seen face was sufficiently different. To determine whether serial dependence operates at the level of identity, our final two experiments manipulated the viewpoint of the sequentially presented faces. We found that identity perception is serially dependent across different face viewpoints, even without any prior associative perceptual training [7]. Some existing models, including Bayesian models of perceptual dependencies [2] and physiologically motivated population coding models [1, 8], can produce serially dependent effects, but importantly, our cross-viewpoint results take these further by demonstrating that the perceptual continuity field [1] can operate at the level of object-centered perceptual representations.

History Effects in Perception

Prior studies have shown that perceptual history can shape current perception, but none predict the existence of a continuity field [1] tuned for face identity. Priming, negative aftereffects, hysteresis, and other phenomena show a type of perceptual dependence on the recent past yet are distinct from our findings. Adaptation studies show that prior exposure to a variety of stimulus features [9–12] results in a stimulus-specific negative aftereffect or perceptual repulsion away from the adapting stimulus (for reviews see [13–15]). Our results, however, indicate a positive perceptual pull toward the recent past and are therefore not a result of known forms of adaptation. Since the serial dependence effect is restricted to faces seen 3–10 s ago, our experiments do not show a long-term positive aftereffect, as described in [16]. There is potentially a small contribution of memory confusion between the current and one-back face, if subjects sometimes mistakenly reported the one-back face in experiments 1 or 4, but additional control experiments show that memory confusions are unlikely and cannot account for our pattern of results (see Figure S2).

Furthermore, our results are not due to typical hysteresis of near-threshold stimuli [17–19] or stabilization of bistable stimuli [20–24], since our stimuli are randomly presented (thus disrupting hysteresis [19]) and are not bistable. Our results may be related to perceptual priming effects [25–29], but there are important differences. Priming generally occurs for reaction time [25–27] and, where relevant, can improve discriminability of primed stimuli [30]. Our results reveal a counterintuitive reduction in the discriminability of sequential stimuli due to serial dependence (Figure S3D). Nevertheless, the possible interaction between priming and serial dependence remains an open question.

Is Serial Dependence in Perception Inevitable?

There are three possible perceptual consequences of prior exposure to a visual stimulus: a negative aftereffect, complete independence in sequential perception, or positive serial dependence. Although negative face aftereffects can emerge following brief adaptation [31, 32], like they sometimes do for basic features [33, 34], serial dependence on the timescale of our experiments trumped any potential negative aftereffects.

Complete independence in moment-to-moment perceptual judgments would theoretically be the most bias-free and accurate. Given independent temporal noise and estimates of object identity, a temporal integration mechanism without serial dependence could better estimate the instantaneous state of the world compared to one that introduces a sequential dependence. Our results suggest that the visual system instead favors perceptual continuity over short periods of time, at the cost of introducing potential perceptual biases.

Conclusions

Our results are consistent with a continuity field [1] yet go significantly beyond related findings on perceptual history effects [1–3] by showing that the perception of faces, and not just features, is serially dependent. Thus, the continuity field is object selective, surviving changes in viewpoint, and reflects a mechanism that produces serially dependent perception of objects for the purpose of visual stability. By recycling previously perceived identities, the object-selective continuity field decreases the neural computations necessary for the identification of perceptually similar objects over time.