Stimulus timing-dependent plasticity in high-level vision

Summary

Humans are able to efficiently learn and remember complex visual patterns after only a few seconds of exposure[l]. At a cellular level, such learning is thought to involve changes in synaptic efficacy, which have been linked to the precise timing of action potentials relative to synaptic inputs[2-4]. Previous experiments have tapped into the timing of neural spiking events by using repeated asynchronous presentation of visual stimuli to induce changes in both the tuning properties of visual neurons and the perception of simple stimulus attributes[5, 6]. Here we used a similar approach to investigate potential mechanisms underlying the perceptual learning of face identity, a high-level stimulus property based on the spatial configuration of local features. Periods of stimulus pairing induced a systematic bias in face identity perception in a manner consistent with the predictions of spike timing-dependent plasticity. The perceptual shifts induced for face identity were tolerant to a two-fold change in stimulus size, suggesting that they reflected neuronal changes in non-retinotopic areas, and were more than twice as strong as the perceptual shifts induced for low-level visual features. These results support the idea that spike timing-dependent plasticity can rapidly adjust the neural encoding of high-level stimulus attributes [7-11].

Results

Plasticity in face identity perception

We began by testing whether a period of stimulation with asynchronously paired face images would lead to a systematic shift in the subsequent perception of identity. We took as a measure of identity perception the perceptual midpoint of a gradually varied stimulus set that was generated by morphing together two individual source faces (Face A and Face B; Fig 1 A). Each session of the experiment included the following stages: (1) a threshold block to establish the perceptual midpoint (that is, the morph level that was equally likely to be classified as Face A or Face B), (2) a pairing block to present repeated stimulus pairs, and (3) a test block to determine whether the perceptual midpoint had been shifted as a result of the pairing (see Supplemental Information for a complete description of the Experimental Procedures). The threshold block (Fig 1B) consisted of 110 trials in which a face stimulus appeared, and the subjects were required to report the identity of the face using a button box. In the pairing block (Fig 1C), subjects fixated on a small crosshair in the center of the screen while a series of 100 rapidly presented face pairs appeared over the fixation point. Each stimulus in the pair appeared for a single monitor refresh cycle (10 ms). Both the temporal order of the pair (A-B or B-A) and the stimulus onset asynchrony (SOA) were held constant throughout a single experimental session. In the test block (Fig 1D), the subjects again judged the identity of seven morph levels centered around the perceptual midpoint. Plasticity induced in the pairing block was measured by assessing what shift in the perceptual midpoint (if any) occurred between the initial threshold block and the final test block.

Figure 1.

Stimuli and sequence of main task blocks. A. Example of a series of morphed face stimuli. Five morph levels are shown, out of a total of 101 (0% to 100%) used in the experiment. B. Threshold block. Psychometric functions were obtained using the method of constant stimuli (110 trials) to characterize the subjects' baseline perception of face identity for each set of morphed faces. The psychometric function was then used to determine the perceptual midpoint (i.e., the morph level that was equally likely to be perceived as Face A or Face B). C. Pairing block. The subjects fixated while 100 rapidly flashed stimulus pairs appeared on the screen. The stimulus onset asynchrony (10 to 100 ms) and pair order (either A-B or B-A) were held constant within a single session of the experiment. D. Test block. The psychometric function was again sampled around the perceptual midpoint (14 trials) to assess the impact of the stimulus pairing on the perception of face identity. If the timing of the flashed A-B pairing matched the permissive window for spike timing-dependent plasticity, the subject's perceptual midpoint was predicted to shift towards face B.

Based on the physiological properties of spike timing-dependent plasticity[4-6], we predicted that face identity perception would be systematically altered by appropriately timed face pairings (Fig 1D). The rationale for this prediction is illustrated schematically in Fig 2 for the case of A-B pairings. During the pairing block, the successively presented face images will evoke temporally offset volleys of synaptic activity. Thus two volleys of excitatory postsynpatic potentials (EPSPs) will reach a face-responsive neuron immediately before and immediately after the neuron starts to fire spikes (Fig 2A). After repeated pairings of EPSPs and spikes that fall within the permissive window for spike timing-dependent plasticity, the synapses carrying input from the first stimulus will be strengthened and the synapses conveying input from the second stimulus will be weakened (Fig 2B). This change in the balance of synaptic weights will make the network more sensitive to Face A input and less sensitive to Face B. In a network of broadly-tuned face-selective neurons that encode face identity in the distribution of activity across the population[12, 13], the perceptual midpoint will correspond to the stimulus that evokes balanced activity in populations of A-selective and B-selective neurons (Fig 2C). After biasing the synaptic weights in the network, a stimulus with more Face B content will be needed to evoke balanced firing. Accordingly, the perceptual midpoint assessed psychophysically is predicted to shift towards Face B. Likewise, a shift in the opposite direction (towards Face A) will be observed after a block of B-A pairing within the permissive window of plasticity.

Figure 2.

Expected impact of spike timing-dependent plasticity on a network of broadly-tuned face-selective neurons. A. A pair of faces flashed in rapid succession (Face A followed by Face B) will evoke temporally offset EPSPs in synapses conveying input from Face A and Face B. If the time interval between the EPSPs and the first evoked spike falls within the permissive window for spike timing-dependent plasticity, the synapses activated before the spike will be strengthened and the synapses activated after the spike will be weakened. B. After repeated pairings, the asymmetric impact of spike timing-dependent plasticity will make the network of neurons more sensitive to Face A content and less sensitive to Face B content. C. The perceptual midpoint corresponds to stimulus level that evokes equal activity in both Face A-selective and Face B-selective neurons. After biasing the inputs to the network, more Face B content is needed to achieve balanced activity. Accordingly, the perceptual midpoint will shift towards B.

The impact of stimulus pairing blocks with 20 ms SOA on face identity perception is shown for a single subject in Fig 3A (average of 10 sessions). The psychometric functions obtained before stimulus pairing (black curve) are shown together with the shifts in perceptual midpoint assessed immediately after the block of stimulus pairings. In agreement with the predictions outlined above, the shift in the perceptual midpoint depended on the order of the pairing, with the curve shifted toward the face that was presented second in the pair, i.e. towards Face A after B-A pairings (blue curve) and towards Face B after A-B pairings (red curve). The magnitude of this effect was strongly timing-dependent, as revealed by the average perceptual shifts observed in sessions with SOAs ranging from -100 to +100 ms (30 subjects, Fig 3B). Plasticity peaked narrowly at ±20 ms SOA, and was considerably weaker or absent altogether in sessions with SOAs beyond ±60 ms. The overall shape of the timing dependency curve shown here is consonant with the hypothesis that face identity perception is susceptible to stimulus timing-dependent plasticity. Since the designations “face A” and “face B” were arbitrary, we pooled equivalent conditions together after flipping the x-axis for all the negative SOAs (Fig 3C). A significant perceptual shift was only observed after pairing with SOAs of 20 ms, 40 ms, and 60 ms (p < 0.001, single-condition bootstrap test).

Figure 3.

Effect of stimulus pairing on perception. A. Example of shifted face identity perception induced in a single subject that was studied intensively using only ±20 ms SOA pairings (10 sessions). Greater values on the x-axis indicate stimulus morph steps with greater Face B content. Error bars are 95% confidence intervals derived from Bernoulli distribution. B. Timing dependency profile of perceptual shift across a separate set of 30 subjects (average of 28 sessions and 14 subjects per condition). Error bars are 95% confidence intervals determined by Monte Carlo simulations. C. Comparison of plasticity in perception of high- and low-level visual features. Timing dependency curves show the perceptual shift induced in a grating orientation task (green) compared to a face identity task (black). Conditions with SOAs of equal magnitude and opposite sign were pooled after inverting the y-axis for the conditions with negative SOAs. The magnitudes of the shift are normalized to the slope of the psychometric functions that were measured before conditioning. D. Peak perceptual shifts for faces and gratings normalized as in (C). Error bars show confidence intervals from Monte Carlo simulation. Asterisks indicate significant effects (single-condition bootstrap test). Crossbar indicates significant difference between conditions (two-condition bootstrap test).

Comparing plasticity in perception of high- and low-level stimulus features

We repeated the same experimental design using the perceptual judgment of orientation, which, unlike face identity, could be made on the basis of information explicitly represented by neurons in early visual cortex (see Supplemental Information). Paired gratings produced a small but statistically significant shift in perceived line orientation at ±40 ms SOA (0.3°±0.16°, p < 0.01, single-condition bootsstrap test). The 0.1° shifts induced by the other SOAs tested (±20 and ±60 ms) were not significant (Fig 3C). This result agrees with a previous finding by Yao and Dan, who reported that stimulus pairings within a 40 ms time window induced a shift in perceived line orientation on the order of 0.2° [5]. To compare the plasticity effects induced by the two different classes of stimuli (faces vs. gratings) we scaled the perceptual shifts by the kernel sigma of the psychometric function, which is equivalent to applying a z-transform to the plasticity effects. When the most effective conditions for both faces and gratings were compared (20 ms and 40 ms SOAs respectively), pairing induced a 77% of sigma shift in face perception, which was significantly greater than the 38% shift induced in orientation perception (p = 0.001, two condition bootstrap test, Fig. 3D). This difference may reflect a general trend towards enhanced plasticity in higher-level visual areas[14, 15], and a correspondingly greater capacity for acquiring visual expertise in higher-level object vision.

Scale invariance of face plasticity

Although we observed the greatest plasticity in experiments that manipulated face perception, it does not necessarily follow that those effects were mediated by changes in face-selective cortical areas such as the fusiform gyrus[16]. An alternative possibility is that the observed changes reflect neural modification in early visual cortex, or at multiple processing stages in the visual system. If the effect does primarily reflect changes in face-responsive neurons in the temporal lobe, a strong prediction derives from the fact that neurons in monkey inferotemporal cortex tend to show broad tolerance to changes in stimulus size[17-19]. As a consequence of this property, a stimulus that drives spikes in one pool of neurons in inferotemporal cortex will also drive mostly the same pool of neurons when the stimulus size is doubled or halved. Accordingly, a perceptual change induced by pairing with stimuli of one size should still be observed when subsequently testing with stimuli of a different size. By contrast, in retinotopic visual areas such as V1, the same change in stimulus size will drive separate pools of neurons with largely non-overlapping receptive fields and therefore plasticity effects will show little scale invariance. To test this prediction, we conducted a variation on our standard experimental design in which there was a stimulus size mismatch between the pairing block as compared to the threshold and test blocks (SOA = 20 ms). In experimental sessions, the stimuli were either small (4 × 5 degrees of visual angle) or large (8 × 10 degrees). In control sessions the large and small stimulus sizes were held constant throughout all blocks (Fig 4A). The magnitude of the plasticity effect did not differ between sessions where the stimulus size was changed or held constant between pairing and testing blocks (p > 0.05, two-condition bootstrap test). This tolerance to changes in stimulus size supports the idea that plasticity in face perception is indeed driven by changes in higher-level visual areas.

Figure 4.

A. The magnitude of perceptual shift was not reduced by size changes between conditioning block and test block (two-condition bootstrap test). B. Pairing induced a stronger perceptual shift for novel stimuli than for familiar stimuli. C. Presenting Face B alone (in place of Face A - Face B pairings) did not induce a significant after-effect. All conventions as in Fig 3D.

Impact of stimulus novelty on plasticity

Previous studies have suggested that stimuli become less effective at driving changes in neural responses as the stimuli become more familiar[20]. Consistent with this notion, our pilot data suggested that the magnitude of conditioning was strongest for the first session, which prompted us to conduct the above experiments exclusively with novel face sets. In a separate set of experiments, we contrasted the capacity of novel and familiar stimuli to induce plasticity by exposing 11 subjects to five different stimulus sets repeatedly over six days (SOA = 20 ms). A significantly greater perceptual shift was induced on day one (when the subjects encountered the stimuli for the first time) than on subsequent days (Fig 4B, p < 0.05, two-condition bootstrap test). This result could indicate that novel stimuli are particularly susceptible to perceptual shifts, or alternatively that the capacity for plasticity is greater in “naive” synapses that were not exposed to previous pairings [21]. The reduced degree of plasticity is not due to saturation, as we alternated pair ordering (A-B on day one, then B-A on day two, etc). A comparison between the perceptual midpoints assessed in the threshold block on successive days revealed no evidence for a residual effect from the pairing block from the previous session (p > 0.05, two-sample t-test).

Can stimulus timing-dependent plasticity be explained by adaptation?

We considered the possibility that the effect of A-B pairing on perception can be accounted for by adaptation, rather than by spike timing-dependent plasticity as we propose. After-effects (perceptual shifts) are commonly observed following prolonged exposure to a single adapting stimulus; thus looking at face B for several seconds will induce a shift of the psychometric function towards B (and likewise for face A) [22, 23]. To test whether a series of 100 flashed faces is sufficient to induce a robust after-effect, we conducted a control experiment (N = 5 subjects, 25 sessions per condition) in which face B was presented by itself (Fig 4C). Conditioning with B alone resulted in a perceptual shift of 0.8%±1.3%. This effect was not significantly different from zero (p > 0.05, single-condition bootstrap test), and crucially, was significantly less than the shift induced by A-B conditioning with 20 ms SOA in the same control subjects (p < 0.05; two-condition bootstrap test). To further investigate the possible effects of adaptation combined with asymmetrical masking under the conditions used in our experimental paradigm, we again repeated the experiment using face images paired with phase-scrambled mask images of equal spatial frequency content (SOA = 20 ms). The magnitude of the perceptual shift measured in the backward masking condition (mask-face B; 2.1%±1.5%) was not greater than the shift induced by the forward masking (face A-mask; -3.8%±1.4%.) condition (p > 0.05, two-condition bootstrap test). These results indicate that the stimulus timing parameters used in the current study were not sufficient to induce a robust after-effect.

Discussion

We demonstrated that the perception of face identity is susceptible to stimulus timing-dependent plasticity. In qualitative terms at least, our results are consistent with a mechanism in which sequential visual stimuli give rise to sequential volleys of spikes that result in the modification of synaptic efficacy in visually selective neurons (see Fig 2). When paired gratings were used to induce shifts in perceived line orientation, the magnitude and timing dependency of the effects observed in the current study were similar to those reported previously[5]. Somewhat surprisingly, pairing with face images resulted in a broader effective timing window for inducing plasticity than the timing window observed for gratings. Although previous physiological studies conducted in vitro found a consistent relation between synaptic changes and the order of synaptic and spiking events, the effective timing window reported for different brain regions (and using different preparations) has varied from 10 ms to >100 ms [4]. In the absence of physiological data from extrastriate visual cortex obtained from awake behaving animals, it is therefore not possible to predict in quantitative terms what the shape of the timing dependency curve should be for plasticity in human subjects.

Our study focused on the timing required to elicit a perceptual shift, but did not track its duration. Results from single unit recording studies show that plasticity induced by paired stimulation may last on the order of several minutes to a few hours. In VI neurons in anesthetized cats, shifts in orientation selectivity driven by stimulus pairings similar to the ones employed here were shown to persist for 15 to 20 minutes [5]. In neurons in inferotemporal cortex of awake monkeys, changes in stimulus selectivity could be induced by pairing the peripheral view of one object with the foveal view of a different object tended to accumulate over time spans as long as 2.5 hours[9]. In human subjects, one study reported that paired pulses of transcranial magnetic stimulation applied to human motor areas induced an increase in the amplitude of evoked motor potentials that persisted for 30 minutes[24]. Taken together, these results support the idea that changes driven by stimulus timing can mediate long-term visual learning, although the impact of intervening visual stimulation on plasticity has yet to be systematically investigated.

Spike timing-dependent plasticity has been proposed as a mechanism for the emergence of invariant feature selectivity in ventral visual cortex[7, 8, 10, 11, 25]. Recent theoretical work exploring this idea has shown that the timing dependency characteristics demonstrated in physiological in vitro studies of synaptic plasticity are well-suited to underlie a “slow learning” algorithm[25], whereby patterns of visual stimulation that change continuously over time (for instance, while viewing a rotating object from gradually changing viewpoints) become associated together[8]. Experimental findings from psychophysical studies are in harmony with this computational framework. In studies of viewpoint invariance in face recognition, subjects were more likely to confuse two distinct individuals after viewing a face that was morphed between the two identities while the face was simultaneously subject to slow changes in either its three dimensional orientation[7, 26], or its illumination[26]. In a rather different study, Cox et al.[27] found that, after swapping the identity of a saccade target in mid-flight, subjects tended to associate the peripheral view of one object with the foveal view of the swapped object. In the current study, we tested a key prediction of the idea that spike timing-dependent plasticity contributes to perceptual learning. The temporal profile of the perceptual shifts we observed is consonant with the permissive window of synaptic changes revealed by physiological studies of stimulus timing-dependent plasticity [5, 6], although as noted above the timing kernel relevant to face perception remains to be measured experimentally.

Our findings extend previous work by demonstrating stimulus timing-dependent plasticity in the perception of complex stimulus attributes. Face identity cannot be derived from local structural features, but instead derives from global relational properties of the facial elements. We found not only that identity perception was influenced by the stimulus pairings, but also that the relative magnitude of the plasticity was substantially greater than the comparable effect induced for orientation perception. Given the need to learn and remember new face identities well into adulthood, this difference may reflect a particularly high degree of plasticity in brain regions that are selective for faces and objects. Consistent with this notion, several physiological studies examining perceptual learning have pointed to a trend towards more robust plasticity in higher-level visual areas than in earlier areas[14, 15, 31, 32]. A parallel conclusion can be drawn by comparing the behavioral aspects of high- and low-level visual learning. In contrast to the rapid time course characteristic of perceptual learning in object recognition, expertise in low-level feature perception tends to be more modest in size and slower to acquire[31, 32]. It is impossible to conclude on the basis of behavioral evidence alone that the changes we observed in face perception were driven by plasticity in ventral visual cortex, and conversely that the effects on perceived line orientation depend exclusively on early visual areas. The fact that the effects on face perception proved to be scale invariant supports this interpretation, but physiological recording studies will be necessary to test this idea directly.

Highlights.

1. Perception of face identity is susceptible to stimulus timing-dependent plasticity.

2. The impact of stimulus timing on face perception shows scale invariance.

3. Face perception is more strongly affected by stimulus timing than orientation perception.