Evidence for rapid localist plasticity in the ventral visual stream: The example of words

Abstract

Our recent work has shown that the Visual Word Form Area (VWFA) in left occipitotemporal cortex contains an orthographic lexicon based on neuronal representations highly selective for individual written real words (RW) and that learning novel words selectively increases neural specificity in the VWFA. But, how quickly does this change in neural tuning occur and how much training is required for new words to be codified in the VWFA? Here we present evidence that plasticity in the VWFA from broad to tight tuning can be obtained in a short time span, with no explicit training, and with comparatively few exposures, further strengthening the case for a highly plastic visual lexicon in the VWFA and for localist representations in the visual processing hierarchy.

Introduction

Understanding how the brain represents sensory stimuli is key to understanding how the brain computes. Localist representations, defined here as neuronal representations in which a set of neurons selectively encodes one particular, meaningful stimulus, are a computationally simple way to represent stimuli, in particular behaviorally important stimuli that have to be recognized from similar foils at a high level of accuracy. Prior experiments (Logothetis & Pauls, 1995) have provided evidence for such neurons in the inferotemporal cortex, IT, of monkeys trained to recognize a small number of “paperclip” objects. In humans, the object class of real words has proved to be a fruitful domain with which to study the existence of localist object representations in the brain (see also the Preface to this special issue). Reading written words is an important cognitive skill that, given the cultural recency of reading and the variability of lexica across languages, arguably needs to depend on neural representations that are acquired through experience with written words (Binder, Medler, Westbury, Liebenthal, & Buchanan, 2006; Dehaene, Cohen, Sigman, & Vinckier, 2005; Vinckier et al., 2007). Recent neuroimaging studies (Dehaene et al., 2004; Vinckier et al., 2007) have provided evidence for a hierarchical organization in the ventral visual pathway (Ungerleider & Haxby, 1994) for the visual word form, leading to the proposal that running posterior to anterior along this pathway, neurons are tuned to increasingly complex word features, viz. from oriented bars, to letters, to bigrams, and finally quadragrams (Dehaene et al., 2005; Vinckier et al., 2007). At the top of this visual hierarchy, numerous human neuroimaging studies have identified an area in the left occipitotemporal cortex that appears to play a key role in whole word reading, termed the “visual word form area” (VWFA) (Baker et al., 2007; Gaillard et al., 2006; Kronbichler et al., 2004; Schurz et al., 2010; Vinckier et al., 2007; Yeatman, Rauschecker, & Wandell, 2012).

The nature of the representation in the VWFA has been the subject of some debate. Recently, we used fMRI rapid adaptation techniques (fMRI-RA) to examine the nature of word representation in the VWFA by systematically altering the visual word form and lexicality between pairs of stimuli (Glezer, Jiang, & Riesenhuber, 2009) to test the hypothesis that the VWFA contained neurons tuned to whole words (localist representation) rather than prelexical letter combinations, in line with theories of reading (e.g., the dual-route cascaded, DRC, model (Coltheart, 2004)) that have argued for the existence of a neural representation for whole real words (i.e., an orthographic lexicon). In fMRI-RA experiments, two stimuli are presented sequentially in each trial. If the two stimuli activate overlapping neuronal populations, neurons that are activated by both stimuli show adaptation on the second presentation (also referred to as repetition-suppression) (Dehaene et al., 2001, 2004; Grill-Spector, Henson, & Martin, 2006). Due to the slow time course of the BOLD signal, the BOLD response recorded in each trial reflects the neural responses to both words in each pair. This BOLD-contrast response to the pair is taken to reflect similarity of the neuronal activation patterns corresponding to the two individual stimuli, with the lowest response for two stimuli activating identical neuronal populations, and maximum signal if the two stimuli activate disjoint groups of neurons. In our experiments, we presented words in prime/target pairs in each trial, and examined three conditions: 1) “same” (S), in which the same stimulus was presented twice (as prime and target) in each trial, 2) “1L”, in which the prime and target differed by one letter and, 3) “different” (D), in which the target shared no letters with the prime (Fig. 1).

Figure 1.

Experimental paradigm. Examples of the different stimulus conditions for the pseudoword (PW) pairs.

For both real words (RW) and orthographically legal non-words (pseudowords, PW) we predicted the lowest signal for the S condition, as the two stimuli were identical and would therefore repeatedly activate the same neural populations, causing maximum adaptation (Grill-Spector et al., 2006) (Fig. 2). Likewise, we predicted the least amount of adaptation for the D condition because two stimuli that share no letters should not activate overlapping populations of neurons, regardless of whether neurons in the VWFA are tuned to whole words or to prelexical letter combinations. The crucial prediction that differentiates a whole-word representation from a part-word or sublexical representation is the response of the 1L condition when comparing the PW pairs and the RW pairs: If neurons in the VWFA are tuned to sublexical features, one would predict a gradual response increase with increasing dissimilarity for both the RW pairs and the PW pairs, as the S condition has maximal sublexical unit overlap, the 1L condition has partial sublexical unit overlap and D has no sublexical overlap. In contrast, if neurons in the VWFA are tightly tuned to whole real words, then two similar but non-identical real words should have minimal neural overlap and therefore no adaptation should occur. This would lead to a total activation in the RW pairs for the 1L condition similar to that caused by the two completely different words in the D condition. In addition, as we have proposed in previous work (Glezer et al., 2009) selective tuning for specific real words would be the result of visual experience with real words and the requirement to discriminate highly visually similar RW (such as “farm” and “form”), producing RW-tuned neurons that show high selectivity to a specific RW, but show little response to other RW, even if they are orthographically similar. In contrast, lack of experience with PW and the lack of a need to discriminate RW from PW (such as “farm” from “tarm”) during normal reading would lead to a lower degree of experience-driven response differentiation of specific RW-tuned neurons to orthographically similar PW. These RW-tuned neurons then not only respond strongly to their preferred RW but also respond to a lesser degree to similar PW, thus accounting for VWFA responsiveness also to PW (Fiez, Balota, Raichle, & Petersen, 1999; Paulesu et al., 2000). For example, neurons tuned to the word “farm” would respond strongly to the word “farm” but very little to the real words “form”, “firm”, “harm”, etc. In contrast, the pseudoword “tarm” would cause low-level activation of neurons tuned to orthographically similar real words (“farm”, “harm”, “term”, etc.), leading to a total signal for PWs that might be equal to or greater than that evoked by RWs. Therefore, assuming that VWFA responsiveness to pseudowords is caused by low-level activation of RW-tuned neurons to orthographically similar PWs, we should see a gradual BOLD response increase with increasing dissimilarity (from S to 1L to D) in the PW pairs, due to the lower degree of selectivity of the RW-tuned neurons for pseudowords.

Figure 2.

Predictions for fMRI-RA response to the different conditions for whole-word (A) and part-word or sublexical (B) representations.

Our fMRI-RA results ((Glezer et al., 2009), replicated in (Glezer et al., 2015)) demonstrated that the VWFA showed high selectivity for whole real words, yet broader tuning to pseudowords. In particular, our results provided evidence that the VWFA contains an “orthographic lexicon” based on neurons highly tuned to familiar words: Two real words that differ by just one letter (1L condition) caused as little adaptation as two words that shared no letters at all, suggesting that a high degree of specificity of tuning exists in the VWFA such that familiar words are represented by disjoint groups of neurons. In contrast, for PW increasing sublexical dissimilarity was associated with a gradual increase in responses in the fMRI-RA experiment, with responses in the 1L condition higher than in the S condition, but in contrast to the RW conditions, significantly lower than in the D condition. These results are compatible with the notion that extensive visual experience with RW (and the need to discriminate between these words), but not unfamiliar PW, results in experience-driven refinement of neuronal tuning to create a visual lexicon.

A direct test of this visual lexicon theory would be to demonstrate that learning to visually recognize novel pseudowords induces RW-like selectivity profiles specific to those words after learning. In (Glezer et al., 2015), we tested this hypothesis by training subjects to recognize novel pseudowords and used fMRI rapid adaptation to compare neural selectivity with RWs, untrained PWs (UTPWs), and trained PWs (TPWs). Before training, PWs elicited broadly tuned responses as in the 2009 study, whereas responses to RWs indicated tight tuning, again as in the 2009 study. After training, TPW responses resembled those of RWs, whereas UTPWs continued to show broad tuning. This change in selectivity was specific to the VWFA. Therefore, word learning appears to selectively increase neuronal specificity for the new words in the VWFA, thereby adding these words to the brain’s visual dictionary.

That study (Glezer et al., 2015) involved training participants on sets of novel PW over the course of 2–3 weeks (with an average of 7.5 training sessions), for an average of 49 exposures per word. Yet, there is behavioral evidence that novel PW become codified like real words within 1 session (with no intervening sleep) (Bowers, Davis, & Hanley, 2005; Coutance & Thompson-Schill, 2014; Shtyrov, Nikulin, & Pulvermüller, 2010). These data suggest faster plasticity of word representations than predicted by the influential complementary learning systems theory (McClelland, McNaughton, & O’Reilly, 1995), which proposes that novel word learning is subserved by two neural systems, the hippocampus which is involved in rapidly acquiring sparse representations, and the neocortex which is involved in a slower, more distributed fashion. One key component to this model is the importance of sleep and time on strengthening and solidifying neocortical representations. A recent study by Davis et al. (Davis, Maria, Betta, Macdonald, & Gaskell, 2010) supports this model by showing two stages in word learning: rapid familiarization which is subserved by the hippocampus and consolidation, which occurs in the cortex after a night of sleep.

Here we reanalyzed data previously presented in (Glezer et al., 2009) and provide evidence that plasticity in the VWFA for novel pseudowords to real word-like selectivity can be obtained in a short time span, with no explicit training just exposure to new words, and with comparatively few exposures, further strengthening the case for a highly plastic visual lexicon in the VWFA.

Experimental Procedures

Participants

In this analysis we reanalyzed data previously presented in (Glezer et al., 2009) and included subjects from Experiment 2 (Group 1; n=12 (8 female)) and 3 (Group 2; n=10 (6 female)) from (Glezer et al., 2009) as these experiments included PW. All subjects were right-handed normal adults who were native English speakers (aged 18–32). Experimental procedures were approved by Georgetown University’s Institutional Review Board, and written informed consent was obtained from all subjects prior to the experiment.

Stimuli

For each group we included in the analysis the conditions that included PW in both prime and target positions, which was a subset of the conditions presented in the original experiments. In the original experiments, Group 1 and Group 2 also had stimulus pairs that included RW. The stimuli used for the current study were termed “PW triplets” and consisted of three different types of primes (conditions: S, 1L, and D). Forty-seven, 3–6 letter PW triplets were generated using the ARC Nonword Database (Rastle, Harrington, & Coltheart, 2002). These PW triplet lists were matched for length, bigram and trigram (where applicable) frequency, and neighborhood size. A final condition was created for the oddball task (see below) and included the same number of trials as all other conditions. None of these oddball trials were included in the analysis.

Tasks

For both groups we used an oddball detection task. We did this to engage subjects’ attention yet avoid potential task-related confounds of the BOLD-contrast response to the conditions of interest (Grady et al., 1996; Sunaert, Van Hecke, Marchal, & Orban, 2000). For Group 1, subjects were asked to press a button whenever a word appeared that referred to either a fruit or vegetable. These “oddball” stimuli were created by selecting common fruit and vegetable words that were 3–6 letters in length. In Group 2, subjects were asked to press a button every time they saw the sequential letters “xyz” or “abc”. These “oddball” stimuli were created by randomly replacing 3 sequential letters at the beginning, middle, or end of either the prime or the target stimuli. Prior to scanning, subjects participated in practice trials familiarizing them with the task. None of the stimuli presented in the practice were presented in the experiments.

All stimuli were rendered in Courier font (36 point size, 100 ppi), average letter size ¼ × ¼ inch (25 × 25 pixels), for an approximate size of 0.5 degrees of visual angle per letter in the scanner.

Functional localizer scans

As in (Glezer et al., 2009), for both groups, separate localizer scans were conducted to identify the VWFA in each subject individually. Using a block design, EPI images from two functional localizer scans were collected. Participants passively viewed blocks of images of written words (high frequency nouns, >50 per million, different from those used in the event-related scans), scrambled words, faces, and objects. Each block lasted 20.4s (stimuli were displayed for 500ms and were separated by a 100ms blank interval), and stimulus blocks were separated by a 10.2s fixation block. Each run consisted of 2 blocks of each group (words, scrambled words, faces, objects) and 8 fixation blocks. The face and object images used in the localizer scans were purchased from http://www.hemera.com and post-processed using programs written in MATLAB (The Mathworks, MA) to eliminate background variations and to adjust image size, luminance, and contrast. The final size of all images was scaled to 200 × 200 pixels. Word stimuli were chosen using the CELEX database (Baayen, Piepenbrock, & Gulikers, 1995). Scrambled images of words were generated by scrambling the word images with a tile size of 4 pixels.

Rapid event-related (ER) scans

To probe the effects of varying orthographic similarity and lexicality on BOLD-contrast response in the VWFA, MRI images from four (Group 1) and six (Group 2) ER scans were collected. For Group 1, each run lasted 448.8s and had two 10.2s fixation periods, one at the beginning and the other at the end. Between the two fixation periods, a total of 110 trials were presented to participants at a rate of one every 4080ms. During each trial, the two stimuli (prime/target) were displayed sequentially (timing: fixation for 1000ms, prime for 33ms, blank for 337ms, target for 500ms, and blank for 2210ms). For Group 2, each run lasted 473.28s and had two 10.2s fixation periods one at the beginning and the other at the end. Between the two fixation periods, a total of 116 trials were presented to participants at a rate of one every 4080ms. During each trial, the two stimuli (prime/target) were displayed sequentially (timing: prime for 300ms, blank for 400ms, target for 300ms, and blank for 3080ms). For both groups the number of repetitions of each word stimulus was counterbalanced across all conditions to control for long-lag priming effects (Henson, Shallice, & Dolan, 2000). Trial order and timing was adjusted using M-sequences (Buračas & Boynton, 2002). Participants were asked to watch all stimuli and respond to instances of the “oddball” stimuli by pressing a button with the right hand. The stimuli of both localizer and ER scans were presented using E-Prime software (http://www.pstnet.com/products/e-prime/), back-projected on a translucent screen located at the rear of the scanner, and viewed by participants through a mirror mounted on the head coil.

MRI acquisition

All MRI data were acquired at Georgetown University’s Center for Functional and Molecular Imaging using an echo-planar imaging (EPI) sequence on a 3 Tesla Siemens Trio scanner. For Group 1 an 8-channel head coil was used (Flip angle = 90°, TR = 2040ms, TE = 30ms, FOV = 205 mm, 64×64 matrix). For Group 2 a 12-channel head coil was used (Flip angle = 90°, TR = 2040ms, TE = 29ms, FOV = 205 mm, 64×64 matrix). For all experiments, 35 interleaved axial slices (thickness = 4.0 mm, no gap; in-plane resolution = 3.2×3.2 mm²) were acquired. At the end, three-dimensional T1-weighted MPRAGE images (resolution 1×1×1 mm³) were acquired from each subject.

MRI data analysis

All preprocessing and most statistical analyses were done using the SPM2 software package (http://www.fil.ion.ucl.ac.uk/spm/software/spm2/). After discarding the first five acquisitions of each run, the EPI images were temporally corrected to the middle slice (for event-related scans only), spatially realigned, resliced to 2 × 2 × 2 mm³ and normalized to a standard MNI reference brain in Talairach space. Images were then smoothed with an isotropic 6mm Gaussian kernel. As in our previous studies (Glezer & Riesenhuber, 2013; Glezer et al., 2015; Glezer et al., 2009), the VWFA regions were identified for each individual subject independently with the data from the localizer scans. We first modeled the hemodynamic activity for each condition (word, scrambled word, face, object, and fixation) in the localizer scans with the standard canonical hemodynamic response function, then identified a word-selective ROI with the contrast of word versus fixation (at least p<0.00001, uncorrected) masked by the contrast of word versus scrambled word (at least p<0.05, uncorrected). This contrast typically resulted in only 1–2 foci in the left ventral occipitotemporal cortex (p<0.05, corrected). ROIs were selected by identifying in each subject the most anterior cluster that was significant at the corrected cluster-level of at least p<0.05 in the ventral occipitotemporal cortex (specifically, the occipitotemporal sulcus/fusiform gyrus region) with at least 10 but not more than 100 contiguous voxels in a location closest to the published location of the VWFA, approximate Talairach coordinates −43 −54 −12 ±5 (Cohen et al., 2002; Kronbichler et al., 2004). In the ER scans, after applying global scaling following verification that there were no correlations of the global signal with the experimental conditions (Aguirre, Zarahn, & D’Esposito, 1998)and removing low frequency temporal noise from the EPI runs with a high pass filter (1/128Hz), we modeled fMRI responses with a design matrix comprising the onset of trial types and movement parameters as regressors using a standard canonical hemodynamic response function (HRF), then extracted the mean percent signal change of the VWFA ROIs for each subject with the MarsBar toolbox (Brett, Anton, Valabregue, & Poline, 2002).

Results

As in our prior studies, we independently localized the VWFA in each subject (average Talairach coordinates Group 1: -39±4 -57±9 -11±6 and Group 2: -40±3 -58±7 -11±6). For the analysis, we pooled VWFA response data from Groups 1 and 2. Figure 3A shows results for the S, 1L, and D conditions in the pooled data set, indicating broad neuronal tuning for PW, as for the individual Group 1 and Group 2 data sets in (Glezer et al., 2009), with graded adaptation (S<1L, p = 0.001, and 1L<D, p=0.02, paired t-test). We then separately analyzed the first and second half of the scans. Over the entire scanning session, Group 1 subjects saw the primes on average 3 times (SD = 0.72, range 2–5) and the targets on average 8 times (SD= 1.12, range 6–10), Group 2 subjects saw the primes on average 2 times (SD = 0.48, range 2–4) and the targets on average 7 times (SD= .76, range 6–8). All words and pairs were evenly distributed over the 6 fMRI runs and therefore approximately ½ of the number of presentations occurred in the first 3 runs and ½ occurred in the last 3 runs. Interestingly, our results show a change in neuronal tuning from the first half to the second half (Fig. 3B). In the first half, we see a graded adaptation pattern similar to that seen with novel words (Glezer et al., 2015; Glezer et al., 2009) with S < 1L < D (S vs. D and 1l, p < 0.006, and 1L vs. D, p = 0.0005, paired t-test) whereas in the second half we see neural tuning similar to what is seen with real words (Glezer et al., 2015; Glezer et al., 2009) with S < 1L = D (S vs. D and 1l, p < 0.007; and 1L vs. D, p = 0.81 paired t-test).

Figure 3.

Plots of mean percent signal change in relation to orthographic similarity in the fMRI-RA scans. (A) Overall results from the full scan. (B) Results from 1^st and 2^nd halves for PW pairs. Error bars represent within-subject SEM. Significance levels: * < 0.05, ** < 0.01, *** < 0.001, **** < 0.0001, paired t-tests.

A 2×2 repeated-measures ANOVA with two factors, half (1^st half vs. 2^nd half) and experimental condition (1L vs. D, as we predicted adaptation to differ between these two conditions for 1st vs. 2^nd half, with D>1L for 1^st half, but D=1L for 2^nd half), revealed a significant interaction between half and conditions (F(1,21)=13.821, p=0.001). Note that these differences were not the result of general response habituation, as there was no reduction in overall amplitude from 1^st to 2^nd half (1^st half vs. 2^nd half S: p = 0.34, 1L: p = 0.09, D: p = 0.81, paired t-test). Rather, the pattern of release from adaptation changed, compatible with the sharpening of neuronal tuning from a broadly tuned PW-like response profile to a word-specific representation.

Discussion

We have shown previously that the VWFA holds a finely tuned orthographic representation that is not modulated by semantics or phonology (Glezer et al., 2009; Glezer et al., 2016), supporting the DRC model of reading (Coltheart, Rastle, Perry, Langdon, & Ziegler, 2001; Coltheart & Rastle, 1994) and other models that posit whole word lexical representations in which orthographic forms are stored as distinct units. In contrast, our results provide less support for models of reading such as the Triangle model (Seidenberg & McClelland, 1989) that have argued against the existence of an orthographic lexicon.

Further supporting the notion of an orthographic lexicon in the VWFA, we also showed previously that word-selective plasticity in the VWFA could be obtained by training subjects on an implicit learning task involving exposure to novel pseudowords over several sessions (Glezer et al., 2015) and dozens of exposures to each novel word. We here show that similar levels of plasticity can already be obtained within one scanning session with just 5–6 exposures while subjects performed an oddball detection task, i.e., were not actively trying to remember the stimuli. Interestingly, these neuroimaging data demonstrating rapid addition of novel words to the brain’s visual lexicon fit well with behavioral data (Salasoo, Shiffrin, & Feustel, 1985) that have shown that novel pseudowords become codified like real words with just five exposures. One question that arises is whether this rapid, word-selective plasticity we observe requires the involvement of other brain areas, e.g., replay from hippocampus to cortex? The timescale of the effects observed (within a single scan session, while subjects were engaged in an oddball task) make this less likely, suggesting a cortical learning mechanism to store highly specific sensory “memories”. In this context, it is interesting to relate our results to the complementary learning systems model (McClelland et al., 1995) and recent work by Davis and colleagues (Davis & Gaskell, 2009; Davis et al., 2010). Specifically, Davis et al., proposed a dual-process model for novel word learning, the first being rapid familiarization with novel words, which is then followed by consolidation (presumably requiring a night of sleep). Based on their results showing more activation in the hippocampus for unfamiliar novel words in combination with equivalent activation in cortex for unconsolidated and untrained words (and both being higher than existing words), Davis et al. suggest that the first stage is subserved by the hippocampus and the second stage by cortex. Our data help to elucidate this process by offering data (using a technique that can probe neural specificity at a much finer level than the average BOLD signal) showing that within this first stage of the dual process suggested by Davis et al., where rapid learning/familiarization is taking place, there is in fact already a change in neural tuning in cortex that cannot be detected via the average BOLD signal. While we did not collect behavioral measures of lexicalization in our study, other training studies have shown lexicalization effects for trained words in the absence of intervening sleep (Bowers et al., 2005; Coutance and Thompson-Schill, 2014). We therefore propose that novel word learning leads to rapid plasticity in the VWFA that does not require intervening sleep. The second stage discussed by Davis et al. (Davis et al., 2010) might involve further refinement or attentional adjustments that bring the overall average BOLD level to the same level as existing words. It will be interesting in the future to examine the additive contributions of sleep consolidation to word learning.

Our data raise further intriguing questions for future studies: What are the capacity limits of this process? What are its attentional requirements? In our study, subjects were just performing an oddball task, which does not require the use of a phonological loop, previously thought to be important for word learning (Baddeley, Gathercole, & Papagno, 1998). Other studies (Laine, Polonyi, & Abari, 2013) have shown word learning effects even with a single exposure. Future studies with more sensitive imaging paradigms or larger word sets are needed to test how many exposures are necessary to obtain RW-like selectivity for novel PW, and whether similar levels of plasticity can be obtained in other brain areas.