Novel domain formation reveals proto-architecture in inferotemporal cortex

Abstract

Primate inferotemporal cortex is subdivided into domains for biologically important categories, like faces, bodies, and scenes, as well as domains for culturally entrained categories, like text or buildings. These domains are in stereotyped locations in most humans and monkeys. To ask what determines the location of such domains, we intensively trained 7 juvenile monkeys to recognize 3 distinct sets of shapes. After training, the monkeys developed regions that were selectively responsive to each trained set. The location of each specialization was similar across monkeys, despite differences in training order. This indicates that the location of training effects does not depend on function or expertise, but rather some kind of proto-organization. We explore the possibility that this proto-organization is retinotopic or shape-based.

In normal adult humans and macaques, inferotemporal cortex (IT) is functionally organized into domains that are specialized for different biologically important object categories, like faces, objects, bodies, and scenes. This organization must be a consequence of visual experience interacting with innate programs. There are two broad themes in Learning Theory that address the mechanisms of how representations arise in the brain: Nativism, which stresses innate factors, and Empiricism, which stresses the influence of experience. The reproducible location of different category-selective domains in humans and macaques^1,2 suggests that some aspects of IT category organization must be innate. However the effects of early experience on face recognition³, changes in fMRI domains during development^4,5, the existence of a Visual Word Form area⁶, the effects of expertise⁷, and our recent finding that novel specializations appear in IT as a consequence of intensive early training⁸ indicate that experience must also be important in the formation or refinement of category-selective domains in IT. What are the restrictions, or initial constraints, on the organization of IT, and how does experience manifest its effect on this organization?

Most normally educated humans exhibit a domain for written text⁶, and this domain is in approximately the same location in most people, irrespective of the language they read. It is unlikely that a domain dedicated to processing text evolved by natural selection, given how recently literacy has been prevalent, so there must be some other explanation for the stereotyped localization of the Visual Word Form Area. “Cultural recycling” theory proposes that this stereotyped localization is due to exaptation⁹ of cortical regions that without education would normally process the kinds of line junctions that are also common in objects and scenes and may be critical for figure/ground segregation¹⁰. The “Connectionist” model proposes that the stereotyped localization arises because processing text requires both visual and linguistic connectivity, and that connectivity is innate¹¹. “Constructivist” theory proposes that the stereotyped localization is a consequence of the timing of experience or training interacting with a programmed developmental trajectory¹². Lastly “Expertise” theory proposes that the stereotyped location is a function of the level of skill or categorization required for reading¹³. We recently reported⁸ that macaque monkeys intensively trained as juveniles to recognize human writing symbols develop specialized regions selectively responsive to the trained symbols, compared to visually similar, but untrained, symbols. This novel domain formation was observed in approximately the same location in all animals that were trained as juveniles, but was absent in monkeys identically trained, but as adults. Here we extended this paradigm using 3 distinct symbol sets and varied training schedules with the goal of disentangling the effects of expertise, function, and order of learning on the localization of training-induced effects in IT. To our surprise we discovered no evidence that any of these factors matters for the localization of the training induced domains, leading us to explore other possible factors, including shape and eccentricity.

Results

Part 1: Effects of training on IT organization

Seven juvenile (1–5 years of age) male macaque monkeys were intensively trained to discriminate 3 distinct sets of 26 shapes each (Fig. 1, top). The ‘Helvetica’ symbol set consisted of standard digits and letters; the ‘Tetris’ set consisted of patterns made by filling 4–5 squares in a 3×3 grid; the Cartoon Face symbol set was derived from the 19 parameter cartoon face set that we previously used to study face tuning in the middle face patch¹⁴. Each Cartoon Face symbol had one parameter set to one of the extreme value used in this previous study, and the other 18 parameters set to neutral value. The monkeys were trained using a touch-screen mounted in their home cage to associate each of the 26 shapes in each set with a particular reward value of 0 to 25 drops of liquid. In each trial they were presented with two symbols, and they were rewarded with a number of drops corresponding to the symbol on whichever side they touched first. They were rewarded no matter which side they chose (except for value zero), but they most often chose the side with the symbol representing the larger reward. The monkeys learned the symbols in a given set in increasing order, until they reached criterion performance on all 26 symbols in the set. It took 6–8 months for them to learn a set, and they were given at least 1 further month of practice once they mastered that set. Different monkeys learned the Helvetica or the Tetris symbol set first. Figure 1 shows the 3 symbol sets and the performance of each monkey on each symbol set. Figure S1 shows the timeline for all testing and scanning. Some monkeys were better than others at learning these symbols, but all 3 symbol sets were learned to approximately the same level of accuracy (no significant differences between average accuracies with a two tailed t-test). The order of learning did not significantly affect final performance.

Fig. 1.

Symbol set learning: (top) The 3 different symbol sets: Helvetica, Tetris, and Cartoon face. Each symbol in each set represents, in order, 0 to 25 drops of liquid reward. At the far right of each symbol set is an image average of all the symbols in each set. (bottom) Percent larger choices averaged over 1 month of daily testing for each monkey (horizontal axis) for each symbol set (indicated by color); chance = 50%. Numerals 1–3 indicate the order in which the 3 symbol sets were learned by each monkey. To the right are shown the % larger choices±sem for each symbol set averaged over all monkeys who learned each set, and the average % larger choices±sem for the first, second, and third learned sets. We found a negative correlation between the size of each symbol set patch and each monkey’s performance on that symbol set, but this correlation was not significant [Pearson’s Linear correlation coefficient r= −0.3828; p = 0.1057]. We found no correlation between the average significance value of a particular patch and the monkey’s performance [Pearson’s Linear correlation coefficient r= 0.0196; p = 0.9366)].

We asked, first, whether learning different symbol sets would result in novel domain formation, as we previously found for juveniles learning the Helvetica symbol set⁸; second, if the location of such training-induced changes would be the same for different symbol sets; and third, whether order of learning within the juvenile period would have any effect on location or size of such artificial domains, as predicted by Constructivist theory¹². To do this we performed alert functional MRI on each monkey before and after learning each symbol set using a non-invasive helmet restraint system as described previously⁸. The alert monkeys sat comfortably in a sphinx position and passively viewed blocks of images. Blocks of the trained symbol set (omitting the three lowest value symbols of each set) were presented in alternation with blocks of visually similar but untrained shapes (Fig. S2) as controls for the Helvetica and Tetris symbol sets, and monkey faces as controls for the Cartoon Face set. Before training, none of the monkeys showed any regions with significantly different responsiveness to any of the symbol sets, contrasted with the appropriate controls (significance criterion: p<0.002; cluster size>=31 voxels). After training with each symbol set, each monkey showed patches in inferotemporal cortex that were significantly more responsive to the trained set than to its control set (Fig. 2 and Figs. S3&S4, for results from individual monkeys). The control and test stimuli did not result in differential activations in V1 (Fig. 2 & ref.⁸) either before or after training.

Fig. 2.

Effects of training on functional organization of IT. (a–c) Each panel shows overlaid activations from each monkey (collapsed across hemispheres; see Methods) aligned onto a lateral and a ventral view of a standard macaque brain^47,48; the major landmarks of the superior temporal sulcus (STS) and V1 are indicated. Each of the overlaid patches represents a region from a single monkey that was significantly more active to one set of images than to control blocks; the patches are transparent so the overlap among different monkeys can be seen, as indicated by the scale. Activations for individual monkeys are shown in Figs. S3–S10. (a) Overlaid significant activations after training to Helvetica (blue) and Tetris (green) from monkeys who learned Helvetica before Tetris (B1, R2, G2, & G1). (b) Overlaid activations after training to Helvetica (blue) and Tetris (green) from the monkeys who learned Tetris before Helvetica (B2 & Y1). (c) Overlaid activations to monkey faces > Tetris AND monkey faces>Helvetica (red) before Cartoon Face training and to Cartoon Faces>monkey faces (cyan) after Cartoon face training from all the monkeys who learned Cartoon Faces (B1, B2, R2, G2, Y1 & Y2). (d) Pre-training vs Post-training responsiveness. Average percent signal change for each image category before and after training was normalized to the response in V1 to that same image set from the same data. Monkey face responsiveness was calculated before and after Cartoon face training. Bars represent mean ± sem of values averaged over monkeys; single asterisks indicate means that differ at p<0.05; double asterisks indicate p<0.01. (e) Average percent signal change in opercular V1 to the same image sets, from the same scan sessions, normalized to the maximum V1 activation among image categories.

A 2×2 ANOVA for trained set vs control × pre- vs. post-training was calculated for each trained-set ROI. Main effects of trained vs control were found in all the trained-symbol patches [Helvetica Patch [F(1,4) = 3.55, p <0.05; Tetris Patch [F(1,4) = 8.6, p<0.01, Cartoon Face patch: F(1,4)=2.17, p<0.05]. Main effects of training were also found in all the patches [Helvetica Patch [F(1,1) = 79.42, p<0.01; Tetris Patch [F(1,1) = 2.7, p <0.05, Cartoon Face patch: F(1,1)=19.29, p<0.01]. Critically, a robust interaction between trained vs control and pre- vs post-training was observed in all ROIs [Helvetica Patch [F(1,4) = 9.88, p<0.01; Tetris Patch [F(1,4) = 6.91, p <0.01, Cartoon Face patch: F(1,4)=5.07, p<0.01]. Hypothesis-driven tests indicated that the all of the training-induced patches were significantly more activated by their trained stimulus category compared to controls after training [Helvetica patch: t(12) = 2.188, p< 0.05; Tetris patch: t(12) = 2.74, p< 0.05; Cartoon Face Patch t(12)= −3.97 p<0.001], but none of the ROIs showed significant differences between their preferred stimulus category and controls prior to training (all, p > 0.05). The pattern of results in the Cartoon Face and Helvetica regions showed a larger response to the trained stimuli post- compared to pre-training [Cartoon Face Patch t(12)= −4.97 p<0.001; Helvetica Patch t(12)= −3.10 p<0.01], and no change in response to control stimuli. The Tetris patch developed a post-training selectivity to Tetris via reduced responsiveness to controls after training [t(12)= −2.60 p=0.02], but no significant change in responsiveness to Tetris. There was no significant difference between monkey faces and cartoon faces pre-training any of the face patches (Ps > 0.05), but post-training, there was a significantly smaller response to cartoon faces vs. monkey faces in the Anterior [t(12)=2.45 p<0.05] and Middle Faces Patches [t(12)=2.19, p < 0.05].

Figures 2, 3, S3 and S4 show that, first, training with different symbol sets did reproducibly result in the appearance of selective responsiveness to the trained shapes compared to control shapes in posterior inferotemporal cortex (PIT). There was no significant relationship between the monkeys’ level of performance and the size or strength of the training-induced selectivities. Second, the training-induced activations by the Helvetica symbol set were on average localized to a similar region as described previously⁸, except slightly more dorsal on average than in our previous study, in which monkeys learned only the Helvetica symbol set (cf Fig. 3c). In addition we observed an occasional second smaller patch more anterior, in anterior or central IT (CIT, AIT, see Fig. 3), which we had not observed in our previous study, between the superior temporal sulcus (STS) and the anterior medial temporal sulcus (AMTS). The new Tetris>control activations that appeared in PIT after training with the Tetris symbol set were usually just ventromedial on each monkey to the location of the Helvetica>control patch, centered along the occipitotemporal sulcus. In two monkeys there was also a small, more anterior Tetris patch in CIT or AIT. The larger PIT Tetris selective region was in approximately the same location that a previous study reported to be scene selective¹⁵. Before Cartoon Face training, none of the monkeys showed any differential localization of Cartoon Face responsiveness compared to monkey face activations. But, after Cartoon Face training, Cartoon Faces activated a patch just ventral to the monkey face activation. The Cartoon Face patch was dorsal to the Helvetica patch, between the STS and posterior medial temporal sulcus (PMTS). All monkeys showed bilateral Cartoon face patches and Tetris patches; two monkeys showed bilateral Helvetica patches, three monkeys showed significant Helvetica responses only on the right hemisphere, and one monkey only on the left.

Fig. 3.

Overall organization of selective responsiveness to trained symbol sets and face patches in 7 monkeys. (a) Patches of significant activations for all 3 symbol sets and monkey faces (each contrasted with its control) from each of the 7 multiple-symbol-set trained monkeys projected onto a standard macaque brain⁴⁸ shown in a semi-inflated lateral view. (b) Same data shown on a flattened standard map of macaque cortex with areal borders⁴⁹. (c) Centers of mass for different selective patches. Dots indicate the center of mass of each of the 3 major monkey faces>shapes patches and each of the trained-symbol selective patches, indicated by color, in the monkeys trained in this and our previous study⁸. For monkeys who learned both Helvetica and Tetris, the squares indicate centers of the first-learned symbol set region immediately after learning that symbol set, and circles indicate centers of the same, first-trained, symbol set, but after learning the second symbol set; the two patches for the first-learned symbol set for each monkey are linked by a line of the same color. +’s indicate the centers of the Helvetica patches for the 3 monkeys from our previous study that were trained as juveniles on Helvetica only⁸. By inspection, the centers of Helvetica patches in Helvetica-first trained monkeys shifted slightly dorsal (away from the Tetris location) after Tetris training, and the Tetris patch centers in the two Tetris-first trained monkeys moved slightly ventral (away from the Helvetica location) after Helvetica training. The indicated dorso-ventral and antero-posterior axes for the flat maps are meaningful only for the lateral surface of the brain.

For each monkey we defined Helvetica, Tetris, and Cartoon Face ROIs using scan data obtained immediately after the end of the training epoch for that symbol set (using contrasts Helvetica>control, Tetris>control, and Cartoon Faces>monkey faces); the anterior, middle, and posterior face patch ROIs were defined using the contrast faces>Helvetica AND faces>Tetris on scans obtained immediately before Cartoon face training. Using independent data sets we calculated the average percent signal change (normalized to the V1 signal change for the same image set, to account for any differences in attention or viewing) in each ROI to each symbol set and its control before training and after all training was completed. Figure 2d shows the normalized responses averaged across hemispheres (except for the unilateral Helvetica regions) and across monkeys in each inferotemporal ROI during the presentation of each symbol category and control both before and after training. A 2 × 2 ANOVA for trained set vs control × pre- vs. post-training was calculated (Fig. 2), and a robust interaction between stimulus (trained symbols vs control) and training (pre- vs post-) was observed in all training-induced regions. All the training-induced patches were significantly more activated by their trained stimulus category compared to controls after training, but none of the ROIs showed significant differences between their preferred stimulus category and controls prior to training. Pre-training, there was no significant difference between monkey faces and Cartoon faces in any of the face patches, but post-training, there was a significantly smaller response to Cartoon faces vs monkey faces in the Anterior and Middle Faces Patches and a significantly larger response to Cartoon faces vs monkey faces in the Cartoon face patch. Though the changes in Fig. 2d are complex, taken collectively, they indicate that extensive training can alter the selectivity of regions in inferior temporal cortex.

Despite some variability (Fig. 2), the locations of the different training-induced patches were similar across monkeys (Fig. 3), irrespective of training order. The three training-induced patches in all the monkeys were distributed along the dorso-ventral axis of the inferotemporal gyrus, with the Cartoon face patch usually lying just ventral to the lip of the STS and dorsal to the Helvetica patch, and the Helvetica patch dorsolateral to the Tetris patch, which usually lay along the ventral surface of the inferotemporal gyrus, or even more medial on its medial surface (Fig. S3–S10).

The localization is easier to see in the computationally flattened map in Fig. 3b where it is clear that the patches are distributed systematically along the dorso-ventral extent of PIT. This can also be seen in the distribution of the centers of mass of each of the patches (Fig. 3c). The within-category distances in inflated spherical coordinates between patch centers were significantly smaller than the between-category distances for all 3 pairwise combinations of categories (Supplementary Table 1), indicating that the different training-induced patches were indeed in different locations. The location of the selective patch for each symbol set did not depend on the relative timing of training, but did differ systematically between different symbol sets (Supplementary Table 2)

Part 2: Exploration of other possible factors

Thus intensive training with different symbol sets resulted in the appearance of novel selectivities that mapped on average to different dorso-ventral locations along PIT. We found no evidence for an effect of training order on the localization of these training-induced patches. Furthermore, the monkeys were on average equivalently good at recognizing the different symbol sets, so the differential localization cannot depend on amount of training, degree of expertise, shared function (value representation), or level of categorization. The location of these novel domains must therefore be a consequence of something unique to each set. The kinds of differences among these 3 symbol sets that could account for this differential localization include shape, discriminability, and resemblance to natural categories. The 3 symbol sets do have different shapes, in that it would be apparent at a glance to which set any particular symbol belonged. The 3 symbol sets may be differentially discriminable, although the monkeys learned all 3 sets at about the same rate, and to the same level of expertise; nevertheless the spatial scale of what distinguishes elements of each set may differ. Certainly Cartoon Faces resemble the natural category of faces, but it is not immediately obvious what natural category Helvetica or Tetris might correspond to. Any of these 3 factors could play a role in novel domain localization in IT since category, eccentricity and shape have all be reported to be systematically organized across IT^15–22. We will explore each in turn.

First, could a pre-existing category organization explain the differential localization of the trained symbol sets? Macaque IT is organized by category in that faces are represented in distinct patches within and along the lower lip of the STS^2,22,23; objects are represented in a series of patches just ventral to faces^8,16, and places or scenes still more ventral, on the inferior surface of IT^15,24. As shown in Fig. 2d, after training with Cartoon Faces, responsiveness of the original face patches to Cartoon Faces relative to monkey faces was reduced, and responsiveness of the new Cartoon Face patch increased to Cartoon Faces relative to monkey faces. If a pre-existing category organization drives the localization of training-induced domains, it is not clear why training should shift Cartoon Face responsiveness from the normal face patch to a more ventral location. Furthermore it is not clear to what natural category Tetris and Helvetica should belong. Thus a pre-existing category organization does not account in any obvious way for the differential localization of these trained domains.

We then considered eccentricity as a potential organizing principle to explain the observed training-induced changes. Early visual areas are retinotopic, with a precise map of visual space across cortex. IT is not as precisely retinotopic as early visual areas, though PIT does show a clear organization of upper vs lower visual field, ipsi vs contralateral visual field, and central vs peripheral visual field^20,25. Furthermore, this retinotopic organization is correlated with category organization, in that face processing is centrally biased, and object and place processing more peripherally biased^17,18,20,25. Malach and colleagues^17,18,21 proposed that the fundamental organizing principle of IT is eccentricity based, with a center-periphery gradient inherited from lower visual areas. Thus because our Cartoon Face patch lies just ventral to the middle face patch, the Helvetica patch just ventral to that, and the Tetris patch still more ventral, these 3 novel domains span PIT along this previously reported retinotopic gradient^20,25,26. To map this eccentricity gradient we scanned three monkeys while they viewed blocks of flickering checkerboard patterns that stimulated the central 3 degrees of visual field, contrasted with blocks of checkerboard patterns that stimulated 4 to 10 degrees of eccentricity (Fig. S2). The maps for this peripheral-field minus central-field contrast confirmed previous reports of an eccentricity bias, with central-field representation (blue) along the lower lip of the STS and peripheral-field (yellow) represented more dorsally and more ventrally (Fig. 4 leftmost column). Note that this swath of central visual field representation, flanked by more peripheral representations, extends not only across early visual areas, opercular V1, through V2, V3, and V4, but also through most of IT: including PIT and is still apparent, though weaker, through CIT and even AIT.

Fig. 4.

Maps of Eccentricity bias, Curvature, and Category selectivity in 3 monkeys, as indicated. T-score maps were averaged over both hemispheres of each monkey for the 3 contrasts indicated and aligned onto a standard macaque brain^47,50 that was then computationally flattened. Light gray areas represent gyri and dark gray sulci. Representative images from the sets used to generate these contrast maps are shown at the top of each column; image sets are shown in Fig. S2. Visual areal borders⁴⁹ are indicated. Outlines of R2’s Cartoon face (cyan), Helvetica (blue), Tetris (green), and middle face (red) patches are overlaid on his eccentricity map.

The outlines of the Cartoon face, Helvetica, Tetris, and middle face patch of monkey R2 are overlaid on his eccentricity map (Fig. 4 upper left map). Thus the Cartoon Face activations mapped to a centrally biased part of IT, and Tetris activations mapped to a peripherally biased region, with the Helvetica symbol set mapping intermediately. The stimuli in the 3 sets were on average the same size, as shown by the overlays of all the stimuli in each set at the far right of Fig. 1, and the most salient image-set difference is that the Cartoon Face symbols do not extend as far to the corners of the average template as the Helvetica and Tetris symbols do. The monkeys learned these symbol sets while they were moving around freely in their home cages, so it is unlikely that such small differences in image size would have resulted in large differences in an eccentricity-biased functional organization. It is more likely that differences in size could have caused differences in activation patterns during scanning, but retinotopic activation differences should be manifest before as well as after training, which they were not, and primarily in early, retinotopic, visual areas, and they were not (Fig. 2: note absence of any differential activations in retinotopic visual areas). Therefore we cannot explain the stereotyped localizations of the different training-induced domains by eccentricity organization, despite the presence of an eccentricity-bias organization in this part of IT.

Lastly we consider shape. It has been proposed^9,27 that the localization of category-selective domains is driven by experience-dependent modification of a pre-existing shape organization, and, in support of this idea, Tootell and colleagues recently found that face-selective regions respond better to curvy stimuli and place-selective regions to rectilinear stimuli²⁸, suggesting that another potential organizing principle in IT is shape-based, along a degree-of-curvature axis. By inspection, the 3 symbol sets do differ in curvature: every symbol in the Cartoon face set has multiple curved contours, half the symbols in the Helvetica set have at least one curved contour and more than half have at least one straight contour, whereas none of the contours in the Tetris set are curved. To look for a correlation between our training-induced domains and shape, or curvature, we also scanned these same monkeys while they viewed blocks of full-field (20° × 20°) patterns that were predominantly curvy or predominantly straight (Fig. S2). Figure 4 (middle column) shows t-score maps for the contrast straight-patterns minus curvy-patterns also thresholded at t = ±2 and averaged over both hemispheres of the same monkeys. Results were indistinguishable for beaded-curvy and wavy-curvy patterns. These maps confirm previous reports²⁸ that the lower bank and ventral lip of the STS, where the face patches lie, are more responsive to curvy (blue) than to rectilinear shapes, whereas more ventral and dorsal regions, that are scene-selective¹⁵, are more responsive to rectilinear shapes (yellow)²⁹; indeed Kornblith et al.¹⁵ found that single units recorded in a scene-selective region in the occipitotemporal sulcus responded strongly to long straight contours but only weakly to short curved contours

By inspection of the first two columns of Fig. 4, there is a similarity between the contrast maps for curvature selectivity and the contrast maps for eccentricity bias, even in early visual cortex. Curvy patterns (blue) tend to activate the same swath of cortex as central-field stimulation (blue), a swath extending from central V1, V2, V3, V4, then along the ventral lip of the STS, whereas straight patterns (yellow) tend to activate the same regions as were activated by the peripheral visual field stimulus (yellow).

Though the similarity between the eccentricity maps and the curvature maps in Fig. 4 is apparent, it is difficult to assess by inspection how similar two maps are, given the thresholding, differences in relative activation strength of the two image sets, and variability between animals. Therefore we used standard correlation analysis³⁰ to quantify the similarity between different contrast maps from the same individuals. To do this we calculated for different visual areas in each monkey the voxel-wise correlation between t-values in different maps (Fig. 5), and used permutation analysis to establish significance (see Methods). First, as proof of principle for this approach, given the well-established relationship between eccentricity and spatial-frequency tuning, we calculated for the same 3 monkeys correlations between eccentricity maps and spatial frequency maps (Fig. S11) for different visual areas. Central visual field has small receptive fields and responds to high spatial frequencies, whereas receptive fields at more peripheral eccentricities are systematically larger and respond better to lower spatial frequencies^31,32. Consistent with this well-established relationship between spatial frequency tuning and eccentricity, we found significantly positive correlations between peripheral-minus-central eccentricity maps and low-minus-high spatial frequency maps in V1, V2/V3, and V4 in all 3 monkeys (Fig. 5a; see Methods for details on spatial frequency mapping). That is, the overall positive correlation in early visual areas is consistent with the well-established relationship between eccentricity and spatial frequency tuning--regions in early visual areas that represent central visual field respond better to higher spatial frequencies than to lower, and regions representing the periphery respond better to lower spatial frequencies.

Fig. 5.

Relationship between eccentricity, spatial frequency, curvature and category. (a–d) Voxel-wise correlations between pairs of contrast maps for visual areas V1, V2&V3, V4, PIT, CIT, and AIT from 3 monkeys (Ba, R2, & Pa); the correlations are calculated between the contrast maps in Fig. 4, with the addition of a map for low-spatial frequency minus high-spatial frequency patterns for the same 3 monkeys (stimuli shown in Fig. S2; maps for spatial frequency in Fig. S11). Dotted line indicates zero correlation. Asterisks at the top indicate correlations that were significantly greater than zero at p<0.05; the asterisk at the bottom of the 3rd map indicates a correlation that was significantly less than zero at p<0.05. The contrast maps used for this analysis were Eccentricity: 4–10° patterns minus 0–3° patterns; Spatial Frequency: 0.4 cpd patterns minus 2.5 cpd patterns; Curvature: straight minus curvy patterns; Category: objects minus faces. (e–l) Spatial frequency power spectra, averaged over each stimulus set, as indicated.

When we then compared the straight-minus-curvy contrast maps (Fig. 4 middle column) with the periphery-minus-central eccentricity contrast maps (Fig. 4 left column), we also found significantly positive correlations in all 3 monkeys in V1, V2/V3, V4, and PIT, and in two monkeys in CIT and AIT (Fig. 5b). This is not surprising, since the similarity is apparent in the maps themselves, though a degree-of-curvature organization has not been previously described in V1 or V2. The positive correlation means that curvy pattern preference was correlated with central visual field, and straight pattern preference with peripheral visual field.

Thus there was a strong correlation not only between eccentricity and spatial frequency (Fig. 5a) in early visual areas but also between eccentricity and curvature (Fig. 5b), such that central visual field regions were more responsive to high spatial frequencies and to curvy patterns, compared to low spatial frequencies and straight patterns, and peripheral visual field showed the reverse preference; this preference was strongest in early visual cortex. Could the differential mapping of symbol sets or curvature be due to the well-established gradient for spatial frequency? Fig. 5 e–l show spatial frequency power spectra from the Fourier transforms averaged over all the images in each set: there is a substantial difference in spatial frequency between the high and low spatial frequency image sets, but a negligible difference between the straight image set and either of the two curvy image sets, or between any of the symbol sets and their controls, except for a small difference between Cartoon faces and Monkey faces. We conclude that the differential mapping of the curvy and straight patterns is due to curvature, not spatial frequency; that is to differences along contours, not across contours, and that the differential localization of the trained symbol domains cannot be explained by differences in spatial frequency.

The rightmost column of Fig. 4 shows for the same 3 monkeys category contrast maps using objects minus faces (see Fig. S2 for stimuli), confirming previous studies showing that face selectivity maps to patches in PIT, CIT and AIT on the lip and ventral bank of the STS, with object selectivity ventral, and sometimes dorsal, to that^2,16,23. There is no clear similarity between maps for eccentricity and category or between curvature and category in early visual areas, but there is in IT. By inspection of Fig. 4 (rightmost column), regions that are face selective (blue) lie along the lower bank and ventral lip of the STS, the same general region that is centrally biased and selective for curvy patterns^25,28, whereas object selective regions (yellow) lie mostly ventral to that, on the inferior temporal gyrus, in the same general region as more peripheral visual field representation and rectilinear bias²⁹. Consistent with this impression and with previous results, correlations between category and eccentricity were not significantly positive in V1 or V2/V3, but were significantly positive for 2 monkeys in V4, for 2 monkeys in PIT, for 2 monkeys in CIT, and one monkey in AIT; these positive correlations indicate that there is a tendency for face-selective regions in IT to have a central visual field bias, as previously reported^20,26,33, and object-selective domains to be more peripherally biased. Correlations between curvature and category were similarly not significantly different from zero in V1, V2/V3, or V4, but were significantly positive for all 3 monkeys in PIT and for 2 monkeys in CIT. This is consistent with a previous report that category selectivity is correlated with a curvature gradient^25,28.

We last asked how the novel training-induced domains were localized along both eccentricity and curvature gradients in monkeys Y1, Y2, R2 and B1. We measured the t-score for each contrast averaged within Monkey Face, Cartoon Face, Helvetica, or Tetris ROIs, for these four monkeys, except monkey Y1, who lacked Tetris training. In Fig. 6 we plot the z-score for each ROI averaged across all 4 monkeys. Across monkeys the ROIs were distributed along both the eccentricity and curvature gradients in central-to-peripheral and curvy-to-straight order: Monkey faces, Cartoon faces, Helvetica, Tetris. The ROIs were distributed along the eccentricity contrast for all 4 individual monkeys, but there was more variability in the individual monkey distribution along the curvature axis, in that in monkey B1 the Cartoon face ROI was the most straight biased ROI, and there was no curvature difference in monkey R2 between any of the ROIs except Monkey Faces. Nevertheless there was, on average and individually, a consistent alignment of the four patches with eccentricity, and a similar, though individually less consistent, distribution according curvature; the distribution was consistent with the hypothesis that the localization of these domains could have been determined either by shape or eccentricity.

Fig. 6.

Average z-scores for eccentricity (peripheral-minus-central) and curvature (straight-minus-curvy) contrasts for Monkey face, Cartoon face, Helvetica, and Tetris ROIs combined across monkeys Y1, Y2, B1 and R2; values are mean across monkeys ±sem. Plots for individual monkeys are shown in Fig. S12.

Discussion

Intensive training of macaque monkeys with three different symbol sets resulted in localized increased selectivity in IT to the trained symbols compared to visually similar but untrained shapes. Many previous studies have already established a role for experience in shaping the selectivity of IT in both humans and monkeys³⁴. The fact that the human Visual Word Form area is more responsive to text in a subject’s own language, compared to visually similar shapes or text in languages unfamiliar to the subject³⁵ indicates that symbol training in particular can alter selectivity in human IT to text. We had previously found⁸ that extensive training of juvenile macaque monkeys with a set of human symbols similarly resulted in localized changes in responsiveness in IT that could be visualized with fMRI. In this previous study we used human symbols for training monkeys because we knew that educated humans acquire domains selectively responsive to these shapes. We used human symbols as one of our symbol sets in the present study because of its clear effects in our previous study.

The goal of the present study was to find out whether training on shapes other than those used in human writing would also result in changes in responsiveness in IT and whether order of learning or degree of expertise would affect the localization of training-induced effects. Our first finding is that training on shape sets other than human symbols does result in changes in IT, such that some regions that were initially equally responsive to those shapes compared to controls became more responsive to the trained shapes than to controls. We therefore conclude that there is not anything special about human symbols that permits the emergence of symbol selectivity in IT, but rather that intensive early experience with other shapes can also result in changes in IT that can be visualized using fMRI.

Our second major finding is that training on different symbol sets produced changes in selectivity in different locations in IT, rather than in the same location for all symbol sets. Changes in a single region might be expected given that the training always involved the same behavioral task, and all the symbol sets represented the same range of reward values. Although an anatomically distinct region might have been predicted for the Cartoon Face symbol set, given the spatial segregation of face processing, finding distinct regions for the Helvetica vs. Tetris symbol sets was unexpected.

Our third, and most surprising, finding was that the locations in IT of these training-induced changes were similar for each symbol set, regardless, as far as we can tell, of the order in which the symbol sets were learned, and despite the monkeys’ being equally expert at recognizing the different symbol sets. This suggests that some inherent characteristics of these symbols determined where expertise-related changes would occur, and supports the hypothesis that plasticity is constrained by some native organization in cortex^9,27,36.

If some native organization determines where training-induced specializations will occur, why should Cartoon face responsiveness shift from the part of IT where it is found in untrained animals to a new location? Studies in humans have occasionally found shifts in selectivity as a function of training: Moore et al.³⁷ found that learning to read an alphabet of human faces induces responsiveness to those faces in the left fusiform area. Moreover, the part of the brain that is selectively responsive to text in literate humans is, in illiterate people, responsive to faces³⁸. These studies thus indicate that inferotemporal cortex can be differentially modified depending on competing influences of experience. Close inspection of Fig. 3c suggests that learning a second symbol set has a small repulsive effect on the localization of the first-learned set, but our sample size is too small for us to draw any firm conclusions. This small effect is consistent with the hypothesis that competitive interactions are involved in domain formation³⁸

Although our symbol sets were not designed to probe shape or retinotopic organization in IT, we nevertheless asked whether any native organization of either retinotopy or curvature could explain the observed localizations. We confirmed correlations in IT between maps of curvature and maps of category selectivity²⁸, as well as correlations in IT between eccentricity and category selectivity^17,20. We, in addition, discovered that our novel, training-induced domains in IT for completely unnatural shape categories are distributed along these same gradients of curvature and eccentricity (see also ref.³⁰).

We discovered a correlation between curvature and eccentricity in early visual cortex as well, and this correlation was even stronger than the correlation between curvature and eccentricity in IT. This correlation could not be accounted for by differences in spatial frequency, suggesting that curvature tuning is a low-level receptive-field property that varies with eccentricity. Different eccentricities have different spatial resolution³⁹; this is usually observed as differences in spatial frequency tuning³², which is assumed to be governed by receptive-field organization perpendicular to the axis of orientation. Yet selectivity for change vs continuity along the axis of orientation, i.e., curvature, is an important feature of higher visual areas, like V4⁴⁰. As far as we know no studies have looked at curvature tuning as a function of eccentricity, but it would be logical for central visual fields to prefer higher curvature, or faster change in orientation, since central receptive fields are smaller than peripheral receptive fields in all dimensions⁴¹, and end inhibition is prevalent in both V1^42,43 and V2⁴⁴. End-inhibited cells respond better to contours with changing orientation than to straight contours of the same length⁴⁵.

Previous studies that found a correlation between category selectivity and curvature pointed out differences in natural image statistics of faces and scenes²⁸, with faces containing more curvy contours and scenes more straight ones. Because our shapes are completely unnatural and behaviorally unrelated to either social or navigational information, our results and those of Op de Beeck et al.³⁰ favor a pre-existing curvature gradient rather than a curvature bias that derives from a category-based organization. Previous studies that found a correlation between category selectivity and eccentricity attributed that correlation to resolution requirements of different categories or ways these categories are generally viewed; that is, faces require scrutiny and are usually foveated, while scenes usually encompass the entire visual field³³. The correlation of our training-induced changes with an eccentricity gradient cannot be explained by viewing bias, since all 3 symbol sets were presented at the same size, in exactly the same manner, for the same behavioral task. Yet, in literate humans, letter strings map to an even more foveally biased region of the fusiform gyrus than faces¹⁸, whereas our monkeys show the reverse order. Our monkeys learned symbols that were always 4 cm high, and humans usually read symbols that are much smaller, suggesting that both viewing bias and shape bias can influence where training effects are localized.

Hasson et al. proposed¹⁷ that IT is organized during development according to a retinotopic map transmitted or inherited from earlier visual areas. Our results further indicate that a shape organization could arise from variations in curvature selectivity with eccentricity. Thus experience with different image categories may produce changes in IT that will be localized according to their shape statistics and/or viewing scale⁴⁶. An appeal of this idea is the plausibility of an extension of existing mechanisms of retinotopic mapping to higher visual areas, and its provision of a secondary shape-based proto-organization on which experience can exert modifying effects. Whether the organization of IT is initially established by an inherited retinotopic map, or by an innate organization for biologically important image categories, or by factors such as connectivity to other structures like the hippocampus or motor system¹¹ is ultimately, however, a developmental question, and the key experiments have not yet been done.

Methods

Behavioral training

Seven experimentally naïve juvenile male Macaca nemestrina monkeys (B1,B2,R2,Y1,Y2,G1,G2) and two juvenile Macaca mulatta monkeys (Pa and Ba) were trained to recognize sets of 26 distinct symbols as representing different reward amounts using a touch-screen (Touch Screens, Inc., St. George, Utah) mounted in their home cage. All training occurred before any of the monkeys reached puberty (~4–5 years, identified by testicle descent), but some of the scanning occurred after some of the animals reached puberty. A reward system dispensed liquid using a gravity feed and a solenoid; each drop was accompanied by a beep. Each symbol set consisted of 26 symbols representing reward values from 0 to 25 drops (or solenoid openings) of fluid. Two symbols were presented simultaneously side-by-side on the screen, and the monkey was rewarded with the number of drops of liquid represented by the symbol on whichever side he touched first. The values presented on each side were randomly chosen from values 0 to 25. The symbols were each 4 cm high. The monkeys were rewarded no matter what side they touched (except for value=0), but they usually chose the larger value side. They worked to satiety daily, during the normal light-on period, over several hours, usually performing several hundred trials per day. For training on each set they started with 0 and 1 (with the solenoid opening set to a long duration) and new symbols were added sequentially when behavior on the previously learned symbol stably exceeded 80% for all choices involving that symbol. During the learning period for each set the solenoid open time was gradually decreased as they learned higher value symbols. It usually took 6–8 months of daily training for them to master all 26 symbols in a set, then they continued daily training on all 26 symbols of that same set for at least 1 more month before scanning. The final behavioral testing for the data in Fig. 1 was at the end of the last-learned symbol set; the monkeys were given at least 1 week refresher training for each set before testing for 1 month with each set. See Fig. S1 for details of the training schedule for each monkey.

Scanning

The monkeys were scanned in the alert state, comfortably lying in a sphinx position in a primate restraint chair that fit into the bore of the scanner. The juveniles were restrained using a non-invasive padded helmet^8,51 and an attached rigid chin strap with an embedded bite bar for fluid reward delivery for fixation. Monkeys B1, R2, and Pa reached puberty after learning their last symbol set, and were implanted with a plastic head-post fastened with ceramic screws to the occipital ridge and to the frontal bone just posterior to the brow, and the scans for these monkeys that were done after puberty were accomplished using this headpost restraint.

All monkeys were scanned in a Tim Trio 3T scanner with an AC88 gradient insert and custom-made 4 channel coil arrays (made by Azma Maryam at the Martinos Imaging Center). Each session consisted of 10–30 functional scans. Stimuli were presented in 20 second blocks with 20 seconds of a gray screen with a fixation spot between each block. Images were presented in random order within each block for 0.5 seconds each. Blocks were presented in different order each session, but the same order within each scan. Scan parameters were: EPI sequence, repetition time (TR) = 2 seconds, echo time (TE) = 13ms, flip angle (α) = 72°, iPAT=2, 1 mm isotropic voxels, matrix size = 96 × 96mm, 67 contiguous sagittal slices. In order to enhance contrast⁵² the monkeys were injected with 12mg/kg monocrystaline iron oxide nanoparticles (Feraheme, AMAG Pharmaceuticals, Cambridge, MA) in the saphenous vein just before scanning. Only scans in which the monkey for fixated >85% of the scan and in which there was no motion >1mm were used for analysis. The maps in Fig. 2 and the activations in Figs. S3–S10 were all calculated from 30–40 blocks of each image category and each control for each monkey. The bar graphs in Fig. 2 were all calculated from 25–30 independent blocks of each stimulus type for each monkey. The eccentricity, curvature, category, and spatial frequency maps in Fig. 4 and Fig. S11 were each calculated from 20–25 blocks each of each pair of image categories. Eye position was monitored by an infrared eye tracker (ISCAN Burlington, MA). The monkeys were rewarded for keeping their gaze within a 2 degree fixation window on which the stimuli were centered.

Stimuli

Visual stimuli were presented on a back projection screen at the end of the scanner 50 cm from the monkeys’ eyes, using an LCD projector. The entire screen subtended 20×20 degrees of visual angle. Helvetica, Tetris, and Cartoon Face symbols, as well as the controls and achromatic monkey face controls were presented at 8 degrees of visual angle in height on a 20×20 degree dark gray background; the full-field spatial frequency stimuli and curvature stimuli were presented as shown in Fig. S2 and filled 20×20 degrees of visual angle; the face and object images for category mapping were presented in color on a pink-noise background, as shown in Fig. S2, covering 20 × 20 degrees of visual angle. The straight patterns were chosen to represent a variety of rectilinear patterns. The wavy curvy patterns were generated by adding waves to the straight patterns, and the beaded curvy patterns were generated by adding circular distortions to the straight patterns. Eccentricity (center/periphery) contrast maps were generated for peripheral flickering checkerboard patterns (4–10 degrees eccentricity) minus central flickering checkerboard patterns (0–3 degrees eccentricity). Spatial Frequency contrast maps were generated for the contrast of low-spatial frequency stimuli (full field dynamic patterns of 0.4 cycles per degree) minus high-spatial frequency (full field dynamic patterns of 2.5 cycles per degree). Curvature contrast maps were generated for the contrast between full-field straight patterns minus full-field curvy patterns. Category contrast patterns were generated for the contrast objects minus faces.

Data Analysis

Functional scan data were analyzed using AFNI⁵³ and CARET^47,48. Functional data from different sessions were first aligned to each monkey’s own average functional template using JIP software (http://www.nitrc.org/projects/jip) and then detrended and motion corrected. Scans with movements more than 1mm were not used for analysis. We calculated the maximum likelihood maps of responses to each learned symbol set using a modified gamma-variate function approximating monkey hemodynamic changes in cerebral blood volume⁵². To correct for multiple comparisons, the values for minimum patch size were derived from a simulation that estimated the probability of false positive or noise-only clusters⁵³. Using this simulation, we calculated the cluster size should be at least 31 voxels to keep the probability of getting a single noise-only cluster under 0.02 for a per-voxel p-value of 0.002. The resulting corrected (for false positives) activations (Fig. S3–S10) were projected onto a monkey template^47,48 (Figs 2&3). In order to visualize the different patches from different monkeys together, for each individual monkey we collapsed the thresholded t-maps for the two hemispheres onto the standard monkey brain (that is, a voxel was counted as belonging to a patch if it was significant for that contrast in either hemisphere). All the patches from all monkeys were color coded by symbol set and overlaid onto a single map using transparency to allow visualization of any pattern common to all monkeys (Figs 2&3). To calculate centers of mass for each symbol type for each monkey, we took the thresholded t-score maps for each hemisphere, and averaged the t-score maps across hemisphere. We then calculated centers of mass for each of these patches and projected them onto a single standard flat map (Fig. 3c). To generate maps of eccentricity bias, curvature, and category selectivity in 3 monkeys (Fig. 4 & Fig. S11), t-score maps for each contrast were calculated for each hemisphere and then averaged over both hemispheres of each monkey and aligned onto a standard flat map^48,50 of macaque cortex.

ROI analysis

ROIs for selective patches for the graphs in Fig. 2d & Fig. 6 were identified using a localizer data set collected after training on each of the symbol sets. Percent signal changes in these ROIs were then measured from independent data sets collected immediately before and training with each symbol set (pre-training) and after training with all symbol sets (post-training). The occipitotemporal ROIs analyzed in Fig. 5 & S11 were identified using the maps of Felleman & Van Essen⁴⁹ that are incorporated into Caret⁴⁸.

Correlation Coefficients

We calculated the similarity between pairs of contrast maps (curvy/straight, faces/objects, high/low spatial frequency, and central/peripheral) by calculating correlation coefficients between the t-values of each voxel in a given area in each pair of contrast maps in Fig. 4 (& Fig. S11). The correlation coefficient can range between −1 and 1; a positive correlation coefficient means the two maps vary in parallel across the area, zero means there is no relationship between the maps, and a negative correlation means the two contrasts are anticorrelated. To estimate confidence limits for the null hypothesis (null hypothesis: the two contrast maps are not correlated at all in a given visual area), we randomly shuffled the identity of the stimuli assigned to the stimulus blocks 32 independent times, then calculated contrast maps for every 1024 possible pairwise combination of identity-shuffled response blocks. We then calculated correlations between pairs of shuffled contrast maps for each area to find the 95% limits. Correlations for the unshuffled data were considered significant if they exceeded this limit.

Confidence limits for average t-score plots in Fig. S12

To estimate confidence limits for the average t-scores across each ROI, we used the same scans and randomly shuffled the identity of the stimulus (curvy vs straight or central vs peripheral) assigned to each stimulus block 1000 times, then calculated t-scores for each voxel in each ROI based on shuffled stimulus blocks. We then found 95% limits for each ROI.

Animals were pair or group housed under a 12-hour light/dark cycle. All procedures conformed to USDA and NIH guidelines and were approve by the Harvard Medical School Institutional Animal Care and Use Committee.