Functional brain mapping during free viewing of natural scenes

Abstract

Previous imaging studies have used mostly perceptually abstracted, idealized, or static stimuli to show segregation of function in the cerebral cortex. We wanted to learn whether functional segregation is maintained during more natural, complex, and dynamic conditions when many features have to be processed simultaneously, and identify regions whose activity correlates with the perception of specific features. To achieve this, we used functional magnetic resonance imaging (fMRI) to measure brain activity when human observers viewed freely dynamic natural scenes (a James Bond movie). The intensity with which they perceived different features (color, faces, language, and human bodies) was assessed psychometrically in separate sessions. In all subjects different features were perceived with a high degree of independence over time. We found that the perception of each feature correlated with activity in separate, specialized areas whose activity also varied independently. We conclude that even in natural conditions, when many features have to be processed simultaneously, functional specialization is preserved. Our method thus opens a new way of brain mapping, which allows the localization of a multitude of brain areas based on a single experiment using uncontrolled, natural stimuli. Furthermore, our results show that the intensity of activity in a specialized area is linearly correlated with the intensity of its perceptual experience. This leads us to suggest that each specialized area is directly responsible for the creation of a feature‐specific conscious percept (a microconsciousness). Hum. Brain Mapp. 21:75–83, 2004. © 2003 Wiley‐Liss, Inc.

INTRODUCTION

A century of anatomic, physiologic, and clinical studies of the cerebral cortex have shown beyond reasonable doubt the association of specific regions with specific functions. This process of localizing functions in the cerebral cortex has reached new heights with the development of techniques of measuring brain activity when humans undertake particular tasks. Subdivisions of the cerebral cortex into many components based on clinical and imaging studies do not address the question of the extent of localization in complex behavioral situations, however, when many areas are responding and interacting to give us our unified behavior. The extent of functional specialization remains disputed, which is reflected in statements such as the following “The process of perceptual analysis in the visual system is not to ‘break down the visual scene’ into basic components. Such analysis is performed interactively by areas and neurons with multipurpose properties” [Schiller, 1996]. One would naturally expect these multipurpose properties of cortical areas to be most apparent during processing in complex situations, when many attributes have to be processed simultaneously and interactively. Most studies that revealed a cortical selectivity for attributes such as color, motion, faces, or even human bodies used abstracted stimuli, such as colored squares, black and white moving dots, or static pictures of faces or bodies to do so, thereby potentially favoring “modular” results [Downing et al., 2001; Kanwisher et al., 1997; Malach et al., 1995; Perrett et al., 1982; Zeki, 1978; Zeki et al., 1991]. We wanted to go beyond and ask whether areas that show specializations under strictly controlled experimental conditions also show specializations in complex situations.

We decided to probe functional specialization in more natural conditions, by allowing our subjects to view freely the initial 22 min of the James Bond action movie Tomorrow Never Dies (EON productions) while brain activity was recorded using functional magnetic resonance imaging (fMRI). The latter was correlated subsequently with the perceived intensity of various attributes. This procedure would at the same time answer a second question, namely, whether activity in the different, functionally specialized areas of the cerebral cortex correlates with intensity of the subjective experience of the attribute corresponding to the specialization of the areas. In other words, would an intense experience of a face result in a more intense activity in the relevant (face‐processing) area of the cerebral cortex than would a less intense subjective experience of this attribute? Moreover, is the variation in response intensity in that area independent of another cortical area, specialized for another attribute? If this novel method of mapping functionally specialized areas during free viewing works, it would open a new way of identifying a multitude of cortical subdivisions based on a single experiment. This can be done by simply correlating fMRI data collected during natural viewing with subjective experience ratings of a given feature, without the need to create new stimuli or record new data, because by definition most features encountered in real life are present in natural viewing conditions. In initiating this new mapping approach we chose to study features whose cortical processing sites had been well characterized previously using conventional stimuli, thus allowing for reliable verification of our results. We chose for this purpose the perception of color, faces, human bodies, and language, whose perceptual intensity we expected to be easy to rate in natural viewing conditions and whose cortical processing substrates are known to be segregated based both on functional imaging studies [Downing et al., 2001; Kanwisher et al., 1997; Malach et al., 1995; Perrett et al., 1982; Zeki et al., 1991; Zeki, 1978] as well as from clinical observations [Broca, 1861; Damasio et al., 1990; De Renzi, 2000; Wernicke, 1874; Zeki, 1990, 1991].

Some recent studies also used more natural stimuli, but not to address the above problem. Zacks et al. [2001] used free viewing of movie clips to identify regions involved in the detection of “event‐boundaries” and Sereno et al. [2001] mapped a topographically organized area in the parietal cortex by attracting their subjects' covert attention to a transparent wedge rotating over an otherwise blackened‐out movie clip; studies on lip‐reading presented movies of talking faces [Calvert et al., 1997]. In contrast to these studies our stimuli were not manipulated experimentally, and our analysis was based on the subjective experience of the intensity of features as perceived by human observers.

SUBJECTS AND METHODS

Informed consent was obtained from all subjects in accordance with the Declaration of Helsinki, and ethical approval was granted by the Ethics Committee of the National Hospital for Neurology and Neurosurgery (London, UK).

Stimuli

Eight normal subjects (four men; age 20–30 years, all right‐handed) viewed the first 22 min of the James Bond movie Tomorrow Never Dies on a translucent back‐projection screen (26 × 19‐degree visual angle) via an angled mirror. The soundtrack was delivered through a set of air‐based headphones. The movie was interrupted every 2.5 or 3 min for 30 sec with a black screen (no sound). The intensity of perception of each attribute was highly correlated across observers, with the exception of color. Observers reported that their rating of color was somewhat arbitrary because its intensity varied very little. This was reflected in a low between‐observer correlation during colored viewing periods (r = 0.48 ± 0.19 SD). We therefore increased the variance of color intensity by switching the image every 30 sec from color to black and white and vice versa; this increased the between‐observer correlation to similarly high values as those obtained for the other attributes (r = 0.76 ± 0.11 SD; see Table I), while not affecting the overall complexity of the stimulation. The natural viewing experience of the film was only little affected by this manipulation. In many instances the changes between black and white and color were barely noticeable, not only because one was too involved in following the action of the film, but also because several scenes in the film were not fully chromatic (e.g., in snow, during darkness, etc.). This procedure allowed us to analyze data in two ways: either including only colored viewing or including both black and white and colored viewing.

Table I.

Correlations of subjective experience ratings

	Color	Faces	Language	Human body
Same featurea	0.76 ± 0.11	0.86 ± 0.06	0.88 ± 0.03	0.66 ± 0.06
Between featuresb
Color	1	0.02 ± 0.04	0.04 ± 0.06	−0.05 ± 0.06
Faces	0.02 ± 0.04	1	0.38 ± 0.03	−0.02 ± 0.12
Language	0.04 ± 0.06	0.38 ± 0.03	1	0.25 ± 0.08
Human body	−0.05 ± 0.06	−0.02 ± 0.12	0.25 ± 0.08	1

Correlation coefficients (r ± SD) among ratings of subjective experience between different observers (top) and between different features (bottom). Different observers perceived the same feature with high correlations while different features were generally uncorrelated. Values are based on viewing of the complete film clip (excluding blank periods and 15 sec of associated transitions).

^aCorrelations between observers averaged over n = 10 pairs.

^bCorrelations within observers averaged over n = 5 observers.

Subjective Ratings

Five normal human observers (two women, age 20–30 years) were asked to view the above movie clip four times each, and rate the intensity of the subjective experience in one of four attributes in each viewing. The attributes examined here were basic visual attributes, whose perception can be expected to be similar in each normal human subject, which we confirmed by demonstrating a high between‐observer correlation for each attribute. The scanning session was done separately from the rating sessions partly because multiple rating sessions were required (one for each attribute), and partly to avoid introduction of an attentional bias and problems of familiarity that would increase with each viewing. Even had the same observers been used, it would have been difficult to obtain their precise subjective experience during the scan in the subsequent rating sessions. Obtaining experience ratings from separate individuals is a stringent approach, as varying subjective experience would weaken the results. It also has the advantage that the activity observed corresponds to that common to the typical human observer, allowing for a generalization of the results.

During each of the four viewings, each observer indicated by key‐presses the perceived intensity of one of the four attributes. Four keys were available: 1, no percept; 2, moderate; 3, medium; 4, intense percept. The attributes and instructions were as follows: (1) color, rate color according to the intensity with which it is perceived; (2) faces, indicate the degree to which you see and look at faces on the screen; (3) language, indicate the degree to which you hear, read, and listen to language (reduce rates for reduced intelligibility, e.g., Chinese, or talking in the background); and (4) human bodies, same as for (2) but for human bodies, and reduce rating if only part of the body (e.g., torso) is visible; disregard faces alone. Ratings were convolved by the hemodynamic response function (HRF) for inclusion in the fMRI analysis and before their correlations were calculated to smooth out rapid alterations and to keep them comparable to the fMRI analysis.

Statistical Analysis

A multiple regression analysis was done using SPM99 (online at http://www.fil.ion.ucl.ac.uk) [Friston et al., 1995]. Each subject was analyzed separately. Overview figures were based on a fixed‐effects analysis, in which the reliability of the observations was assessed relative to within‐subject variance, as is appropriate for establishing typical features of the human brain [Friston et al., 1999]. The parameter estimates were based on a random effects analysis (error bars indicates between‐subject variance) to demonstrate specificity of the regions identified at a population level. The first four regressors corresponded to the mean ratings of subjective experiences of color, faces, language, and human bodies, convolved by the HRF. Blank periods were modeled out by regressor five, and movement‐induced variance was modeled out by the last six regressors, which contained rotation and translation parameters from the spatial realignment. In a second, otherwise identical analysis, scans corresponding to black and white periods were excluded (see above). In a third analysis, two halves of the data were analyzed separately (with two regressors per attribute, one modeling the first, the other the second half of the movie) to determine whether viewing of two separate movie clips would reveal the same neural correlates, allowing for a generalization of the results. In a fourth analysis, data were analyzed separately for each feature regressor, i.e., in four separate analyses that each contained one of the four feature regressors. This allowed each feature regressor to obtain maximal attribution of variance without competition against other regressors, and served in combination with a low statistical threshold to confirm the spatial segregation of regions specialized for processing each of the features.

Data Acquisition and Preprocessing

Whole brain images (48 slices; 1.8 mm thick, 1.2 mm gaps, 64 × 64 pixels, overall resolution 3 × 3 × 3 mm) were collected in a 2‐T Siemens Vision scanner using an echo‐planar imaging (EPI) sequence. Echo time (TE) was 40 msec. For technical reasons, scans for Subjects 1–4 were acquired using a repetition time (TR) of 4.105 sec with 324 whole‐brain acquisitions, Subjects 5–8 with a TR of 3.649 sec and 368 acquisitions. Scans from Subjects 1–4 were subsequently sub‐sampled to match those of Subjects 5–8. Using SPM99, images were spatially realigned to reduce movement artifacts and slices were realigned temporally to compensate for acquisition time lags. Images were normalized spatially to the Montreal Neurological Institute template (approximating to the atlas of Talairach and Tournoux [1988]) and spatially smoothed with an isotropic Gaussian kernel of 10 mm full‐width at half height. Data were low‐pass filtered by convolution with the HRF and high‐pass filtered with a cut‐off period of 400 sec. Each whole brain image was scaled to its mean intensity to compensate for fluctuations of the global signal.

RESULTS

Perception: Independence of Features in the Natural Environment

Correlations of the psychometrically determined subjective experiences of the four attributes (color, faces, language, and human bodies), as rated by five observers on a scale from 1–4 are summarized in Table I. There were two main findings. The first was that the same feature was perceived similarly by different observers in terms of intensity of experience, with an average correlation of r = 0.79 ± 0.10 SD (first section of Table I). The second finding was that different features were perceived very differently over the viewing period, on average with a correlation of r = 0.10 ± 0.17 SD (mean of all positive correlations: r = 0.17 ± 0.17 SD, negatives: r = −0.04 ± 0.03 SD; second section of Table I). The perception of color varied little (see methods), and correlated least with that of the other attributes (mean of abs (r) = 0.04 ± 0.02 SD; colored viewing only: mean of abs (6 r) = 0.09 ± 0.09 SD). The highest correlation was between language and faces, because language often originated from speaking faces (r = 0.38 ± 0.03 SD).

The psychometric findings thus show that the perceived intensities of different attributes vary quite independently from each other in free viewing conditions and that different observers agree substantially in their subjective experience of these attributes.

Cortex: Independence of Activity Time Courses Between Segregated Areas

Brain regions whose activity correlated specifically with the subjective experience of each of the four attributes are shown in Figure 1 for the whole group in the form of glass‐brain projections (P < 0.05, corrected for multiple comparisons). It is evident that the neural correlates of the attributes were anatomically distinct, localized, and corresponded to the regions identified by more traditional studies that have focused on isolated features. The anatomical segregation of these specialized regions is illustrated further in Figure 2a, where activity related to each feature is color‐coded and superimposed on a rendering of the structural of a template brain. White markers indicate the points of peak‐activity in temporal and ventral regions, which correspond to functionally specialized areas identified in previous studies (see below for detailed descriptions). The parameter estimates (expressed as percent BOLD signal change) of the four feature‐regressors obtained for these regions are plotted in Figure 2b. The estimates were averaged over subjects, and error bars indicate the standard error across subjects to demonstrate the feature‐specificity of these areas at a random‐effects population level. Each area showed a preferential response to one of the four features. Note that the zero point is arbitrary, and also that the absolute values of blood‐oxygen level dependent (BOLD) responses vary across areas; they can only be compared relative to each other within the same area. Figure 3 shows the results of four of eight subjects, with color‐coded activity superimposed on the subjects' structural scans and on transverse slices. Experience of each feature activated mostly non‐overlapping regions that corresponded anatomically between subjects.

Figure 1.

Brain activity that correlated with the intensity of experience of color, faces, language, and human bodies during free viewing of the action movie. Shown are glass‐brain views for each of the four feature regressors obtained from a group analysis (P < 0.05, corrected).

Figure 2.

Overview showing the spatial segregation of feature‐specific activity (A), and random‐effects parameter estimates from regions of interest (B). A: Rendered views of a template brain (after removal of the cerebellum), with superimposed group‐activity related to the four features (same data as in Fig. 1, P < 0.05 corrected). B: Parameter estimates of the four feature regressors of subjective experience, expressed as mean percent change in blood oxygen level dependent (BOLD) signal ± standard error (across eight subjects), for the group‐peak voxels indicated by white markers in (A). l, left; r, right; RH/LH, right/left hemisphere; FG, fusiform gyrus; ST, superior temporal cortex.

Figure 3.

Neural correlates of perception of different features during free viewing on the example of four subjects. Significant voxels (P < 0.001 uncorrected) were color‐coded for each of the four feature regressors and shown on renderings of the right hemisphere (left) and on transverse slices (right) for each individual. Note that slices do not reflect the full extent of activity for all features, as this varied from plane to plane. Slices were taken at z = −19, −18, −16, −20 mm.

Because the intensity of experience of distinct features was fairly independent, we expected that there would be correspondingly little correlation in activity between the distinct areas specialized to process these features. At the same time, because different observers agreed in their intensity rating for the same feature, we expected activity in corresponding areas in different observers to be comparably high. To test these two hypotheses, we calculated the correlation coefficients of activity between specialized areas of different observers (for V4, fusiform face area [FFA], the face‐selective region in superior temporal [ST] cortex, Wernicke's area, and the extrastriate body area [EBA]). The subject‐ and area‐specific signals necessary for these calculations were obtained by taking, in each subject, the BOLD responses from the peak‐voxels that were nearest to the group‐peaks specific for each regressor shown in Figure 2a. The standard deviation across subjects for these locations was on average ±(5, 8, 5) mm for the (x, y, z) Talairach coordinates, respectively (see below for coordinates). Figure 4 summarizes the main findings of correlations between corresponding or distinct areas: correlations between different areas were close to zero, whether measured within or between brains (within brains: r = 0.07 ± 0.21 SD, mean of absolute values: r = 0.18 ± 0.12 SD, n = 80; between brains: r = 0.02 ± 0.17 SD, mean of absolute values: r = 0.14 ± 0.10 SD, n = 560), whereas corresponding areas (of different brains) had relatively high correlations (r = 0.32 ± 0.13 SD [same for absolute values], n = 140). Correlations between corresponding areas were thus considerably and significantly higher than correlations between different areas (Wilcoxon rank sum test P < 10⁻¹⁰, two‐sample t‐test P < 10⁻¹⁰, whether applied within or between brains, for real or for absolute values). These findings correspond to the psychometric data. The detailed listing of correlations is given in Table II, which is organized like Table I. It shows in the first section the correlation of the BOLD signal of corresponding areas across different brains, and in the second section the correlations of the BOLD signal between different areas within the same subject. Even though correlations between distinct areas were generally low, some areas were more correlated than others, such as the two face‐specific regions in the fusiform gyrus (FG) and in the ST cortex, and Wernicke's area and the face‐selective regions, a result that was expected from the high correlations between language and face perception.

Figure 4.

Corresponding areas of different brains have higher correlations of activity than different areas within the same brain. Data include all corresponding pairwise correlations across all eight subjects (see also text and Table II). n_{between brains} = 140, n_{within brains} = 80, Wilcoxon rank sum test and two‐sample t‐test: P < e⁻¹⁰, error bars = SEM.

Table II.

Correlations of BOLD signals

	V4	Face ST	FFA	Wemicke	EBA
Corresponding areasa	0.28 ± 0.12	0.22 ± 0.09	0.31 ± 0.10	0.42 ± 0.13	0.37 ± 0.08
Different areasb
V4	1	−0.16 ± 0.10	−0.09 ± 0.11	−0.17 ± 0.13	0.05 ± 0.13
Face ST	−0.16 ± 0.10	1	0.34 ± 0.13	0.21 ± 0.13	0.13 ± 0.13
FFA	−0.09 ± 0.11	0.34 ± 0.13	1	0.22 ± 0.08	0.22 ± 0.09
Wemicke	−0.17 ± 0.13	0.21 ± 0.13	0.22 ± 0.08	1	−0.02 ± 0.09
EBA	0.05 ± 0.13	0.13 ± 0.13	0.22 ± 0.09	−0.02 ± 0.09	1

Correlation coefficients (r ± SD) among BOLD signals between corresponding areas in different brains (top) and between different areas in the same brain (bottom).

^aCorrelations between brains. Each value is average of n = 28 pairs.

^bCorrelations within brains. Each value is average of n = 8 brains.

Correlation of BOLD Signal With Perceptual Intensity

Figure 5 demonstrates that activity in a specialized area is correlated linearly with the subjective experience intensity of the feature that area is specialized to process. BOLD signals obtained for the above analysis were plotted as a function of the subjective experience intensity of the corresponding feature. Figure 5 shows these plots for the regions that were most selective for each feature: V4, the face‐selective region in the ST cortex, Wernicke's area, and EBA [Downing et al., 2001]. The plots for V4α and FFA [Kanwisher et al., 1997] were equally linear and omitted merely for purposes of graphical clarity. The intensity of perception of color, faces, language, and human bodies had a clear linear correlation with activity in the areas specialized to process them. As in Table I and in the previous analysis, the time courses considered exclude blank periods and the associated 15 sec of transition where stimulus on/offsets may have led to unspecific BOLD responses.

Figure 5.

Linear correlation between the intensity of subjective experience of a given feature with the activity in the areas specialized for it. The perceived feature‐intensities (as rated on a scale from 1–4) for color, faces, language and human bodies are plotted against the BOLD signal taken from V4, the face‐selective region in the ST cortex, Wernicke's area, and EBA from all subjects. Corresponding graphs for V4α and FFA were virtually identical and omitted for graphical simplicity. Error bars = SEM.

In the following, we describe activity related to each feature separately. Talairach coordinates in this section are taken from the group analysis.

Color

Subjective experience of color (during viewing of the complete film) correlated with activity restricted to a region in the FG, and, less intensely, the primary visual cortex and an area corresponding to V5, just as in our previous study that used dynamically changing illumination conditions [Bartels and Zeki, 2000]. Two subdivisions are apparent in the medial FG: a posterior one, coinciding with area V4, and an anterior one, corresponding to V4α [Bartels and Zeki, 2000]. Talairach coordinates (x, y, z) and statistical Z‐scores were: V4, left (−32, −70, −10), Z = 7.76 and right (26, −74, −10), Z = 8.70; V4α, left (−36, −44, −22), Z = 6.55 and right (24, −54, −14), Z = 7.72. When only colored viewing periods were considered the involvement of early visual areas was enhanced comparably. This latter result, however, has to be taken with caution because of the low variance in color and the associated rating problems during these periods.

Faces

Face perception correlated with activity in regions found previously to be face selective, in the lateral FG [Kanwisher et al., 1997] and more prominently in the ST gyrus and sulcus, anterodorsal to area V5 [Chao et al., 1999] had Talairach coordinates (x,y,z) and statistical Z‐scores of: FFA, left (−44, −46, −24), Z = 3.36 and right (44, −46, −26), Z = 7.02 [left reached significance only at P < 0.001 uncorrected]; ST, left (−58, −58, 16), Z = 13.95 and right (56, −62, 12), Z = 10.57. Activity spread from the ST along the whole length of the middle temporal gyrus, more prominently so on the right, and included regions of Wernicke's speech reception area, Brodmann Area (BA) 22. This is in accord with findings of previous studies that used dynamically moving faces (eye and mouth movements) as stimuli [Calvert et al., 1997; Puce et al., 1998].

Language

The perception of comprehensible language correlated bilaterally with activity in Wernicke's area (BA 22) and in auditory cortices (BA 41 and 42) bilaterally. Coordinates and Z‐scores were: Wernicke's, left (−66, −26, 4), Z = 24.07 and right (64, −24, 0), Z = 17.73; auditory, left (−60, −12, −2), Z = 22.34 and right (60, −2, −8), Z = 22.80. The overlap with activity related to face perception in the temporal lobe (see Figs. 1, 2, 3) is compatible with previous studies on viewing eye and mouth movements and on lip reading and suggests that these regions help extract meaning related to language, not only from auditory but also from visual cues [Calvert et al., 1997; Puce et al., 1998].

Human Body

The perception of human bodies correlated with activity in two regions. Most prominently activated was EBA, which partially overlaps with V5 [Downing et al., 2001]. Coordinates and Z‐scores were: EBA, left (−44, −82, 4), Z = 13.63 and right (50, −74, 2), Z = 16.69. There was weaker activity in the anterior lateral FG, overlapping partially with the face‐selective region in the group analysis. Coordinates and Z‐scores were: body FG, left (−44, −46, −22), Z = 5.71 and right (40, −44, −22), Z = 7.48. Both these shape‐selective regions in the FG were located lateral to the color‐selective regions as is apparent in the overview given in Figure 2a. The corresponding parameter estimates in Figure 2b show that these regions are also functionally separate and feature selective.

General Nature of Natural Stimuli

The general validity of results obtained during free viewing conditions was examined in a second analysis, in which we simply analyzed the first and the second halves of the movie separately. Despite the relatively small number of scans available in each analysis (corresponding to <9 min film viewing time), the results obtained from either half of the movie were consistent and shown in Figure 6 as glass brain projections, side‐by‐side. The high similarity of these independently obtained results confirms that free viewing of natural scenes, regardless of their precise nature, leads to feature‐specific and characteristic activity in distinct anatomical loci. Because competition of regressors in a regression analysis may emphasize functional segregation, we did additional analyses using each feature‐regressor in a separate analysis. Figure 7 shows that when regressors are assigned maximal variance in this way, the results are virtually indistinguishable from the original results shown in Figure 1, with the locations of peak activations coinciding with those obtained in the original analysis. Even though single‐regressor results are shown at the very low threshold of P < 0.001 uncorrected, distinct, functionally specialized areas are associated with each feature, with the same minimal overlap. This is in itself surprising because no one can doubt that specialized regions interact with each other and are affected by each other's processing (see for e.g., parameter estimates in Fig. 2b). Although we cannot exclude the possibility that a given area may be involved in a multiplicity of processing, our results nevertheless demonstrate a high degree of spatial segregation of feature‐specific processing, even during natural viewing conditions.

Figure 6.

Separate analyses of data from the first (top) and the second (bottom) halves of the movie provided consistent results, suggesting that functional mapping during natural free viewing conditions does not depend on the precise nature of the stimuli used (group, P < 0.05, corrected).

Figure 7.

Spatial specificity of feature‐specific activation. Results from four different analyses, using only one of the four feature regressors in each to maximize their power and shown at a low threshold of P < 0.001 uncorrected to illustrate that activity is contained to the functionally specialized regions identified in the original analysis shown in Figure 1.

DISCUSSION

We exposed human observers to conditions that resemble those we encounter in real life, by allowing them to freely view a James Bond movie, where the brain has to process many features simultaneously and at a rapid rate. Subsequent correlation of the intensity of perception of color, faces, language, and human bodies with the recorded brain activity showed that in natural conditions: (1) functional segregation in the cortex is preserved; (2) the intensity of feature perception correlates linearly with the activity in the area specialized to process it; (3) different features are perceived with a high degree of temporal independence; and (4) functionally specialized cortical regions have activity time courses that are similarly independent and unique to them. Functional mapping can therefore be achieved in natural conditions without the need for specialized stimuli.

Because the functionally specialized areas identified here are the same as those identified using traditional stimuli, our study replicates the results of a multitude of previous conventional studies [Downing et al., 2001; Kanwisher et al., 1997; Malach et al., 1995; Perrett et al., 1982; Zeki, 1978; Zeki et al., 1991]. This demonstrates that the feature‐specific responses of these areas cannot be attributed to the use of perceptually abstracted, tailored stimuli, but that functional segregation is an inherent property of the brain, apparent even in complex situations. Our study thus shows that traditional, more reductionist approaches to human brain mapping lead to results that are valid also in natural conditions and vice versa.

Direct Relationship Between Perceptual and Physiologic Signal Intensity

The close relationship between the intensity of BOLD signal and perceptual intensity is an extension of previous findings, in two ways. First, it has been shown consistently that specialized areas may be active even when there is no conscious percept of the attribute they process, but that they are always more active when a conscious percept arises [Dehaene et al., 2001; Moutoussis and Zeki, 2002; Rees et al., 2000b; Zeki and Ffytche, 1998]. Second, it has been shown that directed attention (which in a sense is enhanced perception) to one of several features or locations enhances activity in the loci processing these features or locations [Bartels and Zeki, 2000; Corbetta et al., 1991; Tootell et al., 1998], that the recognition performance of objects is correlated linearly with activity in object‐sensitive areas [Bar et al., 2001], and that parametric changes in the physical property of an isolated stimulus (such as motion coherence or word‐presentation rate) is related to BOLD signal intensity in V5 or in ST cortex [Rees et al., 1997, 2000a]. Taking all these studies into account, our results allow us to formulate the general rule that the amount of activity in an area directly reflects the intensity of the percept of the attribute it is specialized for, and that a certain minimal amount of activity is required to achieve a percept. Because the only regions in the brain whose activity correlated with the perception of a given feature were those specialized to process it, we conclude that part of the activity in these specialized regions is responsible for the creation of the conscious neural representation of a particular feature (i.e., a microconsciousness) [Bartels and Zeki, 1998; Zeki and Bartels, 1998, 1999b].

Special Role for V5 and Its Satellites in Processing of Dynamic Features

The regions that correlated most with the experience of color were two subdivisions in the medial FG, V4, and V4α, whose color selectivity has been shown previously [Bartels and Zeki, 2000; Beauchamp et al., 1999; Kastner et al., 1998; Zeki and Bartels, 1999a]. Both are involved in color constancy operations and differ in their retinotopic organization [Bartels and Zeki, 2000], and we refer to them as to the human V4 complex [Bartels and Zeki, 2000; Wade et al., 2002]. Weak but interesting activity was found in a region corresponding to the motion‐sensitive area V5. In contrast to studies that used static color stimuli [see for e.g., Beauchamp et al., 1999; Kastner et al., 1998; McKeefry and Zeki, 1997], we found this region to be activated in a previous study that involved dynamically changing illumination conditions [Bartels and Zeki, 2000]. This leads us to suggest that V5, even though not specific to colors per se, may play a role in processing dynamic aspects of changes, including changes in luminance and wavelength composition of otherwise stationary stimuli [Bartels and Zeki, 2000]. Face perception correlated with a face‐selective region in the lateral FG (FFA) [Kanwisher et al., 1997] and, more prominently, with a region located just dorsal and anterior to area V5 in the ST gyrus, a region activated previously with shapes of animate objects (faces and animals, but not tools or houses) [Chao et al., 1999]. A similar pattern was observed with the perception of human bodies: a region in the lateral FG and another, more prominent one, in close proximity to V5 were activated. The latter coincides with an area shown to respond to body shapes (EBA), directly neighboring V5 [Downing et al., 2001]. Together with previous studies that have shown V5 itself to have a higher activity when viewing shapes [Bork and Zeki, 1998; Kourtzi et al., 2002], we interpret our results to reflect the dynamic nature of the more natural stimuli viewed here. We propose that the occipitotemporal cortex in proximity to V5 may be involved in processing dynamic aspects associated with shapes, such as changes in illumination, orientation, viewing angle, size, and in the case of faces, expressions, gaze, etc., which change constantly in real life, whereas the FG might be involved in attributing identities to the shapes seen.

Segregated Versus Distributed Processing and Limitations of fMRI

The cortical regions whose activity correlated with the subjective experience intensity of different features showed minimal or no overlap with each other, even in an analysis that maximized the cortical extent of activation. Like the results of previous conventional studies [Downing et al., 2001; Kanwisher et al., 1997; Malach et al., 1995; Perrett et al., 1982; Zeki, 1978; Zeki et al., 1991], this speaks in favor of a highly segregated processing of the features studied here. We should emphasize, however, that results obtained using functional imaging can never be used to show the non‐involvement of areas, and also that positive results are only of a correlational nature. A region of particular interest in this context is the ventral lateral occipital cortex (LOC), which is involved in visual shape recognition [Malach et al., 1995]. Although some groups emphasize its functional segregation [Epstein and Kanwisher, 1998; Kanwisher et al., 1997] others claim that LOC may represent the vast variety of shapes ranging from faces to houses in an entirely distributed manner [Haxby et al., 2001]. A third alternative may lie somewhere in‐between, where objects belonging to different classes or to different levels of personal expertise are represented in broadly segregated regions [Gauthier et al., 2000; Hasson et al., 2003]. We simply point out here that neither extreme view can be proven sufficiently using functional imaging alone. On the one hand, non‐involvement of functionally specialized regions in a given task cannot be proven formally using fMRI. On the other hand, distributed responses can arise due merely to lateral connections between functionally specialized regions, which may lead to distributed activity patterns in large expanses of cortex, even though it may be a by‐product of activity in distinct, segregated, and specialized regions. Clinical evidence is therefore crucial, as it can provide causal proof of the necessity of a particular region in processing of a particular feature. Lesions in spatially confined regions (that overlap with those identified functionally) lead to feature‐specific deficits that can occur separately, such as prosopagnosia, achromatopsia, object agnosia, akinetopsia, or aphasia [Broca, 1861; Damasio et al., 1990; De Renzi, 2000; Wernicke, 1874; Zeki, 1990, 1991], thus providing causal evidence for functional segregation for these features.

Feature Independence in Time and Space

It is worth mentioning that our measurements of cortical activity relied on the relatively slow BOLD signal, which was only sampled every few seconds. This did not prevent the emergence of BOLD signal time courses that were highly specific for each of the spatially segregated areas, despite the stimulation by rapid, real‐life stimuli of a much faster time scale. We believe that the temporal independence with which different features occur in natural conditions, reflected in the distinct activity time courses (ATCs) in each area, may well have been one of the evolutionary driving forces behind functional specialization [Bartels and Zeki, 2003]. It is therefore the distinctness of ATCs of spatially and functionally segregated areas during natural conditions that allowed us to segregate distinct areas from each other using a hypothesis‐driven analysis. Many more areas can be segregated using data‐driven analyses that rely solely on the distinct ATCs of distinct areas during natural conditions, even though their functions are not revealed directly by such analyses [Bartels and Zeki, 2003]. That the same areas were identified when different halves of the movie were analyzed separately suggests that results obtained during free viewing conditions have a general validity in that they do not depend on the precise nature of the stimuli used. Our findings therefore open a new way of functional imaging, which allows the mapping of multiple functionally specialized brain areas in a single experiment, using completely uncontrolled, complex, and natural stimuli.