Language affects patterns of brain activation associated with perceptual decision

Abstract

Well over half a century ago, Benjamin Lee Whorf [Carroll JB (1956) Language, Thought, and Reality: Selected Writings of Benjamin Lee Whorf (MIT Press, Cambridge, MA)] proposed that language affects perception and thought and is used to segment nature, a hypothesis that has since been tested by linguistic and behavioral studies. Although clear Whorfian effects have been found, it has not yet been demonstrated that language influences brain activity associated with perception and/or immediate postperceptual processes (referred hereafter as “perceptual decision”). Here, by using functional magnetic resonance imaging, we show that brain regions mediating language processes participate in neural networks activated by perceptual decision. When subjects performed a perceptual discrimination task on easy-to-name and hard-to-name colored squares, largely overlapping cortical regions were identified, which included areas of the occipital cortex critical for color vision and regions in the bilateral frontal gyrus. Crucially, however, in comparison with hard-to-name colored squares, perceptual discrimination of easy-to-name colors evoked stronger activation in the left posterior superior temporal gyrus and inferior parietal lobule, two regions responsible for word-finding processes, as demonstrated by a localizer experiment that uses an explicit color patch naming task. This finding suggests that the language-processing areas of the brain are directly involved in visual perceptual decision, thus providing neuroimaging support for the Whorf hypothesis.

Does the language people speak influence their perception of the world? Fifty years ago, the speculations of Whorf on this question led to the hypothesis that the characteristics of a particular language affect the perception and thought of its speakers (1, 2). Over decades, this hypothesis has received mixed evidence from behavioral and linguistic findings (3–18). Studies reporting empirical support have suggested that categorical perception of colors is affected by the presence of lexical codes of these colors. For example, speakers of English judge colors that straddle the English category boundary between green and blue to be less similar than do speakers of Tarahumara, a language of Mexico that uses a single word for these colors (6). Unlike English, Russian makes a distinction between lighter blues (“goluboy”) and darker blues (“siniy”). In a speeded color discrimination task, Russian speakers are faster at discriminating two colors falling into the two linguistic categories (one goluboy and the other siniy) than when the colors are from the same linguistic category, whereas English speakers fail to do so (8). In domains other than color perception, it has also been shown that categorical perception (for animal silhouettes, for example) occurs at lexical boundaries (3). Language may also affect how people represent spatial relations (19, 20) and how they encode viewed objects (21).

Recent psycholinguistic investigations have further established that language is disproportionately engaged in the discrimination of colors presented in the right visual field (RVF) as compared with the left visual field (LVF), confirming the Whorf hypothesis from the perspective of the functional organization of the brain (7, 9). Specifically, discrimination of colors with different names is faster in the RVF than in the LVF, because, inferentially, the lexical distinction in the left cerebral hemisphere sharpens the perceptual difference. Moreover, this laterality effect is attenuated by a concurrent interference task that demands language resources and not on an equally difficult nonverbal interference task. Thus, it seems that perceivers view the right half of their visual field filtered through the lens of their language.

In the present, blocked-design functional magnetic resonance imaging (fMRI) study, we measured brain activity to determine whether the activation of neural systems mediating “perceptual decision” is modulated by language. In a color discrimination paradigm, we exposed simultaneously two colored squares for 100 ms against a gray background, which was followed by a mask for 900 ms. The mask was a gray square identical to the background, which was used to interrupt ongoing processing of the targets (i.e., colors) (22, 23). Subjects were asked to judge whether the two viewed colors were the same or different; therefore, no use of color words was required. There were two experimental conditions. In both conditions, the two color patches to be judged as the same or different were drawn from a group of three colors. In the first condition, all three colors had an easy name to access: the Mandarin Chinese equivalent of red, green, or blue (the “easy-to-name” condition). In the second experimental condition, the three colors used were equally familiar to our subjects as the three easy-to-name colors, but their names were hard to access (the “hard-to-name” condition) (Fig. 1 A and B).

Fig. 1.

Experimental materials and behavioral results. (A and B) Printed-rendered versions of the six colors used. The three colors in A are easy-to-name colors, and three colors in B are hard-to-name colors. (C and D) Behavioral performance in the three conditions. In the color discrimination task, there were no significant differences in reaction time or response accuracy between the easy-to-name and hard-to-name colors. Reaction times (for correct responses only) were faster and accuracies were higher for arrow judgment compared with color discrimination. Error bars indicate SE measurement (SEM). ENC, easy-to-name colors; HNC, hard-to-name colors.

The accessibility of the names of the six stimulus colors was assessed in a preliminary behavioral experiment in which we asked five native Mandarin speakers who did not take part in the formal experiments to name aloud each of the six colored squares within 1 s. The colors were presented one at a time and subjects were asked to name each as quickly as possible with a single Chinese word (or character). All subjects made consistent naming responses to the three easy-to-name colors. For the second, hard-to-name color (Fig. 1B), none of the five subjects made any response within the specified exposure period. For the first and third colors, two subjects made no responses and the other three subjects reported three different names for each of them, indicating that the subjects experienced difficulty in mapping these three colors to lexical codes. Hence, the crucial difference between the two color conditions lies in the ease of access to lexical labels of the colors for native speakers of Mandarin.

In a control (baseline) condition, subjects judged whether a viewed arrow was pointing rightward or leftward (the “arrow judgment” condition). This task controlled for activation owing to the decision-making required by the experimental task itself (i.e., making a single binary discrimination judgment). The two experimental conditions and the control condition were administered in a counterbalanced order.

If lexical codes are automatically involved in color discrimination, we predict that the availability of lexical information in the two experimental conditions will be reflected in the language processing regions of the brain. Brain regions associated with successful lexical search include the left posterior superior temporal cortex and the inferior parietal lobule, which have previously been shown to mediate word-finding processes in primary progressive aphasia (24, 25). In addition, the left mesial occipitotemporal regions, specific to color anomia (26), may also be relevant, because these areas are known to subserve color naming in studies with brain-damaged patients (27, 28). Consequently, some or all of these regions are predicted to be more highly activated by the easy-to-name colors in our discrimination task. The present results indicate that the posterior superior temporal cortex and the inferior parietal lobule in the left hemisphere, both relevant to language processing, were involved in color discrimination.

To further study whether cortical regions showing differential activations between the two color perception conditions indeed subserve specifically linguistic processes (namely, lexical search or word finding), we administered two localizer tasks by using fMRI with the same subjects (29). Subjects had to either name (orally) the three easy-to-name color patches (“color patch naming”) or, with no color stimuli present, read aloud the Mandarin equivalents of the three corresponding color words (red, blue, and green) (“color word reading”) in a block design. Although both tasks require lexical activation, motor programming, and articulation processes, only the color patch-naming task requires a color-to-word mapping process. Thus, some regions recruited by the color patch naming task should overlap cortical areas showing higher activation during the perceptual decision of the easy-to-name colors in the discrimination task. Our comparison of the level of brain activity evoked by the color patch naming and color word reading tasks demonstrated that the posterior superior temporal and inferior parietal circuits are related to the color-to-word mapping procedure.

Results

Behavior.

We performed an analysis of variance on the data from the three conditions (Fig. 1 C and D). For both sets of colors, the color discrimination task took significantly longer than the arrow decision: F(1, 16) = 122.35 and P < 0.001 for the easy-to-name condition; F(1, 16) = 109.7 and P < 0.001 for the hard-to-name condition. Response accuracies followed the same pattern; for both conditions, F(1, 16) > 26 and P < 0.001. However, there were no statistical differences between the easy-to-name and the hard-to-name colors (86% vs. 88% for accuracy and 426 vs. 427 for reaction time); F(1, 16) < 1 in both cases. This suggests that the perceptual judgments of two types of colors were equally difficult, thus ensuring that any differences in the brain areas activated by easy-to-name and hard-to-name colors would not reflect differences in the amount of effort required to identify these stimuli (30, 31).

fMRI Results.

In examining the neural systems mediating color discrimination, we first contrasted brain activation during the “color judgment” and arrow judgment conditions (Fig. 2 and Table 1). Largely common brain regions were recruited for the perceptual decision of easy-to-name and hard-to-name colors, which agreed with the visual cortex critical for color vision, as demonstrated by prior studies (30, 32–34); medial frontal gyrus [Brodmann area (BA) 6]; mid-inferior prefrontal cortex (BA 9 and 44); and insula; all bilaterally. The left superior parietal gyrus (BA 7), right superior temporal cortex (BA 22), thalamus, and cerebellum also contributed to color discrimination. The left superior temporal gyrus (BA 22/40) was uniquely activated by easy-to-name colors.

Fig. 2.

Brain regions with significant activity during color discrimination. (A) Cortical activation associated with perceptual discrimination of easy-to-name colors contrasted with arrow judgment. (B) Cortical activation associated with perceptual discrimination of hard-to-name colors contrasted with arrow judgment. The significance thresholds are P < 0.05 FDR-corrected for both comparisons. All of the functional maps (in color) are overlaid on the corresponding T1 images (in gray scale). Planes are axial sections, labeled with the height (millimeters) relative to the bicommissural line. L, the left hemisphere; R, the right hemisphere.

Table 1.

Coordinates of activation peaks

	Easy-to-name vs. arrow					Hard-to-name vs. arrow
	BA	Z score	x	y	z	BA	Z score	x	y	z
Occipital
Left lingual gyrus	18	6.03	−12	−84	−8	18	6.19	−12	−84	−6
Right lingual gyrus	18	5.42	14	−78	−8	18	5.32	12	−80	−8
Left fusiform gyrus	19	4.99	−30	−72	−12	19	5.03	−32	−71	−13
Right fusiform gyrus	37	5.29	32	−43	−15	37	5.51	34	−45	−16
Left calcarine fissure	17/18	5.35	−12	−83	6	17/18	5.43	−10	−83	2
Right calcarine fissure	17/18	4.93	6	−87	1	17/18	5.40	12	−93	10
Left superior occipital gyrus	19	4.22	−32	−82	23	19	3.52	−32	−82	23
Right superior occipital gyrus	19	4.48	28	−80	26	19	4.36	28	−80	26
Left middle occipital gyrus	18	5.85	−14	−94	16	18/19	5.59	−24	−85	19
Right middle occipital gyrus	18	5.62	18	−94	14	18	5.52	20	−94	14
Left inferior occipital gyrus	18/19	4.27	−34	−72	−6	17/18	4.13	−12	−90	−7
Right inferior occipital gyrus	18/19	4.86	34	−72	−6	18/19	4.65	34	−72	−6
Left cuneus	18/19	5.84	−22	−86	21	18/19	5.6	−22	−84	21
Right cuneus	18	5.88	16	−95	12	18	5.68	14	−93	10
Temporal
Left superior temporal gyrus	22	3.19	−48	−17	6
Right superior temporal gyrus	22	3.64	50	13	−2	22	3.71	48	6	−5
Parietal
Left superior parietal gyrus	7	3.24	−22	−51	60	7	4.60	−26	−60	45
Left inferior parietal gyrus	7/40	3.67	−24	−56	43	40	3.45	−50	−31	42
	40	3.16	−51	−16	21
Right inferior parietal gyrus	40	3.16	55	−35	31
Frontal
Left medial frontal gyrus	6	4.44	−6	12	44	6	5.16	−2	3	51
Right medial frontal gyrus	6	4.65	4	8	53	6	4.65	8	12	45
Left precentral gyrus	6	4.23	−42	5	16	6	2.58	−61	7	18
Right precentral gyrus	6	3.87	55	6	42	6	3.33	34	−7	61
Left postcentral gyrus	2	3.40	−50	−29	51	2	3.76	−53	−29	47
Left superior frontal gyrus						6	3.43	−28	−7	63
Right superior frontal gyrus	6	4.34	22	3	66	6	3.30	28	−4	67
Left middle frontal gyrus	9	3.51	−32	51	18	8	3.44	−51	8	38
Right middle frontal gyrus	10	4.07	34	52	21	9/10	2.79	38	48	23
Left inferior frontal gyrus	44	4.29	−46	7	27	44	5.76	−46	7	25
Right inferior frontal gyrus	44	4.40	50	11	27	44	4.39	50	9	25
Left insula	—	4.35	−38	14	−1	—	3.58	−42	11	−7
Right insula	—	3.28	34	16	−1	—	3.25	32	18	1
Others
Cerebellum	—	5.75	−32	−46	−23	—	5.38	−28	−69	−17
	—	5.24	28	−71	−12	—	5.78	32	−44	−16
Thalamus	—	3.39	20	−19	18	—	3.49	14	−15	6

Next, we compared the two color discrimination conditions directly to look into possible activation differences [Fig. 3 A and B; also see supporting information (SI) Fig. 4 for an illustration of more axial images covering the whole brain]. In comparison with hard-to-name colors, perceptual discrimination of easy-to-name colors elicited unique activation in the posterior portion of the left superior temporal gyrus (BA 22; x = −57, y = −38, z = 18). In addition, activity levels in the left inferior parietal lobule (BA 40; x = −61, y = −32, z = 27), left precuneus (BA 7; x = −12, y = −44, z = 61), and left postcentral gyrus (BA 3; x = −16, y = −32, z = 68) were statistically stronger for easy-to-name colors. The heavy involvement of the superior temporal gyrus and the inferior parietal cortex in easy-to-name color discriminations is corroborated by a further analysis of regions of interest in terms of percent blood oxygen level-dependent (BOLD) signal changes (Fig. 3 C and D). No regions showed stronger activity for the discrimination of the hard-to-name colors.

Fig. 3.

Brain activations elicited by color perception and explicit color naming. (A and B) Areas showing significant activation during perceptual discrimination of easy-to-name colors in comparison with perceptual discrimination of hard-to-name colors. A and B are lateral view and axial sections, respectively. Two regions of greatest interest are the left posterior superior temporal gyrus (BA 22; x = −57, y = −38, z = 18) and the left inferior parietal lobule (BA 40; x = −61, y = −32, z = 27). (C and D) Percentage BOLD signal change (± SEM) at voxels of maximal difference between the two color-discrimination conditions in the two regions of interest. (E and F) Areas showing significant activation in explicit color naming against color word naming as baseline. E and F are lateral view and axial sections, respectively. The left posterior superior temporal gyrus and the left inferior parietal lobule are critically engaged by the color naming task. The significance thresholds are P < 0.001 uncorrected for the perceptual discrimination of easy-to-name colors contrasted with the perceptual discrimination of hard-to-name colors and P < 0.05 FDR-corrected for color patch naming against word naming. Functional maps shown at axial sections (in color) are overlaid on the corresponding T1 images (in gray scale). Error bars indicate SE of measurement (SEM).

To determine whether the strong activation of the left posterior temporal and parietal regions during the discrimination of easy-to-name colors arose from the involvement of the linguistic labels, we compared brain activity during color patch naming with brain activity during color word reading. Because color patch naming taps a color-to-word mapping process that is not required by color word reading, participation of the left posterior temporoparietal circuits in that task would simply reflect their essential role in mediating language processing, particularly lexical access or word finding for color patches. (We need to note that color patch naming presumably involves more visual processing than color word naming, and, hence, regions in the visual cortex for color vision should show stronger activation in the former task.) Our comparison of activations from the two localizer tasks revealed that these cortical regions, centered at BA 22 (x = −55, y = −36, z = 13) and BA 40 (x = −65, y = −42, z = 24), respectively, both exhibited more activity in color patch naming (Fig. 3 E and F).

Discussion

We used a color discrimination task to determine whether cortical regions mediating specific language processes participated in neural networks underlying perceptual decision. We found that perceptual identification of easy-to-name and hard-to-name colors activated largely overlapping brain areas; easy-to-name colors, however, were more related to activation of the left posterior temporoparietal circuits. The localizer experiment using explicit color patch-naming and color word-reading tasks showed that these circuits subserve the word-finding process entailed by color patch naming, consistent with clinical reports of aphasia patients characterized by impaired word finding who show gray matter atrophy in the left posterior temporal cortex and inferior parietal lobule (21, 22), thus confirming our hypothesis that these neural circuits both mediate linguistic processes and participate in color perceptual decision.

These brain-imaging results provide neurophysiological evidence for the Whorf hypothesis. In this study, two colored squares viewed simultaneously on each trial were exposed very briefly (for only 100 ms) and followed by a mask. With this exposure procedure, differential activation of the left temporoparietal circuits most likely reflects their direct and automatic involvement in perceptual decision per se. Our results extend findings of the left-lateralized neural organization of the Whorf effect previously shown in a visual hemifield paradigm with normal subjects and split-brain patients (7, 9). Perceptual discrimination of colors seems to provoke orchestrated brain activity that automatically occurs within a number of neuroanatomical subsystems involving bilateral visual cortices for color identification and left-lateralized temporoparietal sites relevant to language processes. It remains to be determined whether and how the activity of these language areas facilitates and interacts with activation of the visual systems specifically crucial for color vision.

In conclusion, the results of this fMRI study indicate that perceptual decision of colors with easily accessible linguistic terms in a brief discrimination task activates the left posterior temporoparietal regions and that these are the same regions that also contribute to word-finding processes engaged when a color is named aloud. Language appears to affect neural activity patterns activated in the course of color perception.

Subjects and Methods

Subjects.

We scanned 17 native Mandarin speakers, ranging in age from 18 to 33 years (nine males and eight females). They gave informed consent in accordance with guidelines established by the University of Hong Kong and the Queen Mary Hospital. All subjects were native Chinese speakers from Mainland China who had arrived in Hong Kong no longer than 30 months earlier (mean = 14.9 months; SD = 9.3 months). They were tested with the Ishihara test for color blindness; all of them had normal color vision and no history of neurological or psychiatric illness. All subjects were strongly right-handed.

Stimuli and Experimental Design.

A blocked design was used. The study consisted of two experiments, one being a color discrimination experiment and the other a language localizer experiment, which were conducted in one single run. An instruction page was visually presented for 4,000 ms in the beginning of each experiment to inform the subjects of the coming tasks. All instructions were in Chinese. The stimuli were presented via an liquid crystal display projector and back-projected onto a projection screen placed at the end of the scanner bore. Subjects viewed the rear projection screen through a mirror attached to the head coil. The distance from the projection screen to the mirror was ≈61 cm, and the distance from the mirror to the eyes of the subject was ≈12 cm. The inner edge of the colored square was presented 0.67° to the right or to the left of a centrally presented “+.“ Hence, the stimuli were separated by a visual angle of 1.34°. Subjects were asked to perform the task as quickly and as accurately as possible.

Color Discrimination Experiment.

There were two color conditions (experimental conditions): colors viewed were easy to name and colors viewed were hard to name. The former contained three colors (red, blue, and green), and the latter contained three other colors (see Fig. 1). The RGB (red, green, and blue) values of the six colors were as follows (see Fig. 1B): red = 235, 0, and 60; blue = 0, 0, and 255; green = 0, 125, and 115; color a = 88, 50, and 50; color b = 100, 158, and 167; color c = 191, 188, and 143. For the color discrimination conditions in each trial, two square color patches presented side by side against a gray background were simultaneously exposed for 100 ms, followed by a 900-ms mask with the same gray background color. When the images were presented on the projection screen, the size of the color patches was 3 × 3 cm. The RGB values for the background color were 210, 210, and 210. In both the baseline (arrow) and the two experimental conditions, subjects indicated a positive response by pressing a key with the index finger of the right hand and a negative response by pressing a different key with the index finger of the left hand.

An arrow judgment task served as the baseline condition. During each trial, an arrow was presented for 100 ms against a gray background, followed by a 900-ms mask. Subjects judged whether the arrow was pointing rightward or leftward.

Each block had 24 trials and an instruction, and each condition contained four blocks. An instruction requesting a color judgment or an arrow judgment was exposed for 2,000 ms in the beginning of each block. There were in total 96 trials for each condition. For the color conditions, half were “same” and half “different” trials. For the baseline condition, there were half “right” and half “left” correct responses. The two experimental conditions and the baseline condition were administered in a counterbalanced order.

Language Localizer Experiment.

There were two experimental conditions (i.e., color patch naming and color word reading) plus the baseline (arrow) condition. For the experimental conditions, in each trial, either a color patch (printed in the same red, blue or green color as presented in the color discrimination experiment) or a color word (one of three Chinese character printed in black , or , meaning red, blue, and green, respectively) was presented against a gray background (RGB values = 210, 210, and 210) and exposed for 1,000 ms. This was followed by a 500 ms blank screen with the same gray background color. In the experimental conditions, subjects had to either name the color of the color patch verbally or read aloud the Chinese character. The arrow judgment task served as the baseline condition. In each trial, an arrow was presented on the screen for 1,000 ms, followed by a 500-ms blank screen, and the subject was instructed to judge whether the arrow was pointing rightward or leftward as quickly and accurately as possible. Each block had 16 trials and an instruction, and each condition contained 4 blocks. An instruction “name the color” or “read the character” was exposed for 2,000 ms in the beginning of each block. There were a total of 64 trials for each condition. For the naming tasks, among the 64 trials, there were 22 trials of “red,” 21 trials of “green,” and 21 trials of “blue.” For the baseline condition, 32 of 64 trials were “rightward” arrows and 32 were “leftward” arrows. The three conditions were presented in a counterbalanced order.

fMRI Scan and Data Analysis.

Whole-brain fMRI data were acquired on a 3T EXCITE HD MRI System (GE/Signa) in the Department of Radiology, Queen Mary Hospital, by using a T2*-weighted single-shot gradient-echo EPI sequence (repetition time = 2 s; echo delay time = 30.9 ms; flip angle = 77°, respectively; in-plan resolution = 3.4 × 3.4 mm). Each volume consisted of thirty-one 4-mm-thick contiguous axial slices parallel to the anterior commissure–posterior commissure line. A total of 331 volumes were acquired for each participant, with the first three volumes discarded to allow for T1 saturation effects. Processing of fMRI data were performed by using Matlab and the SPM2 software package. The functional images were realigned and unwarped to remove movement-by-susceptibility-induced variance, spatially normalized to standard EPI template based on the Montreal Neurological Institute (MNI) stereotactic space, and resampled into 2 × 2 × 2 mm cubic voxels. The images were spatially smoothed by a 8-mm full-width half-maximum Gaussian kernel by using standard SPM methods.

Statistical analysis was performed for each subject. The activation t map of each subject was generated by using the general linear model, in which time series were convolved with the canonical hemodynamic response function. Adjusted mean images were created for each condition after removing global signal and low-frequency covariates. For the color discrimination experiment, contrasts between color conditions (easy-to-name and hard-to-name) and the arrow judgment baseline were each computed. These contrasts were then entered into a second-level analysis treating subjects as a random effect, by using a one-sample Student's t test against a contrast value of zero at each voxel. Activations that fell within clusters of 20 or more contiguous voxels exceeding the false discovery rate (FDR)-corrected statistical threshold of P < 0.05 were considered significant. For direct comparisons of easy-to-name and hard-to-name conditions, activations that fell within clusters of 20 or more contiguous voxels exceeding the uncorrected threshold of P < 0.001 were considered significant. For direct comparisons between experimental conditions, e.g., comparison of “A versus B,” negative values generated from “A versus baseline” comparison were excluded. For the localizer experiment, contrasts between color patch-naming and color word-naming conditions were computed. Activations that fell within clusters of 20 or more contiguous voxels exceeding the FDR-corrected statistical threshold of P < 0.05 were considered significant. Negative values generated from “color patch naming versus baseline” comparison were excluded. Brain regions were estimated from Talairach and Tournoux (35), after adjustments for differences between MNI and Talairach coordinates.