Words and pictures: An electrophysiological investigation of domain specific processing in native Chinese and English speakers

Abstract

Comparisons of word and picture processing using Event-Related Potentials (ERPs) are contaminated by gross physical differences between the two types of stimuli. In the present study, we tackle this problem by comparing picture processing with word processing in an alphabetic and a logographic script, that are also characterized by gross physical differences. Native Mandarin Chinese speakers viewed pictures (line drawings) and Chinese characters (Experiment 1), native English speakers viewed pictures and English words (Experiment 2), and naïve Chinese readers (native English speakers) viewed pictures and Chinese characters (Experiment 3) in a semantic categorization task. The varying pattern of differences in the ERPs elicited by pictures and words across the three experiments provided evidence for i) script-specific processing arising between 150–200 ms post-stimulus onset, ii) domain-specific but script-independent processing arising between 200–300 ms post-stimulus onset, and iii) processing that depended on stimulus meaningfulness in the N400 time window. The results are interpreted in terms of differences in the way visual features are mapped onto higher-level representations for pictures and words in alphabetic and logographic writing systems.

1. Introduction

Humans have a remarkable ability to quickly identify and categorize visual information. Studies using various tasks and methods have suggested that the neural processes underlying this ability are well underway within 200ms after a visual stimulus is presented (e.g. Curran, Tanaka, and Weiskopf, 2002; Schendan, Ganis, and Kutas, 1998; Thorpe, Fize, & Marlot, 1996), although some likely continue for several hundreds of milliseconds (e.g., McPherson & Holcomb, 1999). One issue that has received a lot of attention in recent years, and is the focus of the current study, is the time-course of the component neural processes that allow perceivers to rapidly differentiate and apply specialized resources to different domains of visual input – specifically, pictures of objects and printed words. Clearly, at some elementary level of processing, all kinds of visual stimuli must be processed in a similar way, and domain-specific processing will start to emerge as processing proceeds to higher levels. Providing precise information about the time-course of this shift from domain-independent to domain-specific processing will provide important constraints on models of visual object identification in general, and how expertise in a particular domain (such as reading expertise in the present work) can modify general purpose mechanisms for visual object processing. One way to address this issue is to directly compare processing in various domains using event-related potentials (ERP), a cognitive neuroscience technique with high temporal resolution. The primary goal of the current study was to track the time-course of cortical processing differences between pictures of common objects and printed words.

1.1 Objects and words

Let us first consider possible differences between words and objects in the context where these are likely to be the greatest – that is with words written in alphabetical scripts. There are several fundamental differences between words and objects that are known to affect the nature of the information processing that subtends their identification. Object parts (e.g., an animal's leg) convey meaning, while the parts of words (letters), at least in monomorphemic words, do not (e.g., the L in “lion” does not tell you anything about the meaning of that word). Global shape information conveys information about object identity (Bar et al., 2006), but not word identity (Paap et al., 1984; Grainger, 2008). These two fundamental differences between words and objects can explain why semantic categorization is faster for objects than for words (e.g., Theios & Amrhein, 1989). On the other hand, words can be named faster than objects (e.g., Potter & Faulconer, 1975), because word parts connect with sound in a way that determines the name of the whole, hence allowing word naming without prior identification. Finally, words written in alphabetic scripts are thought to involve highly specialized processing mechanisms compared with normal everyday object recognition, and particularly with respect to the way in which visual features are mapped onto the elementary parts of each type of stimulus. According to the neuronal recycling hypothesis of Dehaene et al. (2005), when children start to learn to read, they must first exploit the basic machinery for visual object recognition that is already in place. However, as expertise with printed words develops, this basic machinery is adapted in order to optimize processing given the specific characteristics of printed words (Tydgat & Grainger, 2009).

Evidence for such processing differences between words and objects has been seen in the ERP waveforms generated by these two types of stimuli. Several studies have reported evidence for an early word-specific response in the EEG signal that emerges between 150 ms and 200 ms post-stimulus onset. For example, Schendan et al. (1998) found that a positive wave with fronto-central distribution (referred to as the P150 by these authors) was larger to both faces and words than to pictures. Furthermore, pseudo-objects and pseudowords behaved much in the same way as real objects and words in eliciting these effects (Schendan et al., 1998), suggesting that they reflect processing at the sub-object/sub-lexical level of representation. Schendan et al. concluded that these early differences between objects on the one hand, and words and faces on the other, reflect specialization for processing visual inputs.

A parallel body of research has revealed a more localized early differentiation between words and objects (as well as other types of visual stimulus) in a negative-going wave peaking at around 170 ms post-stimulus onset (referred to as the N170 - e.g. Bentin, Allison, Puce, and Perez, 1996; Bentin, Mouchetant-Rostaing, Giard, Echallier, and Pernier, 1999; Rossion, Joyce, Cottrell, and Tarr, 2003; Curran et al., 2002). N170 amplitude has been shown to be modulated as a function of expertise with a given category of stimuli such as words and faces (e.g. Bentin et al., 1996), as well as for other forms of expertise such as bird watchers viewing birds (Tanaka and Curran, 2001). Furthermore, the N170 has a different spatial distribution for different categories. Faces are generally right-lateralized, objects tend to produce a more bilateral distribution, and words result in a more left-lateralized N170 (e.g. Rossion et al., 2003). These early differences in the ERP signal as a function of stimulus category therefore likely reflect the different ways in which basic object processing mechanisms have been adapted to the specificities associated with each type of stimulus as expertise in processing these stimuli is acquired. More recently Joyce and Rossin (2005) reported convincing evidence that the P150 (Schendan et al., 1998) and N170 reflect the same neural source, the different patterns being due to the choice of reference electrode. They showed that when the nose or the average of all scalp sites is used as the reference, than the N170 pattern is obtained. However, with the more traditional mastoid or earlobe referencing scheme a pattern of more anterior positivities (P150) is obtained.

Processing differences between words and objects have also been seen later in the ERP signal. One ERP component with a frontal distribution (the N300 - Holcomb and McPherson, 1994; McPherson and Holcomb, 1999) has been found to pictorial stimuli but not to linguistic ones, suggesting that it reflects object-specific processing (Barrett and Rugg, 1990; Eddy, Schmid, and Holcomb, 2006; Federmeier and Kutas, 2001; Hamm, Johnson, and Kirk, 2002). The N300 has been shown to differentiate between high and low semantic relatedness (Barrett and Rugg, 1990; McPherson and Holcomb, 1994) and has also been suggested to reflect semantic categorization (Hamm et al., 2002). The N350 component reported by Schendan and Kutas (2003; 2007) appeared to be the same component and was proposed to index object model selection processes, by which global shapes of objects are matched to stored visual knowledge.

Finally, the N400, which is one of the most studied ERP components and has most often been related to language processing, has been reported to reflect the process of semantic integration (e.g. Kutas and Hillyard, 1980; Holcomb, 1993). This component was enhanced by semantically anomalous words in a sentence context, by words that were primed with unrelated words, and also by single words (Bentin, McCarthy, and Wood, 1985; Holcomb and Neville, 1990). When elicited by single words, the N400 usually has a more anterior distribution, maximal over frontal or central sites (Bentin et al., 1985). Both pictorial stimuli and linguistic stimuli have been found to elicit similar N400-like effects in response to semantic mismatch (Federmerier and Kutas, 2001; McPherson and Holcomb, 1999).

1.2 Domain and script-specific processing

One challenge to using electrophysiological measures to study the time-course of domain differences for words and pictures is that contrasting stimuli as physically different as printed words and pictures of objects is likely to produce quite large differences in ERPs that have little to do with domain specific processing per se. This is because ERPs, especially early ERP components, are known to be quite sensitive to the low-level featural characteristics of stimuli such as color, size, and complexity (Luck, 2005), and therefore ERP differences between words and objects might just as well reflect differences at this level of analysis as differences due to domain specific processing. To overcome this limitation, the current study included a language comparison that also involves a large difference in low level featural elements -- contrasts between Chinese words written in simplified Han characters and English words written with the Roman alphabet. Given the gross visual differences between these two writing systems, ERP effects that are present to both Chinese and English words would not be due to their featural characteristics. Thus, ERP effects that are seen only to the linguistic stimuli but not to picture stimuli would likely reflect domain differences between words and objects beyond that of low-level differences.

Another reason for comparing English and Chinese words is that while there are undoubtedly similarities in word processing across languages (e.g. Weber-Fox and Neville, 1996), there are reasons to expect fundamental differences in the way printed words are recognized in languages that use very different writing systems, such as English and Chinese (alphabetic vs. logographic). Much prior research using ERPs to compare word processing in these two types of writing system has focused on effects of linguistic expertise seen in the N170 ERP component. Several studies have revealed that patterns of lateralization for the N170 differ for alphabetic and syllabic scripts, compared with logographic scripts. In particular, it has been found that alphabetic stimuli systematically elicit a left-lateralized N170 response (Maurer, Brandeis, & McCandliss, 2005; Maurer, Brem, et al., 2006; Rossion et al., 2003; Bentin et al., 1999), whereas the evidence at present suggests that there might be less left-lateralization for logographic stimuli such as Japanese Kanji and Chinese characters (Kim et al., 2004; Maurer, Zevin, & McCandliss, 2008).

Logographic scripts differ from alphabetic scripts not only in terms of their gross physical resemblance to line drawings of objects, but in several other important ways. First, and most relevant for the present study, is that visual features can map directly onto meaningful units (i.e., the semantic radical) in logographic scripts, whereas an intermediate set of abstract symbols (the letters of the alphabet) always intervene between visual features and meaningful representations (i.e., words and morphemes) in alphabetic scripts (see Grainger, 2008, for a review of the evidence in favor of letter-based word recognition in alphabetic scripts; and Taft, Zhu, & Peng, 1999, for a model of Chinese word recognition). In terms of the neuronal recycling hypothesis of Dehaene et al. (2005), learning to read an alphabetic script would require the development of more word-specific processes than would learning to read a logographic script, since the latter partly retains the more direct mapping of features to meaning-bearing representations that is characteristic of visual object recognition. More precisely, alphabetic writing systems require the setting-up of a specialized system dedicated to parallel, independent, letter identification (Tydgat & Grainger, 2009). Furthermore, many letters of the Roman alphabet support large variations in visual format (e.g., “a” vs. “A”), adding a further specificity to words written with this alphabet compared with the Chinese logographic script. According to this analysis we expect to see early differences in the processing of English words and pictures compared with differences in the processing of Chinese words and pictures. These early differences should reflect the different mechanisms involved in the mapping of visual features onto higher-level linguistic representations in the two types of script, and in particular the greater amount of word-specific processing hypothesized to be involved in processing alphabetic as opposed to logographic scripts.

1.3 The present study

The current study investigated processing differences between words and pictures through three experiments. In Experiment 1, the processing of pictures and Chinese words by expert Chinese readers was compared. The second experiment used the same pictures as Experiment 1 but tested English words with expert English readers. In Experiment 3, the same pictures as Experiment 1 and 2 were tested, as well as the same Chinese words as Experiment 1, but this time with naïve Chinese readers. A comparison of the differences in picture and word processing as revealed in the ERP waveforms (which we will refer to in short as picture-word ERP differences) in these three experiments will allow us to examine the time-course of qualitatively different types of neural processes defined in terms of whether or not they are domain-general (i.e., the same for words and pictures), script-dependent or not (i.e., Chinese vs. English words), and influenced by stimulus meaningfulness (i.e., pictures vs. unknown Chinese words). The following three hypothetical patterns will be particularly informative. These three patterns are described in terms of differences in the spatial distribution of picture-word ERP differences across the scalp (i.e., scalp topographies) for a given time-window across the three experiments.

• 1)Picture-word ERP differences that have distinct topographies in Experiments 1 and 2, and are not visible in Experiment 3, will reflect script-specific processing. Any change in the pattern of picture-word differences across Experiments 1 and 2 will most likely be driven by the change in script that occurs across these two experiments (Chinese words vs. English words). An absence of an effect in Experiment 3 would imply that it is unlikely to be differences in the physical similarity of pictures with Chinese words compared with English words that are the source of any observed difference across Experiments 1 and 2, and would suggest that it is indeed expertise with a given script that is the critical factor.
• 2)Picture-word ERP differences that have the same topography in Experiments 1 and 2 but are not seen with naïve Chinese readers in Experiment 3 will reflect domain-specific, script-independent processing. This precise pattern should reflect fundamental differences between picture and word processing independently of script (logographic or alphabetic) and low-level differences between these two types of stimuli (given the absence of an effect in Experiment 3). This pattern is hypothesized to reflect differences in the way abstract form representations are activated during the processing of pictures and printed words.
• 3)Picture-word ERP differences that are visible in Experiment 3 and that differ from the patterns seen in Experiments 1 and 2 will reveal domain-independent processing of meaningful objects (pictures and known words). The presence of a distinctive pattern of ERP picture-word differences seen in Experiment 3 should reflect the fact that in this Experiment the Chinese words are unknown to the participants (native speakers of English). If this pattern is not the same as the picture-word differences seen in Experiments 1 and 2, then it likely reflects differences in meaning activation independently of stimulus format.

2. Experiment 1

2.1 Introduction

In Experiment 1, expert Chinese readers were presented with pictures of common objects and the corresponding Chinese words in a semantic categorization task. Participants were instructed to press a button to occasional picture and word stimuli that referred to “body part” (e.g., a picture of a nose or the word “nose”), which were presented on a random 12% of trials (6% words and 6% pictures), but to withhold responding to all other non-body part “critical” words (44% of trials) and pictures (44% of trials). ERPs were recorded to the critical word and picture stimuli. This task requires participants to process all stimuli for meaning, but prevents motor contamination of ERPs on the trials of interest.

2.2 Methods

2.2.1 Participants

Twenty native Mandarin speakers from the Tufts University community (13 females, mean age 24.2) who were very familiar with the Chinese written script volunteered to participate and were compensated for their time. All participants were right-handed, with normal or corrected-to-normal visual acuity and no history of neurological insult or language disability.

2.2.2 Stimuli

The picture stimuli consisted of 184 black and white line drawings of common objects, selected from a standardized inventory (Snodgrass & Vanderwart, 1980). The word stimuli were 184 Chinese words that corresponded to the line drawings. Of these, 24 words and 24 images were probe items and were not included in analyses. The stimuli were divided equally into two sets such that pictures and words referring to the same objects were not in the same set. Participants only viewed one set of the two. All stimuli were presented in white on a black background. Both pictures and words were presented together in a mixed block, arranged in a pseudo-random order to prevent expectation and priming effects. The Chinese characters were in the simplified script, which is predominantly used in mainland China. Half of the Chinese words had one character and half had two characters. In Chinese, most common words are compounded from multiple characters, although some characters can stand alone as words. Regardless of the number of characters, each word referred to one object in this stimulus set.

2.2.3 Procedure

Participants were seated in a comfortable armchair facing a computer monitor in a sound-attenuated room for electrode placement. Each trial started with a fixation cross in the middle of the screen for 500ms and a blank screen for 500ms. The stimulus (a picture or a word) then appeared for 400ms, followed by a 1100ms blank screen and a blink signal for 1700ms. This was followed by another blank screen for 500ms and the fixation cross for the next trial (see Figure 2 for examples of both trial types). There were 92 picture trials and 92 word trials in total. Participants were asked to blink during the blink signal if necessary, and minimize eye movements for the rest of the time. There were two scheduled one-minute breaks during the experiment.

Figure 2.

Two sample trials, one with a word stimulus and one with a picture stimulus. For Experiments 1 and 3 the words were presented in Chinese and for Experiment 2 the words were in English.

A go/no-go semantic categorization task was used to ensure that participants were paying attention and processing the stimuli at a semantic level. Participants were instructed to press a button when they saw either a picture or a word referring to a human body part. These stimuli made up 12% of trials with equal numbers of picture and word body parts. No response was required for non-target stimuli, and only non-target items were averaged in the ERPs reported here.

2.2.4. EEG Recordings

Electroencephalograms were collected using 32-channel caps (Electro-cap International). The tin electrodes were arranged according to International 10–20 system (see Figure 3). In addition, an electrode below the left eye (LE) was used to monitor for blinks and vertical eye movements and an electrode beside the right eye (HE) was used to monitor for horizontal eye movements. Two electrodes were placed behind the ears on the mastoid bone; the left mastoid site (A1) was used as a reference for all electrodes, and the right mastoid site (A2) was recorded to evaluate differential mastoid activity. Impedance was kept at less than 5kΩ for all electrode sites except the lower eye channel, which was below 10kΩ. The EEG was amplified using an SA Bioamplifier (SA Instruments, San Diego, CA) operating on a bandpass of .01 and 40Hz. The digitizing computer continuously sampled the EEG at a rate of 200Hz while a stimulus computer simultaneously presented stimuli to a 19-inch CRT monitor located 54 inches in front of the participant (all stimuli subtended less than 7 degrees of horizontal visual angle).

Figure 3.

32-channel electrode montage used in all three experiments. Sites for average reference N170 and P/N270 analyses are the 4 most lateral sites in each hemisphere connected by thin lines. Sites for mastoid reference N400 analyses are the 12 sites with larger circles.

2.2.5. Data Analysis

Averaged ERPs were computed for all word and picture stimuli for each participant at the 29 scalp sites shown in Figure 3. Epochs with eye movement artifacts between −100 and 600 ms post stimulus onset were excluded prior to averaging. The resulting ERPs were baselined between −100 and 0 ms. Two approaches to analyzing the resulting averaged ERPs were taken. In keeping with the norm in studies of the N170, in one set of analyses the averaged ERPs were referenced to the average of the 29 scalp sites (i.e., average reference -- Joyce & Rossion, 2005). The resulting ERP data were measured by calculating mean amplitudes within two latency windows: 150 to 200 ms and 200 to 300 ms. Conversely, in keeping with the norm of studies focusing on the N400 component, in a second set of analyses the ERP data were referenced to the average of the two mastoid electrodes (mastoid reference). The resulting mastoid reference ERP data were measured by calculating the mean amplitude between 300 and 500 ms.

For both sets of data repeated measures ANOVAs were used with three independent variables: DOMAIN (words vs. pictures), ANTERIOR-POSTERIOR (prefrontal vs. frontal vs. temporal-parietal vs. occipital for average reference data and frontal, vs. central vs. parietal vs. occipital for mastoid reference data) and LATERALITY (left vs. right, for average reference and left vs. midline vs. right for mastoid reference – see Figure 3 for the electrode sites included in each analysis). To correct for nonsphericity of the ERP measurements the Geisser-Greenhouse correction (Geisser & Greenhouse, 1959) was applied to all repeated measures containing more than one degree of freedom in the numerator. Finally, because interactions between groups/conditions and scalp site variables (ANTERIOR-POSTERIOR and/or LATERALITY) can result even when the configuration of the underlying neural generators do not differ (McCarthy & Wood, 1985; Ruchkin et al., 1999; Urbach & Kutas, 2002), we also followed up all significant site by group and site by domain interactions by rescaling the ERP data separately within conditions using a z-score normalization procedure (see Holcomb et al., 1999). Rescaling was then followed up with ANOVAs to see if the same interactions were now significant. Here we report the results of statistical analyses for the original (un-rescaled) data, although only when the rescaled and original interactions were both significant.¹

2.3 Experiment 1 Results

2.3.1 ERP analyses

150–200 ms, N170 epoch (average reference)

As can be seen in Figures 4a and 5a (left), differences between ERPs to Mandarin words and pictures were quite small in this epoch. This observation is supported by a lack of both a main effect of DOMAIN, and other interactions involving this variable (all ps > .45).

Figure 4.

ERPs from L1 Chinese speakers, solid lines are ERPs for pictures of objects and dashed lines are for Chinese words. Stimulus onset is the vertical calibration bar and each tic mark on the x-axis is 100 ms. (a) from the eight electrode sites used in the statistical analyses of the N170 and P/N270 (average reference). (b) six of the 12 electrode sites used in the statistical analyses of the N400 (mastoid reference).

Figure 5.

Voltage Maps based on differences waves calculated by subtracting ERPs recorded to pictures of objects from ERPs recorded to words. (left) N170 epoch, (middle) P/N270 epoch, (right) N400 epoch (note that left and middle plots are with an average reference and the right plot is for a mastoid reference). The top panel (a) is from Experiment 1 where the word stimuli were in Chinese and were read by native Chinese speakers. The middle panel (b) is from Experiment 2 where the word stimuli were in English and were read by native English speakers. The bottom panel (c) is From Experiment 3 where the word stimuli were in Chinese and were read by native English speakers who were naïve of Chinese.

200–300 ms, P/N270 epoch (average reference)

Inspection of Figures 4a and 5a (middle) reveals that there were now large differences in the ERPs to Mandarin words compared to pictures, with words producing a more positive-going response than pictures over anterior electrodes sites and the reverse pattern over posterior sites (labeled P/N270 in Figure 4). This pattern is evident in the significant DOMAIN × ANTERIOR-POSTERIOR interaction (F(3,57) = 26.18, p = .0001) as well as the three way interaction of DOMAIN × ANTERIOR-POSTERIOR × LATERALITY (F(3,57) = 3.10, p = .034). The later interaction is due to a larger left-right asymmetry between words and pictures at the anterior, but not the posterior sites (see Figure 4 and 5a middle). In other words, the greater positivity to words compared to pictures is larger over the right than left anterior sites.

300–500 ms, N400 epoch (mastoid reference)

Figures 4b and 5a (right) show that words produced more negative going ERPs than pictures over left hemisphere sites, but that pictures were more negative than words over right anterior sites between 300 and 500 ms. These effects were supported by the significant two way interaction of DOMAIN × LATERALITY (F(2,38) = 3.58, p = .049 – Figures 4b and 5a right). There were no significant interactions of DOMAIN with the ANTERIOR-POSTERIOR factor (all ps > .5).

2.4 Experiment 1 Discussion

The ERP results showed clear differences in the processing of pictures and Chinese words in the 200–300 ms epoch. Word stimuli produced a more positive-going response than pictures in central and frontal sites, and the opposite pattern appeared at occipital sites. Differences between Chinese words and pictures were also visible in the 300–500 ms time-window. Here, responses to the word and the picture stimuli showed an interaction effect with words eliciting a more negative wave in the left hemisphere, and pictures eliciting a bilateral response.

3. Experiment 2

3.1 Introduction

Experiment 2 was identical to Experiment 1, but English words were used instead of Chinese words and native English speakers were used as participants.

3.2 Methods

3.2.1 Participants

Twenty native English speakers (9 females, mean age 19.1) participated and were compensated for their time. Participants had no prior experience in reading Chinese or related scripts such as Japanese Kanji. All participants were right-handed, with normal or corrected-to-normal visual acuity and no history of neurological insult or language disability.

3.2.2 Stimuli and procedure

The pictures were the same as in Experiment 1, but all the Chinese words were replaced by their direct English translations. The word lengths of the English words ranged from 3 to 13 letters. Task and procedure in Experiment 2 were the same as in Experiment 1.

3.2.3 Data analyses

Data analyses in Experiment 2 were performed in the same way as in Experiment 1.

3.3 Experiment 2 Results

3.3.1 ERP analyses

150–200 ms, N170 epoch (average reference)

As can be seen in Figures 6a and 5b (left), there were large differences between ERPs to English words and pictures in this epoch, with words producing a large left lateralized posterior negativity and anterior positivity. Pictures on the other hand tended to produce a more laterally symmetric response in this epoch. These visual impressions were supported by a significant main effect of DOMAIN (F(1,19) = 8.81, p = .008), and, importantly, a three-way interaction between DOMAIN and the two distributional variables (DOMAIN × ANTERIOR-POSTERIOR × LATERALITY: F(3,57) = 8.91, p = .0005).

Figure 6.

ERPs from L1 English speakers, solid lines are ERPs for pictures of objects and dashed lines are for English words. Stimulus onset is the vertical calibration bar and each tic mark on the x-axis is 100 ms. (a) from the eight electrode sites used in the statistical analyses of the N170 and P/N270 (average reference). (b) six of the 12 electrode sites used in the statistical analyses of the N400 (mastoid reference).

200–300 ms, P/N270 epoch (average reference)

Figures 6a and 5b (middle) reveals that there were also large differences in the ERPs to English words and pictures in the middle epoch, with words continuing to produce a more negative-going response than pictures over posterior electrodes sites and pictures producing a more negative-going response than words over anterior sites (labeled P/N270 in Figure 6a). This pattern is supported by a significant main effect of DOMAIN (F(1,19) = 27.09, p = .0001) and an interaction between DOMAIN and ANTERIOR-POSTERIOR (F(3,57) = 111.88, p = .0001). Unlike the previous epoch, there was not a significant three-way interaction, indicating that word-picture ERP differences were not laterally asymmetrical in this epoch.

300–500 ms, N400 epoch (mastoid reference)

Figures 6b and 5b (right) show that words produced more negative going ERPs overall than pictures (main effect of DOMAIN F(1,19) = 6.17, p = .023). However, this difference tended to be larger over the more posterior sites as revealed by the DOMAIN by ANTERIOR-POSTERIOR interaction (F(3,57) = 11.46, p = .0005).

3.4 Experiment 2 Discussion

Unlike Experiment 1, here we found significant differences between pictures and words in the 150–200 ms epoch. On the other hand, the pattern of effects in the two later time-windows more closely followed that found with Chinese words and pictures in Experiment 1 with the exception of the N400 distribution which tended to be very left dominant for Mandarin words and posterior dominant for English words.

4. Experiment 3

4.1 Introduction

Both Experiments 1 and 2 revealed a clearly larger early frontal positivity for words compared to pictures. And while it would seem most parsimonious to attribute this difference to domain processing differences, it is still possible that differences in low-level visual feature processing between words and pictures might be responsible for the observed ERP effects. To test this possibility, Experiment 3 compared the identical stimuli used in Experiment 1 (Chinese words and pictures) with participants that had no prior experience with Chinese (all participants were native speakers of English). Since the participants were naïve Chinese readers, the Chinese characters were novel meaningless symbols to them, but nevertheless were still composed of the same low-level visual features presented to the native Chinese readers in Experiment 1.

4.2 Methods

4.2.1 Participants

Twenty native English speakers (13 females, mean age 19.4) participated in this experiment for monetary compensation. They were shown the identical pictures and Chinese words as the expert Chinese readers in Experiment 1, and will be referred hereon as naïve Chinese readers. Participants had no prior experience in Chinese or related scripts such as Japanese Kanji.

4.2.2 Stimuli and procedure

Stimuli used in Experiment 3 were the same as in Experiment 1. Task and procedure in Experiment 3 were the same as in Experiment 1.

4.2.3 Data analyses

Data analyses in Experiment 3 were performed in the same way as in Experiment 1.

4.3 Experiment 3 Results

4.3.1 ERP analyses

150–200 ms, N170 epoch (average reference)

As can be seen in Figures 7a and 5c (left), there were no large differences between ERPs to Mandarin words and pictures in this epoch. The main effect of DOMAIN and its interaction with both distributional variables did not approach significance (all ps > .24).

Figure 7.

ERPs from L1 English speakers who were naïve to Chinese, solid lines are ERPs for pictures of objects and dashed lines are for Chinese words. Stimulus onset is the vertical calibration bar and each tic mark on the x-axis is 100 ms. (a) from the eight electrode sites used in the statistical analyses of the N170 and P/N270 (average reference). (b) six of the 12 electrode sites used in the statistical analyses of the N400 (mastoid reference).

200–300 ms, P/N270 epoch (average reference)

Figures 7a and 5c (middle) reveals that there was now a large difference in the ERPs to Mandarin words and pictures, with words producing a consistently more negative-going response than pictures over all sites except the right middle region (main effect of DOMAIN (F(1,19) = 14.26, p = .0013; three-way interaction between DOMAIN × LATERALITY × ANTERIOR-POSTERIOR (F(3,57) = 3.54, p = .02).

300–500 ms, N400 epoch (mastoid reference)

As can be seen in Figures 7b and 5c (right) pictures produced a much more negative-going ERP pattern than words across the scalp in this epoch (main effect of DOMAIN: F(1,19) = 36.22, p < .0001). There was, however, a tendency for this difference to be larger at central and right hemisphere sites ((DOMAIN × ANTERIOR-POSTERIOR × LATERALITY interaction (F(6,114) = 2.79, p = .046).

4.4 Experiment 3 Discussion

In Experiment 3, ERPs elicited by unknown Chinese words differed from pictures starting around 200 ms post-stimulus onset. This difference was apparent at all sites, and was mainly due to Chinese words not producing a large N400-like component in the naïve Chinese readers.

5. Between experiment analyses

We also ran two sets of analyses comparing the ERPs across the three experiments. In the first set we used difference waves calculated by subtracting ERPs recorded to picture stimuli from ERPs recorded to word stimuli. These subtraction wave forms remove global participant effects and allow for a direct examination of group differences due to domain effects. In a second set of group comparisons we directly contrasted the ERPs from the three experiments separately for the two domains (note that parallel analyses using z-score rescaling of the data were also performed and only interaction effects significant in both analyses are reported). In both sets of analyses MM are the data from Experiment 1 with native Mandarin participants viewing Mandarin words and pictures, EE are the data from Experiment 2 with native English participants viewing English words and pictures, and EM are the data from Experiment 3 with native English participants viewing Mandarin words (which they did not know) and pictures.

Experiment (Group) Word – Picture Difference Wave Analyses

150–200 ms, N170 epoch (average reference)

In this epoch there were significant overall differences in the Word minus Picture difference waves between the three GROUPs (F(2,57) = 5.06, p = .01) as well as a GROUP × ANTERIOR-POSTERIOR × LATERALITY interaction (F(6,171) = 5.09, p = .0005). As can be seen in Figure 8a these effects are consistent with the pattern reported earlier for the three groups when analyzed separately, as well as a series of follow-up analyses run on the difference waves contrasting the native English reading English with each of the two other groups (EE vs. MM, GROUP × ANTERIOR-POSTERIOR × LATERALITY interaction: F(3,114) = 3.98, p = .018; EE vs. EM, GROUP × ANTERIOR-POSTERIOR × LATERALITY interaction: F(3,114) = 7.54, p = .0008). These analyses indicate that only the native English viewing English produced a posterior, left greater than right hemisphere negativity to words relative to pictures. The other two groups (MM and EM) did not reveal any word-picture differences in this epoch.

Figure 8.

Difference wave (Word minus Picture) ERPs overlapped for the three participant groups. (a) four of the eight sites used in the analyses if the N170 and P/N270 (average reference). (b) six of the 12 sites used in the analysis of the N400 (mastoid reference).

200–300 ms, P/N270 epoch (average reference)

In this epoch there were again significant differences between the groups in the pattern of ERP difference waves across the scalp (GROUP × ANTERIOR-POSTERIOR: F(6,171) = 24.9, p < .00001; GROUP × ANTERIOR-POSTERIOR × LATERALITY: F(6,171) = 2.8, p = .021). However, the previous experiment-wise analyses and a series of follow-up analyses indicated that it was now the two native reader groups (MM and EE) that showed similar anterior positivities and posterior negativities, while the native English viewing Mandarin group showed a completely different pattern (EE vs. EM, GROUP × ANTERIOR-POSTERIOR × LATERALITY: F(6,171) = 3.55, p = .024; MM vs. EM, GROUP × ANTERIOR-POSTERIOR: F(3,114) = 18.96, p < .00001 – see Figure 8a). Participants in this latter group produced slightly more negative difference waves over left hemisphere and anterior sites and slightly more positive waves or no difference over right hemisphere and middle/posterior sites (see Figure 8a).

300–500 ms, N400 epoch (mastoid reference)

In this epoch there were significant differences between the groups for the word minus picture difference waves (main effect of GROUP: F(2,57) = 22.72, p < .00001) and this effect differed across the scalp (GROUP × ANTERIOR-POSTERIOR: F(6,171) = 2.77, p < .043). Follow-up pairwise analyses suggested that both of the native language groups produced larger negativities in the N400 latency range for words compared to pictures, while the native English group reading Mandarin words showed no evidence of such an effect, instead showing a much larger N400-like negativity to pictures (MM vs EM, main effect of DOMAIN: F(1,38) = 25.9, p < .0001; EE vs. EM, DOMAIN: F(1,38) = 38.43, p < .0001 – Figure 8b). Contrasts between the two groups reading words in their native language revealed that the pattern of negativities differed across the scalp (MM vs. EE, DOMAIN × ANTERIOR-POSTERIOR: F(3,114) = 4.47, p = .023). While the Word-Picture waves tended to be more negative at posterior than anterior sites for English words, the Mandarin words in native Mandarin speakers tended to produce a smaller but more widely distributed negative difference effect.

Experiment (Group) Picture and Word Analyses

We also used a series of analyses to contrast the ERPs for the three participant GROUPs for all picture stimuli (Figure 9a and 10a) and a separate set to contrast the ERPs for the three GROUPs for all word stimuli (Figure 9b and 10b).

Figure 9.

ERPs from four of the eight sites used in the analyses of the N170 and P/N270 (average reference) from each of the three participant groups in Experiments 1 to 3. (a) ERPs for picture stimuli. (b) ERPs for word stimuli.

Figure 10.

ERPs from six of the 12 sites used in the analyses of the N400 epoch (mastoid reference) from each of the three participant groups in Experiments 1 to 3. (a) ERPs for picture stimuli. (b) ERPs for word stimuli.

150–200 ms, N170 epoch (average reference)

As can be seen in Figure 9a the differences in ERPs to pictures for the three GROUPS were quite small and did not reveal any significant main effect of GROUP or interactions involving this factor (all ps > .18). However, for the analysis involving the ERP word data, in both of the native language groups (MM and EE) the negativity in this epoch tended to be larger over the left hemisphere than the right, while the comparable negativity in native English speakers reading unfamiliar Mandarin words (EM) was much smaller and tended to be larger over the right than left hemisphere (GROUP × ANTERIOR-POSTERIOR × LATERALITY interaction: F(6,171) = 5.73, p = .0009). Follow-up analysis contrasting just the two native reader groups (MM vs. EE) revealed that the lateral asymmetry in this negativity at the occipito-temporal sites was greater for English than Mandarin words (GROUP × LATERALITY × ANTERIOR-POSTERIOR interaction: F(3,114) = 5.21, p = .015).

200–300 ms, P/N270 epoch (average reference)

In this epoch there were significant differences between the groups across the scalp for picture stimuli (GROUP × ANTERIOR-POSTERIOR × LATERALITY interaction: F(6,171) = 3.91, p = .009). Follow-up analyses indicated that most of this effect was due to differences between the two English speaking groups (EE vs. EM, GROUP × ANTERIOR-POSTERIOR × LATERALITY interaction: F(3,114) = 6.62, p = .005), with the EE group producing more positive-going ERPs at right posterior sites and the EM group producing the opposite pattern (see Figure 9a). There were also clear differences in the overall scalp distribution of ERPs recorded to the word stimuli between the three groups (GROUP × ANTERIOR-POSTERIOR × LATERALITY interaction: F(6,171) = 10.62, p < .00001). Follow-up analyses indicate that this interaction was due the EM group differing from the other two groups (MM vs. EM, GROUP × ANTERIOR-POSTERIOR interaction: F(3,114) = 27.77, p < .00001; EE vs. EM, GROUP × ANTERIOR-POSTERIOR interaction: F(3,114) = 27.75, p < .00001). As can be seen in Figure 9b, the ERPs to words in language naïve readers were quite distinct from those in either L1 language group. On the other hand, comparisons of the two L1 groups showed a similar overall pattern of larger posterior negativities and corresponding anterior positivities (i.e., there was no main effects of GROUP, p > .18). However, there tended to be a larger left more negative than right asymmetry for English compared to Mandarin for the posterior negativity, but a larger right more positive than left asymmetry for Mandarin than English for the anterior positivity (MM vs. EE, GROUP × ANTERIOR-POSTERIOR × LATERALITY interaction: F(3,114) = 10.14, p = .0004).

300–500 ms, N400 epoch (mastoid reference)

The final epoch also showed differences between the groups for pictures across the scalp (GROUP × LATERALITY × ANTERIOR-POSTERIOR interaction: F(12,342) = 2.39, p = .029). Follow-up analyses revealed that the native Mandarin group produced more positive-going ERPs across the scalp than the two English speaking groups (MM vs. EE, GROUP: F(1,38) = 7.3, p = .01; MM vs. EM, GROUP: F(1,38) = 5.65, p = .023). For words there were also differences for the three groups (main effect of GROUP: F(2,57) = 16.0, p < .0001). The EE group revealed the largest negativity (MM vs. EE, GROUP: F(1,38) = 12.88, p = .0009; EE vs. EM, GROUP: F(1,38) = 28.22, p < .0001). While the MM group produced a smaller negativity to words than the EE group, they nevertheless produced a larger negativity than the EM group (MM vs. EM, GROUP: F(1,38) = 4.78, p = .035 – see Figure 10b).

6. General discussion

The experiments in this study used ERPs to track differences in the time-course of cortical processing between pictures of common objects and words written in Chinese or English. In Experiment 1, native Chinese speakers were tested with Chinese words (logographs) and line drawings of common objects. Experiment 2 tested English native speakers with English words and the same set of line drawings. Experiment 3 tested naïve Chinese readers (native speakers of English) with the same Chinese words and line drawings as in Experiment 1. Apart from the differences between pictures and words observed within each experiment, we were particularly interested in finding three specific patterns revealed in the variation of picture-word differences across all three experiments. These across-experiment comparisons provide a means of plotting the time-course of qualitatively different types of information processing related to script (logographic vs. alphabetic), domain (pictures vs. words), and stimulus meaningfulness (unknown Chinese words vs. pictures of known objects). Clear evidence for these three patterns was found. In the following discussion, we highlight each of these patterns and provide an interpretation of the type of processing that they might reflect. Table 2 provides a summary of the key findings with respect to the three predicted patterns.

Pattern 1. Picture-word ERP differences that have distinct topographies in Experiments 1 and 2, and are not visible in Experiment 3, will reflect script-specific processing. This pattern is visible in the 150–200 ms time window, and more specifically on the N170 component. However, the pattern of picture-word differences on the N170 failed to reveal an effect in Experiment 1 with Chinese readers reading Chinese words, a null effect that was also seen in Experiment 3 with naïve Chinese participants and Chinese words. This implies that the effect seen in Experiment 2, with native English readers reading English words, could simply be due to the greater visual differences between English words and pictures compared with Chinese words and pictures. The between-experiment analysis of the N170 effect to word stimuli across the three groups of participants (see Figure 9b) suggests, however, that this was not the case. N170 amplitude was found to be significantly greater to Chinese words being read by native Chinese compared with naïve Chinese readers, and this N170 effect was left-lateralized, although significantly less so than the N170 generated by English words read by native English readers. The different N170 effect seen to English and Chinese words read by native speakers of each language likely reflects fundamental differences in early orthographic processing of alphabetic versus logographic scripts.

Pattern 2. Picture-word ERP differences that appear in Experiments 1 and 2 but are not seen with naïve Chinese readers in Experiment 3 will reflect domain-specific, script-independent processing. This pattern is visible in the 200–300 ms time-window. It takes the form of more negative-going ERPs to words than pictures at occipital sites, with the opposite pattern emerging at frontal sites (P/N270). These effects onset slightly later than the script-specific effect (pattern 1), and likely reflect the mapping of prelexical form representations onto higher-level representations of known words that operates independently of type of script.

Pattern 3. Picture-word ERP differences that are visible in Experiment 3 and that differ from the patterns seen in Experiments 1 and 2 will reveal domain-independent processing of meaningful objects (pictures and known words). This pattern was most evident in the 300–500 ms time window, where the Chinese words seen by naïve participants produced ERPs that differed from the picture ERPs, but in a very different manner compared with the Chinese words seen by Chinese readers in Experiment 1. The unknown Chinese words generated increased negativity in frontal sites, and reduced negativity in right-hemisphere sites, compared with pictures and known words. This divergence likely reflects access to semantic representations that are common to words and pictures, and obviously not available to meaningless stimuli.

6.1 Script-specific processing – N170

Our between-experiment analyses revealed the standard expertise-dependent N170 effect to word stimuli. The results fit perfectly with those reported by Maurer et al. (2008), this time obtained with a semantic categorization task rather than the one-back repetition task used by Maurer et al. A left-lateralized N170 response was found to both English words and Chinese words when read by native readers of each language compared with the smaller and more right-lateralized response to Chinese words read by English speakers with no knowledge of Chinese. Also in line with the pattern reported by Maurer et al. is the fact that the left-lateralization was stronger for English words than Chinese words (although this was only marginal in the Maurer et al. study).

The pattern of N170 effects seen in the present study lends some support to the hypothesis that alphabetic scripts engage more word-specific processes than logographic scripts in early phases of processing. More precisely, we interpret these early differences between English and Chinese words as reflecting the different mechanisms that are involved in mapping visual features onto higher-level linguistic representations in different scripts. Alphabetic scripts are thought to automatically engage a specialized mechanism for parallel independent letter identification that is a key ingredient of skilled word recognition in languages such as English (e.g., Grainger et al., 2010; Tydgat & Grainger, 2009). Although Chinese word recognition also involves the simultaneous mapping of visual features onto a componential structure in the form of radicals (in words with more than one radical) and characters (in multi-character words - see Taft et al., 1999), at least part of this process involves a direct mapping of features onto meaning-bearing units (i.e., semantic radicals). This implies that logographic scripts might retain more of the basic processes associated with visual object recognition than do alphabetic scripts. These conclusions fit with the proposal that the earliest phases of orthographic processing are performed by neural structures in left posterior fusiform gyrus (e.g., Cohen & Dehaene, 2004; Dehaene, Cohen, Sigman, & Vinckier, 2005; Nobre, Allison, & McCarthy, 1994), combined with the evidence that the N170 has its source in posterior fusiform gyri, with the relative involvement of each hemisphere depending on the type of stimulus (e.g., Brem, Bucher, Halder, Summers, Dietrich, Martin, & Brandeis, 2006; Halgren, Raij, Marinkovic, Jousmäki, & Hari, 2000; Rossion et al., 2003; Tarkianen, Helenius, Hansen, Cornelissen, & Salmelin, 1999).

6.2 Domain-specific, script-independent processing – P/N270

The pattern of results seen in the 200–300 ms time window appears to reflect word-specific processing that is independent of script. Both Chinese and English words generated more positive-going waveforms than pictures at frontal sites, and more negative-going waveforms at occipital sites, both peaking between 250 and 300 ms. One obvious candidate for such word-specific, script-independent processing would be abstract lexical representations that are independent of visual format. These could be whole-word orthographic / logographic representations or whole-word phonological representations. In line with this proposal is the “graphemic” priming effect reported by Liu, Perfetti, and Hart (2003) in an experiment testing skilled Chinese readers with Chinese words. They found that the amplitude of an early positive-going component (referred to as the P200) was reduced when primes shared a radical with Chinese target words. We therefore tentatively interpret this commonality in the processing of Chinese logographs and English words, found in the present study, as reflecting rapid access to abstract lexical representations from print. In both logographic and alphabetic scripts, this would involve associating key parts of the word (e.g., a radical or part of a radical for a logograph, and combinations of letters for the alphabetic script) with abstract lexical representations. The time-course and scalp distribution of this ERP activity is consistent with the hypothesis of neural sources located in more anterior regions of the fusiform gyrus, as well as left temporal regions thought to be involved in the processing of whole-word orthographic representations and their connectivity with semantics (Fujimaki et al., 2010;).

The effects seen in this time-window can be linked to prior research examining effects of expertise on an ERP component (the N250) that has a similar latency and spatial distribution as the negative-going part of the P/N270 of the present study. N250 amplitude is modified by presentation of known faces but not unfamiliar faces, but the response to unfamiliar stimuli is modified by stimulus repetition during the experiment (Tanaka, Curran, Porterfield, & Collins, 2006). Furthermore, the N250 was found to be sensitive to training at the subordinate-level of categories (e.g., heron, snowy owl) but not the basic-level (e.g., wading bird, owl - Scott, Tanaka, Sheinberg, & Curran, 2006; 2008). Reading individual words for meaning is analogous to processing subordinate-level category information, where the basic-level corresponds to the category of all words in a given language. Thus the N250 seen in the above-cited studies, and the present P/N270, would both reflect processing of abstract (i.e., view-invariant, script-invariant) representations of known objects, at a level where fine details are critical for discriminating between different exemplars.

The results of the present study can also be compared with prior research reporting the presence of a picture-specific ERP component, a negative-going waveform peaking before the N400, referred to as the N300 (Holcomb & McPherson, 1994). This pattern was seen for the picture stimuli in Experiment 2, with pictures generating two negative peaks at anterior sites (the N300 and the N400) while words generated one negative peak (the N400). Although the pattern was less evident for the picture stimuli in Experiments 1 and 3, the waveforms generated by picture stimuli in a 200–400 ms time-window were systematically very distinct from the word stimuli in all three experiments. The absence of a clear N300 in Experiment 1 testing Chinese-speaking participants could be due to culturally induced differences in picture processing. Indeed, an overall comparison of the ERP waveforms to picture stimuli in all three experiments shows some major differences. For example, in Experiment 1 there is a very pronounced negative-going wave peaking around 250 ms at frontal sites (Figure 4b) that is less apparent in Experiments 2 and 3 where the picture N300 dominates the frontal sites (compare Figures 4b, 6b and 7b). This therefore appears to be evidence that Chinese readers might process pictures differently compared with English readers, either because of an influence on specific language expertise on picture processing, or possibly because of cultural differences in the relative familiarity of the pictures tested in the present study.

6.3 Domain-general processing of meaningful stimuli- N400

Meaningfulness of stimuli was found to have a widespread effect on ERP amplitudes starting around 300 ms post-stimulus onset. N400 amplitude was found to be greatly reduced to the unknown Chinese words tested in Experiment 3, compared with the picture stimuli tested in the same experiment and the known Chinese and English words tested in Experiments 1 and 2. Given the widespread nature of this effect and the fact that it was relatively long-lasting (roughly 300–500 ms), it likely reflects more than just one type of processing. The unknown Chinese words differed from the familiar stimuli on at least two dimensions - visual familiarity and semantic interpretability. Although failure to activate semantic representations in the case of unfamiliar stimuli is likely to be one major cause of the reduced negativity in the time-window of the N400 ERP component, this does not exclude a role for the absence of any type of higher-order form representation for these stimuli. That is, an absence of anything akin to lexical form representations for words, and structural representations for pictures. In support of such an interpretation, Holcomb and Grainger (2006; 2007) have argued that the N400 seen in single word paradigms (as opposed to sentence processing studies) reflects the mapping of whole-word form representations onto semantics.

The results of Experiment 3 might seem to contradict those found in previous studies reporting that pseudo-words and pseudo-objects tend to generate larger rather than smaller N400s (Holcomb & Neville, 1990; McPherson & Holcomb, 1999). However, the results in this experiment make more sense if they are compared to studies that contrasted word and word-like stimuli to items that are not plausible linguistic representations within the reader's language system. For example, Holcomb and Neville (1990) showed that while pseudo-words and real words produced a large N400, random letter strings (without vowels) produced almost no negativity in the N400 epoch. Their interpretation was that participants do not attempt semantic analysis on stimuli that do not follow the compositional rules of language. The Chinese characters in Experiment 3 were outside the familiar writing system of participants and did not resemble real objects, and therefore were unlikely candidates for semantic analysis either as words or objects.

6.4 Conclusions

Processing differences between line-drawings of common objects and English and Chinese words were revealed in the ERP waveforms generated by these different types of stimuli. The observed pattern of picture-word differences were interpreted as reflecting: i) script-specific processing due to differences in the way general object-processing mechanisms are adapted to optimize processing of words written in different scripts; ii) word-specific but script-independent processing due to fundamental differences in the way visual features map onto higher-level representations for pictures and words; and iii) processing that depends on stimulus familiarity due to the absence of higher level structural and semantic representations for unknown stimuli. Finally, from a methodological point of view, the present study demonstrates the gain in interpretational power that can be achieved by introducing words written in physically different formats when evaluating differences between pictures and words.

Figure 1.

Summary of the conditions tested in Experiments 1–3 and the logic behind the cross-experiment comparisons. For a given time-window in the ERP analyses, a picture-word ERP difference is observed in each experiment, and can be associated with differences in the picture and word stimuli presented in each experiment (stimulus differences). These differences can be visual, and therefore not the same for Chinese and English words (visual A, visual B). They can be related to expertise with a given script (script A, script B), related to differences in domain (pictures, words), or whether the stimuli are meaningful or not. Contrasting the observed ERP picture-word differences across experiments can help isolate effects that are likely to be driven by one particular stimulus difference.

Table 1.

Summary of findings supporting the three patterns of picture-word ERP differences.

Pattern 1	Left-lateralized N170 component to words in MM and EE groups only. The component was greater for EE group compared to MM group.	Likely reflects fundamental differences in early orthographic processing of alphabetic versus logographic scripts.
Pattern 2	Posterior negativity and anterior positivity for words compared to pictures (P/N270) seen in MM and EE groups only.	Likely reflects mapping of prelexical form representations onto higher-level word representations that is script-independent.
Pattern 3	Different pattern of N400 elicited by words and pictures in EM group compared to MM group.	Likely reflects access to semantic representations that are common to words and pictures, but not available to meaningless stimuli.