Masked Cross-Modal Repetition Priming: An Event-Related Potential Investigation

Abstract

We report three experiments that combine the masked priming paradigm with the recording of event-related potentials in order to examine the time-course of cross-modal interactions during word recognition. Visually presented masked primes preceded either visually or auditorily presented targets that were or were not the same word as the prime. Experiment 1 used the lexical decision task, and in Experiments 2 and 3 participants monitored target words for animal names. The results show a strong modulation of the N400 and an earlier ERP component (N250 ms) in within-modality (visual-visual) repetition priming, and a much weaker and later N400-like effect (400–700 ms) in the cross-modal (visual-auditory) condition with prime exposures of 50 ms (Experiments 1 & 2). With a prime duration of 67 ms (Experiment 3), cross-modal ERP priming effects arose earlier during the traditional N400 epoch (300–500 ms) and were also larger overall than at the shorter prime duration.

Masked Cross-Modal Repetition Priming: An Event-Related Potential Investigation

There is increasing evidence from studies of word recognition in both the visual and auditory modalities that information associated with the non-presented modality can affect the recognition process. Thus, for example, it is now commonly accepted that recognition of a printed word is influenced by information concerning its pronunciation (e.g., Frost, 1998). This accumulation of empirical evidence has led to the development of models of word recognition that allow strong interactivity across modality-specific representations. For example, Grainger and Ferrand (1994; 1996), proposed an extension of McClelland and Rumelhart’s (1981) interactive-activation model that included both sublexical and lexical-level connections between orthographic and phonological representations (a recent version of this model is shown in Figure 1). In the bimodal interactive-activation model described in Figure 1, presentation of a visual word stimulus generates activation in orthographic codes, which rapidly activate the corresponding phonological codes and subsequently influence the recognition process. The same holds for auditory word recognition, where phonological codes rapidly activate the corresponding orthographic representations.

Figure 1.

Architecture of a bi-modal interactive-activation model of word recognition in which a sublexical/lexical division is imposed on orthographic (O) and phonological (P) representations. In this architecture, orthography and phonology communicate directly at the level of whole-word representations (O-words, P-words), and also via a sublexical interface (O ⬄ P). Higher-level semantic representations (not shown) receive activation from whole-word orthographic and phonological representations (the details of inhibitory within-level and between-level connections are not shown).

Such theoretical developments were initially motivated by the growing evidence, as noted above, that phonological information does indeed affect the recognition of visually presented words. For example, heterographic homophones (orthographically and semantically distinct words that share the same pronunciation, e.g., MAID-MADE) are generally harder to recognize than non-homophonic words (Ferrand & Grainger, 2003; Pexman, Lupker, & Jared, 2001), and are more prone to semantic confusions (Van Orden, 1987). This homophone disadvantage is taken to reflect competition driven by mandatory phonological processing during visual word recognition. Furthermore, the evidence obtained from the masked priming paradigm clearly shows that these phonological influences on visual word recognition arise quite early (Frost, Ahissar, Gotesman, & Tayeb, 2003; Perfetti & Bell, 1991), and lag only an estimated 20–30 ms behind the influence of orthographic variables (Ferrand & Grainger, 1993; Ziegler, Ferrand, Jacobs, Rey, & Grainger, 2000). Rapid and mandatory phonological influences on visual word recognition have also been found in scripts where the link between visual and auditory language is much less transparent (such as Chinese: Tan & Perfetti, 1999; Spinks, Liu, Perfetti, & Tan, 2000).

There is also clear evidence that spoken word recognition is influenced by the orthographic characteristics of word stimuli. Early research had shown orthographic influences on adult performance when judging if two words rhymed, such that orthographically different rhymes (e.g., RYE-TIE) were more difficult to match than orthographically identical rhymes (Seidenberg & Tanenhaus, 1979). Similarly, priming across two auditorily presented words was only found to be robust when the phonological overlap also involved orthographic overlap (Jakimik, Cole & Rudnicky, 1985; see also Slowiaczek, Soltano, Wieting, and Bishop, 2003). Furthermore, Ziegler and Ferrand (1998) have shown that auditory lexical decision responses were longer to words with rimes that can be spelled in different ways (e.g., in French, the phonological rime/o/in the target word “dos” can be written os, ot, aux, eau, eaux) compared to words with rimes having a unique spelling. Ventura, Morais, Pattamadilok, and Kolinsky (2004) replicated and extended this effect of sound-to-spelling consistency in spoken word recognition in Portuguese. In a similar line of investigation, Ziegler, Muneaux, and Grainger (2003) have shown that spoken word recognition is affected by the number of orthographic neighbors of target words, while controlling for their phonological neighborhood.

Masked Cross-modal priming

Another line of recent research that has provided particularly strong support for cross-modal interactions in word recognition has used a cross-modal variant of the masked priming paradigm, in which prime stimuli are presented visually and targets auditorily. Compared to standard cross-modal priming, the masked priming variant has the advantage of preventing contamination from strategic factors that play a role a soon as participants become aware of the priming manipulation (Forster, 1998). In the first published application of the cross-modal version of the paradigm, Kouider and Dupoux (2001) found significant effects of masked visual primes on the recognition of auditorily presented target words with prime durations of 64 ms or longer. Since prime visibility measures showed above chance performance in these conditions, Kouider and Dupoux argued that cross-modal priming effects only arise when prime stimuli are available for conscious awareness.

However, Grainger et al. (2003) extended this observation to even shorter prime durations, finding significant effects of visual primes on auditory target processing at prime exposure durations of only 50 ms. In the Grainger et al. study, cross-modal effects of pseudohomophone primes (e.g., “brane” as a prime for the target word “brain”) emerged with longer exposure durations (67 ms), equivalent to the duration necessary for obtaining within-modality (visual-visual) pseudohomophone priming. Grainger et al., found that the size of cross-modal priming effects did not vary as a function of prime visibility measures.

These behavioral studies provide an important initial estimation of the time-course of cross-modal transfer from the visual to auditory modalities during word recognition. This estimation is based on priming effect sizes that vary as a function of the type of priming manipulation (e.g., repetition vs. pseudohomophones), and prime duration. However, manipulations of prime duration in behavioral experimentation only provide an indirect estimate of the time-course of cross-modal interactions relative to processing within a given modality. This is because a) the prime duration manipulation can only, at best, provide relative information about the time-course of information processing since prime processing most likely extends beyond prime offset; and b) behavioral responses reflect information accumulation over a period of time extending well beyond word recognition. Therefore, the present study uses event-related brain potentials (ERPs), with their enhanced temporal resolution, as a means of providing more precise information about the time-course of cross-modal interactions.

Event-related brain potentials

The enhanced temporal resolution of ERPs comes through their ability to provide a continuous measure of processing starting before stimulus onset and continuing on hundreds of milliseconds later. We therefore reasoned that a combination of masked priming and ERP recordings would provide the measurement sensitivity that would allow us to provide a more accurate description of the time-course of within-modality and across-modality repetition priming.

Evidence that ERPs should be sensitive to such effects comes from a growing literature suggesting that supraliminal priming produces characteristic differences in ERPs. The typical effect is that repeated or semantically related words result in the attenuation of the N400 component compared to unrelated words (e.g., Rugg, 1985; Rugg & Nagy, 1989; Rugg, 1990; Rugg, Doyle & Wells, 1995; Holcomb, 1993; Holcomb, Anderson & Grainger, 2005; Van Petten, Kutas, Kluender, Mitchiner & McIsaac, 1991; Düzel, Yonelinas, Mangun, Heinze & Tulving, 1997; Misra & Holcomb, 2003). Such N400 effects are widely believed to result from the processing of the lexico-semantic properties of the stimulus and its context (Van Petten et al., 1991), the most dominant view being that N400 amplitude is proportional to the difficulty of semantic integration. According to this view the N400 is relatively small when the meaning of a word is easily integrated into a prior context and is larger when integration is more difficult (e.g., Holcomb, 1993).

Relevant to the current study is a series of studies that contrasted ERP priming effects within and across modalities. Anderson and Holcomb (1995) and Holcomb and Anderson (1993) demonstrated that the time-course of semantic priming effects in the N400 region were very similar for spoken word targets whether they were primed by a visual word or another auditory word. In a more recent study, Holcomb et al. (2005a) extended these observations to the case of cross-modality repetition priming. Most important, when the prime was visual there were large and early (between 200 and 300 ms) ERP repetition effects for spoken word targets, and these effects even extended to a condition where the onsets of the visual prime and auditory target were simultaneous. This study suggests that there is a rapid interaction of word processes between the modalities and that ERPs are sensitive to at least some aspect of this interaction.

Also relevant to the present study are four recent reports that ERPs are sensitive to repetition priming effects even when the prime stimulus is masked (Schnyer, Allen, & Forster, 1997; Misra & Holcomb, 2003; Holcomb, Reder, Misra, & Grainger, 2005; Holcomb & Grainger, in press). Importantly, in the Misra and Holcomb study, ERP priming effects with 50 ms masked primes began as early as 200 ms post-target onset and continued through the traditional N400 epoch (300 to 500 ms). Consistent with this pattern is the conclusion by Holcomb, O’Rourke and Grainger (2002), who argued that at least some aspect of the early phase of the N400 is sensitive to word processing at the form-meaning interface. Furthermore, in testing conditions very similar to those used in the present study, Holcomb and Grainger (in press) found evidence for repetition priming effects in an earlier negative-going component peaking at around 250 ms post-target onset, which they called the N250.

The Current Study

In the current study we examined ERP masked repetition priming both within and between modalities. Native French speakers were presented with brief (50 ms in Experiments 1 & 2, 67 ms in Experiment 3) visual prime words that were masked by both a forward mask and a backward random consonant string mask and were rapidly (13 ms) followed by either a repeated target word (e.g., word - word) or an unrelated target word (e.g., hand - word). Targets could be in either the visual or auditory modality (note that all items were in French). In Experiment 1 subjects performed a lexical decision task. This allowed us to replicate the behavioral pattern obtained by Grainger et al. (2003) while simultaneously measuring ERPs. In Experiments 2 and 3, subjects were told to read/listen to all stimuli and press a button to occasional probe words in a particular semantic category (animals – semantic categorization task). ERPs were time-locked to the onset of prime words and recorded for 1000 ms after the onset of target words.

Between-Modality Priming Predictions

The predictions for these experiments are best expressed in terms of the architecture for word recognition described in Figure 1, and our knowledge of the mechanisms governing cross-modal transfer given prior behavioral results. Figure 2 describes the hypothetical strength of activation flow at different points in the architecture following a visually presented prime word at two different prime durations (corresponding to the durations tested in the present study, and those tested in the Grainger et al., 2003, study).

Figure 2.

Hypothesized state of activation flow in the bi-modal interactive-activation network following the visual presentation of a word for 50 ms (left panel) and 67 ms (right panel) followed by backward masking. The darkness and thickness of lines reflects the amount of activation flow, with strongest activation between O-units and O-words and between O-units and the O-P sublexical interface. Intermediate levels of activation are hypothesized to arise at the lexical O-P interface, between O-words and P-words.

This framework makes two clear predictions concerning the relative time-course of within-modality and across-modality repetition priming effects in ERPs.

1) Lexical-level pathways (i.e., O-word – P-word connections) dominate cross-modal transfer

In prior behavioral research, cross-modal repetition priming occurs with lower prime durations than necessary to obtain cross-modal pseudohomophone priming. This is explained within the bi-modal interactive-activation framework by the fact that pseudohomophones mainly exert their influence via the sublexical O-P interface, and this route to whole-word phonology takes longer than the route via whole-word orthographic representations (see Figure 2). Therefore, we expect to observe effects of within-modality repetition priming appearing in relatively early ERP components (pre-N400), whereas cross-modal priming should only be reflected in later components associated with activation at the level of whole-word representations (i.e., the N400). Given prior priming research measuring ERPs (e.g., Misra & Holcomb, 2003), we expect priming effects to be reflected in a reduced negativity in the related prime condition.

2) Cross-modal transfer also operates sublexically

Although the architecture does not allow direct transfer from letters (O-units) to low-level acoustic/phonetic representations involved in spoken word recognition (P-units), the sublexical (O-P) interface does connect up to such representations. Therefore in conditions where pseudohomophones have been shown to influence behavioral responses in cross-modal priming (i.e., 67 prime durations), we expect to observe earlier cross-modal influences on spoken word recognition (see right panel of Figure 2).

Within-modality predictions

We expected that within-modality visual masked repetition effects should replicate those reported by Holcomb and Grainger (in press), and show both an early ERP repetition effect in the N250 as well as a later effect on the main portion of the N400.

EXPERIMENT 1

Method

Participants

Twenty-four volunteers were paid 25 euros to participate in this experiment. All were right-handed native speakers of French reporting normal visual and auditory acuity (17 female, mean age = 21).

Stimuli

The critical stimuli for this experiment were formed from 200 pairs of French words and 200 pairs of nonwords. Pairs of words were matched for length (four to six letters, average length 5.3 letters; auditory word stimuli: mean duration 640 ms, range 237–1007 ms) and frequency (between 10 and 431 per million; New, Pallier, Ferrand, and Matos, 2001) and had no clear semantic or form overlap. The 200 pairs of nonwords were generated from French words and although they could be pronounced according to French grapheme-phoneme conversion rules, they were neither French words nor pseudohomophones. The 200 pairs of nonwords were matched for length (four to six letters, average length 5.3 letters; auditory nonword stimuli: mean duration 610 ms, range 340–980 ms) and had no clear form overlap. For both words and nonword pairs, the first member of each pair was referred to as the prime and the second member as the target. Stimulus lists consisted of 400 trials each containing a prime-target pair of letter strings. In each list, there were 100 pairs of repeated words (e.g. courir-courir), 100 pairs of unrelated words (e.g. paquet-courir), 100 pairs of repeated nonwords (e.g. tudais-tudais), and 100 pairs of unrelated nonwords (e.g. cagnot-tudais). In addition to the two levels of repetition (repeated/unrelated) and the two levels of word type (word/nonword), there was a third factor of target modality (visual and auditory) which resulted in four types of trials (50 pairs/condition – see Table 1). Four counter-balanced lists were created so that across subjects, target words and nonwords appeared once in each of the four conditions. Within subjects, each list (and therefore each target stimulus) was presented once.

TABLE 1.

Experiment 1 Trial Types

	description	target modality	mask	prime	mask	target
1	repeated word	visual	###########	courir	KGBVDPSFZ	courir
2	repeated word	auditory	###########	courir	KGBVDPSFZ	courir
3	unrelated word	visual	###########	paquet	KGBVDPSFZ	courir
4	unrelated word	auditory	###########	paquet	KGBVDPSFZ	courir
5	repeated nonword	visual	###########	tudais	PSKZGNVFJ	tudais
6	repeated nonword	auditory	###########	tudais	PSKZGNVFJ	tudais
7	unrelated nonword	visual	###########	cagnot	PSKZGNVFJ	tudais
8	unrelated nonword	auditory	###########	cagnot	PSKZGNVFJ	tudais

In a lexical decision task, participants were instructed to rapidly classify each target stimulus as a word or a nonword by pressing a ‘oui’ (yes) or ‘non’ (no) button on a handset. The ‘oui’ and ‘non’ response hand was counterbalanced across subjects and lists. Before the experiment, participants were familiarized with the experiment during a practice session.

Procedure

Visual stimuli were presented on a 17” monitor and auditory stimuli binaurally through headphones. Visual stimuli were displayed as white letters on a black background in the Arial font using software written in Microsoft Visual C++ and timing controlled to the nearest millisecond with a photodiode and oscilloscope. All auditory stimuli were recorded by a female native French speaker in a sound-attenuated room and edited using GoldWave digital audio editor version 4.26. Each trial began with the presentation of a fixation point in the middle of the screen. Five hundred milliseconds later, a forward mask was presented (11 hash marks) for a duration of 500ms. The forward mask was replaced at the same location on the screen by a four to six letter lower case prime word for 50ms. The prime was immediately replaced by a 9-letter consonant string (backward mask – e.g., DFWZHMPZC). For each word pair, a fixed backward mask was constructed that was used for all 4 conditions tested with that target. The backward mask had no clear-form overlap with the prime or the target. For within-modality trials (i.e., visual-visual), the backward mask remained on the screen for 13ms and was immediately replaced by the visual target. Visual targets were presented for 600ms. For between-modality trials (i.e., visual prime and auditory target), the auditory target was presented 13ms after the onset of the backward mask, but in this case the backward mask remained on the screen until the end of the trial. The purpose of the enduring backward mask was to provide a fixation point on the screen (which reduces eye movements during the between-modality trials) and also to make sure that the prime was effectively masked (note on visual trials the visual target serves as an additional backward mask). Visual and auditory trials were randomly intermixed. All target words were followed by a 1000ms black screen which was replaced by a 2000ms fixation point marking the end of the trial. Subjects were asked to blink only when the fixation point appeared on the screen in order to minimize eye blink artifact.

After the main ERP experiment, each participant performed a mask-effectiveness test. The purpose of this test was to establish an upper limit on the visibility of masked primes during the ERP experiment and as such, participants were asked to explicitly judge the masked stimuli themselves. Note that this approach is likely to provide an overestimate of the visibility of primes during the ERP experiment as participants were explicitly focused on detecting the identity of the masked items whereas unmasked 600 ms targets were the main focus during the ERP experiment. Fifteen words and fifteen nonwords were presented for 50ms each and preceded by a 500 ms forward mask and a 13 ms backward mask. None of these items appeared as a prime or target in the ERP experiment. Participants were asked to classify each of the masked stimuli as a word or a nonword by pressing a ‘oui’ or a ‘non’ button on a handset in a two-alternative forced-choice procedure.

EEG Recording Procedure

Participants were seated in a comfortable chair in a sound attenuated darkened room. The electroencephalogram (EEG) was recorded from 29 active tin electrodes held in place on the scalp by an elastic cap (Electrode-Cap International). Scalp locations included those over the left and right fronto-polar (FP1/FP2), frontal (F3/F4, F7/F8), frontal-central (FC1/FC2, FC5/FC6), central (C3/C4), temporal (T5/T6, T3/T4), central-parietal (CP1/CP2, CP5/CP6), parietal (P3/P4), and occipital (O1/O2) areas and five midline sites over the frontal pole (FPz), frontal (Fz), central (Cz), parietal (Pz) and occipital (Oz) areas (see Figure 3). In addition, four electrodes were attached to the face and neck area: one below the left eye (to monitor for vertical eye movement/blinks), one to the right of the right eye (to monitor for horizontal eye movements), one over the left mastoid (reference) and one over the right mastoid (recorded actively to monitor for differential mastoid activity). All EEG electrode impedances were maintained below 5 kΩ (impedance for eye electrodes was less than 10 kΩ). The EEG was amplified by an SA Bioamplifier with a bandpass of 0.01 and 40 Hz and the EEG was continuously sampled at a rate of 200 Hz.

Figure 3.

Electrode montage used in the present study.

Data analysis

Averaged ERPs were formed off-line from trials free of ocular and muscular artifact. Separate waveforms were calculated by averaging two-word epoch ERPs to prime and target words based on the two repetition conditions and modality. Mean amplitudes for four post-target onset latency windows were used to quantify the ERPs in this experiment: 0–150 ms, 200–300 ms, 300–500 ms, and 550–750 ms.

Four separate repeated measures analyses of variance (ANOVAs) were conducted using the within-subject factors of repetition (repeated vs. unrelated) and modality (visual vs. auditory): midline (FPz, Fz, Cz, Pz, Oz), column 1 (FC1, C3, CP1, FC2, C4, CP2), column 2 (F3, FC5, CP5, P3, F4, FC6, CP6, P4) and column 3 (FP1, F7, T3, T5, O1, FP2, F8, T4, T6, O2 – see Figure 3 for the locations of each column). Columns 1, 2 and 3 also included a factor of hemisphere (left/right). The Geisser-Greenhouse (1959) correction was applied to all repeated measures with more than one degree of freedom.

Results

Behavioral Data

All correct responses between 200ms and 2000ms post-target onset (95% of the data) were included in the behavioral analyses. The mean reaction times, percentage of errors, and standard error of the mean were calculated for each subject.

Mean RTs and percentage of errors for each of the four target types are presented in Table 2. Across the conditions of target modality and repetition, word targets were responded to more quickly than nonword targets (F(1,23) = 160.178, p < 0.001). There was also a main effect of target modality, with subjects responding more quickly to visual targets than auditory targets (F(1,23) = 180.941, p < 0.001) and a main effect of repetition, with subjects responding more quickly to repeated targets than to unrelated targets (F(1,23) = 7.227, p = 0.013). Simple effects ANOVAs revealed that the repetition effect was reliable for visual words (39ms effect, F(1,23) = 20.176, p < 0.001), and auditory words (40ms effect, F(1,23) = 14.957, p < 0.001), but no interaction between target modality and repetition (F(1,23) = 0.004, p < 0.947).

TABLE 2.

Experiment 1 Mean Reaction Times (ms) and Percentage of Errors

			Within Modality			Across Modality
		Mean RT	Std. Error Mean	% Errors	Mean RT	Std. Error Mean	% Errors
Words	Repeated	744.969	20.87	4	949.140	20.26	4.25
	Unrelated	784.039	19.89	2	988.921	17.95	3.67
Nonwords	Repeated	906.107	24.71	5.47	1105.283	25.66	5.5
	Unrelated	926.62	25.18	5.58	1081.11	21.89	8.42

Subjects made more errors to nonword targets than to word targets (F(1,23) = 10.932, p = 0.003). There were also more errors made to auditory targets than to visual targets (F(1,23) = 4.221, p = 0.051). There were no significant differences in the number of errors made to repeated and unrelated targets (F(1,23) = 0.079, p < 0.781).

In the mask-effectiveness test of forced-choice lexical decision performance with masked words and nonwords, participants correctly classified 53% of words and 57% of nonwords. The average d’ value was 0.11, and this did not significantly differ from zero (t(23)=1.19, p =0.24).

Electrophysiological Data

On average 4.2% of the trials were rejected due to artifact which left an average of 47 trials per subject and per condition.

Visual Inspection of ERPs

The two-word ERPs (prime and target) for the two target modalities for word trials are plotted in Figures 4 and 5. These figures reveal both similarities as well as numerous differences between the two target modalities regarding the morphology and distribution of components across the scalp. It should be kept in mind that modality differences in ERPs are to be expected, especially in the earlier exogenous ERP components (N1, P2) because of the different anatomical locations and demands for visual and auditory processing. However, such modality differences are likely somewhat muted here because the waves in Figure 5 are an amalgamation of the ERPs to the auditory target and the backward visual mask (the latter of which was a constant across conditions). For ERPs to both visual and auditory targets, one of the first visible components is a central-anterior negativity (N1) peaking around 100 ms post-target onset (larger and somewhat later for auditory than visual). Following the N1 there was a large widely distributed positivity which peaked at about 200 ms (P2). The P2 component was followed by a negative-going wave that was centrally located and peaked at approximately 250 ms (N250) for visual targets. For auditory targets, the negativity in this same general latency range was more broadly distributed across the scalp (with the exception of the most posterior sites) and peaked at approximately 275 ms. Following the 250–275 ms epoch there was a broad negative-going component peaking between 400 and 500 ms for visual targets and 450 and 550 ms for auditory targets. This negativity, which overlapped the window normally associated with the N400, had a broad scalp distribution and was largest (i.e. was most negative) at the central-anterior electrode sites. Finally, following the N400 was a positive component, which had a central-posterior distribution for visual targets.

Figure 4.

Experiment 1 ERPs from 29 scalp sites to Repeated (solid) and Unrelated (dashed) prime-TARGET pairs in which the prime was a masked visual word and the target a clearly visible visual word. Target onset is marked by the vertical calibration bar and prime onset was always 67 ms earlier (as indicated by the arrow labeled “P” on the time legend in the lower left hand corner). Relative electrode positions are indicated on the schematic head in the upper left hand corner. Negative voltages are plotted up.

Figure 5.

Experiment 1 ERPs from 29 scalp sites to Repeated (solid) and Unrelated (dashed) prime-TARGET pairs in which the prime was a masked visual word and the target an auditory word. Target onset is marked by the vertical calibration bar and prime onset was always 67 ms earlier (as indicated by the arrow labeled “P” on the time legend in the lower left hand corner). Relative electrode positions are indicated on the schematic head in the upper left hand corner. Negative voltages are plotted up.

Analyses of ERP Data

0–150 ms Target Epoch

An examination of Figures 4 and 5 suggests that differences between repeated and unrelated targets did not begin until the end of this epoch. This observation was supported by a lack of a significant repetition effect in either modality (all Fs < 1.5).

200–300 ms Target Epoch

For visual words unrelated targets elicited a significantly more negative-going ERP (see Figure 4) than related targets (main effect of repetition, visual words: midline: F(1,23)=15.22, p< 0.001; C1: F(1,23)= 15.14, p<0.001; C2: F(1,23)=9.96, p= 0.004; C3: F(1,23)= 5.16, p=0.033) and at lateral sites this effect tended to be larger over the right than left hemisphere at posterior temporal and occipital sites (repetition × hemisphere × electrode site interaction, C3: F(4.92) = 4.37, p = .008). For auditory word targets there were no main effects of repetition, but there were significant interactions between repetition and hemisphere at all three lateral columns (repetition × hemisphere: C1: F(4,92)= 4.382, p= 0.045; C2: F(4,92)= 4.71, p= 0.041; C3: F(4,92)= 5.71, p= 0.025). This interaction resulted from unrelated auditory words having elicited a slightly greater positivity than repeated auditory words, especially over the right hemisphere (see Figure 5).

300–500 ms Target Epoch

For the visual word ERPs, unrelated visual words continued to be significantly more negative-going than repeated visual words at all but the most lateral sites (midline: F(1,23)= 6.36, p= 0.019; C1: F(1,23)= 7.80, p= 0.007; C2: F(1,23)= 3.01, p= 0.096). Typical of the classic N400 effect, for visual word ERPs there was an interaction between repetition and electrode site, with the effect being most pronounced at the more central and posterior sites (repetition × electrode site interaction: midline: F(4,92)= 9.30, p< 0.001 – see Figure 4). For auditory target words there were no significant main effects of repetition in the traditional N400 time period.

550–750 ms Target Epoch

Between 550–750 ms, there were no effects of repetition for visual target words. However, there was a small effect of repetition for auditory target words (midline: F(1,23)= 4.74, p= 0.040; C1: F(1,23)= 3.84, p= 0.062; C2: F(1,23)= 3.14, p= 0.090; C3: F(1,23)= 4.73, p= 0.040 – see Figure 5).

Difference Wave Analysis

Figure 6 illustrates the difference waves obtained by subtracting the repeated from the unrelated waveforms in the visual and auditory modalities at the five midline sites. We quantified these waveforms by measuring two windows (300–500 and 550–750 ms) in these waveforms. In the 300 to 500 ms window visual repetition effects were significantly more negative-going than auditory repetition effects especially at the central and parietal electrodes (modality × electrode site interaction, midline: F(4,92) = 7.48, p = .001). However, in the 550–750 ms window it was the auditory modality which produced the significantly more negative-going priming effect (midline: F(1,23) = 5.30, p = .031 – see Figure 6).

Figure 6.

Difference waves for visual (solid) and auditory (dotted) targets computed by subtracting repeated targets from unrelated targets in Experiment 1. Target onset is the vertical calibration bar and negative is up.

Discussion

Experiment 1 successfully replicated the behavioral pattern of within and cross-modality masked repetition priming obtained in prior studies using the lexical decision task and similar stimulus presentation conditions (i.e., 50 ms visual primes, Grainger et al., 2003). Subjects were significantly faster in classifying both visual and auditory targets as words when these items were preceded in the visual modality by the same word as compared to a different unrelated word, and there was no interaction between target modality and priming. Experiment 1 further confirms the robustness of this particular result with effect sizes very similar to those observed by Grainger et al. (2003).

In the ERP data, however, clearly distinct patterns of priming effects were observed in the cross-modality compared to the within-modality condition. These differences were both qualitative (in terms of scalp distribution and direction of effects) and quantitative (in terms of the time courses and amplitude). While the visual ERP repetition effects started early (200–300 ms) and continued through the traditional N400 epoch (300–500 ms), a comparable effect in the auditory modality was not significant until the offset phase of the typical N400 (550–750 ms), a time period by which the visual effect had completely dissipated. These results therefore confirm the first prediction derived from the bi-modal interactive-activation model, namely that lexical-level pathways dominate cross-modal transfer. Activation engendered by a visual prime word reaches whole-word orthographic representations (giving rise to within-modality priming) and is sent on to corresponding whole-word phonological representations. However, as depicted in the left panel of Figure 2, given the limited prime exposures of Experiment 1, the activation arriving at whole-word phonological representations is weaker and delayed compared to the activation arriving at whole-word orthographic representations.

One unpredicted result in Experiment 1 was a small inverse priming effect (repeated more negative than unrelated) in the cross-modal condition in the 200–300 ms measurement window over the right hemisphere. Our earlier supraliminal cross-modal study (Holcomb et al., 2005a) found priming in this epoch in the expected direction (i.e., unrelated more negative than repeated), suggesting that the current reversed finding is due to masking and the resulting limited processing time for the visual prime.

The results of Experiment 1 replicate prior observations of within-modality (visual-visual) masked repetition priming in event-related brain potentials (Holcomb et al., 2005b; Holcomb & Grainger, in press; Misra & Holcomb, 2003; Schnyer et al, 1997). Like all of these previous studies, target words following the same word presented as a masked prime produced an attenuated centro-posterior N400 response compared to target words preceded by different masked prime words. In line with the results of Holcomb and Grainger (in press), in Experiment 1 we also found evidence for visual-visual repetition effects starting as early as 200–300 ms post-target onset that resulted in a clear negative-going peak at 250 ms (see Figure 4). Holcomb and Grainger referred to this as the N250 component and suggested that it reflects early sublexical processing in the visual word processing system. Like Holcomb and Grainger, the N250 in the current experiment also had a somewhat more anterior distribution than the later N400. The observation of a similar components, using the same masking and prime-target timing, but different tasks (lexical decision versus semantic categorization in Holcomb & Grainger) and a different language (French versus English in Holcomb & Grainger) provides an important confirmation of the generalizability of this component.

Although direct between-study comparisons are difficult to make, it is worth noting that the current 1.2 microvolt auditory priming effect at Cz between 500 and 750 ms is relatively late and somewhat small compared to at least one previous cross-modal priming study. Holcomb et al. (2005a) using a similar prime-target interval (but without masking of the visual primes) found a significant 5.0 microvolt N400 effect for auditory targets at Cz between 300 and 500 ms, as well as a significant 4.5 microvolt effect between 500 and 800 ms. This suggests that it was not the brief interval between visual primes and auditory targets or the relatively long duration of auditory targets that accounts for the small and late priming effects in the current experiment. More likely it was the prime exposure duration used in Experiment 1 which effectively limited cross-modal transfer, causing a much reduced and later effect of repetition priming for auditory targets. An alternative explanation for the relatively small auditory N400 effect, one that is investigated in Experiment 2, has to do with task influences and the timing of priming effects.

EXPERIMENT 2

Experiment 1 used a lexical decision task, which has known limitations in isolating the component processes of word recognition (e.g., Balota & Chumbley, 1984; Grainger & Jacobs, 1996). Moreover, when combined with ERPs, the requirement of making an explicit decision and response on each trial typically results in the modulation of additional ERP components – most notably late positivities (see Figure 4 and 5 in the 400 to 750 ms period). These late positivities overlap in time with the N400 and can make unambiguous separation of these components difficult. This may have been particularly problematic for the ERP repetition effect seen for auditory targets in Experiment 1 as this effect mostly occurred on what we interpreted as the descending phase of the N400. This is exactly the time region when overlapping decision-related positivities might have influenced the ERP. It is therefore possible that the observed N400 effect was modified to some degree by one of these overlapping processes.

Experiment 2 was a replication of Experiment 1 using the same critical visual prime words and visual and auditory target words that were used in Experiment 1. The main difference was that the nonword target trials from Experiment 1 were replaced with a smaller number of “probe” trials (17% of trials) in which the target and/or prime words were members of single semantic category (animal names). Participants were told to rapidly press a single response button whenever they detected an animal name, and to withhold responding to all other (i.e., non-animal) words. In this manner the target words on the critical trials of interest are presumably processed for meaning, but they are not associated with an overt decision and response. In previous studies we (e.g., Misra & Holcomb, 2003; Holcomb et al., 2005) and others (e.g., Rugg., 1990) have shown that this go/no-go semantic categorization task produces large ERP priming effects, but greatly reduces the late positivities associated with overt decisions. The primary goal of Experiment 2 was to replicate the ERP pattern obtained in Experiment 1 using a different task. If the reduced N400 effect for auditory targets in Experiment 1 was due to interference from late positivities associated with the lexical decision task, then we expect to see larger effects in Experiment 2.

Method

Participants

Thirty volunteers were paid 25 euros to participate in this experiment. Of this 30, six were rejected prior to data analysis because they identified more than 30% of masked words in a post-ERP experiment mask-effectiveness test procedure (see below). The remaining 24 participants (22 female, mean age = 20 years) were all right-handed native speakers of French reporting normal visual and auditory acuity. None of the subjects had participated in Experiment 1.

Stimuli

The critical stimuli for this experiment were the same 200 pairs of French words taken from Experiment 1. The non-critical stimulus pairs were formed by combining 40 animal names with 40 unrelated non-animal words. Stimulus lists consisted of 240 trials each containing a prime-target pair of words. As in Experiment 1, there were 100 pairs of repeated critical words (e.g. courir-courir) and 100 pairs of unrelated critical words (e.g. paquet-courir). In addition to the two levels of repetition, there was another factor of target modality (visual and auditory), which resulted in four types of critical trials (50 pairs/condition). Four counter-balanced lists were created so that across subjects, critical target words appeared once in each of the four conditions. Within subjects, each list (and therefore each target stimulus) was presented once. For the 40 non-critical pairs 20 contained animal names in the prime position and repeated animal names in the target position (10 targets presented visually and 10 presented aurally) and 20 pairs contained unrelated words in the prime position and animal names in the target position (10 targets presented visually and 10 aurally). The animal names served as probe items in a go/no-go semantic categorization task in which participants were instructed to rapidly press a single button whenever they detected an animal name. Participants were told to read/listen to all other words passively (i.e., critical stimuli did not require an overt response). A practice session was administered before the main experiment to familiarize the participant with the procedure.

Procedure

The procedure was the same as in Experiment 1 with the exception that participants made semantic categorization judgments to each target word and pressed a single button to occasional animal names. Also, in the mask-effectiveness test administered after the main ERP experiment, participants were asked to try to identify the words flashing briefly (50ms) on the screen between a 500 ms forward and 13 ms backward mask. Responses were given orally and recorded by an experimenter seated next to the participant. This allows us to provide another measure of prime visibility in testing conditions that are identical to Experiment 1. Note, however, that as in Experiment 1, the mask-effectiveness test is likely to have over estimated the visibility of primes during the ERP experiment as it focused attention on detecting the masked items themselves (Kahan, 2000).

Data analysis

The electrophysiological data from Experiment 2 were analyzed using the same procedures and measurement windows employed in Experiment 1.

Results

Behavioral Data

During the main experiment, participants detected 88% of animal probes within a window of 200 to 1200ms post-probe onset. During the post-ERP mask-effectiveness procedure an average of 1.2 (4%) of the 30 primes were correctly identified (SD=2.5). This would be equivalent to 52% performance in a forced-choice task (if one assumes chance level forced-choice performance when no prime is identified), and almost identical to that found in the same testing conditions of Experiment 1 (53% for word primes).

Electrophysiological Data

On average 7.9% of trials were rejected due to artifact which left an average of 40 trials per subject per condition.

Visual Inspection of ERPs

The two-word ERPs (prime and target) for the two target modalities are plotted in Figures 7 (visual targets) and 8 (auditory targets). Overall the ERPs in this experiment are quite similar to those seen in Experiment 1. The biggest difference being that following the N400 the late positivity seen in Experiment 1 was considerably smaller in Experiment 2.

Figure 7.

Experiment 2 ERPs from 29 scalp sites to Repeated (solid) and Unrelated (dashed) prime-TARGET pairs in which the prime was a masked visual word and the target a clearly visible visual word. Target onset is marked by the vertical calibration bars and prime onset was always 65 ms earlier (as indicated by the arrow labeled “P” on the time legend in the lower left hand corner). Relative electrode positions on the scalp is indicated on the schematic head in the upper left hand corner. Negative voltages are plotted up.

Analyses of ERP Data

0–150 ms Target Epoch

For the auditory ERPs, there were no main effects of repetition during this time period at any of the four electrode columns. In contrast, visual target ERPs were significantly different for the repeated and unrelated conditions. ERPs to unrelated visual targets were more negative than to repeated visual targets (main effect of repetition, midline: F(1,23) = 5.244, p= 0.032; C1: F(1,23) = 5.303, p= 0.031; C2: F(1,23) = 4.495, p= 0.045; C3: F(1,23) = 2.967, p= 0.098). To examine the time-course of this effect, more fine-grained analyses were performed for the 0–50 ms, 50–100 ms and the 100–150 ms target epochs. Differences during the 0–50 ms and 50–100 ms epochs did not approach significance (all ps > .2), but the 100–150 ms epoch revealed a significant repetition effect (midline: F(1,23) = 13.454, p =0.001; C1: F(1,23) = 13.105, p = 0.001; C2: F(1,23) = 11.582, p = 0.002; C3: F(1,23) = 6.373, p = 0.019) with unrelated targets producing more negative-going ERPs than repeated targets. These very early effects of repetition, which were not significant in Experiment 1, can clearly be seen in Figure 7 as overlapping the P2 component.

200–300 ms Target Epoch

In this epoch the unrelated visual targets elicited a significantly more negative-going ERPs than the visual repeated targets across the scalp (N250, midline: F(1,23) = 20.89, p= 0.001; C1: F(1,23) = 26.71, p= 0.001; C2: F(1,23) = 22.57, p= 0.001; C3: F(1,23) = 18.99, p= 0.001 – see Figure 7). This is the same basic pattern seen in Experiment 1, although the effect was somewhat larger over the right than left hemisphere in Experiment 1 and here the effect was uniform across the hemispheres (all interactions between repetition and electrode and hemisphere Fs < 1). Conversely, the auditory unrelated targets produced about the same amplitude ERPs as the repeated targets at the midline and C1 electrode columns and a slightly more positive response than repeated targets at the two more lateral electrode columns (C2: F(1,23) = 4.31, p= 0.049; C3: F(1,23) = 5.09, p= 0.034 – see Figure 8). Again, this is the same general pattern found in Experiment 1, although there the reversed priming effect also tended to be larger over the right than left hemisphere.

Figure 8.

Experiment 2 ERPs from 29 scalp sites to Repeated (solid) and Unrelated (dashed) prime-TARGET pairs in which the prime was a masked visual word and the target an auditory word. Target onset is marked by the vertical calibration bar and prime onset was always 65 ms earlier (as indicated by the arrow labeled “P” on the time legend in the lower left hand corner). Relative electrode positions are indicated on the schematic head in the upper left hand corner. Negative voltages are plotted up.

300–500 ms Target Epoch

As can be seen in Figure 7, between 300 and 500 ms, unrelated visual words were again associated with a larger negativity than repeated visual words (midline: F(1,23) = 26.86, p= 0.001; C1: F(1,23) = 20.48, p= 0.001; C2: F(1,23) = 19.70, p= 0.001; C3: F(1,23) = 15.50, p= 0.001). Peaking around 450 ms post-target onset, this N400-like effect tended to be larger over central and more posterior sites especially along the midline (repetition × electrode site interaction: midline: F(4,92) =5.01, p= 0.011). As in the previous epoch, auditory ERPs (Figure 8) were either about the same amplitude for repeated and unrelated targets or were somewhat more positive for unrelated targets. The reversed priming effect was most notable towards the front of the head (repetition by electrode site interaction, midline: F(3,92) = 3.74, p < .031; C3: F(3,92) = 3.92, p = .031).

550–750 ms Target Epoch

For both the visual and auditory modalities, there were no significant main effects of repetition on target words during this time period. However, for the auditory ERPs there were significant interactions between repetition and electrode site across the electrode columns (repetition × electrode site interaction: midline: F(4,92) =6.60, p= 0.005; C1: 3.62, p = .061; C2: F(4,92)= 4.43, p= 0.033; C3: F(4,92)= 6.19, p= 0.011). At the more anterior electrode sites, repeated auditory targets elicited somewhat less positive ERPs than unrelated auditory targets, but at more posterior sites unrelated targets produced greater negativities than repeated auditory targets (i.e., an N400 effect – see Figure 8).

Difference Waves

As in Experiment 1 we compared repetition effects between the modalities by contrasting difference waves at the midline sites in both the 300–500 and the 550–750 ms epochs. In the 300 to 500 ms window visual repetition effects were significantly larger than auditory repetition effects (main effect of modality, midline: F(4,92) = 16.51, p < .001). However, in the 550–750 ms window there were no significant differences in the size of repetition effects between modalities (main effect of modality, F < 1 – see Figure 9).

Figure 9.

Difference waves from Experiment 2 for visual (solid) and auditory (dotted) targets computed by subtracting repeated targets from unrelated targets. Target onset is the vertical calibration bar and negative is up.

Discussion

The ERP data again replicate prior observations of within-modality (visual-visual) repetition priming with unrelated visual target words producing more negative going ERPs than repeated visual target words starting in the 100–150 ms window post-target onset and continuing until the end of the N400 epoch (500 ms). These effects are even earlier than observed in Experiment 1, and among the earliest ERP repetition effects we are aware of. As in Experiment 1 and in Holcomb and Grainger’s (in press) study, these effects took the form of a modulation of an early ERP negativity peaking at about 250 ms (N250) as well as changes in the later N400 as well. There was also evidence for cross-modal priming. However, as with Experiment 1, the cross-modal priming effects were both qualitatively and quantitatively distinct from within-modality priming. Early during the 300–500 ms window there was no evidence of the typical N400 priming effect. As can be seen in Figure 9 there was actually what appears to be a reverse repetition effect (unrelated words more positive than repeated words) at some electrode sites. This would appear to reflect some difficulty associated with having the same word appear in rapid succession in different modalities, and might reflect the operation of mechanisms used to manage the deployment of attentional resources across modalities. The more typical influence of repetition (i.e., larger negativity for unrelated items) was again evident only during the later declining phase of the auditory N400 between 550 and 750 ms. Thus, the reduced N400 effect for auditory targets found in Experiment 1 cannot be attributed to factors specific to the lexical decision task and instead would appear to be due to the masking manipulation. The results of Experiment 2 therefore confirm the pattern of within-modality and cross-modality repetition priming effects reported in Experiment 1.

EXPERIMENT 3

Experiments 1 and 2 show very small and late effects of cross-modal repetition priming in the region of the N400 that arguably represent one end of a continuum, that, given previous results with unmasked cross-modality priming (Holcomb et al., 2005), should eventually lead to quite large and early effects. Furthermore, given prior behavioral evidence that cross-modal priming from pseudohomophone primes emerges at slightly longer prime durations (67 ms; Grainger et al., 2003), we decided to test for cross-modal repetition priming with only a minimal increase in prime duration relative to Experiments 1 and 2. Therefore, Experiment 3 was a direct replication of Experiment 2, the only change being that prime exposure duration was increased from 50 ms to 67 ms (i.e., from three vertical retrace intervals to four). According to the second prediction of the bimodal interactive-activation model, cross-modal priming effects should appear in earlier (pre-N400) ERP components with this longer prime duration (see right panel of Figure 2).

Methods

Participants

Twenty-four volunteers were paid 25 euros to participate in this experiment (12 female, mean age = 21 years). All were right-handed native speakers of French reporting normal visual and auditory acuity. None of the subjects had participated in Experiments 1 or 2 and because primes were likely to be more visible, no participant was rejected because of the post-ERP mask effectiveness test.

Stimuli, Procedure and Data Analysis

The stimuli, procedures and approach to data analysis were virtually identical to those used in Experiment 2. The two exceptions were that the prime duration was increased from 50 ms 67 ms, and 20 animal name probe trials were added to the design as prime stimuli accompanied with non-animal filler words as targets. Since participants are asked to press the response button whenever they detect an animal name, this procedure adds another mechanism for testing participants’ awareness of masked primes, but in this case, during the experimental run itself. Note that exactly the same post-experiment mask-effectiveness test as in Experiment 2 was also used.

Results

Behavioral Data

During the main experiment, participants detected an average of 92% of animal probes in the target position within a window of 200 to 1200ms post-probe onset. In the post-ERP mask-effectiveness procedure participants correctly reported an average of 64% of primes. By contrast, the estimated visibility of the animal probe words in the prime position during the ERP experiment was much lower (10% detected on average).¹ These results support the suggestion made in Experiments 1 and 2 that the post-ERP mask-effectiveness procedure overestimates the visibility of mask primes during the ERP experiment.

Electrophysiological Data

On average 6.7% of trials were rejected because of artifact which left an average of 47 trials per subject per condition.

Visual Inspection of ERPs

The two-word epoch ERPs (prime and target) for the two target modalities are plotted in Figures 10 (visual targets) and 11 (auditory targets). The morphology and scalp distribution of the major ERP components was very similar to those in Experiment 1 and 2.

Figure 10.

Experiment 3 ERPs from 29 scalp sites to Repeated (solid) and Unrelated (dashed) prime-TARGET pairs in which the prime was a masked visual word and the target a clearly visible visual word. Target onset is marked by the vertical calibration bars and prime onset was always 80 ms earlier (as indicated by the arrow labeled “P” on the time legend in the lower left hand corner). Relative electrode positions on the scalp is indicated on the schematic head in the upper left hand corner. Negative voltages are plotted up.

Analyses of ERP Data

0–150 ms Target Epoch

Unrelated visual target ERPs were significantly more negative-going than repeated visual target ERPs at medial posterior sites (main effect of repetition: Column 1: F(1,23) = 4.26, p = .05; midline, repetition by electrode site interaction: F(4,92) = 3.59, p= 0.033). As in Experiment 2, to examine the time-course of these effects, more fine-grained analyses were performed for successive 50 ms epochs between target onset and 150 ms. Only the 100–150 ms epoch produced a significant main effect of repetition in the Column 1 analysis (F(1,23) = 7.14, p < .014; 0–50 and 50–100 Fs < 1.5) and repetition by electrode site interaction in the midline analysis (F(4,92) = 3.36, p = .047). For the auditory ERPs, there were no main effects of repetition or interactions of repetition with electrode site during this time period (all Fs < 1.5).

200–300 ms Target Epoch

Unrelated visual targets elicited a significantly more negative-going ERPs than visual repeated targets, particularly at central-medial sites (repetition by electrode site interaction, midline: F(4,92) = 4.01, p= 0.024; C1: F(2,46) = 3.99, p= 0.041; C2: F(1,23) = 4.56, p= 0.03; C3: F < 1.7 – see Figure 11). Unlike Experiments 1 and 2, there were no significant repetition effects in the auditory analyses (all Fs < 2.0).

Figure 11.

Experiment 3 ERPs from 29 scalp sites to Repeated (solid) and Unrelated (dashed) prime-TARGET pairs in which the prime was a masked visual word and the target an auditory word. Target onset is marked by the vertical calibration bar and prime onset was always 80 ms earlier (as indicated by the arrow labeled “P” on the time legend in the lower left hand corner). Relative electrode positions are indicated on the schematic head in the upper left hand corner. Negative voltages are plotted up.

300–500 ms Target Epoch

As can be seen in Figure 10, between 300 and 500 ms, unrelated visual target words again produced significantly larger N400 responses than repeated visual target words especially over more posterior medial sites (repetition by electrode site interaction, midline: F(4,92) = 4.01, p = .024; C1: F(2,46) = 3.99, p = .041; C2: F(3,69) = 4.56, p = .03). However, unlike the previous two experiments, unrelated auditory target words (Figure 11) also produced a somewhat larger N400-like negativity than repeated auditory words in this epoch (midline: F(1,23) = 3.88, p= 0.061; C1: F(1,23) = 4.74, p= 0.04; C2: F(1,23) = 4.05, p= 0.056; C3: F(1,23) = 3.82, p= 0.063). Peaking near 475 ms post-target onset, this negative-going effect was also larger over the right than left hemisphere in the more lateral electrode columns (repetition by hemisphere interaction: C2: F(1,23) = 6.05, p = .022; C3: F(1,23) = 5.92, p = .023).

550–750 ms Target Epoch

As in the previous two experiments for the visual targets there were no significant effects of repetition during this time period. However, for the auditory targets there continued to be significant main effects of repetition across the scalp (midline: F(1,23) = 11.30, p = .003; C1: F(1,23) = 13.46, p = .001; C2: F(1,23) = 10.17, p = .004; C3: F(1,23) = 6.56, p = .017), although this effect tended to be largest at more central and posterior sites (repetition by electrode site interaction, midline: F(4,92) = 4.75, p = .02; C1: F(2,46) = 7.96, p = 003; C2: F(3,69) = 7.31, p = .005; C3: F(4,96) = 4.39, p = .029).

Difference Wave Analysis

As in Experiments 1 and 2 we compared repetition effects between the modalities by contrasting difference waves at the midline sites in both the 300–500 and the 550–750 ms epochs. Unlike Experiment 1 and 2, in the 300 to 500 ms window there were no significant differences in the size of repetition effects between modalities (main effect of modality F < 1). However, in the 550–750 ms window the auditory modality produced a significantly larger repetition effect than the visual modality (midline: F(1,23) = 5.30, p = .031 – see Figure 12).

Figure 12.

Difference waves from Experiment 3 for visual (solid) and auditory (dotted) targets computed by subtracting repeated targets from unrelated targets. Target onset is the vertical calibration bar and negative is up.

Between experiment comparisons

Finally, we also directly compared auditory repetition effects across the second two experiments (which were procedurally equivalent) to determine if the difference in the size and time-course of priming effects was reliably different as a function of prime duration. For these comparisons we again used difference waves. In the auditory analyses we used mean amplitudes between 300–500 ms and 550–750 ms and in the visual analyses we used 200–300 ms and 300–500 ms. All of these analyses included a single within-subject factor of electrode site (FPz vs. Fz vs. Cz vs. Pz vs. Oz) and a single between-subject factor of experiment (Exp. 2 vs. Exp. 3). For the auditory targets both the initial N400 epoch between 300 and 500 ms, and the later descending phase of the N400 between 550 and 750 ms the auditory difference waves in Experiment 3 were significantly more negative-going than those in Experiment 2 (main effect of Experiment 300–500 ms: F(1,46) = 6.10, p < .017; 550–750: F(1,46) = 4.52, p = .039 – see Figure 13). For the visual targets there were also differences between the experiments. For the 200–300 ms epoch (N250) and the 300–500 ms epoch (N400) the difference waves for Experiment 2 were more negative-going than those in Experiment 3 (main effect of Experiment 200–300 ms: F(1,46) = 7.44, p = .009; 300–500 ms: F(1,46) = 4.52, p = .039).

Figure 13.

Auditory target difference waves in the semantic categorization experiments with 50 ms prime durations (Experiment 2) and 67 ms prime durations (Experiment 3), calculated by subtracting repeated targets from unrelated targets.

Discussion

A small increase in prime exposure duration from 50 ms in Experiment 2 to 67 ms in Experiment 3, and with otherwise identical procedures, has caused a significant change in the pattern of cross-modal repetition priming as seen in ERP recordings (Figure 13). Our between-experiment analyses showed an earlier and much larger modulation of N400 amplitude in cross-modal repetition priming in Experiment 3 compared to Experiment 2. Compared to within-modality priming, the cross-modal priming observed in Experiment 3 was as statistically strong in the window spanning 300–500 ms, and the effects were significantly larger in the 550–750 ms window.

Within-modality repetition priming continued to show an early component (N250) in Experiment 3 that was not present in the cross-modal condition. This provides further evidence that the N250 obtained with visually presented targets reflects early orthographic processing at the interface between sublexical (i.e., letters and letter clusters) and whole-word orthographic representations. Furthermore, the fact that cross-modal effects were still limited to the N400 time window, suggests once again that lexical-level connections dominate cross-modal transfer. This dominance of lexical representations is clearly illustrated in Figure 2 where we see that whole-word phonological representations receive much more activation input from a visual prime than do sublexical phonological representations.

Within-modality repetition priming actually diminished with increasing prime exposure across Experiments 2 and 3. This could well be due to inhibitory reset mechanisms coming into play as prime stimuli are close to the threshold of conscious awareness (which is arguably the case in Experiment 3). Such inhibitory effects of repetition have been reported in conditions of rapid serial visual presentation (RSVP) and referred to as “repetition blindness” (e.g., Kanwisher, 1987; Kanwisher & Potter, 1990). One plausible mechanism for such inhibitory effects is that the word recognition system needs to be reset in order to correctly process new upcoming words. Such a reset mechanism would come into play when primes and targets are perceived as separate perceptual events, which is all the more likely as prime exposure duration increases, and would operate on modality-specific representations (see Grainger & Jacobs, 1999, for a discussion of this proposal).

Interestingly, the small reversed priming effect found in Experiments 1 and 2 for the cross-modal condition was not present in Experiment 3 with slightly longer visual prime durations. One admittedly speculative possibility for the different size and time-course of priming effects across experiments is that the reversal effect in the first two experiments blocked or obscured the earlier phase of the N400 seen in Experiment 3 and that the resulting larger and somewhat earlier effect in Experiment 3 is more a reflection of the removal of the reversal than the addition of a larger/earlier N400. Testing of this hypothesis will have to await further experimentation.

GENERAL DISCUSSION

The present study reports three masked priming experiments that used ERP recordings to investigate the influence of visually presented prime stimuli on visual and auditory target word processing. Primes were either the same word as targets or a different word. In Experiment 1 participants made a lexical decision on every trial, and in Experiments 2 and 3 participants had to detect animal names occurring on non-critical trials. Experiments 1 and 2 used 50 ms prime exposures, and Experiment 2 used 67 ms exposures. The results show a very early effect of repetition priming for visual targets, arising no later than 150 ms post-target onset, and peaking at around 250 ms (the N250 component, Holcomb & Grainger, in press). When targets were presented auditorily, visual primes mostly affected the later N400 component during target processing, and at 50 ms prime durations this effect was much weaker than the within-modality equivalent. However, with slightly longer prime durations (67 ms), the effect of repetition priming on N400 amplitude was just as strong for auditory (cross-modality) as for visual (within-modality) targets, but was still restricted to the N400 window.

Cross-modal interactions in word recognition

The present study provides further evidence that briefly presented visual primes can affect the subsequent processing of auditorily presented target words. Experiment 1 provided a replication of the behavioral results of Kouider and Dupoux (2001) and Grainger et al. (2003) showing that auditory lexical decision latencies are faster when visual primes are the same word as targets compared with different word primes. Grainger et al. (2003) showed that cross-modal repetition priming arises at shorter prime durations than cross-modal priming from pseudohomophones. They argued that this is because cross-modal transfer from a printed word stimulus occurs more rapidly via whole-word representations than via sublexical representations. This analysis implies that cross-modal repetition priming will initially be primarily driven by preactivation of the appropriate whole-word phonological representation. In line with this account, the present study showed that cross-modal priming affected ERP amplitude in the region of the N400, a component thought to reflect lexical-semantic processing. The fact that the cross-modal N400 effect increased in magnitude and initiated earlier as prime exposure was slightly increased, suggests that the longer prime duration allowed activation to reach whole-word phonological representations earlier and therefore to build up to higher levels than at the shorter prime duration (see Figure 2).

Alternatively, one could argue that the different time-courses of within-modality and across-modality repetition priming found in the present study, reflect the operation of fundamentally different mechanisms (cf., Kouider & Dupoux, 2001). Within-modality priming would be more automatic, involving the pre-activation of appropriate representations by the prime stimulus (the approach adopted within the framework of the bimodal interactive-activation model), whereas priming across modalities would reflect the operation of controlled processes involved in establishing connections across modality-specific representations. Holcomb et al. (2005) made just such a distinction in their analysis of the different time-courses of repetition and semantic priming. However, we would argue that while semantic priming may indeed require the conscious linking of related representations, such a mechanism would be superfluous for cross-modal repetition priming since the visual prime and auditory target presumably have the same semantic representation. Furthermore, the semantic priming effect that emerged with longer prime durations in the Holcomb et al. (2005) study was mainly evident on the middle portion and trailing edge of the N400, while cross-modal repetition priming was evident on the rising edge of the N400 at the longest prime duration of the present study.

The present results are compatible with the first prediction of the bimodal interactive-activation model, that lexical-level pathways dominate cross-modal transfer. This model also predicted that cross-modal priming should be evident in early (pre-N400) ERP components in conditions where the sublexical interface between orthography and phonology is hypothesized to be operational (i.e., 67 ms prime durations). However, contrary to this prediction we failed to find any evidence for standard priming effects (i.e., reduced negativity in the related condition) in early ERP components (i.e., the N250) with auditory targets at the longer prime duration of Experiment 3. Within the framework of the bimodal interactive-activation model (see Figure 2), this is likely due to the fact that activation input to sublexical phonological representations (P-units) is still too weak to be detectable in the EEG signal. As can be seen in Figure 2, sublexical representations involved in spoken word recognition (P-units) only receive one source of activation from a visual prime, whereas whole-word phonological representations (P-words) receive two sources of input. Thus, it might well be the case that activation flow to these sublexical units is not strong enough at 67 ms prime exposures to produce measurable effects during auditory target word recognition.

In line with this account, another study of ours using longer (supraliminal) prime exposure durations has found larger and earlier effects of visual primes on auditory word recognition. Holcomb et al. (2005a) investigated cross-modal visual-auditory priming at different prime-target SOAs and 200 ms prime durations. Their results showed a massive and early influence of visual primes on the processing of auditory target words that was maximal at 200 ms SOA (200 ms prime immediately followed by the target) and did not increase significantly with a longer SOA. One explanation for this result within the framework of the bimodal interactive-activation model is that with long enough prime durations, activation can eventually flow from whole-word phonological representations (following cross-modal transfer) down to lower-level representations involved in spoken word recognition. Thus, it remains to be seen in further research whether another small increase in prime duration is all that is necessary to generate cross-modal priming effects in pre-N400 components.

Masked repetition priming within the visual modality

The very early influence of masked visual primes in the within-modality conditions of all three experiments is one of the first demonstrations (with priming methodology) that scalp recordings of the brain’s electrical activity are sensitive to relatively early processes in printed word perception (see also Holcomb & Grainger, in press). Most of the prior priming work using ERPs to study word based processing have produced effects that are primarily centered on the N400. Although N400 effects to visual words have been reported to start as early as 200 ms, along with the Holcomb and Grainger study, this is one of the first reports of a pre-N400 negativity in a visual masked repetition priming paradigm with a clear peak at 250 ms and an onset between 100 and 150 ms. Holcomb and Grainger (in press) referred to this negativity as the “N250.” The replication of this finding in the present study, plus its appearance across all three experiments would appear to strengthen the possibility that this negativity is a real ERP component -- one that, at least in part, is functionally independent from the later N400. Bolstering this possibility are several pieces of evidence. First, at a number of electrode sites in all three experiments there are two clear negativities, one peaking at about 250 ms and the second peaking between 400 and 475 ms. Furthermore, at a number of sites the two negativities are separated by an intervening positivity peaking at about 350 ms (see Figure 4). Second, although both components were widely distributed across the scalp, there were differences in distribution in all three studies. While the N400 had the classic midline posterior maximum, the N250 tended to have a more anterior and/or right sided maximum (see also Holcomb & Grainger, in press).² Third, the most compelling evidence for component independence is when components show differential sensitivity to experimental variables (Rugg & Coles, 1995). In the current series of studies such a dissociation occurred in the comparisons of within and across modality priming in Experiment 3 where within-modality priming revealed an early N250 effect that was absent in the cross-modality condition. On the other hand, priming effects in the region of the N400 appeared to be just as strong cross-modalities as within-modalities in this experiment. Together these data strongly suggest that the N250 and N400 are at least partially distinct and emanate from non-identical neural sources.

The pattern of results obtained in the three experiments suggests that the early influence of repetition primes likely reflects the integration of information across sublexical orthographic representations (letters and letter clusters) and whole-word orthographic representations. We tentatively locate the earliest repetition effects (those on the rising edge of the N250 between 100 to 200 ms) at the level of the first phase of orthographic processing that involves the parallel identification of a set of individual letters. ³ One admittedly speculative possibility is that processing at this level corresponds to what Cornelissen, Tarkiainen, Helenius, and Salmelin (2003) referred to as a Type II response in left occipito-temporal cortex with a latency of approximately 150 ms in MEG recordings. As orthographic processing develops over time, whole-word orthographic representations become more and more activated, and repetition effects at the trailing edge of the N250 (between 200 and 300 ms) would reflect activation at this level of processing. One, again admittedly speculative, possibly is that this activity is generated in the so-called visual word form area of the left fusiform gyrus (Cohen, Dehaene, Naccache, Lehericy, Dehaene-Lambertz, Henaff, & Michel, 2000). Repetition effects in the window of the N400, on the other hand, would reflect integration of information across whole-word representations (either orthographic or phonological) and higher-level semantic representations (Holcomb et al., 2002). Evidence from depth recordings, MEG and fMRI suggest that processing at this level likely relies on widespread activation in the left and right hemisphere including anterior temporal, perisylvian, orbital, frontopolar and dorsolateral prefrontal sites (e.g., Halgren, Dhond, Christensen, Van Petten, Marinkovic, Lewine, & Dale, 2002; McCarthy, Nobre, Bentin, & Spencer, 1995; Kuperberg et al., 2003).

Thus the pattern of within-modality repetition effects observed in the present study are hypothesized to reflect processing at (minimally) three necessary steps for successful word recognition during silent reading: sublexical orthographic, lexical orthographic, and semantic. In future tests of this mapping of ERP time-course effects to levels of word processing it will be important to systematically manipulate variables that differentially affect each processing level independently. Moreover, replication of these finding with MEG and/or parallel fMRI studies should allow for a better evaluation of the spatial localization of the neural processors that generate these different ERP components.

Conclusions

The present study used the precise temporal resolution of ERP recordings to investigate cross-modal transfer from briefly presented, pattern-masked visual prime stimuli to auditorily presented targets. Within-modality repetition priming showed both an early (N250) and later (N400) effect on ERP amplitudes that remained fairly stable across tasks and prime duration. Cross-modal repetition priming, on the other hand, proved to be highly sensitive to a very small change in prime duration (from 50 ms to 67 ms), with repetition effects in the region of the N400 showing an earlier onset and much larger amplitude at the longer prime duration. This study demonstrates how ERP recordings can be usefully combined with the masked priming technique to provide a detailed analysis of the time-course of orthographic and phonological processing in word recognition.