Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2023 Mar 1.
Published in final edited form as: Psychophysiology. 2021 Nov 17;59(3):e13975. doi: 10.1111/psyp.13975

Tracking the time course of sign recognition using ERP repetition priming

Karen Emmorey 1, Katherine J Midgley 2, Phillip J Holcomb 2
PMCID: PMC9583460  NIHMSID: NIHMS1842641  PMID: 34791683

Abstract

Repetition priming and event-related potentials (ERPs) were used to investigate the time course of sign recognition in deaf users of American Sign Language. Signers performed a go/no-go semantic categorization task to rare probe signs referring to people; critical target items were repeated and unrelated signs. In Experiment 1, ERPs were time-locked either to the onset of the video or to sign onset within the video; in Experiment 2, the same full videos were clipped so that video and sign onset were aligned (removing transitional movements), and ERPs were time-locked to video/sign onset. All analyses revealed an N400 repetition priming effect (less negativity for repeated than unrelated signs) but differed in the timing and/or duration of the N400 effect. Results from Experiment 1 revealed that repetition priming effects began before sign onset within a video, suggesting that signers are sensitive to linguistic information within the transitional movement to sign onset. The timing and duration of the N400 for clipped videos were more parallel to that observed previously for auditorily presented words and was 200 ms shorter than either time-locking analysis from Experiment 1. We conclude that time-locking to full video onset is optimal when early ERP components or sensitivity to transitional movements are of interest and that time-locking to the onset of clipped videos is optimal for priming studies with fluent signers.

Keywords: ERPs, N400, repetition priming, sign language

1 |. INTRODUCTION

Rapid recognition and understanding of words is key to language comprehension. During language comprehension, readers and listeners recognize between two and four words per second. Unpacking the perceptual and linguistic mechanisms that underlie such rapid processing has proven challenging. Even using online measures such as event-related potentials (ERPs) and eye-tracking to probe the sequence of events involved in word comprehension has proven somewhat difficult. Although it is possible to record the time course of word processing with high temporal precision, it is not always clear from ERPs recorded to individual words what the perceptual and linguistic significance of various ERP components actually is. One of the biggest problems is that obtaining ERPs requires averaging across many different words which likely overwhelms subtle word-level differences in processing. One attempt to overcome this problem has been to use priming techniques (e.g., Holcomb & Neville, 1990). Priming allows investigators to compare the same words in two contrastive conditions and thus avoids some of the problems that unique words can cause when trying to disentangle item-specific and variable-specific influences during word processing (see Grainger & Holcomb, 2009).

A well-documented finding in cognitive electrophysiology is the sensitivity of the ERP response to words preceded by a repeated or a semantically related word compared to an unrelated word (e.g., Doyle et al., 1996; Kutas & Hillyard, 1980). Such priming effects (the ERP difference between repeated/semantically related and unrelated word primes) have been shown to influence a sequence of ERP components generated by both visual and auditory words, with attenuation in the amplitude of the N400 to primed target words being the most robust finding. This so-called N400 priming effect has usually been interpreted as reflecting easier access to the lexico-semantic properties of a word when it is primed in the related/repeated condition compared to the unrelated prime condition (for review see Kutas & Federmeier, 2011).

Most of the studies using ERP priming techniques have used visually presented words where the entire stimulus is available at stimulus onset—fewer studies have used auditorily presented spoken words (e.g., Anderson & Holcomb, 1995; Holcomb & Neville, 1990) or non-linguistic stimuli such as pictures (e.g., McPherson & Holcomb, 1999) or videos of action sequences (Sitnikova et al., 2003). For dynamic stimuli like spoken words, stimulus presentation and mental processing unfolds over time as the stimulus itself progresses in time. Like their static counterparts, ERPs to dynamic stimuli generate a series of ERP components including robust N400 priming effects. However, because information contained in dynamic stimuli becomes available over the time course of the stimulus, effects like the N400 tend to be more extended in time (Holcomb & Neville, 1990).

Signs in a sign language are perceived visually, but they are also dynamic and unfold over time. Signs are also parallel to spoken words in their linguistic structure. For example, signs, like words, exhibit a level of sub-lexical structure—phonology—that can involve a sequence of segments (e.g., a sequence of handshapes, locations, or movements) and syllables (although multi-syllabic signs are rare; see Brentari, 2019, for a recent review). An important distinction between signs and spoken words is that the linguistic articulators for speech are largely hidden within the vocal tract, but are directly observable for sign. This difference has direct consequences for how individual words and signs are perceived when presented in isolation. For example, the movement of the tongue from rest to the soft palate in the word “gold” is not visible and not perceptible auditorily before voicing begins (note this is not case when words are presented in sentences). Thus, the onset of an audio clip of the individual word “gold” and the onset of the stimulus word itself are identical, namely the onset of voicing of the /g/ consonant. In contrast, as illustrated in Figure 1, the onset of the video clip of the sign GOLD from American Sign Language (ASL) begins with the sign model’s hands at rest (just below the frame) and then her hand transitions to the target location of the sign at the cheek. Thus, the onset of the stimulus video and the onset of the sign are not identical. While speakers cannot hear or see the transition of the tongue at rest to the voiced onset of the word “gold”, signers see the transition of the hand at rest to the onset of the sign GOLD at the cheek. This transitional movement contains useful information about the identity of the stimulus sign, including (a) the number of hands (one vs. two) which can be perceived in the first 1–2 video frames, (b) information about the handshape that is perceived prior to sign onset (contact with the cheek), and (c) the movement speed and trajectory during the transition indicates that the location of the sign is at the face or head (vs. the torso or non-dominant hand).

FIGURE 1.

FIGURE 1

Open in a new tab

The first 12 frames of the video of the sign GOLD, illustrating the transition from rest position to the sign onset location at the cheek. The arrows indicate the time-lock points for video onset and sign onset for Experiment 1

The visibility and dynamic nature of the sign articulators creates a linguistic signal that is distinct from both auditorily and visually presented words. In the present study, we used ERPs and repetition priming to investigate the time course of sign recognition. In one analysis, we time-locked the ERPs to the onset of the video (the beginning of transitional movement), and in the second analysis we time-locked the ERPs to the onset of the signs (e.g., when the hand reached the location of the sign on the body; see Method section for details about how sign onset is determined). For both time-locking analyses, we predict an N400 priming effect in which repeated signs elicit less negativity than unrepeated signs. Our primary interest was to uncover the time course of the N400 priming effect, which will index the time course of lexical-semantic processing during sign recognition.

Time-locking to video onset allows us to examine how early lexical access and semantic processing may occur for signs presented in isolation. Specifically, does lexical processing occur during the transitional movement before the onset of the sign itself? A previous ERP study that investigated N400 priming effects for signs embedded in a sentential context suggests that the answer to this question may be yes. Hosemann et al. (2013) presented deaf signers with sentences in German Sign Language that ended with either an expected or a semantically anomalous sign. Anomalous signs elicited an N400 effect (greater negativity than expected signs), and crucially the onset of the N400 preceded the critical sign onset and began during the transitional movement. Thus, information about the lexical identity of the critical sign was available before sign onset. However, in the present study single signs are presented in isolation, and without a sentence providing contextual cues, it is possible that lexico-semantic processing does not begin until sign onset. Note that the question we address here is unique to sign languages because transitional information for isolated spoken words is not perceivable in the auditory signal. Although some early transitional information, such as lip rounding, may be available for audio-visually presented words, the transitional information for isolated signs extends over a much longer time period, e.g., the transitional movement shown in Figure 1 for the sign GOLD extends about 360 ms.

Time-locking to sign onset may yield results that are more parallel to those found for auditory word processing. We hypothesize that sign onsets (like word onsets) are critical points in the linguist signal for lexical recognition. Some signs onset shortly after video onset (e.g., those produced near the rest position), while others have later onsets (e.g., those produced all the way at the forehead). If ERPs are time-locked to video onset, then the variability of sign onsets may lead to an average ERP response that is “smeared” over time due to a loss of temporal registration. In contrast, if sign onset corresponds to a critical moment with regard to sign recognition, then time-locking to sign onset may generate a larger and/or “tighter” N400 priming effect. The latter may be more likely given that data from partial retrievals during a “tip-of-the-fingers” state suggest that sign onsets are privileged within the lexicon, parallel to word onsets in a “tip-of-the-tongue” state (Thompson et al., 2005).

Finally, we conducted a second experiment in which deaf signers performed the same task with the same signs, but the videos were edited (clipped) so that video onset was aligned with sign onset. Such clipped video stimuli are more parallel to spoken word stimuli because for speech, word onset and audio clip onset are aligned. However, clipped videos may be less natural than full videos because signers typically observe transitions to and from a rest position for isolated signs. Experiment 2 allowed us to investigate the time course of sign recognition when transition information is absent and the timing of stimuli presentation is more parallel to spoken words.

2 |. EXPERIMENT 1: FULL VIDEO WITH TIME–LOCKING TO VIDEO OR SIGN ONSET

2.1 |. Method

2.1.1 |. Participants

Participants included 32 deaf ASL signers (16 female; mean age 29.6 years, SD 7.7 years, range: 20–46). Twenty-two of the participants were native signers (born into a signing family) while 10 learned ASL before the age of six (mean = 2 years). Twenty-nine participants took the ASL Comprehension Test (Hauser et al., 2016), and the average score was 90% correct (SD = 6%), indicating high proficiency in ASL. According to self-report, 30 participants were right-handed and two were left-handed. Although we did not statistically compare the two handedness groups, qualitative comparison of the ERPs did not suggest any notable differences. All participants had severe-to-profound hearing loss (greater than 70 db), used ASL as a primary language, had normal or corrected-to-normal vision, had no history of neurological dysfunction, and were not taking any medications that would affect brain function. Eight more participants were excluded from analyses due to high artifact rejection rates.

Informed consent was obtained from all participants in all experiments in accordance with the Institutional Review Board at San Diego State University. All participants were volunteers who received monetary compensation for their time.

2.1.2 |. Stimuli

Stimuli were 245 short video clips of a native ASL signer producing individual signs, and all videos were obtained from the ASL-LEX database (Caselli et al., 2017; http://asl-lex.org). Videos were presented on an LCD video monitor at a distance of 150 cm and a size of 10 × 13.5 cm. The critical sign features (face, arms, and torso of the signer) were contained within a smaller 6 × 6 cm central portion of each video which subtended a visual angle of 2.3° × 2.3°. These viewing parameters were selected so participants would not have to make eye movements to fully perceive each sign. Each video began with the hands in the lap of the native sign model (see Figure 1) and ended after the hands returned the lap. The average duration of critical videos was 2177 ms (range: 1735–2970 ms; SD = 287), and the average sign onset was 578 ms (range: 334–834 ms; SD = 96) after video onset. The average length of critical signs (see below) within the video was 920 ms (range: 434–2002 ms; SD = 243). Video length, sign onset, and sign offset were all extracted from the ASL-LEX database. Caselli et al. (2017; p. 790) defined sign onsets using criteria similar to those used by other sign language linguists (e.g., Crasborn et al., 2015; Johnson & Liddell, 2011). Briefly, sign onset was defined as the first video frame in which the fully formed handshape contacted the body. If the sign did not have contact, then sign onset was defined as the first video frame in which the fully formed handshape arrived at the target location near the body or in neutral space before starting the sign movement. Caselli et al. (2017) reported that agreement for sign onset coding among three independent coders for a subset of 205 signs in the ASL-LEX database was 91.2%.

2.1.3 |. Procedure

Participants sat in a comfortable chair in a darkened sound attenuated room. They were presented with 238 signs in a single 12-min session while performing a no/go semantic categorization task in which they were instructed to press a button on a response box resting in their lap to occasional probe signs (28 in total, ~12% of trials) that referred to people (e.g., NURSE, CHILDREN). All other signs were viewed passively (i.e., did not require a behavioral response). Instructions were given in ASL, as well as in written English.

During the experimental session participants were presented with 238 sign stimuli arranged as 238 individual trials. A random 70% of trials contained signs preceded by another semantically unrelated sign while the remaining 30% contained signs that were exact repetitions of the sign on the prior trial. For the critical signs, 40 were initial presentations, unrelated to the previous signs, and 40 were identical signs immediately following these initial presentations. The former items served as the unrelated control stimuli since they were preceded by different semantically unrelated (filler) signs while the latter served as the experimental repetition condition. This design allows ERPs to the same items in the same participants to be used in both conditions. Of the remaining signs, 28 were people probe signs (which required a button press) and 130 served as filler signs (see Figure 2). Fillers were included to ensure that trials with repetitions were of comparatively low overall proportion so that participants did not engage repetition-based processing strategies. There were two stimulus lists (A and B) that differed in their order of presentation, and the 40 critical signs were different items in the two lists. Sixteen participants were randomly assigned to the A list the other 16 participants were assigned to the B list.

FIGURE 2.

FIGURE 2

Open in a new tab

Sample trial illustrating filler, first presentation, repeated and probe items (t = time)

Participants were instructed to refrain from blinking during video presentation but were encouraged to blink between videos or during a 3 s blink cue that appeared randomly every 7 to 11 trials. Longer breaks were given every three minutes. A practice list of 16 trials was presented to participants before the experimental run to familiarize them with the task and the video format. After the practice session participants were given an opportunity to ask any questions. As is standard in priming studies, the ISI between trials/signs was set to a constant value (970 ms—e.g., Holcomb & Neville, 1990).

2.1.4 |. EEG recording and analysis

Participants were fitted with an Electro-Cap® equipped with 29 electrodes (for montage see Figure 3). An electrode placed on the left mastoid process was used as a reference during recording and for subsequent analyses. An electrode was placed on the right mastoid to monitor differential mastoid activity as a function of priming (none was observed). An electrode located below the left eye was used to identify blink artifacts. An electrode on the outer canthus of the right eye was used to identify artifacts due to horizontal eye movements. Using a saline gel, all electrode impedances were maintained below 2.5 kΩ. The EEG was sampled continuously at 500 Hz and amplified with SynAmsRT amplifiers (Neuroscan-Compumedics) with a bandpass of DC to 100 Hz.

FIGURE 3.

FIGURE 3

Open in a new tab

Electrode montage with sites plotted in ERP figures interconnected with grid lines and the ANOVA site (Cz) circled

In the first analysis, ERPs were time-locked to the onset of each video and were binned separately for unrelated and repeated signs. In the second analysis, the time-locking point was three frames (100 ms) before sign onset (see Stimuli for how sign onset was determined). Offline, separate ERPs for each condition and each time-lock were averaged for each participant at each electrode site. In the case of the time-lock to video onset, ERPs extended over a 1500 ms epoch, using a 100 ms pre-stimulus-onset baseline. In the case of the time-lock to sign onset, ERPs extended over a 1500 ms epoch, using a 600 ms pre-stimulus-onset baseline. The longer baseline was used because there was greater variability in the ERPs due to the variable information in the sign transitions during the pre-stimulus period. As is standard in our lab, all averaged ERPs were bandpass filtered between 0.01 and 15 Hz prior to analysis.

Trials contaminated by artifact due to eye movements were excluded (7.5% of trials). While the ERP figures plot data from 15 scalp sites (see grid in Figure 3) we restricted our analyses to the Cz site where the N400 is typically largest. To track the time-course of priming effects mean amplitude was calculated for eight consecutive 100 ms time windows, starting at 300 ms (the typical starting latency of the N400) and extending to 1100 ms for analyses of the video onset time-lock data and between −300 and 500 ms for the sign onset time-lock. Eight sequential one-way repeated measures ANOVAs were used to examine the time-course of repetition effects (Unrelated vs. Repeated) at the Cz electrode site. Because of the large number of sequential statistical comparisons, we report false-detection-rate (FDR) corrected p-values (Groppe et al., 2011) in all analyses.

3 |. EXPERIMENT 2: CLIPPED VIDEOS

In this experiment we presented the same signs to another group of deaf signers (with some overlap in participants across the two experiments) who performed the same go/ no-go semantic categorization task. However, the videos in this study were edited to present just the sign without the transitions from rest position or the transition back to rest position.

3.1 |. Method

3.1.1 |. Participants

Participants included 20 deaf ASL signers (9 female; mean age 29.6 years, SD 6.5 years, range: 22–49). Fifteen of the participants were native signers while 5 learned ASL before the age of six (mean = 2.8 years). Nineteen participants took the ASL Comprehension Test (Hauser et al., 2016), and the average score was 91% (SD = 6%), indicating high proficiency in ASL. According to self-report, 17 participants were right-handed and 3 were left-handed. All participants had severe-to-profound hearing loss (greater than 70 db), used ASL as a primary language, had normal or corrected-to-normal vision, had no history of neurological dysfunction and were not taking any medications that would affect brain function. Three more participants were excluded from analyses due to high artifact rejection rates. Fifteen participants in Experiment 2 also participated in Experiment 1, with at least a month gap between the two studies.

3.1.2 |. Stimuli

The sign stimuli were the same as Experiment 1, but the videos were edited so as to start three frames (100 ms) before sign onset (e.g., frame 10 in Figure 1) and to end at sign offset. Clipping the video to just before sign onset created a more natural, less abrupt stimulus onset. Sign offset was defined as the last video frame in which the hand contacted the body for body-anchored signs or the non-dominant hand for two-handed signs, and if the sign did not end with contact, then offset was defined as the last video frame before the hand(s) began to transition to rest position (Caselli et al., 2017). Caselli et al. (2017) reported that agreement for sign offset coding among two independent coders for a subset of 205 signs in the ASL-LEX database was 87.3%. The average duration of the sign stimuli for Experiment 2 was 1020ms (SD = 243).

3.1.3 |. Procedure

The procedure was the same as Experiment 1. Trials contaminated by artifact due to eye movements were excluded for Experiment 2 as well (7.6% of trials).

4 |. RESULTS

We compare deaf signers processing target signs preceded by another unrelated sign with those preceded with the same (repeated) sign. Participants correctly detected 94% (SD = 0.05) of the people probes in Experiment 1 and 91%. (SD = 0.07) in Experiment 2.

4.1 |. ERPs time-locked to video onset (Experiment 1)

Plotted in Figure 4 are the grand mean ERPs at 15 scalp sites to the repeated (solid black) and unrelated (dotted red) signs time-locked to the onset of each video clip. As can be seen, both related and unrelated signs evoked a clear set of early visual ERP components starting with an occipitally distributed P1 peaking near 100 ms post-video onset, a central-anterior maximal N1 peaking around 180 ms, and an occipital maximal P2 peaking at about 220 ms. Two later components can also be seen: a central anterior N300 peaking near 300 ms post-video onset and a later (600–1100 ms), broadly distributed bi-phasic response that appears to differentiate repeated and unrelated signs. Although it is much later than the classic N400 repetition effect seen in written word ERP studies (Grainger & Holcomb, 2009), this effect appears to have a similar scalp distribution and sensitivity to repetition priming as reported in previous ERP language studies (e.g., Holcomb et al., 2005; Pickering & Schweinberger, 2003; Rugg, 1990).

FIGURE 4.

FIGURE 4

Open in a new tab

ERPs to ASL signs from Experiment 1 at 15 electrode sites. Repeated signs are plotted in black (solid line) and unrelated signs are plotted in red (dotted line). Time-lock is video onset. Baseline is −100 to 0 ms pre-video onset. Negative is plotted up

Table 1 reports the results of the eight sequential ANOVAs between 300 and 1100 ms post-video onset. As indicated, significant main effects of repetition priming started in the 600 to 700 ms window and continued on for 500 ms (until 900 to 1000 ms). See Figure 7a below for the voltage maps showing the distribution of the priming effect (unrelated—repeated signs) for these same windows.

TABLE 1.

Repetition effects time-locked to video onset at Cz

Epoch (ms) 300–400 400–500 500–600 600–700 700–800 800–900 900–1000 1000–1100
F ratio 0.28 0.17 2.34 6.45 11.87 10.71 8.23 0.01
Adjusted p* 0.78 0.78 0.22 0.032 0.01 0.01 0.02 0.98
Open in a new tab
*

FDR corrected.

FIGURE 7.

FIGURE 7

Open in a new tab

The time course of ERP repetition priming illustrated in a series of voltage maps calculated by subtracting repeated sign ERPs from unrelated sign ERPs in each of eight consecutive 100 ms time intervals. (a) Contains maps starting at 300 ms post stimulus onset and is for ERPs time-locked to video onset from Experiment 1. (b) Contains maps starting 300 ms pre-time-lock and is for ERPs time-locked to sign onset from Experiment 1. (c) Contains voltage maps to video clips edited so that video onset was aligned with sign onset from Experiment 2. Epochs with significant repetition priming effects (FDR corrected, critical p < .05) are marked with an asterisk (also see Tables 1–3)

4.2 |. ERPs time-locked to sign onset (Experiment 1)

Plotted in Figure 6 are the ERPs re-averaged with a time-lock to sign onset for each individual sign. We used the epoch between −300 and −200 as the baseline because of the variable information across stimuli in the sign transitions prior to sign onset. As can be seen, the early components (P1, N1 and P2) visible in Figure 4 are no longer present in the post-time-lock epoch (the vertical calibration bar). However, the video onset N1 and P2 can be seen in the pre-time-lock epoch to the left of the calibration bar. Of interest is what impact this time locking scheme has on the repetition priming effects. The sequential one-way ANOVAs for eight 100 ms latency ranges starting 300 ms prior to sign onset and continuing on through 500 ms are reported in Table 2. As can be seen in Figure 5 and Table 2 (and Figure 7b below), significant differences in the ERPs between the repeated and unrelated signs begin even before the sign onset time-lock at a latency approximately 100 ms prior to the lock point (−100 to 0 ms). The repetition effect continued to be significant from this point through the 300–400 ms window.

FIGURE 6.

FIGURE 6

Open in a new tab

ERPs from Experiment 2 for repeated and unrelated signs in which videos were clipped so that video onset was aligned to sign onset

TABLE 2.

Repetition effect time-locked to sign onset at Cz

Epoch −300–200 −200–100 −100–0 0–100 100–200 200–300 300–400 400–500
F ratio 0.11 1.01 6.44 10.42 16.15 26.39 10.15 0.14
Adjusted p* 0.74 0.43 0.026 0.007 0.001 0.0004 0.0007 0.74
Open in a new tab
*

FDR corrected.

FIGURE 5.

FIGURE 5

Open in a new tab

ERPs to ASL signs from Experiment 1 at 15 electrode sites plotted with a sign-onset time-lock point. Repeated signs are plotted in black and unrelated signs are plotted in dotted red. Note the −300 to −200 ms pre-sign onset epoch was used for baselining due to variable information in the sign transition prior to sign onset

4.3 |. ERPs time-locked to clipped video/sign onset (Experiment 2)

As can be seen in Figure 6, the ERPs generated a very similar set of early ERP components as seen with video time-locking in Experiment 1 (Figure 4). This includes a posterior P1 and P2 and anterior N1 and N300 components. Of most interest is the time-course of the later N400 effect. As in Experiment 1, mean amplitudes were calculated for consecutive time windows of 100 ms from 200 to 800 ms at the Cz electrode site and were analyzed using one-way ANOVAs with p values FDR corrected across the six epochs. The results are presented in Table 3. As can be seen in Figure 6, Table 3, and Figure 7c below, the time course of N400-like priming effects starts in the 300–400 ms epoch but is not significant until the 400–500 ms epoch. This effect then continues into the 500–600 ms epoch but is largely complete by the 600–700 ms epoch. This time course is both shorter (by approximately 200 ms) and earlier compared to the video-onset time-locked ERPs from Experiment 1.

TABLE 3.

Repetition effect time-locked to clip onset at Cz

Epoch 200–300 300–400 400500 500600 600700 700800
F ratio 4.18 2.67 7.28 11.6 0.35 1.91
Adjusted p* 0.09 0.14 0.035 0.015 0.56 0.18
Open in a new tab
*

FDR corrected.

4.4 |. Comparing full video and sign onset time-locking (Experiment 1) and clipped video time-locking (Experiment 2)

Figure 7 presents the voltage maps showing the scalp distribution of the priming effects (unrelated—repeated signs) for the relevant time windows when time-locking to onset of the full video, to the sign onset within this video, and to the clipped video onset (aligned with sign onset). Figure 8 compares repeated and unrelated sign ERPs for these same time-lock points at the Cz electrode site. As can be seen in these Figures, there was a substantial difference in the absolute time course of priming effects between the time-locking schemes. The difference between time-locking to the full video and to sign onset within this video can best be seen in Figure 7a,b which shows epochs where the repetition effect was significant for each of the two time-locking schemes. The time-course of significant priming effects starts in the 100 ms epoch prior to sign onset and continues on through 400 ms post-sign-onset in the sign onset locked data. After this point there were no longer any significant main effects of priming. This pattern contrasts with the much later 500 to 1000 ms priming effect seen in the video onset data.

FIGURE 8.

FIGURE 8

Open in a new tab

ERPs to ASL signs from Experiment 1 in the repeated (solid black) and unrelated (dotted red) conditions at the Cz electrode site. Left panel shows ERPs time-locked to video onset, and the center panel shows the same data time-locked to sign onset (see text for the definition of sign onset). The right panel shows the ERPs time-locked to the clipped videos from Experiment 2

Careful examination of Figure 7a,b shows that there is actually a remarkable similarity in the relative time-course and as well as the distribution of priming effects across the two time-locking schemes if we align the plots based on the apparent onset of priming. The main difference appears to be that priming in the full video-onset aligned data starts and ends 600 ms later than the sign onset effects. Since 600 ms is almost exactly the same delay as the average sign onset point (578 ms), it would appear that changing the time-lock point did not alter the basic pattern of priming.

Follow-up analyses adding a factor of time-lock type (full video onset vs. sign onset) contrasting each of the five 100 ms epochs where priming effects were found (500–600 ms vs. −100 to 0 ms; 600–700 vs. 0–100 ms; 700–800 vs. 100–200 ms; 800–900 vs. 200–300 ms; and 900–1000 vs. 300–400 ms) all showed the same pattern of main effects of priming which were very similar to what was found individually for the two time-lock schemes. Importantly, there were no trends for an interaction of the repetition priming factor with the time-lock type factor (all ps > .5) which suggests that the time-course and size of priming effects did not significantly differ as a function of time-lock point for Experiment 1. So, while sign onset time-locking seems to be as good at capturing priming as video onset, there was no evidence that sign onset was superior.

As can be seen in Figures 7c and 8c, the time-course of the N400 priming effect for clipped videos (Experiment 2) was shorter than the N400 effect observed with sign-onset time-locking in Experiment 1 (Figures 7b and 8b). However, the N400 effect for clipped videos also onsets several hundred milliseconds later. Together the data from these two experiments suggest that the time course of sign repetition priming effects more closely mirrors results from spoken and written word repetition priming when the videos of signs are edited so that stimulus onset is aligned with sign onset, as in Experiment 2.

5 |. DISCUSSION

The linguistic signal for sign languages differs from spoken languages because the articulators are directly observable (unlike spoken words) and because the visual signal unfolds over time (unlike written words). We used repetition priming and varied the ERP time-lock point (video onset vs. sign onset) and stimulus type (full videos vs. clipped videos of signs) to uncover the time course of the N400 priming effect, which indexes lexical access and semantic processing.

The results of Experiment 1 revealed that when ERPs were time-locked to video onset, the polarity and distributional pattern of repetition effects in the 600 to 1000 ms period were consistent with the N400 repetition priming effect reported in numerous previous language studies using written and spoken words, but the time-course of the effect started substantially later. For word stimuli, typical N400 effects start in the 200 to 300 ms post-word onset epoch and usually continue for about 200 ms in written word studies and perhaps 300 or 400 ms in spoken word studies (e.g., Holcomb et al., 2005; Holcomb & Neville, 1990). Comparable priming effects were not significant until the 500 to 600 ms window when ERPs were time-locked to video onset. Moreover, the effects of sign repetition priming continued until at least 1000 ms, for a total duration of 500 ms. Given that the average length of the transition from the onset of the video to sign onset was 578 ms, it is perhaps not surprising that the onset of the N400 was delayed compared to written or spoken words. The extended duration of the priming effect might not be surprising either since there was substantial variability in the timing of the sign onset across the different videos (SD = 96 ms). In addition, signs may have also differed in the predictive phonological information available within the transition to sign onset. For example, in gating studies in which the “gates” are 1 video frame (33 ms; see Figure 1), signs with marked handshapes can be identified earlier than those with unmarked handshapes (Emmorey & Corina, 1990). In addition, deaf signers, unlike hearing non-signers, use handshape information within transitional movements to identify target pseudosigns, suggesting that transitional movements contain linguistic information for signers (Brozdowski, 2018).

In fact, when ERPs were time-locked to the sign onset within the video, the repetition priming effect started 100 ms prior to this point on average. This result suggests that information within the transition to sign onset was enough to signal whether a target sign was a repeat or not a repeat of the prior sign. An interesting question raised by this result is whether the repetition priming effect within this early (pre-onset) time window represents priming of sublexical features (e.g., handshape) for repeated trials or represents early lexical priming (i.e., initial access to the lexico-semantic representation of the repeated sign). One way to test these possibilities would be run a version of the experiment using semantic as opposed to repetition priming, which more specifically targets the N400 and would involve completely different stimuli for both the related and unrelated contrast. If signers are able to recognize and access isolated signs prior to sign onset (on average), then the N400 priming effect should emerge prior to sign onset within the video.

Another way to investigate this question would be to conduct a gating study to determine the point at which a given sign is recognized, which may be prior to sign onset. For example, Emmorey and Corina (1990) found that on average signs were isolated (i.e., recognized but without confidence) within 240 ms from the start of the video, and signs were recognized (with confidence) after 310 ms (see also Grosjean, 1982). The gating results would provide the “uniqueness” or recognition point for individual signs within a stimulus video. For auditorily presented words, O’Rourke and Holcomb (2002) found that the onset of the N400 was earlier for spoken words with an early uniqueness point (e.g., pupil) than for words with a late uniqueness point (e.g., carriage). In addition, when time-locking was to the recognition point (rather than to audio clip onset), the time course of the N400 was shorter and identical for the two word types. If sign recognition processes are parallel to spoken word processes, then we would predict the same pattern of results.

When time-locking to either video-or sign-onset, the duration of the N400 was relatively prolonged, which we suggest is due to the prolonged and dynamic nature of signed stimuli and in particular to the critical linguistic content of signs taking comparatively long to unfold. The average duration of sign videos in this experiment was 2177 ms (SD = 287). One question is whether this longish duration of the sign videos is the primary culprit in the 500 ms duration of the N400 or whether processing of signs is inherently sluggish compared to visual and spoken words. One clue to answering this question is that realigning the time-lock point from video onset to sign onset did not cause the onset of the N400 priming effect to move to precisely the point of sign onset. As noted above, we interpret this finding as an indication that signers were extracting critical linguistic information prior to sign onset and that they used this information to begin to detect repetitions of the signs. The results of Experiment 2 provide support for this interpretation. In this experiment, the same sign videos were clipped so that stimulus onset and sign onset were at the same point, and this manipulation resulted in a shorter N400 duration.

In sum, the results of Experiments 1 and 2 indicate that signers utilize linguistic information within the observed transition to sign onset (see initial frames in Figure 1) to identify a sign repetition because priming effects began prior to sign onset within a video (Figures 5 and 8). This finding replicates the results of Hosemann et al. (2013) who reported N400 effects prior to the onset of a target sign within a sentence context.

The time-course, amplitude, and distribution of the N400 priming effect did not differ between video-and sign-onset time-locking (see Figures 7 and 8), suggesting that both analyses are equally sensitive for detecting N400 priming effects. Time-locking to sign onset within a video did not yield a larger or tighter N400 priming effect, and this analysis required using a 600 ms pre-stimulus-onset baseline due to variable information during the transition period. In addition, although inter-rater reliability for identifying sign onsets and offsets is relatively high (Caselli et al., 2017), these judgements are likely to be more variable and subjective than identifying stimulus onset for full videos (i.e., the frame before the hand appears or begins to move).

Furthermore, time-locking to sign onset within a video missed or obscured earlier ERP components that were observed in both the full video and clipped video analyses: P1, P2, N1, and N300 (Figures 4 and 6). These components are typically elicited by visual stimuli, including written words and pictures (Luck, 2014). In a previous study, we found that lexical frequency modulates P1 and N1 amplitudes when full sign videos are presented and time-locking is to video-onset (Emmorey et al., 2020). We attributed this very early frequency effect to signers’ sensitivity to the frequency of the handshapes that are perceived during the transitional movement because more frequent signs tend to contain more frequent handshapes. The MEG study by Almeida et al. (2016) also found evidence of signers’ early visual sensitivity to hand configuration on the M100 compnent. Other ERP studies have found differences in the P1 and N1 for deaf signers compared to hearing controls for non-linguistic visual stimuli (e.g., Bottari et al., 2014; Neville & Lawson, 1987). The N300 component is typically observed in experiments that present pictures or gestures and is hypothesized to be involved in processing early visual semantic features (e.g., Hamm et al., 2002; Wu & Coulson, 2007). Recently, some studies have suggested that the N300 component for signs might index the mapping between sub-lexical and lexical representations (Emmorey et al., 2020; Meade et al., 2018; Meade et al., submitted). Given these previous results and those of the current study, we suggest that when full videos are presented as stimuli, time-locking to video-onset, rather than sign-onset, will provide a more comprehensive picture of the processes involved in sign recognition.

The timing and duration of the N400 priming effect for signs was most parallel to that observed for words when clipped videos were presented (Experiment 2). Time-locking to full video-onset or sign-onset within a full video resulted in an N400 component that was much more elongated. This pattern was most likely due to variability in sign onsets within a video and variability in the information content of transitions across signs. By clipping the videos to sign onset, transitional variability was removed, rendering the sign stimuli more like auditory word stimuli and shortening the N400 component by 200 ms. Together these results suggest that clipped videos of signs may be most appropriate for ERP experiments focused on N400 priming effects, and we have successfully used clipped video stimuli to examine phonological and semantic priming in ASL (Lee et al., 2019; Meade et al., 2018), as have others for a different sign language (Spanish Sign Language: Guttiérrez et al., 2012). Full video stimuli may be more appropriate for studies that focus on early visual components of sign recognition or that compare sign recognition across learners and proficient signers who may differ in their ability to utilize transitional information (Mott et al., 2020; Ortega et al., 2020).

ACKNOWLEDGEMENTS

This work was supported in part by a grant from the National Institute on Deafness and Other Communication Disorders (R01 DC010997). We would like to thank Lucinda O’Grady Farnady for help with this study, and we are grateful to all of the deaf participants without whom this research would not be possible.

Funding information

National Institute on Deafness and Other Communication Disorders, Grant/Award Number: R01 DC010997

Footnotes

CONFLICTS OF INTEREST

The authors report no conflicts of interest.

REFERENCES

  1. Almeida D, Poeppel D, & Corina D (2016). The processing of biologically plausible and implausible forms in American Sign Language: Evidence for perceptual tuning. Language, Cognition and Neuroscience, 31(3), 361–374. 10.1080/23273798.2015.1100315 [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Anderson JE, & Holcomb PJ (1995). Auditory and visual semantic priming using different stimulus onset asynchronies: An event related brain potential study. Psychophysiology, 32(2), 177–190. 10.1111/j.1469-8986.1995.tb03310.x [DOI] [PubMed] [Google Scholar]
  3. Bottari D, Heimler B, Caclin A, Dalmolin A, Giard MH, & Pavani F (2014). Visual change detection recruits auditory cortices in early deafness. NeuroImage, 94, 172–184. 10.1016/j.neuroimage.2014.02.031 [DOI] [PubMed] [Google Scholar]
  4. Brentari D (2019). Sign language phonology. Cambridge University Press. [Google Scholar]
  5. Brozdowski CR (2018). Forward modeling in the manual modality: Linguistic and nonlinguistic predictions by American Sign Language users. University of California, San Diego. https://escholarship.org/uc/item/2kk8z7sxescholarship.org/uc/item/2kk8z7sx [Google Scholar]
  6. Caselli NK, Sehyr ZS, Cohen-Goldberg AM, & Emmorey K (2017). ASL-LEX: A lexical database of American Sign Language. Behavior Research Methods, 49(2), 784–801. 10.3758/s13428-016-0742-0 [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Crasborn O, Bank R, Zwitserlood I, van der kooij E, Meijer A, Sáfár A, & Ormel E (2015). Annotation conventions for the corpus NGT, version 3. 10.13140/RG.2.1.1779.4649 [DOI] [Google Scholar]
  8. Doyle MC, Rugg MD, & Wells T (1996). A comparison of the electrophysiological effects of formal and repetition priming. Psychophysiology, 33(2), 132–147. 10.1111/j.1469-8986.1996.tb02117.x [DOI] [PubMed] [Google Scholar]
  9. Emmorey K, & Corina D (1990). Lexical recognition in sign language: Effects of phonetic structure and morphology. Perceptual and Motor Skills, 71, 1227–1252. 10.2466/pms.1990.71.3f.1227 [DOI] [PubMed] [Google Scholar]
  10. Emmorey K, Winsler K, Midgley KJ, Grainger J, & Holcomb PJ (2020). Neurophysiological correlates of frequency, concreteness, and iconicity in American Sign Language. Neurobiology of Language, 1(2), 249–267. 10.1162/nol_a_00012 [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Grainger J, & Holcomb PJ (2009). Watching the word go by: On the time-course of component processes in visual word recognition. Language and Linguistics Compass, 3(1), 128–156. 10.1111/j.1749-818X.2008.00121.x [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Groppe DM, Urbach TP, & Kutas M (2011). Mass univariate analysis of event-related brain potentials/fields I: A critical tutorial review. Psychophysiology, 48(12), 1711–1725. 10.1111/j.1469-8986.2011.01273.x [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Grosjean F (1982). Sign and word recognition: A first comparison. Sign Language Studies, 1032(1), 195–220. 10.1353/sls.1982.0003 [DOI] [Google Scholar]
  14. Gutiérrez E, Müller O, Baus C, & Carreiras M (2012). Electrophysiological evidence for phonological priming in Spanish Sign Language lexical access. Neuropsychologia, 50(7), 1335–1346. 10.1016/j.neuropsychologia.2012.02.018 [DOI] [PubMed] [Google Scholar]
  15. Hamm JP, Johnson BW, & Kirk IJ (2002). Comparison of the N300 and N400 ERPs to picture stimuli in congruent and incongruent contexts. Clinical Neurophysiology, 113(8), 1339–1350. 10.1016/S1388-2457(02)00161-X [DOI] [PubMed] [Google Scholar]
  16. Hauser PC, Paludneviciene R, Riddle W, Kurz KB, Emmorey K, & Contreras J (2016). American Sign Language comprehension test: A tool for sign language researchers. Journal of Deaf Studies and Deaf Education, 21(1), 64–69. 10.1093/deafed/env051 [DOI] [PubMed] [Google Scholar]
  17. Holcomb PJ, Anderson A, & Grainger J (2005). An electrophysiological investigation of cross-modal repetition priming at different stimulus onset asynchronies. Psychophysiology, 42, 493–507. 10.1111/j.1469-8986.2005.00348.x [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Holcomb PJ, & Neville HJ (1990). Auditory and visual semantic priming in lexical decision: A comparison using event-related brain potentials. Language and Cognitive Processes, 5(4), 281–312. 10.1080/01690969008407065 [DOI] [Google Scholar]
  19. Hosemann J, Herrmann A, Steinbach M, Bornkessel-Schlesewsky I, & Schlesewsky M (2013). Lexical prediction via forward models: N400 evidence from German Sign Language. Neuropsychologia, 51(11), 2224–2237. 10.1016/j.neuropsychologia.2013.07.013 [DOI] [PubMed] [Google Scholar]
  20. Johnson RE, & Liddell SK (2011). A segmental framework for representing signs phonetically. Sign Language Studies, 11(3), 408–463. 10.1353/sls.2011.0002 [DOI] [Google Scholar]
  21. Kutas M, & Federmeier KD (2011). Thirty years and counting: Finding meaning in the N400 component of the Event-Related Brain Potential (ERP). Annual Review of Psychology, 62(1), 621–647. 10.1146/annurev.psych.093008.131123 [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Kutas M, & Hillyard SA (1980). Reading senseless sentences: Brain potentials reflect semantic incongruity. Science, 207(4427), 203–205. 10.1126/science.7350657 [DOI] [PubMed] [Google Scholar]
  23. Lee B, Meade G, Midgley KJ, Holcomb PJ, & Emmorey K (2019). ERP evidence for co-activation of English words during recognition of American Sign Language signs. Brain Sciences, 9(6), 148. 10.3390/brainsci9060148 [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Luck SJ (2014). An introduction to the event-related potential technique. MIT press. [Google Scholar]
  25. McPherson WB, & Holcomb PJ (1999). An electrophysiological investigation of semantic priming with pictures of real objects. Psychophysiology, 36, 53–65. 10.1017/s0048577299971196 [DOI] [PubMed] [Google Scholar]
  26. Meade G, Lee B, Massa N, Holcomb PJ, Midgely KJ, & Emmorey K (submitted). Are form priming effects phonological or perceptual? Electrophysiological evidence from American Sign Language. [DOI] [PMC free article] [PubMed]
  27. Meade G, Lee B, Midgley KJ, Holcomb PJ, & Emmorey K (2018). Phonological and semantic priming in American Sign Language: N300 and N400 effects. Language, Cognition and Neuroscience, 33(9), 1092–1106. 10.1080/23273798.2018.1446543 [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Mott M, Midgley KJ, Holcomb PJ, & Emmorey K (2020). Cross-modal translation priming and iconicity effects in deaf signers and hearing learners of American Sign Language. Bilingualism: Language and Cognition, 1–13. 10.1017/S1366728919000889 [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. Neville HJ, & Lawson D (1987). Attention to central and peripheral visual space in a movement detection task: An event-related potential and behavioral study. II. Congenitally deaf adults. Brain Research, 405(2), 268–283. 10.1016/0006-8993(87)90296-4 [DOI] [PubMed] [Google Scholar]
  30. O’Rourke TB, & Holcomb PJ (2002). Electrophysiological evidence for the efficiency of spoken word processing. Biological Psychology, 60(2), 121–150. 10.1016/S0301-0511(02)00045-5 [DOI] [PubMed] [Google Scholar]
  31. Ortega G, Özyürek A, & Peeters D (2020). Iconic gestures serve as manual cognates in hearing second language learners of a sign language: An ERP study. Journal of Experimental Psychology: Learning, Memory, and Cognition, 46(3), 403–415. 10.1037/xlm0000729 [DOI] [PubMed] [Google Scholar]
  32. Pickering E, & Schweinberger SR (2003). N200, N250r, and N400 event-related brain potentials reveal three loci of repetition priming for familiar names. Journal of Experimental Psychology: Learning, Memory, and Cognition, 29(6), 1298–1311. 10.1037/0278-7393.29.6.1298 [DOI] [PubMed] [Google Scholar]
  33. Rugg M (1990). Event-related brain potentials dissociate repetition effects of high-and low-frequency words. Memory and Cognition, 18, 367–379. 10.3758/BF03197126 [DOI] [PubMed] [Google Scholar]
  34. Sitnikova T, Kuperberg G, & Holcomb PJ (2003). Semantic integration in videos of real-world events: An electrophysiological investigation. Psychophysiology, 40(1), 160–164. 10.1111/1469-8986.00016 [DOI] [PubMed] [Google Scholar]
  35. Thompson R, Emmorey K, & Gollan TH (2005). “Tip of the Fingers” experiences by deaf signers: Insights into the organization of a sign-based lexicon. Psychological Science, 16(11), 856–860. 10.1111/j.1467-9280.2005.01626.x [DOI] [PubMed] [Google Scholar]
  36. Wu YC, & Coulson S (2007). How iconic gestures enhance communication: An ERP study. Brain and Language, 101(3), 234–245. 10.1016/j.bandl.2006.12.003 [DOI] [PubMed] [Google Scholar]

RESOURCES