Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2022 Nov 15.
Published in final edited form as: Neuropsychologia. 2021 Oct 5;162:108051. doi: 10.1016/j.neuropsychologia.2021.108051

Matching Pictures and Signs: An ERP Study of the Effects of Iconic Structural Alignment in American Sign Language

Meghan E McGarry a, Natasja Massa b, Megan Mott c, Katherine J Midgley c, Phillip J Holcomb c, Karen Emmorey b
PMCID: PMC8560580  NIHMSID: NIHMS1747582  PMID: 34624260

Abstract

Event-related potentials (ERPs) were used to explore the effects of iconicity and structural visual alignment between a picture-prime and a sign-target in a picture-sign matching task in American Sign Language (ASL). Half the targets were iconic signs and were presented after a) a matching visually-aligned picture (e.g., the shape and location of the hands in the sign COW align with the depiction of a cow with visible horns), b) a matching visually-nonaligned picture (e.g., the cow’s horns were not clearly shown), and c) a non-matching picture (e.g., a picture of a swing instead of a cow). The other half of the targets were filler signs. Trials in the matching condition were responded to faster than those in the non-matching condition and were associated with smaller N400 amplitudes in deaf ASL signers. These effects were also observed for hearing non-signers performing the same task with spoken-English targets. Trials where the picture-prime was aligned with the sign target were responded to faster than non-aligned trials and were associated with a reduced P3 amplitude rather than a reduced N400, suggesting that picture-sign alignment facilitated the decision process, rather than lexical access. These ERP and behavioral effects of alignment were found only for the ASL signers. The results indicate that iconicity effects on sign comprehension may reflect a task-dependent strategic use of iconicity, rather than facilitation of lexical access.

Keywords: sign language, iconicity, ERPs, deaf, structural alignment

The phonological form of spoken words is usually arbitrary, with little or no relationship to the word’s meaning, e.g., the of concept of ‘cow’ is expressed as vaca in Spanish, inek in Turkish, and ko in Danish. Lexical items with a motivated relationship between form and meaning are referred to as iconic. In spoken languages, iconic words have long been believed to be rare and limited to instances such as onomatopoeia. However, it is becoming increasingly apparent that iconicity is more prevalent than previously thought (Dingemanse et al., 2015; Imai et al., 2015; Perniss et al., 2010; Perry et al., 2015). Further, iconicity has been found to be widespread across the lexicons of sign languages (Aronoff et al., 2005; Bellugi & Klima, 1976; Östling et al., 2018; Padden et al., 2013). The apparent difference in the extent of iconicity across spoken and signed languages may be due to the visual three-dimensionality of sign languages which allows for iconic depiction of a wide range of basic concepts, such as object and human actions, movements, locations, and shapes (Taub, 2001).

Iconicity can be viewed as a structured mapping between a mentally-stored conceptual representation and a phonological representation (Emmorey, 2014). This view builds from the Structure-Mapping theory proposed by Gentner and colleagues to account for the nature of analogies and metaphors in spoken languages (Gentner, 1983; Gentner & Markman, 1997). This approach also draws on the Analogue-Building model proposed by Taub (2001) to account for the structure of iconic mappings between form and meaning in both signed and spoken languages. The structure-mapping account proposes that the perception of iconicity involves a comparison process between two stored mental representations: the mental concept denoted by the word or sign (including any distinctive semantic features) and the phonological representation of the lexical item. For example, consider the sign for ‘cow’ in American Sign Language (ASL). The phonological form (a Y handshape located at the temple) depicts the horns on a cow’s head (a salient semantic feature) (see Figure 1 in Methods for an illustration of the ASL sign COW).

Figure 1.

Figure 1.

Open in a new tab

Examples of picture stimuli in each condition. For the English speakers, a video of the same model saying the word “cow” was presented.

The Structure-Mapping model of iconicity can be extended to the mapping between a visual stimulus (i.e., a picture) and the phonology of an iconic sign. For example, when comparing the two different drawings of a cow shown in Figures 1A and 1B (see below), one picture clearly depicts the cow’s horns, and so maps onto the phonological form of the ASL sign COW. The other picture depicts the entire body of the cow such that the horns are not prominent, and so this picture does not map as clearly onto the form of the sign. We refer to the mapping between features depicted in a picture and the phonological form of the sign as structural ‘alignment.’ Pictures that have a clear visual mapping to an iconic sign are considered visually ‘aligned,’ and those that do not are considered ‘non-aligned’. In the present study we used a picture-sign matching task and event-related potentials (ERPs) to investigate the effects of structural alignment on sign comprehension in deaf signers of ASL.

Thompson, Vinson, and Vigliocco (2009) previously investigated whether this type of visual overlap between a picture and an iconic sign impacted sign recognition times in a picture-sign matching task. In this study, researchers presented pictures followed by video clips of ASL signs to deaf and hearing ASL signers. In half of the trials, the picture-prime and the sign-target matched in meaning (e.g., a picture of a bird was followed by the sign BIRD), while in the other half the picture and the sign did not match. For the critical subset of iconic signs, the matching picture was either aligned with the sign (e.g., a picture of a bird in profile with a prominent beak, which maps to the depiction of a bird’s beak in the ASL sign) or the picture was non-aligned (e.g., a picture of a bird in flight). Participants were asked to decide whether the picture and the sign matched. Monolingual English speakers served as a control group and decided whether the same pictures matched videoclips of a spoken word (translations of the ASL signs).

Thompson et al. (2009) found that signers responded significantly faster when the picture-prime and the sign-target were visually aligned than when they were non-aligned. No advantage was observed for the aligned pictures for the English-speaking control participants (Thompson et al., 2009). Vinson et al. (2015) replicated these findings with different pictures and with deaf signers of British Sign Language (historically unrelated to ASL). These authors suggested that the presence of the same visually salient features in both the picture and the sign resulted in faster recognition of the sign form (Vinson et al., 2015). However, the mechanism underlying this facilitation effect is unclear. Faster matching decisions for visually aligned picture-sign pairs could be due to priming, i.e., aligned visual features of the picture prime the phonological features of the sign, thus speeding lexical access. However, it is also possible that participants recognized that structural mapping between the aligned pictures and signs, which facilitated the matching decision. The use of ERPs can help to distinguish between these hypotheses because ERPs provide information about the time-course of lexical access and decision processes, whereas reaction times (RTs) can only reflect the cumulation of these different processes. In the present study, we attempted to replicate the picture-sign alignment effects found by Thompson et al. (2009) and Vinson et al. (2015), while recording EEG from deaf ASL signers.

Recently, we have found picture-sign alignment ERP and RT effects for sign production using a picture-naming task (McGarry et al., 2020). Aligned pictures (e.g., Figure 1A) were named faster than non-aligned pictures (e.g., Figure 1B) for the same concept, and there was no difference in naming times for the control group of non-signing participants naming the same pictures in English. McGarry et al. (2020) also found that naming aligned pictures resulted in a reduced N400 amplitude compared to naming non-aligned pictures. The N400 is an ERP component that is sensitive to lexico-semantic processes, and reduced N400 amplitude (less negativity) is typically interpreted as reflecting less effortful lexical access and retrieval (see Kutas & Federmeier, 2011). McGarry et al. (2020) interpreted the reduced N400 for naming aligned pictures as reflecting a lexical-phonological priming effect due to the overlap between visual elements in the picture and the visual phonological form of the sign. If the RT facilitation effect observed for alignment in the picture-sign matching task is similarly due to priming during lexical access, then we should observe a reduced N400 for target signs preceded by aligned picture primes compared to non-aligned pictures. However, if faster RTs for aligned pictures are due to a post-lexical strategic effect, then alignment should modulate a different electrophysiological component that indexes decisional processes, namely the P3 component.

The P3 component1 is a positive-peaking wave that has been shown to reflect decision difficulty and the effort or resources that are required to perform a given task (Luck, 2014; Twomey et al., 2015; for recent reviews see Hajcak & Foti, 2020 and Verleger, 2020). The time window of the P3 is generally more variable than for the N400, and the two components can overlap in time, although in most language studies the P3 tends to peak after the N400 (Bentin et al., 1999). However, the P3 has a different polarity than the N400, such that reduced amplitude (less positivity, rather than less negativity) is associated with easier decision making. More specifically, the P3 is modulated by the amount of perceived evidence for a given decision 1Herding et al., 2019; Twomey et al., 2015, 2016). The amplitude of the P3 component increases during the pre-decision window as the necessary information is accumulated prior to the response. In the case where a decision is easily made (e.g., because less evidence is required or the evidence is strong), the amplitude of the P3 component is reduced compared to more difficult decisions (e.g., when the evidence is weaker). As the picture-sign matching paradigm requires participants to make explicit decisions on the relatedness between a picture prime and sign target, we predict that P3 amplitude to target signs will reflect the different levels of evidence and demands for cognitive resources in the aligned vs. non-aligned conditions. If structural alignment results in easier decision-making in the picture-sign matching task, we predict faster decision latencies and a reduced P3 in the aligned condition compared to the non-aligned condition because the evidence for the “yes” matching decision is stronger in the aligned condition. While the correct response in both the aligned and unaligned conditions will be a “yes” response, we predict differences in these “yes” responses on the P3 component and the response latencies. Regardless of whether the N400 or P3 components are modulated by the alignment manipulation, we do not expect these components to be modulated by alignment for control participants undertaking the same task with spoken English words since these words are not iconic and have no form overlap with the picture primes.

Finally, regardless of picture alignment, we expect to find faster RTs for matching trials (“yes” responses) than mismatching trials (“no” responses) for both ASL signers and English speakers. Faster RTs for matching trials is well-established and is due in part to semantic priming of the target item by the related prime (Bentin et al., 1993; Brown & Hagoort, 1993). Thus, we also expect to find smaller N400 amplitudes (less negativity) for matched compared to mismatched trials for both deaf ASL signers and English-speaking control participants.

In sum, the present study comprises an exploration into the nature of facilitation (priming) for structural alignment in a picture-sign matching task. In this investigation, we attempt to replicate the finding of reduced RTs when the picture-prime and the sign-target are visually aligned, and we explore whether structural alignment impacts lexico-semantic processing (indexed by N400 amplitude) and/or decision making (indexed by P3 amplitude).

Methods

Participants

Deaf participants included 24 ASL signers (13 female; mean age = 33.6 years, SD = 7.7 years). Twelve participants were native signers born into signing families, and 12 were early ASL signers who began learning ASL prior to 5 years of age2. All had normal or corrected-to-normal vision, and had no history of any language, reading, or neurological disorder. Two participants were left-handed.

Control participants were 24 monolingual hearing English speakers ( 12 female; mean age = 28 years, SD = 6.6 years) who had normal or corrected-to-normal vision, and had no history of any language, reading, or neurological disorders. Three participants were left-handed. Control participants were screened for ASL knowledge and had no exposure to ASL beyond knowing a few signs or the fingerspelled alphabet.

All participants received monetary compensation in return for participation. Informed consent was obtained from all participants in accordance with the Institutional Review Board at San Diego State University. One additional deaf participant and one additional hearing participant were excluded prior to analyses due to excess artifacts present in the EEG recordings.

Materials.

Stimuli consisted of pairs of pictures (primes) and videoclips of signs or spoken words (targets). The primes consisted of digitized black on white line drawings. A total of 360 different pictures were presented representing 180 different concepts. Each picture was followed by a videoclip of a hearing native ASL-English signer, who either produced an ASL sign (for deaf participants) or spoke an English word (for hearing controls). Each ASL stimulus was clipped 3 frames (100ms) before sign onset (see Caselli et al. 2017 for details on how sign onsets and offsets were determined). The English stimuli began with the model sitting silently, and the video was clipped 9 frames (300ms) before voice onset. The sign or word target could either match or not match the preceding picture, with trials matching 50% of the time.

For deaf signers ASL stimuli included 60 video clips of iconic signs. These items comprised the critical experimental stimuli used to form the alignment manipulation. Each iconic sign was presented three times, once preceded by a picture that matched the sign in meaning and was also visually aligned with the target sign (Figure 1A), once preceded by a matching picture that was non-aligned to the target sign (Figure 1B), and once preceded by a picture that did not align or match the meaning of the target sign (Figure 1C). The remaining 60 ASL video clips were filler target stimuli, including both iconic and non-iconic signs. On filler trials, none of the preceding pictures were aligned with the target signs; however, each of these targets was preceded twice by a non-matching picture, and once by a picture that matched the meaning of the sign. For the hearing speakers, the ASL target signs were replaced with audio-visual video clips of the same model producing the equivalent English word. Each participant saw all 360 trials (each of the 120 video clips preceded three times by a unique picture prime), and the order of the prime-target pairs was counterbalanced across participants in two pseudorandom lists.

Using data retrieved from the ASL-LEX database (Caselli et al., 2017; Sevcikova Sehyr, Caselli, Goldberg, & Emmorey, 2020), the 60 critical targets and the 60 filler targets were assessed for ASL frequency and iconicity. For this database, deaf signers rated the frequency of signs in everyday conversation on a scale from 1 (very infrequent) to 7 (very frequent). The frequency of the ASL targets was balanced across the critical stimuli (mean = 3.74, SD = 1.24) and filler stimuli (mean = 3.48, SD = 1.20), p = 0.27. The translated English targets were also balanced for frequency (retrieved from the CELEX database; Baayen, 1995), across the critical stimuli (mean = 2.96, SD = 0.63) and filler stimuli (mean = 2.90, SD = 0.63), p = 0.58. Iconicity ratings were from hearing non-signers who rated how much a sign resembled its meaning on a scale of 1 (not iconic at all) to 7 (very iconic). Iconicity was balanced between critical stimuli (mean = 4.63, SD = 1.55) and filler stimuli (mean = 4.28, SD = 1.43), p = .21.

Prior to inclusion in the study, each picture was rated by 30 Mturk workers through an online Qualtrics survey. Participants were given the English word that matched the picture, and asked to generate a mental image of the concept. Once they had done so, the picture was presented to the participant, and each participant scored the picture on a 5-point scale of how well it matched their mental representation. The average score reflects the prototypicality of the picture, with a high score indicating high prototypicality. Prototypicality scores were high across pictures in both the matching and non-matching conditions (matching = 4.47; non-matching = 4.57, p = .20). Crucially, the aligned and non-aligned pictures were also balanced for prototypicality (aligned = 4.47; non-aligned = 4.47, p = .83). Because the aligned condition requires there to be two unique pictures for a given sign, ensuring that the aligned and non-aligned pictures were equally prototypical helps ensure that any behavioral or electrophysiological effects are not driven by prototypicality differences.

Procedure.

For both signers and speakers, each trial began with a 1000ms purple fixation cross in the center of the screen, followed by a 400ms gray fixation cross, after which the picture appeared for 500ms. After the presentation of the picture and a 300 ms blank screen, the target videoclip was presented, and the final frame was displayed on the screen until the participant pressed a button on a gamepad indicating whether the picture and the word/sign matched or not (left and right button presses were counterbalanced across participants). Hearing participants wore a headset to enable them to clearly hear the spoken English word from the video clips. Response times were calculated from sign onset or spoken word onset (not video onset) to the button press. After making their match/not-match decision, participants then saw a blank screen for 500ms.

Participants were advised that they could blink during the purple fixation cross or during the longer blink breaks that came about every fifteen trials. They were asked to prepare for the upcoming trial upon the presentation of the gray fixation cross and not to blink during stimuli presentation. Participants were also provided with three self-timed breaks during the experiment. Participants were advised to make their decision as quickly and accurately as possible. To familiarize the participants with the task, a practice set with fifteen trials that were not included in the experiment was given to all participants before the experiment.

EEG recording.

During recording, participants wore an elastic cap (Electro-Cap) containing 29 active electrodes (see Figure 2 for an illustration of the montage of electrodes). An electrode located on the participant’s left mastoid was used as the reference.3

Figure 2.

Figure 2.

Open in a new tab

Electrode montage. Shaded sites indicate the electrodes included in the analyses.

Identification and rejection of trials with artifacts, including blinks and horizontal eye movements, was conducted using recordings from electrodes located on the outer canthus of the right eye (A1 reference) and a bipolar montage located above (FP1) and the left eye. Using saline gel (Electro-Gel), all mastoid, eye and scalp electrode impedances were maintained below 2.5 kΩ. EEG was amplified with SynAmpsRT amplifiers (Neuroscan-Compumedics) with a bandpass of DC to 100 Hz and was sampled continuously at 500 Hz. Offline averaged ERPs were computed between −300 and 900 ms with a baseline of −300 to 0 from artifact free single trial EEG in each condition. Averaged ERPs were low passed filtered at 20 Hz prior to analysis.

Mean amplitude was calculated for a typical N400 window (300-500ms after lexical onset), as well as a later P3 (500-700ms after lexical onset). These windows were selected to allow for separate analysis of the N400 lexico-semantic effects and the P3 decisional effects. likely to occur. For each time window, two repeated-measure ANOVAs were conducted. The first analysis was aimed at examining the effects of semantic relatedness (i.e., matching vs. mismatched picture-sign pairs). For this analysis, there were two levels of Relatedness (Matching, Mismatched), three of Laterality (Left, Midline, Right), and five of Anteriority (FP, F, C, P3, O; see Figure 2). The second analysis was aimed at testing for effects of picture-sign alignment, and there were two levels of alignment (Aligned, Non-Aligned), three of Laterality, and five of Anteriority.

Results: Matching vs. Mismatched Trials

To identify semantic relatedness effects between picture-primes and sign-targets, we compared the 180 trials where the picture and the target sign/word referred to the same concept with the 180 trials where the prime and the target sign/word were unrelated. Due to modality differences between signs and words, signing participants had to visually recognize the targets, while English-speaking participants had to recognize the targets auditorily. In light of this difference, we did not directly compare the results from the signers and speakers. EEG data containing ERP artifacts were excluded prior to analysis (7% of data overall).

Behavioral Effects

Trials with naming latencies shorter than 300ms or longer than 2.5 standard deviations above the individual participant’s mean response time (RT) were excluded from analysis (5% of the data). Trials with incorrect responses were excluded from both the RT and the ERP analyses for both the ASL signers (11 trials or 3% on average) and the English speakers (6 trials or 2% on average). To statistically compare the differences in accuracy and RTs between matched and mismatched trials, we used a binomial linear mixed effects model, with items and participants as random intercepts, and relatedness as the fixed effect for both groups of participants. The results are given in Table 1.

Table 1.

Mean response times (ms) and accuracy (percent correct) for matching and mismatching trials. Standard deviations are in parentheses.

Response Time Accuracy
Matching Mismatched Matching Mismatched
ASL Signers 673 (126) 741 (126) 99 (1.42) 95 (2.66)
English Speakers 683 (157) 764 (138) 99 (1.19) 98 (2.53)
Open in a new tab

Response times.

For signers, RTs for matching trials (“yes” responses) were faster than for mismatched trials (“no” responses), t = 12.06 p <0.0001. Similarly, for speakers RTs for matching trials were faster than for mismatched trials, t = 17.61, p <0.0001.

Accuracy.

Mean accuracy was high for both the signing participants and the English-speaking participants (97% and 98% of all trials, respectively); see Table 1. Both signers and speakers demonstrated higher accuracy for mis-matching trials than matching trials, but this difference was significant only for the signing participants, t = 9.212, p <0.0001, and not for the English speakers, t = −0.354, p = 0.71.

Electrophysiological Effects

N400 window (300-500ms).

In this epoch (300 to 500 milliseconds after speech or sign onset), there was a widespread main effect of Relatedness for both signers, F(1,24) = 108.23, p <0.0001, and speakers, F(1,24) = 130.8, p <0.0001). ERPs for signs and words preceded by mismatched pictures elicited larger centro-posterior negativities than signs and words preceded by matching pictures (see Figures 3 and 4).

Figure 3.

Figure 3.

Open in a new tab

EEG recordings and voltage maps for ASL signers. Voltage maps are shown from 300-500ms and 500-700ms after the onset of the target sign. The scale used for voltage maps is +/− 6 microvolts.

Figure 4.

Figure 4.

Open in a new tab

EEG recordings and voltage maps for hearing English speakers. Voltage maps are shown from 300-500ms and 500-700ms after the onset of the target spoken word. The scale used for voltage maps is +/− 6 microvolts.

P3 window (500-700ms).

In this epoch, the significant main effect of matching persisted for both signers, F(1,24) = 37.39, p <0.0001, and speakers F(1,24) = 162.34, p <0.0001.

Results: Aligned vs. non-aligned trials

To test the effect of alignment within matching trials, we compared the 60 aligned picture-sign trials with the 60 non-aligned trials. Trials were only aligned or non-aligned for the signs, and the English speakers served as controls as the form of the spoken words had no relation to any of the pictures. As in the Relatedness analyses above, we used a linear mixed effects model, with items and participants as random intercepts, but alignment, rather than Relatedness, was the fixed effect factor. Results are given in Table 2.

Table 2.

Mean response times (ms) and accuracy (percent correct) for aligned and non-aligned trials, for both signers and speakers. Standard deviations are in parentheses.

Response Time Accuracy
Aligned Non-aligned Aligned Non-Aligned
ASL Signers 640 (104) 659 (109) 97 (1.15) 95 (1.03)
English Speakers 669 (103) 675 (154) 98 (1.66) 97 (3.33)
Open in a new tab

Behavioral effects

Response times.

For signers, RTs for aligned trials were faster than for non-aligned trials, t = 5.09, p <0.0001. For speakers, there was no RT difference between these trial types, t = −0.90, p = 0.49, indicating that the effect for alignment is specific to signing participants.

Accuracy.

Both signers and speakers were highly accurate when determining whether the target sign matched the picture for both aligned and non-aligned picture-word/sign pairs (see Table 2). Signers were more accurate for aligned than non-aligned trials, t = −2.527, p <0.011, whereas there was no difference in accuracy between these trial types for English speakers, t = −0.90, p = 0.36.

Electrophysiological effects

N400 window (300-500ms).

In the typical N400 epoch, sign targets preceded by aligned pictures elicited greater negativities than sign targets preceded by non-aligned pictures, F(1,24) = 8.78, p = 0.007 (see Figure 5). The effect of alignment was strongest over the anterior and central sites, as evidenced by a significant Alignment x Anteriority interaction, F(4,96) = 3.22, p = 0.016). The alignment effect was also strongest over the midline sites, as evidenced by a significant Alignment x Laterality interaction, F(4,96) = 3.72, p = 0.031. In contrast, there were no significant effects of alignment for speakers (all ps > 0.1).4

Figure 5.

Figure 5.

Open in a new tab

EEG recordings and voltage maps for ASL signers. Voltage maps are shown from 300-500ms and 500-700ms after sign onset. The scale used for voltage maps is +/− 2 microvolts.

P3 window (500-700ms).

The greater negativity for signs preceded by aligned pictures found in the N400 epoch continued through the P3 epoch, F(1,24) = 8.99, p = 0.006. There was additionally a significant three way interaction, Alignment x Laterality x Posteriority, F(8,192)= 2.98, p = 0.036), such that the P3 effect was strongest in frontal and central sites. For the English speakers, there were no significant effects of alignment in this window (all ps > 0.11)

Discussion

The primary goal of this study was to investigate the possible role of iconic structural alignment in the recognition and comprehension of ASL signs. To do so, we varied the visual overlap between a picture and an iconic sign in a picture-sign matching task, following Thompson et al. (2009) and Vinson et al. (2015). We replicated these previous studies: ASL signers, but not English speakers, had faster response times and higher accuracy when the picture and iconic sign were structurally aligned (Figure 1A) than when they were non-aligned (Figure 1B). We utilized ERPs to determine whether this behavioral facilitation effect occurred during lexical access, which would predict a reduced N400 amplitude for aligned compared to non-aligned trials, or whether behavioral facilitation was linked to decision processing, which would predict a reduced P3 amplitude for aligned compared to non-aligned trials. Contrary to the lexical access prediction, we found no evidence of a reduced N400 for aligned signs. Instead, we observed a smaller positivity for the aligned than non-aligned trials, and this difference spanned both the N400 and P3 time windows suggesting that the structural alignment between the picture and sign facilitated the matching decision, rather than lexical access. No ERP or behavioral effects of alignment were observed for the English speakers.

In contrast, we found very similar behavioral and ERP effects for ASL signers and English speakers for the matching compared to non-matching trials. Both groups had faster response times for matching trials (“yes” responses) than mismatched trials (“no” responses). In addition, both groups exhibited the expected reduced N400 amplitude for matching trials, compared to mismatched trials, indicative of a priming effect. That is, picture primes that matched the target word/sign in meaning facilitated lexical access in comparison to unrelated picture primes. The N400 priming effect extended into the later 500-700ms (P3) window. This finding could arise due to the robustness of the N400 effect in this condition and to the fact that the signs and spoken words were both dynamic and unfolded over time.

The behavioral results from the structural alignment manipulation in the present study are consistent with our previous study of sign production (McGarry et al., 2020). In that picture-naming study, ASL signers were faster to produce iconic signs elicited by aligned than non-aligned pictures. However, the electrophysiological results were not parallel with the present sign comprehension study. McGarry et al. (2020) found a smaller N400 amplitude for aligned compared to non-aligned pictures, which was interpreted as evidence that visual overlap between a picture and the phonological form of the iconic sign facilitated lexical access. In contrast, in this experiment on sign comprehension, we found larger amplitudes in the typical N400 epoch for signs primed with aligned compared to non-aligned pictures. If interpreted as an N400 effect, this result would suggest that the visual overlap between the aligned picture and iconic sign had an inhibitory effect on sign recognition and comprehension. However, such an interpretation conflicts with the observed behavioral facilitation results.

The lack of clear N400 priming (reduced negativity) for the aligned vs. non-aligned conditions in the present study suggests that picture alignment is not significantly affecting lexico-semantic processing. However, the design of the picture-sign matching paradigm causes the aligned and non-aligned trials to exclusively occur in the matching condition, which resulted in a strong attenuation of the N400 compared to the mis-matched trials (see Figures 3 and 4). The increased N400 amplitude for the matching trials is substantial and highly significant in both groups of participants. Thus, it is possible that a smaller N400 effect for alignment for the signers might have been masked by the large attenuation of the N400 due to semantic relatedness that may effectively be at ceiling. The use of a different paradigm, such as a go/no-go response with critical items in the no-go condition, might reveal an N400 alignment effect, if one exists for comprehension.

A second prediction discussed in the Introduction is that structural alignment might also result in easier decision-making and a reduced P3 for aligned trials in the picture-sign matching task. This prediction stemmed from prior interpretations of the P3 effect, which suggest that the amplitude of this component indexes the accumulation of evidence for a decision, leading to smaller P3 amplitudes and reduced response latencies for easier decisions that require less evidence (e.g., Herding et al., 2019, Twomey et al. 2015). We predicted that if this is the case, we might find reduced P3 amplitude in the aligned condition compared to the non-aligned condition. The findings from the present study are consistent with this prediction. We found reduced response latencies and a reduced P3 amplitude in the aligned condition compared to the non-aligned condition only for signing participants. While the distribution of this effect in the earlier N400 window (300-500ms) is not identical to the classic P3 effect due to its anterior expansion, the centro-parital maximum seen in the later epoch (500-700ms) is consistent with the typical P3 distribution. Consequently, it appears that aligned structural mapping between the picture-prime and the phonological features of the target sign reduces the amount of decisional effort required to complete the matching decision, most likely because less information needs to be compiled prior to reaching a decision. As the spoken English targets are not able to structurally map onto the picture-primes, there were no differences in the P3 window between the aligned and non-aligned conditions for the non-signing controls.

Existing research has questioned whether iconicity affects on-line sign processing, or if observed effects represent a strategic use of iconicity in a task (Emmorey, 2014; Perniss et al., 2010; Thompson et al., 2012). Several studies have shown that iconicity does not impact sign comprehension in paradigms where iconicity is not task-relevant (Bosworth & Emmorey, 2010; Emmorey et al., 2020; Mott et al., 2020; but see Thompson et al., 2010). The results of the present study are consistent with these studies and suggest that iconicity may be used strategically to facilitate decision making in comprehension tasks. To evaluate this hypothesis more thoroughly, a similar picture-sign matching paradigm could be designed, such that both the matching condition and the non-matching condition had iconic and non-iconic sub-conditions. However, as the present study was modeled after the behavioral investigation by Thompson et al. 2009, the present study was not designed with the ability to directly compare iconic and non-iconic signs. If overall iconicity (regardless of structural alignment between picture-prime and sign-target) facilitates sign comprehension, we would expect to find similar reduced P3 amplitude and response latencies for iconic signs than for non-iconic signs.

However, whether iconicity plays a different role in lexical access for comprehension than production needs to be explored further. It is possible that iconicity is only relevant in picture-naming tasks due to the mapping between visual features of the to-be-named picture and the phonological form of the target iconic sign that needs to be produced. If this is the case, we would not expect to find behavioral or ERP evidence for facilitated lexical access when iconic sign production is elicited without pictures, such as with a word translation task or a verb generation task. Ultimately, further explorations of iconicity that employ ERPs in addition to behavioral paradigms should be able to clarify whether potential effects of iconicity occur during lexical access or arise from later strategic processes, and so help determine the source of these effects.

  • Participants indicated whether a picture-prime and a sign/word-target match

  • Signers and non-signers had faster RTs and a reduced N400 for matching trials

  • Signers reacted faster when the picture and iconic sign were structurally aligned

  • Alignment reduced P3 amplitude for signers, indicating easier decision-making

  • No effect of alignment for non-signing participants

Acknowledgements

This work was supported by a grant from the National Institute on Deafness and other Communication Disorders (R01 DC010997). We would like to thank Lucinda O’Grady Farnady and Dan Fischer for their help with the study and all of our participants without whom this research would not be possible.

Footnotes

Publisher's Disclaimer: This is a PDF file of an article that has undergone enhancements after acceptance, such as the addition of a cover page and metadata, and formatting for readability, but it is not yet the definitive version of record. This version will undergo additional copyediting, typesetting and review before it is published in its final form, but we are providing this version to give early visibility of the article. Please note that, during the production process, errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

1

The P3 component is known under several different names including the Late Positivity Complex (LPC) and centro-parietal positivity (CPP – Twomey et al., 2016, and these are umbrella terms for the wide variety of ERP components that seem to be associated with P3 and P3-like effects. The names are therefore frequently used interchangeably. For the sake of clarity, we have elected to refer to this decisional positivity as the P3 to be consistent with previous work in our lab.

2

When separately analyzed, native and early signers demonstrated similar patterns of both behavioral and electrophysiological results, indicating that the observed results are not driven by one group of participants.

3

We regularly record EEG with a left mastoid reference but also record activity from the right mastoid. After inspecting the ERPs at the right mastoid for any differential effects of our experimental variables, the data are re-referenced to the algebraic average of the mastoids if any such effects are found. However, we did not see any differential effects and so the data were not re-referenced.

4

Illustrations of the (nonsignificant) ERP results for the hearing participants are provided as supplementary material on the Open Science Framework (https://doi.org/10.17605/OSF.IO/Y8X9Q).

References

  1. Aronoff M, Meir I, Padden C, & Sandler W (2005). Morphological universals and the sign language type. In Booij G & van Marle J (Eds.), Yearbook of Morphology 2004 (pp. 19–39). Springer; Netherlands. 10.1007/1-4020-2900-4_2 [DOI] [Google Scholar]
  2. Bellugi U, & Klima ES (1976). Two faces of sign: Iconic and abstract. Annals of the New York Academy of Sciences, 280, 514–538. [DOI] [PubMed] [Google Scholar]
  3. Bentin S, Kutas M, & Hillyard SA (1993). Electrophysiological evidence for task effects on semantic priming in auditory word processing. Psychophysiology, 30(2), 161–169. 10.1111/j.1469-8986.1993.tb01729.x [DOI] [PubMed] [Google Scholar]
  4. Bentin S, Mouchetant-Rostaing Y, Giard MH, Echallier JF, & Pernier J (1999). ERP Manifestations of Processing Printed Words at Different Psycholinguistic Levels: Time Course and Scalp Distribution. Journal of Cognitive Neuroscience, 11(3), 235–260. 10.1162/089892999563373 [DOI] [PubMed] [Google Scholar]
  5. Bosworth RG, & Emmorey K (2010). Effects of Iconicity and Semantic Relatedness on Lexical Access in American Sign Language. Journal of Experimental Psychology. Learning, Memory, and Cognition, 36(6), 1573–1581. 10.1037/a0020934 [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Brown C, & Hagoort P (1993). The Processing Nature of the N400: Evidence from Masked Priming. Journal of Cognitive Neuroscience, 5(1), 34–44. 10.1162/jocn.1993.5.1.34 [DOI] [PubMed] [Google Scholar]
  7. Caselli NK, & Pyers JE (2017). The Road to Language Learning Is Not Entirely Iconic: Iconicity, Neighborhood Density, and Frequency Facilitate Acquisition of Sign Language. Psychological Science, 28(7), 979–987. 10.1177/0956797617700498 [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Caselli NK, Sehyr ZS, Cohen-Goldberg AM, & Emmorey K (2017). ASL-LEX: A lexical database of American Sign Language. Behavior Research Methods, 49(2), 784–801. 10.3758/s13428-016-0742-0 [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Dingemanse M, Blasi DE, Lupyan G, Christiansen MH, & Monaghan P (2015). Arbitrariness, Iconicity, and Systematicity in Language. Trends in Cognitive Sciences, 19(10), 603–615. 10.1016/j.tics.2015.07.013 [DOI] [PubMed] [Google Scholar]
  10. Emmorey K (2014). Iconicity as structure mapping. Phil. Trans. R. Soc. B, 369(1651), 20130301. 10.1098/rstb.2013.0301 [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Emmorey K, Winsler K, Midgley KJ, Grainger J, & Holcomb PJ (2020). Neurophysiological Correlates of Frequency, Concreteness, and Iconicity in American Sign Language. Neurobiology of Language, 1(2), 249–267. 10.1162/nol_a_00012 [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Gentner D (1983). Structure-mapping: A theoretical framework for analogy. Cognitive Science, 7(2), 155–170. 10.1016/S0364-0213(83)80009-3 [DOI] [Google Scholar]
  13. Gentner D, & Markman AB (1997). Structure mapping in analogy and similarity. American Psychologist, 52(1), 45–56. 10.1037/0003-066X.52.L45 [DOI] [Google Scholar]
  14. Hajcak G, & Foti D (2020). Significance?… Significance! Empirical, methodological, and theoretical connections between the late positive potential and P300 as neural responses to stimulus significance: An integrative review. Psychophysiology, 57(7), e13570. 10.1111/psyp.13570 [DOI] [PubMed] [Google Scholar]
  15. Herding J, Ludwig S, von Lautz A, Spitzer B, & Blankenburg F (2019). Centro-parietal EEG potentials index subjective evidence and confidence during perceptual decision making. NeuroImage, 201, 116011. 10.1016/j.neuroimage.2019.116011 [DOI] [PubMed] [Google Scholar]
  16. Imai M, Miyazaki M, Yeung HH, Hidaka S, Kantartzis K, Okada H, & Kita S (2015). Sound Symbolism Facilitates Word Learning in 14-Month-Olds. PLOS ONE, 10(2), e0116494. 10.1371/journal.pone.0116494 [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Kutas M, & Federmeier KD (2011). Thirty years and counting: Finding meaning in the N400 component of the Event-Related Brain Potential (ERP). Annual Review of Psychology, 62(1), 621–647. 10.1146/annurev.psych.093008.131123 [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Luck SJ (2014). An Introduction to the Event-Related Potential Technique. MIT Press. [Google Scholar]
  19. McGarry ME, Mott M, Midgley KJ, Holcomb PJ, & Emmorey K (2020). Picture-naming in American Sign Language: An electrophysiological study of the effects of iconicity and structured alignment. Language, Cognition and Neuroscience. https://www.tandfonline.com/doi/abs/10.1080/23273798.2020.1804601 [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Mott M, Midgley KJ, Holcomb PJ, & Emmorey K (undefined/ed). Cross-modal translation priming and iconicity effects in deaf signers and hearing learners of American Sign Language. Bilingualism: Language and Cognition, 1–13. 10.1017/S1366728919000889 [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. Östling R, Börstell C, & Courtaux S (2018). Visual Iconicity Across Sign Languages: Large-Scale Automated Video Analysis of Iconic Articulators and Locations. Frontiers in Psychology, 9. 10.3389/fpsyg.2018.00725 [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Padden CA, Meir I, Hwang S-O, Lepic R, Seegers S, & Sampson T (2013). Patterned iconicity in sign language lexicons. Gesture, 13(3), 287–308. 10.1075/gest.13.3.03pad [DOI] [Google Scholar]
  23. Perniss P, Thompson RL, & Vigliocco G (2010). Iconicity as a General Property of Language: Evidence from Spoken and Signed Languages. Frontiers in Psychology, 1. 10.3389/fpsyg.2010.00227 [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Perry LK, Perlman M, & Lupyan G (2015). Iconicity in English and Spanish and Its Relation to Lexical Category and Age of Acquisition. PLOS ONE, 10(9), e0137147. 10.1371/journal.pone.0137147 [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Taub SF (2001). Language from the Body: Iconicity and Metaphor in American Sign Language. Cambridge University Press. [Google Scholar]
  26. Thompson RL, Vinson DP, & Vigliocco G (2009). The Link Between Form and Meaning in American Sign Language: Lexical Processing Effects. Journal of Experimental Psychology. Learning, Memory, and Cognition, 35(2), 550–557. 10.1037/a0014547 [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. Thompson RL, Vinson DP, & Vigliocco G (2010). The link between form and meaning in British sign language: Effects of iconicity for phonological decisions. Journal of Experimental Psychology. Learning, Memory, and Cognition, 36(4), 1017–1027. 10.1037/a0019339 [DOI] [PubMed] [Google Scholar]
  28. Thompson RL, Vinson DP, Woll B, & Vigliocco G (2012). The road to language learning is iconic: Evidence from British Sign Language. Psychological Science, 23(12), 1443–1448. 10.1177/0956797612459763 [DOI] [PubMed] [Google Scholar]
  29. Twomey DM, Kelly SP, & O’Connell RG (2016). Abstract and Effector-Selective Decision Signals Exhibit Qualitatively Distinct Dynamics before Delayed Perceptual Reports. Journal of Neuroscience, 36(28), 7346–7352. 10.1523/JNEUROSCI.4162-15.2016 [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Twomey DM, Murphy PR, Kelly SP, & O’Connell RG (2015). The classic P300 encodes a build-to-threshold decision variable. European Journal of Neuroscience, 42(1), 1636–1643. 10.1111/ejn.12936 [DOI] [PubMed] [Google Scholar]
  31. Verleger R (2020). Effects of relevance and response frequency on P3b amplitudes: Review of findings and comparison of hypotheses about the process reflected by P3b. Psychophysiology, 57(7), e13542. 10.1111/psyp.13542 [DOI] [PubMed] [Google Scholar]
  32. Vinson D, Thompson RL, Skinner R, & Vigliocco G (2015). A faster path between meaning and form? Iconicity facilitates sign recognition and production in British Sign Language. Journal of Memory and Language, 82(Supplement C), 56–85. 10.1016/j.jml.2015.03.002 [DOI] [Google Scholar]

RESOURCES