Exploring perceptual processing of ASL and human actions: Effects of inversion and repetition priming

Abstract

In this paper, we compare responses of deaf signers and hearing non-signers engaged in a categorization task of signs and non-linguistic human actions. We examine the time it takes to make such categorizations under conditions of 180-degree stimulus inversion and as a function of repetition priming, in an effort to understand whether the processing of sign language forms draws upon special processing mechanisms or makes use of mechanisms used in recognition of non-linguistic human actions. Our data show that deaf signers were much faster in the categorization of both linguistic and non-linguistic actions, and relative to hearing non-signers, show evidence that they were more sensitive to the configural properties of signs. Our study suggests that sign expertise may lead to modifications of a general-purpose human action recognition system rather than evoking a qualitatively different mode of processing, and supports the contention that signed languages make use of perceptual systems through which humans understand or parse human actions and gestures more generally.

1.0 Introduction

While there has been a growing and increasingly sophisticated linguistic analysis of signed languages from around the world, we still have little knowledge of the fundamental properties which underlie the recognition of signs (see Corina & Knapp, 2006; Emmorey, 2002; for discussions). A particularly glaring gap is our lack of understanding of how the initial stages of sign language processing differ from, or are similar to, the processing of other kinds of human actions, something that the present study seeks to address. Using a continuous categorization paradigm, we assess deaf and hearing subjects’ performance for speeded categorization judgments of signs and actions. The effects of two stimulus manipulations, 180-degree inversion and stimulus repetition, are introduced. Stimulus inversion provides an opportunity to assess the relative contributions of configural and featural properties across stimulus class, while repetition priming provides insight into the representations mediating the recognition of these stimuli. Using these manipulations, we test deaf users of sign language and sign-naive hearing subjects to assess the role of linguistic and sensory experience on the perception of human actions.

We begin with an overview of the relationship between sign language and gestures and discuss the evidence for domain-general and language-specific mechanisms in the perception of signed languages.

1.1 Sign language and gesture

The formal relationship between signed languages and human gestural actions is of considerable interest to a range of disciplines. Linguists, psychologists and cognitive scientists have proposed a critical role for manual gesture in the development and evolution of human languages (Arbib, 2005, 2008; Armstrong, Stokoe & Wilcox 1995 Armstrong & Wilcox, 2007; Corballis, 1999, 2003; Kendon, 1997, 2000, 2004; Tomasello, 2005; Wilcox, 2004). Recently, linguists have documented compelling evidence that the development of full-fledged sign languages derives from idiosyncratic gestural and pantomimic systems used by isolated communities, which in some cases may be limited to individual families who have a need to communicate with a deaf child (Kegl, Senghas & Coppola, 1999; Morford & Kegl, 2000; Senghas, 2005; Sandler, Meir, Padden & Aronoff, 2005; Frishberg, 1987; Goldin-Meadow, 2003). Even within full-fledged sign languages of Deaf communities, linguistic accounts of sign language structure have argued that lexical and discourse components of ASL and other signed languages may be best understood as being gesturally based (Liddell, 2003). Thus, diachronic and synchronic evidence from language research supports the contention that signed languages might make use of similar perceptual systems through which humans understand or parse human actions and gestures more generally.

Alternatively, there is reason to suspect that sign language perception may require the attunement of specialized systems for recognizing sign forms. Developmental studies have shown that deaf infants make errors in the use of pronouns despite the fact that the pronominal signs are isomorphic in form to indexical referential pointing gestures commonly used in most cultures, especially for first and second person pronouns (Petitto, 1987; Hoffmeister (1987); Pizzuto (1990). These data suggest that during language development, otherwise iconic actions are reinterpreted anew within a formal linguistic framework. In adults, studies of sign language aphasia have documented, in production and comprehension, dissociations between the use of a formal sign language and pantomimic gestures¹ (Corina et al., 1992; Marshall, Atkinson, Smulovitch, Thacker & Woll, 2004; Poizner et al., 1987). In further support of the attunement hypothesis, there is growing evidence that age of exposure to sign language may affect the fidelity of language processing and suggests vulnerabilities in the initial mappings required for sign language recognition (Mayberry & Fischer, 1989; Mayberry & Witcher, 2006; Morford et al., 2008). Electrophysiological and neuroimaging studies have documented the effects of language experience on brain organization for sign language (Capek et al., 2009; McCollough, Emmorey & Sereno, 2005, Newman et al., 2002). The notion that the perception of human actions may be susceptible to modifications due to linguistic experience is credible, given the fact that human actions themselves may constitute a special class of visual forms.

1.2 Introduction to the effects of inversion on human form and biological action perception

There is substantial evidence that the recognition of human forms, including faces and body postures, entails distinctive recognition processes. Studies have shown that configural processing of these forms factors significantly in their recognition and discrimination. The term “configural processing” is used to refer to perceptual phenomena that involve perceiving the relations among the features of a stimulus, and is often contrasted with “featural processing,” or “componential processing”² (Maurer, Le Grand & Mondloch, 2002).

The evidence for a reliance on configural over featural processing is based, in part, on stimulus inversion effects, whereby the 180-degree inversion of particular classes of stimuli adversely affects the processing of those stimuli. Inversion appears to disrupt the appreciation of the canonical relationships between features of the stimuli. The “inverted face effect” is a paradigmatic case and refers to the fact that inverted faces are recognized far less accurately than other inverted objects (Diamond & Carey, 1986; Bartlett & Searcy, 1993; Murray, Yong & Rhodes, 2000; Bertin & Bhatt, 2004); it widely accepted that inversion of these visual forms disrupts configural processing (Yin, 1969; Diamond & Carey, 1986; Bartlett & Searcy, 1993; Murray, Yong & Rhodes, 2000; Bertin & Bhatt, 2004; Farah, Tanaka & Drain, 1995).

In addition to stimulus characteristics, perceiver characteristics such as expertise are known to modulate inversion effects. Gauthier and colleagues, for example, have shown that inversion effects are obtained when the observer has the necessary expertise to differentiate objects on the basis of their configural properties (Gauthier & Tarr, 1997; Gauthier, Williams, Tarr & Tanaka, 1998). That a learner’s experience interacts with inversion effects for novel stimuli has raised the question of whether the perceptual mechanisms at work in the identification of biological forms are unique to their domains or reflect modifiable general-purpose perceptual strategies (Valentine, 1988; Gauthier et al., 2000; Yovel & Kanwisher, 2005; Haxby et al., 2000).

Reed et al. (2003) noted that bodies, like faces, share many similar stimulus properties. Human bodies are highly symmetric, different exemplars share the same set of parts (e.g., arms, legs, trunk, head), and their recognition requires perceivers to make fine distinctions based on the shapes and sizes of these parts. Exemplars of body positions are distinguished by subtle differences in the spatial relations among parts and are recognized at the subordinate level. Using a forced-choice, same/different inversion paradigm, Reed et al. (2003) observed that images of bodies, like faces and in contrast to houses, were processed faster and more accurately when upright than when inverted.

Perception of point-light displays of biological motion also exhibit great sensitivity to upright as opposed to inverted orientations (Bertenthal, 1993; Pavlova & Sokolov, 2000). Recognition of a human form is reduced, although not eliminated, when a point-light walker is shown upside down (Pavlova & Sokolov, 2000; Sumi, 1984). Detection of point-light walkers in noise is reduced by turning the walker upside down (Bertenthal & Pinto, 1994; Pavlova & Sokolov, 2000). Some researchers have suggested that the disruptive effects of inversion on the perception of biological motion stems from a disruption of the perception of the human form. In an event-from-form model (Pavlova & Sokolov, 2000), biological motion perception is, in part, dependent upon an abstract human form template. Under conditions of inversion, the spatial fit between image and template (or prototype) that is needed for motion recognition is poor. Others have argued that difficulty in recognizing these actions under inversion is due to the fact that the dynamic relations specified by the kinematics become unfamiliar (Shipley, 2003). Evidence in support of each of these positions has been offered, indicating that point-light human motion perception relies upon multiple processes for identification.

Recent work has examined the effects of inversion on videotapes of fully expressed human actions. Loucks and Baldwin (2009) created sets of video triads that included a standard-action video, a video altered solely in featural information, and a video altered solely in configural information. Featural changes were modifications in the local detail of the action performed, with no alteration of the global spatial trajectories of body parts. For instance, for a standard-action video showing a person reaching and grasping a cup with a whole-hand grasp and moving it in a direct trajectory across a table, the featurally-changed version showed the person holding the cup with a pincher-grasp instead. Configural changes consisted of global modifications in the spatial trajectory body parts followed through space, without altering the small action detail. For example, for the standard-action video just described, the configurally-changed version incorporated a change in the large-scale trajectory of the movement, such that the hand and arm lifted the cup higher off the table in its movement to its new location than in the standard version. Subjects viewed upright or inverted pairs of stimuli that were either identical or a pairing of a standard video with its featurally- or configurally-changed counterpart. The researchers observed that sensitivity to featural changes was unaffected by inversion, while sensitivity to configural trajectory changes was significantly impaired by inversion. That is, subjects were significantly less sensitive in detecting configural-level differences in the inverted stimuli, relative to feature-level differences.

In a subsequent experiment, when these stimuli were subjected to a low-pass spatial frequency filter (thus obscuring high spatial frequency information), detection of featural changes were disrupted to a greater extent than configural changes. Based upon this pattern of results, the authors concluded that while both feature-level and configural-level information is brought to bear on the interpretation of dynamic human actions, inversion adversely affects the configural processing of such actions.

While there remain important issues regarding the specification of the relational properties that are subsumed under the term “configural processing” and questions regarding the specificity of these effects, the general consensus from studies of faces, bodies, point-light biological motion displays and human action videos indicates that inversion disproportionally affects the processing of configural information while having relatively less impact on the processing of featural information. We draw upon these findings, and use stimulus inversion, to explore the contributions of configural and featural properties in the processing of linguistic sign forms and non-linguistic gestures, and examine whether perceptual experience with linguistic actions alters the relative weightings of these perceived stimulus properties. We then present data from a repetition priming manipulation in a further effort to understand the roles of semantic and episodic representations in subjects’ categorization performance.

2.0 Methodology

2.1 Continuous categorization paradigm

Using a continuous categorization paradigm, we examined the time it takes for subjects to categorize signs and non-linguistic actions, specifically self-grooming (SG) actions. In this paradigm, subjects see an ASL sign or SG action and must decide as quickly as possible if it is a sign or a non-linguistic action. To make a proper classification, a subject must minimally identify the distinguishing properties of these two classes of stimuli across many exemplars. Given the relatively large number of stimuli used and the inherent similarities in stimulus characteristics, subjects must recognize the actions as belonging to a particular class in order to make this action classification. Evidence in support of this claim is provided below.

In our paradigm, we measure subjects’ response time to each stimulus item, and timing constraints are used to minimize opportunities for strategic responding. In previous work, we have found that this paradigm provides an efficient and engaging testing experience. Two stimulus manipulations, 180-degree inversion and stimulus repetition, are incorporated into the design. The inclusion of signs and SG actions in both upright and inverted orientations permits an assessment of perceived stimulus characteristics, in particular the relative weighting of featural and configural properties of these stimuli. We present our predictions and results of stimulus categorization and inversion in Section 3.0. In our design, the juxtaposition of stimulus items is also used to assess the effects of repetition priming and provides a means to understand the contributions of episodic and semantic representations supporting action categorization (see Figure 1, which is described in more detail below). Our introduction, results and discussion related to repetition priming are presented in Section 4.0.

Figure 1.

A schematic sequence of the video stimuli (some items have been suppressed for clarity). Starting at the left: (A) inverted gesture target, (B & C) related sign prime–target pair, (D & E) inverted related gesture prime-target pair, (F) filler sign, (F & G) unrelated sign prime-target pair, (G & H) related sign prime-target pair. The word above each image is the correct categorization of the stimulus.

2.2 Subjects

In this task, a total of 76 participants took part, of whom 43 were deaf (25 female) and 33 were hearing non-signers (19 female). There were thirteen left-handed subjects, eight of whom were deaf. Twenty-two deaf subjects were considered “early-exposed,” having reported exposure to ASL before age 8. All deaf subjects reported using ASL as their preferred means of communication and varied in their age of exposure (13 native, 9 early (prior to age 8), 21 late (between 9–16 years)). The majority of the hearing subjects were undergraduate students at the University of California at Davis, and 25 of the deaf subjects were students at Gallaudet University in Washington, D.C. The other subjects were residents of northern California who were recruited through advertisements and word-of-mouth. Subjects were given either course credit or a small fee for participating. All gave informed consent in accordance with established Institutional Review Board procedures at Gallaudet University and U.C. Davis.

2.3 Stimuli

Stimulus items were presented as short video clips (mean=1029 ms; range=[701 ms, 1635 ms]), each showing one of two sign models (native ASL signers, one male and the other female) performing an action which was either an ASL sign or a non-linguistic self-grooming gestural action such as head scratching. Each action was articulated and filmed in isolation, not as part of a longer utterance. The sign clips were somewhat shorter than the gestures on average (mean duration = 984 ms for signs, 1094 ms gestures), but across the stimulus set this difference was not statistically significant. All stimuli were closely edited to remove non-critical transitional movements. Liddell & Johnson (1989) discuss the fact that phonetically, signs often exhibit a brief stasis where a sign’s acceleration is effectively zero; on video with 30 frame-per-second encoding, this appears as a moment in which the intended target handshape of the sign is clearly articulated and non-blurred. In our laboratory's editing scheme, our sign stimuli started two frames prior to this initial hold segment of the sign. The ending frame of each sign stimulus was chosen as the point at which the target handshape began to fall out of its intended form. We applied this same approach to the editing of the non-linguistic SG stimuli, closely cropping them so that they clearly displayed the SG action without including a large amount of transitional movement.

A number of constraints were used to select the stimuli, which needed to be easily identifiable, have no major occlusions, and be balanced overall between the two performers. Sign stimuli had to be mono-morphemic and relatively frequent.³ During filming, each sign model was prompted by an experimenter who stood behind the front-facing camera, so all signing actions were articulated “to the camera.” The non-linguistic SG gestures included some which were planned in advance of filming and others which occurred spontaneously during the filming session between takes while the cameras were still running. As we sought to capture naturalistic gestures, models were not asked to perform the SG actions for the camera per se, as this would tend to make the gestures appear unnatural. It is important to bear in mind that in this study we are not attempting to test the minimal differences by which signs may be differentiated from non-linguistic gestures, but wish to explore the nature of the processing modes that are brought to bear upon the identification of linguistic and non-linguistic gestural signals in situ.

The entire sequence of stimuli consisted of approximately 450 video clips, lasting a total of approximately 13 minutes. A given action could appear from one of four orientations or Viewpoints: straight-ahead front upright view (“F”), inverted (i.e., Upside-down, “U”), or from the side (Left or Right).⁴ The Upside-down clips were created by inverting the corresponding Front-view upright clips using Final Cut Pro software; for example, the stimulus item DOLL-Front-view was inverted to create the item DOLL-Upside-down-view.

A total of 28 actions (14 signs and 14 SG-gestures, listed in Appendix 1, henceforth referred to as critical actions) were chosen to appear in unprimed and primed contexts. ⁵ The critical action signs included nine one-handed signs and five two-handed signs (four involving a stationary base hand, one in which both hands moved). This set of signs incorporated nine distinct dominant handshapes, four major body places of articulation (six on the face, three on the head, one on the trunk, and four in “neutral space,” the area in front of the signer) and seven movement types. The critical action SG gestures included nine one-handed forms and five two-handed forms (four involved both hands moving; the other was performed on a stationary arm). The SG actions incorporated six distinct handshapes (though these would be termed “laxed” variants compared to the ASL handshapes); five major body places of articulation (six on the face, three on the trunk, three on the head, one on the neck, and one in neutral space) and five distinct movement types (see Appendix 2 for details). While there is some homogeneity in the signs and SG actions selected for our experiment, the stimuli do vary upon a number of dimensions that subjects could use to differentiate these forms. However, each subject group observed the same set of physical stimuli; thus all subjects had an equal opportunity to make use of the naturalistic cues available to differentiate them.

Appendix 1. List of stimuli.

As is customary in the sign language literature, glosses of ASL signs are given in capital letters.

Gestures	Signs
Adjust Shirt	BACHELOR
Brush Hair	BEER
Crack Knuckles	BITTER
Pull Ear	DOLL
Rub Eye	GLASSES
Rub Nose (Signer 1)	HOME
Rub Nose (Signer 2)	ISLAND
Scratch Face	LIKE
Scratch Head (Signer 1)	NATURALLY
Scratch Head (Signer 2)	ORANGE
Scratch Neck	PARENT
Scratch Nose	TREE
Stretch Arms	WEEK
Tug Shirt	WONDER

Appendix 2. Description of Stimuli.

Signs	Hands	Handshape	Specific Location	Body Part	Movement	Eyes
BACHELOR	1	B	chin	FACE	2 downward taps	camera
BEER	1	B	cheek	FACE	2 downward brushes	camera
BITTER	1	1	chin	FACE	twists	camera
DOLL	1	X	nose	FACE	2 downward brushes	camera
GLASSES	1	oG-bO	eye	FACE	rightward, with handshape change	camera
HOME	1	fO	cheek	HEAD	tap near mouth and cheekbone	camera
ISLAND	2	1/B	neutral space	NEUTRAL	circular	camera
LIKE	1	o8 to 8	chest	TRUNK	outward, with handshape change	camera
NATURALLY	2	N & B	neutral space	NEUTRAL	circular to tap	camera
ORANGE	1	O	side of mouth	FACE	handshape clenches 2X	camera
PARENT	1	5	mouth and forehead	HEAD	one tap at each location	camera
TREE	2	5/B	elbow on back of hand	NEUTRAL	twists	camera
WEEK	2	1/B	palm of non-dominant hand	NEUTRAL	straight movement	camera
WONDER	2	1	forehead	HEAD	circular	camera

Gestures	Hands	Handshape	Specific Location	Body Part	Movement	Eye gaze
Adjust Shirt	2	bO	shoulders	TRUNK	two tugs	camera/shirt
Brush Hair	1	lax 5	side of face	HEAD	brush downward	camera/shirt
Crack Knuckles	2	5's	neutral space	NEUTRAL	arms straight, fingers intertwined	to hands
Pull Ear	1	bO	ear	FACE	rub ear 2X	stage left
Rub Eye	1	A	eye	FACE	rub to side	eye close
Rub Nose (Signer 1)	1	A	nose	FACE	rub upward 2X	camera/shirt
Rub Nose (Signer 2)	1	1 (lax 5)	nose	FACE	rub leftwards 1X	camera
Scratch Face	1	A-dot	cheek/nose	FACE	downward scratch 2X	forward
Scratch Head (Signer 1)	1	5-claw	side of head	HEAD	scratches (multiple)	forward
Scratch Head (Signer 2)	1	5-claw	side of head	HEAD	scratches 2X	camera
Scratch Neck	2	5-claw	neck	NECK	scratches 3X	forward-downward
Scratch Nose	1	1	side of nose	FACE	finger bends 1X	averted down
Stretch Arms	2	C	elbow	TRUNK	left hand pulls right arm leftward	stage left
Tug Shirt	2	C	sleeve	TRUNK/ARM	right hand pulls up left sleeve	forward

An additional 40 actions (20 signs and 20 gestures) served as “filler” items, and were interspersed among the critical action trials. In “unprimed contexts,” critical action targets were preceded by filler items of the opposite type (e.g., the target sign DOLL being preceded by a filler item depicting a SG action). In Section 3.0, we report results of trials in which unprimed targets appeared in either the upright or inverted orientation.

The assessments of repetition priming reported in Section 4.0 contrasted the effects of “related” and “unrelated” prime-target stimulus pairs showing actions in the upright view. In the unrelated condition, the primes differed from the critical action targets but were drawn from the same action class (e.g., the target sign ORANGE being preceded by a prime showing a different sign like HOUSE). Unrelated primes were drawn from the pool of filler items. In the related condition, the prime-target pairs were identical (e.g., the sign target WEEK being preceded by a prime stimulus showing the same sign WEEK). Finally, in Section 5.0 we consider an additional repetition priming condition in which the prime is presented upside down and the identical target presented right-side up (e.g., the sign prime BITTER-Upside-down-view followed by the target sign BITTER-Front-view).

Critical action pairs were always separated by a sequence of one to three filler clips. Filler clips could show either SG gestures or ASL signs and could show those actions from any viewpoint. For each sequence of fillers, the number of fillers (one, two or three), the actions they showed, and each of their viewpoints were determined randomly. Over the course of the task, each critical action (whether a sign or gesture) was presented in a prime-target pair a total of four times (i.e., eight occurrences total: four times as a prime and four times as a target). For each subject, randomization was performed on the entire set of prime-target pairs, so that different subjects saw the same critical actions, but in a different order. The viewpoints used for particular critical action target-prime pairs also varied from subject to subject.

2.4 Task

The subject was told that he/she would be watching a series of short video clips and that for each clip, the task was to decide as quickly as possible whether the action shown was an ASL sign or a non-linguistic gesture. Responses were registered by pushing one of two buttons. The task was balanced for response hand (i.e., whether the left- or right-handed button was used to respond to signs) and for handedness of participant. A head-chin mount (UHCO HeadSpot, Houston, TX) was used to maintain a consistent 24-inch distance from the subject’s eyes to the computer screen. The task was administered and data collected using Presentation software (Neurobehavioral Systems, Albany, CA).

Before the experiment began, each subject was given a short training exercise, identical in format to the actual experiment, during which the subject viewed and categorized 30 video clips. The signs and gestures in the training clips were different from those shown during the actual experiment and feedback for correctness was given during the training task. Subjects were told after the warm-up task that the actual experiment would not provide feedback for correctness, only for appropriately-timed responses (not too fast or slow, discussed below). Hearing subjects, who had no formal knowledge of ASL, were instructed to simply make their best judgment and respond as quickly as possible even if they were uncertain.

Figure 1 shows a schematic portion of a stimulus sequence that illustrates our conditions of interest, along with the expected answers to each item. As indicated in the figure, stimuli were always signs or SG gestures, and could be Front upright, “F,” or upside-down, “U.” The SG actions tended not to resemble signs when seen in their entirety (as opposed to the still shots in Figure 1), and as will be seen in the accuracy data, hearing subjects were generally quite successful in making these categorizations. Examples of the video stimuli can seen in the supplementary materials.

Subjects were encouraged to respond as quickly as possible, and did not have to wait to the end of the video to register a response. All reaction times are measured from the onset of each video. As noted, feedback for correctness was not given to subjects during the experiment. However, in order to encourage them to respond quickly, two rapidly-shortening horizontal green lines were displayed during each trial, one above and one below the video clip being presented, indicating the amount of time remaining for the subject to respond.

In addition, a brief (approx. 300 ms) message screen reading “TOO SLOW” was shown after a trial if the response for that trial came more than 400 ms after the end of that video clip. Similarly, to prevent subjects from speeding through the experiment without sufficient attention to the task, a similar message screen reading “TOO FAST” was given if response time for a given trial was under 300 ms, measured from the onset of the video. Responses outside the RT windows just described were deemed “invalid”; if such a response was given during the presentation of a prime-target pair, or if no response at all was given, that pair was repeated later. Overall, the incidence of such invalid trials was about 1%. After each valid trial, a brief (approx. 300 ms) screen with a small green square was presented, indicating to the subject that they had responded within the appropriate time window for that trial and that the next trial was forthcoming.⁶ The interval between trials (from end of feedback screen to start of next video) was 218 ms. Therefore, the total ISI (end of one video to start of next video)—whether between filler, prime or target items—was always about 500 ms.

2.5 Data analysis

Accuracy and RT for categorizing video clips served as dependent measures. We examined accuracy and reaction time with ANOVA by items (F1) and subjects (F2). For our main comparisons in Sections 3.0 and 4.0, we report significance results in terms of the minF’ statistic (Clark, 1973), a conservative measure that seeks to ensure that results are simultaneously generalizable to both new subjects and new items. We then follow up with planned and post-hoc comparisons when such results are germane to the discussion. The analyses for accuracy were performed after applying an arcsine transformation (y=arcsin(sqrt(x)) on the initially-obtained percentage data, but for clarity of presentation the summary information given for accuracy has been back-transformed into percentage scores (Wheater & Cook, 2000). Reaction time analyses were performed on correct responses only.

Response times were considered to be outliers if they were more than 3 standard deviations (SD) away from the mean among the results for a given subject (over all actions of that Action-type, i.e., sign or gesture) or a given action (over all subjects of the same group, i.e., deaf or hearing). Outliers were replaced with the average of two quantities—the subject’s average RT for that Action-type, and the average RT over all subjects of the same group (deaf or hearing) associated with that critical action. Overall, the proportion of RT values classified as outliers and so replaced was under one percent. Moreover, data for both members of a video clip pair were rejected if the subject did not respond to both the prime and target correctly, or did not make each response within the requisite time window.⁷ In rare cases, this conservative procedure resulted in a subject not having any valid data points for a specific condition of interest; in these cases a group mean for that condition was used to fill in the missing value.⁸

3.0 Classification and inversion of signs and self-grooming gestures

3.1 Effects of classification and inversion

We reasoned that if deaf signers use language-specific perceptual routines in the early identification of signs that differ substantially from those used in identifying the novel self-grooming actions, we might observe response differences in reaction time and accuracy during stimulus classification. We predicted that because of their rich experience with signing, the deaf subjects’ categorization of the sign language stimuli would be faster and more accurate than the identification of the novel SG gestures. In contrast, for hearing subjects, for whom all stimuli are novel, we predicted no differences in the time to categorize signs and SG actions. Planned comparisons are used to assess stimulus class differences.

Of additional interest are the effects of 180-degree inversion of these stimuli. We expect that inversion will significantly disrupt the identification and classification of both signs and gestures, but that the costs associated with inversion may differ for the two classes of stimulus items. If so, this will provide insight into the relative contributions of configural and featural information used in the service of the classification of these different classes of stimuli. Our cross-group comparison will allow us to determine whether the relative weighting of these dimensions is common to all perceivers or is modulated by linguistic experience. To the extent that expertise modulates inversion effects, one would predict that deaf subjects would show greater inversion costs for the sign language stimuli relative to the hearing subjects.

3.1.1 Results

To assess whether there are behavioral differences in the classification of ASL signs and SG-gestures, and to examine the effects of inversion on subjects’ performance of this task, we examine RTs for critical actions in an unprimed context. A summary of the data is given in Table 1. Means and standard errors (SE) are given for the RT data. SE’s for accuracy were uniformly small (probably due to the overall high accuracy scores), so for clarity of presentation they are not shown in this or later tables.

Table 1.

Overview of RT means (and standard errors) and accuracy means for the deaf and hearing groups for critical-action items immediately preceded by fillers of the opposite Action-type.

	RT (ms)				Accuracy (%)
Deaf	F	U	Δ	All views	F	U	All views
Signs	709 (13.1)	774 (17.2)	65	742 (13.9)	93.8	92.6	93.3
Gestures	732 (13.7)	818 (17.6)	86	775 (14.0)	88.3	92.6	90.6
Overall	721 (12.3)	796 (15.0)	75	758 (12.7)	91.3	92.6	91.9
Hearing
Signs	842 (14.9)	874 (19.6)	32	858 (15.9)	82.0	75.8	78.9
Gestures	830 (15.7)	927 (20.1)	97	879 (16.0)	87.4	91.9	89.8
Overall	836 (14.0)	901 (17.1)	65	868 (14.5)	84.8	84.7	84.8

A three-way ANOVA with factors of Group (deaf vs. hearing), Action-type (sign vs. gesture) and stimulus Viewpoint (Front upright vs. Upside-down) was used to establish significant patterns in these data. We observed main effects of Group (minF’(1,96.9)=26.4, p<0.001) and Viewpoint (minF’1(1,76.8)=39.8, p<0.001), and a two-way Action-type by Viewpoint interaction (minF’1(1,65.1)=4.14, p<0.05).

The main effect of Group is due to the fact that overall, deaf subjects were faster at categorizing the stimuli (mean RT = 758 ms for deaf and 868 ms for hearing). The main effect of Viewpoint reflects, not surprisingly, that categorization of Upside-down stimuli was slower than categorization of upright stimuli (mean RT for F = 778 ms vs. 848 ms for U). These main effects must be qualified in light of the two-way Action-type by Viewpoint interaction. This interaction is due to the fact that overall, for both groups, inversion slowed the categorization of SG-gestures to a greater extent than the categorization of sign stimuli (respective mean RTs for upright and inverted gestures = 781 ms vs. 873 ms, ΔRT = 91.4 ms; for signs the corresponding values are 775 ms vs. 824 ms, ΔRT = 48.7 ms).

The accuracy analysis found only a marginal main effect of Group (minF’1(1,83.5)=3.24, p=0.075), reflecting the fact that deaf subjects were generally more accurate than hearing subjects (91.9% vs. 84.8%).

3.1.2 Discussion: Classification and inversion of signs and self-grooming gestures

These data demonstrate that deaf and hearing subjects differ in their ability to categorize linguistic and non-linguistic action stimuli. In addition, for both groups, the inversion manipulation led to stimulus-class differences that disrupted categorization of SG gesture stimuli more than sign stimuli. We discuss the implications of each of these results in turn.

Deaf signers were significantly faster than hearing subjects at categorizing these actions. While it is not surprising that the deaf subjects were faster in categorizing the sign stimuli, this effect was not limited to the sign language items. In fact, deaf subjects were over 100 ms faster at categorizing the SG gestures than the hearing subjects were (ΔRT = 104 ms, p<0.001) (see Figures 2A & 3A). This pattern of results provides compelling evidence that deaf subjects’ ability to discriminate human actions differs from that of hearing non-signers. This finding suggests that experience with a signed language may enhance the discrimination of dynamic human actions more generally. However, we cannot rule out that such enhancements may emerge as a consequence of deaf individuals’ greater reliance on visual processing—for example, the need to efficiently monitor a wide range of human actions, both potentially linguistic and non-linguistic (Corina et al., 2007). It is also possible that the faster performance of deaf subjects may reflect differences in the allocation spatial attention necessary to perform this classification task. Deafness is known to alter the spatial distribution of visual attention (for a review see Bavelier, Dye & Hauser, 2006).

Figure 2.

Reaction times and standard errors in ms for deaf signers’ categorization of sign (red) and self-grooming action (green) targets. Panel A: unprimed upright targets; Panel B: unprimed inverted targets; Panel C: unrelated primed targets; Panel D: related (i.e. repeated) primed targets; Panel E: targets preceded by inverted-related primes. * p<0.05; n.s. = not significant.

Figure 3.

Reaction times and standard errors in ms for hearing non-signers’ categorization of sign (red) and self-grooming action (green) targets. Panel A: unprimed upright targets; Panel B: unprimed inverted targets; Panel C: unrelated primed targets; Panel D: related (i.e. repeated) primed targets; Panel E: targets preceded by inverted-related primes. * p<0.05, n.s. = not significant.

The data from deaf signers also show some evidence of language-specific processes. Planned comparisons indicate that deaf subjects were significantly faster in the categorization of upright sign targets than SG actions (p<0.05). In contrast, while slower than the deaf respondents, hearing subjects showed no significant difference in the time required to categorize upright signs relative to SG gestures (see Figures 2A & 3A). This finding suggests that the activation of lexical knowledge may have aided deaf signers’ performance on this task. This is an important finding as it indicates, at least for deaf signers, that this continuous categorization task is not simply tapping low-level perceptual discriminations but has likely engaged lexical recognition processes, albeit implicitly.

A second finding was that for both deaf signers and hearing non-signers, stimulus inversion disrupted the ability to categorize these actions, with inversion of SG actions showing the greatest inversion costs. This indicates that the identification of SG actions, relative to signs, may be more reliant on configural properties. This seems plausible given that the recognition of SG actions may be more reliant on the holistic form of the actions, in contrast to more highly constrained and systematic set of features which differentiate one sign from another. For example, whether one scratches one’s cheek with one or three fingers would seem to be less significant from the perspective of an interested observer than the repetitive act of scratching itself.

Of further interest are the inversion costs observed for the sign stimuli across our subject groups. We conjectured that due to their expertise with signs, the deaf subjects might show greater inversion costs for signs relative to hearing subjects. The data provide support for this prediction. Planned comparisons indicate that the inversion effects for signs were more detrimental for the deaf signers compared to hearing subjects (mean RT for deaf U-view sign minus mean RT for F-view sign = 65 ms, p<0.001; mean RT for hearing U sign minus mean RT for F sign = 32 ms, p<0.05) (see Figures 2B & 3B). That such effects were not more robustly observed may be due to methodological factors, as the strongest expertise-inversion interactions are most often observed in cases where experts differentiate objects within a given class of objects (Gauthier & Tarr, 1997; Gauthier, Williams, Tarr & Tanaka, 1998). Note that our paradigm requires between-class discriminations rather than within-class discrimination of stimulus items. This leads to the testable prediction that under conditions of lexical decision, with deaf subjects required to differentiate between signs and pseudo-signs, the costs of inversion would be greater than those observed in this categorization task. However, a test of this prediction must await further study.

In the present study, while all deaf signers reported using ASL as their daily and preferred language, they varied in the age and manner of ASL acquisition. A question of interest is whether expertise effects are differentially observed as a function of age of exposure to sign language. The prediction is that early-exposed signers may show greater inversion costs for inverted signs compared to late-exposed learners. A further analysis of our data, in which we compare the categorization times and inversion costs for early signers (n = 22) and late signers (n = 21), shows that early signers were overall faster than late signers in categorization of the stimuli (early signers = 726 ms vs. late signers = 793 ms, p<0.01); however the magnitude of inversion costs showed a non-significant trend opposite of the aforementioned prediction (for early signers, mean RT for upright signs = 683 ms, inverted signs = 728 ms, Δ = 45 ms; mean RT for upright gestures = 708 ms, for inverted gestures = 785 ms, Δ = 77 ms; and for late signers, mean RT for upright signs = 737 ms, for inverted signs = 824 ms, Δ = 87 ms; mean RT for upright gestures = 758 ms, for inverted gestures = 853 ms, Δ = 95 ms). Further research is needed to better understand these effects; as noted above, the use of within-stimulus class manipulation might provide more robust results.

In summary, these data show that deaf signers are faster and more accurate at categorizing both sign language and SG gestures than hearing non-signers. The faster identification of SG gestures by the deaf subjects suggests that experience with sign language may enhance the ability to discriminate different classes of human actions. Consideration of the inversion effects indicates that for both groups, the identification of the SG gestures, relative to the sign forms, is more reliant upon configural than featural properties. In addition, relative to the hearing subjects, deaf subjects exhibited greater inversion costs for sign language stimuli. This may be indicative of the effects of expertise with sign language. Taken together, these findings suggest that at the early stages of recognition, deaf signers may make use of an efficient and more uniform mode of discrimination in the identifications of these divergent forms of human action. To explore further the encoding of signs and SG actions across our two populations, we now turn to an examination of the effects of repetition priming.

4.0 Introduction to repetition priming

Repetition priming has been studied extensively by cognitive psychologists, and refers to the observation that subjects are faster and more accurate when identifying a repeated stimulus. In the cognitive literature, competing (though not mutually exclusive) proposals for sources of priming effects include the activation of previously acquired semantic representations, as well as newly constructed episodic perceptual traces. These representations serve to enable the subsequent recognition of repeated items (Bowers, 2000; Tenpenny, 1995). In addition to these representational accounts, psychological mechanisms, such as lowered recognition thresholds (Morton, 1979) and shifting decisional criteria (Ratcliff & McKoon, 1997) have been considered as factors in the improved accuracy and reduced response times for recognizing a repeated stimulus. Recently, repetition effects have garnered the interests of neuroscientists, who have used physiological and neuroimaging techniques to explore the anatomical signatures of stimulus repetition. The term repetition suppression has been used to describe the attenuation of hemodynamic signals as well as the reduction in single-cell firing rates that occur with repeated stimulus presentation. The exact relationship between behavioral priming effects and repetition suppression is an area of active research (see Grill-Spector, Henson & Martin, 2006, for a review). The lack of a transparent relationship between behavioral repetition priming and repetition suppression is underscored by evidence of repetition enhancement. Repetition enhancement refers to the observation that repetition of unfamiliar stimuli may lead to localized increases in neural activity (Henson, 2000; Fiebach, Gruber & Gernot, 2005). Repetition effects for unfamiliar objects have been associated with a smaller magnitude of priming than that observed for familiar objects (Vuilleumier, Henson, Driver & Dolan, 2002).

4.1 Effects of repetition priming

Repetition priming effects provide the means to probe further the specificity of the mechanisms subserving sign and SG-gesture identification. To the extent that the categorization of our stimuli is dependent upon common perceptual mechanisms, we would predict that repetition priming would affect each stimulus class in a similar fashion. However, given the differential contributions of episodic and semantic representations and stimulus familiarity, we expect a different outcome. Specifically, we predict that deaf subjects, who have lexical representations for signs and are highly familiar with these forms, would show greater repetition priming effects for sign stimuli relative to the novel SG gesture stimuli. For hearing subjects, we conjecture that the grooming action stimuli, while novel, nevertheless represent a somewhat familiar kind of action relative to the sign stimuli, and thus we predict greater repetition priming for SG gestures relative to signs for this group. In our analysis, planned comparisons are used to evaluate stimulus class differences across subject groups.

4.1.1 Results

To evaluate repetition priming effects, we contrast reaction times to target stimuli immediately preceded by unrelated primes of the same action-type, compared to cases in which target stimuli were preceded by an identical prime⁹. ANOVA with factors of Group (deaf, hearing), Action-type (sign, gesture) and Prime Relation (prime unrelated, prime related) was used to analyze the data. Table 2 provides an overview of the results for the two groups in our conditions of interest.

Table 2.

Overview of RT means (and standard errors) and accuracy means for the deaf and hearing groups for F-view critical-action items immediately preceded by F-view stimuli showing identical content or F-view filler items of the same Action-type.

	RT (ms)			Accuracy (%)
Deaf	Unrelated	Related	Overall	Unrelated	Related	Overall
Signs	682 (18.5)	522 (14.9)	602 (14.8)	96.6	99.3	98.2
Gestures	745 (18.7)	520 (15.2)	633 (14.8)	94.3	99.2	97.3
Overall	713 (15.4)	521 (14.7)	617 (13.7)	95.5	99.3	97.8
Hearing
Signs	833 (21.2)	615 (17.0)	724 (16.9)	90.2	92.4	91.3
Gestures	853 (21.3)	575 (17.3)	714 (16.9)	95.5	97.3	96.5
Overall	843 (17.6)	595 (16.8)	719 (15.6)	93.1	95.2	94.2

The RT analysis revealed a Main effect of Group (minF’1(1,97.6)=19.3, p<0.001), a main effect of Relatedness (minF’(1,85.5)=304.9, p<0.001), a Relatedness by Action-type interaction (minF’(1,91.3)=9.05, p<0.01), and a Relatedness by Group interaction (minF’(1,80.1)=4.50, p<0.05). The main effect of Group showed that again, deaf subjects were faster than hearing subjects (deaf, 617 ms; hearing, 719 ms). The effect of Relatedness indicated that targets preceded by identical stimuli were responded to faster than targets proceeded by unrelated primes (unrelated, 778 ms; related, 558 ms). The Relatedness by Action-Type interaction indicated that in the unrelated condition, signs were uniformly classified faster than the SG-grooming actions, while under conditions of repetition priming, SG gestures were classified faster than signs (all p’s<0.05). The Relatedness by Group interaction shows that hearing subjects showed substantially more priming under the condition of relatedness than did deaf subjects (hearing unrelated vs. related mean ΔRT = 248 ms, deaf unrelated vs. related mean ΔRT = 192 ms).

The accuracy analysis found a main effect of Relatedness (minF’(1,57.2)=6.33, p<0.05), due to subjects performing more accurately in the related condition than in the unrelated condition (97.6% vs. 94.4%). Although the deaf group’s mean accuracy was higher than the hearing group’s (97.8% vs. 94.2%), this difference did not reach significance.

4.1.2 Discussion: Repetition priming

The repetition priming analysis produced some expected findings, but also some surprises. Consistent with data presented in Section 3, deaf subjects were significantly faster than hearing subjects at classifying each type of action. Once again, we observe that this rapidity of response was not limited to signs, but extended to the categorization of SG gestures as well. This provides additional confirmation that deaf subjects’ recognition of human actions differs from that of hearing non-signers (see Figures 2C& 3C). This suggests that experience with a signed language, and/or conditions of deafness, may enhance the discrimination of human dynamic actions.

Under conditions of repetition priming, for both groups, we found that SG actions benefited more from repetition priming than signs. In hearing subjects, this effect was predicted on the basis of familiarity, as the SG gestures were likely to appear more familiar than the novel sign language stimuli. This group’s less robust priming for signs may reflect repetition enhancement effects, whereby the repetition of unfamiliar objects is associated with decreased priming relative to more familiar objects. For deaf subjects, the benefits of repetition priming for SG actions relative to signs are unexpected under both the lexical representation and familiarity accounts.

However, we must use some caution in interpreting these results, as floor effects may have masked evidence for differential priming. Planned comparisons of the effects of Action-type across subject groups add some support to this concern. For the hearing subjects, we note that in the unrelated prime condition, RTs for categorization of signs and gestures does not differ significantly (mean RTs for signs = 833 ms, for gestures = 853 ms, ΔRT= 20.5 ms, p=0.39). In contrast, for deaf subjects, signs preceded by unrelated sign primes were classified significantly faster than the SG gestures (mean RTs for signs = 682 ms, for gestures = 745 ms, ΔRT=63.7 ms, p<0.01; see Figures 2C & 3C). This replicates the pattern seen for classification of upright sign and gesture targets reported in Section 3.1. These data suggest that the lexical representation of signs enabled the classification of these stimuli. SG actions, lacking this representational substrate, failed to benefit to the same degree.

In the related condition, hearing subjects classified SG actions more quickly than signs (575 ms vs. 615 ms), while in contrast, deaf subjects’ categorizations of signs and gestures in this condition were essentially numerically identical (mean RTs for signs = 522 ms, for gestures = 520 ms; see Figures 2D & 3D). Evidently, deaf signers were able to use the sign and gesture prime stimuli effectively to achieve equally fast categorizations of the targets. We consider two interpretations of this result. First as mentioned earlier, these data may reflect floor effects; that is, deaf subjects may be at the limit of response time for classification of these stimuli, thus masking differential effects between the two conditions. Alternatively, as seen with the hearing subjects, these data may in fact reflect relatively greater priming of the SG gestures compared to the signs. Under this interpretation, for deaf subjects the additional priming afforded by the SG actions essentially compensates for the otherwise slower categorization of these (unprimed) forms, resulting in equivalent RTs for signs and SG actions. In support of this interpretation, we note that for both groups the magnitude of priming between conditions is quite similar (deaf: unrelated sign vs. related sign ΔRT = 160 ms, unrelated gesture vs. related gesture ΔRT = 225 ms, ΔRT difference = 65 ms; hearing: unrelated sign vs. related sign ΔRT = 218 ms, unrelated gesture vs. related gesture ΔRT = 278 ms, ΔRT difference = 60 ms). Thus while deaf signers are overall faster than their hearing counterparts, both groups derive similar benefits from the prior occurrences of stimuli of each Action-type.

Finally, we observed that hearing subjects exhibited significantly greater priming than deaf subjects. While this again may be an artifact related to floor effects, it may also reflect the contributions of episodic factors in the assessment of these actions. The question of how hearing persons come to recognize and discriminate different forms of human action is not well understood. The recognition of novel actions, whether familiar SG actions or unfamiliar signs in the present case, is often subsumed under theories of event perception (Niesser, 1976; Turvey, 1977), which place a significant burden of action understanding on perceptual prediction, working memory and episodic representations (Zacks, Speer, Swallow, Braver & Reynolds, 2007). In the present experiment, the repetition trials analyzed were zero lag (i.e., the target immediately followed the prime), a condition manipulation often associated with robust episodic effects (Bentin & Moscovitch, 1988; Kersteen-Tucker, 1991). The repeated novel actions provide a basis for episodic representations, which are used in the service of this classification task. The relatively greater priming observed for SG gestures over signs in the case of deaf signers is not inconsistent with this interpretation, as deaf subjects too may rely upon episodic information in the interpretation of these non-linguistic actions.

In summary, data from repetition priming replicates and extends the findings from the classification data discussed in Section 3.1. Here we observe that deaf signers were consistently faster than hearing subjects in categorizing both signs and SG actions. These data show that in comparison to hearing non-signers, deaf signers show an enhanced discrimination of human actions, leading to significantly faster identification and classification of actions. In spite of the rapidity of these responses, deaf signers maintained faster classification of signs over the SG actions, which is indicative of lexical recognition effects. In considering the effects of repetition priming, we found data from hearing subjects was consistent with a familiarity-based account, whereby SG actions exhibited more priming than the unfamiliar signs. The data from the deaf subjects was more difficult to reconcile, perhaps in part because of floor effects that may have masked differential effects of these two Action-types. Under one interpretation, it was suggested that episodic representations contributed in a similar fashion to the classification performance of both deaf and hearing subjects for SG actions. Taken together, unprimed classifications revealed performance advantages for the deaf subjects’ classification of human actions. Repetition priming data, however, was less conclusive in revealing representational differences across these two stimulus types.

5.0 Combining inversion and repetition priming

In a final effort to assess the processing of signs and gestures, we combined stimulus manipulations of inversion and repetition priming. Specifically, we asked whether inverted signs and SG actions would serve as equally effective primes in the classification of targets presented in their upright orientation. Two factors motivate this empirical investigation. First, one well-known property of episodic representations is their sensitivity to perceptual variation; specifically, perceptual differences in form will attenuate priming effects across instances of the same object (Schacter, 1990). By presenting inverted primes, we hoped to reduce episodic contributions in our primed classification task. Second, we saw in Section 3 that inversion differentially disrupted the classification of signs and SG actions. The costs of 180-degree stimulus inversion indicated that relative to signs, the identification of SG actions may rely to a greater extent on configural stimulus properties. Across our subject groups, deaf signers showed more inversion costs for signs relative to hearing subjects. We speculated that to the extent that deaf signers were using similar perceptual strategies in the classification of signs and SG actions—for example, by placing a greater reliance on configural properties—they might show similar priming benefits across Action-types, especially in cases where episodic priming influences might be attenuated.

5.1 Effects of inverted primes on repetition priming

5.1.1 Results

For this exploratory analysis, ANOVA was conducted on the RT data, restricted to cases in which targets were preceded by inverted primes, with the factors of Group and Action-type. The analysis revealed main effects of Group (F1(1,74)=15.2, MSE=24515, p<0.001; F2(1,26)=29.3, MSE=5335, p<0.001) and Action-type (F1(1,74)=12.6, MSE=7629, p<0.001; F2(1,26)=2.23, MSE=7152, p=0.15), as well as a Group by Action-type interaction (F1(1,74)=4.13, MSE=7629, p<0.05; F2(1,26)=2.18, MSE=5335, p=0.15). An overview of the results is given in Table 3.

Table 3.

Overview of RT means (and standard errors) and accuracy means for the deaf and hearing groups for F view critical-action items immediately preceded by U-view identical items.

Deaf	RT (ms)	Accuracy (%)
Signs	576 (20.8)	99.8
Gestures	555 (17.7)	99.9
Overall	565 (16.8)	99.8
Hearing
Signs	705 (23.7)	94.9
Gestures	625 (20.2)	97.9
Overall	665 (19.2)	96.6

As can be seen in Figures 2E & 3E, deaf subjects responded faster than hearing subjects (mean RT to signs = 576 ms for deaf vs. 705 ms for hearing, p<0.001; mean RT to gestures = 555 ms for deaf vs. 625 ms for hearing, p<0.05). We also see that the inverted primes result in nearly comparable RTs for sign and gesture targets on the part of the deaf subjects (respective means = 576 ms vs. 555 ms, ns), while for hearing subjects the categorization times for signs are significantly greater than those for gestures (respective means = 705 ms vs. 625 ms, p<0.001).

The accuracy analysis found significant only a main effect of Group (F1(1,74)=10.9, MSE=0.074, p<0.01; F2(1,26)=13.7, MSE=0.045, p<0.01), again reflecting the more accurate performance of the deaf subjects relative to the hearing group (99.8% vs. 96.6%).

5.1.2 Discussion: Combined effects of inversion and repetition priming

In a now familiar pattern, we observe that deaf subjects were faster than hearing subjects and that these effects are found for both sign and SG actions. Of particular interest are the differential effects observed across groups. Specifically, under conditions of priming, deaf subjects’ categorizations of gestures and signs were equally fast, in a similar fashion to what was observed in Section 4.1. In contrast, hearing subjects showed faster categorization of the SG actions (see Figures 2E & 3E).

These novel data provide additional evidence that the perceptual properties used in deaf signers’ identification of signs and SG actions may be more similar in kind. This stands in contrast to the hearing subjects, who continue to show slower classification times for the unfamiliar signs relative to the SG gestures.

6.0 General discussion

In this paper, we used a continuous classification task to examine whether early stages of sign language perception are similar to or differ from the perception of a class of human action, specifically self-grooming actions. Using stimulus inversion and repetition priming, we probed the properties that supported speeded classification judgments. By testing both deaf signers and hearing individuals unfamiliar with sign language, we were able to evaluate how differences in linguistic and sensory experience may contribute to the early stages of sign and human action recognition.

A consistent finding across our analyses was that deaf subjects were not only faster and more accurate than hearing subjects in the classification of signs, but that this performance extended to the classification of the novel SG actions as well. These data suggest that deaf signers may have enhanced abilities to discriminate and recognize different classes of human actions. Visual signals such as gestures, body postures, and facial expressions profoundly influence communication in ecological contexts. For deaf signers, there is increased importance in distinguishing linguistic and other communicative signals from other forms of human action in the visual modality. It is plausible that for this population, these increased visual demands are manifested as perceptual efficiencies in the assessment of human actions.

We examined the effects of stimulus inversion and repetition priming to further understand the perceptual properties and representations supporting the recognition of signs and SG actions. Consistent with previous reports, 180-degree stimulus inversion significantly disrupted recognition and categorization of these human action stimuli (Loucks & Baldwin, 2009). Inversion costs have been reported for a variety of biological stimuli, including faces, bodies, point-light displays and complex actions; our study extends these findings to signed language. Inversion costs have been used to index the relative contributions of configural properties in the perception of complex visual forms. In our data, for both deaf and hearing subjects, inversion impacted the categorization of SG actions differently than the sign stimuli, suggesting that the global relational forms of these actions play a greater role in their recognition than the particular fine details of the actions. Self-oriented actions, such as scratching one’s head or rubbing one’s eyes, while highly frequent and commonly encountered, may not trigger elaborate conceptual associations in the observer. Impressionistically, in everyday interactions one tends to “look past” these gestures, perhaps because they are largely irrelevant for the viewer (Corina et al., 2007). It seems reasonable to assume that in contrast to the highly specific feature-level details which differentiate one sign form another, that the recognition of SG actions may be more reliant on the holistic form of the actions. The lack of specificity for such actions is reflected in linguistic usage; for example, a wide range of articulatory actions fall under terms such as “to roll up one’s sleeves” or “he scratched his chin.” Generally, it is the goal of the action that is encoded linguistically, not the particular form by which these SG actions are executed. The action’s name reflects the level at which the action referent needs to be categorized (see for example Brown, 1958).

In our group comparisons, deaf subjects showed indications of greater inversion costs for the sign language stimuli, relative to the hearing subjects. This may be evidence that for deaf subjects, proficient sign recognition is dependent upon appreciation of configural properties of signs. This is of great interest as many psycholinguistic studies have been based on the notion that sign recognition may require the decomposition of signs into constituent parts. To our knowledge, our data is the first to suggest the importance of relational properties in the skilled recognition of sign language. The observation that deaf signers show greater inversion-costs for signs is consistent with theories of expertise which have shown that experts, relative to novices, exhibit greater holistic processing and encode information about the spatial relations between the parts of an object (Gauthier et al., 1998). Experts encode information over a wider spatial extent than do novices and they tend not to attend selectively to single parts of the object (Bukach, Gauthier & Tarr, 2006). In the present study, while all deaf signers reported using ASL as their daily and preferred language, they varied in the age and manner of ASL acquisition. A question left for future research is whether these expertise effects are differentially observed as a function of age of exposure to sign language.

Repetition priming was used to explore the representational properties of these human actions. It was speculated that at least for deaf subjects, the availability of lexical semantic representations for signs might affect response times for repeated signs relative to the SG actions; this was not observed, however. Robust repetition effects were observed for both signs and SG actions, consistent with reported effects of episodic representations in our zero-lag prime-target manipulation. For hearing subjects, primed contexts resulted in overall faster categorizations for SG actions compared to signs. These results are consistent with data showing that repetition priming is enhanced for familiar stimuli, while the repetition of unfamiliar stimuli may evidence repetition enhancement effects, leading to slower reaction times (Vuilleumier, Henson, Friver & Dolan, 2002). In the case of deaf subjects, priming contexts resulted in equivalent categorization times for SG actions and signs. This may be an indication that deaf signers, in contrast to hearing subjects, were able to use the SG actions and signs in a similar fashion to reach rapid classification of these differing Action-types. A similar pattern was also observed during the categorization of upright targets preceded by 180-degree inverted primes showing identical content, where the perceptual mismatch between prime and target is apt to attenuate perceptually-based episodic effects. While overall priming effects were reduced, hearing subjects continued to show more robust priming for SG gestures than for sign language stimuli. In contrast, deaf subjects’ reaction times for SG gestures and signs were not significantly different. These outcomes provide additional support for our claim that relative to sign-naive hearing subjects, deaf subjects are processing both Action-types in a similar fashion.

Overall, our data provide compelling evidence that deaf signers’ rapid and accurate recognition of signs is not limited to sign language forms, but extends to the processing of non-linguistic human actions as well. More subtle effects suggest that deaf subjects, relative to hearing non-signers, may be more reliant upon configural properties in the perception of signs. These data indicate that experience with a sign language and/or the heightened visual-sensory experience related to deafness may alter the perceptual processes used in the identification of human actions.

The present study extends earlier related work reported in Corina, Grosvald and Lachaud (2011). That study examined evidence for language-specific processing effects by examining the degree to which deaf signers’ and hearing non-signers’ perception of signs and human actions were differentially robust to changes in perceptual viewpoint. A repetition priming manipulation was used to compare the magnitude of priming in cases where the repeated stimuli were filmed from the same forward-facing viewpoint, to cases where the repeated items differed in viewpoint (e.g., a prime showing a side view of the signer articulating the sign DOLL, followed by a target showing the front facing view of the sign DOLL). While robust effects of viewpoint were found, there was little evidence of group-related or action-class differences that could be interpreted as language-specific effects of perceptual invariance. Instead, the processing costs as a function of viewpoint were uniform across both subject populations. The current study suggests that under more extreme perceptual manipulations (i.e., 180-degree inversion), group and stimulus-class differences may be observed. These differences in reaction time, and subtle differences in the weighting of perceptual properties, however, are cast against a more global pattern in which both hearing non-signers and deaf signers show largely similar effects across action categories. Taken together, these studies suggest a scenario by which sign expertise may lead to minor modifications of a general-purpose human action recognition system, rather than evoking a qualitatively different mode of processing. Our findings support the contention that signed languages make use of perceptual systems through which humans understand or parse human actions and gestures more generally.

Highlights.

Deaf users of ASL and Hearing non-signers categorized ASL signs and human actions.

Effects of repetition priming and stimulus inversion were evaluated.

Deaf subject are faster than hearing subjects in recognizing sign and human actions.

Deaf subjects are more sensitive to the configural properties of signs.