How to distinguish gesture from sign: New technology is not the answer

Abstract

Linguistic and psycholinguistic tests will be more useful than motion capture technology in calibrating the borders between sign and gesture. The analogy between motion capture (mocap) technology and the spectrograph is flawed because only vocal articulators are hidden. Although information about gradience and variability will be obtained, the technology provides less information about linguistic constraints and categories. Better models are needed to account for differences between co-speech and co-sign gesture (e.g., different degrees of optionality, existence of beat gestures).

Goldin-Meadow and Brentari (G-M&B) call for new technology to analyze motion and location as a means to distinguish between co-speech gestures and signs. I am not so optimistic about this approach. G-M&B suggest an analogy between the development of motion analysis tools for sign/gesture and the importance of the development of the spectrograph for advancing our understanding of speech. However, this analogy is flawed because (1) unlike sign/gesture, the articulators are not observable for speech and thus visualizing acoustic information was particularly crucial for spoken language, and (2) spectrograms and motion capture data provide a great deal of information about variability in the signal, but less information about linguistic or cognitive categories. For example, G-M&B argue that if the variability of signers’ movements is less than speakers’ movements when describing the same motion event, this finding would constitute evidence that signers’ movements are generated by a different system (possibly linguistic) than speakers’ gestures. However, reduced variability could simply be due to the motor expertise of signers who have much more experience producing communicative information with their hands (see Hilger et al. 2015). Although motion capture technology may be essential for investigating the phonetic and phonological properties of sign language (e.g., Jantunen 2013; Tyrone & Mauk 2010), this technology is less likely to provide the data necessary to understand the relationship between gesture and sign.

Rather, I suggest that the field will be advanced more by linguistic analyses (e.g., assessing whether syntactic or semantic structures constrain the interpretation of variations in location or motion, such as Schlenker 2011) and psycholinguistic experiments (e.g., testing whether and how signers or nonsigners categorize gradient information expressed in signs/gestures, such as Emmorey & Herzig 2003). Even then, more theoretical work is needed to establish models of language and gesture processing in order to determine how sign and gesture are combined and whether this combination is parallel to how speech and gesture are integrated.

For example, according to the gestures as simulated action (GAS) framework (Hostetter & Alibali 2008), gestures arise from perceptual and motor simulations that underlie embodied cognition, and they are produced when the level of motor and premotor activation exceeds a preset threshold (influenced by individual and contextual factors). Such a framework assumes that gestures are not obligatory, and this seems to be true except under certain (rare) deictic circumstances (e.g., “I caught a fish this big” is ill-formed without a size-illustrating gesture). In contrast, as noted by G-M&B, most linguistic analyses of sign language assume that directional (“agreeing”) verbs and pronouns comprise both a linguistic and a gestural component, although whether the gestural component (indicating the location of a referent) is always expressed is an empirical question. G-M&B suggest that the difference in the optionality of gestures may be a matter of degree, but nonetheless the difference is quite large – many more signed than spoken language expressions are ill-formed without the gestural referential component. Further, the size of this difference indicates that perceptual and motor simulations are unlikely to be the source of both co-speech and co-sign gesture production. The point here is that it is unclear how current models of co-speech gesture production – including other proposals, such as the interface model (Kita & Özyürek 2003) – account for the high degree of obligatory gestural expression in signed compared to spoken language. It is unlikely that motion capture technology can help much with this question.

Interestingly, G-M&B do not discuss beat gestures which are small movements of the hand (or head) that contain little semantic information (unlike deictic or iconic gestures) but have pragmatic or discourse functions such as marking prominence (McNeill 1992). Beat gestures are ubiquitous for speakers, but it is unclear whether the parallel exists for signers. One possibility is that the head movements of signers constitute beat gestures. Puuponen et al. (2015) recently used motion capture technology to identify the prosodic, grammatical, and discourse functions of head movements in Finnish Sign Language. They found that some head movements (e.g., a head thrust or pull) were produced primarily as prosodic cues signaling the prominence of a manual sign in an utterance. However, it is unclear whether these head movements should be analyzed as beat gestures or as prosodic markers in Finnish Sign Language. The primary problem is that sign and gesture cannot be separated by articulatory systems, unlike speech and gesture. Although motion capture technology was able to characterize small changes in head movements, Puuponen et al. (2015) were unable to find noticeable differences between head movements that seemed more gestural (e.g., nods indicating affirmation) and those that were likely to be more linguistic (e.g., nods occurring at syntactic boundaries).

Linguistic analyses may be more fruitful in determining whether particular head or manual movements constitute part of the prosodic system in a sign language, and psycholinguistic experiments can help determine how signers interpret these movements. Perhaps more importantly, better models of the relation between language and gesture can provide clearer hypotheses about how gradience is expressed in both the vocal and manual modalities and whether certain questions might even be ill-posed, e.g., perhaps there is no functional difference between beat gestures and prominence marking for sign languages, and spoken languages simply have the ability to spread prominence marking across two articulatory systems.