Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2019 Aug 9.
Published in final edited form as: Brain Lang. 2019 Jan 21;190:31–37. doi: 10.1016/j.bandl.2018.12.008

Gesture height reflects common ground status even in patients with amnesia

Caitlin Hilverman a,*, Sarah Brown-Schmidt b, Melissa C Duff a
PMCID: PMC6688473  NIHMSID: NIHMS1044181  PMID: 30677621

Abstract

When we communicate, we alter our language and gesture based on the mutually shared knowledge – common ground – that we have with our listener. How memory supports these alterations remain unclear. We asked healthy adults and patients with hippocampal amnesia to engage in a referential communication task. Previous work suggests that common ground can be encoded by distinct memory systems; Amnesic patients show normal learning and referential label use as common ground increases, but inconsistently mark these labels with definite determiners (e.g., the vs. a windmill). Which memory systems support the ability to mark common ground via hand gesture? We found that gestures of both healthy participants and amnesic patients reflected common ground status. Both groups produced high gestures when common ground was lacking, and were less likely to do so as common ground increased. These findings suggest that gesture can reflect common ground status during conversation, potentially via non-declarative memory.

1. Introduction

Speakers alter their communication based on their memory of mutually shared information that they have with their listener. This mutually shared information– called common ground – affects speakers’ language in a multitude of ways. Speakers with more common ground with the listener tend to reduce their spoken language; they shorten their utterances, reuse specific lexical items, and increase their use of definite reference (e.g., the windmill vs. a windmill; Clark & Wilkes-Gibbs, 1986). Although these alterations rely on memory representations of common ground, little is known about the biological memory systems that support these communicative changes through spoken language and gesture. Recent progress has been made in understanding the biological memory systems that support use of common ground in spoken language (e.g., Brown-Schmidt & Duff, 2016).

The connection between memory and common ground has inspired multiple lines of behavioral research and theoretical proposals. The classic view focuses on memory for past events, with episodic memory for these past events comprising the information needed to infer whether something is in common ground, or not. On this view, if a speaker and a listener have experienced a past event together (rationally and attentively), the speaker can presume that the listener has common ground for that event (Clark & Marshall, 1978). Once common ground is established, a definite referring expression – e.g., “the duck” – can be used to signal that what is being referred to is in common ground; the listener in turn can search through their episodic memory for the referent (Clark & Marshall, 1981). This account was later challenged by Horton and Gerrig (2005a, 2005b) who posited that episodic, person-based memories may not be necessary to account for common ground in all cases. Rather, they proposed a resonance-based account of common ground where components of the conversation resonates with information in long term memory rapidly and can support use of common ground in the absence of an explicit search of past events (Horton, 2007; Horton and Gerrig, 2005a, 2005b, 2016).

More recently, our group has proposed a psychologically and biologically integrated account of the memory determinants of common ground that incorporates components from both accounts (Brown-Schmidt & Duff, 2016). We approach the relationship between memory and common ground from a multiple memory systems framework supported by decades of neuropsychological research. This framework posits two anatomically and functionally distinct systems: declarative memory – memory for facts and events, supported by the hippocampus and surrounding medial temporal lobe (MTL) – and non-declarative memory – memory for skills and habits supported by areas outside of the MTL (e.g., Cohen & Squire, 1980). Because the classic view proposed by Clark implicates episodic (declarative) memory processes, we can test this view by examining how the declarative memory system supports common ground use. We have done this by working with patients who have hippocampal damage and declarative memory impairment, or hippocampal amnesia. Because their brain damage is restricted to the medial temporal lobes and the hippocampus – neural areas critical for declarative memory – these patients have a behavioral dissociation of impaired declarative but intact non-declarative memory that serves as an ideal test case of Clark and Marshall’s (1978, 1981) proposal.

Initial tests with this population used a collaborative reference task involving abstract “tangram” images (Clark & Wilkes-Gibbs, 1986). In this task, the amnesic participant (the “director”) and a familiar communication partner (the “matcher”) have matching sets of tangram images. A barrier between the director and matcher then blocks the view of each other’s workspace. The director’s tangrams are in a fixed order that they must describe to the matcher. During this task, patients with declarative memory impairment shortened their utterances and reused lexical items in a manner indistinguishable from healthy individuals (Duff, Hengst, Tranel, & Cohen, 2006), suggesting that acquisition of common ground may not rely solely on declarative (episodic) memory.

As common ground is acquired in this task, healthy participants also increase their use of definite reference over time; once a referent is in common ground, the director uses the definite article “the” rather than the indefinite article “a” to refer to a particular tangram (e.g., “the siesta man”, rather than “a shape that looks like a man taking a siesta”). This use of definite reference signals to the matcher early on in the utterance that they are referring to a referent already in common ground (Clark & Marshall, 1981). However, despite the patients’ success in developing increasingly concise labels and reusing those lexical items, they were less likely to use definite reference as common ground accumulated (Duff, Gupta, Hengst, Tranel, & Cohen, 2011). The observation that different aspects of common ground seem to be supported by different forms of memory leads to our multiple memory systems view of common ground (Brown-Schmidt & Duff, 2016; Duff & Brown-Schmidt, 2012; Duff et al., 2011). Furthermore, amnesic patients’ failure to consistently use definite reference suggests that declarative memory may be a key contributor to the explicit signaling of common ground through speech. However, there is more to communication than speech; in addition to speaking, we also produce spontaneous hand gestures that can communicate information. Can gesture serve as a visual re-flection of common ground status, and if so, what memory system supports this communicative behavior?

Gesture, like spoken language, becomes more concise as common ground is established (for a review see Holler & Bavelas, 2017). As common ground accumulates in a conversation, the gesture to speech ratio typically remains constant, indicating that both speech and gesture are being reduced in tandem. Additionally, gestures are rated as less informative and less precise with increased common ground (Gerwing & Bavelas, 2005), suggesting that speech and gesture may be co-affected by changes in common ground over time. However, gesture differs from spoken language in other ways that may be informative about the memory systems affecting these changes: gesture communicates visually, with the ability to readily convey spatial and motor information iconically. Furthermore, multiple features of gesture can be modulated simultaneously (i.e., form, height, speed, etc.; see Hilliard & Cook, 2017). As such, changes in gesture as a function of common ground are visually available to the listener, potentially providing a signal of common ground status that can be encoded independently of spoken language.

One property of gesture form that has been found to be affected by common ground is gesture height. Hilliard and Cook (2015) had speakers complete a computerized Tower of Hanoi task which involved moving a stack of disks peg to peg without putting a larger disk on top of a smaller disk. The disks could be moved on a computer in one of two ways: by dragging the disks across the pegs or lifting them up and over the pegs. When the speaker and listener had different knowledge about how to move the disks, speakers’ gestures were higher relative to the body than if they had the same knowledge. This aligns with work on American Sign Language demonstrating that signs are produced lower when they are presented later in the sentence (Tyrone & Mauk, 2010). Together, these findings suggest that the height at which gesture is produced may reflect common ground status, with information not in common ground paired with higher, potentially more visible gestures.

If gesture height is indeed modulated systematically as a function of common ground, this would provide speakers with the opportunity to mark whether information is new or not without encoding this information verbally. Indeed, Hilliard and Cook (2015) found no difference in properties of spoken language as a function of common ground. Furthermore, when asked during debriefing, speakers reported not knowing that they made this height modulation with their gestures. It is an open question as to which memory mechanisms support this modulation of gesture during communication.

Links between gesture and non-declarative memory mechanisms have been hinted at in recent work using learning tasks. Patients with hippocampal amnesia – who have preserved non-declarative memory – perform significantly better on new word learning tasks when they gestured at encoding (Hilverman, Cook, & Duff, 2018). Additionally, patients with Parkinson’s Disease and non-declarative (procedural) memory deficits do not show the same benefits in learning from producing and viewing gestures that are evident in healthy participants (Klooster, Cook, Uc, & Duff, 2015). Healthy individuals have also been shown to rely on non-declarative motor mechanisms during comprehension. For example, when viewing gesture during sentence comprehension, these sentences are remembered better if they are accompanied with gesture; however, this benefit goes away if the hands are engaged in an additional motor task while comprehending the sentences, suggesting that the motor component of gesture is critical for learning (Ianì & Bucciarelli, 2017). It remains unknown if gesture reveals this same link with non-declarative memory in communicative contexts.

2. Experiment

The present study is a re-analysis of the data described in our earlier work (Duff et al., 2006, 2011). The original analysis of the dataset focused on spoken utterance form, and those analyses revealed that amnesic patients, like comparison participants, shorten their utterances but are less successful in their use of definite reference to signal common ground (Duff et al., 2006, 2011). Building on prior research by Hilliard and Cook (2015), we return to this dataset to examine the gesture height and spoken language in the interactive setting of the collaborative referencing task of healthy adults and patients with severe declarative memory impairment – but intact non-declarative memory.

We hypothesize that if gesture height is modulated for common ground status via non-declarative memory mechanisms, gesture should be systematically lowered as common ground increases throughout the task by both healthy adults and patients with amnesia. Alternatively, if amnesic patients do not alter gesture like the healthy participants, this would suggest that the modulation of gesture height with common ground may be supported by declarative (episodic) memory. In a supplemental analysis, we examine whether healthy participant’s use of indefinite reference (e.g. “a dancer”) co-occurs with raised gestures, as both reflect a lack of common ground. We note that these hypotheses were developed prior to the current re-analysis of hand gesture in the dataset.

3. Methods

The data were collected as part of an earlier study that investigated the acquisition of referential labels and use of definite reference (Duff et al., 2011, 2006). Prior work did not code or examine gesture production in these data.

3.1. Participants

Participants included 6 patients with amnesia and 6 healthy comparison participants. Each participant selected a familiar partner (e.g., a friend or family member) that completed the task with them.

Structural MRI examinations confirmed bilateral hippocampal damage in 5 of the 6 patients. Participant 2563 could not undergo MRI examination due to a pacemaker; damaged confined to hippocampal region confirmed by computer tomography. Of the 6 patients, 3 sustained bilateral hippocampal damage from an anoxic (hypoxic) event, 2 sustained more extensive bilateral medial temporal lobe damage bilaterally – including the hippocampus – from herpes simplex encephalitis, and 1 sustained shearing lesions in the white matter tracts surrounding the hippocampus following a closed head injury (see Table 1 for demographic information).

Table 1.

Demographic, neuroanatomical, and neuropsychological characteristics of participants with hippocampal amnesia.

Participant Demographic Characteristics Neuropsychological Scores
Etiology Sex Handedness Education (years) Age (years) WAIS-III FSIQ WMS-III GMI AVLT Trial 5/ Delay CFT Delay Boston Naming Test MAE Token Test
2563 Anoxia M Let 16 47 102 75 7/1 7 52 44
2363 Anoxia M Right 16 45 98 73 9/0 5 58 44
0001 Anoxia F Right 9 54 90 54 7/0 5 56 44
1951 HSE M Right 16 51 106 59 6/2 4 49 44
2308 HSE M Left 18 46 87 45 5/0 0 52 44
0002 CHI F Right 20 45 126 49 3/0 0 59 44
Mean 15.8 (3.7) 48.0 (3.68) 101.5 (13.4) 59.2 (12.4) 6.2 (2) / 0.5(0.1) 3 (2.8) 54.3 (3.9) 44.0 (0)
Open in a new tab

Note: HSE = herpes simplex encephalitis; CHI = closed head injury; F = female; M = male. Standard deviations are given in parentheses. Neuropsychological data include Full Scale IA (FSIQ) from the Wechsler Adult Intelligence Scale – Third Edition (WAIS-III; Wechsler, 1997a), General Memory Index (GMI) score from the Wechsler Memory Scale – Third Edition (WMS-III; Wechsler, 1997b), the Trial 5 and Delay scores from the Rey Auditory and Verbal Learning Test (AVLT; Schmidt, 1996), Delay scores from the Complex Figure Test (CFT; Meyers & Meyers, 1995), Boston Naming Test score (Goodglass, 2000), and Token Test score from the Multilingual Aphasia Examination (MAE; Benton, Sivan, Hamsher, & Varney, 1994).

All patients with amnesia had severe and selective declarative memory impairment with generally preserved intelligence, cognition, and non-declarative memory (see Cavaco, Anderson, Allen, Castro-Caldas, & Damasio, 2004; Duff et al., 2011; Duff, Wszalek, Tranel, & Cohen, 2008). All patients were free of any motor deficits that could affect gesture production. General Memory Index scores from the Wechsler Memory Scale – Third Edition were at least 25 points lower than the Full-Scale IQ from the Wechsler Adult Intelligence Scale – Third Edition (see Table 1 for the neuropsychological scores for patients with amnesia). Healthy comparison participants were matched with amnesic patients on sex, age, education, and handedness. The familiar partners of both patients and comparisons were all neurologically healthy and similar in age (amnesic partners: 57.6 years; comparison partners: 56.5 years) and education (amnesic partners: 14.0 years; comparison partners: 13.7 years).

3.2. Procedure

Participant pairs were instructed that they would be playing a matching game. Each pair sat at a table facing each other with a partial barrier in between them that was 14 in. in height. This barrier allowed them to see each other’s faces and shoulders but obscured the view of each other’s workspaces on the desk in front of them. Each partner had a playing board with 12 numbered spaces and identical sets of 12 playing cards displaying unlabeled tangrams (see Fig. 1). The amnesic patient or comparison participant was always the director, and the familiar partners were always the matchers. The director began with their cards in a predetermined order on the board that changed each trial and told the matcher which cards to place in which numbered spaces; the goal was to make both boards identical. Each pair matched all 12 cards 24 times across four sessions (six games per session with two sessions on each of two days). Here, we focus on data only from the first session, as multiple participants did not gesture in session 2 and beyond, likely because common ground was well-established and tangrams could be communicated using short, concise spoken labels.

Fig. 1.

Fig. 1.

Open in a new tab

Example of a director, right, producing a visible gesture over the barrier. The barrier occluded each participant’s workspace while allowing them to see each other’s faces and shoulders.

4. Coding

4.1. Reference

To examine the co-occurrence of indefinite reference and high gesture production, we used coded referential data reported in Duff et al. (2011). Two coders who were blind to hypotheses transcribed the referential game, and identified the director’s first attempt during each trial at labeling each card (e.g., “Number one kinda looks like a dragon reading a book”). These references were coded as definite (e.g., “the camel,” “the one reclining in the chair with his feet stickin’ out, against the tree”) or indefinite (e.g., “looks like a couple of hills,” “it has a diamond at the top of the figure”; see Wilkes-Gibbs & Clark, 1992; Hengst, 2003). On later trials, pairs often produce a label that functioned like a proper noun (e.g., “siesta man” instead of “the siesta man”); following the original coding system, these instances were coded as definite (also see Wilkes-Gibbs & Clark, 1992). See Duff et al. (2011) for further details.

4.2. Gesture

Gestures were annotated in ELAN (Lausberg & Sloetjes, 2009). A hand movement was categorized as a gesture if it was produced while speaking, in rhythm with the speech that it accompanied. Hand movements that were produced while not speaking and self-adaptive movements (e.g., scratching face, fixing hair) were not coded.

Each gesture was categorized as being visible or not visible. A gesture was considered visible if the gesture produced could be viewed by the listener. Therefore, if a gesture was produced at shoulder height – which may not have been above the barrier – but the listener was positioned high enough to still perceive the gesture then it was considered visible (see Fig. 1 for an example of a visible gesture). The listener did not have to be looking at the gesture for it to be categorized as visible. Inter-rater reliability of the coding for visibility was 92%; in cases of difference, a third coder made a decision regarding whether the gesture was visible to the listener or not. Finally, while both members of a pair (directors and matchers) produced gestures given the interactional nature of the task, coding was conducted on the gestures of the directors (amnesic patients and comparisons).

4.3. Analysis

Our analyses of reference and gesture visibility consisted of mixed effect regression models that predicted the dimension of interest as a function of group (patients with amnesia vs. healthy comparison participants), tangram card number (1–72, standardized), and their interaction. This allowed us to examine whether there were indeed differences in gesture visibility by group and how this changed as common ground increased (i.e., as tangram card number increased). We determined the random effect structure for each model by using log-likelihood ratio testing; if the testing resulted in non-significant chi-square values then the random effect being tested was not included in the model. We used dummy coding in all models, with healthy comparison participants serving as the reference group.

5. Results

5.1. Reference

A prior analysis of reference use was conducted by Duff et al. (2011). However, since we focus only on data from the first session – and use it as a variable in our analyses below - we analyze it here as well using mixed effects regression models (the original report used ANOVA models). We assessed performance trial by trial by creating a variable for tangram card number across all games (12 cards per game and 6 games for 72 total cards). To assess participants’ reference use, we used a binomial mixed effect model that predicted the use of definite reference (e.g., the windmill) as a function of group (comparison/amnesic), tangram card number (standardized), and their interaction. We included random by-subject, and by-tangram card number intercepts.

There was a significant main effect of tangram card number (β = 3.96, z = 3.25, p = .001); as tangram card number increased, the probability of producing a definite reference also increased. There was also a significant main effect of group (β = −3.43, z = 3.37, p < .001); patients with amnesia were less likely than comparison participants to produce a definite reference (Fig. 2). The interaction of group and card number was marginally significant (β = −1.38, z = −1.80, p = .07); the increase in definite reference across cards was marginally larger for healthy comparison participants than it was for patients with amnesia.

Fig. 2.

Fig. 2.

Open in a new tab

Logistic probability curves demonstrating the probability of producing definite reference (“the”) over the course of the experiment. Patients are shown in a grid with their demographically-matched healthy comparison participants. Although both groups demonstrated an increase in the probability of definite reference use over the course of the experiment, patients were marginally less likely to produce definite reference as the card number increased.

5.2. Gesture

All participants (both patients and comparisons) gestured. Participants also produced ample visible gestures, although the quantity decreased across games (Table 2). To evaluate whether the groups gestured in similar amounts, we used a mixed effect regression model to predict the total number of gestures produced. There was a fixed effect for group and random by-participant and by-game intercepts. Patients with amnesia produced a similar amount of total gestures to the comparison group (β = −0.06, t(10) = −0.12, p = 0.90).

Table 2.

Mean number of gestures (and standard deviation) produced in each session in each of the groups. Patients with amnesia and healthy comparison participants gestured at similar rates.

Game 1 Game 2 Game 3 Game 4 Game 5 Game 6
Comparison Visible 1.0 (1.8) 0.6 (1.7) 0.4 (0.9) 0.1 (0.6) 0.2 (1.1) 0.1 (0.2)
Total 2.6 (3.8) 1.9 (1.3) 1.0 (1.8) 0.5 (1.2) 0.6 (1.8) 0.3 (0.8)
Amnesic Visible 1.5 (3.0) 0.5 (1.0) 0.5 (2.2) 0.2 (0.6) 0.3 (0.6) 0.1 (0.3)
Total 2.4 (3.5) 1.3 (2.4) 1.2 (2.7) 0.7 (1.5) 0.7 (1.4) 0.2 (0.5)
Open in a new tab

To determine if the probability of producing a visible gesture decreased throughout the task, we used a binomial mixed effect model that predicted gesture visibility as a function of group (comparison/ amnesia), tangram card number (standardized), and their interaction. We included a covariate for number of non-visible gestures to ensure that we were indeed assessing the probability of visible gestures and not gesture production in general. There was random by-participant intercept.

There was a significant main effect of tangram card number (β = −3.26, z = −2.95, p = .003); as card number increased, the likelihood of producing a visible gesture decreased (Fig. 3). The main effect of group was not significant (β = 2.52, z = 1.30, p = 0.19), indicating that patients with amnesia and healthy comparison participants were similarly likely to produce visible gestures. The covariate for non-visible gesture number was not significant (β = −0.03, z = −0.54, p = 0.59). The interaction of group and card number was also not significant (β = 1.86, z = 1.37, p = 0.17), indicating that the likelihood of producing a visible gesture across the task was similar for patients and comparison participants (Fig. 2).

Fig. 3.

Fig. 3.

Open in a new tab

Logistic probability curves demonstrating the probability of producing a gesture above the barrier (visible to the listener) over the course of the experiment. There were 12 tangrams described in each game. Amnesic patients and comparison participants had similar probabilities of producing a gesture above the barrier as card number increased.

5.2.1. Gesture and reference co-occurrence

We next assessed whether reference and gesture height were related; if producing visible gestures is indeed a way to signal common ground, then this behavior should co-occur with indefinite reference use. We predicted that gesture visibility and reference use would be related only in the healthy comparisons. We did not predict a relationship between gesture production and definite reference use because of the patients’ impaired declarative memory, patients’ use of reference varies both within and across patients (Fig. 2), making it difficult to evaluate the relationship of the two behaviors over the course of the experiment at a group level.

To assess the relationship between reference use and gesture visibility, we conducted two-way contingency table analyses to evaluate whether there was a relationship between the two behaviors in each group separately (see Table 3). In patients with amnesia, gesture visibility and reference were not significantly related (χ2(1) = 0.0165, p = 0.898). In healthy comparison participants, gesture visibility and reference use were significantly related (χ2 (1) = 20.699, p < .001).

Table 3.

Contingency tables demonstrating the frequency of reference use (definite and indefinite) and gesture visibility (gesture visible or not visible) by group. Patients with amnesia did not show a significant relationship between reference use and gesture visibility, while healthy comparison participants did.

Reference Total
Indefinite Definite
Amnesic patients
Gesture Not 207 142 349
Visible 48 34 82
Total 255 176 431
Healthy comparisons
Gesture Not 88 262 350
Visible 42 41 83
Total 130 303 433
Open in a new tab

We next examined the relationship between gesture visibility and reference use over the course of the experiment in healthy comparison participants. We used a binomial regression model that predicted the likelihood of producing a gesture above the barrier as a function of referential form (indefinite, definite), with definite reference serving as the reference group. We included random by-participant and by-tan-gram card number intercepts (standardized). Reference significantly predicted the likelihood of a visible gesture being produced (β = 1.50, z = 2.06, p = .04); indefinite reference (a) was significantly more likely than definite reference (the) to predict the production of a visible gesture.

6. Discussion

Communication changes as common ground is formed in conversation, with speakers reducing and changing what is said in both spoken language and gesture. Our findings indicate that gesture reflects common ground status during conversation, with high, co-visible gestures co-occurring with new information in spoken language. This is consistent with Hilliard and Cook (2015), who demonstrated that speakers gestured higher in space when their listener lacked information about the task. However, in the work reported here, rather than all gestures being visible to the listener, the presence of a barrier meant that speakers had to raise their gestures up over the barrier for their listener to view it. This indicates that speakers may utilize gesture height to signal that the common ground status of the information being communicated. Furthermore, our analysis of gesture and reference use in the healthy comparison participants demonstrates that high co-visible gestures co-occur with indefinite reference. Interestingly, the fact that patients with amnesia can signal common ground status with gesture but not reliably with reference supports the idea that there are multiple ways to display shared knowledge. While they co-occur in healthy individuals, they are distinct and independent communicative channels.

Open questions surround which memory systems support the representation and use of common ground during conversation. Here, we demonstrate for the first time that this modulation of gesture visibility across the course of the conversation is present in both patients with amnesia and healthy comparison participants; when common ground was lacking, gestures were more likely to be produced up above the barrier such that the listener had visible access to them. The fact that patients with amnesia systematically lowered their gestures just as healthy participants did is consistent with the patients’ ability to arrive at increasingly concise labels for the cards and reuse lexical items across the task in a manner indistinguishable from healthy adults (Duff et al., 2006).

Our findings provide additional evidence concerning the complex relationship between memory and common ground. Recent work has demonstrated that common ground is not a unitary phenomenon that maps onto a single memory system (Duff et al., 2011; see Brown-Schmidt & Duff, 2016 for review). Rather, multiple memory systems are recruited in forming and using common ground in communication. Hippocampal-dependent memory in part supports the ability to consciously access and explicitly signal common ground using definite reference (Duff et al., 2011). However, non-declarative forms of memory appear sufficient to support the gradual acquisition of information needed to form and use concise labels (Duff et al., 2006), tailor unique utterances for specific partners (Yoon, Duff, & Brown-Schmidt, 2017), and as demonstrated here, signal common ground status via gesture height.

Growing evidence for a link between gesture and non-declarative memory has been observed in other paradigms (Hilverman et al., 2018; Hilverman, Clough, Duff, & Cook, 2018; Ianì & Bucciarelli, 2017; Klooster et al., 2015). This link is bolstered by the similarities between the properties of non-declarative memory and gesture. Gesture is often produced and understood without conscious awareness – participants report not being able to remember whether or not they gestured and what they gestured about – similarly to the characterization of non-declarative memory as being largely inaccessible to conscious awareness. Gesture is also a form of action. This feature of gesture is reminiscent of the connection between some forms of procedural memory and the motor system, and is also similar to characterizations of non-declarative memory as resulting directly from experience (Eichenbaum & Cohen, 2004; Reber, Knowlton, & Squire, 1996).

Despite this link between gesture and non-declarative memory, prior work has also demonstrated a relationship between gesture and declarative memory. Hilverman, Cook, and Duff (2016) found that patients with amnesia gestured significantly less than healthy comparison participants when describing past events and information from the remote past. Why was there not a significant difference in amount of gesture in the present study? We posit that the gesture rate difference in Hilverman et al. (2016) was due to the task demands; in our previous work participants were prompted to reconstruct past events – a heavily hippocampally-mediated task – and then describe that event. Because their representations were impoverished (e.g., their verbal descriptions contained significantly fewer episodic details), their gesture production was impoverished as well. Indeed, when the prompted event required less hippocampal involvement (i.e., they described a procedural task, like making a sandwich, rather than an event, like the death of JFK), their gesture rate was less affected. Here, everything that was being discussed was visually present and accessible, not requiring hippocampal reconstruction, and thus patients with amnesia could readily gesture about the tangram cards. Moving forward, it will be critical to consider the task demands when assessing the relationship between gesture and memory.

In sum, language use is inherently shaped by representations of common ground we hold with others; on the classic view, these representations are formed on the basis of episodic memory for joint experience (Clark & Marshall, 1978, 1981). Language use is also inherently multimodal, with speech and hand gestures often conveying unique aspects of the contents of our memory. The characterization and understanding of how hand gesture can engage multiple memory systems offers a novel window into the relationship between memory and common ground, and the relationship between memory and language more broadly. Speakers can alter specific properties of their gesture – like gesture height and visibility – independently of what is said, and we show that this modulation of common ground status is possible even in the face of profound declarative (episodic) memory impairment. Most broadly our findings point to communication as a process by which information is conveyed through multiple communicative channels, each of which draws on the human biological memory systems in distinct ways.

Acknowledgments

Supported by NIDCD F32-DC016580 to CH and NIDCD R01-DC011755 to MCD and SBS.

References

  1. Benton AL, Sivan AB, deS Hamsher K, & Varney NR (1994). Contributions to neuropsychological assessment: A clinical manual. USA: Oxford University Press. [Google Scholar]
  2. Brown-Schmidt S, & Duff MC (2016). Memory and common ground processes in language use. Topics in Cognitive Science, 8(4), 722–736. 10.1111/tops.12224. [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Cavaco S, Anderson SW, Allen JS, Castro-Caldas A, & Damasio H (2004). The scope of preserved procedural memory in amnesia. Brain, 127(8), 1853–1867. 10.1093/brain/awh208. [DOI] [PubMed] [Google Scholar]
  4. Clark HH, & Marshall C (1978). Reference diaries. In Theoretical issues in natural language processing (pp. 57–63). Retrieved from http://portal.acm.org/citation.cfm?id=980262.980272.
  5. Clark HH, & Marshall CR (1981). Definite reference and mutual knowledge In A. Joshi, Weber B, & Sag I(Eds.). Elements of discourse understanding (pp. 10–63). Cambridge: Cambridge University Press. [Google Scholar]
  6. Clark HH, & Wilkes-Gibbs D (1986). Referring as a collaborative process. Cognition,22, 1–39. [DOI] [PubMed] [Google Scholar]
  7. Cohen N, & Squire L (1980). Preserved learning and retention of pattern-analyzing skill in amnesia: Dissociation of knowing how and knowing that. Science, 210(4466), 207–210. 10.1126/science.7414331. [DOI] [PubMed] [Google Scholar]
  8. Duff MC, & Brown-Schmidt S (2012). The hippocampus and the flexible use and processing of language. Frontiers in Human Neuroscience, 6(April), 69 10.3389/fnhum.2012.00069. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Duff MC, Gupta R, Hengst J.a., Tranel D, & Cohen NJ (2011). The use of definite references signals declarative memory. Psychological Science, 22(5), 666–673. 10.1177/0956797611404897. [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Duff MC, Hengst J, Tranel D, & Cohen NJ (2006). Development of shared information in communication despite hippocampal amnesia. Nature Neuroscience, 9(1), 140–146. 10.1038/nn1601. [DOI] [PubMed] [Google Scholar]
  11. Duff MC, Wszalek T, Tranel D, & Cohen NJ (2008). Successful life outcome and management of real-world memory demands despite profound anterograde amnesia. Journal of Clinical and Experimental Neuropsychology, 30(8), 931–945. [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Eichenbaum H, & Cohen NJ (2004). From consolidation to conscious recollection:Memory systems of the brain. Retrieved from Oxford: Oxford University Press; http://doi.apa.org/psycinfo/2001-06744-000%5Cnhttp://psycnet.apa.org/index.cfm?fa=search.displayRecord&uid=2001-06744-000. [Google Scholar]
  13. Gerwing J, & Bavelas J (2005). Linguistic influences on gesture’s form. Gesture, 4(2),157–195. 10.1075/gest.4.2.04ger. [DOI] [Google Scholar]
  14. Goodglass H (2000). Boston diagnostic aphasia examination: Short form record booklet. Lippincott Williams & Wilkins. [Google Scholar]
  15. Hengst JA (2003). Collaborative referencing between individuals with aphasia and routine communication partners. Journal of Speech, Language, and Hearing Research, 46(August), 831–848. [DOI] [PubMed] [Google Scholar]
  16. Hilliard C, & Cook SW (2015). Bridging gaps in common ground: Speakers design their gestures for their listeners. Journal of Experimental Psychology: Learning, Memory, and Cognition, 42(1), 91–103. 10.1037/xlm0000154. [DOI] [PubMed] [Google Scholar]
  17. Hilliard C, & Cook SW (2017). A technique for continuous measurement of body movement from video. Behavior Research Methods, 49, 1–12. 10.3758/s13428-015-0685-x. [DOI] [PubMed] [Google Scholar]
  18. Hilverman C, Clough S, Duff MC, & Cook SW (2018). Patients with hippocampal amnesia successfully integrate gesture and speech. Neuropsychologia. [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Hilverman C, Cook SW, & Duff MC (2016). Hippocampal declarative memory supports gesture production: Evidence from amnesia. Cortex, 85, 25–36. [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Hilverman C, Cook SW, & Duff MC (2018). Hand gestures support word learning in patients with hippocampal amnesia. Hippocampus, 28(6), 406–415. [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. Holler J, & Bavelas J (2017). Chapter 10. Multi-modal communication of common ground. Why gesture? How the hands function in speaking, thinking, and communicating (pp. 213–240). 10.1075/gs.7.11hol. [DOI] [Google Scholar]
  22. Horton W (2007). The influence of partner-specific memory associations on language production: Evidence from picture naming. Language and Cognitive Processes, 22(7), 1114–1139. 10.1080/01690960701402933. [DOI] [PMC free article] [PubMed] [Google Scholar]
  23. Horton WS, & Gerrig RJ (2005a). The impact of memory demands on audience design during language production. Cognition, 96(2), 127–142. 10.1016/j.cognition.2004.07.001. [DOI] [PubMed] [Google Scholar]
  24. Horton WS, & Gerrig R (2005b). Conversation common ground and memory processes in language production. Discourse Processes, 40(1), 57–82. 10.1207/s15326950dp4001. [DOI] [Google Scholar]
  25. Horton WS, & Gerrig J (2016). Revisiting the Memory-Based Processing Approach to Common Ground (pp. 1–16). 10.1111/tops.12216. [DOI] [PubMed]
  26. Ianì F, & Bucciarelli M (2017). Mechanisms underlying the beneficial effect of a speaker’s gestures on the listener. Journal of Memory and Language, 96(96), 110–121. 10.1016/j.jml.2017.05.004. [DOI] [Google Scholar]
  27. Klooster NB, Cook SW, Uc EY, & Duff MC (2015). Gestures make memories, but what kind? Patients with impaired procedural memory display disruptions in gesture production and comprehension. Frontiers in Human Neuroscience, 8(January), 1–13. 10.3389/fnhum.2014.01054. [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Lausberg H, & Sloetjes H (2009). Coding gestural behavior with the NEUROGES-ELAN system. Behavior Research Methods, 41(3), 841–849. 10.3758/BRM.41.3.841. [DOI] [PubMed] [Google Scholar]
  29. Meyers JE, & Meyers KR (1995). Rey complex figure test under four different administration procedures. The Clinical Neuropsychologist, 9(1), 63–67. [Google Scholar]
  30. Reber PJ, Knowlton BJ, & Squire LR (1996). Dissociable properties of memory systems: Differences in the flexibility of declarative and nondeclarative knowledge. Behavioral Neuroscience, 110(5), 861–871. 10.1037/0735-7044.110.5.861. [DOI] [PubMed] [Google Scholar]
  31. Schmidt M (1996). Rey auditory verbal learning test: A handbook. Los Angeles: Western Psychological Services. [Google Scholar]
  32. Tyrone ME, & Mauk CE (2011). Sign lowering and phonetic reduction in American Sign Language. NIH Public Access, 38(2), 317–328. 10.1016/j.wocn.2010.02.003.Sign. [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. Wechsler D (1997a). Wechsler adult intelligence scale-III (WAIS-III).
  34. Wechsler D (1997b). Wechsler memory scale (WMS-III), Vol. 14 San Antonio, TX:Psychological Corporation. [Google Scholar]
  35. Wilkes-Gibbes D, & Clark HH (1992). Coordinating beliefs in conversation. Journal of Memory and Language, 31, 183–194. [Google Scholar]
  36. Yoon S, Duff MC, & Brown-Schmidt S (2017). Learning and using knowledge about what other people do and don’t know despite amnesia. Cortex, 94, 164–175. 10.1016/j.cortex.2017.06.020. [DOI] [PMC free article] [PubMed] [Google Scholar]

RESOURCES