Hippocampal contributions to language: Evidence of referential processing deficits in amnesia

Abstract

A growing body of work suggests the hippocampus contributes to a variety of cognitive domains beyond its traditional role in memory. We propose that the hippocampus, in its capacity for relational binding, representational flexibility, and on-line maintenance and integration of multimodal relational representations, is a key contributor to language processing. Here we test the hypothesis that the on-line interpretation of pronouns is hippocampus-dependent. We combined eye-tracking with neuropsychological methods, where participants (4 patients with bilateral hippocampal damage and severe declarative memory impairment, 4 patients with ventromedial prefrontal cortex (vmPFC) damage, and healthy comparison participants) viewed a scene while listening to short dialogs introducing two characters; e.g., “Melissa” is playing violin for “Debbie”/”Danny” as the sun is shining overhead. She is wearing a blue/purple dress. Consistent with previous work, analysis of eye-gaze showed that younger and older healthy comparison participants and the vmPFC patients rapidly identified the intended referent of the pronoun when gender uniquely identified the referent, and when it did not, they showed a preference to interpret the pronoun as referring to the first-mentioned character. By contrast, hippocampal patients, while exhibiting a similar gender effect, exhibited significant disruptions in their ability to use information about which character had been mentioned first to interpret the pronoun. This finding suggests that the hippocampus plays a role in maintaining and integrating information even over a very short discourse history. These observed disruptions in referential processing demonstrate how promiscuously the hallmark processing features of the hippocampus are used in service of a variety of cognitive domains including language.

Hippocampal contributions to language: Evidence of referential processing deficits in amnesia

The role of the hippocampus (and related medial temporal lobe structures) in the formation of new long-term memories and their subsequent retrieval is well established (Eichenbaum & Cohen, 2001; Gabrieli, 1998; Ranganath, 2010; Squire, 1992). Critical to the role the hippocampus plays in supporting declarative memory use are its hallmark processing features of relational binding and representational flexibility. The hippocampus supports the creation and integration of event representations including information about the co-occurrences of people, places, and things, and the ability to link spatial, temporal and interactional relations across time (Eichenbaum & Cohen, 2001). These relational representations are uniquely flexible, permitting rapid integration across representations and processing systems, and facilitating use of old representations in novel contexts (Eichenbaum & Cohen, 2001; O’Keefe & Nadel, 1978; Squire, 1992). Recent evidence suggests the hippocampus also plays a role in on-line processing; patients with hippocampal amnesia show deficits across minimal delays and when all the necessary information is immediately available (e.g., Barense, Gaffan, & Graham, 2007; Hannula, Tranel & Cohen, 2006; Warren, Duff, Tranel, & Cohen, 2011). These results converge with fMRI findings of hippocampal activation for relational learning over short delays (Hannula & Ranganath, 2008).

A growing body of work suggests the hippocampus contributes to a variety of cognitive domains beyond its traditional role in memory. Duff and Brown-Schmidt (2012) have proposed that the hippocampus is a key contributor to language use. The proposal argues that the same hippocampal processes used in support of memory (i.e., relational binding, representational flexibility, and on-line use of multimodal relations) support rapid access and integration of contextual and experiential information that the language processing system relies on to create meaning in the moment. Evidence for this proposal comes, in part, from findings that patients with hippocampal amnesia have difficulty establishing, recovering, maintaining and using relational memory representations during conversation (e.g., Duff et al; 2008; 2011). What this work has not shown, however, is the contribution of hippocampus to on-line, or real-time language-processing.

The goal of the present research is to investigate hippocampal contributions to on-line referential processing. Establishing and maintaining reference is a central component of language processing, as most of what we talk about involves referring to entities. To understand and use reference requires the ability to maintain a representation of the unfolding discourse history and the ability to integrate information about referential form with rich representations of the context. We propose that the hallmark relational processing capacity of the hippocampus makes critical contributions to on-line referential processing and that patients with hippocampal damage will show deficits, even over short discourse histories.

To examine this prediction, we monitored the eye movements of patients with hippocampal damage, a brain damaged comparison group (who have damage outside MTL), and demographically matched healthy comparison participants as they listened to a two-sentence discourse while viewing a corresponding scene. We asked whether amnesic patients can use immediately-available information in the scene, along with discourse information from the immediately preceding sentence to resolve a potentially ambiguous referring expression. We focus on interpretation of pronouns such as he and she, which require the listener to use features of the pronoun (gender, animacy, etc.) and representations of potential discourse referents to identify the speaker’s intended referent. While this process requires complex, multidimensional calculations, evidence from multiple methodologies indicates that resolution of referential ambiguity usually begins within 200–400ms of pronoun onset (Arnold, Eisenband, Brown-Schmidt, & Trueswell, 2000; Van Berkum, Koornneef, Otten, & Nieuwland, 2007; Kaiser, Runner, Sussman, & Tanenhaus, 2009).

This line of research also shows that different types of information, with potentially different memory demands, guide on-line processing in healthy young adults. For example, Arnold, et al., (2000, Exp. 1) presented participants with brief stories in which two characters were introduced, and then one was subsequently referred to with a pronoun:

1. Donald is bringing some mail to Mickey / Minnie.

   while a violent storm is beginning.

   He’s / She’s carrying an umbrella,

   and it looks like they’re both going to need it.

In the few hundred milliseconds immediately following the pronoun (underlined), Arnold, et al. (2000) examined the eye movements that listeners made to a scene that featured the two characters. When the characters were of different gender, listeners quickly fixated the intended referent of the pronoun, regardless of whether that referent had been mentioned first (e.g., Donald) or second (e.g., Minnie) in the story. By contrast, when the characters were of the same gender, listeners initially interpreted the pronoun as referring to the first-mentioned character, as personal pronouns typically refer to the more salient potential referent (Gundel, Hedberg, & Zacharski, 1993; cf. Kaiser & Trueswell, 2008).

By tracking amnesic participants’ gaze as they process language in real time, we gain novel insights into if and how hippocampus-dependent representations contribute to on-line processing. If the hippocampus underlies on-line referential processing, patients with hippocampal amnesia will be impaired in interpreting ambiguous references and this may attenuate preferential viewing of target referents. Such an outcome would expand our understanding of the dynamic network of neural substrates and cognitive processes that support on-line language processing and the resolution of ambiguity in everyday language use

Methods

Participants

Four amnesic patients (1 female) with bilateral hippocampal damage and severe memory impairment participated (Table 1; Supplemental materials). Etiologies included anoxia/hypoxia (1846, 2363) and herpes simplex encephalitis (HSE) (1951, 2308). Coronal sections from magnetic resonance imaging, through the midsection of hippocampus, are shown in Figure 1. A brain-damaged comparison (BDC) group (n=4; 1 male), with damage outside hippocampus and MTL also participated. All BDC participants had frontal lobe damage with the greatest area of overlap in the ventromedial prefrontal cortex. The BDC group was significantly older than amnesic patients (t = 4.26) and performance on neuropsychological testing was within normal limits. BDC performance on neuropsychological tests of memory was in normal limits and significantly better than amnesic participants.

Table 1.

Demographic and neuropsychological characteristics of the amnesic and brain-damaged comparison participants

		Demographic				Anatomical			Neuropsychological
								Intelligence	Memory	Perception		Language		Executive Function
Patient	Group	Sex	Hand	Age	Ed.	ET	HC Vol	WAIS-III FSIQ	WMS-III GMI	Faces	JoL	TT	BNT	WCST PE	WCST Cat
1846	AM	F	R	45	14	anoxia	−4.23	84	57	45	22	41	43	6	6
1951	AM	M	R	56	16	HSE	−8.10	106	57	44	30	44	49	16	6
2308	AM	M	L	52	16	HSE	N/A	98	45	50	30	44	52	N/A	N/A
2363	AM	M	R	52	18	anoxia	−2.64	98	73	47	26	44	58	12	6
Amnesic Summary		3 M 1 F	3 R 1 L	51.3 ± 4.6	16.0 ± 1.6		−5.0 ± 2.8	96.5 ± 9.2	58.0 ± 11.5	46.5 ± 2.7	27.0 ± 3.8	43.3 ± 1.5	50.5 ± 6.2	11.3 ± 5.0	6.0 ± 0.0
318	BDC	M	R	70	14	MR	N/A	143	109	43	30	44	60	4	6
2025	BDC	F	R	62	16	ACoA	N/A	115	114	43	28	44	59	4	6
2352	BDC	F	R	61	14	SaH: ACoA	N/A	106	109	43	27	44	54	9	6
2391	BDC	F	R	64	13	MR	N/A	109	132	49	31	43	57	7	6
BDC Summary		1 M 3 F	4 R 0 L	64.3 ± 4.0	14.3 ± 1.3		N/A	118.3 ± 16.92	116.0 ± 10.92	44.5 ± 3.0	29.0 ± 1.8	43.8 ± 0.5	57.5 ± 2.7	6.0 ± 2.5	6.0 ± 0.0
Difference (p value)				0.005	0.14			0.08	0.0003	0.36	0.40	0.56	0.1	0.2	1.0

Note. AM = Amnesic patients; BDC = brain-damaged comparison patients; M = Male; F = Female; ET = Etiology; HSE = Herpes simplex encephalitis; SaH = Subarachnoid hemorrhage; ACoA = Anterior communicating artery aneurysm; MR = meningioma resection; HC Vol = studentized residual differences in hippocampal volume relative to amatched comparison group (see Allen et al., 2006; Buchanan et al., 2005); WAIS-III = Wechsler Adult Intelligence Scale; FSIQ = Full Scale Intelligence Quotient, WMS-III GMI = Wechsler Memory Scale-III; GMI = General Memory Index; Faces = Benton Facial Recognition Test; JoL = Judgment of Line Orientation Test; TT = Token Test; BNT = Boston Naming Test; WCST = Wisconsin Card Sorting Task; PE = Perseverative errors; Cat = Number of categories achieved out of six.

Figure 1.

Magnetic resonance scans of hippocampal patients. Images are coronal slices through the midportion of the hippocampus from T1-weighed scans. Volume changes can be noted in the region of the hippocampus bilaterally.

Healthy comparison participants (CP) (n=15), matched pairwise to amnesic and BDC participants on sex, age, handedness, and education, with no history of neurological or psychiatric disease participated. To verify that we could replicate previous findings (i.e., Arnold et al., 2000) using our materials, we also recruited healthy undergraduates (N=12).

Materials

Items consisted of scenes (similar to Arnold et al., 2000) and narratives; the participants’ task was to decide if the scene and narrative matched. Scenes contained known Disney characters (referred to for publication as “Melissa” and “Manny”, “Debbie” and “Danny”; Table 2). For each of 32 target trials, the narrative and scene always matched. Four variants of each target item manipulated (a) gender (same versus different) and (b) order-of-mention (1^st versus 2^nd). These variables were manipulated within-subject and within-item, creating 128 critical trials, plus 32 filler items for a total of 160 total trials per participant.

Table 2.

Narrative Design

		Order of Mention
		First	Second
Gender	Same	S1: “Melissa” is playing the violin for “Debbie” as the sun is shining overhead. She is wearing a yellow bracelet and it looks like the song is being played well.	S2: “Melissa” is playing the violin for “Debbie” as the sun is shining overhead. She is wearing a yellow bracelet and it looks like the song is being played well.
	Different	D1: “Melissa” is playing the violin for “Manny” as the sun is shining overhead. She is wearing a yellow bracelet and it looks like the song is being played well.	D2: “Melissa” is playing the violin for “Manny” as the sun is shining overhead. He is wearing a yellow bracelet and it looks like the song is being played well.

In target narratives (Table 2), the first clause introduced two characters (e.g. “Melissa” is playing violin for “Debbie”/”Danny”), establishing their relative salience (i.e., “Melissa” is more salient because she was mentioned first and in subject position). The second clause mentioned another object (e.g. as the sun is shining overhead.), and was designed to shift gaze away from the characters. The third clause began with the critical pronoun (e.g. She/He is wearing a yellow bracelet), and maintained the potential for ambiguity for at least 4 words, at which point the sentence uniquely identified which character was the intended referent (e.g., only the target referent wore a yellow bracelet; see Table 2). The average time between the onset of the pronoun and the disambiguating word was 690ms (SD = 23ms). Practice trials introduced the characters and allowed participants to practice the task. Fillers had a similar structure, but did not contain ambiguous pronouns. Twenty-four of the fillers did not match the picture, requiring participants to indicate a lack of match.

Procedure

On each trial, the picture appeared and the narrative began 3 seconds later; gaze was recorded throughout the trial. Following the narrative, participants indicated whether the narrative matched the picture by pressing `yes' or `no'. Following 3 practice trials, each participant completed 128 critical trials and 32 fillers, presented in a set random order. In order to have enough data for statistical analysis, three of the amnesic patients and all of the BDC participants completed all trials twice¹. Due to scheduling constraints, the fourth amnesia participant (1951) completed the task once. All healthy comparison participants (including undergraduates) completed one session of all trials.

Analysis

The primary measure was the eye-fixations participants made following the critical pronoun. For each trial, we calculated the proportion of fixations to the target and competitor referents between 200ms to 1000ms following pronoun onset. The time-window is offset by 200ms to account for the time needed to program and launch an eye movement (Hallett, 1986).

The dependent measure was the log of the ratio of target to competitor fixations. Positive values indicate a target preference and negative values indicate a competitor preference. All data were analyzed with mixed-effects models. Effect-size estimates for critical comparisons (Cohen’s d) are based on by-participant condition means. See Supplementary Materials for analysis details, supplementary analyses of fixations, and analysis of offline response data.

Results

Replication Check with Healthy Undergraduates

To verify that our version of the task was consistent with previous work, we first replicated Arnold and colleagues’ (2000) study with 12 healthy undergraduates (see Supplementary Results). Each undergraduate participated in one session consisting of 163 trials; the results were consistent with Arnold, et al.’s (2000) findings.

Hippocampal Amnesia Disrupts On-line Referential Processing

To test the hypothesis that referential processing requires hippocampal contributions, our primary analysis directly compared the performance of amnesic patients and demographically matched comparison participants. To address hippocampal specificity, we examined the performance of four brain-damaged comparison (BDC) participants who have damage outside of MTL. Because the BDCs were significantly older than the amnesic patients, we analyzed the BDCs performance against their own age matched comparison participants (n=7).

Eye-movement Data

We test for group differences in the use of discourse context by analyzing gaze during interpretation of the potentially ambiguous pronoun. We directly compared eye-fixations for the amnesic patients and their comparisons in one analysis, and the BDC participants to their comparisons in a second analysis. These analyses included gender, order of mention and participant group (patient vs. comparison) as orthogonal factors, as well as time-window, with the baseline window coded as reference (see supplemental results for complete results tables).

Our first analysis found that amnesic patients did not perform like their comparisons (Supplementary Table 2). Critically, this analysis revealed a significant gender * mention * participant group interaction (t=2.80, p<0.05; see Figure 3, Figure 4), demonstrating that the amnesic patients and comparison participants were differentially sensitive to these cues. We now turn to examine the interpretation process for the amnesic patients and comparison participants separately, to understand the locus of their processing differences.

Figure 3.

Time-course of fixation preferences plotted as the difference between target and competitor fixations (proportion target minus proportion competitor), separately by condition and group. Positive values indicate target preference. 0ms= pronoun onset. Panel A = undergraduate participants; Panel B = healthy comparison participants matched to amnesia patients; Panel C = amnesic patients. Panel D = brain-damaged comparison participants; D1 = Different Gender, First Mention; D2 = Different Gender, Second Mention; S1 = Same Gender, First Mention; S2 = Same Gender, Second Mention. Vertical lines denote analysis time-windows.

Figure 4.

Log ratio of target to competitor fixations for each group during the critical pronoun time-window (200ms to 1000ms following pronoun onset). This is the dependent measure in the statistical analyses; positive values indicate a target preference (see Supplementary Materials for plots at baseline and late time-windows). Error bars indicate one SD.

The healthy comparison participants matched to the amnesic patients performed similarly to undergraduates (Supplementary Table 3), with a larger target preference when the characters were of different gender (t=−7.14, p<0.0001; d=0.71), and when the target was the first-mentioned character (t=−3.74, p<0.01; d=0.28). In addition, a gender*order interaction (t=−5.14, p<0.001) was due to a clear order effect (first-mentioned preference) when the characters were of the same gender (order t=−4.81, p<0.001; d=0.53), and a large target preference, regardless of order, when the characters were of different genders (order t=0.46, p=0.63; d=0.04). Thus like undergraduates, these healthy older participants rapidly integrated information about the discourse context and character gender to identify the intended referent.

In striking contrast to all other participants groups (healthy participants and the BDC group; see below), during interpretation of the pronoun, amnesic patients showed neither an effect of order (t=−0.48, p=0.67; d=0.04), nor a gender*order interaction (t=−0.78, p=0.45). Instead, there was only an effect of gender (t=−4.75, p<0.01; d=0.52), due to a larger target preference when the characters were of different genders than when they were of the same gender (Supplementary Table 4).

Closer inspection of the data reveals performance variability in both the amnesia patients and their matched comparisons (Figure 5). These individual data demonstrate that the disruption in referential processing in individual amnesic patients ranges from marked attenuation of the effect (1846, 1951) to significant impairment (2308, 2363). Note that the participant with the worse performance in the same-gender condition is an anoxic patient, suggesting that patients with more extensive medial temporal lobe damage (1951, 2308) are not driving the effect.

Figure 5.

Individual data for each amnesic participant (the bar graphs) with the their demographically matched comparison data superimposed on top (the black squares).

Our final analysis compared BDCs to their healthy comparison participants and found that both groups performed similarly to undergraduates, with a larger target preference when the characters were of different gender (t=−6.36, p<.0001; d=0.36), and when the target was the first-mentioned character (t=−3.19, p<0.01; d=0.04). In addition, a gender*order interaction (t=−4.86, p<0.01) was due to a clear order effect (first-mentioned preference) when the characters were of the same gender (order t=−4.16, p<0.0001; d=0.02), and a large target preference, regardless of order, when the characters were of different genders (order t=−0.58, p=0.56; d=0.08). There were no effects or interactions with participant group (ts<1.0, ps>0.3) (Supplementary Table 5). Thus, these BDCs and their healthy older comparison participants rapidly integrated information about the discourse context to identify the intended referent.

Discussion

Our findings demonstrate that amnesic patients experienced difficulty in integrating and maintaining information even over a very short discourse history. Amnesic patients were significantly impaired in their ability to use information about the relative salience of two very recently mentioned discourse referents to disambiguate a pronoun. By contrast, young adults, BDC participants, and healthy older adult participants recruited this information, and used it to begin guiding the on-line interpretation of the pronoun within the first second of pronoun onset. That amnesic patients performed significantly worse than the BDCs suggests a strong link between the functionality of the hippocampus and demands of referential processing. The observed disruption in referential comprehension in amnesia, but not in frontal lobe patients, is consistent with previous work on language production (e.g., Kurczek & Duff, 2011, 2012; also see MacKay, Johnson, & Hadley, 2013).

These deficits are most likely the inability of amnesic patients to bind order-of-mention information to the first- and second-mentioned characters to disambiguate the pronoun. This interpretation is consistent with role of the hippocampus in binding more generally—and in the present case, binding order-of-mention information in the previous discourse with the appropriate discourse referent and the representation of that referent in the visual scene—and then maintaining this information to support resolution of a potentially ambiguous referring expression in the subsequent sentence. This interpretation fits with work on hippocampal involvement in the binding and memory for the temporal order of events (Heuer & Bachevalier, 2013; Jenkins & Ranganath, 2010; Tubridy & Davachi, 2011) and in the maintenance of information over very short timescales (e.g., Hannula & Ranganath, 2008; Warren et al., 2010) outside the language domain. Our findings suggest that the hippocampus contributes to language processing including use of all but the most recent discourse information and integrating information across the discourse (see Oztekin, Davachi, & McElree, 2010; McElree, 2006; Lewis & Vasishth, 2005).

Importantly, however, when the gender of the intended referent disambiguated the gender-marked pronoun, amnesic patients successfully identified the intended referent, with no significant difference in the time-course of this process compared to healthy participants. This result suggests that amnesic patients can integrate language with scene information so long as all the key information is readily available in a co-present scene, and is marked in the immediate linguistic input (also see Rubin, Brown-Schmidt, Duff, Tranel, & Cohen, 2011; Trude, Duff, & Brown-Schmidt, 2012). These islands of success, along with normal performance on standard measures of language, likely contributed to the traditional view that language comprehension is intact in amnesia (e.g., Milner, Corkin, & Teuber, 1968; although see MacKay et al., 1998). The full set of results here, however, suggest serious deficits in the ability to follow the thread of a discourse when doing so relies on representations of the salience of discourse referents, with catastrophic impairments in comprehension with the passage of time or intervening items.

In addition to expanding our understanding of the hippocampal declarative memory system, our findings also contribute to theories of language use and processing. While there is considerable agreement as to the importance of referential form in discourse coherence, there remain long-standing, unresolved theoretical debates as to the specific factors and mechanisms that facilitate reference comprehension (e.g., Beaver, Wolters, & Zeevat, 2003; Chambers & Smyth, 1998; Gordon, Grosz & Gilliom, 1993). Much of this work has focused on attention, working memory and/or executive control processes, functions putatively associated with prefrontal cortex mechanisms, (e.g., Gibson, 1998; Green et al., 1994; Novick et al., 2005; Walker, 1996; cf. MacDonald & Christiansen, 2002). Consistent with our proposal regarding hippocampal contributions to language (Duff & Brown-Schmidt, 2012), the work here supports the notion that the cognitive and neural basis of referential processing extends beyond the frontal lobes to include the hippocampal declarative memory system. In this way, this study serves as an important step in expanding the language network to include the hippocampus and in defining its contributions to language. Future work examining which other aspects of referential processing also depend critically on the hippocampus for effective operation, and the nature and time-course of the interactions between hippocampus and other systems known to support reference use and understanding is warranted.

In summary, the present research tracked participants’ eye movements as they processed language in real time. We demonstrated that hippocampal damage affects communication and language use not just off-line or over long stretches of discourse but in the moment and during the incremental understanding of talk. These findings are striking given the traditional view of hippocampus contributing exclusively to long-term memory and of referential processing as relying on the frontal lobes and its putative functions (e.g., attention, working memory). Linking deficits in language processing to the hippocampus demonstrates how promiscuously the hallmark processing features of the hippocampus are used in service of a variety of cognitive domains including language.

Figure 2.

Lesion overlap of the four brain damaged comparison (BDC) participants. The colorbar indicates the number of lesion overlaps (range = 0–4). The vertical line through the left and right mesial views is through the midportion of the hippocampus indicating that no BDC had MTL lesions.