Structural and semantic constraints on the resolution of pronouns and reflexives

Elsi Kaiser; Jeffrey T Runner; Rachel S Sussman; Michael K Tanenhaus

doi:10.1016/j.cognition.2009.03.010

. Author manuscript; available in PMC: 2010 Jul 1.

Published in final edited form as: Cognition. 2009 May 7;112(1):55–80. doi: 10.1016/j.cognition.2009.03.010

Structural and semantic constraints on the resolution of pronouns and reflexives

Elsi Kaiser ¹, Jeffrey T Runner ², Rachel S Sussman ³, Michael K Tanenhaus ⁴

PMCID: PMC2763941 NIHMSID: NIHMS106595 PMID: 19426968

Abstract

We present four experiments on the interpretation of pronouns and reflexives in picture noun phrases with and without possessors (e.g. Andrew’s picture of him/himself, the picture of him/himself). The experiments (two off-line studies and two visual-world eye-tracking experiments) investigate how syntactic and semantic factors guide the interpretation of pronouns and reflexives and how different kinds of information are integrated during real-time reference resolution. The results show that the interpretation of pronouns and reflexives in picture NP constructions is sensitive not only to purely structural information, as is commonly assumed in syntactically-oriented theories of anaphor resolution, but also to semantic information (see Kuno, 1987; Tenny, 2003). Moreover, the results show that pronouns and reflexives differ in the degree of sensitivity they exhibit to different kinds of information. This finding indicates that the form-specific multiple-constraints approach (see Kaiser, 2003; Kaiser, 2005; Kaiser & Trueswell, 2008; Brown-Schmidt, Byron & Tanenhaus, 2005), which states that referential forms can exhibit asymmetrical sensitivities to the different constraints guiding reference resolution, also applies in the within-sentence domain.

Introduction

Successful language comprehension requires a reader or listener to rapidly interpret referential forms such as it, him and herself. Although these forms are extremely frequent, they are referentially underspecified. A form such as her can, in principle, refer to any human feminine singular referent—in other words, to any member of a very large set. As a consequence, successful comprehension of these forms requires a narrowing of the domain of potential referents. This raises the question of what constrains the referential domains for different referring forms: how do comprehenders know which entities to exclude, and which to treat as possible referents?

For some referential forms, it has been argued that structural/syntactic information determines the choice of antecedent. For example, it is commonly assumed that the choice of referent for a reflexive like herself is specified by the structural configuration of the sentence (e.g. Chomsky, 1981 and many others). In contrast, for many other cases of reference resolution (especially when the referential form and its referent are in different clauses) non-structural information appears to play a crucial role. For example, many researchers have argued that semantic information influences the interpretation of pronouns with cross-clausal antecedents (e.g., see Garvey and Caramazza, 1974; McDonald & MacWhinney, 1995, Stewart et al., 2000; Koornneef & Van Berkum, 2006; Stevenson, Crawley & Kleinman, 1994; Arnold, 2001, for research on effects of implicit causality and thematic roles). Stevenson and colleagues (Stevenson et al., 1994) found that the thematic role of an entity (e.g., agent, goal, source, experiencer) influences its likelihood of being the antecedent for a subsequent pronoun (see also Arnold, 2001). In sum, existing work suggests that referential expressions tend to fall into two groups: Some are interpreted in accordance with structural principles, whereas others rely more on non-structural factors such as semantic role as well as other discourse factors (e.g., Kehler, 2002; Wolf et al., 2004).

Even though many occurrences of referentially underspecified forms fall into one of these two groups, we argue that regarding structural information and discourse/semantic information as separate aspects of reference resolution is an oversimplification. We build on and extend what we call the form-specific multiple-constraints framework, first introduced in Kaiser (2003; see also Kaiser, 2005; Kaiser & Trueswell, 2008; Brown-Schmidt, Byron & Tanenhaus, 2005). Like existing multiple-constraints approaches to reference resolution (e.g., Arnold, 1998; Ariel, 1990), the form-specific approach assumes that anaphor resolution is not determined by a single constraint but rather is the result of the interaction of multiple constraints. Crucially, the form-specific approach allows for the multiple constraints that guide reference resolution to be weighted differently for different referential forms. The approach was originally formulated on the basis of data from Finnish showing that pronouns and demonstratives, both of which can be used to refer to human antecedents mentioned in a preceding clause, do not show the same level of sensitivity to the antecedent’s syntactic role and linear position (Kaiser, 2003; Kaiser & Trueswell, 2008). The asymmetrical sensitivities indicate that not all anaphoric forms are equally sensitive to the same kinds of information. This finding is also supported by cross-clausal data from Estonian (Kaiser & Hiietam, 2004; Kaiser, 2003; Kaiser & Vihman, 2006), Dutch (Kaiser & Trueswell, 2004) and English (Brown-Schmidt et al., 2005).

Most of the previous studies that found different anaphoric forms exhibiting asymmetrical sensitivities focused on cross-clausal anaphora, for which it is not surprising that non-structural (discourse/semantic) factors are relevant. One of our main aims here is to investigate whether the form-specific multiple-constraints approach also applies in the domain of within-clause reference resolution. In the clause-internal domain, many anaphor-antecedent dependencies are in structural configurations that are considered to be constrained by structural factors (e.g. the large body of work on Chomskyan Binding Theory, though see Gordon & Hendrick 1997), and the role of non-structural factors has received less attention than in the cross-clausal domain. Moreover, although existing work within the form-specific framework indicates that anaphoric forms differ in their sensitivity to different kinds of information, it does not make specific predictions regarding the range of variation that will occur for different anaphoric forms within and across languages. The ultimate goal of the form-specific approach is to develop detailed testable hypotheses about the conditions of use and weightings of constraints for each anaphoric form. This will require more research on different anaphoric forms in different languages. This paper aims to contribute to that goal.

We present four experiments investigating the processing of pronouns and reflexives in picture noun phrases (PNPs) with and without possessors (e.g. Andrew’s picture of him/himself, the picture of him/himself). The picture NP construction is widely acknowledged to represent a stumbling block for existing structure-driven approaches of within-clause reference resolution. Since this construction has been hypothesized to be influenced by both syntactic factors and discourse/semantic factors (see for example Kuno, 1987; Pollard & Sag, 1992; Reinhart & Reuland, 1993; Tenny, 1996; Tenny, 2003), it is a useful tool for investigating how structural and non-structural information interact during within-sentence anaphor resolution. When discussing the effects of non-structural factors, we will refer to them as ‘semantic factors’ for ease of exposition. However, we leave open the question of whether the source/perceiver manipulation (described below) is best regarded as a semantic, thematic role manipulation or a discourse-level/pragmatic manipulation (e.g., having to do with perspective-taking).

We demonstrate that in picture NPs, pronouns and reflexives differ in the degree of sensitivity they exhibit to structural and non-structural information, and that these differences emerge very early during processing. Our findings are compatible with the form-specific multiple-constraints framework, but not with multiple-constraints approaches in which the relative weights of structural and semantic constraints are the same for reflexives as for pronouns, nor with approaches that propose an initial structure-only stage of processing.

Background

The picture NP (PNP) construction has long been acknowledged to be problematic for standard structural accounts of within-clause reference resolution, such as classic Chomskyan Binding Theory (e.g., Chomsky, 1981) because picture NPs do not exhibit the complementary distribution of pronouns and reflexives that is the cornerstone of most of these approaches. As a result, some researchers have suggested that reflexives and pronouns in PNPs are exempt from structural principles, such as Binding Theory, and are resolved in accordance with semantic and pragmatic constraints (e.g. Pollard & Sag, 1992; see also Reinhart & Reuland, 1993; Safir, 1999 for related claims).

Despite the acknowledged shortcomings of structural accounts in general and Binding Theory in particular, we begin this section with a brief overview of the predictions that Binding Theory makes regarding pronouns and reflexives in PNPs. This is partly for historical reasons: Structurally-based accounts of reference resolution have had an enormous impact on syntactic research and there exists a large body of work on PNP constructions within Binding Theory that ought to be acknowledged. Furthermore, as will become clear later, even though structural constraints may not be sufficient to explain the referential properties of pronouns and reflexives in PNPs, our results indicate that they are nevertheless necessary.

To understand the predictions that Binding Theory makes for PNPs, we first consider sentences without PNPs. In most contexts, pronouns are in complementary distribution with reflexives. As illustrated in (1), the pronoun her cannot refer to Julianna, the subject of the clause containing the pronoun, whereas the reflexive herself must be interpreted as referring to the subject. (Subscripted indices are used to mark coreference, with a star * indicating that coreference with the indexed referent is not acceptable.)

(1) Julie_i said that Julianna_j tickled her_i/*j/herself_*i/j.

Principles A and B of traditional Binding Theory provide a structural account of this complementarity. According to Principle A, a reflexive must be bound in a local domain. In other words, the noun phrase that a reflexive refers to (is “bound by”) – its antecedent – must be sufficiently close to the reflexive itself (in this case, in the same clause) and must stand in a particular relation of structural superiority to the reflexive. Thus, in (1), the reflexive herself can only refer to the subject of the same clause, namely Julianna. Principle B, conversely, states that a pronoun must be free in a local domain, i.e., its antecedent cannot be in a position of structural superiority in the local domain. Thus, in (1), her cannot refer to the local subject, Julianna, but can be coreferential with Julie, as Julie is not in the local domain.

For PNPs in sentences like (2), the Binding Theory principles presented in Chomsky (1981) predict similar complementary patterns. The indices in (2) show the coreference relations that classic Binding Theory predicts to be possible, as well as those which are predicted to be impossible (marked with *).

(2a) Peter_i saw the picture of himself_i/him_*i. [possessorless PNP]
(2b) Mary_i saw Lisa_j’s picture of her_i/*j/herself_*i/j. [possessed PNP]

Recall that according to Principle A, reflexives need to be bound by an antecedent in the local domain. According to Binding Theory, the relevant local domain in a possessorless PNP, as in (2a), is the entire clause. As a result, the reflexive himself in the PNP construction must be bound by (i.e., coreferential with) the subject of the clause (Peter). For pronouns, Principle B states that local antecedents are not allowed, and thus the pronoun him in (2a) cannot refer to the subject of the clause. (Instead, him refers to some third person not mentioned in the local domain.)

In the possessed PNP in (2b), the presence of a possessor limits the local domain to the picture NP. As a result, the reflexive herself (which needs an antecedent in the local domain, according to Principle A) must refer to the possessor Lisa. The pronoun her (which cannot have an antecedent in the local domain, according to Principle B) cannot refer to the possessor, but can refer to the subject of the sentence (Mary) because the subject is outside the local domain.

In sum, Principles A and B of classic Binding Theory predict there to be no overlap in the antecedents of pronouns and reflexives in possessed and possessorless PNPs. However, it is now well-established that the pattern in (2a–b) does not fully reflect people’s interpretations, especially for possessorless PNPs. A number of researchers, including Jackendoff (1972), Chomsky (1986), Williams (1987), Reinhart and Reuland (1993), Keller and Asudeh (2001) and Tenny (2004, 2003), have observed that pronouns in possessorless PNPs can – at least in certain contexts – refer to the subject of the sentence, just like reflexives.

More recent work has tested whether reflexives and pronouns in possessed PNPs show a similar breakdown in complementarity. Keller and Asudeh (2001) and Runner, Sussman and Tanenhaus (2003) found, using magnitude estimation and visual-world eye-tracking respectively, that reflexives in possessed PNPs can refer to the subject of the sentence, contrary to the predictions of the structural approach (see also Jaeger, 2004). Runner et al. present eye movement data showing that it is not the case that these Binding-theory-incompatible interpretations arise later, after an initial stage of processing during which Binding theory is strictly followed. Rather, the eye movements show the same time course for structurally expected and unexpected interpretations of reflexives. These results, as well as more recent work by Runner et al. (2006) and self-paced reading data from Badecker & Straub (2002), argue against previous claims that there exists an early processing stage of purely Binding-Theory-compatible processing (e.g., Nicol & Swinney, 1989; Sturt, 2003, see also Clifton, Kennison & Albrecht, 1997).

In sum, a range of studies suggest that a structural account along the lines of standard Binding Theory is not sufficient to capture the referential properties of pronouns and reflexives. These kinds of findings led researchers to adopt one of two main approaches: (i) augment Binding Theory in order to maintain a primarily structural account of the distribution of pronouns and reflexives (e.g., Chomsky, 1986 and many others), or (ii) treat some anaphors (especially reflexives in possessorless PNPs) as being exempt from Binding Theory altogether (e.g. Pollard & Sag 1992) and guided by semantic and discourse constraints.

Semantic and discourse factors

Research regarding the effects of semantic and discourse constraints on the interpretation of pronouns and reflexives (e.g. Cantrall, 1974; Kuno, 1987; Zribi-Hertz, 1989; Pollard & Sag, 1992; Reinhart & Reuland, 1993, Tenny, 1996; Tenny, 2003) has focused primarily on possessorless PNPs, and has led to contrasting claims for reflexives and pronouns.

Reflexives in possessorless PNPs have been argued to be sensitive to factors such as the potential antecedents’ point of view, degree of awareness and semantic role (e.g., Pollard & Sag, 1992; Kuno, 1987). In this paper, we focus on the notion of ‘source of information’, which is based on Kuno’s observation that in a sentence like (3a) the reflexive can felicitously refer to Mary because she is the one who provides John with the information. This contrasts with example (3b), where Mary is not the source of information. We will examine the hypothesis in (4). (Our use of the term ‘source’ draws on Sells (1987)’s definition of source as the one who is the intentional agent of the communication.)

(3a) John heard from Mary about a damaging rumor about herself that was going around. (Kuno, 1987:175)
(3b) John told Mary about a damaging rumor about ^??herself that was going around.
(4) Source hypothesis:

Reflexives in PNPs prefer antecedents that are sources of information.

The source hypothesis for reflexives provides an interesting counterpart to claims that have been made regarding pronouns in possessorless PNPs. Tenny (2003, 2004) observes that pronouns, which she argues are sensitive to point of view, have a preference for antecedents that are perceivers of information. She notes that “verbs that provide a sentient, perceiving antecedent are especially conducive to SDPs” (Tenny, 2003:14). The abbreviation SDP refers to short-distance pronouns, i.e., pronouns that, contrary to the requirements of the standard Binding Theory, have antecedents in the local domain. The effect of perceiver status is illustrated by the examples in (5). Example (5a) (from Reinhart & Reuland, 1993), has a perceiving subject and sounds better than (5b), which has a subject that is not a perceiver (see also Chomsky, 1986:166–167, Bhatt & Pancheva, 2006). So, for pronouns, the hypothesis in (6) can be formulated.

(5a) Max_i heard the story about him_i. (Reinhart & Reuland, 1993:685)
(5b) *Max_i told the story about him_i. (Reinhart & Reuland, 1993:685)
(6) Perceiver hypothesis:

Pronouns in PNPs prefer antecedents that are perceivers of information.

The question of whether pronouns and reflexives in possessed PNPs are also influenced by non-structural information has not received as much attention in the literature. Jaeger (2004) uses evidence from a magnitude estimation experiment to claim that pronouns in possessed PNPs cannot refer to agentive subjects—more specifically, cannot refer to the creator of the picture. For example, a sentence such as Manray_i burned Mary’s photo of him_i is judged to be less acceptable than Mary_i burned Manray’s photo of her_i. Jaeger attributes this to the creator/agent status of Manray, who is a famous photographer. However, it is not yet known whether the claims made by Kuno and Tenny (see hypotheses (4) and (6) above) also apply to possessed PNPs.

In sum, in light of the observation that pronouns and reflexives in PNPs are not in complementary distribution, it seems that a purely structural approach is insufficient to capture their referential properties, especially in possessorless PNPs. Existing research suggests that semantic properties of the antecedent, such as source and perceiver, play a role in guiding the interpretation of pronouns and reflexives.

General aims of this paper

Our research tests how structural and semantic constraints influence participants’ final interpretations of pronouns and reflexives in PNPs, and also investigates the time course with which different constraints influence the processing of these forms. We aim to complement and extend existing work by testing whether the source and perceiver hypotheses in (4) and (6) are supported by experimental data for the on-line interpretation of possessorless PNPs as well as possessed PNPs. This research will (i) enable us to test whether the asymmetrical sensitivity that the form-specific multiple-constraints approach permits also applies to the within-sentence domain, and will (ii) shed light on whether structural and semantic constraints show qualitatively different behavior during real-time processing.

We conducted a series of comprehension experiments in which we used the verb to manipulate the source/perceiver status of the subject and object in sentences with PNPs. Compare, for example, the possessorless picture NPs in (7a) and (7b):

(7a) Peter told John about the picture of himself/him on the wall
(7b) Peter heard from John about the picture of himself/him on the wall.

In (7a), Peter is the subject and the source, and John is the object and the perceiver. In contrast, in (7b) Peter is the subject and the perceiver, and John is the object and the source. The verb manipulation allows us to create situations in which structural constraints and non-structural (source/perceiver) constraints are pitted against each other as well as situations in which they are aligned and favor the same antecedent.

From a purely structural perspective – e.g., the structural constraints articulated in classic Binding Theory – the verb manipulation should not affect reference resolution. According to this classic view, the subject noun phrase is the NP that the reflexive should be bound by and that the pronoun should be free from. We refer to the constraint that the reflexive be bound by the subject as the subject constraint, and the constraint that the pronoun be free from the subject as the anti-subject constraint. Crucially, the structural preferences of reflexives and pronouns are predicted to be unaffected by the verb manipulation.

However, the verb manipulation presented above introduces an additional difference between sentences (7a) and (7b); namely, the object of hear from is preceded by a preposition whereas the object of tell is not. One might expect the presence of the preposition to render the object of hear from (‘John’ in (7b)) incapable of binding the reflexive inside the PNP, due to the object being syntactically more deeply embedded in (7b) than in (7a). However, according to Pollard and Sag (1992) and Jackendoff (1990), this is not the case. They show that objects inside prepositional phrase arguments of verbs such as hear (e.g., ‘John’ in (7b)) have the same binding abilities as the direct objects of verbs such as tell (e.g., ‘John’ in (7a), see also Runner, 1998). In other words, if one regards the object as a potential antecedent, it is an equally possible antecedent with tell and with hear. In sum, according to Pollard and Sag (1992) and Jackendoff (1990), the presence/absence of the preposition does not have an effect on the object’s ability to act as an antecedent for a reflexive. Thus, comparing tell to hear from is not problematic.

Furthermore, as will become clear later, our results show that the object of hear from is actually a better antecedent for a reflexive inside the PNP than the object of tell is – which is exactly the opposite of what one would expect if the preposition were interfering with the binding abilities of the object.

Nature of relation between structural and semantic constraints

So far, we have considered the interpretation of pronouns and reflexives in possessorless PNPs from a purely structural perspective. However, as we discussed earlier, existing research suggests that semantic constraints are also at play. If both syntactic and semantic constraints play a role, we are faced with the question of how they interact. One fundamental question concerns the relative weights of syntactic and semantic information: How much of an influence does semantic information exert on the interpretation of pronouns and reflexives, as compared to the influence of syntactic information?

This formulation of the question presupposes that whatever the relative weights of structural and semantic constraints are, these weights are the same for pronouns and reflexives. Let us consider what predictions follow from this plausible starting point. If both syntactic and semantic constraints play a role, and if their relative weights are the same for pronouns as for reflexives, then we predict that in sentences like (7a) and (7b),

with pronouns, there will be more subject interpretations with hear than with tell (based on the perceiver-preference hypothesis in (6)).
with reflexives, there will be more object interpretations with hear than with tell (based on the source preference hypothesis in (4)), and
if the relative weights of structural and semantic information are the same for pronouns and reflexives, the proportion of subject and object choices triggered by one form should be ‘mirrored’ by the proportion of subject and object choices triggered by the other form. In other words, whatever levels of sensitivity reflexives show to structural and to semantic information, pronouns will show the same levels of sensitivity.

However, in light of existing results revealing form-specific effects in cross-clausal reference resolution (e.g., Kaiser, 2003; Kaiser & Trueswell, 2008; Brown-Schmidt et al., 2005), one should also consider the possibility that syntactic and semantic constraints are not weighted equally for pronouns and reflexives. If the form-specific approach applies to pronouns and reflexives in possessorless PNPs as in (7), then it could be the case that one form is guided by a relatively more powerful syntactic constraint, perhaps modulated by a weaker semantic constraint (or vice versa), while the other form might have the reverse situation or might perhaps be guided by more evenly weighted structural and semantic constraints. Thus, while we would still expect to see effects of perceiver status for pronouns and source status for reflexives (assuming that the semantic constraints are not so weak as to be indetectable), the magnitude of the effects could be different for pronouns and reflexives, thus ruling out the state of affairs outlined in (iii) above.

To further clarify how semantic constraints interact with structural constraints, we also investigate possessed PNPs (e.g. Lisa’s picture of herself/her). Due to the presence of the possessor phrase, possessed PNPs are subject to additional structural constraints which affect reflexives and pronouns. Most structural binding theories include a structural constraint penalizing coreference between a pronoun and the possessor, as well as a structural constraint requiring coreference between a reflexive and the possessor in possessed PNPs (e.g., Chomsky 1981; Pollard & Sag, 1992). We will refer to these constraints as the anti-possessor constraint (for pronouns) and the possessor constraint (for reflexives). (We follow existing work in assuming that constraints that make reference to syntactic/grammatical roles such as ‘subject’, ‘object’ and ‘possessor’ are structural constraints, but acknowledge that these structural roles are also connected to semantic and/or discourse-level properties.)

By examining both possessorless and possessed PNPs, we can investigate the relative strengths of different structural constraints, and compare their impact to that of semantic constraints. This allows us to see whether the relative weighting of a constraint is determined by its linguistic properties (structural vs. semantic, such that all constraints of a particular linguistic type are predicted to be equally weighted) or whether different structural constraints can have distinct weights – a question which has implications for our view of the syntax-semantics interface.

The experiments presented in this paper investigate these issues not only by looking at comprehenders’ final choices, but also by probing the time-course of what referents comprehenders consider over time, before converging on their final choice. A large body of recent research suggests that real-time language processing is continuously guided by multiple weighted constraints (e.g. MacDonald, Pearlmutter & Seidenberg, 1994; Trueswell, Tanenhaus & Garnsey, 1994; see also Badecker & Straub, 2002), and thus we expect to see early effects of both structural and semantic constraints and can observe when potential asymmetries in the constraint weights for pronouns and reflexives emerge during real-time processing. This contrasts with two-stage models of processing (e.g., Sturt, 2003 in the real-time processing of reflexives, see also Frazier & Fodor, 1978; Nicol & Swinney, 1989) which predict that effects of the source/perceiver verb manipulation will not emerge until after an initial stage of processing guided only by syntactic information.

We present four experiments. The first three experiments (Experiments 1, 2a and 2b) investigate possessorless PNPs, and the last experiment (Experiment 3) examines possessed PNPs. Experiment 1 is a picture verification task that tests whether the judgments predicted by the source and perceiver hypotheses can be confirmed experimentally. Experiments 2a and 2b aim to shed more light on the detailed time-course of the source/perceiver effects. Experiment 2a is an off-line study that pilots the picture-choosing methodology used in Experiment 2b, which is an eye-tracking study. Experiment 3 is an eye-tracking experiment that probes the sensitivity of pronouns and reflexives in possessed PNPs to the source/perceiver manipulation. By using eye-tracking, we can gain insights into the on-line process of reference resolution and thus investigate when different constraints play a role in the comprehension process.

Experiment 1: Picture verification

Method

Participants

Twenty-four native English speakers from the University of Rochester community participated in this experiment in exchange for $7.50.

Materials

The visual stimuli consisted of displays depicting two characters and a framed picture of one of the characters (see Figure 1). The images were generated from a large repository of purchased clip-art images and were arranged and edited using Adobe Photoshop. There were five male and five female characters. Before the start of the experiment, participants were familiarized with the names of the characters. The sound files that participants heard while viewing the visual displays were recorded using Praat speech software (Boersma & Weenink, Institute of Phonetics Sciences, University of Amsterdam) on an iMac G4 computer. The same female English speaker’s voice was used for all sound files, and the sentences were spoken with neutral intonation.

Sample display for Experiment 1, picture-verification experiment

A total of 32 target items (display-sentence pairs) and 32 filler items were constructed. The target displays always contained two characters, one on each side of the display, and a framed picture of one of the pictured characters in the middle (see Figure 1). The filler displays also had two characters and a picture in the same positions as the targets, but the picture depicted a third character. While viewing the displays, participants heard sentences with picture NPs without possessors, as shown in (8). For the target items, we crossed verb type (told/heard), anaphoric form (himself/him), and visual display (picture of subject/picture of object) to create eight conditions. Filler sentences had the same structure and used the same verbs, but contained no pronouns or reflexives.

(8) Peter {told/heard from} Andrew about the picture of {him/himself} on the wall.

With told, the subject of the sentence is the source of information and the object is the perceiver, whereas with heard, this pattern is reversed.

Eight presentation lists were constructed by combining the 32 target items with the 32 filler items. Within a presentation list, a participant saw 16 target trials with told and 16 with heard. Eight of the told items appeared with a pronoun and eight with a reflexive. Similarly, eight of the heard items appeared with a pronoun and eight with a reflexive. Half of the pronoun items were paired with a display showing a picture of the object of the sentence and half were paired with a display showing the picture of the subject. The same was done for the reflexives. This resulted in eight presentation lists in total. Overall, half of the items on any given presentation list contained two male referents and half contained two female referents. The pictures were counterbalanced for position of subject (left/right) and position of the source-of-information (left/right).

Procedure

Participants listened to pre-recorded sentences while looking at displays on a computer screen. The participants’ task was to indicate whether or not the sentence they heard matched the visual display, by pressing ‘y’ or ‘n’ on the computer keyboard. The displays were presented on an eMac G4 using Psyscope (Cohen, MacWhinney, Flatt & Provost, 1993), which was also used to record participants’ responses. The sound files were played over external speakers.

Predictions

If structural constraints (the subject constraint for reflexives and the anti-subject constraint for pronouns) fully determine reference resolution, differences in verb semantics should not lead to differences in anaphor resolution in picture NPs. Thus, the preferred antecedents for pronouns and reflexives should remain the same regardless of whether the sentence contains hear or tell. However, if semantic constraints modulate the effects of structural constraints for both pronouns and reflexives, as predicted by the source and perceiver hypotheses ((4) and (6)), a main effect of the verb manipulation is predicted. Reflexives are predicted to result in more structurally-dispreferred object choices with hear (object=source) than with tell, and pronouns are predicted to result in more structurally dispreferred subject choices with hear (subject=perceiver) than with tell. In other words, whereas the structural and semantic constraints converge in the tell conditions, they conflict in the hear conditions.

Crucially, the magnitude of the verb effect depends on the relative weights of the structural and semantic constraints. If the weights of structural and semantic constraints (whatever these relative weights turn out to be) are the same for pronouns as they are for reflexives, the prediction is that the magnitude of the verb effect will be the same for reflexives and pronouns. In other words, the proportion of subject and object choices in the reflexive conditions will be paralleled by the proportion of object and subject choices triggered in the pronoun condition: Whatever levels of sensitivity reflexives show to structural and to semantic information, pronouns are predicted to show the same levels.

However, if the form-specific multiple-constraints approach applies in the within-clause domain, the syntactic and semantic constraints might not be weighted equally for pronouns and reflexives. If the weightings for these two forms are different, one form could be subject to a powerful syntactic constraint and a weaker semantic constraint (or vice versa), whereas for the other form, the weights of the constraints might be the same or perhaps reversed. We do not have specific hypotheses regarding the directions of these effects, but want to emphasize that the form-specific approach allows for a situation in which the effects for pronouns and reflexives are asymmetrical, contrary to the ‘equally weighted’ approach sketched above.

Results and Discussion

We first provide a general overview of the results and then report in more detail the statistical outcomes of mixed-effects logistic regression models. As can be seen in Figure 2, in the reflexive conditions, there was a general preference to interpret the reflexive as referring to the subject: overall, collapsing across verb types, there were more yes answers when the subject was pictured (91%) than when the object was pictured (32%). However, the proportion of subject and object choices was modulated by the verb manipulation. When the object was pictured, there were 37.5% yes answers with reflexives when the object was the source-of-information (with hear), and 27% yes answers when the object was the perceiver-of-information (with told). When the subject was pictured, there were 93.7% yes answers when the subject was the source and 87.5% yes answers when it was the perceiver.

Percentage of ‘yes’ answers in the picture verification task in Experiment 1 (error bars represent +/− 1 SE).

In the pronoun conditions, there was a general preference to interpret the pronoun as referring to the object: collapsing across verbs, there were more yes answers when the object was pictured (78%) than when the subject was pictured (43%). The verb manipulation also influenced participants’ choices. When presented with a picture of the subject, participants gave 55.2% yes responses when the subject was the perceiver of information (with heard) and 30% yes responses when the subject was the source of information (with told). When presented with a picture of the object, participants gave 86.5% yes responses when the object was the perceiver of information and 69.8% yes responses when it was the source.

We used a mixed-effects regression model to analyze the proportion of yes answers as a function of anaphor type (pronoun vs. reflexive), verb (hear vs. tell) and picture (subject vs. object), with participant and item as random effects. The variable ‘picture’ refers to whether the subject or the object was pictured, i.e., whether participants were being asked to judge a subject-referring interpretation or an object-referring interpretation. The independent variables were centered in order to avoid collinearity in the interaction terms (see Jaeger, 2008 and others).

The analyses reveal a significant main effect of picture (β = −0.82, Wald Z = −3.9, p<.001), as well as a significant picture x anaphor interaction (β = 5.52, Wald Z = 12.94, p<.001), but no main effect of anaphor (p>.1). Here and in the rest of the paper, β is used to denote the estimated regression coefficient. Wald’s z-score (Wald, 1943) is calculated by dividing β by the estimate for its standard error and is a measure of how far the estimated regression coefficient is from zero in terms of its standard error. If this distance is great enough – i.e., the coefficient is judged to be significantly different from zero – the factor is considered to contribute significantly to the model (see Jaeger, 2008 for further details).

Further analyses show that in the pronoun conditions, a picture of the object was significantly more likely to be accepted than a picture of the subject (β =1.89, Wald Z=7.44, p<.001). In the reflexive conditions, a picture of the subject was significantly more likely to be accepted than a picture of the object (β =−3.50, Wald Z =−10.5, p<.001). Thus, we find a significant structural preference with both pronouns and reflexives, but in opposite directions: pronouns prefer (pictures of) object antecedents and reflexives prefer (pictures of) subject antecedents.

In addition, we also see a significant picture × anaphor × verb interaction (β =3.84, Wald Z=4.6, p<.001). Further analyses show that in the pronoun conditions, there is a significant picture x verb interaction (β =2.35, Wald Z=4.62, p<.001): with a picture of the object, there were more ‘yes’ answers with tell than hear, but when a picture of the subject was shown, there were more ‘yes’ answers with hear than tell. There is a weaker, but still significant picture x verb interaction in the reflexive conditions (β =−1.36, Wald Z =−2.08, p<.05): with a picture of the object, there were more ‘yes’ responses with hear than tell, but when a picture of the subject was shown, the verb manipulation did not have a very strong effect (Fig 2). In sum, the likelihood of yes responses in both the pronoun and the reflexive conditions depends not only on whether the picture shows the subject or the object but also on the verb used. The preference that pronouns and reflexives exhibit for objects and subjects respectively was significantly influenced by the source/perceiver status of the subject/object.

In order to compare the sensitivity of pronouns vs. reflexives to the verb manipulation more directly, we also fitted a model in which the dependent variable was the proportion of structurally-expected vs. structurally-unexpected responses. For reflexives, according to the subject constraint, responding ‘yes’ to a picture of the subject is a structurally-expected response (responding ‘no’ is unexpected), and responding ‘no’ to a picture of the object is also a structurally-expected response (responding ‘yes’ is unexpected). For pronouns, according to the object constraint, responding ‘yes’ to a picture of the object is a structurally-expected response, and responding ‘no’ to a picture of the subject is also a structurally-expected response. As before, the independent variables were centered to avoid collinearity. This approach makes it possible to evaluate the reflexive and pronoun conditions uniformly by allowing us to compare directly the strength of the (different) structural preferences of pronouns and reflexives, as well as how strongly these preferences are influenced by the verb manipulation

This analysis reveals a significant main effect of anaphor (β =−0.80, Wald Z =−3.92, p<.001), signaling an asymmetrical sensitivity to structural information: Pronouns resulted in more structurally-unexpected responses (acceptance of a subject antecedent) than reflexives (acceptance of an object antecedent). We also see a significant main effect of verb (β =0.95, Wald Z =4.62, p<.001), due to hear triggering more structurally-unexpected responses than tell. This fits with the hypothesis that reflexives are subject to a source constraint (and thus objects are more likely to be accepted with hear than tell) and pronouns to a perceiver constraint (and thus subjects are more likely to be accepted with hear than tell).

In addition, there is a significant anaphor-picture interaction (β =2.97, Wald Z =7.18, p<.001), showing that reflexives are more sensitive to the structural subject/object distinction than pronouns. However, there is no significant anaphor-verb interaction (p>.2), indicating that there is no significant difference in how sensitive pronouns and reflexives are to the verb manipulation.

Further analyses reveal a strong effect of the verb manipulation for pronouns (β =1.23, Wald Z =4.74, p<.001). The reflexive conditions show a weaker but nevertheless significant verb effect as well (β =0.88, Wald Z =2.41, p<.05). Thus, semantic constraints have a significant effect on both forms.

Overall, the results of the picture-verification task used in Experiment 1 revealed a significant effect of structure, with pronouns preferring objects and reflexives preferring subjects. The verb manipulation also had an effect on both anaphoric forms, with reflexives preferring sources and pronouns preferring perceivers – as predicted by the hypotheses in (4) and (6). However, there was a difference in the strength of the effects, with reflexives being more strongly guided by structural information than pronouns. This asymmetry supports a model such as the form-specific multiple-constraints approach, in which multiple constraints of differing types and strengths interact and can be differently weighted for different anaphoric forms. In particular, we see that in the case of reflexives in PNPs, the structural subject constraint is weighted more heavily than the semantic source constraint – although the semantic constraint still has a modulating influence. In contrast, pronouns in PNPs exhibit more evenly-matched competition between the structural anti-subject constraint and the semantic perceiver constraint. This pattern of data is compatible with the form-specific multiple-constraints framework, but not an approach in which the relative weights of the structural and semantic constraints are the same for both pronouns and reflexives. This finding is also consistent with Runner et al. (2003) who found that a simple discourse manipulation had a much greater effect on pronouns than on reflexives.

Experiments 2a and 2b. Picture choosing and eye-tracking

Even though the results of Experiment 1 shed light on how structural and semantic constraints influence the interpretation of pronouns and reflexives in picture NPs, yes/no answers reflect participants’ final choices and do not tell us about other interpretations participants may have considered before reaching a final choice. In order to learn about what entities participants consider as potential referents for pronouns and reflexives in picture NPs, and how these choices emerge and/or change over time, we use the visual-world eye-tracking paradigm (Cooper, 1974; Tanenhaus, Spivey-Knowlton, Eberhard & Sedivy, 1995).

In order for eye movements to provide meaningful information about which referents participants are considering as potential antecedents, the visual display presented to participants must include at least two pictures – in this case a picture of the subject and a picture of the object. While such displays can easily be designed, they represent a departure from the picture verification task because the picture-choosing task forces participants to choose between two directly competing interpretations. This differs from the picture verification task, in which participants are asked whether a particular sentence matches a particular display. Thus, the picture-choosing task asks participants to choose the best interpretation out of two possibilities, whereas the picture verification task asks participants whether a particular interpretation is possible. A priori, we might expect these tasks to generate slightly different results, especially for interpretations that are possible but dispreferred.

As a preliminary step towards our aim of using eye-tracking to investigate the on-line processes leading to participants’ final interpretations of pronouns and reflexives in picture NPs, we first conducted an off-line picture choosing experiment to investigate the effects of a design switch from picture verification to picture choice (Experiment 2a), followed by a visual-world eye-tracking study (Experiment 2b).

Experiment 2a. Picture-choosing

Experiment 2a investigates whether the effects of the source/perceiver verb manipulation persist in a context where people need to choose between two directly competing interpretations. If both the structurally-preferred and the structurally-dispreferred interpretation are visually salient, will we still see effects of the semantic constraints? To test this question, we used the same kinds of auditorily-presented sentences as Experiment 1, but instead of being shown a computer display consisting of the two mentioned characters and a picture of one of the characters, participants saw (on a sheet of paper) a display with two characters and a picture of each character (Fig 3). The task was to select one of the pictures by circling it.