Little houses and casas pequeñas: message formulation and syntactic form in unscripted speech with speakers of English and Spanish

Abstract

During unscripted speech, speakers coordinate the formulation of pre-linguistic messages with the linguistic processes that implement those messages into speech. We examine the process of constructing a contextually appropriate message and interfacing that message with utterance planning in English (the small butterfly) and Spanish (la mariposa pequeña) during an unscripted, interactive task. The coordination of gaze and speech during formulation of these messages is used to evaluate two hypotheses regarding the lower limit on the size of message planning units, namely whether messages are planned in units isomorphous to entire phrases or units isomorphous to single lexical items. Comparing the planning of fluent pre-nominal adjectives in English and post-nominal adjectives in Spanish showed that size information is added to the message later in Spanish than English, suggesting that speakers can prepare pre-linguistic messages in lexically-sized units. The results also suggest that the speaker can use disfluency to coordinate the transition from thought to speech.

Speaking begins with the formulation of a non-verbal message that must undergo linguistic encoding to be produced as a comprehensible, sequentially ordered string of words (Bock & Levelt, 1994). In principle, speakers can start talking after encoding some minimal chunk of the message they wish to convey. The size of these chunks has been under scrutiny in the literature on language planning for some time, with most findings pointing to a phrasal scope of planning during the early stages of utterance planning (e.g., Dell & O’Seaghdha, 1992; Ferreira, 1991; Levelt & Maassen, 1981; Smith & Wheeldon, 1999). Here, we examine the formulation of the message itself—its scope and its coordination with subsequent utterance planning.

Currently we know very little about the time-course of message formulation and how messages are coordinated with linguistic processing (Bock, Irwin, & Davidson, 2004). Some have argued that the entire message might be available before linguistic encoding begins (e.g., Wundt, 1900), while others argued that messages themselves might be prepared incrementally as speech unfolds (e.g., Paul, 1880). More recently, it has been suggested that initial formulation may be rapid for simple messages, as speakers can initiate picture descriptions within 1 second of picture onset (Bock, Irwin, Davidson, & Levelt, 2003; Griffin & Bock, 2000). With respect to linguistic planning, there is also debate about the size of the units that can undergo linguistic encoding simultaneously. Speech errors involving exchanges of words from different phrases suggest that speakers’ linguistic preparation can be extensive (Garrett, 1975). Other work suggests that early linguistic planning, or semantic encoding, may be more incremental, encompassing words from different phrases in simple sentences (e.g., Meyer, 1996; Smith & Wheeldon, 2004), but limited to words occurring in the first phrase of more complex utterances, with only rudimentary processing of words in subsequent phrases (Smith & Wheeldon, 1999, 2004; also Bock et al., 2003; Ferreira & Swets, 2002; Griffin, 2001).

In communicative settings, the preparation of messages often involves incorporating information from the referential context. In the case of referring expressions, which are the focus of this paper, messages must include enough information to uniquely identify the intended referent; e.g., by using modifiers (the small/big butterfly) to distinguish between potential referents (Olson, 1970; Osgood 1971; Roberts, 2003). So, understanding the scope of planning for referring expressions requires understanding how, and when, the speaker evaluates the referential context, formulates an adequately informative message, and then passes this message to utterance planning processes.

Brown-Schmidt and Tanenhaus (2006) used the visual world eye-tracking technique (Cooper, 1974; Pechmann, 1989; Tanenhaus, Spivey-Knowlton, Eberhard & Sedivy, 1995) to examine the coordination of message formulation and utterance planning in a paradigm where speakers planned simple messages online. Speakers were asked to describe selected pictures (e.g., a peach) in contexts that sometimes contained a contrast picture (e.g., a bigger peach), requiring the use of a modifier to uniquely identify the target referent (the small peach). The results indicated a tight temporal link between fixations to the relevant pictures and the ensuing linguistic expression. Early fixations to the contrast picture were associated with pre-nominal modification (the small peach), delayed fixations were associated with repairs (the peach…uh small one), and intermediate fixations with disfluent productions (thee uh small peach). Initial contrast fixations were also delayed for post-nominal modifiers (the square with small triangles), suggesting that the part of the message that expresses size was planned after the part that expressed the noun.

Thus in this paradigm, the timing of fixations to unmentioned, but message-relevant entities can provide insight into the coordination of processes at the interface of message formulation and utterance planning. The Brown-Schmidt and Tanenhaus (2006) results show that messages can be passed to utterance planning processes in roughly phrase-sized units. Both message planning and utterance planning appear to be highly incremental, with speakers formulating aspects of messages on the fly and encoding information linguistically as soon as it becomes available.

The primary goal of this paper is to examine the lower-limit on the size of incrementally-planned message-level units. We compare the coordination of message and utterance planning for size-modified expressions in languages that differ in phrasal syntax: scalar adjectives precede the noun in English (the small butterfly), but follow the noun in Spanish (la casa pequeña). This difference leads to a strong prediction concerning the relationship between message planning and linguistic planning, and thus the relationship between message planning and the uptake of visual information in a referential communication task. If message planning is incremental at the phrasal level, there should be no difference between the two languages in terms of when size information is added to the message: the timing of first contrast fixations (relative to NP onset) should be equivalent for English and Spanish size-modified expressions. However, if messages can be passed to utterance planning processes in units smaller than what is needed to formulate an entire phrase, we should observe later planning for size information when speaking in Spanish compared to English. Since these planning units correspond to individual lexical items, we refer to this process as lexically-incremental message planning.

Specifically, lexically-incremental planning predicts that first contrast fixation times for fluent, size-modified expressions (e.g. the timing of the first contrast fixation, relative to NP onset) should occur later, on average, in Spanish than in English. This is because for Spanish, relatively late fixations to the contrast (within about 200ms of NP onset, given our materials) still allow time for size to be included in the message and incorporated into a fluent, modified expression, as planning of the size term can occur during production of the determiner and noun. In contrast, when speaking English, such late additions of size information do not provide enough time to add this information to the message and plan a pre-nominal modifier. Thus the range of first contrast fixation times that affords fluent, modified expressions in Spanish is wider (including more fixations just before and after NP onset) than the range that affords fluent, modified expressions in English. Therefore, on average, first contrast fixations should delayed for Spanish compared to English (e.g. not as far before NP onset, compared to English).

A secondary goal of this paper is to examine the role of disfluency in the interfacing of message and utterance planning. Specifically, we test the hypothesis that speakers use disfluent repairs (the butterfly…small one; la mariposa, eh, pequeña) to add late planned message elements after utterance planning is underway (Brown-Schmidt & Tanenhaus, 2006). If so, first contrast fixation times for utterances with repairs should be delayed compared to fluent utterances.

Method

Participants

Thirty-two early Spanish-English bilinguals participated in exchange for $20 or course credit¹. Data from 13 additional participants were excluded due to equipment problems (n=3), failure to follow instructions (n=5), or late acquisition of English (n=5).

A language questionnaire evaluated language use and experience. All participants started learning English and Spanish before age 8. The average age of exposure was 3.67 years (SE=.47) for English and .28 years (SE=.14) for Spanish. Participants rated how often they used each language on a 1–5 scale (1=never, 5=several hours every day). Average use of English and Spanish was 4.78 (SE=.07) and 4.13 (SE=.09), respectively. Participants also reported being proficient speakers of both languages (1=poor, 5=excellent), M=4.71 (SE=.08) for English, and M=4.65 (SE=.09) for Spanish.

Procedure

The experiment consisted of two one-hour sessions (one in English, one in Spanish) conducted on different days. Half of the participants completed the English session first, and the other half completed the Spanish session first. For the English session, the experimenter was a monolingual English speaker. For the Spanish session, the experimenter was a native Spanish speaker fluent in English.

On each trial, the participant and experimenter saw 20 pictures on their respective computer screens (see Figures 1a–b). On the first trial, the participant saw a yellow box around one picture (the target referent). The task was to describe the target for the experimenter. Once the experimenter found the target, both the participant and experimenter clicked on it and the next trial began. On this trial, it was the experimenter’s turn to tell the participant which picture to find. The experimenter modeled her instructions after typical responses by participants in the task. The participant and experimenter alternated in describing pictures across 480 trials in two sessions.

Figure 1.

a–b. Example scene from the speaker’s perspective (Fig. 1a) and the listener’s perspective (Fig. 1b). The pictures are arranged differently on the participant’s and experimenter’s computer screens in order to prevent use of locatives (e.g., top right picture).

Materials

The pictures were easily identifiable, colorized images, selected from a full-color version (Rossion & Pourtois, 2004) of a large corpus of pictures (Snodgrass & Vanderwart, 1980) and similar clip-art pictures. A norming study was conducted to select target pictures which would be easily nameable in both languages.

On half of the trials, the target picture was presented with a contrast picture that differed in size, color or numerosity. On the remaining trials, the contrast picture was replaced with an unrelated picture; these no-contrast trials were used to evaluate the rate of modifier use in the absence of contextual ambiguity. Our analyses focus on size-contrast trials (and the no-contrast controls), as use of size adjectives is highly sensitive to the referential context (Sedivy, 2005).

Each picture was seen approximately 10 times during the experiment but no picture was a target more than once. Picture locations were randomized, with the exception that target and contrast pictures were never adjacent in order to minimize the chance that participants would notice the contrast in visual periphery while fixating the target.

The 480 trials (240 experimenter-speaking, 240 participant-speaking, half contrast-present, half contrast-absent) were arranged in two lists (presented in one of four random orders), which counterbalanced the target and contrast pictures for participant-speaking, contrast-present trials.

Results

The participant’s description of each target picture was transcribed. For trials with a size-contrast, the timing of expression onset and the first fixation to the size-contrast were measured. Trials during which the participant failed to name the target, used the wrong size adjective, or the experimenter interrupted the participant were excluded from analysis (12% of trials).

Modification and referential context

When participants described pictures on size-contrast trials, modification rates were significantly higher compared to similarly designed no-contrast trials, F₁(1,31)=2427.94, F₂(1,67)=1671.33, demonstrating that use of a size adjective strongly depends on the presence of a size-contrast in the scene (Table 1). Modification rates did not differ by language, F₁(1,31)=.51, F₂(1,67)=2.71, and there was no interaction, F’s <.2.

Table 1.

Modification rates for contrast-present and contrast-absent trials.

		English	Spanish
Contrast-absent Trials		8% (n=593, SE=1%)	7% (n=570, SE=1%)
Contrast-present Trials		90% (n=563, SE=2%)	89% (n=533, SE=2%)
	contrast fixated	95% (n=511, SE=1%)	94% (n=478, SE=2%)
	contrast not fixated	40% (n=52, SE=8%)	47% (n=55, SE=8%)

Fluency and Speech Latency

On trials with a size-contrast in the display, speakers began their expressions approximately 2 seconds after display onset (Table 2). Across all trials, there was no cross-linguistic difference in expression onset time (ts<.7), nor was there a difference for the subset of fluent, modified trials (ts<.5). The most frequent type of disfluency was post-nominal size repairs in English (the butterfly, uh small one), which had earlier expression onsets compared to fluent English expressions, t₁(38)=4.50, t₂(29)=2.03, suggesting that when the speaker made a repair to add size, the size adjective was not planned before expression onset².

Table 2.

Noun phrase (NP) onset times for contrast-present trials only.

		n	NP onset (SE)
English (all trials)		563	1955 (84)
pre-nominal modifier (all)		336	2005 (93)
	pre-nominal, fluent (the small house)	292	2021 (96)
	pre-nominal, disfluent determiner (thuuh small house)	44	2003 (201)
disfluent repair (the house uh small one)		173	1791 (83)
no modifier (the house)		54	2013 (153)
Spanish (all trials)		533	2013 (91)
post-nominal modifier (all)		473	2013 (92)
	post-nominal, fluent (la casa pequeña)	414	2008 (104)
	post-nominal, disfluent determiner (laaa casa pequeña)	25	2000 (284)
	disfluent repair (la/laaa casa eh pequeña)	34	2522 (537)
no modifier (la casa)		60	2042 (182)

Note. Data shown include all contrast-present trials, regardless of whether there was a fixation to the contrast. NP onset times indicate by-participant means of the time from display onset to NP onset (ms).

Referential form and timing of fixations

The typical pattern of fixations consisted of an initial look to the highlighted target, followed by looks to the size-contrast and/or unrelated pictures. The percentage of trials speakers looked at the contrast was equivalent in English (91%) and Spanish (90%). When a size-contrast was present, modification was significantly more likely if the speaker fixated the contrast than if they did not (Table 1); the 95% confidence interval of the difference was ±16% for both English and Spanish (the magnitude of this effect was equivalent for English and Spanish, 95% CI of the difference= ±17%). The link between modification and contrast fixations demonstrates that the first fixation to the contrast is a reasonable indicator of when the part of the message that encodes size is first planned.

If message planning proceeds in lexically sized units, when producing a fluent expression, speakers should look at the contrast earlier, relative to NP onset, in English compared to Spanish. Consistent with this hypothesis, on fluent trials (trials with word repeats, lengthenings, or lexical disfluencies were excluded), speakers looked at the size-contrast significantly earlier when speaking in English (M=−482ms) compared to Spanish (M=−108ms), t₁(31)=2.91, t₂(39)=3.86 (Figure 2).

Figure 2.

Latency of first size-contrast picture fixations, relative to NP onset, in English and Spanish. Trials with NP onsets more than 2 SD outside the grand mean were trimmed (M=2022ms, SD=1160ms), as were disfluent trials. Error bars indicate standard errors.

However, it is possible that size-contrast fixations occurred, on average, later in Spanish, compared to English, because some of the Spanish utterances had non-lexical disfluencies between the noun and the adjective: in fact, some expressions contained silent pauses (e.g., la mariposa…pequeña; mean pause length=440ms). Nevertheless, some pausing is normal in fluent production, and the distribution of first contrast fixation times for Spanish expressions with and without a post-nominal pause did not differ (Figure 3), t₁(25)=.65, t₂(34)=1.56, suggesting that pausing is not responsible for the delay in Spanish. Further, if disfluent pauses were the source of the cross-linguistic effect, the effect should disappear when comparing Spanish trials (with and without pauses) to English pre-nominal trials with and without pre-nominal disfluencies; importantly, the cross-linguistic difference remains, t₁(31)=2.91, t₂(39)=3.87. Lastly, if we consider only Spanish trials with no discernible pause, the difference between English and Spanish size-modified expressions is still 237ms (compared to a 373ms difference for the full dataset), t₁(25)=1.78, p<.05, t₂(34)=1.39, p=.09, both one-tailed. The by-items effect is most likely marginally significant due to the reduced amount of data in this analysis.

Figure 3.

Relative frequency of first contrast fixations for size-modified expressions in English (pre-nominal and post-noun repairs), for Spanish post-nominal expressions containing a pause between noun and adjective, and for Spanish post-nominal expressions with no discernible pause between noun and adjective.

Next, we examined disfluent utterances to test the hypothesis that disfluent size-repairs occur when size information is planned well after information about the noun. Consistent with this hypothesis, for English expressions with a repair (the butterfly, uh small one), speakers looked at the contrast well after NP onset (657ms, n=154), a significant delay compared to fluent English expressions, t₁(29)=8.83, t₂(38)=6.77³.

General Discussion

Communicating a well-formulated message requires tight coordination between conceptual, message-level representations and the linguistic processes which implement messages into speech. Previous work on message planning showed that elements of a message appearing in different phrases are planned incrementally, in the order in which they appear within an utterance (Brown-Schmidt & Tanenhaus, 2006). In this experiment, we tested the hypothesis that the lower-limit on the size of message planning units is lexical. This lexically-incremental planning hypothesis predicts that when producing fluent, size-modified expressions, size-contrast fixations should occur, on average, later in Spanish than English because post-nominal modifier syntax affords delayed planning of the size adjective in Spanish. We observed the predicted cross-linguistic difference, with first fixations to size-contrasts occurring later in the production of Spanish than English descriptions. When English speakers looked late at the contrast, they used disfluent repairs to accommodate the late-planned size information into the utterance. Such fine tuning of the timing of speech, achieved through pauses, word lengthening, and the addition of optional words (Arnold, Fagnano, & Tanenhaus, 2003; Ferreira & Dell, 2000; Fox-Tree & Clark, 1997), may be better viewed as tools, rather than problems, of typical language production.

We propose that these results can inform theories of message planning and the coordination of this information with linguistic encoding. A growing number of studies (e.g., Meyer, 1996, 1997; Pechmann, 1989; Smith & Wheeldon, 1999, 2004) have suggested that utterance planning is highly incremental, with a phrasal scope of planning during early, semantic encoding. Our results point to a high degree of incrementality in message planning as well, and suggest that the scope of planning at this level can be lexically incremental. Lexically incremental planning of message elements allows message-planning of size information separately from message-planning of the noun. Cross-linguistic differences in phrasal syntax result in a cross-linguistic difference in the range of contrast fixation times that afford fluent production: Spanish phrasal syntax affords later planning of size information while preserving fluency. Consequently, the set of fluent, Spanish post-nominally modified utterances have, on average, later first contrast fixation times relative to English. Because the effect was observed with fluent speech, i.e., when speakers produced all the necessary message elements without fillers or repairs, we propose that this type of incrementality is a characteristic of message planning proper, rather than merely a phenomenon masking difficulties in conceptual or linguistic processing⁴.

These results have direct implications for models of the interface between conceptual and linguistic representations. We propose that speaking begins with activation of conceptual representations associated with the speaker’s communicative intent. In referential-communication tasks, conceptual activation begins with fixations to the target. Each element of the pre-linguistic message is eventually passed to linguistic encoding in a lexically-sized unit, via interfacing representations (Bock, 1982), and finally to the articulators. Figure 4 presents a mechanistic sketch of this process.

Figure 4.

Schematic of the proposed lexically-incremental message-to-utterance planning interface for picture descriptions in English and Spanish. Arrows represent proposed connections (dotted arrows represent feedback connections). Time=0 is NP onset. The process begins with an initial fixation to the target (small butterfly), followed by a fixation to the size-contrast (large butterfly). When the contrast is fixated early, the resulting expression is fluent in both languages. When the contrast is fixated just after NP onset, the result is a fluent expression in Spanish, and a disfluent repair in English (see Figure 3; the modal first contrast fixation for both expression types is immediately following NP onset).

How might lexical incrementality extend to other types of utterances? Here we show that message planning units can be as small as what is needed to express size, which may be facilitated by an isomorphism of meaning between size and size terms. More complex meanings expressed through multiple words, such as verb phrases with verbs requiring multiple arguments, may undergo more complex, hierarchical patterns of message planning (see Bock et al., 2003). In this case, lexical incrementality in message planning may involve sequentially selecting the meanings for anticipated arguments. Based on our results, we predict that verb arguments are planned in lexically-sized units, and that the order in which these units are planned should affect both fluency and syntactic structure. Further examination is necessary to identify the boundary conditions of message-level incrementality as well as other language-specific effects on linguistic planning (see Janssen, Alario, & Caramazza, 2008, for a demonstration of cross-linguistic differences in phonological encoding).

In sum, examining how speakers coordinate message planning with utterance formulation as they construct simple messages shows that message elements are passed to utterance planning processes in units smaller than what is needed for an entire phrase, possibly in lexically-sized units. As message elements are prepared over time, properties of the phrasal syntax used to express those messages constrain whether a fluent expression can be produced. When message plans are inconsistent with phrasal syntax, disfluency provides the flexibility needed to execute the linguistic plan. The obtained pattern of lexical incrementality in message planning points to very fine temporal coordination between message and utterance planning.