Gestural coordination at prosodic boundaries and its role for prosodic structure and speech planning processes

Abstract

Prosodic structure is a grammatical component that serves multiple functions in the production, comprehension and acquisition of language. Prosodic boundaries are critical for the understanding of the nature of the prosodic structure of language, and important progress has been made in the past decades in illuminating their properties. We first review recent prosodic boundary research from the point of view of gestural coordination. We then go on to tie in this work to questions of speech planning and manual and head movement. We conclude with an outline of a new direction of research which is needed for a full understanding of prosodic boundaries and their role in the speech production process.

1. Introduction

The goal of this review is to investigate the coordination of articulatory gestures at prosodic boundaries and to examine how these coordinations can inform us about prosodic structure and the broader context of language use, such as speech planning. The focus is predominantly on articulatory gestures, but recent work on manual and head gesturing is also examined. This is a necessarily limited selection of research; for reviews focusing on other aspects of prosodic structure (e.g. the acoustic properties of prosodic structure, the syntax–prosody interface, different accounts of prosodic structure that have been proposed) see Shattuck-Hufnagel & Turk [1], Wagner & Watson [2] and Fletcher [3].

The linguistic term prosody refers to the suprasegmental structure of the utterance, encoding prominence and phrasal organization [4–7]. Here, the focus is on phrasing and prominence above the word level (see e.g. [6,8,9] for discussion of prosodic properties at the word level). In English, prominence highlights important or new information and is also used for rhythmic purposes.^1,2 Phrasal organization serves to group words together into appropriate chunks that are used by listeners and speakers for language processing.

The focus of this review is on phrasal organization, specifically on the properties of prosodic boundaries that mark these phrases. An example of prosodic phrasing and the importance of prosodic boundaries in deducing the meaning of these utterances is given in the sentences below. Note that sentence (ii) has two more boundaries, marked by commas, than sentence (i).

• (i) She knew David thought about the present.

• (ii) She knew, David thought, about the present.

The phrases form a prosodic hierarchy, with larger units dominating smaller units. We start with a brief outline of the prosodic hierarchy in §2, describe how the π-gesture model [7] accounts for the temporal properties of prosodic boundaries (§3a), and then in §3b,c turn to more recent studies examining how linguistic gestures at prosodic boundaries are coordinated (e.g. [13–15]). A discussion of the role of pauses in speech planning and of the articulatory properties of pauses is given in §4a,b. We bring these lines of research together by investigating the coordination of body movements (e.g. manual gesturing) with prosodic structure [16,17] in §4c and by presenting a small study of head and manual gesturing at prosodic boundaries in §4d. In the remainder of the paper, we discuss the implications that gestural coordinations have for our understanding of prosodic boundaries in a broader context. We suggest directions for research that will help address some of the central issues for prosodic theory by taking gestural coordination at boundaries as a starting point.

2. The prosodic hierarchy

Within an utterance, words group together into phrases, which in turn group into larger phrases, forming a prosodic hierarchy. Research approaches differ as to how they define these prosodic units. Within one approach, they are determined based predominantly on syntactic structure [18,19], whereas another approach identifies them based on intonational properties (e.g. [20,21]; see [22,23] for a discussion of these two approaches).

As to factors influencing prosodic structure, it is known that syntactic structure determines prosodic structure to a large extent. Additional factors have been found to influence prosodic phrasing in terms of both occurrence and strength of prosodic boundaries. For example, speech rate (for English [24] for French [25] and for German [26]) and phrase length [27–30] are well known to have an effect on prosodic phrasing.³ More recently, phrase boundaries across an utterance have been shown to influence how individual boundaries are produced or perceived, indicating global effects of prosodic structure on speech production and perception [30–34]. For example, listeners’ interpretation of ambiguous sentences depends on the relative strength of prosodic boundaries, i.e. on the strength of one boundary in relation to another [31].

Over the years, a number of different prosodic hierarchies have been proposed (see the overview in [1]; see also [9,35,36] for a discussion on a more gradient approach to the prosodic hierarchy). For English, there are at least two categories above the word level although further additional categories have been proposed. We follow Beckman & Pierrehumbert [20] and refer to these two categories as the intermediate phrase (ip) and intonation phrase (IP). A schematic representation of the hierarchy is given in figure 1. In this model, the IP is the highest prosodic category. Specific tonal properties mark the phrasal units. The ip has at the minimum a nuclear pitch accent. Pitch accents signal prominence, and an ip can have additional pitch accents; the nuclear pitch accent is by definition the last pitch accent in the ip. The end of the ip is marked by a phrase accent. The IP consists of one or more intermediate phrases, and its end is marked by a boundary tone (and, although rarely, the beginning can also be marked with a boundary tone).

Figure 1.

Prosodic structure for English ([20] model). ‘T*’ stands for pitch accents, ‘T-’ for phrase accents and ‘T%’ for boundary tones. The nuclear pitch accent is circled. The structure represents a production with a weak (ip) boundary after ‘Whittaker’ and a strong boundary (IP) after ‘late’. (Online version in colour.)

In addition to tonal properties, prosodic phrases have characteristic temporal properties. In the acoustic domain, boundaries are marked by lengthening of segments, i.e. segments at the boundary are longer compared with the same segments phrase-medially [37–40]; this phenomenon is also referred to as phrase-final and phrase-initial lengthening. Strong boundaries (IP boundaries) often have pauses in addition to the lengthening. Work in articulation has shown that boundary–adjacent gestures become longer and less overlapped [41–49]. The temporal effects grade in magnitude, reflecting hierarchical embedding such that prosodically higher categories show more lengthening than lower categories, and distinguishing several degrees of lengthening (e.g. phrase-final lengthening: [30,38,42,44]), and phrase-initial: (e.g. [30,41,42,44,48–50]).

3. Prosodic gestures

(a). The π-gesture model

The π-gesture model provides an account of the properties of prosodic boundaries [7]. While various other conceptualizations of prosodic boundaries have been proposed [18–20] (see reviews in [1,3]), the π-gesture model is discussed here because it clearly defines prosodic boundaries (for a discussion of the lack of definition of boundaries in the literature, see [9,51]). Furthermore, it has explicit temporal properties and allows the examination of the coordination of prosodic events. Finally, the model allows for a structurally more gradient prosodic hierarchy, which is in line with experimental evidence (see discussion in [9,35,52]). The π-gesture model has been developed within the Articulatory Phonology framework [53–55], where the basic phonological unit is a gesture, which specifies a constriction target as its goal (e.g. for alveolar consonants, a tongue tip constriction constitutes the constriction target). Gestures in Articulatory Phonology are both units of information, specifying lexical contrast, and units of action, with specified temporal and spatial information. That is, gestures are lexical units parametrized both phonetically and phonologically, such that there is no need for a translational component that traditionally might be posited as mediating between phonology and phonetics. Byrd & Saltzman [7] propose that prosodic boundaries are viewed as gestures. The prosodic gesture (π-gesture) does not instantiate constriction targets, but instead, the effect of the π-gesture is to slow the time course of the constriction gestures that are active at the same time as the π-gesture. Like constriction gestures, π-gestures extend over a temporal interval. As a consequence of these properties, gestures at boundaries become temporally longer and less overlapped. Stronger prosodic boundaries have a stronger π-gesture activation, which in turn leads to boundary effects becoming stronger. That is, at hierarchically higher boundaries, there is more lengthening and less overlap than at lower boundaries, thus accounting for the empirical findings on the temporal properties of boundaries. As mentioned above, the model also allows for a structurally gradient or categorical prosodic hierarchy [52]. Byrd [52] discusses that the gradient structure could arise by allowing the π-gesture to have a continuum of activation strength values, whereas the categorical structure could be achieved by allowing a small number of attractors for the activation strength of the π-gesture. Alternatively, the π-gesture could have a continuum of activation strength values, but different tonal properties (phrase accent, boundary tone) would distinguish between different types of prosodic categories (see [30,52] for further discussion). The π-gesture suggested a novel way of viewing prosodic boundaries as inherently temporal, and it makes specific predictions about the behaviour of gestures at boundaries. One of the predictions of the model is that the temporal effects of the boundary extend over a period of time, and that the slowing effects decrease with distance from the boundary, corresponding to the activation shape of the π-gesture, shown in figure 2 (see [7]). The predictions of this model have been born out in a number of articulatory (e.g. [14,30,56]) and acoustic studies (e.g. [57–59] for Hebrew and [60] for Dutch).

Figure 2.

A schematic representation of the π-gesture [7]. The arrow indicates the activation strength of the gesture, and the scope of its effect is indicated by the grey area. Darkness in the area reflects the strength of the effect.

(b) Tone gesture and μ-gesture

Recent work has further developed the idea that prosodic structure can be understood in terms of gestures. Saltzman et al. [61] have modelled lexical stress using a temporal modulation gesture (μ-gesture), which, like the π-gesture, slows the constriction gestures that are co-active with it. Gao [13] extends the gestural approach to tones in Mandarin. Thus, tones are viewed as dynamical systems [62–64], that, like other gestures, extend in time. They have as their goal tonal targets and are coordinated with constriction gestures [13]. This approach has also been used to investigate Catalan and German pitch accents [65], and boundary tones in Greek [14,66].

These developments allow us to consider the relationship among prosodic events, and between prosodic events and constriction events, in terms of gestures and gestural coordination (where gestural coordination refers to the timing of gestures to each other). We can also address the question of how prosodic events and articulatory constriction events are coordinated with other body movements. Examining these relationships is critical for any theory of prosodic structure, but the Articulatory Phonology framework lends itself to these questions owing to the dynamic properties of the model. Gestural coordination has been implemented in Articulatory Phonology with a coupled oscillator system, in which gestures are associated with oscillators and gestural coordination emerges through the coupling of these oscillators [67–69].

(c). The coordination of prosodic events

The relationship between prominence and boundaries is a crucial area of research, because a model of prosodic structure can be developed only if the interactions between various prosodic events are known. It has been suggested, for example, that the onset of prosodic boundaries might be triggered by lexical stress (this could be achieved by coordinating the π-gesture with the last lexically stressed syllable in a prosodic phrase; see [30,52,66]).⁴ It has also been argued that prominence and phrasing have similar functions cross-linguistically [5,6,9,22,70]. The current discussion is limited to the interaction of boundary and prominence in the pre-boundary domain, as the relationship between the two phenomena has been less examined in the post-boundary domain (but see [15,71]).

Systematic investigations of the interaction between boundaries and prominence have started only recently. In an acoustic study of English, Turk & Shattuck-Hufnagel [72] examined final lengthening in words with final, penultimate and antepenultimate lexical stress and found that final lengthening starts earlier in the word when the stressed syllable is further away from the boundary. Unstressed syllables between the stressed and the final syllable were not always lengthened, or were lengthened less than the stressed syllable. Rusaw [73] replicated these findings. Similar results were obtained in White's [74] study, but contrary to Turk & Shattuck-Hufnagel [72] and Rusaw [73] the intervening unstressed syllables in his study also lengthened. Byrd & Riggs [15] in an articulatory magnetometer study also found that final lengthening can start earlier when the prominent syllable is earlier in the phrase-final word. In an articulatory magnetometer study of Greek, Katsika [14] found that the onset of final lengthening was during the final syllable in words with final stress, but that final lengthening started earlier when the prominent syllable was earlier in the phrase-final word (consistent with [15,72–74]). Like White [74], she also found that intervening unstressed syllables lengthened. Katsika [14] further examined the onset of the boundary tone in IP boundaries. She found that the boundary tone always occurred during the final vowel, but that it occurred earlier within the vowel if the stressed syllable occurred earlier in the word.

While the results of these studies are not consistent, they all suggest an interaction of prosodic events. Rusaw [73] modelled the interaction between boundary and prominence with an artificial central pattern generator neural network model. In the model, prosodic boundaries increase the effects of prosodic prominence, achieving precisely the findings of Turk & Shattuck-Hufnagel [72] and Rusaw [73]. Within the Articulatory Phonology framework, Byrd & Riggs [15] suggested two possible accounts of their findings. Either the π-gesture shifts towards the prominent syllable, keeping the scope of lengthening constant, or, alternatively, the π-gesture extends towards the pre-boundary prominent syllable, which would mean that the scope of the π-gesture increases if the prominent syllable is further away from the boundary. In the most comprehensive account of prosodic boundaries to date, Katsika [14,66] argues that the boundary properties arise through the coordination of the μ-gesture, the boundary tone and the π-gesture. In this account, the π-gesture is coordinated with the phrase-final vowel gesture and it is also coordinated, although more weakly, with the temporal modulation gesture of the stressed syllable (the μ-gesture) of the phrase-final word. These coordinations account for the flexible scope of lengthening in that the onset of lengthening will depend on the position of the stressed syllable, i.e. the onset will shift towards the stressed syllable owing to the coordination of the π-gesture with the μ-gesture. The boundary tone gesture is activated when the π-gesture reaches a certain threshold activation level; that is, it will be activated only at strong boundaries (see also [52] for the relationship between π-gesture activation and boundary tone).⁵ The onset of the boundary tone will also, owing to its being triggered at a certain level of activation of the π-gesture, shift with stress: given that the π-gesture will reach its activation earlier if it starts earlier, the boundary tone will also be activated earlier, accounting for the observed boundary tone shift (for further discussion, see [14] and [66]). Thus, Katsika suggests that prosodic boundaries can be conceived of as a set of prosodic gestures that interact in a dynamic way [14,66]. Through this interaction, the temporal and tonal properties that characterize boundaries emerge.

These empirical studies [14,15,72–74] , the accounts within the Articulatory Phonology approach [14,15,66] and Rusaw's artificial central pattern generator neural network model [73] all show that prosodic events are interdependent and need to be investigated in a holistic manner for a full understanding of prosodic structure.

4. Cognitive functions of prosodic boundaries

(a). Pauses and speech planning

At boundaries, acoustic segments and articulatory gestures become longer (i.e. phrase-final and phrase-initial lengthening), and strong prosodic boundaries can contain pauses. Pauses are of particular interest because they can inform us about the cognitive function of prosodic boundaries in speech production, specifically about the speech planning processes. In what follows, we review work on pauses in speech planning and continue with an examination of how recent work on articulation during pauses has the potential to illuminate the process of speech planning and the distinction between prosodic boundaries and other types of pauses.

Boundaries mark linguistic organizational units or ‘constituents’ for both listeners and speakers. Listeners can use prosodic boundaries in sentence disambiguation (e.g. [31,75–80], for an overview, see [81]). In terms of the speaker, starting with work by Goldman Eisler (see [24] for a summary), it has been argued that speakers plan speech at prosodic boundaries. Thus, pause duration and sentence initiation times increase when the upcoming chunk of speech is syntactically more complex [27,82], prosodically more complex [29,30,83]), or when the upcoming chunk of speech is longer in terms of number of syllables [28–30,82,84,85]. Cooper & Paccia-Cooper [86] examined both pauses and final lengthening and found that upcoming syntactic complexity led to increased final lengthening, even though the pauses were short and did not themselves show lengthening. These findings indicate that pauses, and more generally prosodic boundaries, allow speakers to plan the upcoming chunk of speech; the more planning that chunk requires (owing to its complexity, length, etc.), the longer the pause (and the boundary) is going to be [24,29,30,82–88].

Planning is not the only factor affecting pause duration. As mentioned in §2, it is known that the structure of the utterance preceding the pause (and therefore not related to planning) also affects pause duration. This has been found for syntactic structure [86,89], for prosodic structure [29,30], and for phonological length [28–30]. It has been suggested that these effects are owing to the speaker's need to deactivate processed information [29,86,88].

Drawing on the distinction between structural and planning effects, Ferreira [82,87,88,91] suggests that pauses consist of two parts. The first part of the pause is the grammatical part, and the second is the planning, or performance, part. This suggestion is based in part on her findings that boundary-induced final lengthening is determined by the prosodic structure of the preceding phrase, rather than the syntactic complexity of the following phrase, indicating that final lengthening is not affected by speech planning [82,87]. By contrast, pause duration seems to be affected by both the complexity of the preceding phrase and the complexity of the upcoming phrase, indicating that planning takes place here. Thus, Ferreira suggests that final lengthening and pause duration are evidence of different properties, with final lengthening and the first part of the pause reflecting prosodic structure, and the second part of the pause reflecting the planning of the upcoming speech. Ferreira's findings [82,87] about final lengthening not being affected by upcoming material, and therefore not reflecting speech planning, are in contrast with findings from other studies (e.g. [86] see also [72]). It is also known that planning is a continuing process that takes place throughout the sentence [84,90,92,93] and not just at the boundary. Thus, the differential account of the lengthening and pausing phenomena put forth by Ferreira [88,91] might be too strong. Nevertheless, the idea that a boundary can be understood as divided into different ‘cognitive’ parts is intriguing.

Related to the question of the cognitive function of prosodic boundaries is the distinction between prosodic boundaries and disfluent pauses. Some cases are clear-cut (for example, pauses in read, highly practiced speech, are typically prosodic boundaries; similarly, pauses containing uh, um, are clearly disfluencies). A disfluency should not count as a boundary as it is an indicator of a breakdown in the planning or production process. However, the fact that planning also takes place at boundaries, and that not every disfluent pause contains uh or um, makes boundaries and disfluencies often difficult to distinguish. The question arises then whether there are articulatory correlates of the different processes taking place at boundaries. These aspects of pauses cannot be distinguished using acoustic data only (see also [88] on this point), as any articulatory behaviour indicative of these differences in the functions of the pauses will be too minute to observe in the acoustic signal. In order to learn more about the correlates of planning, a better understanding of articulation during prosodic boundaries is needed.

(b). Articulation in pauses

Advances in articulatory measurement techniques, particularly in real-time magnetic resonance imaging (MRI) [94], have made it possible to obtain a full view of the midsagittal vocal tract, which in turn allows for an adequate examination of changes that the vocal tract might be undergoing during pauses. A real-time MRI study of spontaneous speech by Ramanarayanan et al. [95,96] has found that the position of the articulators during grammatical pauses (defined as pauses that occur between syntactic constituents and excluding hesitation and word search pauses, in other words, pauses that can be assumed to be prosodic boundaries) show least variability, followed by ready-to-speak postures (i.e. postures of articulators that speakers adopt immediately before starting to speak, see also [97,98], followed by rest positions). As pointed out by Ramanarayanan et al. [95], lower variability can be understood as more control over the vocal tract, indicating that pauses at prosodic boundaries are planned, whereas the ready-to-speak pauses are less planned. In another real-time MRI study of spontaneous speech, Ramanarayanan et al. [99] found that grammatical pauses, but not ungrammatical ones (such as hesitation pauses), showed a significant decrease in speed of articulators at the pause compared with the pre-pausal region. The speed of articulators increased for the post-pause region for both types of pauses. Ungrammatical pauses also showed larger variability than grammatical pauses. These findings indicate, as Ramanarayanan et al. [99] point out, that grammatical, but not ungrammatical, pauses are planned. This work has the potential to indicate detailed vocal tract positioning during boundaries, which in turn could be taken as indicators of speech planning processes. For example, a decrease in the speed of articulators at the pause could be an indicator of the start of a predominantly planning stage in the speech production process, whereas an increase in the speed could be an indicator of a predominantly articulatory stage.

Katsika [14,66] has also examined articulation in pauses. She has identified for Greek a pause posture, which is a specific configuration of the vocal tract during grammatical pauses. The achievement of the pause posture and the end of phonation occur at a stable time from the onset of the boundary tone across all prosodic conditions examined. Katsika suggests that the pause posture is triggered by a π-gesture with a specific, very high activation level (a higher level than the one that triggers boundary tones), and therefore occurs only at strong intonation phrases and at a stable interval from the onset of the boundary tone. The achievement of the pause posture deactivates the glottal gestures (boundary tone and phonation). Katsika's [14,66] study adds a further articulatory component to the study of boundaries.

The studies of Ramanarayanan et al. [95,99] and Katsika [14] begin to illuminate articulation during pauses. Research of this kind will be crucial for gaining insights into the nature of prosodic boundaries and has already started informing us as to what constitutes a grammatical boundary and what is less planned articulatory behaviour during silent intervals.

(c). Gesturing and prosodic structure

In addition to research on vocal tract articulation, examining movement of other parts of the body, such as hand, head and torso movements at prosodic boundaries can shed further light on the properties of planning at prosodic boundaries and of disfluencies. We will refer to these movements as co-speech gestures or gesturing (to be distinguished from the linguistic gestures discussed so far). Gesturing research has typically focused on movement that contributes to the utterance meaning in some way (e.g. [100], p. 109 defines co-speech gestures as ‘actions that are treated by coparticipants in interaction as part of what a person meant to say’; see overview in [101] for what constitutes a gesture in gesturing research). Other kinds of movements have also been examined in relation to speech, such as finger tapping, head movements and torso movements.

Previous research has demonstrated the relevance of gesturing in communication [101–106]. A number of studies have examined the role of gesturing in speech planning, arguing that it facilitates lexical retrieval [107–109] and conceptual planning of an utterance [110]. Evidence that gesturing is related to speech planning is also found in studies examining speech errors. For example, in disfluent speech, gesturing is suspended before speech is interrupted and speakers hold their co-speech gestures during disfluencies [111]. Tiede et al. [112] found that head movement amplitude increased near speech errors.

Recently, gesturing has been examined in relation to prosodic structure. An early study by Kendon [102] shows that there is a correspondence between hand, arm and head movements and the prosodic and the discourse hierarchical organization (see also [113] for a relationship between discourse structure and gesturing). It has also been shown that manual gesturing and head movement is timed with prominent syllables [16,17,114–117]). Interestingly, co-speech gestures at pitch accented words are affected by prosodic boundaries in a manner parallel to the way pitch accents are affected by boundaries: in a study of Catalan, Esteve-Gibert & Prieto [118] found that, like intonation peaks in pitch accents, gesturing targets were produced earlier (retracted) when a prosodic boundary followed the prominent word. Furthermore, and unlike pitch accents, this retraction was also affected by a preceding prosodic boundary.

The above studies provide strong evidence for a link between prosodic structure and gesturing, but a number of questions remain. For example, it is not clear which part of the speech signal and which part of the co-speech gesture is targeted in alignment, although for the gesturing the apex seems to be emerging as the most relevant point (see [118,119]; the apex is the peak of a gesturing movement, e.g. the furthest point a finger reaches in a pointing movement). It is also unclear how systematic the co-occurrence of gesturing with prominence is in conversational speech (see also [116]). Thus, Ferré [120] in a study of conversational French found that only 171 of 1289 hand movements and 104 of 2520 head movements aligned with a prosodically prominent element. Similarly, for Dutch, Swerts & Krahmer [116] found that out of 228 head nods only 60 (26.3%) nods co-occurred with a strongly accented (i.e. highly prominent) word, whereas the remaining 168 nods occurred on words with a weak accent (124 nods) or no accent (44 nods). It is worth pointing out, however, that the most prominent words co-occurred with a head nod in 89.6% of the cases (60 of 67 words), whereas words without an accent co-occurred with a head nod in only 7.2% of the cases (44 of 609 words). For English, on the other hand, Loehr [115,121] found that most hand and head apexes occurred within approximately one word of the prominent word. Similarly, Mendoza-Denton & Jannedy [117] found a tight integration of gesturing and prominence in that almost all gesture apexes (in hand, arm and torso movements) occurred together with pitch accented words. It is worth keeping in mind that co-occurrences are difficult to interpret, because prosodically prominent elements might have other co-speech gestures associated with them (torso movements, eyebrow movements; on eyebrow movements see [116]) and on the other hand, gesturing might be marking prominence other than prosodic prominence (syntactic prominence for example, see discussion in [120]). However, the co-occurrence question will need to be addressed for a full understanding of the relationship between prosodic structure and gesturing.

Finally, work on finger tapping at prominent syllables has shown that the movement of both fingers and oral gestures lengthen under prominence [122,123]. In Dutch, Krahmer & Swerts [124] found that words exhibit acoustic properties of prominence (e.g. lengthening) when occurring with various types of gesturing (e.g. head nods and manual gestures). This work indicates that the influence of prosodic structure extends beyond vocal tract gestures and includes body movements.

(d). Gesturing at prosodic boundaries

The relationship between prosodic boundaries and gesturing has received little attention to date. A study by Barkhuysen et al. [125] found that Dutch speakers were better at distinguishing phrase-medial words from words at prosodic phrase boundaries when they could both see and hear speakers (rather than just seeing or just hearing the speakers). Barkhuysen et al. [125] further conducted an analysis of half of the visual stimuli they used in their study and found that facial gesturing differed between phrase-final and phrase-medial words (e.g. there were differences in the direction of the gaze and the head, in the position of the eyebrows and in the amount of blinking and nodding). Similarly in Japanese, Ishi et al. [126] found that head nods occurred frequently towards ends of strong prosodic phrases. Thus, there is evidence of boundary-related gesturing. There is also evidence of alignment of gesturing and boundaries. A study of one speaker by Shattuck-Hufnagel et al. [127] has found indication of alignment of intonation phrases and torso leans. Loehr [115,121] examined alignment of gesturing with intermediate phrases and stated that in the majority of the data, gesturing aligned closely with intermediate phrases. These studies lend strong support to the idea that body movements are in some way coordinated with prosodic phrasing. However, research in this area is only beginning. In the remainder of this section, we explore the question of whether properties of body movements at prosodic boundaries can inform us about the nature of linguistic boundaries. Specifically, the goal is to evaluate whether the questions raised in §4a about the distinction between grammatical and ungrammatical boundaries, and about different cognitive processes at boundaries, can be further illuminated by examining body movement. To begin addressing this issue, we present a small study of gesturing during pauses. The goal of this study, which uses data from Myers [128], is to examine whether body movements display lengthening at boundaries.

Myers [128] examined gesturing at pauses. She analysed short video clips of six speakers engaging in a spontaneous monologue (the speakers were answering a question). One of the six speakers was excluded, however, because she had only four gestures. For different speakers, different movements were analysed, depending on where an initial screening of the participants revealed the participants to gesture most. For two subjects, manual gesturing was analysed, and for three subjects, head movement was analysed. The movements were analysed irrespective of a possible contribution to meaning. The only movements that were not analysed were those that were clearly not related to speech (e.g. head scratching). Movements were analysed in terms of ‘onset’ and ‘target’ for each distinct motion. The onset was defined as the movement of the head or the hand as it leaves one position or changes direction from a previous movement, and the target was defined as the point when the hand or head stops moving or changes direction (in that case it would be also the onset of the next movement).⁶ In total, 82 pauses were analysed (19 pauses for speaker 1, 16 for speaker 2, 19 for speaker 3, 15 for speaker 4 and 13 for speaker 5). Myers divides movement at pauses as follows: bridging in movement, which is movement that starts during speech and ends in a pause, bridging out movement, which is movement that starts during a pause and ends in the following speech utterance, bridging movement (referred to as pre-to-post movement in [128]), which is movement that starts in the speech preceding the pause and ends in the speech following the pause, and in pause movement, which is movement that starts and ends in a pause. Myers finds that most of the movement is of the bridging out type, that is, starting in the pause and ending during the following speech part. She suggests that this might indicate that speakers use movement to help initiate speech, thereby functioning ‘as a cognitive bridge between pauses and fluent speech’ [128]. For additional analyses not reported here, she also classified the pauses as fluent—indicating a prosodic boundary—or disfluent, based on the perception of two listeners (Myers and one naive listener). The labellers agreed in 78.3% of the cases. In the case of disagreement, we have used Myers' scoring, as, based on a check of a sample of the data, her judgement seemed more accurate. Julia Myers has kindly provided her data to us for further analysis.

Dividing the movement data as they occurred between fluent and disfluent pauses shows that in disfluent pauses the most common gesturing type was again bridging out (26 gestures), followed by bridging in (18 gestures) and in pause (15 gestures), and the least common was bridging (five gestures). In fluent pauses, the most common type of gesturing was also bridging out (18), followed by bridging (14) and bridging in (13) and then in pause gesturing (6). The hand and head movements were further analysed for lengthening. If body movements behave like articulatory constriction gestures, in the sense that they are under the control of the π-gesture, then they should exhibit lengthening at boundaries, and the lengthening should increase as boundary strength increases. Thus, a positive correlation is expected between co-speech gesturing duration and pause duration: because more lengthening is expected at stronger boundaries, with an increase in boundary strength (evidenced by an increase in pause duration) co-speech gesturing should also increase in duration. In the case of bridging gestures, this correlation would be trivial, because they encompass the pause. Note, however, that the prediction is that lengthening will occur only at fluent pauses, because only a fluent pause can be considered to be a part of a prosodic boundary, whereas a disfluent pause cannot. While this is clearly too small a dataset to draw firm conclusions, we conducted a preliminary analysis on the relationship of these manual/head gesturing types with prosodic boundaries (except for bridging disfluent and in pause fluent gesturing, where there were fewer than 10 data points, five for bridging disfluent and six for in pause fluent).

To pool the speakers, pause durations were z-scored for each subject separately, as were gesturing durations. A regression analysis was conducted on each type of gesturing, examining whether pause duration (as an indicator of prosodic boundary strength) and gesturing duration are correlated. There is a nearly significant positive correlation only for the bridging in gesturing in fluent pauses (r² = 0.242, p = 0.0504). No other correlations were significant or even approached significance. Thus, there is suggestive evidence of lengthening of one kind of gesturing, namely the bridging in gesturing in fluent pauses. This suggests that, like oral constriction gestures, the movements at the end of prosodic phrases at large boundaries (which have fluent pauses) show lengthening. This in turn indicates—within the limitations that this is a very small set of data—that body movements are under the control of the π-gesture. The lack of a correlation with the bridging gesturing is surprising, given that this type of gesturing encompasses the pause. However, the pauses where this type of gesturing occurred were particularly short (−1.35 to 0.2 standard deviations), whereas the gesturing was not (0.35 to 2.5 standard deviations). This indicates that the gesturing was probably too long, and the boundaries too weak, for the lengthening effect of boundaries to become evident, even if lengthening of the manual/head movement took place. The lack of a correlation between bridging out gesturing and boundary strength suggests that the bridging out gesture is not controlled by prosodic structure. One explanation could be that bridging out gesturing is related to planning. It has been suggested that gesturing facilitates lexical retrieval [107,108,129] or is involved in the conceptual planning of an utterance [110]. Both events can be understood as non-structural and therefore might not be under control of the π-gesture.

One question that arises for future research is whether pause duration and gesturing could reflect different types of planning. For example, pause duration might reflect the planning process of the rough representation of the whole utterance, together with the full planning of an initial part of the utterance (as would be expected in Keating and Shattuck-Hufnagel's model of speech planning [93]). This could be considered linguistic planning. Hand and head movement could reflect conceptual planning or lexical retrieval, which are not structural in nature (see [85] on the related idea that pauses and F0 reflects different planning levels).

Work discussed in this section provides clear evidence that gesturing is related to linguistic structure but that this relationship needs to be further examined. Combining co-speech gesturing research with the work of Ramanarayanan et al. [95,96,99], which shows that articulation during pauses differs depending on the type of pause, we arrive at the following questions: (i) at which types of pauses does co-speech gesturing occur (e.g. is there a difference in co-speech gesturing at prosodic boundaries in comparison with disfluent pauses)? (ii) what kind of co-speech gesturing occurs with which kind of pauses (e.g. is there a difference in the occurrence of semantically emptier in comparison with semantically more specified co-speech gesturing at different kinds of pauses; is the use of different body parts in co-speech gesturing related to a specific type of pause)? (iii) how is co-speech gesturing coordinated with oral constriction gestures and prosodic gestures (i.e. where in the boundary or disfluent pause do co-speech gestures occur and with which specific linguistic gestures are they coordinated)? (iv) can we gain information about speech planning from the answers to the first three questions, in the sense that different types of planning (e.g. conceptual or linguistic) might be reflected in different types of co-speech gesturing or different coordinations? and (v) how are boundaries, prominence and gesturing integrated? The answers to these questions will inform us about what constitutes a prosodic boundary, how different linguistic gestures and co-speech gesturing combine to form a prosodic boundary, and what the role of co-speech gesturing is in speech planning. This information can in turn contribute significantly to understanding the distinction between prosodic boundaries and planning-driven behaviour.

5. Conclusion

Research on prosodic structure has made significant advances over the past decades and new directions of research are continually emerging. From the large number of fascinating findings and exciting research questions, we have selected a small set of questions that are centred on the coordination of events at prosodic boundaries. The emerging picture suggests that prosodic boundaries can be understood as arising from a set of coordinated prosodic gestures, namely π-gestures, boundary tone gestures, and μ-gestures [7,13,14,61]. Studies examining the relationship between prosodic structure and body movement provide evidence of their tight integration, and there is indication that some body movement is under the control of prosodic structure (e.g. [118,119,125] and the small analysis presented here). We suggest that further examination of the vocal tract gestures and body movements at prosodic boundaries promises to yield a fuller understanding of processes at prosodic boundaries and a distinction between structurally controlled and planning-driven behaviour. On the whole, research on gestural coordination at prosodic boundaries allows us to refine our understanding of prosodic structure and how it is reflected in and shaped by the nature of speech production processes.

Data accessibility

The conclusions in this paper are based on video data of speakers. As these videos contain identifying features, we cannot provide these videos as additional data. For more information, please contact the corresponding author.

Funding statement

Research supported by NIH grant DC002717 to Doug Whalen and NIH grant P01 HD 001994 to Jay Rueckl.