A Theoretical Framework for Human and Nonhuman Vocal Interaction

Abstract

Vocal communication is an important feature of social interaction across species; however, the relation between such behavior in humans and nonhumans remains unclear. To enable comparative investigation of this topic, we review the literature pertinent to interactive language use and identify the superset of cognitive operations involved in generating communicative action. We posit these functions comprise three intersecting multistep pathways: (a) the Content Pathway, which selects the movements constituting a response; (b) the Timing Pathway, which temporally structures responses; and (c) the Affect Pathway, which modulates response parameters according to internal state. These processing streams form the basis of the Convergent Pathways for Interaction framework, which provides a unified conceptual model for investigating the cognitive and neural computations underlying vocal communication across species.

INTRODUCTION

In humans, conversation is central to social interaction. Although speaking with a partner may seem effortless, this behavior requires complex sensory, cognitive, and motor function, and devastating disorders can result from brain damage affecting any aspect of these processes (Pedersen et al. 2004). However, the neural and behavioral mechanisms of interactive language use remain poorly understood due to two main factors. First, most speech production studies are performed in noninteractive contexts using controlled behaviors, and relatively few investigations focus on conversation (Castellucci et al., 2022). Consequently, the extent to which established models of speech production (Guenther 2016, Levelt et al. 1999, Redford 2015, Saltzman & Munhall 1989, Turk & Shattuck-Hufnagel 2019) can be applied to naturalistic language use is unclear. Second, the methodologies available for studying the human brain are limited in their ability to uncover circuit- and cellular-level insights into language generation. Meanwhile, a wider array of techniques can be leveraged to study nonhuman species that engage in flexible interactive communication (Pika et al. 2018) mediated by neural substrates with similarities to those related to human speech (Benichov et al. 2016, Miller et al. 2015, Okobi et al. 2019, Roy et al. 2016, Taglialatela et al. 2008). However, we lack a systematic method for relating vocal and social interaction across species – therefore, the specific parallels between these behaviors and human communication remain largely unknown.

To address these lacunae, we review the scientific literature concerning interactive language use to develop a conceptual framework constituting the cognitive processes required for human communication. We then demonstrate how various interactive behaviors across a range of species can be mapped onto individual components of the framework, thus providing direct connections between features of interactive language use and nonhuman vocal communication. We intend for this model to increase cross-talk between the fields of animal behavior, linguistics, and neuroscience and to enable the sophisticated technological and analytical tools of each discipline to advance our neurobiological understanding of language and social behavior.

A FRAMEWORK FOR HUMAN SPOKEN INTERACTION

In the context of human conversation, interactants produce a spoken response in reaction to sensory cues generated by a vocal partner. Enabling this motor response is a network of multi-step cognitive pathways which formulate a speaker’s intended message. We schematize this network with the Convergent Pathways for Interaction (CPI) framework (Figure 1), a conceptual model comprised of the cognitive modules we hypothesize to be critical for vocal communication.

Figure 1.

Diagram of the Convergent Pathways for Interaction framework, which depicts the set of cognitive processes involved in interactive vocal behaviors. The sensory information generated by a vocal partner or the environment is processed in a Sensory Comprehension module, and the generation of motor action is modeled as discrete Motor Processing modules. Between the sensory and motor components of the framework are three parallel pathways that shape an organism’s response (i.e., the Timing, Content, and Affect Pathways). Each pathway is composed of multiple discrete processing modules, and links between modules are labeled with numbers to facilitate discussion within the text.

The CPI framework is organized around three pathways which mediate different features of an interactant’s response. First, the Content Pathway encodes a response’s conceptual substance as action – enabling both linguistic and nonverbal communication (see Glossary) (Abner et al. 2015, Kendon 1997, Pezzulo et al. 2013). Second, the Timing Pathway controls response initiation, thus allowing for a high degree of coordination between speakers (Heldner & Edlund 2010, Levinson & Torreira 2015, Stivers et al. 2009) while also temporally structuring the individual elements within a response (e.g., syllables and words) (Shattuck-Hufnagel & Turk 1996). Third, the Affect Pathway modulates response production according to emotional and motivational state; in spoken language this concept most notably manifests as affective prosody (Banse & Scherer 1996, Hammerschmidt & Jurgens 2007). In the CPI framework, these pathways converge on motor processing operations to generate communicative actions.

The bulk of our review is dedicated to explaining the structure of the CPI framework and its relevance to interactive language use. We begin with an overview of the framework’s sensory and motor components, as these processes (e.g., sensory perception, articulation, motor control) have been longstanding topics of research in multiple disciplines. We then provide a focused discussion on the three aforementioned cognitive pathways, and we motivate the organization of each pathway (i.e., its modules and associated interconnections) using human behavioral and neuropsychological evidence. Finally, we conclude by considering specific nonhuman vocal behaviors through the lens of the CPI framework to identify potential parallels across species (Nieder & Mooney 2020, Petkov & Jarvis 2012).

SENSORY COMPREHENSION

To generate an appropriate response during spoken conversation, an interactant must first decode the communicative actions produced by their vocal partner. In this context, communication primarily occurs through acoustic means (i.e., via speech), but information is also conveyed through other channels, such as vision. For example, a speaker’s co-speech gestures (e.g., pointing and other hand movements) provide additional semantic or contextual information about the spoken message (Goldin-Meadow & Sandhofer 1999) and can indicate the proper timing for their partner to respond (Holler et al. 2018). Therefore, while perception of communicative actions can occur using a single sensory modality (e.g., sign language or telephone calls), it typically relies on multiple sensory streams (Gick & Derrick 2009, McGurk & MacDonald 1976).

The neuropsychological basis for sensory processing, especially as it relates to speech perception and language comprehension, is an active area of investigation in neuroscience and linguistics. Several influential models for speech processing exist (Fowler 1986, Hickok & Poeppel 2007, Kotz & Schwartze 2010, Marslen-Wilson 1975, McClelland & Elman 1986), and we direct readers to numerous informative reviews on this topic (Diehl et al. 2004, Fowler et al. 2016, Hickok & Poeppel 2016, Samuel 2011, Skipper et al. 2017). It is important to note that the processing of language relies on multiple sensory and comprehension-related operations (Davis et al. 2007, Obleser et al. 2007, Phillips 2001), which function in a highly interactive fashion (Heald & Nusbaum 2014, McClelland et al. 2006, Samuel 2011). Therefore, we model these operations with the combined Sensory Comprehension module in the CPI framework, which subsumes the processes required to decode a vocal partner’s message and relevant information from the environment. Therefore, the specific function of this module is to translate sensory stimuli into percepts in the brain of an interactant, which provide the necessary input for the downstream mechanisms enabling response generation.

Sensory comprehension is strongly influenced by top-down processes (Ganong 1980, McClelland et al. 2006). For instance, listeners can understand highly degraded speech (Miller et al. 1951, Stickney & Assmann 2001) and perceive phonemes that are not present in an acoustic signal if their occurrence is contextually predictable (Samuel 1996, Warren 1970). Such phenomena indicate that sensory processing is strongly biased by its expected conceptual content. Likewise, perception of sensory signals can be influenced by an interactant’s internal state (Kelley & Schmeichel 2014, Niedenthal & Wood 2019) as well as fluctuations in attention related to their decision to execute a response (Bronkhorst 2015, Cherry 1953, Talluri et al. 2018). These modulatory effects on perception are modeled in the CPI framework with a bidirectional connection between Sensory Comprehension and the Content, Timing, and Affect pathways (see Link 1 in Figure 1). The specific role of the framework’s individual modules in the comprehension process will be further discussed as each pathway is presented.

MOTOR PROCESSING

During interactive language use, the motor system controls the execution of communicative actions. Motor-related neural circuitry is organized into two subnetworks related to vocalization: a direct motor control system generating articulatory movements and an upstream system enabling volitional control over vocal production (Hage & Nieder 2016, Jurgens 2009, Nieder & Mooney 2020). Humans possess both circuits, which control the articulation of innate vocalizations—such as laughs and cries—and learned speech in a largely independent fashion (Jurgens 2009). The CPI framework models this functional and anatomical division within the vocal motor system as a two-step Motor Processing operation consisting of Motor Programming and Movement Encoding modules.

Motor Programming

In humans, representations of the speech articulators are observed in both motor cortex and brainstem motor nuclei (Bouchard et al. 2013, Zemlin 1998), and damage to either region can result in disorded articulation (Duffy 1995). The cortical aspects of the speech motor system are specifically involved in programming articulatory movements rather than activating musculature directly (Guenther 2016). For example, damage to speech-related premotor cortex can result in a selective impairment in the motor planning and sequencing of speech movements which largely spares the execution of nonverbal vocal tract movement (Deger & Ziegler 2002, Graff-Radford et al. 2014). However, this division between the programming and execution of action is not speech-specific but instead appears to be common to learned motor behaviors – a notion illustrated by the symptomology of ideomotor apraxia (Wheaton & Hallett 2007). We model this higher-level aspect of motor execution in the CPI framework with the Motor Programming module, which translates response plans – both vocal and nonvocal – into commands within the motor system.

Input to the Motor Processing modules is provided through the cognitive systems responsible for response formulation. In addition, several lines of evidence demonstrate that the sensory system can bypass these processing steps and feed into motor planning operations directly. For example, both infants and adults can imitate spoken material without understanding its linguistic content (Kuhl & Meltzoff 1996, Vaquero et al. 2017). Furthermore, because brain lesions can disrupt speech repetition but largely spare other language functions and visa-versa (Gorno-Tempini et al. 2008, Davis et al. 1978), it is likely that speech imitation and propositional speech rely on distinct neural circuitry. Likewise, speech output is unconsciously monitored to correct errors in real-time (Abbs & Gracco 1984, Burnett et al. 1998, Tourville et al. 2008) and to adapt speech motor programs when repeated sensory errors are detected (Houde & Jordan 1998, Lametti et al. 2012). These motor control processes rely heavily on interactions between the sensory and motor planning operations (Golfinopoulos et al. 2011, Guenther 2016) and further reinforce the tight link of these systems during speech production. In the CPI framework, we model this sensorimotor interfacing as a bidirectional connection between the Sensory Processing and Motor Programming modules (see Link 2 in Figure 1).

Movement Encoding

Prior to innervation of the musculature, the final processing step for centrally generated motor actions in vertebrates occurs in brainstem and spinal cord circuits (Jurgens 2009, Nielsen 2016). In the case of human speech production, neural signals from motor cortex converge along monosynaptic projections to vocal tract motor nuclei in the pons and medulla whose outputs drive articulatory movement (Zemlin 1998). Processing in these low-level motor circuits can operate independently from those in higher-order motor centers; for instance, motor cortical activity continues to reflect articulatory parameters in patients who are unable to speak due to brainstem stroke (Guenther et al. 2009). Conversely, brainstem vocal motor circuitry can drive the articulation of innate vocalizations in patients displaying disordered speech production due to lesions of motor cortical regions (Groswasser et al. 1988, Mao et al. 1989). In the CPI framework, we model this low-level motor processing step with the Movement Encoding module, which acts to transform the motor commands output by the Motor Programming step (see Link 3 in Figure 1) into patterns of muscle movement.

Sensory information can directly modulate subordinate motor execution processes as well as the higher-level operations modeled with the Motor Programming module. For example, humans unconsciously increase vocal intensity in the presence of background noise. This compensatory reflex—known as the Lombard effect (Lombard 1911)—is observed widely in learned and innate vocalizations across species and is thought ot be mediated by brainstem mechanisms (Luo et al. 2018). The CPI framework accounts for such phenomena by allowing the Sensory Comprehension module to influence the Motor Encoding step via a direct connection (see Link 4 in Figure 1). Together with the feedback connection targeting the Motor Programming step (see Link 2 in Figure 1), these sensorimotor interfaces provide a theoretical mechanism for interactants to communicate robustly by compensating for environmental and internal perturbations (Golfinopoulos et al. 2011, Tourville et al. 2008).

CONVERGENT COGNITIVE PATHWAYS

Having reviewed the sensory and motor components of the CPI framework, we now turn to the pathways that interconnect these processes and formulate communicative action. In the context of human conversation, interactants produce responses to their vocal partners that display three key features: they relay a conceptual message represented by articulatory movements and concurrent nonvocal actions (Content Pathway), they are timed appropriately (Timing Pathway), and their execution is modulated by affective state (Affect Pathway).

THE CONTENT PATHWAY

Once a partner’s message is processed by the Sensory Comprehension module, an interactant must select an appropriate response, prepare the actions comprising it, and potentially hold those actions in working memory. In the CPI framework, this translation from abstract meaning into intended movement proceeds via the Content Pathway, which is subdivided into three separate modules: Response Selection, Response Planning, and Plan Buffering.

Response Selection

To react to a vocal partner, an interactant must first select and formulate a conceptual (i.e., nonlinguistic) response (Levelt et al. 1999). Evidence for this abstract level of representation is provided by both neuropsychology and psycholinguistics. For example, lesions in lateral regions of frontal cortex can result in severe deficits to linguistic output while largely sparing the ability to comprehend speech and respond nonverbally (Corina et al. 1992, Graff-Radford et al. 2014, Groswasser et al. 1988, Marshall et al. 2004). In addition, humans have been found to process events independently of their native language’s word order, indicating the existence of a nonlinguistic level of representation (Papafragou & Grigoroglou 2019, Trueswell & Papafragou 2010) which furthermore appears to be accessed during speech production (Bunger et al. 2013).

In the CPI framework, we model the abstract formulation of a response’s conceptual content with the Response Selection module. This module receives input from the Sensory Comprehension step and also provides feedback to it (see Link 1 in Figure 1), consistent with previous research suggesting that action selection is highly interconnected with sensory perception via top-down and bottom-up mechanisms (Cisek & Kalaska 2005, Cisek & Kalaska 2010). In the context of interactive language use, we hypothesize that an interactant collects information from their partner’s communicative actions via Sensory Comprehension until the most appropriate response is selected for generation via separate downstream planning operations (de Ruiter 2000, Fuchs et al. 2013, Indefrey 2011, Lee et al. 2013, Meyer 1996, Smith & Wheeldon 1999). At the same time, an interactant’s expectations regarding the content of an appropriate response guides sensory processing via predictive mechanisms (Ganong 1980, McClelland et al. 2006, Miller et al. 1951, Samuel 1996, Stickney & Assmann 2001, Warren 1970).

Response Planning

Following the selection of a response, an interactant must encode its conceptual content into action. During conversation, these actions primarily consist of articulatory vocal tract movements and a variety of associated nonvocal actions (Abner et al. 2015, Goldin-Meadow & Sandhofer 1999, Kendon 1997, Pezzulo et al. 2013). Speakers use nonverbal communication, such as co-speech gestures (e.g., head nods, manual signs, and facial expressions), alongside their spoken output and can generate responses that are completely nonvocal (Kogure 2007). To enable this flexible usage of multimodal communicative actions, we posit a general-purpose Response Planning module in the CPI framework, which computes action plans from the conceptual-level output generated by the Response Selection step (see Link 5 in Figure 1).

Neuropsychological evidence suggests that the Response Planning module consists of distinct subprocesses. Specifically, language planning is at least partly compartmentalized from general premotor mechanisms, as demonstrated in patients with expressive aphasia (Corina et al. 1992, Graff-Radford et al. 2014, Groswasser et al. 1988, Marshall et al. 2004). Furthermore, different linguistic features are likely planned via distinct operations (Guenther 2016, Levelt et al. 1999). For example, both speech and sign language exhibit linguistic structure and linguistic prosody (Dachkovsky & Sandler 2009, Shattuck-Hufnagel & Turk 1996); disorders such as fluent aphasia can selectively affect a patient’s ability to generate linguistic content while leaving their prosody relatively unaffected (Seddoh 2004, Van Lancker Sidtis et al. 2010), thus suggesting these features are planned separately. Furthermore, linguistic structure is thought to be generated via serially organized processes encoding semantic, syntactic, and phonological structure (Fuchs et al. 2013, Indefrey 2011, Lee et al. 2013, Meyer 1996, Smith & Wheeldon 1999). The number and identity of language planning processes remains an active area of investigation in psycholinguistics and neurolinguistics, and we refer readers to several influential models and reviews on this topic (Guenther 2016, Levelt et al. 1999, Redford 2015, Saltzman & Munhall 1989, Turk & Shattuck-Hufnagel 2019). However, for the purpose of the CPI framework, we hypothesize that these speech planning functions as well as nonvocal planning operations are performed by the Response Planning module, which generates multimodal action plans that are then translated into low-level motor commands via Motor Programming.

Plan Buffering

Once the actions comprising a response have been planned, they can be executed as movement. However, an interactant may be required to temporarily buffer their planned response in order to properly time its execution (Levinson & Torreira 2015). For example, gaps between speakers engaged in conversational turn-taking are typically less than 200 ms in duration (Heldner & Edlund 2010, Stivers et al. 2009) – which is considerably shorter than reaction times typically observed in single-word production tasks (Bates et al. 2003). To achieve this degree of temporal coordination, interactants must plan their responses prior to the completion of their partner’s turn and store this plan until the appropriate time to respond (Bögels et al. 2015, Levinson & Torreira 2015). In the CPI framework, this working memory process is modeled with the Plan Buffering module, which is immediately downstream of Response Planning (see Link 6 in Figure 1) and acts to temporarily hold action plans before Motor Processing commences. As we discuss below, the Plan Buffering module is also necessary following the initiation of a spoken response, where it acts to maintain the complete outgoing message while the motor system serially reads out its individual components (e.g., syllables within a word) (Guenther 2016). The Plan Buffering module in the CPI framework therefore resembles working memory steps in other models (Baddeley 1992, Bohland et al. 2010); however, we hypothesize this module also stores nonvocal communicative actions (e.g., co-speech gestures) and therefore subsumes other domain-specific buffers (Poletti et al. 2008, Smyth & Pendleton 1989).

THE TIMING PATHWAY

During interactive language use, both the timing of response initiation (Levinson & Torreira 2015) and the internal temporal structure of a response (Shattuck-Hufnagel & Turk 1996) are used for communication (Breen et al. 2010, Mücke & Grice 2014, Pomerantz & Heritage 2013, Snedeker & Trueswell 2003). However, speech timing is controlled by mechanisms which appear largely distinct from the processes encoding its linguistic structure. For example, prelinguistic children can time vocal responses with near adult-like latencies (Jasnow & Feldstein 1986), and aphasics with severely disordered speech production can use laughter as responses during conversational turn-taking (Norris & Drummond 1998, Rohrer et al. 2009). These results indicate that linguistic content is not required for temporally coordinated vocal interactions. Conversely, cerebellar lesions can result in speech timing abnormalities but largely spare other features of linguistic structure (Ackermann et al. 1992), thus demonstrating that neurological damage can selectively disrupt the temporal structure of speech. Similarly, while the perturbation of speech-related motor cortex in neurosurgical patients via mild focal cooling results in articulatory disruptions, cooling upstream frontal areas instead modulates speech rate (Long et al. 2016). To account for these observations, the CPI framework includes a Timing Pathway separate from the Response Planning operations of the Content Pathway. The Timing Pathway is divided into two modules—Decision to Act and Motor Initiation—which are described below.

Decision to Act

During rapid conversational turn-taking, the high degree of temporal coordination across interactants is achieved via turn-end prediction, which relies on several linguistic features (de Ruiter et al. 2006, Holler et al. 2018). For example, articulation becomes systematically longer in duration as a speaker nears the end of a fluent utterance (Wightman et al. 1992), and interactants monitor their partner’s speech for this cue to precisely time their responses (Bögels & Torreira 2015). Likewise, speakers can rapidly suppress ongoing speech (Tilsen 2011), and they can use this abilty to prevent overlapping talk when faced with an interruption (Schegloff 2000).

In the CPI framework, the processes important for deciding when to initiate a response – such as turn-end prediction – is modeled with the Decision to Act module, which we posit to also include general-purpose decision-making operations (Rangel et al. 2008, Rilling & Sanfey 2011) in addition to language-specific ones. This hypothesis is supported by the existence of disordered turn-taking behavior in neurodevelopmental conditions typified by increased impulsivity, such as attention deficit hyperactivity disorder (Giddan 1991). Furthermore, as decision-making and predictive mechanisms are well known to modulate perceptual processes (Ganong 1980, Samuel 1996, Sohoglu et al. 2012), we model the connection between the Decision to Act and Sensory Comprehension modules as bidirectional (see Link 1 in Figure 1). We likewise postulate a bidirectional connection between the Decision to Act and Response Selection modules (see Link 1 in Figure 1) since active control of response onset can carry communicative value – for example, a speaker may delay response onset to signal that their response might be met with disapproval (Pomerantz & Heritage 2013).

Motor Initiation

After deciding to respond to a partner, an interactant must rapidly execute their planned response to achieve the precise inter-speaker coordination observed in human conversation (Levinson & Torreira 2015). Action initiation is thought to be controlled by cognitive operations which are separate from those underlying planning (Klapp & Maslovat 2020); likewise, the initiation of motor plans has been shown to rely on distinct neural circuitry (Bohland et al. 2010, Dacre et al. 2021, Dick et al. 2019, Guenther 2016, Zimnik et al. 2019). In the CPI framework, we model these processes with the Motor Initiation module, which we hypothesize acts to gate the release of response plans stored in Plan Buffering to the downstream Motor Programming module (see the ⊗ in Link 7 in Figure 1). To accurately time response onset, we therefore posit that the Decision to Act module provides direct input to the Motor Initiation module (see Link 8 in Figure 1).

During conversation, spoken responses typically consist of connected speech (Farnetani & Recasens 1997), and interactants can modulate the temporal structure of these multi-word responses to communicate (Breen et al. 2010, Mücke & Grice 2014). For example, speakers can produce the same series of words at a variety of rates (Goldstein et al. 2007) and can modify pause and word durations for emphasis or to signal differences in linguistic structure (Snedeker & Trueswell 2003). These behavioral features suggest that the mechanisms controlling speech timing receive input from the processes encoding the content of a response. In the CPI framework, we model this interaction between timing and planning functions as a direct connection between the Response Selection module in the Content Pathway and Motor Initiation (see Link 9 in Figure 1). This link therefore allows for the conceptual content of an interactant’s response to be signalled through a series of actions as well as its timing. Specifically, connected speech consists of multiple speech planning units (e.g., syllables) (Guenther 2016). During articulation, these planning units must be buffered as the motor system sequentially executes the vocal tract movements corresponding to each unit. We hypothesize the Motor Initiation module releases individual planning units stored in Plan Buffering to the Motor Programming step and thereby acts to encode the timing of each unit’s articulation. Through control of this mechanism by the Response Selection module, our model therefore provides a theoretical account for the generation of linguistic prosody as it relates to speech timing (Turk & Shattuck-Hufnagel 2019).

In the CPI framework, Motor Initiation operates on both vocal and nonvocal action plans in the Plan Buffering module; therefore, our framework predicts a multimodal initiation process for all communicative action plans. This hypothesis is supported by several experimental and clinical studies which have found a tight link between the timing of speech and nonspeech movements. For example, perturbation of cortical loci important for the initiation of speech (Ferpozzi et al. 2018) via direct electrical stimulation results in the inhibition or slowing of both articulation and hand movements (Breshears et al. 2019). Likewise, lesions of the supplemental motor area may result in slowed speech and nonvocal movements (Laplane et al. 1977). Finally, the kinematic properties of natural speech usage also support a domain-general initiation mechanism, as the execution of nonvocal movements has been found to be temporally coordinated to articulation and linguistic prosody in many contexts (Krivokapic et al. 2017, Mendoza-Denton & Jannedy 2011, Parrell et al. 2014, Shattuck-Hufnagel & Ren 2018). For example, prolongations in word duration that signal a prominent word are also realized on co-occurring manual gestures (Krivokapic et al. 2017).

THE AFFECT PATHWAY

During speech communication, the affective and physiological state of an interactant can strongly influence their behavior. For instance, a speaker’s internal state – which is informed by the content of their partner’s message – can directly affect their choice of response (Saslow et al. 2014). Likewise, a speaker’s emotional state is signalled by the affective prosody of their spoken responses (Banse & Scherer 1996, Hammerschmidt & Jurgens 2007). The processing of emotion and arousal occurs in limbic brain regions (LeDoux 2000) which are distinct from those underlying the volitional motor production (Hage & Nieder 2016, Jurgens 2009, Mao et al. 1989, Pichon & Kell 2013). For instance, damage to language-critical neural circuitry can severely disrupt speech production but largely spare the ability to communicate affective state using innate vocalizations (Fruhholz et al. 2015, Groswasser et al. 1988). This separation between emotional and linguistic aspects of human spoken communication is modeled in the CPI framework with the Affect Pathway. During interactive language use, this distinct pathway influences processing in the Content and Timing pathways via its ‘State Processing’ and ‘Emotional Readout’ modules.

State Processing

The internal state of an organism modulates a wide array of processes related to sensory perception (Kelley & Schmeichel 2014, Niedenthal & Wood 2019), motor behavior (Barrett & Paus 2002, Reilly et al. 1992), and decision-making (Lerner et al. 2015). In the context of human vocal communication, a speaker’s affective and physiological state can therefore be expected to influence how they perceive their partner’s communicative actions, as well as whether and how they respond. While it remains unclear how internal state specifically affects each of these processes individually, human interaction is known to be heavily modulated by factors related to emotion (Lopes et al. 2005, Weightman et al. 2014). Likewise, previous research has demonstrated that increased stress levels result in higher rates of disfluencies in stutterers (Blood et al. 1997) and reduced linguistic complexity in typical speakers (Saslow et al. 2014). These results indicate that language generation itself can be specifically influenced by a speaker’s affective state.

To account for the effects of emotional and physiological state on interactive behavior, the CPI framework models the processes involved in monitoring and regulating internal state with the State Processing module. While physiological state is expected to affect the brain and behavior globally (e.g., stress can affect working memory capacity) (Sorg & Whitney 1992), we explicitly model the aspects of this modulation that are particularly important for communicative action generation with connections from the State Processing module (see Link 1 in Figure 1) to: 1) Sensory Comprehension, allowing for the communicative message of a vocal partner(s) to affect an interactant’s state, 2) Response Selection, allowing for an interactant’s state to influence the content of their response, and 3) Decision to Act, which provides a mechanism for emotional state to influence whether and when to initiate a response. Finally, this four-way link is hypothesized to be bidirectional, allowing for internal state to shape comprehension processes (Kelley & Schmeichel 2014, Niedenthal & Wood 2019) and enabling language generation itself to affect an interactant’s own internal state (Lerner et al. 2015).

Emotional Readout

An interactant’s affective status can result in communicative action via two general mechanisms: 1) production of discrete behaviors and 2) modification of ongoing movement. In the first case, specific emotional or physiological states are biologically preprogrammed to result in innate vocalizations such as a groan (Hage & Nieder 2016, Jurgens 2009). These behaviors are generated through a circuit consisting of phylogenetically ancient limbic structures that directly project to brainstem motor nuclei (Jurgens 2009, Nieder & Mooney 2020). However, this system also mediates the execution of innate nonvocal behaviors such as defensive posturing (Deng et al. 2016, Evans et al. 2018), thus indicating that it is not specific to vocalization. To model the production of these affective vocalizations, the CPI framework allows for specific internal states to be read out as distinct behavioral units via a direct link between the Emotional Readout and Movement Encoding modules (see Link 11 in Figure 1).

While innate vocalizations can be used communicatively during interaction (Vettin & Todt 2004), the more prevalent reflection of internal state in human communication is affective prosody, which is realized as modulations in prosodic features such as pitch, duration, and voice quality according to a speaker’s emotional status (Banse & Scherer 1996, Barrett & Paus 2002, Hammerschmidt & Jurgens 2007). Although superficially related to linguistic prosody, the generation of affective prosody appears independent of operations encoding the linguistic structure of speech. For example, akinetic mutism can resolve over longer timescales such that patients recover their ability to speak, albeit with disordered affective prosody (Jurgens & von Cramon 1982). Similarly, the speech of schizophrenic individuals has been observed to display abnormal affective prosody, while linguistic prosody is generally unaffected (Lucarini et al. 2020, Murphy & Cutting 1990).

While most studies of affective prosody focus on its vocal properties, a speaker’s internal state is also signaled during natural language production by nonverbal movements such as facial expressions (Attardo et al. 2011, Gironzetti et al. 2016). Furthermore, a speaker’s affective status can also influence the execution of ongoing nonvocal actions. For example, co-speech gestures of the head display increased velocity when a speaker is experiencing stress (Giannakakis et al. 2018). Likewise, affective prosody is realized in sign language through modulations of linguistically meaningful movements of the head and face (Hietanen et al. 2004, Reilly et al. 1992). Such phenomena demonstrate that emotional processes that generate affective prosody influence both vocal and nonvocal aspects of communicative motor behavior. In the CPI framework, affective state exerts modulatory effects on all communicative actions through the Emotional Readout module, which influences the generation of motor commands by the Motor Programming step (see Link 12 in Figure 1).

APPLICATION OF THE CPI FRAMEWORK TO NONHUMAN VOCAL INTERACTIONS

In the CPI framework, the generation of communicative action is schematized as a set of modules constituting distinct yet interacting pathways. These modules and their connections were motivated using behavioral and neuropsychological features of human spoken communication, but this model can also be used to contextualize the vocal behaviors of other organisms (Figure 2). The CPI framework therefore provides a means for systematically identifying analogous cognitive processes across species, which may in turn rely on shared neural mechanisms. To demonstrate this important feature of the CPI framework, we will now discuss how various communicative behaviors across species can be related to specific model components and pathways.

Figure 2.

Using the Convergent Pathways for Interaction Framework to investigate vocal communication in nonhuman species. Behaviors across four organisms are highlighted: (a) house mouse, (b) zebra finch, (c) singing mouse, and (d) vervet monkey. An example sonogram depicting the vocal behavior of each species is shown at the top of each panel. Below the sonograms, the aspects of the framework we hypothesize to be recruited by each vocal behavior are highlighted. The audio data in panel a was collected by Gregg Castellucci and in panels b and c by Michael Long. The finch and singing mouse images were adapted from artwork created by Julia Kuhl. The vervet monkey image and sonogram in panel d were provided by Julia Fischer (German Primate Center). The framework shown below each sonogram is illustrated in greater detail in Figure 1.

Affective Vocal Production (Mus musculus)

The production of innate vocalizations can be automatically driven by factors related to an organism’s affective state (Hage & Nieder 2016, Jurgens 2009). These affective vocalizations are widespread among vertebrates and have been well studied in laboratory mice (Mus musculus), which generate a variety of ultrasonic and audible vocalizations (Portfors 2007). While mice have not been observed to use their calls in structured vocal interactions (e.g., turn-taking), these vocalizations are used communicatively and elicit complex behavioral responses from conspecifics (Ehret 1992, Ehret & Bernecker 1986, Hammerschmidt et al. 2009, Liu et al. 2013). Mice therefore represent a valuable species for the study of innate calls reflecting emotional state, which are modeled by the CPI framework as arising through the Affect Pathway.

Recent studies have demonstrated that stimulation of vocalization-related neurons in the periaqueductal gray (PAG) of male mice results in the production of stereotyped courtship calls (Chen et al. 2021, Tschida et al. 2019), while stimulation of upstream centers in the lateral preoptic area leads to vocalizations that vary naturalistically in amplitude and duration via modulation of PAG dynamics (Chen et al. 2021). Likewise, stimulation of amygdalar regions projecting to PAG can suppress vocal output (Michael et al. 2020). In the CPI framework, modulation of PAG by these upstream regions may reflect State Processing inputs to the Emotional Readout module; as PAG in turn projects to the brainstem pattern generators that drive vocal production (Jurgens 2009), we likewise hypothesize that this circuit corresponds to the direct input from the Emotional Readout module to Movement Encoding (Figure 2a). Further research is needed to determine whether these connections represent universal mechanisms enabling an organism’s internal state to influence the production of preprogrammed behaviors. However, we anticipate such investigations to be broadly relevant given that affective state drives and modulates innate vocal output in a range of organisms, including humans (Liao et al. 2018, Stewart et al. 2013).

Forebrain Vocal Motor Processing (Taeniopygia guttata)

Human speech production relies on a two-step motor system consisting of a cortical network converging onto brainstem motor regions (Hage & Nieder 2016, Jurgens 2009). Unlike mice (Hammerschmidt et al. 2015, Nieder & Mooney 2020), humans and songbirds – such as the zebra finch (Taeniopygia guttata) – exhibit this hierarchical relationship. In humans and finches, lesions of this vocal forebrain system lead to selective dysfunction in the production of speech and courtship song, respectively, while the ability to produce other innate vocalizations is largely unaffected (Groswasser et al. 1988, Mao et al. 1989, Simpson & Vicario 1990). Furthermore, experimental perturbations of these forebrain regions with electrical stimulation (Penfield & Boldrey 1937, Vu et al. 1994) and focal cooling (Long & Fee 2008, Long et al. 2016) can result in similar behavioral outcomes during vocalization. In the CPI framework, this forebrain control over speech and song is hypothesized to correspond to the Motor Programming module, which outputs motor commands to the Motor Encoding module to drive articulatory musculature.

The common organization of the zebra finch and human vocal motor pathway is further reflected by several intriguing parallels. For example, zebra finch courtship song – like human speech – is learned primarily through vocal imitation (Tchernichovski et al. 2001), which may be enabled in the CPI framework via direct input from Sensory Comprehension to the Motor Programming module (Figure 2b). The connection between these modules also provides a theoretical account for the ability of finches to execute compensatory articulatory movements when sensory feedback is perturbed (Leonardo & Konishi 1999, Sober & Brainard 2009, Tumer & Brainard 2007), which we hypothesize to underlie similar speech motor control processes in humans (Abbs & Gracco 1984, Burnett et al. 1998, Houde & Jordan 1998, Lametti et al. 2012, Tourville et al. 2008). Furthermore, involvement of the Motor Programming module in birdsong production allows for vocal output to be modulated by the animal’s internal state in a manner parallel to the generation of affective prosody (Figure 2b). Consistent with this prediction, finches exhibit context-dependent changes in singing behavior; for example, songs directed to females differ in acoustic structure from those produced in isolation (Sossinka & Böhner 1980). Taken together, these features of the zebra finch song system reinforce its utility for examining how forebrain motor circuits control vocal behavior and are modulated by affective processes.

Coordinated Vocal Timing (Scotinomys teguina)

A key feature of human vocal interaction is rapid turn-taking between speakers (Stivers et al. 2009). However, such vocal turn-taking is not exhibited by house mice or observed during zebra finch courtship song production, although it is common in various forms across other species (Pika et al. 2018). For example, a form of antiphonal calling in the neotropical singing mouse (Scotinomys teguina) called counter-singing displays a high degree of temporal coordination between vocal partners . Specifically, singing mice produce extended advertisement calls (~10 s in duration) that can elicit a vocal response from conspecific males within hundreds of milliseconds. This vocal behavior is mediated by forebrain motor circuits, and an animal’s ability to rapidly respond to a partner is abolished when this system is inactivated (Okobi et al. 2019). However, perturbations of the forebrain vocal regions do not eliminate spontaneous calling or disrupt their articulation, indicating that brainstem motor circuits encode the vocal tract movements required for advertisement call production (Okobi et al. 2019).

The forebrain control of vocalization timing in singing mice can be modeled with the Motor Programming module, which provides commands to the Motor Encoding step (i.e., brainstem structures) to drive call articulation. According to the CPI framework, engagement of the Motor Programming module enables the Timing Pathway to provide input to the vocal motor system (Figure 2c), thus allowing for response timing to be controlled by decision-making operations (i.e., Decision to Act). For example, during counter-singing, the singing mouse must recognize and monitor a partner’s call for acoustic features (e.g., elapsed time, song termination) relevant for response initiation. Once these cues have been detected, an affirmative Decision to Act can be made and Motor Initiation of downstream Motor Processing steps can commence. Similarly, recent work examining the usage of short, largely innate calls in the zebra finch has demonstrated that the forebrain song system is also required to finely time vocal initiation during interaction (Benichov et al. 2016). Therefore, Timing Pathway control over innate vocal behavior appears to be present in zebra finches (Figure 2b) and may represent a widespread phenomenon for precise response timing during interaction.

Response Selection (Nonhuman Primates and Beyond)

Human language is symbolic, with individual words linked to conceptual information through experience (Woodward & Markman 1998). The extent to which singing mice and zebra finches learn to associate their vocalizations to distinct concepts is unknown; however, a limited version of this phenomenon is observed in nonhuman primates. Specifically, some primate species possess vocal systems acquired through usage learning, where animals learn to produce a call in a specific context (Gultekin & Hage 2017, Seyfarth et al. 1980, Wegdell et al. 2019). While words are further linked by semantic and syntactic relations in human language, these indexical calls may represent the precursors of symbolic vocal use (Nieder 2009). For example, vervet monkeys produce distinct warning calls for different types of predators (Seyfarth et al. 1980); while the warning calls themselves are preprogrammed, juvenile vervets learn to produce them in response to the appropriate stimulus (Seyfarth & Cheney 1986). This behavior suggests that vervets possess distinct conceptual representations of predator classes and can select the warning call corresponding to each class. The neurophysiology underlying vocal usage learning and the generation of calls acquired through this process is poorly understood; however, juvenile macaques have been trained to produce their innate vocalizations in response to an arbitrary cue (Coude et al. 2011, Gavrilov et al. 2017, Hage & Nieder 2013, Hage et al. 2013). In this behavior, premotor activity is observed prior to the production of cued calls but not before the same calls are produced spontaneously (Coude et al. 2011, Gavrilov et al. 2017), suggesting that cortical input mediates the volitional control of call production (Hage & Nieder 2016). Similar premotor activity has been observed in marmosets during antiphonal calling (Miller et al. 2009b, 2015; Roy et al. 2016), indicating that this forebrain pathway is also important for vocal control during interaction. In the CPI framework, volitional selection and regulation of specific vocal outputs is hypothesized to occur through the Response Selection module; therefore, nonhuman primates may provide a valuable model to examine the neural correlates of the Content Pathway (Figure 2d), including during vocal interaction (Lemasson et al. 2011, Miller et al. 2009a).

Humans can learn new words and speech sounds even as adults (Brysbaert et al. 2016), however vocal usage in nonhuman primates is limited to a species-specific repertoire of innate calls (Hage & Nieder 2016). Consequently, additional model organisms are required to study how conceptual information can be linked to novel vocal elements. For example, several avian species are capable of both vocal usage learning and imitation, including corvids (Brecht et al. 2019) and parrots (Osmanski et al. 2021, Pepperberg 2002). Parrots in particular can associate numerous human words to objects (Pepperberg 2002), demonstrating their ability to signal an object’s mental representation with learned vocal output. The neural mechanisms enabling this behavior remain unknown, but investigation of this topic may lead to important insights into how conceptual processes related to meaning (i.e., Response Selection) interact with forebrain motor processes (i.e., Motor Programming) – which is crucial for human language.

CONCLUSION

In this review, we synthesize research from the fields of linguistics, neuropsychology, animal behavior, and systems neuroscience with the goal of uncovering a species-general cognitive architecture underlying communicative action. The CPI framework represents the first step in this effort, and we anticipate that future findings will update this model by providing additional detail to specific modules or uncovering modules and connections that were not originally included. Overall, the development of the CPI framework is intended to foster communication across disciplines by facilitating cross-species comparative investigations of communicative behaviors and their neural dynamics. By integrating studies from a variety of model organisms, we can achieve a more mechanistic understanding of the circuit elements (e.g., cell types, synaptic connectivity) underlying vocal interaction that is presently unavailable for the human brain. The CPI framework therefore provides an essential device for organizing and contextualizing these efforts.

GLOSSARY

Affective Prosody

Acoustic cues, such as modulations in pitch, amplitude, and voice quality, which reflect a speaker’s emotional state

Akinetic Mutism

An acquired neurological condition characterized by a dramatic reduction in the motivation to speak and generate movements

Antiphonal Calling

A behavior in which multiple organisms produce alternating vocalizations, usually over a distance

Communicative Action

All vocal and nonvocal motor behaviors used by an interactant to communicate

Connected Speech

Spoken series consisting of multiple words

Expressive Aphasia

An acquired neurological condition characterized by severely disrupted spoken and sign language but largely preserved nonlinguistic action generation

Fluent Aphasia

An acquired neurological condition typified by the production of fluent sounding speech which lacks proper linguistic structure

Ideomotor Apraxia

An acquired neurological condition typified by deficits in the motor planning of learned skills and communicative gestures

Innate Vocalizations

Vocalizations whose production is genetically programmed in a species and thus do not need to be learned

Linguistic Communication

Communication through language, including spoken language and sign language

Linguistic Prosody

Features of an utterance’s words which relate to its linguistic meaning, including amplitude, pitch, and timing (see Timing Pathway)

Linguistic Structure

Grammatical features of an utterance’s words, including: semantics (meaning), syntax (hierarchically organized word-order), morphology (word-internal structure), and phonology (abstract sound-structure)

Nonverbal Communication

Nonlinguistic communication used to convey information, including nonvocal behaviors (e.g., co-speech gestures, body language) and nonspeech vocalizations (e.g., crying, laughing)

Phonology

Subfield of linguistics dedicated to the study of language’s abstract sound-structure rather than its articulatory realization

Planning Unit

Linguistic units used during speech planning. Speech production models posit different planning units, such as syllables, words, and fluent phrases

Propositional Speech

Meaningful linguistic output whose content is generated voluntarily and spontaneously

Semantics

Subfield of linguistics dedicated to the study of meaning

Syntax

Subfield of linguistics dedicated to the study of the hierarchical rules governing word order

Turn-end Prediction

A predictive process used by human interactants to anticipate the end of their partner’s spoken turn

Turn-Taking

The behavior exhibited during conversation in which interactants attempt to respond to vocal partners quickly while avoiding overlap