Neural Modeling and Imaging of the Cortical Interactions Underlying Syllable Production

Abstract

This paper describes a neural model of speech acquisition and production that accounts for a wide range of acoustic, kinematic, and neuroimaging data concerning the control of speech movements. The model is a neural network whose components correspond to regions of the cerebral cortex and cerebellum, including premotor, motor, auditory, and somatosensory cortical areas. Computer simulations of the model verify its ability to account for compensation to lip and jaw perturbations during speech. Specific anatomical locations of the model’s components are estimated, and these estimates are used to simulate fMRI experiments of simple syllable production.

Introduction

The advent of functional brain imaging techniques that are safe for use on human subjects has led to an explosion in the amount of data concerning brain activity during speech and language tasks. The current article details a neural model of speech production that provides a conceptual and computational framework for interpreting many of these datasets. The model is a neural network model of speech acquisition and production, called the DIVA model (Directions Into Velocities of Articulators), that utilizes a babbling cycle to learn to control movements of simulated speech articulators in order to produce phoneme strings. Over the past decade, our laboratory has used numerical simulations to show how the model provides a relatively simple, unified account of a very wide range of speech production phenomena, including motor equivalence, contextual variability, anticipatory and carryover coarticulation, velocity/distance relationships, speaking rate effects, and speaking skill acquisition (e.g., Guenther, 1994; Guenther, 1995; Guenther, Hampson, & Johnson, 1998; Guenther & Ghosh, 2003; Nieto-Castanon, in press). Predictions concerning speech production in normal adults have been drawn from the model and tested using electromagnetic articulometry (e.g., Guenther et al., 1999; Perkell et al., 2004). The model has been used to account for issues in child development (e.g., Guenther, 1995), including a demonstration of its ability to deal with the dramatic changes in size and shape of the speech articulators that take place during the first three years of life (Callan, Kent, Guenther, & Vorperian, 2000). The model has also been used to investigate the role of auditory feedback in speech production in normally hearing individuals, deaf individuals, and individuals who have recently regained some hearing through the use of cochlear implants (Perkell et al., 2000), and to investigate stuttering (Max, Guenther, Gracco, Ghosh, & Wallace, 2004). Because the DIVA model is defined as a neural network, its components can be interpreted in terms of brain function in a straightforward way. The model thus provides an ideal framework for interpreting data from functional imaging studies of the human brain during speech tasks. Preliminary associations of the model’s components with specific brain regions have been presented elsewhere (e.g., Guenther, 1998; Guenther, 2001; Guenther et al., 2003); a primary goal of the current paper is to provide a more thorough treatment of the hypothesized neural bases of the model’s components.

A second purpose of the current work is to extend the model to incorporate realistic neural processing delays, and therefore more realistically address the issue of combining feedforward and feedback control strategies. Earlier versions of the DIVA model effectively assumed instantaneous transmission of neural signals. However the nervous system must cope with potentially destabilizing delays in the control of articulator movements. For example, a motor command generated in the primary motor cortex will typically take 40 ms or more before it effects movement of the associated speech articulator. Similarly, sensory information from the articulators and cochlea are delayed by tens of ms before they reach the primary sensory cortices. These transmission delays can be very problematic for a system that must control the rapid articulator movements underlying speech. Most adults can pronounce the word “dilapidated” in less than one second; this word requires 10 transitions between phonemes, with each transition taking less than 100ms to complete. A purely feedback-based control system faced with the delays mentioned above would not be able to stably produce speech at this rate. Instead, our speech production system must supplement feedback control with feedforward control mechanisms. In this article we address the integration of feedback and feedforward control subsystems in the control of speech movements with realistic processing delays, and we provide model simulations of perturbation studies that probe the temporal response properties of feedback control mechanisms.

Several aspects of the DIVA model differentiate it from other models in the speech production literature (e.g., Levelt, Roelofs, & Meyer, 1999; Saltzman & Munhall, 1989; Morasso, Sanguineti, & Frisone, 2001; Westermann & Reck, 2004). Whereas the Levelt, Roelofs, & Meyer (1999) model focuses on linguistic and phonological computations down to the syllable level, the DIVA model focuses on the sensorimotor transformations underlying the control of articulator movements. Thus, the DIVA model focuses on speech control at the syllable and lower motor levels. The task dynamic model of Saltzman et al. (1989) is a computational model that provides an alternative account of the control of articulator movements. However, unlike the DIVA model its components are not associated with particular brain regions, neuron types, or synaptic pathways. Of current biologically plausible neural network models of speech production (e.g., Morasso et al., 2001; Westermann et al., 2004), the DIVA model is the most thoroughly defined and tested, and it is unique in using a pseudoinverse-style control scheme (from which the model’s name is derived) that has been shown to provide accurate accounts of human articulator kinematic data (e.g., Guenther et al., 1998; Guenther et al., 1999; Nieto Castanon et al., in press). It is also unique in using a combination of feedback and feedforward control mechanisms (as described in the current article), as well as embodying a convex region theory for the targets of speech that has been shown to provide a unified account of a wide body of speech acoustic, kinematic, and EMG data (Guenther, 1995).

An overview of the DIVA model and description of its components are provided in the next section. Subsequent sections relate the model’s components to regions of the cerebral cortex and cerebellum, including mathematical characterizations of the model’s components and treatment of the relevant neurophysiological literature. Computer simulations of the model producing normal and perturbed speech are then presented, followed by a more precise account of fMRI activations measured during simple syllable production in terms of the model’s cell activities.

OVERVIEW OF THE DIVA MODEL

The DIVA model, schematized in Figure 1, is an adaptive neural network that learns to control movements of a simulated vocal tract, or articulatory synthesizer (a modified version of the synthesizer described by Maeda, 1990), in order to produce words, syllables, or phonemes. The neural network takes as input a speech sound string and generates as output a time sequence of articulator positions that command movements of the simulated vocal tract. Each block in the model schematic (Figure 1) corresponds to a set of neurons that constitute a neural representation. In this article, the term map will be used to refer to such a set of cells. The term mapping will be used to refer to a transformation from one neural representation to another (arrows in Figure 1), assumed to be carried out by filtering cell activations in one map through synapses projecting to another map. The synaptic weights are tuned during a babbling phase in which random movements of the speech articulators provide tactile, proprioceptive, and auditory feedback signals that are used to learn the mappings between different neural representations. After babbling, the model can quickly learn to produce new sounds from audio samples provided to it, and it can produce arbitrary combinations of the sounds it has learned.

Figure 1.

Hypothesized neural processing stages involved in speech acquisition and production according to the DIVA model. Projections to and from the cerebellum are simplified for clarity.

In the model, production of a phoneme or syllable starts with activation of a speech sound map cell, hypothesized to lie in ventral premotor cortex, corresponding to the sound to be produced. After a speech sound map cell has been activated, signals from premotor cortex travel to the auditory and somatosensory cortical areas through tuned synapses that encode sensory expectations for the sound. Additional synaptic projections from speech sound map cells to the model’s motor cortex (both directly and via the cerebellum) form a feedforward motor command.

The synapses projecting from the premotor cortex to auditory cortical areas encode an expected auditory trace for each speech sound. They can be tuned while listening to phonemes and syllables from the native language or listening to correct self-productions. After learning, these synapses encode a spatiotemporal target region for the sound in auditory coordinates. During production of the sound, this target region is compared to the current auditory state, and any discrepancy between the target and the current state, or auditory error, will lead to a command signal to motor cortex that acts to correct the discrepancy via projections from auditory to motor cortical areas.

Synapses projecting from the premotor cortex to somatosensory cortical areas encode the expected somatic sensation corresponding to the active syllable. This spatiotemporal somatosensory target region is estimated by monitoring the somatosensory consequences of producing the syllable over many successful production attempts. Somatosensory error signals are then mapped to corrective motor commands via pathways projecting from somatosensory to motor cortical areas.

Feedforward and feedback control signals are combined in the model’s motor cortex. Feedback control signals project from sensory error cells to the motor cortex as described above. These projections are tuned during babbling by monitoring the relationship between sensory signals and the motor commands that generated them. The feedforward motor command is hypothesized to project from ventrolateral premotor cortex to primary motor cortex, both directly and via the cerebellum. This command can be learned over time by averaging the motor commands from previous attempts to produce the sound.

The following sections present the model’s components in further detail, including a mathematical characterization of the cell activities in the cortical maps and a treatment of relevant neuroanatomical and neurophysiological findings (with a more detailed neurophysiological treatment provided in Appendix A). For purposes of exposition, the model’s premotor and motor cortical representations will be treated first, followed by treatments of the feedback and feedforward control subsystems.

MOTOR AND PREMOTOR REPRESENTATIONS

Premotor Cortex Speech Sound Map

Each cell in the model’s speech sound map, hypothesized to correspond to neurons in the left ventral premotor cortex and/or posterior Broca’s area¹, represents a different speech sound². A “speech sound” is defined here as a phoneme, syllable, word, or short phrase that is frequently encountered in the native language and therefore has associated with it a stored motor program for its production. For example, we expect all phonemes and frequent syllables of a language to be represented by unique speech sound map cells. In contrast, we expect that infrequent syllables do not have stored motor programs associated with them; instead we expect they are produced by sequentially instating the motor programs of the phonemes (or other sub-syllabic sound chunks, such as demisyllables cf. Fujimura & Lovins, 1978) that form the syllable. In terms of our model, infrequent syllables are produced by sequentially activating the speech sound map cells corresponding to the smaller sounds that make up the syllable.

Speech sound map cells are hypothesized to lie in ventral premotor cortex because of their functional correspondence with “mirror neurons.” Mirror neurons are so termed because they respond both during an action and while viewing (or hearing) that action performed by another animal or person (Rizzolatti et al., 1996; Kohler et al., 2002). These cells have been shown to code for complex actions such as grasping rather than the individual movements that comprise an action (Rizzolatti et al, 1988). Neurons within the speech sound map are hypothesized to embody similar properties: activation during speech production drives complex articulator movement, and activation during speech perception tunes connections between the speech sound map and sensory cortex (described further below; see Arbib, in press for a different view of the role of mirror neurons in language.)

Demonstrations of mirror neurons in humans have implicated left precentral gyrus for grasping actions (Tai, Scherfler, Brooks, Sawamoto, & Castiello, 2004), and left hemisphere opercular inferior frontal gyrus for finger movements (Iacoboni et al., 1999). Recently, mirror neurons related to communicative mouth movements have been found in monkey area F5 (Ferrari et al., 2003) immediately lateral to their location for grasping movements (Di Pellegrino, Fadiga, Fogassi, Gallese, & Rizzolatti, 1992). This area has been proposed to correspond to the caudal portion of ventral inferior frontal gyrus (Brodmann’s area 44) in the human (see Rizzolatti & Arbib, 1998). We therefore propose that the speech sound map cells lie in ventral lateral premotor areas of the left hemisphere³, including posterior portions of the inferior frontal gyrus.

The equation governing speech sound map cell activation in the model is:

$$\begin{array}{l}{P}_i(t)=1if\hspace{0.17em}{i}^{th}\hspace{0.17em}sound is being produced or perceived\\ P_i(t)=0otherwise\end{array}$$ (1)

Each time a new speech sound is presented to the model (as an acoustic sample) for learning, a new cell is recruited into the speech sound map to represent that sound. There are several aspects to this learning, described further below. After the sound has been learned, activation of the speech sound map cell leads to production of the corresponding sound via the model’s feedforward and feedback subsystems.

The model’s speech sound map cells can be interpreted as forming a “mental syllabary” as described by Levelt and colleagues (e.g., Levelt & Wheeldon, 1994; Levelt et al., 1999). Levelt et al. (1999) describe the syllabary as a “repository of gestural scores for the frequently used syllables of the language” (p. 5). According to our account, higher-level brain regions involved in phonological encoding of an intended utterance (e.g., anterior Broca’s area) sequentially activate speech sound map cells that correspond to the syllables to be produced. The activation of these cells leads to the readout of feedforward motor commands to the primary motor cortex (see Feedforward Control Subsystem below), as well as a feedback control command if there is any error during production (see Feedback Control Subsystem). The feedforward command emanating from a speech sound map cell can be thought of as the “motor program” or “gestural score”, i.e., a time sequence of motor gestures used to produce the corresponding speech sound (cf. Browman & Goldstein, 1989).

According to the model, when an infant listens to a speaker producing a new speech sound, a previously unused speech sound map cell becomes active, and projections from this cell to auditory cortical areas are tuned to represent the auditory signal corresponding to that sound. The projections from the premotor speech sound map cells to the auditory cortex represent a target auditory trace for that sound; this auditory target is subsequently used in the production of the sound (see Feedback Control Subsystem below for details), along with feedforward commands projecting from the speech sound map cell to the motor cortex (detailed in Feedforward Control Subsystem below).

Motor Cortex Velocity and Position Maps

According to the model, feedforward and feedback-based control signals are combined in motor cortex. Three distinct subpopulations (maps) of motor cortical cells are thought to be involved in this process: one population representing positional commands to the speech articulators, one representing velocity commands originating from the feedforward control subsystem, and one representing velocity commands originating from the feedback control subsystem.

Cells in the model’s motor cortex position map correspond to “tonic” cells found in motor cortex electrophysiological studies in monkeys (e.g., Kalaska, Cohen, Hyde, & Prud'homme, 1989). Their activities at time t are represented by the vector M(t). The motor position cells are formed into antagonistic pairs, with each pair representing a position command for one of the eight model articulators. Thus M(t) is a 16-dimensional vector, and it is governed by the following equation:

$$M(t)=M(\text{0})+{\alpha }_{ff}\underset{0}{\overset{t}{\int }}{\dot{M}}_{Feedforward}(t)g(t)dt+{\alpha }_{fb}\underset{0}{\overset{t}{\int }}{\dot{M}}_{Feedback}(t)g(t)dt$$ (2)

where M(0) is the initial configuration of the vocal tract when starting an utterance, α_fb and α_ff are parameters that determine how much the model is weighted toward feedback control and feedforward control⁴, respectively, and g(t) is a speaking rate signal that is 0 when not speaking and 1 when speaking at a maximum rate. The 16-dimensional vectors ${\dot{M}}_{Feedforward}(t)$ and ${\dot{M}}_{Feedback}(t)$ constitute the model’s motor cortex velocity maps and correspond to “phasic” cells found in electrophysiological studies in monkeys (e.g., Kalaska et al., 1989). ${\dot{M}}_{Feedforward}(t)$ encodes a feedforward control signal projecting from premotor cortex and the cerebellum, and ${\dot{M}}_{Feedback}(t)$ encodes a feedback control signal projecting from sensory cortical areas; the sources of these command signals are discussed further in later sections (Feedback Control Subsystem and Feedforward Control Subsystem).

The model’s motor position map cells produce movements in the model’s articulators according to the following equation:

$$Artic(t)=f_{MAr}(M(t-{\tau }_{MAr}))+Pert(t)$$ (3)

where f_MAr is a simple function relating the motor cortex position command to the Maeda parameter values (transforming each antagonistic pair into a single articulator position value), τ_MAr is the time it takes for a motor command to have its effect on the articulatory mechanism, and Pert is the effect of external perturbations on the articulators if such perturbations are applied (see Computer Simulations of the Model below). The eight-dimensional vector Artic does not correspond to any cell activities in the model; it corresponds instead to the physical positions of the eight articulators⁵ in the Maeda articulatory synthesizer (Maeda, 1990). The resulting vocal tract area function is converted into a digital filter that is used to synthesize an acoustic signal that forms the output of the model (e.g., Maeda, 1990).

Roughly speaking, the delay τ_MAr in Equation 3 corresponds to the time it takes for an action potential in a motor cortical cell to affect the length of a muscle via a subcortical motoneuron. This time can be broken into two components: (1) the delay between motor cortex activation and activation of a muscle as measured by EMG, and (2) the delay between EMG onset and muscle length change. Regarding the former, Meyer, Werhahn, Rothwell, Roericht, and Fauth (1994) measured the latency of EMG responses to transcranial magnetic stimulation of the face area of motor cortex in humans and found latencies of 11–12 ms for both ipsilateral and contralateral facial muscles. Regarding the latter, time delays between EMG onset and onset of the corresponding articulator acceleration of approximately 30 ms have been measured in the posterior genioglossus muscle of the tongue (Majid Zandipour and Joseph Perkell, personal communication); this estimate is in line with a more thorough investigation of bullfrog muscles which showed average EMG to movement onset latencies of approximately 24 ms in hip extensor muscles, with longer latencies occurring in other leg muscles (Olson & Marsh, 1998). In keeping with these results, we use τ_MAr = 42 ms in the simulations reported below. When an estimate of EMG onset latency is needed in the simulations, we use a 12 ms estimate from motor cortical cell activation to EMG onset based on Meyer et al. (1994).

The next two sections describe the feedback and feedforward control subsystems that are responsible for generating the motor commands ${\dot{M}}_{Feedback}(t)$ and ${\dot{M}}_{Feedforward}(t)$.

FEEDBACK CONTROL SUBSYSTEM

The feedback control subsystem in the DIVA model (blue portion of Figure 1) carries out the following functions when producing a learned sound. First, activation of the speech sound map cell corresponding to the sound in the model’s premotor cortex leads to readout of learned auditory and somatosensory targets for that sound. These targets take the form of temporally varying regions in the auditory and somatosensory spaces, as described below. The current auditory and somatosensory states, available through sensory feedback, are compared to these targets in the higher-order auditory and somatosensory cortices. If the current sensory state falls outside of the target region, an error signal arises in the higher-order sensory cortex. These error signals are then mapped into appropriate corrective motor commands via learned projections from the sensory error cells to the motor cortex.

The following paragraphs detail these processes, starting with descriptions of the auditory and somatosensory state maps, continuing with a treatment of the auditory and somatosensory targets for a speech sound, and concluding with a description of the circuitry involved in transforming auditory and somatosensory error signals into corrective motor commands.

Auditory State Map

In the model, the acoustic state is determined from the articulatory state as follows:

$$Acoust(t)=f_{ArAc}(Artic(t))$$ (4)

where f_ArAc is the transformation performed by Maeda’s articulatory synthesis software. The vector Acoust(t) does not correspond to brain cell activities; instead it corresponds to the physical acoustic signal resulting from the current articulator configuration.

The model includes an auditory state map that corresponds to the representation of speech-like sounds in auditory cortical areas (BA 41, 42, 22). The activity of these cells is represented as follows:

$$Au(t)=f_{AcAu}(Acoust(t-{\tau }_{AcAu}))$$ (5)

where Au(t) is a vector of auditory state map cell activities, f_AcAu is a function that transforms an acoustic signal into the corresponding auditory cortical map representation, and τ_AcAu is the time it takes an acoustic signal transduced by the cochlea to make its way to the auditory cortical areas. Regarding τ_AcAu, Schroeder and Foxe (2002) measured the latency between onset of an auditory stimulus and responses in higher-order auditory cortical areas posterior to A1 and a superior temporal polysensory (STP) area in the dorsal bank of the superior temporal sulcus. They noted a response latency of approximately 10 ms in the posterior auditory cortex and 25 ms in STP. Based in part on these numbers, we use an estimate of τ_AcAu = 20 ms in the simulations reported below.

Regarding f_AcAu, we have used a variety of different auditory representations in the model, including formant frequencies, log formant ratios (e.g., Miller, 1989), and wavelet-based transformations of the acoustic signal. Simulations using these different auditory spaces have yielded similar results in most cases. In the computer simulations reported below, we use a formant frequency representation in which Au(t) is a three-dimensional vector whose components correspond to the first three formant frequencies of the acoustic signal.

Somatosensory State Map

The model also includes a somatosensory state map that corresponds to the representation of speech articulators in somatosensory cortical areas (BA 1,2,3,40,43):

$$S(t)=f_{ArS}(Artic(t-{\tau }_{ArS}))$$ (6)

where S(t) is a 22-dimensional vector of somatosensory state map cell activities, f_ArS is a function that transforms the current state of the articulators into the corresponding somatosensory cortical map representation, and τ_ArS is the time it takes somatosensory feedback from the periphery to reach higher-order somatosensory cortical areas. Regarding τ_ArS, O'Brien, Pimpaneau, and Albe-Fessard (1971) measured evoked potentials in somatosensory cortex induced by stimulation of facial nerves innervating the lips, jaw, tongue, and larynx in anesthetized monkeys. They report typical latencies of approximately 5–20 ms, though some somatosensory cortical cells had significantly longer latencies, on the order of 50 ms. Schroeder and Foxe (2002) noted latencies of approximately 10 ms in inferior parietal sulcus to somatosensory stimulation (electrical stimulation of a hand nerve). Based on these results, we use an estimate of τ_ArS = 15 ms in the simulations reported below.

The function f_ArS transforms the articulatory state into a 22-dimensional somatosensory map representation S(t) as follows. The first 16 dimensions of S(t) correspond to proprioceptive feedback representing the current positions of the 8 Maeda articulators, each represented by an antagonistic pair of cells as in the motor representation. In other words, the portion of f_ArS that determines the first 16 dimensions of S(t) is basically the inverse of f_Mar. The remaining 6 dimensions correspond to tactile feedback, consisting of palatal and labial tactile information derived from the first five Maeda articulatory parameters using a simple modification of the mapping described by Schwartz and Boë (Schwartz & Boë, 2000).

Motor-to-sensory pathways encode speech sound targets

We hypothesize that axonal projections from speech sound map cells in the frontal motor cortical areas (lateral BA 6 and 44) to higher-order auditory cortical areas⁶ in the superior temporal gyrus (BA 22) carry auditory targets for the speech sound currently being produced. That is, these projections predict the sound of the speaker’s own voice while producing the sound based on auditory examples from other speakers producing the sound, as well as one’s own previous correct productions. Furthermore, projections from the speech sound map cells to higher-order somatosensory cortical areas in the anterior supramarginal gyrus and surrounding cortex (BA 40; perhaps also portions of BA 1, 2, 3, and 43) are hypothesized to carry target (expected) tactile and proprioceptive sensations associated with the sound currently being produced. These expectations are based on prior successful attempts to produce the sound, though we envision the possibility that some aspects of the somatosensory targets might be learned by infants when they view a speaker (e.g., by storing the movement of the lips for a bilabial).

The auditory and somatosensory targets take the form of multidimensional regions, rather than points, that can vary with time, as schematized in Figure 2. The use of target regions is an important aspect of the DIVA model that provides a unified explanation for a wide range of speech production phenomena, including motor equivalence, contextual variability, anticipatory coarticulation, carryover coarticulation, and speaking rate effects (see Guenther, 1995 for details).

Figure 2.

Auditory target region for the first three formants of the syllable “ba” as learned by the model from an audio sample of an adult male speaker.

In the computer simulations, the auditory and somatosensory targets for a speech sound are encoded by the weights of the synapses projecting from the premotor cortex (specifically, from the speech sound map cell representing the sound) to cells in the higher-order auditory and somatosensory cortices, respectively. The synaptic weights encoding the auditory target for a speech sound are denoted by the matrix z_PAu(t), and the weights encoding the somatosensory target are denoted by the matrix z_PS(t). These weight matrices are “spatiotemporal” in that they encode target regions for each point in time from the start of production to the end of production of the speech sound they encode. That is, each column of the weight matrix represents the target at one point in time, and there is a different column for every 1 ms of the duration of the speech sound.

It is hypothesized that the weights z_PAu(t)become tuned when an infant listens to examples of a speech sound, e.g. as produced by his/her parents. In the current model the weights are algorithmically tuned⁷ by presenting the model with an audio file containing a speech sound produced by an adult male. The weights z_PAu(t) encoding that sound are then adjusted so that they encode upper and lower bounds for each of the first three formant frequencies at 1 ms intervals for the duration of the utterance.

It is further hypothesized that the weights z_PS(t) become tuned during correct self-productions of the corresponding speech sound. Note that this occurs after learning of the auditory target for the sound since the auditory target can be learned simply by monitoring a sound spoken by someone else Many aspects of the somatosensory target, however, require monitoring of correct self-productions of the sound, which are expected to occur after (and possibly during) the learning of feedforward commands for producing the sound (described in the next section). In the model the weights z_PS(t)are adjusted to encode upper and lower bounds for each somatosensory dimension at 1 ms intervals for the duration of the utterance.

In the motor control literature, it is common to refer to internal estimates of the sensory consequences of movements as “forward models”. The weight matrices z_PAu(t) and z_PS(t) are examples of forward models in this sense. Although not currently implemented in the model, we also envision the possibility that lower-level forward models are implemented via projections from the primary motor cortex to the primary somatosensory and auditory cortices, in parallel with the z_PAu(t) and z_PS(t)projections from premotor cortex to higher-order somatosensory and auditory cortices. Such projections would not be expected to significantly change the model’s functional properties.

Auditory and somatosensory error maps

The sensory target regions for the current sound are compared to incoming sensory information in the model’s higher-order sensory cortices. If the current sensory state is outside the target region error signals arise, and these error signals are mapped into corrective motor commands.

The model’s auditory error map encodes the difference between the auditory target region for the sound being produced and the current auditory state as represented by Au(t). The activity of the auditory error map cells (ΔAu) is defined by the following equation:

$$\mathrm{\Delta }Au(t)=Au(t)-P(t-{\tau }_{PAu})z_{PAu}(t)$$ (7)

where τ_PAu is the propagation delay for the signals from premotor cortex to auditory cortex (assumed to be 3ms in the simulations⁸), and z_PAu(t) are synaptic weights that encode auditory expectations for the sound being produced. The auditory error cells become active during production if the speaker’s auditory feedback of his/her own speech deviates from the auditory target region for the sound being produced.

The projections from premotor cortex represented in Equation 6 cause inhibition⁹ of auditory error map cells. Evidence for inhibition of auditory cortical areas in the superior temporal gyrus during one’s own speech comes from several different sources, including recorded neural responses during open brain surgery (Creutzfeldt, Ojemann, & Lettich, 1989a; Creutzfeldt, Ojemann, & Lettich, 1989b), MEG measurements (Numminen et al., 1999a; Numminen et al., 1999b), and PET measurements (Wise et al., 1999). Houde, Nagarajan, Sekihara, and Merzenich (2002) note that auditory evoked responses measured with MEG were smaller to self-produced speech than when the same speech was presented while the subject was not speaking, while response to a gated noise stimulus was the same in the presence or absence of self-produced speech. The authors concluded that “during speech production, the auditory cortex (1) attenuates its sensitivity and (2) modulates its activity as a function of the expected acoustic feedback” (p. 1125), consistent with the model.

The model’s somatosensory error map codes the difference between the somatosensory target region for a speech sound and the current somatosensory state:

$$\mathrm{\Delta }S(t)=S(t)-P(t-{\tau }_{PS})z_{PS}(t)$$ (8)

where τ_PS is the propagation delay from premotor cortex to somatosensory cortex (3 ms in the simulations), and the weights z_PS(t) encode somatosensory expectations for the sound being produced. The somatosensory error cells become active during production if the speaker’s somatosensory feedback from the vocal tract deviates from the somatosensory target region for the sound being produced. To our knowledge, no studies have looked for an inhibitory effect in the supramarginal gyrus during speech production, although this brain region has been implicated in phonological processing for speech perception (e.g., Caplan, Gow, & Makris, 1995; Celsis et al., 1999), and speech production (Geschwind, 1965; Damasio & Damasio, 1980).

Converting sensory errors into corrective motor actions

In the model, production errors represented by activations in the auditory and/or somatosensory error maps get mapped into corrective motor commands through learned pathways projecting from the sensory cortical areas to the motor cortex. These projections form a feedback control signal that is governed by the following equation:

$${\dot{M}}_{Feedback}(t)=\mathrm{\Delta }Au(t-{\tau }_{AuM})z_{AuM}+\mathrm{\Delta }S(t-{\tau }_{SM})z_{SM}$$ (9)

where z_AuM and z_SM are synaptic weights that transform directional sensory error signals into motor velocities that correct for these errors, and τ_AuM and τ_SM are cortico-cortical transmission delays (3 ms in the simulations). The model’s name, DIVA, derives from this mapping from sensory directions into velocities of articulators. Mathematically speaking, the weights z_AuM and z_SM approximate a pseudoinverse of the Jacobian of the function relating articulator positions (M) to the corresponding sensory state (Au, S; see Guenther et al., 1998 for details). Though calculated algorithmically in the current implementation (see Appendix B for details), these weights are believed to be tuned during an early babbling stage by monitoring the relationship between movement commands and their sensory consequences (see Guenther, 1995 and Guenther, 1998 for simulations involving learning of the weights). These synaptic weights effectively implement what is sometimes referred to as an “inverse model” in the motor control literature since they represent an inverse kinematic transformation between desired sensory consequences and appropriate motor actions.

The model implicitly predicts that auditory or somatosensory errors will be corrected via the feedback-based control mechanism, and that these corrections will eventually become coded into the feedforward controller if the errors are consistently encountered (see next section for learning in the feedforward control subsystem). This would be the case if a systematic auditory perturbation was applied (e.g, a shifting of one or more of the formant frequencies in real time) or a consistent somatosensory perturbation is applied (e.g., a perturbation to the jaw). Relatedly, Houde and Jordan (1998) modified the auditory feedback of speakers (specifically, shifting the first two formant frequencies of the spoken utterances and feeding this shifted auditory information back to the speaker with a time lag of approximately 16 ms) and noted that the speakers compensated for the shifted auditory feedback over time. Tremblay, Shiller, and Ostry (2003) performed an experiment in which jaw motion during syllable production was modified by application of a force to the jaw which did not measurably affect the acoustics of the syllable productions. Despite no change in the acoustics, subjects compensated for the jaw force, suggesting that they were using somatosensory targets such as those represented by z_PS(t) in the DIVA model. The DIVA model provides a mechanistic account of these sensorimotor adaptation results.

FEEDFORWARD CONTROL SUBSYSTEM

According to the model, projections from premotor to primary motor cortex, supplemented by cerebellar projections (see Figure 1), constitute feedforward motor commands. The primary motor and premotor cortices are well-known to be strongly interconnected (e.g., Passingham, 1993; Krakauer & Ghez, 1999). Furthermore, the cerebellum is known to receive input via the pontine nuclei from premotor cortical areas, as well as higher-order auditory and somatosensory areas that can provide state information important for choosing motor commands (e.g., Schmahmann & Pandya, 1997), and projects heavily to the motor cortex (e.g., Middleton & Strick, 1997). We believe these projections are involved in the learning and maintenance of feedforward commands for the production of syllables.

Before the model has any practice producing a speech sound, the contribution of the feedforward control signal to the overall motor command will be small since it will not yet be tuned. Therefore, during the first few productions, the primary mode of control will be feedback control. During these early productions, the feedforward control system is “tuning itself up” by monitoring the motor commands generated by the feedback control system (see also Kawato & Gomi, 1992). The feedforward system gets better and better over time, all but eliminating the need for feedback-based control except when external constraints are applied to the articulators (e.g., a bite block) or auditory feedback is artificially perturbed. As the speech articulators get larger with growth, the feedback-based control system provides corrective commands that are eventually subsumed into the feedforward controller. This allows the feedforward controller to stay properly tuned despite dramatic changes in the sizes and shapes of the speech articulators over the course of a lifetime.

The feedforward motor command for production of a sound is represented in the model by the following equation:

$${\dot{M}}_{Feedforward}(t)=P(t)z_{PM}(t)-M(t)$$ (10)

The weights z_PM(t) encode the feedforward motor command for the speech sound being produced (assumed to include both cortico-cortical and trans-cerebellar contributions). This command is learned over time by incorporating the corrective motor commands from the feedback control subsystem on the previous attempt into the new feedforward command (see Appendix B for details).

As mentioned above, once an appropriate feedforward command sequence has been learned for a speech sound, this sequence will successfully produce the sound with very little, if any, contribution from the feedback subsystem, which will automatically become disengaged since no sensory errors will arise during production unless unexpected constraints are placed on the articulators or the auditory signal is perturbed.

COMPUTER SIMULATIONS OF THE MODEL

This section describes new computer simulations that illustrate the model’s ability to learn to produce new speech sounds in the presence of neural and biomechanical processing delays, as well as to simulate the patterns of lip, jaw, and tongue movements seen in articulator perturbation experiments. Introducing perturbations during a speech task and observing the system response provides information about the nature of the controller. In particular, the time course and movement characteristics of the response can provide a window into the control processes, including neural transmission delays and the nature of the transformation between sensory and motor representations.

The simulations utilize Equations 1–10, with the delay parameters in the equations set to the values indicated below each equation. Prior to the simulations described below, the model’s synaptic weight parameters (i.e., the z matrices in the equations) were tuned in a simulated “babbling phase”. In this phase, the cells specifying the motor cortex movement command (M) were randomly activated in a time-varying manner, leading to time-varying articulator movements (Artic) and an accompanying acoustic signal (Acoust). The motor commands M were used in combination with the resulting auditory (A) and somatosensory (S) feedback to tune the synaptic weight matrices z_AuM and z_SM (see Appendix B for details regarding the algorithms used to tune the model’s synaptic weights).

After the babbling phase, the model was trained to produce a small corpus of speech sounds (consisting of individual phonemes, syllables, and short words) via a process meant to approximate an infant learning a new sound by hearing it from an adult and then trying to produce it a few times. For each sound, the model was first presented with an acoustic example of the sound while simultaneously activating a speech sound map cell (P) that was chosen to represent the new sound. The resulting spatiotemporal auditory pattern (A) was used to tune the synaptic weights representing the auditory target for the sound (z_PAu). Then a short “practice phase”, involving approximately 5–10 attempts to produce the sound by the model, was used to tune the synaptic weights making up the feedforward commands for the sound (z_PM). Finally, after the feedforward weights were tuned, additional repetitions were used to tune the somatosensory target for the sound (z_PS).

Simulation 1: “good doggie”

For this simulation, an utterance of the phrase “good doggie” was recorded at a sampling rate of 10 kHz. Formants were extracted from the signal and were modified slightly to form an auditory target that better matched the vocal tract characteristics of the Maeda synthesizer. The auditory target was represented as a convex region for each time point (see Guenther, 1998 for a discussion of convex region targets). Figure 3 shows the results of the simulations through the spectrograms of model utterances. The top plot shows the original spectrogram. The remaining plots show the 1^st, 3^rd, 5^th, 7^th, and 9^th model attempts to produce the sound. With each trial, the feedforward system subsumes the corrective commands generated by the feedback system to compensate for the sensory error signals that arise during that trial. As can be seen from the figure, the spectrograms approach the original as learning progresses.

Figure 3.

Spectrograms showing the first three formants of the utterance “good doggie” as produced by an adult male speaker (top panel) and by the model (bottom panels). The model first learns an acoustic target for the utterance based on the sample it is presented (top panel). Then the model attempts to produce the sound, at first primarily under feedback control (Attempt 1), then with progressively improved feedforward commands supplementing the feedback control (Attempts 3, 5, 7, and 9). By the 9th attempt the feedforward control signals are accurate enough for the model to closely imitate the formant trajectories from the sample utterance.

Simulation 2: Abbs and Gracco (1984) lip perturbation

In this simulation of the lip perturbation study, the model’s lower lip was perturbed downward using a steady force during the movement toward closure of the lips when producing the utterance /aba/. Figure 4 shows a comparison of the model’s productions to those measured in the original experiment for normal (no perturbation) and perturbed trials. The experiment results demonstrated that the speech motor system compensates for the perturbation by lowering the upper lip further than normal, resulting in successful closure of the lips despite the downward perturbation to the lower lip. The corresponding model simulations are shown in the right panel of Figure 4. The model was first trained to produce the utterance /aba/. After the sound was learned, the lower lip parameter of the model was perturbed with a constant downward force. The onset of perturbation was determined by tracking the velocity of the jaw parameter. The vertical black line marks the onset of perturbation. The position of the lips during the control condition is shown with the dashed lines while the position during the perturbed condition is shown with the solid lines. When the lips are perturbed, the tactile and proprioceptive feedback no longer matches the somatosensory target, giving rise to a somatosensory error signal and corrective motor command through the model’s feedback subsystem. The commandis generated approximately 60 ms (the sum of τ_ArS, τ_SM, and τ_MAr) after the onset of perturbation. This is within the range of values (22–75 ms) measured during the experiment.

Figure 4.

Abbs and Gracco (1984) lip perturbation experimental results (left) and model simulation results (right). Far left panels show upper and lower lip positions during bilabial consonant production in the normal (top) and perturbed (bottom) conditions of the Abbs and Gracco (1984) experiment; shown to the right of this is a superposition of the normal and perturbed trials in a single image. Arrows indicate onset of perturbation. [Adapted from Abbs and Gracco (1984).] The right panel shows the lip heights from model simulations of the control (dashed lines) and perturbed (solid lines) conditions for the same perturbation, applied as the model starts to produce the /b/ in /aba/ (vertical line). The solid lines demonstrate the compensation provided by the upper and lower lips, which achieve contact despite the perturbation. The latency of the model’s compensatory response is within the range measured by Abbs and Gracco (1984).

Simulation 3: Kelso, Tuller, Vatikiotis-Bateson, and Fowler (1984) jaw perturbation

In the experiment, the jaw was perturbed downward during the upward movement of the closing gesture in each of the two words: /baeb/ and /baez/. Their results demonstrate that the upper lip compensated for the perturbation during the production of /baeb/ but not during the production of /baez/ (top panel of Figure 5). These results indicate that compensation to perturbation does not affect the whole vocal tract but primarily affects articulators involved in the production of the particular phonetic unit that was being perturbed. Since the upper lip is not involved in the production of /z/, it is not influenced by the jaw perturbation in /baez/.

Figure 5.

Top: Results of Kelso et al. (1984) jaw perturbation experiment. Dotted lines indicate normal (unperturbed) trials, and solid lines indicate perturbed trials. The vertical line indicates onset of perturbation. Lower lip position is measured relative to jaw. Subjects produce compensatory downward movement of the upper lip for the bilabial stop /b/ but not for the alveolar stop /d/. [Adapted from (Kelso, Tuller, Vatikiotis-Bateson, & Fowler, 1984).] Bottom: Corresponding DIVA simulation. As in the Kelso et al. (1984) experiment, the model produces a compensatory downward movement of the upper lip for the bilabial stop /b/ but not for the alveolar stop /d/.

In the model simulations (bottom panel of Figure 5), we used the words /baeb/ and /baed/ to demonstrate the effects of jaw perturbation¹⁰. A steady perturbation corresponding to the increased load in the experiments was applied during the upward movement of the jaw. The perturbation was simulated by adding a constant value to the jaw height articulator of the vocal tract model. The perturbation remained in effect through the end of the utterance, as in the experiment. The onset of the perturbation is indicated by the vertical line in the simulation diagrams of Figure 5 and was determined by the velocity and position of the jaw displacement. The dotted lines indicate the positions of the articulators in the normal (unperturbed) condition. The solid lines indicate the positions in the perturbed condition. As in the experiment, the upper lip compensates by moving further downward when the bilabial stop /baeb/ is perturbed, but not when the alveolar stop /baed/ is perturbed.

COMPARING THE MODEL’S CELL ACTIVITIES TO THE RESULTS OF FMRI STUDIES

As stated in the Introduction, a major goal of the current modeling work is to provide a framework for interpreting the results of neuroimaging studies of speech production, and for generating predictions to help guide future neuroimaging studies. To this end, we have identified likely neuroanatomical locations of the model’s components based on the results of previous neurophysiological studies as well as the results of functional magnetic resonance imaging experiments conducted by our laboratory. These locations allow us to run “simulated fMRI experiments” in which the model produces speech sounds in different speaking conditions, and the model cell activities are then used to generate a simulated hemodynamic response pattern based on these cell activations. These simulated hemodynamic response patterns can then be compared to the results of fMRI and/or positron emission tomography (PET) experiments in which human subjects produce the same (or similar) speech sounds in the same speaking conditions. In this section we describe this simulation process and the resulting hemodynamic response patterns, including a comparison of these patterns to the results of an fMRI experiment of simple syllable production performed in our laboratory. The results in this section are meant to illustrate the degree to which the model can currently account for the brain activities seen in human speech production experiments, and to serve as a baseline for future simulations involving additional speaking conditions that will test specific hypotheses generated from the model.

In Appendix A we detail the hypothesized anatomical locations of the model’s components, with particular reference to the brain of the canonical single subject provided with the SPM2 software package (Friston, Ashburner, Holmes, & Poline, 2002). These locations are given in Montreal Neurological Institute (MNI) normalized spatial coordinates in addition to anatomical descriptions with reference to specific sulci and gyri. Figure 6 illustrates the locations of the model’s components projected onto the lateral surface of the standard SPM brain, with the corresponding MNI coordinates provided in Table 1 of Appendix A.

Figure 6.

Rendered lateral surfaces of the SPM standard brain indicating locations of the model components as described in the text. Medial regions (anterior paravermal cerebellum and deep cerebellar nuclei) are omitted. Unless otherwise noted, labels along the central sulcus correspond to a motor (anterior) and a somatosensory (posterior) representation for each articulator. Abbreviation key: Aud = auditory state cells; ΔA = auditory error cells; ΔS = somatosensory error cells; Lat Cbm = superior lateral cerebellum; Resp = motor respiratory region; SSM = speech sound map. *Palate representation is somatosensory only. ^†Respiratory representation is motor only.

Table 1.

Montreal Neurological Institute (MNI) normalized spatial coordinates of DIVA model components mapped onto the left and right hemisphere of the canonical single brain provided with the SPM2 analysis software package. SPT = Sylvian-parietal-temporal region as described by Hickok et al. (2004).

	Left			Right
Model Components	x	y	z	x	y	z
Motor Tongue
1	−60.2	2.1	27.5	62.9	2.5	28.9
2	−60.2	3.0	23.3	66.7	2.5	24.9
3	−60.2	4.4	19.4	64.2	3	22
Motor Lip
Upper	−53.9	−3.6	47.2	59.6	−7.2	42.5
Lower	−56.4	0.5	42.3	59.6	−3.6	40.6
Motor Jaw	−59.6	−1.3	33.2	62.1	3.9	34.0
Motor Larynx	−58.1	6.0	6.4	65.4	5.2	10.4
Motor Respiration	−17.4	−26.9	73.4	23.8	−28.5	70.1
Cerebellum
Anterior Paravermis	−18	−59	−22	16	−59	−23
Anterior Lateral	−36	−59	−27	40	−60	−28
Deep Cerebellar Nuclei	−10.3	−52.9	−28.5	14.4	−52.9	−29.3
Speech Sound Map
Inf. Prefrontal Gyrus	−56.5	14.8	4.8
Sensory Tongue
1	−60.2	−2.8	27.0	62.9	−1.5	28.9
2	−60.2	−0.5	23.3	66.7	−1.9	24.9
3	−60.2	0.6	20.8	64.2	0.1	21.7
Sensory Lip
Upper	−53.9	−7.7	47.2	59.6	−10.2	40.6
Lower	−56.4	−5.3	42.1	59.6	−6.9	38.2
Sensory Jaw	−59.6	−5.3	33.4	62.1	−1.5	34.0
Sensory Larynx	−61.8	1	7.5	65.4	1.2	12
Sensory Palate	−58	−0.7	14.3	65.4	−0.4	21.6
Somatosensory Error Cells
Supramarginal Gyrus	−62.1	−28.4	32.6	66.1	−24.4	35.2
Auditory State Cells
Heschl’s gyrus	−37.4	−22.5	11.8	39.1	−20.9	11.8
Planum temporale	−57.2	−18.4	6.9	59.6	−15.1	6.9
Auditory Error Cells
SPT	−39.1	−33.2	14.3	44	−30.7	15.1
Post. Sup. Temporal Gyrus	−64.6	−33.2	13.5	69.5	−30.7	5.2

FMRI and PET studies of speech production typically involve one or more “speaking conditions”, in which the subject produces speech, and a “baseline condition”, in which the subject rests quietly. The brain regions that become “active” during speech (i.e., those that have a larger hemodynamic response in the speech condition compared to the baseline condition) are typically interpreted as being involved in speech production.

In the model simulations, the “speaking condition” consisted of the model producing simple syllables. That is, speech sound map cells corresponding to the syllables were activated (Equation 1), and Equations 2–10 were used to calculate the activities of the model’s cells (with the same model parameters used in the jaw and lip perturbation simulations described above). In the “baseline condition” all model cell activities were set to zero, corresponding to a resting state in which no speech is being produced. To produce the simulated hemodynamic response for each condition, model cell activities were first normalized by the maximum possible activity of the cell; this was done to correct for differences in the dynamic ranges of the different cell types in the model. The resultant activity was then convolved with an idealized hemodynamic response function, generated using default settings of the function ‘spm_hrf’ from the SPM toolbox. This function was designed by the creators of SPM to approximate the transformation from cell activity to hemodynamic response in the brain. For brain locations that include more than one cell at the same location (i.e., those with the same MNI coordinates in Table 1 of Appendix A) the overall hemodynamic response was simply the sum of the responses of the individual cells. A brain volume was then constructed with the appropriate hemodynamic response values at each position. Responses were smoothed with a Gaussian kernel (FWHM=12mm) to approximate the smoothing carried out during standard SPM analysis of human subject data¹¹. The resultant volume was then rendered using routines from the SPM toolbox.

In order to qualitatively compare the model’s simulated activations with those of actual speakers, we conducted an fMRI experiment in which ten subjects produced simple consonant-vowel (CV) syllables that were read from a display screen in the scanner. Blood oxygenation level dependent (BOLD) responses were collected in 10 neurologically normal, right-handed speakers of American English (3 female, 7 male) during spoken production of vowel-vowel (VV), consonant-vowel (CV), and CVCV syllables which were presented visually (spelled out, e.g. “pah”). An event-triggered paradigm with a 15–18 second interstimulus interval was used wherein two whole head functional scans (3 seconds each in duration) were collected shortly after each syllable production, timed to occur near the peak of the speech-related hemodynamic response (approximately 4–6 seconds after the syllable is spoken). Since no scanning was done while the subject was pronouncing a syllable, this paradigm avoids confounds due to scanner noise during speech as well as image artifacts due to articulator motion. One to three runs of approximately 20 minutes each were completed for each subject. Data were obtained using a whole head coil in Siemens Allegra (6 Subjects) and Trio (4 subjects) scanners. Thirty axial slices (5 mm thick, 0 mm skip) parallel to the anterior and posterior commissure covering the whole brain were imaged with a temporal resolution of 3 sec using a T2*-weighted pulse sequence (TR=3s, TE=30ms, flip angle=90°, FOV=200mm and interleaved scanning). Images were reconstructed as a 64 × 64 × 30 matrix with a spatial resolution of 3.1×3.1×5 mm. To aid in the localization of functional data and for generating regions of interest (ROIs), high-resolution T1-weighted 3D MRI data were collected with the following parameters: TR=6.6ms, TE=2.9ms, flip angle=8°, 128 slices in sagittal place, FOV=256mm. Images were reconstructed as a 256 × 256 × 128 matrix with a 1 × 1 × 1.33 mm spatial resolution. The data from each subject were corrected for head movement, coregistered with the high-resolution structural image and normalized to MNI space. Random effects analysis was performed on the data using the SPM toolbox. The results were thresholded using a false discovery rate of p<0.05 (corrected).

Brain activations during syllable production (as compared to a baseline task involving passive viewing of visually presented X’s on the display) are shown in the left half of Figure 7. The right half of Figure 7 shows brain activations derived from the DIVA model while producing the same syllables, with the model’s components localized on the cortical surface and cerebellum as described in Appendix A. For the most part, the model’s activations are qualitatively similar to those of the fMRI subjects. The biggest difference in activation concerns the supplementary motor area in the medial frontal lobe. This area, which is active in the experimental subjects but is not included in the mode at this time, is believed to be involved in the initiation and/or sequencing of speech sounds (see Concluding Remarks for details). Another difference concerns the respiratory portion of the motor cortex, high up on the motor strip, which is more active in the model than in the experimental subjects. This may be due to the fact that the model has no activity in this area during the baseline condition (quiet resting), whereas experimental subjects continue breathing during the baseline condition, perhaps controlled in part by motor cortex. The reduced baseline respiratory motor cortex activity in the model would result in greater activity for the model than for subjects in the speech – baseline comparison.

Figure 7.

fMRI activations measured in human subjects while they read simple syllables from a screen (left) and simulated fMRI activations derived from the model’s cell activities during simple syllable production (right). See text for details.

Although it is informative to see how much of the fMRI activity in human subjects producing simple syllables can be accounted for by the model, it is perhaps more informative to generate novel predictions from the model and test them in future neuroimaging studies. We are currently performing two such fMRI studies, one involving somatosensory perturbation during speech (using a pneumatic bite block) and one involving real-time auditory perturbation of the subject’s acoustic feedback of their own speech. According to the model, somatosensory perturbation should lead to activity of somatosensory error cells in the anterior supramarginal gyrus (ΔS in Figure 6) due to a mismatch between the somatosensory target region and the incoming somatosensory feedback. Such activity would not be expected during unperturbed speech since the feedforward command in adults is well-tuned and thus few if any somatosensory errors should arise without perturbation. Similarly, auditory perturbation during speech should lead to more activation of auditory error cells in the superior temporal gyrus and planum temporale (ΔA in Figure 6) than unperturbed speech. The results of these fMRI studies should help us further refine our account of the neural bases of speech production. We also plan to investigate quantitative techniques for comparing model and human activations. One possible measure is mutual information (e.g., Maes et al., 1997), which describes the degree of agreement between two datasets in a way that is more robust than other comparable measures such as correlation.

CONCLUDING REMARKS

In this article we have described a neural model that provides a unified account for a wide range of speech acoustic, kinematic, and neuroimaging data. New computer simulations of the model were presented to illustrate the model’s ability to provide a detailed account for experiments involving compensations to perturbations of the lip and jaw. With the goal of providing a computational framework for interpreting functional neuroimaging data, we have explicitly identified expected anatomical locations of the model’s components, and we have compared the model’s activities to activity measured using fMRI during simple syllable production and with and without a jaw perturbation.

Although the model described herein accounts for most of the activity seen in fMRI studies of speech production, it does not provide a complete account of the cortical and cerebellar mechanisms involved. In particular, as currently defined, the DIVA model is given a phoneme string by the modeler, and the model produces this phoneme string in the specified order. Brain structures involved in the selection, initiation, and sequencing of speech movements are not treated in the preceding discussion; these include the anterior cingulate area, the supplementary motor area (SMA), the basal ganglia, and (possibly) the anterior insula. The anterior cingulate gyrus lies adjacent to the SMA on the medial surface of the cortex in the interhemispheric fissure. This area is known to be involved in initiation of self-motivated behavior. Bilateral damage to the anterior cingulate area can result in akinetic mutism, characterized by a profound inability to initiate movements (DeLong, 1999). The anterior cingulate has also been implicated in execution of appropriate verbal responses and suppression of inappropriate responses (Paus, Petrides, Evans, & Meyer, 1993; Buckner et al., 1996; Nathaniel-James, Fletcher, & Frith, 1997). Several researchers have posited that the supplementary motor area is particularly involved for self-initiated responses, i.e., responses made in the absence of external sensory cues, whereas lateral premotor cortex is more involved when responding to external cues (e.g., Goldberg, 1985; Passingham, 1993). As the model is currently defined, it is not possible to differentiate between internally generated and externally cued speech. Diseases of the basal ganglia, such as Parkinson’s disease and Huntington’s disease, are known to impair movement sequencing (Stern, Mayeux, Rosen, & Ilson, 1983; Georgiou et al., 1994; Phillips, Chiu, Bradshaw, & Iansek, 1995; Rogers, Phillips, Bradshaw, Iansek, & Jones, 1998), and single-cell recordings indicate that cells in the basal ganglia in monkeys and rats code aspects of movement sequences (Kermadi & Joseph, 1995; Aldridge & Berridge, 1998). The basal ganglia are strongly interconnected to the frontal cortex through a set of segregated basal ganglia-thalamo-cortical loops, including a loop focused on the SMA (DeLong & Wichman, 1993; Redgrave, Prescott, & Gurney, 1999). Like the SMA, the basal ganglia appear to be especially important when movements must be selected and initiated in the absence of external cues (Georgiou et al., 1994; Rogers et al., 1998). Also, stimulation at the thalamic stage of the basal ganglia-thalamo-cortical loops has been shown to affect the rate of speech production (Mateer, 1978). The lesion study of Dronkers (1996) indicated that the anterior insular cortex, or insula, buried in the sylvian fissure near the base of premotor cortex, plays an important role in speech production since damage to the insula is the likely source of pure apraxia of speech, a disorder involving an inability to select the appropriate motor programs for speech. Others have identified insula activation in certain speech tasks (e.g., Wise et al., 1999; Nota & Honda, 2003). The fMRI study of Nota and Honda (2003) suggests that the insula becomes involved when different syllables have to be sequenced in a particular order, as opposed to repetitive production of the same syllable. Based on these studies, we hypothesize that the insula plays a role in selecting the proper speech sound map cells in the ventral lateral premotor cortex.

Some additional factors limit the biological plausibility of the model in its current form. First, as described herein, all model cells of a particular type (e.g., the motor position cells) typically become active simultaneously. However, studies of primate cortex typically identify “recruitment curves” that show a more gradual onset of cells in a particular brain region (e.g., Kalaska & Crammond, 1992). Second, we make a sharp distinction between premotor cortex and primary motor cortex, with premotor cortex involving higher-level representations (the speech sound map) and motor cortex involving low-level motor representations (the articulator velocity and position cells). Neurophysiological results indicate that, instead, there appears to be a continuum of cells from motor to premotor cortex, with the complexity of the motor representation increasing as one moves anteriorly along the precentral gyrus into the premotor cortex (e.g., Kalaska et al., 1992). Future work will involve modifications that make the model more compatible with these findings.

Finally, it is interesting to note that the current model provides a more detailed account of the “mental syllabary” concept described by Levelt and colleagues (e.g., Levelt et al., 1994). In our account, the speech sound map cells can be thought of as the primary component of the syllabary, but additional components include the feedforward command pathways to motor cortex (the “gestural score”), and the auditory and somatosensory target projections to the higher-order auditory and somatosensory cortices. Thus in our view the syllabary is best thought of as a network of regions that together constitute the sensorimotor representation of frequently produced syllables.

APPENDIX A: ESTIMATED ANATOMICAL LOCATIONS OF THE MODEL’S COMPONENTS

In this appendix we describe hypothesized neuroanatomical locations of the model’s components, including a treatment of the neurophysiological literature that was used to guide these location estimates. Table 1 summarizes the Montreal Neurological Institute (MNI) coordinates for each of the model’s components; these coordinates were used to create the simulated fMRI activations shown in Figure 7. Unless otherwise noted, each cell type is represented symmetrically in both hemispheres. Currently there are no functional differences between the left and right hemisphere versions of a particular cell type in the model. However future versions of the model will incorporate hemispheric differences in cortical processing as indicated by experimental findings (e.g., Poeppel, 2003; Tallal et al., 1993; Zatorre et al., 1992, 2002).

Motor Position and Velocity Maps

Cells coding for the position and velocity of the tongue parameters in the model are hypothesized to correspond with the Motor Tongue Area (MTA) as described by Fesl et al. (2003). The region lies along the posterior bank of the precentral gyrus roughly 2–3 cm above the Sylvian fissure. The spatial localization of this area is in agreement with imaging (Fesl et al., 2003; Corfield et al., 1999; Urasaki, Uematsu, Gordon, & Lesser, 1994; also see Fox et al., 2001) and physiological (Penfield & Rasmussen, 1950) studies of the primary motor region for tongue/mouth movements. We designated a motor (and somatosensory) tongue location for each degree of freedom in the model. This expanded representation is consistent with the large tongue sensorimotor representation

A region superior and medial to the tongue region along the posterior bank of the precentral gyrus has been shown to produce lip movements in humans when electrically stimulated (Penfield & Roberts, 1959). Comparing production of syllables involving tongue movements to those involving lip movements, Lotze et al. (2000b) found the lip area to be approximately 1–2 cm from the tongue area in the directions described by Penfield. In another mapping study of motor cortex using fMRI, Lotze et al. (2000a) showed the lip region inferolateral to the hand motor area, consistent with the Penfield electrical stimulation results. This area is hypothesized to code for the motor position and velocity of the model lip parameters. Upper and lower lip regions have been designated along the precentral gyrus superior and medial to the tongue representation. Data indicating the relative locations of upper and lower lip motor representations in humans is scarce. Currently, we have placed the upper lip motor representation dorsomedial to the lower lip representation, mirroring the somatosensory organization (see Somatosensory State Map below).

Physiological recordings by Penfield and Roberts also indicate a primary motor region corresponding to jaw movements that lies between the lip and tongue representations along the posterior bank of the precentral sulcus, and a region corresponding to larynx motor control inferolateral to the tongue area (Penfield et al., 1959p. 200). Further evidence of the location of a motor larynx representation near the Sylvian fissure is provided by electrical stimulation in primates (e.g., Simonyan & Jurgens, 2002).

Fink et al. (1996) demonstrated dorsolateral precentral gyrus activation during voluntary breathing using PET. The bilateral region noted in that study lied along the superior portion of primary motor cortex, well above the ventral motor representations of the articulators. In an fMRI study, Evans, Shea, and Saykin (1999) found a similar activation association with volitional breathing along superior precentral gyrus medial to the Fink et al. findings and only in the left hemisphere. In the current study, we found activity in approximately the same regions as that described by Fink et al.: bilateral activation superior to and distinct from ventral motor activation (see left half of Figure 6). We hypothesize that this activity is associated with the control of breathing (e.g., maintenance of appropriate subglottal pressure) required for speech production and therefore place cells in this region that correspond to voicing control parameters in the model (specifically, parameter AGP of the Maeda articulatory synthesizer).

While the studies mentioned above indicate bilateral primary motor involvement during articulator movements, they do not explicitly show bilateral involvement of these areas during speech production (though Penfield & Roberts report a bilateral precentral gyrus region that causes “vocalization”). However, Indefrey and Levelt (2004), in their review of neuroimaging studies of speech note bilateral activation of ventral pre- and postcentral gyri during overt speech when compared to silence. In our fMRI results (left half of Figure 6) we found activation along both banks of the central sulcus in both hemispheres, but with stronger activation in the left hemisphere than the right. This finding is consistent with a report of bilateral primary motor activity during overt speech, but stronger activation in the left hemisphere (Riecker et al., 2000). In keeping with these findings, the model’s motor position and velocity cell populations are assumed to contain 20% more cells in the left hemisphere than the right hemisphere, resulting in the leftward bias of the model’s motor cortical activations in the right half of Figure 6.

We hypothesize that the model’s feedforward motor command (specifically, the product P(t)z_PM (t)) involves a cerebellar contribution. Based on the lesion study by Ackermann, Vogel, Petersen, and Poremba (1992), the anterior paravermal region of the cerebellar cortex appears to play a role in the motor control of speech. A contribution to speech production by the medial anterior region of the cerebellum is also supported by a study of dysarthria lesions (Urban et al., 2003). Though not visible in Figure 6 because of the overlying cortex, our fMRI results also show superior medial cerebellum activation during CV production. Recent imaging studies (e.g., Riecker et al., 2000; Riecker, Wildgruber, Grodd, & Ackermann, 2002; Wildgruber et al., 2001) indicate bilateral cerebellum activation during speech production that lies posterior and lateral to the anterior paravermal activity. Our production results reveal distinct bilateral activations that lie behind the primary fissure and lateral to the cerebellum activity already mentioned, in roughly the same region described in these earlier studies. We have therefore placed model cells in two cerebellar cortical regions: anterior paravermal and superior lateral areas. Finally, we identify a region within the medial portion of the sub-cortical cerebellum where the deep cerebellar nuclei (the output cells of the cerebellum) are located.

Speech Sound Map

As described above, we believe the model’s speech sound map consists of mirror neurons similar to those described by Rizzolatti and colleagues. Cells that behave in this fashion have been found in the left inferior premotor F5 region of the monkey (Rizzolatti et al., 1988; Rizzolatti et al., 1996a). Accordingly, we have designated a site in the left ventral premotor area, anterior to the precentral gyrus, as the speech sound map region. This is also consistent with our fMRI results (left half of Figure 6). The designated region, within ventral Brodmann’s area 44 (the posterior portion of Broca’s area), has been described as the human homologue of monkey area F5 (Rizzolatti & Arbib, 1998; Binkofski & Buccino, 2004)¹². We expect that the speech sound map spreads into neighboring regions such as the precentral sulcus and anterior portion of the precentral gyrus.

Somatosensory State Map

Tactile and proprioceptive representations of the articulators are hypothesized to lie along the inferior postcentral gyrus, roughly adjacent to their motor counterparts across the central sulcus. Boling, Reutens, and Olivier (2002) demonstrated an anatomical marker for the tongue somatosensory region using PET imaging that built upon earlier work using electrical stimulation (Picard & Olivier, 1983). They describe the location of the tongue region below the anterior apex of the triangular region of the inferolateral postcentral gyrus approximately 2 cm above the Sylvian fissure. This region of the postcentral gyrus was found to represent the tongue in a somatosensory evoked potential study of humans (McCarthy, Allison, & Spencer, 1993), a finding further supported by a similar procedure in the macaque (McCarthy & Allison, 1995). By generating potentials on either side of the central sulcus, both studies by McCarthy and colleagues demonstrate adjacent motor-somatosensory organization of the tongue representation.

McCarthy et al. (1993) also mapped the primary sensory representations of the lip and palate. The lip representation was located superior and medial to the tongue representation along the anterior bank of the postcentral gyrus at the apex of the inferior postcentral triangle and below the hand representation. Nakamura et al. (1998) localized the lip and tongue sensory representations to nearly identical regions of the postcentral gyrus using MEG. The palatal representation was located inferolateral to the tongue region roughly 1 cm above the Sylvian fissure. The relative locations of the lip, tongue, and palate were confirmed in the macaque (McCarthy et al., 1995). Consistent with early electrophysiological work (Penfield et al., 1950) and a recent MEG study (Nakamura et al., 1998), we have placed the upper lip representation dorsomedial to the lower lip representation.

Graziano, Taylor, Moore, and Cooke (2002) report early electrical stimulation work (Fulton, 1938; Foerster, 1936) which depicts a sensory representation of the larynx at the inferior extent of the postcentral gyrus, near the Sylvian fissure. This location mirrors the motor larynx representation that lies on the inferior precentral gyrus.

Using the same reasoning as outlined above for the primary motor representation of articulators, we hypothesize bilateral somatosensory representations for each of the articulators, with a 20% leftward bias. As was the case for precentral activation, our fMRI results (Figure 6) show greater involvement of the left hemisphere postcentral gyrus.

Somatosensory Error Map

The DIVA model calls for the comparison of speech motor and somatosensory information for the purpose of somatosensory target learning and feedback-based control. We hypothesize that this component of the model, the somatosensory error map, lies within the inferior parietal cortex along the anterior supramarginal gyrus, posterior to the primary somatosensory representations of the speech articulators. Similarly, Hickok and colleagues (e.g., Hickok & Poeppel, 2004) have argued that speech motor commands and sensory feedback interface in the ventral parietal lobe, analogous to the visual-motor integration of the dorsal parietal lobe (Andersen, 1997; Rizzolatti, Fogassi, & Gallese, 1997). Reciprocal connections between area F5 and inferior parietal cortex has been demonstrated in the monkey (Luppino et al., 1999). These connections are believed to contribute to the sensorimotor transformations required to guide movements (see Rizzolatti & Lupppino, 2001) such as grasping. We hypothesize that similar connections are employed to monitor and guide speech articulator movements. Reciprocal connections between posterior inferior frontal gyrus and both the supramarginal gyrus and posterior superior temporal gyrus in the human have been demonstrated by Matsumoto et al. (2004) using a cortico-cortical evoked potential technique involving direct cortical stimulation in epilepsy patients.

Auditory State Map

The auditory state cells are hypothesized to lie within primary auditory cortex and the surrounding auditory association cortex. Therefore we have localized auditory state regions along the medial portion of Heschl’s gyrus and the anterior planum temporale (Rivier & Clarke, 1997; Morosan et al., 2001). These locations are consistent with fMRI studies of speech perceptual processing performed by our group (Guenther, Nieto-Castanon, Ghosh, & Tourville, 2004).

Auditory Error Map

Hickok and colleagues have demonstrated an area within the left posterior Sylvian fissure at the junction of the temporal and parietal lobes (area SPT) and another in the lateral posterior superior temporal gyrus/sulcus that respond during speech perception and production (Buchsbaum et al., 2001; Hickok et al., 2004). The former area was also noted by Wise et al. (2001) in a review of several imaging studies of speech processing as being “engaged in the motor act of speech.” Thus these areas could compare efferent motor commands with auditory input as in the model’s auditory error map. Presently, insufficient information is available to differentiate between the two sites. Therefore we have placed auditory error cells at both locations.

The Buchsbaum and Hickok studies indicated that these regions might be lateralized to the left hemisphere. However, using delayed auditory feedback, Hashimoto and Sakai (2003) showed bilateral activation of the posterior superior temporal gyrus and the inferior supramarginal gyrus. Moreover, activity within the posterior superior temporal gyrus and superior temporal sulcus was correlated with size of the disfluency effect caused by the delayed auditory feedback. Based on this result, we have placed the auditory error cells bilaterally; however we consider it possible that these cells are left-lateralized, and further investigation of this issue is being carried out in ongoing studies of auditory and articulatory perturbation in our laboratory.

As mentioned above, Matsumoto et al., (2004) demonstrated bi-directional connections in humans between posterior inferior frontal gyrus and the two regions proposed to contain the speech error map. Evidence of modulation of the posterior superior temporal gyrus by speech production areas in the human is also provided by the Wise et al. (1999) positron emission tomography study which demonstrated reduced superior temporal gyrus activation during a speech production task compared to a listening task. Single unit recordings from primate auditory cortex provide further support. Eliades & Wang (2002) noted suppression of marmoset auditory cortex immediately prior to self-initiated vocalizations. Based on these results we propose that projections from premotor to higher-order auditory cortical areas exist either directly or via an intermediate area (e.g., anterior supramarginal gyrus).

Although we have treated the auditory and somatosensory error maps as distinct entities in this discussion, we believe there probably exist combined somato-auditory cells, and somato-auditory error maps, that involve relatively highly processed combinations of speech-related somatosensory and auditory information. Thus we expect a continuum of sensory error map representations in and between the superior temporal gyrus, sylvian fissure, and supramarginal gyrus, rather than entirely distinct auditory and somatosensory error maps as described thus far.

APPENDIX B: TUNING THE SYNAPTIC WEIGHTS IN THE MODEL

For the simulations reported above, the model’s synaptic weight parameters (i.e., the z matrices in the equations) were tuned as follows.

The synaptic weights z_AuM and z_SM, which encode the transformation from auditory (z_AuM) and somatosensory (z_SM) errors into corrective motor commands, were calculated by an explicit algorithm that determines the local pseudoinverse for any configuration of the vocal tract. A more biologically plausible method for tuning these weights was described in Guenther et al. (1998). The pseudoinverse (J(M)⁻¹) is determined by applying a perturbation (δM) of the motor state (M), measuring the resulting change in sensory space (δSensory = [δS,δAu]), and calculating the Jacobian and its inverse as follows:

$$\begin{array}{l}J(M)=\delta Sensory/\delta M\\ z_{AuM}(M)=J{(M)}^{-1}.\end{array}$$

The matrix z_PAu corresponds to the auditory expectation for a given speech sound target. This matrix was set to encode upper and lower bounds, for each 1 ms time slice, of the first three formants that were extracted from the acoustic sample.

The matrices z_PM and z_PS, which encode the feedforward command and somatosensory target for a sound, respectively, were updated during the practice phase. The matrix z_PM was updated using the feedback commands ( ${\dot{M}}_{Feedback}$) generated by the auditory portion of the feedback control subsystem, while the matrix z_PS was tuned based on the somatosensory error (ΔS). To account for temporal delays, these tuning processes align the auditory error or somatosensory error data slice with the appropriate time slices of the weight matrices. The weight update rules include this temporal alignment.

$$\begin{array}{l}{\dot{z}}_{PM}[t-{\tau }_{PAu}]={\dot{M}}_{Feedback}(t)\\ {\dot{z}}_{PS}[t-{\tau }_{PS}]=\mathrm{\Delta }S(t).\end{array}$$