Action knowledge, visuomotor activation, and embodiment in the two action systems

Abstract

Scientific interest in the relationship between cognition and action has increased markedly in the past several years, fueled by the discovery of mirror neurons in monkey prefrontal and parietal cortex and by the emergence of a movement in cognitive psychology, termed the embodied cognition framework, which emphasizes the role of simulation in cognitive representations. Guided by a functional neuroanatomic model called the Two Action Systems account, which posits numerous points of differentiation between structure- and function-based actions, we focus on two of the major issues under recent scrutiny: the relationship between representations for action production and recognition, and the role of action in object representations. We suggest that mirror neurons in humans are not critical for full action understanding, and that only function-based (and not structure-based) action is a component of embodied object concepts.

Action and cognition in transition: seeds of the embodied cognition movement

Recent years have witnessed increasing scientific excitement about the role of sensorimotor information in knowledge representation. This interest has coalesced into a movement known as the embodied cognition approach, which, among a number of other tenets, claims that “cognition is based on action,”¹ and in particular, that cognitive processes are simulations of previously acquired sensory and motor experiences. Recent studies of embodiment phenomena have not often made contact with a rich tradition of prior cognitive neuropsychological studies in the domains of “action semantics” and apraxia, which have long been devoted to understanding the relationship of action and cognition. In addition, the embodied cognition approach has been largely mute with respect to the functional neuroanatomic systems underlying embodiment phenomena (but see Grafton² for an expection).

The overarching goal of this paper, then, is to assist us in seating the embodied cognition movement relative to these two other reasonably well-developed streams of scientific inquiry, namely, the domain of action semantics and the functional-neuronatomic study of the brain’s action systems. We will apply our framework, called the “Two Action Systems” (2AS) account, to the question of whether there is sufficient evidence to conclude that action information is a component of object representations. We will also review the claims of the 2AS account with respect to the functional neuroanatomy of action production and recognition. To help focus the endeavor, we will not address the interesting question of the relationship of action representations to language (e.g., verbs). A secondary goal, given a recent scientific climate that has favored evidence from functional neuroimaging,³ is to point out the unique understandings enabled by cognitive neuropsychological studies of lesioned patients; consequently, we will occasionally make note of important case and group studies with patients.

It is hoped that by situating the investigation of embodiment more robustly with respect to other powerful bodies of evidence in cognitive neuroscience, we may sharpen our ability to predict the circumstances under which embodiment phenomema may (and may not) occur, as well as to lay the groundwork for future directions of the study of the relationships between cognition and action.

Functional neuroanatomy of cognition and action: a brief recent history

Current understandings of the cognition–action relationship owe a great deal to the important contributions of Ungerlieder and Mishkin,⁴ whose demonstrations of dorsal and ventral visual-processing routes (devoted to spatial localization and visual representation/knowledge, or “where” and “what,” respectively) served to frame subsequent theoretical development as well as countless investigations. Perhaps most prominently, the “two visual routes” account directly paved the way for the work of Goodale and Milner,⁵ who characterized the dorsal route as a system concerned with programming the body for the purpose of object acquisition (i.e., “how” rather than “where”). This distinction is well exemplified in the case of patient DF, a visual form agnosic with lesions in the ventral stream due to carbon monoxide poisoning who was unable to recognize objects. DF could not perceive the orientation of a slot-shaped hole due to deficits in recognizing boundary orientation, yet could correctly orient her hand relative to the slot.⁶ This case was complimented by Jeannerod’s description of patient AT,⁷ an optic ataxic patient with dorsal stream lesions who made significant grasping errors with novel three-dimensional shapes, yet improved markedly when objects of the same shape and size were familiar (e.g., a lipstick case). This double dissociation suggested that the dorsal stream acting alone (in the case of DF) is not well-equipped for object recognition, whereas the ventral stream on its own (in the case of AT) is poorly suited for object acquisition.⁷ Following from these observations, Jeannerod characterized the two routes as the “pragmatic” and “semantic” routes, respectively.^7–9 Subsequent investigations (some to be reviewed later) suggest that the basic framework of the two visual systems model requires elaboration.

Another important contribution to the understanding of the functional neuroanatomy of action came with the discovery of “mirror neurons” in monkey ventral prefrontal and parietal cortex cortex that fire when the monkey observes a human or other monkey interacting with objects, as well as “canonical neurons” that fire at the sight of manipulable objects alone.^10,11 Mirror neurons have been held to be critically involved in various aspects of social cognition, including action understanding, empathy, and “theory of mind.”^12,13 Relevant to this review, mirror neurons have recently been implicated in embodiment phenomena. We will return later to this issue.

Investigations of semantic memory

Contemporary study of the question of whether actions inform cognition has its genesis in the important work of Warrington and colleagues.^14–16 In 1983, Warrington and McCarthy¹⁴ reported a patient known as VER who was disproportionately impaired in knowledge of objects as compared with animals, food, and flowers. This was followed in 1984 by Warrington and Shallice’s report of two patients,¹⁶ JBR and SBY, who exhibited the opposite pattern. To explain this double dissociation, Warrington and colleagues advanced the sensory/functional theory, suggesting that objects are disproportionately represented in terms of their functional significance, whereas the representations of animals, food, and flowers are weighted toward sensory attributes (e.g., shape, size, characteristic sound, etc.). In a further refinement, Warrington and McCarthy¹⁵ reported the case of YOT, who was disproportionately impaired with small manipulable objects, and suggested that functional information in itself is derived from somatosensory information and from action. Since then, there have been numerous reported cases of dissociations of animal and artifact knowledge. Deficits in artifact knowledge are most frequently associated with fronto-parietal lesions, consistent with the involvement of sensorimotor information.^17–20

Around this time, Allport²¹ advanced an account of a distributed semantic memory system emphasizing the role of visual, kinesthetic, tactile, auditory, and action-oriented elements in knowledge representations. Critically, knowledge is represented by different sensory and motor elements to the degree that these elements were activated during knowledge acquisition. Similar accounts were subsequently proposed by Saffran and Schwartz²² and Simmons and Barsalou,²³ among others. These theoretical contributions paved the way for the current scientific excitement surrounding embodiment phenomena.

Action semantics and apraxia

At the same time, a number of investigators were attempting to address the interesting patterns of performance exhibited by patients with ideomotor apraxia (IMA). These patients typically have no difficulty with object recognition, are deficient in performing skilled actions with objects, and even more tellingly, are impaired in recognizing object-related actions. The impact of IMA clearly extends beyond laboratory tasks. IMA patients make more errors with implements while eating than subjects without apraxia,²⁴ and gesture recognition and tool manipulation knowledge are strongly significant, independent predictors of sequencing errors in multistep naturalistic action.²⁵

A long history in the apraxia literature attributes object misuse errors to impaired “action semantics,” specifically, loss of knowledge of the manner in which particular objects are manipulated. This comports with accounts of conceptual knowledge proposing that conceptual information is distributed across the same network of sensory and motor attribute domains activated when the information was first acquired.^26–30 As noted, these accounts posit that knowledge of manipulable objects should be weighted toward sensorimotor elements that record information about how the objects are held and moved (or, more precisely, how the body is moved while using them). Consistent with this, behavioral work from our laboratory suggests that patients with IMA tend to be impaired in knowledge of object manipulation, although relatively spared in knowledge of object function. The IMA literature, then, is broadly consistent with the idea that sensorimotor information about skilled object use is a component of object representations. There is corroborating evidence to this effect from functional neuroimaging from our lab and others.^31–33 Later, we will distinguish manipulation knowledge from other types of sensorimotor information that may not be so closely linked to conceptual knowledge of objects.

Embodying cognition

Planning actions orients attention to action-relevant object features

Against this backdrop of inquiry in the domains of semantic memory and apraxia, the embodied cognition movement in cognitive psychology has focused in part on the claim that action information is a component of object representations. Two types of observations have been cited in support of this claim. The first is the observation that the intention to perform a specific action may affect the salience of objects affording that action, a phenomenon that has been termed “motor-visual attention.” Thus, action plans affect the action-compatible features of objects that are attended. The second is that viewing objects may prime motor responses. With respect to motor-visual attention, it has been demonstrated, for example, that preparation of a specific grasp facilitates attention and response times to detect visual stimuli whose orientations are congruent with the planned grasp, but not incongruently oriented stimuli.^34–36 In addition, categorizing small-or large-sized objects as natural or man-made by performing a precision or power grip results in frontal and parietal functional activations that covary with compatibility between performed and “afforded” grips.³⁷

In several studies performed in our laboratory^38,39 and in a corresponding computational model,³⁸ we have demonstrated that the absence or presence of a motor plan to perform a specific action on a target object substantially modifies the amount and type of interference caused by distractor objects. For example, when a participant knows in advance that the trial will contain a handled cup to be grasped with a power grip, then a distractor object that also affords a power grip is strongly interfering. On the other hand, suppose the participant knows prior to the beginning of a trial only that he or she will act on a target, whatever it may afford (and location of the target is signaled by an irrelevant feature, such as its color), and discovers upon finding the target that it affords a power grip. In this case, distractors affording a different action than the target are strongly interfering (see Fig. 1). The observed interaction reflects the principle that a distractor object can affect reach execution by impacting either object selection or action selection, with the weighting between these determined by the nature of the task or plan. When the action to be performed is known and prepared from the outset, object selection becomes the rate-limiting process, and reach execution is slowed by a distractor that affords the preplanned action. When the action to be performed is instead contingent on the target object, the process of action selection contributes more significantly to reaction time, and reach completion is slowed by a distractor that compels a different action than the target object.

Figure 1.

(Left) Interference (in milliseconds) by distractors matching and not matching the target in structure (shape and size) from Botvinick et al.³⁸ Interference was calculated by subtracting the mean of a no-distractor condition from the corresponding match and nonmatch distractor conditions. (The interference score enables comparison of the unique effects of distractors for trials with the same movement requirements.) In Fixed blocks, subjects knew in advance the precise response they would make. In Variable blocks, subjects were cued to their response by the structure of the object (precisiongrip to a slender projectile object; powergrip to a handle). The pattern of interference by distractors varied markedly as a function of task demands. (Right) Differences between interference scores for distractor conditions (nonmatch minus match) for each block type.

The evidence that action-related interference effects (i.e., the aspects of objects that slow selection processes) can be tuned by intention sounds a cautionary note for embodied cognition theories claiming that action information is a core part of object representations. The waxing and waning of actionrelated effects as a function of task demands suggests the need for a detailed account of the contexts and situations under which these effects may be expected to emerge.

Note that in the experiment just described, as well as other similar studies, subjects intend to act upon objects, and a reaching action toward the objects is prepared in advance of viewing the display. Under such circumstances, we may conclude that activation of action information is observed. The question of whether objects evoke actions when they are passively viewed remains controversial. It is to this topic that we turn next.

Processing objects induces motor resonance

A typical claim in the embodied cognition literature is that identifying a graspable object includes the processing of its action-related attributes,⁴⁰ a phenomenon termed “motor facilitation” or “motor resonance.” For example, graspingactions are potentiated when participants view handled objects,^41,42 and the shape of distractor objects may influence hand shape in reaching to grasp.⁴³ Similarly, distractor objects that strongly affording potential grasping actions (e.g., distractors with handles facing toward the acting hand) slow target selection more than distractors with weaker affordances for grasping.^39,44 However, whether motor activation occurs spontaneously and obligatorily without the intention to act, or rather is mediated by task goals is a matter of debate. In either event, motor resonance effects appear transient in nature, requiring current vision of the objects.⁴² This is an important point to which we will return later.

A number of functional imaging studies showing activation of motor and premotor cortex when objects are viewed or named have been cited in support of motor resonance, and ultimately, the embodied cognition framework.^45–47 At the same time, these studies are unable to provide evidence relevant to whether sensorimotor activation is critical to conceptual knowledge of objects.

Most behavioral studies purporting to show “automatic” (spontaneous, non-task-related) activation of motor actions required subjects to prepare specific hand actions or view hand picture primes prior to target object display. For example, in a recent study by Bub and colleagues,⁴⁸ evidence for motor activation from pictures and words occurred only when subjects performed grasping actions on a manipulandum, and not when they merely touched the manipulandum. Although subjects in the Bub and colleagues study did not act on the stimulus objects themselves, they performed “as if” they were acting on the objects. In light of the evidence for “motor-visual attention” just reviewed, this creates an important ambiguity. It is not possible to tell in these circumstances whether the intention to perform specific hand actions throughout the block of trials (albeit on a “displaced” manipulandum) summoned attention to those features of the objects that afford (or conflict with) those actions, resulting in “top down” priming from that intention.

In another recent study purporting to show automatic activation of motor information from objects, Borghi and colleagues⁴⁹ asked subjects to perform category decisions (man-made or living) on pictured objects. On each trial, display of the object was preceded by a picture of a hand prime in a precision or power grip. They assessed the hypothesis that category decisions, signaled by a button press response, would be faster for objects graspable with a power grip when preceded by a power grip prime, and faster for objects graspable with a precision grip when preceded by a precision grip prime, even though the grip was irrelevant to the category decision task. Critically, the predicted compatibility effect emerged only when subjects were explicitly trained in advance to link the hand posture primes to an actual manual action (i.e., during training they explicitly performed the gesture associated with the prime picture).

What do these results tell us about the “obligatoriness” or “automaticity” of task-irrelevant motor activation? They strongly suggest that motor activation need not be associated with object recognition unless specific motor features are primed. It might be argued that the very fact that motor features may be primed indicates that, indeed, motor features are part of the object representation and tied to the identity of the object. In fact, however, the same pattern of results could be obtained even if motor priming operates external to the object representation.

Consider Figure 2, a simplified box and arrow schematic of some of the cascading processes that may occur during an experiment in which a picture of a hand (trained to be associated with a motor response) precedes some other stimulus to which a response must be made (the imperative stimulus). Assume, for the sake of the thought experiment, that the processes involved in the motor priming effect occur in one processing stream (perhaps the dorsal stream, mediated by mirror neurons), and the processes involved in the object categorization effect occur in another stream (perhaps the ventral stream). The dorsal stream “matches” the shape and size of a shallowly processed imperative stimulus to the prior motor representation evoked by the hand prime. The successful resolution of the visual-motor “matching” process is either facilitation in the case of a match (as when, e.g., the prime is a power grip, and the shape and size of the shallowly-processed object are compatible with a power grip) or alternatively, interference when there is a nonmatch (as when the prime is a power grip and the object is compatible with a precision grip). At the same time, the ventral cascade is “deeply” processing the object, categorizing it, and ultimately sending a signal (perhaps to prefrontal systems) about the resolution of the categorization choice. A motor response is required to signal the choice; however, and so the choice must be “sent” to the motor cascade, which by this point is either operating in a state of facilitation or interference. The ventral cascade converges on a response that is either (relatively) faster or slower as a result.

Figure 2.

Schematic illustration of a possible scenario in which preparation for action may prime responses to an object without necessitating that the action is a component of the object representation. See text for additional explanation.

Even if this particular scenario is incorrect, it serves as an illustration that preparation for action may affect response times on cognitive tasks without requiring that motor features are “part” of an object representation.

In the next sections, we turn to behavioral and functional neuroanatomic evidence with the potential to constrain predictions of the conditions under which we expect evidence of activation of sensorimotor features to reflect processes intrinsic to the identity of an object, rather than an epiphenomenon of processing in a parallel system or stream.

Two routes to action

The observation of a particularly telling behavioral dissociation in the apraxia literature was made by Rothi and colleagues,⁵⁰ who noted that some apraxic patients can imitate gestures they are unable to recognize. They proposed that this pattern suggests that there are two distinct routes to action. A “direct” route that enables translation of visual input (viewed gestures or objects) into motor programs for gesture production, bypassing semantic information, and an “indirect” or “semantic” route that contacts meaning (see Fig. 3). On this model, action imitation can proceed via the direct route even in patients who cannot comprehend the gestures.

Figure 3.

An illustration of the apraxia model of Rothi et al.⁵⁰

The behavioral model of Rothi and colleagues has rightfully been extremely influential in the study of apraxia, but it is probably fair to say that it has not permeated the study of normal action control as much as deserved (but see Noy et al.⁵¹ and Tessari et al.,⁵² for exceptions). The model contains an important insight, namely, that there are different pathways by which action may be processed. The apraxia model has potential implications for theories of embodiment of concepts, as some actions produced in response to gestures or objects may not be conceptual (i.e., not a component of the gesture or object representation).

More recently, we developed an expanded model of two routes to action, based in part on the behavioral model of Rothi and colleagues, and informed by the neurophysiologic evidence for two visual-processing streams (Ungerlieder and Mishkin,⁴ Goodale and Milner⁵). Our expanded model was prompted by the need for a model of disordered action that made contact with the cognitive neuroscience literature on the functional neuroanatomy of the action system. The model was also informed by the observation that most apraxics perform normally when reaching to and grasping currently visualized objects, but show substantial deficits in pantomiming object-related gestures, and reduced, but still significant impairments when using objects.⁵³ This pattern of better online grasping than pantomime/object use doubly dissociates from the performance of patients with optic ataxia, who are frequently impaired in grasping objects, but perform normally in gesture tasks.^54,55

The Two Action Systems model

In response to these observations, we proposed the 2AS theory, which postulates functionally and neuroanatomically separable action systems based on object structure (“affordances for action”) versus functional manipulation.⁵⁶ Related and similar accounts have since been proposed by Johnson-Frey,⁵⁷ Fridman and colleagues,⁵⁸ Glover,⁵⁹ Pisella and colleagues,⁶⁰ and Vingerhoets and colleagues.⁶¹

The bilateral “Structure” system is specialized for acquiring objects based on currently visualized visual information about object shape, size, and location that is constantly updated based on a complex system of spatiomotor transformations of the positions of objects with respect to the retina, eye, head, torso, limb, and hand.⁶² It is activated by objects even when they are not consciously perceived, suggesting that it may not be a “cognitive” system in the traditional sense.⁶³ It appears to respond to structural object properties, consistent with its specialization for prehensile (grasping, pinching) actions upon objects, and to possess only a rapidly decaying sensorimotor memory, consistent with its specialization for online processing.⁶⁴ It appears to be unaffected by priming of previous views of congruent objects.⁶⁵

The left lateralized “Function” system is relatively reliant upon on long-term, conceptual representations.^60,66,67 It computes and stores representations of the core features of skilled actions. Thus, when an action is performed multiple times, the Function system extracts the features of the action that remain constant across instances. By this mechanism, a “hammering” gesture is identifiable regardless of whether it is performed on the wall or ceiling, with large or small strokes, and with a large or small power grip. Its specialization comes to the fore in tasks entailing retrieval of action representations from memory, or in establishing associative links between actions and objects.⁶⁴

The two systems are richly interactive, and in many everyday cases, both contribute to processing. The relative involvement of each system in conceptual and motor tasks is a conjoint function of the capacities of each system, the nature of the task and environment, and the actor’s goals and intentions. These factors bias processing toward one system or another, while not precluding the participation of both systems.^66,68–72

Rizzolatti and Matelli⁷³ recently proposed, from neurophysiological evidence in the monkey, that the dorsal stream is actually subdivided into two streams, a “dorso-dorsal” stream specialized for online control of grasping, and another, “ventro-dorsal” stream specialized for skilled action and action recognition. Both dorsal streams are distinguished from the ventral stream, specialized for object recognition. The neurophysiologic data lend further plausibility to the 2AS theory. Table 1 denotes the specialization of the two systems for different aspects of action, and Figure 4 provides a schematic overview of its functional neuroanatomy.

Table 1.

Overview of the characteristics of the structure and function action systems

Action system	Primary aspects of object coding	Activation of motor information without motor intention	Persistence of information	Relationship to conceptual knowledge	Probable mapping onto apraxia routes	Neuroanatomic Substrate
Structure	Nonarbitrary “affordances” related to current visual information	Yes, motor responses may be activated outside of conscious awareness	Short (msec)	Weak	Direct route	Dorso-dorsal stream: bilateral IPS, dorso-lateral fronto-parietal
Function	Canonical actions; may be distantly related to structure	No, requires relevant intention/goal	Long (min)	Strong	Indirect route	Ventro-dorsal stream: left superior temporal/inferior parietal

Figure 4.

Schematic drawing of the functional neuroanatomy of the posterior components of the dorso-dorsal, dorso-ventral, and ventral streams.

The key neuroanatomic components of the model in the human are the posterior parietal lobe, divided into inferior parietal lobe (IPL) and superior parietal lobe (SPL) by the intraparietal sulcus (IPS), the premotor cortex (PM), middle frontal gyrus (MFG), inferior frontal gyrus (IFG), and posterior middle temporal gyrus (pMTG). In the monkey, the dorso-dorsal circuit runs from the SPL and middle portion of the IPS (areas MIP and AIP) to the dorsal premotor areas and on to area F5 (putative homologue of Broca’s Area; in particular, Brodmann’s Area 44). This circuit is involved in reaching, grasping, and the execution of action.⁷⁴ The anterior intraparietal sulcus (aIPS) in humans appears to represent the grasps afforded by an object, whereas the IFG drives selection of an affordance appropriate to the context.⁷³ The AIP and ventral PM (PMv) are also involved in action observation.^75,76

The ventro-dorsal circuit in the monkey runs from the medial superior temporal (MST) area to the inferior parietal lobule, and there diverges further into a caudal route through areas PG and F7 and a ventral route through F5 and F2.⁷³ According to Rizzolatti and Matelli,⁷³ the mirror neuron system in the monkey falls within the purview of the ventro-dorsal stream. In the next section, we review evidence from human behavioral and functional neuroimaging studies consistent with the 2AS theory.

A sample of relevant evidence from our laboratory

In one of the first studies we conducted to investigate possible differential integrity of functional versus structural actions in patients, we (Buxbaum and colleagues⁷⁷) investigated the ability of patients with IMA to match hand postures to neutral shapes and familiar objects. The task was to choose from an array of four pictures the hand posture “most appropriate for contacting the object.” Despite the deliberately neutral instructions, 10 healthy control subjects and 6 nonapraxic LCVA chose function-appropriate postures on trials with familiar objects normally associated with a functional response (e.g., poke gesture when presented with a typewriter key). With these same objects, nine patients with IMA due to left IPL damage responded erroneously by choosing a structural responses (clench or pinch) that matched the structural, but not functional, properties of the object (e.g., pinch gesture in response to the typewriter key). In contrast, no group differences were observed when subjects responded to the neutral objects. These data suggest the possibility that in the face of degradation of the functional representations associated with familiar, but not novel, objects, IMA patients rely upon their intact Structure system for calculating responses.

In a follow-up study with healthy participants using fMRI, we (Buxbaum et al.⁷⁸) assessed the hypothesis that a left-hemisphere-lateralized system including the IPL is specifically activated by the requirement to recognize hand postures for functional object use. Fifteen healthy subjects viewed pictures of manipulable objects and made yes/no decisions about whether it would be appropriate to respond to that object with (1) a pinch or clench for a structural grasping response (Grasp condition), (2) a pinch or clench for a functional use response (Prehensile Use condition), or a palm or poke hand posture for a functional use response (Nonprehensile Use condition). Despite the fact that the conditions were equated for behavioral difficulty, significantly greater activations were observed in the left IFG, posterior superior temporal gyrus (STG), and inferior parietal lobule (IPL) in Nonprehensile Use trials as compared to Grasp trials. In comparison to Prehensile Use trials, Nonprehensile Use trials were associated with greater activation in the left IPL only, suggesting that additional parietal processing is required for nonprehensile Use judgments. These data confirmed the importance of the left IPL in storing knowledge of hand postures unambiguously associated with functional object responses, and comported well with our earlier findings in patients.

Our earlier studies with patients (e.g., Ref. 77) suggested that damage to left IPL structures mediating functional responses forces IMA patients to rely on structural object information extracted by the dorso-dorsal stream. One way to frame this benefit of structure is in terms of “affordances” for action.⁷⁹ We (Barde et al.⁸⁰) explored whether the “affordance match” between novel objects and novel gestures might play a role in apraxics’ abilities to learn the gestures. We trained 12 IMA patients, 4 LCVA nonapraxics, and 6 controls to match novel gestures to novel tools. Stimuli were eight black-and-white line drawings of archaic Finnish tools that were novel to our subjects,⁸¹ each paired with a video clip of a meaningless gesture. On half the trials gestures were highly afforded by their associated object and on half poorly afforded as judged by prior piloting work. In 20 blocks of trials over two sessions, patients imitated each gesture followed by a multiple-choice recognition test. Apraxics, but not LCVA patients or controls, demonstrated better recognition of High- than Low-Afforded gestures, and apraxics’ performance with High-Afforded (but not Low-Afforded) gestures was normal. Lesion overlap analysis revealed that patients benefiting from High-Affordance were more likely to have a relatively inferior pattern of damage focused in the STG that spared Brodmann’s 44 and 45 and superior portions of the parietal lobe. These results indicate that apraxics may rely abnormally on object structure when learning to associate novel gestures and tools.

Thus, there are two routes to action. Patients with left hemisphere lesions and apraxia exhibit the symptoms of damage to the Function system (the ventro-dorsal stream) in the context of intact structure-based motor control in the Structure system. In the next sections, we consider the perspective of the 2AS account on the questions of (1) the neural regions subserving action production and recognition, and (2) the types of actions that may be embodied in object representations.

Two routes to action, gesture representations, and mirror neurons

Driven by the discovery of mirror neurons in inferior frontal and parietal regions in the macaque, one of the questions that has inflamed scientific interest of late is whether the same representations subserve action production and recognition. Although claims about the capacities of mirror neurons vary widely, at the most extreme is the suggestion that mirror neurons subserve action semantics, namely “what the action is about, what its goal is, and how it is related to other actions” (Nelissen et al.,⁸² p. 332). We will examine this type of strong claim in this section, with an eye toward the need to develop a more precise analysis of what mirror neurons may and may not accomplish.

In the last decade, numerous studies with brain lesioned patients have provided evidence for the involvement of the left IFG and adjacent premotor cortex (PM) as well as the inferior parietal lobule (IPL) in a range of action-related tasks. Apraxic patients with IPL lesions are impaired in the execution of meaningless gestures,^83–87 imitation of intransitive^87–90 and transitive gestures,^83,91 and performance of transitive gestures to the sight of objects.^83,90,91 In addition, apraxics with IPL lesions patients show deficits in recognition of pantomimes^84,86 and limb action-related sounds.⁸⁸ Similarly, lesions in IFG and PM have been related to impaired performance in pantomime execution⁹² and recognition^89,93–95 as well as mouth actionrelated sounds.⁸⁸ Finally, action production and recognition are significantly correlated (Buxbaum et al.;⁸⁴ but see Negri et al.⁹⁶). Taken as a whole, these findings appear to corroborate the general interpretation that mirror mechanisms within the left inferior frontal and parietal cortices are responsible for both action imitation and understanding.

We suggest that this interpretation merits refinement. In a review of this issue, Hickok⁹⁷ raised several objections to the claim that mirror neurons support action understanding by “transforming visual input into knowledge” (Rizzolatti and Craighero,¹⁰ p. 172), and among other arguments, suggested that nonmirror mechanisms (e.g., in the temporal lobe) may support action understanding. Most importantly, he argued that “mirror” responses may merely represent a facilitation of the motor system by learned associations without subserving action semantics (see Mahon and Caramazza,⁹⁸ for a similar argument).

We are sympathetic to these arguments, and will argue later that the lesion evidence to date provides little to no support for the claim that mirror neurons, in isolation, are sufficient for the understanding of actions. At present, however, two major issues hinder our ability to draw definitive conclusions from the patient literature. The first is that there are a number of different methods used to delineate and statistically compare lesions, each with differing sensitivities and weaknesses. The second issue (as in the fMRI literature) is that the use of slightly different tasks may lead to very different conclusions about the relevance of a given brain region for the hypothesized function. Accordingly, before describing a small sample of studies relevant to the question of the substrates of action production and recognition, we first briefly review the strengths and weakness of the lesion analysis methods in current use.

Mapping cognitive deficits to specific brain regions using lesion analyses: benefits and limitations

Although lesion analyses have the merit of providing relatively informative evidence on the question of whether a given brain region is necessary to perform a given task, different lesion analysis methods may lead to differing conclusions. In group comparison studies, behavioral measures are typically related to anatomical sites in three ways: (i) by using patient groups with lesions limited to a broad region of interest (e.g., Halsband et al.⁹⁰); (ii) by extracting the critical anatomical differences between two groups differentiated on behavioral scores, such as apraxic and nonapraxic participants (i.e., lesion subtraction, e.g., Buxbaum et al.⁸⁴); (iii) by comparing the degree of overlap in the brain areas that are the most frequently altered in the behaviorally impaired group with the incidence of damage in the same areas in the unimpaired group (i.e., lesion overlap, e.g., Haaland et al.⁸⁷).

Group comparisons commonly identify a set of cortical areas that are involved in a particular ability. Although providing important probabilistic information, the group comparison methodology is likely to emphasize the role of relatively broad brain regions in a given cognitive process. For example, the 17 apraxic patients in the study of Haaland and colleagues who were impaired in gesture imitation show 80–100% overlap in the middle frontal gyrus and 60–80% overlap in the superior and inferior parietal cortex surrounding the intraparietal sulcus. From these results, we can conclude that one or several areas within the fronto-parietal cortex, considered in isolation or in interaction with other regions, may be critical for the ability to imitate gestures. However, the role of delimited frontal and parietal structures cannot be determined.

Voxel-based methods such as voxel-based lesion-symptom mapping (VLSM) (e.g., Bates et al.⁹⁹) can provide greater specificity. At each voxel, VLSM compares the scores of participants with and without damage and determines whether the scores differ significantly. This results in a statistical map of the voxels (vs. the regions) that are significantly associated with performance on a particular behavioral task. VLSM has the merit of using continuous behavioral measures such as percent correct responses, obviating the need to artificially group patients by lesion or behavior. VLSM’s major drawbacks are that it requires large sample sizes for significance, as well as the fact that it may fail to detect networks of regions involved in a given task. Thus, if the task relies on a distributed architecture in regions A, B, and C, but with different weightings on these three regions in different subjects, VLSM analyses may fail to achieve significance for any of the regions.

With these caveats in mind, we next review two recent studies of gesture recognition that have been interpreted as strong support for neural systems subserving a direct matching between action production and action understanding.

Determining the role of the IFG in gesture recognition: further caveats

Pazzaglia and colleagues⁸⁹ recently used VLSM to demonstrate that impairment in a gesture recognition task is critically associated with lesions of the IFG but not IPL. Similarly, Fazio and colleagues⁹³ demonstrated using lesion overlap analyses that action comprehension deficits were associated with IFG and not IPL lesions in a small group of six aphasic patients. These findings contradict previous results from numerous group comparison studies that have shown that apraxic patients with IPL lesions are more impaired than other patient groups in gesture imitation and recognition tasks (e.g., Buxbaum et al.;⁸⁴ Heilman et al.;¹⁰⁰ Rothi et al.;¹⁰¹ Ferro et al.;¹⁰² Halsband et al.⁹⁰). Aside from the fact that lesion analysis methods vary widely across studies, subtle but important differences in task requirements may contribute to the differences between studies. Given that our goal is to understand the precise role of frontal and parietal cortices in action understanding, it is critical to consider these task differences, and we do so in some detail here.

In the task of Pazzaglia and colleagues,⁸⁹ subjects, including 33 left hemisphere stroke patients, watched a video of a transitive or intransitive gesture and decided on each trial whether it was correct or incorrect. Transitive gestures depicted actions performed with objects; on “incorrect” trials, an incorrect object (either semantically related or not) was substituted for the correct object. Intransitive gestures were symbolic actions such as hitch-hiking; “incorrect” trials were created by changing the spatial hand or finger configuration. Thus, the required discriminations were apparently multifactorial. In addition, the task was quite difficult: apraxic subjects performed at only 68% correct (Mn. 20.4 of 30) overall.

The task administered by Fazio and colleagues⁹³ required temporal sequencing of four pictures extracted from a short action movie. Performance was compared for low-level, simple transitive and intransitive human actions on the one hand (e.g., grabbing a bottle, turning one’s head and pointing) and physical events on the other hand (e.g., door closing). Note that the simple actions presented in the human action condition differ quite markedly from the transitive and intransitive actions classically used in the gesture recognition literature (e.g., toothbrushing, hitch-hiking). Patients were more impaired than controls for ordering human actions but not physical events. Moreover, their capacity to order transitive human actions was correlated with sequencing of linguistic materials (sentence segments and word syllables).

The findings from these two studies contrast with the results we obtained in two large studies of left hemisphere stroke patients with a different task.^84,103 In both studies, participants heard and simultaneously read an action word and then watched two transitive pantomime videos (no objects were used). The written word remained in view during the entire trial. The task was to select the video associated with the action word (e.g., hammering). In the semantic condition, the foil corresponded to a semantic error (e.g., “sawing” instead of “hammering”). In the spatial condition, the foil was incorrect by virtue of the posture of the body part or the amplitude and timing of the movement. In our first study, we found that apraxic patients with overlapping lesions in the IPL and the IFG were more impaired (Mn. 80%) correct than nonapraxic patients (Mn. 95% correct) in both recognition tasks, and that in the spatial recognition task, the results were not a function of general language comprehension impairment. Lesion subtraction analysis showed that poor performance in the spatial recognition task was associated with IPL but not IFG lesions; the latter did not even approach significance.

In a recent study, we confirmed the absence of critical involvement of the IFG in these same gesture recognition tasks using VLSM methodology.¹⁰³ Importantly, in this study subjects were also tested on a verb comprehension control task requiring participants to match the very same action words (e.g., hammering) to pictures of objects (e.g., hammer). Even controlling for verb comprehension, semantic gesture recognition was associated with temporal lesions, whereas spatial gesture recognition was associated with IPL damage. Again, there was absolutely no evidence for IFG involvement.

Both the studies of Pazzaglia and colleagues and Fazio and colleagues were elegant and carefully performed. What, then, might explain the differences between all of the studies reviewed? One possibility is that the differences between Pazzaglia’s findings and ours reflect, at least in part, differences in task difficulty: the IFG is well-known to mediate difficult selection tasks (e.g., Thompson-Schill et al.¹⁰⁴). Second, the lesion analyses of Pazzaglia and colleagues considered transitive and intransitive gestures as an aggregate, the former requiring semantic discrimination and the latter requiring spatial discrimination. It is conceivable that as a result there was reduced power to detect lesions (e.g., in the parietal lobe) related to deficits in either type of discrimination alone, along with increased power to detect meta-task capacities (such as executive function). Finally, the Pazzaglia task—to judge the “correctness” of an action—could arguably be accomplished based on recognition of the familiarity of a structural description of the action without the necessity of contacting full action meaning. Similar arguments have been made in the case of patients who are able to judge whether stimuli are real objects or not, but nevertheless are unable to name them or match them to their names (Warrington and Taylor).¹⁰⁵ For review, see Warrington et al.¹⁰⁶). Thus, the full semantic network may not need to be contacted in the “correctness judgment” task.

The task used by Fazio and colleagues emphasizes action sequencing, putatively supported by motor simulation (imagery). As with the Pazzagila task, the low-level tasks used by these investigators assess goal-directed actions, but could plausibly be accomplished without full access to functional action meaning (i.e., action semantics). Interestingly, Fazio and colleagues note that when their subjects with IFG lesions were queried as to the global meaning of the action sequences, they were largely correct.

Consistent with the possibility that the IFG may encode sequential aspects of action, Fadiga and colleagues¹⁰⁷ proposed that the IFG and the adjacent ventral premotor cortex play a key role in the processing of syntactic structure of sequences of stimuli whether in the language or motor domain. Indeed, both linguistic stimuli and motor acts are hierarchically organized and composed of simpler elements. In the case of action processing, the claim is that the IFG supports the organization of sequences of elementary gestures that compose complex actions such as pantomimes. On this account, this syntactic role of the IFG reflects an evolution of the original capacities of the mirror neuron system, which functioned to store a “vocabulary of motor acts” available during both action execution and observation. Thus, the IFG (as part of a largely evolved mirror system in humans) may be involved in concatenating logical sequences of goal-directed movements composing skilled actions.

A related account of the role of the IFG in action was proposed by Newman-Norlund and colleagues,¹⁰⁸ who on the basis of a failure to demonstrate in an fMRI study that IFG distinguishes meaningful from meaningless actions, suggested that IFG is involved in encoding any goal-directed actions, whether meaningful or not (IPL, on the other hand, was found to distinguish semantically meaningful from meaningless actions).

On the basis of the available evidence, we suggest that the representation (and thus, recognition) of skilled functional action is a distributed cognitive function supported by the ventro-dorsal stream (the Function system), acting in concert with the Ventral Stream (see Valyear and Culham¹⁰⁹). Mirror neurons in the IFG are likely to be involved only in circumscribed aspects of the processing of human actions, particularly those based on goal-directed and/or sequential movements, even if relatively low level (e.g., object grasping). The interplay between parietal, temporal, and frontal systems in the processing of different aspects of action remains an exciting area for further investigation.

Two routes to action: relevance for embodied cognition

Bearing in mind the different types of action information computed by the dorso-dorsal (Structure) and dorso-ventral (Function) systems, let us now return to the question of whether object perception obligatorily entails activation of all associated action representations. To address this question, Steve Jax and I¹¹⁰ recently examined the degree to which response conflict might be observed when objects (such as a pump soap dispenser, toaster, or calculator) specify different functional and structural responses, and whether recent prior experiences with the objects influenced this pattern. To do this, we elicited functional and structural responses by instructing participants to produce hand postures based on how the object is skillfully used (functional response) or grasped to be moved (structural response). Data from 14 healthy college-aged subjects show that time to initiate functional use responses (preparing to poke a keyboard or drink from a glass) are significantly slower with objects such as a computer keyboard whose functional response (poke) differs from its structural response (clench; i.e., it is a Conflict Object) than in objects such as a drinking glass in which the two responses are the same (Non-conflict Object). In other words, structural responses interfere with the preparation of functional responses.

We also assessed the time course of activation of structural and functional information. The data showed that structural actions are made quickly, except when subjects have prior conflicting experience making functional responses with those objects. In this latter case, persistent activation of functional representations appears to affect planning of structure-based actions. We interpreted these results in terms of a race between the activation of function- and structure-based actions wherein structure-based actions are rapidly elicited by objects but quickly degrading, whereas function-based activations are slower but maintained over an interval of at least several minutes, producing short-term interference effects.

To our knowledge, these are the first studies to explore the temporal characteristics of interference between distinctly different, but extremely common, action representations evoked in a naturalistic context by single objects (see Bub et al.,¹¹¹ for related work).

These data have several implications for theories of embodiment of concepts. First, they indicate that single objects may evoke more than one type of action representation that may be in conflict; thus, the claim that object perception is associated with activation of motor information must be tempered to specify which type(s) of motor information is at issue. Second, the data show that these two types of action representation have different temporal characteristics, likely as a function of the action system from which they emanate (also see concept of “temporary” vs. “stable” affordances; Borghi and Riggio⁷²). Functional representations may induce long-lasting interference that lasts for many minutes, as is typical of semantic memory (cf. Damien and Als¹¹²). Thus, unlike structural representations, which are rapidly decaying and computed de novo based on the position and size of the object with respect to the viewer, functional use information has characteristics that make it a likely component of distributed object representations.

Conclusions and suggestions for future study

We have argued that the study of embodiment phenomena would be much enriched by consideration of prior literature in the domains of action semantics and apraxia, as well as by a detailed consideration of the functional neuroanatomy of action. Accordingly, we have proposed a functional neuroanatomic model of two action systems that are differentially specialized for stored functional manipulation knowledge versus online calculations of structural information; this model appears to parallel a more recent explication of a subdivision of the dorsal stream in the monkey. We have explored the implications of our distributed account of action representation for theories positing that mirror neurons play a critical role in action understanding and action semantics. Finally, we have provided data to suggest that only the representations of the functional use system have characteristics that render them likely candidates for the role of embodied components of object concepts.

A number of areas appear to require additional exploration. Given that action systems are relatively widely distributed, additional studies in patients with well-characterized lesions, using carefully designed tasks, are needed to understand whether mirror neurons in humans play a necessary role; this question is not readily addressed with functional neuroimaging. In the realm of embodiment phenomena, there should be greater consideration of the role of task demands in altering the phenomena under study. One open question is whether passive viewing of objects may under any circumstance induce motor resonance phenomena without a prior intention to act in object-compatible ways. Another area of inquiry concerns fleshing out the details of the competition that may occur between transient and more stable action representations. For example, of clear relevance to naturalistic action, additional studies are needed to elucidate the characteristics of activation of target and distractor action representations in displays of multiple objects as a function of task demands. Are patterns of interference from nontarget objects different when the actor intends to interact with an object based on its structure versus function? Finally, given that functional action representations appear to have many of the characteristics of semantic memory representations (and may in fact be semantic memory representations), a major question is whether semantic “proximity” effects (e.g., greater interference by closely related distractors than far distractors) may be observed in the domain of functional use representations. Addressing this question will enable us to begin to describe the cognitive architecture of “action semantic space” in much the same way as has been developed in other semantic domains.^113,114