The functional neuroanatomy of actions

Abstract

Our current understanding of the neural basis of semantic memory is informed primarily by studies of concrete objects. However, conceptual knowledge encompasses many other, albeit less concrete, domains. This article reviews evidence from neuroimaging and patient studies that speaks to the neural basis of action concepts and the words that refer to them. These data highlight 2 important principles governing the neural instantiation of semantic knowledge. First, the organization of conceptual representations in the brain parallels perception and action. Action concepts are at least partially represented within modality-specific areas responsible for the perception and execution of dynamic actions. Second, unimodal sensory and motor cortices act as “points of entry” for more abstract action knowledge. Increasingly abstract conceptual knowledge derived from these modalities is represented in brain areas located anterior and centripetal to modality-specific regions. Extending research on the neural basis of semantics to include dynamic and relational aspects of the world gives us a more complete appreciation of the range of cognitive and communication impairments that may be experienced by patients with neurologic disease.

How is knowledge about the world represented in the mind and brain? Most research on this knowledge, referred to as semantic memory, focuses on understanding objects, concrete entities that clearly correspond to verbal labels. Broadly speaking, 2 ways of thinking about the organization of semantic knowledge have emerged from studies of healthy and neurologically impaired participants. On one perspective, semantic representations exist within a unitary system. These representations are amodal and organized according to their relative similarities.^1,2 Patients with semantic dementia who exhibit general semantic impairments across many tasks (e.g., naming, drawing) are taken as evidence for an amodal semantic system.² Semantic dementia is associated with progressive focal atrophy of bilateral anterior temporal cortex,³ an area of the brain with many connections to posterior association areas⁴ and thereby proposed as the location of amodal conceptual representations of objects.³

An alternative perspective posits that the semantic system is not unitary or amodal and instead consists of multiple subsystems that each represents information from different sensorimotor modalities.⁵ Conceptual knowledge of objects arises from the weighted contributions of these modality-specific subsystems. Neuropsychological data also support “sensory/functional” accounts, as evidenced by patients with “category-specific” semantic deficits who show disproportionate impairments for one category of objects. The most widely reported category dissociation is between living things (e.g., animals)⁵ and nonliving things (e.g., tools, artifacts).⁶ By a sensory/functional account, our knowledge of living things is derived primarily from visual attributes (e.g., the shape or color of a camel), while our knowledge of nonliving things is derived from “functional” attributes that describe ways objects are used (e.g., the movements associated with a hammer). Damage to either source of knowledge will produce category-specific semantic deficits (see⁷ for an alternative interpretation). Lesions associated with living vs nonliving things impairments tend to exhibit a coarse temporo-occipital vs frontoparietal division, respectively,⁸ a division that coincides with visual processing streams responsible for the identification of objects and visually guided actions with objects.⁹ More recently, functional neuroimaging studies of healthy participants have also shed light on theories of semantic organization. In particular, the meanings of objects appear to be represented in distributed networks of brain areas that parallel the organization of sensory and motor systems (for a review, see¹⁰); for instance, naming pictures of animals compared to tools activates left medial occipital cortex, an area of the brain also involved in early visual processing.¹¹

Neuroimaging and neuropsychological findings for the involvement of modality-specific cortical regions in the representation of object concepts are also consistent with “embodied” or “grounded cognition” accounts.^12,13 According to this perspective, sensory and motor states are acquired during experience with objects and partially re-enacted during later conceptual processing, a process referred to as “simulation.”¹² An open area of debate concerns whether activity within sensory and motor systems is necessary or sufficient to represent the meanings of objects^14,15; at present, the data do not rule out the existence of additional, amodal representations that work in concert with modal areas to represent object semantics. For instance, according to Damasio's “convergence zone” theory,¹⁶ the sensorimotor attributes of objects are represented in primary and first-order sensory association and motor cortices. Other brain areas implement “convergence zones” that bind high-level conjunctions between patterns of neural activity in low-level, modality-specific cortices (see also^12,17).

By and large, theories of the organization of semantic knowledge have been motivated by research on objects. However, this historic focus on object knowledge obscures the extent of what we understand about the world: we also have knowledge about relations between objects. For instance, some concepts correspond to dynamic actions that unfold over space and time (“kick,” “jump,” and “run”). Knowledge about “entities” (e.g., objects) contrasts with this knowledge about “events” (e.g., actions), the spatial and temporal relationships into which entities can enter.¹⁶ As discussed above, object concepts appear to be—at least in part—represented within modality-specific areas of the brain. Does a similar representational scheme underlie the meanings of actions? The obvious problem that arises is the degree to which sensorimotor attributes are useful in understanding relational rather than object concepts. Actions can be recognized even when they involve diverse, perceptually dissimilar actors or contexts. Thus, one can recognize that a child, a cat, and even financial costs can “jump,” not solely because of their perceptual similarity but because they describe similar relationships between entities in space and time. While members of the same entity category (objects) are intrinsically similar and have richly correlated features, members of a relational category (e.g., actions) may not be obviously similar and share only a sparse resemblance (“jumping-ness”).¹⁸

In this review, we consider what neuroimaging and neuropsychological data bring to bear on the organization of action concepts in the brain. Specifically, how far can sensory/functional and embodied accounts of semantics go toward explaining the observed patterns of data? Given that we recognize actions irrespective of the entities involved, our knowledge of action concepts must extend beyond the rich, specific sensorimotor attributes important for the meanings of objects. Does the neural basis of this more abstract knowledge lie within or extend beyond modality-specific cortices?

The neural basis of object knowledge is at least partially grounded in modality-specific representations of static, intrinsic properties: i.e., color, shape¹⁰ (although the necessity of this grounding is unclear¹⁴). These attributes are stable across space and time. Conceptual knowledge of actions, on the other hand, must draw upon dynamic information that changes rapidly in space and over time.¹⁹ Patients with brain damage often show double dissociations between their ability to name objects and actions,²⁰ suggesting that entity and event knowledge are mediated by different neural networks. Similarly, neuroimaging studies comparing objects with actions implicate distinct networks for these 2 types of knowledge (for a review, see²¹).

Although actions differ from objects, activity within sensory and motor cortices may nevertheless be an integral part of their conceptual representations. Neuroimaging and patient data suggest that actions will recruit a different set of modality-specific areas—those that represent information about dynamic spatial and temporal relations. In particular, 2 kinds of sensorimotor attributes might contribute to our conceptual knowledge of actions: visual motion areas and the motor system.

VISUAL MOTION

In neuroimaging studies, the perception of visual motion—any moving visual stimulus—consistently activates an area at the junction of the inferior temporal gyrus and occipital gyrus, termed MT/MST.²² In fact, even static depictions of actions that imply motion activate MT/MST,²³ and lesions to this area cause akinetopsia, or “motion blindness.”²⁴ Concrete actions (“jump,” “run”) unfold dynamically over time and space, and their perception involves the perception of visual motion. Therefore, motion perception areas of the brain would be a good candidate for a modality-specific semantic subsystem that contributes to action concepts.

In support of this possibility, evidence from fMRI suggests that making conceptual decisions about static depictions of actions (e.g., is a line drawing of someone “digging” more similar to one of “shoveling” or “sewing”?) is associated with activity in bilateral areas in posterior temporal-occipital cortex, including MT/MST (primarily Brodmann areas 37 and anterior 19) relative to the same kind of judgment with objects.²⁵ Other fMRI studies support the finding that visual motion areas are active during conceptual processing of action pictures.^26,27 For instance, naming pictures of actions relative to objects was found to activate bilateral MT.²⁷ Even stronger evidence for area MT's role in action concepts comes from patient studies. In one study, the 2 patients with damage to area MT were both impaired on a test of action picture naming,²⁸ although their performance on other categories (e.g., objects) was not tested. Additionally, lesions to the white matter underneath lateral occipital-temporal cortex are associated with impaired judgments about pictured actions.²⁹ However, damage to this posterior middle temporal area is also associated with impaired tool naming, so the involvement of visual motion areas in conceptual processing is not strictly limited to the category of actions. Rather, area MT may participate in the representation of any concept for which visual motion is a critical feature.²⁹

Although visual motion areas may be involved in deriving the meaning of actions from visual input, these same areas, in principle, need not be involved when meaning is accessed via language. On a strong embodied view, words will activate the same sensory and motor neural structures that were active during initial experience with the words' referents in the world.¹² Consistent with this view, recent neuroimaging studies examining the neural substrate of action word (verb) comprehension have found increased activity in posterior lateral temporal cortex near area MT.^25,30,31 Similarly, a recent neuropsychological study found that lesions to posterior middle temporal gyrus impaired patients' ability to match an action word to a video of that action.³² However, many fMRI studies have not functionally localized area MT within each participant's brain, so they are unable to determine whether or not the meanings of action words recruit the same area involved in visual motion processing. In some studies that have functionally localized MT, action words yielded greater activity in an area not identical to MT,³³ but located slightly more anteriorly and centripetally.^25,30 However, Bedny and colleagues³³ found no evidence that this area encodes visual motion properties and propose instead that the region responds to the grammatical category of verbs or abstract event semantics (e.g., thematic role). In contrast to these studies with single words, area MT does seem to be active when action words are presented within sentences.³⁴ One possibility is that the deeper the engagement of the semantic system by language, the more likely that posterior, lower-level visual motion areas will be engaged, perhaps because visual images of actions are more likely to be evoked.

With few exceptions (e.g.,³⁴), most investigations into the neural basis of action verbs have presented words in isolation. However, without a sentence context, it is difficult to appreciate the role of verbs as the linchpin of natural language: they tell us who is doing what to whom and relate the constituents of a sentence to each other. Some patients appear to have deficits that specifically affect “thematic role” information about actions,^35,36 but little is known about the neural underpinnings of this more abstract kind of action knowledge and the degree to which it is independent from other aspects of action semantics. A recent voxel-based lesion-symptom mapping (VLSM) study addressed this question by testing the intactness of aphasic patients' thematic role knowledge relative to their knowledge about spatial relations (locative knowledge, e.g., “above,” “under”).³⁷ VLSM techniques correlate behavioral impairments with the likelihood that a given voxel (in standardized brain space) is lesioned.³⁸ In the experiment, patients were asked to match the meaning of a simple sentence (e.g., “The circle kicked the square”) to one of 4 line drawings that depicted the event (e.g., a circle character kicking a square character); this task required patients to make thematic role discriminations, for instance, to avoid selecting a drawing of a square kicking a circle. The VLSM analyses revealed that thematic role knowledge deficits were associated with lesions to the middle and superior temporal gyri. This area in the lateral temporal lobe was located anterior to the area active during processing of single action words in previous neuroimaging studies,³⁰ and even more anterior relative to area MT itself. Similarly, comprehending sentences in which action verbs were used metaphorically (e.g., “The man fell under her spell”) relative to literal sentences (e.g., “The child fell under the slide”) activated areas of the brain anterior and dorsal to MT (left posterior middle temporal gyrus and inferior frontal gyrus).³⁹ The proximity of the brain regions implicated by thematic role knowledge and the figurative use of action verbs to visual motion processing areas suggests that more abstract knowledge of actions—knowledge not tied to specific sensorimotor attributes—may be derived from visual motion information.

MOTOR SYSTEM

Action understanding frequently draws upon visual motion information, but other sources of sensorimotor information may contribute, as well. In particular, many actions are associated with specific motor programs: for instance, “to jump,” “to hammer,” and “to run.” On sensory/functional and embodied accounts of semantics, activity in the motor system may be a useful source of information about an action during the acquisition of conceptual knowledge.¹³ On embodied accounts, in particular, conceptual processing of actions will involve the partial reenactment of motor states.^12,13

One way in which this motor system activity may arise is through the “mirror neuron” system. The mirror neuron hypothesis originated with the observation that neurons in the F5 sector of the macaque prefrontal cortex discharged both when a monkey performed actions and when the monkey observed actions⁴⁰; neurons with similar properties were also identified in the inferior parietal lobule.⁴¹ In humans, frontal and parietal circuits similarly appear to be recruited when healthy participants perform and observe actions.⁴² Activity within these mirror system areas differs depending upon whether the actions are transitive or intransitive: while both types of actions activate frontal areas, actions that involve an object additionally activate parietal areas.⁴² Recent evidence suggests that the mirror neuron system is more distributed than previously thought; neurons that respond to both action execution and observation have been identified in humans via single-cell recordings in medial frontal and medial temporal cortices.⁴³ In sum, the mirror neuron hypothesis posits that activity within these mirror neuron ensembles allows us to understand the actions of others.

Whatever the mechanism (mirror neurons or otherwise), activity within the motor system is a good candidate for contributing to action concepts. Consistent with this view, naming static depictions of actions is associated with activity in a network of frontoparietal regions, including left motor and premotor cortex.^44,45 However, not all neuroimaging studies of visually presented actions have found activity in frontal or parietal cortex,²⁵ suggesting that the motor system's participation in action concepts varies by task or with the specific stimuli used. Furthermore, although damage to premotor and prefrontal areas is associated with impaired action relative to object naming, lesions to other brain areas (i.e., temporo-occipital cortex and underlying white matter) also produce action naming deficits.²⁹

Activity within the motor system has also been observed when the meanings of actions are accessed by language^46,47 (but see^25,30). The patterns of neural activity produced by action words differ according to the specific effector involved (e.g., hand vs foot verbs),^48,49 though not necessarily in accordance with the traditional somatotopy of the primary motor system.¹⁵ Moreover, individuals with ALS and accompanying aphasia have selective difficulty with action verb processing,⁵⁰ and the degree of their action knowledge deficit correlates with cortical atrophy in motor-associated brain regions.⁵¹

However, unlike parallel findings for the role of visual motion areas in action language, studies investigating the role of the motor system frequently lack the anatomic specificity needed to make claims about the location and extent of motor engagement during the conceptual and linguistic processing of actions.¹⁵ Additionally, not all neuroimaging studies of action words find motor system involvement.^25,30 One explanation for this discrepancy is that the motor system is active primarily during the comprehension of actions with which we have direct experience.⁵² Consequently, only words referring to this subset of actions will engage the motor system.⁵³ Because we can speak about and understand actions with which we do not have motor experience—like “slithering snake” or “swarming bees”—motor attributes may make a lesser contribution to the meanings of actions than visual ones. However, the relative weightings of different types of sensory and motor information to action concepts remains an open question. Although we predict that visual motion information will be important for a larger number of verbs than motor information, both aspects appear to contribute to the conceptual processing of actions. For instance, Kemmerer and Tranel⁵⁴ report a patient with impaired knowledge of action verbs relative to locative prepositions. Although this patient has a fully functioning area MT and surrounds, extensive damage to left premotor and prefrontal cortex appears to cause his action verb deficit. Thus, at least for this patient and the stimuli on which he was tested, intact area MT alone was not sufficient to support accurate action verb processing.

SOME PRINCIPLES OF NEURAL ORGANIZATION UNDERLYING ACTIONS

The findings reviewed here speak to the neural basis of actions, an aspect of conceptual knowledge that historically has been overlooked. Taken together, neuroimaging and neuropsychological results highlight 2 important principles governing the neural organization of semantic knowledge. First, conception parallels perception and action. Object concepts are at least partially represented in areas of the brain devoted to the visual processing of static properties (i.e., color, shape),¹⁰ and, similarly, action concepts appear to rely on areas of the brain devoted to visual motion processing (posterolateral temporal cortex) and the motor system (a frontoparietal network). The presence of activity in these modality-specific areas during action concept and action word processing is consistent with a modality-specific, embodied view of semantics. On this same view, however, visual motion and motor movement need not be the only kinds of information relevant for action concepts. For instance, proprioceptive information about the position of one's body in space also plays a role in action understanding.⁵⁵ Further, other sensory modalities like touch and audition may also be relevant for the conceptual representations of some actions. In this way, the neural basis of action concepts lies within a network of areas that are also implicated by the perception and execution of dynamic actions.

A second neural principle that emerges from the evidence reviewed here is that sensory and motor modality-specific cortices act as “points of entry” for the acquisition of more abstract action knowledge. Subsequent to learning, abstract action semantics are represented within areas that lie anterior or centripetal to these same modality-specific regions; this “shift” in the location of semantic knowledge may, in fact, be in the direction of peri-Sylvian language areas in the left hemisphere.¹⁹ For example, the same area of the brain involved in visual motion processing (area MT) is active when individuals need to access the meaning of actions from visual input.²⁵ More abstract action knowledge, like accessing the meanings of words referring to actions,³⁰ or thematic role information,³⁷ is located more centripetally toward peri-Sylvian cortex relative to the areas involved in perception, as revealed by neuroimaging and patient studies. This “point of entry” hypothesis is related to sensory/functional⁵ and embodied^12,13 accounts but with the addition of a topographic organization of knowledge based on degree of abstraction. Because semantic regions anterior and centripetal to modality-specific cortices encode more abstract knowledge than the modalities themselves, these areas may be viewed as “amodal” by comparison. A similar account has been proposed by Plaut,⁵⁶ who used a computational model to investigate the structure of semantic knowledge after learning. Parts of the semantic system proximal to particular input or output modalities developed relative degrees of functional specialization for modality-specific knowledge, while parts of the semantic system located farther away became progressively more amodal.

DISCUSSION

Research on the neural basis of actions reveals the richness of the human conceptual system. Neuroimaging and patient studies will be critical for understanding the way in which our knowledge of relational concepts is represented in the brain, with each methodology offering complementary and convergent evidence to the other.⁵⁷ This research extends our understanding of the cognitive deficits associated with neurologic disease or injury: by shifting the focus from tests of object knowledge to include assessments of actions and other relational knowledge, we will more adequately understand the depth of communication or comprehension problems experienced by patients and design treatments that target these other types of semantic knowledge. Our ability to learn abstract, relational knowledge has been claimed to be at the heart of “why we're so smart.”⁵⁸ Continued research in this vein will illuminate the ability of the human mind to understand both concrete and abstract aspects of the world and the patterns of its breakdown following neurologic disease.

DISCLOSURE

Dr. Watson received postdoctoral fellowship support from an NIH/NINDS training grant. Dr. Chatterjee served as an Associate Editor for the Journal of Cognitive Neuroscience and receives research support from the NIH and the NSF.