Neural bases of action abstraction

Abstract

There has been recent debate over whether actions are processed primarily by means of motor simulation or cognitive semantics. The current study investigated how abstract action concepts are processed in the brain, independent of the format in which they are presented. Eighteen healthy adult participants viewed different actions (e.g., diving, boxing) in the form of verbs and schematic action pictograms while functional magnetic resonance imaging (fMRI) was collected. We predicted that sensorimotor and semantic brain regions would show similar patterns of neural activity for different instances of the same action (e.g., diving pictogram and the word ‘diving’). A representational similarity analysis revealed posterior temporal and sensorimotor regions where specific action concepts were encoded, independent of the format of presentation. These results reveal the neural instantiations of abstract action concepts, and demonstrate that both sensorimotor and semantic systems are involved in processing actions.

1. Introduction

How do we understand and conceptualize actions? Using neuroimaging studies, researchers attempt to explain how the human brain processes actions—whether they are performed, observed, or represented by words or other symbols. Some researchers theorize that actions are processed by engaging our own sensorimotor networks—in other words, that we understand actions by vicariously simulating perceptual, sensory, and motor states associated with an action (Barsalou, 2008; Gallese & Sinigaglia, 2011; Rizzolatti & Sinigaglia, 2010). This view is closely tied to the mirror neuron theory of embodied action understanding (Gallese, 2013; Rizzolatti & Sinigaglia, 2010) according to which similar neural systems are active whether an action is observed or performed. Other theories of action understanding hold that action simulation is not a primary mode of action understanding, but rather that actions are processed cognitively—i.e., that they are categorized and accessed without reliance on the motor system (Wurm, Ariani, Greenlee, & Lingnau, 2015; Wurm & Lingnau, 2015).

Simulation theories of action comprehension postulate that actions are understood through vicarious activation of the motor system. This simulation can take two forms. Actions can be understood in terms of how the observer would herself carry out the action. Such “action implementation” (Quandt & Chatterjee, 2015) is largely reliant on dorsal streams, including frontal and parietal regions such as the premotor cortex (Michael et al., 2014), primary sensorimotor cortices, and the posterior parietal lobe. From the action simulation viewpoint, the regions of the brain involved in understanding actions are the same neural circuits that instantiate the motor and sensory features of actions (Avenanti, Candidi, & Urgesi, 2013). Similarly, actions can also be understood by simulating memories of having observed them. In this case we expect that neural circuits in or adjacent to visual motion area MT+ would be engaged. There is evidence for action simulation throughout the motor systems in regions including the supplementary motor area, primary somatosensory cortex, premotor cortex, the supramarginal gyrus, and the superior parietal lobe (Grezes & Decety, 2001; Avenanti, Bolognini, Maravita, & Aglioti, 2007).

Other brain regions process actions as cognitive semantic entities rather than by means of simulation, and these regions may also process other categories (e.g., objects or animals) in a similar manner. In this cognitive action semantics framework, posterior regions near the visual system, such as the lateral occipital cortex (LOC), and the posterior middle temporal gyrus (pMTG), along with inferior parietal regions are generally considered to be hubs of action representation (Leshinskaya & Caramazza, 2015; Wurm & Lingnau, 2015). For example, the inferior posterior parietal cortex (Binder et al., 2009) and the MTG (Bedny, Caramazza, Grossman, Pascual-Leone, & Saxe, 2008; Wu, Morganti, & Chatterjee, 2008) are associated with conceptual action associations, while the IFG (Thompson-Schill et al., 1998) and anterior temporal lobes are associated with domain-general semantic processing (Abel et al., 2015). pMTG has been shown to be causally involved in action understanding in patient studies (Urgesi, Candidi, & Avenanti, 2014; Wu, Waller, & Chatterjee, 2007).

While these two frameworks are sometimes pitted against one another, they are not mutually exclusive. Some meta-analyses of action processing demonstrate that both the simulation and the cognitive model of action processing are involved in action processing, depending on the task, the stimuli, and one’s experience with a given action. One such meta-analysis identified regions that are uniquely activated by the observation of actions (Caspers, Zilles, Laird, & Eickhoff, 2010). This analysis revealed that action observation involved both traditional “mirror system” regions (premotor cortex, IPL, primary somatosensory cortex) and also the supplementary motor area, pMTG, and extrastriate visual area. Another meta-analysis identified the left supramarginal gyrus and the left pMTG as subserving “action semantics” (Binder, Desai, Graves, & Conant, 2009). Notably, a recent meta-analysis identified regions involved in conceptual action processing of action words and pictures (Watson, Cardillo, Ianni, & Chatterjee, 2013). Temporal cortex regions adjacent to the visual motion area (MT+) and inferior and superior parietal regions were implicated in conceptual action representations. Along the left lateral temporal cortex, a gradient of abstraction was found, with more abstract action representations housed in more anterior parts of the posterolateral temporal lobe. While these meta-analyses provide important starting points for approaching the question of conceptual action processing, they are inherently limited to explaining specific types of experimental stimuli. For instance, the meta-analysis conducted by Watson et al (2013) included studies that used static depictions of action (e.g., line drawings), but excluded action execution (e.g., participants producing actions) and moving stimuli. Additionally, meta-analyses typically use whole-brain analyses, which identify common activations, but do not directly test specific hypotheses.

Most action-related experiments identify brain regions that process actions when presented in a specific format. For instance, a study might ask participants to view action videos (Calvo-Merino, Glaser, Grezes, Passingham, & Haggard, 2005; Kirsch & Cross, 2015; Quandt & Marshall, 2014; Wurm & Lingnau, 2015), static action pictures (Kable, Lease-Spellmeyer, & Chatterjee, 2002; Kourtzi & Kanwisher, 2000; Watson, Cardillo, Bromberger, & Chatterjee, 2014), or action words (Kable, Kan, Wilson, Thompson-Schill, & Chatterjee, 2005; Papeo & Lingnau, 2015; Willems, Toni, Hagoort, & Casasanto, 2010). While these studies contribute to the understanding of how we process actions presented in a certain format, they are limited in their ability to answer the broader question: what regions of the brain are involved in processing actions, regardless of the format in which they’re presented? For instance, what brain systems process the format-independent concept of “boxing”, rather than simply the English word boxing or a video of a person boxing?

This question of how the brain processes format-independent information can be addressed experimentally, in part thanks to advances in functional neuroimaging analysis techniques. By examining similarities in the neural response to different versions of a concept, one can make inferences about processing in the brain that is common to different instances of the same concept, rather than simply comparing neural activity in response to different stimuli. Researchers have identified brain regions involved in processing format-independent conceptualizations of objects (Devereux, Clarke, Marouchos, & Tyler, 2013; Fairhall & Caramazza, 2013), distances (Parkinson, Liu, & Wheatley, 2014), and letters (Rothlein & Rapp, 2014). In these studies, neural activity during one condition (e.g., viewing object nouns) is correlated with neural activity during another condition (e.g., viewing object images) to see which brain regions have similar responses to different formats of the same object concept (e.g., “cup”).

A critical issue in the action processing literature concerns the format of the actions in question. A given action may be seen in real life (e.g., watching someone throw a ball), or may be referred to by representational means, such as an action verb or a picture of an action. Different formats of action-related stimuli vary in their levels of abstraction. For instance, a high-definition video feed of a football game is richly detailed, and is not particularly abstract, other than being rendered on a flat screen. On the other hand, the word “football” is a symbolic, and highly abstract, representation of the same action concept. One intermediate mode of representing actions is schematic pictograms of action, such as line drawings or stick figures like those used to identify sports during the Olympic games. These are intermediate in the sense that they share some symbolic properties of words and some analog properties of pictures (Amorapanth et al., 2012; Chatterjee, 2001; Kranjec, Ianni, & Chatterjee, 2013). Schematic representations of spatial relations may be the foundation upon which we understand abstract spatial information. Unlike pictures, they are abstract by virtue of being types rather than tokens of actions or relations. Unlike words, they are understood easily and are less subject to cultural variations. For instance, a left-facing arrow conveys spatial direction more readily than the word “left”. Such image schemas (e.g., arrows, lines, or circles representing abstract concepts) may provide a structure that allows us to conceptualize more complex relations between abstract entities (Lakoff & Johnson, 1999; Talmy, 1983). Schematic representations of spatial relations may especially rely on the right supramarginal gyrus (Amorapanth et al., 2012). Other recent work demonstrates that symbolic stimuli preferentially activate the left inferior frontal gyrus (Muayqil, Davies-Thompson, & Barton, 2015).

We aimed to investigate the format-independent processing of actions, by examining the neural processing common to different formats of action stimuli: lexical and schematic action representations. Action schemas preserve the fundamental structure of action concepts, while abstracting away perceptually-rich details present in a less symbolic format, such as a color photograph or a video (Chatterjee, 2001). By examining the similarities in action processing evoked by schematic action images and action words, we characterize the regions of the brain involved in format-independent conceptual action processing (Barsalou, 2008). We designed a functional neuroimaging study in which we showed participants schematic action images and corresponding verbs. We then conducted a representational similarity analysis (Kriegeskorte, Mur, & Bandettini, 2008) in order to test the hypothesis that format-independent action processing would evoke neural activity in brain regions devoted to motor and sensory simulation of these actions as well as brain regions implicated in associative processes (Quandt & Chatterjee, 2015).

2. Materials and methods

2.1 Participants

Twenty participants (12 females; mean age = 25.79, SD = 5.23) volunteered to participate in the study in exchange for monetary compensation. All participants gave their informed consent before participation, and none reported history of neurological abnormality. All were right-handed and native speakers of English. Two participants were excluded from all analyses because of excessive movement throughout data acquisition, for a final sample of eighteen (11 females).

2.2 Stimuli

Four classes of stimuli were created: action pictograms, object pictograms, action words, and object words. All stimuli were presented in black on a white background in E-Prime 2.0. Action pictograms consisted of six schematic images depicting actions (boxing, diving, golfing, fencing, skating, and skiing), taken from the set of 1972 Olympic pictograms designed by Otl Aicher (bottom of Figure 1). These copyrighted images were used with permission from ERCO GmbH (© 1976 by ERCO GmbH, Lüdenscheid, Germany). The six object pictograms (globe, telescope, beehive, beaker, shoe, teapot; Figure S1 in Supplemental Materials) included some images from the Aicher pictogram set, and some clip art images found via web search. Action and object words were verbs and nouns, respectively, which corresponded to the actions or objects in the pictograms.

Figure 1.

A) The Pictogram Similarity comparison, showing regions that have higher correlations for pictogram-pictogram pairs compared to cross-format pairs; B) the Word Similarity comparison, showing regions that have higher correlation for word-word pairs compared to cross-format pairs; and C) the Action Similarity comparison, showing regions that have higher correlations for same action cross-format pairs, compared to different action cross-format pairs. Note that only two actions are shown in A–C, but actual analyses included all six action pictograms (shown in D) and their corresponding action verbs. Action pictograms used with permission, © 1976 by ERCO GmbH, www.aicher-pictograms.com.

2.3 Norming and matching

2.3.1 Pictograms

All pictograms were rated by an independent sample of undergraduate students (two groups of raters, N = 36 in each). Participants in this rating study were shown each pictogram, and asked to provide a name describing what they saw. They then rated the ease of naming, and their own personal experience with seeing and doing the actions. Six action pictograms were included in the stimuli set. Action pictograms were rated as easy to identify (M = 6.28, SD = .45 where 1 = I did not know what to enter and 7 = There was only one entry I considered). The rating groups reported performing actions rarely (M = 1.88, SD = .27 where 1 = never and 7 = very often) and seeing them performed somewhat more frequently (M = 2.63, SD = .52 where 1 = never and 7 = very often).

A group of object pictograms was also included in the experiment, statistically matched to the action pictograms (and words, below) for ease of naming and experience. These object data were ultimately removed from the analysis (see Section 2.6.3 and the Supplemental Materials).

2.3.2 Words

Words were selected based on the names given to the pictograms by the participants from 2.1.1. The six words (boxing, diving, fencing, golfing, skating, and skiing) had a mean length of 6.5 letters (SD = .54), log frequency of 8.08 (SD = 1.08), and naming reaction time of 652.82 ms (SD = 86.38). Lexical characteristics were obtained from the English Lexicon Project website (Balota et al., 2007).

2.4 Procedure

The experimental design involved relatively passive viewing of schematic images and words. While participants saw images and words, they were instructed to prepare to make a response indicating their own personal experiences with the actions. We wanted the participants access their knowledge of the action concept, with no particular goal as to how or why they were referring to it. For action words and images, participants responded with how often they had performed the action themselves (“never”, “at some point in life”, or “in the past month”). Responses were given by pressing a button with the index, middle, or ring finger. Response mappings were counterbalanced across participants so that for half the participants the mapping went from left-to-right across the fingers to indicate never-sometime-recently, and for half that mapping went from right-to-left. Lastly, half the participants used their left hand to answer while half used their right. For object words and images, participants responded with how often they had touched or used the object themselves. The same responses were used, and response mappings were once again counterbalanced. Before the experiment began, participants were led through a small number of practice trials to ensure that they knew the correct response mappings. Practice trials used the same type of task as used in the subsequent scanning session, but a different set of action and objects retrieved from the Aicher pictogram set.

Stimuli were projected onto a screen, which was viewable by subjects in the scanner through a mirror mounted on the head coil. On each trial, participants saw a stimulus (action or object in word or pictogram format) displayed for 2 s. For most trials, this was followed by 4s of a fixation cross (i.e., 4 s inter-trial interval) before the next trial began. For a small number of trials (3 or 4 trials per run, 15.4% of trials overall), the stimulus was followed by a probe question. The assignment of a probe question to a particular trial was entirely random and determined by the presentation software. On these trials, after the stimulus (2 s), a screen showed the text “How often?” (2 s), and participants responded with how frequently they had performed the action or touched the object. The probe question was then followed by 2 s fixation cross. Participant responses were accepted at any time during the display of the probe question or the fixation cross.

Trials were presented in ten runs with a fast event-related design during a single scanning session. Each run was made up of 24 trials – 6 exemplars × 2 formats (picture and word) × 2 categories (action and object), and 6 null events in which a fixation cross was shown for 3 s. The ordering of the trials and the null events was designed to extract optimal signal from the brain in response to each event and to avoid the problem of multicollinearity, as determined using the software program OptSeq2 (Dale, 1999; https://surfer.nmr.mgh.harvard.edu/optseq/). In total, participants completed 300 trials across 10 runs, of which 60 were null events, and 240 were trials of interest. Each run was 2.7 minutes long, and began with 6 s of “ready” screens to prepare the participant for the upcoming run.

After leaving the fMRI scanner, participants completed a questionnaire to assess a) their memory of the items they saw in the scanner and b) their familiarity with the actions and objects they saw. First, participants saw an array of 12 schematic action pictures, 12 object pictures, and 12 each of action words and objects words. They were instructed to circle only those items they had seen in the scanner. Next, they were instructed to give familiarity ratings for each stimulus, using the same scale as in the experimental task.

2.5 Data acquisition and processing

Functional and structural MRI data were collected on a Siemens Magnetom Trio 3T scanner (Siemens AG, Munich, Germany). Functional images were acquired using echo-planar T2*-weighted scans. We collected 50 transversal slices acquired in an ascending interleaved order (TR = 3000 ms; TE = 30 ms; flip angle = 90°; FOV = 220 mm; voxel size = 3.4 × 3.4 × 3 mm) covering the entire cerebral cortex. Functional images were collected in ten runs consisting of 56 volumes each. Structural images were acquired for each participant with T1-weighted MP-RAGE scans (transversal slice orientation; voxel resolution = 0.9 × 0.9 × 1.0 mm; FOV = 240 mm; TR = 1630 ms; TE = 3.87 ms; flip angle 15°). Participants’ heads were fixed with foam pads to minimize head motion.

Data were pre-processed using the afni_proc.py script in the AFNI software package (afni.nimh.nih.gov, Cox, 1996). Specific AFNI commands are given parenthetically below. After discarding the first 2 volumes to allow for magnet stabilization, data were checked for outliers in each volume (3dToutcount). Data then underwent slice-timing correction (3dTshift), alignment to skull-stripped anatomical images in Talaraich-Tornaux space, and warping to standard Talaraich-Tournoux space (3dvolreg and 3dAllineate). A 6.0 mm full-width/half-maximum smoothing kernel was applied to each functional volume (3dmerge). A regression analysis (3dDeconvolve) modeled the time series against the stimuli, and demeaned motion parameters were included as regressors of no interest. TRs with excessive head (> .3 mm) were excluded.

2.6 Data analysis

2.6.1 Univariate Analysis

For the univariate analysis examining Action Pictogram (AP) and Action Word (AW) conditions, a statistical map for each participant was created for each condition. We then computed group-level statistics using the 3dANOVA+ command in AFNI to calculate both the overall effect of AP and AW greater than baseline (null trials), and the comparison between AP and AW.

2.6.2 Multivariate Analysis

We performed whole brain searchlights equipped with representational similarity analysis using in-house script implemented in MATLAB 2014 (MathWorks, Natick, MA, USA). To this end, we used timecourses of BOLD activity in data smoothed with a 6 mm full-width-half-maximum Gaussian kernel. These timecourses were high-pass filtered with a 0.008 Hz cutoff and mean-centered within runs. For each stimulus, we constructed activation vectors by concatenating timepoints that solely segregated signals driven by the stimulus from preceding stimuli. To this end, we relied on canonical hemodynamic function (HRF) such that those times points above the mean intensity of HRF were chosen (Lee et al., 2015). These typically corresponded to second (3 sec) and third TRs (6 sec) after stimulus presentation. We then constructed searchlight spheres consisting of one center voxel and its neighboring voxels within a 3-voxel radius. Within each sphere, we computed Pearson correlations within actions (e.g., skating-word and skating-pictogram) and correlations between actions (e.g., skating-word and boxing-pictogram). The correlation coefficients were averaged across stimulus pairs, such that there was one within-action value computed (the average of all six within-action correlations) and one between-action value computed (the average of all 36 between-action correlations). Then, the coefficients were converted to Fisher’s Z scores. Within vs. between action maps (all within-action correlations, averaged, compared to all between-action correlations, averaged; see Figure 1C) were then constructed by storing the difference between the two comparisons at the center voxel of each sphere for every subject. These individual searchlight maps were submitted to a random-effects analysis (one-sample t-test) to test the group-level effect in SPM8. A voxel-wise significance threshold was set at p < 0.001 and only those clusters that met a cluster-extent based correction threshold (family-wise error rate controlled at a < 0.01; k ≥ 24) across the whole brain were included in the results.

We also ran two multivariate analyses on the action data that were designed to test basic hypotheses concerning stimulus format: Pictogram Similarity and Word Similarity (see Figure 1A–B). These analyses used the same basic logic as the Action Similarity analysis described above, but aimed to see what regions of the brain showed higher correlations for within-format pairs compared to between-format pairs. For the Pictogram Similarity analysis, we computed correlations within format (pictogram-pictogram pairs) and compared these to the between-format correlations. For the Word Similarity analysis, we computed correlations within format (word-word pairs) and compared these correlations to correlations between formats (word-pictogram pairs). The same statistical thresholds were used for these analyses as for the Action Similarity analysis.

2.6.3 Data quality analysis

After running the multivariate analysis on the object data, it became clear that these data were corrupted or otherwise unusable. See Supplementary Materials for a detailed description of the analyses that warranted this conclusion. We ruled out the possibilities that this data quality problem arose from any individual subject, any individual stimulus item, or any errors in scripts or data coding. We performed these quality control analyses on action data as well, and the action data showed none of the same problems as the object data. After exhaustively investigating this problem, we concluded that the problem with the object data was intractable and excluded object data from further analyses. Since the object trials were included as a control condition, we acknowledge a loss of explanatory power of the original design, but remain able to make meaningful comparisons and conclusions based on analysis of the action data.

3. Results

3.1 Behavioral results

The post-scan questionnaire revealed that participants successfully recognized the stimuli from the experiment. Ten participants made no errors. Accuracy scores on the recognition test ranged from 87.5% to 100% correct (mean = 97.2%, SD = 3.78%). Participant experience with the actions was relatively low. The participants rated how recently they had performed each action, on the following scale: 1 = never; 2= at some point in life; 3 = in the past month. The average overall rating for the six actions was 1.51. Exact proportions of responses for the six actions are shown in Figure 2. Data was collected during the summer, making it unlikely that any participants had skated or skied in the past month.

Figure 2.

Stacked bar graph showing participants’ (N = 18) ratings of how often they had performed each action.

3.2 Univariate results

Univariate analyses revealed many regions that were more activated by AP and AW compared to the low-level baseline condition (viewing fixation cross; See Figure 3). These regions included a large cluster centered at the left pMTG, and extending throughout the bilateral occipital and posterolateral temporal cortices (see Table 1). Other regions that were more active for AP and AW than for baseline include bilateral insulae, left precentral gyrus, bilateral cerebellum, and the left IPL.

Figure 3.

A) Results from univariate analyses showing regions activated by Action Pictograms (AP) and Action Words (AW) more than baseline. Warm colors indicate regions more active for AP+AW. B) Regions showing differences in presentation format. Warm colors show regions more active in response to AP compared to AW. Cool colors show regions more responsive to AW than AP. All results are significant at p < .001, cluster corrected to α = .01. Results are presented on the Colin brain in standard Talairach space.

Table 1.

Results from three univariate analyses, showing coordinates, t statistics, Pearson’s r effect sizes, and cluster sizes (k). AP: action pictogram; AW: action word; MTG: middle temporal gyrus; IPL: inferior parietal lobule

Effect	Peak coordinates
	Region	X	Y	Z	t(17)	r	k
AP + AW > baseline	Bilateral occipital and posterolateral temporal cortex
	L pMTG	−43	−58	−1	11.19***	.93	2932
	L insula	−31	17	2	7.55***	.87	124
	L cingulate gyrus	−7	2	44	8.11***	.89	97
	L precentral gyrus	−46	−1	41	6.05**	.82	89
	R cerebellar tonsil	11	−61	−31	7.11**	.86	36
	L MTG	−52	−40	5	7.59***	.89	36
	R thalamus	17	−31	−1	7.85***	.89	33
	L IPL	−42	−39	38	4.83*	.76	29
	L uvula of vermis	−4	−61	−31	8.04***	.89	25
AP > AW	Bilateral occipital and posterolateral temporal cortex
	R cuneus	14	−91	8	11.48***	.94	2750
	R thalamus	20	−28	2	11.00***	.94	25
AW > AP	R caudate / cingulate	20	−42	14	6.70***	.85	31

All coordinates are given in Talairach space.

^*p < .001;

^**p < .0001;

^***p < .00001.

When comparing action word and picture conditions, AP resulted in greater activity compared to AW in the bilateral occipital and posterolateral temporal cortices (the effect was strongest at the right cuneus; see Table 1), and the right thalamus. AW led to greater activity compared to AP in one cluster located in the right caudate and cingulate.

3.3 Multivariate results

Three multivariate analyses were carried out. Two analyses (Pictogram Similarity and Word Similarity) were designed to test basic predictions concerning the format of the stimuli (see Figure 4 and Table 2). The Pictogram Similarity analysis revealed several significant clusters (p < .001, cluster corrected to α = .01) in the occipital cortices bilaterally. The Word Similarity analysis revealed one significant cluster (p < .001, cluster corrected to α = .01) in the right middle occipital cortex and lingual gyrus. It is worth noting that this locus is on the opposite side of where the visual word form area is in the left hemisphere. In the left visual word form area, the Word Similarity analysis approached significance but did not meet our criteria.

Figure 4.

Results from multivariate analyses of stimulus format. A) Regions showing significantly higher correlations for pictogram-pictogram pairs, compared to cross-format pairs (see Figure 1A for example). B) Regions showing significantly higher correlations for word-word pairs, compared to cross-format pairs (see Figure 1B for example). Both A and B show axial brain slices. Activation is thresholded at p < .001, cluster corrected to α = .01. Results are shown on a standard brain in MNI space in Mango (Research Imaging Institute, University of Texas Health Science Center).

Table 2.

Results from three multivariate analyses, showing coordinates, t statistics, Pearson’s r effect sizes, cluster sizes (k), and p values. Coordinates are listed in MNI space.

Comparison		Peak coordinates
	Region	X	Y	Z	t(17)	r	k	puncorr	pFWE-corr
Pictogram Similarity	L occipital pole	−13	−91	4	6.33	.83	116	<.001	.218
	R middle occipital gyrus (BA 18/19/37)	39	−71	19	6.24	.83	285	<.001	.260
	L middle occipital gyrus (BA 19)	−37	−74	10	4.03	.70	33	<.001	1.00
Word Similarity	R middle occipital (BA 19) / lingual gyrus	36	−71	7	5.69	.81	64	<.001	.524
Action Similarity	Bilateral frontal and temporal cortex, cerebellum	−16	22	−8	9.88	.92	6299	<.001	.001
	L dorsal precentral gyrus	−54	−9	49	7.39	.87	68	<.001	.031
	L ventral postcentral gyrus	−54	−19	22	4.87	.76	31	<.001	.944
	L supramarginal gyrus	−64	−36	25	4.74	.75	24	<.001	.970

The third multivariate analysis (“Action Similarity”) revealed regions where within-action correlations (e.g., between golfing word and golfing pictogram) were greater than between-action correlations (see Figure 5). A wide-ranging network of regions showed this pattern. The largest cluster extended through the bilateral precentral gyri, the right ventral post-central gyrus, bilateral medial frontal gyri, right frontal gyri (BA 8/10/11), right pMTG (with activation extending somewhat into the pMTG in the left hemisphere), bilateral middle occipital regions, bilateral superior, inferior, and anterior temporal lobes, certain subcortical structures (e.g., bilateral amygdala, bilateral hippocampus), and part of the cerebellum. Three more clusters were located in the left dorsal post-central gyrus, left ventral post-central gyrus, and the left SMG. The opposite comparison (regions where between-action correlations are stronger than within-action correlations) yielded no significant clusters, which is in line with predictions. Cluster labeling was completed using xjView toolbox in SPM (http://www.alivelearn.net/xjview) in consultation with a neurologist (author A.C.).

Figure 5.

Results of the multivariate Action Similarity analysis. Regions showing significantly higher correlations for the six within-action pairs (e.g., golfing pictogram and golfing word) compared to between-action pairs (e.g., golfing pictogram and fencing word; see Figure 1C). Left and right surfaces are shown on a standard brain in MNI space in Mango. Activations up to a depth of 5mm from the cortical surface are shown with a threshold of p < .001, cluster corrected to α = .01.

4. Discussion

4.1 Format-independent action processing

How do we process the idea of an action? That is, does the brain abstract actions beyond any specific instance and format of presentation? To address these questions, we used a representational similarity analysis of fMRI data to identify regions that show similar neural activity during observation of different exemplars of the same action, across word and schematic pictogram formats. Such common neural activity cannot be explained by visual features since features within formats (word-to-word) are more similar than across formats (word-to-pictogram). A level of abstraction that transcends both pictures and words would presumably drive such neural activity. We found regions displaying this property in bilateral posterolateral temporal, anterior temporal, and primary sensorimotor cortices. We also found this pattern of activity in the right posterior superior temporal/supramarginal gyrus.

As mentioned in the introduction, two broad hypotheses attempt to explain how actions are processed. First, actions might be processed by simulating the production of those actions in our own motor systems, or activating our memories of having seen them (Watson et al., 2014). Second, an action might be understood by retrieving cognitive (e.g., semantic) information associated with the action (Wurm & Lingnau, 2015). For instance, processing a stimulus referring to boxing might call upon semantically-related concepts, such as fighting or punching. These two kinds of action knowledge have their own neural signatures (Quandt & Chatterjee, 2015). We consider each possibility as it relates to our findings and propose that both kinds of processing occur when processing the concept of an action that is independent of the form of its presentation.

4.2 Action simulation

The first of the two action processing frameworks we aimed to investigate was action simulation, in which vicarious activation of the motor system facilitates the processing or understanding of an action concept. The Action Similarity analysis was designed to test whether vicarious sensorimotor activity or visual motion responses were sensitive to the concept of the action, across different formats of presentation.

The findings presented here are consistent with the idea that abstract representations of actions, independent of format, engage these embodying systems. We found neural activity within pre- and post-central gyri, and premotor cortex/middle frontal cortex. The primary motor and somatosensory cortices, in particular, may be processing these action-related stimuli by vicariously simulating the motor plans or sensory consequences associated with producing the action (Gazzola & Keysers, 2009; Kirsch & Cross, 2015; Michael et al., 2014). In our experiment, when a participant saw the boxing pictogram, his motor plan for how he would box was activated, and this motor plan is similar to the activity evoked by seeing the word “boxing”. This finding corresponds with prior work demonstrating that primary sensorimotor cortices respond similarly to different exemplars of the same action (Watson et al., 2014), and with recent work linking premotor cortex activity to retrieval of specific action concepts (Lin et al., 2015).

The middle temporal gyrus is included in the results of the Action Similarity analysis, suggesting that this region is processing actions based upon their action content, regardless of format of presentation. These patterns are consistent with hypothesis that action ideas evoke neural activity in and around structures sensitive to visual motion. Pictures of actions, semantic judgments of action pictures, action verbs, and sentences describing action events activate human MT+ and adjacent areas (Kable et al., 2005; Kourtzi & Kanwisher, 2000; McCullough, Saygin, Korpics, & Emmorey, 2012; Papeo & Lingnau, 2015). We previously argued that human MT+ serves as a perceptual point of entry for action semantics with levels of abstraction being instantiated more anteriorly from visual motion areas (Kable et al., 2005). The results of the current study indicate that neural activity within and close to visual motion areas is not unstructured sensitivity to motion memories, but represents patterns of motion for specific actions distinguishable from other actions.

We also found that regions within the cerebellum showed format-independent processing of action stimuli, including the declive, tuber, nodule, and cerebellar tonsil. The cerebellum is not typically considered to be involved in higher-level cognitive or conceptual tasks (although see Schmahmann, 1991, 2010). In the current study we see that it has similar neural activity during visually dissimilar, but conceptually similar, action-related stimuli. While not an area we predicted and not considered part of the action simulation system, it is possible that the cerebellum evokes motor plans associated with the action, in a manner similar to simulation, as has been found previously in several studies (Calmels, Pichon, & Grezes, 2014; Calvo-Merino, Grèzes, Glaser, Passingham, & Haggard, 2006; Molenberghs, Cunnington, & Mattingley, 2012).

We note one possible limitation of the findings in these sensorimotor regions, particularly those in the primary motor cortex. During the processing of all stimuli, regardless of whether a probe question followed or not, participants may have been engaging in motor preparation in anticipation of giving a response. For instance, when seeing a boxing pictogram, the participant may have primed her response for “never” (e.g., the right middle finger), and when she saw the word boxing she may have primed the same response again, in preparation for answering “never” if a probe were presented. This preparatory response using the same finger in response to two different instances of an action may have resulted in primary motor cortex activity (Cunnington et al., 2002; Lotze et al., 1999). The counterbalancing of response mappings and hands likely reduces the effect of such anticipatory motor preparation, but it is a possibility that should be acknowledged.

4.3 Action association

The second action processing framework we aimed to investigate was a conceptual semantics framework, in which cognitive, semantic representations of action are processed without the use of vicarious simulation. We predicted that the Action Similarity analysis would reveal activation in the inferior posterior parietal cortex (Binder et al., 2009), the MTG (Bedny, Caramazza, Grossman, Pascual-Leone, & Saxe, 2008; Wu, Morganti, & Chatterjee, 2008), the IFG (Thompson-Schill et al., 1998) and anterior temporal lobes (Abel et al., 2015).

Our multivariate Action Similarity analysis shows high correlations in several of these regions for format-independent action concepts. The left SMG, and to a lesser extent the right SMG, responded similarly to pictograms and words referring to the same action. This finding is in accord with prior work suggesting that the left SMG is uniquely involved in processing action semantics (Binder et al., 2009). We found support for the IFG responding to format-independent action concepts as well. The IFG, particularly the pars orbitalis (BA 47), is often considered to be part of a domain-general semantic system (Binder et al., 2009). In our study, this region is likely responding similarly to different exemplars of the same action due to naming, or other verbally-encoded semantic information. Damage to the IFG has previously been linked to deficits in action understanding, suggesting that this region may play an imperative role in action processing (Urgesi et al., 2014). Similarly, IFG shows similar neural activity during observation of action and reading of action words (Rueschemeyer, Ekman, van Ackeren, & Kilner, 2014). The IFG is also known to be involved in domain-general processes such as selection and inhibition of incorrect information (Moss et al., 2005; Swick, Ashley, & Turken, 2008). Thus, it is unknown whether the involvement of the IFG in format-independent action processing, as shown in our study, is due to domain-general or domain-specific functions.

We also found format-independent activations in the anterior ventrolateral temporal regions. This region is often considered part of a semantic hub, but those claims have been based largely on object semantics (Jackson, Lambon Ralph, & Pobric, 2015; Visser, Jefferies, & Lambon Ralph, 2010). Our data suggest that the anterior temporal pole may serve as a semantic hub for actions as well.

4.4 Format-specific action processing

Both action words and schematic pictograms are relatively abstract stimuli. They are symbolic and represent types of actions rather than specific instances of action. While not the major thrust of this study, we looked for neural activations that were more similar within formats, compared to between formats. These analyses revealed which regions were selective for pictograms and for words.

The Pictogram Similarity analysis, as predicted, resulted in activation clusters in visual cortex—specifically, the bilateral extrastriate cortex (Brodmann Area 18/19/37), and the left occipital pole (Brodmann Area 17). This analysis was intended to serve as a data-quality check, ensuring that the predicted (visual) regions showed the pattern of having high correlations between pairs of pictograms, compared to the correlations between word-pictogram pairs. We confirmed this prediction. This observation is in accord with the established role of early visual cortex in visual recognition (Grill-Spector & Malach, 2004) and the role of extrastriate cortex (BA 18/19/37) in object perception (Pennick & Kana, 2012; Schintu et al., 2014).

The Word Similarity analysis resulted in an activation cluster in the right middle occipital gyrus (BA 19) and the lingual gyrus. We predicted that this analysis would result in highest correlations in the left inferior temporal cortex (e.g., the visual word-form area), however this is not what we found. While there was one small cluster in the left occipito-temporal region (k = 17) that showed high correlations for word pairs compared to word-pictogram pairs, this cluster was too small to be considered statistically significant by our criteria. One explanation for having found this pattern in the right hemisphere is that the right middle occipital gyrus and the lingual gyrus may contribute to processing a string of graphemes as a visual object rather than a string of graphemes that relate to lexical semantics. This possibility is supported by evidence that right middle occipital gyrus is implicated in object processing (Schintu et al., 2014), and the right lingual gyrus is involved in recognition of letter forms and words (Muayqil et al., 2015; Perani et al., 1999). Thus the similarity in neural activity in this region may be driven by the presence of graphemes on the screen, rather than any deeper level of processing.

In our univariate analyses, the pMTG showed a high degree of activity in response to action pictograms, compared to action words. This may be because the pMTG is especially sensitive to schematic representations of action (Watson et al., 2014). One recent study (Devereux et al., 2013) showed that the functional role of left pMTG differs between words and objects—here we confirm that there is a difference in pMTG activity between words and images.

4.5 Experience and action processing

The action stimuli included in our study were relatively uncommon. Participants had little-to-no personal experience carrying out most of the actions (see Figure 2). There is accumulating evidence that motor experience with actions changes the way in which action stimuli are processed, particularly in the fronto-parietal action processing systems (Calvo-Merino et al., 2005; Cross et al., 2012; Kirsch & Cross, 2015; Quandt & Marshall, 2014). Given that the current study included a narrow range of actions, which were relatively uncommon, we have a limited ability to comment on the role of prior motor experience in the current study. Interestingly, despite the participants’ paucity of experience with these actions, we saw format-independent processing in primary sensory and motor cortices. This suggests that even with little personal experience with an action, participants interrogate their motor systems for how he or she might carry out an action. This simulation may be based on prior experiences with similar actions, or experience viewing others performing the action (Kirsch & Cross, 2015; Quandt, Marshall, Bouquet, Young, & Shipley, 2011). An important next step in exploring this question would involve assessing conceptual-level neural activity for actions that systematically varied with regard to participants’ own experiences. We predict that conceptual processing for more familiar actions might rely more on premotor and inferior parietal regions, in addition to the regions found in the current study.

4.6 Conclusion

Little is known about how format-independent action concepts are processed in the human brain. We sought to examine whether neural systems implicated in simulation accounts and associative accounts would be engaged in a format-independent manner in response to action words and schematic action pictograms. Using functional neuroimaging, we found that a bilateral network including inferior frontal regions, primary sensorimotor cortices, and anterior and posterior temporal cortex showed similar activity in response to different exemplars of the same action. This new finding complements the existing literature of conceptual object knowledge networks in the brain (Devereux et al., 2013; Lin et al., 2015), as well as other recent work discussing the contributions of sensorimotor and non-sensorimotor aspects of representations (Leshinskaya & Caramazza, 2016).

We thus conclude that both fronto-parietal action implementation networks and posterolateral temporal action association regions contribute to format-independent action processing. Future research should continue to investigate the nature of conceptual action processing, including its similarities to and differences from conceptual object processing, and the role that self-experience plays in action knowledge.

Highlights.

• fMRI collected as participants viewed action words and action pictograms

• Representational similarity analysis showed network of action abstraction

• Both semantic and sensorimotor regions contribute to action abstraction