Sensorimotor Experience and Verb-Category Mapping in Human Sensory, Motor and Parietal Neurons

Abstract

Semantic grounding is the process of relating meaning to symbols (e.g., words). It is the foundation for creating a representational symbolic system such as language. Semantic grounding for verb meaning is hypothesized to be achieved through two mechanisms: sensorimotor mapping, i.e., directly encoding the sensorimotor experiences the verb describes, and verb-category mapping, i.e., encoding the abstract category a verb belongs to. These two mechanisms were investigated by examining neuronal-level spike (i.e. neuronal action potential) activities from the motor, somatosensory and parietal areas in two human participants. Motor and a portion of somatosensory neurons were found to be involved in primarily sensorimotor mapping, while parietal and some somatosensory neurons were found to be involved in both sensorimotor and verb-category mapping. The time course of the spike activities and the selective tuning pattern of these neurons indicate that they belong to a large neural network used for semantic processing. This study is the first step towards understanding how words are processed by neurons.

1. INTRODUCTION

Upon hearing the word grasp, we quickly understand the action described. In everyday language, words are commonly used to point to real-world experiences (Meteyard, Cuadrado, Bahrami, & Vigliocco, 2012). The process of establishing “meaning-to-symbol” pointers in the brain is called semantic grounding. The semantic grounding mechanisms for concrete action verbs were investigated in this study at the neuronal level. Specifically, we asked how human neurons map action concepts experienced in the real world to language, and vice versa.

We propose that two complementary grounding mechanisms are involved in the mapping between verbs and their meanings. The first of these is Sensorimotor Mapping: under this mechanism, verb concepts are grounded through direct reference to concrete sensorimotor experience (Hauk, Johnsrude, & Pulvermüller, 2004). This mechanism is what is behind embodied approaches to verb meaning. The second mechanism is Verb-Category Mapping: in this mechanism, verb meanings are grounded in their relationship to other verb meanings, based on the overlapping set of sentence contexts that related verbs may occur in (Mahon & Caramazza, 2008; Pulvermüller, 2013a:). This mechanism is connected to the taxonomy of verb classes that has been the basis of much linguistic and psycholinguistic work on verb meaning (e.g., Levin, 1993; Pinker, 1994).

The distinction between these two mechanisms can be illustrated by considering their relationship to the meaning of a single verb, kick. In relation to sensorimotor mapping, the concept for the verb kick is derived (in part) from the Body Part or Force associated with performing the action of kicking. Body Part or Force, therefore, are defined as Features of sensorimotor mapping, as each of these features describes one important sensorimotor aspect of the verb’s meaning. These features relevant to sensorimotor mappings are referred as sensorimotor features for the remainder of this paper. Verb-category mapping, on the other hand, requires a taxonomy of verbal categories, based on the sentence contexts in which a set of verbs may appear. These verb categories may be used to facilitate learning of new verbs and concepts (Kemmerer, 2006; Mahon & Caramazza, 2008; Meteyard et al., 2012; Pulvermüller, 2013a). Take the same verb kick. It belongs to the same category as slap and knock in a verb taxonomy, because it occurs in similar sentence contexts as kick (e.g., Levin, 1993). This grouping of similar verbs (kick with slap and knock) defines a category, one which is different from verb groupings based on sensorimotor features: kick, slap, and knock have different Body Part sensorimotor features but belong to the same abstract taxonomic verb category. We will refer to the taxonomic verb categories that verbs belong to as verb-category features for the remainder of this paper. They form an abstract, grammatically-active semantic representation that is no longer strictly tied to the specific sensorimotor experiences associated with verbs.

These two kinds of semantic mappings have been argued to tap into two different levels of verb meanings (Kemmerer & Gonzalez-Castillo, 2010), and to recruit different neural networks. The sensorimotor mapping is proposed to involve motor and somatosensory areas. For example, previous functional Magnetic Resonance Imaging (fMRI) studies found that a sensorimotor feature of verbs, Body Part, modulated the activations in the motor cortex by somatotopic organizations (Buccino et al., 2001; Hauk et al., 2004; Kemmerer, Castillo, Talavage, Patterson, & Wiley, 2008; van Ackeren, Schneider, Musch, & Rueschemeyer, 2014). For example, the hand area in the motor cortex activated selectively in response to hand verbs (e.g. pick, grasp) and the mouth area activated selectively in response to mouth verbs (Hauk et al., 2004). Magnetoencephalography (MEG) studies further show that these responses occurred earlier than the typical semantic processing window for words (Mollo, Pulvermüller, & Hauk, 2016). This supports the hypothesis that sensorimotor mapping is not merely from post-comprehension processes (see also Vukovic, Feurra, Shpektor, Myachykov & Shtyrov, 2017, for evidence that repetitive Transcranial Magnetic Stimulation (rTMS) to motor cortex impairs retrieval of actions verbs). Rather, these sensorimotor mappings may in fact be a fundamental neural mechanism of verb-semantic processing. Some aspects of sensorimotor mapping have also been found to be connected to associative cortices such as the inferior frontal (Barrós-Loscertales et al., 2012; Kemmerer & Gonzalez-Castillo, 2010; Moody & Gennari, 2010), temporal (Papeo et al., 2014; Romagno, Rota, Ricciardi, & Pietrini, 2012) and parietal areas (Buccino et al., 2001; Cattaneo, Maule, Tabarelli, Brochier, & Barchiesi, 2015; Fogassi et al., 2005).

Verb-category mapping, on the other hand, is hypothesized to primarily recruit associative cortices such as inferior frontal cortex (Bak & Chandran, 2012), temporal lobe (Damasio, Grabowski, Tranel, Hichwa, & Damasio, 1996; Kiefer & Pulvermüller, 2012), and inferior parietal cortex (Kemmerer, 2006; Noordzij, Neggers, Ramsey, & Postma, 2008). Damage to these regions in post-stroke aphasia is commonly associated with deficits in access to these abstract verb-category features (e.g., Kim & Thompson, 2000, 2004; Kemmerer, 2006).

These previous studies provide evidence of the existence of the two kinds of mapping at macroscopic level, in terms of broad cortical regions. However, due in part to the difficulty in examining activities of individual neurons in human participants, there is still debate regarding whether these two grounding mechanisms are distinct, and whether they are jointly active during online language processing (Arévalo, Baldo, & Dronkers, 2012). This difficulty is magnified by the fact that much data to date regarding sensorimotor and verb-category mapping comes from either fMRI evidence or lesion-deficit studies, neither of which directly addresses how these mechanisms affect real-time processing or are connected to neuronal-level activity. Similarly, it remains unclear whether the two grounding mechanisms overlap in their cortical distribution and temporal properties (Mahon & Caramazza, 2008; Meteyard et al., 2012). Some previous findings have suggested that they may overlap cortically, for example in temporal or parietal associative cortex. Understanding how these hypothesized grounding mechanisms are represented at the neuronal level, and whether they overlap in their temporal or cortical distributions, is a critical next step in understanding the neural bases of verb-semantic representation.

To elucidate these issues, four questions need to be answered. First, do the sensorimotor experiences described in verbs directly modulate spike activities (i.e. neuronal action potentials) of individual neurons in the primary motor and somatosensory areas, as actual body movement or sensation does (Georgopoulos, Schwartz, & Kettner, 1986; Wang, Chan, Heldman, & Moran, 2010)? Second, does verb-category information of the verbs modulate spike activities of individual neurons, specifically in associative areas like parietal cortex (Kemmerer, 2006; Noordzij, Neggers, Ramsey, & Postma, 2008)? Third, what is the time window of these neuronal responses? Does this time window precede or overlap with semantic processing time windows (Kutas & Hillyard, 1980b), and is it similar for the two different grounding mechanisms (sensorimotor and verb-category)? Fourth, are those neurons involved in semantic grounding (either sensorimotor or verb-category) specialists or generalists? That is, does a neuron encode a variety of semantic features, or specialize in just one or two? This question is related to a longstanding issue in neuroscience: are individual neurons broadly or narrowly tuned (Anderson, Bruni, Lopopolo, Poesio, & Baroni, 2015; Quiroga, Fried, & Koch, 2013).

To answer these questions, we recorded spike activities from implanted microelectrode arrays in two human participants, in response to visually-presented verbs. Spike activities were recorded from neurons in the primary motor cortex, the primary somatosensory cortex, and the superior and inferior parietal lobules near the intraparietal sulcus. The recorded spike activities from each channel of the implanted arrays were sorted into units based on spike waveform morphology similarity. A unit is presumed to be a collection of spikes from one or a small number of neighbor neurons. Four specific hypotheses were tested. Hypothesis I: the spike activities of somatosensory and motor units will be modulated by sensorimotor features (e.g. Body Part, Duration) of presented verbs (Kemmerer & Gonzalez-Castillo, 2010). This would be evidence of embodied verb meaning at the neuronal level. We anticipate that some of these modulated units will also be modulated by attempted motor movements, exhibiting “bimodal” properties. Hypothesis II: the spike activities of parietal units will be modulated by both sensorimotor features and verb-category features of presented verbs (Aziz-Zadeh & Damasio, 2008; Cattaneo et al., 2015; Jirak, Menz, Buccino, Borghi, & Binkofski, 2010). This would be evidence that two mapping mechanisms (sensorimotor and verb-category) overlap in their cortical distribution, recruiting different neuronal populations within the same macroscopic region. Hypothesis III: these spike modulations will occur prior to the semantic processing window, consistent with previous MEG findings (Mollo et al., 2016). This would be evidence that sensorimotor activations are not simply epiphenomenal but are critical to verb-semantic processing (see also Vukovic, et al. 2017). We do not have strong hypotheses regarding whether spike modulations in response to verb-category features will also occur in early time windows, or will overlap in time with sensorimotor spike modulations; these questions have not been examined to date. Hypothesis IV: units are more likely to be Specialists, modulated only by a very selective set of sensorimotor features or verb-category features, similar to the neuronal tuning properties to other kinds of complex information, such as persons or objects (Anderson, Bruni, Lopopolo, Poesio, & Baroni, 2015; Quiroga, Fried, & Koch, 2013) by highly specialized neurons (Quiroga et al., 2013).

2. METHODS

2.1. Human participants and intracortical microelectrode array placements

The implanted device used in this study was under an Investigational Device Exemption (IDE) granted by the US Food and Drug Administration and registered on clinicaltrials.gov (http://clinicaltrials.gov/ct2/show/NCT01364480andNCT01894802). The study was approved by the Institutional Review Board of the University of Pittsburgh (Pittsburgh, PA, USA), the Space and Naval Warfare Systems Center Pacific (San Diego, CA, USA) and the Department of Defense. Informed consents were obtained from the participants.

Participant A was a 53-year old female, diagnosed with a variant of spinocerebellar degeneration without cerebellar involvement 15 years prior to the experiment (Wodlinger et al., 2014). This participant had previously been implanted with two 100-channel intracortical microelectrode arrays (96 recording channels, Blackrock Microsystems, Salt Lake City, UT, USA) in the arm and hand areas of the left primary motor cortex for a neuroprosthetic control study (Collinger et al., 2013). Participant B was a 28-year-old male with a C5 level spinal cord injury. He was implanted with four arrays: two 96-channel arrays in the somatosensory area and two 32-channel arrays in the superior and inferior parietal lobules respectively, near the intraparietal sulcus (Blackrock Microsystems, Salt Lake City, UT, USA), also for a neuroprosthetic control study (Flesher et al., 2016). Both participants had corrected-to-normal vision and no known cognitive impairments. They both have normal verb comprehension, as indicated by 100% accuracy on the verb comprehension subtest of the Northwestern Assessment of Verbs and Sentences (NAVS) (Cho-Reyes & Thompson, 2012). Figure 1A shows the locations of the implanted microelectrode arrays.

Figure 1.

The microelectrode array locations and the experimental paradigm. 1A: Array locations of the two subjects, registered on their respective MRI scans. Left: Subject A, one array was implanted in the finger area of the primary motor cortex (precentral gyrus, labeled ‘F’) and another was in the hand area (labeled ‘H’). Both arrays have 96 recording channels. CS: central sulcus. Right: Subject B, two square arrays (96 recording channels) were implanted in the primary somatosensory area (postcentral gyrus, labeled ‘S’) while another two rectangle arrays (32 recording channels) were placed on superior and inferior parietal lobules, respectively (labeled ‘P’). IPS: intraparietal sulcus. 1B: Experimental Paradigm for the video-following task: videos with different hand, wrist, elbow and shoulder movements were shown to the subjects to map sensorimotor cortex responses on the same day as participants completed the verb-reading task. These videos were made following the same timeline: each video consisted of one second of movement and two seconds of holding. Each video consisted of five such repetitions. An example is shown here. The red dashed line indicates the kinematics of index finger movement. Both subjects were instructed to watch all of the videos and to ‘attempt’ the movements in the video as if they could perform them, at the same pace as the video. 1C: Experimental Paradigm for the verb-reading task: the subjects were instructed to read verbs silently. The verbs were presented for 1 second with an inter-stimulus interval of 1 second. Catch trials were included to keep the subjects attentive. In a catch trial, they were cued to make a sentence with the verb just shown.

2.2. Stimuli

Two kinds of stimuli were created for this experiment: hand/arm movement videos, and action verbs. The purpose was to see whether the units modulated by sensorimotor features described in verbs were also modulated by actual attempted movements.

Thirteen hand/arm videos were made, which included the actions Thumb flexion-extension, Index finger flexion-extension, Middle finger flexion-extension, Ring finger flexion-extension, Little finger flexion-extension, Grasp, Wrist Flexion-extension, Wrist Interior-Exterior rotation, Elbow Flexion-extension, Shoulder Abduction-Adduction, Shoulder flexion-extension, Shoulder shrug, and Shoulder Interior-Exterior rotation. Each video had five movement repetitions, and each repetition consisted of one second of moving phase and two seconds of holding phase. An example is shown in Figure 1B, with the red dashed line indicating the kinematic movements: the slopes indicate the moving phase (1 second) while the horizontal lines indicate the holding phase (2 seconds). The whole cycle lasted 6 seconds.

The verb stimuli consisted of 400 verbs satisfying three criteria: 1) they described concrete actions instead of cognition, attitude, or emotions; 2) they were used more frequently as verbs than as nouns, based on the CELEX database (Baayen, Piepenbrock, & Rijn, 1993); 3) they could be understood by people with a high school level education, as assessed by a high school student and online grade-level vocabulary classification (https://www.vocabularya-z.com/).

2.3. Video-following task

Participants were instructed to watch the hand/arm videos and to attempt the movements in the video with their dominant hands, at the same pace as the video. Because the participants had conditions that prevented them from actually moving their hands/arms, this involved visualizing these movements and mentally attempting them. The video-following task was presented on the same day of recording with the verb reading task.

2.4. Verb-reading task

For the verb-reading task, each verb was presented in white type face at the center of a black screen for one second, separated by one-second inter-stimulus intervals. Participants were not required to do any explicit task except to read the verbs silently and naturally.

The 400 verbs were randomized and separated into eight blocks. Each block also includes 4–8 catch trials. For each catch trial, a visual cue (a picture of a mouth) was presented one second after the verb. Upon the appearance of the visual cue, the participants were required to make a sentence using the previously presented verb. These catch trials made sure that the participants paid attention to the meaning of the verbs. This paradigm was illustrated in Figure 1C.

Each block lasted about 3–4 minutes with each verb presented once. The participants were allowed to take breaks between the blocks.

2.5. Sensorimotor and verb-category features

To study the two mapping mechanisms, two kinds of semantic features were coded for each verb: 7 sensorimotor features (like Body Part and Force) and 11 verb-category features (like [THROW] or [REMOVE]). The purpose of coding is to convert the two kinds of verb-meaning mappings (i.e. from verbs to sensorimotor experiences, and from individual verbs to a larger verb-category taxonomy) into numerical vectors, so that subsequent semantic modeling could be conducted on these numerical vectors. This is similar to semantic vectors generated in Latent Semantic Analysis (Duamis, 2004); the primary difference is that these coded feature vectors (for sensorimotor and verb-category features) are human interpretable.

Specifically, seven sensorimotor features (Body Part, Object, Duration, Boundedness, Force, Decomposability, and Complexity) were coded into categorical variables for each of the 400 verbs by two language experts (one licensed speech language pathologist and one advanced student in linguistics) and reconciled by a third linguist (the second-to-last author). The definitions and coding rules for these sensorimotor features are listed in Table 1. Examples of the coded values for the sensorimotor feature vectors are shown in Table 2.

Table 1.

Definition, citation and examples of the sensorimotor features

Feature Name	Definition	Coding rules
Body Part (Hauk, Johnsrude, & Pulvermüller, 2004)	Whether hand/arm/finger is used to carry out the action.	Grasp is coded as 1: the hand is used (non-hand/arm/finger verbs are coded as 0)
Object (Hopper & Thompson, 1980)	Whether an object is affected by the action.	Eat is coded as 1: whenever someone is eating, something is being eaten.
Duration (Verkuyl, 1972)	Whether the action goes on for some period of time.	Swim is coded as 1: it is durative.
Boundedness (Vendler, 1957;Verkuyl, 1972)	Whether the action described has an inherent stopping point.	Kick is coded as 1: the action is inherently bounded.
Force (Moody & Gennari, 2010)	How much physical effort is required to carry out the action described by the verb.	Buy is coded as None Touch is coded as Weak cough is coded as Moderate Beat is coded as Strong
Decomposability (Bennett & Partee, 1978)	Whether the action holds true for every moment that action is taking place.	Collect is coded as 1: every subpart of an action of collecting is also an action of collecting.
Complexity	Whether the action has multiple distinct sub-parts.	Bake is coded as 1: it has multiple sub-actions.

Table 2.

Each verb was assigned a value for seven sensorimotor features. Example verbs are shown below. Duration: whether the action is continuous or punctual, coded 1 for durative verbs and 0 for punctual verbs; Boundedness: whether the action has a definite end, coded 1 for boundary presence and 0 for absence; Object: whether the action involves manipulating or affecting an object, coded 1 for requiring objects and 0 for not requiring objects; Body Part: whether hand is involved, coded 1 for involving hand, and 0 for not; Complexity: whether the action involves complex movement sequences involving distinguishable sub-parts (complex) or not (simple), coded 1 for complex and 0 for simple; Decomposability: whether the action can be decomposed into sub-actions (such as an activity and change of state associated with completion of that activity), coded 1 for yes and 0 for no; and Force: how strong a force the action requires, coded 1 for verbs with weak force, coded 2 for verbs with intermediate force, coded 3 for strong verbs and coded 0 for verbs that do not involve physical force

Example verbs	grasp	breathe	knock
Sensorimotor Features
Duration	0	1	0
Boundedness	1	0	1
Object	1	0	1
Body Part	1	0	1
Complexity	1	0	1
Decomposability	1	1	1
Force	3	1	2

Because many aspects of sensorimotor experiences in concrete actions are linked (e.g. punctual actions usually have an inherent boundary, durative actions usually often involve less physical force than punctual actions), we examined the correlations between these sensorimotor features. The correlation coefficients between each pair of the sensorimotor features and the p-value of the correlations are shown in Figure 2A.

Figure 2.

A: Correlations between sensorimotor features. * indicated significant correlations. B: The 11 verb categories used in this study and some example verbs of each category. The coding of these features was derived from Levin (1993) and was used as to examine the verb-category mapping mechanism.

Verb-category features were based on an independently-motivated verb taxonomy (Levin, 1993). This taxonomy was derived from the set of sentence contexts in which different verbs may occur. This taxonomic verb-semantic information corresponds to the abstract event-semantic representations that govern the range of grammatical operations and sentence frames that different verbs may participate in (Levin, 1993; Pinker, 1994). These verb-semantic properties are hypothesized to be represented in general semantic processing areas, such as superior temporal, inferior frontal and parietal areas (Kemmerer & Gonzalez-Castillo, 2010). Our 400 verbs covered enough instances to permit analysis (i.e. more than 15 verbs per category) in 11 categories. Verb-category names and examples for each of these 11 verb-category features are listed in Figure 2B. The set of verb categories each verb belonged to was converted into a 11-element binary vector: a verb was coded as 1 if that verb belonged to a particular category, and 0 if the verb does not belong to that category. For example, the verbs throw and kick were coded with a 1 for the verb category [THROW], while the verb walk would be coded with 0 for this verb category. These verb categories were correlated with the sensorimotor features to see if verb categories were formed based on any of the sensorimotor experiences. Indeed, these two sets of features were significantly correlated; these correlations and their influences on the neuron tuning pattern will be discussed in the Results section below.

2.6. Recording Setup

Spiking events crossing a threshold (−4.5 times the root-mean-square (RMS) value of the noise floor for Participant A and −5.25 RMS for Participant B) were recorded by a NeuroPort data acquisition system (Blackrock Micro systems) on two separate days, for both participants. The recoded spikes were synchronized with video or verb presentations through digital input.

Because the dimension of the electrode pitch on the microelectrode array (~400μm) is larger than the average size of a typical neuron (~20 μm), it is possible that spikes from a few neighboring neurons were recorded in the same electrode channel. To separate these spikes, the recorded spikes from each electrode channel were loaded in Central Play (Cerebus Central Suite, BlackRock MiscroSystems, the interface is shown in Figure 3A) offline. The spike events with their respective waveforms were displayed in the main panel of Central Play (as shown in the center panel of the Central Play interface, Figure 3A). If the displayed waveforms had distinctively separate peaks (in terms of their peak timing and amplitude), these peaks were manually marked with reference points. The spatial differences between these reference points were subsequently used to separate all the spike events in the recorded data, as indicated by the different marked colors in Figure 3A. These color-marked spikes were considered to come from different neurons, based on the assumption that different neurons have different spike waveform morphology. This separation procedure of grouping spike events coming from each channel into presumed “neurons” is defined as sorting. To summarize, spike events coming from the same channel can be sorted into different units based on their waveform morphology. This criterion is based on the distinctive timing and amplitude of the spikes’ peaks.

Figure 3.

A: Screenshot of the Central Play software, in which the manual sorting procedure was performed as the recorded spike files were played. As shown in the central panel, waveforms with very different peaks (in terms of timing and amplitude) were labeled by references and separated into different color groups. This procedure is defined as sorting, B: Raster plots of the sorted units, exhibiting synchrony with the video kinematics.

This sorting procedure was applied to all the electrode channels. The sorted spikes were defined as units. More specifically, a “unit” is a collection of spikes recorded in one electrode channel that exhibit very similar waveform morphology. This collection of similar spikes is therefore assumed to come from either one neuron or a small number of very similar neurons. Because the recorded units may change from day to day, due to small changes in the physical placement of the electrodes as well as fluctuation in the behavior of individual neurons, the two participants’ units from the two recording days were pooled together as independent sets. The total numbers of units that fired faster at rates faster than 1Hz were 193 in the primary motor cortex, 70 in the somatosensory cortex and 58 in the parietal area.

A portion of the recorded units were found to be modulated by the video-following task. This modulation is illustrated in the spike raster plots for some representative units shown in Figure 3B; these units responded to index and ring finger movements. The raster plots show the timing of each spike by marking its occurrence with a black tick. The eight subplots in Figure 3B illustrate eight different units. The y-axis of each subplot (unit) represents single trials, and the black ticks are the temporal occurrences of spikes during those trials. The red dashed lines follow the kinematics of the actions depicted in the videos (as shown in Figure 1B). As revealed by the raster plots in Figure 3B, the units not only responded selectively to the movements but were also time-locked in their spike rates to the different phases of the movements. For example, unit 9 for index finger movement (top row, right in Figure 3B) was time-locked to the holding phase, whereas unit 21 for ring finger (bottom row, right, Figure 3B) was time-locked to the moving phase.

2.7. Firing rates and unit-level encoding model

To compute the firing rate for each unit, the number of the spikes was first counted in small time bins (30ms), and the counts in each time bin were converted into firing rates by dividing the number of spikes by the duration of the time bin. These binned firing rates were subsequently low-pass filtered with a 450ms-wide exponential smoothing window, incorporating the unit’s firing rate history with exponentially decaying weights (Collinger et al., 2013). These low-pass filtered firing rates served as the dependent variables for all the subsequent analysis.

Unit-level encoding models were based on the regression of the low-pass filtered firing rates against semantic features. This analysis was done by sliding a 200ms window with a step size of 30ms across all the trials. At each step, the mean firing rates of a unit within the window across all the 400 verbs were regressed against the coded values of semantic features of the verbs in a simple linear regression model:

$$\mathrm{f}={\mathrm{b}}_0+\sum _{\mathrm{i}=1}^{\mathrm{d}}{\mathrm{b}}_{\mathrm{i}}{\mathrm{s}}_{\mathrm{i}}$$

“f” denotes the firing rate of a unit within the time window; “d” is the number of semantic features (d=7 for the sensorimotor feature encoding model, and d=11 for the verb-category encoding model); s_i represents the value of the ith semantic feature (either sensorimotor or verb-category feature). b₀ and b_i are regression coefficients.

This model is motivated by previous models of motor cortical neuronal activity (Georgopoulos, Schwartz, & Kettner, 1986; Wang, Chan, Heldman, & Moran, 2007, 2010) and models of semantic feature encoding from hemodynamic signals (Huth, Nishimoto, Vu, & Gallant, 2012; Mitchell et al., 2008). If the variance in the spike rates of one unit was reliably predicted by one or more semantic features (sensorimotor or verb-category) within a given time window, as determined by the F-statistics of the regression model (adjusted for multiple comparisons with the false-discovery rate method; Benjamini & Hochberg, 1995), then this unit was identified as a tuned unit. Furthermore, the time window in which the tuned unit’s spike rate was reliably predicted by a semantic feature was defined as that unit’s tuning window.

2.8. Tuning pattern analysis

To tease apart the contribution of each feature (sensorimotor or verb-category) in explaining the variance of spike activities in the regression model, one-way Analyses of Variance (ANOVAs) and Analyses of Covariance (ANCOVAs) were also conducted. The ANOVAs addressed the role of sensorimotor features in modulating spike rates of tuned units, whereas the ANCOVAs addressed the role of verb-category features in modulating spike rates of tuned units. In each of the ANOVAs, the independent variables were the individual sensorimotor features, and the dependent variables were the corresponding firing rates of each tuned unit.

Because all the verb-category features were correlated with three sensorimotor features (Object, Body Part and Force), the tuning of units for the verb-category features was analyzed with ANCOVA, with these three sensorimotor features as covariates. In each of the ANCOVAs, the independent variables were the individual verb-category feature, the covariate variables were the three sensorimotor features (Object, Body Part and Force) that were significantly correlated with the verb-category features, and the dependent variables were the corresponding firing rates of each tuned unit. Then the numbers of units that were significant for each sensorimotor feature/verb-category were counted for tuning pattern analysis. If an ANOVA or ANCOVA was significant with respect to a feature (corrected for multiple comparison), then the tested unit was counted as a tuned unit for the relevant feature.

3. RESULTS

We present three sets of results: 1) neuronal activity changes (i.e. firing rate modulation) with respect to both different semantic features (sensorimotor and verb-category) and attempted movements in the verb-reading and video following tasks; 2) the spatiotemporal pattern of the neuronal tuning: when and where the firing rate variances were explained significantly by the coded sensorimotor features or verb-category features; and 3) tuning pattern analyses: which and how many semantic features each unit was tuned to.

3.1. Firing Rate Modulation

To examine whether neuronal firing rates changed in relation to semantic features in the verb-reading task and to movement cycles in the video-following task, unit firing rates were plotted against verb features and kinematics of the videos. Three prototypical units that responded to both the attempted movements in the video following task and a specific sensorimotor feature (Duration) in the motor (Unit A), somatosensory (Unit B), and parietal areas (Unit C) are shown in Figure 4A. The firing rates of these three units (indicated by the blue curves) were modulated by the attempted movements (indicated by the red dashed line plotting the kinematics of the movement cycle) when the participants were watching the videos of Index Finger Flexion (Unit A), Grasping (Unit B) and Middle Finger Flexion (Unit C), respectively. The firing rates of these units were time-locked to the movement cycles. Furthermore, the same units were modulated by the Duration feature in the verbs. Specifically, the red curve indicates the averaged firing rates over 245 durative verbs (e.g. breathe, swim), and the blue curve indicates the averaged firing rates over 155 punctual verbs (e.g. grasp, knock). These three units showed higher firing rates in response to durative verbs than to punctual verbs, thus responding to the sensorimotor feature of Duration. This suggests that these units responded to sensorimotor experiences in actual movements (duration in the videos) and in language symbols (duration concept in verbs) in similar ways.

Figure 4.

A: Firing rates of three prototypical units from motor, somatosensory and parietal areas that responded to the index-finger flexion, grasping and middle finger flexion in the video-following task (top) and to the Duration feature in the verb-reading task (bottom). For the top figure (video-following task), the blue curve represents the averaged firing rate of the unit over 5 repetitions. In each repetition, subjects watched the videos of index-finger flexion, hand grasping and middle-finger flexion respectively, and attempted in their minds to perform the actions at the same pace as in the videos. The red dashed line traces the kinematics of the action depicted in the video. It can be seen that these units locked their firing activities to the kinematics of the videos. For the bottom figure (verb-reading task), the grey dashed lines indicate stimulus onsets, and the two colors indicate verbs with different values of the sensorimotor feature Duration. Specifically, the red curve indicates the averaged firing rate of the unit over the 245 durative verbs (e.g. breathe, swim), and the blue curve indicates the averaged firing rate over the 155 punctual verbs (e.g. grasp, knock). It can be seen that these units showed higher firing rates in response to durative verbs than to punctual verbs. In other words, these units responded to the sensorimotor feature of Duration. B: Prototypical sensory and parietal units responding to verb-category features in the verb-reading task. Red line: average firing rates for verbs that belong the labeled category; Blue lines: average firing rates for verbs outside the labeled category. If a unit’s firing rates were modulated by the categories, the red curve should be significantly different from the blue following the stimulus onset; this is indicated by a grey shaded area.

Out of the 11 verb-category features, we found units whose firing rates were significantly modulated by seven categories ([REMOVE], [SEND/BRING], [MIX] [PUT], [GIVE/EQUIP], [THROW], and [CUT]; see Figure 6 for details). Some prototypical tuned somatosensory units (Unit D and E) and parietal units (Unit F and G) are shown in Figure 4B. The red curves show each unit’s firing rates in response to verbs belonging to the specified verb category, and the blue curves showed the same unit’s firing rates in response to other verbs.

Figure 6.

Tuning Patterns. 6A: Left Column: percentage of units tuned to each sensorimotor feature; lower row: percentage of units tuned to each verb-category feature. Upper row: motor units; middle row: somatosensory units; bottom row: parietal units. Sensorimotor features: DR-Duration, BD-Boundedness, OB-Object, BD- Body Part; CP-Complexity, DP-Decomposability, FC-Force. Right Column: percentage of tuned units to each verb-category feature. Verb-category features: [P]: [PUT] verbs; [R]: [REMOVE] verbs; [S/B]: [SEND/BRING] verbs; [G/E]: [GIVE/EQUIP] verbs; [T]: [THROW] verbs; [M]: [MIX] verbs; [C]: [CUT] verbs. Red dashed line indicated significance level. 6B: the number of units tuned to one, two or four semantic features. Most units were tuned to only one or two features.

3.2. Unit-level mapping

We regressed neuronal firing rates for each unit against the sensorimotor features or verb-category features for the 400 action verbs in a sliding-window fashion (constructing a regression model in every 200ms window and moving the window in 30ms steps). This regression model systematically assessed whether the variances of unit firing rates can be explained by either sensorimotor or verb-category features. If the unit-level encoding model was significant within a given time window (corrected for multiple comparisons using False Discovery Rate, Benjamini & Hochberg, 1995), this window was defined as a tuning window. Units with significant tuning windows were defined as tuned units. The spatiotemporal distributions of p-values in the unit encoding models for units that had at least one significant tuning window are shown in Figure 5. In all the subplots of Figure 5, the x axis represents time, with 0 being the stimulus onset (indicated by the white dashed line), and the y axis represents different units, sorted by the onset of their first tuning window. The color of the plots shows the p-values of the encoding model: red-orange colors indicate that the model is significant at that particular time point for that particular unit.

Figure 5.

Unit-level encoding model. Left Column: the p-values of the unit firing rates regressed against the sensorimotor features, indicating whether each individual neuron participated in sensorimotor mapping; Right Column: the p-values of the unit firing rates regressed against verb-category features, indicating whether each unit participated in verb-category mapping. Top row: motor units; middle row: somatosensory units; bottom row: parietal units. In all the subplots, the units are sorted by the temporal order of their first tuning windows. The white dashed line indicates the onset of the stimuli. The color scales are consistent across subplots: red and orange colors indicated that the unit-level encoding model was statistically significant during this time window.

We found significant numbers of units that were tuned to the sensorimotor features in all three regions: 74 units from the primary motor cortex (38% of the total recorded motor units), 28 units from the somatosensory cortex (40% of the total recorded somatosensory units), and 32 (units from the parietal area (55% of the total recorded parietal units) were tuned to one or more sensorimotor features. We also found significant number of units that were tuned to verb-category features in two regions: seven somatosensory units (10% of the total recorded somatosensory units) and twenty-five parietal units (43% of the total recorded parietal units). The number of motor units that showed tuning to verb categories was small (five); this number is sufficiently small that it may be attributable to chance, as measured by False Discovery Rate (Benjamini & Hochberg, 1995). From Figure 5, it can be seen that the tuning windows for these tuned units were widely distributed, extending both before and after the canonical semantic processing window of ~400 ms post stimulus onset (Kutas & Hillyard, 1980a).

We also investigated how many units were “bimodal,” responding to both the movement cycles in the video-following tasks and the sensorimotor features in the verb-reading tasks. A notable proportion of the units were found to be bimodal: 30 units (41%) from motor cortex, 10 units (36%) from somatosensory cortex and 7 units (22%) from parietal area.

3.3. Tuning Patten Analyses

To tease apart the contribution of each sensorimotor/verb-category feature, the firing rates of each unit were analyzed using one-way ANOVA with respect to each feature. Correlation analyses examining the relationships between the sensorimotor and verb-category features revealed that all the verb-category features were significantly correlated with three sensorimotor features (Object, Body Part and Force). Therefore, the tuning of units to the verb-category features was analyzed using ANCOVAs, with these three sensorimotor features as covariates. If an ANOVA or ANCOVA was significant with respect to a feature (corrected for multiple comparison), then the unit was counted as being tuned to that feature. The number of units tuned to each feature was counted. The results are shown in Figure 6A. We observed a decreasing number of units that were tuned to sensorimotor features and an increasing number of units tuned to verb-category features when moving from motor and somatosensory to parietal areas.

The percentages of units tuned to sensorimotor features are plotted in the left column of Figure 6A for motor, somatosensory and parietal areas, with the red dashed line indicating a significance level of p=0.05 (determined by random permutation tests). All seven sensorimotor features had significantly tuned units in the motor areas, all sensorimotor features except Force had significantly tuned units in the somatosensory areas, and only Duration and Force had significantly tuned units in the parietal areas. The percentages of units tuned to the different verb-category features are plotted in the right column of Figure 6A. Seven verb-category features (out of 11) had units that were significantly tuned to them. For the motor units, none of the verb-category features reached significance. For both the somatosensory and parietal units, seven verb categories had significantly tuned units: [PUT], [REMOVE], [SEND/BRING], [MIX], [GIVE/EQUIP], [THROW], and [CUT]. However, the parietal units had more units tuned to each of these verb-category features.

To see whether these tuned units were specialists or generalists, we counted how many features each unit was tuned to. Figure 6B shows the histogram for this distribution: around 80 units (the majority of all the tuned units) were tuned to only one or two features. This indicates that the majority of units tend to be specialists: they respond to only a very specific kind of information.

Due to the physical constraints on actions as performed in the external world, sensorimotor features of verbs are not independently distributed. For example, verbs describing punctual actions usually have a well-defined boundary or endpoint. As a result, a number of sensorimotor features are significantly correlated with other sensorimotor features (as shown in Figure 2A). These correlations may inflate the apparent number of generalist units: a unit tuned to a number of highly correlated semantic features may still be a specialist. To check whether this is the case, we examined the tuning preferences of units that were tuned to two or more sensorimotor features (generalist units). We found that the majority of generalist units were indeed tuned to highly correlated sensorimotor features. The largest group of generalists tuned to the feature combination of Duration and Boundedness. This co-occurrence may be attributed to the fact that the majority of punctual actions (i.e., actions with no duration) also have a well-defined boundary (i.e., have boundedness). Therefore, these units may actually tune to “punctual actions that have a physical boundary.” In this sense, these units may still be specialists rather than generalists: they are tuned to the interfaces between sensorimotor features that usually occur together in the physical world.

4. DISCUSSION

4.1. Spatial distribution of the two grounding mechanisms

We found that neurons in the motor, somatosensory, superior and inferior parietal lobules selectively responded to sensorimotor features or verb-category features when human participants silently read action verbs. These responses are the neuronal reflex of verb-related embodied cognition (Hauk, et al., 2004), and of higher-order verb taxonomies (Levin, 1993). Furthermore, these units’ tuning properties followed a trend: gradually decreasing representation of sensorimotor mapping and increasing representation of verb-category mapping when moving from primary motor to somatosensory to superior/inferior parietal lobules. Because the participants were instructed not to attempt or imagine actions described in the verbs, the tuning to sensorimotor features in motor and somatosensory neurons are unlikely to be due to post hoc responses from imagined movements. Rather, the tuning to these features occurred naturally and automatically. This is consistent with the hypothesis that action verbs prompt the same neuronal response as performing or sensing a movement. The existence of bimodal units responding to both visual depictions of actions and verbs describing actions (discussed further below) provides additional evidence consistent with this hypothesis. Also consistent with this is the fact that transcranial magnetic stimulation to motor areas interferes with action-verb processing (Vukovic, et al., 2017).

Also as expected, verb-category information was processed in neurons near the intraparietal sulcus. This is consistent with our hypothesis that parietal neurons will play a critical role in more abstract verb-semantic processing (Aziz-Zadeh & Damasio, 2008; Cattaneo et al., 2015; Jirak, Menz, Buccino, Borghi, & Binkofski, 2010). Some of the verb-category features (e.g. [THROW], [GIVE/EQUIP]) that the parietal units were significantly tuned to align well with one general cognitive bias of the parietal area: spatial trajectory processing or action planning (Culham & Kanwisher, 2001). For example, [THROW] verbs generally involve a spatial change of location.

The sensorimotor and verb-category features do not form a clear-cut dichotomy. Rather, the verb categories can be viewed as high-level abstractions over sensorimotor experiences. Consistent with this view, the verb-category features do in fact reliably correlate with three sensorimotor features (Object, Body Part and Force). Previous studies have indicated that parietal cortex is responsible for integrating sensorimotor information (Andersen, 1997; Huk & Shadlen, 2005), and distilling these experiences into higher-level taxonomic information (Fogassi, et al., 2005). Viewed from this perspective, our results suggest one way that categorical semantic representations may emerge from sensorimotor information: parietal cortex may mediate this capacity for abstraction. Unexpectedly, some somatosensory neurons were also tuned to verb-category features, though to a lesser degree than parietal neurons. There are two possible explanations for this finding. First, verbs in the same taxonomy category may share some similar sensory information (e.g., some but not all verbs in the [THROW] category share the same Body Part). These verb-category-tuned neurons may be responsive to this shared sensory similarity. Second, these somatosensory neurons may receive top-down input from superior/inferior parietal lobules as part of semantic processing circuits, as discussed in the literature on mirror neurons (Pineda, 2008). This top-down input from cortex that is tuned to more abstract features may be responsible for the observed modulation of these somatosensory units’ units firing rates. Further work is needed to tease apart these possibilities.

4.2. Distribution of tuning windows

The distribution of tuning windows – periods in which there were units that were significantly tuned to one or more semantic features – covered the entirety of the typical time-course of semantic processing, from very early to very late processes, as shown in the previous electrophysiology studies (Lau, Phillips, & Poeppel, 2008). This broad coverage indicates that different neurons may participate in different stages of semantic processing: some units’ tuning windows correspond to the earliest stage of semantic comprehension, in which all the related meanings of the current stimulus are automatically activated (100–250ms) (Greenwald, Draine, & Abrams, 1996). The tuning windows for other units corresponds to the stage of selecting the best-fitting meaning (250–550ms) (Lau, Phillips, & Poeppel, 2008). Finally, some late-responding neurons may contribute to post-comprehension processes, such as priming the sensorimotor system after a related concept is comprehended (Mahon & Caramazza, 2008; Pulvermüller, 2013b).

One caveat that needs to be raised is that we do not know the empirical time course of comprehension (i.e. exactly when comprehension occurred in these two participants). The above-mentioned comprehension windows are therefore based on findings from the previous electrophysiology literature. It is not impossible that comprehension time windows of these two participants may be slightly different from the comprehension windows proposed in other studies. If so, this may call for modification to the interpretation of the findings offered above. Further work using tasks that permit empirical verification of the time course of participants’ comprehension (for example, lexical-decision or semantic-judgment tasks) is required to settle this question definitively.

Interestingly, we did not observe differences in tuning windows for units involved in sensorimotor mapping and those involved in verb-category mapping. This suggests that there is no strict temporal ordering for these two levels of semantic abstraction, and that sensorimotor experiences and more abstract verb categories may be accessed and processed in parallel. The broad time coverage of tuned units also suggests that these areas may belong to a larger semantic network, constantly sending and receiving sensorimotor or verb-category information among different regions/processes (Papeo et al., 2014; Romagno et al., 2012).

4.3. Neuronal tuning properties

Interestingly, the neurons were very selective in the features they were responsive to: most responded to one or only a few features. This is analogous to sparse coding of complex visual scenes in the neurons of primary visual cortex (Vinje & Gallant, 2000). Furthermore, the units’ tuning to sensorimotor experiences may also reflect the physical constraints of accomplishing actions in the real world. For example, for neurons that are tuned to two or more sensorimotor features, those features are highly likely to be correlated in the physical world.

Though the neurons appeared to be selective to the sensorimotor features they respond to, they may not be selective to input modalities. A good portion of the tuned neurons were bimodal: they responded both to actual sensorimotor experience in the video-following task, and to the sensorimotor information encoded in verbs in the verb-reading task. Interestingly, such bimodal responses were not found in a previous fMRI study examining activation of motor cortex (Postle, McMahon, Ashton, Meredith, & de Zubicaray, 2008). However, the difference between the BOLD signals in this fMRI study and the firing rate signals in the current study may lie in the different spatial-resolution scales of these two methods. The observation of bimodal neurons at the microscopic level did not correspond to the observation of such bimodal responses at the macroscopic level because the BOLD signals in the fMRI studies usually requires that a cluster of neurons fire in synchrony. The bimodal neurons in this study did not form any spatial clusters (i.e., they typically were not neighbors to each other). The implications of the existence of these bimodal neurons for language learning and evolution need further study.

4.4. Summary, limitations and future directions

In summary, this study has three main findings. First, semantic features associated with verbs (both sensorimotor and verb-category features) directly modulated neuronal activities in three different areas (primary motor, somatosensory and superior/inferior parietal lobules) in two human participants. Because our task – natural reading – is a highly automatic process in adult readers (Rayner, 1998), it is reasonable to conclude that neuronal-level semantic grounding occurred naturally as well. Second, this study identified the type of semantic grounding associated with each area. Sensorimotor mapping was represented in a decreasing degree from motor/somatosensory to parietal areas, whereas verb-category mapping was represented in an increasing degree across these areas. This trend is consistent with a hierarchical semantic processing mechanism, moving from more specific to more abstract semantic representations (Kemmerer & Gonzalez-Castillo, 2010). Third, the study characterized the neuronal tuning patterns for the two types of semantic grounding. Neurons were very selective for the features they were tuned to, and the populations of tuned units had broad temporal coverage, encompassing both early and late semantic processes. Both properties (broad temporal coverage and selective responsiveness) suggest that these areas are active members of a larger semantic network that is responsible for action-concept processing.

Overall, the results of the current study are consistent with a weak version of the Embodiment Theory (Mahon & Caramazza, 2008; Meteyard et al., 2012). It appears that sensorimotor experiences may be processed by both sensorimotor regions and associative regions, but that not all neurons that are responsive to verbs process sensorimotor information.

The current study has limitations that need to be acknowledged. First, given the small sample size, the differences in neuronal modulation across regions found in the current study (such as the engagement of the somatosensory neurons to the verb-category features and the distribution of the tuning time windows) may be attributed in part to participant differences. Additional data from other participants are needed to test this possibility systematically. In addition, due to the inherently variable nature of microelectrode array recording (different neurons are recorded each day), caution should be taken regarding generalization from the current results. Second, the current study tested only real-word stimuli, unlike other studies of word processing (such as lexical-priming studies). Nonsense word control stimuli may be included in future studies, to further validate that the neuron tuning was indeed specific to real words. Third, as a first-step estimation of verb-category mapping, the verb taxonomy used in the current study (Levin, 1993) may not necessarily be the most suitable category structure to estimate neuronal representations. More realistic and neurally-based verb-category features still need to be explored. Fourth, it is possible that neurons which responded to the video-following task are tuned to conceptual movement schema rather than pure sensorimotor experiences. If so, this would provide an alternative explanation of what appear to be bimodal neurons. However, we consider this possibility is rather unlikely, since the raster plot of the spike events (Figure 3B) showed temporal locking to movement cycles, strongly suggestive of tuning to the actual sensorimotor experiences.

This study provides the first set of evidence regarding how individual neurons in the human brain represent meaning in language. The current findings thus have the potential to advance our understanding of the neurobiology of language learning and development. Specifically, they may shed light on the specific neuronal processes that translate externally-grounded, sensorimotor experience into internal representation and manipulation of abstract symbols (e.g. language) in humans. The current findings suggest that parietal neurons may play an important role in this translation, integrating verbal stimuli that are associated with different sensorimotor experiences into higher-order, abstract categories that are partially independent of those specific sensorimotor experiences. If this is correct, these findings provide initial evidence regarding the neuronal processes that may be responsible for the emergence of abstract verb categories during language development (e.g., Gleitman, 1990; Pinker, 1994). Furthermore, the current findings may also inform treatment of verb-related semantic processing deficits in aphasia (Engelter et al., 2006) or Alzheimer’s disease (Kim & Thompson, 2004), pointing to the potential value of gesture in revitalizing or augmenting impaired verb-semantic mapping mechanisms (Boo & Rose, 2011). For instance, verb-retrieval deficits in Alzheimer’s disease are associated with degradation of access to sensorimotor features that enable participants to distinguish among specific verbs within the same taxonomic categories (e.g., go vs. walk vs. stroll; Kim & Thompson, 2004). The current findings suggest that neurons in motor and somatosensory cortex are not only tuned to sensorimotor features of verb meaning, but are also responsive to visually-presented gestures. Gestures that reinforce relevant sensorimotor features may therefore activate the same neurons that are involved in sensorimotor aspects of verb meaning, facilitating retrieval of semantically-specific verbs that are distinguished by just such sensorimotor features.

5. Conclusion

Neuronal spike activities in the motor, somatosensory, superior and inferior parietal lobules respond to either sensorimotor or verb-category features of concrete action verbs. The pattern of spike-rate modulation revealed gradually decreasing sensitivity to sensorimotor features and increased tuning to verb-category features moving from motor to somatosensory to superior/inferior parietal lobules. Furthermore, the majority of neurons were found to be specialists: they were tuned to only one or a few number of features. In addition, the broad temporal coverage at the neuronal population level and the selective tuning properties of units are consistent with a large semantic network responsible for verb and action-concept processing, one that processes multidimensional semantic features of each verb in parallel.

This study is the first attempt to examine how neurons in the human brain represent meaning in language. Further work is needed to determine how the neuronal mechanisms identified in the current study may explain how abstract linguistic categories are derived from specific sensorimotor experience. In addition, further work should target how these findings may be leveraged to develop efficacious verb-retrieval treatments, focused of either sensorimotor or more abstract aspects of verb meaning.