Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2018 Dec 1.
Published in final edited form as: Brain Cogn. 2017 Sep 9;119:1–9. doi: 10.1016/j.bandc.2017.09.001

Not so secret agents: Event-related potentials to semantic roles in visual event comprehension

Neil Cohn 1,2, Martin Paczynski 3, Marta Kutas 1
PMCID: PMC5759974  NIHMSID: NIHMS905597  PMID: 28898720

Abstract

Research across domains has suggested that agents, the doers of actions, have a processing advantage over patients, the receivers of actions. We hypothesized that agents as “event builders” for discrete actions (e.g., throwing a ball, punching) build on cues embedded in their preparatory postures (e.g., reaching back an arm to throw or punch) that lead to (predictable) culminating actions, and that these cues afford frontloading of event structure processing. To test this hypothesis, we compared event-related brain potentials (ERPs) to averbal comic panels depicting preparatory agents (ex. reaching back an arm to punch) that cued specific actions with those to non-preparatory agents (ex. arm to the side) and patients that did not cue any specific actions. We also compared subsequent completed action panels (ex. agent punching patient) across conditions, where we expected an inverse pattern of ERPs indexing the differential costs of processing completed actions as a function of preparatory cues. Preparatory agents evoked a greater frontal positivity (600–900ms) relative to non-preparatory agents and patients, while subsequent completed actions panels following non-preparatory agents elicited a smaller frontal positivity (600–900ms). These results suggest that preparatory (vs. non-) postures may differentially impact the processing of agents and subsequent actions in real time.

Keywords: visual language, event cognition, visual narrative, agents, event anticipation

1. Introduction

Within the structure of transitive two-participant events (e.g., X punches Y or X grasps Y), agents, the doers of actions (punchers), typically hold an advantage over patients (punchees), the receivers of actions (Dowty, 1991; Gruber, 1965). Arguably, this leads to an “agent advantage” across all human communication and comprehension (Strickland, 2016). Agents typically precede patients in the canonical sentence structures of most (89%) human languages (Dryer, 2011; Greenberg, 1966; Kemmerer, 2012). This ordering also persists in the signs of deaf children who have not learned a sign language (Goldin-Meadow, 2003; Goldin-Meadow & Feldman, 1977) and the gestures of non-signing adults asked to communicate without speaking (Gershoff-Stowe & Goldin-Meadow, 2002), independent of their native spoken language (Goldin-Meadow, So, Ôzyûrek, & Mylander, 2008). In addition, agents are typically recognized faster than patients in pictures and films of events (Robertson & Suci, 1980; Segalowitz, 1982; Webb, Knott, & Macaskill, 2010), even when these agents are represented by geometric shapes (Verfaillie & Daems, 1996).

Based on a series of behavioral experiments with events depicted in comic strips, we have argued that agents provide more information about event structure than do patients, and thereby facilitate event processing (Cohn & Paczynski, 2013). For example, we observed longer self-paced viewing times to preparatory agents (like a figure reaching back an arm to punch) than to patients, regardless of their relative position within a sequence (i.e., in agent-patient vs. patient-agent orders), at panels prior to those wherein semantic roles would be assigned (i.e., at the completed punch). Moreover, completed actions following agent-patient orderings or agents alone are viewed for shorter durations than those following patient-agent orderings or just patients. This is consistent with the possibility that preparatory agents may afford frontloading of event processing and thereby facilitate processing of action events when they occur. Though examined in the context of visual narrative sequences, we have argued that this “event builder” role may motivate the preference for agents over patients across many domains.

These sorts of findings raise the question of what visual features comprehenders might use to interpret characters’ semantic roles (agent, patient) prior to their appearance in completed actions, when those roles become actualized. Not only does body posture cue action preprocessing, e.g., sports or dancing, particularly for viewers with greater expertise with those actions (Aglioti, Cesari, Romani, & Urgesi, 2008; Smith, 2016; Urgesi et al., 2010; Urgesi, Savonitto, Fabbro, & Aglioti, 2011), but even 5 month old infants seem able to distinguish action-based cues from static postures (Shirai & Imura, 2016). Moreover, postural cues allow comprehenders to discern agent and patient roles (Wilson, Papafragou, Bunger, & Trueswell, 2011), even in rapidly presented (37ms, 73ms) action photographs (Hafri, Papafragou, & Trueswell, 2012).

Given that postural cues can serve to distinguish semantic roles during an event, we hypothesize that similar cues can signal upcoming agents for preparatory actions (i.e., reaching back an arm in order to punch), which might then afford predictions about upcoming actions (Urgesi et al., 2010). Indeed, static figure postures implying specific upcoming movements activate motor brain areas (Kourtzi & Kanwisher, 2000; Senior et al., 2000) similarly active during viewing of those actual movements (Dupont, Orban, De Bruyn, Verbruggen, & Mortelmans, 1994; Zeki et al., 1991).

Such observations align with mounting evidence of neural prediction during event comprehension. For example, fMRI activation has been observed prior to event boundaries during event segmentation (Zacks, Braver, et al., 2001), and participants have been found to generate more accurate subjective predictions about subsequent events from within a segment than after a segment boundary (Zacks, Kurby, Eisenberg, & Haroutunian, 2011). Centrally-distributed ERP negativities have been observed in anticipation of incongruous dance motions, prior to their full manifestation, an effect that is larger for experienced dancers than novices (Amoruso et al., 2014). In addition, an increase in motor-evoked potentials has been observed when participants view people shoot basketballs, even before the ball leaves a shooter’s hands (Aglioti et al., 2008).

Despite indications that comprehenders preactive event information, little work has examined what motivates these expectancies. Postural kinematic cues do motivate anticipation of actions, particularly for expert observers (Aglioti et al., 2008; Smith, 2016; Urgesi et al., 2010; Urgesi et al., 2011). However, research on event predictions has not directly manipulated these potential cues. Extant work has focused on specific actions (basketball, dancing) rather than generalizing across different actions, and the frequent use of video stimuli has made it difficult to distinguish precisely which cues are critical given that events unfurl over an extended time period (although, see Webb et al., 2010). We hypothesize that the postural cues of expected agents may provide one source for frontloading of event processing.

Preparatory visual cues are recognized as such to the extent they are linked to a completed action, constituting an “event schema” entrenched in semantic memory (Lasher, 1981; Strickland & Keil, 2011). Jackendoff (2007) argues that an abstract schema generalizes across specific events; a completed “head” is preceded by a “preparation” and followed by a “coda” (see also Moens & Steedman, 1988). For example, shaking hands involves a preparation (extending a hand), a head (grasping and shaking another hand), and a coda (releasing and withdrawing). This process can be recursive, with whole structures serving as preparations or codas (ex. walking up to a person may be a preparation for shaking hands). Such hierarchies have been well established in psychological research (Zacks & Tversky, 2001; Zacks, Tversky, & Iyer, 2001). An event “script” (Schank & Abelson, 1977) reflects a concatenation of numerous event schemas for specific situations and scenarios (e.g., the event schemas comprising restaurant behavior). While psychological theories of event comprehension include such schemas, they generally leave both representations and their contributions to processing unspecified (e.g., Zacks, Speer, Swallow, Braver, & Reynolds, 2007).

Because of the tripartite preparation-head-coda structure of this schema, viewing parts of an event should frame inferences of other parts of events. For example, viewing both the preparation and coda provide enough information to infer an unseen completed head of an event (Strickland & Keil, 2011), as schematized in Figure 1a. Preparations also seem to afford some forward predictions of subsequent actions (Aglioti et al., 2008; Smith, 2016; Urgesi et al., 2010; Urgesi et al., 2011). Thus, as in Figure 1b, extension of a hand would be recognized as a preparation, thereby activating the generalized event schema for a subsequent head of handshaking. We here ask whether the specific preparatory cues offered by an agent-to-be motivate such predictions.

Figure 1.

Figure 1

Open in a new tab

Inference and prediction of event states from a tripartite event schema. Peanuts is © Peanuts Worldwide LLC.

We used event-related brain potentials (ERPs) to explore the contribution of semantic roles and/or visual preparatory cues to the processing of visual events. We drew upon visual narratives (comics) that depict events statically in their prototypical states, as in prior work (Cohn & Paczynski, 2013). These stimuli enabled us to isolate and manipulate the preparatory actions of characters across various different types of actions.

We followed up on prior behavioral work (Cohn & Paczynski, 2013) by recording ERPs to visual narrative sequences with completed action panels preceded by either a preparatory agent or a patient for the event. We expanded this design by manipulating the postural cues that signaled the preparatory actions taken by the agent, and adding a condition with “non-preparatory agents” in passive postures as well (see Figure 2). Because we hypothesized that these cues motivate the building of event structures, we expected their absence (vs. presence) to affect action-related (event) processing.

Figure 2.

Figure 2

Open in a new tab

Example sequences where the completed action is preceded by a preparatory Agent, a passive Patient, or a Non-Preparatory-Agent in a non-preparatory state. Peanuts is © Peanuts Worldwide LLC.

We hypothesized that, on the basis of these cues, preparatory agents would initiate the building of an event structure more than patients would. Event processing would thus be frontloaded to the processing of the preparatory action panel, which in turn would facilitate processing downstream at the completed action. Based on our behavioral results showing longer viewing times to agents than patients (Cohn & Paczynski, 2013), we expected ERP activity to preparatory agents to reflect greater processing effort. Without the cues indicating a preparatory action, the semantic role of this “agent-to-be” would not be recognized, and thus should render these non-preparatory-agents as indistinguishable from patients in their accessing event structures prior to the completed action. We thus expected similar ERP effects for patients and non-preparatory-agents compared to preparatory agents.

At the subsequent completed action panel, we expected to see a different pattern of results. If, as we hypothesize, preparatory agents afford event processing frontloading which in turn leads to downstream facilitation, processing of completed actions preceded by preparatory agents might differ from that following both non-preparatory agents and patients. One possibility is that ERPs to completed actions following preparatory agents would differ from those following non-preparatory agents and patients, which should not differ from each other. Alternatively, completed actions following non-preparatory-agents may seem unusual given the preceding (presumed preparatory) context, and thus invoke more effortful processing than following truly informative preparatory agents. This would be akin to greater ERP negativities to unanticipated versus anticipated events (Reid et al., 2009), only in this case to a congruent completion preceded by ultimately uninformative preparation. Finally, if completed actions following both types of agents differ from those following patients, it would suggest that preparatory cues do not facilitate the processing of event information. Such a result would imply no predictive processing, and only a backward-looking process, since semantic roles would be treated as equal given preceding postural cues. As our prior behavioral results showed shorter viewing times to actions following agents than patients (Cohn & Paczynski, 2013), ERP amplitudes patterning with patients could be assumed to require more effort.

While we did predict differential activity for stimuli as a function of the presence or absence of preparatory cues, how these would manifest in ERPs was less clear, given that the specific neural areas involved in action understanding remain a matter of debate (Kilner, 2011). Indeed, heterogeneous results have been observed to action predictions (see Smith, 2016 for review) and for ERPs related to event comprehension (Amoruso et al., 2013). Postural cues related to event anticipations have elicited both sustained central negativities (Amoruso et al., 2014) and fronto-central positivities (Bach et al., 2009; van Elk, Bousardt, Bekkering, & van Schie, 2012). In addition, late positivities have been argued to constitute a family of responses indexing the integration of information into a mental model (Brouwer, Fitz, & Hoeks, 2012; Donchin & Coles, 1988; Kuperberg, 2013), and have been observed to violations of event structure (Amoruso et al., 2013; Bach, Gunter, Knoblich, Prinz, & Friederici, 2009; Sitnikova, Holcomb, & Kuperberg, 2008). Accordingly, we considered the possibility that negative or positive ERP components could be elicited by our manipulations.

2. Material and Methods

2.1. Stimuli

Sixty sequences were created drawing from a corpus of panels culled from twelve volumes of the Complete Peanuts by Charles Schulz (1952–1974), expanding on the stimulus set used previously (Cohn & Paczynski, 2013). These experimental strips combined existing Peanuts strips with novel strips created by recombining existing panels into new combinations. Sequences varied in length from 5 to 7 panels long. Stimuli depicted several types of events: characters acting on inanimate objects (ex. throwing, catching, and/or kicking balls, slamming piano keys), characters interacting with other characters (ex. punching, shaking hands), and characters using inanimate objects as instruments to act upon another character (ex. popping a bag to frighten someone, hitting them with a newspaper).

Our three sequence types manipulated the content of the panel prior to the completed action (Figure 2), situated in either the second or third panel position of the sequence. Preparatory agents clearly depicted a posture initiating a subsequent action, while patients were the receiver of that subsequent action. Non-preparatory-agents either repeated the passive state of the agent-character from the previous panel or altered the preparatory agent panel to no longer show the relevant postural cues. For example, we may have lowered hands from reaching back to punch to hanging at a figure’s side, along with erasing motion lines which are indicative of event structures (Cohn & Maher, 2015). Our analysis focused on both the manipulated panel (critical panel 1) and the completed actions (critical panel 2). Sequences were counterbalanced using a Latin Square Design into three lists such that each list included 20 trials of each sequence type (preparatory agent, patient, non-preparatory-agent) without repeating strips, along with 120 filler sequences featuring varying degrees of coherence.

2.2. Participants

Thirty-six comic readers (13 male, 23 female, mean age: 20.3) from the UC San Diego campus and surrounding neighborhoods participated in the study. All participants were English-speaking right-handed individuals with normal or corrected vision and no history of head trauma. Each participant gave their informed written consent according the guidelines of the UCSD Human Research Protections Program. Self-defined “comic readers” were chosen to ensure that participants were “fluent” in this manner of assimilating graphic events. To assess this fluency, all participants completed the Visual Language Fluency Index (VLFI) questionnaire (see http://www.visuallanguagelab.com/resources.html) which produces a metric that has provided a strong predictor of both behavioral and neurophysiological effects in online comprehension of visual narratives (Cohn & Kutas, 2015; Cohn & Maher, 2015; Cohn, Paczynski, Jackendoff, Holcomb, & Kuperberg, 2012). An idealized average along this metric would be a score of 12, with low being below 7 and high above 20. Participants’ mean VLFI score was 17.43 (SD=7.1, range: 7.5–37.5), a high average.

2.3. Procedure

In a room separate from the experimenter and computers, participants sat in a comfortable chair facing a computer screen. White panels on the black screen creates a “flashing” effect that induces blinks, so the lights were kept on throughout the experiment. Participants first saw a screen reading READY, where they pressed a button to start each trial. After a subsequent fixation-cross, each panel of the sequence subsequently appeared one at a time on the center, until the trial ended with a question mark where participants assessed whether or not the sequence made sense. Each screen persisted for 1350ms separated by a 300ms ISI that ensured panels did not become animated. A short practice list acclimated participants to the procedure and stimuli.

2.4. Data Analysis

We recorded EEG from 26 tin electrodes evenly distributed across the scalp. Electrodes were referenced online to the left mastoid and then re-referenced to the average of the right and left mastoids in offline analyses. We monitored horizontal eye movements and blinks with electrodes placed beneath and next to each eye. Impedances were kept below 5 kΩ for all electrodes.

Analyzed ERPs were time-locked to the onset of critical panels and averaged across a 1500ms epoch relative to a 500ms pre-stimulus baseline. Our analysis focused on the ERPs recorded from the preparatory state (critical panel 1) and the subsequent panel depicting a completed action (critical panel 2) at epochs of 300–400ms, 400–600ms, 600–900ms, as in prior research (Cohn et al., 2012; West & Holcomb, 2002). As in Figure 3, Sequence Types (agent, patient, non-preparatory-agent) were analyzed across 16 electrode sites divided into factors of Hemisphere (left, right), Laterality (lateral, medial), and Anterior-Posterior Distribution (prefrontal, frontal, parietal, and occipital) as in our prior work (Cohn & Kutas, 2015, 2017). Our within-subjects ANOVA looked for main effects and interactions of Sequence Type, Hemisphere, AP Distribution, and Laterality with a Greenhouse-Geisser correction for multiple comparisons. We followed omnibus ANOVAs with analyses targeted within each of the four quadrants (right/left - anterior/posterior) for pairwise relations between Sequence Types.

Figure 3.

Figure 3

Open in a new tab

Electrode montage, illustrating 16 electrode sites analyzed across Hemisphere, Laterality, and Anterior-Posterior Distribution, as well as Quadrants used in follow up analyses.

3. Results

3.1. Critical panel 1

We first compared critical panels depicting preparatory agents, non-preparatory agents, and patients. Omnibus analyses in all epochs revealed interactions between Sequence Type, Hemisphere, Laterality, and/or Anterior-Posterior Distribution (all Fs > 2.9, all ps < .05). Follow up analyses in each epoch compared pairwise relations between Sequence Types in each quadrant to further assess the distribution of effects (see Table 1 for statistics).

Table 1.

Results of ANOVAs in each quadrant comparing each sequence type at critical panel 1 and 2.

300–400ms 400–600ms 600–900ms
Anterior Posterior Anterior Posterior Anterior Posterior
Left Right Left Right Left Right Left Right Left Right Left Right
Critical panel 1
Preparatory Agent – Patient ST 2.8 1.1 0.93 0.32 0.05 5.2* 0.58 0.91 0.74 8.8** 0.36 0.31
ST* AP 6.1* 0.55 0.02 0.08 1.1 2.5 1.2 0.01 1.7 0.4 2.8 0.14
ST* L 6.2* 6.1* 1.6 1.9 8.8** 3.6^ 1.8 0.43 13.5** 3.0^ 1.2 0.08
ST* AP* L 9.3** 2.6 0.23 3.5^ 4.3* 10.5** 0.05 0.03 1.1 7.1* 2.6 0.3
Preparatory Agent – Non-preparatory Agent ST 1.8 0.95 0 0.02 0.08 1.6 0.41 0.62 2.5 5.0* 5.0* 3.8^
ST* AP 2.9^ 0.19 0.24 0 0.06 1.6 0.64 93 0.37 0.8 4.7* 4.9*
ST* L 2.4 0.01 1.7 0.19 5.0* 0.28 0.49 0.03 9.4** 3.9^ 0.18 3.5^
ST* AP* L 2.6 0.003 0 0.79 0.1 3.5^ 2.2 0.58 0 2.7 10.7** 4.8*
Patient – Non-preparatory Agent ST 0.05 3.3^ 0.88 0.39 0.35 1.1 2.5 3.2^ 1.4 0.26 1.9 1.6
ST* AP 0.02 0.77 0.4 0.1 0.65 0.18 0.2 1.2 1.1 0.04 0.8 2.8
ST* L 0.34 6.8* 0.03 0.5 0.09 9.7** 0.31 0.54 0.43 15.7*** 0.26 4.4*
ST* AP* L 0.29 1.2 0.21 0.94 1.4 0.42 1.8 1.2 1.5 0.3 2.5 2.6
Critical panel 2
Preparatory Agent – Patient ST 3.36^ 0.5 1.2 0.1 4.2* 0.34 0.01 1.8 0.8 0.45 0.25 0.45
ST* AP 0.8 0 0 0.54 2.4 1.1 0.05 0.24 1.2 0.63 0.97 1.3
ST* L 2.8 4.0^ 0.01 0.81 7.0* 0.48 0.04 0.22 6.3* 0.24 0.04 0.34
ST* AP* L 5.5* 2.2 0.02 2.4 3.4^ 2.41 4.12* 0.11 3.0^ 1.8 2.2 0.44
Preparatory Agent – Non-preparatory Agent ST 0.6 0.004 1.4 2.7 2.2 0.45 0.26 2.9^ 2.7 3.2^ 0.13 0.44
ST* AP 3.7^ 0.41 0.93 1.5 3.0^ 1.9 0.08 0.62 4.2* 1.2 0.16 2.2
ST* L 1.9 0.38 0.17 1.7 5.9* 0.03 0.003 3.5^ 0.77 0.15 0.67 4.3*
ST* AP* L 7.9** 0.19 0.45 0.01 6.8* 2.5 3.9^ 0.08 7.9** 4.8* 0.21 0.21
Patient – Non-preparatory Agent ST 0.96 0.53 4.6* 1.5 0.002 1.9 0.13 0 1.1 4.0^ 0.05 0.07
ST* AP 1.09 0.27 0.43 3.8^ 0.34 0.19 0.25 1.6 1.6 0.19 0.32 5.1*
ST* L 0.45 4.18* 0.22 0.01 1.01 0.21 0.02 2.2 4.2* 0.01 0.7 1.9
ST* AP* L 0.02 1 0.29 3.05^ 0.04 0.05 0.29 0.01 0.47 0.03 2.8 2
Open in a new tab

ST=Sequence Type, AP=Anterior–Posterior Distribution, L=Laterality. F-values are given. All df=1,35.

^

p<.1,

*

p<.05,

**

p<.01,

***

p<.001

A negativity in the left anterior region to preparatory agents compared to patients was suggested by significant interactions in the 300–400ms and 400–600ms epochs between Sequence Type, AP Distribution, and/or Laterality for preparatory agents and patients. In addition, an interaction appeared between Sequence Type and Laterality between preparatory and non-preparatory agents in the 400–600ms epoch, but no difference between non-preparatory agents and patients. A focal negativity greater to patients than non-preparatory agents, which in turn was greater than to preparatory agents in the right prefrontal electrodes sustained from the 300–400ms epoch through the 600–900ms epoch, as suggested in all epochs by significant interactions between Sequence Type and Laterality and/or AP Distribution.

Between 500 and 900ms, we observed a slightly leftward fronto-central positivity that was greater to preparatory agents versus non-preparatory agents and patients (Figure 4). This manifested in contrasts between preparatory agents and both non-preparatory agents and patients in interactions between Sequence Type and Laterality in left anterior regions, and in main effects of Sequence Type in the right anterior regions. Main effects also appeared in the left posterior region between preparatory agents and non-preparatory agents, along with interactions between Sequence Type and AP Distribution, and Sequence Type, AP Distribution, and Laterality in both posterior regions. A slight central positivity for patients compared to non-preparatory agents was found in the rightward regions in Sequence Type by Laterality interactions.

Figure 4.

Figure 4

Open in a new tab

ERPs time-locked to critical panel 1 of panels with Preparatory Agents, Non-Preparatory Agents, and Patients.

3.2. Critical panel 2

We next examined the ERPs at a subsequent critical panel depicting a completed action (Figure 5). Omnibus analyses found interactions between Sequence Type and Laterality in all epochs (all ps > 5.2, all ps <.01), Sequence Type and Hemisphere in the 300–400ms, F(2,70)=4.4, p<.05, and 400–600ms epochs, F(2,70)=5.1, p<.01, and Sequence Type, Hemisphere, AP Distribution, and Laterality in the 400–600ms, F(6,210)=2.4, p<.05, and 600–900ms epochs, F(6,210)=2.3, p=.054.

Figure 5.

Figure 5

Open in a new tab

ERPs time-locked to critical panel 2 following panels with Preparatory Agents, Non-Preparatory Agents, and Patients.

Follow up analyses (Table 1) showed a larger positivity appeared to preparatory agents than patients and non-preparatory agents in the left anterior region across epochs. In the 300–400ms epoch, this was suggested by an interaction between Sequence Type, AP Distribution, and Laterality in the left anterior region between preparatory agents and both non-preparatory agents and patients, and by a near significant interaction between Sequence Type and Laterality (p=.053) in the right anterior region between preparatory agents and patients. This extended in the 400–600ms epoch between preparatory agents and both non-preparatory agents and patients in the left anterior and posterior regions with interactions between Sequence Type and Laterality and/or AP Distribution.

In the 300–400ms epoch, patients were also more positive than non-preparatory agents posteriorly, as suggested by an interaction between Sequence Type and Laterality in the right anterior region, a main effect of Sequence Type in the left posterior region, and trending interactions between Sequence Type and AP Distribution (p=.06), and with Laterality, (p=.09) in the right posterior region.

Finally, in the 600–900ms epoch, preparatory agents evinced a larger positivity than patients, which in turn were less positive than non-preparatory agents in a centro-anterior distribution, as depicted in Figure 4. This was suggested by interactions between Sequence Type and Laterality at the left anterior region between all three sequence types, and between non-preparatory agents and both preparatory agents and patients in the right posterior regions. The positivity to patients also extended to a more widespread rightward distribution relative to both types of agents, as suggested by trending main effects of Sequence Type in the right anterior regions (all ps < .072) and interactions between Sequence Type by AP Distribution in the right posterior regions (all ps < .054).

4. Discussion

We hypothesized that postural cues for preparatory actions taken by agents-to-be allow for a potential frontloading of event structure processing with the consequence of facilitating comprehension of the completed action/event panel when it occurs. We thus examined whether or not, and if so to what extent, the processing of completed action panels was influenced by preparatory cues in agents-to-be as compared to non-preparatory agents-to-be and semantic patients, which did not carry such preparatory cues. We observed greater anterior positivity (600–900ms) to preparatory agents as compared to non-preparatory agents and patients (critical panel 1); this positivity indicates differential processing of preparatory agents consistent with the hypothesized frontloading of event structure processing. At the subsequent image (critical panel 2), a larger positivity appeared to preparatory agents compared to non-preparatory agents or patients, followed by a contrast between all three types in a later epoch (600–900ms). We take this to suggest that the absence of preparatory cues has an effect on event structure processing, presumably deleterious, whenever the agent does not readily fit with the completed action given the context.

We posited that agents serve as “event builders” by providing greater access to event structures—e.g., as their postures cue preparatory actions—than do patients (Aglioti et al., 2008; Shirai & Imura, 2016; Smith, 2016; Urgesi et al., 2010; Urgesi et al., 2011). At the critical panel, preparatory agents elicited a larger frontal positivity than either non-preparatory agents or patients, which only differed in a focal centro-right posterior region. This was consistent with frontal positivities in comparable epochs observed to postures matching or mismatching anticipated actions (Bach et al., 2009; van Elk, Bousardt, Bekkering, & van Schie, 2012). While the scalp maximum of an ERP effect does not necessarily align with the location of the responsible neural generators, taken at face value the centro-frontal distribution of the positivity is consistent with the suggested link between event comprehension and frontal brain areas (Sitnikova, Rosen, Lord, & West, 2014; Wood & Grafman, 2003; Zalla, Pradat-Diehl, & Sirigu, 2003).

We note that preparatory and non-preparatory agents both depict the same figures, and thus are more physically similar than they are to patients. Yet, the ERP amplitudes to patients and non-preparatory agents differed only minimally; both depict passive actions which provide less event information than preparatory agents, as in Cohn and Paczynski (2013). Since in that work we interpreted longer viewing times to agents than patients as a sign of more effortful processing, we take a similar view here. Accordingly, we tentatively take this late frontal positivity for preparatory agents to index the integration of preparatory cues into the event structure, relative to comparatively passive actors (i.e., non-preparatory agents and patients).

At the subsequent critical panel, completed actions following preparatory agents were more positive between 300 and 900ms than those following non-preparatory agents or patients. In a later time window (600–900ms) there was a greater positivity for preparatory actions than patients, which in turn were more positive than for non-preparatory agents. This later difference parallels the reported pattern in viewing times: completed actions following patients or patient-agent sequences were viewed longer than those following preparatory agents or agent-patient sequences (Cohn & Paczynski, 2013). However, if this positivity indexes updating a mental model (Brouwer et al., 2012; Donchin & Coles, 1988; Kuperberg, 2013) as argued above, it would imply the opposite of our expectations: that completed actions require more effort to integrate into an event model following preparatory agents than those absent event cues. It would also reverse the inference where longer viewing times to patients were interpreted as greater processing effort, not (preparatory) agents. This is further complicated by the ERPs to completed actions following patients, which were more positive posteriorly in an early (300–400ms) epoch than non-preparatory agents, and in a later epoch (600–900ms) had a more widespread posterior and sustained anterior positivity compared to both types of agents. If completed actions following patients undergo the same processing as agents, and if positivity implies effort, this suggests that incorporating patients into event structure may require more sustained effort than agents of either type, but that preparatory agents required more effort than non-preparatory agents.

It is also possible that despite some surface similarities, the neural responses at the critical panels are distinct. Indeed, upon close scrutiny, the ERPs do appear more sustained across epochs at critical panel 2 than the positivity at critical panel 1. This attenuated positivity (or relative centro-frontal negativity) to non-preparatory actions compared to preparatory agents may index demands of integrating a completed event with a semantically impoverished or ambiguous prior action. Completed actions are somewhat less expected when preceded by (non-preparatory) agents that do not set them up, and thus may mismatch with the prior context. By comparison, actions following preparatory agents and patients are presumably easier to process because a warranted event structure is fulfilled, albeit with different levels of ease (agents being easier than patients). Such an interpretation would be in line with reports of sustained centrally distributed negativities to the disruption of hierarchic structures of novel events during event comprehension (Pace et al., 2013), to unanticipated event completions (Reid et al., 2009), and to anticipatory processing during viewing of tango dancing by experts (Amoruso et al., 2014). To the extent that this attenuation is an index of relative ease of processing, the pattern of results suggests that cues in preparatory agents modulate processing of an upcoming completed action more than patients, which in turn are easier than non-preparatory agents.

Our results suggest that preparatory actions differentially influence the processing of events compared to passive postures (both patients and non-preparatory-agents), however, they do not prove that preparatory actions evinced a prediction for the subsequent action. Our findings could reflect backward-looking reanalysis of prior information at the completed action, rather than facilitation from incoming information based on forward-looking anticipations. Recent research has shown that comprehenders can anticipate event information (Abreu et al., 2012; Aglioti et al., 2008; Amoruso et al., 2014; Hard, Tversky, & Lang, 2006; Zacks, Braver, et al., 2001), and this aligns with the proposition that a comprehender can frontload processing to further facilitate comprehension of subsequent events (Kurby & Zacks, 2008; Zacks et al., 2007). The cues manipulated here could provide the basis for such anticipatory processing, though we cannot be sure without further investigation.

Finally, these results are consistent with the existence of generalized event schemas related to preparatory and completed actions stored in memory across numerous actions (Jackendoff, 2007; Lasher, 1981; Moens & Steedman, 1988). Specific events (punching, shaking hands, etc.) would thus be manifestations of this generalized event structure: viewing a puncher activates the punching schema which itself is embedded in a generalized preparation-head-coda schema. However, our results at present only hint at such a generalized structure. Future work could examine this by crossing preparatory states of one event (with or without their cues) with a subsequent head or coda of another.

4.1. Conclusions

These findings suggest that specific cues motivate the initial and subsequent processing of visual events, possibly linking to more generalized event schemas (Jackendoff, 2007; Lasher, 1981; Moens & Steedman, 1988). Such schemas could allow for efficient processing of known events stored in memory (Strickland & Keil, 2011)—which would likely be enhanced by expertise (Abreu et al., 2012; Aglioti et al., 2008; Amoruso et al., 2014)—and could enable recognition of violations in the structure of novel events (Pace et al., 2013). Such stored event knowledge thus combines with the monitoring of transient meaningful changes in the situation in the course of event comprehension (Zacks et al., 2007).

Highlights.

  • Across domains, agents typically have a processing advantage over patients

  • We hypothesized that agents are cued by preparatory postures prior to completed actions

  • Preparatory agents evoked a greater frontal positivity than non-preparatory agents or patients

  • Completed actions after non-preparatory agents elicited a reduced frontal positivity

  • Preparatory postures cue the processing of agents and subsequent actions

Acknowledgments

Thanks go to Ben Amsel and Mirella Manfredi for aid in data gathering and analysis. Fantagraphics Books is thanked for their generous donation of The Complete Peanuts. This research was supported by NIH grant #5R01HD022614, and NIH funded (T32) postdoctoral training grants through the Center for Research in Language and the Institute for Neural Computation at UC San Diego.

Footnotes

Declaration of interest

The authors declare no conflicts of interest.

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

References

  1. Abreu AM, Macaluso E, Azevedo RT, Cesari P, Urgesi C, Aglioti SM. Action anticipation beyond the action observation network: a functional magnetic resonance imaging study in expert basketball players. European Journal of Neuroscience. 2012;35(10):1646–1654. doi: 10.1111/j.1460-9568.2012.08104.x. [DOI] [PubMed] [Google Scholar]
  2. Aglioti SM, Cesari P, Romani M, Urgesi C. Action anticipation and motor resonance in elite basketball players. Nature Neuroscience. 2008;11(9):1109–1116. doi: 10.1038/nn.2182. [DOI] [PubMed] [Google Scholar]
  3. Amoruso L, Gelormini C, Aboitiz F, Alvarez González M, Manes F, Cardona J, et al. N400 ERPs for actions: Building meaning in context. Frontiers in Human Neuroscience. 2013:7. doi: 10.3389/fnhum.2013.00057. [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Amoruso L, Sedeño L, Huepe D, Tomio A, Kamienkowski J, Hurtado E, et al. Time to Tango: Expertise and contextual anticipation during action observation. NeuroImage. 2014;98(0):366–385. doi: 10.1016/j.neuroimage.2014.05.005. [DOI] [PubMed] [Google Scholar]
  5. Bach P, Gunter TC, Knoblich G, Prinz W, Friederici AD. N400-like negativities in action perception reflect the activation of two components of an action representation. Social Neuroscience. 2009;4(3):212–232. doi: 10.1080/17470910802362546. [DOI] [PubMed] [Google Scholar]
  6. Brouwer H, Fitz H, Hoeks J. Getting real about Semantic Illusions: Rethinking the functional role of the P600 in language comprehension. Brain Research. 2012;1446:127–143. doi: 10.1016/j.brainres.2012.01.055. [DOI] [PubMed] [Google Scholar]
  7. Cohn N, Kutas M. Getting a cue before getting a clue: Event-related potentials to inference in visual narrative comprehension. Neuropsychologia. 2015;77:267–278. doi: 10.1016/j.neuropsychologia.2015.08.026. [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Cohn N, Kutas M. What’s your neural function, visual narrative conjunction? Grammar, meaning, and fluency in sequential image processing. Cognitive Research: Principles and Implications. 2017;2(27):1–13. doi: 10.1186/s41235-017-0064-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Cohn N, Maher S. The notion of the motion: The neurocognition of motion lines in visual narratives. Brain Research. 2015;1601:73–84. doi: 10.1016/j.brainres.2015.01.018. [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Cohn N, Paczynski M. Prediction, events, and the advantage of Agents: The processing of semantic roles in visual narrative. Cognitive Psychology. 2013;67(3):73–97. doi: 10.1016/j.cogpsych.2013.07.002. [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Cohn N, Paczynski M, Jackendoff R, Holcomb PJ, Kuperberg GR. (Pea)nuts and bolts of visual narrative: Structure and meaning in sequential image comprehension. Cognitive Psychology. 2012;65(1):1–38. doi: 10.1016/j.cogpsych.2012.01.003. [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Donchin E, Coles MGH. Is the P300 component a manifestation of context updating? Behavioral and Brain Sciences. 1988;11(03):357–374. [Google Scholar]
  13. Dowty D. Thematic proto-roles and argument selection. Language. 1991:547–619. [Google Scholar]
  14. Dryer MS. Order of subject, object and verb. In: Dryer MS, Haspelmath M, editors. The World Atlas of Language Structures Online. Munich: Max Planck Digital Library; 2011. [Google Scholar]
  15. Dupont P, Orban G, De Bruyn B, Verbruggen A, Mortelmans L. Many areas in the human brain respond to visual motion. Journal of neurophysiology. 1994;72(3):1420–1424. doi: 10.1152/jn.1994.72.3.1420. [DOI] [PubMed] [Google Scholar]
  16. Gershoff-Stowe L, Goldin-Meadow S. Is there a natural order for expressing semantic relations? Cognitive Psychology. 2002;45:375–412. doi: 10.1016/s0010-0285(02)00502-9. [DOI] [PubMed] [Google Scholar]
  17. Giglio ACA, Minati L, Boggio PS. Throwing the banana away and keeping the peel: Neuroelectric responses to unexpected but physically feasible action endings. Brain Research. 2013;1532(0):56–62. doi: 10.1016/j.brainres.2013.08.017. [DOI] [PubMed] [Google Scholar]
  18. Goldin-Meadow S. The Resiliance of Language: What Gesture Creation in Deaf Children Can Tell Us About How All Children Learn Language. New York and Hove: Psychology Press; 2003. [Google Scholar]
  19. Goldin-Meadow S, Feldman H. The development of language-like communication without a language model. Science, New Series. 1977;197(4301):401–403. doi: 10.1126/science.877567. [DOI] [PubMed] [Google Scholar]
  20. Goldin-Meadow S, So WC, Ôzyûrek &, Mylander C. The natural order of events: How speakers of different languages represent events nonverbally. Proceedings of the National Academy of Sciences. 2008;105(27):9163–9168. doi: 10.1073/pnas.0710060105. [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. Greenberg JH. Some universals of grammar with particular reference to the order of meaningful elements. In: Greenberg JH, editor. Universals of Grammar. Cambridge, MA: MIT Press; 1966. pp. 73–113. [Google Scholar]
  22. Gruber JS. Unpublished Dissertation. Massachusetts Institute of Technology; 1965. Studies in lexical relations. [Google Scholar]
  23. Hafri A, Papafragou A, Trueswell JC. Getting the Gist of Events: Recognition of Two-Participant Actions From Brief Displays. Journal of Experimental Psychology: General. 2012;142(3):880–905. doi: 10.1037/a0030045. [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Hard BM, Tversky B, Lang DS. Making sense of abstract schemas: Building event schemas. Memory and Cognition. 2006;34(6):1221–1235. doi: 10.3758/bf03193267. [DOI] [PubMed] [Google Scholar]
  25. Jackendoff R. Language, Consciousness, Culture: Essays on Mental Structure (Jean Nicod Lectures) Cambridge, MA: MIT Press; 2007. [Google Scholar]
  26. Kemmerer D. The cross-linguistic prevalence of SOV and SVO word orders reflects the sequential and hierarchical representation of action in Broca’s area. Language and Linguistics Compass. 2012;6(1):50–66. [Google Scholar]
  27. Kilner JM. More than one pathway to action understanding. Trends in Cognitive Sciences. 2011;15(8):352–357. doi: 10.1016/j.tics.2011.06.005. [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Kourtzi Z, Kanwisher N. Activation in Human MT/MST by Static Images with Implied Motion. Journal of Cognitive Neuroscience. 2000;12(1):48–55. doi: 10.1162/08989290051137594. [DOI] [PubMed] [Google Scholar]
  29. Kuperberg GR. The pro-active comprehender: What event-related potentials tell us about the dynamics of reading comprehension. In: Miller B, Cutting L, McCardle P, editors. Unraveling the Behavioral, Neurobiological, and Genetic Components of Reading Comprehension. Baltimore: Paul Brookes Publishing; 2013. pp. 176–192. [Google Scholar]
  30. Kurby CA, Zacks JM. Segmentation in the perception and memory of events. Trends in Cognitive Science. 2008;12(2):72–79. doi: 10.1016/j.tics.2007.11.004. [DOI] [PMC free article] [PubMed] [Google Scholar]
  31. Lasher MD. The cognitive representation of an event involving human motion. Cognitive Psychology. 1981;13(3):391–406. [Google Scholar]
  32. Moens M, Steedman M. Temporal ontology and temporal reference. Computational linguistics. 1988;14(2):15–28. [Google Scholar]
  33. Pace A, Carver LJ, Friend M. Event-related potentials to intact and disrupted actions in children and adults. Journal of Experimental Child Psychology. 2013;116(2):453–470. doi: 10.1016/j.jecp.2012.10.013. [DOI] [PMC free article] [PubMed] [Google Scholar]
  34. Reid VM, Hoehl S, Grigutsch M, Groendahl A, Parise E, Striano T. The Neural Correlates of Infant and Adult Goal Prediction: Evidence for Semantic Processing Systems. Developmental Psychology. 2009;45(3):620–629. doi: 10.1037/a0015209. [DOI] [PubMed] [Google Scholar]
  35. Robertson SS, Suci GJ. Event perception by children in the early stages of language production. Child Development. 1980;51(1):89–96. [PubMed] [Google Scholar]
  36. Schank RC, Abelson R. Scripts, Plans, Goals and Understanding. Hillsdale, NJ: Lawrence Earlbaum Associates; 1977. [Google Scholar]
  37. Segalowitz NS. The perception of semantic relations in pictures. Memory and Cognition. 1982;10(4):381–388. doi: 10.3758/bf03202430. [DOI] [PubMed] [Google Scholar]
  38. Senior C, Barnes J, Giampietroc V, Simmons A, Bullmore ET, Brammer M, et al. The functional neuroanatomy of implicit-motion perception or ‘representational momentum’. Current Biology. 2000;10(1):16–22. doi: 10.1016/s0960-9822(99)00259-6. [DOI] [PubMed] [Google Scholar]
  39. Shirai N, Imura T. Emergence of the ability to perceive dynamic events from still pictures in human infants. Scientific Reports. 2016;6(37206):1–11. doi: 10.1038/srep37206. [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. Sitnikova T, Holcomb PJ, Kuperberg GR. Two neurocognitive mechanisms of semantic integration during the comprehension of visual real-world events. Journal of Cognitive Neuroscience. 2008;20(11):1–21. doi: 10.1162/jocn.2008.20143. [DOI] [PMC free article] [PubMed] [Google Scholar]
  41. Sitnikova T, Rosen BR, Lord LD, West WC. Understanding human original actions directed at real-world goals: The role of the lateral prefrontal cortex. NeuroImage. 2014;103(0):91–105. doi: 10.1016/j.neuroimage.2014.09.012. [DOI] [PMC free article] [PubMed] [Google Scholar]
  42. Smith DM. Neurophysiology of action anticipation in athletes: A systematic review. Neuroscience & Biobehavioral Reviews. 2016;60:115–120. doi: 10.1016/j.neubiorev.2015.11.007. [DOI] [PubMed] [Google Scholar]
  43. Strickland B. Language Reflects “Core” Cognition: A New Theory About the Origin of Cross-Linguistic Regularities. Cognitive Science. 2016 doi: 10.1111/cogs.12332. n/a-n/a. [DOI] [PubMed] [Google Scholar]
  44. Strickland B, Keil F. Event completion: Event based inferences distort memory in a matter of seconds. Cognition. 2011;121(3):409–415. doi: 10.1016/j.cognition.2011.04.007. [DOI] [PMC free article] [PubMed] [Google Scholar]
  45. Urgesi C, Maieron M, Avenanti A, Tidoni E, Fabbro F, Aglioti SM. Simulating the Future of Actions in the Human Corticospinal System. Cerebral Cortex. 2010;20(11):2511–2521. doi: 10.1093/cercor/bhp292. [DOI] [PubMed] [Google Scholar]
  46. Urgesi C, Savonitto MM, Fabbro F, Aglioti SM. Long- and short-term plastic modeling of action prediction abilities in volleyball. Psychological Research. 2011;76(4):542–560. doi: 10.1007/s00426-011-0383-y. [DOI] [PubMed] [Google Scholar]
  47. van Elk M, Bousardt R, Bekkering H, van Schie HT. Using Goal- and Grip-Related Information for Understanding the Correctness of Other’s Actions: An ERP Study. PLoS ONE. 2012;7(5):e36450. doi: 10.1371/journal.pone.0036450. [DOI] [PMC free article] [PubMed] [Google Scholar]
  48. Verfaillie K, Daems A. The priority of the agent in visual event perception: On the cognitive basis of grammatical agent-patient asymmetries. Cognitive Linguistics. 1996;7(2):131–148. [Google Scholar]
  49. Webb A, Knott A, Macaskill MR. Eye movements during transitive action observation have sequential structure. Acta Psychologica. 2010;133(1):51–56. doi: 10.1016/j.actpsy.2009.09.001. [DOI] [PubMed] [Google Scholar]
  50. West WC, Holcomb P. Event-related potentials during discourse-level semantic integration of complex pictures. Cognitive Brain Research. 2002;13:363–375. doi: 10.1016/s0926-6410(01)00129-x. [DOI] [PubMed] [Google Scholar]
  51. Wilson F, Papafragou A, Bunger A, Trueswell J. Rapid Extraction of Event Participants in Caused Motion Events. Proceedings from the 33rd Annual Meeting of the Cognitive Science Society; Hillsdale, NJ: Erlbaum; 2011. [Google Scholar]
  52. Wood JN, Grafman J. Human prefrontal cortex: processing and representational perspectives. Nature Reviews Neuroscience. 2003;4(2):139–147. doi: 10.1038/nrn1033. [DOI] [PubMed] [Google Scholar]
  53. Zacks JM, Braver TS, Sheridan MA, Donaldson DI, Snyder AZ, Ollinger JM, et al. Human brain activity time-locked to perceptual event boundaries. Nature Neuroscience. 2001;4(6):651–655. doi: 10.1038/88486. [DOI] [PubMed] [Google Scholar]
  54. Zacks JM, Kurby CA, Eisenberg ML, Haroutunian N. Prediction error associated with the perceptual segmentation of naturalistic events. Journal of Cognitive Neuroscience. 2011;23(12):4057–4066. doi: 10.1162/jocn_a_00078. [DOI] [PMC free article] [PubMed] [Google Scholar]
  55. Zacks JM, Speer NK, Swallow KM, Braver TS, Reynolds JR. Event perception: A mind-brain perspective. Psychological Bulletin. 2007;133(2):273–293. doi: 10.1037/0033-2909.133.2.273. [DOI] [PMC free article] [PubMed] [Google Scholar]
  56. Zacks JM, Tversky B. Event structure in perception and conception. Psychological Bulletin. 2001;127(1):3–21. doi: 10.1037/0033-2909.127.1.3. [DOI] [PubMed] [Google Scholar]
  57. Zacks JM, Tversky B, Iyer G. Perceiving, remembering, and communicating structure in events. Journal of Experimental Psychology. 2001;130(1):29–58. doi: 10.1037/0096-3445.130.1.29. [DOI] [PubMed] [Google Scholar]
  58. Zalla T, Pradat-Diehl P, Sirigu A. Perception of action boundaries in patients with frontal lobe damage. Neuropsychologia. 2003;41(12):1619–1627. doi: 10.1016/s0028-3932(03)00098-8. [DOI] [PubMed] [Google Scholar]
  59. Zeki S, Watson J, Lueck C, Friston KJ, Kennard C, Frackowiak R. A direct demonstration of functional specialization in human visual cortex. The Journal of neuroscience. 1991;11(3):641–649. doi: 10.1523/JNEUROSCI.11-03-00641.1991. [DOI] [PMC free article] [PubMed] [Google Scholar]

RESOURCES