Event perception and memory

Abstract

Events make up much of our lived experience, and the perceptual mechanisms that represent events in experience have pervasive effects on action control, language use, and remembering. Event representations in both perception and memory have rich internal structure and connections one to another, and both are heavily informed by knowledge accumulated from previous experiences. Event perception and memory have been identified with specific computational and neural mechanisms, which show protracted development in childhood and are affected by language use, expertise, and brain disorders and injuries. Current theoretical approaches focus on the mechanisms by which events are segmented from ongoing experience, and emphasize the common coding of events for perception, action, and memory. Abetted by developments in eye-tracking, neuroimaging, and computer science, research on event perception and memory is moving from small-scale laboratory analogs to facing the complexity of events in the wild.

Introduction

Event perception is the set of mechanisms by which an organism represents the activity in which it is immersed. Research on event perception has roots in Gestalt psychology (Köhler 1929), in Michotte’s (1946) studies of perceptual causality, in the ecological psychology of human behavior (Barker 1963), and in the social psychology of attribution (Newtson 1976). Miller and Johnson-Laird (1976), in their broad and original overview of language and perception, articulated what would come to be some of the central issues in the field of event cognition: the relationships between objects and events, the fact that events are ephemeral, disappearing even as they occur, and the tight relationships between goals in human action and the parts of events in perception.

In the 1970s and early 1980s there was a burst of interest in the visual perception of simple motion events. In 1980, Gunnar Johansson and his colleagues wrote a chapter for the Annual Review of Psychology titled “Event perception” (Johansson et al. 1980). Around the same time, Gibson’s (1979) volume on ecological perception discussed events in some detail, Cutting (1981) published six tenets for event perception, and the first International Conference on Event Perception took place (Warren & Shaw 1985). Interest then waned until the turn of the century, when a new wave of research took a broader view that encompassed the other sensory modalities, language perception, and the integration of perception and action. Since then, interest in event perception and its interaction with other aspects of psychology has grown steadily.

The mechanisms of event perception are fundamental organizers of conscious experience, of active behavior, of language use, and of the experience of remembering the past. The broader term event cognition refers to event perception and its interfaces with action control, language, and memory. In this review, I will first summarize some of the key empirical phenomena in event cognition, focusing on perception and memory. I will then give an overview of current theoretical models and situate them relative to other ideas in psychology and neuroscience. I will conclude by describing some of the methodological developments that are shaping the field.

The psychological study of event perception is broad, and the study of event memory even broader. This review will necessarily be selective, focusing on the understanding of everyday events across the lifespan.¹ Some topics will receive short shrift here but have been recently been reviewed elsewhere (Radvansky & Zacks 2014): I will say little about how event perception and memory are affected by neurological and psychological disorders (for a review, see Zacks & Sargent 2010). I will also say little about the applications of event cognition to diagnosing and improving comprehension and memory (see Richmond et al. 2017, and commentaries). Finally, I will say little about event cognition in text comprehension—though it is important to note that text comprehension research has had a strong influence on current theories and research programs in event cognition (Radvansky & Zacks 2014; Zwaan & Radvansky 1998).

The structure of events in the world and in perception

Everyday experience can be described as chaotic—both in the usual sense of the word and in the technical sense of dynamical systems theory. The specific movements of leaves in the breeze or of people in a crowd are unpredictable and are sensitive to small fluctuations. At the same time, everyday experience can be described as highly structured, regular, and predictable. The larger patterns of blowing leaves or crowds of people are often quite straightforward; people’s actions can be predicted from their goals, from their stable characteristics, and from the situation around them. What is special, perhaps, about human activity is that the variables that organize the activity are removed from the relevant physical variables—they track actions and intentions rather than movements, and track objects rather than shapes or surfaces (Richmond et al. 2017). Human perceptual and conceptual systems are geared to track these regularities even when this requires abstracting considerably from the sensory input.

Continuous experience is segmented into continuous events

One thoroughgoing form of abstraction in perception is segmentation—dividing a continuum into parts. (Segmentation is sometimes described as unitization—grouping some of a continuum into a gestalt. These are two sides of the same coin.) Much research on event segmentation has utilized variants of a procedure in which observers press a key to mark boundaries between events while watching a movie or slide show, or while reading or listening to a story (Newtson 1973). Studies applying this procedure to depictions of everyday activities have revealed some key features of event segmentation behavior, summarized in Table 1. Recent studies of event segmentation have extended the range of presentation media and materials, exploring the segmentation of very simple displays and quite complex ones. On the simple end, when people viewed animations of flashing objects, the introduction of new objects is sufficient to induce event boundaries (Tauzin 2015), and minimal displays of human body motion are sufficient to support robust segmentation, even when the displays were inverted and object information was removed (Hemeren & Thill 2011). On the complex end, a recent study of the relationship between editing techniques in commercial film and segmentation revealed that both changes in the narrated situation (shifts in space, characters, and time) and editing transitions (dissolves, fades, and wipes) are associated with higher levels of segmentation (Cutting 2014).

Table 1:

Characteristics of behavioral event segmentation (see Radvansky & Zacks, esp. Ch. 5).

Intersubjective agreement	Observers tend to identify similar boundaries.
Correlated with change	Observers tend to identify boundaries when more features of the stimulus are changing.
Varying grain size	People can adjust the temporal grain at which they report event boundaries.
Partonomic hierarchy	Fine-grained events cluster into larger coarse-grained events.

Most previous studies of visual event segmentation have presented activity from a third-person perspective. Two recent studies have explored the first-person perspective. Magliano and colleagues (2014) presented movies recorded from a first-person video game and found good segmentation agreement, replicating what is seen with other stimuli. Also replicating previous results, event boundaries tended to occur when actors’ goals changed. Swallow and colleagues (2018) directly compared first-person and third-person perspectives, using two cameras, one head-mounted and one tripod-mounted, to record everyday activities. They found strong agreement between segmentation from the two perspectives.

Event representations have internal structure

The events that are distilled from the stream of behavior are not themselves undifferentiated clumps. Rather, they have considerable structure within them. An event unfolds within a spatiotemporal framework and includes entities such as people and objects and the relations amongst them. The representation of this structured segment of experience is called a working event model (Radvansky & Zacks 2014). (I will reserve the term “working event model” for representations that are immediately accessible, whereas the broader term “event model” includes immediately accessible representations and representations in long-term memory.) Working event models are perceptual representations, because they represent currently-unfolding activity. At the same time, they are also working memory representation, because they maintain information throughout the duration of an event. There is strong converging evidence that event models are rapidly established during event comprehension and are supported by specialized neural mechanisms that are distinct from representations of event components (Stawarczyk et al. under review).

For events that are the perception of a human action, the configuration of an agent (who is doing the action), a patient (to whom or what is the action being done), and an action (what action is being performed) is particularly important. Observers can extract this configuration from pictures presented as briefly as 37 ms and subsequently masked (Hafri et al. 2012), and information about the roles of agent and patient is quickly bound to other features of the entities involved (Hafri et al. 2018). Information about an entity’s role (Hafri et al. 2012) and what it might do (Cohn et al. 2017) is often available from its pose: People acting as agents generally have more stretched-out body poses than those on whom they are acting, and the windup to a kick appears quite different from the windup to a throw. There appears to be a regularity to the time course of establishing an event configuration, with agents usually being identified first (Webb et al. 2010). A key feature of agents is that they are animate, and humans appear to have specialized perceptual mechanisms to identify animate agents based on static features and movements. Neurophysiologically, these mechanisms draw on brain areas including ones in the ventral temporal cortex specialized for representing face and body features, on areas in the lateral temporo-occipital cortex specialized for movement processing, and on networks associated with theory of mind (Heberlein 2008). In the perception of comic strips, viewers look at agents longer than they do patients, and are more willing to make predictions about what will happen in a subsequent frame based on pictures of agents than of patients (Cohn & Paczynski 2013). In short, event perception rapidly establishes the configuration of agent, patient, and action, with agents having primacy.

When comprehenders process information about the action itself, they appear to focus on the action’s goal rather than on the particulars of its surface structure. Goal focus is evident early in infancy; studies using imitation and habituation paradigms have shown that infants attend to adults’ goals at the expense of the specifics of their movements (Woodward 2009). By age four, children can better recognize pictures depicting the goal of an action than other pictures, and this advantage is present in adulthood (Papafragou 2010). Adults are better able to detect changes in movie clips when those clips change goal-relevant movement features than when they change features that are not associated with the action’s goal (Loucks & Pechey 2016). This goal processing contributes to observers’ ability to make predictions about upcoming action. For example, observers’ eye gaze anticipates which object is the goal of an actor’s reach, such that the eyes land on the target object well before the actor’s hand arrives; this is true both for observers and for actors themselves (e.g., Eisenberg et al. 2018; Flanagan & Johansson 2003; Hayhoe & Ballard 2005). One important aspect of goals is that they are a component of event structure to which observers and actors have quite different access. Someone performing an action has access to information about their goals and plans that observers cannot access (though such information is by no means without error). Differences in the information available to actors and to observers can lead to differences in event segmentation (Fournier & Gallimore 2013).

Building a working event model requires cognitive work. This can be seen in slower processing of event boundaries in reading (e.g., Pettijohn & Radvansky 2016) and self-paced slide shows (Hard et al. 2011). In language processing, priming studies suggest that reading a word naming one part of an event primes other components of that event. For example, verbs prime the agents, patients, and instruments of their associated events (Ferretti et al. 2001), and are in turn primed by those components (McRae et al. 2005). There is evidence that the effort to build event structures is re-used when possible: Successive sentences sharing a goal-subgoal decomposition were read more quickly than those that did not share goal-subgoal structure (Allen et al. 2010). Event representations can have more or less complex internal structure, and building more complex structures may place unique demands on the system. For example, if one reads that “The chef will weigh the onion. And then she will smell the onion,” the onion is in the same state throughout. However, if one reads that “The chef will chop the onion. And then she will smell the onion,” the onion changes from intact to chopped, requiring the representation of two object states. Solomon and colleagues (2015) compared the processing of these two types of passages during fMRI scanning. They focused on a region of the ventrolateral prefrontal cortex that is selectively active during instances of representational conflict, and found that it was more active when objects needed to be represented in two states. Data from eye-tracking and EEG measures show that during the processing of event descriptions, information from the visual environment is integrated rapidly and continuously (e.g., Altmann & Mirkovic 2009; Knoeferle et al. 2008). This supports the proposal that event comprehension operates on a common representational format that integrates information from multiple sources including language and perception.

Event perception is structured by partonomy and other relations

Event models not only have internal structure; they also form rich connections across events. Partonomic hierarchy (see Table 1) can be thought of as either an aspect of within-event structure or in terms of relationships across events: Viewed “downward” from an event to its sub-events, the parts are aspects of the within-event structure. Viewed “upward” from an event to the larger event of which it is a part, the sibling and parent events are cross-event relations. For goal-directed human activity, the part-subpart structure of events is conditioned strongly on the goal-subgoal structure that generates action. Hierarchical organization is ubiquitous in event segmentation data and in descriptions of events (Zacks et al. 2007). It appears to increase with familiarity with a particular event sequence (Hard et al. 2006) and with domain expertise (Bläsing et al. 2009).

Hierarchically organized event models are more complex than simple strings, but they retain the constraint of being continuous. An activity such as “walking a dog” may consist of sub-parts such as “putting on shoes,” “opening the door,” and “walking up the block,” but all of the temporal parts of the activity occupy a continuous stretch of time. Some activities, however, appear to form discontinuous events—for example “training for a marathon,” or “taking a calculus class.” Kubovy (2015) proposes that the mental representation such discontinuous events constitute projects that cohere despite their temporal discontinuity. Projects in turn are subcomponents of strands, which are large-scale structures such as “sports” or “school,” which are themselves temporally discontinuous. On this view, projects and strands run through lives as do strands through a weaving. An important question for future research is whether such structure is present during online perception of ongoing activity or is a construction established as part of the act of remembering.‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬

Event perception is heavily informed by knowledge

When someone experiences a particular event, they can bring to bear knowledge distilled from their many experiences with varying event types. A longstanding proposal is that event schemas or scripts capture knowledge about the temporal structure of events, entities, roles, and locations (Abelson 1981). How a perceiver understands a particular moment depends on how they can bring such knowledge to bear on the immediate stimulus. By monitoring eye movements while participants watched excerpts from a feature film, Loschky and colleagues (2015) showed that viewers’ looking patterns were more systematic when the clip they were viewing could be placed in a larger context. Knowledge about event categories can affect perception rapidly and automatically, as can be seen in a recent study of change detection (Strickland & Scholl 2015). Viewers were shown clips in which one object either went into another object or behind it. Participants were better able to detect a change an object’s width than its height when the object was shown going into a vertical container; when the object was going behind the other object there was no difference. This suggests that viewers’ visual systems tracked object width more carefully when viewing the event type—vertical containment—for which width is relevant.

The opportunity to deploy knowledge during event comprehension is associated with specific neural mechanisms. A study using multivoxel pattern analysis fMRI identified brain areas whose activity patterns distinguished between schemas for eating at a restaurant or flying from an airport (Baldassano et al. 2018). The analysis targeted areas that had consistently different patterns for the two schemas across multiple movie clips and story excerpts. Schema information was represented in medial components of the default mode network: medial PFC, superior frontal cortex, and medial posterior cortex. Another fMRI study of movie viewing found that presenting clips with a context that allowed for a schema to influence processing was associated with sustained fMRI responses in the inferior frontal cortex, middle temporal gyrus, and angular gyrus, all in the left hemisphere (Keidel et al. 2017). In contrast, when encoding under schema-poor conditions the parahippocampal gyrus and retrosplenial cortex were more strongly activated. One possibility is that the schema representations are stored in the medial network, and the left lateral network plays a role in deploying those representations to guide ongoing processing.

Event perception is predictive and inferential

One thing that knowledge does for event perception is to enable predictions about how events will unfold. If one is familiar with going to rock concerts, one can predict that after the lights go down the band will take the stage. Even in situations where perceivers have little relevant schema knowledge, their brains may make predictions based on more general knowledge such as knowledge about how bodies and objects move, about the relationships between goals and actions, and the like. In scene perception, Intraub and colleagues have shown that viewers quickly fill in information outside the field of vision, leading to systematic errors in picture memory (Intraub 2010). One kind of knowledge that can support strong inferences is knowledge about the causal consequences of actions. For example, kicking a soccer ball leads to the ball moving. When viewers see a sequence of video clips that show a player running up to a ball and then show the ball in flight, they often falsely recognize a clip showing the moment of contact that was not presented, after even quite brief delays or even when asked to monitor for the moment of contact while watching (Papenmeier et al. 2019; Strickland & Keil 2011).

The predictive nature of event perception can be seen in the previously-discussed fact that viewers look ahead to the goals of reaching motions (Eisenberg et al. 2018; Flanagan & Johansson 2003). Viewers can learn to predict even arbitrary, randomly-determined behavior sequences if they are given repeated experience, and this can allow them to look ahead to the targets of actions (Monroy et al. 2018). Predictive processing in event perception also can be seen in evoked EEG responses, particularly the N400 (for a review, see Amoruso et al. 2013; but see also Brouwer et al. 2012). In paradigms that use video editing, picture presentation, or language to tightly control the timing of event stimuli, researchers have shown that unpredicted happenings in event comprehension produce larger N400 responses.

There is a tight temporal relationship between prediction and event segmentation. The more unpredictable activity becomes, the more likely participants are to identify event boundaries (Huff et al. 2014). Making explicit predictions about the near future is more difficult near event boundaries, and attempting to do so is associated with increased fMRI activity in midbrain structures associated with signaling prediction error to the rest of the brain (Zacks et al. 2011). Predictive eye movements are also less prevalent near event boundaries (Eisenberg et al. 2018).

Events structure consciousness

The predictive nature of event perception illustrates a point highlighted by William James (1890): The conscious experience of the present is not an infinitesimal point but has temporal extent. The fact that event segmentation tasks are so easily learned, produce such reliable data, and correspond with ongoing neural events supports the conclusion that conscious experience is segmented into events (Zacks et al. 2007). The time course of other functions provides converging evidence. When participants are asked to monitor a complex situation for a visual target, the presence of an event boundary transiently disrupts performance (Huff et al. 2012). Consistent with this result, mind-wandering while watching a film was less likely at event boundaries than during the middles of events (Faber & D’Mello 2018). These findings suggest that model updating is resource-intensive: at event boundaries, those resources are less available for secondary tasks or off-topic wanderings of the mind.

Summary

In sum, perception is characterized by the segmentation of ongoing experience into meaningful events. Event representations have rich internal structure, and also rich relationships between them. The construction of these relationships is strongly influenced by knowledge, which enables online predictive inference about the unfolding of activity. The construction of working event models and the predictive nature of event perception highlight that the experience of “now” is smeared in time—there is no bright line between now, a moment ago, and a moment hence. This entails that there is no bright line between perceptual systems and memory systems (Christophel et al. 2017). A more fundamental distinction than that between perception and memory is that between the “expanded present” represented by a working model and events outside of that expanded present.

From perception to memory

Rubin and Umanath (2015) define event memory as “the mental construction of a scene, real or imagined, for the past or the future.” In other words, we can distinguish between representations of events that one is currently experiencing and representations of events that are based on other sources; event memories are the latter. One attractive feature of this conception of event memory is its emphasis on the constructive nature of memory. Another attractive feature is its acknowledgement that when one attempts bring an event to mind, the result depends on multiple signals and active strategic mechanisms; event memories are constructions rather than the mere reloading of previous event representations stored immutably in the brain’s vault.

This approach entails that experiencing a current event and remembering a previous event are underwritten by a common representational substrate: a working event model. The difference is that in perception we focus on how one’s current working model is populated by currently-available perceptual information, whereas in remembering we focus on how one’s event model is populated by retrieval of information from long-term memory. This analysis offers a novel take on the relationship between perception, working memory, and long-term memory.

Working memory representations are updated at event boundaries

Most theories of human memory distinguish between short-term (or primary or working) memory and long-term memory. An outstanding puzzle has been: what makes a memory long-term? The data presented in the previous section, in supporting the proposal that the brain maintains working event models and updates them periodically, suggests that in many cases the boundary between working memory and long-term memory may be the end of one event and the beginning of the next. Striking support for this proposal has come from a series of studies by Radvansky and colleagues on the memorial consequences of walking through doorways. In the initial experiments (Radvansky & Copeland 2006a; Radvansky et al. 2010), participants navigated a virtual reality environment in which they picked up objects and put them in a backpack, and later put down the objects. From time to time, they were probed to report which object was currently in their backpack. Controlling for distance traveled and time elapsed, participants were slower to respond if they had walked through a doorway after picking up an object. This effect has been found not just in virtual reality but also in real rooms (Radvansky et al. 2011) and in imagined ones (Lawrence & Peterson 2016). It does not appear to reflect simply a temporary disruption due to crossing the doorway because it persists when a constant delay is added to the retention interval (Pettijohn & Radvansky 2015); nor does it seem to be due to the dissimilarity in context between the encoding and retrieval conditions because it remains when participants return to the original room (Radvansky et al. 2011). These interactive paradigms converge with results from studies of narrative reading (e.g., Rinck & Bower 2000), and with studies of memory for recently-seen objects in movies (Swallow et al. 2009), which show that after a shift in location or of narrative time, memory retrieval is often less efficient.

Updating event models at boundaries would seem to be a resource-intensive operation. It may include comparing the contents of the old model with new perceptual information. Evidence supporting this idea comes from a study showing that, while viewing films, changes in actors’ clothing were noticed better at event boundaries than event middles (Baker & Levin 2015). If information that is present at event boundaries receives special processing, this may have consequences for later retrieval. Using fMRI, Swallow and colleagues (2011) showed that successful retrieval of visual object information encoded during an event boundary was associated with selective activity in the hippocampus and parahippocampal gyrus—structures associated with long-term memory in many studies in humans and other species. The delay in these studies was only 5 s, which supports the notion that what makes a memory long term may be whether it requires reaching back across an event boundary.

Thus, an event boundary may involve the segregation of the immediate present—subserved by working event models—from events in long-term memory. A natural implication of the hierarchical structure of activity, described in the previous section, is that event model updating happens on multiple timescales; some updates may only affect finer-grained event models, whereas others additionally update coarser-grained event models. If so, then updating at coarser-scale event boundaries should have larger effects on memory retrieval and brain activity. This hypothesis has yet to be tested.

Event boundaries also may condition what sorts of representations are formed in long-term memory. Recent studies of memory retrieval with longer delays have provided evidence that the experience of event structure in the present leaves heavy footprints on subsequent memory.

The structure of events in experience is mirrored in long-term memory

One idea that has become dominant in the psychology of human memory holds that the experience of remembering is fundamentally one of intentionally searching for information in the mind. While there is no doubt that people sometimes go into a mode of deliberate memory search (Tulving 1983), it may be that a more basic and more frequent mode of experiencing event memories results from involuntary associative cuing (Berntsen 2010). Features such as locations, people, or objects may quickly and automatically bring to mind events that are associated with those features. For example, a child, upon arriving in a classroom on Tuesday morning, might retrieve events of Monday’s class in that room, and upon encountering a friend one might retrieve features of the last conversation had with them. Such associative retrieval is adaptive: If a new event induces memories of similar previous events, those previous events are likely to provide valid predictive information about how the new event will proceed. But what if things change? What if there is a substitute teacher and as a result the events of yesterday’s lesson do not predict what will happen today? In such cases, event memory may impair rather than facilitate comprehension. However, if the child can register the discrepancy as part of the memory for Tuesday, this can result in a highly effective form of memory that embeds what happened on both days, the temporal relationship between them, and something about the variability associated with that context. A recent study of memory for changes in events provides evidence for such mechanisms (Wahlheim & Zacks in press).

Event memory is structured by the same dimensions as perception

What is the relationship between event structure in perception and in memory? There is strong evidence that the segments that are identified during event perception correspond to the representational units in subsequent memory. First, the boundaries themselves are remembered exceptionally well. For example, Huff and colleagues (2014) showed viewers episodes of a situation comedy, asked them to segment them, and tested their memory later. Event boundaries tended to occur at points in time when many features were changing, and those points were remembered better. Ezzyat and Davachi (2011) used a narrative priming paradigm to show that sentences cued memory retrieval of subsequent sentences more strongly when the two belonged to the same narrative event than when they did not, and that the binding of sentences within an event was associated with selective activity in brain areas including the ventromedial prefrontal cortex. Pettijohn and colleagues (2016) showed that breaking up the experience of studying a word list into two events (by walking through a doorway) increased the total number of studied words that were remembered later. They interpreted this manipulation as decreasing the degree to which words from the first half of the study experience interfered with those from the second half, and vice versa.

Such data support the existence of integrated event models in memory. However, event memory also depends on other representations—for example, representations of people and objects. Furthermore, even bound event models can be confused with similar events during memory retrieval. When one attempts to remember a past event, one constructs a working model that depends on these confusable event representations and other long-term memory representations, interference between similar representations can lead to confusions. For example, when people see a number of actors involved in a number of actions, confusions about which person performed which action are common (Earles et al. 2008; Kersten et al. 2013).

Just as there are structural relationships across events in perception, there are relationships across events in memory. One of the strongest organizers of relationships in event memory appears to be causal connection (Radvansky 2012). In fact, the organization of memory by causal relations is likely stronger in memory than in perception, because post-encoding processes elaborate causal relations. When people read narratives of event sequences, components with more causal connections are remembered better (Trabasso & Stein 1997). When people are asked to recall multiple events from distant periods in their lives, events that are causally related tend to cue each other (Brown 2005).

Spatial location is another powerful organizer of events in memory. It is much easier to remember the association between multiple objects and a single location than the association between multiple locations and a single object (e.g., Radvansky et al. 2017). For example, it is easier to remember that a potted plant, an ATM, and a poster are in a lobby than it is to remember that there is a potted plant in a lobby, a library, and a café. This suggests that a collection of objects in a location can be represented in a single event model, whereas representing the same object in multiple locations requires multiple event models.

Time is also a potential organizer of events in memory. Just as it is easier to associate multiple objects with a single location than multiple locations with a single object, it is easier to associate multiple objects with a single time period than multiple time periods with a single object (Radvansky et al. 1998). Hierarchical organization, prevalent in perception, is also prevalent in autobiographical memory. The same autobiographical cueing procedure that reveals causal relations in memory also shows clustering of events by membership in larger events (Brown 2005). If events are organized into part-subpart hierarchies in memory, this entails representing a certain amount of information about temporal distance, because two events that are both part of the same larger event will generally be closer in time than two events that are parts of different larger events. In addition, causal relations can scaffold memory for temporal order because causes precede their effects. However, some aspects of temporal organization, such as the order of events within a larger unit, can be quite weak in memory. For example, Wyer and Bodenhausen (1985) found that people recalling stories showed good memory for the order of actions within an event (possibly because these tend to be causally linked) but poor memory for the larger order of events.

For both space and time, there are a range of scales on which things can be organized. Temporal and spatial scales that have natural organization are probably represented—or reconstructed—better than those that are more arbitrarily organized, because those natural organizations facilitate prediction. For example, the arrangement of objects within a room is systematically related to the locations of doors and windows, and to the actions that take place within the room. However, the order of rooms on a hallway or of buildings on a street is much less consistently tied to objects and actions. This may be why, after watching many episodes of a television series, people have relatively good memory for the layout of rooms but poor memory for the spatial relations amongst rooms (Levin 2010). Similarly, the order of actions in making a sandwich is structured by causal and conventional relations that can facilitate predictions, but the order of which larger activities may precede or follow sandwich-making is less systematic (Wyer & Bodenhausen 1985). Gravina and Sederberg (2017) suggest that these relationships of systematicity and the predictions they allow for account for the temporal and spatial similarity gradients seen in memory representations in the medial temporal lobes (Nielson et al. 2015).

Memory for the duration of events is often important for planning future activities—and it is fascinatingly affected by features other than actual duration. In spatial navigation, routes that have more turns are remembered as being longer in space and also are mentally replayed more slowly (Bonasia et al. 2016). This appears to be an instance of a more general phenomenon: Intervals more filled with “stuff” are remembered as having taken longer when looking back on them. Wang and Gennari (2019) showed viewers animations, asked them to describe the animations, and then asked them to recall their duration. They found that those animations which elicited more extensive descriptions were remembered as having taken longer, controlling for actual duration. Similarly, routine events are usually described less richly than unusual ones, and such events are remembered as having been shorter (Avni-Babad & Ritov 2003). What is the “stuff” that goes into remembering duration? At least one source appears to be the number of sub-events in an interval. Jeunehomme and D’Argembeau (2018) asked participants to perform activities of daily living while wearing cameras, to segment their recordings, and then to estimate the durations of particular activities. They found that those activities which were segmented into more events were remembered as longer, controlling for actual duration. Finally, Bangert and colleagues (in press) observed a similar effect of event boundaries on judgments of the durations of intervals within movies of everyday activity.

Event memory, like perception, is heavily informed by knowledge

The fact that relationships within and amongst events are systematic means that they can be the subject of knowledge. We saw previously that knowledge affects event perception. The effects of knowledge on event memory are even more striking. Inspired by the influence of scripts and event schemas on memory for narrative text, Brewer and colleagues conducted several studies showing that memory for filmed events is influenced by knowledge about event categories (e.g., Brewer & Dupree 1983). Information that is congruent with an event schema is often remembered better than information that is irrelevant, in part because the schema can act as a bias during recall. For example, if one were asked to remember whether at a wedding the couple recited vows, one could respond based on a schema for weddings in general in addition to representations specific to that event. However, suppose the couple did not recite their vows, but instead had them tattooed onto their forearms. This might also be memorable, and indeed under many circumstances such discrepant information is also well remembered compared to schema-unrelated information—for example, the kind of car one took to the wedding as a guest. In a recent study, Bonasia and colleagues (2018) found that movie clips that were either congruent with a schema or discrepant were both remembered well compared to clips that were not strongly related to a script. Remembering schema-congruent clips led to more medial prefrontal fMRI activation, whereas remembering discrepant clips led to more medial temporal activation. This suggests that the influence of schemas on event memory may be mediated in part by the medial prefrontal cortex, and that the medial temporal lobe system is particularly taxed when memories need to bind arbitrary relations. In most studies of event comprehension and knowledge, the schemas in question are assumed to be well-learned. (We will see exceptions to this in the study of child development.) However, recent evidence suggests that event knowledge can be acquired from a few experiences (MacLean et al. 2018). In this study, participants tasted various foods in the laboratory, with a different experimenter conducting each tasting. After just a few of these events, a new food tasting experience in which the experimenter’s behavior deviated from the others was better remembered.

Remembering events is constructive and destructive

If event memory is the mental construction of a scene (Rubin & Umanath 2015), this suggests that the act of searching one’s memory for a particular episode is not so different from imagining a novel event or thinking about a counterfactual event. Indeed, neuroimaging and neuropsychological data strongly support a common mechanism for constructing events from memory, imagination, and reasoning (Addis et al. 2007; Schacter et al. 2012). A reasonable proposal is that all of these tasks depend on constructing an event model and their common neural correlates reflect either the representational medium of event models, the systems that are needed to construct such representations, or both. Barbey and colleagues (2009) used fMRI to examine the mechanisms of counterfactual reasoning and found that multiple dimensions of events were represented in the spatial locus of activation in the medial prefrontal cortex. This is consistent with the results of Baldassano et al. (2018), who found schema-specific patterns in this area during movie viewing. The coupling of this region with other default network regions during event encoding is associated with better subsequent memory of narrative sequences (Simony et al. 2016); This suggests that the common activations seen in event perception, event imagining, and event memory tasks reflect a functional, causal relationship.

The view of event memories as constructive also entails that remembering is reconstructive and destructive. That is, when one constructs an event memory, the operations of retrieval and the activation of knowledge during that retrieval affects one’s future remembering. For events, this can sometimes produce straightforward benefits to subsequent memory: For example, after viewing movies of everyday events, being reminded of them with pictures or titles reduces forgetting. However the same reminding also can impair memory for other events that were not reminded (Koutstaal et al. 1999). Eye witness memory studies show how a given retrieval attempt can produce negative effects on subsequent memory: For example, retrieving an event while trying to decide whether a mug shot matches the perpetrator of the event can lead the person pictured to be falsely incorporated into subsequent memories for the event (e.g., Kersten & Earles 2017). Post-encoding operations may affect not just the contents of long term memory event models, but also how activity is segmented in long term memory (Hohman et al. 2013).

Summary

In sum, although there is no bright line between perception and memory, there is good evidence for a distinction between information that is maintained in one’s current working event models and event information that is represented in other neural systems. Event memory can be conceived as the construction of a working model based on that other information—including episode-specific representations, knowledge, and new information generated during the construction process. The act of constructing a working event model creates new representations in these memory systems, which side-affect subsequent remembering. These operations have specialized neural mechanisms. The PM/AT framework (Ranganath & Ritchey 2012) summarizes and integrates current knowledge about these mechanisms. It proposes that one brain network, including the lateral temporal cortex and perirhinal cortex, supports the use of object knowledge and perceptual features in memory formation. Another network, including the medial prefrontal cortex, retrosplenial cortex, and parahippocampal cortex supports the organization of entities within a spatial framework into an event model, drawing on event schemas. This account fits well with the neurophysiological data reviewed here.

Group and individual differences in event cognition

There are substantial group and individual differences in event perception and event memory, which can inform theories of their mechanisms (Zacks & Sargent 2010). Here, I will focus on group-level effects of healthy aging and early Alzheimer’s disease, and of language, and on how individual differences in event perception relate to differences in memory.

The experience of events changes through adulthood, and is impaired in Alzheimer’s disease

We saw previously that event segmentation is characterized by agreement across observers and by hierarchical organization (“Segmentation of continuous experience into events”). Both of these features can be used to develop measures of group and individual differences. When asked to segment movies of everyday activities, older adults usually show lower segmentation agreement than younger adults (e.g., Kurby & Zacks 2018; but see Sargent et al. 2013). Older adults also show less hierarchical organization (Kurby & Zacks 2011). Interestingly, a recent study of the segmentation of filmed everyday events found no difference in segmentation agreement or alignment between young and older adults (Kurby & Zacks 2018). One possibility is that age differences in event segmentation may be reduced when older adults can use knowledge to construct richer event representations (Radvansky & Dijkstra 2007). Consistent with this proposal older adults, like younger adults, show robust updating at situational changes in narrative memory updating paradigms (Radvansky et al. 2003).

Early Alzheimer’s disease reflects a divergence from the path of healthy aging that is characterized by impairments in memory and thinking, which can initially be subtle but increase in severity and scope with disease progression. These include deficits in event segmentation and memory (e.g., Bailey et al. 2013b). They also include deficits in the ability to perform everyday activities, in and out of the laboratory (e.g., Giovannetti et al. 2008; Gold et al. 2015). One possibility is that these deficits result from disruption of event knowledge, or of the ability to use that knowledge effectively.

Language affects some aspects of event perception and memory—especially when using language

The language one speaks is a group difference that is of particular interest for perception and memory in general due to debates over whether and how language shapes thought. The effect of language on event perception and memory has attracted sustained attention because different languages represent aspects of event structure differently. For example, languages including English and Arabic encode in the form of their verbs whether an activity is viewed as ongoing over time (“is walking”) or as a whole (“walks”), whereas German does not. Because of this grammatical difference, German speakers are more likely to explicitly describe the locations of the endpoints of actions than are speakers of English or Arabic. (This is because whole actions tend to make less sense without their endpoints, whereas ongoing actions make sense with or without endpoints; for example “Stacey walks” sounds a little odd without a destination specified, whereas “Stacey is walking” sounds fine.) Corresponding with this linguistic difference, German speakers look more at action endpoints than do Arabic speakers (Flecken et al. 2014), and show larger P3 EEG responses to unexpected action endpoints than do English speakers (Flecken et al. 2015), even in tasks not involving language. However, such differences are not always observed. Papafragou and colleagues conducted several tests comparing Greek, which highlights path information more, to English, which highlights path information less, in designs similar to those of Flecken and colleagues. In these studies, no effects of language were found unless language was being used (Papafragou 2010; Papafragou et al. 2008; see also Gennari et al. 2002).

In addition to affecting attentional selection, language can affect segmentation. French verbs are more likely to represent motion paths more than do German verbs. For example, in a situation where a French speaker might say “La voiture a traversé le pont” (roughly, “The car crossed the bridge”), a German speaker would more likely say “Das Auto fuhr über die Brücke” (“The car drove over the bridge”). Whereas French verbs more often describe path (traverse/cross), German verbs more often describe manner of motion (fuhr/drive), conveying path information in a preposition (über/over). Given that information in a verb is obligatory, one might expect that changes in path would be more likely to lead to event model updating for speakers of French than of German; this has indeed been observed (Gerwien & von Stutterheim 2018).

The potential for language to influence event segmentation also can be seen in the gestures that speakers use to talk about events. For example, Avatime is a language that can use serial verbs to package a string of smaller actions into one event. For example, the sentence “lɛ̌ be-dzì e-mu-i” (roughly, “then return ascend”) means “Then they climbed up again.” When Avatime speakers use this construction, if they gesture while uttering a serial verb string the gesture spans the entire verb string, suggesting they conceive of it as a single unit (Defina 2016).

There are strong relationships between individual differences in event perception and in event memory

In addition to evidence for group differences, there is evidence for substantial individual differences in event perception—and individual differences in perception predict individual differences in memory. In a large-scale study of event segmentation and memory across the lifespan, people with higher segmentation agreement had better subsequent recall and recognition for events (Sargent et al. 2013). This held after controlling for individual differences in processing speed, working memory, crystallized knowledge, and laboratory episodic memory. Event knowledge also was a significant independent predictor of event memory. Within older adults, including those with early Alzheimer’s disease, better segmentation is associated with better event memory and with better ability to perform everyday actions (Bailey et al. 2013a,b; Kurby & Zacks 2011). Experimental interventions on event segmentation suggest that the relationship between event perception and event memory is causal: Interventions that improve event segmentation by instruction or by editing event stimuli improve subsequent event memory (Flores et al. 2017; Gold et al. 2016).

In short, event perception and event memory vary across individuals and groups. Moreover, differences in these two abilities are related to each other and also to other domains including everyday action performance. Intervening to improve event segmentation improves event memory, which is consistent with the previously-discussed evidence that segmentation during perception shapes the representational units of event memory. Differences between young and older adults highlight the importance of adult development in event perception and memory. But of course, these abilities do not emerge fully formed in 18-year-olds. How do they develop in childhood?

The development of event cognition reflects growth in knowledge, language use, and deliberate rehearsal

For many years, Piagetian accounts dominated the study of event understanding and event memory. These held that infants possess little of the conceptual structure to support adult-like performance, and that this conceptual structure emerges incrementally over years of experience in the world. Modern views, however, identify early competence in areas of both event perception and event memory, as well as extended growth. One important aspect of growth in perception and memory, which was noted by early theorists as well as current ones, is growth in schematic knowledge about events (Bauer 2006).

Psychology’s picture of the development of event perception is much hazier than its picture of the development of event memory. There is substantial evidence for early competence in infancy, but little data on how infant abilities develop into those of adults. Well within the first year of life, infants can individuate actions within a continuous stream of behavior (Sharon & Wynn 1998; Wynn 1996), and show evidence of segmenting activity at points that are identified by adults as event boundaries (Baldwin et al. 2001; Hespos et al. 2009, 2010; Saylor et al. 2007). Infants are sensitive to causal interactions such as in the Michotte launching effect (Cohen & Amsel 1998), and can use experiences with recent events (Nakano & Kitazawa 2017) and the statistics of extended experience with event types (Monroy et al. 2017) to guide their ongoing processing of current events. However, the infant studies have used almost completely different measures of event perception than have adult studies, and little is known about the developmental trajectory of event perception between infancy and young adulthood.

Much more is known about the development of event memory. This is true, in part, because of a strong applied interest in what children can remember for the sake of legal testimony. Whereas older views held that children lacked the ability to form event memories before the advent of language, a new generation of research has found evidence for early competence in event memory formation and continuity of development through the early phases of language acquisition (Bauer 2006). By 18–24 months, toddlers can recall elements of an otherwise-forgotten event if they are cued with features of that event (Sheffield & Hudson 1994); this indicates that the elements are bound into a coherent whole. At the same age, toddlers show better memory for event boundaries than for event middles, and show selective memory impairment from occluding event boundaries during encoding (Sonne et al. 2016, 2017). From ages 2 to 10, the development of event memory is linked to growth in event knowledge (Fivush 1997; Hudson et al. 1992). Young children tend to misremember events as having conformed to the schemas they have learned, but by 7 or 8 years, children have more ability to recall deviations from schemas and to recall schema-irrelevant details, especially with environmental support (Brown & Pipe 2003). One important aspect of children’s event knowledge is goal relations, which have strong early and continuing effects on event memory (Loucks et al. 2016). By age 10, children’s event memory looks qualitatively like that of adults, though encoding efficiency and completeness continue to increase with age. Together, the limited perceptual data and more extensive memory data indicate that very young children make use of event models that are in some ways quite adult-like, but that these become more elaborated over development, in part due to the development of knowledge structures.

Summary

In short, contrary to earlier views, there is good reason to think that infants construct event representations that are similar in form and content to those of adults, though more limited in many aspects. The development of event memory is conditioned strongly on the development of knowledge about event classes. An important issue for future research is the role of knowledge and other factors in the development of event perception.

Theoretical approaches to event cognition

In previous generations, theories of event cognition tended to deal with perception (Gibson 1979; Johansson et al. 1980; Michotte 1946), memory and inference (Abelson 1981), or action control (Miller et al. 1960). The memory and action control theories emphasized structure in mental representations, whereas the perceptual theories emphasized structure in the environment. In contrast, current theories tend to bridge at least two of these domains, and to consider structure both in the mind and in the world.

Common, multimodal coding of events and actions

One problem taken on by current theories is the relationship between people’s roles as perceivers and as actors in the stream of events. One such theory, which has been highly influential, is the Theory of Event Coding (TEC) proposed by Hommel and colleagues (2001). TEC is a qualitative account of how perception and action control are integrated in events on a short time-scale, from tens of milliseconds to a few seconds. It can be described in terms of a set of proposals about how immediate events are represented, listed in Table 2. The third and fourth proposals state that event representations are composites of feature codes. The range of possible codes is determined by previous experience encoded as knowledge. For example, most animals will have access to codes such as “red” and “short” to characterize a cup, but as one interacts with a cup one might add codes for its previous location and the level of liquid inside. The final proposal governs how event representations can be shaped by current interests: If one intends to pick up a cup to drink from it, shape features will be highly weighted; if one is selecting a cup to purchase in a gift shop, features related to attractiveness and desirability will receive more weight.

Table 2:

Posits of the theory of event coding (TEC; Hommel et al., 2001).

1. Shared representations	Representations for perceiving and action planning are functionally equivalent; both are correspondences between brain states and anticipated interactions of the actor/observer with external events.
2. Distal coding	Events are coded in terms of distal features such as objects and their movements, rather than in terms of proximal features such as the feel of touching an object or a sequence of limb movements planned to move it.
3. Feature codes	Stimuli and objects are both represented as temporary composites of feature codes.
4. Activation and integration	The formation of an event code is composed of two phases: activation and integration.
5. Intentionality	The combination of feature codes is weighted by current goals and intentions.

A key set of findings accounted for by TEC involves situations in which actions or intentions interact with perception. Planned actions can affect how visual or auditory stimuli are weighted, and irrelevant features of visual or auditory stimuli can affect action execution. TEC’s proposal that feature codes are first activated and then bound has the advantage that it can account for both facilitation from overlapping features and also for interference. However, in any given situation it is often not clear whether to predict facilitation or interference. As evidence has accumulated, it appears that additional mechanisms are needed to account for the complex patterns of facilitation and interference that are observed (Zwickel & Prinz 2012).

One of the most grounded ways to build a model of cognitive representation for perception and action is to explicitly build a controller for an agent. The REtrospective and PRospective Inference SchEme (REPRISE) model (Butz et al. 2018) does just that. REPRISE is a recurrent neural network that uses a bank of contextual neurons to represent which of a number of potential event states the network is currently experiencing, and to bias processing in the rest of the network based on the currently-inferred event state. The model identifies which state it is in by looking backwards at the recent dynamics of the system, and uses the current state and the values of the physical variables describing the environment to predict the consequences of its actions. Butz and colleagues applied REPRISE to a simulated environment in which the model learned to control a set of vehicles, and the vehicle it was driving could be changed without warning. Thus, at any given time, the model had to simultaneously infer which vehicle’s dynamics were in play and had to drive the vehicle. It could learn to identify vehicle changes (event boundaries) and to update its contextual neurons appropriately, improving driving performance.

Event segmentation and the formation of event memories

The application of REPRISE to detecting state changes illustrates the importance of modeling temporal structure in activity. Event segmentation theory (EST; Zacks et al. 2007) proposes that event segmentation occurs as a side-effect of constructing working event models that improve perceptual prediction. EST starts from a perceptual processing stream that takes a representation of the current state of the world (which may include perceptual information, language, and other sources) and produces predictions about what will happen a short time in the future. Such predictive processing is assumed to be an ongoing component of comprehension, which facilitates more effective and timely behavior. This predictive processing stream is modulated by a working event model that maintains a stable representation of “what is happening now.” The architecture retains its current working model as long as prediction error is low, and updates its working model when prediction error spikes. A simplified version of EST was implemented as a gated recurrent network by Reynolds and colleagues (2007). The model was trained using back-propagation on a series of inputs representing the position of an actor’s body over time while completing a sequence of goal-directed actions. The model was able to learn to predict the actor’s motions and to use spikes in prediction error to update its working models, improving prediction performance. EST proposes that the error-based updating mechanism could be implemented by phasic activity of the midbrain dopamine system, a system with broad projections throughout the cortex via direct connections to the prefrontal cortex and via the basal ganglia. This proposal has received support from neuroimaging (Zacks et al. 2011). Another neurophysiological hypothesis was that event model maintenance depended heavily on the lateral prefrontal cortex. This proposal has fared less well empirically; instead, components of the brain’s default network including the medial prefrontal and posterior cortex and parts of the lateral inferior parietal cortex are shaping up as better candidates (Stawarczyk et al. under review).

The event horizon model of Radvansky and colleagues incorporates EST as a front-end mechanism to account for a number of features in long-term memory for events (Radvansky 2012; Radvansky & Zacks 2014; see Table 3). The model provides an integrated descriptive account of many of the features of event memory described in “Event memory is structured by the same dimensions as perception” above. Segmentation during perception leads to units in event memory. The occurrence of similar event features in multiple events leads to better memory for those features, but to worse memory for identifying which particular features occurred in a specific event. Across events, causal relations are a major organizing feature that determines the likelihood of remembering a particular event and which other events that event will bring to mind. An important question left unresolved by EST and the event horizon model is how event structure evolves with forgetting, repeated retrieval, and subsequent experience. A recent study by Hohman and colleagues (2013) found that, when pressed, people retrieving information from autobiographical memory could retrieve additional sub-events from a remembered event and that doing so extended the remembered event’s boundaries. One possibility is that the temporal continuum that is sliced by perception can be resliced by the actions of memory retrieval. Another possibility is that such malleability of event boundaries in memory does not reflect the reslicing of a stored continuous experience, but rather a process in which the originally stored representations are discrete, and new discrete event representations are constructed through the act of retrieval.

Table 3:

Posits of the Event Horizon model (from Zacks & Radvansky, 2014).

1. Segmentation	Continuous ongoing activity is segmented into discrete events, and an event model is constructed for each event.
2. Working models	The event model corresponding to the event you are currently experiencing at any particular timescale has special status. It is actively maintained by recurrent neural activity.
3. The causal network	Long-term memory links event models by their causal relations.
4. Noncompetitive attribute retrieval	When elements of events are represented in multiple event models, access to those elements is facilitated.
5. Competitive event retrieval	When several event models are similar, accessing any specific event model is more difficult.

The segmentation of events is one aspect of event structure; other aspects include the order of events, their hierarchical organization, and the roles that actors and objects play in an activity. All of these, we saw, are influenced by knowledge about different event types. Elman and McRae (in press) have recently proposed a model of the acquisition of event knowledge that takes on all of these aspects of structure. The model is a recurrent connectionist network that codes for information about agents, actions, patients, instruments, locations, and recipients. The model learns associations amongst these elements within and across timepoints. Both kinds of associations are subject to variability in natural experience; for example, when changing a tire, one may loosen the lug nuts before or after jacking up the car; and might pull the car into a driveway or not. Trained on a corpus of event sequences from people’s descriptions, the model learns the most typical patterns and also information about the alternatives and the degree of variability. It is able to use this information to make predictions about what will happen next in a novel sequence—a key feature of comprehension.

Perhaps the broadest-scope model of event cognition to date is the structured event memory (SEM) model (Franklin et al. 2019). SEM uses a hybrid architecture in which the dynamics within events are represented using recurrent neural networks and relations across events are modeled as a partially observable discrete process that generates a sequence of persisting states which correspond to event types. The model learns a library of possible event dynamics, stored in the weights of a set of recurrent networks, and learns at each point in time to apply a previously-stored weight set or to create a new one if none of the existing weight sets fit. The model segments ongoing activity into events, forms online event representations, and can retrieve event information later. It provides a novel account of the learning of event schemas: When the model first creates a new neural network to represent an event that is not well fit by existing learned dynamics, this representation looks like an episodic memory. As it encounters more instances that activate the same dynamics, the representation grows more and more schematic. Like the Reynolds et al. (2007) model, SEM can segment ongoing activity based on a dynamic video (in this case, full motion video rather than a body-tracking recording). It is sensitive to the statistical structure of activity subunits in a way similar to human observers. It shows memory updating at event boundaries in an adaptation of the “walking through doorways” paradigm (Radvansky & Copeland 2006b), the long-term memory benefit of splitting encoding across two events (Pettijohn et al. 2016), and stronger memory cueing within an event than across events (Ezzyat & Davachi 2011).

The hierarchical organization of activity in time is a central feature to be accounted for. In the models described thus far, hierarchical organization is addressed only implicitly. For example, in EST, segmentation on different timescales can be achieved by varying the time constant of integration of the prediction error signal: Longer time constants do more smoothing on the error signal and produce less frequent event model updating in response to large, slow error spikes; shorter time constants produce more frequent event model updating in response to quicker error spikes that need not be as large. The hierarchical process memory theory (Hasson et al. 2015) explicitly addresses how different time scales of representation might be implemented in the brain. Rather than view memory as a specialized system or set of systems, the authors note that all of the brain’s dynamical systems have characteristic timescales. In the retina and the cochlea, neural activity tracks the current state of the world closely, and only retains the influence of previous states of the world for tens or hundreds of milliseconds. This fast temporal fading carries through the earliest stages of cortical processing, but as sensory information is processed through successive cortical stages temporal dependencies grow longer and longer. Complementary patterns of time-dependence are seen in motor control: Cortical systems that are close to synapsing on muscles show fast-fading temporal integration, whereas earlier stages of motor control show longer timescales. Brain areas that show the longest temporal dependencies tend to be multimodal and tend to represent both perceptual and action-related features of activities; these areas overlap with the default network, and include the angular gyrus and areas in the medial posterior and medial frontal cortex.

Summary

In sum, recent theoretical approaches to event cognition have focused on two topics: the multimodal integration of representations for perception and for action, and the processing of temporal structure in events. Most current theories integrate computational descriptions with neurophysiological descriptions in accounting for behavior with respect to events.

New methodological developments shaping event cognition

A key to recent progress in event cognition has been the deployment of new methodological tools and approaches. One such development is eye-tracking techniques for working with complex, dynamic stimuli. Whereas for decades eye-tracking has made important contributions to reading, scene perception, and attention, improvements in hardware and software have now made it tractable to present naturalistic movies or live interactive experiences, record eye movements, and calculate features including looking to target objects (Hayhoe & Ballard 2005) and gaze synchronization across viewers (Loschky et al. 2015). These improvements have also made it tractable to track the characteristics of saccades over time in conjunction with stimulus features (Eisenberg et al. 2018). Eye tracking is particularly promising for studies of young children and others who are unable to perform complex tasks under instruction.

The current resurgence of interest in event perception has corresponded with the rise of fMRI as a means to study brain activity. Many of the key contributions of this method have already been discussed: fMRI initially provided a noninvasive means to test hypotheses about the operations of segmentation during ongoing visual event and narrative event comprehension (Speer et al. 2007; Whitney et al. 2009; Zacks et al. 2001). More recently, multivariate pattern analysis, intersubject synchronization measures, and functional connectivity analysis have allowed researchers to test sophisticated new hypotheses (Baldassano et al. 2017, 2018; Hasson et al. 2008). These methods have provided evidence that the brain segments ongoing experience at a range of temporal scales, with later perceptual processing stages specializing in longer time-scales. They also have shown that phasic activity at the boundaries of events is predictive of online memory updating and of the organization of subsequent long term memory (Baldassano et al. 2017; Ben-Yakov & Dudai 2011; Ben-Yakov et al. 2013, 2014; DuBrow & Davachi 2016; Ezzyat & Davachi 2011; Hsieh et al. 2014). New developments in EEG theory and methods also have made key contributions. An important theoretical development is a reinterpretation of the P600 component. Previous accounts held that this component reflected syntactic reanalysis, but a recent proposal is that it reflects integration of new information into an event model (Brouwer et al. 2012). New EEG methods include multivariate pattern analysis such as those that have been applied to fMRI. For example, Sols and colleagues (2017) recently used pattern-based EEG to provide evidence that sequential structure within an event is recapitulated at event boundaries, and that this replay predicts subsequent memory (see also, e.g., Knoeferle et al. 2008). Over time, neurophysiological studies have increasingly embraced naturalistic materials and experimental designs (Maguire 2012).

Meanwhile, in computer vision, a new generation of neurally-inspired machine learning models have transformed object and action recognition from naturalistic stimuli (Herath et al. 2017; see also Google Cloud Vision API; FaceBook Detectron 2019). These methods provide tools to test behavioral and neural hypotheses about human event perception at scale. Crucial for these investigations will be the creation of large corpora of coded event stimuli (McNamara et al. 2017). Finally, event cognition is starting to move outside the laboratory to confront the richness of perception and memory “in the wild.” One ongoing development is the use of wearable sensors and lifelogging devices to measure behavioral structure on the scale not just of seconds or minutes but of hours to weeks (Nielson et al. 2015; Zhuang et al. 2012). These data suggest that events over the course of the day have segmental structure consistent with that attested by observers’ segmentation of relatively brief events in the laboratory.

Looking ahead in event perception

So, where do we stand and how do things look? Since Johansson and colleagues (1980) canvassed the study of event perception in the Annual Review there has been a dramatic growth and widening in the empirical phenomena encompassed by the field. First, event perception has become more cognitive, embracing the investigation of mental representation as a complement to characterizing the structure of the stimulus. Second, event perception and event memory have converged on a number of features of event structure. Third, new measures—especially neurophysiological ones—have been brought to bear. Finally, the sheer volume of the empirical database on event perception and memory has grown dramatically. These developments warrant using the broader term “event cognition” to describe the new state of the art.

These empirical developments have been accompanied by new generations of theories. These theories have attempted to account for the interaction of perception, action, and memory, for the segmentation of ongoing activity, and for the temporal organization of events on multiple scales. Though the models reviewed here vary in their scope, assumptions, and format, they share a concern with the structure of event representations in the mind and brain. That is, like the empirical research programs, the theoretical programs have become very cognitive.

What comes next? One possibility is that the models will scale up to something more like the full complexity of event comprehension, as the empirical work has already started to do. We can look forward to theoretical and computational models that can “experience” the same environments that our participants confront and mechanistically account for their comprehension and memory for those environments in terms of representational form and neurophysiological instantiation. Ideally, a model would take in the same stimuli as a participant and produce a set of representations that would enable it track activity in real time, and also have event memories and event knowledge. Such a model should not be a passive perceiver but should also be able to act on its environment, closing the loop between perception and action, using event memory and knowledge as guides.

Another encouraging development is that event cognition appears to be drawing together research from multiple areas of psychology with neurophysiology, linguistics, and computer science. As the field of cognitive science has grown over the last several decades, its transdisciplinary strength has dissipated. One may hope that event cognition will develop into a point of newfound cognitive science consilience.

Terms and Definitions

Event schema

Structured knowledge representation about an event type. The term script has a similar meaning but is sometimes reserved for events with social conventional structure

Event segmentation

The mechanisms by which the brain segments ongoing activity into meaningful events. Also sometimes used to refer to tasks intended to measure these mechanisms

Functional connectivity analysis

A collection of techniques for measuring the degree to which different brain regions’ activity (usually measured with fMRI) rise and fall together

Intersubject synchronization

The degree to which a behavioral or neurophysiological measure is correlated across participants

Launching effect

A configuration of two moving objects such that one is perceived to cause the other to start moving. Michotte (1946) found that the perception of launching depends precisely on spatial arrangement, speed, and timing of motion onset and offset

Lifelogging

Recording ongoing information about one’s life using wearable sensors and software

Medial temporal lobe

The hippocampus and surrounding structures in the temporal lobe, on the medial-ventral surface of the forebrain

Multivoxel pattern analysis

A collection of techniques for analyzing the spatiotemporal patterns of fMRI within a brain region that are associated with task parameters or stimuli; in distinction to techniques that analyze the overall level of fMRI signal in a region

N400:

A negative-going electroencephalographic (EEG) response that peaks near 400 ms after stimulus onset. The N400 is associated with processing unexpected stimuli or stimuli that are difficult to integrate semantically

P3, or P300

An early positive-going electroencephalographic (EEG) response associated with mechanisms of attention and target detection

P600

A late positive-going electroencephalographic (EEG) response associated with syntactic analysis or building a representation of a discourse

Piagetian

The school of thought in developmental psychology pioneered by the Swiss psychologist Jean Piaget

Prediction error

The difference between a system’s prediction about a variable or state and what it subsequently observes. Prediction errors play key roles in theories of learning of event perception

Prefrontal cortex

The anterior part of the frontal lobes

Recurrent neural networks

A family of neurally-inspired computational models consisting of a large number of simple computing units that influence each other by connections analogous to axons. Whereas in feed-forward networks information flow goes in only one direction, recurrent networks include information flow in the opposite direction