Event Boundaries in Memory and Cognition

Abstract

Research on event cognition is rapidly developing and is revealing fundamental aspects of human cognition. In this paper, we review recent and current work that is driving this field forward. We first outline the Event Horizon Model, which broadly describes the impact of event boundaries on cognition and memory. Then, we address recent work on event segmentation, the role of event cognition in working memory and long-term memory, including event model updating, and long term retention. Throughout we also consider how event cognition varies across individuals and groups of people and consider the neural mechanisms involved.

Events are at the center of human experience, and event cognition is the study of how people perceive, conceive, talk about, and remember them [1]. A current focus of interest is how cognitive systems form and update representations of events, namely event models. The Event Horizon Model [1, 2, 3] provides a framework for how such representations are created, structured, and remembered. It consists of five principles, illustrated in Figure 1: (1) people segment an ongoing stream of activity into a succession of event models; (2) only the current event model is in working memory; (3) the dominant dimension organizing relations between event models in long term memory is the causal connectivity among elements; (4) people better remember information stored across multiple events in noncompetitive attribute retrieval; and (5) people have retrieval interference for information stored across multiple events. Here, we will highlight recent developments in event cognition in light of this framework. (For systematic reviews, see [1, 4].)

Figure 1.

Illustration of the basic principles of the Event Horizon Model. In the figure, each small rectangle corresponds to an event model. The lettered bits within each model are entities or other elements of an event model, such as people, objects, etc. The large dashed rectangle corresponds to long-term memory. (1) Event segmentation (red) involves parsing the ongoing action into separate events. At an event boundary, the current model is stored in long-term memory. (2) Only the current model (green) is active in working memory at any given time. (However, because events are represented on multiple time-scales, the “current model” can correspond to a hierarchy of models ranging from short to longer durations.) (3) Causal connections (blue), indicated by dotted lines, organize the structure of long-term event memory. Note that some elements, even those that occur in multiple models, may not be part of a larger causal structure, but are only present in those events. Also, note that some events are antecedents for multiple other events, and that some events may be caused by multiple prior antecedents. (4) Elements that are part of a recall set (orange) are easier to manage and remember if they are segregated into multiple events. Finally, (5) retrieval interference occurs when elements are stored across multiple competing event models. Consider the example colored in purple: Suppose one observed Henry (H) and Archie (A) throwing darts at a dartboard (D). Retrieving the fact that Henry was involved would be relatively easy because there is only a single event model in long-term memory with him in it. In contrast, retrieving that Archie was involved would be more difficult due to retrieval interference, because Archie is present in four event models.

The first principle states that people parse action into events. An account of how this is done is given by Event Segmentation Theory [5, 6, 7] which proposes that people’s event comprehension systems form predictions about upcoming happenings based on the current event model. When important situation features change, such as new movements, spatial location, characters, objects, causes, and goals, then prediction error spikes. As a result, the current event model is updated and this is experienced as an event boundary [8, 9, 10, 11, 12, 13, 14]. This can be revealed by explicitly asking people to indicate the boundaries between events during comprehension [15, 16]; this has been done for videos of everyday activities [17, 18], dance movements [19] and visual and written narratives [20, 21]. Event boundary identification is partially determined by conceptually driven factors, such as comprehenders’ current interests and attention [20], action predictability [21], and default expectations [22].

For the second principle, when memory is probed, information in the current event model is more available than information from previous events. This can be revealed as faster and more accurate responses to probes for elements from either the current or a prior event [23]. Moreover, for the third principle, processing emphasis is given to causally connected elements, which are better processed and remembered. This can be revealed by faster processing times and better memory for more causally connected material [24].

For long-term memory, the fourth principle captures the idea that event structure can be a chunking mechanism to improve memory. Some experiments show that when features are distributed across multiple events they are remembered better than if they occur in a single event, controlling for exposure [25]. Finally, for the fifth principle, when long-term memory contains multiple overlapping event models that share features, but a task involves the retrieval of only one of them, retrieval interference occurs [26].

1. Event Segmentation as a Trigger of Attention

The updating of a current event model at an event boundary entails a transient increase in computation. Neuroimaging studies have found that there is a large, distributed response at event boundaries during ongoing comprehension, independent of whether viewers attend to segmentation [6, 27, 28, 29]. This activity can be tied to processing of changes in action features, including movement [30], and event dimensions such as location, characters, and objects [6, 31]. This has behavioral consequences. Recent studies have shown that, at event boundaries, people are less likely to mind wander [32] and are more likely to detect changes in objects [33, 34]. Moreover, placing information in separate events can benefit cognitive control [35]. This suggests that building explicit event representations into computer systems may facilitate adaptive control [36]. It should also be noted that expectations and attention can be guided linguistically, such that language may affect how we think about events [32, 33]. An important question for future research is whether such effects are transient, or persist to affect event cognition outside of conversational situations.

Observers tend to agree in where they place event boundaries [31]. Consistent with this, the neural dynamics in many brain areas show high levels of agreement across observers, rising and falling in similar places [37], and appear to be robust with brain changes in healthy aging [38]. That said, there are individual differences in the agreement with these group norms, both in terms of the placement of boundaries and in their hierarchical organization. These differences not only reflect on-line processing, but also predict memory quality [39, 40] and the ability to do everyday activities [41]. For example, older adults’ event boundaries show less agreement with each other than do younger adults’, and older adults with early-stage Alzheimer’s disease show even lower agreement [42, 43, 44, 45]. Deviations from normative segmentation also have been reported for people with traumatic brain injury [46], lesions to prefrontal cortex [47], post-traumatic stress symptoms [48], schizophrenia [49], and intellectual disability [50]. When interpreting these group differences, it is important that in some cases group differences in observed segmentation could arise from differences in cognitive capacities that are not strictly related to segmentation, such as the ability to interpret and retain task instructions, or to remain on task. Interestingly, Parkinson’s disease, which impairs some aspects of action performance and cognitive function, may spare event segmentation [51].

Of course, people are usually active participants in events, not simply observers. When event perception is coupled to action control, updating an event model could elicit a change in ongoing action. The Fluid Events Model [52, 53] incorporates this, accurately predicting when a person will switch from using one action during an activity to using a different one. As an everyday example, if a football coach was using a given strategy during a game, and that strategy became less effective, what is the probability that there will be a switch to a different strategy? Importantly, such action switches are guided more by factors related to a person’s prior experience than factors related to the event structure of the environment. Thus, overall, it is possible to predict how people parse events though their own behavior.

2. The Primacy of the Current Event Model in Working Memory

Experiences, whether real or fictional, are rarely about a single scene or event. The action moves from one event to another, and comprehension requires people to update their understanding. The Event Horizon Model proposes that information in the current event model is highly available, whereas information that is part of a prior model is less available. As one example, people are less able to recognize recently-seen objects in movies following an event boundary [54] and are slower to recognize recently-read words after a boundary [23, 55]. As another example, people also find it harder to resolve anaphors if the referent is in a prior event model as compared to the current one [56]. Recent work [57] has found that anaphor resolution is guided by the structure of the described events, and is only minimally influenced by extra-narrative event boundaries (such as pauses in reading).

A recent neuroimaging study revealed a neural concomitant of behavioral event updating: Many brain areas form stable patterns that transition abruptly, as would be expected for event models, and whose transitions correspond with event boundaries in a movie or story [58]. Areas showing such transitions overlapped with areas showing event boundary responses, and with areas associated with episodic memory retrieval including regions in the default mode network [59] and hippocampus. These dynamics may reflect representation of the underlying meaning of an event rather than the surface structure of the stimuli; brain dynamics when participants recalled movies tracked the dynamics of encoding. Other work has shown a transient increase in hippocampal activity at the ends of events, the magnitude of which corresponds to subsequent memory [60]. Thus, the hippocampus may be involved in transforming experience into a form that can be remembered over long delays.

Individual dimensions of a current model can be updated selectively. For example, suppose one is maintaining a current model of “checking out at the grocery store.” If the clerk is called away to answer a question on the phone and the manager steps in to finish the transaction. One possibility is that this triggers an event boundary and global updating of the current model. Alternatively, the current model would remain intact but the “clerk” attribute would be updated incrementally. Two recent reading comprehension studies [5, 61] assayed these two kinds of updating, and found evidence for both. Moreover, compared to younger adults, older adults showed evidence of reduced incremental updating but similar global updating.

3. The Organization of Long-Term Memory by Causal Connections

The Event Horizon Model proposes that event models processing is heavily influenced by causal connectivity. One recent study [62] used a narrative reading paradigm to test this. During comprehension, readers must access long-term memory to resolve anaphoric references in a text, and causal breaks could contribute to slowing in memory access. When readers experience a failure of their predictions they update their event models, resulting in slower reading times [e.g., 63]. It was hypothesized that foreshadowing an upcoming causal break would eliminate this updating-related slowing by eliminating the prediction error. However, it was hypothesized that information presented prior to the event break would still be less accessible because the relevant causally-related features would be less salient. This is what was found.

4. Events as a Means of Organizing and Chunking Information

Effects of event structure on long-term memory are stable over long periods. While memory for verbatim and propositional meaning units are lost quite rapidly, event model memory is much better retained [64]. The Event Horizon Model proposes a coupling between the structure of ongoing experience and long-term event memory: Events form the episodes in episodic memory. In text memory, after reading, narrative sentences from the same event prime each other more than sentences from adjacent events [65]. Moreover, memory for the order of encounters is better within events than across them, with judgments of temporal order across events often being quite poor [66, 67]. This suggests that within-event relations are represented in event models, but those between models are less well-defined, especially if there is no causal relationship. Finally, removing information from event boundaries impairs subsequent memory for those actions [68, 69].

According to the Event Horizon Model, elements that are learned across multiple events are more accessible in long-term memory, because the event structure provides a means of organizing and chunking. If online segmentation forms the units of subsequent episodic memory, the improving segmentation should improve memory. Initial results supporting this hypothesis were correlational, showing the people and groups with better segmentation had better memory [16, 40, 42, 43]. More recent studies have shown that intervening to support event segmentation results in better memory, as long as one month later [39, 70]. Memory improvement also occurs if a set of information is learned by having it distributed across multiple events [25]. For example, if people learn lists of words either within the same location, or spread across multiple location, such as rooms in a lab or windows on a computer screen, they remembered more when the set was divided between multiple events, rather than being part of a single event, complementing prior work on context and retroactive interference [71].

One exciting new research direction involves linking the temporal dynamics of event encoding and retrieval. One study examined interactions between the hippocampus and the default mode network while people watched the second half of a movie [72]. The first half was presented either immediately before or one day before. With the one-day delay, longer-term memory was needed to understand the second half, and this was associated with increased interactions between the hippocampus and the default mode network. Another study showed that the hippocampus represents the spatial and temporal distance between events experienced over weeks in real life [73]. The dynamics of these representations appears to determine subjective experience: When two events have similar hippocampal representations, they are experienced as having been closer in time [74].

5. Events and Retrieval Interference

The influence of event structure impeding memory is clearly seen with the location updating effect. This phenomena is found when people walk through doorways. In an initial experiment, people navigated a virtual environment and were probed for the identity of objects they were carrying [75]. After picking up an object and walking a fixed distance, memory was poorer if that walk included a doorway. This effect occurs both when the probes are pictures as well as labels, when people need to remember word pairs [76], with small computer screens, and real world movement [77]. As shown in Figure 3, this is not simply a context effect: If people leave a room and then return to it, memory is not improved. It is also not solely a consequence of event segmentation: moving from one room into a second room and then into a third room leads to worse memory than returning to the original room. In both cases, there are two event boundaries but in the former case there are three retrieval contexts, whereas in the latter case there are two. Thus, at least part of the memory decrement reflects retrieval interference. The location updating effect is also not a matter of sensory-perceptual processing: it occurs for imagined doorways [78], for doorways separated by a transparent “glass” wall [79], and occurs when there is time to allow participants to process the sensory and perceptual transients [80]. Finally, it should be noted that recent work [81] has shown that this pattern of performance was similar in younger and older adults suggesting that they manage their event models in long-term memory similarly.

Figure 3.

In this study, people navigated from room to room, picking up objects on tables and then setting them down on the next table. (A) presents a schematic of a subset of trials illustrating four conditions, and (B) is the error rate data. The location updating effect is the finding that memory is better when there is No Shift (red) from a location compared to when there is a Shift (green). In the Return condition (orange), after moving from one room to the next room, halfway through a person was told to return to the room from whence they came prior to having memory probed. Finally, in the Double Shift condition (blue), after moving from one room to the next, halfway through a person is told to move on to yet another room.

The effects of retrieval interference on long-term memory not only emerge quickly; they also are durable. This durability recently has been seen in a study using the differential fan effect [82]. The differential fan effect is the finding that when information can be integrated and organized into a common event model, such knowing that the potted palm, the pay phone, and the bulletin board are in the museum, then there are no competing event models, and there is no retrieval interference. In contrast, when information cannot be integrated into a common event model, such as knowing that the welcome mat is in the airport, the barber shop, and the movie theater (because these all refer to separate events), then during memory retrieval, these event models, which share the common element of the welcome mat, interfere with one another. The more event models there are, the greater the interference, and the slower the retrieval time. This increase in retrieval time with an increase in the number of associations is called a fan effect. Recent work [82] has shown that the differential fan effect persists over long periods of time, largely unchanged. This can be seen in Figure 4 in which a differential fan effect is present both immediately after learning, as well as two weeks later, with only minor changes.

Figure 4.

Results showing a differential fan effect for immediate testing and after a two week delay. Level of fan is the number of associations with a concept. The differential fan effect is the finding of retrieval interference (a fan effect) in one condition (common object), but not the other (common location). The pattern is largely unchanged after two weeks of retention, with only minor variations, although response time has slowed some during this time.

Conclusions

Event cognition is a rapidly emerging field of study that has implications for a wide range of cognitive phenomena. This includes the processing of actions as they are unfolding in the moment, the using of event knowledge to manage information in working memory, and the retrieval of knowledge from long-term memory. Finally, this field of study has matured to the point that it has provided some revealing insights about various individual and group differences.

Figure 2.

A. In two experiments [39], participants watched movies that were edited to cue information at event boundaries, event middles, or were left unedited. In both experiments, cueing event boundary information led to better recall memory after a brief delay. B. In 5 experiments [40], participants either watched movies with the intent to remember later (orange) or segmented them to mark meaningful events (green). Recognition memory for pictures from the movies was better after segmenting at all delays except for the immediate delay. Immediate memory may rely on perceptual details that are better encoded without a secondary task, but which are quickly lost.

Highlights.

• Event elements changes are event boundaries, thus creating new event models.

• Information beyond the current event model is less available.

• Long-term memory is influenced by the structure of event models.

• Event cognition research provides insights into individual differences.

• Neuroscience research continues to support theoretical advances in event cognition.