Precis of “Flicker: Your Brain on Movies”

Abstract

This article is a précis of the book Flicker: Your Brain on Movies (Zacks 2014). Flicker aims to introduce a broad readership to the psychology and neuroscience that underlies their experience in the movie theatre. The book’s topics include including: emotional experience, adaptation from texts to films, memory and propaganda, movie violence, film editing, and brain stimulation. Cutting across the specific topics are a few broad themes: the evolution of the brain and mind, the role of automatically evoked responses in film viewing, and the role of behavioral and neural plasticity in everyday experience.

Why this book?

Flicker: Your Brain on Movies aims to introduce a broad general readership to the psychology and neuroscience of film. This is an exciting time for those of us study how film works in the mind and the brain. The Society for the Cognitive Study of the Moving Image was founded in 1997. This journal was re-launched in 2006, with an explicit aim of bringing together scientists and humanists working on film (Konigsberg 2007). To describe the emerging scientific study of the experience of film art, Art Shimamura coined the term “psychocinematics” (Shimamura 2013b), and Uri Hasson coined the term “neurocinematics” (Hasson et al. 2008). And, of course, emergence of psychocinematics and neurocinematics would not have been possible without foundational work in cognitive film theory (e.g., Bordwell 1989; Anderson 1996; Grodal 2009).

It’s not just these institutions and coinages. We also have experienced a rapid maturation of new methods for studying the way film and brains interact. Two developments that I think have been particularly important are naturalistic eye-tracking (Henderson 2003) and event-related functional MRI (Dale & Buckner 1997). As a result of all this, there has been an efflorescence of research and theory, represented in this journal and elsewhere. The community doing this research is of course an interdisciplinary one, spanning film and media studies, cognitive science, psychology, and neuroscience. It has drawn heavily on a broader literature in cognitive neuroscience and in turn has given back to that literature. One manifestation of this surge of activity is the excellent recent volume edited by Shimamura (2013a).

I wrote Flicker because I thought it was time to bring all of this together in a single voice. More importantly, I wanted to “give away the science” (to steal a phrase from the late George Miller) to a broader audience. The book’s explicit goal is to explain to interested readers what we know about how movies work in the mind and the brain; however, it also has a covert mission to teach a bit of psychology and neuroscience. In short, Flicker aims to bring the research exemplified by Projections to a wider audience.

The book consists of two major parts of five chapters each, plus a prologue (trailer), an intermission, and an epilogue (stinger). In the following précis, I have briefly abstracted each of the chapters. To do so, I have had to trim out the majority of the empirical results and the stories that I use to bring those results to life. However, I hope you will forgive me for retaining the conversational tone of the book, though it is something of a departure for a scholarly journal.

Part I: From Up on a Screen to Inside Your Head

1). Your Brain Wasn’t Built For Movies

Sitting in a theatre, it is not at all uncommon to experience bodily responses—to feel as if you are preparing to move, perhaps even twitching a little in your seat. What is going on? Your eyes and ears are telling you that something exciting is happening in front of you and your brain is preparing you to react. Of course, you know it’s just a movie. But large parts of your brain don’t make that distinction in how they process the information. This makes sense—our brains evolved long before movies were invented, and our perceptual systems are honed to deal with the problems posed by the real world. Our brains didn’t evolve to watch movies—movies evolved to take advantage of the brains we have. Our tendency to want to respond physically to something that can’t touch us reflects its lack of ecological naturalness.

The ways our brains are built to respond to situations in the real world profoundly shape our response to film. Two broad principles are particularly relevant; I will call these the mirror rule and the success rule. The mirror rule says “Do what you see.” Everyday life is replete with examples of the mirror rule, though we often don’t notice them. Say you and I are sitting across a desk. If I cross my arms or legs you are much more likely to cross yours. Neither of us are likely to notice it, but it is easy to see in the lab. Once you start looking for them, the effects are dramatic and actually pretty funny (Chartrand et al. 2005). If you go to an action movie with a lot of kids, you can literally see the mirror rule operating as the children flinch, punch, and kick in their seats. More subtle but even more ineluctable, watch the faces of an audience sometime when you go to the movies. If a character on screen is grinning, people will tend to smile. If a character is angry their brows will knit. If someone starts crying, mouths will turn down and you may even see tears.

The success rule says, “Do what has worked.” It is obvious why we would be built to follow the success rule—doing what works is better than the alternative. The success rule is a pervasive factor in everyday action, and a significant component of how we learn new skills. Psychologists talk about the success rule in terms of stimuli (the things the world presents to us) and responses (the actions we take). Its formal name is “operant conditioning.” The success rule says that if you experience a stimulus, make a particular response to it, and things work out, then the next time you experience that stimulus you should be more prone to making the same response. You can feel this working powerfully in situations where you need to react fast. When a traffic light turns red and your foot presses the brake seemingly before you could think of it, that’s the success rue.

Is there a mirror rule brain center? A success rule brain center? It turns out the answer is no. Rather, these rules emerge from interactions across systems, and they are properties of may interacting systems. To understand how this plays out, we need to consider three principles of brain function. First, many brain areas are neural maps of our visual world. A neural map is a representation in the brain that is constructed such that if two locations are nearby in the world they will activate groups of neurons that are nearby in your brain. Second, brain areas specialize, dividing up the labor of processing. In the visual system, different visual areas focus on different visual features. Some areas are very sensitive to relative brightness, others to color, others to line orientation or shape. Third, brain systems are arranged hierarchically in successive levels. Each level feeds forward to the next level, but also feeds back to the previous level. This feedback is critical for the sophisticated processing our visual systems do. As we go up the perceptual hierarchies, the features to which brain areas respond become broader and more complex. Also, interactions across hierarchies increase; increasingly, cells at higher levels respond not just to vision or hearing or touch but also to complexes of these modalities.

The brain’s hierarchies are changing and adapting all the time. Neuroscientists call this “plasticity,” and it is critical for adapting behavior to a changing environment. Plasticity is responsible for much of the operation of the mirror rule and the success rule. For example, in The Red Violin, the orphan prodigy Kaspar Weiss is instructed to practice a piece over and over, slowly increasing the tempo, until he can play the piece perfectly. This sort of practice modifies the neural connections throughout Kaspar’s motor cortex, strengthening connections that work together and weakening connections that interfere with the performance. Over time, this huge number of small adjustments can have significant effects on the function—and even the physical layout—of our brains. One especially neat thing about violin-playing is that it is asymmetric—the left arm does fine work with the hand on the fingerboard, while the right hand does large movements with the upper arm. Does this produce hand-specific changes in the brain? Yes. A group of researchers in Germany found that touching the left hands of violinists leads to larger brain responses than does touching the left hands of non-violinists (Schwenkreis et al. 2007). It turns out that the hand area in primary motor cortex usually sits on a distinctive knob-shaped fold in the brain’s surface.

If we are constantly acting so as to mirror others’ actions and to perform previously successful actions, how do we ever do what we want to do? If we are sitting in a movie theater, why are we not waving our arms whenever the onscreen characters do, smiling or grimacing in response to their expressions, or reaching for the steering wheel whenever we see an obstacle in a car chase? The first answer is that we do, a little bit. As I have noted, subtle overt actions do leak out. But for the most part we are able to modulate neural connections depending on the task at hand, allowing us to contain inappropriate motor responses. This switching of rules depends in part on the prefrontal cortex, the front-most part of the brain. The prefrontal cortex is just about the slowest part of the brain to develop as we grow up. Thus, it makes sense that if we look around a movie theater, it is the kids that we see duck, grimacing, and squirming the most in response to the action on the screen. As entertainment becomes increasingly interactive, overt actions may become more appropriate and the kids may have a leg up on the adults!

2). The Movie in Your Head

What does it mean to understand a story? What is it about what happens in our heads that is the same whether we read a story on a page or watch it on a screen? When you get lost in a book or in a movie, what is going on? In this chapter I propose an account that answers these questions by building on the mirror rule and the success rule. The basic idea is this: To understand a story, we construct models of the story’s events. An event model is a representation in your head that corresponds systematically to the situation in the story. It is not a perfect copy—it simplifies a lot and distorts some things, but it is accurate enough that you can use it to run simulations that can tell you about parts of the situation you may have missed and to infer what might happen soon (Radvansky & Zacks 2014).

Think about reading J.K. Rowling’s The Goblet of Fire and compare that to watching Mike Newell’s film adaptation. The event model you would construct would not be exactly the same, but it would have a lot of the same elements: Harry, the dragon, and so forth. An event model is a representation of what the situation is about, not of how you learned about the situation. It ought to be more or less the same whether you witnessed the situation yourself, watched it in a movie, or read about it in a newspaper or a novel.

Differences between reading and watching arise, first, because filmmakers change elements of the situation when they adapt a story for the screen. But this is not really about the difference between the media; the same would hold if you compared two different written versions of a folk tale or two different movie adaptations of a book. More subtly, differences can arise because moving pictures and written language tend to convey different sorts of information. Movies almost always give you information about the color, shape, and size of the people and objects they depict. In a book, it is the author’s choice whether to describe those features or not. When a dragon appears in the Harry Potter movie, you get information about its color, shape, and size “for free,” whereas in the book, the author had to decide which aspects of the creature to describe. On the other hand, books almost always tell you about the categories to which objects and people belong (house, girl) and often tell you their names (Green Gables, Anne), whereas movies don’t have to.

Event models build on neural maps, as described in Chapter 1. They also are built on top of relationships acquired by the mirror rule and the success rule, which allow us to represent parts of the situation that are not explicit in a movie or text—to build inferences into our models that line up with the structure of the world about which we are inferring. Event models also have componential structure. Components are critical for running simulations. A good model has parts that correspond to the real situation in relevant ways, so that by manipulating the parts you can produce a result that corresponds to what would happen in the world. A good model also is somewhat abstract; it does not specify every perceptual detail. A representation that is perfect copy is no more or less useful than the thing it represents.

My colleagues and I think that whether you experience an event live, watch it on a screen, or read about it, you wind up with a model that includes information about how things look and feel and sound, where objects and people are located, and how you might act. When you understand a story, you simulate the events in the story in your event models. This is a pretty strong claim, but it has been supported by recent results in the psychology and neuroscience of reading and film viewing (Speer et al. 2009; Zwaan 2004). Most striking is that the brain’s response to specific sensory and motor features is quite similar across film viewing and reading.

One idea that has attracted a lot of recent interest is that we have special adaptations to process narratives—that we think in stories (Guber 2011). This claim is difficult to evaluate empirically, but it does make a neat suggestion: We may enjoy some kinds of storytelling because they push us to construct event models from sparse, incomplete information.

3). Tearjerkers and Sitcoms

Emotional responses to movies are striking in part because they come on with such rapidity and force. A good piece of slapstick can quickly induce laughter and euphoria, and a good crying scene can just as quickly have you weeping. A good melodrama can really make you sad—sometimes, apparently, sadder than you would feel if you encountered the same situation in real life. What I find especially perplexing about movie emotion is that it is in some sense so inappropriate. What’s the point in getting angry at a fictional villain? You can’t do anything to thwart his nefarious aims. First of all, the situation is fictional (assuming we’re talking about a fiction film). Second, nothing you do in your seat can change the outcome.

To explain the strength and speed of emotional responses to film, I appeal first to the mirror rule. Just as when you see someone wave you have a tendency to wave back, when you see someone smile you will find yourself smiling as well. See someone frown and your lips will tend to droop. These responses happen fast and automatically, and they are hard to suppress—if you try to frown just as someone else starts smiling it is quite difficult. In one experiment, researchers asked people to watch a series of pictures showing people with neutral expressions. Each was presented for 5 seconds. Unbeknownst to the participants, just before each of the slow faces was shown, another face was presented very briefly, for 25 to 35 milliseconds. This face was presented so briefly that none of the participants was even aware it was there; it was completely masked by the slow face that immediately followed. The briefly presented face sometimes displayed a neutral expression just like the subsequent face, but sometimes it was smiling or frowning. The researchers measured contraction of the viewers’ facial muscles using a technique called electromyography. When viewers were shown brief faces that were smiling, their muscles took on more of a smiling expression. When they were shown brief pictures of a face that was angry, they took on more of an angry expression. They were imitating the expressions of faces they couldn’t even detect!

A second piece of the explanation appeals to the success rule. In Marathon Man, when the audience sees shots of bad guys chasing Dustin Hoffman, the success rule predicts that audience members’ bodies will prepare to flee or fight because in the past that has been a successful response to seeing people engaged in chasing. They’ll hunch forward, their pulse will go up, and their pupils will dilate.

The mirror rule and the success rule can explain the smiling and grimacing, but not the subjective experience of the emotion. To explain that I appeal to two mechanisms: appraisal and the direct activation of emotion programs by bodily poses. Appraisal corresponds to the intuitive theory that many of us have of emotion: If we identify with a character and infer that something good has happened to that character we are happy; if we infer that something bad has happened we are sad. One important component of this is mind reading—reasoning about other people’s mental states. This mechanism is broadly supported, but it can’t account for the rapidity of some emotional reactions. The direct action mechanism was championed by William James, and it says that bodily changes follow directly on perception and that they are integral parts of the emotion program: “We feel sorry because we cry, angry because we strike, afraid because we tremble” (James 1890, pp.449–450). This sounds perhaps farfetched, but it too has received empirical support. For example, in studies in which people were instructed to pose their faces into smiles or frowns without realizing that was the pose they were adopting, they still felt happier or sadder (Strack et al. 1988; Larsen et al. 1992).

How are these mechanisms instantiated by the brain? There is no “emotion center” in the brain; instead, emotion depends on a broad network of cortical and subcortical structures. However, some components can be singled out as making unique contributions to the experience of emotion. These include the insula, which is particularly associated with the experience of disgust, the ventromedial prefrontal cortex, which may be important for translating a cognitive representation of a situation into an emotional one, and the amygdala, which is necessary for some aspects of experiencing and learning from fear.

The distributed processing of emotion is absolutely essential to the experience of film viewing. As Greg Smith (2003) argues, movies are machines to manipulate viewers’ emotional systems. When people talk about how the select movies to view, often they describe a film’s anticipated emotional effects. By titrating jolts of acute emotional response and integrating them with slower inferential processes, a well-made film can establish, maintain, and change a viewer’s emotional tone over its course.

4). How Movies Make Memories

In a film, a 121-year life can be played out in 139 minutes (Little Big Man) or an evening dinner can unfold in essentially real time (My Dinner With Andre). Movies can tell story elements out of order with flashbacks (Citizen Kane) or even backwards (Memento). Nonetheless, the structure of memory for stories told in movies or novels is strikingly similar to the structure of memory for real life. One reason may be that we have scripts that specify the typical structure of a life story, which we use for remembering our lives and those of others, including fictional characters (Bohn & Berntsen 2010).

Given the similarity between story memories and memories for real events, it is perhaps not surprising that people sometimes attribute to real life information that they actually learned from a movie. This confusion can be exaggerated in people with neurological injury or disease, but it happens to all of us and it forms a basis for innocent mistakes and for propaganda. One set of issues comes up in the context of biographical and historical movies. Filmmakers sometimes simplify and alter historical data to tell a better story. Laboratory studies show that viewers learn the inaccurate information along with the accurate information and have a hard time telling the two apart even if the inaccuracies are pointed out and they are instructed to be vigilant about them (Butler et al. 2009; Umanath et al. 2012). Another set of issues comes up when filmmakers deliberately set out to persuade viewers with in propaganda. In World War II, the social psychologist Carl Hovland studied the effects of Frank Capra’s The Battle of Britain, which was made to motivate U.S. troops entering the war in Europe (Hovland et al. 1949). Most importantly, Hovland found that although memory for facts decayed with time, some of the film’s effects on attitudes grew over time—a pattern he dubbed the “sleeper effect.” Sleeper effects can occur when memory for information becomes dissociated from memory for the source of the information. Capra’s recruits may have been skeptical of anything that came down from the army brass but after a while they just remembered his story, not its source.

Source memory is important for using your previous experiences to guide your behavior. It is supported by robust activation of modality-specific brain areas. It also depends on areas in the parietal lobes that are associated with the sensation that “I was there” and areas in the prefrontal cortex that appear to be important for maintaining retrieved bits of information while juggling them to evaluate their source. People with damage to these structures have particular trouble keeping sources separate—but it’s hard for all of us (Johnson 2006). What can we do to protect ourselves? Fact-checking when the bogus information crops up helps, and specific warnings about what might be wrong help (Marsh & Fazio 2006).

The similarities between memory for movies and memory for real life suggest that Wordsworth may have gotten in backwards when he wrote that appreciating fiction requires a “willing suspension of disbelief.” Perhaps instead what is required is a suspension of belief (Prentice et al. 1997). By default, we process movies with the same mechanisms that we bring to bear to process real life. When we watch a film, we undergo many of the same brain responses as we would were we to experience the film’s situation in real life. To the extent that we are transported into the movie, deeply engaged in comprehension, we experience these responses even more. Something extra is required to suspend those mechanisms. That kind of top-down control—suspending belief—likely depends on the prefrontal cortex.

5). The Dark Side

From the notion that movies and real life are remembered using the same mechanisms, it’s just a small step to wondering how watching movies changes our behavior in real life. For about four decades, scientists and the public have worried about one particular effect: Does watching violent movies (and TV) make us more violent?

There have been a lot of studies. Most have focused on kids because children’s behavior is likely to be more malleable than that of adults. The studies fall into three main categories, each with their own limitations. Longitudinal studies look at individual differences in media consumption and ask whether they predict subsequent individual differences in violent or aggressive behavior. With these, the main challenge is that people who differ in the media consumption may differ on other dimensions as well, and it can be difficult to control for all of these potential confounds. Quasi-experimental studies were done mostly early in the game, when TV was being introduced. The strategy here is to compare two communities, one of which is undergoing a change in its media consumption (say, due to the introduction of TV). As with longitudinal designs, there may be other differences between the two communities that are really driving any effects. Finally, experimental studies are ones in which people are randomly assigned to one of several treatment groups—for example, watching violent movies or nonviolent ones. Experimental designs are the gold standard for establishing causal relationships. They face two challenges. First, experimentally manipulating exposure to violence on a chronic basis over a long time is impractical and unethical. Second, measuring real violence as an outcome variable is unethical. As a result, most experimental studies are briefer and milder than the situations to which we would like to generalize.

The weaknesses of these three methods are complementary; if we see the same results coming from each of them we should be pretty convinced. And in fact that is what we do see: No matter the method, more often than not a study assessing exposure to violent media and violent or aggressive behavior finds a link between the two (Bushman & Anderson 2007; but see also Ferguson & Kilburn 2009). It’s not 100% of the time, but we would not expect it to be. There are important moderators: Movie violence affects different people differently, and how it is portrayed modulates its effect.

The research has identified three mechanisms by which watching violent movies can lead to violent behavior. First is observational learning: Humans can learn to do many things by watching, and some of them are bad. Second is habit. For example, if one watches many movies in which angry words are followed by violence, this will produce a tendency to respond to angry words by readying yourself for a violent situation. This need not be intentional or conscious. The third mechanism is desensitization. Violence is, for most people, intrinsically aversive. But violence may work like hot peppers: With repeated exposure consumers’ tolerance increases, and they may even develop an appetite for it.

Some of the toughest questions about media violence concern public perception and policy. If the effects of violent media on behavior are pretty clear, why are so many in the general public unaware of this? Part of the reason is that, as with any scientific question, there are dissenters (e.g., Ferguson 2009), and news outlets do their best to portray both sides of the issue. Doing so while at the same time explaining that a preponderance of the evidence falls on one side is hard, and it gets harder when you also have to explain that the effects vary in size and are moderated by individual and situational variables. Another part of the reason is that many of us enjoy film art that includes violence, and we resist data that conflict with our preferences. And finally the fact that large commercial interests have a stake in this question probably has something to do with it.

So what to do? Current movie ratings systems are ineffective in helping parents make decisions about their children’s exposure to media violence, but more specific and comprehensive ratings systems may prove more useful. One organization that is making great strides in this area is Commonsense Media (commonsensemedia.org).

Part II: The Tricks That Make Movies Work

6). Action!

A movie is a succession of still images. We call them movies, but why do they move? A compelling—but incorrect—explanation can be traced back to Roget’s analysis of a curvature illusion in motion perception (Roget 1825). He proposed that each image exposed to the eyes leaves a trace that lingers for a moment, and if the successive images are fast enough they fuse. This came to be known as persistence of vision. Persistence of vision happens; it turns out to be a side-effect of the time lags in the retina, optic nerve, and the lateral geniculate nucleus, which relays visual information to the brain. The problem with persistence of vision is that, although it is a good explanation for Roget’s curvature illusion, it doesn’t explain anything about how movies move. Persistence of vision can produce blurring, but not motion.

Another intuitive idea is that our visual system recognizes objects or shapes in each frame, compares their locations, and then infers the movement from that comparison. The problem with this idea is that it is wrong, and there is solid evidence against it. On the one hand, people can have lesions to a brain region called “area MT,” leading to a selective deficit in motion processing with no deficit in shape perception (Heywood & Zihl 1999). On the other hand, there are a number situations in which we can perceive motion without comparing shapes. (Google “waterfall illusion” for one of the more compelling examples.)

The real explanation for why movies move is that we have detectors tuned to the spatiotemporal pattern of retinal stimulation that is associated with motion. We can think of motion across the retina as a three dimensional phenomenon with two spatial dimensions (vertical and horizontal position) plus time. We have detectors that are tuned for particular “shapes” in this three-dimensional manifold. For example, one detector might respond to a patch moving slowly from left to right in the upper-middle part of your visual field, whereas another detector might respond to a patch moving quickly from top to bottom in the lower-right part of your visual field. When successive frames stimulate these detectors the way real motion would, we have the illusion that the shapes on the screen are moving. Early in our visual system, these detectors respond to motion of simple features, particularly moving edges: adjacent light and dark patches, with the boundary between light and dark changing over time. Later stages of visual processing, particularly area MT, have detectors tuned to more complex patterns such as a moving texture boundary.

Understanding how motion processing works explains why, when the frame rate is too low, motion can look jerky. Jerkiness happens because neurons in early visual areas respond to smaller regions of visual space, whereas neurons in area MT integrate over larger regions. If you have an object moving fast and the frame rate is too low, the signal “jumps over” the early visual neurons but still stimulates cells in area MT. The conflict between the signals—early visual cortex says “no motion,” but MT say “motion”—leads to a feeling that the motion is weird.

Right now, the neurobiology of motion perception is colliding head-on with the technology of film projection, as filmmakers explore shooting and projecting at higher frame rates. Can viewers even tell the difference between 24 and 48 frames per second? It turns out the answer is “yes”—in fact viewers can discriminate frame rates up to 200 frame per second. That doesn’t mean they always prefer higher frame rates; Peter Jackson’s use of high frame rate in the Part I of his Hobbit trilogy was panned by critics and fans.

7). Cut!

Most cuts are continuity edits, which join shots that are meant to depict an ongoing scene that is continuous in time and space. The amazing thing about continuity edits is how invisible they are. A cut is a massive visual change—why do we sometimes miss it?

I account for the invisibility of continuity edits by considering how processing is distributed across components of the visual system. A key distinction is between the dorsal and ventral visual streams. These two processing streams diverge anatomically from the primary visual cortex, with the ventral stream projecting to regions on the bottom of the cortex (hence its name), and the dorsal stream projecting to regions on the lateral and superior surfaces. The ventral stream, on the one hand, is specialized for visual thinking—for recognizing objects, for categorizing configurations of objects, for problem-solving. Neurons in parts of the ventral stream respond selectively to particular kinds of things—say, cats versus dogs, or even “Fido” versus “Spot.” The dorsal stream, on the other hand, is specialized for controlling fast and simple actions. It is built to guide your body right now, to allow you to respond quickly to things in your immediate environment. Both the dorsal and ventral stream are made up of a number of brain areas, each of which specialized to extract particular kinds of signals from its inputs. Cuts work because they hide in the gaps between the expertise of each of the components in these networks: If there is no region that is sensitive to a particular kind of disruption, the disruption goes unnoticed.

Some theorists have proposed that cuts literally hide—that good edits are placed when people tend to blink or move their eyes and thus are insensitive to the visual information (Murch 2001). The data don’t bear this out: Eyetracking studies show that blinks and eye movements do tend to happen around cuts, but they usually happen after the cut, so they are probably a reaction to it rather than a way to hide it (Smith & Henderson 2008).

One way to hide takes advantage of a natural gaze pattern: Often, something in the scene creates a visual question and we turn our gaze to get the answer. If the film cuts to what our visual system would have tended to look at given the opportunity, it may go unnoticed. Another way to fit is by masking. Masking happens when a big sensory change reduces your sensitivity for a nearby sensory change. If a cut is followed by a lot of motion (as in a match-action cut), the motion renders the cut less noticeable. These principles give a reasonable account of the heuristics used by filmmakers.

Of course, some cuts don’t work. In a jump cut, an object appears to “jump” suddenly from one place to another. One kind of jump cut happens when the camera angle changes too little across a cut. These happen because objects in the scene are close enough on screen before and after the cut to stimulate motion detectors in area MT—but not close enough to stimulate motion detectors earlier in the visual system. Filmmakers have evolved a good feel for when this discrepancy occurs, though of course they don’t talk about it in these terms. The “30 degree rule” says never cut within a scene without moving the camera more than 30 degrees. That distance is usually enough to prevent stimulating motion detectors in area MT.

Viewed one way, it is surprising that cuts work at all: In a cut, a big piece of our visual field is instantaneously replaced with something completely different—a phenomenon that is utterly new in the evolutionary history of our species. Do we have to learn how to handle such changes? One recent series of experiments suggests that the answer is mostly “no” but with a dash of “yes” (Ildirar & Schwan 2014; Schwan & Ildirar 2010). Sermin Ildirar and Stephan Schwan created a set of simple films and showed them to people living high in the mountains of Turkey, who had grown up without any television. Their subjects had no trouble describing what was happening on either side of a cut, and in some cases they accurately described the relationship between the shots. But in some cases they had trouble understanding how a series of shots formed a coherent narrative. Familiar actions helped, suggesting that understanding a story can help viewers learn how to process the editing.

Another surprising thing about cuts is how rapidly we can process them. Editing rates have increased dramatically over time; a wide sample of commercial films suggests that the mean shot length decreased from more than 11 seconds in the 1950s to less than 6 seconds in the 1990s (Salt 2009). At the same time, the rhythm of editing has changed, adapting the distribution of shot lengths to distributions that characterize fluctuations in human attention (Cutting et al. 2010). One might worry that this is dramatically reshaping our brains—but in fact the evidence suggests any effects of watching fast editing would likely be subtle.

8). Bottlenecks, Spotlights, and Chunks

Movie viewing—and human perception more broadly—presents us with a paradox: Despite the huge capacity of our sense organs, the processing load that our central nervous system can handle is really quite modest. When you look at your TV or your computer, does it feel as though you are processing every little pixel on the screen? For most of us most of the time, the answer is “no.” Instead, we seem to inhabit a world consisting of a modest number of objects and events. The question then is: How is the vast data stream that impinges on our sense organs transformed to something that is tractable for our central nervous system and that corresponds to our subjective experience of the world? This question has a two-part answer: (1) a lot of what happens in our sensory fields is filtered out before it even gets to the central processor, forming a perceptual “bottleneck;” and (2) what does get through is radically reshaped, so that what our brain represents is not billions of pixels but something much more modest and much more useful.

One helpful way to think about how the filtering happens is in terms of a spotlight: A small region in the spotlight is processed intensely; the rest of the frame is not. Filmmakers intuitively know this: They spend a lot of effort on the details of the one or two parts of the frame that viewers are likely to attend to and let the rest slide. Humans have finely tuned neural mechanisms to control their spotlights (Wolfe 2003). One set of mechanisms depends on salience, which is the degree to which a visual element is different from its neighbors. Another set of mechanisms depends on the viewers’ goals at the time—if you are searching for your keys, then your spotlight is drawn to keylike features. These mechanisms depend in part on specialized structures in two lobes in the back of the brain: the parietal and temporal lobes. They can be disrupted by disease or injury. One such case is a neurological disorder known as neglect, which often occurs after a stroke affecting the back of the brain just in front of the visual cortex, particularly areas on the right side (Robertson 2004). Patients with neglect due to right-hemisphere brain damage might eat only the food on the right side of their plates, leaving the left untouched. If asked to copy a picture, they might draw a rich image on the right but only a few squiggles on the left.

Filmmakers have known from the beginning how important it is to direct the viewer’s attention and how powerful a medium film is for doing so. In recent years, that craft knowledge has been supplemented by data from the laboratory. When people look at raw footage of other people going about their lives, they tend to look at the same things, but not necessarily in the same order, and there is a fair amount of variability. In a well-made commercial film sequence, however, pretty much everybody is looking in the same one or two places at any given time (Hasson et al. 2008; Smith 2012; Smith & Mital 2013).

Selective attention accounts for part of the solution to the problem of our limited central processing capacity. A second critical component is chunking. In the spatial domain, we segment a scene into figure and ground, and we segment the figure into parts. In the temporal domain, we segment a continuous stream of activity into events. Chunking necessitates that we abstract away a lot of information. This abstraction can produce distortions in our memory. Spatial memory is one particularly good case study, because a lot is known about the neural mechanisms underlying different aspects of spatial representation. Recently, my laboratory has been focused on the case study of chunking in time. We have found that people chunk movies into meaningful events as an ongoing concomitant of normal viewing. Across members of an audience, we find good agreement about the boundaries between events. When we record brain activity during event viewing with functional MRI, we find that a collection of regions in the posterior cortex and a region in the right prefrontal cortex increase transient activity at the event boundaries. Both spatial chunking and temporal chunking are hierarchically structure, with small chunks clustering into bigger chunks (Kurby & Zacks 2008).

The relationship of cuts to event boundaries is of particular interest. It turns out that cuts by themselves are not perceived as event boundaries (Magliano & Zacks 2011; Zacks et al. 2010). So what happens in the brain when a cut presents us with a massive visual discontinuity but we perceive the event as continuing smoothly? The account given in Chapter 7 would suggest that our processing systems have strategies to bridge over discontinuities in the signal during natural viewing, and these strategies might come into play here. Functional MRI data provided support for this idea: Early visual areas were massively activated at all edits, but a smaller collection of higher-level visual areas was selectively activated for continuity edits; these activations might reflect the bridging process (Magliano & Zacks 2011). This result suggests that filmmakers give viewers cues as to how to process edits, telegraphing which are the big scene breaks so that our visual systems don’t try to look for correspondences across those discontinuities.

9). Sleight of Hand

One kind of visual discontinuity that drives filmmakers and fans crazy is a continuity error. For example, in The King’s Speech, Derek Jacobi, playing the Archbishop of Canterbury, is first wearing one cross and then wearing a quite different cross in the next shot. It’s not unusual for film-fan web sites to document upwards of 200 such continuity errors in a feature film, despite the presence of dedicated staff on the set to try to keep them from happening. But most go unnoticed. Why?

Studies of change blindness confirm that viewers are insensitive to even large changes in movies (Levin & Simons 1997), and moreover that they miss big changes in real life. In my favorite example (Levin et al. 2002), you show up to participate in an experiment. The paperwork tells you it’s a study of visual memory. You sign the form and return it. The experimenter ducks behind the counter, saying “Let me just get you these forms.” A different experimenter pops up with the forms. The forms ask you to describe everything you saw since you exited the elevator, and to mention whether you noticed anything funny. Of the 20 students who participated, only 5 indicated that they noticed the switched experimenter.

Change blindness is counterintuitive because it feels as though we have a high-fidelity visual representation in memory but we actually have only a filtered, chunked, and abstracted representation, as described in the previous chapter. We have this wrong intuition in part because most of the time when something changes in our environment (think of a balloon popping or a person walking into a room) there is a large transient signal—a color change or moving contour. We detect the transient signal, not the change itself. The kinds of changes we do detect are changes to the gist or meaning of a scene.

My colleagues and I have studied the effects of chunking on memory for objects in movies (Swallow et al. 2009; Swallow et al. 2011). We hypothesized that when viewers experience a boundary between chunks, they updated their memory for the objects in the scene. To test this, we first asked one group of viewers to tell us explicitly where they boundaries between meaningful events (chunks) while watching excerpts from commercial films. We then asked a new group of viewers watch the excerpts and tested their visual memory for objects that appeared on screen just 5 seconds before each test. Sometimes, 5 seconds ago was still part of the same chunk; sometimes not. The results were dramatic: Accessing an object from the previous event—even though it was only five seconds ago—was much more difficult than accessing an object from the current event. For certain kinds of objects they could do it; but for others it’s basically impossible. When they were successful recognizing objects from previous events they tended to activate brain regions that we usually think of as being associated with long term memory, particularly a region called the hippocampus. This pattern suggests that chunking forms a major boundary between the conscious present and everything else in your experience of a film (or of real life).

These findings might give the impression that visual memory is only sketchy and weak. That’s really not true: We are not blind to all sorts of changes (Hollingworth et al. 2001), and once something does make it into long term memory we actually have pretty amazing ability to recognize it weeks later (Standing 1973).

What does all this say for filmmakers? On the one hand, it suggests not sweating the details too much: Any given viewer is unlikely to notice any particular continuity error. On the other hand, if you consider the large number of potential continuity errors in a feature film and you think about the large number of viewers a commercial film hopes to attract, the cumulative number of detected errors adds.

10). Virtual Futures

A recurring movie plot point is that we will someday be able to “jack in” to the central nervous system and directly edit memories. A few examples: virtual vacations in Total Recall, memory erasure in Eternal Sunshine of the Spotless Mind, a whole illusory world in The Matrix, direct-connect video gaming in eXistenZ, and neurophysiological corporate espionage in Inception. At some point in the future, will we be plugging in to download the latest release?

Well, no—but I wouldn’t rule out recreational brain stimulation as part of the movie experience at some point. Filmmakers and film promoters have always looked for ways to augment the movie experience. From the beginning there were strategies for adding color and sound to movies; these eventually became standard. Stereoscopic presentation (“3D”) has had several incarnations, the earliest of which goes back to the 1920s. Filmmakers have experimented with smells, vibrations, puffs of wind, tilting chairs and vibrating seat cushions in order to add something extra.

There is no doubt that direct stimulation of the brain can give rise to sensations that are to varying degrees controllable. In 1934, the neurosurgeon Wilder Penfield stimulated a patient’s brain as part of a surgery to treat her epilepsy, and she reported things like: “I can see the most wonderful lights.” A little later, “Did you pour cold water on my hand?” Then, “I can smell burnt toast” (Penfield & Rasmussen 1950). More recently, researchers have developed noninvasive brain stimulation techniques. The technique that is closest to being plausible as an entertainment vehicle is transcranial magnetic stimulation, or TMS. TMS works by placing a strong electromagnet near the head and rapidly changing the strength of the magnet. This induces an electric current adjacent to the magnet, within the brain. No current technique can give you anything like the control envisioned in the movies I mentioned, but TMS can produce crude visual, auditory, or touch sensations that might augment a big explosion or a car chase scene. Less frivolously, brain stimulation techniques are proving quite valuable in treating some psychiatric and neurological disorders, and may prove effective in augmenting some kinds of learning.

Looking at current brain stimulation techniques in the context of historical developments such as stereoscopic projection and color suggests an important lesson: We’re not tweaking our brains only if we get zapped by a transcranial magnetic stimulator—we’re tweaking our brains every time we sit down in a theatre. The issue is just whether the tweak gets in through our eyes and ears or by other means.

Giving away neurocinematics and psychocinematics to a wider audience

To sum up: Flicker makes a broad argument that the human fascination for moving pictures can be explained in terms of how the brain and mind evolved to solve the problems posed by the real world. Many relevant mechanisms are fast and automatic, and as a result are not very accessible to introspection. Two broad principles that cut across these automatic mechanisms are the mirror rule (do what you perceive) and the success rule (do what has worked in the past). These principles are implemented in brain systems that are distributed and hierarchical. Psychological and neural mechanisms and principles can help us understand how movies evoke bodily responses, provoke emotion, and make memories. They also give insight into the tricks that make movies work: the illusion of motion, the functions of editing, and the limitations in our ability to tracking continuity across shots.

Finally, let me emphasize the Flicker is primarily written for a general reader. I hope it may be useful for Projections readers in teaching and in introducing our colleagues to psychocinematics and neurocinematics. But even more I hope it will be vehicle for sharing the work of the Projections community with a wider audience.

Biography

Jeffrey M. Zacks is Professor of Psychology and Radiology at Washington University in Saint Louis, where he chairs the Cognitive, Computational, and Systems Neuroscience Pathway, serves as Associate Chair of the Department of Psychology, and directs the Dynamic Cognition Laboratory. He has served as Associate Editor of Cognition and as Chair of the governing board of the Psychonomic Society. He is the recipient of scientific awards from the American Psychological Association and the American Psychological Foundation, and is a fellow of the Association for Psychological Science and the Midwest Psychological Association.