Leveraging the power of media to drive cognition: a media-informed approach to naturalistic neuroscience

Abstract

So-called ‘naturalistic’ stimuli have risen in popularity in cognitive, social and affective neuroscience over the last 15 years. However, a critical property of these stimuli is frequently overlooked: Media—like film, television, books and podcasts—are ‘fundamentally not natural’. They are deliberately crafted products meant to elicit particular human thought, emotion and behavior. Here, we argue for a more informed approach to adopting media stimuli in experimental paradigms. We discuss the pitfalls of combining stimuli that are designed for research with those that are designed for other purposes (e.g. entertainment) under the umbrella term of ‘naturalistic’ and present strategies to improve rigor in the stimulus selection process. We assert that experiencing media should be considered a task akin to any other experimental task(s) and explain how this shift in perspective will compel more nuanced and generalizable research using these stimuli. Throughout, we offer theoretical and practical knowledge from multidisciplinary media research to raise the standard for the treatment of media stimuli in neuroscience research.

Psychology and neuroscience have four main goals for the study of human behavior: describe, explain, predict and control. Indeed, the journals of these disciplines are filled with nuanced descriptions and explanations of human cognition, emotion and behavior. However, if someone sought an expert to make them laugh or cry, would it be better to turn to a psychological scientist or, say, a screenwriter or movie producer? When it comes to predicting and controlling our thoughts and emotions, entertainment media like film, television, books and music have long outpaced the social sciences.

Recognizing the power of media, neuroscientists have increasingly adopted them as stimuli in human neuroimaging research where they are commonly referred to as ‘naturalistic tasks’.¹ The adoption of media stimuli, especially audio and audiovisual narratives, for neuroscience has led to a wealth of new insights into phenomena such as memory (e.g. temporal receptive windows, Hasson et al., 2008b; shared signatures for real-world memory encoding, Heusser et al., 2021; Chen et al., 2017), language (e.g. narrative-level comprehension, Yeshurun et al., 2017), emotion (e.g. emotion classification from functional magnetic resonance imaging (fMRI) data, Kragel and LaBar, 2015) and social cognition (e.g. the role of the superior temporal sulcus in processing dynamic social information, Lahnakoski et al., 2012; Lee Masson and Isik, 2021). It has also methodologically benefitted neuroimaging research by providing more attentionally engaging experiences for participants, which is especially useful for studying children and developing brains (Vanderwal et al., 2019) and other special populations (Naci et al., 2017; Laforge et al., 2020). Many researchers now accept the utility and benefit of naturalistic stimuli (Hasson et al., 2008; Sonkusare et al., 2019; Vanderwal et al., 2019). However, to date, the process of choosing a specific piece of media for ‘naturalistic’ neuroscience research appears largely ad hoc: researchers choose movies or stories that they are familiar with and that they intuitively believe will evoke certain processes of interest but typically do not consider formal features as defined by filmmakers, writers and other creators. While decades of scholarship in fields such as communication (focused on message processing and effects), communications (focused on communication technologies) and media studies have established theoretical and empirical links between these features and the audience experience, most experimental psychologists and neuroscientists are not familiar with this literature and are therefore unable to leverage lessons from it in choosing stimuli and designing experiments.

Here, we aim to fill that gap by providing a practical guide for human neuroscientists outlining what to consider from a media perspective when selecting a preexisting media stimulus for a given experimental goal. We argue that rather than being ‘naturalistic’ in the sense of spontaneous and unstructured, works of media offer complex and hierarchically nested information that is deliberately crafted to elicit a particular audience experience. Although the act of consuming media without additional constraints is arguably naturalistic, the media content itself is often less so. With this lens, although watching a film in an magnetic resonance imaging scanner seems more externally valid than many traditional cognitive tasks, in some ways, an action film that is designed to entertain and thrill an audience has more in common with a Gabor patch than a conversation with a friend. Therefore, if a neuroscientist wishes to study a particular cognitive process using a movie stimulus, having a basic knowledge of formal cinematic features, editing techniques and genre conventions will allow them to more effectively realize the many advantages that media offer because they are crafted for a specific purpose.

Moving beyond the ‘naturalistic’ label

According to Sonkusare et al. (2019), naturalistic paradigms ‘provide a reasonable approximation to how we encounter stimuli in everyday life’ (pp. 699) and can be ‘heuristically defined as those that employ the rich, multimodal dynamic stimuli that represent our daily lived experience, such as film clips, TV advertisements, news items, and spoken narratives, or that embody relatively unconstrained interactions with other agents, gaming environments, or virtual realities’ (pp. 700). Given the phrase ‘everyday life’ in this definition, there are strong parallels between the term ‘naturalistic’ and the ongoing discussion over the definition and role of ecological validity in the psychological and social sciences broadly (Araujo et al., 2007; Shamay-Tsoory and Mendelsohn, 2019; Holleman et al., 2020, 2021; Osborne-Crowley, 2020; Kihlstrom, 2021). Although we draw on ideas from this discussion, we do not argue for any specific definition of what it means for an experimental paradigm to be naturalistic or have ecological validity. Instead, we seek to expound one category of stimuli, namely media, which often falls under the umbrella term of ‘naturalistic’. We argue that stimuli such as film clips and TV advertisements are not representative of our daily lived experience in the same way as, say, conversations with colleagues or people watching in a park. We acknowledge that film and TV are representations of life but with the added distinction that they are heightened representations designed to serve some underlying function and achieve a specific goal.

Here, we differentiate between stimuli that were designed for research purposes and stimuli that were originally crafted for a specific function outside of a research context. The first category includes—to adopt terms commonly used in the literature—‘traditional’ or ‘typical’ experimental stimuli, such as Gabor patches, auditory tones, isolated letters or words, colored or textured shapes and static images. The second category includes audio, visual and/or written works that, importantly, were designed not for research purposes but rather to inform, persuade or entertain an audience; these can include film, television, advertisements, books and music. These stimuli, which we refer to as media stimuli throughout this paper, were not produced for scientific research or experimentation. Notably, the first category can include stimuli like movies and written narratives, but the key is that they were created with a research function in mind, and the additional variables inherent to these complex stimuli have been, at least in part, considered by the researchers. This offers the researchers an expanded degree of control. In contrast, when an experiment relies on stimuli created for nonresearch purposes and the original function of the adopted stimulus goes unacknowledged, this can introduce a host of unexpected and therefore unchecked variables. However, in these instances, researchers can exert a higher degree of control by understanding how and why a stimulus was originally created and describing and justifying how and why it is used in an experimental context, which further benefits the interpretation of results and how they might generalize.

The term ‘naturalistic’ has become a catch-all classification for experimental stimuli that are more similar in any regard to the lived human experience regardless of the original purpose for the stimuli. Evidence for this is present in the current literature. To ascertain a snapshot of how media stimuli are treated in naturalistic neuroimaging, we conducted a content analysis of Methods sections of all articles included in the recent NeuroImage special issue, ‘Naturalistic Imaging: The use of ecologically valid conditions to study brain function’, edited by Drs. Tamara Vanderwal, Emily Finn, Enrico Glerean and Uri Hasson (articles published between October 2019 and January 2021; Finn et al., 2022). Of the 50 articles associated with this special issue, nine were review or opinion pieces, and six used stimuli or paradigms specifically designed for research purposes. Importantly, the remaining 35 used stimuli that were not originally created for research and included text, images, music, animated shorts, films, video clips, and audio recordings from a variety of sources. Altogether, this illustrates the prevalent practice of adopting media stimuli for research and how these stimuli are not distinguished from, say, live interactions or bespoke experimental narratives. We highlight three issues with using the term ‘naturalistic’ in this way.

Naturalness is not a single dimension

First, the reference points for what makes something natural are unclear and highly relative to common practices within particular subfields (Holleman et al., 2020). For example, a cognitive scientist might justify a study’s importance by claiming that their use of a naturalistic stimulus, like a movie of someone smiling, improves upon older methods that used only static images of smiling faces because we encounter more moving faces in everyday life than still faces. But if that smiling face is cropped to show no background, is that more or less natural than a static headshot of someone smiling that does include background? Moreover, in vision science, naturalistic is often used to refer to static images with real-world properties (Puckett et al., 2020; Allen et al., 2021). In this way, the term ‘naturalistic’ is akin to other adjectives (like ‘difficult’ or ‘open-ended’) used to describe stimuli relative to subfield standards. Although many scholars use the descriptor ‘naturalistic’ with the knowledge that there are many dimensions on which a stimulus can be more or less natural than some other stimulus, without further explanation of those specific dimensions as continuous spectra, ‘naturalistic’ becomes a categorical bin with the assumption that the readers understand the subfield-specific reference point. This leads to a logical trap wherein the ease with which we use the ‘naturalistic’ label becomes a shortcut for thinking more deeply about how or why these stimuli represent novel or improved conditions for testing a research question.

Media are meticulously crafted for form and function

Some previous work specifies that the word naturalistic refers to the viewing or listening task itself such that there are no additional demands on participants when a researcher asks that they attend to a media stimulus. In other words, the participants will watch a movie in the lab just as they would in their living room, and therefore the task is natural. In many ways, this is a valid assertion (notwithstanding that many film directors and media scholars would argue that watching a movie while supine in an MRI scanner is a fundamentally different experience from being in a theater or at home on the couch). However, our impression is that most neuroscientists using media adopt the stronger sense of ‘naturalistic’ in the sense of ‘similar to real life’ and are attracted to these stimuli for the promise of more closely mimicking perception and cognition in the natural (i.e. nonlaboratory) environment. Indeed, movies, like everyday life, present viewers with sights and sounds as continuous streams of multimodal information that brains synthesize into the perception of, e.g. faces and speech (Mesulam, 1998). As the faces move and speech unfolds over time, audiences comprehend the characters and dialogue that comprise the larger scenes and the overall narrative.

Unlike real life, however, mediated messages have form, meaning that the cues a message uses to engage audiences’ senses, percepts, cognition and emotions are deliberately structured as a system of interrelated features (Bordwell et al., 2016). This underlies a second issue with the current usage of the term naturalistic: the media products that are commonly subsumed under the term ‘naturalistic’ are fundamentally not natural. Broadly speaking, if we take ‘natural’ to refer to anything existing spontaneously in nature, the act of translating something to screen or text—expressing it through a medium—makes it unnatural. In other words, that which is mediated cannot be natural. More specifically, media products like movies, television or books represent subjective and goal-directed tellings of fictional or nonfictional events. They intentionally sacrifice ‘natural’ in service of a particular function(s). A video might be designed to inform, to persuade or to entertain (Johnson-Sheehan and Paine, 2016) or serve a more abstract mass communicative function, such as surveilling the environment, interpreting major events or transmitting cultural norms (Lasswell, 1948). Choices about elements, from wording in a script to lighting to minor background objects, are often painstakingly considered by creators to ensure that every detail contributes to the intended effect on the audience. Therefore, while the ‘experience’ of the stimulus may be considered naturalistic (in the sense that it reflects an activity—e.g. watching a movie, listening to a podcast—that people might choose to do outside the laboratory), the stimulus itself should not be considered natural in the sense of other, less structured inputs.

The ‘naturalistic’ label limits interpretability and generalizability

What are the elements of form in media, and how can we understand and harness them for research? Media form is typically highly complex, with rich spatiotemporal information that, from a neuroscientist’s perspective, makes media much more difficult to parameterize than traditional stimuli. This, in turn, makes it hard to describe and compare media stimuli in terms of their formal features. Yet, simply labeling a stimulus ‘naturalistic’ sidesteps an explanation of why a particular piece of existing media is fit for a particular research goal and what potential influences (positive and negative) its features may have on experimental results. This brings up the third issue with the ‘naturalistic’ label: without any reference to why a particular piece of media is fit for an experiment, the underlying assumption becomes that all similar pieces of media will elicit the reported effect (otherwise referred to as a stimulus-as-fixed-effect fallacy; Clark, 1973; Yarkoni, 2020). In other words, by neglecting to describe how, for example, a video’s form and/or content suits the research question, the authors imply that the choice of video does not matter, and the results can be reproduced with any video that is ‘naturalistic’. On its face, anecdotal evidence is sufficient to show that this assumption cannot be true: not all superhero movies are box office hits, and not all celebrity-endorsed public service announcements inspire corporate change like the backlash against plastic straws. What specific feature(s), and in what combinations, account for these varied outcomes? While each particular piece of media is unique, media are built from a set of common features that can be quantitatively and/or qualitatively described. Describing and justifying stimuli in terms of these features would allow neuroscientists to begin to abstract away from a specific experiment to predict how results will generalize.

Given the importance of describing and justifying naturalistic experimental stimuli, we further content analyzed the Methods sections of each of the previously mentioned 35 empirical articles reporting the use of media stimuli (Figure 1). Within each Methods section, two independent coders sorted the sentences that were clearly related to stimuli into categories based on whether the sentence dominantly pertained to: (i) stimuli settings or editing (e.g. screen resolution or volume), (ii) procedure (e.g. presentation order), (iii) summarizing the stimuli (e.g. overviewing stimulus set or citing previous work for more details), (iv) describing the content (e.g. what was on screen) or (v) justifying the content (e.g. how and why were stimuli selected; see Supplementary Materials for examples of all five categories). The first three fall under the umbrella of experimental protocol and logistics, while the latter two fall under the umbrella of explaining the suitability of the stimuli for the particular research question as we argue for here. For comparison, the coders also identified the sentences describing brain image data acquisition in the subsections on data acquisition. For each manuscript, to assess space dedicated to each category, we expressed sentences belonging to each category as a percentage of the total number of characters (without spaces) in the overall Methods section. In terms of percentage of total Methods sections, there was no difference between the percentage of characters dedicated to all stimuli-related information and brain image acquisition information (paired t-test within articles, t = 0.21, P = 0.83), but there was a significantly higher percentage dedicated to brain image acquisition than to content description and justification combined (t = 3.85, P = 0.0005; Figure 1). For more details on this content analysis, see the Supplementary Materials.

Fig. 1.

Summarizing the current treatment of media stimuli in psychological and neuroscientific research. Thirty-five articles from the NeuroImage Special Issue on Naturalistic Neuroimaging that reported using media stimuli were assessed. Left panel: For each article, relevant sentences from Methods sections were sorted into two categories: stimuli-related and image acquisition-related, and the number of characters devoted to each category was expressed as a percentage of total Methods section characters. Each data point represents one article. There was no difference in number of characters devoted to stimulus-related versus image acquisition-related information (paired t-test; t = 0.21, p = 0.83). Right panel: The stimuli-related category was further broken down into subcategories: one concerned with describing the experimental protocol (e.g., how and when stimuli were presented), and one concerned with describing and justifying the actual content of the stimuli. This analysis showed that articles devoted significantly more characters to describing imaging acquisition than to describing and justifying stimulus content (paired t-test; t = 3.85, P < 0.001). The outlier noted here was removed solely for visualization (not statistical) purposes.

From this analysis, which is meant to serve as a brief descriptive snapshot of the current treatment of media stimuli in neuroscience, there seems to be great variance in how media stimuli are treated when adopted for neuroscientific study. Notably, all 35 Methods sections had more than zero characters dedicated to brain image acquisition, whereas seven (or 20% of all analyzed Methods sections) had no sentences categorized as either content description or content justification.

By overlooking or underestimating the importance of stimulus content, researchers obscure experimental validity and limit theoretical and methodological nuance. Reporting the aspect ratio of a video is important to gauge what the experimenters did to deliver the stimulus to participants, but without any description or explanation of what was on screen, readers will fail to understand what the participants did during the experiment. Given the novelty and diversity of media stimuli compared to well-known cognitive tasks, we argue that more Methods section real estate should be dedicated to describing and justifying the choice of specific stimuli.

Of course, this is more easily said than done; there are entire fields of study dedicated to understanding media, films and narratives, and most neuroscientists do not—and cannot be expected to—have comprehensive knowledge of this literature. However, there are some basic principles from these fields that, if considered, can go a long way toward helping researchers improve the theoretical and methodological validity of their experiments. The rest of this article is intended to serve as an introductory guide for neuroscientists on what to think about when designing experiments involving media stimuli, including a brief primer on media design and construction, how to treat low-level audiovisual information, how ‘media priors’ influence participants’ expectations and pointers toward further scholarship from media and communication studies that can immediately enhance their own work.

A media primer for naturalistic neuroscience

Form matters

Every media product has a form or a structured and dynamic system of relationships between its constitutive elements (Bordwell et al., 2016). Elements, here, refer to anything that comprises the mediated message. To further motivate why neuroscientists should care about form in choosing media stimuli, we first briefly mention some empirical work that demonstrates why, and in many ways how, the form imparted on media through cinematic practices meaningfully influences cognitive processes. In some of the earliest work using movie stimuli in neuroimaging experiments, Hasson et al. (2008) demonstrated that different types of media (e.g. a western movie, a comedy and a video of a park) lead to distinct patterns in group brain responses. Research on event segmentation using movie stimuli has shown that event beginnings and endings are not purely driven by editing cuts but also higher-order changes in the narrative (Magliano and Zacks, 2011). Attention research using eye tracking demonstrates that films drive cross-subject synchrony in eye gaze (Smith and Mital, 2013), and attention in film contexts is in turn driven by both the narrative and lower-level cinematic features like motion and color (Loschky et al., 2015; Hinde et al., 2018). Although it may be sufficient for a study of face perception to simply have any moving faces on screen, our judgments of those faces as mediated fictional characters are contextually dependent on the other characters on screen (Grizzard et al., 2020). This work and much more (see Shimamura, 2013 for a collection of examples) paint a complex picture wherein a viewer’s perception of the elements used in a film is tied to their comprehension of the film. Moreover, these media practices should be taken into account when adopting media for study. It is with this in mind that we discuss how media are designed and constructed to understand how the movie (or audio story, etc.) becomes the task.

The movie is the task: media design and construction

Broadly speaking, there are four major categories of formal elements often referenced in film and television. These are: (i) cinematography, which refers to manipulating camera angle, distance or movement when capturing information; (ii) mise-en-scene, which refers to any element that appears on the screen from a stationary prop to an actor’s movement; (iii) editing, which refers to how images or film are stitched together; and (iv) sound, which refers to the audio elements, such as diegetic (sounds originating from within the film world that characters would hear, like cars on a busy street) and nondiegetic sound (sounds detached from the film world like the soundtrack, which characters would not hear).

Because media have form, one can tease apart how they engage audiences in particular tasks, what those tasks are and how specific elements of that form influence the biological processes and psychological experience of viewers in a predictable and therefore useful way for scientific study. Close-ups effectively engage social processes such as empathy (Bálint et al., 2020; Lankhuizen et al., 2020); the number of film cuts has been steadily increasing over many decades, which seems to reflect an increased alignment with the timescale of human attention (Cutting, 2016; Cutting et al., 2010); and personal narratives collectively engage audiences by synchronizing brain responses more so than other forms of messages (Grall et al., 2021). The editing technique of montage overcomes physical limitations of time and space, as well as cognitive limitations of attention and memory capacity, by presenting only the most relevant information in a condensed time frame (Dudai, 2012). From the world of film, one of director Alfred Hitchcock’s most prominent lessons is on how to build tension and suspense by giving the audience information the characters do not know (Hitchcock, 1948). For example, if the audience knows that a bomb is hidden under a table where the protagonist sits, this engages the audience in an emotionally laden theory-of-mind task as the viewers infer the mental state of the character. The structured relationships among features create an enhanced representation of everyday life that is optimized for and drives human cognition.

Mediated messages allow scientists to research these processes because media are not natural and because they engage viewers in specific tasks. Therefore, the more that a scholar carefully considers the system of relationships among elements of a media stimulus, the more they can use this knowledge to their scientific advantage. When two seemingly unrelated video clips are edited together, such as a clip of a sweating man and a clip of a door, the audience is engaged in an inference task in which they extract meaning that was not explicitly shown (e.g. the man nervously waits for someone to come through the door). When a script repeatedly references a character or an object, this engages the audience in a repeated exposure and memory reinstatement task. These are the same relationships that critics notice when analyzing an audiovisual work: How are elements repeated or varied? How are events segmented, and how do they progress from one to another? How well do the elements fit together? By applying the language of cognitive and social science to these questions and emphasizing psychological and brain responses as the dependent variable (e.g. how does the brain segment events and integrate information over long timescales? Baldassano et al., 2017), this builds a beginning framework for a more nuanced adoption of media stimuli for scientific study.

Low-level confounds or formal features?

One major critique against adopting media stimuli is the unchecked influence of the many ‘low-level confounds’, such as brightness or volume, that might account for large portions of variance in neural activity. For example, abrupt movement on screen can create large spikes in visual cortex that obscure effects of interest when trying to observe, for example, more subtle computations involved in spatial navigation. These ‘artifacts’ also undermine generalizability, raising concerns that observed effects might only be due to the low-level properties of a specific stimulus. One popular solution to this problem is to regress out the effects of these features in preprocessing or first-level general linear model analyses and/or to limit analyses to higher-order regions, which are assumed to be less closely driven by purely perceptual features.

However, as alluded to previously, what a neuroscientist considers to be a confound, a filmmaker would consider a formal feature. On a film or media production staff, there are whole teams dedicated to designing a particular feature, such as the lighting or audio mixing, that make up the majority of roles. This is because those exact features, and how they vary and covary over time, fundamentally influence the experience of the product as a whole. Neuroscientists are quick to include some proxy for volume as a control in analyses, yet the different facets of sound, including perceived loudness, influence our psychological experience (Coutinho and Dibben, 2013; Ma and Thompson, 2015). Drawing on the earlier editing example, the cut between two unrelated video clips is the thing that creates a third, unrepresented meaning in the mind of a viewer. This inspires an important question for scientists to consider when using media stimuli. What is lost when ‘controlling for’ low-level confounds that, to a filmmaker, are the fundamental building blocks of the story? Colloquially speaking, are we throwing the baby out with the bathwater?

Addressing this issue necessitates nuanced decision-making on multiple fronts. Computationally, scientists will continue to develop techniques for parsing particular effects in data. As those techniques evolve, however, we can further finesse stimulus selection and design. Prior to launching a full-scale study with a given media stimulus, one could extract continuous regressors for low-level (e.g. luminance and audio envelope), mid-level (e.g. presence of certain characters on screen) and high-level (e.g. mood) features to determine the degree of collinearity between features and the extent to which this might confound testing of a particular hypothesis using that stimulus. The potential ‘confounding’ features will vary based on the research questions and phenomena of interest. For an example of assessing feature collinearity profiles as part of the stimulus selection process, see Figure 2.

Fig. 2.

Example assessment of feature collinearity profiles to aid media selection using automatic feature extraction (pliers; McNamara et al., 2017). The following examples use a set of two- to four-minute movie clips from the Human Connectome Project 7 T acquisition (Van Essen et al., 2013) and extracted ‘faces on screen’ as a proxy for social content. (A) If an experimenter wanted to choose a media clip in which social content is minimally correlated with low-level features, they could extract these features (such as brightness, optical flow and audio root mean square or RMS) and assess the collinearity profiles within each clip across time. The correlation matrices show that in the Social Network clip, faces are associated with moments of louder audio and less motion, and in the Star Wars clip, the frames containing faces tend to be darker. The Ocean’s 11 clip shows the weakest collinearities between faces and low-level features (Low-level features [brightness, optical flow and audio RMS] have been smoothed with a Gaussian kernel and z-scored for visualization purposes.) (B) & (C) If an experimenter wants to use a variety of video clips, they can also assess collinearities across clips to determine, say, whether videos with higher social content (averaged across time) also have certain low-level properties that distinguish them from videos with lower social content. In the above example, there is a strong negative relationship between social content and the standard deviation in brightness such that clips higher in social content tend to be darker overall and to fluctuate less in brightness across frames. This analysis was inspired by and adapted with permission from Finn and Bandettini (2021).

Additionally, one might use multiple stimuli that differ in low-level features (and/or the degree of collinearity between low-level and high-level features) but evoke a similar higher-order experience. If stimuli similar in their higher-order properties, like two sentences that have the same meaning, elicit similar patterns of brain activity despite differences in low-level properties, like difference in phonemes, one can more safely conclude that the effect is associated with a more general cognitive or affective process. Using multiple media stimuli will also benefit generalizability and help overcome the stimulus-as-fixed-effects fallacy. Alternatively, one could embrace the stimulus-specific effects. One might rely on a constrained set or single media stimulus to characterize the degree of heterogeneity of a particular behavior in some clinical population, like eye gaze patterns in autism (Keles et al., 2021). Although this would necessitate the aggregation of findings across many studies to build the general picture of some phenomenon, it also represents a reasonable strategy if the goal is to develop or select just one or a handful of audiovisual stimuli with an eye toward creating a new diagnostic test for a clinical condition (Eickhoff et al., 2020).

With this in mind, there are several efforts to overcome the challenges of adopting preexisting media for research by creating bespoke stimuli. ‘Inscapes’, for example, was developed to minimize head motion and keep participants awake during resting-state scans without offering cognition-driving content like a typical movie (Vanderwal et al., 2015). This same team is currently producing a movie to elicit OCD symptoms (Naturalistic Neuroimaging Lab, 2020) and has an associated studio dedicated to creating and testing media stimuli that evoke particular neural processes (https://www.headspacestudios.org/headspace-studios). Additional examples include custom games (Huskey et al., 2018) or original stories crafted by researchers (Finn et al., 2018) or professional writers (Chang et al., 2021). It should not be underestimated how much skill and effort it takes to create custom stimuli, which requires knowledge about both media production and experimental design or effective communication between partners with complementary expertise. However, in the authors’ experience, there seem to be many willing scholars from both media and neuroscience who are eager to take advantage of the research opportunities that a partnership would afford.

Media priors: what expectations do your participants bring with them?

While so far we have considered elements intrinsic to a single media stimulus, media products sit in a broader context as reflections of and influences on culture. This means that when selecting an audiovisual stimulus for research purposes, experimenters should consider how participant characteristics, prior life experiences and, particularly, prior media experiences may impact their responses. A documentary about the Vietnam War will feel very different to an 18-year-old compared to an 80-year-old, and previous work shows age-related differences in how audiences engage in, or are affected by, media viewing (Kirkorian et al., 2008; Depp et al., 2010). The movie Citizen Kane is renowned by critics for being innovative for the time period in which it was made (Brown, 2011), but a movie made today in a similar style might feel derivative or old fashioned. People often relay how well something ‘holds up’ against the test of time when discussing media with peers. This is not to say that older media make bad stimuli; they hold just as much utility as any new media. However, for research purposes, scientists should carefully assess the relationship between the context (e.g. year) the media was created in or represents and the life experiences of participant samples. This will ensure that no senior investigator selects a movie stimulus based on childhood nostalgia with the expectation that it will affect current 18–22-year-olds the same way it affected them.² Environmental influences are also worth considering, such as the impact of co-viewing on media processing (Tal-Or, 2016; Cheong et al., 2020; Dziura et al., 2021).

Beyond participant characteristics and life experiences, there is an outsized influence of prior media experience on how viewers engage with novel media, or what we informally refer to as ‘media priors’. Audiences learn how to watch a screen from a very young age (Anderson and Hanson, 2010). Despite the inherent artificiality of cuts, film editing is often considered ‘the invisible art’ because it goes unnoticed when done well (Smith and Henderson, 2008). This is, in part, attributable to viewer familiarity with classic and popular continuity filmmaking guidelines, such as shot/reverse shot and the 180° rule.³ Importantly, this means that viewers will notice or have to adjust to deviations from common techniques. For example, given the utility of editing for efficiently representing the passage of space and time, if a researcher chooses a media stimulus with no cuts to minimize visual transients, participants may expend conscious efforts to comprehend the novel filmmaking technique. Similarly, displaying fisheye lens footage may seem odd to a participant because it collapses recordings from a wide field of view onto a screen placed at the center of the participant’s field of view, which is atypical.

In addition to priors for how media are generally constructed, there are also influential priors for the content and form of specific media. The act of viewing a movie once fundamentally changes how a movie is viewed the second time due to the knowledge of the story. This has important implications for experimental design but also in and of itself creates fertile ground for research. For example, for mystery and suspenseful media, it could be a confound if a participant had previously seen the selected stimulus and the researcher wanted to study surprise. However, watching a film twice lends itself to studying how social and emotional processes might change with context (for an additional example, see Box 1: Paradox of Suspense). Screening participants for their familiarity with a media stimulus (i.e. Have you seen this movie before?) is a researcher’s first line of defense against media priors to maintain experimental consistency across a sample. This also impacts the common analytical technique of averaging participant responses across repeated exposures: experimenters cannot assume that they are capturing the same processes between repeated viewings of a media stimulus because the conditions have changed such that the participant knows what to expect and may focus on unique features. Therefore, for experimental designs that call for repeated testing of the same individuals (e.g. within-subject context manipulations, reliability and/or longitudinal studies), scientists should carefully consider whether the processes of interest are confounded by the changes inherent to watching a movie a second time. One solution for this would be to obtain multiple exemplars of media that all evoke the processes of interest to avoid showing any stimulus twice. Although this type of design precludes some first-order analyses that cannot be performed across different stimuli (e.g. intersubject correlation), one could compare repetitions at the level of second-order statistics (e.g. intersubject correlation values across a population and beta weights associated with stimulus features derived from a general linear model-based approach). Importantly, in service of our larger argument, this solution is made possible by understanding and specifying what it is about a stimulus that is driving the desired process so that one can choose stimuli with similar effects.

Box 1.

The paradox of suspense

There is no terror in the bang, only in the anticipation of it.—Alfred Hitchcock One big influence on a person’s response to a mediated message is their previous exposure to that message or similar messages. Knowing the endings to movies like The Sixth Sense and Saw will fundamentally alter how someone comprehends story information on a second viewing because there is new context. However, viewers have long reported that they will still feel suspense when watching a suspenseful film a second time (Carroll, 1996; Gerrig, 1989). This is referred to as the paradox of suspense, or the idea that repeat viewers feel the same emotion despite knowing a story’s outcomes. The paradox of suspense has been contemplated across disciplines from literary studies to quantitative communication research for many decades. The paradox can be summarized in three premises (Yanal, 1996): Suspense requires uncertainty. If one consumes a narrative a second time, that person is certain of the outcomes. People feel suspense when they consume a narrative a subsequent time. Given that these three premises cannot logically all be true, scholars have weighed in on whether they believe the first premise to be wrong (Comisky and Bryant, 1982; de Wied et al., 1997; Zillmann, 1996), the second (Brewer and Lichtenstein, 1982; Gerrig, 1989) or the third (Uidhir, 2011; Yanal, 1996). Despite being a relatively untouched topic in recent years, audiences still go back to experience their favorite media again and again. Modern methodological and analytical tools are poised to shed new light on this phenomenon, such as by determining how intraindividual patterns of brain activity evolve from one viewing to the next.

Media priors can also be managed by understanding film and media conventions (Bordwell et al., 2016). Take genre conventions, for example. Common conventions in action films include a clear hero and villain, plus fast-paced editing with lots of movement and explosions. Westerns are also typified by clear heroes and villains, but the protagonist is often motivated by revenge and the action often involves guns, bandits and dust. Conventions establish expectations in audience minds and allow viewers to suspend disbelief such that the rules that govern the real world need not apply. In fantasy films, viewers will expect the protagonist to overcome supernatural obstacles using supernatural means to complete a quest. If the protagonist fails to complete the quest, however, this can create uncertainty, dissatisfaction or curiosity. In this way, the expectations established by conventions are particularly powerful because, by understanding how conventions create meaning and emotion in viewers’ minds, media producers are able to blend or deviate from the norm to create novel experiences. In other words, conventions represent relatively stable priors useful for manipulating and predicting viewer responses (see Box 2 for a thorough example).

Box 2.

Undertale: testing morality by defying convention

Undertale is a 2D role-playing game (RPG) developed by Toby Fox released in 2015 (Fox, 2015). RPG refers to a genre of games with a common ancestry traceable to the tabletop game Dungeons and Dragons. Characteristic conventions of RPGs include (i) embodying a character that (ii) grows over time (i.e. levels up) by (iii) defeating enemies and completing quests. In the game Undertale, the player controls a child that fell into the magical world of the Underground and must make their way back to the surface. Genre convention dictates that any enemy that engages the child in combat should be defeated to then level up and reach the ending. However, game developer Fox advertised the game as ‘The RPG game where you don’t have to destroy anyone’. With this tagline, Undertale becomes an experiment in morality wherein a player’s prior experience with RPG convention is pitted against the knowledge that no ‘enemy’ in the game must perish. It is exceedingly easy to defeat enemies early in the game, but this dynamically changes the shape of the narrative where nonplayer characters guilt the player with dialogues like, ‘You killed him. He had a family, just like you. Did you ever think about that?’ Executing the no-kill (pacifist) version of the narrative takes a surprising amount of effort, creativity and trust in the knowledge that there is indeed a way to progress through the game without felling anyone. Moreover, this also strips away the typical RPG reward system because, without killing enemies, the player does not level up as they would otherwise, which disincentivizes this mode of play by increasing the relative challenge of each encounter. By understanding and defying genre convention, Fox effectively created a game that artificially manipulated the extent to which a participant will expend effort to uphold moral decision-making (i.e. don’t kill) despite the ease of and experience with making the immoral decision (i.e. kill) to achieve personal goals (i.e. level up and beat the game). This is a prime example of a convention-defying media stimulus that could be used as inspiration for behavioral and/or neuroimaging experiments to study moral decision-making processes in rich first-person scenarios.

Don’t reinvent the wheel: leveraging existing media scholarship

Another way to predict viewer responses in designing psychology and neuroscience experiments is by drawing on the previous testing conducted by media scholars. Scholarship on media in its many forms is exceedingly multidisciplinary and highly heterogeneous in its philosophical approaches, theories and methods. In an academic institution, media research might take place in departments of Communication, Film and Media Studies, Digital Humanities or Literature and fall under a host of labels, including media psychology, media effects, narratology, advertising, etc. This should not deter scientists from other disciplines from engaging with these literatures, however, but instead reassure them that a lot of preparation has already been done. For example, if one is interested in using media stimuli to study dynamic changes in empathy and impression formation over time, there are popular and useful resources from screenwriting on the necessary conditions to get an audience invested in a character (Snyder, 2005). Additionally, there is classic media theory and updates that describe how viewer judgments of character behavior interact over time to shape impressions of the character and the overall story (Zillmann, 2000; Tamborini et al., 2018). Furthermore, previous media research might have readily available stimuli that are already pretested, which is exceedingly useful because pretesting is the empirical path to justify stimuli selection by ensuring validity. For instance, there is a body of work in media psychology on inspirational media (Oliver et al., 2018) in which scholars characterized the types of media that elicit feelings of inspiration and transcendence (Dale et al., 2017; Ji et al., 2019), the audiences who are most likely to consume inspirational media (Raney et al., 2018) and physiological responses to inspirational media (Clayton et al., 2019). From this research, an emotion scholar can adopt the inspirational videos, which are already pretested, or use published findings as guidelines for selecting new stimuli.

Conclusion

We know that media—the movies, books and music we consume every day—influence our thoughts, emotions and behavior, and our responses depend on the qualities and content of what we consume. This has been written about since the early days of academia (see Aristotle’s Poetics). Because of this, it comes as no surprise that when scientists show a movie in an MRI scanner, it elicits unique, rich and dynamic patterns of brain activity. However, over 15 years since the Hasson et al. (2004) landmark study, it is time to move beyond the umbrella term ‘naturalistic’ when referring to media stimuli, which is overly broad at best and misleading at worst. It is no longer sufficient to conclude that observed brain activity or behavior is simply ‘evoked by the (particular) stimulus’ when the tools are available to answer ‘what about the stimulus?’

As scientists, we are equipped to explain how media elicit these observable effects and, more importantly, how we can use media for more nuanced research. It is not uncommon for a scholar using media narratives in their research to claim that their ultimate goal is to ‘use science to make better stories’. However, that fails to acknowledge the immense and successful industries (e.g. film, publishing, gaming and advertising), with their long histories of innovation, dedicated to the art of mediated storytelling and communication. Regardless of whether they cite Karl Popper, there is a science to the art that manifests in every creator’s room and editing suite. It is we, the academics, who fall short in asking the more necessary question, ‘How can we use their stories to do better science?’

Based on the arguments presented here, we offer a set of questions to guide researchers in choosing media stimuli for a future experiment:

1. Why is this particular media stimulus the right task for this research question?

2. How have you ensured your stimuli engage the processes you intend for them to engage? What previous testing supports your choice of stimuli?

3. Which formal media elements can you notice in your stimulus, how might these elements engage certain cognitive processes and can you model these elements at the analysis stage to draw more nuanced inferences?

4. How much collinearity is there among ‘low-level’, ‘mid-level’ and ‘high-level’ features in your stimulus? How much of a problem does this collinearity pose for your particular scientific purpose(s)?

5. How do participant characteristics like age or clinical diagnosis influence the ways they engage in or are affected by your media stimuli?

6. How do any edits to the stimulus (e.g. cutting out a scene for length purposes) help or hinder your observation of the processes of interest?

7. How do these media stimuli uphold or violate convention to your advantage or disadvantage? If there is a disadvantage, how do you ensure this doesn’t impact your results?

8. Have participants seen this movie before? Will previous experience with this or similar stimuli influence your results? Does the age of the film (and associated differences in filmmaking techniques or representations of events) affect your inferences?

9. What research on these or similar stimuli has been conducted in the fields of communication, media psychology, film and media studies, etc.?

We look forward to ongoing progress in social, cognitive, and affective neuroscience with the thoughtful use of media stimuli.

Funding

C.G. and E.S.F. are supported by National Institutes of Health grant R00MH120257.

Conflict of interest

The authors declared that they had no conflict of interest with respect to their authorship or the publication of this article.

Supplementary data

Supplementary data are available at SCAN online.