Imagining predictions: mental imagery as mental emulation

Abstract

We argue that the primary function of mental imagery is to allow us to generate specific predictions based upon past experience. All imagery allows us to answer ‘what if’ questions by making explicit and accessible the likely consequences of being in a specific situation or performing a specific action. Imagery is also characterized by its reliance on perceptual representations and activation of perceptual brain systems. We use this conception of imagery to argue that all imagery is simulation—more specifically, it is a specific type of simulation in which the mental processes that ‘run’ the simulation emulate those that would actually operate in the simulated scenario. This type of simulation, which we label emulation, has benefits over other types of simulations that merely mimic the content of the simulated scenario.

1. Introduction

For cognitive scientists, the term mental imagery typically first brings to mind either the protracted debate over the nature of the representations used in imagery or the role of imagery as a mnemonic (see Paivio 1971, 1986; Kosslyn et al. 2006). The ‘imagery debate’ frequently overshadows the question of the everyday functions of mental imagery, which are at least as important as the questions that have received the most attention. However, research on this topic has often been hobbled by a key problem, namely the sparse stimuli and artificial tasks that imagery researchers contrive for their experiments (e.g. the rotation of geometric shapes; Shepard & Metzler 1971). The minimalist character of the sorts of imagery evoked in most laboratory studies may obscure the vivid, rich character of everyday imagery.

In this paper, we move beyond questions such as ‘what is imagery?’ and ‘can imagery enhance memory?’ to ask ‘what is the primary psychological function of imagery?’. In doing so, we argue that mental imagery affords us more than the mental rotation of stacked cubes—it allows us to simulate reality at will, and, because of this, allows us to predict what we would experience in a specific situation or after we perform a specific action. This ability not only allows us to reconstruct the past, but also to anticipate what may occur in the near and distant future.

2. Mental imagery: levels of analysis

Mental imagery occurs ‘when a representation of the type created during the initial phases of perception is present but the stimulus is not actually being perceived; such representations preserve the perceptible properties of the stimulus and ultimately give rise to the subjective experience of perception’ (Kosslyn et al. 2006, p. 4). Critically, this characterization of mental imagery implies that multiple forms of imagery exist: every type of perception should have a corresponding type of imagery. And in fact, there is evidence for distinct object-based imagery (e.g. of shapes and colours) versus spatial imagery (e.g. of locations; Farah et al. 1988; Kozhevnikov et al. 2005); there is evidence for auditory imagery (e.g. Zatorre et al. 1996), and for what is commonly called ‘motor imagery’ (which actually appears to be proprioceptive or kinaesthetic imagery—one experiences the bodily sensations of movement, not the movement commands themselves; Jeannerod 1994).

Although the above characterization captures essential features of mental imagery, it does not explain how imagery works or what it does. In his seminal analysis of vision, Marr (1982) argued compellingly that in order to understand fully an information processing system, psychologists must analyse it from three distinct levels: the computational level (which focuses on what the system is designed to accomplish); the algorithmic level (which focuses on the system's structures and processes and how they are drawn upon to perform specific computations); and the implementation level (which focuses on the system's physical substrate). As Kosslyn & Maljkovic (1990) pointed out, Marr's levels of analysis are interdependent, not independent: theory and research focused exclusively on one level of analysis constrain theories and informs research focused on other levels of analysis.

In an effort to characterize mental imagery more completely as an information processing system, we consider it below using Marr's levels of analysis.

(a) Computation: the functions of mental imagery

When we consider the computational level, we are led to ask first and foremost: what is the function or functions of mental imagery? In our view, the primary function of mental imagery is to allow us to generate specific predictions based upon past experience. Imagery allows us to answer ‘what if’ questions by making explicit and accessible the likely consequences of being in a specific situation or performing a specific action. For example, imagery can allow us to predict the path of a projectile, the consequences of a chess move, the scariness of a snake and the feelings of a friend.

We want to make a very strong claim, namely that all imagery allows us to generate specific predictions based upon past experience. Consider the most trivial example of imagery: close your eyes and imagine1 a dot. What did your dot look like? Round, we hope. Because a dot is by definition round, however, the roundness of your imaged dot hardly predicts anything. Now, what colour was your dot? A dot is not defined as having a specified colour—it can be an infinite number of colours, from crimson to cobalt. In all likelihood, however, your imagined dot was black, not crimson or cobalt. The reason is simple: most dots you have previously seen (e.g. the full stops on this page) were black, and imagery draws automatically on this past experience. Thus, even the bare image of a dot transforms past experience into explicit, specific predictions: in this instance, you used imagery to predict the undefined colour of the dot we asked you to visualize; put differently, you used imagery to predict the colour of a typical dot.

Mental imagery not only allows us to predict the imminent or distant future, but also to consider many possible futures—or even many possible worlds. For example, when deciding the best way to drive home from a friend's house, you might imagine several alternative routes, and compare the likely traffic patterns, the number of stoplights and so on. Moreover, we can use imagery to predict events that we may never actually experience in the future (e.g. Einstein's imagery of travelling at light speed), or events in the counterfactual present (e.g. imagining life as a lottery winner). This same mental machinery allows us to revisit events that have already occurred (e.g. the images of a murder scene formed by someone who witnessed the crime). In our view, the oft-cited role of imagery in memory (e.g. Paivio 1971; Kosslyn 1980) is in fact an application of a more general function, namely the ability to simulate2 what one would perceive in a specific situation.

(b) Algorithm: representations and processes used in imagery

Considering imagery at the algorithmic level leads us to focus on how it accomplishes its computational functions. Specifically, at this level, we ask: what mental structures and processes does imagery engage? Many empirical investigations attest to the fact that imagery relies on many of the same representations that give rise to perception. There is good evidence that at least in some circumstances, imagery recruits even mechanisms used in the early phases of perceptual processing. Such mechanisms rely in part on depictive representations, where configurations of densely arranged points (with size, intensity and colour) in a coordinate space are arranged so that the pattern ‘resembles’ the appearance of the referent (see Kosslyn et al. 2006).

Because imagery relies on perceptual representations, it makes explicit and accessible the same types of information that are registered by the senses during perception (including proprioception and kinaesthetic information). In object-based visual imagery, for instance, depictive representations make available the full set of visual information (i.e. size, intensity and colour values for each point). Critically, the information made explicit and accessible by perceptual representations during imagery supports its computational functions. For instance, if asked the shape of a cat's ears, most people will visualize the feline and ‘look’ at the shape—which they had never considered explicitly before but which is implicit in the representation.

However, we must note that representations by themselves do no work. They must be processed in some way, or they may as well not exist. Imagery invokes at least four distinct types of processes. First, memorial processes must retrieve and later encode episodic information (e.g. to imagine telling a risqué joke during a toast at your wedding, you would retrieve memories of past weddings; furthermore, you encode your episode of wedding imagery). Much evidence supports this interplay between memory and imagery. For example, neuroimaging studies have revealed that most of the perceptual regions activated by mental imagery also become active during episodic retrieval, and do so in a modality-specific manner (for reviews, see Mellet et al. 1998; Cabeza & Nyberg 2000; Buckner & Wheeler 2001; Wagner et al. 2005). Furthermore, neuropsychological evidence suggests that this overlap is not merely correlational: amnesic patients often exhibit severe deficits in their imagery (e.g. O'Connor et al. 1992; Ogden 1993; Hassabis et al. 2007b; for review, see Rubin & Greenberg 1998; see also Rosenbaum et al. 2004).

Second, imagery processes must draw on retrieved episodic information to generate explicit, accessible representations in working memory (e.g. an auditory image of yourself telling the joke and a visual image of your attentive bride; for review, see Kosslyn (1980)).

Third, automatic associative processes must guide the imagery realistically, both by providing new content (e.g. a look of horror on your bride's face upon hearing the punch line) and generating affective or physiological responses (e.g. a feeling of embarrassment, a racing heart). Just as a percept can bring to mind associated images and memories (as evidenced by innumerable studies of ways that linguistic prompts can prime visual processing), so can an image. Furthermore, just as a percept can induce associated affective and physiological events (e.g. Watson & Rayner 1920), so can an image. For example, victims of childhood sexual abuse report greater negative affect and undergo greater physiological stress when imagining scenarios of abuse than when imagining neutral scenarios (Shin et al. 1999).

Finally, top-down executive processes must direct the processes that initiate, inspect, manipulate and terminate the imagery when it has allowed you to accomplish your goal (e.g. upon embarrassment, ending the imagery or re-imagining the toast with a different joke). Substantial evidence supports the role of top-down processing in imagery (for review, see Kosslyn et al. 2006).

(c) Implementation: core network

At the level of the implementation we are led to ask: how is imagery realized physically in the brain? As with perception, the neural mechanisms that underlie imagery surely depend on the content and purpose of particular instances. Nevertheless, we make three strong claims. First, because it relies on perceptual representations, imagery should activate perceptual cortices in a predictably specific manner. For example, whereas individuals who imagine preparing a banana split should reliably activate their visual cortex (and possibly motor cortex), individuals who imagine the sensation of eating a banana split should reliably activate their gustatory cortex (and possibly motor cortex). Indeed, much evidence supports this claim. Visual imagery activates the visual cortices, including, in some cases, the earliest cortex (e.g. Kosslyn & Thompson 2003), auditory imagery activates the auditory cortices (e.g. Zatorre et al. 1996), motor imagery activates motor cortices (e.g. Porro et al. 1996) and gustatory imagery activates gustatory cortices (e.g. Kobayashi et al. 2004).

Second, because of the complex and temporally extended nature of imagery, it should activate a broad and diverse set of brain regions. More specifically, imagery should activate—in addition to perceptual cortices—regions involved in episodic memory retrieval (e.g. the hippocampus), top-down processing (e.g. prefrontal cortices) and associative processing (e.g. the retrosplenial complex; Bar & Aminoff 2003). Considerable evidence supports these claims (for review, see Cabeza & Nyberg 2000), although baseline conditions in many neuroimaging studies—in conjunction with the sparse stimuli often used by researchers—frequently obscure the full range and degree of brain activation elicited by mental imagery. In a noteworthy recent study, Hassabis et al. (2007a) investigated which brain areas become active during the sort of rich imagery individuals use in everyday life, using a simple imagery task as baseline; their results revealed a broad network that included the ventromedial prefrontal cortex, hippocampus, retrosplenial complex and posterior parietal cortex.

Finally, the activation of these distinct regions should unfold temporally in a pattern that mirrors the temporal sequence of imagery processing. For example, the prefrontal activation associated with top-down control should precede the hippocampal activation associated with episodic memory retrieval which, in turn, should precede the perceptual activation associated with perceptual representation. Although neuroscientists are beginning to track the temporal–spatial unfolding of mental imagery (e.g. Sack et al. 2008), this hypothesis has yet to be tested.

The network of regions implicated in imagery often resembles the ‘core network’ (for review, see Buckner et al. 2008), which is also sometimes called the ‘default network’ because of its association with task-unrelated thought. The core network supports processing for a set of imagery-dependent tasks such as navigation, episodic recall and perspective-taking (see Buckner & Carroll 2007; Hassabis & Maguire 2007), and has been characterized as a distinct network after researchers observed that seemingly dissimilar tasks activate strikingly similar brain regions.

That said, we note that some of the brain areas that are sometimes activated during visual mental imagery are not activated in the core network, particularly areas in the medial occipital lobe (for a summary, see Kosslyn et al. 2006). To a large degree, this difference may reflect the baseline conditions that researchers tend to employ in their neuroimaging contrasts. Imagery researchers typically use baselines conditions that do not require visual processing (e.g. Slotnick et al. 2005), whereas researchers who investigate the core network typically use baseline conditions that do require visual processing (e.g. Addis et al. 2004). When contrasted with target conditions that involve imagery, such baseline conditions surely mask activation in lower sensory cortices. Furthermore, medial occipital areas are not activated during all types of imagery (such as spatial imagery). Moreover, even during object-based visual mental imagery, these areas tend to be activated only when high-resolution representations are required to perform the task (see Kosslyn & Thompson 2003); if a task does not require making fine-grained judgements about shape, these areas tend not to be activated.

Thus, baseline problems aside, we cannot identify a single ‘imagery network’—the areas that are activated depend, to some extent, both on the specific type of imagery (e.g. of objects versus spatial relationships) and on the requirements of the task. Nonetheless, the core network is very similar to the network of areas that is activated during many imagery tasks, particularly those that do not require making subtle judgements about shape.

3. Mental imagery, mental simulation and mental emulation

Mental imagery may best be understood in the context of mental simulation, specifically as a kind of mental emulation. As a psychological construct, mental simulation has been considered theoretically and researched empirically in many different contexts: self-regulation (e.g. Taylor et al. 1998); memory (e.g. Ingvar 1979); mental practice (e.g. Driskell et al. 1994); decision-making (e.g. Kahneman & Tversky 1982); mechanical reasoning (e.g. Hegarty 2004); consciousness (e.g. Hesslow 2002); creativity (Clement 2008); social cognition (e.g. Gordon 1986); affective regulation (e.g. Gilbert & Wilson 2007); and mental imagery (e.g. Kosslyn 2008).

Nearly all of these differing treatments of simulation converge on the two essential features of mental simulations. First, simulations are, in the words of Fisher (2006), ‘epistemic devices’ (p. 419). In other words, they make available or generate knowledge. For example, in simulating the sound of a police siren, you can access stored information about its acoustical properties to answer questions such as ‘does a police siren have a constant pitch?’. In addition, mental simulations can be used to generate knowledge, allowing you to answer presumably novel questions such as ‘how does a police siren differ from an ambulance siren?’. However, we note that many mental processes other than simulation make available or generate knowledge (e.g. semantic memory, deduction), and hence this feature cannot by itself define simulation.

Second, simulations operate by sequential analogy. That is, the steps of the simulation mimic the corresponding steps of the represented situation. Mental simulations are ‘run’ such that intermediate steps in the process correspond to intermediate states in the event being simulated. This correspondence is not necessarily one to one (i.e. an isomorphism; see Goldman 1995); not every step in the event must correspond to a distinct step in processing. But each of the intermediate states of the simulation must approximate an intermediate state of the to-be-simulated event (see Fisher 2006). For example, in simulating the drive from one location to another, one need not simulate every turn of the steering wheel or curve of the road; instead, one can merely simulate a sequence of key turns. Importantly, this loose correspondence in intermediate states is a necessary but not sufficient condition of simulation: the ordering of intermediate states in the simulation must also mirror the ordering of the corresponding process or event. In the navigation example, therefore, the sequence of simulated turns must correspond to the sequence of turns in the actual journey.

Furthermore, the sequence of states is functional, not epiphenomenal: each step generates or makes accessible information that critically constrains succeeding steps. A simple example is mental rotation: as an object rotates, each intermediate orientation represents an intermediate orientation of the corresponding object. Or, to take a richer example, when visualizing how far one could hit a ping-pong ball with a baseball bat, one does not engage in simulation by merely imagining the ball at some distance from its initial position; instead, one must also imagine the initial set-up, the swing of the baseball bat and the full trajectory of the ball. Furthermore, the swing of the baseball bat depends on the initial set-up, the trajectory of the ball depends on the swing and the initial set-up, and the final resting place of the ball depends on the initial set-up, the swing and the trajectory. Any process that lacks a functionally dependent sequence does not qualify as a mental simulation. Thus, in sum, we define mental simulation as an epistemic device that operates by sequential analogy.3

Critically, we can distinguish between two fundamentally different types of simulations: instrumental simulations and emulative simulations. In the former, the algorithms that transform successive states in the simulation differ categorically from the processes that transform successive states in the simulated event. For example, you could simulate a social conversation by using conceptual knowledge or hypotheses about the participants to approximate their dialogue. Although this simulation generates knowledge via sequential analogy, the explicit third-person theorizing that drives the simulation bears little resemblance to the first-person socializing that would characterize the actual encounter. These simulations are instrumental in the sense that their algorithms serve merely as the means to produce successive states (and a final outcome) given a set of initial parameters.

In contrast to instrumental simulations, emulative simulations (or, simply, ‘emulations’) mimic not only the intermediate states of the simulated event, but also rely on algorithms that mimic the processes that transform successive states of that event. To simulate a conversation via emulation, for example, you could place yourself in the ‘mental shoes’ of those conversing, predicting their dialogue based upon how you would respond (based on your emotions and the associations that are triggered) in their respective situations. Unlike instrumental simulations, the processes that generate successive states of emulative simulations are not merely instruments used to produce these successive states—they also function as a second layer of simulation. Put differently, whereas instrumental simulations can be thought of as first-order simulation (in that they imitate content), emulations can be thought of as second-order simulations (in that they imitate the processes that change content as well as the content itself).

In defining mental emulation as a type of simulation in which the psychological processes that drive the simulation mimic the processes involved in the simulated event, we imply a close connection to mental imagery. Images, by definition, mimic what we perceive, and hence images can easily capture a sequence of states that underlie an event. Others have also implied such a connection between imagery and simulation, oftentimes without clarifying explicitly the relationship between the two concepts. For example, Roese (1997) defined simulation as ‘imaginative mental construction’ (p. 134), Buckner et al. (2008) defined it as ‘imaginative constructions of hypothetical events or scenarios’ (p. 20) and Taylor et al. (1998) defined it as ‘imitative representation of some event or series of events’ (p. 430). But how tight is the relationship between mental emulation and mental imagery? Could they be one and the same? Or might mental emulation merely sometimes draw on imagery? Or might there be no functional connection between the two?

(a) Similarities between imagery and mental emulation

On all three levels of analysis—computation, algorithm and implementation—imagery and mental emulation are fundamentally similar. In fact, based on the evidence reviewed below, we argue that although all mental emulations may not involve imagery, all imagery is mental emulation.

(i) Predictive function

Because all mental simulations make available or generate knowledge about specific events, they make specific predictions. Thus, in terms of computational function, imagery and mental simulation (instrumental and emulative) are fundamentally similar. Indeed, the fact that the terms ‘simulation’ and ‘imagery’ can often be used interchangeably reveals their functional similarity. For example, if we asked you to ‘imagine seeing a cat's head on a dog's body’ and ‘simulate seeing a cat's head on a dog's body’ you, in all likelihood, would understand the task similarly, if not identically. In fact, many experiments that ostensibly investigate mental simulation explicitly instruct their participants to engage in imagery. For example, Bruzzo et al. (2008) studied the ‘mechanisms underlying mental simulation’ (p. 145) by asking participants to ‘imagine’ movements, and Green et al. (2006) studied ‘how mental simulation alters the impact of previously presented arguments’ (p. 32) by asking participants to ‘imagine a future state of affairs’ (p. 38). Clearly, imagery and simulation can be used interchangeably to instruct research participants, which implies that the terms often have comparable meanings.

(ii) Perceptual representation

An abundance of evidence supports the claim that many forms of mental simulation rely on perceptual representations and, in doing so, connects mental emulation with mental imagery. For starters, individuals engaged in simulation often report the defining phenomenological feature of imagery: ‘seeing with their mind's eye’ (e.g. Clement 1994). However, even if we take such reports at face value, the co-occurrence of mental imagery and mental simulation does not imply that the latter requires the former: mental imagery may play an epiphenomenal role in simulation, just as the trajectory of a projectile visualized on a computer monitor plays no functional role in the underlying simulation.

Interference and individual differences paradigms provide evidence that many forms of mental simulation depend on perceptual representations. For example, Sims & Hegarty (1997) found that a visuospatial task (compared with an equally difficult verbal task) selectively interfered with participants' ability to simulate mechanical motion. And using an individual differences approach, Hegarty & Sims (1994) found that performance on a mechanical simulation task correlated strongly with performance on spatial imagery tasks. If these forms of simulation did not depend on perceptual representations, one could not explain easily either of these findings.

Moreover, neuroscientific evidence also implicates perceptual representations in many types of mental simulation. On the whole, the perceptual regions associated with actual movement are also associated with simulated movement (for review, see Grezes & Decety 2001; Jeannerod 2001). Furthermore, simulated movement reduces the amount of transcranial magnetic stimulation (TMS) required to induce actual motion (Fadiga et al. 1999; Hashimoto & Rothwell 1999), which is just as expected if the simulated movements engage the same neural structures that are stimulated by TMS—and hence boost the effects of TMS.

The forms of simulation investigated in the studies cited above are all, arguably, examples of mental emulation. The ‘mental witnessing’ of mechanical motion and experiencing of bodily motion, for example, rely fundamentally on perceptual processes. Whereas emulative simulations of these events apparently rely on perceptual representations, instrumental simulations of the same events do not.

(iii) Neural implementation

In addition to activating perceptual cortices, mental simulation activates all other regions of the core network, as noted above. In fact, in coining the term ‘core network’, Buckner & Carroll (2007) described its unifying function in terms of simulation: ‘the processes of the network are characterized by a personal, internal mode of mental simulation’ (p. 49).

Thus, from the perspective of the brain, mental simulation and mental imagery are similar. Again, we argue that this similarity applies specifically to emulative simulations. Only with emulations, would one expect such a strong overlap between the neural correlates of mental imagery (which mimics perception) and simulation, as well as the overlap between simulation (e.g. in episodic memory) and perception (e.g. Wheeler et al. 2000).

(b) Differences between imagery and emulation

Given the fundamental similarities between mental emulation and imagery, one could be tempted to conclude that they are not merely overlapping constructs, but instead are identical. However, before making this leap, we must reflect on several potential distinctions between emulation and imagery. The following distinctions have been raised in the literature.

(i) Simple versus complex

Schacter et al. (2008) defined simulation as the ‘imaginative constructions of hypothetical events or scenarios… that involves more than simple imagery’ (p. 42). Whether simulation differs qualitatively from ‘simple imagery’ in their view, however, remains unclear, as does their precise notion of simple imagery. But even if it does, there may be no difference between complex imagery and simulation (cf. Hassabis et al. 2007a).

Along similar lines, Hegarty (2004) argued that simulation does not simply involve visual representations. Her argument is based on the evidence that concurrent motion and body position affect simulation. As far as we know, however, no one has claimed that mental simulation involves exclusively visual representations, and for good reason: even performance on apparently simple tasks of ‘visual’ imagery (e.g. mental rotation) often relies on motor representations (e.g. Ganis et al. 2000) and is affected by body position (Mast et al. 2003).

(ii) Holistic versus piecemeal generation

Citing evidence that individuals construct mental simulations of mechanical systems in a piecemeal fashion, Hegarty (2004) claimed that simulation differs from imagery. She argued that if individuals used imagery to simulate mechanical motion, they could create and inspect holistic images of mechanical motion, and would not need to build up their simulations piece by piece. However, she failed to consider evidence that individuals construct images in stages. For example, individuals generate images of novel patterns serially, based upon the sequence in which they originally encoded the parts into memory; moreover they generate images of block letters serially, based upon writing sequence (Kosslyn et al. 1988). Depending on the image, sequential image generation probably reflects a variety of constraints, including limited attentional resources, the ‘refresh rate’ of co-opted perceptual hardware, the encoding of object parts (rather than entire objects) and the reliance on relative (rather than absolute) spatial information (see Kosslyn 1994).

(iii) Conscious versus unconscious

Barsalou (2008) drew a clear distinction between imagery and the broader concept of simulation: ‘whereas mental imagery typically results from deliberate attempts to construct conscious representations in working memory, other forms of simulation often appear to become active automatically and unconsciously outside working memory’ (p. 619). Kent & Lamberts (2008) echoed this distinction, arguing that whereas explicit simulation involves mental imagery, implicit simulation requires neither ‘consciously experienced analogue reasoning [nor] explicit episodic recall’ (p. 93). Relying on this same explicit–implicit/deliberate–automatic distinction, Gallese (2003) argued that imagery and simulation are wholly distinct constructs and goes so far as to claim that all simulations are implicit and automatic.

The distinction between instrumental simulation and emulative simulation is pertinent here. It is possible that only emulations necessarily rely on working memory. Indeed, ample evidence indicates that people are aware of at least some simulations, and that they intentionally use such simulations (e.g. Clement 1994). Thus, we can easily reject the idea that all simulations are implicit and automatic.4 We cannot reject as easily, however, the possibility that some implicit simulations exist—specifically in implicit memory, high-level perception, sensorimotor coordination, conceptual knowledge (Barsalou 1999, 2003, 2008), language comprehension (Pulvermuller 2005) and social cognition (Gallese 2006).

However, in order to qualify as simulation, and thereby serve as an epistemic device, a simulation must feed into processes used in working memory. Even if an implicit process were to operate via sequential analogy, it would not qualify as an emulation unless it produces consciously accessible information. As Fisher (2006) stated, ‘a simulation is supposed to work by providing an epistemically available process that reflects the relevant aspects of some process that is not so epistemically available’ (p. 419).

In short, we are led to conclude that all mental imagery is mental emulation. However, we do not assert that all mental emulation (or simulation) necessarily must be mental imagery; we leave open the possibility that implicit simulation exists and that it does not rely on imagery.

(c) Advantages of imagery-based mental emulation

Mental emulation via mental imagery offers several functional advantages over instrumental simulation and implicit emulation (if such a thing in fact exists).

For one, because imagery mimics perception (including the perception of movement, both of the body and of objects), it evokes similar associations (including emotional responses), and, in turn, can generate accurate predictions (see Kosslyn 2008). Thus, one can try out alternative scenarios, varying key aspects of an anticipated situation (e.g. the person to whom one asks specific questions). Implicit simulation is by definition rigid: it is a response to a specific stimulus and cannot be varied at will.

In addition, imagery can reveal conceptual knowledge that informs prediction. For example, one can recover information stored tacitly in memory and use that information to guide future behaviour. One example is our ability to visualize spatial layouts and then to use this information to plan routes.

Furthermore, because imagery-based predictions are mediated by working memory (and hence we are aware of them), they can be explicitly reported, shared, remembered and violated.

Finally, imagery can aid prediction by creating or modifying implicit memories. As Bar (2007) noted, ‘We simulate, plan and combine past and future in our thoughts, and the result might be ‘written’ in memory for future use’ (p. 286). In fact, imagery can actually build in conditioned responses (Dadds et al. 1997).

4. Concluding remarks

In this brief article, we have made the case that mental imagery plays a key role in many forms of mental simulation, specifically in emulative simulations. We have by necessity only skimmed the surface. We have not considered, for example, the circumstances in which such mental emulations rely on different forms of imagery, such as object versus spatial. Nor have we considered the alternatives to imagery-based simulation in detail, or reviewed the abundant evidence that imagery-based simulation plays a key role in episodic memory, episodic future thinking, counterfactual thinking, spontaneous cognition and mentalizing. We leave the door open to the possibility that some simulations rely on implicit, non-imagery processes, but we argue strongly that whenever imagery is used, it is used in the service of simulation. One uses imagery to simulate what one would perceive if one were in a specific situation; this is as true of imagery used to retrieve memories as it is of imagery used to predict the future. Mental images are a way to move the world into the head, and then to run models to observe possible implications for the actual world. As such, imagery and simulation are joined at the hip, and should be studied together.