Predictions penetrate perception: Converging insights from brain, behaviour and disorder

Abstract

It is argued that during ongoing visual perception, the brain is generating top-down predictions to facilitate, guide and constrain the processing of incoming sensory input. Here we demonstrate that these predictions are drawn from a diverse range of cognitive processes, in order to generate the richest and most informative prediction signals. This is consistent with a central role for cognitive penetrability in visual perception. We review behavioural and mechanistic evidence that indicate a wide spectrum of domains—including object recognition, contextual associations, cognitive biases and affective state—that can directly influence visual perception. We combine these insights from the healthy brain with novel observations from neuropsychiatric disorders involving visual hallucinations, which highlight the consequences of imbalance between top-down signals and incoming sensory information. Together, these lines of evidence converge to indicate that predictive penetration, be it cognitive, social or emotional, should be considered a fundamental framework that supports visual perception.

Introduction

Visual perception is not a passive or exclusively stimulus-driven process. Instead, there is a proactive interplay between incoming stimuli and predictions based on internally generated models, which shapes our conscious perception of the world around us (Bar, 2004; Bullier, 2001; Engel, Fries, & Singer, 2001). This enables our perceptual system to harness a lifetime of experience with the world, leveraging our past to aid our interpretation of the present. Along these lines, a predictive coding framework for visual perception is well established, fitting with a broader general principle of the brain as a predictive machine (Clark, 2013; Dayan, Hinton, Neal, & Zemel, 1995; Friston, 2010). Echoed across perceptual modalities, motor outputs and learning systems, hierarchically organised top-down pathways convey information about what to expect based on our prior experience. These predictions are combined with incoming sensory information to sculpt the most likely interpretation of the world around us. Sparingly, only mismatches between the descending predictions and lower-level sensory information are carried forward in the form of prediction error signals, to be reconciled at higher levels of the processing hierarchy, where new hypotheses are generated to accommodate the incoming input (Friston, Stephan, Montague, & Dolan, 2014).

Predictive coding is both a biologically and computationally plausible description of information flow in the brain, yet the intimately related process of cognitive penetration in perception is hotly debated (Firestone & Scholl, 2015, in press; Macpherson, 2012; O’Callaghan, Kveraga, Shine, Adams, & Bar, in press; Raftopoulos, 2014; Stokes, 2013; Vetter & Newen, 2014; Zadra & Clore, 2011). Cognitive penetration entails that a myriad of information from other modalities can influence the earlier stages of the perceptual process. Access to a wide variety of information processing systems when generating predictions will ensure that the predictions will be rich and most effective for minimising global prediction error (Lupyan, 2015), emphasising a central role for cognitive penetration in visual perception.

The predictive nature of the visual processing system itself is apparent from the basic tenets of object perception, (Bar et al., 2006; Enns & Lleras, 2008), but there is less clarity regarding the extent to which other cognitive systems shape perception. Here we start by using object recognition to demonstrate the mechanisms of top-down influence on perception. We then extend this framework to show that beyond features intrinsic to an object itself, contextual factors, as well as other cognitive and affective processes, all converge to provide a source of top-down prediction that influences visual perception. We explore these patterns both in social and affective neuroscience, and then describe novel insights on how a failure of this system can manifest neuropsychiatric disorders such as visual hallucinations.

Object recognition

The premise for a predictive coding account of visual perception relies on descending neural pathways throughout the visual processing system (Angelucci et al., 2002; Bullier, 2001). This structure allows for feed-forward projections, originating in the primary visual cortex and ascending via dorsal and ventral streams, to be matched with reciprocal feedback connections (Felleman & Van Essen, 1991; Gilbert & Li, 2013; Salin & Bullier, 1995). Under this scheme, top-down predictions are generated at higher levels and compared with lower-level representations of sensory input, with any resultant mismatch coded as a prediction error (Clark, 2013; Friston, 2005b; Friston et al., 2014; Rao & Ballard, 1999). In the case of object recognition, the orbitofrontal cortex has emerged as a source of such top-down expectations (Bar et al., 2006).

Earlier work characterised the ascending dorsal and ventral visual pathways as dominated respectively by magnocellular and parvocellular cells (Goodale & Milner, 1992; Ungerleider & Mishkin, 1982). The magnocellular (M) pathway carries achromatic, low spatial frequency information rapidly and is sensitive to high temporal frequencies, whereas the parvocellular (P) pathway transmits fine detail in the luminance channel, low spatial frequency information in the colour channel, has slower conduction velocities and can only process lower temporal frequencies (Maunsell, Nealey, & DePriest, 1990; Nowak & Bullier, 1997; Tootell, Switkes, Silverman, & Hamilton, 1988). Studies have since capitalised on the different properties of M and P pathways to reveal the time course and sources of top-down modulation of vision. Using low spatial frequency (LSF) stimuli to preferentially recruit M pathways, a combined approach of magnetoencephalography (MEG) functional magnetic resonance imaging (fMRI) revealed early activity was evident in the orbitofrontal cortex ~130 ms post stimulus presentation, well before object recognition-related activity peaks in the ventrotemporal cortex (Bar et al., 2006). A similar time window for top-down responses to categorising LSF scenes versus HSF scenes, where fMRI showed increased activity in the left prefrontal cortex and left middle temporal cortex, was associated with ERP activation in the 140–160 ms period (Peyrin et al., 2010). Although these time scales do not represent the earliest temporal component in visual processing (i.e., 0–100 ms), they are occurring at the short latency between 100–200 ms, suggesting that these top-down influences on object recognition modulate processing from reasonably early stages.

In order to provide a clearer picture of the spatiotemporal trajectory of top-down feedback, an fMRI study using dynamic causal modelling confirmed that early activation in the orbitofrontal cortex in response to M-biased stimuli then initiated top-down feedback to the fusiform gyrus (Kveraga, Boshyan, & Bar, 2007). P-biased stimuli in the same study were associated with a different connectivity pattern where only feedforward information flow increased between occipital cortex and fusiform gyrus. By comparison, orbitofrontal cortex activity only predicted M- and not P-biased stimuli, which resulted in faster recognition of M stimuli by ~100 ms. Similar evidence of early orbitofrontal activation preceding temporal lobe activity has been found when individuals make initial judgements about the coherence of degraded visual stimuli (Horr, Braun, & Volz, 2014) as well as during categorisation of face versus non-face images (Summerfield et al., 2006). These studies highlight the orbitofrontal cortex as an origin for top-down effects that can modulate processing in earlier visual processing areas.

The above findings support a predictive coding account of object recognition, whereby cursory visual information is sufficient to engage a memory store of information about the basic category of the object, used to generate the most likely prediction about the object’s identity (Bar, 2003; Oliva & Torralba, 2007). Rapidly extracted predictions about the general category of an object is fed back to lower levels of the visual processing hierarchy to enable faster recognition by constraining the number of possible interpretations. Fitting with the predictive coding framework that we generate top-down hypotheses based on our knowledge of stored regularities in the world, another fMRI study revealed this orbitofrontal cortex facilitation of object recognition was triggered only for meaningful LSF images (Chaumon, Kveraga, Barrett, & Bar, 2014). Meaningful LSF images resulted in a top-down feedback signature of increased functional connectivity between the orbitofrontal cortex and ventral visual pathway, whereas meaningless images did not. This suggests that generating strong top-down predictions requires connection with a learned representation of an object. In typical circumstances, guiding perception with top-down expectations facilitates the accuracy and speed of object recognition. When sensory input is ambiguous, top-down guidance is even more important for supplying information to resolve the ambiguity (O’Reilly, Wyatte, Herd, Mingus, & Jilk, 2013; Panichello, Cheung, & Bar, 2012). In a predictive coding framework, the integration of top-down signals with the feed-forward information flow is an iterative process geared toward progressive error reduction. Initial feed-forward sweeps associated with coarse (LSF) categorical processing are followed by graded HSF recurrent processing, which reflects the formation of increasing detailed recognition judgments (Clarke, Taylor, & Tyler, 2011; Poch et al., 2015). Such a pattern supports the idea that more complex information may require further iteration in the error reduction process.

Over and above this predictive mechanism characterising visual perception, considering the orbitofrontal cortex as an origin site for top-down effects reveals how diverse information streams may penetrate visual processing from the initial stages. The orbitofrontal cortex is well connected with the visual system, receiving projections from the temporal visual regions (inferior temporal cortex, superior temporal sulcus and temporal pole) (Barbas, 1988, 1995; Carmichael & Price, 1995) and is a target of visual information relayed via the pulvinar (Bridge, Leopold, & Bourne, 2015; Guillery & Sherman, 2002). Recordings in non-human primates reveal neuronal responses to visual stimuli in the orbitofrontal cortex (Rolls & Baylis, 1994; Thorpe, Rolls, & Maddison, 1983). Further to this, however, the orbitofrontal cortex is a richly connected association region, receiving inputs from visceral, limbic and other sensory modalities (Rolls, 2004). The orbitofrontal cortex is therefore ideally positioned to integrate cross-modal information relevant to the interpretation of incoming visual stimuli. Studies relating orbitofrontal activation to top-down modulation of object recognition chiefly identified activation in the inferior orbitofrontal cortex (Bar et al., 2006; Bar et al., 2001; Chaumon et al., 2014). This region overlaps with the lateral orbitofrontal region (area 12) where sensory input from the visual system terminates (Kringelbach & Rolls, 2004).

The orbitofrontal cortex has been associated with an array of functions, including inhibition, emotion regulation and reward processing. Evidence suggests the unifying computational nature of the orbitofrontal cortex is to signal the predicted value of choice or action outcomes, by encoding both the sensory features of outcomes and their biological value relative to the current state of the organism (Passingham & Wise, 2012; Rudebeck & Murray, 2014; Schoenbaum, Roesch, Stalnaker, & Takahashi, 2009). Exactly how this computational process supports early categorical predictions in object recognition is not yet clear, but activity related to expected versus observed outcomes during perceptual decision making has been observed in the orbitofrontal cortex (Summerfield & Koechlin, 2008), consistent with a comparative process extending to visual predictions. The inter-connectedness of the orbitofrontal cortex may allow object interpretations to be evaluated on the combined basis of sensory information, emotional information and information regarding an organism’s current internal state and environmental context. The convergence of sensory, emotional and contextual information is consistent with more information-rich prediction error signals being a most effective means of minimising global prediction error. The winning interpretation is then fed back to bias processing in earlier visual regions (Trapp & Bar, 2015). In the following section we consider contextual associations as one of the critical sources of information harnessed in ongoing visual processing.

Contextual processing

A fundamental assumption is that our cognitive and perceptual processes do not operate in a vacuum, but are subject to the constraints of a changeable environment that requires flexible and context-dependent behaviour (Engel et al., 2001). Responses to contextual cues dictate much of human behaviour, from influencing our social interaction to determining levels of reward seeking or avoidant behaviours (Bartz, Zaki, Bolger, & Ochsner, 2011; Koob & Volkow, 2010; Pennartz, Ito, Verschure, Battaglia, & Robbins, 2011). An ability to both encode contextual associations and utilise them to guide behaviour is hard-wired across animals and humans, and visual perception leverages this same mechanism. In this way, scene perception can be understood in terms of the broader cognitive mechanism of associative processing (Aminoff & Tarr, 2015; Bar, 2004; Summerfield & Egner, 2009).

Visual objects tend to occur in stereotypical settings. For example, before even walking into a bathroom we could predict seeing certain objects – a shower, basin, lavatory, towels, and so on. Experience affords the opportunity to learn such associations, which can in turn become a rich source of information to guide predictions about object identification. In the case of ambiguous visual input, information about the contextual surrounding will facilitate its interpretation (Bar & Ullman, 1996). For example, the same object can be perceived as a hairdryer or a drill, depending on whether it appears in a bathroom or a workshop context (Bar, 2004). Regions anchored in the ventral visual stream, such as the parahippocampal cortex, along with hippocampal and retrosplenial complexes, have a well-established shared involvement in place-processing and episodic memory (Burgess, Maguire, & O’Keefe, 2002; Epstein, Harris, Stanley, & Kanwisher, 1999). Studies exploring contextual processing in visual perception unified these disparate functions to reveal their role for penetration in the visual processing system, as described next.

By contrasting stimuli that are strongly versus weakly associated with a specific context, or by training associations between novel visual stimuli, a network of activation across parahippocampal, retrosplenial, and medial orbitofrontal cortices has been uniquely associated with contextual processing (Aminoff, Gronau, & Bar, 2007; Aminoff, Kveraga, & Bar, 2013; Bar & Aminoff, 2003). In contrast to the sites associated with early processing of LSF information, more medial areas of the orbitofrontal cortex were identified in relation to contextual processing. Although broadly involved in the computational processes outlined earlier, a more fine-grained distinction is that lateral orbitofrontal regions evaluate options independently of each other, and medial regions are involved in comparing the options to directly guide decisions (Rudebeck & Murray, 2014). Consistent with a comparative process that draws upon multiple information streams, activity in overlapping regions of the medial orbitofrontal cortex tracks both an object’s valence and its associative strength (Shenhav, Barrett, & Bar, 2013). Continued careful examination of the exact role of the orbitofrontal cortex in generating predictions to guide visual processing may reveal a lateral-medial gradient related to the information processing demands of the incoming visual stimulus. The significance of context in object processing is further highlighted when identical objects that were included in different task sets (i.e., manipulating the context) resulted in different patterns of ventral temporal and prefrontal cortex (Harel, Kravitz, & Baker, 2014). In this case, activation in higher processing regions reflected the nature of the task set (context) rather than object identity.

Critically for a predictive coding account, MEG with phase synchrony analyses revealed that top-down contextual influences occur during the formation stages of a visual percept, extending all the way back to early visual cortex (Kveraga et al., 2011). Substantiating an account of top-down modulation drawn from associative processing, the reduced responses in early visual areas shown in the context of cued or expected visual stimuli (Marois, Leung, & Gore, 2000; Summerfield & Egner, 2009; Yoshiura et al., 1999) result in sharper representations of the expected stimulus in those early regions (Kok, Jehee, & de Lange, 2012) – suggesting that the learnt association directly affects the perceptual process. This has been interpreted as neural activation of expected events being silenced as it is constrained by prediction feedback from higher-level regions (Murray, Kersten, Olshausen, Schrater, & Woods, 2002; Summerfield & de Lange, 2014)

In object recognition, the orbitofrontal context emerges as an origin of neural prediction that incorporates cross-modal information and utilises associative memory, in concert with parahippocampal and retrosplenial complexes. Top-down information from these higher-level regions is consistent with an initial global ‘context signal’, which incorporates a range of information from LSF input and learned associations to facilitate perceptual processing in earlier visual areas (Trapp & Bar, 2015). This rich set of information and associations penetrates the earlier processing stream in the form of top-down predictions about the most likely interpretation of the visual world. However, the scope of information that penetrates visual perception is by no means limited to features intrinsic to an object or scene. Information is drawn from a range of sources in order to formulate the predictions that mould ongoing visual processing. Predictions in visual perception are drawn not only from learned associations and context, but also from cognitive biases, interoceptive feedback and affective states, as emphasised by the roles of social cognition and emotion in visual processing.

Social cognitive penetration

There is a long history in social psychology of empirical work in favour of the claim that social, cultural and motivational factors can directly influence visual perception (Bruner, 1957). For instance, social status is found to affect perceptions of the size of coins (Bruner & Goodman, 1947). Motivational factors that centre on how desirable an object is to the perceiver are suggested to influence perception of the size, distance, steepness, salience, and brightness of visual stimuli (Alter & Balcetis, 2011; Balcetis & Dunning, 2006; Banerjee, Chatterjee, & Sinha, 2012; Bhalla & Proffitt, 1999; Den Daas, Häfner, & de Wit, 2013; Radel & Clément-Guillotin, 2012; Song, Vonasch, Meier, & Bargh, 2012). Social categorisation, including judgements about gender and race, is also thought to be strongly influenced by well-learnt cultural stereotypes (Correll, Wittenbrink, Crawford, & Sadler, 2015; Levin & Banaji, 2006; MacLin & Malpass, 2001; Macrae & Martin, 2007; Payne, 2006).

Many of these behavioural findings, however, have been effectively criticised as failing to provide true evidence of dynamic cognitive penetration of visual perception. Pitfalls in the methodological approaches of many of these studies mean that task-specific effects of judgement, memory and response bias cannot be ruled out as potential explanations of the findings, as opposed to being true examples of top-down effects altering visual perception (Firestone & Scholl, 2014, in press). This criticism highlights the continued need to validate behavioural evidence of cognitive penetration in vision against neural markers, to establish whether they are indeed consistent with top-down effects on ongoing visual processing.

Electrophysiological and fMRI studies of face perception, which is central to social interactions, go some way to support a role for diverse social cognitive effects on visual perception. When making judgements about the gender of a face, activity in the fusiform cortex tracks closely with the linear gradations between morphed male and female faces, whereas orbitofrontal cortex responses instead reflect subjective perceptions of face gender (Freeman, Rule, Adams, & Ambady, 2010). Greater neural activity in upstream top-down processing areas, including the fusiform face area of the ventral visual stream and the orbitofrontal cortex, has been found in response to faces from a individual’s own ‘group’ (i.e., their racial group, or a group they have been arbitrarily assigned to by an experimenter) (Golby, Gabrieli, Chiao, & Eberhardt, 2001; Van Bavel, Packer, & Cunningham, 2008). Neural activity in the fusiform face area during processing of ‘in-group’ faces is functionally distinct from the pattern of activity in lower-level processing regions of the primary visual cortex (Van Bavel, Packer, & Cunningham, 2011). Consistent with the effects of ‘in-group’ categorisation influencing visual processing from fairly early stages, the N170 component of the event-related potential (i.e., the earliest component reflecting perceptual processing of a face (Rousselet, Husk, Bennett, & Sekuler, 2008)) is larger for ‘in-group’ faces (Ratner & Amodio, 2013). Together, these findings support that social category cues modulate early visual processing of faces, and that the influence of these top-down effects is particularly evident in the fusiform gyrus of the ventral visual stream (Amodio, 2014). Importantly, these neural sites supporting top-down effects in facial processing overlap with those identified for object recognition and contextual processing.

It is perhaps not surprising then that social cues, such as gender, culture, race and eye gaze have all been shown to influence the perception of, and neural responsiveness to, facial displays of threat (Adams, Gordon, Baird, Ambady, & Kleck, 2003; Adams, Hess, & Kleck, 2015; Adams & Kleck, 2005). Consistent with the notion that the visual system might be penetrable by a range of information processing streams, such effects have been found to be modulated by individual differences in trait anxiety and progesterone levels; this is presumably linked to threat vigilance, as demonstrated in both perceptual tasks (e.g., (Conway et al., 2007; Fox, Mathews, Calder, & Yiend, 2007) and in amygdala responsiveness to threat cues (e.g., (Ewbank, Fox, & Calder, 2010)).

Other social factors that are claimed to influence ongoing visual perception, such as morality and social norms, remain to be validated against neuroimaging and electrophysiological markers to substantiate that they do exert top-down effects on visual processing (Gantman & Van Bavel, 2015). In time, social vision may emerge as a particularly salient example of cognitive penetration of visual processing, as social interactions are amongst the most information-rich human activities. Drawing from a wealth of cognitive resources to help interpret social situations would be an effective way of generating top-down predictions and lowering global prediction error. This is likely to be particularly relevant in social vision, where we are required to interpret complex, and often very subtle, social cues.

Emotional penetration

Until quite recently, emotion was thought to be independent (and usually studied separately) from perception and cognition – something that came after the stimulus was recognised. Rather than being separate or competing processes, however, emotion and perception interact to maximise the probability of correctly identifying a stimulus. In this way, a person’s affective state is presumed to be a source of top-down penetration of visual perception (Barrett & Bar, 2009). Effects of valence during visual perception are detectable at a reasonably early time scale, with the earliest ERP modulations as a function of valence emerging at 120–160 ms (Carretié, Hinojosa, Martín-Loeches, Mercado, & Tapia, 2004; Olofsson, Nordin, Sequeira, & Polich, 2008; Schönwald & Müller, 2014; Smith, Cacioppo, Larsen, & Chartrand, 2003). Rapid extraction of affective information about a visual stimulus may be highly adaptive, signalling whether an object should be approached or avoided (Barrett & Bliss-Moreau, 2009). Multiple neural routes exist whereby the affective value of a stimulus can be rapidly processed, including the magnocellular system and projections to the orbitofrontal cortex (Pessoa & Adolphs, 2010).

Supporting the integration of affective state and conscious visual awareness, in a binocular rivalry paradigm pairing faces against houses, unseen faces with an affective value (i.e., scowling or smiling) modulated perception of neutral faces, such that they were imbued with affective value (Anderson, Siegel, White, & Barrett, 2012). The temporal course for visual information entering conscious awareness is also found to be modulated by positive versus negative mood induction (Kuhbandner et al., 2009). These effects are apparent at the neural level, where negative mood induction produced lower recognition reaction times and decreased the latency of responses in a number of visual and affective brain regions, from the primary visual cortex, to fusiform, medial temporal and insular cortex, to the orbitofrontal cortex (Panichello, Kveraga, Chaumon, Bar, & Barrett, under revision). These effects of emotional valence became apparent at around 80–90 ms in V1 and LOC, suggesting at least some early-stage penetrability of early and mid-level visual regions (Panichello et al., under revision). Evidence from fMRI suggests that the orbitofrontal cortex is a site for triggering top-down predictions about the affective value of visual stimuli, as activity in overlapping regions in the medial orbitofrontal cortex indexes both an object’s associative strength and whether is has positive or negative valence (Shenhav et al., 2013). The amygdala may be providing a critical source of input for orbitofrontal cortex computations about the affective value of stimuli, given the amygdala’s connectivity with the orbitofrontal cortex, its position as a target of visual pathways and its role in encoding emotionally salient information (Pessoa & Adolphs, 2010; Roy, Shohamy, & Wager, 2012). Consistent with ERP time scales cited above describing the time course of valence effects in human vision, cellular recordings from the monkey amygdala in response to visual stimuli range from 100–200 ms (Gothard, Battaglia, Erickson, Spitler, & Amaral, 2007; Leonard, Rolls, Wilson, & Baylis, 1985; Nakamura, Mikami, & Kubota, 1992), with differential effects depending on valence detectable in the range of 120–250 ms (Gothard et al., 2007).

Affective predictions play an important role in facilitating visual recognition of not just familiar faces and objects, but also in quickly identifying danger. Viewing threatening visual stimuli is accompanied by increased activity in regions implicated in threat detection and initiating “fight-or-flight” responses in the sympathetic system, such as the amygdala, periaqueductal grey, and the orbitofrontal cortex (Kveraga et al., 2015). However, the activity in these regions, as well as behavioural threat ratings, were found to be dependent on the context of the threat: images of someone handling a gun at a shooting range vs. being robbed at gunpoint evoked vastly different response patterns. Threatening objects or animals oriented towards the observer heightened the personal feeling of danger compared to when they were oriented towards someone else. Moreover, in a threat detection task, context influenced how quickly threatening objects or animals were recognised: threatening objects and animals were recognised quickest in direct-threat contexts (Kveraga et al., 2015, Supp. Mat.). These findings suggest that affective information in threatening situations is integrated with visual processing, modulating both how objects are interpreted and how quickly they are recognised.

Another avenue supporting the penetration of affective information in ongoing visual processing is derived from neuropsychiatric syndromes where visual recognition is impaired in the context of reduced emotional certainty about a decision. Patients with Capgras syndrome have apparently intact visual recognition abilities (though their recognition latencies are unknown), but insist their family members and friends (and sometimes, familiar objects) have been replaced by impostors or “doubles” (Anderson, 1988; Ellis & Lewis, 2001; Ellis, Young, Quayle, & De Pauw, 1997). Capgras syndrome can emerge in the context of varied disorders and its neural substrate is not precisely defined (Thiel, Studte, Hildebrandt, Huster, & Weerda, 2014). A consistent finding in these patients, however, is the absence of an increased skin conductance response when viewing familiar faces, indicating no covert recognition by their autonomic system (Ellis et al., 1997). This highlights an integrative role for affective responses and visual processing facilitating complete recognition – i.e., beyond recognising a physical likeness, but recognising a person or object as one that is known and familiar.

Visual hallucinations

It becomes clear from the preceding sections that given the multiple avenues of top-down penetration into visual processing, accurate perception requires striking a balance between the weighting assigned to our predictions versus that assigned to incoming sensory input. Visual hallucinations illustrate the consequences of an imbalance in this system. To be adaptive, the relative influence of top-down versus bottom-up sensory input must be flexible and context-sensitive (Lupyan & Clark, 2015). For example, navigating our homes in darkness we will rely heavily on a stored model of the topography and less on sensory information, but in a novel, challenging environment – say, navigating precipitous terrain on a bushwalk – reliance on sensory information is paramount, though not exclusive. Predictive coding frameworks allocate precision weighting measurements that signal how much confidence should be assigned to prediction errors, in effect this allows information from the most relevant channels to be weighed higher (Friston et al., 2014). An emerging theme in neuropsychiatry is that imbalances in this system cause undue reliance on either top-down or on incoming sensory information, contributing to a range of symptoms (Corlett, Honey, Krystal, & Fletcher, 2011; Fletcher & Frith, 2009; Stephan et al., 2015; Teufel et al., 2015). Hallucinations are linked to a relative increase in the weighting of prior predictions (Adams, Stephan, Brown, Frith, & Friston, 2013; Friston, 2005a), such that top-down influences tend to dominate visual processing in hallucinations and erroneous perceptions prevail in the face of contradictory sensory evidence. The predictive coding framework provides a mechanistic account of how aberration in the visual processing hierarchy results in hallucinations. However, considering the nature of cognitive penetration in visual perception gives a much richer picture of the qualitative aspects of visual hallucinations and clues to their underlying anatomy.

Phenomenological enquiry in conditions that manifest visual hallucinations, including schizophrenia, psychosis, Parkinson’s disease and dementia with Lewy bodies, suggests that the content of hallucinatory percepts can be furnished by contextual associations drawn from the environment or by autobiographical memories. For example, identifiable faces or objects can be construed from ambiguous scenery as is the case with pareidolic hallucinations (Uchiyama et al., 2012), and hallucinations of familiar people or pets are commonly reported (Barnes & David, 2001). Furthermore, the frequency and severity of visual hallucinations has been linked to mood and physiological states (e.g., stress, depression and fatigue), with mood also playing an important role in determining the content of hallucinations (e.g., when the image of a deceased spouse is perceived during a period of bereavement) (Waters et al., 2014). Together, the influence of contextual associations, autobiographical memories and emotion on the hallucinatory percept dovetails with the above sections to show that cognitive penetration of visual processing accommodates diverse information streams.

Convergent evidence across disorders suggests that abnormal activity in those regions supplying top-down influences during normal visual processing, identified in the previous sections to include medial temporal and prefrontal sites, is instrumental in generating hallucinatory visual percepts (Shine, O’Callaghan, Halliday, & Lewis, 2014). Hippocampal and parahippocampal regions of the medial temporal lobe, combined with midline posterior cingulate and medial frontal sites comprise the default network (Andrews-Hanna, Reidler, Sepulcre, Poulin, & Buckner, 2010). This functionally defined network sits at the nexus of dorsal and ventral attentional networks responsible for top-down direction of attention and orientation to salient events, respectively (Corbetta & Shulman, 2002; Fox et al., 2005). Across disorders with visual hallucinations increased instability, or elevated activity and connectivity, within the default network have been identified (Franciotti et al., 2015; Jardri, Thomas, Delmaire, Delion, & Pins, 2013; Shine et al., 2015a; Yao et al., 2014). More broadly this is in keeping with observations that an over-active/poorly supressed default network may underscore many positive and ruminative symptoms in neuropsychiatric disorders (Anticevic et al., 2012; Whitfield-Gabrieli & Ford, 2012). Collectively the default network is associated with memory, self-referential processing, construction of mental imagery and creative cognition, mind wandering and emotional processing (Beaty, Benedek, Silvia, & Schacter, 2015; O’Callaghan, Shine, Lewis, Andrews-Hanna, & Irish, 2015).

As outlined above, key nodes in the default network, including the medial prefrontal cortex and parahippocampal cortex, are sources of the top-down input that directly influences ongoing visual perception. It follows that abnormal activity in these input zones could conceivably produce the combination of visual, autobiographical, and emotive qualities that characterise complex visual hallucinations (O’Callaghan, Muller, & Shine, 2014). Importantly, the emergence of hallucinatory phenomena with abnormal activity in these sites has been verified independently in direct electrical stimulation studies conducted during neurosurgery. Stimulation to medial temporal areas, including parahippocampal regions, as well as to the prefrontal cortex, is capable of generating complex visual hallucinations (Blanke, Landis, & Seeck, 2000; Mégevand et al., 2014; Selimbeyoglu & Parvizi, 2010). Aside from abnormal activity in default network regions emerging as a neural signature, patients with visual hallucinations due to Parkinson’s disease and schizophrenia exhibit higher functional and structural connectivity from the visual cortex to medial temporal and medial prefrontal sites (Amad et al., 2014; Ford et al., 2014; Yao et al., 2015) and increased coupling between visual cortex and default network (Shine et al., 2015b). Hyperconnectivity of these pathways suggests a possible route whereby top-down recurrent feedback may be overactive, and dominating early visual regions in people with chronic visual hallucinations.

Examining cognitive penetration in visual hallucinations at the macro scale of brain regions and large-scale neural networks gives important clues to the brain systems involved and explains the convergence of sensory, memory and emotional based content that comprises hallucinations. More research is needed to marry this with neural function at the micro scale associated with predictive coding and precision weighting. Current evidence suggests that pyramidal cells are critically involved in comparing prediction error against top-down expectations at each level of the processing hierarchy, but neuromodulatory control flexibly influences the weighting (or precision) of these comparisons (Friston et al., 2014). Therefore ‘leaky’ systems that result in improper weighting of incoming sensory versus top-down input in perception could result from neuropathology at levels of the cortical hierarchy or imbalances in neuromodulatory systems. Emerging evidence from schizophrenia has begun to relate predictive coding disruptions to neural changes at high levels in the processing hierarchy, including prefrontal cortex and medial temporal lobe, as well as disruption to key neuromodulators (glutamate, GABA and dopamine) (Adams et al., 2013). Continued understanding of the specific predictive coding deficits in visual hallucinations will help explain what causes the system to be so vulnerable to cognitive penetration into the visual perception process.

The timescale of top-down penetration in vision

Much of the empirical evidence we have discussed here has not directly addressed the timescale of cognitive penetration in vision, and where timing of effects have been reported they are in the vicinity of 120–200 ms post stimulus onset. Indeed, it has been argued that the first 100 ms of visual processing are cognitively impenetrable (Raftopoulos, 2009, 2014), and there is certainly scarcer evidence to support penetration of this very early processing stage. However some investigations have highlighted that the C1 component (originating in V1, with onset around 50 ms post-stimulus and peak latencies before 100 ms) is penetrable by top-down influence (Rauss, Schwartz, & Pourtois, 2011). For example, C1 amplitude is increased when subjects actively attend to a location (Kelly, Gomez-Ramirez, & Foxe, 2008), and using MEG with source localisation modulation of V1 as a function of attentional engagement is evident within the first 100 ms post-stimulus (although not the first 50 ms) (Poghosyan & Ioannides, 2008; Poghosyan, Shibata, & Ioannides, 2005). Aside from attentional engagement, modulation of the C1 is also found in response to attentional load (Rauss, Pourtois, Vuilleumier, & Schwartz, 2012; Rauss, Pourtois, Vuilleumier, & Schwartz, 2009). These human findings are supported and extended by cell recordings in monkeys that show effects of attention (Ito & Gilbert, 1999) and task context (Li, Piëch, & Gilbert, 2004) are evident in V1 from the very onset of neural response. Such findings raise the possibility that V1 is not impenetrable, but that cells there are dynamic and their functional properties and ensemble organisation is influenced by top-down effects (Gilbert & Li, 2013; Ramalingam, McManus, Li, & Gilbert, 2013) and that these effects are evident from the earliest stages.

It is important to note, however, that the detection of these very early top-down effects is much less robust that those top-down effects detected after 100 ms (Rauss et al., 2011). Furthermore, support for very early top-down influences mostly comes from studies where attention is found to have modulatory effects. Attention is, arguably, not necessarily an example of cognitive penetration as its effect on perception may only be the extent to which it influences the visual input we receive, but not directly modulating how the visual processing stream operates (Firestone & Scholl, in press). As attention could represent a pre-perceptual process, to help resolve the current debate about cognitive penetrability in very early vision more thorough investigation of different cognitive processes is warranted. Given the inconsistencies in detecting effects at this time-scale (Rauss et al., 2011) there may also be methodological challenges to be addressed. One possibility is that the effects of very early top-down modulation on V1 may be mostly evident in the subtle tuning of cells’ functional properties (Gilbert & Li, 2013; Nienborg & Roelfsema, 2015), as opposed to net effects of increased neural/BOLD activity that are documented in higher cortical regions to indicate the origins of top-down effects. Careful consideration of how to detect these potentially subtler effects in human studies is needed. However, one could speculate that, given the initial evidence and the fact that the neural architecture is in place to support penetration even at these earliest levels, additional evidence of cognitive penetration at this earliest stage may be uncovered.

Summary and outlook

The existence of top-down predictions in visual processing has received substantial empirical support, and fits with the broader notion of predictions as a framework for explaining much of the information processing in the brain. However, the penetration of other cognitive and behavioural processes into ongoing visual perception is often contested. The basis of top-down predictions is to utilise past experience to streamline visual processing, it follows that information-rich top-down signals will be best suited to forming accurate predictions. Cognitive penetration into visual perception enables diverse sources of information to contribute to the formulation of predictions that influence ongoing visual perception.

We have described examples at the behavioural level demonstrating that affective states and cognitive biases can alter visual perceptual judgements. At a mechanistic level, a prominent role for the orbitofrontal cortex emerges as a site for early generation of top-down predictions, alongside higher-level visual regions in the lateral temporal cortex, and subcortical memory and limbic structures (hippocampus; amygdala). Much of the work to date characterising orbitofrontal function has focused on value-based decision making. An important goal for future research in visual processing will be extending what is currently known about orbitofrontal computational mechanisms (signalling and evaluating potential outcomes or choices), to prediction in vision. The involvement of the orbitofrontal cortex in early top-down predictions for vision may ultimately be a similar computational process of signalling potential interpretations of a visual stimulus and selecting the most tenable. However, accommodating insights drawn from reinforcement theories of choice and behaviour into a predictive processing framework remains an ongoing challenge (Pezzulo, Rigoli, & Friston, 2015).

Going forward, the work reviewed here highlights the continued need to validate claims of cognitive penetration on vision against neural markers. This is particularly relevant for behavioural experiments in social and emotional vision where methodological pitfalls have called in to question whether these experiments reveal true top-down effects (Firestone & Scholl, 2014, in press). An obvious extension would be to combine these behavioural paradigms with temporally sensitive brain imaging and recording techniques, to establish neural evidence of top-down effects. The ongoing debate regarding the time course of cognitive penetration in vision emphasises a need to combine both spatial (e.g., fMRI) and temporal (e.g., EEG, MEG) information sources. Furthermore, work in non-human primates can make valuable contributions to this debate; however future studies using simultaneous recordings in both higher-level regions and V1 sites are needed to assess both the origins and targets of top-down effects (Nienborg & Roelfsema, 2015).

The broad range of cognitive and affective information streams that converge to influence visual perception emphasises the highly dynamic and adaptive nature of visual perception. Neuropsychiatric conditions, however, confirm that this system relies on a finely tuned balance between top-down predictions and sensory input, where imbalances can have significant consequences. The various lines of evidence discussed here suggest that predictions in visual perception are intimately tied to a process of cognitive penetrability, as the very formation of those predictions is not only based on features intrinsic to a visual stimulus, but also on prior knowledge and current affective state. Going forward, prediction and cognitive penetration should be considered as complementary processes and mechanistic frameworks to describe them must account for the vast range of information that is encapsulated in top-down predictions for vision.