Hallucinations as top-down effects on perception

Abstract

The problem of whether and how information is integrated across hierarchical brain networks embodies a fundamental tension in contemporary cognitive neuroscience, and by extension, cognitive neuropsychiatry. Indeed, the penetrability of perceptual processes in a ‘top-down’ manner by higher-level cognition—a natural extension of hierarchical models of perception—may contradict a strictly modular view of mental organization. Furthermore, some in the cognitive science community have challenged cognitive penetration as an unlikely, if not impossible, process. We review the evidence for and against top-down influences in perception, informed by a predictive coding model of perception and drawing heavily upon the literature of computational neuroimaging. We extend these findings to propose a way in which these processes may be altered in mental illness. We propose that hallucinations - perceptions without stimulus - can be understood as top-down effects on perception, mediated by inappropriate perceptual priors.

Introduction

Imagine you are walking home on a warm early summer night. The sights and sounds that greet you are familiar—the bark of your neighbor’s dog, the old oak tree on the corner, the echo of your footsteps as you get closer to your destination. Now imagine you are walking the same route after watching a scary movie. The same things might now seem strange and menacing to you. The dog’s bark might seem like a growl; the oak tree’s shadows may seem more prominent; those echoed footsteps might sound louder—or maybe they might not seem to be your own. Your aroused state makes you search for hidden threats, and your beliefs guide where you search (1). Could your fear even lead you to hallucinate footsteps when there are none? We consider these questions of how cognition alters perception, in light of recent advances in computational psychiatry.

Present-day cognitive scientists have argued that cognition does not influence perception (2). However, work in computational neuroscience calls this claim into question. We will argue that hallucinations too challenge strict, encapsulated modularity. Instead, they indicate penetrability of perception by cognition. We will illustrate these claims with phenomenology as well as neuro-computational work.

Modules and the Mind

In The Modularity of Mind (1983), Jerry Fodor sketched a blueprint of mental architecture comprised of modules—systems that process a single specific kind of information (3). For example, the early vision module takes in ambient light and outputs color representations. Fodor never gave a specific definition of a module (nor have other modularists), which makes the theory difficult to falsify. Twyman and Newcombe write: “Given this lack of agreed-upon definition, the modularity position becomes analogous to the Hydra, the many-headed monster that Heracles found difficult to combat because there were too many heads to take on simultaneously, and, worse, because other heads grew while he addressed a specific one” (4). Ultra-cognitive neuropsychologists even claim to shun brain localization, calling it mere hardware and irrelevant to the software they are interested in (5). However, even the most ardent ultra-cognitivists utilize some lesion location information in rendering their arguments (6).

A strictly modular approach focuses on functional segregation, with brain regions responsible for discrete mental faculties that can be damaged in isolation (7). Such an approach eschews functional integration, which posits that complex mental functions are based on interactions among distributed regions (7). Human lesion cases also support integration (8–10). We appeal to functional and effective connectivity data for insights into integration. We view this work via a model of mind and brain that itself challenges encapsulated modularity—namely, predictive coding (11).

Predictive Perception Implies Cognitive Penetration

An encapsulated perceptual system, kept separate from the influence of beliefs, could have the advantage of keeping our beliefs grounded in the truth offered by our senses (12). However, a cognitively penetrable perceptual apparatus may be equally adaptive, despite misperceiving and misbelieving, as long as the resulting behavior is adaptive (13, 14). Hume (15) and Helmholtz (16) appreciated this: we perceive what would need to be present in order for our sensations to make sense. To solve this inverse problem the brain uses an internal generative model of its environment to infer what it is sensing (16), combining feed-forward “bottom-up” information from sensory organs with feedback or “top-down” predictions from higher-level regions to compute precision-weighted prediction errors that guide formation of an optimal estimation of the surroundings (17–20). Combining top-down expectation and bottom-up input to explain perception has a rich history. McClelland and Rumelhart proposed early models with this motif (21, 22). Rao and Ballard added neural data and Bayes theorem (23). Maia and Cleeremans proposed that perception solves a ‘global constraint satisfaction’ problem via the interplay between current top-down prefrontal cortical modulation and prior knowledge through learned synaptic connections across a hierarchy (24). Friston first described how these mechanisms may be generalized to a broad model of brain function in terms of predictive coding (25).

Contrary to encapsulated modularity, some studies claim that early visual processing (i.e., not “post-perceptual processing”) is influenced by non-perceptual information (27–30). For example, semantic priming increases the speed and accuracy of detection by minimizing prediction error (28). Behavioral and neurophysiological evidence shows prediction error signals generated in early visual regions in response to violations of semantic expectation (31, 32). Word contexts result in ambiguous shapes being perceived as the missing letters that complete a word (33, 34). Semantic categories including letters and animals improve accuracy and response-times in orientation identification (28, 35). Audiovisual integration induces response changes in primary sensory cortices, such that auditory stimuli engage V1 and visual stimuli activate A1 (36). These activations evolve via prediction error-driven learning (36). These phenomena challenge the informational encapsulation of perception (11).

The Burden of Proof: Establishing Top-Down Influences in Perception

Studies that comprise the so-called New Look movement, purporting to demonstrate effects of language and culture on perception, have recently come under scrutiny. Firestone and Scholl (2) asserted such studies may be plagued by confounds that can be avoided by following these guidelines: 1) Disentangle perceptual from decisional processes; 2) Dissociate reaction time effects from primary perceptual changes; 3) Avoid demand characteristics 4) Ensure adequate low-level stimulus control; and 5) Guarantee equal attentional allocation across conditions.

These suggestions address issues inherent to tasks where perception guides a behavioral decision. However, Bayesian formulations do not accept such separation (37). Signal detection theory appears to sharply distinguish sensation from decision. However, it too allows cognition to influence perception (38). Top-down processes can even alter the mechanical properties of sensory organs (39) by altering the signal-to-noise ratio (40). As we will argue in the following sections, top-down influences may be clearest when sensory input is completely absent: when experiences are hallucinated.

Hallucinations as Examples of Top-Down Penetration

Hallucinations (41) can have contents consistent with one’s affective state (42). When people are depressed, hallucinations can contain themes like guilt and disease. Those experienced during mania center around extraordinary powers (43). Changes in the content of hallucinations can be wrought by experimental mood manipulations (44). Thus affect may penetrate perception. However, auditory-verbal hallucinations (AVH) represent a derangement of normal function. Perhaps perception is normally impenetrable.

The existence of “non-clinical voice-hearers”—who have auditory hallucinations but do not reach clinical criteria for a psychotic disorder—argue against this hypothesis. Hallucinations of a loved one are common following bereavement (45–47). They are typically comforting and do not impair functioning (45, 46) (although see (47)); thus, hallucinations may not be abnormal per se. Non-clinical hallucinations also occur in the general population (48–52). Estimates of their prevalence are as high as 28% (53) and only 25% of those meet the diagnostic criteria for a psychotic disorder (54). Thus, hallucinations may best be described as an extreme of normal functioning (48) rather than a failure of modularity.

Are hallucinations top-down processes? In a recent investigation (55), prior knowledge of a visual scene conferred an advantage in recognizing a degraded version of that image. Patients at risk for psychosis were particularly susceptible to this advantage. A bias toward top-down information is the basis for “sensory conditioning” (56–60), wherein a visual stimulus is established as a predictor of a difficult-to-detect auditory stimulus and participants begin to report auditory stimuli that were not presented on the basis of the visual cue. This effect is amplified in individuals who hallucinate (57). Experiences of uncertainty can increase top-down effects. When a participant’s sense of control over the environment is intentionally decreased (with spurious feedback), they tend toward illusory pattern perception, seeing nonexistent signal in noise and detecting illusory trends in the stock market (61).

The guidelines proffered by Firestone and Scholl (2) may serve as a useful roadmap for future studies of perceptual decision-making tasks. However, studying the penetrability of perception by way of hallucinatory experiences may circumvent these pitfalls. Participants report spontaneous, vivid experiences rich with sensory information that are unlikely to result from attentional biases. Neuroimaging data may likewise circumvent some critiques. We now try to integrate our understanding of hallucinations with notions of neural modularity and connectivity.

Brain lesions, modularity, connectivity and hallucinations

We propose that inter-regional effects mediate the penetration of perception by cognition. Some have discussed these top-down effects in terms of attention (2). Predictive coding theory conceives of attention in terms of the precision of priors and of prediction errors. Formally this could be implemented with a Kalman Filter (62, 63), which combines the predictive relationship between states (reliability), together with how this relationship is expected to change over time (the uncertainty) (64). This formalism can explain behavioral neuroscience observations in predictive learning (64). There are also encouraging human data. (65). However, precision weighting awaits more extensive empirical investigation in humans (particularly with regards to neurobiology and neurochemistry).

The precision of priors, in our example from the Introduction, may guide the search for threats in the shadows after having watched a scary movie. We contend that extremely precise (i.e., strongly weighted) priors could result in perception of particularly dark shadows, and—if priors are weighted strongly enough—could result in a hallucination. The supportive data are threefold: (i) a single case where a lesion caused hallucinations; (ii) functional connectivity between the lesion location and other regions; and (iii) effective (directional) connectivity in patients who suffer AVH. With these data, we will argue that top-down priors influence perception, thus violating a strictly encapsulated modular architecture.

(i) Lesion-Induced Hallucinosis

In graph-theory approaches to network analysis, hubs are defined from network structure based on structural, functional or effective connectivity data. FMRI data analyzed using graph theoretic metrics parses the brain into hubs (sub-networks), where a subset of regions connects those sub-networks (“connectors” in Figure 1, above). A module in graph theory has more within-group than between-group connections. A rich club is a collection of high degree hubs, more highly interconnected than predicted by chance. Lesions that impair cognition are more likely to be in rich-club hubs, or regions that mediate the long-range connectivity between connected information processing hubs (66). One such rich-club hub, the limbic system (including anterior insula cortex), has recently been implicated in the global specification of precision weighting or gain control (67) and has consistently been implicated in symptom capture fMRI studies of AVH (68, 69). It would be a challenge to call these circuits part of early perception. Rather, we will show regions like orbitofrontal cortex penetrate perceptual processing in primary sensory cortices giving rise to hallucinations.

Figure 1. Hubs, Connectors and Modules.

Connectors (black circles) and hubs (white squares) in modules. Hubs have many within-module connections. Connectors have many between module connections

Geddes and colleagues describe visual peduncular hallucinosis in a 66-year-old man following an ischemic stroke in the left pons, encompassing locus coeruleus (LC) (70). They found that functional connectivity between the LC and visual association cortex was reduced in the hallucinating patient. Conversely, functional connectivity between brainstem and visual association cortex, and between visual association cortex and prefrontal cortex (PFC), was significantly increased. In rodents, LC sends inhibitory noradrenergic inputs to visual cortex that improve perceptual acuity by gating target neurons (71). Norepinephrine may mediate the neuronal gain-control that implements this gating (72). Gain-control involves the weighting of conflicting cortical states to arbitrate which state governs experience (73). With higher gain, neuronal responses become more feature-selective and learning rate increases (74). This would be a candidate mechanism for our search for threatening figures on our walk home: if gain is driven up too high we may perceive villains where there are none.

The hallucinating patient had hyper-connectivity between visual association cortex and orbitofrontal cortex, a region that specifies top-down priors (75). PFC stimulation produces (76)—and frontal leucotomy ameliorates—hallucinations (77). These regions are outside of early perceptual modules.

(ii) Lesion effects on graph-theory metrics

Cognitive neuropsychology depends on the assumption that brain damage predominantly affects only the function of the damaged region. However, focal damage can disrupt whole-brain network organization (78). Intriguingly, damage to between-module connections (so-called rich club hubs, characterized as especially densely connected nodes (79) – see Figure 1) has more profound effects on cognitive performance (78) and, crucially, lesions alter connectivity in the opposing, un-lesioned hemisphere (78). The latter finding is difficult to reconcile with cognitive neuropsychology, which depends on very circumscribed damage effects (7).

We suggest that the rich-club hubs that alter global network function (and renounce encapsulation) are also the hubs involved in specifying global precision and therefore updating of inference in predictive coding (67). Patients with schizophrenia do have perturbed connectivity in rich club hubs in regions of the Default Mode Network and salience network (80), which themselves have been linked to predictive coding (81). However, hallucinations have not been specifically related to these metrics (82), and this will be an important direction for future inquiry. Rich club hubs seem well placed to mediate the top-down influence of one module on another (83). Dehaene and Changeux embrace this idea in their global workspace model of conscious cognitive functions, in which connector hub brain regions mediate informational integration into a coherent whole (84). Dysfunction of this process has been extended to understand some of the problems in schizophrenia (although not hallucinations per se (85)).

In our predictive coding approach informational integration (between modules) is mediated via precision weighting of priors and prediction errors, perhaps through rich club hubs. Others before us have asserted congruence between neural and mental modules (88). However, the exact relationship between psychological modularity (86) and modularity in functional connectivity (87) remains an open empirical question.

(iii) Directional effects

Thus far, the lesion effects we have discussed do not say anything about directionality. Dynamic causal modeling (DCM) is a means of making inferences about directional or effective connectivity of fMRI data as a function of extrinsic inputs (e.g., tasks) or intrinsic network states (89). Curcic-Blake and colleagues (90) examined inner speech processing in patients with schizophrenia with and without auditory hallucinations using DCM. They found a relative dearth of connectivity from Wernicke’s to Broca’s areas in patients with auditory hallucinations, suggesting that the precision of processing in Broca’s was higher than in Wernicke’s, consistent with a relative reliance on top-down versus bottom-up signals in those who were hallucinations. These data suggest information from higher regions is penetrating lower region. On a finer scale, single unit recording data have demonstrated rich-club architecture between groups of cells and that rich-clubs are key mediators of transfer entropy (directional information flow) between other hubs (83).

Chanes and Barrett (83) observe that predictions flow from less to more laminated cortices (top-down) and prediction errors flow in opposite direction (bottom-up) (91). Limbic regions like the insula have the simplest laminar structure and as such they represent good candidates for the specification of priors. Insula may be a rich club hub (92), given its membership in a number of intrinsic circuits it is well placed to mediate the communication between intrinsic networks in the brain (92). Eldar, Cohen and Niv showed that perceptual gain (as measured by pupilometry) was correlated with clustering coefficients in functional connectivity of fMRI data (74). Correlations were particularly strong in regions with high clustering coefficients; voxels that had higher inter-correlation with other voxels (like rich club hubs). Taking these findings together, we speculate that rich club hubs are by virtue of their connectivity and functional activation, well placed to implement changes in gain control as a function of the precision of predictions and prediction errors.

Studies of bi-stable perception – percepts that switch in dominance, apparently under the influence of intrinsic factors (since nothing changes in the sensory input (93)), also support this notion. Pupil dilation predicts switches in bi-stable perception (both auditory and visual) (94). Bi-stable percepts switch more frequently in manic patients (95) and patients with schizophrenia have less volitional control over the switches between bi-stable percepts (96). With regards binocular rivalry, Tononi and Edelman made similar observations in patients with schizophrenia (97), linking their observation to alterations in functional integration and segregation in the nervous system (98). Thus far, neither changes in bistability nor rivalry correlate with hallucinations. This will be a key empirical focus in future work.

DISCUSSION & FUTURE DIRECTIONS

We have argued that hallucinations demonstrate the top-down penetration of perception by cognitive processes. We presented behavioral and lesion data as well as functional and effective connectivity neuroimaging data that support our thesis. These data seem incommensurate with an encapsulated modularity of mind.

We take seriously the criticisms and failed replications of New Look experiments (2). However, we argue that the data we raise here cannot be dismissed in terms of demand characteristics or changes to post-perceptual judgment. Furthermore, we support our case with neural connectivity suggesting that those who hallucinate have enhanced directional top-down connectivity that modulate gain control mechanisms in sensory cortices. These mechanisms go beyond enhancement of selected inputs. They may sculpt percepts in the absence of sensation.

We framed these observations in terms of predictive coding, whose broad computational goal is to minimize prediction error (25). Predictive perception is penetrable to the extent that penetration minimizes overall long-term prediction error (99), and what we know will change what we see (99): that I know the lines in the Muller-Lyer illusion are identical does not render my perception veridical, because in the long term, the illusion is Bayes optimal (99), and knowing in this case does not minimize prediction error.

At odds with our theory, there are empirical data that demonstrate a reduced sensitivity to some illusions in patients with schizophrenia (100, 101). On the other hand, there is evidence that the influence of top-down priors is enhanced in psychotic states (102). These seem to be contradictory observations. Neither behavioral effect correlated with hallucinations specifically (weakened hollow-mask illusion correlates with disorganization symptoms, release from contextual suppression correlates with schizotypy, and enhanced impact of prior experience with stimuli correlates with schizotypal symptoms (102)).

Furthermore, other illusions may be enhanced in people with schizophrenia (for example the three flash illusion – in a manner that correlates with positive symptoms (103)). Clearly, a simple explanation of psychosis in terms of weak priors does not apply and illusions don’t consistently fail in these patients. Likewise, our proposal challenges theories that posit a failure of prediction or corollary discharge in generating hallucinations. These theories have not fared well when tested empirically. Patients with schizophrenia have impaired corollary discharge, but that effect is not related specifically to hallucinations (104).

Again, hierarchy may be crucial here (105). Whilst we eschew a strict informational separation between perception and cognition, we appreciate that perceptual systems are hierarchically arranged and that information processing can be impaired at different levels of the hierarchy. Even staunch modularists allow for priors to influence perception within the perceptual module and argue that these processes mediate illusions (86). It is possible that illusions could fail at this level and hallucinations could be generated higher in the hierarchy.

Our own work with ketamine may also adjudicate. Ketamine does not usually engender hallucinations, but rather, illusions (alterations of stimuli that are present). We recently reported actual hallucinations (of music and voices) on ketamine (106). These hallucinations occurred in the MRI scanner, which is perceptually denuded (dark, still, rhythmically noisy). In prior work, we suggested (based on our functional imaging data) that ketamine enhances bottom up noise (107). In the same work, we argued sensory deprivation induces hallucinations, via top-down priors. This is similar to the paradoxical effect of hearing loss and vision loss on hallucinations. In response to low-level prediction errors (108) higher-level precision increases to accommodate, producing non-veridical percepts. The isolation of the scanner provides an ambient sensory context that interacts with the bottom-up noise under ketamine, culminating in hallucinations.

Taken together, we argue that the dynamic interaction between priors and prediction errors give rise to hallucinations (109). This dynamic interaction is weighted differently at different levels of different hierarchies. Low-level impairments may give rise to illusions (and their failure in patients with schizophrenia) and higher-level perturbations may portend hallucinations. A brain hungry for priors (102) at the lowest levels of sensory input may try to satisfy that hunger by imposing more precise priors higher in the hierarchy—weighting perception towards expectation (rather than input) in a “listening attitude,” as Arieti put it (110)—which produces hallucinations (111). Critically, these effects high and low in the hierarchy, could be somewhat separated. Similar arguments have been put forth by Deneve & Jardri (112).

Recent apologists for Fodorian modularity do allow for attention to function as a back door through which some top-down effects can occur (2). In the predictive coding framework, attention involves the precision-weighting of predictions and prediction error (17). These gain-control mechanisms may be mediated by LC noradrenergic signaling as well as corticopetal cholinergic signals, which have also been implicated in hallucinatory phenomena (113). On the other hand, GABA and glutamate signaling may be involved in divisive normalization processes within a region or set of regions (114); this processing gates the impact of particular signals (114). Thus, within-module gain control is also possible. Our contention is that over-engagement of these processes renders priors overly precise, creating percepts in the absence of stimulus. Modulating noradrenergic, cholinergic, glutamatergic and GABAergic function may, therefore, have antipsychotic effects (115).

One criticism of cognitive penetration is that it is not under volitional control: we cannot conjure villains before our eyes at will. Some of the earliest phenomenological descriptions of schizophrenia actually describe such conjuring (of large insects for example) (116). A more recent example is found in the rise in popularity of the Tibetan spiritual practice of creating tulpas—a created companion which, through meditative practice, the creator is able to actually perceive (117). So-called ‘tulpamancy’ appears to involve a volitional choice to perceive specific agents (117). Whether tulpamancers are to be believed and whether objective evidence can be provided will require further work. By using functional imaging data and multi-voxel-pattern analysis (MVPA) to train classifiers, it may be possible to discern the neural correlates of perception from imagination (118). If tulpamancy and, more broadly, hallucinations involve truly perceptual rather than imaginative or malingering processes, they ought to engage perceptual neural correlates and be classified as such.

Further, hallucinations often involve a lack of insight although in Charles-Bonnet Syndrome they are often recognized as lacking reality (119). Seth and others have argued that the sense of perceptual presence entails not only sensory but also motor predictions (120). Even in mice, motor cortical representations modulate gain control in primary sensory cortices (121). If a percept has affordances—signifiers for action—and if those signifiers interact appropriately with our sensory feedback, then we perceive it as present (120). In this way, some hallucinations have a presence whereas others do not (119). Comparing and contrasting these phenomena is a key direction for future work. Furthermore, behavioral tasks could be devised that explore the impact of tulpas and hallucinations on behavior – rather like the studies of synesthetes that rendered them a population who may illuminate how perception is organized (122, 123).

CONCLUSION

Using functional imaging data that focus on inter-regional connectivity, a computational perspective, and focusing on hallucinations, we have made the case that perception is not encapsulated. Rather, it can be penetrated by expectations, beliefs and emotions. While this may seem an academic exercise, a deeper understanding of the neural and cognitive architecture is essential for the more precise treatment of symptoms that are distressing and portend poor functional outcome.