The brain's predictive prowess revealed in primary visual cortex

We humans carry around in our heads rich internal mental models that constitute our construction of the world, and the relation of that world to us. These models can be expressed at multiple levels of abstraction, including beliefs about sensory stimuli and the output of our motor programs, or higher-level beliefs about self. Advancing our understanding of the brain’s internal processing states, however challenging, could lead to breakthroughs in understanding states such as dreaming, consciousness, and mental disorders (1). Computational theories propose that internal models are broadcast throughout the brain, including to sensory areas of the brain (2). Techniques to read out internal mental models will deliver insight into how our brains use beliefs or predictions to “construct” the environment. Internal models in higher brain areas are complex though, creating a paradox that they are difficult to constrain if we want to decipher the brain’s coding of them. However, when internal models are fed back to sensory cortex, we assume that they are translated into sensory predictions that would intuitively have simpler content. In PNAS, Chong et al. (3) provide empirical evidence needed to drive this theory forward. Using brain reading, they show that the brain constructs new plausible predictions of expected sensory input, and that these predictions can be read out in sensory cortex.

There are two information streams that coincide in V1: the high-precision retinal input that is processed with high spatial acuity and the broad, abstract, less-precise strokes painted by cortical feedback. V1 is retinotopically organized and has small classical feed-forward receptive fields. Conversely, feedback conveys information about larger portions of the visual field, giving rise to extraclassical or contextual receptive fields. By eliminating the feed-forward input to the classical receptive field, we can investigate internal signals carried by feedback, which originate from diffuse brain areas (4). There are many challenges that we need to understand, including what kind of features are signaled in feedback. In a novel variation of the apparent motion illusion, Chong et al. (3) used inducer stimuli that allowed them to show that the motion prediction fed back to V1 contained detailed texture information. The information that they were able to read out from V1 was created by the brain's prediction about an object's rotational movement as it moved through the visual field. But, in fact, this experience of rotation was never present in the sensory input to the brain—it was fabricated by the brain out of acquired world knowledge. A previous study used objects rotating in depth to show a similar construction of an unseen (but implied) object position, but only in higher areas of the visual hierarchy (5). A number of other studies have shown that internal processing is contextual, both in higher (6) and lower visual areas (7, 8). The innovation of the data of Chong et al. (3) is in showing that internal processing is constructive and dynamic and contains motion and feature information even in primary visual cortex V1.

An important distinction to make is that during the long-range apparent motion used by Chong et al. (3) observers do not fill the intermediate space with a perceptual illusion of the rotated object. Wertheimer (9) extensively discussed the “visibility” of an object on the “Bewegungsfeld” (apparent motion trace). Although observers sense that the object is continuously crossing the field, they also know that it is an illusion and not real motion. This is an example of amodal completion: the knowledge that a figure continues to exist even when placed behind an occluded object (10–12) or that motion is continued behind objects (13). The important notion for the data of Chong et al. (3) is that, whereas only the gray surface was visible, observers’ knowledge constructed the object that was passing by (i.e., the intermediate grating). That the constructed grating can be read out of the activation pattern in V1 even though subjects do not actively perceive the “hidden object” shows that visual perception and internal constructions of visual content exist in independent neural codes. These codes might be compared with one another as is proposed in predictive coding frameworks, or they might be preserved in the system simultaneously to allow for an internal code distinct from an external.

In the data of Chong et al.(3), knowledge about the two objects is used to construct a prediction of a new, rotated object, which was never seen. However, our knowledge of the world spans several levels of complexity. For example, we recognize objects and can understand their interactions and their social meanings for us. Whereas sensory predictions are more fixed because perception leaves little room for ambiguity, our knowledge of the world that forms more complex mental models has more flexibility. These models can guide how we interact with the world and govern emotional responses. The significance of their flexibility is that maladaptive higher-level models can be altered with psychotherapeutic or pharmacological intervention. In depression, undesirable internal models bias attention and memory toward negative stimuli (14), leading to the abnormal processing of reward and punishment, and the confirmation of maladaptive internal mental models. Deepening our understanding of how the brain internally represents the world may then be crucial for guiding treatment intervention in mental disorders.

One question that arises from the data of Chong et al. (3) is this: What is the neural mechanism responsible for integrating model-based feedback signals with sensory-driven feed-forward signals? Rodent data of Larkum (15) and others suggest that this integration might happen within individual pyramidal neurons. Pyramidal neurons have two functionally distinct compartments to process feedback and feed-forward inputs, in the apical tuft and basal dendrites, respectively. These functional compartments reside in different layers of cortex, which in turn can be studied noninvasively in humans. For example, using high-field, high-resolution functional brain imaging, it has been shown that superficial layers of human V1 contain contextual feedback about natural scenes (16). We can conjecture that the predicted object representation found in the data of Chong et al. (3) may also be localized to the superficial layers that represent the nonstimulated apparent motion trace (Fig. 1, Right). It may also be present in deep layers; for example, while monkeys perform figure ground segregation, feedback arrives to V1 in the superficial layers but also in layer 5 (17). It remains to be tested how feedback effects in specific layers (superficial and/or deep) are modulated by the complexity of the feed-forward stimulation, and by the distance from V1 that feedback originates.

Fig. 1.

(Left) Apparent motion illusion stimulation similar to that in the experiment of Chong et al. (3). Observers reconstructed the intermediate dynamic representation on the apparent motion trace (dashed box, thought bubble). (Right) Possible cortical layer distribution of feed-forward and feedback information in the data of Chong et al. (3).

In his novel 1Q84, author Haruki Murakami (18) writes, “What we call the present is given shape by an accumulation of the past.” Chong et al. (3) use a simple and clever paradigm to demonstrate that during the interpolation of motion, the brain reconstructs object features in V1 that were never shown to the retina. Indeed, it seems the brain can “give shape” to its environment.