Desirability, availability, credit assignment, category learning, and attention: Cognitive-emotional and working memory dynamics of orbitofrontal, ventrolateral, and dorsolateral prefrontal cortices

Abstract

Background:

The prefrontal cortices play an essential role in cognitive-emotional and working memory processes through interactions with multiple brain regions.

Methods:

This article further develops a unified neural architecture that explains many recent and classical data about prefrontal function and makes testable predictions.

Results:

Prefrontal properties of desirability, availability, credit assignment, category learning, and feature-based attention are explained. These properties arise through interactions of orbitofrontal, ventrolateral prefrontal, and dorsolateral prefrontal cortices with the inferotemporal cortex, perirhinal cortex, parahippocampal cortices; ventral bank of the principal sulcus, ventral prearcuate gyrus, frontal eye fields, hippocampus, amygdala, basal ganglia, hypothalamus, and visual cortical areas V1, V2, V3A, V4, middle temporal cortex, medial superior temporal area, lateral intraparietal cortex, and posterior parietal cortex. Model explanations also include how the value of visual objects and events is computed, which objects and events cause desired consequences and which may be ignored as predictively irrelevant, and how to plan and act to realise these consequences, including how to selectively filter expected versus unexpected events, leading to movements towards, and conscious perception of, expected events. Modelled processes include reinforcement learning and incentive motivational learning; object and spatial working memory dynamics; and category learning, including the learning of object categories, value categories, object-value categories, and sequence categories, or list chunks.

Conclusion:

This article hereby proposes a unified neural theory of prefrontal cortex and its functions.

1. Introduction: towards a mechanistic theoretical understanding of prefrontal functions

1.1. Functional roles of orbitofrontal, ventrolateral, and dorsolateral prefrontal cortex

The prefrontal cortex (PFC) contributes to many of the higher cognitive, emotional, and decision-making processes that define human intelligence, while also controlling the release of goal-oriented actions towards valued goal objects. As noted in the Wikipedia article about PFC,

The most typical psychological term for functions carried out by the prefrontal cortex area is executive function. Executive function relates to abilities to differentiate among conflicting thoughts, determine good and bad, better and best, same and different, future consequences of current activities, working toward a defined goal, prediction of outcomes, expectation based on actions, and social ‘control’ (the ability to suppress urges that, if not suppressed, could lead to socially unacceptable outcomes).

Elliott et al. (2000) discussed how the PFC contributes to generating behaviours that are flexible and adaptive, notably in novel situations, and to suppressing actions that are no longer appropriate, thereby freeing humans and other primates from being forced to respond more reflexively to current sensory inputs. These authors also review the various terms that have been used to describe PFC functions, including planning (Luria, 1966), memory for the future (Ingvar, 1985), executive control (Baddeley, 1986), working memory (Goldman-Rakic, 1987), supervisory attention (Shallice, 1988), and top-down modulation of bottom-up processes (Frith and Dolan, 1997).

Miller and Cohen (2001) reviewed data that are consistent with these concepts, noting that these properties result

from the active maintenance of patterns of activity in the prefrontal cortex that represent goals and the means to achieve them. They provide bias signals to other brain structures whose net effect is to guide the flow of activity along neural pathways that establish the proper mappings between inputs, internal states, and outputs needed to perform a given task. (p. 167)

Said in yet another way, the PFC is involved with predicting future outcomes and enabling animals and humans to respond adaptively to them.

Wise (2008) espoused a similar view that he vividly summarised as follows: ‘The long list of functions often attributed to prefrontal cortex may all contribute to knowing what to do and what will happen when rare risks arise or outstanding opportunities knock’ (p. 599).

Even this brief heuristic summary of some of the multiple functions of the PFC illustrates the challenge facing any theorist who wishes to model this, or indeed any, part of the brain. The challenge is that various functionally distinct parts of the PFC are connected to each other in complex ways, in addition to being widely connected with multiple other brain regions. Broadly speaking, the dorsal PFC is interconnected with brain regions involved with attention, cognition, and action (Goldman-Rakic, 1988), whereas the ventral prefrontal cortex is interconnected with brain regions involved with emotion (Price, 1999). These facts do not, however, explain how these brain circuits give rise to these distinct psychological functions as emergent properties that arise from interactions among brain regions that work together as functional systems.

A critical question for any theorist of mind and brain is thus: How can the divide between brain mechanisms and psychological functions be bridged? How can this be done with sufficient mechanistic precision to explain and predict challenging interdisciplinary data? Section 1.3 summarises a well-established theoretical method whereby the emergent properties of brain mechanisms may be linked to the mental functions that they control.

Before summarising this theoretical method, Section 1.2 will review some of the prefrontal properties that recent experiments have reported and that will be explained in later sections using this theoretical method. In particular, many prefrontal cortical properties can be subsumed under two unifying mechanistic themes: cognitive-emotional dynamics and working memory dynamics. A macrocircuit of the brain regions that embody these processes in a unified predictive Adaptive Resonance Theory (pART), a model that herein explains and predicts many prefrontal cortical data, is shown in Figure 1.

Figure 1.

Macrocircuit of the main brain regions, and connections between them, that are modelled in the unified predictive Adaptive Resonance Theory (pART) of cognitive-emotional and working memory dynamics that is described in this article. Abbreviations in red denote brain regions used in cognitive-emotional dynamics, whereas abbreviations in green denote brain regions used in working memory dynamics. Black abbreviations refer to brain regions that process visual data during visual perception and are used to learn visual object categories. Arrows denote non-adaptive excitatory synapses. Hemidiscs denote adaptive excitatory synapses. Many adaptive synapses are bidirectional, thereby supporting synchronous resonant dynamics among multiple cortical regions. The output signals from the basal ganglia that regulate reinforcement learning and gating of multiple cortical areas are not shown. Also not shown are output signals from cortical areas to motor responses. V1: striate, or primary, visual cortex; V2 and V4: areas of prestriate visual cortex; MT: middle temporal cortex; MST: medial superior temporal area; ITp: posterior inferotemporal cortex; ITa: anterior inferotemporal cortex; PPC: posterior parietal cortex; LIP: lateral intraparietal area; VPA: ventral prearcuate gyrus; FEF: frontal eye fields; PHC: parahippocampal cortex; DLPFC: dorsolateral hippocampal cortex; HIPPO: hippocampus; LH: lateral hypothalamus; BG: basal ganglia; AMGY: amygdala; OFC: orbitofrontal cortex; PRC: perirhinal cortex; VPS: ventral bank of the principal sulcus; VLPFC: ventrolateral prefrontal cortex. See text for further details.

Section 2 will summarise relevant data and models of cognitive-emotional dynamics and Section 3 will summarise relevant data and models of working memory dynamics. Cognitive-emotional dynamics, and models thereof, include how orbitofrontal cortex (OFC) interacts with brain regions like temporal cortex, amygdala, hippocampus, and cerebellum to regulate processes like category learning and adaptively timed motivated attention and action to acquire a valued goal. Working memory dynamics, and models thereof, include how sequences of events are temporarily stored in ventrolateral and dorsolateral prefrontal cortex, how these sequences are unitised, or chunked, into cognitive plans, and how interactions of prefrontal regions with other brain regions like perirhinal cortex (PRC), parahippocampal cortex (PHC), and basal ganglia (BG) enables predictions and actions to be chosen that are most likely to succeed based on sequences of previously rewarded experiences.

These sections will also compare and contrast the neural models that are used herein with other models in the literature and will make testable predictions to further clarify the brain mechanisms that underlie these data.

Section 4 will provide concluding remarks that highlight some of the article’s main themes.

A unified theoretical explanation with such ambitious goals will necessarily be presented in stages. The text will state key data, and outline model explanations of them, at the beginning of each section to help readers to frame subsequent, more detailed, explanations. The text will also attempt to provide a self-contained explanation of all the models that it uses. This explanation cannot be exhaustive because the theoretical and experimental literatures that fall within the scope of the models are vast. The exposition will nonetheless provide enough information for the reader to appreciate how the explanations work, and why they are compelling.

The exposition will also try to deal with another, even more basic, problem: How can any theory penetrate the complexity of a behaving brain? Indeed, a few facts, taken alone, can have multiple explanations. They provide insufficient guidance to rule out plausible, but incorrect, explanations. In contrast, the current theoretical method confronts hundreds of facts about a particular behavioural function, taken from the entire experimental literature on multiple organisational levels from behavioural to cellular, with all known modelling principles, mechanisms, microcircuits, and architectures. When this is done properly, every such fact, and every modelling hypothesis about it, is put under enormous ‘conceptual pressure’ that typically allows only one possible explanation to survive. The text tries to summarise enough of these constraints to clarify how and why the explanations work and to motivate readers to pursue further reading about topics that particularly interest them in related articles.

1.2. Prefrontal desirability, availability, credit assignment, and feature-based attention

These models help to understand and mechanistically explain recent data about the OFC, ventrolateral prefrontal cortex (VLPFC), and dorsolateral prefrontal cortex (DLPFC). The text will first focus upon a striking conclusion that has arisen about roles for OFC and VLPFC that were derived from behavioural experiments in monkeys who had experienced excitotoxic lesions of these brain regions. The conclusion is that OFC and VLPFC are involved in updating a predicted outcome’s desirability versus its availability, respectively (Rudebeck et al., 2017). Recent data have also claimed that DLPFC neurons encode a solution of the credit-assignment problem (Assad et al., 2017). What the concepts of desirability, availability, and credit assignment mean will be operationally defined in the sections where these data are explained.

Some of the results on desirability that are reported by Rudebeck et al. (2017) were predicted, indeed simulated, by a well-established neural model of cognitive-emotional interactions, notably reinforcement learning and motivated attention, in Grossberg et al. (2008). Aspects of the role of VLPFC in updating availability are clarified by a well-established neural model of cognitive working memory and the learning of predictive cognitive plans, or list chunks (Cohen and Grossberg, 1986, 1987; Grossberg, 2017b; Grossberg and Pearson, 2008). This model has been further developed in Huang and Grossberg (2010) to quantitatively simulate data about sequential decisions during contextually cued visual search and in Silver et al. (2011) to simulate prefrontal cortical neurophysiological data about sequences of eye movement decisions. Data about credit assignment are explained by combining the model of cognitive working memory and list chunks with the model of reinforcement learning and motivated attention to show how chunks that lead to successful predictions are amplified, while those that do not are suppressed.

In Sections 2 and 3, the article will first summarise the above kinds of data that are explanatory targets of the article, then review the models that explain them, and go on to use these models to provide explanations and predictions of additional interdisciplinary data. It will then extend them in a consistent way to mechanistically and functionally explain data about prefrontal sources of feature-based attention in monkeys (Bichot et al., 2015) and humans (Baldauf and Desimone, 2014).

The PFC has been studied with multiple methods. Some articles have studied the PFC of humans with functional neuroimaging in normal subjects or clinical patients, while others have studied monkeys or rats with neurophysiological or anatomical methods. Important functional conclusions have also been derived by combining selective lesions with behavioural studies in monkeys. Recent studies have, however, also shown that different lesion methods can yield quite different results. For example, monkeys with selective excitotoxic lesions of the OFC, unlike monkeys who have received aspiration OFC lesions, are unimpaired in learning and reversing object choices based on reward feedback (Rudebeck et al., 2013). Neurotoxic lesions of the amygdala (Izquierdo and Murray, 2007) have also led to results that challenge earlier demonstrations using aspiration and radiofrequency lesions that the amygdala is needed for object reversal learning (Aggleton and Passingham, 1981; Jones and Mishkin, 1972; Spiegler and Mishkin, 1981).

Why do different lesion methods yield such different results? One main reason is that, unlike excitotoxic lesions, other lesion methods, including aspiration and radiofrequency lesions, may also damage fibres of passage to adjacent cortical areas. The fact that OFC activity has been reported during reversal learning (Fellows and Farah, 2003; Morrison et al., 2011; Rolls, 2000; Rolls et al., 1994) suggests that several neuronal regions and pathways may be involved in this behavioural competence.

This picture is complicated further by different definitions of the brain areas that constitute OFC and VLPFC. The conclusions above hold if OFC is understood to consist of areas 11, 13, and 14. If, however, area 12o is also included, which overlaps what Chau et al. (2015) call the lateral OFC (lOFC), then various properties of what Rudebeck et al. (2017) would assign to VLPFC get attributed to OFC. Herein, the convention will be followed that OFC consists of areas 11, 13, and 14. Another caveat is that there appear to be species-specific variations. For example, unlike old world monkeys, excitotoxic lesions in new world monkeys such as marmosets (Roberts, 2006) can impair these animals on reversal tasks. These variations will not be analysed herein.

1.3. A theoretical method for linking brain to mind: method of minimal anatomies

One successful method for linking brain mechanisms to behavioural functions has been developed and applied during the past 60 years. This method has led to neural models that often anticipated psychological and neurobiological data about the PFC, among other brain regions. It continues to do so, as this article will illustrate.

A key theme of this theoretical ‘method of minimal anatomies’ is that one cannot derive a theory of an entire brain in one step. Rather, one does so incrementally in stages. This incremental theoretical method embodies a kind of design evolution whereby each model embodies a certain set of design principles and mechanisms that the evolutionary process has discovered whereby to cope with a given set of environmental challenges. Then, the model is refined, or unlumped, to embody an even larger set of design principles and mechanisms and thereby expands its explanatory and predictive power. This process of evolutionary unlumping continues unabated, leading to current models that can individually explain psychological, anatomical, neurophysiological, biophysical, and biochemical data about a given faculty of biological intelligence.

Using this method, the pART theory of cognitive-emotional and working memory dynamics has been discovered and incrementally elaborated over the past 60 years (Figure 1). The current exposition will emulate the theoretical method by first summarising the simplest models that can explain nontrivial amounts of prefrontal data, before unlumping them to explain even more data. Although, for expository reasons, multiple models will be mentioned, it needs to be understood that there is just one unified theory behind all the explanations that joins together cognitive-emotional dynamics and working memory dynamics.

The theoretical derivation always begins with behavioural data because brain evolution needs to achieve behavioural success. Starting with behavioural data enables models to be derived whose brain mechanisms have been shaped during evolution by behavioural success. Starting with a large database helps to rule out incorrect, but otherwise seemingly plausible, answers.

Such a derivation leads to the discovery of novel design principles and mechanisms with which to explain how an individual, behaving in real time, can generate the behavioural data as emergent properties. This conceptual leap from data to design is the art of modelling. Once derived, despite being based on psychological constraints, the minimal mathematical model that realises the behavioural design principles has always been interpretable in terms of brain mechanisms. Sixty years of modelling have hereby supported the hypothesis that brains look the way that they do because they embody computational designs whereby individuals autonomously adapt to changing environments in real time. The link from behaviour-to-principle-to-model-to-brain has, in addition, often disclosed unexpected functional roles of the derived brain mechanisms that are not clear from neural data alone.

A ‘minimal’ model is one for which if any of the model’s mechanisms is removed, then the surviving model can no longer explain a key set of previously explained data. Once a connection is made top-down from behaviour to brain by such a minimal model, mathematical and computational analysis discloses what data the minimal model, and its variations, can and cannot explain. Such an analysis focuses attention upon design principles that the current model does not yet embody. These new design principles and their mechanistic realisations are then consistently incorporated into the model by unlumping it to generate a more realistic model. If the model cannot be refined in this way, then that is strong evidence that the current model contains a serious error and must be discarded. The unified pART model that is discussed herein, which explains key functional processes in the brain regions depicted in Figure 1, has withstood multiple stages of unlumping.

2. Cognitive-emotional dynamics and the OFC

2.1. Orbitofrontal coding of desirability as probed by selective satiation

This section of the article summarises an explanation of how the desirability of an outcome is computed in our brains. The relevant data will first be reviewed before the model that can explain, indeed anticipated, them is summarised, along with other data explanations and predictions. An outline of the model’s explanation of desirability will first be given to frame the subsequent exposition of the model mechanisms that accomplish this.

As noted above, the experiments of Rudebeck et al. (2017) support the hypothesis that the OFC, but not the VLPFC, plays a necessary role in choices that are based on outcome desirability. In contrast, the VLPFC, but not the OFC, plays a necessary role in choices that are based on outcome availability. What desirability means is explained operationally by an experiment (their Experiment 2) that manipulates the subjective value of different food rewards with a stimulus-based reinforcer devaluation, or satiation, procedure that was earlier used by Málková et al. (1997), while keep the probability and magnitude of reward stable. The monkeys in this experiment were trained with some conditioned stimulus (CS) objects that were associated with Food 1, and others associated with Food 2. The two foods acted like unconditioned stimuli (US) in the experiment. Following the selective satiation procedure, monkeys were presented with pairs of objects, one object associated with Food 1 and the other with Food 2. The effects of devaluation were measured by calculating how much monkeys shifted their choices towards objects associated with a higher value food, relative to baseline choices.

Both unoperated control monkeys and monkeys with excitotoxic VLPFC lesions could update and use the current value of food reward to guide their choices. In contrast, monkeys with excitotoxic OFC lesions chose stimuli that were associated with the sated food at a much higher rate. Various tests led unambiguously to the conclusion that this deficit in monkeys with OFC lesions arose from their inability to link objects with the current value of the food in guiding their choices.

Málková et al. (1997) had earlier used the same devaluation procedure in rhesus monkeys to test whether excitotoxic basolateral amygdala (AMYG) lesions lead to an inability to shift decisions based upon current food value. Their experiments followed up earlier work of Hatfield et al. (1996) and Holland and Straub (1979) in rats. The results on desirability that are reported by Rudebeck et al. (2017) on OFC lesions were earlier predicted and simulated by the MOTIVATOR (Matching Objects To Internal VAlues Triggers Option Revaluations) neural model in Grossberg et al. (2008) as part of an explanation of the Málková et al. (1997) results.

In particular, Grossberg et al. (2008) wrote the following as part of the caption of their Figure 8 that shows data and simulations about reaction time, choice behaviour, and reward value:

Choices made between the two CSs reflect preferences between the different food rewards. Devaluation of a US by food-specific satiety (FSS) shifts the choices of the animal away from cues associated with the devalued rewards (reprinted with permission from Málková et al., 1997). Málková et al. (1997) report the effects of basolateral amygdala lesions using a difference score. The difference score is calculated by measuring the percent of the trials in which the to-be-devalued food is chosen over other foods, before and after FSS. The ‘difference score’ reflects the difference between these two percentages … Using FEF activity to determine cue choice, the intact model (CTL) shows a similar shift in CS preference when the US associated with it is devalued by FSS. Food-specific satiety is implemented by lowering selected DRIVE inputs to the LH … The automatic shifting of visual cue preference when an associated US is devalued by FSS is lost after AMYG lesions (AX) and ORBl lesions (OX). [italics mine]

Figure 8.

A temporal sequence of inputs creates a spatial pattern of activity across item chunks in an Item-and-Order working memory (height of hatched rectangles is proportional to cell activity). Relative activity level codes for item and order. A rehearsal wave allows item activations to compete before the maximally active item elicits an output signal and self-inhibits via feedback inhibition to prevent its perseverative performance. The process then repeats itself. Solid arrows denote excitatory connections. Dashed arrows denote inhibitory connections.

Source: Adapted from Grossberg (1978a).

The devaluation procedure that was used by Málková et al. (1997) and Rudebeck et al. (2017) fed monkeys a lot of Food 1 before testing their choice between Food 1 and an alternative food, Food 2, that has not be devalued. The MOTIVATOR model explains how devaluing Food 1 reduces the drive input (Figure 2(a)) from the lateral hypothalamus (LH) that is needed to activate its value category in the amygdala (AMYG), thereby reducing its ability to compete with the value category of Food 2. The value category of Food 2 can hereby win the competition between value categories, and release an incentive motivational signal to the OFC, which enables the OFC to choose Food 2 with increased probability. Either AMYG or OFC lesions eliminate this pathway to motivated choice of Food 2.

Figure 2.

(a) CogEM (Cognitive-Emotional-Motor) neural model circuits and their anatomical interpretation. CogEM models how invariant object categories in sensory cortex can activate value categories, also called drive representations, in the amygdala and hypothalamus, and object-value categories in the orbitofrontal cortex. Converging output signals from an object category and its value category are needed to vigorously fire the corresponding object-value category. Achieving such convergence from the amygdala requires prior conditioned reinforcer learning and incentive motivational learning. Activation of a value category also requires converging signals: from its object category and its internal drive input. When an object-value category fires, it can send positive feedback to its object category and attentionally enhance it with value-modulated activation. The motivationally enhanced object category can then better compete with other object categories via a recurrent competitive network (not shown) and draw attention to itself. Closing the feedback loop between object, value, and object-value categories causes a cognitive-emotional resonance that can induce a conscious percept of having a particular emotion, or feeling, towards the attended object, as well as knowing what it is. As this resonance develops, the object-value category can generate output signals that can activate cognitive expectations and action commands through other brain circuits. Adapted with permission from Grossberg and Seidman (2006). (b) The neurotrophic Spectrally Timed Adaptive Resonance Theory, or nSTART, model macrocircuit is a further development of the START model in which parallel and interconnected networks support both delay and trace conditioning. Connectivity from both the thalamus and the sensory cortex occurs to the amygdala and hippocampus. Sensory cortex interacts reciprocally with prefrontal cortex, specifically orbitofrontal cortex. Multiple types of learning and neurotrophic mechanisms of memory consolidation cooperate in these circuits to generate adaptively timed responses. Connections from the sensory cortex to the orbitofrontal cortex support category learning. Reciprocal connections from orbitofrontal cortex to sensory cortex support motivated attention. Habituative transmitter gates modulate excitatory conductances at all processing stages. Connections from sensory cortex to amygdala support conditioned reinforcer learning. Connections from amygdala to orbitofrontal cortex support incentive motivation learning. Hippocampal adaptively timed learning and brain-derived neurotrophic factor (BDNF) bridge temporal delays between CS offset and US onset during trace conditioning acquisition. BDNF also supports long-term memory consolidation within sensory cortex to hippocampal pathways and from hippocampal to orbitofrontal pathways. The pontine nuclei serve as a final common pathway for reading-out conditioned responses. Cerebellar dynamics are not simulated in nSTART. Arrowhead: excitatory synapse; hemidisc: adaptive weight; square: habituative transmitter gate; square followed by a hemidisc: habituative transmitter gate followed by an adaptive weight. Reprinted with permission from Franklin and Grossberg (2017).

In order to better understand how this decision process is proposed to work, the subsequent text will review three model properties: (1) how the brain can learn different value categories that can be selectively activated by different foods; (2) how internal drive inputs, notably satiety signals, interact with conditioned or unconditioned reinforcing sensory inputs before such sensory-drive combinations compete to determine an incentive motivational output; and (3) how conditioned reinforcer pathways can habituate due to frequent activation by a particular food and thus create progressively smaller inputs to their value categories. The loss of these factors due to an amygdala lesion may prevent an animal from shifting its visual cue preference away from a devalued food.

2.2. Reinforcement learning, motivated attention, resonance, and directed action

The Rudebeck et al. (2017) concept of desirability of predicted outcomes is related to earlier concepts such as the ‘somatic marker hypothesis’ which proposes that decision-making is a process that depends upon emotion (Bechara et al., 1999). The Cognitive-Emotional-Motor, or CogEM, model (Figure 2(a)) that will be described in this section, in several incrementally unlumped versions that include the MOTIVATOR model, provides a mechanistic neural explanation of some aspects of emotionally modulated decision-making by describing different properties of, and interactions between, sensory cortex, amygdala, and OFC in making these decisions (Baxter et al., 2000; Bechara et al., 1999; Schoenbaum et al., 2003; Tremblay and Schultz, 1999). CogEM also clarifies how the OFC contributes in this circuit to the expression of the incentive value of rewards and their sensitivity to reward devaluation (Gallagher et al., 1999), and how acquired value in OFC depends on input from basolateral amygdala (Schoenbaum et al., 2003).

These properties of the CogEM model arise as emergent, or interactive, properties of the neural mechanisms that regulate reinforcement learning among the sensory cortex, amygdala/hypothalamus, and OFC; the allocation of motivated attention to chosen options in the sensory and orbitofrontal cortices as a result of this learning; and the release of motivated actions by the OFC to acquire valued goal objects that can realise these options (Grossberg, 1971, 1972a, 1972b, 1982, 2000b; Grossberg and Levine, 1987; Grossberg and Schmajuk, 1987; Grossberg and Seidman, 2006).

In particular, as summarised in Figure 2(a), the CogEM model explains how invariant object categories, in sensory cortical regions like the anterior inferotemporal cortex (ITa), and object-value categories, in cortical regions like the OFC, interact with value categories, in subcortical emotional centres like amygdala and hypothalamus. These brain regions are linked by a feedback loop which, when activated for sufficiently long time, can generate a cognitive-emotional resonance. Such a resonance can support conscious feelings while conditioned reinforcer learning pathways (from sensory cortex to amygdala; Gore et al., 2015) and incentive motivational learning pathways (from amygdala to OFC; Arana et al., 2003) focus motivated attention upon valued object and object-value representations. These attended object-value representations can, in turn, release commands to perform actions that are compatible with these feelings.

It needs immediately to be noted, however, that the CogEM circuit in Figure 2(a) cannot, by itself, maintain motivated attention during an adaptively timed interval that is sufficiently long to enable reinforcement learning to effectively occur in paradigms where rewards are delayed in time, as happens during trace conditioning and delayed match-to-sample, and to enable a conscious ‘feeling of what happens’ to emerge (Damasio, 1999). The hippocampus is needed to support both of these properties (Figure 2(b)), as sections 2.5 and 2.8 will further discuss.

The next Sections say more about these several types of categories and the learned interactions between them.

2.3. Object, value, and object-value categories

Three different types of learned representations are included in the CogEM circuit of Figure 2(a): invariant object categories respond selectively to objects that are seen from multiple views, positions, and distances from an observer. They are learned by inferotemporal (IT) cortical interactions between anterior IT (ITa), posterior IT (ITp), and prestriate cortical areas like V4 (Figure 1; Desimone, 1998; Gochin et al., 1991; Harries and Perrett, 1991; Mishkin, 1982; Mishkin et al., 1983; Seger and Miller, 2010; Ungerleider and Mishkin, 1982). How such invariant object categories may be learned as the eyes scan a scene is modelled by the ARTSCAN Search neural model that is discussed in Sections 3.28 and 3.29 (Cao et al., 2011; Chang et al., 2014; Fazl et al., 2009).

Value categories are sites of reinforcement learning that control different emotions and incentive motivational output signals. They occur in amygdala and hypothalamus in cells where reinforcing and homeostatic, or internal drive, inputs converge to generate emotional reactions and motivational decisions (Aggleton, 1993; Aggleton and Saunders, 2000; Barbas, 1995, 2007; Bower, 1981; Davis, 1994; Gloor et al., 1982; Halgren et al., 1978; LeDoux, 1993).

Object-value categories respond to converging signals from object categories and value categories. They are proposed to occur in OFC. The properties of object-value categories will be particularly discussed in this exposition. How they interact with representations in other brain regions is also an essential part of such an analysis.

Finally, motor representations (M) control discrete adaptive responses. They include multiple brain regions, including motor cortex and cerebellum (Evarts, 1973; Ito, 1984; Kalaska et al., 1989; Thompson, 1988). More complete models of the internal structure of motor representations and how they generate movement trajectories are described elsewhere (e.g. Bullock et al., 1998; Bullock and Grossberg, 1988; Cisek et al., 1998; Contreras-Vidal et al., 1997; Fiala et al., 1996; Gancarz and Grossberg, 1998, 1999) and can readily be incorporated into CogEM model extensions.

2.4. Conditioned reinforcer, incentive motivational, and motor learning: wanting

Three types of learning are shown in Figure 2(a) between these representations: conditioned reinforcer learning strengthens the pathway from an invariant object category to a value category. Incentive motivational learning strengthens the pathway from a value category to an object-value category. Motor learning enables the performance of an act aimed at acquiring a valued goal object.

Reinforcement learning, such as classical, or Pavlovian, conditioning (Kamin, 1968, 1969; Pavlov, 1927), occurs within conditioned reinforcer pathways (Figure 2(a)). A neutral event is called a CS when it is paired with an emotion-inducing, reflex-triggering event that is called an unconditioned stimulus (US). A CS can become a conditioned reinforcer when its object category is activated sufficiently often just before the value category is activated by an US. As a result of conditioned reinforcer learning, a CS can, on its own, subsequently activate a value category via this learned pathway. When this happens, the CS is called a conditioned reinforcer because it can trigger many of the same reinforcing and emotional effects as an US.

During classical conditioning, incentive motivational learning also occurs from the activated value category to the object-value category that corresponds to the CS, incentive motivational learning enables an active value category to prime, or modulate, the object-value categories of all CSs that have consistently been correlated with it. It is the kind of learning that primes an individual to think about places to eat when feeling hungry.

Motor, or habit, learning enables the sensorimotor maps, vectors, and gains that are involved in sensory-motor control to be adaptively calibrated, thereby enabling a CS to read out correctly calibrated movements via its object-value category.

These conclusions about the CogEM model also hold for operant, or instrumental, conditioning, where rewards or punishments are delivered that are contingent upon particular behaviours (Skinner, 1938). In fact, the CogEM model was introduced to explain data about operant conditioning (Grossberg, 1971). Many reinforcement learning and motivated attentional mechanisms exploit shared neural circuits, even though the experimental paradigms and behaviours that activate these circuits may differ.

The incentive motivational and motor learning pathways contribute to the process that various researchers call wanting. As Berridge et al. (2009) note, ‘By “wanting” we mean incentive salience, a type of incentive motivation that promotes approach towards and consumption of rewards’ (p. 67). See also Smith et al. (2011) for a further discussion of the different brain substrates of pleasure and incentive salience.

2.5. Category learning and memory consolidation: effects of lesions

A fourth kind of learning, category learning, adapts the connections from thalamus to an object category in sensory cortex, and/or from an object category to an object-value category in OFC. This category learning process enables external objects and events in the world to selectively activate object and object-value categories. Category learning was not simulated in the original CogEM model (Figure 2(a)), which focused on reinforcement learning, motivated attention, and the release of actions towards valued goal objects. Category learning does play a key role in extensions of CogEM, such as the neurotrophic Spectrally Timed Adaptive Resonance Theory, or nSTART, model (Figure 2(b)) of Franklin and Grossberg (2017). nSTART augments CogEM to include category learning, as well as adaptively timed learning in the hippocampus that can bridge between CS and US stimuli that are separated in time by an interval that can be hundreds of milliseconds in duration, as can occur during trace conditioning and delayed non-match to sample. Interactions between these two processes, augmented by all the other processes of CogEM, enable nSTART to explain and simulate how memory consolidation of recognition categories may occur after conditioning ends. nSTART supports this explanation by mechanistically explaining and simulating data about the complex pattern of disruptions of memory consolidation that occur in response to early versus late lesions of thalamus, amygdala, hippocampus, and OFC.

2.6. Polyvalent constraints and competition interact to choose the most valued options

Both the CogEM and nSTART circuits in Figure 2 need to have two successive sensory processing stages, an invariant object category stage in the temporal cortex, and an object-value category stage in OFC, in order to ensure that the object-value category can release motivated behaviour most vigorously if both sensory inputs from temporal cortex and motivational inputs from the amygdala are simultaneously delivered to the object-value category. A polyvalent constraint on an object-value category means that it fires most vigorously when it receives input from its invariant object category and from a value category. In other words, an object-value category is amplified when the action that it controls is valued at that time. Only when an object-value category wins a competition with other object-value categories can it trigger an action. After learning occurs, a conditioned reinforcer can, by itself, satisfy this polyvalent constraint by sending a signal from its object category directly to its object-value category, and indirectly via the (conditioned reinforcer)-(incentive motivational) pathway (Figure 2). Converging pathways from sensory cortical areas and amygdala to OFC are well known anatomically (e.g. Barbas, 1995).

The firing of each value category in the amygdala/hypothalamus is also regulated by a combination of polyvalent constraints and competition. Here, the polyvalent constraint (Figure 2(a)) is realised by two converging inputs to each value category: a reinforcing input from an US or conditioned reinforcer CS via a conditioned reinforcer pathway and a sufficiently large internal drive input (e.g. hunger, satiety). Each value category can only become active enough to reliably win the competition with other value categories when it receives sufficiently big converging inputs, and only a winning value category can generate large incentive motivational signals to object-value categories. In particular, even if visual cues such as a familiar food generate a strong conditioned reinforcer inputs to a value category, it cannot fire if its internal drive input is reduced by eating a lot, since then the hunger drive input will decrease and the satiety drive input will increase, thereby preventing its value category from winning the competition.

Because both the value categories and the object-value categories obey polyvalent constraints and compete to determine either incentive motivational or motor outputs, a CogEM circuit tends to choose options for action that are currently the most desired ones. These CogEM dynamics hereby clarify necessary conditions for the computation of desirability by the OFC.

Many issues need to be discussed to better understand how these circuits work in practice. These issues include the following: Why is the amygdala called a value category? How does a value category differ from just an internal drive such as hunger? In particular, can value categories represent specific hungers that can be selectively devalued by eating a lot of a particular food? How is the hypothalamus involved in learning a value category? Finally, are there also pathways for expressing valued goals that do not require the amygdala? Such issues will be discussed as the exposition proceeds.

2.7. Predicting what happens next in a changing world: blocking and unblocking

In the real world, a foraging animal may be confronted with multiple possible sensory cues that predict different kinds of available foods. The case of choosing Food 1 or Food 2 in response to different CSs is a special case of this situation. To more completely understand how a choice of one option over another occurs, along with how its desirability is computed, requires an understanding of how attentional blocking and unblocking occur (Kamin, 1968, 1969; Pavlov, 1927). Blocking and unblocking experiments also clarify how humans and other animals discover what combinations of cues are causal and which are predictively irrelevant. Causal feature combinations tend to be attended and used to control subsequent actions. The following explanation of how this happens can be used to design ecologically more realistic neurobiological studies of desirability.

Blocking and unblocking experiments show that the discovery of true environmental causes is an incremental process. Unless sufficiently many actions based upon unblocked cue combinations are made, and their consequences used to match and mismatch learned expectations, an irrelevant cue can be erroneously thought to be predictive, much as superstitious behaviours may be learned and maintained (Skinner, 1948). Predictive errors hereby play a crucial role in shaping the learning of environmental causes, as popular recent books have noted; for example, Schulz (2010). Section 2.14 discusses how unblocking works in greater detail.

A food’s desirability in the real world, where there may be multiple possible food options to choose from, can only be computed if it is not blocked. For example, suppose that a buzzer sound (CS₁) is paired with a food reinforcer (US₁) until an animal salivates to this sound in anticipation of eating the food. On subsequent learning trials, suppose that, before the food occurs, the buzzer sounds at the same time that a perceptually equally salient light flashes (CS₂). Under these circumstances, the flashing light does not become a source of conditioned salivation because it does not predict, indeed does not cause, any consequence that the buzzer sound alone did not already predict. In other words, the flashing light is predictively irrelevant and is thus attentionally blocked.

On the other hand, suppose that, whenever the flashing light occurs with the buzzer sound, the amount of subsequent food (US₂) is much greater than when the buzzer sound occurred alone. Then, the animal does learn to anticipate food in response to the flashing light, with the usual salivation and expectation of subsequent food, because it causally predicts an increase in food. If, instead, the amount of food that is presented when the flashing light and buzzer occur together is much less than when the buzzer sound occurred alone, then a wave of frustration (Amsel, 1962, 1992), which is a negative reinforcer, may be experienced, even though some food, which is a positive reinforcer, has been presented. The process whereby the CS₂ becomes predictively relevant is called attentional unblocking, and it may become a source of either a conditioned appetitive response or conditioned frustration, depending on whether the amount of food is more or less ample after the simultaneous cues than in response to the buzzer alone. Unexpected consequences can hereby lead to the discovery of new cue combinations that cause valued consequences. Attention can then be focused on the unblocked cues, which can then be associated with appropriate new responses.

Blocking and unblocking experiments show that humans and many other animals behave like minimal adaptive predictors who can learn to focus their attention upon events that causally predict important affective consequences, while ignoring other events, at least until an unexpected consequence occurs.

The same CogEM dynamics that enable the OFC to release actions aimed at acquiring desirable goal objects also carry out blocking. Blocking can be understood in the CogEM model as a result of how a cognitive-emotional resonance using the feedback pathways between temporal cortex, amygdala, and OFC (Figure 2) triggers competition among the representations in each of these brain regions to choose, and thereby attend to, the object that has the most salient combination of sensory input and motivational feedback from its value category.

For blocking to work properly, the cells in sensory cortex and OFC need to obey the membrane equations of neurophysiology, also called shunting interactions, and to compete with each other using recurrent, or feedback, inhibitory interactions. Such recurrent shunting on-centre off-surround networks tend to conserve, or normalise, the total activity that is shared among their active cells (Grossberg, 1973, 1980, 2013b). Thus, if the activity of one object category gets most amplified by a favourable combination of feedforward and feedback signalling, then the activities of the object categories with which it is competing will be inhibited, leading to attentional blocking. Grossberg and Levine (1987) simulated attentional blocking using these CogEM interactions. An explanation of unblocking requires additional mechanisms that will be described in Section 2.14.

It should be noted that blocking and unblocking experiments share some properties with the Asaad et al. (2017) experiments on credit assignment. Section 3.19 clarifies why this is so.

2.8. Cognitive-emotional resonances, feeling of what happens, and somatic markers

Previous articles review additional psychological and neurobiological data that the CogEM model has explained and predicted, and how it compares with other models of cognitive-emotional dynamics; for example, Grossberg (2013a, 2017b). All of these data explanations are relevant, moreover, to the computation of desirability.

One such comparison relates to the ability of the CogEM model to explain and predict clinical data. Damasio (1999) has derived from clinical data a heuristic version of the CogEM model and used it to describe cognitive-emotional resonances that support ‘the feeling of what happens’. Each processing stage in Damasio’s model (see his Figure 6.1) corresponds to a processing stage in the CogEM circuit of Figure 2(a). In particular, Damasio’s ‘map of object X’ corresponds to the sensory cortical stage where invariant object categories are represented. His ‘map of the proto-self’ becomes the value category and its multiple interactions. His ‘second-order map’ becomes the object-value category. And his ‘map of object X enhanced’ becomes the object category as it is attentively amplified by feedback from the object-value category. Feedback from an object-value category to its object category closes an excitatory feedback loop between object, value, and object-value categories. Persistent activity through this loop – maintained long enough with the help of adaptively timed feedback from sensory cortex to OFC via the hippocampus (Figure 2(b)) – enables the attended object to achieve emotional and motivational significance and to drive the choice of motivated decisions that can trigger context-appropriate actions towards valued goals.

Figure 6.

(a) A gated dipole opponent process can generate habituative ON responses and transient OFF rebounds in response to the phasic onset and offset, respectively, of an input to the ON channel. These mechanisms are a phasic input to the ON channel that is turned on and off through time (input J), nonspecific arousal that is delivered equally to the ON and OFF pathways (input I), cell activities at each of the three processing stages in the ON and OFF pathways (variables x_i for the six cells that are labelled with indices i = 1–6), activity-dependent habituative transmitters for the ON and OFF pathways (variables z_i with i = 1 and 2, and denoted by the square synapses), competition across the ON and OFF pathways (pathways with plus and minus signs), and output thresholds that rectify the ON and OFF pathway output signals. The antagonistic rebound in response to offset of a phasic input, such as a fear-inducing shock to the ON pathway (variable J), is the transient OFF-response (e.g. relief) at the output stage of the OFF pathway. This rebound is energised by the tonic input I that equally arouses both the ON and the OFF pathways, even after the phasic input shuts off. The activities at the dipole’s several processing stages react to the phasic and tonic inputs in the following way: the ON and OFF cell activities x₁ and x₂ respond to the sum of tonic-plus-phasic ON pathway input I + J, and the tonic OFF pathway input I, respectively, before they generate output signals f(x₁) and f(x₂) to the next processing stage. Before they reach the next processing stage, these signals are multiplied, or gated, by the habituative transmitters z₁ and z₂, respectively. The gated output signals f(x₁)z₁ and f(x₂)z₂ excite the ON and OFF cell activities x₃ and x₄, respectively, at the next processing stage. The habituative transmitters convert the step-plus-baseline activity pattern x₁ in the ON channel into the overshoot-habituation-undershoot-habituation pattern at activity x₃. The baseline activity pattern x₂ in the OFF channel is converted into the habituated baseline activity x₄. Next, opponent competition occurs across the ON and OFF channels. As a result, the habituated baseline activity x₄ in the OFF channel is subtracted from the ON activity x₃ to yield x₅. The overshoot and undershoot in x₅ are now shifted down until they lie above and below the equilibrium activity zero, respectively, of x₅. Then, activity x₅ is thresholded to generate an ON output signal. This output signal has an initial overshoot of activity, followed by habituation. The negative activity in the undershoot of x₅ is inhibited to zero by the output threshold. When the signs of excitation and inhibition are reversed in the OFF channel, the activity x₆ is caused. Activity x₆ is simply the flipped, or mirror, image of x₅ with respect to the zero equilibrium activity. Positive (negative) activities in x₅ become negative (positive) activities in x₆. Thresholding x₆ again cuts off negative activities, thereby allowing only the flipped undershoot to generate the OFF channel output. This rectified output is the transient antagonistic rebound. Grossberg (1972b) mathematically proved that a sudden increment in arousal can also trigger an antagonistic rebound. Since ‘novel events are arousing’, this property enables an unexpected event to trigger an antagonistic rebound. See Grossberg and Seidman (2006, Appendix A) for a review of this proof. In summary, an antagonistic rebound is due to interactions between a phasic input, tonic arousal, habituative transmitter gating, competition, and thresholding. (b) A READ (REcurrent Associative Dipole) circuit is a gated dipole with excitatory feedback, or recurrent, pathways between activities x₇ and x₁, and activities x₈ and x₂. Feedback enables the READ circuit to maintain a stable motivational baseline to support an ongoing motivated behaviour. A sensory representation S_k sends conditionable signals to the READ circuit that are multiplied, or gated, by conditioned reinforcer adaptive weights, or long-term memory (LTM) traces, w_k7 and w_k8 to the ON and OFF channels, respectively. Read-out of previously learned adaptive weights is dissociated from read-in of new values of the learned weights. This dissociation allows new weight learning to be generated by teaching signals from the ON or OFF channel that wins the opponent competition. The combination of recurrent feedback and associative dissociation enables the adaptive weights to avoid learning baseline noise, while they maintain in short-term memory the relative balance of ON and OFF channel conditioning during a motivated act, and preserve their learned conditioned reinforcer associations until they are disconfirmed by predictive mismatches if and when new learning contingencies are experienced. Reprinted with permission from Grossberg and Schmajuk (1987). (c) Gated dipole opponent processes exhibit an Inverted-U behavioural response as a function of arousal level, as explained in the text. Reprinted with permission from Grossberg and Seidman (2006).

Such sustained activation through a positive feedback loop gives rise to a resonant brain state. In the present instance, it is called a cognitive-emotional resonance because it binds cognitive information in the object category to emotional information in the value category. A resonance is a dynamical state during which neuronal firings across a brain network are amplified and synchronised when they interact via excitatory feedback signals during a matching process that occurs between bottom-up and top-down pathways. Grossberg (2013a, 2017b) explains in detail how resonating cell activities focus attention upon a subset of cells, and how the brain can become conscious of attended events during a resonant state. It is called an adaptive resonance because the resonant state can trigger learning within the adaptive weights, or long-term memory (LTM) traces, that exist at the synapses of these pathways. In a cognitive-emotional resonance, these adaptive weights occur in conditioned reinforcer and incentive motivational pathways, among others. Several different types of adaptive resonances will be described below.

CogEM hereby embodies and anticipated key concepts of the ‘somatic marker hypothesis’ which proposes that decision-making is a process that depends upon emotion, while also providing a mechanistic neural explanation and simulations (e.g. Grossberg et al., 2008) of various different properties of amygdala and OFC in making these decisions (Baxter et al., 2000; Bechara et al., 1999; Schoenbaum et al., 2003).

CogEM has also proposed explanations of additional clinical data, such as the data about memory consolidation that were mentioned above (Franklin and Grossberg, 2017); data about Fragile X syndrome and some types of repetitive behaviours that are found in individuals with autism (Grossberg and Kishnan, 2018); and data about emotional, attentional, and executive deficits that are found in individuals with autism or schizophrenia (Grossberg, 2000b; Grossberg and Seidman, 2006).

2.9. MOTIVATOR: amygdala and basal ganglia dynamics during conditioning

In order to mechanistically explain how devaluation works, and to answer the questions posed at the end of Section 2.6, the MOTIVATOR model, which unlumps the CogEM model, is needed (Dranias et al., 2008; Grossberg et al., 2008; Figure 3(a)). The MOTIVATOR model combines the CogEM model with a model of how the basal ganglia (BG) learns to respond to expected and unexpected rewards (Figure 4(a)–(c); Brown et al., 1999). Thus, in addition to clarifying how value categories are learned, the MOTIVATOR model begins to explain how the amygdala and the BG interact with one another and with the temporal and orbitofrontal cortices to focus motivated attention and trigger choices aimed at acquiring valued goal objects.

Figure 3.

(a) The MOTIVATOR neural model explains and simulates key computationally complementary functions of the amygdala and basal ganglia (SNc) during conditioning and learned performance. The basal ganglia generate Now Print signals in response to unexpected rewards. These signals modulate learning of new associations in many brain regions. The amygdala supports motivated attention to trigger actions that are expected to occur in response to conditioned or unconditioned stimuli. Object Categories represent visual or gustatory inputs in anterior inferotemporal (ITA) and rhinal (RHIN) cortices, respectively. Value Categories represent the value of anticipated outcomes on the basis of hunger and satiety inputs, in amygdala (AMYG) and lateral hypothalamus (LH). Object-Value Categories resolve the value of competing perceptual stimuli in medial (MORB) and lateral (ORB) orbitofrontal cortex. The Reward Expectation Filter detects the omission or delivery of rewards using a circuit that spans ventral striatum (VS), ventral pallidum (VP), striosomal delay (SD) cells in the ventral striatum, the pedunculopontine nucleus (PPTN) and midbrain dopaminergic neurons of the substantia nigra pars compacta/ventral tegmental area (SNc/VTA). The circuit that processes CS-related visual information (ITA, AMYG, ORB) operates in parallel with a circuit that processes US-related visual and gustatory information (RHIN, AMYG, MORB). (b) Reciprocal adaptive connections between lateral hypothalamus and amygdala enable amygdala cells to become learned value categories. The bottom region represents hypothalamic cells, which receive converging taste and metabolite inputs whereby they become taste-drive cells. Bottom-up signals from activity patterns across these cells activate competing value category, or US Value Representations, in the amygdala. A winning value category learns to respond selectively to specific combinations of taste-drive activity patterns and sends adaptive top-down priming signals back to the taste-drive cells that activated it. CS-activated conditioned reinforcer signals are also associatively linked to value categories. Adaptive connections end in (approximate) hemidiscs. See text for details.

Source: Adapted with permission from Dranias et al. (2008).

Figure 4.

(a) Model circuit for the control of dopaminergic Now Print signals in response to unexpected rewards. Cortical inputs (I_i), activated by conditioned stimuli, learn to excite the SNc via a multi-stage pathway from the ventral striatum (S) to the ventral pallidum and then on to the PPTN (P) and the SNc (D). The inputs I_i excite the ventral striatum via adaptive weights W_iS, and the ventral striatum excites the PPTN via double inhibition through the ventral pallidum, with strength W_SP. When the PPTN activity exceeds a threshold GP, it excites the SNc with strength W_PD. The striosomes, which contain an adaptive spectral timing mechanism (x_ij, G_ij, Y_ij, Z_ij), learn to generate adaptively timed signals that inhibit reward-related activation of the SNc. Primary reward signals (I_R) from the lateral hypothalamus both excite the PPTN directly (with strength W_RP) and act as training signals to the ventral striatum S (with strength W_RS) that trains the weights W_iS. Arrowheads denote excitatory pathways, circles denote inhibitory pathways, and hemidiscs denote synapses at which learning occurs. Thick pathways denote dopaminergic signals. Reprinted with permission from Brown et al. (1999). (b) Dopamine cell firing patterns: Left: data. Right: model simulation, showing model spikes and underlying membrane potential. (A) In naive monkeys, the dopamine cells fire a phasic burst when unpredicted primary reward R occurs, such as if the monkey unexpectedly receives a burst of apple juice. (B) As the animal learns to expect the apple juice that reliably follows a sensory cue (conditioned stimulus, CS) that precedes it by a fixed time interval, then the phasic dopamine burst disappears at the expected time of reward, and a new burst appears at the time of the reward-predicting CS. (C) After learning, if the animal fails to receive reward at the expected time, a phasic depression, or dip, in dopamine cell firing occurs. Thus, these cells reflect an adaptively timed expectation of reward that cancels the expected reward at the expected time. The data are reprinted with permission from Schultz et al. (1997). The model simulations are reprinted with permission from Brown et al. (1999). (c) Dopamine cell firing patterns: Left: data. Right: model simulation, showing model spikes and underlying membrane potential. (A) The dopamine cells learn to fire in response to the earliest consistent predictor of reward. When CS2 (instruction) consistently precedes the original CS (trigger) by a fixed interval, the dopamine cells learn to fire only in response to CS2. Data reprinted with permission from Schultz et al. (1993). (B) During training, the cell fires weakly in response to both the CS and reward. Data reprinted with permission from Ljungberg et al. (1992). (C) Temporal variability in reward occurrence: When reward is received later than predicted, a depression occurs at the time of predicted reward, followed by a phasic burst at the time of actual reward. (D) If reward occurs earlier than predicted, a phasic burst occurs at the time of actual reward. No depression follows since the CS is released from working memory. Data in C and D reprinted with permission from Hollerman and Schultz (1998). (E) When there is random variability in the timing of primary reward across trials (e.g. when the reward depends on an operant response to the CS), the striosomal cells produce a Mexican Hat depression on either side of the dopamine spike. Data reprinted with permission from Schultz et al. (1993). Model simulation reprinted with permission from Brown et al. (1999).

The BG need to be discussed along with the amygdala because it plays an important role in both the cognitive-emotional and working memory learning that are relevant to this article’s explanatory goals (Figure 1). Indeed, the amygdala and BG seem to play computationally complementary roles (Grossberg, 2000a), with the BG triggering Now Print learning signals in response to unexpected rewards, and the amygdala learning to activate incentive motivational signals with which to help acquire expected rewards. A Now Print learning signal is a signal that is broadcast broadly to many brain regions where it can modulate learning at all of its recipient neurons (Grossberg, 1974; Harley, 2004; Livingston, 1967; McGaugh, 2003).

As in CogEM, the model amygdala and LH interact to calculate the expected current value of the subjective outcome that the CS predicts, constrained by the current state of deprivation or satiation. The amygdala then relays the expected value information to orbitofrontal cells (ORB in Figure 3(a) and OFC in Figure 1) that receive visual inputs from anterior inferotemporal cells (ITA; Amaral and Price, 1984; Ghashghaei and Barbas, 2002; Öngür and Price, 2000; Reep et al., 1996) and to medial orbitofrontal cells (MORB in Figure 3(a)) that receive gustatory inputs from rhinal cortex (RHIN) (Barbas, 1993, 2000; Barbas et al. 1999; Reep et al., 1996). The activations of these orbitofrontal cells code the expected subjective values of objects. These values guide behavioural choices.

The review of Levy and Glimcher (2012) discusses the OFC computation of expected subjective value from the perspective of neuroeconomic theory. Also of neuroeconomic interest is the Grossberg and Gutowski (1987) exposition of how the CogEM model explains and simulates data of Kahneman and Tversky (1979) about decision-making under risk, thereby explaining Prospect Theory axioms using neural designs that are essential for survival, as well as data that Prospect Theory cannot explain, such as data about preference reversals.

The model BG, or Reward Expectation Filter (Figures 3(a) and 4(a)), detects errors in CS-specific predictions of the value and timing of rewards (Ljungberg et al., 1992; Schultz, 1998; Schultz et al., 1992, 1993, 1995, 1997). Excitatory primary rewarding inputs from the LH reach the substantia nigra pars compacta (SNc), via the pedunculopontine nucleus (PPTN). The SNc also receives adaptively timed inhibitory inputs from model striosomes in the ventral striatum. Mismatches between these signals can trigger widespread dopaminergic burst and dip Now Print signals from cells in SNc (Figure 4(a)) and the ventral tegmental area (VTA in Figure 3(a)). Learning in cortical and striatal regions is strongly modulated by these Now Print signals, with dopamine bursts strengthening conditioned links and dopamine dips weakening them. In particular, such a dopaminergic Now Print signal modulates learning in the two pathways that are activated by a CS in Figure 4(a). After learning occurs, one pathway, via the ventral striatum to the SNc, enables the CS to activate the SNc and generate its own Now Print signals. The other pathway, via the striosomes to the SNc, enables the CS to inhibit responses to an expected US from the LH. This SNc circuit will be important in considering the kinds of reinforcement learning that are spared when amgydala and/or OFC are lesioned.

MOTIVATOR was used to explain and simulate psychological and neurobiological data from tasks that examine food-specific satiety, Pavlovian conditioning, reinforcer devaluation, and simultaneous visual discrimination, while retaining the ability to simulate neurophysiological data about SNc. Model simulations successfully reproduced the neurophysiologically recorded dynamics of hypothalamic cell types, including signals that predict saccadic reaction times and CS-dependent changes in systolic blood pressure.

In order to more fully understand how food-specific satiation occurs, more needs to be said about how value categories in the amygdala are learned as a result of adaptive feedback interactions with the LH, and why, in order to properly regulate reinforcement learning and affective prediction, some of these pathways need to receive internal drive inputs and/or habituative reinforcing inputs. This explanation will be given in Section 2.11.

2.10. Temporal difference models of BG responses to unexpected rewards

The Reward Expectation Filter of the MOTIVATOR model was first published in Brown et al. (1999) who used it to explain and simulate many experiments about how SNc reacts to rewards whose amplitude or timing is unexpected. Several other models have attempted to describe the SNc cell behaviour using a temporal difference (TD) algorithm (Montague et al., 1996; Schultz et al., 1997; Suri and Schultz, 1998). These models suggest that the dopaminergic SNc cells compute a temporal derivative of predicted reward. In other words, they fire in response to the sum of the time-derivative of reward prediction and the actual reward received. These models have not been linked to structures in the brain that might compute the required signals. As Brown et al. (1999) have noted, the Suri and Schultz (1998) model has simulated some of the known dopamine cell data. However, their model can only learn a single fixed interstimulus interval (ISI) that corresponds to the longest-duration timed signal (xlm(t) in their model). If the ISI is shorter than this, dopamine bursts will strengthen all of the active stimulus representations predicting reward at the time of the dopamine burst or later. Thus, their model generates inhibitory reward predictions beyond the primary reward time, and predicts a lasting depression of dopamine firing subsequent to primary reward, which is not found in the data, data that the Brown et al. (1999) model explains.

In contrast to TD models that compute time derivatives immediately prior to dopamine cells, our model uses two distinct pathways: the ventral striatum and PPTN for initial excitatory reward prediction, and the striosomal cells for timed, inhibitory reward prediction. The fast excitation and delayed inhibition are hereby computed by separate structures within the brain, rather than by a single temporal differentiator. This separation avoids the problem of the Suri and Schultz (1998) model by allowing transient rather than sustained signals to cancel the primary reward signal, thereby enabling precisely timed reward-cancelling signals to be trained, and preventing spurious sustained inhibitory signals to the dopamine cells.

2.11. Learning value categories for specific foods and effects of their removal

Figure 3(b) diagrams how the MOTIVATOR model conceptualises the learning of a value category as a result of reciprocal adaptive interactions between the LH and the amygdala (AMYG). This figure summarises how the model embodies a network that calculates the drive-modulated affective value of a food US (Cardinal et al., 2002); notably, how selective responses to different foods can be acquired. Animals have specific hungers that vary inversely with blood levels of metabolites such as sugar, salt, protein, and fat (Davidson et al., 1997). Similarly, the gustatory system has chemical sensitivities to tastes such as sweet, salty, umami, and fatty (Kondoh et al., 2000; Rolls et al., 1999). An AMYG value category (top level in Figure 3(b)) learns to respond to particular LH combinations of these metabolites and tastes (bottom level in Figure 3(b)) in a selective fashion, hence can represent specific hungers.

MOTIVATOR begins its computation of food-specific selectivity with the lower layer of the model’s LH cells in Figure 3(b). These cells perform pairwise multiplications, each involving a taste and its corresponding drive level. They are therefore called taste-drive cells. LH neurons such as glucose-sensitive neurons provide examples of LH cells that are both chemical- and taste-sensitive. Indeed, glucose-sensitive neurons are excited by low glucose levels, inhibited by high glucose levels, and respond to the taste of glucose with excitation (Karadi et al., 1992; Shimizu et al., 1984).

The activation pattern across all these taste-drive cells is projected via converging adaptive pathways to a higher cell layer and summed there by an AMYG value category cell that represents the current value of the specific food US. These cells are therefore also called US-value cells. Such food selective US-value representations can be learned from a competitive learning process (Grossberg, 1976a, 1978a) that associates distributed activation patterns at the taste-drive cells with compressed representations at the US-value cells that survive the competition at the AMYG processing level. The resulting US-value cells in the AMYG help to explain data about neurons in the AMYG that respond selectively to specific foods or associated stimuli in a manner that reflects the expected consumption value of the food (e.g. Nishijo et al., 1988a, 1988b).

Figure 3(a) and (b) illustrates the hypothesis that a visual CS becomes a conditioned reinforcer by learning to activate a US-value representation in the AMYG during CS-US pairing protocols. Despite the fact the CS generates no gustatory inputs to the taste-drive cells and is not actually consumed, the model can use this CS-US association to compute the prospective value of the US, given current drives, during the period between CS onset and the delivery of the food US. The model can do this if the CS-activated US-value representation in the AMYG can, in turn, activate the taste-drive cells in the LH that have activated it in the past, when the US was being consumed.

This is accomplished, as depicted in Figure 3(a) and (b), by adaptive top-down pathways, or learned expectations, from the US-value cells in the AMYG to the taste-drive cells in the LH. The resultant bidirectional adaptive signalling between taste-drive LH cells and integrative US-value AMYG cells can prime the taste-value combinations that are expected in response to the conditioned reinforcer CS. Such reciprocal adaptive interactions have also been shown, as part of Adaptive Resonance Theory (ART), to be capable of stabilising category learning and memory, in whatever brain systems they occur (see Section 2.12; Carpenter and Grossberg, 1987, 1991; Grossberg, 1980, 2013a, 2017b). An ART circuit complements the bottom-up adaptive filter of a competitive learning model with adaptive top-down expectation signals, among other extensions. ART was introduced to overcome the catastrophic forgetting that occurs when only bottom-up learning occurs (Grossberg, 1976a, 1976b). Without top-down expectations to dynamically stabilise the learning of value categories, memory instability could become as great a problem in LH-AMYG dynamics as it would be in the learning of invariant object categories by the inferotemporal cortex (Figures 2 and 3).

These properties of AMYG value categories help to explain how an animal’s behaving changes when its AMYG is lesioned. When the AMYG is lesioned, the ability to selectively respond to specific foods is eliminated. The reduced drive inputs of a satiated food will then not be able to cause a smaller activation of its AMYG value category. Also lost will be the competition among value categories that would determine the choice of a non-satiated food in a normal animal.

If a food was visually presented to a normal animal in order to satiate it, then both reduced internal drive and external cue inputs could contribute to the choice of a non-satiated food (Figures 2 and 3). In particular, as above, eating a lot of food would lead to shrinking appetitive inputs and growing satiety drive inputs to the LH. Seeing the food repeatedly during each eating event could also habituate the conditioned reinforcer inputs that activate the corresponding AMYG value category.

It should immediately be noted that not all responsiveness to reinforcing cues is eliminated by lesions of the AMYG. The BG can still be fully functional. As illustrated by Figure 4(a), LH inputs can still regulate learning of dopaminergic Now Print signals from the SNc to large parts of the brain, including the PFC, in response to unexpected rewards. As will be discussed in Section 3, these Now Print signals can support learning of representations that are sensitive to the probability of reward over a series of previous trials.

2.12. Another mismatch mechanism for correcting disconfirmed behaviour: ART

This BG mechanism for processing an unexpected reinforcing event is not the only way that unexpected events are processed by the brain. Another fundamental process is also at work in perceptual and cognitive processes that greatly influences how the PFC makes predictions that are sensitive to the probability of reward over a series of previous trials. This process enables humans and other primates to rapidly learn new facts without being forced to just as rapidly forget what they already know, even if no one tells them how the rules of each environment differ or change through time. When such forgetting does occur, it is often called catastrophic forgetting.

Grossberg (1980) called the problem whereby the brain learns quickly and stably without catastrophically forgetting its past knowledge the stability-plasticity dilemma. ART was introduced to explain how brains solve the stability-plasticity dilemma. Since its introduction in Grossberg (1976a, 1976b), ART has been incrementally developed into a cognitive and neural theory of how the brain autonomously learns to attend, recognise, and predict objects and events in a changing world, without experiencing catastrophic forgetting. ART currently has the broadest explanatory and predictive range of available cognitive and neural theories. See Grossberg (2013a, 2017b) for reviews.

ART prevents catastrophic forgetting by proposing how top-down expectations focus attention on salient combinations of cues, called critical feature patterns. When a good enough match occurs between a bottom-up input pattern and a top-down expectation, a synchronous resonant state emerges that embodies an attentional focus. Such a resonance is capable of driving fast learning that incorporates the attended critical feature pattern into the bottom-up adaptive filters that activate recognition categories, and the top-down expectations that are read out by them – hence the name adaptive resonance – while suppressing outliers that could have caused catastrophic forgetting.

In contrast, when a bottom-up input pattern represents an unexpected event, it can cause a big mismatch with the currently active top-down expectation. As illustrated in Figure 5, such a mismatch, or disconfirmed expectation, can trigger a burst of nonspecific arousal (‘novel events are arousing’). It is called nonspecific arousal because it affects all category representations equally, since the orienting system that triggers such an arousal burst has no information about which active categories caused the mismatch, and thus must be reset (Figure 5(c)). A nonspecific arousal burst can reset whatever categories caused the mismatch (Figure 5(d)) and initiate a search for a more predictive category. In this way, rapid reset of an object category can occur when it leads to disconfirmed behaviours, and a more predictive category can automatically be chosen in its stead.

Figure 5.

ART search and learning cycle. This figure summarises how ART searches for and learns a new recognition category using cycles of match-induced resonance and mismatch-induced reset due to interactions of an attentional system and an orienting system. (a) Input pattern I is instated across feature detectors at level F₁ of the attentional system as an activity pattern X, at the same time that it generates excitatory signals to the orienting system A with a gain ρ that is called the vigilance parameter. The activity pattern X is represented by a shaded region in (a) and (d). Activity pattern X generates inhibitory signals to the orienting system A as it generates a bottom-up input pattern S to the category level F₂. A dynamic balance within A between excitatory inputs from I and inhibitory inputs from S keeps A quiet. The bottom-up signals in S are multiplied by learned adaptive weights to form the input pattern T to F₂. The inputs T are contrast-enhanced and normalised within F₂ by recurrent lateral inhibitory signals that obey the membrane equations of neurophysiology, otherwise called shunting interactions (see section 3.15). This competition leads to selection and activation of a small number of cells within F₂ that receive the largest inputs. The chosen cells represent the category Y that codes for the feature pattern at F₁. In this figure, a winner-take-all category is chosen, represented by a single cell (population). (b) The category activity Y generates top-down signals U that are multiplied by adaptive weights to form a prototype, or critical feature pattern, V that encodes the expectation that the active F₂ category has learned for what feature pattern to expect at F₁. This top-down expectation input V is added at F₁ cells using the ART Matching Rule, whereby object attention activates a top-down, modulatory on-centre, off-surround network. If V mismatches I at F₁, then a new STM activity pattern X* (the grey pattern in (b) and (c); white regions represent inhibited cells) is selected at cells where the patterns match well enough. In other words, X* is active at I features that are confirmed by V. Mismatched features (white area) are inhibited. When X changes to X*, total inhibition decreases from F₁ to A. (c) If inhibition decreases sufficiently, the orienting system A releases a nonspecific arousal burst to F₂; that is, ‘novel events are arousing’. Within the orienting system A, a vigilance parameter ρ determines how bad a match will be tolerated before a burst of nonspecific arousal is triggered. This arousal burst triggers a memory search for a better-matching category, as follows: arousal resets F₂ by inhibiting Y. (d) After Y is inhibited, X is reinstated and Y stays inhibited as X activates a different winner-take-all category Y*, at F₂. Search continues until a better matching, or novel, category is selected. When search ends, a feature-category resonance triggers learning of the attended data in adaptive weights within both the bottom-up and top-down pathways, at the same time that it supports conscious recognition of the attended object (Grossberg, 2013a, 2017b). As learning stabilises, inputs I can activate their globally best-matching categories directly through the adaptive filter, without activating the orienting system.

Source: Adapted with permission from Carpenter and Grossberg (1987).

Figure 5 illustrates the limiting case in which the category level makes a winner-take-all choice of a single winning cell, or cell population. In general, the category level compresses the distributed patterns on the feature level, but does not necessarily chose winner-take-all categories; for example, Carpenter (1997). The two processing levels in Figure 5 schematise a kind of dynamics that may be repeated in multiple brain regions within the What cortical stream. Figure 18 below illustrates how this can happen in the laminar circuits of neocortex.

Figure 18.

The LAMINART model clarifies how bottom-up, horizontal, and top-down interactions within and across cortical layers in V1 and V2 interblob and pale stripe regions, respectively, carry out bottom-up adaptive filtering, horizontal grouping, and top-down attention to carry out perceptual grouping, including boundary completion. Similar interactions seem to occur in all six-layered cortices. See text for details.

Source: Reprinted with permission from Raizada and Grossberg (2001).

How do brain circuits selectively respond to a nonspecific arousal burst in order to inhibit unpredictive categories and trigger a memory search? One mechanism is by causing antagonistic rebounds in opponent processing circuits. Such rebounds can occur in perceptual, cognitive, and affective brain circuits. They will now be explained for the special case of LH-AMYG interactions that are needed to explain how devaluation and reversal learning work.

2.13. Opponent processing by gated dipoles in object and value categories

The LH-AMYG pathways in Figures 2 and 3 include circuits that control opponent emotional and motivational states, such as fear versus relief, and hunger versus satiety. These opponent circuits can trigger an antagonistic rebound in response to two kinds of input changes: changes in the amount of reward or punishment, such as a sudden increase or reduction in the amount of food or shock level; or to an unexpected event, such as the non-occurrence of an expected shock. The rebound that is caused by the non-occurrence of an expected event is triggered by a burst of nonspecific arousal to the dipole (Figure 6(a)) due to a mismatch with the expectation of the non-occurring event (Figure 5(c)). These rebounds can rapidly reset a currently active value category, and the amount of incentive motivation with which it was supporting an ongoing valued behaviour, while simultaneously helping to choose more predictive representations with which to learn and perform the new environmental contingency.

These opponent circuits are modelled by gated dipoles (Dranias et al., 2008; Grossberg, 1972a, 1972b, 1980, 1984a, 2000b; Grossberg et al., 2008; Grossberg and Seidman, 2006). The simplest gated dipole circuit is depicted in Figure 6(a) (Grossberg, 1972b). It has non-recurrent, or feedforward, ON and OFF channels, or pathways. ON and OFF cells within the opponent pathways of a gated dipole can represent an opponent pair of emotional and motivational states. Gated dipoles have helped to explain many data about both classical and operant conditioning, including conditioned extinction, learned escape and avoidance, attentional unblocking, partial reinforcement acquisition effect, gambling behaviours, and self-punitive behaviours – behavioural properties that currently seem to have no other mechanistic neural explanations. Simple mechanisms, occurring in a prescribed order, enable gated dipoles to cause antagonistic rebounds either in response to changes in reinforcer amplitude or to disconfirmations of cognitive expectations of reward. As explained by ART, disconfirmation of a cognitive expectation can cause a nonspecific arousal burst that is broadcast throughout the brain (Figure 5(c)). When such an arousal burst is received by a hypothalamic gated dipole, it can cause an antagonistic rebound. The caption of Figure 6(a) explains how an antagonistic rebound occurs.

Neurophysiological data from hypothalamic ‘opposite cells’ match affective gated dipole properties, including their opponent and rebound properties (Nakamura et al., 1987; Nakamura and Ono, 1986; Ono et al., 1986). These hypothalamic properties have been simulated by the MOTIVATOR model (Figure 3(a); Dranias et al., 2008; Grossberg et al., 2008). The ON and OFF channels within a gated dipole in the LH of the MOTIVATOR model delivers inputs to the AMYG value categories which, in turn, provide incentive motivational signals to OFC object-value categories, and thereby influences what actions are taken to achieve valued goals. Animals with an intact AMYG and OFC can hereby use hypothalamic rebounds to flexibly choose value and object-value categories that can track changing reinforcement contingencies, and thereby update an option’s desirability. Indeed, the primate AMYG contains separate cell populations that respond to positively and negatively valued visual stimuli (Paton et al., 2006), can carry out moment-by-moment tracking of state value (Belova et al., 2008; Morrison and Salzman, 2010), and are modulated by unexpected events (Belova et al., 2007).

Opponent processing circuits with gated dipole rebound properties also occur in perceptual and cognitive brain regions, and reflect the ability of these representations to be rapidly reset, and indeed reversed, when stimulus conditions change. For perceptual examples of how a change in stimulus level can cause a rebound, consider negative aftereffects: Offset of sustained viewing of intersecting radial lines leads to an opponent MacKay negative aftereffect of concentric circles. Offset of sustained viewing of a red surface leads to an opponent green surface aftereffect. Offset of sustained viewing of a video of water flowing downwards leads to an opponent waterfall aftereffect of motion upwards. See Francis and Grossberg (1996) for a model of boundary grouping and surface filling-in that explains and simulates such perceptual aftereffects.

Rebounds also help to carry out tasks in response to changes in perceptual cues. As one example, consider the task for humans of pushing a buzzer, or for pigeons of pecking a key, as fast as possible when a red light shuts off. If the only thing that happened in the brain when the light shuts off was the termination of activity within a category that codes for red (among other features), then there would be no internal signal at stimulus offset to activate the buzzer press. If, however, offset of the ON cell (population) that codes for red triggers an antagonistic rebound in an associated OFF cell (population), then activation of the OFF cell can learn to be associated with the buzzer press, or key peck, command.

2.14. Affective antagonistic rebounds, unblocking, and reversal learning

Antagonistic rebounds facilitate tracking and learning about changing reinforcement contingencies in many other types of situations, including the learning of escape behaviours, attentional unblocking (Section 2.7), and reversal learning. For example, either a sudden reduction of a fearful shock, or the non-occurrence of an expected shock, can cause a relief rebound (Denny, 1971; Masterson, 1970; Reynierse and Rizley, 1970) that can be used to motivate new conditioned responses. The unexpected non-occurrence of food can, in contrast, cause a frustrative rebound (Amsel, 1962, 1992) that can be used to suppress unsuccessful responses. Thus, rebounds can occur from negative to positive affects, such as from fear to relief, or from positive to negative affects, such as from hunger to frustration. These antagonistic rebounds enable the brain to learn responses that quickly adapt to changing reinforcement contingencies.

For example, suppose that an animal who is experiencing a shocked Skinner box floor accidentally pushes a red button that causes the shock to stop and opens a door whereby to escape from the Skinner box. The sudden termination of shock causes a relief rebound within its value category. The object category that is activated by the red button can then be associated with the relief-activated value category, leading to new conditioned reinforcer learning, incentive motivational learning, and motor learning (Figure 2(a)). In this way, conditioned relief can be used to motivate the learned escape behaviour in the future. In the opposite direction, the frustrative rebound that occurs after expected food does not occur can drive forgetting, or extinction, of motivational support for the consummatory actions that no longer lead to food, thereby releasing exploratory behaviours to search for more productive sources of food.

Using the same mechanisms, the unexpected change in reinforcer amplitude in an unblocking experiment, or an unexpected change in reward schedule in a reversal experiment, can cause antagonistic rebounds that immediately modify the net incentive motivation that is controlling ongoing behaviour, while also triggering rapid relearning, via the newly activated value categories, of the conditioned reinforcer, and incentive motivational, and motor learning pathways that will control subsequent motivated choices. These hypothalamic rebounds can also help reversal learning to occur in circuits that do not involve AMYG and OFC, such as the hypothalamic pathway via the SNc that causes Now Print signals to be broadcast to multiple brain regions, including PFC (Figures 3(a) and 4(a)).

In an unblocking experiment, at least two kinds of events, one cognitive and one cognitive-emotional, occur in parallel. The cognitive-emotional event is due to the antagonistic rebounds that have just been described. They cause activation of a new combination of value categories. The cognitive event enables a more predictive set of perceptual features and object categories to be attended, including those that would have remained blocked if there was no change in the expected reinforcer (see Section 2.7). Unblocking happens when a burst of nonspecific arousal is triggered by the unexpected reinforcer and causes new features and categories to be selected and attended (Figure 5(c) and (d)). Then, the new features and categories can be associated with the newly activated value categories to cause new conditioned reinforcer, incentive motivational, and motor learning that can, for example, lead to escape from the shocked Skinner box floor.

When a shock is unexpectedly reduced, it can cause a rebound in the relief channel that can more than cancel the direct effect of the reduced shock input to the fear channel. In this way, an unexpectedly small shock can act as a positive reinforcer, despite the fact that shock is a negative US (Grossberg, 1972b; Section 6). Sudden flips of affective sign can occur in other ways as well in a gated dipole opponent processing circuit. One such effect is called by Robinson and Berridge (2013) an ‘instant transformation of learned repulsion into motivational “wanting”’ (p. 282). In this kind of experiment, a lever is associated with an intensely unpleasant concentration of salty water. The ensuing aversive behaviours after presentation of the lever can be immediately transformed into positive ‘wanting’ responses by injections of deoxycorticosterone and furosemide that mimic sodium deficiency.

Wanting can also be increased in hyperdopaminergic mutant mice who embody a dopamine transporter (DAT) knockdown mutant (Zhuang et al., 2001) that causes 70% higher extracellular dopamine levels in the striatum (Peciña et al., 2003). One way that wanting can be increased in these mice is that larger dopaminergic Now Print signals (Figures 3(a) and 4(a)) during learning may strengthen the conditioned reinforcer, incentive motivation, and motor learning pathways that support stronger ‘wanting’ responses.

2.15. READ circuits: feedback enables stable motivation and secondary conditioning

Once rebound mechanisms of a gated dipole are understood, it is then possible to explain how rebounds can lead to new reinforcement learning, including reversal learning and changes in object desirability. To fully achieve these properties, the non-recurrent dipole of Figure 6(a) needs to be replaced by a recurrent dipole, with positive feedback pathways within the ON and OFF opponent channels. The recurrent gated dipole in Figure 6(b) is called a READ circuit, for REcurrent Associative Dipole (Grossberg and Schmajuk, 1987). READ circuits model the hypothalamic gated dipoles in the MOTIVATOR model. They are omitted in Figure 3(a) for simplicity.

Due to recurrent feedback in both the ON and OFF channels of a READ circuit, activity x₇ reactivates x₁ in the ON channel, while activity x₈ reactivates x₂ in the OFF channel. As in Figure 6(a), there are also habituative transmitter gates z₁ and z₂ in the ON and OFF channels, respectively. In addition, adaptive weights, or LTM traces, w_k7 and w_k8, within the black hemidisc synapses, sample the ON and OFF channels, respectively, in response to sampling signals S_k that are activated by different CSs. These CSs become conditioned reinforcers when they are associated with reinforcing US inputs at the gated dipole (input J in Figure 6(b)). Any number k of sampling signals, that are activated by different CS signals S_k, can converge on a single READ circuit. In this way, multiple CSs, with different levels of desirability, can all benefit from the same emotional and motivational hypothalamic circuits.

Properties of reversal learning and changes in object desirability build upon the ability of a READ circuit to simulate data about primary and secondary excitatory and inhibitory conditioning, among other important properties. The utility of a recurrent anatomy is vividly illustrated by the case of secondary inhibitory conditioning. Suppose that CS₁ is associated with a US shock input J until it becomes a source of conditioned fear. For this to happen, the adaptive weights of CS₁ must occur after the position in the ON channel where the US input J is registered, so that they can sample the activity caused by the US. After CS₁ has become a source of conditioned fear, suppose that onset of a different CS₂ is associated with the offset of CS₁ so that it can sample the antagonistic rebound in the relief channel, and thereby learn how to become a source of conditioned relief. For this to happen, CS₁ must deliver its input before the habituative gates, so that its offset can cause the rebound in the OFF channel. In contrast, CS₂ must occur after the habituative gates, where it can sample the rebound, and thereby achieve secondary inhibitory conditioning.

What does this have to do with recurrent connections? The following sleight of hand makes this clear: this experiment could have been done with any CS₁ and CS₂. If we now interchange the cues that are used as CS₁ and CS₂, then by the above argument, each CS must occur both before and after the habituative gates. This can only happen if the network has recurrent connections within the ON and OFF channels.

A READ circuit exhibits several other basic functional properties that enable it to learn and perform motivated behaviours in a naturalistic environment. These properties were demonstrated with computer simulations in Grossberg and Schmajuk (1987). First, it can maintain steady motivation while a behaviour is being performed, even during sufficiently small environmental distractions. This happens because the READ circuit is a specialised recurrent on-centre off-surround network that is capable of storing activity patterns in short-term memory (STM). Section 3.15 explains how storage occurs.

Second, a READ circuit can rapidly switch to support a new behaviour with a different motivation if the distraction, or change in reinforcement contingency, is big enough. This is just a reset of STM in response to external inputs that change enough to overcome the hysteresis caused by the recurrent excitatory feedback.

Third, a READ circuit enables affective learning to remain sensitive to any number of reinforcing events throughout the lifespan; its LTM traces do not saturate. In other words, if conditioning continued to occur over many trials, one could imagine all the LTM traces reaching their maximum values, after which no future conditioning would be possible. LTM saturation is prevented from happening in a READ circuit in the following way: the LTM traces sample dendritic activities (the thick black bars in Figure 6(b)) of cells with activities x₇ and x₈. These dendrites receive teaching signals in the form of retrograde calcium spikes from the cell bodies (Grossberg, 1975; Magee and Johnston, 1997; Markram et al., 1995, 1997). These teaching signals occur after the ON and OFF channels have undergone opponent competition. Due to opponent competition, if both ON and OFF channels had the same activity, their teaching signals would be inhibited to zero, so that the LTM traces that sample them would also approach zero. When the ON and OFF channels have different activities, the competition computes teaching signals that are normalised net activities because the activities in such a recurrent on-centre off-surround network are normalised. The LTM traces can continue to learn these net values throughout life, without ever saturating.

Fourth, a READ circuit enables affective memories to be preserved for a long time, even years, until reward or punishment schedules change, or cognitive expectations are disconfirmed. Such stable memories help to explain the persistence of instrumental avoidance behaviours and why Pavlovian conditioned inhibitors do not extinguish, among other conditioning data (Grossberg, 1972a; Grossberg and Schmajuk, 1987; Kamin et al., 1963; Lysle and Fowler, 1985; Maier et al., 1969; Miller and Schachtman, 1985; Owren and Kaplan, 1981; Solomon et al., 1953; Witcher, 1978; Zimmer-Hart and Rescorla, 1974). The stable affective memory happens because, when LTM traces are read into STM by gating the signal from a recall probe S_k in Figure 6(b), they instate in STM the normalised net pattern that they have learned. The sampled LTM traces then, in turn, sample this STM pattern and thereby ensure their stability under recall.

2.16. Inverted-U: emotional depression in mental disorders

Gated dipoles have other properties without which reinforcement learning and motivated attention would not be possible. These properties would profitably be more deeply probed by modern neurophysiological methods.

A property of major importance is that the activity of a gated dipole circuit exhibits an Inverted-U as a function of its arousal level (Figure 6(c)). The Inverted-U is a consequence of the same mechanisms that enable a gated dipole to trigger antagonistic rebounds and to thereby quickly adapt to changing reinforcement contingencies. The Inverted-U can be traced to how the state of habituation in the dipole’s transmitter gates (square synapses in Figure 6(a) and (b)) divide the effects of signals through the dipole. This division creates a Weber Law of dipole responsiveness.

In particular, a gated dipole can support normal behavioural dynamics if its tonic arousal level, as distinct from phasic arousal bursts in response to unexpected events, remains within an optimal range that causes peak values of the Inverted-U to occur (Figure 6(c)). These intermediate arousal input sizes generate a Golden Mean of responding that enables sufficient activation of AMYG value categories to occur. Maintaining this optimal range during waking hours is a major achievement of the affective brain. Future experiments are needed to probe how this example of homeostatic plasticity is maintained. Failure to do so is reflected in behavioural symptoms of several mental disorders, including autism and schizophrenia (Grossberg, 2000b; Grossberg and Seidman, 2006), as well as attention deficit hyperactivity disorder, or ADHD (see below).

Gated dipoles play a role in causing symptoms of these mental disorders when the arousal level remains either abnormally small or abnormally large. At either extreme, gated dipole outputs are depressed, but in different ways. Abnormally small arousal causes an underaroused depressive syndrome, whereas abnormally large arousal causes an overaroused depressive syndrome.

In an underaroused gated dipole, the response threshold to inputs is abnormally high but, after input intensity exceeds this elevated threshold, further increments in input intensity lead to hypersensitive emotional responses (Figure 6(c)). This happens because the habituative transmitter that divides dipole responses in abnormally small. In an overaroused gated dipole, the threshold to inputs is abnormally low but, despite the ability of inputs to easily exceed this threshold, all dipole responses are hyposensitive, leading to a flat emotional affect, insufficient incentive motivation, and a hypofrontal condition that is insufficient to support normal executive functions. This happens because the habituative transmitter that divides dipole responses is abnormally large. Underarousal may be one cause of behavioural symptoms in individuals with autism (Baker et al., 2017; Bujnakova et al., 2016) that are explained in Grossberg and Seidman (2006), whereas overarousal may be one cause of behavioural symptoms in schizophrenia (Ban, 1973; Depue, 1974; Haralanova et al., 2011) that are explained in Grossberg (2000b).

Other mental disorders may also reflect these underaroused and overaroused depressive properties. For example, many individuals with ADHD seem to be underaroused; for example, Mayer et al. (2016). The underaroused transmitter has been reported to be dopamine, and it gives rise to the kind of hypersensitivity that gated dipole affective dynamics predict (Sikström and Söderlund, 2007). Moreover, pharmacological ‘uppers’ like Ritalin are often used to bring individuals with ADHD ‘down’ (e.g. Weyandt et al., 2014). In a gated dipole, this happens because such an upper will increase tonic arousal from underaroused hypersensitivity to a Golden Mean of more moderate sensitivity and threshold reactivity (Figure 6(c)).

These clinical symptoms will have predictable effects on reinforcement learning and motivated behaviour, notably on factors like desirability and reversal learning, that the reader can derive using the earlier explanations of these behavioural properties.

2.17. Some other recent models of orbitofrontal functioning

One way to appreciate the explanatory value of the cognitive-emotional processes that have been summarised in Section 2 is by contrast with other recent models of OFC. For example, Wilson et al. (2014) and Schuck et al. (2016) have proposed a model which proposes that OFC is a ‘cognitive map of task space’ during animal and human decision-making. They claim that ‘OFC is critical for representing task states in … partially observable scenarios’ and that ‘OFC is unique in its ability to disambiguate task states that are perceptually similar but conceptually different, for instance by using information from working memory’ (Wilson et al., 2014: 267). One of the types of data that they use the model to simulate is how OFC dysfunction ‘is impaired during reversal learning’ (Wilson et al., 2014: 268). However, as was earlier reviewed in Section 1.2, monkeys with selective excitotoxic lesions of the OFC, unlike monkeys who have received aspiration OFC lesions, are unimpaired in learning and reversing object choices based on reward feedback (Rudebeck et al., 2013). Neurotoxic lesions of the amygdala (Izquierdo and Murray, 2007) have also led to results that challenge earlier demonstrations using aspiration and radiofrequency lesions that the amygdala is needed for object reversal learning (Aggleton and Passingham, 1981; Jones and Mishkin, 1972; Spiegler and Mishkin, 1981).

How did the model lead to the wrong explanation? A basic weakness of the model is that is does not describe any neural mechanism. It does not model the functions of any brain region, including OFC. Instead, it describes abstract state space representations to fit any particular set of data, with different states hypothesised to explain different data sets, and ad hoc assumptions that are tailored to explain each data set about how these abstract states will change due to an OFC lesion. One can go so far to criticise the definition of model states as being circular, designed, and modified until they simulate a small piece of data, even if the data are wrong, as in the case of reversal learning. These states are typically connected by abstract edges in a feedforward network. There are none of the feedback loops of identified neural interactions with critical functional properties that are described herein, for example, Figures 1–8, 10, 12, 15, 17, and 18. A typical feedforward diagram in Wilson et al. (2014) is between one abstract state and two outcomes, or two successive states and an outcome.

Figure 10.

LIST PARSE model. (Left panel) The brain processes that are modelled by LIST PARSE are written in red and underlined. These processes are a Cognitive Working Memory, assumed to be in VLPFC; a Motor Working Memory, assumed to be in DLPFC; a VITE, or Vector Integration To Endpoint, Trajectory Generator, assumed to be in motor cortex; Motor Volition, assumed to be in the basal ganglia substantia nigra pars reticulata, or SNr; and Rehearsal Timing, assumed to be in the basal ganglia and cerebellum. (Right panel) The Cognitive Working Memory network is assumed to be within the deeper layers 4–6 of VLPFC, and the corresponding list chunking network is assumed to be within the superficial layers 2/3 of VLPFC. Green solid arrows are excitatory, red dashed arrows are inhibitory, and blue lines ending in hemidiscs are adaptive. Only 1-item chunks (C_i) and their feedback connections within a single Cognitive Working Memory channel are shown, whereas the model simulates chunks corresponding to words of variable lengths in layer 2/3. Learned positive feedback signals from layer 2/3 to layer 5/6 of the cognitive working memory are broadly distributed from each list chunk across the working memory so that an active chunk can reinstate the activity pattern that caused it into working memory. Also, only the excitatory projections from Cognitive Working Memory to the Motor Working Memory (Y_i→F_i) are shown. Several volitional gain control signals determine model dynamics. For example, the gain control signal F determines whether or not a sequence will be stored within the Cognitive Working Memory. The volitional signals V and G control variable-rate performance of the stored sequence. In particular, a rehearsal wave R is influenced by the volitional signal V when the network begins to recall items and by the deceleration signal (B – A) that allows another rehearsal burst to occur when a previously activated movement trajectory is almost completed. The text described more completely how the deceleration signal is computed.

Source: Adapted with permission from Grossberg and Pearson (2008).

Figure 12.

The lisTELOS model macrocircuit: each grey box represents a brain region within which fields of cells, represented by white inset boxes, share similar functional roles, which are summarised in the box. Arrowheads denote excitatory connections between cells, and filled circles represent inhibitory connections. Curved branches at the ends of connections represent one-to-many fan-out connections that impact all other cells in the field. Half-filled boxes at the ends of connections represent habituative gates that exhibit activity-dependent changes in synaptic efficacy. White circles containing a multiplication sign (×) represent multiplicative interaction between two signals. Boxes containing a sigma (Σ) represent the sum of outputs from all cells in the field that gave rise to the projection. Stacked field representations denote populations of rank-sensitive cells. SC: superior colliculus; PPC: posterior parietal cortex; PFC: prefrontal cortex; BG: basal ganglia; FEF: frontal eye fields; SEF: supplementary eye fields. Note the three BG loops gating the release of output signals from different brain regions. See the text for discussion of model function and dynamics.

Source: Reprinted with permission from Silver et al. (2011).

Figure 15.

Regulation of BG-gating of SC saccadic commands by frontal-parietal resonance. (a) When multiple stimuli exist as potential saccade goals, the corresponding PPC representations specifically excite striatal spiny projection neurons (SPNs; shown in the rectangle within the BG rectangle) and nonspecifically (convergently) excite feedforward inhibitory interneurons (labelled with a capital sigma) via corticostriatal projections. If more than one saccade plan is active, then striatal feedforward inhibition from all active plans prevents any one plan from activating its corresponding striatal SPNs to open the BG gate. This is because the pooled inhibitory input to each SPN can overwhelm the specific excitatory input. Therefore, the SC is not released from inhibition from the SNr, and movement is prevented while the conflicting cortical plan activity remains unresolved. (b) Targets compete in PPC via inhibitory interactions. When competition resolves so that the movement plan is unambiguous, the PPC’s excitatory input to striatal SPNs eventually exceeds striatal feedforward inhibition, which wanes as competing plans lose activation and stop convergent excitation of striatal inhibitory interneurons. The winning SPN’s discharge inhibits SNr (opens the normally closed BG gate), which disinhibits part of the SC map. (c) If the FEF plans a saccade goal whose position differs from that of a strong visual stimulus, the competing frontal and parietal activities collectively drive striatal feedforward inhibition to keep the BG gate shut until the conflict resolves. (d) As the frontal cortex imposes its saccade goal on the parietal cortex, the competition between saccade goals resolves, leading to a frontal-parietal resonance and BG gate opening to generate the unambiguous saccade.

Source: Reprinted from Brown et al. (2004).

Figure 17.

How ARTSCAN maintains spatial attention in the Where cortical stream upon an object surface via a surface-shroud resonance between V4 (Object Surface) and PPC (Spatial Attention), while the What cortical stream learns view-specific categories in ITp that get associated with view-invariant object categories in ITa. While the attentional shroud in PPC maintains spatial attention upon the object surface, it also inhibits reset cells in PPC (dashed red connection from Spatial Attention to Category Reset). When spatial attention shifts to another object, this inhibition shuts off, thereby disinhibiting the reset cells and enabling them to send a transient inhibitory burst to the invariant category representation (dashed red connection from Category Reset to Object Category), which is then also inhibited. A shift of spatial attention and learning of an invariant object category for the newly attended surface can then proceed. Feedback interactions between the object’s boundaries and surfaces include surface contour signals in V2 (red rectangular boundary) that assure perceptual consistency while also initiating figure-ground separation within cortical area V2. Because they are computed using a contrast-enhancing on-centre off-surround networks, surface contour signals are largest at high curvature boundary positions (see purple dots at rectangle corners). These high curvature positions mark locations of salient features, and are converted into eye movement target positions in V3A, which are relayed to brain regions like LIP and FEF (not shown) that enable the eyes to explore the object surface while it is attended.

Their model proposes no decision-making dynamics. Instead, it is a variant of the kind of formal, discrete time, statistical learning theories that were popular in the 1950s–1970s, such as stimulus sampling theory (Estes, 1950), the Rescorla–Wagner learning rule (Rescorla and Wagner, 1972), and the Luce (1977) choice rule. The Rescorla–Wagner (1972) model was, however, shown long ago to be inadequate even to offer a formal explanation of key classical conditioning data, such as why conditioned excitors extinguish, but conditioned inhibitors do not (Lysle and Fowler, 1985; Owren and Kaplan, 1981; Witcher, 1978; Zimmer-Hart and Rescorla, 1974), which the CogEM model does explain (Grossberg and Schmajuk, 1987). Thus, any model, including the Wilson et al. (2014) model, that attempts to explain data about extinction using this learning law is building on an incorrect foundation. Even if its states happen to be chosen to simulate a fixed piece of data, the learning law by which it does so cannot be trusted.

The second formal rule in the model, the Luce choice rule, is a simple ratio rule that, being a ratio, conveniently represents probabilistic outcomes, but includes no mechanistic explanatory power. This kind of formal rule has been obsolete for 40 years, ever since the self-normalising properties of recurrent shunting on-centre off-surround networks started to be used to explain STM and decision-making in the brain (Douglas et al., 1995; Grossberg, 1973, 1978a, 1978b, 1980; Heeger, 1992; see Section 3.15).

None of the explanations in Wilson et al. (2014) hold up to scrutiny. For example, it is assumed that ‘pressing the lever led to 1 U of reward during conditioning and to −0.2 U in extinction’. None of the real-time dynamics of how an unexpected non-reward counter-conditions a learned action is described. In contrast, the MOTIVATOR model explains in detail how recurrent opponent processes, modelled by gated dipoles, undergo antagonistic rebounds in response to unexpected events (Sections 2.13–2.15), whose matching process is modelled by ART (Figure 5). Each of these model circuits is supported by psychological, anatomical, and neurophysiological data and explains and predicts data that cannot be explained within a Rescorla–Wagner–Luce framework.

Wilson et al. (2014) claim to simulate how immediate versus delayed reward, including delayed alternation (p. 269), influence learned actions, but there is no representation of a delay in their model, unlike the adaptively timed learning in the pART model SNc, VTA, and hippocampus (Figures 2(b), 3(a), and 4(a)) that all share the same substrate of spectral timing which is realised by Ca⁺⁺-modulated metabotropic glutamate (mGluR) dynamics (Brown et al., 1999). Significantly, our SNc/VTA model provides a real-time neural explanation of dopamine release in response to unexpected rewards. Despite there being no model VTA in the Wilson et al. (2014), the authors nonetheless discuss dopamine firing by VTA neurons (pp. 271–272) in terms of a purely descriptive state diagram that is created to match the experiment.

Schunk et al. (2016) espouses the same model as Wilson et al. (2014), but it primarily reports interesting functional magnetic resonance imaging (fMRI) data from humans who are presented with a series of images, each of which contains a face superimposed on a house. The task is to perform an age judgment of faces or houses on separate blocks of trials:

The age (young or old) of the first trial defined the age of the current ‘mini-block’ and participants were instructed to continue judging the same category as long as the age in that category stayed the same. Upon encountering a trial in which the age in the judged category was different (e.g. a change from ‘young’ to ‘old’), the task rules required participants to switch to judging the age of the other category, starting a new mini-block on the next trial. (p. 1403)

All of their data can be explained by the following pART neural mechanisms, none of which is modelled in their own article. In particular, the first trial in a block can activate an invariant object category for faces or houses (Figures 2(a) and 3(a)). Choosing low vigilance enables the choice of a general category that can match against many faces, or many houses, but not both (Figure 5). This invariant object category can be maintained in STM during the block of trials by, among other circuits, the feedback loop between visual cortex, amygdala, and OFC (Figures 2 and 3). Sustained top-down attentive feedback from the invariant category to lower cortical and thalamic levels is matched against the bottom-up images via the ART Matching Rule (Figure 5(b)). These top-down signals are priming signals that can select face or house features from their overlay in each image, while suppressing mismatched features, thereby creating a match state. When there is a big enough mismatch with the category of the top-down prime, say due to a change in the age of the face in the current image, the mismatch can reset the invariant object category (Figure 5(c)) and the next block can begin.

These category matching events use the invariant object categories in anterior inferotemporal cortex (ITa in Figure 1), which is called sensory cortex in Figures 2 and 3. Each active invariant category reads out a top-down prime against which input exemplars are matched at lower cortical levels, for example, posterior inferotemporal cortex (ITp in Figure 1). In addition, it activates its specific connection from ITa to OFC (Figures 1–3). Because the ITa-to-OFC connections are specific, changes in matching of an active invariant category at ITa are also represented at OFC, where they could be decoded by the support vector machine classifier that the authors applied to their data.

Instead of providing such a mechanistic account, variants of which have explained scores of other experiments about visual search and top-down attentive matching (e.g. Grossberg, 2013a, 2017b), including how low vigilance categories are learned and matched against image morphs (Akrami et al., 2009; Grossberg et al., 2011), the authors show that different choices of abstract states can create fits to the data, thereby illustrating the arbitrary nature of state choices. In particular, the authors write (p. 1404) ‘It is interesting to note that we did not find any evidence that the observable component “current age” could be classified, while its unobservable counterpart “previous age” was decodable in OFC’. This finding can be explained by the fact that the task is to match a current image exemplar against a stored invariant prototype of a face or house category that is young or old. Once that category is stored, all matching throughout a block of trials is against the stored ‘previous age’ until a mismatch causes reset of the active category, leading to choice of a category with which to match exemplars in the next block of trials.

These OFC articles illustrate how all the design principles, mechanisms, circuits, and architectures that are summarised in Section 2 play useful roles in explaining data about OFC and its interactions with multiple other brain regions, as in Figures 1–3.

3. Working memory, chunking, and reinforcement in PFC and related areas

This section will explain how different parts of the PFC (Figure 1) interact to regulate the availability of outcomes (Rudeck et al., 2017), solve the credit assignment problem (Asaad et al., 2017), and control aspects of feature-based attention (Baldauf and Desimone, 2014; Bichot et al., 2015). All the explanations derive from properties of prefrontal working memory circuits, the cognitive plans, or list chunks, which are learned from them, and the regulatory cognitive and emotional machinery that enables prefrontal circuits to predict outcomes that can maximise reward based on previously experienced sequences of events.

3.1. VLPFC lesions cause a deficit in learning probabilistic stimulus-outcome associations

Rudeck et al. (2017) concluded that VLPFC encodes the availability, rather than the desirability, of outcomes. They based this conclusion on experiments in which excitotoxic lesions of the VLPFC led to a profound deficit in the ability of lesioned monkeys to learn probabilistic stimulus-outcome associations. Their Experiment 1 first tested the ability to update likelihood estimates for predicted outcomes by training a group of unoperated control monkeys and a group of monkeys with excitotoxic OFC neurons to perform a three-choice probabilistic learning task. Four of the unoperated control monkeys were tested before they also received excitotoxic VLPFC lesions. This procedure enabled monkeys with OFC lesions to be compared with controls, whereas monkeys with VLPFC lesions were compared with their own preoperative performances.

Each session consisted of 300 trials on which monkeys were presented with three novel stimuli on a touch screen monitor. Monkeys sampled different stimuli over trials to learn which stimulus was associated with the highest probability of reward. Reward delivery was based on one of four different reinforcement schedules, called stable, variable, forward, and backward, which use a predetermined series of reward/no-reward outcomes for each option on each trial over the 300 trial testing session. The likelihood of receiving a reward for choosing an option on each trial was calculated using a moving 20-trial window.

Unoperated controls and monkeys with OFC lesions quickly learned which image predicted the highest probability of reward and could track that image as it changed with the reward schedule on each session. In contrast, the VLPFC-lesioned monkeys were severely impaired on this task, except with the schedule wherein one option had a very high probability of reward compared to the others. Thus, the deficit is greatest when the probabilistic difference between the options is small. Unlike unoperated controls and OFC-lesioned animals, VLPFC-lesioned animals were more likely to change their choices between trials than they were before their lesions. After their lesions, they were more likely to switch choices after a rewarded choice than were controls or OFC-lesioned monkeys.

Rudeck et al. (2017) traced these effects to a reduced effect of the long-term effects of reward history, also called contingent learning. Choices on the five preceding trials were analysed. In monkeys with VLPFC lesions, associations between previous choices and the outcomes that contingently followed had essentially no influence on subsequent choices, except when one option consistently had a very high probability of reward. In this last condition, monitoring which previous sequences of stimuli predicted higher reward is not essential to doing the task.

3.2. Some classical data about sequential dependencies influencing future choices

The Rudeck et al. (2017) experiments contribute to a long history of experiments in psychology and psychobiology that study how probabilistic choices are determined by previous sequences of events. These kind of data are often attributed to the short-term storage of sequences of events and the influence of these stored sequences on current choices. The short-term storage of event sequences is accomplished by a working memory (e.g. Baddeley, 1986, 1996, 2012; Baddeley and Hitch, 1974; Cowan et al., 2005; Engle, 2002; Grossberg, 1978a, 1978b; Grossberg and Pearson, 2008; Oberauer, 2009; Silver et al., 2011). The term ‘working memory’ is used here to describe short-term storage of event sequences, not just the persistent storage of one event. Data and neural models will be summarised below that clarify how multiple items can be simultaneously stored in working memory and use these models to explain the Rudeck et al. (2017) data.

An early discovery was that pupil dilation increases with the number of items that are stored in working memory, for example, Kahneman and Beatty (1966) and Unsworth and Robison (2015). Oddball experiments also activate working memory. In an oddball paradigm (Banquet and Grossberg, 1987; Squires et al., 1975), a subject receives a Bernoulli series of two types of stimulus with complementary probabilities: a frequent distractor stimulus, and a rare target stimulus, with unsignalled switches occurring in the probabilities of these stimuli. The subject has to perform a task such as releasing a motor response to each target stimulus, or counting target stimuli. Various measures indicate that a subject tracks the probabilistic sequential dependencies of distractor and target stimuli, for example, distractors 80% of time and targets 20% of time. These include the P300 event-related potential (ERP; Picton, 1992; Sutton et al., 1965), also called the P3b, whose amplitude tends to vary inversely with target stimulus probability (Duncan-Johnson and Donchin, 1977; Tueting et al., 1971). Longer sequences of distractors elicit larger P300s (Remington, 1969; Squires et al., 1976), and the P300 is amplified by practice as an expectation of sequence structure is learned (Banquet and Grossberg, 1987). Adding an ERP manipulation to neurophysiological recordings can be a useful way to coordinate classical psychophysiological explanations with modern neurophysiological methods.

After an object or event sequence is stored in working memory, it can activate the learning of a cognitive plan, or list chunk, by sending output signals through a bottom-up adaptive filter to a category learning level (Figure 7). The adaptive filter obeys the same laws as the one from the feature level F₁ to the category level F₂ in the ART circuit of Figure 5(a) and (d). Such an adaptive filter can learn to categorise any spatial pattern of activity across a network of feature detectors. In the case where the activity pattern represents a sequence, or list, of items stored in working memory, the category uses the adaptive filter to learn list chunks that selectively respond to subsequences of these stored items. The dynamics depicted in Figure 7 of these two levels and their interactions will be explained below, including how Masking Fields learn to choose the list chunks that predict the most likely outcomes. For now, it suffices to note that learning by the adaptive weights, or LTM traces (hemidiscs in Figure 7), can learn a list chunk using the same kind of ART dynamics that can learn an object category or a value category, but a list chunk codes sequences, or lists, or events rather than objects or affective values.

Figure 7.

An Item-and-Order working memory (lower level) for the short-term sequential storage of item chunks (e.g. M, Y, S, E, L, F) can activate a multiple-scale Masking Field list chunking network (upper level) through a bottom-up adaptive filter. The larger cell sizes and interaction strengths of the list chunks that categorise longer lists (e.g. MYSELF vs MY) enable the Masking Field to choose the list chunk that currently receives the largest total input and thus best predicts the sequence that is currently stored in the Item-and-Order working memory. The chosen list chunk can then read out the most likely prediction of what will happen next in that temporal context. Green connections are excitatory. Red connections are inhibitory. Arrowheads at the ends of Masking Field inhibitory recurrent pathways denote connections that undergo no learning. Hemidiscs denote connections that can undergo learning, both in the bottom-up filter connections and the top-down expectation connections. Recurrent on-centre off-surround connections in the Item-and-Order working memory are not shown, for simplicity. Recurrent self-excitatory connections in the Masking Field are also not shown, again for simplicity.

3.3. Cognitive working memory and list chunks in VLPFC

This and the next few sections will present experimental and modelling facts that lead to an explanation of how VLPFC working memory and chunking dynamics compute the availability of outcomes. A large cognitive neuroscience experimental literature has implicated the VLPFC in working memory tasks, notably tasks that engage verbal and language working memory properties (e.g. Awh et al., 1996; Schumacher et al., 1996), as well as in temporal sequencing properties that are dissociated from the specific stimulus type (Gelfand and Bookheimer, 2003). Nozari, Mirman, and Thompson-Schill (2016) review a large number of competences that all require such a temporal sequencing property at their core, whatever other control structures also exist in order to convert the stored sequences into observable behaviours.

The theoretical discussion below will propose why so many cognitive competences may exploit a similar temporal sequencing property. It will explain how and why all linguistic, spatial, and motor working memories are predicted to exploit variations and specialisations of a similar circuit design (Section 3.15; Grossberg, 1978a, 1978b; Grossberg and Pearson, 2008; Silver et al., 2011), along with psychological and neurophysiological data that support this hypothesis. Henceforth, the kind of working memory that is found in VLPFC will be called a cognitive working memory to distinguish it from the kind of motor working memory in DLPFC that converts the VLPFC stored sequences into sequences that are monitored to choose and perform properly ordered and timed sequential behaviours (Petrides et al., 2002).

3.4. List chunks and reinforcement interact in a probabilistic choice environment

A list chunk that is active when reinforcement occurs can later be amplified by learned feedback interactions from reinforcement-sensitive midbrain structures. The MOTIVATOR model (Figure 3(a)) has already noted how this can happen due to interactions between OFC (or ORB) and the amygdala and BG. Amplification helps this list chunk to win the competition with other possible list chunks.

In particular, area 12o of macaque monkeys interacts with the LH (Öngür et al., 1998), and this area was spared in the excitotoxic OFC lesions of Rudeck et al. (2017), but not the VLPFC lesion. Interactions with PRC may also play a role (Section 3.26; Petrides and Pandya, 2002). The winning list chunk can then be preferentially associated with a rewarded stimulus by the Now Print signals that are emitted by SNc when this stimulus unexpectedly occurs (Figures 3 and 4). On later trials, output signals from this list chunk can be used to prime, and thus help to choose, the next stimulus to be attended and chosen. List chunks that are active when the most rewards occur will thus be favoured to win the competition and to determine future choices.

On the other hand, when a winning list chunk no longer best represents the correct choice, then its choice of an unexpectedly unrewarded stimulus can cause two parallel processes to occur that reduce the probability of choosing that list chunk in the future. First, the unexpected outcome can cause hypothalamic rebounds that can begin to extinguish the motivational support for the list chunk (Sections 2.14 and 2.15; Figure 6(a)). Second, SNc-mediated dopaminergic dips when a chosen stimulus is unexpectedly unrewarded (Figure 4) can begin to weaken the association between the list chunk and its learned prime. These reinforcing events in response to predictive successes and failures are cumulative across learning and performance trials, thereby flexibly shifting the list chunk that has the most support, and thus the choice that the newly chosen list chunk reads out, across trials. When a list chunk influences future choices based on past event sequences and their reinforcement history, it may be said to function as a cognitive plan.

This article argues that such working memory representations and their list chunks exist in the VLPFC, albeit in different cortical layers (see Figure 10 below; Grossberg and Pearson, 2008), and can be amplified by reinforcing events even if the AMYG and OFC are excitotoxically lesioned. In this way, animals with AMYG or OFC lesions can compute the availability of a valued outcome, even if they can no longer estimate the desirability of such an outcome.

Different probabilistic reinforcement schedules require that an animal be sensitive to, and thus store in working memory, different numbers of preceding rewarded and unrewarded choices. Choosing among lower reinforcement probabilities requires sensitivity to longer preceding sequences, other things being equal. When reinforcement is increased, and in the limit rewards every choice of a particular image, then tracking previous sequences of choices is no longer needed to make a correct choice on the next trial. This sort of consideration clarifies why a VLPFC working memory is not needed to adapt to such a (nearly) deterministic reinforcement schedule.

This consideration also calls attention to a basic problem that the brain needs to solve: how does the brain know how to choose the length of preceding event sequences that a chosen list chunk should encode in order to optimise choices that maximise reward? In particular, how does the brain learn list chunks of stored working memory sequences of variable length? And how do these list chunks compete to choose the most predictive list chunk? A brain design that accomplishes this is called a Masking Field (Cohen and Grossberg, 1987; Grossberg, 1978a; Grossberg and Kazerounian, 2011; Grossberg and Myers, 2000). Masking Fields will be discussed in Sections 3.21–3.23 after the predicted link between working memory and list chunking is explained.

3.5. The predicted link between Item-and-Order working memory and list chunking

In order to explain how Masking Fields work, it is first necessary to summarise how the working memories that input to them are themselves designed. Grossberg (1978a, 1978b) introduced a neural model of working memory which posits that a temporal stream of inputs is stored through time as an evolving spatial pattern of content-addressable item representations (Figure 8). These working memory patterns are, in turn, unitised through learning into list chunk representations that can control context-sensitive behaviours. This working memory model is called an Item-and-Order model because, in it, individual nodes, or cell populations, represent list items, and the temporal order in which the items are presented is stored by an activity gradient across the nodes (Figure 8).

The classical work of Miller (1956) on working memory showed that a fundamental functional unit in speech and language is abstract, namely the chunk, that ‘the memory span is a fixed number of chunks [and] we can increase the number of bits of information that it contains simply by building larger and larger chunks, each chunk containing more information than before’. Chunks can thus be learned from multiple types of acoustic inputs that vary in size. Miller (1956) proposed that approximately 7 ± 2 chunks could be simultaneously stored in working memory. He called this immediate memory span the Magical Number Seven, Plus or Minus Two.

In order to work properly, the networks that use Item-and-Order working memories need to be composed of the correct functional units at successive levels of the brain’s speech and language hierarchy. Some early models proposed levels that process phonemes, letters, and words (e.g. McClelland and Rumelhart, 1981), but these levels do not fit basic psychophysical data and cannot self-organise using unsupervised learning. These problems were solved by the Item-and-Order model levels which represent distributed features, item chunks, and list chunks (Grossberg, 1978a, 1978b, 1984b, 1986). An item chunk selectively responds to one pattern of activity, or a small set of similar patterns of activity, across the distributed feature detectors within a brief time interval (e.g. a phoneme). A list chunk selectively responds to the temporal order of a single sequence of item chunks that are stored in working memory. The properties of these functional units have been used to explain data about word superiority effect, list length effect, and related speech phenomena that are incompatible with alternative processing levels (see Section 3.23).

3.6. Correct temporal order is stored in working memory by a primacy gradient

A primacy gradient stores items in working memory that will be recalled in the correct temporal order. In a primacy gradient, the first item in the sequence activates the corresponding item chunk with the highest activity, the item chunk representing the second item has the second highest activity, and so on, until all items in the sequence are represented (Figure 8). For example, a sequence ‘A-B-C’ of items is transformed into a primacy gradient of activity with cells encoding ‘A’ having the highest activity, cells encoding ‘B’ having the second highest activity, and cells encoding ‘C’ having the least activity.

3.7. Rehearsal and inhibition-of-return

How is a stored spatial pattern in working memory converted to a temporally performed sequence of items during recall? A rehearsal wave that is delivered uniformly, or nonspecifically, from the BG to the entire working memory enables read-out of stored activities (Figure 8). The node with the highest activity is read out fastest because it exceeds its output threshold fastest. As it is read out, it also self-inhibits its working memory representation via a recurrent inhibitory interneuron. This self-inhibitory process mechanises the cognitive concept of inhibition-of-return, which prevents perseverative performance of the most recent item (Posner et al., 1985). Self-inhibition of the last item to be performed is repeated until the entire sequence is performed. These operations may also be influenced by different rehearsal strategies, as illustrated by performance differences during immediate free recall experiments, during which subjects attempt to recall items in any order after a single hearing, versus immediate serial recall (ISR) experiments, during which subjects attempt to recall items in the order that they were heard. These different kinds of data are explained and quantitatively simulated using an Item-and-Order model in Grossberg and Pearson (2008).

3.8. Competitive queuing and primacy models

After Grossberg (1978a, 1978b) introduced the Item-and-Order model, it was used in various forms in a number of other studies (e.g. Boardman and Bullock, 1991; Bohland et al., 2010; Bradski et al., 1994; Bullock and Rhodes, 2003; Grossberg and Pearson, 2008; Houghton, 1990; Page and Norris, 1998). For example, Page and Norris (1998) used a Primacy Model to explain and simulate cognitive data about immediate serial order working memory, including data about word and list length, phonological similarity, and forward and backward recall effects. Properties of the Item-and-Order model were also used in the Competitive Queuing model (Houghton, 1990) which describes how the most active item that is stored in working memory self-inhibits its stored activity when it is performed.

3.9. Supportive psychological and neurophysiological data for Item-and-Order networks

Both psychophysical and neurophysiological data have supported the Item-and-Order predictions that working memories encode item order with relative activity levels and are reset by self-inhibition. For example, Farrell and Lewandowsky (2004) did psychophysical experiments in humans that studied the latency of responses following serial performance errors. They concluded that

Several competing theories of short-term memory can explain serial recall performance at a quantitative level. However, most theories to date have not been applied to the accompanying pattern of response latencies … Data from three experiments … rule out three of the four representational mechanisms. The data support the notion that serial order is represented by a primacy gradient that is accompanied by suppression of recalled items. [italics mine] (p. 115)

Electrophysiological experiments have also directly recorded Item-and-Order working memory properties. For example, recordings in the posterior principal sulcus of the DLPFC of macaque monkeys were done by Averbeck et al. (2002) while monkeys performed learned arm movement sequences that copy geometrical shapes. These authors did extracellular recording from the areas near, but outside the depths of, the caudal portions of the principal sulcus in DLPFC of macaque monkeys during the performance of a sequential motor task. In this task, monkeys used an X-Y joystick to copy concurrently presented geometrical shapes (triangle, square, inverted triangle, trapezoid) on an LCD screen in a prescribed order. Copying proceeded counter-clockwise starting at the top middle of each shape. The recorded cell responses were pooled on the basis of the movement in the sequence with which their firing pattern most correlated. The population response for each movement in the series is shown in Figure 9(a).

Figure 9.

Neurophysiological data and simulations of monkey sequential copying data. (a) Each plot shows the recorded cell activity profiles that control drawing of each segment for each time bin (at 25 ms) of the task. The number of movement segments is due to the starting positions of each movement sequence on the corresponding geometrical figure. Time 0 indicates the onset of the template. Lengths of segments were normalised to permit averaging across trials. Plots show parallel representation of segments before initiation of copying. Furthermore, rank order of strength of representation before copying corresponds to the serial position of the segment in the series. The rank order evolves during the drawing to maintain the serial position code. At least four phases of the Averbeck et al. (2002; Figure 9(a)) curves should be noted: (1) presence of a primacy gradient, that is, greater relative activation corresponds to earlier eventual execution in the sequence during the period prior to the initiation of the movement sequence (period −500 to 400 ms); (2) contrast enhancement of the primacy gradient to favour the item to be performed (greater proportional representation of the first item) prior to first item performance (period ~100–400 ms); (3) inhibition of the chosen item’s activity just prior to its performance and preferential relative enhancement of the representation of the next item to be preformed such that it becomes the most active item prior to its execution (period ~400 ms to near sequence completion); and (4) possible re-establishment of the gradient just prior to task completion. Reprinted with permission from Averbeck et al. (2002). (b) Simulations of item activity across the motor plan field of the LIST PARSE model for 3, 4, and 5 item sequences versus simulation time. In both (a) and (b), line colours correspond to representations of segments as follows: yellow, segment 1; green, segment 2; red, segment 3; cyan, segment 4; magenta, segment 5. Reprinted with permission from Grossberg and Pearson (2008).

The predicted properties of a primacy gradient and a self-inhibitory form of inhibition-of-return are evident in these data. Figure 9(a) depicts stored working memory activities for drawing several different geometrical shapes. In each example, the movement with the largest activity is performed first, before it self-inhibits its activity. Then, the next largest activity reads out its movement while self-inhibiting its activity. After the next-to-last movement is performed, the activity of the final movement grows greater than that of any prior movement command because it has no competitors left that are stored in working memory. How this competition works, and why it exists, is explained in Section 3.15.

These properties were simulated (Figure 9(b)) using an Item-and-Order working memory by the laminar cortical LIST PARSE model of Grossberg and Pearson (2008). This model proposes how a cognitive working memory in VLPFC is converted into a motor working memory in DLPFC (cf. Figure 1) which, when properly monitored, can perform the stored sequence at variable rates that are under volitional control from the BG (Figure 10). In all, the LIST PARSE model shows how a prefrontal linguistic working memory in VLPFC can quantitatively simulate human psychophysical data about ISR, and immediate, delayed, and continuous distractor free recall; whereas a similarly designed prefrontal motor working memory in DLPFC can quantitatively simulate the Averbeck et al. (2002) neurophysiological data about sequential recall of stored motor sequences (Figure 9). This property illustrates the fact that all linguistic, spatial, and motor working memories use variations of the same network design in VLPFC and other prefrontal areas, for reasons that are explained in Section 3.14.

3.10. Bowed and recency gradients during free recall and probabilistic choice behaviours

Before explaining why all working memories are embodied by a similar circuit design, it is important to realise when and why such working memories do not accurately encode or perform temporal order. Some of these data are summarised in this section and the next. Why this happens is explained in Section 3.12, which links this property to the ability to learn and stably remember list chunks. These explanations and predictions would benefit from further neurophysiological studies using longer lists.

Item-and-Order working memories provide principled answers to the following basic questions about working memory: What is the longest list that the brain can store in working memory in the correct temporal order? Why can only relatively short lists be stored with the correct temporal order in vivo? In an Item-and-Order working memory, this question translates into: What is the longest primacy gradient that the working memory can store? How is a primacy gradient altered when longer lists are stored? Free recall data illustrate how primacy gradients change if a longer list is stored. Then, a bowed serial position curve is often observed (Figure 11; for example, Murdock, 1962) such that items at the beginning and the end of the list are performed earliest, and with the highest probability of recall.

Figure 11.

Simulation of cognitive working memory activity gradients by the LIST PARSE model. (Left panel) A short list of six items generates a primacy gradient. (Right panel) A longer list of 20 items, which exceeds the transient memory span of this network, generates a bowed gradient with extended recency. Note the smaller activities that are stored in response to 20 items than six items due to the normalising effect on total activity of the recurrent shunting on-centre off-surround dynamics that store these lists in working memory.

Source: Adapted with permission from Grossberg and Pearson (2008).

Grossberg (1978a, 1978b) noted that these free recall properties can readily be explained if the working memory gradient that stores the list items is also bowed. Then, the first and last items have the largest activities, and items in the middle have less activity. The temporal order of recall is explained as follows: the item with the largest activity is read out first, whether at the list beginning or end, because it exceeds the output threshold first of its output pathway when the rehearsal gate opens (Figure 8). As its output is read out, it then self-inhibits its working memory activity via a recurrent inhibitory interneuron (Figure 8) to prevent preservation. Then, the next largest item will be read out and so on in the order of item relative activity. The probability of recall has the following explanation: items that are stored with larger activities have greater resilience against perturbation by cellular noise. Transpositions of order during recall are explained in the same way because transposed items have similar stored activities.

If even longer lists get stored, then the bow increasingly resembles a recency gradient, such that items at the end of the list are performed earliest, and progressively earlier items are stored with less and less activity (Figure 11). In a probabilistic choice experiment with low probabilities of reward, such a recency gradient may be expected to develop. This working memory property helps to explain the Rudebeck et al. (2017) data that unoperated and OFC-lesioned ‘monkeys were making contingent associations between their specific choices and subsequent outcomes. This effect diminished with increasing distance from the current trial, suggesting that monkeys preferentially used the most recent feedback to guide future choices’ (p. 1211). The way that primacy gradients become bows and then recency gradients as stored list length increases hereby reconciles the different working memory properties that have been reported in experiments like Averbeck et al. (2002) and Rudebeck et al. (2017). It would be interesting to record the transformation from primacy gradient, to bow, to extended recency gradient by systematically changing the number of choices and choice probabilities in this kind of experiment. Another informative kind of experiment could alter the strength of recurrent inhibition across the network, say with a GABA antagonist, and test how it alters the transient and immediate memory spans that will now be explained.

3.11. Magical numbers four and seven: immediate and transient memory spans

What is the longest primacy gradient that can be stored? The classical Magical Number Seven, or immediate memory span, of 7 ± 2 items that is found during free recall (Miller, 1956) estimates the upper bound. Section 3.23 offers an explanation of the Magical Number Seven using an Item-and-Order network. Grossberg (1978a) distinguished between the immediate memory span and the transient memory span. The transient memory span was predicted to be the longest list for which a primacy gradient may be stored in working memory solely as the result of bottom-up inputs (see Figure 11). In contrast, the immediate memory span was predicted to arise from the combined effect of bottom-up inputs and top-down LTM read out from list chunks that could be activated by the working memory (Figure 7). Grossberg (1978a) mathematically proved that the read-out of top-down long-term memories can only increase the maximal primacy gradient that can be stored, and thus that the immediate memory span is longer than the transient memory span. Given an estimated immediate memory span of approximately seven items, it was estimated that the transient memory span should be approximately four items. Cowan (2001) has summarised data that support this prediction by showing that, when the influences of LTM and grouping effects are minimised, there is a working memory capacity limit of 4 ± 1 items. There is thus also a Magical Number Four.

If technically possible, reversible cooling or other method for silencing the list chunk level should therefore lead to a shortening of the longest primacy gradient that the working memory can store.

3.12. LTM Invariance Principle: learning stable list chunks

Why is the transient memory span so short? The proposed answer to this question suggests that neurophysiological experiments be done that combine recordings of working memory storage and list chunk learning. In particular, a network for STM storage of sequences in working memory can only realise its full potential if it can also support the learning and LTM of list chunks. Indeed, without stable list chunk learning and memory, it would be impossible to learn and perform language, motor skills, or navigational routes. Grossberg (1978a, 1978b) predicted that all working memories are designed to enable learning and stable memory of list chunks and showed that two simple postulates imply these properties: the LTM Invariance Principle and the Normalisation rule. Grossberg (1978a, 1978b) also explained how these postulates can be mathematically realised by Item-and-Order working memories and how they generate primacy and bowed gradients to explain free recall data. Since this early derivation, the understanding of how these working memories are realised in vivo has been incrementally refined (e.g. Bradski et al., 1992, 1994), leading to laminar cortical models of how prefrontal circuits realise Item-Order-Rank working memories (Grossberg and Pearson, 2008; Silver et al., 2011) that can store sequences with repeated items in working memory, for example, ABACAD (see Sections 3.16–3.18).

The LTM Invariance Principle implies that novel sequences of items may be stored in working memory and chunked through learning in a way that does not destabilise memories of previously learned chunks. It explains, for example, how a sequence of the item chunks of the longer word MYSELF can be stored in working memory without forcing catastrophic forgetting of previously learned, shorter list chunks for the words MY, SELF, and ELF. Said in another way, the LTM Invariance Principle shows how, if bottom-up inputs activate a previously learned chunk of the word MY, then storage in working memory of the remaining portion SELF of the novel word MYSELF will not erode the previously learned adaptive weights that support activation of the list chunk of MY. Thus, the LTM Invariance Principle begins to explain how sequences of variable length may be stored in working memory and induce learning of list chunks that selectively respond to them. As noted in Section 3.10, longer sequences may need to be stored and selectively chunked when reinforcement probabilities are lower in order to predict the most likely stimulus that will next be rewarded. As Figure 7 illustrates, variable-length list chunks can be learned by a network that is called a Masking Field.

The LTM Invariance Principle is achieved mathematically by preserving the relative activities, or ratios, between previously stored working memory activities as new items are presented and stored in the working memory through time. Newly arriving inputs may, however, alter the total activity of each active cell across the working memory. How does preserving activity ratios help to stabilise the adaptive weights of previously learned categories? These activities send signals to the next processing stage, where the category cells are activated (Figure 7). The signals are multiplied, or gated, by adaptive weights, or LTM traces (in the synaptic knobs with hemidiscs in Figure 7), before the net adaptively gated signals activate their target categories. Multiplicative gating of the bottom-up signals by LTM traces converts the bottom-up pathways into an adaptive filter. The total input to a category thus multiplies a pattern, or vector, of activities times a pattern, or vector, of LTM traces. This kind of multiplication of two vectors is said to carry out an inner product, or dot product, operation. By preserving relative activities of the stored working memory activities, the relative sizes of these total inputs to the category cells do not change through time, and thus nor do the corresponding LTM patterns that track these activities when learning occurs at their category cells.

For example, suppose that bottom-up acoustic inputs are stored in working memory and activate their corresponding list chunks. As these inputs arrive, a chunk such as ‘MY’ may become active once it receives all or most of its expected bottom-up input. If the acoustic inputs are then followed immediately by silence, the chunked representation of ‘MY’ could stably learn from the stored STM pattern of activity that first supported it. On the other hand, if the acoustic inputs are rapidly followed by further acoustic signals (e.g. the inputs corresponding to ‘MYSELF’), then these newly arriving inputs could drastically alter the pattern of activation that represents MY in STM if the LTM Invariance Principle did not hold. If this could happen, then the LTM traces that activate chunk ‘MY’ could change in response to the now altered STM pattern in working memory. The LTM Invariance Principle prevents this from happening, since the newly arriving inputs (corresponding to ‘SELF’) then leave intact the relative pattern of activity in working memory of the already occurring acoustic inputs (corresponding to ‘MY’). A new list chunk for the full word (corresponding to ‘MYSELF’) could then be learned without destabilising the already learned LTM pattern for its subset components (e.g. ‘MY’).

The Normalisation Rule ensures that the total activity that is stored in working memory has an upper bound that tends to be independent of the number of items that are stored. Thus, if more items are stored, then each item tends to be stored with less activity (see Figures 9 and 11). This normalisation property implies the limited capacity of working memory (Baddeley and Hitch, 1974; Grossberg, 1978a, 1978b) by redistributing, rather than adding, activity when new items are stored. Storing more items in working memory thus causes each of them to have less activity. When a sufficiently long list length is reached, trying to store more items will prevent some of them from exceeding the storage threshold.

Consistent with this conclusion, application of a GABA antagonist to DLPFC, and thus a weakening of recurrent inhibition, causes an overall increase in cell activity (Rao et al., 2000). In the opposite direction, excess GABA-mediated inhibition of prefrontal neuronal activity has been identified as a contributor to working memory dysfunction during the first 4 months following a cortical impact injury in rats (Hoskison et al., 2009).

3.13. Bowed gradients for long lists follow from self-stabilising memory

Properties such as the transient memory span were mathematically proved in Grossberg (1978a, 1978b) to follow if both the LTM Invariance Principle and the Normalisation Rule hold. If a list is longer than the transient memory span, then the primacy gradient that is initially stored will evolve into a bowed gradient as more items are stored, as illustrated in Figure 11.

In other words, the ability of a working memory to enable learning and stable memory of stored sequences implies an upper bound on the length of lists that can be temporarily stored in the correct temporal order. The bowed serial position curves of free recall and ISR data can thus be understood as the price paid for being able to rapidly learn, and stably remember, language and sequential spatial and motor skills.

These results about primacy gradients hold when the same amount of attention is paid to each item as it is stored in working memory. If attention is not uniform across items, then multi-modal bows can occur. These Von Restorff (1933) or isolation effects (Hunt and Lamb, 2001) enable items in the middle of a list to be remembered and recalled before other list items.

3.14. Similar circuits for linguistic, spatial, and motor working memories

It can now be seen from the deeper perspective of chunk learning why VLPFC and DLPFC are involved in storing so many types of event sequences, including the probabilistic choice sequences that are activated during the Rudebeck et al. (2017) experiments. Namely, if all working memories obey the LTM Invariance Principle and the Normalisation Rule, then all linguistic, motor, and spatial working memories should have a similar design, and thus should therefore exhibit similar data properties, such as error distributions. Data that support this prediction include the following: Jones et al. (1995) reported similar performance characteristics to those of verbal working memory for a spatial serial recall task in which visual locations were remembered. Agam et al. (2005) reported psychophysical evidence of Item-and-Order working memory properties in humans as they performed sequential copying movements, and Averbeck et al. (2002, 2003a, 2003b) reported neurophysiological evidence for such a working memory in monkeys during performance of sequential copying movements (Figure 9(a)).

Model explanations of working memory data also support the prediction of a universal working memory design for all kinds of input sequences. It has already been noted that the LIST PARSE model of Grossberg and Pearson (2008) has simulated the Averbeck et al. data using a prefrontal motor Item-and-Order working memory (Figure 9(b)) and has used a prefrontal linguistic working memory to quantitatively simulate human psychophysical data about ISR, and immediate, delayed, and continuous distractor free recall. The lisTELOS model of Silver et al. (2011) has, in addition, used a prefrontal Item-Order-Rank spatial working memory to quantitatively simulate neurophysiological data about the learning and planned performance of saccadic eye movement sequences.

The LTM Invariance Principle and Normalisation Rule also imply that there is an intimate connection between the process of storing sequences temporarily in working memory and the learning of list chunks by the next processing stage. Data that support this prediction include the following: Agam et al. (2007) reported data about the formation of list chunks as movement sequences are practiced. Psychophysical experiments on speech perception have also successfully tested this prediction (e.g. Auer and Luce, 2008; Goldinger and Azuma, 2003; Luce and McLennan, 2008; McLennan et al., 2003, 2005; Vitevitch and Luce, 1999).

3.15. Recurrent shunting on-centre off-surround networks embody working memories

What is this shared working memory design? In particular, are postulates such as the LTM Invariance Principle and the Normalisation Rule too sophisticated for evolution to discover? In fact, both the LTM Invariance Principle and the Normalisation Rule are embodied within a ubiquitous neural design, thereby clarifying how such a working memory could arise through evolution: a recurrent on-centre off-surround network (Figure 8) whose cells obey the membrane equations of neurophysiology, also called shunting dynamics. Such networks occur ubiquitously because they enable the brain to process and store distributed patterns of inputs without being degraded by noise – when their inputs are small – or saturation – when their inputs are large, thereby solving the noise-saturation dilemma that is faced by every brain network (Grossberg, 1973, 1980).

How such recurrent shunting networks process ratios (LTM Invariance Principle) and conserve total activity (Normalisation Rule) was mathematically proved in Grossberg (1973). Also see Grossberg (1978a, 1980) for reviews. Bradski et al. (1994) went on to prove theorems about how Item-and-Order recurrent shunting on-centre off-surround networks generate primacy and bowed gradients, among other properties, as a function of network parameters.

The excitatory feedback due to the recurrent on-centre interactions in such a network helps to store an evolving spatial pattern of activities in response to a sequence of inputs. The recurrent shunting off-surround, in concert with the on-centre, helps to preserve the relative activities that are stored. A volitional rehearsal signal from the BG enables the highest stored activity to be read out first, and self-inhibitory feedback prevents perseverative performance of this most highly activated cell population, thereby enabling less active populations to be performed (Figure 8), while the network as a whole gradually renormalises its activity through time. For example, the Normalisation rule clarifies why, in Figure 9, after the next-to-last item that is stored in working memory has been performed, the population that stores the last item is disinhibited. By being freed from inhibitory normalisation, it can reach the highest activity through time of any population that stored the list.

3.16. Storing lists with repeated items: Item-Order-Rank working memory

In a probabilistic learning situation such as the one used by Rudebeck et al. (2017), where a subject chooses one of three images to receive reward, each image may repeat itself over trials. In order to estimate the choice that will most probably lead to reward, an animal needs to keep track of such a contingency. This state of affairs raises the question: How are sequences with repeated items stored in working memory? Can recordings be made during experiments on availability that more completely characterise the predicted circuits whereby repeated items are stored in a PFC working memory? The need for combined prefrontal-parietal recordings of availability in the broadest sense is also suggested by the prediction in Section 3.17 of how PFC working memories that can store repeated occurrences of each experienced option receive inputs from parietal mechanisms for numerically estimating how much of a valued goal object is available during foraging in naturalistic environments.

In its simplest form, an Item-and-Order working memory cannot represent the same item in multiple positions, or ranks, of a list. However, there are many examples in human cognitive data of sensitivity to list position (e.g. Henson, 1998). For example, when presented with the sequence ABC (pause) DEF, exchanges between items at B and E are more common than exchanges between items at B and F. In addition, phonemes or syllables in similar positions in different words may be selectively interchanged. These examples include spoonerisms, for example, ‘hissed my mystery lesson’. Error data in human serial recall experiments also indicate that rank information is available, which some models of serial recall have incorporated (see Grossberg and Pearson, 2008 for a review). In monkeys, some PFC neurons respond, not only to a given item but also to the rank of that item within a sequence of items. Such a cell may respond to a specific target that is presented in a specific list position (e.g. Averbeck et al., 2003a; Barone and Joseph, 1989; Funahashi et al., 1997; Inoue and Mikami, 2006; Kermadi and Joseph, 1995; Ninokura et al., 2004).

3.17. From parietal numerical map to prefrontal Item-Order-Rank working memory

Given that both psychophysical and neurophysiological data also support Item-and-Order models, it remains to explain how rank information may be integrated into such a working memory. Bradski et al. (1994) proposed the first Item-Order-Rank working memory model that incorporated rank-order coding into an Item-and-Order working memory that is capable of storing item repeats at arbitrary list positions, for example, ABACBD. The LIST PARSE model (Grossberg and Pearson, 2008) predicted where in the brain this rank-order coding arises, namely, the analogue spatial representations of numbers that exist in the parietal cortex. The model specifies how an Item-Order-Rank working memory can be created in PFC using parietal-prefrontal projections from this parietal numerical representation. This prediction built upon the Spatial Number Network, or SpaN, model of Grossberg and Repin (2003) which simulated how the analogue map of ordered numerical representations in inferior parietal cortex enables animals and humans to estimate and compare sufficiently small numerical quantities. Such a capability can have life-saving consequences in terrestrial animals who forage for food. For example, choosing a tree with more fruit, or a flower with more honey, illustrates how survival may be enhanced by being able to estimate and compare numerical quantities. The predicted properties of SpaN model parietal neurons were supported by neurophysiological data of Nieder and Miller (2004a), who also studied the prefrontal projections of these parietal numerical representations (Nieder and Miller, 2004b). It would be very interesting to combine the Nieder and Miller experimental manipulations with manipulations of option availability.

In an Item-Order-Rank working memory, a spatial gradient of activity still represents temporal order, with the most active cell population being performed first. The main new idea is that each item representation in an Item-Order-Rank working memory has multiple map positions, or slots, that can store occurrences of the item at multiple list positions. For each item, the cell population that codes all the possible list positions, or ranks, forms a numerical hypercolumn. The parietal number map hereby broadcasts a numerical hypercolumn that is incorporated into multiple PFC item representations using a conjunction of item and numerical information.

This parietal-prefrontal projection enables the correct position in the hypercolumn to be activated when the item occurs with a given rank in the list. The numerical hypercolumn that represents a specific list item can hereby store that item in multiple list positions, just as a positional hypercolumn in the primary visual cortex can selectively respond to multiple orientations at a given position in space (Hubel and Wiesel, 1962, 1963). Thus, to store the list ABAC, item A would be stored in the first and third slots within its hypercolumn, item B would be stored in the second slot within its hypercolumn, and item C would be stored in the fourth slot within its hypercolumn.

A primacy gradient of activity (Figure 11) would still represent the temporal order of a short stored list, whether or not it had repeated items. Working memory gradients would be created in the same way as in an Item-and-Order working memory. The recurrent on-centre off-surround network that stores items in an Item-Order-Rank working memory can still use self-excitatory feedback from each cell population to itself, and a broad off-surround can still equally inhibit all other populations in the working memory, including all the slots of all the items. A similar concept has been used by Davis (2010) to model letter repetitions during visual word identification.

3.18. lisTELOS: an Item-Order-Rank spatial working memory in PFC

How to combine parietal and prefrontal manipulations to get more insight into how option availability is computed may be guided by available models of neural architectures in which Item-Order-Rank working memories play a key role. For example, Silver et al. (2011) implemented an Item-Order-Rank model of spatial working memory in DLPFC that is part of the larger lisTELOS neural architecture (Figure 12) that also includes posterior parietal cortex (PPC), including the lateral intraparietal cortex (LIP), as well as the supplementary eye fields (SEF), frontal eye fields (FEF), BG, thalamus, and superior colliculus (SC). The pART macrocircuit in Figure 1 includes four of these brain regions. The lisTELOS architecture simulates the temporal dynamics of how the brain may control working memory storage, choice, and execution of saccadic eye movement sequences, as they are used to carry out different kinds of tasks. In this spatial working memory, the ‘items’ are target positions to which saccades are commanded to move. An activity gradient across these target positions can store the sequence of target positions to which the eyes will move in the stored temporal order.

The lisTELOS Item-Order-Rank working memory was used to simulate neurophysiological data about SEF and FEF cells that are rank-sensitive (Isoda and Tanji, 2002, 2003). Given that all working memories have a similar network design in order to realise the LTM Invariance Principle and Normalisation Rule, the Item-Order-Rank working memory of the lisTELOS model may be considered a prototype for all linguistic, spatial, and motor working memories in which repeated items may occur.

lisTELOS was also used to learn and perform saccades in five experimental paradigms that did not require storage of a sequence of saccadic target positions (Hikosaka et al., 1989). These paradigms had earlier been simulated by the TELOS model of Brown et al. (2004). Four of these paradigms are shown in Figure 13. A fifth paradigm, the fixation task, requires an animal to learn not to move its eyes from a foveated fixation cue while the cue remains on. This ability is then used to also learn the other four tasks, which all require that the monkey foveate the fixation cue until it turns off, and then to move to an extrafoveal target position. The delayed saccade task, in particular, requires PFC storage of a target light, since the target shuts off 500 ms before the fixation cue does.

Figure 13.

Four of the five benchmark oculomotor tasks that are learned by the lisTELOS model: the saccade task, gap task, overlap task, and memory-guided saccade task. Saccade latencies in the four tasks are consistent with those observed in the literature. In particular, the model reproduces the gap effect by generating saccades at a much lower latency during the gap 500 task (64 ms) than during the saccade task (gap 0; 137 ms) and overlap task (137 ms). Furthermore, saccade latencies are dramatically increased on the memory-guided delayed saccade task (256 ms). These latencies of the model are measured by the timing of its SC burst. F = fixation cue; T = target; E = eye movement.

Source: Reprinted with permission from Silver et al. (2011).

Both TELOS and lisTELOS were able to learn all the tasks. After learning was complete, they could perform all the tasks with the model’s learned parameters, which the models used to reproduce, and predict distinct functional roles for, the neurophysiologically recorded dynamics of 17 different cell types, including properties of FEF dynamics and of sustained cell responses in PFC during the delay period. These predicted functional roles remain to be experimentally tested.

Several kinds of simulations tested how the PFC working memory interacts with the other simulated brain regions. One such test simulated the ISR task, which has served as a benchmark for models of working memory for many years. The ISR task can be divided into two phases. First, a sequence of cues is presented that must be remembered in order. Second, the cues are reproduced in the order in which they were presented. In the lisTELOS simulations, a fixation cue was first presented towards which the model must execute a saccade. Then, a sequence of cues was presented at various spatial positions while fixation was maintained. Finally, the fixation cue was removed. Its disappearance acts as a GO signal during which a BG gate opens, thereby initiating saccades to each of the cued positions in order. This task illustrates, among other things, how SEF chooses the next saccade to be performed from a stored sequence, and how three BG loops sequentially open their gates to enable three different kinds of operations to occur: the storage of a target position sequence in working memory, the choice of the next saccade to be performed, and the performance of that saccade. This gating function of the BG is controlled by the substantia nigra pars reticulata (SNr; Alexander and Crutcher, 1990; Alexander et al., 1986; Grahn et al., 2009; Hikosaka and Wurtz, 1983, 1989; see Grossberg, 2016, for a review).

lisTELOS also simulated DLPFC working memory dynamics that quantitatively reproduced behavioural data, including reaction times, from two microstimulation paradigms (Histed and Miller, 2006; Yang et al., 2008). These experiments focused on the SEF as a region that chooses the next saccade to perform from the sequence of target positions that is stored in the DLPFC working memory (Nachev et al., 2005; Parton et al., 2007; So and Stuphorn, 2010; Taylor et al., 2007). Electrode stimulations altered the order in which eye movements were carried out, but not the target positions to which the eyes moved (Figure 14(a) and (b)). Quantitative simulations of how the saccadic order was changed by microstimulation provided strong evidence for Item-Order-Rank coding in a spatial working memory by showing how the relative activities, and thus the order of recall, were altered. The quite different findings of Histed and Miller (2006) and Yang et al. (2008) about how saccade latency responded to miscrostimulation were also given a unified explanation. These miscrostimulation results followed from how the stimulating electrode increased activity-dependent habituation within SEF feedback pathways (Figure 12) near the electrode position and gradually less with increasing distance, as depicted by the spatial gradient from black (most active) through grey to white (least active) in Figure 14(c).

Figure 14.

SEF microstimulation causes saccade trajectories to converge. Each arrow denotes the initial and final position of a saccadic eye movement during a memory-based saccadic task. In this task, Histed and Miller (2006) trained monkeys to perform a task in which two spatial positions were sequentially cued during an initial fixation phase, remembered during a memory delay, and then visited in order with a sequence of saccades following the offset of the fixation point. Microstimulation did not disrupt the monkey’s ability to correctly saccade to the remembered target positions, thereby supporting the hypothesis that a spatial working memory encodes the saccadic target positions. However, microstimulation did disrupt the order of saccades. The bias of these movements to converge to a position at the upper left target position illustrates the effect of microstimulation. (a) Observed saccade trajectories that converge toward the upper left target. (b) Model simulations reproduce the convergence effect. (c) In model simulations, microstimulation habituates synapses according to a two-dimensional Gaussian function centred over the microstimulation site. The depth of habituation diminishes in a Gaussian fashion across space from black (maximal) to white (minimum). Saccade trajectories after following microstimulation tend to ‘climb’ the Gaussian habituation gradient. Saccades that are furthest from the microstimulation site are least affected by it and thus most likely to serve as the first saccade target position. Data adapted from Histed and Miller (2006). Another important finding of Histed and Miller (2006) was that microstimulation had no effect on saccade accuracy, peak velocity, or latency. These observations suggest that SEF is not involved in the storage, generation, or fine timing of saccades. A target selection role is consistent with these data because SEF microstimulation manipulates only the potency with which plans compete and, during fixation while basal ganglia SNr gates are closed, do not generate new saccade targets. Moreover, if SEF selects targets but does not issue the motor commands that move the eyes, saccade velocity remains unchanged. So long as the duration of the selection process is not changed by microstimulation, saccade latency will also remain unchanged. Reprinted with permission from Silver et al. (2011).

3.19. DLPFC credit assignment by selective and sustained working memory storage

The above examples illustrate some of the working memory properties that are needed to solve the credit assignment problem that Asaad et al. (2017) discuss in the light of their own DLPFC neurophysiological data. As the authors note (p. 6995), ‘credit assignment is the process by which we infer the causes of successes and failures’. In particular, a solution of the temporal credit assignment requires ‘a stable representation of relevant information over time … so that reinforcement received when an outcome becomes apparent can be applied to the same neural ensemble that earlier signaled the causal features’ (p. 6996). A solution of the structural credit assignment problem requires the ability to select causal features when multiple potentially relevant features are simultaneously available.

In the cue learning experiment reported by Asaad et al. (2017) that was used to test credit assignment, rhesus macaques learned which cue was correct by trial and error in blocks of trials, before an unsignalled new block was begun in which a new cue was correct. During each trial, animals were required to maintain central fixation as four cue objects were presented peripherally for 500 ms. The monkeys had to continue maintaining fixation for 1 s after the cues shut off and then to make a choice of the correct cue by making a saccade to the position where the correct cue had appeared. Visual feedback was then presented that indicated whether the choice was correct or not (a green circle or a red X), followed by a juice reward if the choice was correct. Feedback did not reveal which cue had been at the selected position, so the animal had to maintain in working memory a representation of the correct cue against which to match one of the four cues and then saccade to its remembered position. The position of the correct cue was changed across trials within the block.

In the spatial version of the task, the correct choice was determined solely by a cue’s spatial position, and not by the identity of the cue that had previously appeared at that position. In both tasks, an animal’s attention to a particular cue had to be learned from the correct outcomes across trials.

Blocking and unblocking experiments (Section 2.7) also require that an animal infer what cues are predictive based on expected or unexpected consequences that occur after these cues shut off.

A necessary condition for correct credit assignment is thus the ability of the PFC to maintain a representation of the correct cue throughout each trial. Item-Order-Rank working memories accomplish this using the positive feedback in their recurrent shunting on-centre off-surround networks (Section 3.15 and Figure 8). As noted below, such storage is permitted when the appropriate SNr gates open (cf. Figure 11). Another necessary condition is that reinforcing feedback can increase the probability that predictive list chunks will be chosen. This feedback can act either via the amygdala through incentive motivational feedback (Figures 2 and 3) or via the BG through feedback circuits that are strengthened by unexpected rewards. These circuits may, or may not, include the amygdala (Figures 3, 4, and 11).

Reinforcing feedback that enhances the activity of predictive PFC list chunks, combined with SNr gating, are not the only ways that the PFC can selectively choose and store causal contingencies. Sections 3.25 and 3.26 will summarise how the PRC and PHC are proposed to also contribute to this selectivity (Figure 1) by enabling object and spatial contexts, respectively, to influence the storage process. Additional mechanisms regulate whether items will be allowed to be stored in working memory in the first place. Explaining how this occurs requires mechanisms that have not previously been modelled, including functional roles of the ventral bank of the principal sulcus (VPS) and the ventral prearcuate gyrus (VPA; Figure 1). These regions have been shown to enable monkeys to attend and foveate objects that possess task-relevant feature combinations (Bichot et al., 2015). A related role of the inferior frontal junction (IFJ) in humans has also been reported (Baldauf and Desimone, 2014). Here, it will be proposed how these regions may also regulate selective PFC working memory storage.

Before an explanation is offered of how this is proposed to happen, it is useful to summarise some of the data which demonstrate that working memory storage is indeed selective. PFC storage may be prevented on tasks that do not require storage of visual information, consistent with data demonstrating that PFC working memory cells do not fire during such tasks (Fuster, 1973; Kojima and Goldman-Rakic, 1984). Selective storage is also consistent with the observation that, given the presentation of identical stimuli, neural selectivity in PFC depends on subsequent task demands (Warden and Miller, 2010). Awh and Vogel (2008), describing imaging data from McNab and Klingberg (2008), noted that success on working memory tasks was associated with an individual’s ability to selectively identify and store task-related stimuli from a larger sequence of stimuli. Tsushima et al. (2008) showed that subliminal distractors can damage performance in attention tasks, but that making distractors supra-threshold can alleviate performance deficits, perhaps by facilitating the ability to filter them out. Suzuki and Gottlieb (2013) showed similarly that, during a memory saccade task in which a salient distractor was flashed at a variable time and position during the memory delay, responses to the salient distractor were more strongly suppressed and more closely correlated with performance in DLPFC than in LIP. The brain may hereby learn to ‘blacklist’ distracting stimuli before they can be stored in PFC, allowing all other information to be stored. When all of these functions are realised, they embody a solution of the structural and temporal credit assignment problem that the DLPFC has been proposed to solve (Asaad et al., 2017).

3.20. Two processes regulate whether items will be stored in working memory

It is now possible to distinguish two distinct processes that can influence working memory storage, even after all earlier preprocessing has taken place. The first process, as noted above, carries out task-sensitive filtering of individual items before they reach the working memory. This process selects only those items for storage whose feature combinations are compatible with task requirements; for example, only red objects from a sequence of objects with various colours (cf. Egeth et al., 1984; Grossberg et al., 1994; Treisman and Gelade, 1986; Wolfe et al., 1989). This process will be further discussed in Section 3.31 which proposes a mechanistic explanation of the roles of the VPS and the VPA in enabling monkeys to attend and foveate objects that possess task-relevant feature combinations (Bichot et al., 2015) and the related role of the IFJ in humans (Baldauf and Desimone, 2014).

As noted above, the second process enables all the items that get through the filter to be stably stored after they reach the working memory. This corresponds to keeping an SNr gate open during list storage and option prediction (Figure 12). Closing the SNr gate can rapidly reset, or delete, the entire stored sequence from working memory when there is an attention shift to do a different task.

3.21. Masking field working memory chunks variable-length lists

Before discussing the feature-selective gating of objects before they can be stored in VLPFC, it is important to explain how the list chunking networks learn how to selectively categorise sequences of variable length that are stored in working memory; for example, A versus AB versus ABC. How is the most predictive list chunk in a given situation chosen? In particular, in a probabilistic learning situation, how is the most predictive list chunk chosen so that it can read out the best top-down prime with which to predict the most likely stimulus to be reinforced on the next trial, and thereby guide the choice of this stimulus? A Masking Field network has these properties (Figure 7; Cohen and Grossberg, 1986, 1987; Grossberg, 1978a, 1984b, 1986; Grossberg and Kazerounian, 2011; Grossberg and Myers, 2000).

A Masking Field is a specialised type of Item-and-Order working memory that can store chosen list categories through time. As with all Item-and-Order working memories, it is defined by a recurrent on-centre off-surround network whose cells obey the membrane equations of neurophysiology (Section 3.15). The stored ‘items’ of a Masking Field are list chunks that are selectively activated (e.g. chunks representing ‘MY’ and ‘MYSELF’ in Figure 7), via a bottom-up adaptive filter, by prescribed sequences of items that are stored in an Item-and-Order working memory at an earlier processing level (e.g. M, Y, S, E, L, F in Figure 7). The network in Figure 7 thus first compresses spatial patterns of feature detectors into item chunks starting in inferotemporal cortex, and then sequences of the item chunks that are stored in a VLPFC working memory are compressed into list chunks there, among other cortical regions (Figure 1).

3.22. Temporal chunking problem: learning words of variable length

How are the variable-length list chunks of a Masking Field learned, so that they can be used to effectively predict subsequent outcomes? Masking Fields were introduced to solve this Temporal Chunking Problem (Cohen and Grossberg, 1986, 1987; Grossberg, 1978a, 1986) which concerns how a list chunk of an unfamiliar list of familiar speech units – for example, a novel word composed of familiar phonemes or syllables – can be learned under the type of unsupervised learning conditions that are the norm during daily experiences with language.

In particular, before a novel word, or list, can fully activate the adaptive filter, all of its individual item chunks must first be presented. What prevents the familiarity of smaller lists (e.g. MY, ELF, and SELF), which have previously learned to activate their own list chunks, from forcing the novel longer list (e.g. MYSELF) to always be processed as a sequence of these smaller familiar chunks, rather than being able to drive learning of its own list chunk (Figure 7)? How does a not-yet-established word representation overcome the salience of already well-established phoneme, syllable, or word representations to enable learning of the novel word to occur?

3.23. Self-similar growth and competition solve the temporal chunking problem

A Masking Field accomplishes this using cells with multiple cell and receptive field sizes, or scales (Figure 7), that are related to each other by a property of self-similarity; that is, each scale’s properties, including its cell body sizes and their excitatory and inhibitory connection lengths and interaction strengths, are a multiple of the corresponding properties in another scale.

Such a self-similarity property can develop as a result of simple activity-dependent growth laws (Cohen and Grossberg, 1986, 1987). Here is one possible scenario: suppose that item chunk cells in the working memory are endogenously active during a critical period of development. As a result, Masking Field cells that receive inputs from a larger number of item chunk cells receive a larger average total input activity through time. Activity-dependent cell growth causes the Masking Field cell bodies and connections to grow approximately proportionally. This property is called self-similar growth. Cell growth terminates when the cell bodies become large enough to dilute their activities sufficiently in response to their inputs so that they no longer exceed a growth-triggering threshold. Cells that receive more inputs grow larger as a result, so that the effects of individual inputs are smaller on larger cells. In effect, self-similar growth normalises the total effect of all the inputs that converge on a Masking Field cell. Consequently, such a cell fires vigorously only if it receives active inputs from all of its item chunk cells.

Due to self-similar growth, larger list chunks can selectively represent longer lists because they need more inputs, and thus more evidence, to fire. Once they fire, their stronger inhibitory interaction strengths than those of smaller list chunks can inhibit the smaller list chunks more than conversely (Figure 7). This asymmetric competition embodies the intuitive idea that, other things being equal, the longest lists are better predictors of subsequent events than are shorter lists, because a longer list embodies a more unique temporal context.

Asymmetric competition also enables learning of these longer list chunks to occur, because the stronger inhibition from list chunks of longer, but unfamiliar, lists (e.g. MYSELF) enables them to inhibit the chunks that represent shorter, but familiar, sublists (e.g. MY), more than conversely, so that tuning of the LTM traces within the adaptive filter that activates the longer list chunk can occur. Kazerounian and Grossberg (2014) have simulated how variable-length list chunks of a Masking Field can be learned as a list of item chunks is stored in working memory in real time.

Masking Fields have been used to explain cognitive data such as the Magical Number Seven (Miller, 1956), the word length effect in word superiority experiments (Samuel et al., 1982, 1983), phonemic restoration (e.g. Warren and Sherman, 1974), and various other percepts where, in response to a sequence of word inputs, future linguistic context can influence the word sequences that are consciously heard (e.g. Repp et al., 1978).

How does a Masking Field explain the classical Magical Number Seven, or immediate memory span (Section 3.5)? Because a Masking Field can chunk working memory sequences of variable length, it contains cells of variable size. The self-similarity of cell size and asymmetric competition together imply that a larger cell can inhibit a smaller cell more than conversely. This implies that only the largest, and most predictive chunks, will determine Masking Field outputs. Because of self-similarity, the same number of cells of any fixed size can be simultaneously stored. This size will be approximately seven if the transient memory span is chosen to be approximately four (Section 3.11).

How does a Masking Field explain the word length effect in word superiority studies? This effect shows that a letter is progressively better recognised when it is embedded in longer words of lengths from 1 to 4. The word length effect may also seem to follow from self-similarity, since larger chunks are more potent and predictive than smaller chunks. However, self-similarity implies that the list chunk of a familiar multi-letter word can inhibit the list chunk of a shorter word, including a familiar letter (Figure 7), which seems to contradict the property that the word can facilitate perception of its constituent letters, which is the main result of word superiority studies. This problem is resolved in ART systems that use item chunk and list chunk processing levels (Figure 7). In particular, although chunks that represent lists of multiple length compete within the Masking Field that categorises list chunks, the top-down expectations from the list chunk level to the item chunk level are excitatory. By self-similarity, list chunks that represent longer words generate larger inhibitory and excitatory signals (Figure 7). List chunks that represent longer lists will therefore send larger top-down excitatory priming signals to the item chunk level, thereby explaining both how the Magical Number Seven can arise due to asymmetric inhibition among list chunks and how a word length effect in word superiority can arise due to asymmetric top-down excitation from list chunks to the item chunks that activate them.

How future linguistic contexts can influence conscious percepts of previously occurring linguistic items is explained and simulated in articles such as Grossberg et al. (1997) and Grossberg and Kazerounian (2016). Although many neurobiological experiments have shown the importance of activity-dependent plasticity during brain development (e.g. Penn and Shatz, 1999), there seems to be much less experimental evidence that directly studies its effects on neocortical neuron size (e.g. Benders et al., 2015). It would be useful for additional experiments to characterise whether multiple-scale list chunking cells with self-similar properties exist in VLPFC, as predicted by the Masking Field model, particularly in the superficial layers of VLPFC, as suggested by the LIST PARSE model (Grossberg and Pearson, 2008). In particular, if a GABA agonist applied to VLPFC reduces the number of items that can be simultaneously stored there, it should also reduce the longest list that a list chunk can learn and use to predict outcomes from there.

3.24. Visual search: efficient versus inefficient, bottom-up versus top-down

In naturally occurring environments, animals need to search scenes in order to discover the contextual information that can support actions that lead to reward. How this may be achieved will now be discussed, first by reviewing relevant data and then by showing how the pART model explains them, including data about prefrontal mechanisms of feature-based attention and how they may regulate selective storage of items in VLPFC.

Visual attention and eye movements can explore scenes without any goals in mind. Just as often, however, visual searches seek out valued goal objects that are embedded in complex scenes. Common examples include finding a friend in a crowd or locating a menu board in a fast food restaurant. Neurophysiological data from monkeys that illustrate this distinction have been collected by simultaneously recording from multiple electrodes in the parietal and prefrontal cortices (Buschman and Miller, 2007). These experiments used simple stimulus materials to distinguish bottom-up versus top-down processes of attentional control whereby to search a scene. The distinction between a fast automatic bottom-up sweep of activation versus a slower controlled top-down flow of activation has been described in many publications (e.g. Desimone and Duncan, 1995; Grossberg et al., 1994; Hochstein and Ahissar, 2002; Sarter et al., 2001; Treisman and Gelade, 1980). In all conditions of the Buschman and Miller (2007) experiments, a target was randomly located in an array of four stimuli. In the bottom-up, or pop-out, condition, the distractors were identical and differed from the target along the dimensions of colour and orientation. In this case, the target’s salience automatically drew attention to it. In the top-down, or search, condition, each distractor differed independently from the target, and the target matched some of the distractors in each dimension. Memory of the target, rather than its salience, had to be used to find it.

In the pop-out condition, LIP neurons in the PPC were activated first, followed by neurons in the FEF and DLPFC (Figure 1). This kind of search thus proceeded in a primarily bottom-up way. In the search condition, the reverse order of activation was observed, and with a longer latency. Here, search proceeded top-down from prefrontal to lower cortical areas.

In the classical visual search literature, pop-out searches were often called efficient searches. These searches typically yielded zero reaction time (RT) slopes as a function of the number of distractors, hence the term ‘pop-out’ search. The Buschman and Miller (2007) search task illustrates an inefficient search, which in the classical search literature often used targets that are described by a conjunction of features. During an inefficient search, RT increased with the number of distractors (e.g. Treisman and Gelade, 1980). Albeit suggestive, the dichotomy of efficient versus inefficient search based on RT slopes was later shown to be inadequate (e.g. Thornton and Gilden, 2007; Townsend, 1972) because a continuum of flat to steep slopes can be obtained by varying saliency factors (Wolfe, 1998; Wolfe et al., 1989). In particular, search efficiency increases with decreased similarity of targets to distractors and increased similarity between distractors (Duncan and Humphreys, 1989). By proper choice of stimuli, a conjunction search can be rendered efficient, and a feature search can be rendered inefficient, all depending on the degree to which a target can be distinguished from distractors.

Before the eyes move to search a scene, its gist can be rapidly identified (e.g. coast, forest, mountain, countryside; Oliva and Torralba, 2001) if the scene contains enough familiar elements. Gist can be learned as a large-scale texture category in IT (cf. Figure 5) in a manner that is explained and simulated by the ARTSCENE model (Grossberg and Huang, 2009). When the gist texture is supplemented by one to three texture categories of smaller regions of a scene that are learned as the eyes scan the scene, these textures can vote to predict scenic type with up to 91.85% correct on a test set, a benchmark that outperformed alternative models in the literature at that time by 16.5%.

3.25. Object and spatial contexts and reinforcement influence predictive choices

The computation of gist begins the process whereby human observers deploy visual attention using attention shifts and eye movements in a global-to-local and coarse-to-fine manner (Navon, 1977; Schyns and Oliva, 1994), thereby accumulating evidence about a scene, including where a target lies within it (Gold and Shadlen, 2007; Grossberg and Pilly, 2008; Heekeren et al., 2008; Irwin, 1991; Jonides et al., 1982). As part of this evidence accumulation process, object and spatial contexts provide important information that enables working memories and list chunks to carry out more effective visual searches and to more successfully predict which options will be rewarded.

For example, when looking for a friend in a beach picture, our eyes typically fixate the sand at the bottom of the scene before the sky at its top. Such knowledge about the spatial layout of a scene is called spatial contextual cueing (e.g. Chun and Jiang, 1998). Spatial contextual information is not, however, always available in a novel environment. For example, when searching for a beverage in a friend’s refrigerator for the first time, we may not even know where the kitchen is located in the house until we glimpse related objects such as a stove and a sink. In this situation, we may have prior knowledge about which objects may be correlated in a scene like a kitchen, so continue to move towards the room where the stove and sink were glimpsed. However, we may not know the position where the refrigerator is located in this particular kitchen. This is an example of object contextual cueing (e.g. Chun and Jiang, 1999).

Many psychological experiments have described how humans use spatial and object contexts to efficiently search scenes. Such contextual cueing effects are typically measured using the RT for visually searching a familiar scene, subtracted from the RT for more slowly searching a novel scene, as described more fully below. These data have inspired the development of many visual search models (e.g. Backhaus et al., 2005; Brady and Chun, 2007; Grossberg et al., 1994; Itti and Koch, 2000; Torralba et al., 2006; Treisman and Gelade, 1980; Wolfe, 1994). Huang and Grossberg (2010) review how search materials are chosen and how different search models differ.

In general, these models typically try to explain where eye movements fixate to discover targets, and how fixated non-targets lead to the next eye movement. The ARTSCENE Search neural model (Huang and Grossberg, 2010) proposes, in addition, how an eye fixation on an object triggers learning about both its identity and its position, while also matching learned top-down expectations against the object and its position to determine whether it is a target or non-target. Sequences of eye movements also lead to storage of sequences of object and positional representations. These object and spatial contexts are associated through learning with currently fixated objects as the search continues. Associative strength is commensurate with the co-occurrence frequency of the contextual information and the target, the magnitude and frequency of reward of correct target acquisitions, and the attentional valence of both the search target/position and a context object/position. Attentional valence is defined as the degree to which an object attracts attention in response to both bottom-up and top-down factors. For example, a familiar moving object in a scene can attract bottom-up attention due to its activation of transient cells in the Where cortical stream, after which both top-down spatial attention from the PPC and top-down object attention from its learned category in the What cortical stream can further increase its salience, as modelled by Foley et al. (2012). By combining attentionally modulated object and spatial information, each eye movement also helps to accumulate learned contextual evidence about object and spatial contexts that can be used to determine where to look next to most efficiently find the target.

3.26. Perirhinal and parahippocampal cortices store object and spatial contexts

What brain regions carry out these processes? The sequences of scanned objects and their spatial positions are proposed to be stored in object and spatial working memories within the model VLPFC and DLPFC, respectively. Sequences of fixated objects and their spatial positions are also stored in the model PRC and PHC, respectively (Figure 1). Stored PRC and PHC sequences define object and spatial contexts that interact with the VLPFC and DLPFC working memories via bottom-up adaptive filters. The proposed role of PRC and related cortical areas in defining object contexts, and of PHC and related cortical areas in defining spatial contexts, is consistent with neuroimaging data about the dissociation of item and context information by these regions in humans (Aminoff et al., 2007; Diana et al., 2007; Libby et al., 2014).

Learning in the ARTSCENE Search model from a stored object or position in PRC or PHC, respectively, to a stored object or position in VLPFC or DLPFC, respectively, is modulated by a dopamine burst from the model BG (Figures 3(a) and 4(a)) when a target is foveated and reinforced. In this way, predictively successful associations between PRC and VLPFC, and between PHC and DLPFC, can amplify the stored working memory item chunks and list chunks that led to predictive success. The spatial attentional focus can be broadened or narrowed, as task constraints demand, to determine what objects or positions will influence the winning prediction.

Model interactions of IT, PRC, and VLPFC (Figure 1) also clarify neurophysiological data from monkeys that are recorded when they learn, using a delayed match-to-category paradigm, to categorise morphs of image exemplars into two categories; for example, cats versus dogs (Freedman et al., 2001, 2003; Roy et al., 2010). Supervised learning is needed because exemplars that are close to the category boundary, but on opposite sides of it, could be visually more similar than stimuli that belonged to the same category, for example, a cheetah and a housecat. Contextual cueing using sequences of image exemplars, and of a category-predicting discriminative stimulus in cases where it is also presented, help to explain these data. In particular, it was found that IT seems to have properties consistent with ART mechanisms of ITp-ITa category learning (Figure 1), notably attention to critical features of each exemplar (Figure 5), whereas VLPFC seems to have properties consistent with ITa-PRC-VLPFC contextually cued learning (Figure 1), notably sustained activity during the delay period before reward, and greater match/mismatch effects. These studies did not, however, record from PRC, or interactions between PRC and VLPFC. Such additional measures are much to be desired.

3.27. Parietal-prefrontal resonance controls choice of reactive versus planned targets

These last experiments study category learning when stimuli are foveally presented from the start. During daily life, in contrast, an animal typically needs to move its eyes, head, and/or body to foveate important objects. Here, a major design problem needs to be solved: how do planned movements to an extrafoveal object compete with reactive movements to a different extrafoveal object? Rapid reactive movements in response to bottom-up sensory demands are often needed to ensure survival in response to unexpected dangers. Planned movements, that may require PFC executive control and top-down attention, often take longer to select and release. How does the brain prevent reactive movements from being triggered prematurely in situations where a more slowly occurring planned movement would be more adaptive?

A movement gate that is controlled by the SNr is typically tonically active until a movement command inhibits it. Then, the cells that control the corresponding movement can be activated (Figure 15(a)). When a visual cue occurs, the fastest response would be an orienting response to look at it. For this to happen, the cue needs to open the appropriate SNr gate to launch a reactive movement to its position (Figure 15(b)). If the cue is a discriminative cue to do a planned action as quickly as possible, then, as noted above, it may take longer to fully process the features of the discriminative cue to determine the adaptive conditional response. How does the brain know that a plan is being elaborated, even before it is chosen, so that the reactive gate can be kept shut, yet also allow a reactive movement command to open its gate as rapidly as possible when no planned movement command is being formed?

The TELOS model (Brown et al., 2004) explains and simulates how the brain may achieve this balance between reactive and planned movements by predicting how the distribution of excitation and inhibition that converges on the BG when a plan is being elaborated can keep the reactive gate closed (Figure 15(c)). When a movement plan is finally chosen, there is agreement between cells in the FEF and the PPC representation of target position. This agreement is expressed by a synchronous FEF–PPC resonance (Figures 1 and 15(d)) that changes the excitatory–inhibitory balance as it inhibits outlier PPC positions. This resonance is predicted by TELOS to signal attentive consistency between a finally selected movement plan and the target position of the conditional movement. Then, the new balance of excitation and inhibition enables the appropriate FEF-commanded movement gate to open and release the context-appropriate action (Figure 15(d)). As part of their study of bottom-up and top-down attention, Buschman and Miller (2007) reported such prefrontal–parietal resonances, including between FEF and LIP, during movement control, and Pasupathy and Miller (2004) reported that the different time courses of activity in the PFC and BG are consistent with how BG-mediated gating of prefrontal cortical commands are learned in the TELOS model.

3.28. RTs in behavioural data and simulations about object and spatial searches

Computer simulations of contextual cueing data provide considerable support for the ARTSCENE Search model’s proposal of how PRC and PHC help to guide sequential searches. In particular, ARTSCENE Search quantitatively simulates RT data about positive/negative, spatial/object, and local/distant contextual cueing effects during visual search. Figure 16 summarises six of the many experimental conditions about contextual cueing that ARTSCENE Search has successfully simulated. Each panel in the figure depicts RT data (left) and a computer simulation of it (right). These various conditions lead to expectations about what to measure if the neurophysiological methods of Buschman and Miller (2007) are added to traditional psychophysical experiments on contextually cued searches.

Figure 16.

Some simulations of contextual cueing behavioural paradigms using the ARTSCENE Search model. (a)–(c) simulate spatial cueing paradigms, whereas (d)–(f) simulate object cueing paradigms. In all graphs, the x-axis represents training epoch grouped from blocks of trials, and the y-axis represents search reaction time for completing a trial. (a) Positive spatial cueing effects are the RT reductions for search in a familiar spatial context (Old condition) compared to a novel context (New condition). Data reprinted with permission from Chun (2000). (b) Spatial cueing effects can be mainly attributed to target-predictive positions closer to the target, such as those in the same visual hemifield. In the graphs, the conditions New, GlobalCxt, LRCxt, and SRCxt refer to novel, repeated, long-range, and short-range spatial contexts, respectively, with respect to the target position. Data reprinted with permission from Olson and Chun (2002; Experiment 2). (c) Negative cueing effects or context-induced search RT increases can arise at the single subject level due to focused attention. At the group level in which search RTs were averaged across subjects, there was no significant RT difference for search in a familiar spatial context (Old condition) or a novel one (New condition). Data reprinted with permission from Lleras and Von Mühlenen (2004; Experiment 3). (d) Object cueing effects are the RT reductions for search in a congruent or familiar object context (Old condition) compared to an incongruent or a novel context (New condition). Data reprinted with permission from Chun (2000). (e) Selective feature-based attention modulates contextual cueing. In the experiment and simulation, a search trial consisted of red and green items including the target whose colour was maintained and attended to throughout the entire session. Across blocks, the spatial configuration of distractors in a trial was randomly varied in the Control condition, but fully repeated in the Both-old condition. The Ignored-old and Attended-old conditions preserved spatial locations across blocks for distractors in the ignored or attended colour, respectively. Data reprinted with permission from Jiang and Chun (2001; Experiment 3). (f) Task-irrelevant colours did not affect spatial cueing. In the experiment and simulation, the target colour was non-predictable, either red or green. The context layouts were varied in the New condition, but preserved across blocks for half of the items that shared the target colour in the Old-Match condition. In contrast, the Old-Oppose condition preserved locations for half items that differed in colour from the target. Data reprinted with permission from Olson and Chun (2002; Experiment 4). Adapted with permission from Huang and Grossberg (2010).

Figure 16(a) summarises RT data and a simulation of positive spatial cueing. Positive spatial cueing effects are the RT reductions for search in a familiar spatial context compared to a new context. In this paradigm (Chun and Jiang, 1998), a fixed target position was chosen from a grid search display without replacement and presented in one trial per block. Across blocks of search trials, a target position was accompanied by either a repeated spatial configuration of distractors (Old condition) throughout the entire experiment or a random configuration that was newly generated in each block (New condition). In Figure 16(a) (left panel), the x-axis represents search epochs grouped from blocks of trials, and the y-axis represents search RT for completing a trial. Since the upper and lower curve in each panel of this figure corresponds, respectively, to the New and Old spatial context condition, the separation between these two curves indicates the amount of contextual facilitation in search RT that derives from a regular spatial context. Notice that the RT in the New spatial context condition dropped across epochs, and a further RT reduction in the Old spatial condition also developed as the session progressed.

ARTSCENE Search replicates spatial cueing effects by learning pairwise associations between a context position in PHC and a target position in DLPFC. Specifically, when a search display is presented, it activates model PPC and, from there, both DLPFC and PHC as the eyes search a scene. Each context position that is stored in PHC sends biasing signals to all the target positions with which it has been associated in the past that are currently stored in DLPFC. When the contextual information combines with the intrinsic Masking Field dynamics of DLPFC, the chosen DLPFC list chunks encode the likelihood of seeing a target at each position. Top-down feedback from these list chunks to the FEF (Figure 1) then biases attention and eye movements toward likely target positions given the current scene layout. As a consequence, an eye-scan path becomes more target-based rather than saliency-based. Accordingly, the probability of fixations on salient distractors is reduced, reflected as spatial cueing effects.

Why does the New curve also decrease with increased training? This happens because the strongest pairwise associations learned by the model are typically from a target position to itself due to its perfect self-correlation. Unlike positions where a target never occurs, a target position itself, once it re-appears in a search trial, signifies target presence and strongly attracts overt attention. Therefore, search RT can still decrease during the course of training even if a target position is presented in combination with a new context.

The five other panels in Figure 16 depict other spatial contextual cueing (Figure 16(b) and 16(c)) and object contextual cueing (Figure 16(d)–(f)) paradigms that the model successfully simulates. The object contextual cueing results depend upon model interactions between the model’s IT, VLPFC, and PRC regions (Figure 1). These and other successful simulations of the model strongly support its proposed mechanisms. It should, however, be noted that the simulations compute the decreasing number of eye fixations in the New and Old conditions, rather than absolute RTs. By assigning an RT to each fixation duration, a fit to RTs can also be achieved. However, that would still leave open the question about whether each of these fixation steps takes that amount of time in associated brain regions. Such a mixed experimental and modelling study remains to be done.

3.29. Where-to-What and What-to-Where interactions learn and search for objects

The ability to carry out a search for a desired object requires a solution to a major design problem that sufficiently advanced brains have solved. The What cortical stream learns recognition categories that tend to be increasingly independent of object view, size, and position at higher cortical levels, with ITa cells, among others in the temporal cortex, exhibiting such invariance (Bar et al., 2001; Sigala and Logothetis, 2002; Tanaka et al., 1991). The cortical magnification factor does, however, limit the degree of positional invariance, as reflected by the neurophysiologically recorded trade-off between object selectivity and position tolerance in ITa cells (Zoccolan et al., 2007). Grossberg et al. (2011) have used ART to explain and simulate this trade-off.

Invariant recognition categories avoid a combinatorial explosion in the number of categories that are needed to represent an object. Instead of having to learn a different category for each object view, position, and size, the brain just uses a small population of cells in ITa to code an invariant object category that is significantly invariant under changes in view, size, and position (e.g. Hung et al., 2005). This invariant category can then easily interact with other brain processes, such as reinforcement learning and working memory storage. In particular, an invariant object category in ITa may be attended with higher probability if it receives motivated attention via an ITa-AMYG-OFC cognitive-emotional resonance (Figures 1 and 2).

In becoming positionally invariant, however, ITa recognition categories lose information about the positions in space of the objects that they represent. The Where stream represents target positions and controls actions aimed at acquiring them, but does not represent featural properties of the objects themselves. These What and Where stream properties are computationally complementary (Grossberg, 2000a, 2013a). Interactions between the What stream and the Where stream overcome these complementary computational deficiencies. Using What-to-Where interactions, invariant object categories in the What stream can use Where stream spatial representations to control actions towards desired goals in space. Section 3.31 explains how this is proposed to occur.

In addition to What-to-Where interactions that help to search for an object, the ARTSCAN Search model has simulated how Where-to-What interactions enable invariant object categories to be learned as the eyes freely scan a scene. Along the way, this model has successfully explained and predicted many data about interactions between multiple brain regions, including V1, V2, V3A, V4, ITp, ITa, PPC, LIP, PFC, FEF, and SC (Cao et al., 2011; Chang et al., 2014; Fazl et al., 2009; Foley et al., 2012).

In particular, the model explains how, as the eyes freely scan a scene, learning occurs from prestriate visual cortex (e.g. V4) to posterior inferotemporal cortex (ITp) of categories that combine both featural and positional information, and how multiple ITp categories learn, in turn, to be associated with an emerging invariant ITa category (Figure 1). Category learning from ITp-to-ITa uses ART bottom-up adaptive filters. This learning is dynamically stabilised by learning top-down expectations from ITa-to-ITp (Figures 1 and 5).

For invariant category learning to work, the brain needs to solve a basic View-to-Object Binding Problem: as the eyes scan a scene, two successive eye movements may focus on different parts of the same object or on different objects. How does the brain avoid erroneously classifying views of different objects together into an invariant object category, even before the brain knows what the object is? ARTSCAN Search proposes how spatial attention in the Where stream and object attention in the What stream interact to solve this problem.

The category learning process begins when a view- and position-specific category of a novel object is learned in ITp and activates cells in ITa. These ITa cells will learn to encode an invariant object category as multiple specific ITp categories are associated with it as the eyes explore the object surface. When the object view changes enough, the previously active ITp category gets reset to enable a new one to be activated and learned. The emerging invariant category in ITa cannot, however, get reset because it needs to remain active while it is associated with multiple ITp categories of that object.

Why is the invariant category not reset? An attentional shroud in PPC is predicted to inhibit reset of an invariant object category while the eyes scan different views and positions of the object (Figure 17). An attentional shroud (Tyler and Kontsevich, 1995), or form-fitting distribution of spatial attention, can form pre-attentively even before the brain learns to recognise the surface as representing a particular object. Such a shroud is part of a surface-shroud resonance (Figure 17) that arises due to positive feedback interactions between a surface representation (e.g. in cortical area V4) and spatial attention (e.g. in PPC). A surface-shroud resonance maintains sustained spatial attention upon the object to be learned and triggers the process whereby the attended surface qualia becomes consciously visible (Grossberg, 2017b). While the shroud is active, it inhibits the parietal Category Reset mechanism that would otherwise inhibit ITa (Figure 17) while helping to select eye movement target positions whereby to foveate salient features on the object surface. The process whereby sequential eye movement targets are chosen occurs from V2-to-V3A-to-LIP and FEF (Figure 17). When spatial attention shifts to focus on another object, the reset mechanism is transiently disinhibited, and its burst of activity inhibits the active ITa category (Figure 17), so that a new object can be attended and its invariant category learned. See Grossberg (2013a, 2017b) for extensive discussions of this process and the psychological and neurobiological data that it explains and predicts.

3.30. What working memory filtering and activation of Where target positions

After Where-to-What stream interactions help to learn invariant object categories, What-to-Where stream interactions regulate how to foveate valued target objects in a scene. Both ARTSCAN Search and ARTSCENE Search proposed a minimal anatomy (Section 1.3) that could carry out this function, while also simulating challenging RT data about visual search, for example, Figure 16.

Such an anatomy proposes how an invariant object representation in the What stream can activate a positional representation in the Where stream that can be used to foveate a valued target object in a scene. However, it did not try to solve the problem of how the brain can selectively filter desired targets from a stream that also contains distractors, so that it only attends, stores, and foveates matched targets. The minimal anatomy is summarised here for multiple reasons. First, it was sufficient to quantitatively simulate challenging RT data in many contextual cueing experiments. Second, it may have evolved before the prefrontal mechanisms of selective working memory storage did, may operate in parallel with them, and may be unmasked if the prefrontal mechanisms are lesioned. This last possibility may be worth testing directly.

In the minimal anatomy of ARTSCENE Search, winning VLPFC activities send a top-down attentional prime to ITa. An open SNr gate lets the primed ITa cells fire. These ITa cells can then prime the positionally sensitive categories in ITp with which they were associated when ITa was being learned using resonant bottom-up and top-down interactions (Figure 5). If one of the primed ITp categories also receives a bottom-up input from an object at its position, then it can fire and activate positional representations in LIP and FEF. These positional representations can move the eyes to the position in space that they represent.

3.31. Feature-based attention, saccadic choice, and selective working memory storage

More is needed for VLPFC to also be able to selectively filter desired targets from a stream that also contains distractors, and to enable it to selectively attend, store, and foveate matched targets. Neurophysiological data about the role of VPA as ‘a source for feature-based attention’ (Bichot et al., 2015: 832) may be understood in the light of these additional functional properties, notably how and why cells in VPA selectively match desired combinations of object features, resonate with a target that matches these features, and then rapidly activate a positional representation in FEF that can command a saccade to this target. These properties were recorded in experiments where fixating monkeys were presented with a central cue object that defined the search target, which was followed by a delay during which the monkeys held a representation of the target in memory. Then, an array of eight stimuli appeared that contained the search target and seven distractors. The monkeys found the target using free gaze and were rewarded for maintaining fixation on it for 800 ms.

Bichot et al. (2015) conducted experiments in which they simultaneously recorded from IT, VPA, and FEF in two monkeys, and VPS, VPA, and FEF in two other monkeys (Figure 1). The following summary proposes a mechanistic and functional explanation of how these cells interact together, notably how they enable matched and mismatched objects to be selectively processed in PFC (Figure 1):

1. Both ITp (TEO) and ITa (TE) project to PFC (Barbas and Pandya, 1989; Tanaka, 1996; Webster et al., 1994).

2. ITp topographically projects to VPA, whose cells exhibit significant sensitivity to extrafoveal positions (Bichot et al., 2015), as do those in ITp (Tanaka, 1996).

3. ITa topographically projects to PRC, as in the ARTSCENE Search model, and also to VPS, which in turn projects to VLPFC. As noted in Bichot et al. (2015), VPS had the largest spatial tuning curve of any of the cells that they recorded, consistent with invariance properties of ITa.

4. VLPFC outputs top-down signals to both VPS and VPA, where they learn modulatory top-down expectations when they are associated with the currently active VPS and VPA cells. These expectations are assumed to obey the ART Matching Rule (Figure 5(b)).

5. The activity of cells in VPA that are receiving an active VLPFC-to-VPA prime are enhanced when a currently presented extrafoveal object matches target features in their receptive field, and is suppressed when such an object mismatches expected target features, consistent with the ART Matching Rule.

6. The enhanced VPA activity is sufficient to trigger an output signal to FEF at the corresponding positional representation in FEF (Figure 1), consistent with data of Bichot et al. (2015) showing VPA activating around 20 ms before FEF. FEF can then elicit a saccade to the matched target, leading to it being foveated. By inhibiting inputs from objects that mismatch the VPA expectation, mismatched objects are not foveated.

7. The activity of cells in VPS that are receiving an active VLPFC-to-VPS prime are enhanced when an invariant object category from ITa matches their receptive field, and are suppressed when such an object mismatches it, consistent with the ART Matching Rule. When a match occurs, a synchronous resonance develops that enables the category to be stored in VLPFC (Figure 1). This resonance propagates through multiple cortical areas, in the manner described in Section 3.32, and supports conscious recognition of the object.

The mapping between VPA and FEF positions is assumed to have been learned in response to series of objects that have, in the past, activated the What and Where streams in parallel (Figure 1). The kind of learning that can associate corresponding VPA and FEF positions has previously been simulated in the FACADE (Form-And-Color-And-DEpth) model of 3D vision and figure-ground perception, where it was used to associate corresponding positions in the boundary and surface representations within the interblob and blob cortical streams, respectively, through V1, V2, and V4 (Figure 1) as they were activated in parallel by a series of objects (Grossberg et al., 2002). Once these cortical streams were associatively linked, output signals from oriented boundaries to the colours with which they were associated in the surface stream provided an explanation of many properties of the McCollough effect, a striking long-term, oriented, chromatic aftereffect, among other percepts (McCollough, 1965).

The pART macrocircuit in Figure 1 contains two cortical areas, VPA and VPS, that have not previously been simulated. However, interactions and functional roles for all the other brain regions in Figure 1, and some that are not included in this Figure (cf. Figures 3–5, 12, 15, and 17), have been extensively simulated in earlier models, and used to explain large psychological and neurobiological data bases. These models provide a secure foundation for including VPA and VPS in a unified model of prefrontal dynamics. Models of cognitive-emotional dynamics, such as CogEM, MOTIVATOR, and nSTART, have already been mentioned above, as have What stream invariant category learning and search models such as ART, ARTSCAN Search, and ARTSCENE Search. In addition to these models, all the Where stream brain regions in Figure 1 have been extensively modelled, including a model of motion processing, form-to-motion interactions, and directional attention (3D FORMOTION model); motion-based decision-making (MODE model); visual navigation using heading, obstacle avoidance, and route selection (ViSTARS model); and target tracking by combinations of saccadic and smooth pursuit eye movements (SAC-SPEM model) that together simulate properties of Where stream cortical areas V1, middle temporal cortex (MT), medial superior temporal area (MST), PPC/LIP, FEF, and PFC, and areas with which they interact (Baloch and Grossberg, 1997; Berzhanskaya et al., 2007; Browning et al., 2009a, 2009b; Elder et al., 2009; Grossberg et al., 1999, 2001, 2012; Grossberg and Pilly, 2008; Pack et al., 2001; Srihasam et al., 2009).

3.32. Synchronisation of multiple cortical regions for feature-based attention

The proposed role of VPA for ‘feature-based attention’ should not be conflated with the ‘feature-based attention’ that supports conscious seeing and knowing about a familiar object. In this regard, ART and ARTSCAN predicted – and thereby explained a lot of data – about how percepts of visual qualia may become conscious due to surface-shroud resonances that are triggered between V4 and PPC (Figures 1 and 17), before propagating both bottom-up and top-down to other cortical areas; how familiar objects may be recognised due to a feature-category resonance that is triggered between V4 and IT (Figures 1 and 17), before propagating both bottom-up and top-down to other cortical areas; and how an observer may consciously see and know about a familiar object when these two types of resonances synchronise (Grossberg, 2013a, 2017b).

VPA processing carries out a type of top-down ‘feature-based attention’ in a strict mechanistic sense because its circuit seems to embody the ART Matching Rule (Figure 5(b)), as do multiple stages of feature-based attention (Grossberg, 2013a, 2017b). Multiple cortical stages that compute ‘feature-based attention’ can synchronise during a match state, as illustrated by MEG and fMRI data of Baldauf and Desimone (2014) in humans. See also Buschman and Miller (2007), Engel et al. (2001), Gregoriou et al. (2009), and Pollen (1999).

The LAMINART (Laminar Adaptive Resonance Theory) model (Grossberg, 1999; Grossberg and Raizada, 2000; Raizada and Grossberg, 2001, 2003) clarifies how multiple cortical stages can synchronise (Figure 18). This model proposes how all granular neocortical areas may combine bottom-up, horizontal, and top-down interactions that embody variations of the same canonical laminar cortical circuitry. Because of this shared circuitry across cortical areas, the ART Matching Rule circuit in Figure 5(b) may be realised using a similar circuit design at multiple stages of cortical processing. For example, in Figure 18, the top-down attention pathway from layer 6 in V2 projects to layer 6 in V1, which sends bottom-up signals to layer 4. These bottom-up signals are sent via a modulatory on-centre (note the balanced excitatory and inhibitory pathways to layer 4) surrounded by a driving off-surround network. The top-down signals from V2 are hereby ‘folded’ at layer 6 in V1 in order to reach layer 4. This property shows that object attention is carried by a top-down, modulatory on-centre, off-surround network whose off-surround is activated by a folded feedback network.

Figure 18 illustrates how a top-down, task-selective priming signal from PFC can propagate through multiple lower cortical areas via their layer 6, which can then activate their layer 6-to-4 modulatory on-centre, off-surround networks via folded feedback. In this way, an entire cortical hierarchy, including multiple stages of ‘feature-based attention’, may get primed to process incoming bottom-up signals to accommodate the bias imposed by the prime. When a matched bottom-up target is received by this cortical hierarchy, multiple processing stages can rapidly go into gamma synchrony, as simulated by Grossberg and Versace (2008), and can support recognition of the target.

In summary, as Figure 1 illustrates, section 3 has summarised how selective, context-sensitive storage of event sequences in cognitive and motor working memories occurs, and how learning and performance of the most predictive cognitive plans, or list chunks, builds upon the event sequences that are stored in working memory. Selectivity is achieved both by feature-based attention and SNr BG gating. Attentive matching can trigger orienting movements to foveate a target object and can cause a feature-category resonance to propagate through multiple cortical levels and to support conscious event recognition and working memory storage. Opening an SNr gate enables storage in working memory to occur, and closing the gate inhibits storage and prepares the working memory to store future sequences without bias. Choice of list chunks that can best predict valued outcomes is supported by a combination of Masking Field competitive dynamics (Figure 7), PRC and PHC contextual associations (Figure 1), and reinforcement learning mechanisms that operate via both the AMYG and OFC (Figures 1–3) and via SNc Now Print signals in response to unexpected rewards and non-rewards (Figures 3(a), 4(a), and 12).

4. Concluding remarks

This article describes an emerging neural architecture, called pART (Figure 1), that embodies explanations and predictions about the neural mechanisms and functional properties of several prefrontal cortical areas, including OFC, VLPFC, DLPFC, VPS, VPA, and FEF as they carry out cognitive-emotional and cognitive working memory dynamics. The explanations describe how these prefrontal areas may realise their functional properties as emergent properties due to interactions among themselves and with other brain regions, including the amygdala, BG, cerebellum, V1, V2, V3A, V4, ITp, ITa, MT, MST, LIP, PPC, and SC. These functional properties include the computation of desirability by OFC and availability by VLPFC (Rudebeck et al., 2017), a solution of the credit assignment problem by DLPFC (Asaad et al., 2017), and how feature-based attention by VPS and VPA may filter expected versus unexpected objects and thereby both help to direct saccadic eye movements to expected objects (Baldauf and Desimone, 2014; Bichot et al., 2015) and select items for storage in the VLPFC cognitive working memory. Cognitive-emotional interactions, including reinforcement learning and incentive motivational learning; object and spatial working memory dynamics; and category learning, including the learning of object categories, value categories, object-value categories, and sequence categories, or list chunks, are among the processes that are carried out by this architecture. Several functionally distinct types of attention (prototype, surface, and motivated attention) help to dynamically stabilise these learning processes as well as to predictively prime their target representations. Prototype attention focuses upon the critical feature patterns that are attended during the feature-category resonances that support object recognition (Figure 5); surface attention focuses on an object surface during the surface-shroud resonances that support conscious seeing of its visual qualia (Figure 17); and motivated attention focuses on valued objects during conscious cognitive-emotional resonances and supports conscious feelings about them (Figures 2 and 3; Grossberg, 2013a, 2017b).

Many of the pART explanations in this article can be viewed as confirmed predictions, because the model mechanisms that are used to explain the data anticipated the experiments that reported them. The article also suggests multiple additional predictions whereby to further test these explanations. Some of these predictions are stated explicitly in the text, but every model interaction and emergent property implicitly embodies additional predictions. In particular, the article’s unified explanation of cognitive-emotional and cognitive working memory dynamics enables each laboratory to imagine new kinds of experimental tests that would be amenable to its own particular laboratory capabilities.

It should also be emphasised that, once models sufficiently explicate the actual design principles, mechanisms, circuits, and architectures that are used by our brains, the models can lead to explanations and predictions in totally different areas of psychology and neuroscience than those that were their original explanatory targets. That is how, for example, explanations have been discovered showing how particular breakdowns in normal brain functioning may give rise to behavioural symptoms of mental disorders that afflict millions of individuals, including Alzheimer’s disease (Grossberg, 2017a), autism and Fragile X syndrome (Grossberg, 2013a; Grossberg and Kishnan, 2018; Grossberg and Seidman, 2006), schizophrenia (Grossberg, 2000b), and medial temporal amnesia (Carpenter and Grossberg, 1993; Grossberg, 2013a). Here too, multiple predictions are made for which additional experimental tests would be of great value.