Computational Models of Cognitive Control

Abstract

Cognitive control refers to the ability to perform task-relevant processing in the face of other distractions or other forms of interference, in the absence of strong environmental support. It depends on the integrity of the prefrontal cortex and associated biological structures (e.g., the basal ganglia). Computational models have played an influential role in developing our understanding of this system, and we review current developments in three major areas: dynamic gating of prefrontal representations, hierarchies in the prefrontal cortex, and reward, motivation, and goal-related processing in prefrontal cortex. Models in these and other areas are advancing the field further forward.

Introduction

Computational models are important for making explicit links between biological mechanisms and the cognitive and behavioral phenomena that they produce. In the domain of research on cognitive control (e.g., the ability to perform task-relevant processing in the face of other distractions or absence of strong environmental support), there is a rich history of computational modeling that has served to focus empirical and other theoretical work on specific biological mechanisms and their functional roles. For example, early models showed how active maintenance of information in a working memory system could be accounted for in terms of reverberatory excitation among a set of interconnected neurons [1, 2], and how this actively maintained information can provide a strong “top down bias” a to influence processing throughout the brain [3, 4, 5, 6]. These models resonated with a growing body of data from many methodologies, and associated theoretical ideas, to provide a coherent account of the involvement of the prefrontal cortex (PFC) in cognitive control [7, 8, 9]. This work has been highly influential in the field, and represents one of the most significant success stories for the contributions of the computational modeling approach.

Building upon this first wave of synthesis, a new generation of computational models have been extending the theoretical and computational framework to include multiple other systems that the PFC interacts with to achieve cognitive control, and elaborating the contributions that the PFC itself makes. Some particularly active areas of research include:

• Monitoring and feedback mechanisms that can regulate the application of PFC-mediated cognitive control, associated with the anterior cingulate cortex [10, 11, 12].

• Dynamic gating mechanisms that can influence the updating and active maintenance of information in PFC, associated with the basal ganglia [13, 14, 15] and the neuromodulator dopamine [16, 7].

• The hierarchical organization of PFC representations as a means of efficiently performing complex cognitive tasks [17, 18, 19].

• The interactions between PFC and subcortical areas involved in reward and motivation, to organize cognitive control around desired goals, and provide higher-level control and integration of reward and punishment signals [20, 21, 22, 23].

We review some key ideas from the last three of these areas below.

Dynamic Gating of PFC Representations

A remarkable consensus across multiple different levels of analysis has converged on the general idea that an important element of cognitive control is a dynamic gating mechanism that can determine when PFC representations are updated to reflect new information, versus when they continue maintaining older information. From a purely computational perspective, the LSTM (Long Short-Term-Memory) model advanced this gating idea to provide a more robust active memory system [24]. From a biological perspective, the notion that the basal ganglia act as a gating system in the context of motor control has long been accepted [25, 26, 27], and this can be directly extended to account for a dynamic gating role in working memory [13, 14]. Interestingly, these neurally-based ideas have converged with more top-down driven cognitive modeling work in the ACT-R framework, which postulates a gating-like role for the basal ganglia as well [15]. The ACT-R modeling work shows how basal ganglia gating can initiate the firing of cognitive productions, as in the classical production system models from symbolic AI. Thus, it seems that several major threads of computational modeling work are on the verge of a significant new integration [28, 9].

Several recent models have explored the dynamically-gated working memory system in the context of rich mathematical frameworks such as bilinear mappings and partially-observable markov decision processes (POMDP), which provide important insights into the computational properties of these models [21, 29, 30]. The recent work of Dayan [21, 29] is particularly interesting in attempting to bridge between basic gating-like working memory updating mechanisms and the ability of the system to perform arbitrary tasks rapidly through verbal instruction. Dayan draws nice connections between the ACT-R work and the habit versus goal-directed action framework in animal learning [31]. In this context, most existing neural models can be considered to be performing according to a complex habit that is inflexible and cognitively impenetrable, whereas much of human performance is more goal-directed or rule-governed (flexible and cognitively driven). Dayan shows how this rule-governed behavior can emerge from more basic gating-like mechanisms, though many difficult problems remain to be solved in this area.

At a more cognitive level, the work of Ashby and colleagues has shown how a PFC/BG system can explain important findings in the categorization literature, adding further empirical avenues for testing such models [32]. More generally, the empirical data on biological and cognitive neuroscience studies of this PFC/BG system is growing rapidly, and providing considerable support for the central ideas behind these computational models [33, 34, 35, 36]. Thus, this area of research and modeling represents another important emerging success story for the computational modeling approach.

One outstanding question in this area is the relative role of dopamine as a gating signal in the basal ganglia and PFC. Some models focus on the role of dopamine in training a selective gating signal in PFC that is capable of gating new information into some regions of PFC, while leaving others to actively maintain older information [9, 37]. Other models focus on the role of a global dopamine signal in PFC for gating [16, 38]. Future work should examine the interplay between a global dopamine signal, and more focal gating signals from BG.

Hierarchies in PFC

Models of behavioral control have long involved hierarchies [39]. Only recently, however, have hierarchical models made explicit contact with relevant neuroscientific data, which is accumulating quickly [40, 18, 41]. Convergent findings suggest a hierarchy organized along the posterior-anterior anatomical axis of the PFC, with more anterior areas providing higher-level control representations, consistent with the original ideas of Fuster [42]. However, the best way to characterize this hierarchy is still a matter of considerable debate, and computational models are providing some important insights.

Botvinick [17] showed how learning hierarchically structured tasks in a network with connectivity similar to PFC can give rise to a semi-hierarchical organizational structure. The shortest route from stimulus to response in the model was directly from stimuli to premotor areas, with increasingly more indirect routes available through more anterior PFC areas (consistent with anatomical data [40, 42]). When this network learned several complex behavioral tasks, such as making tea or coffee, representations of more temporally protracted task elements developed in higher-level layers. Critically, however, the model was not strictly hierarchical, and thus can capture important subtleties in human performance, such as the context sensitivity of subordinate actions.

Biologically detailed models of PFC active maintenance with basal ganglia gating mechanisms have also been shown to develop hierarchical task representations [19], in the context of the hierarchically-structured 12-AX task [14]. Two architectural variations of the model were used. In the first, a posterior region of PFC received input directly from sensory areas, and projected to motor areas, while an anterior PFC region received only from posterior PFC, similar to the architecture used by [17]. The second architectural manipulation made the BG connectivity hierarchical, with anterior PFC driving posterior BG gating, but not vice-versa. This manipulation is consistent with findings that PFC/BG loops exhibit a spiral-like structure [43, 44]. Both manipulations caused specialization for more temporally extended (“outer loop”) representations in anterior PFC, with additive effects of each.

These studies show how initial connectivity, combined with learning, can explain some aspects of a hierarchical organization in PFC, within an overall framework of common underlying mechanisms across the PFC. However, the empirical literature is unclear about the precise nature of the PFC hierarchical structure: e.g., is it about rule complexity and relational integration [45, 46, 18], or is it more about abstract representations [47, 48, 49], or both? One intriguing prospect for future research is that basic differences in duration of active maintenance across PFC areas (e.g., due to more anterior areas being further removed from sensory perturbations) may interact with learning mechanisms to naturally produce a gradient of abstraction [9, 50].

Reward, Motivation, and Goals

The PFC is considered the “executive” of the brain for multiple reasons, among them that it is strongly concerned with the “bottom line” – satisfying the basic needs and goals of the organism. This occurs through bidirectional interactions with subcortical and other “limbic” brain areas that are known to be involved in processing basic affective signals such as reward, punishment, fear, etc. The ventral and medial areas of the PFC are most directly involved in these interactions, and considerable attention has been focused on the roles of the orbital prefrontal cortex (OFC), and the anterior cingulate cortex (ACC).

The OFC is strongly interconnected with the basolateral amygdala (BLA), which is known to be important for processing a range of basic emotions. From a computational perspective, the OFC may play a similar role in this system as PFC does more generally: active maintenance and top down biasing, but in this case maintenance of reward-related information, and biasing of function in the BLA. Also, because subcortical areas may have relatively slow synaptic learning, rapid working memory updates in the OFC may be required to keep track of recent changes in outcome probabilities and magnitudes [51, 22, 52]. Models implementing this idea have been able to account for a range of different data [22, 52]. For example, patients with OFC damage have been shown to exhibit poor decision making in gambling tasks, which seems to be due to an underlying difficulty in integrating reward magnitude with its probability, and/or reversing initial positive reward associations [53]. The OFC model of Frank & Claus [22] was able to account for this data in terms of a reduced active maintenance system that otherwise facilitates reversal learning of emotional associations, in a manner consistent with other such reversal learning models [49, 54], and OFC lesions in rats [55, 52].

Although the anterior cingulate cortex (ACC) is often thought of as a conflict monitoring area [12], a growing body of data suggests that this may be a subset of a more general computational function as an OFC-like area that is concerned with associations between motor actions and reward/punishment outcomes [56] (whereas OFC is concerned with associations between stimuli and reward/punishment outcomes). One recent model of ACC suggests that it arbitrates between model-based and model-free responses encoded in the dorsomedial vs. dorsolateral striatum, respectively [20, 21]. Model-based responding is goal directed, and is sensitive to manipulations such as devaluation of the unconditioned stimulus, whereas model-free responding is more habitual and insensitive to outcome-related manipulations [57]. A neural network model of instrumental and Pavlovian conditioning shows how widely-accepted biological learning mechanisms can result in a shift from dorsomedial goal-directed behaviors early in skill acquisition to dorsolateral stimulus-response solutions during consolidation of a skill [23].

More work is needed in this area to explore how different types of affective and goal-related signals are encoded and processed in different PFC areas — the ventral and medial areas of PFC are extensive, and current research has focused somewhat narrowly on reward-related processing within a restricted range of behavioral paradigms.

Conclusions

In effect, computational models act like very precise theories, working along with other theoretical ideas at various levels of description. A crucial benefit of computational models as theoretical tools is that they enable simulation of complex cognitive processes, and principled generation of novel hypotheses for testing. The models reviewed above, covering a wide range of different levels of abstraction, contribute important insights and predictions that are being actively tested. This new body of data is then informing new versions of the models, perpetuating the classic cycle of theory development, testing, and revision that underlies all good science.

In addition to the domains mentioned above that have received significant coverage in the literature, we suggest that there are several important areas that are ripe for future exploration, including models exploring the interactions between hippocampal episodic memory and PFC working memory [58, 59, 60], and more work examining the nature of learning and rapid vs. slow adaptation of PFC representations over time — the prevalent focus on active maintenance as the main form of memory in the PFC may be causing us to miss out on a range of other important memory mechanisms that may have important explanatory roles in a wide range of cognitive control phenomena, especially as one considers the developmental spectrum [61, 62].