Cortical and subcortical contributions to context-control learning

Abstract

“Cognitive control” describes our ability to strategically bias information processing in line with internal goals. Traditionally, research has focused on delineating the sources of top-down biasing, implicating the lateral prefrontal cortex. The past two decades, however, have seen increasing interest in the regulation of control, that is, how learning processes guide the context-sensitive application of top-down biasing. Here, we review and synthesize recent research into the cognitive and neural mechanisms of this type of “context-control learning”. We first discuss a fast-growing cognitive psychology literature documenting how specific cognitive control states can become associated with, and subsequently triggered by, contextual cues. We then review neuroimaging studies that speak to the neural substrates of contextual adjustments in control, with a particular focus on recent work that explicitly modeled context-control learning processes. We conclude that these studies suggest an important subcortical extension of the traditional frontal control network, as they indicate a key role for the caudate nucleus in forming associations between contextual cues and appropriate control settings.

1. Introduction: From ‘will over habit’ to associative learning of cognitive control

Cognitive control refers to a collection of mechanisms that allows us to process and act on information in accordance with our current internal goals and external context (Egner, 2017; Miller & Cohen, 2001). It includes, for example, the ability to maintain and process task-relevant information (selective attention), the ability to shield ongoing task sets from interference by minimizing task-irrelevant processing (conflict control), the ability to update task sets in response to changing goals (task switching), and the ability to cancel prepared responses (response inhibition). The processes supporting these functions are thought to be resource- (or computation-) intensive and capacity-limited. Historically, “controlled processing” is regarded as operating in direct opposition to “automatic processing” - routine behavior that is mediated by well-practiced stimulus-response (S-R) associations, which shortcuts effortful and time-consuming computations (Posner & Snyder, 1975; Schneider & Shiffrin, 1977).

This juxtaposition of “top-down” intentional control that is occasionally required to override “bottom-up” driven habitual responding is well-captured in the case of conflict control in the classic Stroop task (MacLeod, 1991; Stroop, 1935). This task pits an unpracticed temporary goal (“name the ink color in which color-words are printed”) against the overlearned process of word-reading, with the result of reliable behavioral costs (“congruency effects”) when the output of the automatic word-reading process is incongruent with the instruction-based response (e.g., when responding to the word BLUE printed in red ink) compared to when it is congruent (e.g., the word BLUE printed in blue). A large congruency effect is therefore taken as evidence for strong interference from habitual processing and a low efficacy or engagement of top-down control, and vice versa for a small congruency effect. Many variants of this task have been developed over the years to track the deployment of control (for an example of a variant of the Stroop task using face stimuli, see Fig. 1a).

Figure 1. Context-control learning protocols and the caudate nucleus.

(a) In a face-name Stroop task, participants are asked to identify the face and ignore the overlaid written name. Congruent trials are trials where the name matches the face whereas incongruent trials are trials where the name does not match the face, thus creating conflict. The difference in response time between incongruent and congruent trials is an index of the efficiency of cognitive control.

(b) In a typical context-control learning study, there are two proportion congruence conditions – one “frequently incongruent” condition where there are more incongruent trials (e.g., 75% incongruent trials) and another “rarely incongruent” condition where there are fewer incongruent trials (e.g., 25% incongruent trials). These proportion congruence conditions can be linked to various stimulus contexts, including spatial location, stimulus identity (item-based context-control) and time frames defining temporal contexts (e.g., blocks of trials).

(c) Behavioral signature of context-control learning. The bar graph illustrates the canonical behavioral finding where there is a reduced mean congruency effect (response time for incongruent minus congruent trials) for contexts that signal more incongruent trials (frequently incongruent) than for those that signal fewer incongruent trials (rarely incongruent). This type of finding suggests that participants learn to associate a more focused attentional state with the contextual cue signaling the frequently incongruent condition.

(d) Three recent fMRI studies observed blood-oxygen level dependent (BOLD) signals in the caudate nucleus of the basal ganglia that specifically tracked long-term and short-term temporal, and item-based context-control learning processes (Chiu & Egner, 2017; Jiang et al., 2015a; Jiang et al., 2015b).These findings, based on distinct experimental manipulations and analytic approaches, converge on a zone of the caudate that is thought to form part of the “associative” cortico-striatal loop, and which receives inputs from the dlPFC (Redgrave et al., 2010).

Traditionally, models and investigations of control have focused on the effects of control in terms of biased stimulus processing and response selection, while putting aside the thorny question of how control itself is regulated over time (Botvinick & Cohen, 2014; Norman & Shallice, 1980). The past two decades, however, have seen a steep increase in interest and research into this question, which has led to the realization that cognitive control, while being able to override well-learned behavior, must itself be guided by associative learning processes in order to be applied adaptively. We here refer to this basic idea as “control learning” (Egner, 2014). This type of learning is outlined in the influential conflict monitoring model of cognitive control (Botvinick et al., 2001), which proposes that the selectivity of top-down attention for task-relevant stimuli is dynamically adjusted in response to the level of performance difficulty or “conflict” (interference) encountered in an ongoing task. For instance, encountering an incongruent Stroop trial in the previous trial tends to nudge up the level of top-down control on the current trial, resulting in a smaller mean congruency effect on trials following an incongruent trial than following a congruent one, a phenomenon referred to as conflict adaptation (Egner, 2007; Gratton, Coles, & Donchin, 1992). Note that in this case of trial-by-trial, conflict-driven adjustments in control, people first running into trouble (experiencing conflict), and then adjust their control settings reactively to overcome this momentary performance difficulty. By contrast, a burgeoning recent literature has documented other cases of control learning where people exploit statistical regularities of the task environment for more extended and/or context-specific anticipation of control demands. Thus, contextual cues can be used to guide the adjustment of control settings in a manner that prevents people from running into trouble in the first place. We here refer to this form of control learning as “context-control learning”.

An early inkling of this possibility can be found in the longstanding observation that mean congruency effects are reduced in a temporal context (e.g., a block of trials) where incongruent trials are more frequent than congruent ones (e.g., Logan and Zbrodoff, 1979), known as the proportion congruent (PC) effect. However, in these studies it was not always clear whether the observed modulation of congruency effects arose from learned adjustments in control settings to different contexts or whether they were driven by stimulus-response learning (i.e., contingency learning) or priming effects (Schmidt, 2013). By contrast, a more recent crop of well-controlled studies (reviewed below) has produced a rich set of clear demonstrations of context-control learning. The present article is dedicated to first providing an overview of this fast-growing behavioral literature and some relevant computational models, and to then review and synthesize relevant neuroimaging literatures for sketching out an emerging model of the functional neuroanatomy of context-control learning that connects a dorsal striatum-centered subcortical learning machinery with the “traditional” cognitive control network of the medial and dorsolateral prefrontal cortex.

2. Context-control learning

2.1. Behavioral evidence for context-control learning

In the past few years, the basic idea of learning-guided control has witnessed a crucial conceptual expansion, as a number of behavioral studies have demonstrated that particular control settings such as a high level of attentional selectivity, or a readiness to switch tasks, can become directly associated with specific contextual cues (for recent reviews, Abrahamse et al., 2016; Braem and Egner, in press; Bugg and Crump, 2012; Egner, 2014; Verguts and Notebaert, 2008). For instance, in one of the first studies of its kind, Crump and Milliken (2009) adapted a Stroop task to probe whether participants could exploit a contextual cue (here, stimulus location) to reduce the congruency effect. Specifically, on each trial, a Stroop stimulus was shown in one of two screen locations and, unbeknownst to the participants, location was predictive of the proportion of incongruent trials (or conflict-likelihood). It was found that congruency effects for stimuli presented in the location with a higher proportion of incongruent trials were significantly reduced (Fig. 1b, c). This suggests that a spatial location can come to serve as a bottom-up predictive cue for retrieving a context-appropriate top-down control set (for additional discussion, see Crump et al., 2017; Hutcheon and Spieler, 2017). Other recent studies have demonstrated this type of modulation of the congruency effect using a wide variety of different contextual cues, including stimulus location (Brosowsky & Crump, 2016; Corballis & Gratton, 2003; Crump, 2016; Crump et al., 2017; Crump, Gong, & Milliken, 2006; Hübner & Mishra, 2016; King, Korb, & Egner, 2012; Weidler & Bugg, 2016), font (Bugg, Jacoby, & Toth, 2008; Crump, 2016), shape (Crump, Vaquero, & Milliken, 2008), color (Vietze & Wendt, 2009), social categories (Cañadas, Rodríguez-Bailón, Milliken, & Lupiáñez, 2013) and incidental sematic cues (Blais, Harris, Sinanian, & Bunge, 2015). This kind of learned contextual modulation of congruency effects, where the frequency of congruent/incongruent trials is linked to an incidental contextual feature of the task, is referred to as the context-specific proportion congruent (CSPC) effect.

A key contribution of a number of these studies is that they rule out the possibility that this type of context-control learning is mediated by biased stimulus-response contingencies, whereby the frequent pairing of particular context-stimulus-response ensembles could produce a CSPC pattern without involving learned control settings (c.f. Dishon-Berkovits and Algom, 2000; Melara and Algom, 2003; Schmidt and Besner, 2008). Specifically, contrary to the assumption of specific context-stimulus-response associations driving this effect, the context-specific performance benefits generalize to new and unbiased stimuli (Bugg, Jacoby, & Chanani, 2011; Crump & Milliken, 2009; Surrey, Dreisbach, & Fischer, 2017). For instance, in the above-mentioned study by Crump & Milliken (2009), the authors created the proportion congruency bias at each location using one subset of stimuli while another subset of stimuli was presented in congruent or incongruent form with equal likelihood at both locations. Crucially, despite the fact that these “transfer items” were unbiased, congruency effects for these items were reduced when they were presented in the location of high conflict-likelihood. Thus, people can learn to associate a specific context (e.g., stimulus location) with a high demand for top-down control, and the contextually cued control settings are then generalized to frequency-unbiased, or even novel items (e.g., King et al., 2012).

Another clear example of a learned control context generalizing to unbiased items can be found in the so-called “list-wide proportion congruent” (LWPC) effect. It has long been known that task blocks comprising of a larger proportion of incongruent trials incur smaller mean congruency effects than blocks with a smaller proportion of incongruent trials (i.e., the PC effect; Kane and Engle, 2003; Lindsay and Jacoby, 1994; Logan et al., 1984; Logan and Zbrodoff, 1979; Lowe and Mitterer, 1982), which supplies evidence for a temporal context becoming associated with heightened top-down control. Importantly, this notion has been confirmed in task designs using unbiased transfer items (Bugg & Chanani, 2011; Bugg et al., 2008). For instance, Bugg & Chanani (2011) employed a picture-word Stroop task where “lists” (blocks of trials) were designed to have a low or high proportion of incongruent trials via a subset of frequency-biased stimuli. Crucially, those frequency-biased items were inter-mixed with frequency-unbiased ones, and the congruency effect for those frequency-unbiased, transfer items was significantly smaller in the mostly-incongruent compared to the mostly-congruent list context. Thus, people can associate a temporal context (or an episode) with context-sensitive but generalizable control settings.

In addition to CSPC and LWPC effects, priming of conflict control has been demonstrated at the level of individual stimuli, called the “item-specific” proportion congruent (ISPC) effect (e.g., Bugg et al., 2011; Bugg & Hutchison, 2013). As an example, consider the task shown in Fig. 1b. In this face-name Stroop paradigm, participants are asked to identify well-known actors’ faces (e.g., pushing one button for Matt Damon’s face, another one for Leonardo DiCaprio’s face, etc.), while trying to ignore congruent or incongruent names written across the face stimuli (Chiu, Jiang, & Egner, 2017). Unbeknownst to the participants, some face identities (e.g., the face of Matt Damon) were associated with a high likelihood of being paired with incongruent names, and other face identities (e.g., the face of Leonardo DiCaprio) were associated with a low likelihood of being paired with incongruent names. The face identity thus served as a predictive cue to signal a demand for high (or low) attentional selectivity. Congruency effects were found to be reduced for the items (face identities) that were frequently incongruent as compared to items that were rarely incongruent (Fig. 1b, c) (Chiu et al., 2017), replicating a number of similar ISPC effect studies (Bugg et al., 2011; Bugg and Hutchison, 2013; Hutchison et al., 2016). Of particular note here is that the participants do not know which stimulus will occur on an upcoming trial, which means that the reduced congruency effects must stem from a rapid, bottom-up triggered retrieval of a cue-associated top-down control set at the item level.

Taken together, the studies reviewed above provide compelling evidence that specific contextual cues, be they stimulus locations, features or individual items (identities), as well as temporal episodes, can become associated with particular conflict control settings and trigger them in a cue-driven, bottom-up manner. Importantly, though, context-control learning is not limited to conflict-control in Stroop-type tasks, as equivalent effects have also been documented in tasks that probe other components of cognitive control or attention, including task-switching (e.g., Chiu and Egner, 2017; Crump and Logan, 2010; Leboe et al., 2008), response inhibition (e.g., Verbruggen and Logan, 2008), dual-tasking (e.g., Fischer et al., 2014; Surrey et al., 2017), Simon task (e.g., Hübner and Mishra, 2016) and attention capture (e.g., Crump et al., 2018). For instance, switch costs - longer response times and lower accuracy when one has to switch rather than to repeat a task set – are thought to reflect control processes of reconfiguring a task-set (Rogers & Monsell, 1995) and/or overcoming interference from a previous set (Allport, Styles, & Hsieh, 1994). Recent studies have shown that switch costs are reduced in blocks where switches are frequent compared to when they are rare, akin to the LWPC effect (e.g., Dreisbach and Haider, 2006), and when screen location provides a contextual cue of switch-likelihood, akin to the CSPC effect (Crump & Logan, 2010; Leboe et al., 2008). Moreover, in a recent test of item-specific control learning in task switching, Chiu and Egner (2017) showed that specific stimuli that predict a high probability of switching (e.g., a particular object that appears on 75% switch trials and 25% repeat trials) incur significantly smaller switch costs than stimuli associated with a neutral or low probability of switching. Importantly, this finding of bottom-up cuing of cognitive flexibility was not mediated by stimulus-task or stimulus-response associations, and was even observed in the context of switching between three different tasks, suggesting that what participants learned was a general switch-readiness rather than a readiness to switch to one specific alternate task. Finally, comparable effects can also be observed in choice data, as seen in enhanced voluntary switch rates under conditions of likely (Fröber & Dreisbach, 2017; Chiu, Frober, & Egner, under review) or rewarded switches (Braem, 2017).

2.2. Computational modeling of context-control learning

Building on an influential model of how congruency effects arise in the Stroop task (Cohen, Dunbar, & McClelland, 1990), the conflict monitoring model offered the first formal account of adaptive control within a computational neural network model (Botvinick, Braver, Barch, Carter, & Cohen, 2001). The model explains both trial-by-trial conflict adaptation effects and PC effects by positing a mechanism that detects conflict (mutually incompatible response activation) to drive commensurate adjustments in top-down control, via biasing of the task-relevant processing pathway (in the Stroop task: color processing) over the task-irrelevant one (word-reading). The model is grounded in reinforcement learning (RL), whereby the degree of adjustment to top-down control is proportional to the degree of conflict (which can be conceptualized as a “control prediction error”), weighted by a learning rate. The learning rate determines the degree to which the most recent experience of conflict (a higher learning rate) vs. a longer history of conflict (a lower learning rate) drives adjustments in control. Note that the conflict monitoring hypothesis also posits specific neural substrates that support these control learning effects, and those are discussed in Section 3.1.

A number of subsequent models have made additions or modification to this basic framework but have retained the fundamental idea of an RL type learning mechanism for driving control adjustments. For instance, Blais and colleagues (2007) extended the model from pathway-level (i.e., color processing pathway) to item-level control, where enhanced top-down attention becomes linked to specific relevant stimulus features (e.g., the color blue) rather than the (color processing) pathway. This model could successfully stimulate both the standard PC effect as well as the ISPC effect (Blais, Robidoux, Risko, & Besner, 2007). Along similar lines, Verguts & Notebaert (2008) argued that while the original conflict monitoring model accounts for when a control adjustment (or learning) is required, it does not specify where to apply control. To resolve this limitation, Verguts & Notebaert (2008, 2009) proposed a conflict-modulated Hebbian learning rule that acts “locally”, that is, at the level of specific stimulus features. Specifically, conflict enhances the (Hebbian) association between top-down attention (or task demand units) and currently activated input units, which - similar to the Blais et al. (2007) model - allows for attention to be bound to specific stimuli or stimulus features. The combination of the basic conflict-monitoring hypothesis with item-specific biasing and/or a Hebbian learning rule has thus successfully expanded the types of control-learning effects that can be formally simulated (see also Blais & Verguts, 2012).

A different type of extension to the conflict-monitoring model was proposed by Jiang and colleagues (2014), who noted that the fixed learning rate used in the original model poses limitations on the flexibility of contextual adjustments in control. These authors developed a version of the conflict-monitoring model that adjusts its learning rate flexibly from trial-to-trial, on the basis of estimating the volatility (rate of change) of control demand in the current task (Jiang, Heller, & Egner, 2014). When demands are stable (e.g., a reliably low or high incidence of conflict), the learning rate is held low (relying on a longer history of trials), whereas when demands are volatile (fast-changing conflict-likelihood), the learning rate gets pushed up (cf. Behrens, Woolrich, Walton, & Rushworth, 2007). This model has successfully simulated simultaneous short- and long-term control learning effects (Jiang, Heller, & Egner, 2014). In sum, a number of computational models have successfully simulated a variety of control-learning phenomena. The most consistent motif across these models is the shared assumption of an RL type learning process that uses conflict prediction error to adjust top-down control settings. These models lay the foundation for incorporating computational parameters in neuroimaging or electrophysiological studies to identify the neural correlates of the control learning process, an approach that the field has only recently begun to pursue. We will return to reviewing these recent studies in Section 3.2.

2.3. Interim summary

To summarize, internal control settings, just like motor actions, appear to be subject to associative learning, such that they can become linked to external contextual cues like stimulus locations, features, identities (or specific items), and temporal contexts that predict changing cognitive demands over time (Fig. 1b, c) (Abrahamse et al., 2016; Egner, 2014). A key advantage of mnemonically binding internal control states to external cues is that the latter can subsequently trigger an appropriate “top-down” setting in a “bottom-up” manner, such that we do not have to wait for performance difficulty to elicit control adjustments – we can instead preempt performance difficulty by exploiting external cues that are predictive of cognitive challenges to trigger the rapid retrieval of an appropriate control strategy. Moreover, because the retrieved control settings are generalizable, their utility is not tied to specific tasks, stimuli or actions. To wit, if a spatial context cues the retrieval of a heightened attentional focus in a Stroop task, this will result in reduced distracter interference regardless of what the exact target color or the distracter word is that occurs in this context, and independently of the exact motor response required. Context-control learning thus holds the promise of combining the speed of automatic, associative processing with the flexibility and generalizability of controlled processing. We next turn to the question of how the brain mediates context-control learning.

3. Neural mechanisms of control learning

3.1. The cortical circuitry of cognitive control

While there is a vast neuroimaging literature assessing the neural substrates of cognitive control in terms of comparing mean activation levels between high vs. low demand conditions (e.g., contrasting incongruent with congruent trials), there are comparably few studies that have measured brain activity in relation to contextual adjustments in control, and only a handful of very recent studies have begun to explicitly model the trial-by-trial neural learning processes involved in forming context-control associations. We here review studies that fall into the latter two categories in order to outline a functional neuroanatomy of context-control learning.

When it comes to delineating the neural substrates of cognitive control operations, just as in the domain of cognitive control theory, the traditional research focus has been to identify brain sources of top-down biasing signals, rather than examining the regulation of control processes. A large literature has indicated a key role for dorsolateral prefrontal cortex (dlPFC) in implementing control (Bunge, Dudukovic, Thomason, Vaidya, & Gabrieli, 2002; Corbetta, 1998; D’Esposito et al., 1995; Diamond & Goldman-Rakic, 1989; J Duncan & Owen, 2000; Fuster, 2000; Mesulam, 1986; E K Miller, 2000; Earl K. Miller, 1999; A M Owen, Evans, & Petrides, 1996; Passingham, 1995; Ridderinkhof, Ullsperger, Crone, & Nieuwenhuiss, 2004; Schumacher & D’Esposito, 2002; Wallis, Anderson, & Miller, 2001; Wise, Murray, & Gerfen, 1996). This region has generally been conceptualized as the source of keeping active – and protecting from interference – task-relevant goal or rule information that is used to guide top-down biasing of sensory and motor processing (Miller & Cohen, 2001). Accordingly, the effects of PFC damage are most apparent when cognitive control is most needed – when the task or action cannot be carried out automatically but instead requires appropriate mappings between inputs, internal states and outputs according to contextually determined rules (Chao & Knight, 1995; Dias, Robbins, & Roberts, 1996; John Duncan, 1986; Milner, 1963; Adrian M. Owen, Downes, Sahakian, Polkey, & Robbins, 1990; Shallice & Burgess, 1991). More recently, with the advent of the control-regulation perspective of the conflict monitoring theory, interest has shifted towards the dlPFC’s interaction with medial frontal regions, most notably the dorsal anterior cingulate cortex (dACC). This region is thought to play a complementary role to dlPFC, supporting moment-to-moment adjustments in control by detecting conflict in information processing and signaling the need for control adjustments to the dlPFC (Botvinick et al., 2001), thus conferring a role in control learning onto the dACC. This idea has found substantial support in fMRI studies of short-term, trial-by-trial conflict adaptation effects, where the dACC is more active on incongruent trials when they follow a congruent trial (where behavioral interference effects are large) than when they follow an incongruent trial (where behavioral interference effects are small), and the opposite activity pattern is observed in dlPFC (Botvinick, Nystrom, Fissell, Carter, & Cohen, 1999; Egner & Hirsch, 2005; Kerns et al., 2004). Moreover, in a cued Stroop paradigm, the dlPFC (but not the dACC) has been found to be more active in (cued) anticipation of color-naming than word-reading trials, whereas the dACC (but not dlPFC) was found to be more activated by the subsequent target stimuli when they were incongruent than when they were congruent (MacDonald, Cohen, Stenger, & Carter, 2000). These studies thus indicate that the dlPFC is more closely involved in the implementation of anticipatory (proactive) control, and the dACC in the monitoring of performance difficulty for recruiting reactive control adjustments. A corresponding psychological distinction between proactive versus reactive forms of control has been formalized in Braver’s dual mechanisms of control (DMC) framework (Braver, 2012; Braver, Paxton, Locke, & Barch, 2009).

The proposal that the dACC is involved in gauging the need for up-regulating control has also been supported in studies of control learning effects in the context of longer-term temporal variations in cognitive demand, that is, in fMRI studies of the PC and ISPC effect (e.g., Blais & Bunge, 2010; Carter et al., 2000; Grandjean et al., 2012; Wilk, Ezekiel, & Morton, 2012) as well as in electrophysiological studies of the CSPC and ISPC effects (e.g., Blais, Hubbard, & Mangun, 2016; Panadero, Castellanos, & Tudela, 2015; Whitehead, Brewer, & Blais, 2017).For instance, Carter and colleagues (Carter et al., 2000) employed a Stroop color-naming task and manipulated the proportion of congruent trials between blocks (20% vs. 80%). The classic behavioral PC pattern of reduced congruency effects in blocks with frequent incongruent trials was accompanied by greater dACC activity elicited by incongruent trials in the 80% congruent than in the 20% congruent condition, in line with the proposed role of this structure in detecting conflict and signaling the need for control (Carter et al., 2000). The same pattern of behavioral and dACC effects were also observed with similar task designs in other studies (Grandjean et al., 2012; Wilk et al., 2012). By contrast, these studies did not observe much evidence for a role of the dlPFC in sustaining tonically higher activation in contexts of more frequent incongruent trials. Moreover, other studies have painted a somewhat different picture of the respective contributions of dlPFC and dACC to cognitive control, observing a more phasic, “control-initiating” activity pattern in the dlPFC, and a more tonic, task-set sustaining pattern in dACC (Dosenbach et al., 2008, 2006; but see Seeley et al., 2007). It should be noted, however, that these latter studies did not create systematic time-varying changes in control demand and were thus not optimally suited to track neural substrates of context-dependent, adaptive cognitive control processes.

Another line of work that speaks to the notion of context guiding control signals is the literature on the AX version of the Continuous Performance Task, or AX-CPT (Cohen & Servan-Schreiber, 1992). In this task, each trial consists of a cue and a probe letter presented sequentially. Participants are instructed to make a target response to a probe (X) only if it is preceded by appropriate particular context cue (A) but not to other cue-probe sequences (e.g., A-Y, B-X, or B-Y). The task is typically comprised of 70% of AX trials with 10% of each of the other combinations, thus creating a prepotency for responding following an A cue, which renders performance on B-X and A-Y trials diagnostic of the mode of cognitive control (proactive versus reactive) participants are engaged in. Functional MRI studies employing the AX-CPT have prominently observed context cue-related activity in dlPFC (Braver & Bongiolatti, 2002; MacDonald & Carter, 2003; Paxton, Barch, Racine, & Braver, 2008), supporting the idea that PFC is involved in implementing context-dependent cognitive control. Note, however, that in the case of the AX-CPT, participants are instructed to rely on context in responding, which is supplied explicitly by external cues. The explicit nature of the contextual cue in the AX-CPT thus differentiates this protocol from control learning as assessed in the behavioral literature that we reviewed above (and as assessed in conflict adaptation and PC paradigms), where contextual changes in demand are inferred from experience and are often acquired implicitly (e.g., King et al., 2012).

In sum, studies of trial-by-trial adaptation, the PC effect, and the AX-CPT paradigm have collectively implicated the dACC and dlPFC in the regulation and implementation of cognitive control in relation to changing contexts. However, it can be argued that these studies have not provided particularly sensitive assessments of the (presumed) learning processes that are supposed to mediate context-control learning. In particular, while the conflict-monitoring model and the more recent control learning perspectives (Abrahamse et al., 2016; Egner, 2014) describe both conflict adaptation and PC effects as a results of a associative learning mechanisms that update predictions of control demand from trial to trial (Botvinick et al., 2001), this learning process was not explicitly modeled in any of the above neuroimaging studies. Therefore, the core learning signals driving context-control, namely the presumed trial-wise updating of predictions (based on “control prediction error”) of forthcoming control demand, were not estimated and employed in the neuroimaging analyses, leaving the neural substrates of this learning process unexplored.

3.2. Subcortical contribution to the learning of context-control associations

A small set of recent studies that adapted behavioral context-control learning protocols to fMRI and explicitly focused on modeling the associative learning process itself have now provided converging evidence for an additional key subcortical component of the neural circuitry of cognitive control, namely, the caudate nucleus of the dorsal striatum (DS) (Fig. 1d). While the DS has long been seen as crucial for gating inputs to working memory (Frank, Loughry, & O’Reilly, 2001; Hazy, Frank, & O’Reilly, 2006), these more recent studies suggest an additional role, in particular for the caudate, in linking contextual cues to appropriate top-down attentional control settings. For example, as reviewed above, Jiang and colleagues built a computational model of how people learn to adapt top-down control to temporal contexts, that is, to time-varying cognitive demands in a volatile task environment (Jiang, Heller, & Egner, 2014). Inspired by the conflict-monitoring theory (Botvinick et al., 2001), this model explicitly simulates how participants learn to estimate the volatility (rate of change) of control demand and use this estimate to inform a prediction of the likelihood of forthcoming conflict (e.g., an incongruent Stroop stimulus) on each trial. In a subsequent model-based fMRI study, trial-by-trial variations in the model’s parameters were regressed against multivariate fMRI data patterns. It was found that while volatility was represented by the anterior insula and inferior frontal gyrus, it was neural activity in the caudate that tracked the trial-by-trial predictions of control demand or conflict (Jiang et al., 2015a). Moreover, those predictions appeared to be translated into proactive control adjustments by the dlPFC and dACC, as reflected by brain-behavior correlations. This work suggests that the classic cingulo-prefrontal cognitive control network identified by the studies discussed in the previous section may be informed or guided by a subcortical learning mechanism centered on the DS.

Other recent studies corroborate this interpretation. First, in another fMRI experiment, Jiang and colleagues took a new approach for interrogating the neural substrates of trial-by-trial conflict adaptation effects (Jiang et al., 2015b). Specifically, they designed a protocol that allowed them to distinguish between brain regions that process information about whether the current trial matches the previous trial in terms of exact stimulus features, in terms of stimulus categories, or in terms of control demand (e.g., an incongruent trial following an incongruent trial in the absence of any overlap in basic stimulus features). Multivoxel pattern analyses showed that activation patterns in both the DS and in anterior hippocampus encoded information about all three types of information (basic stimulus feature, stimulus categories, and control requirements), suggesting that the DS and the hippocampus both play a role in binding together external stimulus features with internal attentional control states in memory, in the service of appropriately matching control states to external demands in subsequent trials (Jiang et al., 2015b).

A very recent fMRI study provides further evidence for the caudate being central to linking appropriate levels of control to specific external cues (Chiu et al., 2017), here by employing an item-specific proportion congruent (ISPC) manipulation based on the design of Bugg et al., (2011). Specifically, as described above, Chiu and colleagues (Chiu et al., 2017) had participants identify famous actors’ faces while trying to ignore congruent or incongruent overlaid name labels (Fig. 1a). Unbeknownst to the subjects, the faces were predictive of congruency, such that some of them were more likely to occur as part of incongruent than congruent stimuli. Participants learned these item-level congruency predictions, as evidenced by reduced congruency effects for the frequently incongruent faces (i.e., the classic ISPC effect). Chiu et al. (2017) then employed a temporal difference learning algorithm (Sutton & Barto, 1998) to capture subjects’ formation of individual stimulus-control (S-C) associations. This approach is conceptually very similar to the item- or context-specific control models of Blais, Verguts and colleagues (Blais & Verguts, 2012; Verguts & Notebaert, 2008, 2009) mentioned in Section 2.2. In essence, the learning process that is associated with each face was modeled by 𝑝_i+1 = 𝑝_i + α (𝑐_i – 𝑝_i)where 𝑝_i+1 is the predicted congruency at trial 𝑖+1, which is given by the predicted congruency in the last trial, , and an updating term based on the prediction error (𝑐_i – 𝑝_i) experienced on the previous-trial weighted by a learning rate . The trial-by-trial prediction error estimates of congruency generated by the model were then regressed against fMRI signal to identify brain regions that track the updating of S-C associations. The caudate nucleus was the only brain region whose activation followed this model parameter, suggesting that the DS is responsible for updating the associations between specific stimuli and appropriate control settings. Importantly, this S-C association signal was independent of the tracking and updating of low-level stimulus-response (S-R) associations, as the study also included an S-R learning control condition, against which the S-C learning model-based analysis was contrasted (Chiu et al., 2017).

Taken together, these recent studies, employing different tasks and examining different types of context-control learning (long-term temporal context (Jiang et al., 2015a) vs. short-term temporal context (Jiang et al., 2015b) vs. item-based context (Chiu et al., 2017)) provide convergent evidence to motivate a conceptual extension of the type of learning processes that the caudate/DS have traditionally been implicated in. On the one hand, the animal literature has considered the caudate to mediate “goal-directed” action learning by linking motor actions to their consequences (Balleine & O’Doherty, 2010; Devan, Hong, & McDonald, 2011; Dolan & Dayan, 2013; Yin, Knowlton, & Balleine, 2006; Yin, Ostlund, Knowlton, & Balleine, 2005). In contrast to habitual actions, which are controlled by stimulus-response (S-R) associations, goal-directed actions are controlled by response-outcome (R-O) associations. Importantly, goal-direction actions (as opposed to habitual actions) are sensitive to changes in the value of the consequences (e.g., positive vs. negative outcomes) associated with the action. Therefore, goal-directed actions are naturally “context-sensitive” because contexts (e.g., motivational state, availability of positive outcomes in the environment) often determine whether the consequences of an action is valuable or not (Tricomi, Balleine, & O’Doherty, 2009; Valentin, Dickinson, & O’Doherty, 2007). However, studies of goal-directed learning have typically focused on linking contextual cues with specific motor actions (e.g., “press the left button”) and subsequent outcomes. The studies we have highlighted here build on the same idea of learning optimal, context-sensitive “actions” to achieve desirable outcomes, but here those context-sensitive actions are “cognitive” rather than motoric in nature, and the outcomes are represented by efficient task performance. In particular, the cognitive action here pertains to the retrieval and application of specific control states, such as attentional selectivity or switch-readiness, which have generalizable effects, whereby the positive outcome (enhanced task performance) is not tied to a specific stimulus or motor action. In that sense, the control-learning effects we reviewed here can be seen as (more abstract) expressions of classic goal-directed learning.

On the other hand, the caudate has also been held to play a key role in gating working memory inputs and outputs to ensure that only the currently most relevant information is allowed to enter working memory and to drive action selection (Chatham, Frank, & Badre, 2014; Cools, Miyakawa, Sheridan, & D’Esposito, 2010; Frank et al., 2001; Gruber, Dayan, Gutkin, & Solla, 2006). Previous conceptualizations of the subcortical contributions to cognitive control have thus focused on the gating of information into working memory and the selection of concrete actions based on predicted outcomes. The recent investigations of context-control learning now suggest that the kinds of actions the caudate learns about should be conceived more broadly to also include high-level “cognitive actions”, as the caudate appears to play a key role in learning to link generalizable control strategies to stimuli and performance outcomes. More generally, these studies motivate the proposal that the DS provides an associative learning machinery to guide the cortical cingulo-prefrontal control network in producing context-sensitive changes in top-down biasing that exploit bottom-up cues for the rapid retrieval of appropriate control settings. Many intriguing questions in this budding research area remain unanswered, however.

4. Remaining challenges

Behavioral context-control learning has been demonstrated for distinct control operations (including task-switching, response inhibition, etc.), but neural substrates of the underlying learning process have to date been explored exclusively in the context of conflict-control (Chiu et al., 2017; Jiang et al., 2015a, 2015b). Thus, whether neural substrates of control learning vary between different control processes remains to be answered in future studies. For example, it is presently not certain whether the caudate would play the same role in context-control learning of switch-readiness as it does in conflict-control. Second, the exact kind of memory that is acquired during the context-control learning is rather equivocal. Some evidence suggests that context-control learning is implicit (Egner, 2014; Musen & Squire, 1993) and may perhaps be considered a form of procedural memory. However, item-based context-control learning effects appear to emerge very swiftly (Chiu & Egner, 2017; Jacoby, Lindsay, & Hessels, 2003) or with single experiences (Brosowsky & Crump, 2018), which is not a typical characteristic of implicit/procedural learning. It is therefore plausible that control settings can be linked to contextual cues both via cumulative associative learning and via “instance-based” episodic memory processes, and that both of these learning processes might occur at the same time (Hartley & Burgess, 2005), but this possibility has not yet been explored in empirical studies.

Third, not all contextual cues are created equal: some are more effective than others in becoming associated with control settings (Crump et al., 2006), but the precise reasons for this have not yet been determined. Moreover, how control is guided in the case of cue redundancy and/or competition is an important question that has only recently been begun to be explored (Bugg & Dey, 2018; Crump, 2016). More generally, the principles that determine to which degree stimulus processing and action selection is driven by “lower-level” stimulus-response associations or by “higher-level” stimulus-control associations are currently not known (Egner, 2014). Last but not least, context-control learning appears to be a highly important process for supporting adaptive behavior, but it is not yet known to what degree this type of learning may be impaired in or diagnostic of specific psychiatric conditions. For instance, it is well known that patients with schizophrenia have deficits in implementing context-dependent control (e.g., Barch et al., 2002), but to what degree these deficits might be attributable to impaired context-control learning is presently unknown. Context-control learning may also potentially serve as a powerful aid for treating clinical populations with deficits in goal-directed control (e.g., addiction, obsessive compulsive disorder). Although there is empirical evidence for the generalizability of control learning to untrained stimuli, the promoting and impeding factors for such generalization have only just begun to be explored (e.g., Bejjani, Zhang, & Egner, 2018; Bugg et al., 2011; Crump & Milliken, 2009; Surrey et al., 2017; Weidler & Bugg, 2016; Wühr, Duthoo, & Notebaert, 2014). Future studies should thus investigate the possibility of transferring control learning across different, associated contexts in order to facilitate possible translation of laboratory findings to clinical applications.

5. Conclusions and implications

While cognitive control research has traditionally focused on the effects of top-down biasing, recent work has highlighted both the importance of understanding how the application of control is matched to a predictable but changing environment. We have here reviewed a burgeoning area in cognitive psychology research that is beginning to delineate the determinants and boundary conditions of context-control learning, which may serve as a powerful means for combining the speed of well-practiced, associative processes with the flexibility and generalizability of cognitive control operations. A synthesis of recent neuroimaging studies that have specifically targeted the neural substrates of the associative learning processes involved in binding control states to contextual cues suggests an extended model of the functional neuroanatomy of control, where the caudate/DS plays the role of guiding the application of control by the “classical” prefrontal control network in line with learned contextual predictions. In conclusion, by putting these recent fMRI studies side by side with the larger behavioral work on context-control learning, we hope to highlight promising future avenues of research to ultimately gain a better understanding of cognitive control in the healthy and diseased brain.

Highlights.

• Specific control states can be triggered by contextual cues in a bottom-up manner.

• Context-control learning recruits the dorsal striatum.

• Context-control learning enables flexible yet fast processing.