Integrated intelligence from distributed brain activity

Abstract

How does organized cognition arise from distributed brain activity? Recent analyses of fluid intelligence suggest a core process of cognitive focus and integration, organizing the components of a cognitive operation into the required computational structure. A cortical “multiple-demand” (MD) system is closely linked to fluid intelligence, and recent imaging data define nine specific MD patches distributed across frontal, parietal and occipitotemporal cortex. Wide cortical distribution, relative functional specialization and strong connectivity suggest a basis for cognitive integration, matching electrophysiological evidence for binding of cognitive operations to their contents. Though still only in broad outline, these data suggest how distributed brain activity can build complex, organized cognition.

Organizing distributed brain activity

Organized cognition of any kind arises from widely distributed brain activity. An immediate question is how such activity is integrated, allowing just the right cognitive contents to be combined in just the right way for current purposes. Though much is certainly unknown, an outline view of the relevant psychological and physiological mechanisms is beginning to appear. In this paper, we describe recent progress towards a whole-brain understanding of cognitive integration.

We begin with recent work on the cognitive mechanisms of fluid intelligence (see Glossary). Theoretical accounts of fluid intelligence focus on processes of cognitive control [1] [2] [3] and cognitive integration [4], and based on recent findings, we suggest a synthesis of these two approaches. Results from brain imaging [5] [6] [7] and lesion [8] studies relate fluid intelligence to a well-known control network in the brain, which previously we have called the multiple-demand or MD system [9] [10]. We describe recent studies on the detailed anatomy and physiology of MD activity, and how they begin to illuminate the physiological underpinning of cognitive control and integration. In broad outline, these findings suggest how distributed brain activity builds organized cognition. We conclude with some of the many questions that this scheme raises for future work.

Fluid intelligence and attentional integration

A fundamental psychometric discovery is positive manifold – to some extent, all tests of different cognitive abilities tend to have positive correlations [11] [12], even those that on the surface are quite dissimilar. In his foundational work, Spearman [11] proposed that some general or g factor contributes to success in any task. If this model is fit to correlational data, novel problem-solving tests turn out to be excellent measures of g – reflecting the fact that, in a diverse task battery, it will be these tests that have the largest average correlations with a wide range of others. Well known examples are matrix problems (Figure 1A) [13] [14], series completions [15] etc. The ability measured in such tests has been called “fluid intelligence”. Later, we consider several possible contributors to positive manifold, but meanwhile, the broad ability of fluid intelligence tests to predict success in many kinds of activity, from laboratory tests to life achievements, suggests cognitive mechanisms of widespread importance in mental life.

Figure 1. Fluid intelligence and cognitive segmentation.

(A) Matrix problem. The task is to choose which of the response alternatives (bottom) would correctly complete the matrix (top). (B) Goal-subgoal hierarchy. (C) Modified matrix problem in typical format. The task is to decide what figure would fill the empty cell of the matrix (dotted outline), with the answer to be drawn in the response box below. (D) Same problem segmented into separate parts. (E) Proportion of problems correctly solved as a function of fluid intelligence. Blue – typical format; red – segmented format. Adapted with permission from [28].

We begin with models of fluid intelligence based on cognitive or executive control. Following early work linking fluid intelligence to frontal lobe functions [1] [16] [17], multiple aspects of control have been implicated in fluid intelligence models – maintenance in working memory in the face of distraction [2], avoiding lapses and mind-wandering [18] [3], inhibiting unwanted mental content [2] and so on. Though this broad focus is appealing, and the link to frontal lobe functions is clear, conceptions of control often seem underdeveloped. Undoubtedly, simple concepts such as maintenance in working memory [19], attentional biasing [20] [21] [22], inhibition [23] [24] and the like bear on important aspects of cognition. On their own, however, they fall far short in addressing what would be needed to generate even a simple everyday activity, such as planning and carrying out a trip to the grocery store.

A different perspective on cognitive control comes from classical work in artificial intelligence, going back to the early problem-solving programs of Newell et al. [25]. As this work made clear, most complex problems cannot be solved in a single processing step. Instead, they must be divided into simpler parts, with these simpler parts solved in turn to produce final success on the whole problem. A familiar example organized as a goal-subgoal hierarchy is shown in Figure 1B. It is not possible simply to set the goal of travelling to Japan and then move immediately to the question of how to move one’s hand; the statement of the goal brings too few constraints to lead immediately to motor commands. But beginning with the goal of travel to Japan, the problem solver can set progressive subgoals of flying, buying a ticket, and logging into the internet – and now, sitting at a laptop, it becomes possible to plan a movement [26]. This reasoning suggests that a core aspect of “control” must be cognitive segmentation – based on knowledge of the problem domain and moves available to the problem solver, a complex whole must be organized into separate, more easily soluble parts.

Early theoretical work [27] emphasized the importance of cognitive segmentation in fluid intelligence problems. In a recent study [28], we modified traditional matrix problems so that little was left beyond this segmentation requirement. As shown in Figure 1C, entries in each cell of the matrix had 3 parts (in this case, short line straight or curved, longer line center or right, arrow pointing left or right). To minimize the working memory demands of traditional problems (Figure 1A), the participant simply drew their answer into a response box, allowing each part of the answer to be drawn as soon as it was determined. The problem appears trivial once attention is focused on a single part of the figures – for example, focusing just on the arrow – but nevertheless, participants with low scores on a standard fluid intelligence test struggled to solve these simplified problems (Figure 1E, blue). To confirm that segmentation was the only significant requirement, we used problems that were pre-segmented, with each part of the figure presented in its own matrix (Figure 1D). Though component problems were exactly the same as those that would be produced by attention to a single part in the original figures, now all participants performed well (Figure 1E, red).

Sometimes, accounts of fluid intelligence based on cognitive control have been contrasted with an account based on cognitive integration [4] [29]. According to integration accounts, the key process in fluid intelligence is binding together the different components of a cognitive process or representation. In this light it is instructive to consider what is needed to “attend” to one part of the problem in Figure 1C, e.g. to determine that the arrow should point left. Evidently, information is needed on arrow direction in each cell of the matrix. This direction information must be correctly bound to positions within each figure and in the matrix as a whole. The layout of left and right arrows must be related to an overall conception of the problem to be solved, and what it means for the solution to “look right”. Doubtless, problem solving is guided by internal reward signals bound to each successful step. To create attention, multiple cognitive fragments must be integrated into precisely the correct combinations and relationships. As these computational structures are built, the task as a whole is segmented into useful parts, each representing a step closer to the overall goal.

The need for attentional integration – creating the steps of complex behavior, each consisting of components assembled into just the right computational structure - is obvious in fluid intelligence problems. We would argue, however, that attentional integration lies at the heart of all organized cognition (Figure 2), helping to explain why fluid intelligence tests predict success in such a wide variety of behavior. Segmentation into parts is minimized in very simple tasks, but even here, just the right cognitive elements, organized in just the right way, must be assembled for behavioral control. Rather than conflicting, we suggest that accounts of fluid intelligence based on control and integration reflect two views of the same process. Focused attention, resistance to distraction, and integration are all important aspects of brain activity that defines and assembles the contents of an elementary cognitive operation. Our approach thus places popular accounts of fluid intelligence within a broader view of how “cognitive control” in general should be conceived.

Figure 2. Integration and segmentation in complex cognition.

(A) Attention to one part of a complex task integrates multiple component fragments (right). As a series of steps is created, the problem is progressively segmented into simpler sub-problems (left). This example is the “travel to Japan” problem from Figure 1B. (B) To create each step, fragments must be selected from many potential candidates and assembled into precisely the correct computational structure.

Integrating distributed brain activity

Of course, the material to be integrated in creating a cognitive structure will be represented in widely distributed brain activity. Even attention to a visual object requires integration of activity across multiple cortical and subcortical brain areas, representing that object’s different properties and action affordances [30] [31]. For a complex cognitive structure such as a single problem-solving step, relevant activity may be distributed across much or perhaps most of the brain. This calls for an integration mechanism with widespread, flexible access to whatever neural activity is needed in current cognition.

Along with regions of brain activity linked to specific cognitive domains, brain imaging studies often show activity in a widely-distributed “multiple-demand” or MD system – a set of brain regions whose activity increases with almost any kind of cognitive load ([32] [9] [10] [33]; for an exception see [34] [35]). MD activity is generally seen in several regions on the lateral frontal surface, in and around the anterior insula, in dorsomedial frontal cortex including pre-supplementary motor area and dorsal anterior cingulate, within and to either side of the intraparietal sulcus, and often also in a region at the occipitotemporal border. From the early days of brain imaging, MD activity has been linked to cognitive control [36] [37] and integration [37] [38]. A core role in fluid intelligence [9] [39] [40] is indicated by strong MD activity during work on fluid intelligence problems [5] [6] [7], and losses of fluid intelligence associated with MD damage [8] [41] [42] [43]. In this section we link new findings on MD anatomy and physiology to the computational requirements of attentional integration.

Though MD-like activity has been known for many years, its precise anatomy has remained uncertain. To address this limitation, we turned recently to the methods and data of the Human Connectome Project (HCP) [44]. In the HCP approach, surface-based processing using multimodal MRI features improves brain coregistration, much sharpening delineation of cortical areas. Multimodal data are further used to parcellate the cerebral cortex of each participant into 180 distinct regions per hemisphere [45]. In a sample of 449 HCP participants, we used the conjunction of 3 cognitive contrasts – high versus low working memory load, relational reasoning versus perceptual matching, and arithmetic versus story comprehension - to examine MD regions and their properties.

The results clarify the anatomy of known MD regions, as well as suggesting new regions. Across the cortex we see 9 tightly-defined patches of MD activity, distributed across frontal, parietal and temporo-occipital cortex (Figure 3A, B). The HCP parcellation further divides MD patches into 27 individual regions, which we separate into a 10-region core with strongest activity and a surrounding penumbra (Figure 3C). Five MD patches lie on the lateral frontal surface, the most dorsal lying just anterior to the frontal eye field (Figure 3A, i6-8/6a), the most ventral incorporating anterior insula and adjacent regions of the frontal operculum (Figure 3A, AVI), and between these, a chain of three extending from the inferior frontal junction posteriorly into the rostrolateral frontal cortex (Figure 3A, IFJp/6r; p9-46v; a9-46v). An additional frontal patch lies on the dorsomedial frontal surface (Figure 3A, SCEF/8BM). In the lateral parietal lobe, the primary MD patch is centered in the depths of the intraparietal sulcus (Figure 3A, IP1/IP2), with an additional patch in dorsomedial parietal cortex (Figure 3A, POS2/7Pm). Confirming previous indications – and despite the fact that one of our three contrasts used auditory rather than visual materials – we find additional MD activity in a region at the junction of anterior occipital and posterior temporal cortex (Figure 3A, TE1p).

Figure 3. Anatomy and physiology of the MD system.

(A) Patches of cortical MD activity defined in data of 449 participants from the Human Connectome Project, using a conjunction of fMRI contrasts for working memory, reasoning and arithmetic. Left hemisphere data are shown; largely similar patches are also seen on the right. Regional parcellation (black outlines) and selected anatomical labels are taken from [45]. (B) The same data shown on a flat map of the left hemisphere. Numbering shows 9 MD patches distributed across lateral frontal (1-5), dorsomedial frontal (6), lateral (7) and medial (8) parietal, and temporo-occipital (9) cortex. (C) Individual MD regions using the HCP parcellation. Data are averaged from left and right hemispheres, and for illustration projected onto the left. The extended MD system (27 regions) is divided into core (10 regions, yellow), with activity above the mean of all 27 regions in at least 2/3 contrasts, and penumbra (remaining 17 regions, red). Adapted with permission from [44].

With this fine-scale definition of MD regions, we can ask also about functional differentiation. Despite very frequent coactivation of MD regions, there have also been many suggestions of functional differentiations (e.g. [46] [47] [48]), though with no clear consensus emerging. Across the full set of 27 MD regions, Figure 4A shows profiles of activity for our 3 task contrasts. The results illustrate both sides of coactivation-differentiation picture. On the one hand is a strong pattern of coactivation, with strongest activity for arithmetic, and weakest for relational reasoning, across almost all MD regions (Figure 4A, left). On the other is differentiation, with exact activity profiles differing between contrasts, and with this large group of participants, even small differences are highly reliable (Figure 4A, right). Though the MD system is commonly recruited as a whole, the exact pattern of this recruitment differs from task to task (see also [49]).

Figure 4. Functional profiles and connectivity of MD regions.

(A) Profiles of activation across the extended MD system for each task contrast. To show reliability, right panels show overlaid plots for 2 independent groups of 210 participants each. (B) Resting state connectivity (correlation of time series), calculated for every pair of cortical regions and then averaged for connections of each type. Left – left hemisphere; right – right hemisphere. Adapted with permission from [44].

HCP resting state data also allowed us to examine patterns of functional connectivity. Commonly, resting state studies define a “frontoparietal control network” (FPN), substantially overlapping with the MD system that is defined by activation data [50]. Based on HCP resting state data, Figure 4B shows average connectivity for all possible types of connection between core, penumbra and non-MD regions. Strikingly, core-core connections were strongest, followed by connections of core to penumbra. Comparison with a canonical FPN, previously extracted by Ji et al. [51] from the same HCP data, showed that all regions in our MD core lie within the FPN, and within that network, core-core connections are especially strong. Penumbra regions, in contrast, are spread across several networks.

Recent analyses of brain connectivity have emphasized its small world structure [52], with strong connectivity within local modules, and long-range connector hubs linking between modules. Graph theory and other approaches suggest that MD regions are among the brain’s widely-connected hubs [53] [54], with dynamic, task-dependent connections to other brain networks [55] [56]. Several previous accounts link fluid intelligence to this widespread MD connectivity [57], including a recent “network neuroscience” proposal [58]. In line with such ideas, our results suggest that the MD system is well placed for attentional integration (Figure 5A). Because local connections dominate in the brain, a system needing access to many modules needs widely distributed parts. Locally, we suggest, different kinds of information are preferentially fed into different MD regions, accounting for partial functional differentiations. Similarly, local connections allow MD outputs to influence processing in many modules. Creating integrated cognitive structures, however, requires that all types of information can be creatively selected and combined, in precisely the roles and relationships required by a particular cognitive demand. Strong connectivity between MD regions – in particular between regions of the core – suggests a medium for information exchange and integration, and an explanation for the strong element of co-activation seen across many different task demands.

Figure 5. The MD system and cognitive integration.

(A) Depending on its pattern of connectivity, each MD region (colored circles) has direct access to different information and brain operations, here illustrated with just a single link (black bidirectional arrows) for each region. Strong connectivity between MD regions allows assembly of these fragments into the required computational structure. The scheme suggests a basis for partial functional differentiations within a broad context of coactivation. (B) With their varying external connectivity, MD regions may show quantitative differences in neural coding for different task features. Against this background, however, communication between MD regions provides a strong basis for mixed selectivity.

A final aspect of the HCP findings is worth emphasis. Though our primary analyses focused on cerebral cortex, MD activity was also seen in associated subcortical structures, including parts of the caudate and thalamus, and in specific regions of the cerebellum. Though details are unknown, it is likely that attentional integration is achieved through interacting cortical and subcortical activity.

Task representation: Contents, roles and relations

If MD regions are to create cognitive episodes, the contents of these episodes must be represented in the firing of MD neurons. In human brain imaging, multivoxel pattern analysis confirms widespread MD encoding of different aspects of a current task, including discrimination of relevant stimuli, rules, responses and more [59] [60] [61]. In contrast to more dedicated regions, such as visual or auditory cortex, MD representation of relevant task contents is often weaker [62] but also broader [63]. Matching a role in selective attention, MD representations favor what is relevant to a current task [64] [65] [66]. Matching univariate findings, MD representations may become stronger as task difficulty is increased [61], likely related to the experience of more careful attention.

Corresponding findings come from electrophysiology in the behaving monkey. While exact homologies between human and monkey are uncertain, imaging data suggest a monkey MD network somewhat resembling that of the human brain [67] [68], including regions of lateral frontal, dorsomedial frontal and inferior parietal cortex. In behaving animals, neurons in these regions show properties strongly reminiscent of human findings, with firing rates of many neurons discriminating the important events of a current task, and adjusting their properties to favor currently relevant over irrelevant information [69] [70] [71] [72] [73] [74]. Again, these data suggest a distributed network with widespread access to task-relevant information, and bidirectional communication between network nodes [75] [76] [77] [78].

Most important for current purposes, the properties of MD neurons suggest a central role in integration. As we have discussed, formation of a cognitive episode requires combining relevant contents into exactly the required roles and relations. The rules of a task may determine how stimuli and responses should be paired, the required order of several responses, how alternatives are linked to available rewards etc. Similarly, in a goal-subgoal tree like that of Figure 1B, subgoals must be bound to goals such that, if the subgoal fails, an alternative route to the higher goal can be sought. Linking contents to roles is the classic computational problem of variable binding [79], and for neural representation, suggests a requirement for conjunctive coding of contents and roles. In recent years, conjunctive coding in frontal and other regions has come to prominence under the heading of “nonlinear mixed selectivity” – neural activity driven by nonlinear combinations of multiple task features [80] [81].

The literature contains many remarkable examples of mixed selectivity. If an animal must remember a sequence of objects, for example, many frontal neurons may encode object identity; a neuron’s object preference at one serial position, however, can be quite independent of its preference at another [82] [83]. If a cue indicates that the animal should search for a particular target stimulus, activity in the delay between cue and choice display may encode the upcoming target, but this encoding can be quite independent of activity when the target finally appears [84]. In a study of frontal activity during problem-solving [85], animals solved an on-screen maze, planning movements of a cursor along open maze paths from a central start position to a peripheral goal. Once the path had been planned but before movement began, individual frontal cells could be selective for the direction of just the first, just the second or just the third movement in the plan.

In line with the requirements of cognitive integration, information exchange between MD regions is well placed to create mixed selectivity (Figure 5B). Exchange between MD regions allows conjunctive coding, binding the components of a cognitive operation into the correct computational structure.

An example from our own recent work [86] illustrates binding of cognitive operations to their target objects. In this task (Figure 6A), displays of 4 objects were shown on a touchscreen, and on each trial, the monkey selected a single object by touching it. For each new problem, one or two objects were rewarded targets; the monkey had to find these targets by trial and error, and then, once targets were found, could re-select them on subsequent trials. We analysed data from 2 trial periods – the choice phase (CH), during which monkeys were shown the object display and awaited a go signal indicating that they could make their choice; and the feedback phase (FB), when following the touch, a cue indicated whether the selected object was target or nontarget.

Figure 6. Neurophysiology of binding object to role.

(A) Object selection task, two-target version. On each trial, the monkey touches a single object in a visual display. For each new problem, in the first set of trials (cycle 1), the monkey selects one object after another, learning which 2 objects (targets) are associated with reward. Targets are indicated here by green circles (not present on actual display). In subsequent trials (cycles 2-4), the animal can reselect the same targets for further rewards. After 4 cycles, targets are redefined for the next problem. In another task version, problems have only a single target. Recording areas in each animal are shown at upper right. (B) Correlation of population firing patterns for feedback (FB) and choice (CH) periods, separated by target object and cycle. Data from correct trials only. For CH, data are shown only for cycles 2-4, as correct choices were not known in cycle 1. Frontal lobe data are separated into dorsal and ventral regions (separated at fundus of principal sulcus). Small negative correlations between FB and CH patterns are an artefact of data normalization [74]. (C) Independence of object preference at CH and FB. For each recording area, left panel shows data for neurons identified as object-selective during FB. For each neuron, “best” (highest firing rate) and “worst” (lowest) were identified based on FB data. Plots show activity for these “best” and ”worst” objects while selecting them at CH (normalized for each neuron, averaged across neurons). Across the population, “best” and “worst” objects defined at FB did not give significantly different responses at CH. Right panels show reverse analysis, defining “best” and “worst” at CH and plotting data from FB. Adapted with permission from [86].

Recordings were made in two MD-like regions, lateral frontal and inferior parietal cortex (Figure 6A, upper right; recordings in superior parietal cortex not considered here). For each region, we used two kinds of analysis to compare activity at FB and CH. First, we considered patterns of activity across the whole recorded cell population (Figure 6B). Within one phase, FB or CH, population activity patterns were strongly correlated for different objects and cycles. Between phases, however, correlations were close to zero. For each neuron, in other words, firing rate in one task phase was approximately independent of firing rate in the other. Our second analysis showed that this same independence extended to object selectivity. At both CH and FB phases, there were frontal and parietal neurons that encoded the identity of the selected object. However, a neuron’s object preference during one task phase, CH or FB, was unpredictive of its preference at the other phase (Figure 6C). As MD regions direct the cognitive operations of successive task stages (Figure 5B), orthogonal representations may minimize confusion, allowing the correct cognitive operations to be executed at each stage [87]. Meanwhile, mixed selectivity for conjunctions of object and phase binds object information to these different operations.

Positive manifold

Earlier we noted positive manifold – the finding of ubiquitous positive correlations between different cognitive tests – and Spearman’s original proposal that some general or g factor contributes to success in any cognitive activity [11] [12]. One simple interpretation is that g reflects the attentional integration functions of the MD system, and in agreement with Spearman’s hypothesis, we suggest that MD functions contribute very broadly to effective cognition. At the same time, it seems likely that the full explanation for positive manifold is more nuanced, with multiple contributory factors [39] [58].

For any task, creating the appropriate control structure is not simply a matter of MD function. Undoubtedly, MD activity combines with activity in multiple, more specialized systems involved in operations of this particular task, and it seems likely that the quality of the resulting computational structure will depend on all collaborators and their interaction. In many studies, fluid intelligence has been found to correlate with different aspects of brain structure, broadly distributed across many brain regions [88] [89]. In a recent study, for example, intelligence correlated with the size and complexity of dendritic trees in tissue taken from the temporal lobe of patients undergoing epilepsy surgery [90]. Plausibly, many neural properties will be correlated across regions of the cortex, and large dendritic trees in the temporal lobe may be predictive of large dendritic trees in MD regions and elsewhere. In this case, a broadly-distributed property of neural function may facilitate the general process of creating cognitive operations, whatever their particular content. Intelligence may also correlate with whole-brain functional properties, such as stability of functional networks [91] [92]. Very likely, there is much overlap between the core cognitive mechanisms required in fluid intelligence tests, the functions of the MD system, and sources of individual differences leading to positive manifold, but these three are not likely to be identical.

The role of long-term knowledge also bears on positive manifold, and a distinction that is often drawn between fluid and crystallized intelligence. While fluid intelligence concerns current problem-solving, crystallized intelligence reflects the accumulated body of a lifetime’s learning [93] [94]. The two are generally correlated, as expected if more useful knowledge is acquired during better constructed, more focused learning episodes – and if a lifetime of learning leads to a large body of knowledge, with parts of this knowledge applicable to many new problems, then “crystallized intelligence” will also contribute broadly to success in new activities and thus to positive manifold. A related perspective is provided by the idea of mutualism in development – that growth in one ability or domain may have positive influences on growth in others [95] [96]. Often, it seems, this must be true – learning to represent relationships as graphs, for example, must surely encourage effective mental representation and hence learning in many future contexts. If positive manifold reflects the broad ability to construct good cognitive structures, it is likely influenced by many aspects of lifetime experience.

Concluding remarks and future directions

Many issues are raised by the integration account. Here we discuss two – the interface of short-term cognitive activity and long-term knowledge, and the nature of attentional capacity limitations.

As implied by our discussion of positive manifold, a core question is interface between on-line cognition and long-term knowledge. As in classical symbolic artificial intelligence (e.g. [97]), a complex problem is divided into simple parts on the basis of long-term knowledge of the structure of the world and relations within it. It is knowledge that tells us how travel to Japan can be divided into component steps, how a useful move can be made in proving a mathematical theorem, or where we should look in seeking a solution to a spatial puzzle. In the brain, knowledge that might shape current cognition is distributed across multiple brain systems. Semantic memory, for example, may be based around a proposed hub in the temporal pole [98], while episodic memory, spatial knowledge and social knowledge are linked to distinct components of the default mode network [99]. To understand MD activity in constructing solutions to cognitive problems, we need to know how multiple aspects of knowledge feed into this process. Again this is reminiscent of the widespread connectivity of MD regions (Figure 5), and our finding that multiple networks have representatives in the MD penumbra.

In classical artificial intelligence, problem solutions were often built up in an unlimited working memory, keeping track of a progressively more complex structure of goals and subgoals. For biological cognition this is not plausible; for goals such as travel to Japan or solving a scientific problem, only a small fraction can be represented in active neural firing at any one time, with the rest of the structure in long-term memory, ready for retrieval when required. At the same time, the current active focus of attention must remain bound to the long-term structure, so that, for example, a failure to progress to a goal by one route can trigger search for an alternative. The issue is reminiscent of recent biological accounts of working memory, combining active neural firing with storage through short-term synaptic change [100] [101]. It is presently unknown how the focus of attention in active cognition can be situated within a complex, long-term representation of the larger-scale problem.

A further open issue concerns the well-known capacity limitations of “attention”, reflected in difficulty carrying out several tasks at once [102] [103]. Shared demands on MD activity could provide an obvious basis for such limits, and indeed, various authors have linked capacity limitations to the functions of frontal and parietal cortex [16] [22] [104] [105]. Such proposals find support in neurophysiological studies, showing that, in frontal and parietal cortex, there is interference between representations of different visual stimuli [106], working memory items [107] or task components [86] [108]. Further work is needed, however, to understand the physiological basis of this interference. In the visual system, capacity limits in representing multiple stimuli are thought to arise through a process of competition or divisive normalization [109] [110] [111]. In such models, each stimulus attempts to drive the activity of a neuron to a particular value, appropriate to representing the properties of this stimulus; with multiple stimuli in the field, opposing forces bring activity to a compromise value, reducing the fidelity of representation for any one. Similar patterns can be seen in the visual responses of prefrontal neurons [112] [113], raising the possibility that divisive normalization is a general principle in MD cortex. Recurrent neural networks have become popular as models of working memory and cognitive control (e.g. [114]), and in a recent model, divisive normalization is the basis for limited working memory capacity [115]. Further experimental work is needed to test whether divisive normalization models may be extended to the broader attentional limits of MD activity and cognitive control.

Of course, our account of cognitive integration leaves much unknown. That said, like an early map of the globe, it provides an outline sketch of how distributed brain activity can assemble complex cognition. This sketch, we suggest, provides the skeleton we need to guide future, more detailed physiological study (see Outstanding Questions).

Glossary

Fluid intelligence

The ability measured in psychometric tests of novel problem-solving, including matrices, series completions etc. Though many tests use simple shapes and figures, fluid intelligence tests can also involve verbal or numerical materials.

Multiple-demand (MD) system

A distributed set of cortical regions showing widespread increase of activation associated with many different cognitive demands.

Nonlinear mixed selectivity

A neural response pattern in which firing rate is driven by a conjunction of task variables, e.g. a particular object presented only at a particular point in a memory list.