Biological constraints on neural network models of cognitive function

Abstract

Neural network models are potential tools for improving our understanding of complex brain functions. To address this goal, these models need to be neurobiologically realistic. However, although neural networks have advanced dramatically in recent years and even achieve human-like performance on complex perceptual and cognitive tasks, their similarity to aspects of brain anatomy and physiology is imperfect. Here, we discuss different types of neural models, including localist, auto-associative and hetero-associative, deep and whole-brain networks, and identify aspects under which their biological plausibility can be improved. These aspects range from the choice of model neurons and of mechanisms of synaptic plasticity and learning, to implementation of inhibition and control, along with neuroanatomical properties including area structure and local and long-range connectivity. We highlight recent advances in developing biologically grounded cognitive theories and in mechanistically explaining, based on these brain-constrained neural models, hitherto unaddressed issues regarding the nature, localization and ontogenetic and phylogenetic development of higher brain functions. In closing, we point to possible future clinical applications of brain-constrained modelling.

Introduction

Cognition calls for mechanistic explanation. The superb and specific cognitive capacities of humans and higher mammals may depend on specific structural and functional features of their brains. If so, these neurobiological features must play a decisive role in explanations of cognitive capacities. Despite substantial progress in understanding brain function in general, explaining how structural and functional features of neural tissue bring about cognition, language and thought has remained a challenge. We propose that such explanation can only be achieved if neural networks for modelling cognition incorporate a broad range of features that make them similar at different levels to real neurobiological networks. This article will discuss recent attempts to progress towards this goal and highlight seven brain constraints that may help to make neural networks more neural (Fig. 1). We discuss established types of cognitive neural models (Fig. 2) and give examples of neural simulations that integrate several brain constraints in novel ways. These networks offer perspectives on modelling neurocognitive mechanisms in and across multiple brain areas using huge numbers of realistic neurons and their local and global interactions through short- and long-range neuroanatomical connections, and hence provide opportunities for better understanding the mechanisms of cognition, including perception, categorization, attention, memory, language and semantic-conceptual processing.

Fig. 1. Seven constraints for making neural networks models more biologically plausible.

Constraints address the integration of modelling from the microscopic to the macroscopic scale of brain structures (1), and specifically highlight the nature of the neuron model (2), the implementation of synaptic plasticity and learning (3), regulation and control by way of interplay between excitatory and inhibitory neurons on the local and global level (4), gross anatomical structure and area subdivision (5) and local within-area structure and connectivity (6), as well as global between-area connectivity (7). Most current network models used for modelling cognition focus on only one or a few of these aspects, whereas brain-constrained modelling works towards networks integrating all of them. The diagram in the middle is redrawn after ref. ¹⁰⁹ . The brain displays on the right were provided by R.T. by using WU-Minn open-source data from the Human Connectome Project (HCP) database and the MRtrix software (https://www.mrtrix.org). HCP Data acquisition and preprocessing are described in ¹⁵⁵ . The parcellation into areas of the Schaefer Atlas219 was used for the analysis.

Fig. 2. Networks for modelling cognitive functions.

a | A localist network model includes nodes representing cognitive entities, for example word forms (middle layer), phonemes (bottom layer) and semantic features (top layer). Lines indicate links between nodes. Nodes sum up their inputs linearly, such that the activation of the phoneme nodes of /d/, /o/ and /g/ activate the word node for ‘dog’, which, in turn, activates semantic feature units (filled circles at the top) characterizing the related concept. b | Auto-associative networks include connections between their neurons, such that reverberating activity is possible; they are inspired by the local connectivity between adjacent cortical pyramidal cells ^20,22 . This panel shows the connectivity matrix between five artificial neurons, α to ε. These neurons make up an auto-associative network that includes two discrete representations indicated in magenta (neurons α-to-γ) and cyan (neurons γ-to-ε). Numbers specify the presence (1) or absence (0) of a connection from the neuron listed on the left of the matrix to the neuron indicated at the top. Each neuron becomes active if and only if it receives at least two simultaneous inputs, thus resulting in the discrete representations maintaining activity over time. c | In heteroassociative networks, neuron populations ordered in ‘layers’ project onto each other serially, resembling connectivity in some neural structures. The typical three-layer networks used in many parallel-distributed processing (PDP) models include input and output layers plus a ‘hidden’ layer in-between. d | Deep neural networks include several hidden layers. The number of neurons per layer can vary substantially. Representations are activation vectors across all neurons of a layer. e | Whole-brain models implement global between-area connectivity. Here, between-area connectivity of the right hemisphere is shown for a single subject (top left panel) and a group of subjects both in anatomical topology (bottom left) and in matrix form (right). The matrix gives all connections between pairs of areas of the network model; colors indicate connection weights (fiber densities). Part a is adapted with permission from ref. ¹¹ . Part e is adapted with permission from ref. ⁹¹ .

Neural models of cognition

Localist networks

A standard way to present theories about cognition has been in the form of abstract box-and-arrow diagrams, with each ‘box’ indicating a module specialized for one cognitive sub-process (for example, sound processing) and arrows indicating information transmission between modules. These models were usually based on behavioural experiments involving healthy participants — for example, in tasks processing differences between speech and other sounds, which motivated the postulate of different modules for speech and nonlinguistic acoustic stimuli ^1,2 . Although some of these models were refined in light of studies in individuals with neurological impairments ^3,4 , they were generally formulated without explicit reference to neuronal circuits, despite researchers’ demands for biological foundations of models of mental processing ^5–7 .

An important step towards addressing the neural substrate was taken by so-called localist models of cognition and language ^8–12 , which filled the boxes of modular models with single artificial ‘neurons’ thought to locally represent cognitive elements ¹³ such as perceptual features and percepts, phonemes, word forms, meaning features, concepts and so on (Fig. 2a). The 1:1 relationship between the artificial neuron-like computational–algorithmic implementations and the entities postulated by cognitive theories made it easy to connect the two types of models. However, the notion that individual neurons each carry major cognitive functions is controversial today and difficult to reconcile with evidence from neuroscience research ^14,15 . This is not to dispute the great specificity of some neurons’ responses ¹⁶ , but rather to highlight the now dominant view that even these very specific cells “do not act in isolation but are part of cell assemblies representing familiar concepts”, objects or other entities ^17,18 . A further limitation of the localist models was that they did not systematically address the mechanisms underlying the formation of new representations and their connections.

Auto-associative networks

Neuroanatomical observations suggest that the cortex is characterized by ample intrinsic and recurrent connectivity between its neurons and, therefore, it can be seen as an associative memory ^19,20 . This position inspired a family of artificial neural networks, called ‘auto-associative networks’ or ‘attractor networks’ ^21–32 .

Auto-associative network models implement neurons with connections between their neuron members, so that each neuron interlinks with several or even all of the other neurons included in the set. This contrasts with the hetero-associative networks discussed below, where connections run between sub-populations of network neurons without any connections within each neuron pool. To simulate the effect of learning in auto-associative networks, so-called learning rules are included that change the connection weights between neurons as a consequence of their prior activity. For example, biologically founded unsupervised Hebbian learning, which strengthens connections between co-activated neurons ⁵ , is frequently applied and leads to the formation of strongly connected cell assemblies within a weakly connected auto-associative neuron pool (Fig. 2b). These cell assemblies can function as distributed network correlates or representations of perceptual, cognitive or ‘mixed’ context-dependent perceptual–cognitive states ^6,30,32–34 . Therefore, the observations that cortical neurons work together in groups and that representations are distributed across such groups ^14,18 can both be accommodated by this artificial network type, along with learning mechanisms, thus overcoming major shortcomings of localist networks.

Additional cognitively relevant features of auto-associative networks include the ability of a cell assembly to fully activate after only partial stimulation — a possible mechanism for Gestalt completion; that is, the recognition of an object (such as a cat) given only partial input (tail and paws). The mechanism is illustrated in Fig. 2b, where stimulation of neurons α and β is sufficient for activating the cell assembly formed by neurons α-to-γ.

Furthermore, auto-associative networks integrate the established observations that: cortical neural codes can be sparse (that is, only a small fraction of available neurons respond to a given (complex) stimulus) ^{15,18,22,35,36} ; and that some (other) neurons respond to elementary and frequently occurring features of several stimuli (thus behaving in a less-sparse manner) ³⁷ . The reason for this lies in cell assembly overlap; that is, the possibility that two or more such circuits can share neurons while remaining functionally separate. This is illustrated in Fig. 2b, by the ‘overlap neuron’ of cell assemblies α-to-γ and γ-to-ε.

Auto-associative networks can model a wide spectrum of cognitive processes, ranging from object, word and concept recognition to navigation, syntax processing, memory, planning and decision making ^{21,22,27,28,30,32,36,38–40} . Some models use several interlinked auto-associative network components to model the interaction between multiple cortical areas in cognitive processing ^36,41–50 . Furthermore, auto-associative components can be included in more complex networks, for example the deep recurrent networks discussed below.

Hetero-associative networks

Hetero-associative or feedforward networks include two or more neuron pools or ‘layers’ with connections between, but not (as in auto-associative networks) within, layers ^51,52 , and offer a different pathway towards distributed representations and learning. These models consist of different neuron pools connected in sequence, a connection scheme inspired by neurobiological structures; for example, by next-neighbour feedforward connections between layers of the retina or between cortical areas ⁵³ .

A frequently used class of models called multiple-layer perceptrons, which are frequently used in parallel-distributed processing, includes three neuron layers, with one layer receiving the input, one generating the output and one ‘hidden’ layer in between (Fig. 2c) ⁵² . In such networks, representations of cognitive entities are dense and nonsparse; that is, they are distributed across all neurons of the hidden layer, such that an activation vector across the entire layer is the network correlate of an object, word, meaning or thought ⁵⁴ . Note that the dense and ‘fully distributed’ nature of these representations contrasts with the sparse (but also distributed) representations characterizing many auto-associative networks.

To implement a form of learning in these networks, input and output ‘teacher’ signals are fed to the input and output layers, respectively ⁵² , and synaptic weights are modulated to make the network learn the relationships between inputs and teacher outputs. To adjust the synaptic weights, supervised learning algorithms are applied that adjust weights according to the error gradient that is calculated backwards from the output layer to the synapse in question. Initially, an algorithm called ‘error back-propagation’ ⁵⁵ was applied; however, more recently, a range of learning rules based on error gradients across the network have become available, thus allowing for different variants of ‘gradient-dependent’ learning ⁵⁶ .

In order to implement memory processes, one additional layer was added to the three-layer architecture and connected reciprocally to the ‘hidden’ layer ⁵⁷ ; this ‘simple recurrent network’ architecture allows for reverberant activity. Similar to auto-associative networks, feedforward networks including three or four layers have successfully addressed a broad range of cognitive functions ^58–60 .

Deep neural networks

To further enhance their computational power, more layers were added to hetero-associative networks, thus resulting in deep neural networks (DNNs; Fig. 2d) ^56,61–64 . A neurobiological motivation for increasing the number of layers from three to six and more comes from the neuroanatomical structure of the ventral occipitotemporal stream for object processing, in which neuroanatomical connections lead from the primary visual cortex (V1) to adjacent areas and, via multiple levels, to anterior temporal areas ⁶⁵ .

DNNs have been varied in several important ways. For example, convolutional DNNs include topographic projections between (some of) their layers to facilitate the joint processing of adjacent inputs, and recurrent DNNs include reciprocal connections between layers and/or recurrent links within layers that make it possible to maintain information over time. These modifications have produced extremely powerful devices that have reached near-human performance levels in several cognitive domains, including, for example, object classification ^66,67 and speech recognition ^68–70 . DNNs have been proposed to perform an overparameterized blind process of directly fitting complex data sets with immanent regularization, and to allow for generalization based on local interpolation, thus taking on functions previously attributed to discrete rules ⁷¹ .

Despite their broad success, specific limitations of DNNs have recently been pointed out. They sometimes show inappropriate generalization behaviour ^72–75 , such as classifying, with high confidence, images completely unrecognizable to humans as instances of specific objects ⁷³ . Similarly, an extensive body of research on so-called ‘adversarial examples’ has shown that minimal perturbations to images can, despite being imperceptible to humans, nonetheless cause gross misclassifications by DNNs ^72,76 . It remains to be investigated whether these partial failures point to relevant differences between DNN-immanent and brain-internal perception mechanisms that can be remediated by algorithmic improvements, or rather reflect limitations of the stimulus sets applied during training, which differ from the realistic stimuli available to living beings during their ontogeny. A recent study found that introducing a hidden layer that better matches primate V1 improved robustness to adversarial examples in a neural network for image classification, suggesting that a higher degree of biological realism might be key ⁷⁷ .

Today, there are various neural network architectures that use different types of processing components, coding and learning rules. Supervised and unsupervised learning, sparse and dense distributed coding, and hetero-associative and auto-associative network components co-exist across approaches, and some models even mix these features — for example, by integrating both hetero-associative and auto-associative layers, as in the case of deep-recurrent and reservoir-computing networks ^48,69,78,79 . The choice of network features is driven by processing efficiency, including error minimization, learning speed, effective use of computing resources and so on. Neurobiological plausibility is frequently used as a source of inspiration, as mentioned, but artificial neural networks are usually not designed to structurally and functionally resemble specific parts of the brain.

Whole-brain networks

The connections between the layers of neural networks are typically simple, with neighbouring areas being linked; however, this differs from the connectivity structure of the cortex, in which numerous areas are connected in intricate ways. Approximating the complex subdivision of brains into areas and nuclei, and implementing the neuroanatomical connectivity between these components, is a main goal of ‘whole-brain’ modelling. Rather than modelling all parts of the brain, most ‘whole-brain’ models focus on cortical areas and forebrain nuclei, or selections thereof.

Neuroanatomists have mapped the areal structure of the cortex and brain at coarse and finegrained levels ^80–82 . Information about structural brain connectivity is available from invasive tracer studies in animals ^83,84 , which have revealed fibre tracts and their directionality, and from non-invasive brain-imaging applicable in both animals and humans. The latter use diffusion tensor or diffusion weighted imaging (DTI/DWI) along with deterministic or probabilistic tractography ^85–88 . By parcellating the cortex into a set of areas and translating its structural connectivity into between-area links, a whole-brain or multiple-area network model can be obtained in which each cortical area and anatomical connection has its corresponding network node and link, respectively (Fig. 2e).

Whole-brain models can be applied to investigate the functional consequences of neuroanatomical structure and connectivity. Multiple studies have simulated the spatial and temporal dynamics of brain activity that spontaneously emerges in resting conditions to explain the interplay between and functional coupling of areas ^89–93 . It transpires that the structural connectivity imposed by cortical anatomy, along with dynamic parameters, determines and explains the emergence of functional communities of areas in the resting state ^94–99 .

Whole-brain networks provide an important tool for understanding aspects of brain activity. However, it is not uniformly functioning whole areas, but rather neurons and their interaction in neuronal circuits that are the functional units that carry brain function and cognition. The crucial neuronal circuits and their interconnecting links are in part determined by neuroanatomical structure, but an equally important contribution to circuit formation comes from synaptic plasticity. Therefore, to move towards improved biological plausibility, it is necessary to incorporate neurons and plasticity into whole-brain network simulations, thus taking advantage of the constraints realized by the aforementioned associative neural networks (for example, see refs. ^{57,62,100,101} ). In essence, whole-brain network models can be improved by adding more detail at the sub-area and neuronal levels; correspondingly, DNNs can be made more realistic by adding neuroanatomical information about within-area and between-area connectivity.

Brain constraints

To make neural networks of neurocognitive function more realistic, it is necessary first to develop models of neuronal mechanisms across different levels, ranging from the micro-levels of neurons and local cortical circuits to the macro-level of cortical areas and global connectivity, and second to apply constraints from neuroscience at these different levels (Fig. 1). Previous proposals have already discussed various different constraints on neural networks ^{7,38,100,102,103} , but have frequently listed neurobiological constraints side-by-side with technical or practical criteria (for example ‘computational efficiency’ or ‘scalability’ ¹⁰⁰ ), or addressed biological plausibility very generally (for example as ‘biological realism’ in ref. ⁷ ).

Below, we spell out point-by-point what ‘biological realism’ implies and discuss a list of specific neurobiological constraints applicable to network models of cognitive functions — some of which are also suitable for neural networks more generally. As cognitive processes are in focus, mechanisms in the forebrain and especially in the cortex receive special attention. The proposed constraints are not thought to represent categorical features that must necessarily be met by models in a binary fashion. Rather, we conceive them as dimensions along which neural models can be adjusted gradually, in view of their specific purposes.

Integration at different levels

Most previous modelling has focused on one specific grain size, aiming to approximate neuronal function at the level of either single neurons ^104,105 , neuronal interaction in local cortical circuits106-109 or global interplay between cortical areas (see Whole-brain networks). To simultaneously apply constraints at different levels of brain structure and function, these different levels must be addressed and integrated into a single model. Furthermore, multi-level modelling is required to exploit multiple sources of experimental data for model validation ^{101,110–112} . These different sources include behavioural performance (for example, accuracy and response times in cognitive experiments), and neurophysiological activity, where, once again, different spatial scales come into play, ranging from the micro-level of single-cell and multi-cell recordings to that of local field potentials and macroscopic local area activations as revealed by non-invasive neuroimaging techniques, such as electroencephalography (EEG), magnetoencephalography (MEG) and functional MRI. These physiological data also come at different temporal scales, ranging from millisecond delays between neurons and high-frequency oscillations, to slow potential shifts and neurometabolic changes. The activity of recorded neurons or imaged areas, and even the similarities between these activation patterns ^113,114 or the transient functional interactions they reveal ^115,116 , can be related to activation patterns of artificial neurons, local neuron clusters or ‘area’ components of a model. Likewise, spatio-temporal patterns of activity within a local cortical neighbourhood or across different cortical areas can be compared with the patterns produced and integrated by networks. Such comparison of activity patterns across different scales requires that components of the brains and networks resemble each other structurally, and that it is possible to identify brain parts with network parts. Only in this case can the physiological measures be used to validate the neural models. The constraints below address different aspects of such similarity.

Neuron models

The functional units of the cortex and brain are neurons. These neurons receive inputs that are translated into postsynaptic potentials and finally into an output of discrete action potentials, whereby action potential frequency can vary continuously ^117,118 . All neural networks are composed of artificial correlates of neurons, but the level of detail with which neuronal function is simulated varies considerably ^104,105,119 .

Mean field models use neurons with continuous inputs and outputs (thus ignoring the spiking of most real neurons), together with a transfer function that transduces the former into the latter ^120,121 . They can be interpreted to simulate the firing probability of single neurons or the cumulative activity of local neuron circuits. The more sophisticated spiking ‘integrate-and-fire’ neurons model the summation of postsynaptic potentials and resultant neuronal firing, and can be extended to integrate dendritic compartments and non-linear interactions between their inputs ^{104,122–125} . Multi-compartment and biophysical neuron models can include even more detail, including subdivisions of the dendritic tree, postsynaptic ion-channel dynamics and dendritic action potentials ^{104,126–129} . Thus, a dimension of progression addressing the degree of realism of the neuron model may advance from mean field to integrate-and-fire to biophysical neuron models. Furthermore, some neuronal activity is difficult to explain by the input and can thus be seen as noise ¹³⁰ . The addition of noise, along with adaptation (that is, the reduction of activity with prolonged activation), can thus further increase the degree of realism of neuron models.

However, please note that the most detailed neuron model is not always the best choice for a given research question. Not only can relatively basic neuron models yield excellent descriptions of neuronal activity ¹⁰⁴ ; the greater computational resources required by sophisticated neuron models also currently limit their applicability to large-scale simulations of within-area and across-area interactions relevant for cognition.

Synaptic plasticity and learning

Evidently, the inclusion of learning mechanisms is a crucial biological ingredient of biologically plausible networks. As mentioned, localist and whole-brain models typically lack this feature. To model multiple learning systems in the brain, the implementation of both major forms of learning, supervised and unsupervised, is necessary.

Learning based on biologically plausible Hebbian principles is relevant for all cognitive domains. It is approximated by various different learning rules ¹³¹ . Some of these rules are elementary; for example, implementations of the ‘fire-together-wire-together’ principle lead to long-term potentiation (LTP) of connections between co-activated neurons. Other rules include more neurophysiological detail, in particular ones that can implement both LTP and, as a consequence of uncorrelated or anticorrelated activation, long-term depression (LTD) ^132,133 . Synaptic plasticity dependent on the timing of action potentials, known as spike-timing dependent plasticity (STDP), is a consequence of Hebbian learning realized in sophisticated implementations ^134–137 . Thus, a progression can be seen from the absence of learning, to LTP-based Hebbian learning and, ultimately, Hebbian LTP-plus-LTD learning rules and the addition of STDP.

Supervised learning involves the use of a feedback signal that informs the individual or network) whether performance was appropriate, or wrong or erroneous. However, the choice of algorithms used in supervised-learning simulations has been guided not only by biological plausibility ^7,138 ; the computational efficacy of gradient-dependent learning has played a major role ^55,56,62 . It is controversial whether these latter algorithms are biologically realistic and applicable to sophisticated learning in specific cognitive domains. Some researchers have criticized the lack of strong biological support for mechanisms that compute and feed error gradients back through feedforward neuron networks ^7,139 , whereas others point to recent emerging evidence for neurobiological mechanisms that could, in part, support aspects thereof ^56,140–142 . The mechanisms by which error gradients might gradually propagate backwards through a population of biological neurons remain a target of ongoing research.

A further putative problem concerns the type of feedback required for gradient-dependent learning. In DNN simulations, thousands of learning trials are available, whereby, for example, object pictures are classified with regard to category membership (for example, whether they show cats or cups). However, in many cases of classification and language learning in humans, such feedback does not play a major role — and is, if at all, only rarely available ¹⁴³ . It rather seems that, in some cases of cognitive learning, the absence of certain inputs is sufficient. For example, to ‘pre-empt’ the use of specific linguistic constructions — that is, to block them from the learner’s repertoire — it is sufficient to hear other constructions in contexts where the pre-empted constructions could in principle be used ^144,145 . This type of cognitive learning is not explained by feedback-driven supervised learning but invites modelling based on Hebbian plasticity ¹⁴⁶ . Nevertheless, explicit feedback is important in some types of learning (such as reinforcement learning) and its biologically realistic implementation is crucial ^7,48,138 .

Inhibition and regulation

Brains are regulated systems, and cortical activity is regulated by control mechanisms at different levels. These include the microscopic local circuit level, at which excitatory cortical neurons interact with local inhibitory cells, and the macroscopic, more global level of interacting brain parts, at which cortical activity is regulated through information exchange with the thalamus, basal ganglia and other subcortical structures ^6,147,148 . Many distributed neural networks used for simulating cognition are composed of excitatory units only and lack inhibition mechanisms. Other modelling frameworks implement regulation at an abstract level, for example by predefining a maximum number of neurons in an area that are allowed to become active at one time ^36,119 . More realistic neural network models include excitatory and inhibitory neuronal elements, such that activity regulation and control is achieved by their interplay, a mechanism crucial for arousal and attention (see Perspectives section). Few models have realized both local and global regulatory mechanisms ^{42–44,109,110,149–154} . Inclusion of inhibition and regulation mechanisms at both the local circuit level and more global levels (such as the area level) is an important feature of making neurocognitive networks biologically plausible.

Area structure

The cortex is structured into a set of areas, whereby area definition is primarily based on anatomical criteria and sometimes refined using functional information (see, for example, the Brainnetome Atlas) ^80–82,155 . Depending on the question to be addressed by a simulation, a network model may implement one, a specific selection of, or all cortical areas along with subcortical nuclei. Each area or nucleus can be realized as a separate ‘layer’ or model area including a predefined number of artificial neurons. Dimensions of progressing towards biological realism include the range of brain parts and regions covered by the model (from one, to several, to whole-brain) and the granularity of the modelled areas, moving from coarse to more fine-grained area subdivisions.

Within-area local connectivity

Pyramidal cells, the most common excitatory neurons in cortex, each carry 10,000–40,000 dendritic spines ^20,156 , each of which in turn typically contain one synapse ²⁰ . Therefore, one of these cells may make contact with a few tens of thousands of other cortical cells — within a pool of 15–32 billions of neurons in human cortex overal^l157. The number of connections (more than 10 ¹⁴ ) is too huge to be determined, item-by-item, by the genetic code alone, and therefore stochastic principles must also co-determine whether a specific pair of neurons is connected. The probability that two adjacent pyramidal neurons are connected decreases with the distance between them ^{20,101,158,159} . In sum, neuroanatomical studies indicate that local excitatory connections within a cortical area are sparse and show a neighbourhood bias towards links between adjacent neurons ^20,160 .

In view of these features, the lack of within-layer connections of hetero-associative networks does not seem biologically realistic. Many networks that include auto-associative layers or areas ^21,22,28,161 include full connectivity between all neurons within these areas and, similarly, the memory layers of simple recurrent networks57 have all-to-all recurrent connections. Such full auto-associativity likewise contrasts with the sparseness of intrinsic local cortical connections. The brain constraint of sparse, local and partly random connections with a neighbourhood bias has been realized in neural networks that connect auto-associativity with between-area hetero-associative connections ^{24,149,150,162} . Most advanced with regard to the local connectivity constraint are microcircuit models that realize different cell types and their location in layers of the neocortex ^{109,110,152,154,163} . Nonetheless, for most neural networks available today, the implementation of within-area connectivity constraints still leads to an increase in biological realism ¹⁰¹ .

Between-area global connectivity

The connections between areas of cortex follow some general rules. Most links are reciprocal. Adjacent areas are almost always interlinked, and second-next neighbours are connected in many cases ^20,164 . However, longer-distance links are sparser, and much effort has been spent to map them precisely using invasive techniques (for example, with tracers) and non-invasive techniques (such as DTI/DWI) ^{101,165–172} .

If two areas are interlinked, their connections are in most cases reciprocal and show topographic projections, such that local neighbouring relationships are preserved. Between-area connections are carried by long axon branches of cortical pyramidal cells. These axon branches pass through the white matter and can reach neurons in distant areas, where they branch and make contact with a local neighbourhood of neurons. Most hetero-associative artificial networks implement between-area links as all-to-all connections. An advance in biological realism can be achieved by introducing random connectivity with biases towards topographical projections and neighbourhood links, as, for example, in multi-area auto-associative and convolutional deep networks ^43,44,173 . At the macroscopic level, it is biologically motivated to replace the typical linear lineup of next-neighbour-connected areas (Fig. 2c,d) with realistic connectivity schemes (see, for example, refs. ^65,168,174 and section on ‘Whole-brain models’), ideally taking into account any connection asymmetries and even the number of axons per fibre bundle along with axonal conduction delays. Therefore, essential brain constraints on artificial neural networks come from the connectivity structure of between-area links as documented by neuroanatomical research.

Additional constraints on global connectivity may be taken from measures of functional interaction, including correlation-based undirected functional connectivity and directed effective connectivity ^175–178 , although we suggest applying such data as neurophysiological evidence for validating anatomically constrained models rather than as a-priori constraints (see ‘Integration of modelling at different levels’).

Recent progress and trends

We propose that models of brain function should attempt to integrate several and, ideally, all of the seven brain constraints mentioned above. The integration of microscopic and macroscopic levels is crucial to this endeavor (see constraint ‘Integration of modelling at different levels’). Microcircuit networks provide a detailed picture of the functional interplay between connected excitatory and inhibitory neurons located in different cortical layers within one local cluster of neurons (constraints ‘neuron models’, ‘inhibition and regulation’ and ‘within-area local connectivity’) ^{109,110,152,153} . Several microcircuit models have been integrated into models of multiple cortical areas by taking into account long-distance connectivity as well (constraints ‘area structure’ and ‘between-area global connectivity’) ^{154,174,179–181} .

The model by Schmidt et al. ¹⁵⁴ convincingly integrates local-circuit with area-structure and global-connectivity constraints and simulates resting-state activity of the human brain at a hitherto unprecedented level of detail (Fig. 3). However, this model is computationally demanding and touches the limits of simulation capacities of current cutting-edge computing equipment, even though it restricts itself to modelling 32 areas, each of which simulates neurons below just 1 mm² of the cortical surface. These restrictions indicate that, in view of keeping computational efforts manageable, it is necessary to carefully select the degree to which constraints can gradually be met. Furthermore, applying all constraints in their most extreme form could result in computationally unrealizable models, which may, even if realized, be too complex to be helpful as tools for better understanding brain function.

Fig. 3. Multi-level network for explaining neural dynamics based on neuroanatomical constraints.

Neurophysiological activity in 32 cortical areas of the visual system of the macaque was modelled ¹⁵⁴ . Probabilities of between-area connectivity (bottom left panel) were based on anatomical data and general connectivity principles. Applying further neuroanatomical constraints, each area was modelled as a 1 mm² patch of cortex with excitatory and inhibitory spiking cells (right panel) arranged in layers and specific connection probabilities within and between layer-specific cell populations (top left panel). The model was applied to reproduce and explain spiking neuronal activity from neurophysiological recordings, activity propagation across areas and causal dynamic interactions between neuron populations and areas. The model unifies local and large-scale accounts of the cortex and clarifies how the detailed local and global connectivity of the cortex shapes its dynamics at multiple scales. Synaptic plasticity and learning were not modelled, although all of the other six constraints discussed in the main text were applied. Figure adapted, with permission, from ref. ¹⁸⁰ .

Multiple-area networks including microcircuits open novel perspectives for neural modelling of cognition. After inclusion of biological learning mechanisms (constraint ‘synaptic plasticity and learning’), they may be applied to simulate and potentially explain higher cognitive functions. Below, we address putative benefits of adding the aforementioned neurobiological constraints to neural models of cognition.

Linking cognitive theory to the brain

Cognitive models may perfectly fit the data obtained in cognitive tasks, independent of whether they are implemented algorithmically or in a neural network. However, there is typically more than one theory, and hence algorithm or network, that can model a given data set. An advantage of neural models is that they allow the introduction of functional neuroscience constraints that can be used to decide between these alternatives ⁹ .

Object recognition and classification

As mentioned, the evidence for cognitive neural model evaluation can come from single-cell and multiple-cell recordings ^111,182 , fMRI activation patterns of areas, or spatio-temporal activation patterns revealing the orchestration of neural activity across different neuron populations or areas ^{162,183–185} . For example, a large range of models of object recognition are able to describe human object classification performance, but only neural models with some similarily to brain structure can be tested using neurophysiological data recorded at different levels of human and monkey inferior temporal cortex ^186,187 . For such testing, it is particularly useful to apply not only measures that reveal the activated loci during object perception and classification, but also the more sophisticated method of representational similarity analysis (RSA). RSA allows one to relate the degrees of similarity between brain activation patterns elicited by different stimuli to the similarity structure of the stimuli themselves, which can be measured at different levels (for example, perceptual or semantic) ¹¹³ . Notably, RSA has revealed that neurometabolic activity in inferior temporal cortex reflects semantic similarities of visually perceived objects ¹⁸⁶ . Application and comparison of alternative neural (and non-neural) models have shown that the best fit of area-specific activation patterns and pattern similarities in temporal cortex signifying object perception was achieved with a feedforward convolutional DNN trained by applying supervised learning ¹⁸⁴ . Furthermore, recent research indicates that inclusion of recurrent connections in DNNs is crucial for capturing aspects of the fast neurophysiological dynamics observed in temporal cortex ¹⁸⁸ . Over and above the structural and functional constraints already addressed by these models, further structural constraints on within- and between-area connectivity may be considered in future (constraints ‘inhibition and regulation’, ‘within-area local connectivity’ and ‘between-area global connectivity’).

Attention

Research on attention offers another example for the application of neuroscience constraints to cognitive theory. Psychologists had long theorized about the fact that visual selective attention biases object and feature perception and that different perceptual elements compete with each other for attention, an interaction described by the biased competition model of attention ¹⁸⁹ . This cognitive theory was translated into a neurocomputational model of early and higher visual cortex (including V2 and V4). Pools of excitatory and inhibitory neurons with within-area and between-area connections inspired by neuroscience data were used to model the interaction between representations, attentional bias and resource-limiting inhibition and to help to explain results of behavioural and neurophysiological recording experiments in non-human primates ^182,190 . In a separate line of work using a convolutional DNN, the effects of attention on visual object classification were recently shown to depend on the level at which biases are applied, with modulation at higher layers being relatively more effective than modulation at lower layers in biasing attention toward specific object categories ^191,192 . This observation may help to clarify the role of anterior temporal and prefrontal areas in visual attention processing ¹⁹³ . Future biological modelling may marry within-area inhibition with deep architectures, constraining the latter using the neuroanatomical connectivity of the visual ‘what’ and ‘where’ streams, and/or may apply Hebbian learning instead of backpropagation for weight tuning (constraints ‘synaptic plasticity and learning’, ‘area structure’, ‘within-area local connectivity’ and ‘between-area global connectivity’).

Language processing and reasoning. A relatively straightforward way of modelling cognitive brain functions in a neurobiological format is to select a set of areas and nuclei, model each using a set of neurons, assign cognitive functions to model areas and determine the interactions among them using predefined or learned mappings between area-specific activation vectors. This strategy has given rise to biologically inspired models of cognitive functions, including concept, language and number processing ^47,194–198 . One such model47 is particularly impressive because of its size (2.5 million neurons) and implementation of the neuron-model constraint (integrate-and-fire neurons) in various cognitive tasks, addressing, for example, number recognition, question-answering and even fluid reasoning ¹⁹⁹ . Twenty brain regions (areas and nuclei) are represented in the model, each by an ensemble of neurons, although global connections are sometimes idealized (for example, showing a linear line-up of areas from primary visual to anterior-temporal). Within areas, there is either no or full connectivity, and inhibitory neurons are missing. Therefore, three of the seven constraints have been addressed (‘Neuron models’, ‘Area structure’ and ‘between-area global connectivity’). Other brain-constrained models using smaller vocabularies and task ranges than those used in this large model have successfully integrated conceptual and language processing with action and perception mechanisms and have been applied in robots ^{194,197,198,200} .

Explaining neuroscience findings

Beyond putting cognitive theories into a neurobiological environment, brain constrained modelling can also provide answers to long-standing questions in neuroscience about how specific capacities of the brain are mechanistically implemented, how they emerge in ontogeny, how they came about in phylogeny and why they are situated in the specific brain parts where they are observed to be situated. We now turn specifically to this explanatory perspective.

Memory

Some models discussed above have provided a mechanistic explanation for the competition aspect of attention, which emerged from local inhibitory connections and between-area projections ¹⁸² ; however, the other component of the biased-competition model of attention, the bias (such as that towards an object or part of visual space), is typically presented to the model from the outside and thus remains without model-immanent explanation. A mechanism for biases may come from network models of memory that account for memory dynamics based on neuronal function and structural connectivity. Here, memory mechanisms at the subcellular level (such as synaptic plasticity) are important, but in these models the questions of why certain types of memory develop in specific species, brain parts and circuits are still partly open. With appropriately adjusted activation thresholds and synaptic weights, a fully connected auto-associative memory is characterized by persistent attractor states, a possible correlate of working memory ^22,201,202 . In models using spiking model neurons, auto-associative networks with all-to-all connectivity within their ‘areas’ gave rise to firing patterns resembling those recorded from neurons in inferior temporal lobe during working memory experiments ^39,46,203 .

Further work has addressed the question of why different cortical areas typically show different predominant memory activity patterns ^46,204 . Hebbian learning principles suggest that memory circuits are built from neurons firing together during sensory stimulation and thus located in areas where the to-be-stored information arrives; that is, in primary areas. However, most neurons with memory characteristics develop in multimodal areas (in prefrontal, anterior-temporal and posterior-parietal cortex) distant from primary areas. A deep neural model implementing six frontal and temporal areas and their neuroanatomical connectivity structure showed that unsupervised Hebbian learning applied to this network gives rise to firing patterns and realistic distributions of memory cells across primary, secondary and multimodal areas, thus providing an explanation of cortical memory topographies and dynamics based on neuron function and corticocortical connectivity ^185,204 . On a related thread, a recent study asked why a specific form of memory, verbal working memory (that is, memory for spoken words and language-like acoustic stimuli), developed in the context of specific evolutionary changes of the connectivity structure of fronto-temporal areas and is specific to humans. It was shown that the specific increase in fronto-temporal connectivity observed in primate evolution led to the emergence of distributed neuronal circuits for articulatory-acoustic units ²⁰⁵ , thus offering an explanation of a specifically human trait based on phylogenetic structural change (Fig. 4). This model and the previously described one realize the seven constraints discussed in this article (with special emphasis on ‘between-area global connectivity’), although the number of areas and the level of detail of local microcircuit simulation could still be increased.

Fig. 4. Model of evolutionary connectivity change in left fronto-temporal cortex and its functional consequences.

Verbal working memory is a feature of humans that apes and monkeys apparently lack. a | ‘Monkey’ and ‘human’ models of six areas of fronto-temporal cortex involved in articulation and auditory perception (middle panel) were used to address this issue. Areas were modelled as sets of 625 mean-field excitatory neurons, each projecting randomly to local neighbourhoods of other excitatory units (coloured); each excitatory cell has a corresponding inhibitory ‘cell’ (in grey) projecting to a narrow local neighbourhood (left panel). Between-area connections implementing comparative DTI results ^{170,171,221,222} included next-neighbour connections between areas (green) and the second-next area connections specific to human perisylvian cortex (violet links in the 'human’ model only). Correlation-based Hebbian plasticity was applied to imitate early sound and sign learning and to interlink articulatory and auditory information. b | After stimulation with learnt auditory–articulatory patterns, the monkey model showed weak and short-lived sequential activation of the model areas: A1, primary auditory cortex; AB auditory belt cortex; M1, primary articulatory motor cortex; PB, auditory parabelt cortex; PF, inferior prefrontal cortex; PM, premotor cortex. c | The same stimulation led to strong and long-lasting parallel activation in the areas of the human model. This prolonged activity can be interpreted as verbal working memory, a mechanism necessary for human language. The model applies all seven constraints discussed above. The left portion of part a is adapted with permission from ref. ²¹¹ . Parts a-c are adapted from ref. ²⁰⁵ .

Concepts

Similar explanatory advances could be achieved in the domain of language and conceptual processing. It is well-known that multimodal areas in frontal, temporal and parietal lobe are important for conceptual and semantic processing generally, whereas modality-preferential sensory and motor areas contribute to the processing of specific semantic categories ^206–209 . An explanation of why these particular sites are relevant for concepts generally or specifically had been missing. Brain theory suggests that the convergence of multimodal information is essential for conceptual mechanisms ²¹⁰ . To get from the visual perception of a cat to the related concept and, for example, the knowledge of what kind of sound a cat typically produces, information from different modalities needs to be integrated. Such integration of multimodal information requires connections bridging between modality-specific neural systems in different cortical areas (which are distant from each other), which implies a role of large-scale connectivity structure in conceptual processing. Modelling of word learning processes in frontal and temporal cortex that applied realistic early language learning scenarios, unsupervised learning and anatomically constrained local and global connectivity, revealed a distribution of neuronal circuits for conceptual and semantic processing consistent with the data. Specifically, most neurons of conceptual circuits, which formed as a consequence of co-processing information about signs and their related objects or actions (Fig. 5), were housed in the model’s multimodal convergence hubs, thus explaining the general ‘pull’ away from the areas where sensorimotor information reaches the cortex towards the most strongly connected connector hub areas. By contrast, the stimulation of specific modality-specific areas during conceptual learning and the resultant correlated neural activation topographies helped to explain the category-specific contribution of modality-preferential areas (Fig. 5) ^{185,211–213} . Again, realistic between-area connectivity was a crucial factor of these explanations; all seven constraints were addressed.

Fig. 5. Brain-constrained model of semantic grounding.

a | For simulating the infant’s learning of the meaning of object-related and action-related words, a 12-area model was created including the six inferior-frontal and superior-temporal perisylvian areas of Fig. 4 (A1, primary auditory cortex; AB auditory belt cortex; M1, primary articulatory motor cortex; PB, auditory parabelt cortex; PF, inferior prefrontal cortex; PM, premotor cortex), plus a ventral temporo-occipital visual stream (in green: AT, anterior-temporal cortex; TO, temporo-occipital cortex;V1, primary visual cortex) and a dorsolateral frontal action stream (in yellow-brown: lateral PF (PFL), lateral PM (PML) and lateral M1 (M1L)). Between-area connectivity is shown by arrows. Semantic learning and grounding of object and action words was modelled by co-presenting acoustic and articulatory information along with either semantic-referential object-related information or action-related information. This was done by co-activating specific patterns of spiking neurons in the different ‘primary’ areas of the model (M1 and M1L, V1 and A1) and Hebbian correlation learning. After learning, ‘auditory word comprehension’ was simulated by presenting specific previously learned auditory patterns to area A1. As a result, specific circuits of neurons distributed across the network were activated, as indicated by the coloured dots in the insets (1 dot indicates 1 active model neuron; blue, object–word circuit; red, action–word circuit; yellow, both), shown in the black boxes representing areas. b,c | Distribution of circuit neurons across model areas. Bars give average numbers of neurons per area for object- (dark grey) and action-word circuits (light grey); whiskers give standard errors. Note the relatively stronger representation of object–word circuits in ventral-visual areas and that of action–word circuits in dorsolateral-frontal areas in part a ^50,213 , which offer an explanation for well-known differences between the cortical mechanisms underlying action-related and object-related concepts ^{207,223–226} . All seven constraints discussed in the main text were implemented. Figure adapted, with permission, from ref. ¹⁸⁵ .

Future applications

Besides the aforementioned theoretical and explanatory advances offered by brain-constrained modelling, this novel strategy offers very practical application strategies for the future. One such application addresses neuroplasticity, aiming at predicting and explaining reorganization of cognitive functions after lesion or deprivation. In this context, recent modelling attempts have targeted, for example, altered language processing in patients with focal lesions of language-relevant cortical areas214 and sensory deprivation in blind people ⁵⁰ . These modelling experiments have led to accounts of well-documented neuroplasticity phenomena; for example, the takeover of visual areas during language and cognitive processing in congenitally blind individuals ⁵⁰ .

In future, it may become possible to perform neurocomputational modelling constrained by specific features of individual brains. Results obtained with individually constrained neural networks may open new perspectives on predicting future neuroplastic dynamics ²¹⁵ and may be used for planning personalised therapy or surgery, for example for individuals with brain tumours ^216–218 . Brain-constrained modelling applied to particular populations and even individual cases may thus open fruitful future perspectives.

Conclusion

Based on a brief overview of neural network simulations of cognition and their successes, we here suggest a move towards more biologically oriented modelling. According to the position put forward in this Perspective, neuroscience constraints have priority over other aims, such as processing efficacy and big data processing. Beyond the imperative to build models that bridge between the macro-scale and micro-scales, in order to be testable against physiological data at different levels, the proposed constraints address neuron models, learning algorithms and regulation mechanisms along with neuroanatomical constraints on model architecture that address area subdivisions and short-range and long-range connectivity. As concrete examples of brain-constrained modelling, we have reviewed biologically motivated neurocomputational implementations of neurocognitive theories and model-based explanations of the brain mechanisms underlying object perception attention, memory, concepts and language along with their ontogenetic and phylogenetic development. The future outlook for brain-constrained modelling includes the generation of networks that realize features of individual brains, which may, for example, be useful in assessing which parts of cortex are crucial for retaining specific cognitive functions in a given individual.