Decoding the brain: from neural representations to mechanistic models

Abstract

A central principle in neuroscience is that neurons within the brain act in concert to produce perception, cognition, and adaptive behavior. Neurons are organized into specialized brain areas, dedicated to different functions to varying extents, and their function relies on distributed circuits to continuously encode relevant environmental and body-state features, enabling other areas to decode (interpret) these representations for computing meaningful decisions and executing precise movements. Thus, the distributed brain can be thought of as a series of computations that act to encode and decode information. In this perspective, we detail important concepts of neural encoding and decoding and highlight the mathematical tools used to measure them, including deep learning methods. We provide case studies where decoding concepts enable foundational and translational science in motor, visual, and language processing.

eTOC

Understanding how the brain works has been a formidable challenge in neuroscience but advances at the computational level are now driving us closer. This Perspective discusses core concepts of neural encoding and decoding, and how moving towards causal modeling can enable translational and foundational insights into the neural code.

Introduction

Imagine sitting at a piano reading the sheet music, taking in the dark notes on the crisp white page of Rachmaninoff’s Piano Concerto No. 2. You have a mental (internal) model of how the piano works, thus the task for your brain is how to translate the notes into motor actions. The tactile sensation of the keys provides critical proprioceptive feedback, and your auditory senses are heightened as they immerse themselves in the melodic contours and harmonic progressions, listening intently to each note as you play and hum along. From this multi-sensory input, you constantly refine your next movements that ultimately create the rich tapestry of a motor skill in action. This scenario showcases the brain’s ability to both encode sensory stimuli and decode this representation into meaningful actions, while modulating your play with memories and emotions.

Within the brain, sensory areas must encode stimuli, such as the edges and dark contrast of the notes on the page, and downstream areas must decode these features to build an internal model of yourself and the environment (1), transforming the statistical spiking properties of input neurons to construct new useful representations within other neurons (Figure 1A)¹. For example, internal models ultimately serve to select control policies that enable goal-directed actions (1), having integrated multi-sensory information with the prior state of the body. Collectively, this process of neural encoding and decoding lies at the heart of one fundamental question in neuroscience, that is how the brain computes to perceive, act and learn.

Figure 1. Encoding-Decoding across scales.

A: An encoder represents the neural response of population K(t) to stimulus x(t) via P(K|x), and a decoder aims to recover x(t) given the neural activity K(t) via P(x|K). B: Systems neuroscience spans scales of descriptions and decoding algorithms can target any individual level, and even span across scales. Here, we outline example scales (from genes to environment), the types of data we can collect (from genetic sequencing to whole-animal video analysis), and the classes of models the field has developed. On the far right is our mapping of scales, example data, and example tools to levels of understanding. Inset images adapted from: (4–8).

“Decoding the brain” therefore has two meanings: one, as described above, is how neural dynamics decode and transform incoming information across distributed circuits to represent meaningful information about sensory and other task stimuli (Figure 1A). The other is how we can build “decoder” algorithms to measure information in the brain (representational level analysis) and use it for translational approaches like Brain-Computer Interfaces (BCI) (2, 3) (Figure 1A), but this does not necessarily link to neural mechanisms. Nonetheless, both avenues require recording from neurons and transforming action potentials (or other signals gleaned from fMRI, EEG, etc.) into lower dimensional representations of the data or latent factors.

However, the inherent challenge lies in the brain’s complexity, from its vast scale to the intricacies of its internal language, spanning from the 302 neurons in a worm to the 80 billion in the human brain (9). Fortunately, technological advances allow us to record from large numbers (10, 11) of neurons and build powerful machine learning models (12, 13). Deciphering the neural code involves grappling with nonlinear, dynamic systems distributed across brain regions, functioning across temporal scales to integrate past experiences, the current state, and future predictions.

A significant challenge of decoding is determining the necessary dataset scale and even timescale to train and evaluate the performance and generalization of the learned model. Namely, for constructing a decoder for BCI applications in motor control or language, for example, shorter time bins of spiking data from local neuronal populations are likely sufficient. However, learning hierarchical behavioral representations that span orders of magnitude in both time (from seconds to years) and space (from local environments to the entire world) in order to build foundational internal world models for downstream decoding tasks definitely requires richer, larger datasets. These datasets should ideally encompass recordings from individual neurons to the entire brain across multiple time scales (Figure 1B).

In this perspective, we begin by reviewing core principles of neural encoding-decoding, mathematical concepts, and discuss work on using data-driven and normative machine learning models. Then, we provide case studies that highlight aspects of how neural decoding can allow for translational and foundational insights into the neural code. Lastly, we argue that ultimately the field should move towards causal modeling that allows us to infer and test causality in neural circuits.

Neural encoding

Decoding information from neural activity hinges on the assumption that the information is, in fact, encoded by the neural population in the first place (Figure 1). Thus, we begin by covering neural encoding principles as it serves as a foundation for our understanding of how neurons mechanistically encode information and how we interpret neural activity. Foundational research involving frogs, cats, mice, non-human primates, and humans has demonstrated that neurons convey information through their action potentials. As neuroscientists, we develop encoding models to quantify, from an information-theoretic perspective, the extent to which neural activity can be explained by externally observable or internally estimated variables (14, 15).

From a mathematical perspective an encoder represents the neural response of population K to stimulus (or event) x:

$$P(K\mid x)$$ (1)

Here K is a vector representing the activity of N neurons, and each entry represents, e.g., the number of spikes in some time bin or the rate response of that particular neuron. Fundamentally, this statistical relationship summarizes how a group of neurons respond to an event x. As we discuss, there are different approaches for estimating these models.

Techniques such as linear regression, generalized linear models (GLMs), and artificial neural networks (ANNs) enable us to assess how individual neurons encode information (15–23). This understanding is crucial because it forms the basis for constructing models of neural population dynamics, whether through population vectors, latent factor dynamical systems, or sophisticated hierarchical neural network models (which will be discussed in depth below).

In brief, linear regression models provide a basic framework by predicting neural responses through a linear relationship with stimulus features, while GLMs offer more flexibility by accommodating non-normal response distributions and nonlinear link functions, making them well-suited for a broader range of neural data (16, 17). To quantify the amount of information neural responses convey about stimuli, information theory models such as mutual information are utilized, offering a measure of predictive accuracy without assuming a specific relationship form. ANNs consist of (multiple) layers of simplified (computational) neurons, whose connectivity patterns mimic the hierarchical, integrative properties of biological circuits. They are universal function approximators (24) and thus, have emerged as powerful non-linear encoding models (18–23).

Decoding within the brain

Once information is encoded, downstream areas must integrate information from upstream ensembles of neurons. For example when reading the sheet music, the retina processes photons and retinal ganglion cells transmit activity via the lateral geniculate nucleus to primary visual cortex. Even at the level of the retina, encoding and decoding approaches have been powerful to better estimate the variance in neural responses (16).

The information encoded by specific groups of neurons, such as local contours in an early visual area of the visual hierarchy like V1 (although not all (25)), is processed or decoded by downstream neurons in higher visual areas like V4 to transform information and encode higher-order features like contours or textures (22, 26). Therefore, from the brain’s point of view, neurons are encoding new information (i.e., representing specific latent world variables) by decoding and transforming information from upstream neurons. The encoding-decoding process must be thought of as two sides of the same coin, where neurons encode, transform, and process information from upstream neurons to more easily decode high-level relevant features of the world and drive behavior. Of course, sensory information cannot increase along a processing hierarchy, and thus all information about the visual world at a given instance is in the photon patterns impinging on the retina. However, the question lies in the simplicity of the decoder to extract relevant information, both from the experimenter’s point of view and downstream neurons (19, 27).

For instance, the representation of a specific friend under all possible poses, lighting conditions, clutter, scales, etc., is encoded in the patterns of activity in the retina, forming a non-linear neural manifold embedded in a high-dimensional space. However, decoding or extracting the identity of that specific friend from all other possible images in a generalizable way requires a complex, non-linear decoder at the level of the retina (27). In contrast, neurons in the inferotemporal (IT) cortex may allow for simpler decoding, potentially even with a linear decoder (19, 21, 28). This progression through the visual processing hierarchy illustrates the shift from implicit to explicit encoding of information (19, 27). Early visual processing involves implicit encoding that does not directly convey object identity, whereas higher visual areas like the IT cortex provide more explicit representations tied to identifiable objects, making them easier to decode and more “human-interpretable.” This view is, of course, oversimplified, as the visual cortex contains abundant redundant reciprocal connections, and this yin-yang of encoding and decoding is not simply done in a feed-forward manner.

One elegant line of work on how neurons might decode upstream spikes comes from studies on decision-making in rodents. Watabe-Uchida, Uchida, and colleagues have shown that dopaminergic neurons in the ventral tegmental area (VTA) encode reward prediction errors (RPEs), and GABAergic neurons encode a function akin to the estimated state value (29). Notably, this maps exceedingly well to reinforcement learning algorithms (29, 30). Then, to address how dopamine neurons come to compute these RPEs, they measure their anatomical “inputome” (31), and go on to record from these upstream neurons to find partially computed RPEs (32), suggesting that dopamine neurons must decode these partially computed RPEs and inputs from GABAergic neurons in order to fully compute then broadcast RPEs (33). This elegant example highlights how theoretical models and the neural decoding framework allows us to estimate what neurons decode in order to compute. Moreover, it offers us a mathematical framework to formalize mapping neural representations to neural computations.

Neural decoding: mathematical principles

As outlined, information processing in the brain can be conceptualized (in a simplified fashion) as a series of cascading encoding-decoding operations. Through these operations, the brain extracts relevant information from the environment, transforms it, and ultimately uses it to guide behavior. Decoding models serve as powerful tools in this context. This section goes into the mathematical principles of these decoders.

Consider a population of neurons encoding a stimulus x as described by Eq. (1): P(K|x). A natural question to ask is, can we predict x from a spike count vector K? The aim of a decoder is to predict x from the neural response K. Mathematically, a decoder is a function that maps K to some estimate $\widehat{x}(K)$. Naturally, many different decoders are possible and linear decoders are particularly simple. First, we want to start with describing the linear decoder in more detail. A linear decoder combines the activities of the different neurons in a linear fashion, i.e.,

$$\widehat{x}(K)=w_1\cdot K_1+w_2\cdot K_2+\dots +w_N\cdot K_N,$$ (2)

where the different w_i are weight vectors that indicate how much the activity of neuron k_i contributes to the estimate. This decoder is biologically plausible, as it’s rather natural to think of neurons to linearly combine their inputs.

A particularly instructive and simple example is given by the cercal system of the cricket (34). The cricket has four neurons that integrate information of hair cells that are responsive to planar wind (Figure 2A). By combining their activity in a linear fashion one can estimate the wind direction in a simple and effective way (Figure 2B). This linear combination is commonly called population vector, i.e., the weighted sum of the activity of all neurons. We will later see that the population vector enables strong movement decoding in primates (35), and that linear decoders are essential for assessing learned representations of models (linear probing). Furthermore, linking to the earlier example, object identity can be accurately decoded from IT, but not from upstream brain areas with linear decoders (19, 28).

Figure 2. Encoding model and decoding methods.

A: The cercal system of the cricket has four interneurons that represent the wind direction. The preferred wind directions of the neurons are pointing in four cardinal directions and can be represented by orthogonal vectors (on the left). Each neuron responds with a firing rate approximated by a half-wave rectified cosine function. The maximum firing rate is elicited when the wind is blowing in the preferred direction. B: The wind direction x can be decoded as the direction of the population vector $\widehat{x}$. This vector is the sum of the four preferred orientations scaled by their firing rate. An example is shown for neurons responding with activity [36, 12, 2, 1]^T. Note how the population vector closely matches the wind direction. C: In the k-nearest neighbors (k-NN) decoding method, neural activity K is represented within a neural activity space, which is illustrated here in 2D for two neurons for clarity (neuron 1 and neuron 2 from panel A). With these two neurons, angles between 0° and 225° can be represented. For simplicity, we focus on a nearest neighbor variant with k = 1. As 1-NN is only able to decode discrete variable, we classify the angles in three ranges: 0°−45°, 45°−135° and 135°−225°. Previously observed trials are color-coded by their associated wind direction ranges (L = 13). To decode the wind direction for a new trial (unfilled triangle), the k-nearest neighbors (here, k = 1) in the activity space are identified. The decoded wind direction corresponds to the wind associated with the nearest neighbor, highlighted by the sample connected to the observed sample via a dashed line. D: Bayesian decoders incorporate a prior P(x) (dashed line) that reflects the probability of different wind directions before taking neural evidence into account and influences the decoded angle. For instance, if mainly wind directions around 125° have been experienced (mean of the prior P(x)), the decoded angle will be shifted towards this direction. The likelihood P(K|x) (green-blue line) describes the probability of observing a particular neural response K given a specific wind direction. Following Bayes’ theorem Eq. (3), the prior P(x) and the likelihood P(K|x) are multiplied to obtain the posterior distribution P(x|K) (solid black line). The posterior can be used to decode the wind direction, here 270°, based on highest value for the observed neural activity K.

Another simple yet powerful decoder is given by the k-nearest neighbors algorithm (k-NN; illustrated in Figure 2C). Consider L recorded neural responses K^l each associated with a stimulus x^l: (x¹, K¹), (x², K²), … (x^L, K^L). To decode the stimulus from a new trial with neural response K, the one-nearest neighbor algorithm finds the neural pattern K^j which is closest to K and assigns the new trial to the corresponding stimulus x^j (Figure 2C). In the more general case, one considers the k-nearest neighbors and can also average their events.

Bayesian decoders directly use the probabilistic encoding model (15, 36). Concretely, a Bayesian decoder uses Bayes’ theorem to compute the probability that, given a response K, the stimulus x was presented. Mathematically, let P(x) denote the probability of a stimulus x and P(K|x) the conditional probability of obtaining the population response K given the stimulus x (as in Eq. (1); illustrated in Figure 2D)). Bayes theorem states that:

$$P(x\mid K)=\frac{P(K\mid x)\cdot P(x)}{P(K)},$$ (3)

with $P(K)={\sum }_{x}P(K\mid x)\cdot P(x)$. Using this expression of the posterior probability P(x|K), we can predict the most likely stimulus: the x that maximizes P(x|K). Bayesian decoders such as the Naive Bayes decoder have been popular for position decoding from hippocampal activity, and here one can utilize priors representing the a priori expected location of the animal (37). Another classic decoder builds on the Kalman filter (38) and allows levering the dynamics of the system.

How can we assess the quality of a decoder? For a continuous variable, like wind direction, we can just check how well it reconstructs the original stimulus x:

$${\mathbb{E}}_{K~P(K\mid x)}\Vert \widehat{x}(K)-x{\Vert }^2,$$ (4)

where we average over samples K (15). This is the variance of the decoder. We think that one particular decoder is better than another decoder, if it has a lower variance, or in other words if it is more accurate at estimating x from the neural response of the population.

Given the large number of potential decoders, establishing lower bounds to the variance is crucial to studying theoretical components of representations. The Cramèr-Rao inequality serves exactly this purpose. It states that the variance of any unbiased estimator can be bound from below by the inverse Fisher information (15). Notably, the Fisher information can be calculated directly from the encoding model, P(K|x), enabling the extraction of valuable insight for simple tuning curves (such as parameterized place and grid cells) within this framework (39, 40). For an excellent review on this topic see Kriegeskorte and Wei (41).

However, in many instances where the encoding model P(K|x) is complex or not even explicitly known, calculating the posterior probability P(x|K) is computationally challenging. As we discuss next, this is where data-driven and normative models, taking advantage of machine learning, can learn powerful statistical models to facilitate decoding.

Data-driven machine learning

As delineated above, the key to a good decoder algorithm is a solid encoding model. There are several approaches to do so, namely data-driven or task-driven models. Data-driven decoding approaches, which we cover in this section, build powerful statistical models to assess how stimuli or behavior are encoded in neural activity, not necessarily focusing on mechanistic realism (Figure 3). Concretely, by mechanistic models we mean they aim to model the biological basis (implementation level) of a given neural function, such as the Hodgkin-Huxley model (42).

Figure 3. Data-driven models: statistical power vs. mechanistic realism.

Ultimately, as a field we want to map mechanism to computation in order to have causal, testable models. On one side we have mechanistic models, and on the other statistical models that aim to best encode neural dynamics, but there is a large gap between them. We provide a non-exhaustive selection of contributions: (13, 16, 36, 42–46).

Historically, data-driven models were hindered by, one, the lack of large-enough datasets, but modern methods allow for single-neuron resolution of nearly 1M neurons at a time (10), and two, modeling approaches that could combine data across sessions and animals without averaging data (12, 13). Combined, this paradigm shift now greatly enables data-driven modeling approaches capable of capturing the complexity of high-dimensional population activity while remaining computationally feasible.

Two main categories of data-driven statistical models have emerged in neuroscience: fully observed models and latent variable models (13, 47–52). Fully observed models, such as GLMs and Vine Copula models, strive to explicitly delineate the interactions among neurons by directly modeling the joint activity of the population, operating under the assumption that the recorded population encompasses all relevant neuronal activity without the need to account for unrecorded neurons contributing to the neural manifold. On other hand, latent variable models, which we will mostly focus on, infer hidden (i.e., latent) variables that capture the underlying structure of the observed neural data through a joint probability distribution, acknowledging the possibility that unrecorded neurons or other unseen factors may contribute to the observed data. The goal of both approaches is to measure how much information, often measured through decoding or other information-theoretic approaches, about a given stimulus or behavior is captured by a model.

Given the intractability of modeling the entire space of neural activity, fully observed models rely on efficient descriptions of neuronal dynamics, often a priori ascribing the dynamics for simplicity (50, 53). On the other hand, latent variable models operate under the assumption that population activity is typically constrained on a low dimensional manifold and can be summarized by a compact set of variables known as latent factors (or variables²).

How can we extract neural latents? Classic dimensionality reduction techniques such as Principal Component Analysis (PCA) (54) and Independent Component Analysis (ICA) (55) simplify neural data by revealing the underlying (linear) structure essential for understanding the stimuli that are encoded. Of note, while such linear methods enhance interpretability, they often sacrifice performance and may over-estimate the true dimensionality (56, 57). A correct estimation of the intrinsic dimensionality is arguably critical for scientific interpretability. Namely, our aim is to reduce the dimensionality of the high-dimensional neuronal space into a latent space of reduced dimensions where the geometry of the latent space is interpretable with respect to behaviorally meaningful features like objects, actions, decisions, cognitive states, etc. Emerging frameworks have successfully illustrated this in many different domains (13, 57–61).

These latent factors are the mathematical representation underlying dynamics that give rise to the observable data (spikes) that we can directly record (Figure 4A). Thus, rather than explicitly modeling neuron correlations, these models capture neuronal relationships through the activation of these latent factors, often through non-linear machine learning approaches such as variational auto-encoders (VAEs) (62), or contrastive learning (13, 63, 64). The aim of these methods is to generate a latent space, or so-called embedding, that captures the variance of the observed neural data within a smaller number of factors. This is akin to a dimensionality reduction procedure.

Figure 4. Learnable latent variable models.

A: On the path to building more causal models are new frameworks, such as CEBRA (13), that allow for learning a mapping from the observable data K to the latent dynamics Z. Here, the aim is to use identifiable models with contrastive learning g(Z) (the encoder), then invert this model or use another decoder framework to probe the relationship between the estimated latents, Z¹, and a variable such as an externally observable state (behavior), internal, or sensory (i.e., recover some stimulus space (x)).

With those frameworks, complex, nonlinear relationships between stimuli and neural responses can be assumed. ANNs are particularly useful for quantifying non-linear relationships in neural data (12, 18, 19, 25, 28). Recent strides in non-linear disentangled representation learning and self-supervised learning have paved the way for new methods that can be jointly applied to behavioral and neural recordings (63, 65), unveiling meaningful lower-dimensional neural population dynamics. Recently a new dimensionality reduction method called CEBRA: Consistent EmBeddings of high-dimensional Recordings using Auxiliary variables, introduced a new paradigm for joint modeling of time-series data with a generalized contrastive learning algorithm (13) (Figure 4A). Critically, the data-sampling scheme (the selection of “positive” samples of paired (x, K) data vs. the “negative” samples that are ultimately contrasted against, ϕ) and the model optimization (minimally denoted by ϕ(·,·)) are directly linked such that the resulting latent factors can be interpreted based on the input auxiliary variables (such as behaviors (kinematics), animal identification, rewards, estimated internal states, etc).

Given an auxiliary variable x, such as the continuous position of an animal, and neural data K, one can select the positive distribution (samples) of paired data, p(x|K), to explicitly test the relationship of the auxiliary variable to the neural data. This is then contrasted with a negative distribution q(x|K) to optimize an ANN, (f). It was empirically shown that if x does not influence K, the model cannot falsely fit the data (in fact, it collapses on the manifold used for training (13)). Thus, this method can be used to construct data-driven models that allow for hypothesis testing, and this model can be simply linked to a decoder algorithm of choice for downstream use, whether for brain-computer interfaces (BCIs) or for interpretation of the latent variables (13, 66, 67).

Note, the goal of this contrastive approach, in comparison to autoencoders (like VAEs), is not to reconstruct the input data (i.e., spikes) but rather invert the data generating process to extract latent variables that give rise to the recorded data (whether from spikes, fMRI, ECoG, or calcium imaging (13, 66)). A promising direction of this work is to interrogate how neurons across time contribute to latent variables (67). Here one can build on advances in interpretable machine learning. Moreover, we can begin to derive the underpinning ordinary differential equations (ODEs) that govern the latent representation (45, 46, 68). Critically, this approach could be additionally merged with normative approaches (see below) in order to ultimately link mechanistic to statistical approaches to build causal models (Figure 3).

Normative, task-driven machine learning

Normative models address the question of why a system exhibits particular features (69, 70), aiming to better link mechanism to function. For example, Horace Barlow proposed the efficient coding hypothesis, which postulates that a sensory system minimizes the number of action potentials (energy constraints) to efficiently represent sensory information (71). One can, for instance, deduce that simple cells emerge as a consequence of sparsely representing natural scenes (72). Sparse coding also explained coding properties in many other systems.

Machine learning has elevated normative modeling to new heights in the name of task-driven modeling, where the normative principle is the goals (or computational task, i.e., Figure 1B) that the system is trained to achieve (19, 70). It is important to keep in mind that just 15 years ago, it was considered challenging to train machines to recognize objects in natural scenes (27). Yet, while robustness issues persist (73), in the meantime these models are the best models at predicting the neural representations of the ventral pathway in non-human primates (19, 21). Importantly, for decoding the brain, this suggests that we can gain insights into neural coding, with a complementary approach to data-driven modeling. This approach is particular successful, in the (neural) data-poor regime.

The normative approach leverages the computational power of ANNs to explore how neural circuits implement complex tasks. The underlying assumption is that neural responses emerge as an interplay of task objectives, neural circuit architecture and learning constraints. This approach involves training models on specific tasks to test hypotheses regarding the computations carried out by neural circuits. By doing so, they provide insights into the functional significance of neural activity patterns and their role in mediating behavior. The emerging representations are then compared neural data via linear probing, i.e., one checks how well the activity on test stimuli can be decoded from the learned representations via a linear readout (19, 28).

This normative framework is general and has been applied beyond vision (18–20, 22, 23), in studies on audition (74), proprioception (75, 76), heat perception (77), and path integration (78, 79). In contrast to neural-data-heavy (data-driven) approaches, here one needs large scale stimulus datasets, tasks and architectures. Thus, for instance in the context of path integration, one simply needs movement trajectories (79). In the context of object recognition, one needs large-scale image datasets with annotations of objects (18, 19). If primary data for a particular stimulus is not easily available, physics engines can help to generate the relevant large-scale datasets (75, 76, 80). For instance, Vargas et al. (76) used musculoskeletal simulators to generated synthetic proprioceptive inputs resulting from passive, naturalistic movement at scale, which they then used to train neural network models on computational tasks reflecting hypotheses about proprioception. Subsequently, they tested if the network’s learned internal representations resembled those of proprioceptive brain areas and found that task-driven models could more accurately predict single-trial neural dynamics than classical encoding models. For the majority of computational tasks, architectures that performed better at solving the computational tasks based on biomechanical data, were better at explaining the neural data (76). This work highlights that we can gain insight into neural processing via constraints from the body including muscles.

However, as our capabilities to scale up recording techniques improve and we enter the regime of very large-scale data in neuroscience (10), including in behaving non-human primates (81), we are beginning to see data-driven models outperform task-driven ones (82). This shift opens exciting research directions to develop new normative models to bridge these gaps. One such intriguing principle is the idea that biological neural systems have evolved to reflect the symmetries of the natural world. For example, in object recognition, translation, scale, and 3D pose represent group symmetries or transformations that do not alter the identity of the object. In the context of group theory, a symmetry is characterized as a transformation that maintains the identify of an object, through the relevant operations. Leveraging the principle that neurons learn representations invariant to 2D translation, recent works have derived filters similar to those of V1 neurons, proposing an alternative normative model to sparse coding that predicts the characteristics of neurons in V1 (83, 84).

Visual feature decoding

Understanding the brain’s algorithms of visual perception requires comprehending how natural visual scenes translate into neural activity. This subject has been explored using two complementary methods: encoding methods, which describe neural responses based on the stimuli, and decoding methods, which aim to reconstruct the visual scene or specific attributes from neural data without learning an encoding model first.

Historically, predicting how higher order visual neurons respond to visual stimuli has long been an open challenge (27). Yamins et al. (28) proposed the task-driven modeling approach and could substantially improve our ability to predict the neural activity of V4 and IT neurons. Subsequently progress was fast and more powerful encoding models were created gaining insights into core-object recognition in primates. This approach was also successful for modeling neural responses to static images in mice (23, 25, 85) and studying how brain states modulate neuronal tuning (86).

These models have mostly focused on static images, and learning encoding models for dynamic scenes has been a challenge. By following the data-driven foundation model paradigm (87), recent works have built a video encoding model that was trained on large-scale data (> 70,000 neurons) combing data from many mice and visual areas (88). Foundation models, by definition, are built in order to provide a strong encoder model for many downstream decoding tasks. Indeed, the model improved the predictive power for natural videos, and many other stimulus domains that the model was not trained on (88). It could also predict synaptic connectivity in the MICrONS data (89), which combined functional imaging with synaptic connectivity measured through electron microscopy (90). Conversely, anatomy can also be used to build better encoding-decoding models. The recent emergence of a partial connectome in Drosophila has also already been leveraged in models of vision using a connectome-constrained deep mechanistic network that was able to predict neural responses across the fly visual system at single-neuron resolution (91).

When modeling more complex neural circuits, such as those in the neocortex, deep learning models that lack implementation-level details remain the gold standard. Introducing mechanistic level details of biological circuits to build more accurate predictive models is challenging. This is formidable due to inherent complexity of cortical circuits and the difficulties associated with global connectivity mapping. Moreover, even though the architecture and elements of networks might not limit the range of functions they can realize, they significantly influence which function is selected during the learning process when data is limited. This means that the inductive bias of architectural elements is intricately linked to the learning rule, underscoring that having the correct circuit structure may not be enough without the appropriate learning rules to guide the model towards better predictions.

A common criticism of using ANNs to model the brain is that we are simply swapping one complex system (the brain) for another (ANNs). However, unlike the brain, we have a degree of control over how ANNs are constructed (70). While we may not fully grasp the nuances of their capabilities, ANNs are to some extent explainable through their architecture, task performance, and training data. Furthermore, encoding-decoding models can also be leveraged for closed-loop physiology experiments to gain insights into neural coding (Figure 5A). A few years ago, several groups made exciting breakthroughs in this domain (25, 92, 93).

Figure 5. Examples of decoding from motor, vision, and language areas.

A: Close loop experiments using digital twins: Schematic of an inception loop, depicted clockwise from the upper left: 1) Presentation of large entropy natural stimuli and tasks while recording large-scale neural activity, 2) Deep learning models accurately predict neural activity, creating a functional digital twin of the recorded neurons, 3) The in silico model facilitates unlimited experiments and employs mechanistic interpretability tools to characterize neural tuning, and 4) Images and hypotheses synthesized in silico are validated back in vivo. B: Illustration of decoded images from fMRI using Diffusion Models: Ground truth (GT) vs. decoded images generated by Chen et al. (94) from human fMRI with a diffusion model. Note that decoded images share similar color, shape and semantics. C: Multi-modal speech decoding: Adapted from Metzger et al. (95), this panel shows the decoding pipeline, where neural activity was used to train a ANN to predict phone probabilities, speech-sound features and articulatory gestures. A decoder was then constructed to produce text, generate audible speech and animate an avatar, respectively.

The development of accurate predictive models of neural activity which function as digital twins of brain circuits, has opened the door to conducting in silico experiments. These models, combined with emerging tools from the field of interpretability research (or mechanistic interpretability) (96) enable the generation of hypotheses that can be tested in vivo through closed-loop experiments (inception loops, see Figure 5A). These models have proven effective in predicting how the brain responds to novel images (25, 92, 93) and have paved the way for creating synthetic images that maximally activate specific neurons or selectively drive a particular neuron while inactivating another group (25, 92). This approach has yielded significant insights: for example, evidence for a columnar organization of tuning to spatial patterns in visual area V4 (22), the characterization of single-neurons invariances (97), and the exploration of contextual modulations at the single-neuron level (98). Recent advancements, such as the use of diffusion-mechanisms, have further improved the process of predicting the most exciting stimuli (82), offering faster and images that can generalize better across varying model architectures compared to earlier gradient-based (25, 92).

In parallel, different groups also succeeded in reconstructing visual input from neural activity. Using CEBRA with a k-NN decoder to predict the best matching video frame from neural activity in the visual cortex of mice (13). The predicted frame was close to the frame that the mice actually saw (with ≈ 95% accuracy). Here, the authors did not decode images at the pixel level. Yet several other teams, based on paired data of images and fMRI activity, trained data-driven diffusion models to tackle this question (94, 99, 100). While these demonstrated that a diffusion model could decode realistic and semantically correct images (Figure 5B), others have observed a significant decrease in performance when applying these methods to datasets that were specifically designed to prevent category overlaps between training and test sets, underscoring the limitations of these fMRI approaches (101).

Taken together, the fusion of large-scale neural activity recording with the latest advancements in machine learning has significantly enhanced the precision of encoding-decoding models, even in the context of higher visual processing. Although encoding models are evaluated based on how accurately they predict neural responses, interpreting these models and what specific groups of neurons represent becomes complex when dealing with natural images due to the highly non-linear nature of neuronal tuning.

Decoding models are instrumental in translating neuronal group activities into understandable stimulus features, such as stimulus orientation, motion direction, or the detection of specific objects. The effectiveness of these decoding approaches is heavily influenced by the selection of a quality metric or training loss function, which guides the estimation of these decoded features. For tasks like identifying an object’s class or the motion direction of a stimulus, the choice of loss function is straightforward. However, when tackling more complex challenges, such as the reconstruction of whole visual scenes, defining an appropriate loss function becomes more daunting.

Metrics based on image properties, like the widely utilized mean-squared-error of pixel intensities, often fail to align with how humans perceive similarity. This issue becomes particularly pronounced when attempting to develop an effective loss function for reconstructing intricate natural scenes in higher visual areas. These areas are proficient at extracting specific latent features from visual scenes, including textures, shapes, colors, and faces. However, the specific visual features encoded are not known in advance. The key measure of success lies in the ability of the decoded stimuli to accurately reflect the brain activity that initiated their reconstruction, essentially creating visual equivalents, or metamers, that are indistinguishable to a given neuronal population when compared to the original image.

Lastly, in visual decision making tasks, decoding can be very powerful during closed-loop experiments. An elegant example is work from Peixoto et al. (102) where the authors record from motor cortex during a visual discrimination (motion dot) task. They constructed a “decision-variable” (DV) decoder such that at each time step, in real-time, they have a continuous readout of the decoder’s prediction (concretely, the logistic model’s log odds ratio) if the macaque will choose left or right. They found that the within-trial DV fluctuations could predict behavioral choices (starting within only 250 ms) substantially better than the condition averaged DV or the visual stimulus alone (and it correlated with the strength of the motion coherence). Moreover, analogous to the inception-loop paradigm, they used the DV to terminate the trial at a particular DV threshold to test how accurate the model was: if they terminated when the model strongly predicted the macaque would “choice left”, for example, it was over 90% accurate, suggesting that this readout from motor cortex was tightly linked to the perceptual decision-making of the animal.

Decoding the motor cortex

The study of the motor system testifies to the diversity and evolution of approaches to decode the brain: from deciphering the role of individual neurons to decoding the computations underlying motor control. Pioneering work in the early 20th century established the fundamental link between individual neuron activity and motor function. Singular motor neurons firing in the spinal cord activate specific muscles directly. Upstream, in the primary motor cortex (M1), the activity of single neurons was found to be correlated with a range of movement-related variables (103) such as force (104), muscle activity (105) and joint kinematics (105). Given that layer 5 M1 neurons project directly onto spinal alpha motor neurons (in many primates), it is expected to find the representation of low-level movement variables.

If the activity of some neurons in M1 relates to lower-level kinetic and kinematic features, what about others? In the 1980’s, Georgopolous and colleagues showed that the direction of whole-arm movements in monkeys could be predicted by simultaneously recording from multiple M1 neurons (106, 107). They found many neurons that are broadly tuned to a preferred direction of hand movement. By constructing a population vector—a weighted sum of the firing rates of multiple neurons—the direction of the hand movement can be accurately decoded (akin to the linear decoder in Figure 2). This framework highlighted a simple mathematical relationship between the reaching direction and population neural activity (cf. Eq. (2)). The population vector approach could successfully find representations of higher-level kinematic variables in the motor cortex (107–109).

BCIs seek to build a link between neural activity and various tools, e.g., in order to control computer cursors (110–114), robotic arms (115–117) or prosthetic devices for paralyzed patients (118, 119). Another consideration, especially for BCIs, is the impact of volitional control on neural activity. Subjects have been shown to quickly learn to optimize control by modulating neural activity, allowing for effective operation of devices like robotic arms or functional electrical stimulation onto muscles (114, 120), irrespective of the neurons’ original encoding.

BCIs thus rely on brain decoders and are ideal for testing the performance of different algorithms. Early BCIs primarily relied on linear decoders for extracting movement kinematics from neural population activity (121). However, incorporating Kalman filters has demonstrably enhanced accuracy (117, 122–124). As described above, the Kalman filter (38) helps estimate the evolution of the state (e.g. cursor velocity) over time and updates its estimates according to the observations (neural recordings) and evolving predictions. The success of Kalman filter decoders highlights a key limitation of the classic population vector approach. While population vectors offer a static snapshot of neural activity, movements are inherently dynamic. Furthermore, identifying representations of movement-related variables within a population vector does not illuminate how either the representation or the movement itself is generated.

The dynamical systems perspective focuses on how the state of a neural population evolves over time. Here, the representation of a neural population is viewed as a dynamical system that performs computations, such as generating movements, through its temporal evolution. External inputs and intrinsic neural dynamics, which dictate how the current neural state influences the next, govern the evolution of neural activity (125, 126). Let the vector K(t) describe the firing rates of N neurons at time t. We can express the evolution of this vector with the following equation

$$\frac{dK(t)}{dt}=f(K(t),u(t))$$ (5)

where the vector $\frac{dK}{dt}$ is the temporal derivative of K, u is a vector describing the external inputs to the neural population and f is a function that defines the dynamics of the neural population.

This perspective shifted the focus from deciphering the information encoded by the motor cortex to understanding how it generates movement. Within this framework, how does equation Eq. (5) aid in decoding the computations underlying movement control? Let’s consider the following: to execute a specific movement at time t_m, the neural activity, represented by K(t_m), needs to reside within a certain subset of states—a specific configuration of activity across individual neurons. Prior to target presentation, nothing prevents the neural activity K to vary across trials. However, as the brain prepares for movement, Eq. (5) governs the evolution of K(t), effectively constraining it towards the required subspace for the intended movement. This convergence towards a specific subspace manifests itself as the reduction in variability across trials. Importantly, this analysis offers valuable insight into the computational goal of motor preparation. From a dynamical systems perspective, the goal becomes driving the neural activity towards the subspace before movement onset to produce the necessary motor commands for the desired movement.

The dynamical systems perspective has been further leveraged for studying movement preparation (127–130). Kaufman and colleagues identified distinct subspaces within the dynamics of the overall neural population during movement preparation and execution (131, 132). These subspaces were termed the “null space” and the “potent space”. During preparation, neural activity can evolve within the null space without triggering unwanted movement. In contrast, movement execution is characterized by neural activity primarily in the potent space, which controls muscle activity. However, a part of the neural activity does not drive the movement directly. Instead, it serves a supportive role “untangling” the neural dynamics—separating the neural states that can result in different future behaviors (133). Further analysis of movement-related neural activity has revealed that the dynamics have a strong rotational component and that their phase and amplitude is determined by the neural state reached during movement preparation (3, 53, 132, 134).

So far, we have reviewed how the dynamical systems perspective sheds light on experimental recordings of neural populations within cortical areas. By analyzing these recordings, researchers have been able to formulate principles governing the computations that take place during motor preparation and generation. In parallel, computational models have provided valuable insights.

Data-driven models based on recurrent neural networks (RNNs) have been used to model the system’s dynamics (Eq. (5)). These RNNs embody abstract representations of the underlying neural circuit while driving movement. For instance, RNNs have been trained to generate dynamics of muscle activity patterns (135, 136). Remarkably, even without being trained to fit neural data, these models can reproduce responses from individual cortical neurons and capture features of the observed population dynamics (135–137). Furthermore, training RNNs on different motor tasks can test hypotheses about the computations performed by distinct motor areas (138, 139). Additionally, RNNs have shown promise in increasing decoder robustness to temporal variations (2, 140, 141). More recently, latent factors have been leveraged to train spiking neural networks to perform two distinct motor tasks (142) and data-driven latent variable models that use time contrastive learning have been shown to have excellent performance for decoding movement in sensorimotor areas (13).

In the past decade, the convergence of advancements in understanding cortical dynamics and powerful machine learning tools has opened exciting avenues for BCIs (119, 141). Incorporating latent factors and their dynamics into decoding algorithms has improved performance (141, 143, 144). For instance, data-driven machine learning has played a critical role in achieving long-term control of a four-limb exoskeleton by a tetraplegic individual using a BCI (145). Transformers, with their ability to be pre-trained on a wide range or motor BCI datasets, offer the potential to enhance BCI adaptability across experimental contexts (146).

Beyond therapeutic applications, BCIs have emerged as powerful tools for studying motor learning and adaptation (120, 147–149). In a typical setup, awake monkeys control a computer cursor using their neural activity while receiving visual feedback of the cursor’s position (150). Unlike traditional motor tasks, where complex and largely unknown transformations convert cortical activity to muscle activity and movement, in a BCI, the decoder characterizes this transformation. By manipulating this mapping, experimenters can study the adaptation required to compensate for the perturbation (150). For instance, it was found that monkeys could learn to compensate for perturbations within the original intrinsic manifold within a session, but took many sessions to learn perturbations that required control in directions outside the manifold (off-manifold) (148). This finding reinforces the idea that although neural state space is high-dimensional, neural population dynamics lives in a low-dimensional manifold reflecting intrinsic constraints.

Beyond intrinsic constraints, the physical properties of muscles, tendons, and bones as well as the sensory feedback streams, play a significant role in determining how neural signals create movement (151–154). Recently, a study in fruit flies has highlighted how the brain controls head movements by adding a bias to proprioceptive feedback loops (155). In a BCI setting in humans, stimulating the somatosensory cortex to mimic tactile feedback has been shown to improve robotic arm control (156). Going forward, we need better models that incorporate biomechanics and biophysics. Advances in biomechanics simulators (157–159) and continuous control learning algorithms (160–162) are being actively combined to further address these challenges (162–166).

Language decoding

The power of machine-learning fueled decoding algorithms has perhaps been most clearly been illustrated for speech decoding, where the action spaces are naturally very high dimensional. Speech is a fundamental mode of human communication, intricate in its orchestration of neural signals and motor actions. The advent of speech neuroprosthetics promise to restore communication capabilities by bridging the gap between neural activity and speech production. Central to understanding speech neuroprosthetics is deciphering the complex neural mechanisms orchestrating speech production. From conceptualization to articulation, speech involves a finely choreographed interplay of neural circuits spanning cortical and subcortical regions. Broca’s area, Wernicke’s area, primary motor cortex, and supplementary motor area are among the key brain regions implicated in speech and language generation. Perhaps the best characterized so far is the motor and premotor cortices in the precentral gyrus (167, 168).

The lateral aspect of the precentral gyrus features a somatotopic organization of orofacial and unique dual laryngeal vocal tract articulator representations (169, 170). Accumulating evidence suggests that the precentral gyrus is not only critical for executing but also for planning speech movements, a function that is commonly mis-attributed to Broca’s area (171). Kinematic analyses have revealed the encoding of dynamical, low-dimensional patterns of speech movements called ‘gestures’ (172). One example is how the specific movements of forward tongue raise and jaw closure are coordinated to create a ‘d’ sound at one local site in the precentral gyrus. The motor cortex contains a complete inventory of speech-related movements to create all of the sounds of a given language (172).

The core objective of speech neuroprosthetics is to decode neural signals associated with speech production and translate them into intelligible output. Traditional approaches have focused on text output and synthetic voice. This entails harnessing BCI technologies coupled with machine learning algorithms to interpret neural activity patterns and reconstruct spoken words or phrases. The first successful demonstration of speech decoding of full words and sentences was carried out in 2021 in a man with severe paralysis after a brainstem stroke (173). The efficacy of speech neuroprosthetics critically hinges on the development of robust decoding algorithms capable of discerning nuanced patterns of neural activity corresponding to different phonetic units. Recent strides in machine learning, particularly deep learning architectures such as RNNs, have propelled the field forward by enabling more precise and efficient decoding of speech-related brain signals.

More recent BCI approaches decode subword linguistic units, such as phonemes or characters, rather than individual words or sentences. This is a common technique in automatic speech recognition where language models — trained to capture the statistical patterns of subword units and words — are used to convert decoded phoneme or character sequences into sentences.

Progress has been rapid with recent demonstrations of fast and large vocabulary decoding. The approaches leveraged RNN models trained to map an input sequence of neural activity to an output sequence of phonemes without the need for any alignment (95, 174). Language models then mapped decoded phoneme sequences into text words and sentences.

Metzger et al. (95) also used a similar approach to decode neural activity into synthesized speech; however, rather than regressing the acoustic mel-spectrogram, they decoded input sequences of neural activity into output sequences of discrete acoustic-speech units. During training, a large self-supervised audio model (HuBert (175)) was adapted to convert target waveforms (generated from a text-to-speech model) into sequences of discrete acoustic-speech units (Figure 5C). During online inference, the decoded discrete acoustic-speech unit sequences were decoded into intelligible sentence-level speech. They first used this voice-conversion approach to personalize synthesized speech for an individual with vocal-tract paralysis using speech samples recorded before the injury (95).

While still in its nascent stages, speech neuroprosthetics hold immense potential for revolutionizing the rehabilitation and communication landscape for individuals with speech impairments (176). Beyond restoring speech functionality, these neurotechnological innovations may pave the way for novel therapeutic interventions targeting a range of neurological disorders affecting speech and language. However, several challenges, including achieving high decoding accuracy, minimizing invasiveness, and ensuring long-term reliability, warrant further investigation to realize the full clinical potential of speech neuroprosthetics.

Towards causal models

Overall, encoding-decoding has been a powerful mathematical framework that enables understanding perception, action, and cognition, as illustrated by several examples. But there are still challenges and opportunities ahead for better understanding the brain and building better BCI decoders.

One major thrust towards this challenge will be coming from foundation models (87). Training large models on diverse data gives rise to stupendous capabilities, and those models can be adapted for many tasks relevant for neuroscience such as behavioral analysis via in-context learning (177), and speech recognition (178). As we outlined above, within neuroscience training large scale models on data from many studies will also open up many possibilities.

Furthermore, we are advocating for advancing models that are mechanistic (biologically plausible) and statistical (can account for neural data) at the same time (Figure 3). Encoding-decoding models lie at the heart of this pursuit. As we have summarized, neuroscience has created many powerful statistical models of brain function for vision, motor control, and language. Beyond neuroscience, advances in machine learning have led to models with high explanatory power grounded in statistical learning theory (179). What all examples have in common is that these models have good (statistical) explanatory power, while being grounded via a mechanism from the lower level(s) of analysis (Figure 1B).

One excellent avenue is exploring the relationship between neural physiology and gene expression, which naturally describe a neuron at different levels, but are linked via biophysical mechanisms such as ion-channel models (42, 44). They created mechanical ion-channel models that can be constrained statistically from transcriptomics data (called NPE-N), thus also linking across scales (Figure 1B). Given that (classically) neuronal cell types are characterized via their anatomical and physiological characteristics, singe-cell transcriptomics continues to refine our taxonomies of cell types (180).

Another interesting example is studying the conditions under which hexagonal firing fields emerge in recurrent neural networks that were trained to path integrate (79). Classical mechanistic models for path integration are built on attractor models (181). The connectivity of these models is designed to allow path integration. Remarkably, the normative models learned similar mechanisms to represent the location of animals (via center-surround connectivity), and to integrate movement updates (via shift circuits). They also provide a mathematical mechanism, grounded in pattern formation theory, why hexagonal tuning curves emerge in the abstract ANN models (79).

How might we continue to push further? Combining statistical models and mechanistic models may require new mathematical tools. Relevant approaches are being developed in other fields such as in causal machine learning. Schölkopf and von Kügelgen (182) propose using causality to build systems that do not only rely on statistical dependencies of data and that generalize better to out of distribution examples. In particular, causal representation learning aims to extract the relevant causal variables from data and learn representations that contain not just statistical information but support interventions, reasoning and planning. We can think of this approach as an extension of the classic latent variable analysis. Indeed, just as with latent variables, we can apply a decoder to the causal variables to extract the relevant behavioral variables. However, causal variables also come with the corresponding mechanisms and a causal graph that capture the relationships between each other. Concretely, let X₁, …, X_d be some experimental variables (with some of them corresponding to neural data K for different brain areas, others to sensory inputs, and motor outs), they possess a causal factorization:

$$p\left(X_1,\dots ,X_d\right)=\prod _{i=1}^{d}p\left(X_i\mid P{A}_i\right),$$ (6)

where PA_i denotes the direct causes of X_i and p(X_i|PA_i) represents the mechanisms of the model. Although learning this causal factorization from the recorded data is a challenging task, if we have some knowledge of the causal structure (which variable is caused by which) then only the mechanisms need to be inferred (183). As the relevant behavioral variable x can be decoded from the causal variables x = g(X₁, …, X_d), the causal factorization can ultimately link the mechanisms with behavior. Although, this approach is promising and has already generated concrete tools for machine learning (causal auto-encoders, selfsupervised causal representation learning or independent mechanism analysis), it might take further refinements to apply to the field of neuroscience (184, 185).

Another avenue to explore is merging learning dynamical systems (such as ODEs or PINNs, see below) with latent variable models. Historically, in latent variable models, the dynamical system has been a priori ascribed to the generative model (50). One notable exception is CEBRA (13), where the authors explore the implicit dynamics of the learned latent variables, but do not explore the explicit equations that best model the latents. A potential challenge could be having several set of differential equations (solutions), even for these theoretically identifiable models.

Here, we can turn to a recent success within the machine learning field regarding modeling dynamical systems via neural ODEs (N-ODE) (45). Yet, despite N-ODE’s success in fitting temporal data for various applications like forecasting, control, and system identification, the approach lacks interpretability, as one simply gets a neural network that fits the data.

Another important direction is building knowledge into neural network models, e.g., via physics-informed neural networks (PINNs), which offer a different approach to bridge the gap between data-driven decoding of brain activity and understanding the underlying mechanisms (186, 187). Unlike traditional black-box models (in machine learning), PINNs can incorporate physics knowledge, such as biomechanics or neural dynamics, as constraints during training. By combining the data-fitting power of ANNs with physical constraints, PINNs are guided towards solutions that align with real-world principles. This not only enhances model interpretability but also allows PINNs to achieve good performance even with limited data. Furthermore, these physical constraints promote better generalization to unseen scenarios and offer mechanistic insights into the decoded information by revealing the potential processes at play. Demonstrating remarkable success in modeling physical systems (188–190), and, more recently, biological processes (191, 192), PINNs hold promise for understanding neural mechanisms. For instance, recent work has developed models of muscle spindles that can accurately predict neural dynamics, while also including biophysical mechanisms (193). By bridging the gap between biomechanics and neural dynamics, this model offers a comprehensive understanding of muscle spindle function as adaptable signal processors in sensorimotor control.

Historically, differential equations provided the gold standard for understanding a system. An alternative method is symbolic regression (SR), which provides human-readable mathematical expressions directly from data, and has recently shown some success in (re-)discovering natural laws (194, 195). SR has mostly been used for discovering f(x) from paired observations (x,f(x)), but has been extended for dynamics $(\text{ODE}\dot{x}=f(x))$, even with multi-modal and noisy data (68). Here, transformers can be trained to predict the differential equation from raw data alone (68). This approach, called ODEFormer, reaches state of the art performance on a number of classic problems such as Lotka-Volterra, a binocular rivalry model, a cell cycle model or a Van der Pol oscillator (68).

Ultimately, we would like to build meaningful “standard models” for causal understanding that link statistical firing properties of neural populations to their underlying mechanisms. A formidable goal would be to develop the neural dynamics equivalent to gold standards in physics, such as Newton’s laws of motion.

Conclusion

Where will we be in 50 years?.

The progress of machine learning in the last year alone makes it a daunting task to look 50 years ahead. Yet, the future certainly will be a place where we can build powerful causal models that predict across scales. Thus, we must consider how the framing of causal encoding-decoding can be natively built into foundation models for (Neuro)Science. Current efforts to build foundation models revolve around tokenizing a multitude of datatypes, leveraging GPT4-style (inferred) training with human alignment and reinforcement learning. The goal is to build good “next prediction” models for all these datatypes in relevant downstream tasks. However, the future might dig deeper: can we build fully mechanistic models that predict, behave, or even show signs of cognitive, compositional thinking? Perhaps this hinges on what the ultimate tasks will be for GPT-42 (or such). This answer will lie in the values of society 50 years from now, and our understanding of the neural mechanisms that give rise to action, perception, and cognition.