Active inference and artificial reasoning

Abstract

This paper considers the sampling of outcomes that provide the greatest amount of information about the structure of underlying world models. This generalisation furnishes a principled approach to structure learning, and specifically rule discovery, under a plausible set of generative models or hypotheses. In active inference, policies—i.e., combinations of actions—are selected based on their expected free energy, which comprises expected information gain and value. Information gain corresponds to the Kullback-Leibler divergence between predictive posteriors with, and without, the consequences of action. Posteriors over models can be evaluated quickly and efficiently using Bayesian Model Reduction, based upon accumulated posterior beliefs about model parameters. The ensuing information gain can then be used to select actions that disambiguate among alternative models, in the spirit of optimal experimental design. We illustrate this kind of active selection or reasoning using partially observed discrete models; namely, a three-ball paradigm used previously to describe artificial insight and aha moments via (synthetic) introspection or sleep. We focus on the sample efficiency afforded by seeking outcomes that resolve the greatest uncertainty about the world model, under which outcomes are generated.

Humans are remarkably adept at learning from sparse data, in part because they selectively sample information that is most informative about the rules governing their environment. Here, the authors demonstrate that uncertainty about the structure of one's internal model can be used to guide behavior, facilitating the efficient discovery of hidden rules.

Introduction

Active inference offers a first principles approach to sense and decision-making under generative or world models. It distinguishes itself from (model-free) reinforcement learning—and related approaches in machine learning—in several respects. First, it draws a clear distinction between the unknown variables entailed by a generative model; namely, latent states, parameters, and structure, c.f.,¹. This calls for inference over the three kinds of unknowns, leading to active inference, (parameter) learning, and selection, respectively². Crucially, latent states can include an agent’s action or policies, leading to planning as active inference, c.f.,^3,4. This speaks to another distinction between active inference and learning state-action policies; in the sense that planning rests on free energy functionals of beliefs about states, as opposed to value functions of states per se. This means that active inference optimises (Bayesian) beliefs about latent variables. A key consequence is that an optimal policy resolves uncertainty—an attribute of a Bayesian belief—through exploratory, information-seeking actions that cannot be cast as optimising a (value) function of states.

The ensuing information-seeking is underwritten by the information gain expected under any given combination of actions, i.e., policy. The expected information gain over states is often referred to in terms of salience, corresponding to notions of Bayesian surprise in the visual search literature and accompanying salience maps^5,6. In robotics, this kind of motivation is sometimes referred to as intrinsic—c.f.,⁷—that manifests as artificial curiosity⁸. When applied to the parameters of a generative model, the expected information gain is sometimes referred to as novelty; namely, the epistemic affordance offered by resolving uncertainty about the parameters of a generative model^9,10. In this paper, we introduce a third sort of information gain that pertains to the structure of the generative model^11,12. Equipping agents with this kind of intrinsic motivation is in line with the principles of optimum Bayesian design^13–15; namely, designing an experiment to elicit data that maximally disambiguate hypotheses or generative models.

The ability to select observations that minimise uncertainty or ambiguity is a hallmark of natural intelligence and self-organisation of certain natural kinds¹⁶. One could argue that such behaviours rest upon inference—i.e., variational free energy minimising processes—that distinguish between artificial intelligence and natural intelligence; sometimes cast in terms of System one versus System two thinking, e.g.,¹⁷. One might further argue that the homologue of ‘thinking’ in active inference is the planning (as inference) in which counterfactual futures are evaluated under plausible policies. This evaluation is in terms of the (path integral of) expected free energy that comprises expected information gain and expected value¹⁶. Value—in the setting of active inference—corresponds to the log probability of a priori preferred outcomes. Expected information gain or epistemic value scores the degree of belief updating an agent expects under each policy. It is the expected information gain that is beyond the reach of conventional learning schemes because the information gain depends upon posterior beliefs that, by definition, are time, context, and experience dependent. In other words, there is no learnable mapping from observations to expected free energy—and implicitly policies—because the mapping per se changes with learning. In short, although one can learn to infer, e.g.,¹⁸, one cannot learn to be curious. Although it is possible to amortise (i.e., learn), time-independent mappings from observations to posteriors over states, one cannot learn a time-dependent mapping from a posterior over states and parameters to expected free energy. This is because the mapping changes with learning. Heuristically, one cannot learn to do something one has never done before. Having said this, it is perfectly possible to learn epistemic habits¹⁹; e.g., sitting down to watch the evening news. So, does planning with epistemic intentions provide a sufficient account of reasoning?

One would probably argue no. Reasoning—especially as used in the current large language model literature²⁰—implies a degree of recursive introspection that, in the active inference literature, is better considered in terms of Bayesian model reduction. Bayesian model reduction is a fast and efficient form of Bayesian model selection that allows one to evaluate the evidence (i.e., marginal likelihood) of alternative models based upon the prior and posterior of some full or parent model^21,22. It is usually implemented offline as a variational free energy minimising process of the sort that has been proposed to occur during introspection or sleep^23,24. One could imagine this manifesting biologically through the dynamic formation and termination of connections between dendritic spines and axonal boutons^25–27.

Technically, variational free energy is model complexity minus accuracy, where complexity is the KL divergence between the posterior and prior²⁸. Active inference and learning therefore seek the posterior that minimises the divergence from prior beliefs, while providing an accurate account of observed data. However, after all the data have been observed, one can minimise variational free energy by selecting a prior that minimises the divergence from the posterior. This post-hoc selection is generally from a set of priors—over model parameters—that constitute a space of models or hypotheses. By post-hoc, we mean that this form of model selection takes place after data assimilation into an existing model. This is because Bayesian Model Reduction allows for assessment of the evidence for—and posteriors one would have obtained—under a ‘reduced’ version of the model, without having to repeat the process of model fitting. However, it requires the posterior parameters estimated from a ‘full’ model and can only be applied post-hoc. One can regard this form of selection as the basis of the scientific process in the following sense: a scientist elaborates a small model space (e.g., an alternate and null hypothesis), acquires some data under uninformative priors over her model space, and then finds the hypothesis that best explains her experimental data^13,29. On this view, the best model is that which renders the data the most likely; namely maximises model evidence or marginal likelihood.

One can generalise this post-hoc variational free energy minimising process to any number of hypotheses—or reduced models—that can be specified in terms of priors over model parameters. Having identified the best hypothesis or model—following some suitable period of evidence accumulation—an agent can proceed with an informed and optimal model structure. In what follows, we will refer to this form of structure learning^30,31 as applying Occam’s razor. This application of Bayesian model reduction has been used to simulate ‘aha moments’ and ‘abstract reasoning’ using a three-ball paradigm, in which (synthetic) subjects had to discover the rule underlying ‘correct’ choices²³. Here, we reprise this paradigm to demonstrate active reasoning or reduction, where the focus is on selecting the right data—c.f., performing the right experiments—that resolve uncertainty about the models or hypotheses entertained. For the purposes of this paper, we will read ‘reasoning’ as selecting the best explanation for observations under competing hypotheses about how observations are generated. Active reasoning can then be read as seeking observations that best disambiguate competing hypotheses for subsequent ‘reasoning’.

We will see that the requisite expected information gain can be computed quickly and efficiently—using Bayesian model reduction—and used to supplement the expected free energy that subtends action selection. This extension of active inference, to cover abstraction—in the sense that a rule is an abstraction of the behaviour of a system, applicable to multiple specific instances of that system—or reasoning³², was inspired largely by the ARC-AGI-3 challenge (ARC-AGI-3); namely, the challenge to discover a rule, symmetry, invariance, law, or abstraction for which no prior information is available. The procedures described in this paper address part of this challenge by describing an efficient way of querying or acting upon the world to discover the rules or contingencies in play. This is only a partial solution to the larger challenge, in that we assume that the agent has already learned a generative model of the puzzle or game at hand but does not know what constitutes a correct response. However, knowing the latent cause-effect structure of the game admits a set of hypotheses or models that conform to some symmetry constraints. In particular, we will consider hypothetical likelihood mappings from latent causes to successful outcomes that feature certain isomorphisms. This allows the agent to successively eliminate unlikely mappings—through active reduction or reasoning—until the rule is discovered. The resulting scheme shares key commitments with probabilistic inductive logic programming^33,34; especially, the distinction between parameter and structure learning. However, its focus is on the sample efficiency of the implicit inductive reasoning.

Our focus in this paper is on artificial reasoning, as opposed to an explicit neurobiological theory of reasoning. Having said this, the theoretical contributions of this paper rest upon a biologically grounded framework derived from neuroscience^35–37, and it will be interesting to see how this applies more generally in cognitive neuroscience. Although we defer a more thorough treatment of this interesting and richly studied topic [see³⁸ for an overview] to future work, we will briefly touch upon this in the discussion. We also note that a core contribution of this paper is the use of action to sample a small number of well-chosen datapoints sequentially to best resolve uncertainty about model structure. This is much like the foveal sampling that natural selection has identified as an efficient means of solving the same problem. Reasoning, as we use the term here, is the process of optimising a model structure to effectively discover rules. The role of active inference is to combine beliefs about model spaces—with priors capturing the relevant symmetries and constraints—with information seeking through action.

The specific theoretical contributions of this paper are as follows:

• One can engage in active structure learning by using expected information gain about model structures to guide epistemically motivated behaviour. This dramatically extends the applicability of active inference to the domain of structure learning and offers an efficient means of data-selection.

• Use of Bayesian Model Reduction techniques allows one to do this efficiently by evaluating a model average and then post-hoc selecting between the models contributing to that average without having to reinvert the original model.

• One can further improve the efficiency of this procedure through a principled formulation of the model space to be evaluated. This will vary somewhat between different applications. However, in general, all possible models are not equally plausible. We illustrate how one can exploit the symmetries implicit in sensory and action modalities to constrain this prior model space and thereby ensure tractability.

Our methods section provides a brief review of belief updating and propagation under the partially observed (categorical) Markov decision processes used in the numerical studies below. This review focuses on the expected information gain over models that induces active reasoning and how this component of expected free energy can be evaluated simply and automatically using Bayesian model reduction. The ensuing scheme is applied to the three-ball paradigm described in the first part of the results section. This paradigm has several features that call for the epistemic foraging. In particular, the problem is partially observed; in the sense that the agent can only see the colour of each ball by selecting it (e.g., foveating the location of each ball). In turn, this calls for sequential policy optimisation that would elude conventional (e.g., reinforcement learning) schemes that seek state-action policies. This subsection simulates Bayes-optimal behaviour, in the dual sense of optimal Bayesian decision theory and experimental design. The latter part of the results section repeats the simulations with and without various components of expected free energy to characterise their contribution to sample efficiency and performance. The paper concludes with a brief discussion of what is, and is not, covered by this application of active inference.

Results

The three-ball paradigm

The three-ball problem was originally used to introduce active learning and model selection as a mathematical homologue of insight and ‘aha moments’ that evince (artificial) curiosity²³. Here, we use a nuanced version of this paradigm to demonstrate the active sampling of data that renders Bayesian model reduction as efficient as possible, through supplementing expected free energy with expected information gain over models (Table 1).

Table 1.

Glossary of terms

Term	Description
State	Latent or hidden states are unobserved variables from which observations may be predicted. These are typically dynamic and may evolve from one time-step to the next as governed by transition probability matrices. For the purposes of this paper, they are assumed to be categorical variables, each taking on one of a countable number of discrete values.
Observation	Observations are the sensory data (or outcomes) one can use to draw inferences about the states, models, and parameters that cause them. Again, for the purposes of this paper, these are assumed to be categorical variables.
Factor	States may be structured according to distinct factors. This greatly simplifies a state-space in that one can represent different kinds of variable independently of one another, as opposed to having states representing all possible combinations of variables. For instance, one might represent one’s location along discretised North-South and East-West factors, rather than having a single state factor that can take on the role of any combination of these.
Modality	We will use the term ‘modality’ as the observational analogue to state factors. This reflects the notion of sensory modalities (vision, audition, etc) in biology.
Path	Paths index, for each state factor, sets of transition probabilities (matrices as subtensors of transition probability tensors) that determine alternative probabilistic trajectories for that variable.
Policy	Policies are combinations of controllable paths and therefore can have implications for the paths followed across multiple state factors.
Action	An action is the effect of selecting among alternative (controllable) paths at a given point in time.
Parameter	Parameters are a subset of latent states that do not vary dynamically (although inferences about parameters will do so). Typically, in this setting, parameters are Dirichlet distributed variables that determine prior or conditional probabilities of states, paths, and observations.
Belief	Beliefs are probability distributions describing states, parameters, and model structures. When used without qualification, they will tend to refer to (marginal) posterior probabilities.
Inference	Inference is the process of optimising (i.e., updating) beliefs about states to extremise variational free energy—thereby achieving (inference to) the best explanation as to the causes of one’s observations.
Learning	Learning is inference about either parameters (parameter learning) or models (structure learning).

In brief, an agent is presented with three coloured balls and invited to choose a colour. If the agent makes a choice (e.g., through a button press), then it receives feedback that it was correct or incorrect. The agent’s prior preferences are set such that the relative log probabilities of no feedback, correct and incorrect are 0, 2 and −6 natural units, respectively. These are specified in much the same way one might define value functions, with greater probability assigned to outcomes one expects to (behave to) achieve and a low probability assigned to outcomes one expects to (behave to) avoid. In other words, it prefers to receive correct feedback and finds an incorrect outcome highly surprising and therefore aversive. The rule that the agent has to discover—to secure preferred outcomes—is context sensitive: if the centre ball is green, then the correct colour is green. However, if the centre ball is red the correct colour is on the left, while if the centre ball is blue, the correct colour is on the right. This rule is installed in the likelihood mapping between latent states encoding the colours of the three balls—and the colour of choice—to the feedback modality. Therefore, the agent must learn and select the likelihood tensor that encodes the contingencies constituting the rule. So, how might the agent discover this rule?

If the agent could see all three balls at the same time, then it would be a relatively straightforward matter to learn the rule using reinforcement learning; e.g.,³⁹. For example, there are only 3×3×3×4 = 108 combinations of ball colours and responses (i.e., three colour choices and a ‘no response’ option). This means that after 500 training trials, the probability of seeing at least one instance of every combination is at least 99% =1 – (1 – 1/108)⁵⁰⁰ and the likelihood mapping could be learned almost certainly. However, problems like the three-ball paradigm preclude a learnable state-action policy because the agent can only see one ball at a time, depending upon where it is looking. This converts a Markov decision problem into a partially observed problem, where the agent not only has to choose the correct colour but also choose where to look.

Interestingly, the foveation mechanism in our setup means that one needs a degree of ‘postdiction’ or smoothing such that inferences about past states can be recontextualised by new information. This would be particularly important in deep hierarchical models, where retrospective inferences about initial states determine ascending messages⁴⁰ [although, see for contrast⁴¹ in which access to the entire visual field at all times precludes a need for backwards smoothing]. The model selected through the procedure outlined here will have profound implications for this kind of postdiction—particularly as increased confidence in context state factors would retrospectively influence confidence in criterion states.

This ubiquitous kind of problem calls for sequential policy optimisation. Intuitively, knowing that there is a red ball on the left does not specify the correct response because the centre ball could be green or blue. This means that the optimal policy is to first resolve uncertainty about the context (by looking at the centre ball) and respond immediately (if it is green) or look to the right or left (if it is blue or red). And then choose the observed colour. The ensuing problem can now be specified with the following generative model.

The latent states of this model are fivefold. There are three sorts of states (i.e., factors) encoding the colour of the ball at each location (we will refer to these as location factors). The remaining (controllable) factors are where the agent is looking (start, left, centre, or right) and the agent’s choice (no choice, red, blue or green). These hidden states are sufficient to generate two outcome modalities; namely, a single visual modality (seeing nothing, a red, green or blue ball) and a feedback modality (no feedback, correct, incorrect). The rule is encoded in the likelihood tensor ${\mathbf{A}}^g\in {\mathbb{R}}^{3\times 3\times 3\times 4\times 4}$ mapping from hidden states to the feedback modality. The transition priors for this model are very simple. There are no transitions among the colours of the balls at each location within a given trial, so the transition tensors are identity matrices ${\mathbf{B}}^f=I\in {\mathbb{R}}^{3\times 3}$. The controllable or action-dependent factors depend upon action, ${\mathbf{B}}^f\in {\mathbb{R}}^{4\times 4\times 4}$; e.g., looking to the left moves the location state to left, and so on. Similarly for the choice factor. On each trial, the colours of the balls are selected at random from ${\mathbf{D}}^f\in {\mathbb{R}}^3$ with the agent looking at the start location. Policies are selected on the basis of expected free energy and prior preferences over the feedback modality are specified by ${\mathbf{c}}^g=\sigma \left(\left[0,2,-6\right]\right)\in {\mathbb{R}}^3$. The agent is allowed to forage and choose for 6 time points, after which a new trial starts, with randomly selected ball colours.

Figure 1 illustrates the behaviour of an agent equipped with the above generative model and the correct rule. This figure reports belief updating and choice behaviour in terms of the posterior densities over latent states and policies, in image format. In this example, the colour of the centre ball was red, meaning the correct choice was the colour of the red ball on the left. With informative preferences and active inference—looking two moves into the future—the agent first resolves uncertainty about the colour of the informative ball by looking from the start location to the centre ball. Having resolved its uncertainty about the context in play, it looks at the left ball to infer its colour. Following this, it then chooses the red option for the remaining moves, thereby maximising the number of preferred (i.e., correct) outcomes. However, while continuing to choose red, it resolves further uncertainty about the uninformative ball on the right. In other words, it continues visual exploration at the same time as exploiting its posterior beliefs about the correct response. The question we now address is: what would happen if the agent did not know the rule? And what kind of prior should it use in the absence of this knowledge?

Fig. 1. Choice behaviour.

This figure reports belief updating for a single trial under the three-ball paradigm. The panels on the upper left report posterior beliefs in image format over five latent state factors and six time points. Two of these latent factors are controllable and are equipped with posteriors over paths, with four levels each. These posteriors are shown on the upper right panel. The lower left panel reports the outcomes solicited by the agent. Of these outcomes, the third (feedback) outcome has informative prior preferences, such that the agent prefers to be correct or right and finds being incorrect or wrong highly surprising. Darker colours indicate higher probability and the dots represent the actual states and outcomes. The lower right panel illustrates the response in terms of where the agent was looking (Visual search) and choice behaviour (Choices). In this example, the agent first looks to the centre location and observes the colour is red. It then knows that the informative (criterion) location is on the left. Looking to the left, it observes a red ball and subsequently chooses red. The agent then looks to the right to resolve the remaining uncertainty about the colour of the right ball. Once the correct colour has been inferred, the agent continues to choose this colour, to maximise preferred feedback.

For simplicity, we will assume that the generative model has been learned by the agent, with the exception of the likelihood tensor mapping from latent states to the feedback modality. It is this likelihood mapping that contains the rule the agent has to discover. So, what alternative likelihood mappings could the agent entertain?

When specifying (or inheriting) priors over the parameters of a generative model (see Fig. 2), one is implicitly specifying a set of reduced models that can be derived by removing parameters from the full prior. Alternatively, one could argue that the initial priors are, effectively, a mixture—or Bayesian model average—over reduced models that all satisfy some constraints. But what constraints? One can now turn to the observation that to maximise model evidence (or minimise expected free energy)—with respect to model parameters—is to maximise mutual information (under constraints: see Eq. 3 in Methods). This is satisfied when each column of the likelihood tensor has a unique nonzero entry. One can demonstrate this by noting that mutual information between states and observations (given likelihood parameters) can be decomposed into the prior entropy of states minus the conditional entropy of states given observations. Only the latter depends upon the likelihood parameters. This means the maximum mutual information is obtained when the conditional entropy of states, for any plausible observation, is zero—i.e., when there is a single plausible state for each observation. With multiplication by the prior vector for states and renormalisation, one can attain the sparsity constraints for the likelihood matrix (or tensor). This also ensures maximum sparsity, in the sense that the number of zero entries attains its supremum.

Fig. 2. Plausible rules.

This figure is designed to give a graphical representation of the steps taken to limit the space of models under which only plausible rules are in play. The graphic on the left shows the relative number of plausible rules (with the area of the circles scaling with the log number of rules) following application of each new constraint. The graphics on the right illustrate the kinds of rules included in each constraint. The initial sparsity constraint only allows for rules in which the feedback predicted is definitively positive or negative for any combination of choice and criterion states. The injectivity constraint says that, when the feedback is positive, there is an injective mapping from criterion to choice states. The isomorphism constraint says that one of the criterion factors that contains the same annotations (i.e., ‘red’, ‘green’, ‘blue’) as the choices. However, the specific factor (left, centre, or right cue) is here determined by some unknown context which may depend upon all three. Finally, the context constraint says that one of the criterion states (shown here as the centre state) provides all the contextual information required to determine the relevant criterion factor (shown here as the left state) that injectively and isomorphically maps to the choice state that ensures the feedback is positive. Please see the main text for more detail and pseudocode illustrating this.

Sparsity is perhaps the simplest constraint, where all models feature a unique mapping from every combination of latent states to some response; e.g., correct or incorrect. This could be a minimal definition of a rule. For example, if we consider the likelihood tensor as a joint distribution over latent states and outcomes, one can specify priors by conditioning on a correct outcome and consider the structure of the corresponding subtensor: e.g., ${\mathbf{A}}_{correct}^g\in {\mathbb{R}}^{1\times 3\times 3\times 4\times 4}$. In the current example, there are 3×3×3×4×4 = 432 combinations of latent states that can be assigned a concentration parameter of one or zero, depending upon whether any given combination leads to a correct or incorrect outcome, respectively. This would lead to 2⁴³² models or rules. Clearly, this would be a weak kind of a rule, in the sense a rule should compress explanations for observations into a small set, in the spirit of minimising algorithmic complexity or description length: e.g.,^8,42–44. In short, sparsity is a necessary but not sufficient condition for maximising mutual information. To maximise the mutual information, it is necessary for some combinations of hidden states to generate a correct outcome and others not. Beyond this, the mutual information constraint tells us little about plausible rules.

However, the latent states have a factorial structure that distinguishes between states that can be controlled or chosen and those that cannot. We can refer to these as choice and criterion states, respectively. One could consider a rule to implement the constraint that, for any combination of criterion states, there is a unique choice state that generates correct feedback. This guarantees maximum mutual information. For example, there are 3×3×3 = 27 combinations of criterion (i.e., location) states, each of which can be associated with one of 4×4 = 16 choice states. This reduces the number of rules to 16²⁷ = 2¹⁰⁸. So, can we go further and impose more symmetries to define a model space?

One can now consider symmetries among factors and hypothesise an isomorphic mapping between a choice and criterion state if, and only if, they share the same support or domain. For example, here, the three location factors (left, centre and right) share annotated states with the choice factor; namely, (red, blue and green). This isomorphism may or may not depend upon the context; namely, a combination of criterion states. In other words, for every feedback outcome, there is a slice of the likelihood tensor (chosen such that one dimension is taken from the choice factors, and the other from the criterion factors) that has the form of an identity mapping. This, in the current example, leads to 3×3×3 = 27 contexts that could specify one of three criterion states that would support an isomorphism with a choice state. This constraint gives 3²⁷ models, which is still too large. So, can we turn to weaker—i.e., sparsity—constraints?

A weaker symmetry or isomorphism pertains to the size of hidden state factors. Here, there are several context states that could uniquely specify a criterion state, in virtue of the fact that the size of each context state equals the number of criterion states. Recall the context states are the set of criterion states and that, in our setup, there are 3 criterion state factors comprising the context with 3 possible values each of the context states might take (red, green, or blue). One could now hypothesise that a context state specifies a unique criterion state, thereby satisfying a sparsity constraint. In the current example, there are three possible criterion states (left, centre, or right), each of which could be specified by one of three context states (red, green, or blue) in 3³ ways (e.g., left red → left, left red → centre, …, centre red → left, centre red → centre, …, right blue → right). This leads to 3×3³ = 81 possible rules (3 possible values the indicated criterion state factor might take by the 3³ ways of specifying a mapping from context to criterion). Some of these rules will be degenerate, in the sense there are only 79 unique rules. These symmetry constraints now provide a plausible hypothesis space, $\mathrm{M}$ that (i) jointly specifies a full prior [see Eq. 8], (ii) affords the opportunity for active reduction or reasoning via expected information gain [see Eq. 10] and (iii) subsequent model selection [see Eq. 12].

Generally speaking, rules or likelihood models can be specified automatically—in terms of Dirichlet tensors—by identifying a (controlled) choice factor that shares more than one annotated state with (noncontrolled) criterion factors. One then identifies candidate context factors whose levels can uniquely specify the criterion factor. Finally, for every n×m^m combination of n context and m criterion factors, a putative likelihood mapping is generated by assigning a Dirichlet account to a correct outcome, whenever the choice matches the criterion state specified by the context state. Algorithmically, this can be expressed as follows, where, $s_i^f$ denotes the level of factor f under the ith combination of states:

For n context states and m^m specifications of m criterion states

            For every combination of states, i

$$\begin{array}{c}\hfill criterion=s_i^{context}\hfill \\ \hfill a_{correct,s_i^1,s_i^2,\dots }^{feedback}=\left\{\begin{array}{cc}\hfill 1\hfill & \hfill s_i^{choice}=s_i^{criterion}\hfill \\ \hfill 0\hfill & \hfill s_i^{choice}\ne s_i^{criterion}\hfill \end{array}\right)\hfill \\ \hfill a_{incorrect,s_i^1,s_i^2,\dots }^{feedback}=\left\{\begin{array}{cc}\hfill 0\hfill & \hfill s_i^{choice}=s_i^{criterion}\hfill \\ \hfill 1\hfill & \hfill s_i^{choice}\ne s_i^{criterion}\hfill \end{array}\right)\hfill \end{array}$$

            end

end

For simplicity, we have omitted some housekeeping details when generating model spaces. For example, it is necessary to supplement the likelihood mappings with contingencies that fall outside isomorphisms. In the current example, one has to add Dirichlet parameters mapping from the no choice to a no feedback outcome. This is straightforward to implement automatically by ensuring every column of the likelihood tensor has at least one entry. The ensuing model space produces rules of the form:

“If context is [context state] then the [criterion state] is the correct choice”:

In this instance, rule 65 was the rule used to generate feedback:

If centre is red then the left state is the correct choice:

if centre is green then the centre state is the correct choice:

if centre is blue then the right state is the correct choice:

In this form, rule discovery can be seen as probabilistic inductive logic programming (PILP)^33,34. Probabilistic program induction logic combines inference with first-order logic to learn probabilistic logic programs from data. This entails learning the structure (rules) and parameters (probabilities) of a program, simultaneously or as separate subproblems. Here, the expected information gain over models renders structure learning as efficient as possible, while the expected information gain over parameters ensures efficient parameter learning.

Above, we have considered the likelihood mapping to a modality that underwrites relatively sparse, uninformative feedback. This kind of feedback is typically encountered in reward learning paradigms (where correct and incorrect can be associated with reward and punishment). Crucially, sparse rewards are only delivered when the ‘stars are aligned’. On this view, the rules correspond to different ways in which the stars (i.e., states) could be aligned (i.e., in an isomorphic fashion). It is interesting to consider the equivalent generation of model spaces for more informative modalities. For example, one could have used the isomorphism between the levels of the ‘where’ factor and the locations of the three balls—in conjunction with the isomorphism between the location and outcome colours—to specify the likelihood mapping to the first (visual) modality. In this instance, the location factor would correspond to the context state that specified which of the three criterion states generates outcomes. Here, there is only one model that satisfies these isomorphic constraints, and therefore no need for model selection. Interestingly, this kind of context sensitive model structure is ubiquitous, in the sense that it rests upon a factorisation into ‘what’ (i.e., criterion) and ‘where’ (i.e., context) resulting in the ability to generate the answer to “what would I see if I looked over there?”⁴⁵.

Clearly, committing to the above (isomorphic) rules does not offer a universal way of specifying a model space. For example, we could consider the context state to be specified by another context state leading to 3rd order hypotheses. Or, we could have added a further constraint that each context state specifies a unique criterion state, leading to 3×3! = 18 possible rules. However, this would preclude simple (first-order) rules; i.e., a fixed criterion state. From the perspective of the contextual rules, a first-order rule simply means the criterion state is specified when the context state is in the first level OR the second OR the third, and so on. For example:

If centre is red OR green OR blue then the left state is the correct choice

One could also consider constraints afforded by multiple choice factors to increase the size of the model space, if deemed plausible a priori. For example, we could have considered rules of the form in which a correct response can only be obtained when looking at the start location:

If centre is red AND location is start, then the left state is the correct choice:

If centre is green AND location is start, then …

In summary, under the hypothesis that correct feedback depends upon context-sensitive isomorphisms, the three-ball paradigm offers 81 possible likelihood mappings between latent states and a feedback modality. This set of likelihood models contains the true rule, which has the form: if the state of some contextual factor is this, then the state of the implied criterion factor determines the state of the control factor that satisfies the rule. In this instance, the colour of the central ball determines the location of the ball whose colour corresponds to the correct choice. By construction, this is a relatively simple rule that involve objects having different attributes, such as colour and position. In other words, we did not have to worry about isomorphisms between the location, pose or colour of objects; however, in principle, similar isomorphic constraints can be applied to more expressive generative models. In what follows, the ensuing priors over models are used to simulate artificial reasoning in which an agent has to discover the rule.

Artificial reasoning

This section repeats the simulation in Fig. 1 but starting with prior beliefs—about the likelihood model that entails the rule—based on the Bayesian model average of the above rules. Figure 3 shows the results of 64 trials, each comprising six time points (i.e., five moves), starting with Dirichlet priors over the feedback likelihood based upon the Bayesian model average in Eq. 8, normalised to a sum of one. Initially, policies—i.e., combinations of eye movement and choice—are evaluated with uninformative prior preferences based upon their expected free energy. This means that visual search behaviour is driven to resolve uncertainty about the colours of the balls at each location, and which of the 81 rules are in play. This means the agent looks at every location and chooses different colours to see whether the outcome is correct or incorrect. As trials progress, the agent accumulates Dirichlet counts in the likelihood mapping, effectively accumulating evidence that speaks to the contingencies mapping from latent states to outcomes. In turn, these are used to evaluate the expected information gain over models using Bayesian model reduction, starting from the initial or full priors.

Fig. 3. Active reasoning.

This figure reports belief updating and subsequent model selection over 32 trials of the game reported in Fig. 1. Each trial starts with randomly selected ball colours, which serve as instructional or contextual cues that enable the agent to resolve uncertainty about the nature of the game. In particular, the agent starts with (81) hypotheses about the likelihood mapping between latent states and the feedback modality and accumulates evidence for or against these hypotheses until one is selected (on trial 18). The upper panel reports the confidence the agent has about its responses (i.e., the negative entropy or precision of beliefs over policies). The second panel shows the outcomes. Note that prior to model selection, the agent chooses as many outcomes as it can; irrespective of whether they are correct or incorrect (blue and red lines, respectively). However, after it has committed to a particular model (here, the correct model, 65) it avoids soliciting surprising or aversive outcomes. It can do this because it now has a precise knowledge of the rule, as shown in the lower panels. The posterior and true likelihoods represent a slice of the likelihood tensor reporting correct outcomes over all combinations of latent states. Following model selection, the posterior likelihoods converge on the true likelihoods and the divergence (between the two) in the third panel falls to zero. At this point, the distribution over models (model comparison) becomes definitive, as reflected in a threshold crossing of Occam’s razor, scoring the log odds ratio of the winning model being more likely than the remainder.

As evidence accumulates, certain hypotheses are rejected, and the remaining plausible rules are successively compared and eliminated. When the agent is sufficiently confident that the most likely model supervenes over all others, it applies Occam’s razor: see Eq. 13. In other words, it replaces its posterior beliefs about the likelihood mapping with the mapping selected on the basis of Bayesian model reduction. At this point, informative preferences are adopted and the agent plans, with greater confidence, two steps into the future. This point can be regarded as an ‘aha moment’ or the dénouement of some data gathering that licences a commitment to a particular hypothesis. This transition is illustrated in Fig. 3 in terms of the confidence with which the agent executes its policies. Here, confidence is simply the precision or negative entropy of posterior beliefs over policies. Interestingly, this precision has been used as an explanation of dopaminergic discharges in human neurophysiology^46,47.

A marked change in behaviour accompanies this implicit rule discovery or insight. The second panel of Fig. 3 demonstrates this in terms of outcomes. During epistemic foraging—to solicit the most informative outcomes—the agent almost makes a choice on every move, which is either correct or incorrect. Both outcomes inform posterior beliefs over rules or hypotheses. However, once uncertainty about the rule has been resolved, the agent secures the maximum number of correct feedbacks, and completely avoids the aversive incorrect outcomes. Note that the maximum number of correct outcomes is either three or four, depending upon whether the central colour is green or not: if the central colour is green, the agent knows to choose green on the next move; otherwise, it has to spend an additional epistemic move looking at the criterion location.

During evidence accumulation in the first 18 trials, posterior beliefs about the likelihood mapping remain far from the mapping ultimately selected; although there is a slow reduction in uncertainty, on average, as Dirichlet counts are accumulated. The accumulation of these Dirichlet counts is summarised in the lower panels in terms of a slice through the likelihood tensor; namely, the subtensor mapping from all combinations of latent states to the correct outcome in Eq. 14. As time proceeds, the implicit probability distribution becomes increasingly precise until the (true) mapping is selected. At this point, the posterior likelihoods and true likelihoods coincide and the KL divergence in the third panel falls to zero. The panel entitled “model comparison” shows the probability distribution over models based upon Bayesian model reduction; i.e., the likelihood of each model or rule based upon the current posterior, following Eq. 9. In this instance, the true rule is number 65 and becomes increasingly predominant as evidence is accumulated. This is reflected in the graph entitled Occam’s razor, reporting the log odds ratio in Eq. 12. At trial 18, this exceeds a threshold of 16 nats and Occam’s razor is implemented. This concludes an illustration of active reasoning, in which action selection or choice behaviour is intrinsically motivated by the expected information gain over both latent states (i.e., the colour of unseen balls) and likelihood mappings (i.e., the rules, contingencies or models that determine whether response is correct or not).

Figure 4 summarises the results of repeating the simulation in Fig. 3, 64 times using randomly selected initial conditions. The upper panel reports the KL divergence between the posterior likelihood mapping and the true rule. When the true rule is discovered, this divergence falls to zero, with a threshold crossing of Occam’s razor at 16 nats (shown in the second panel). The third panel reports performance in terms of a cumulative score; namely, the number of correct minus incorrect outcomes. Note that prior to rule discovery, the performance score tends to drift down. This is because there are more ways of being wrong than right, but both are informative. Histograms reporting the performance and discovery time are shown in the lower panels. These numerical studies suggest that the number of trials needed to discover the rule depend upon the (random) initial conditions of each trial and subsequent epistemic foraging. In this example, the agent can infer the correct rule after just nine trials, in a couple of instances, but sometimes requires more than 40 trials. The mode of discovery time was about 14 trials with a cumulative score of about 150. But, how much of this sample efficiency can be attributed to active reasoning?

Fig. 4. Hypothesis testing.

This figure reports simulations of 64 games, each comprising 32 trials, using the setup described in the previous figure. The upper panel reports the KL divergence between the posterior over the likelihood mapping to the feedback modality and the true likelihood mapping. There is, on average, a decrease in divergence until it falls to zero at the point of Bayesian model selection; i.e., application of Occam’s razor. This application rests upon a threshold crossing of the log odds ratio testing for the probability of the most likely model, relative to others. The third panel reports the cumulative score; namely, the difference between correct and incorrect outcomes. The histograms report the distribution of the cumulative scores and discovery time; i.e., the number of trials before the true model was selected. Note that in one game, Occam’s razor was applied prematurely (at trial six) and the agent selected the wrong model. This means the KL divergence increases, and the discovery time is arbitrarily set to 64.

Contribution analyses

Figure 5 provides an intuition about the efficacy of expected information gain in rule discovery or hypothesis testing. The left panels reproduce the evolution of the distribution over models in Fig. 3—and accompanying Occam’s razor—when the expected information gain includes models, parameters and states. The equivalent results, for exactly the same cues, are provided in the remaining panels when successively removing information gain terms from the expected free energy. When information gain is restricted to parameters and states, active model selection fails to disambiguate between two plausible hypotheses after 32 trials. If we retain just the expected information gain over states, this failure is more pronounced, with two hypotheses being equally plausible after 32 trials. Furthermore, the agent nearly selects the wrong model after 15 trials. In the absence of any expected information gain (i.e., under random behaviour), Occam’s razor is nearly zero at trial 32; suggesting that there is no evidence—in the outcomes elicited—to suggest that the winning model is any more likely than the remaining models.

Fig. 5. Expected information gain and model selection.

This figure reproduces the results in Fig. 3 but in the absence of expected information gain over models, parameters and states. The upper row shows the posterior distribution over models as a function of trials as we remove the expected information gain over models, then parameters, and finally states. The lower panels show the corresponding effect on Occam’s razor—and consequent failures to reach threshold—as successive components of expected information gain are removed.

These numerical studies suggest that although expected information gain over parameters is sufficient to eliminate many implausible models, the expected information gain over models becomes important at later stages of evidence accumulation; i.e., resolving uncertainty about a small number of equally plausible models. In what follows, we will take a closer look at the relative contribution of the different sorts of information gain.

We repeated the above numerical study—using the same random initial conditions for each trial—but precluded any expected information gain over models, parameters and states when evaluating expected free energy. The effect on belief updating and performance are summarised Fig. 6. The first row shows the discovery time and performance with and without expected information gain over models. Here, removing expected information gain over models precluded rule discovery in two games that were otherwise discoverable within 30 trials (red dots in the upper left panel). The second row shows the effects of removing expected information gain over models and parameters. With the exception of seven instances, the discovery time was shorter when including expected information gain over models and parameters, reducing from 28.5 to 19.4 trials. A smaller discovery time means that the agent can exploit its insights to secure more correct responses (and avoid incorrect outcomes). This is shown on the middle row of Fig. 6, with an increase in cumulative performance from 89.2 to 150 points.

Fig. 6. The importance of being curious.

This figure shows the discovery time and performance when repeating the simulations in Fig. 3 with and without expected information gain over models (upper row), models or parameters (middle row), and models, parameters or states (lower row). Each dot corresponds to a game with the same sequence of cues. The green dots report worse performance in the absence of expected information gain. A failure to discover the rule after 64 trials is arbitrarily assigned a discovery time of 64 (red dots). For example, the upper row shows that removing expected information gain over models precluded rule discovery in two of the 64 games (one game was not discovered in either case). These results illustrate the synergistic effect of expected information gain over models, parameters and states, with almost universal improvement in rule discovery when all three are informing each other. The improvement in performance becomes less marked with each additional expected information gain added in, although we anticipate the precise pattern would vary with the specific task.

An interesting aspect of this information seeking behaviour is that the expected information gain over models and parameters depends upon resolving uncertainty about latent states. This is because the posterior predictive densities—used to evaluate the expected free energy—are more precise when the agent has resolved uncertainty about latent states. The ensuing interdependency can be illustrated by repeating the simulations with and without any expected information gain. The third row of panels in Fig. 6 shows the corresponding results in terms of discovery time and performance. In this case, the improvement afforded by information seeking is more marked, with a failure to discover the rule after 64 trials in about half of the simulations (the single failure with expected information gain corresponds to a premature commitment to the wrong model). The corresponding performance fell from an average score of 150 to 51 points. This speaks to a synergistic effect of resolving uncertainty about both states and model parameters.

Discussion

Active inference provides a formal account of sentient behaviour that is driven purely by maximising the evidence for generative models of exchange between an agent and the sensed world. This is sometimes neatly summarised as self-evidencing⁴⁸. Technically, this first principles account of self-organisation can be cast as a free energy minimising process—over separable timescales—where variational free energy provides an upper bound on (negative) log evidence (a.k.a., marginal likelihood). One useful perspective on this formulation is that one can convert an intractable (exact) Bayesian inference problem into an optimisation problem, through minimising variational free energy^42,49–51.

A useful perspective on this optimisation rests upon the fact that the free energy in question is a functional of (approximate) posterior and prior beliefs, where beliefs are used in the sense of Bayesian belief updating and belief propagation^52–54. This means one can minimise variational free energy with respect to the posterior and, crucially, the prior. If one associates different model structures with a set of prior constraints on the parameters of a generative model, one has a straightforward mechanism to absorb structure learning into the minimisation of variational free energy. In particular, one can use Bayesian model reduction to optimise posterior beliefs—and consequent behaviour—under a particular model and optimise the model in and of itself⁵⁵. The current paper addresses the problem of how one should gather data to optimise this kind of structure learning.

In short, by starting with a set of priors over model parameters, one can start with uniform priors over the implicit model structure and update posterior beliefs as evidence—for or against different models—is accumulated. Furthermore, as illustrated above, one can sample those data that best disambiguate between two or more plausible models. Technically, this just means extending the expected free energy—that underwrites action selection—to cover the expected information gain over models. We have illustrated this using Bayesian model reduction to evaluate a posterior over models—before and after acting under a particular policy—to evaluate the expected information gain, in accord with the principles of optimal experimental design¹³.

The resulting procedures were demonstrated analytically, and with numerical studies, using discrete state-space models; namely, partially observed Markov decision processes. Here, partial observability was inherent in constraints on the sampling of the sensorium (e.g., looking at one place at a time, as in active vision)⁵⁶. As evidence is accumulated for alternative models, the ensuing distribution over reduced models becomes increasingly precise, until one model supervenes and can be selected via an application of Occam’s razor.

Clearly, there is no guarantee that selected model will account for data that will be encountered in the future. Indeed, if one returns to Fig. 4, one of the 64 simulations committed to the wrong model prematurely, after just six trials (the instance of a KL divergence increasing at the point of model selection). In short, there is always a possibility of jumping to conclusions, even with a stringent threshold for applying Occam’s razor. This ‘jumping to conclusions’ is a particularly interesting phenomena, which speaks to the possibility of applying the three-ball paradigm in schizophrenia research, where computational phenotyping suggests that people with schizophrenia have a tendency to jump to conclusions^57–59.

Although our simulations concern a relatively simple problem, it is one that has been chosen because of its large hypothesis space (even under sparsity constraints, of size 2⁴³²). This addresses a key problem in designing practical tools: scaling to situations in which there are a very large number of possible solutions to adjudicate among. The solution we propose here is to note that many (most) possible solutions are not plausible. By imposing a series of priors that gradually restrict this solution space until the problem is tractable, one can address the scalability problem. Does this guarantee a correct solution? Clearly not. It may be that the priors chosen are overly restrictive and exclude an a priori unlikely, but ultimately correct, rule. As shown in Fig. 2, the chance of inappropriately excluding a model increases with each successive symmetry that is imposed. In some situations, it may be appropriate to impose some, but not all, of these prior constraints—in other situations, it may be necessary to impose more. Each carries with it the balance between tractable inference in scalable settings compared to the possibility of excluding a ‘true’ model.

Are there other approaches one could adopt in place of, or in parallel with, this approach to ensure greater scaling potential? One option might be to apply the same principles of factorisation we use for hidden states to the model space itself. If there are some clear dimensions along which families of rules can be defined, could apply standard variational methods^52,60,61 and assume a mean-field factorisation across these dimensions. In principle, this lets one iterate between model comparisons for each model-space dimension, rather than having to evaluate all possible combinations in a single comparison. The challenge in applying a method of this sort is in characterising the right dimensions along which to factorise a model, in which comparisons for each factor can be made approximately independent. However, the compatibility of approaches like this with the methods outlined in this paper offer an opportunity to mix methods and potentially achieve even greater gains in scalability.

While this sort of model is relatively generic, dealing with categorical (and discretised continuous) data, one limitation of this sort of model is the inefficient representation of high-dimensional continuous systems. This is because the degree of discretisation required can mean that one needs categorical states with very large numbers of possible values. This raises the question as to whether the methods outlined here can be applied directly to continuous state-space models that afford more efficient representations of such systems. In principle, the answer is yes, in that Bayesian Model Reduction—under a Laplace approximation—has been applied extensively to continuous systems⁵⁵ and one can simply formulate information gains for such systems^15,62, at least, to a reasonable approximation. However, the outstanding challenge to make this practical—for continuous models—is in identifying a set of candidate paths, as opposed to a continuum of possible paths, along which expected information gain should be evaluated. While plausible, this is less straightforward than in the categorical domain as presented here.

As noted in the introduction, this work—and the notion of active reasoning—owes much to the artificial intelligence community’s focus on System 2 thinking and, in particular, the ARC-AGI-3 challenge. However, active reasoning, as described in this paper, only addresses a small part of this challenge. First, we have assumed that the agent knows the generative model under which the (unknown) rule is implemented; as opposed to having to learn the generative model from sampling some unsegmented sensory input, such as a pixel array. Furthermore, it has reduced the problem of rule discovery, or abstract reasoning, to discovering the form of a likelihood mapping to a rule-defining outcome. This means that the kind of reasoning demonstrated above rests upon learning a generative model of the world—and the consequences of acting upon that world—before rule learning.

Other advances that might help fill in the gaps here include other perspectives on structure learning, ranging from generic schemes like hierarchical Dirichlet process modelling^63,64 through proposed neural implementations^30,31 and applications of active inference approaches that learn the models used here^65–67. The challenges one has to solve—when building a model of this sort—include determination of the appropriate hierarchical depth, the dimensionality of each state factor, the number of state factors, the number of plausible paths, and so on. These are challenging problems. However, in principle they can be addressed by making a series of changes to a model (either to grow or reduce it) and by assessing the model evidence before and after each change. This, of course, raises further problems, including the way in which one determines the sequence of changes to test and, if within one’s control, the curriculum of observations on which to train a model. Solutions to these problems will be necessary to develop the models on which active reasoning rests.

Once the functional form and structure of the generative model have been learned, or are known, then this (factorial) structure can be used to specify a space of lawful likelihood models that instantiate an unknown rule or program. We have introduced this specification in terms of hypothetical likelihood mappings that feature some context-sensitive symmetry and isomorphisms, which may or may not be applicable to any generative model with a suitable factorial structure. Irrespective of the generality of context-sensitive isomorphic constraints, the fact that these constraints inherit from a factorial structure suggests that the rules about which we reason rest upon carving nature at its joints in the form of conditionally independent factors such as ‘what’ and ‘where’, or the attributes of ‘objects’. In turn, this speaks to the importance of object-centric representations and requisite factorial models in being able to apply the symmetries or constraints that are variously ascribed to rules, symmetries, equivariances and laws.

Finally, although our focus has been on artificial reasoning, we conclude with some brief observations on human reasoning³⁸. There are a broad range of definitions, and categories, of reasoning in cognitive science. However, common tasks used in human behavioural experiments are based upon inference of rules, such as the Raven’s matrix task⁶⁸. Here, as in our setting, the challenge is to take a series of cues and to infer from these a rule that allows one to determine the likely next cue. This endorses our focus upon rule learning as a core element of reasoning. Anatomically, there is evidence that reasoning of this sort relies upon a lateralised, primarily left-hemispheric, frontoparietal network with basal ganglia involvement⁶⁹. The basal ganglia element of this is particularly relevant here, in that these subcortical nuclei are often closely linked to action, with lesions associated to neurological disorders of action initiation^70–74. One could read this evidence as an endorsement of active reasoning, that the same structures involved in generating behaviour—motivated by preference satisfaction and information seeking—also support rule learning. One could assess whether our account of artificial reasoning might also serve as a (partial) theory of human reasoning through asking human participants to perform the 3-ball task, and by determining which of the patterns shown in Fig. 5 best matches their behavioural trajectory.

At a more fundamental level, there are some constraints built into the biological machinery we have for sensing and acting upon the world. These constraints support the use of symmetries in constructing plausible, and tractable, priors for the space of models. For instance, colour perception depends upon the same retinal cone cells whether they register a cue or a response. Tactile sensing of an object in a spatial location also facilitates motoric manipulation targeted at the same spatial location. More generally, our musculoskeletal effectors are paired with proprioceptive sense organs. This implies a degree of isomorphism between the responses one might make and the sensory modalities that report upon these, supporting models in which these isomorphisms are respected. Some symmetries are relatively generic [e.g., see¹⁶ for an analysis of the need for self-organising systems to exist in minimally ambiguous environments, supporting sparse models] while others may need to be relaxed depending upon context. Ultimately, for a creature to be a successful rule-learner, they must hold a set of priors consistent with rule generation. The challenge for reasoning neuroscience is to determine the form these priors take.

Methods

Active inference, learning and selection

Active inference rests upon a generative model of observable outcomes. This model is used to infer the most likely causes of outcomes in terms of expected states of the world. These states—and their transition or paths—are latent or hidden because they can only be inferred through observations. Some paths are controllable, in that they can be realised through action. Therefore, certain observations depend upon action or policies, which requires the generative model to entertain expectations about the consequences of action. These expectations are optimised by minimising variational free energy while the prior probability of a policy depends upon its expected free energy².

Figure 7 provides an overview of the generative models typically used for sense and decision-making under uncertainty; namely, a partially observed Markov decision process (POMDP). Outcomes at any time depend upon hidden states, while transitions among hidden states depend upon paths. In this model, paths are random variables that depend upon action. This POMDP is specified by a set of tensors. The first set A, maps from hidden states to outcome modalities; for example, exteroceptive (e.g., visual) or proprioceptive (e.g., eye position) modalities. These parameters encode the likelihood of an outcome given hidden states. The second set B encodes transitions among the hidden states of a factor, under a particular path. Factors correspond to different kinds of states; e.g., the location versus the class of an object. The remaining tensors encode prior beliefs about paths C, and initial conditions D and E, sometimes referred to as hidden causes. The tensors are generally parameterised as Dirichlet distributions, whose sufficient statistics are Dirichlet counts, which count the number of times a particular combination of states and outcomes has been inferred. We will focus on learning the likelihood model, encoded by Dirichlet counts, a.

Fig. 7. Generative models.

A generative model specifies the joint probability of observable consequences and their hidden causes. Usually, the model is expressed in terms of a likelihood (the probability of consequences given their causes) and priors (over causes). When a prior depends upon a random variable it is called an empirical prior. Here, the likelihood is specified by a tensor A, encoding the probability of an outcome under every combination of states (s). Priors over transitions among hidden states, B depend upon paths (u), whose transition probabilities are encoded in C, where D and E specify the prior over initial states and paths, respectively. Certain paths are more probable a priori if they minimise their expected free energy (G). For clarity, we have assumed that there is a single factor and path in this model. The blue panel provides the functional form of the generative model in terms of categorical (Cat) distributions that are parameterised as Dirichlet (Dir) distributions. The lower equalities list the various operators required for variational message passing in Fig. 8. These functions operate on each column of their tensor arguments. $\psi \left(\cdot \right)$ is the digamma function. The right panel depicts the generative model as a probabilistic graphical model and the lower panels list the functional form of approximate posteriors (Q) and their optimisation.

The generative model in Fig. 7 means that outcomes are generated by selecting a policy from a softmax function of expected free energy. Sequences of hidden states are then generated using the probability transitions specified by the selected combination of paths (i.e., policy). Finally, hidden states generate outcomes in one or more modalities. Model inversion updates the sufficient statistics $\left(\mathbf{s},\mathbf{u},\mathbf{a},\dots \right)$ of posterior beliefs $Q\left(s,u,\mathbf{A},\dots \right)=Q\left(s\right)Q\left(u\right)Q\left(\mathbf{A}\right),\dots$ that are factorised over hidden states, paths and parameters. This mean field factorisation effectively partitions belief updating into inference, planning and learning.

Variational and expected free energy

In variational (a.k.a., approximate) Bayesian inference, model inversion entails the minimisation of variational free energy with respect to the sufficient statistics of the approximate posterior. For clarity, we will deal with a single factor, such that the policy (i.e., combination of paths) becomes the path. Omitting dependencies on initial states and certain parameters, for model m we have:

$$Q\left(s_{\tau },u_{\tau -1},\mathbf{A}\right)=Q\left(s_{\tau }\right)Q\left(u_{\tau -1}\right)Q\left(\mathbf{A}\right)=\arg {\min }_{Q}FF={\mathrm{E}}_Q\left[\ln {\underbrace{Q\left(s_{\tau },u_{\tau -1},\mathbf{A}\right)}}_{\mathrm{Posterior}}-\ln {\underbrace{P\left(o_{\tau }\mid s_{\tau },\mathbf{A}\right)}}_{\mathrm{Likelihood}}-\ln {\underbrace{P\left(s_{\tau }\mid s_{\tau -1},u_{\tau -1}\right)P\left(u_{\tau -1}\right)P\left(\mathbf{A}\right)}}_{\mathrm{Prior}}\right]={\underbrace{{\mathrm{E}}_Q\left[D_{KL}\left[Q\left(s_{\tau },u_{\tau -1},\mathbf{A}\right)\mid \mid P\left(s_{\tau },u_{\tau -1},\mathbf{A}\mid s_{\tau -1},o_{\tau }\right)\right]\right]}}_{\mathrm{Divergence}}-{\underbrace{\ln P\left(o_{\tau }\mid m\right)}}_{\mathrm{Log\; evidence}}={\underbrace{{\mathrm{E}}_Q\left[D_{KL}\left[Q\left(s_{\tau },u_{\tau -1},\mathbf{A}\right)\mid \mid P\left(s_{\tau },u_{\tau -1},\mathbf{A}\mid s_{\tau -1}\right)\right]\right]}}_{\mathrm{Complexity}}-{\underbrace{{\mathrm{E}}_Q\left[\ln P\left(o_{\tau }\mid s_{\tau },\mathbf{A}\right)\right]}}_{\mathrm{Accuracy}}$$ 1

We have conditioned the model evidence term explicitly upon a model, m, to emphasise its interpretation. However, all generative model terms are implicitly conditioned on m. Here, the expectation is under $Q=Q\left(s_{\tau -1}\right)Q\left(s_{\tau },u_{\tau -1},\mathbf{A}\right)$. Variational free energy is a function of observations and a functional of current beliefs, and could be written as $F\left[o_{\tau },Q\left(s_{\tau -1},\dots \right),P\left(o_{\tau },\dots ,\mid m\right)\right]$. However, we omit these functional dependencies for clarity. Because the (KL) divergences cannot be less than zero, the penultimate equality means that free energy is minimised when the approximate posterior is the true posterior. At this point, the free energy becomes the negative log evidence for the generative model⁶¹. This means minimising free energy is equivalent to maximising model evidence. Planning emerges under active inference by placing priors over paths to minimise expected free energy⁷⁵. In the simple case of one-step-ahead policies (and limiting our focus to the likelihood parameters), we have:

$$G\left(u\right)={\mathrm{E}}_Q\left[\ln Q\left(s_{\tau +1},\mathbf{A}\mid u\right)-\ln Q\left(s_{\tau +1},\mathbf{A}\mid o_{\tau +1},u\right)-\ln P\left(o_{\tau +1}\mid c\right)\right]=-{\underbrace{{\mathrm{E}}_Q\left[\ln Q\left(\mathbf{A}\mid s_{\tau +1},o_{\tau +1},u\right)-\ln Q\left(\mathbf{A}\mid s_{\tau +1},u\right)\right]}}_{\mathrm{Expected}\mathrm{information}\mathrm{gain}\left(\mathrm{Parameters}\right)}-{\underbrace{{\mathrm{E}}_Q\left[\ln Q\left(s_{\tau +1}\mid o_{\tau +1},u\right)-\ln Q\left(s_{\tau +1}\mid u\right)\right]}}_{\mathrm{Expected}\mathrm{information}\mathrm{gain}\left(\mathrm{States}\right)}{\underbrace{-{\mathrm{E}}_Q\left[\ln P\left(o_{\tau +1}\mid c\right)\right]}}_{\mathrm{Expected}\mathrm{cost}}=-{\underbrace{{\mathrm{E}}_Q\left[D_{KL}\left[Q\left(\mathbf{A}\mid s_{\tau +1},o_{\tau +1},u\right)\mid \mid Q\left(\mathbf{A}\mid s_{\tau +1},u\right)\right]\right]}}_{\mathrm{Novelty}}+{\underbrace{D_{KL}\left[Q\left(o_{\tau +1}\mid u\right)\mid \mid P\left(o_{\tau +1}\mid c\right)\right]}}_{\mathrm{Risk}}{\underbrace{-{\mathrm{E}}_Q\left[\ln Q\left(o_{\tau +1}\mid s_{\tau +1},u\right)\right]}}_{\mathrm{Ambiguity}}$$ 2

Here, $Q=:Q\left(o_{\tau +1},s_{\tau +1},\mathbf{A}\mid u\right)=P\left(o_{\tau +1},s_{\tau +1},\mathbf{A}\mid u,o_0,\dots ,o_{\tau }\right)=P\left(o_{\tau +1}\mid s_{\tau +1},\mathbf{A}\right)Q\left(s_{\tau +1},\mathbf{A}\mid u\right)$ is the posterior predictive distribution over parameters, hidden states and outcomes at the next time step, under a particular path. Note that the expectation is over observations in the future, hence, expected free energy. This means that preferred outcomes— parameterised by c—are prior beliefs, which constrain the implicit planning as inference: c.f.,^3,4,76. Planning over longer time horizons simply involves accumulating the path integral of expected free energy over successive actions—or combinations of actions; i.e., policies—as described in ref. ⁷⁷. One could construct a more complete expected free energy through inclusion of other parameters (e.g., transitions B or initial states D) simply by replacing all A’s with A, B, D, and so on.

Expected free energy augments expected information gain with costs or constraints. These constraints reflect the fact that we are dealing with systems with characteristic outcomes that are scored by their cost $\mathrm{C}\left(o\right)=-\ln P\left(o\mid c\right)$ or inverse value. Statistically speaking, expected free energy underwrites Bayes optimal behaviour, in the dual sense of Bayesian decision theory⁷⁸ and optimal experimental design¹³. Its functional form inherits from the path integral formulation of the free energy principle: please see ref. ¹⁶ for details. One can also express a prior over parameters in terms of an expected free energy where, marginalising over paths:

$$P\left(\mathbf{A}\right)=\sigma \left(-G\left(\mathbf{A}\right)\right)G\left(\mathbf{A}\right)={\mathrm{E}}_Q\left[\ln P\left(s\mid \mathbf{A}\right)-\ln P\left(s\mid o,\mathbf{A}\right)-\ln P\left(o\mid c\right)\right]=-{\underbrace{{\mathrm{E}}_Q\left[\ln P\left(s\mid o,\mathbf{A}\right)-\ln P\left(s\mid \mathbf{A}\right)\right]}}_{\mathrm{Expected}\mathrm{information}\mathrm{gain}}{\underbrace{-{\mathrm{E}}_{Q_a}\left[\ln P\left(o\mid c\right)\right]}}_{\mathrm{Expected}\mathrm{cost}}=-{\underbrace{{\mathrm{E}}_Q\left[\right)D_{KL}\left[P\left(o,s\mid \mathbf{A}\right)\mid \mid P\left(o\mid \mathbf{A}\right)P\left(s\mid \mathbf{A}\right)\right]}}_{\mathrm{Mutual}\mathrm{information}}{\underbrace{-{\mathrm{E}}_{Q_a}\left[\ln P\left(o\mid c\right)\right]}}_{\mathrm{Expected}\mathrm{cost}}$$ 3

Here, $Q=:P\left(o\mid s,\mathbf{A}\right)P\left(s\mid \mathbf{A}\right)=P\left(o,s\mid \mathbf{A}\right)$ is the joint distribution over outcomes and hidden states, encoded by Dirichlet parameters. Note that the Dirichlet parameters encode the mutual information, because they encode the joint distribution over outcomes and hidden states: when normalising each column of the a tensor, we recover the likelihood distribution. However, we could normalise over all entries of the tensor to recover a joint distribution. The prior in the first line of Eq. 3 can be read as implementing a constrained principle of maximum mutual information or minimum redundancy^79–82. In machine learning, this kind of prior underwrites disentanglement^83,84, and generally leads to sparse representations^82,85–87. We will call on this prior later when specifying priors over models.

Active inference

In variational inference, sufficient statistics—encoding posterior expectations—are updated to minimise variational free energy. Figure 8 illustrates these updates in the form of variational message passing^52,60. For example, expectations about hidden states are a softmax function of messages that are linear combinations of other expectations and observations.

$${\mathbf{s}}_{\tau }^f=\sigma \left({\mu }_{\uparrow \mathbf{A}}^f+{\mu }_{\to B}^f+{\mu }_{\leftarrow B}^f+\dots \right){\mu }_{\uparrow \mathbf{A}}^f={\sum }_{g\in ch\left(f\right)}{\mu }_{\uparrow \mathbf{A}}^{g,f}{\mu }_{\uparrow \mathbf{A}}^{g,f}={\mathbf{o}}_{\tau }^g\odot \phi \left({\mathbf{a}}^g\right){\odot }_{i\in pa\left(g\right)\backslash f}{\mathbf{s}}_{\tau }^i$$ 4

Fig. 8. Belief updating and variational message passing.

The right panel presents the generative model as a Forney factor graph, where the nodes (square boxes) correspond to the factors of the generative model (labelled with the associated tensors). Boxes containing the “=” symbol are Kronecker delta tensors that enforce equality between the connecting edges, ensuring the graph is bipartite. The edges connect factors that share dependencies on random variables. The leaves (filled circles) correspond to known variables, such as observations (o). This representation is useful because it scaffolds the message passing—over the edges of the factor graph—that underwrite inference and planning. The functional forms of these messages are shown in the left-hand panels. For example, the expected path—in the first equality of panel C—is a softmax function of two messages. The first is a descending message ${\mu }_{\downarrow \mathbf{E}}^f$ from E that inherits from expectations about hidden states at the level above. The second is the log-likelihood of the path based upon expected free energy, G. This message depends upon Dirichlet counts scoring preferred outcomes—i.e., prior constraints on modality g—encoded in ${\mathbf{c}}^g$: see Eq. 2. The matrix π encodes the probability over paths, under each policy. Subscripts pertain to time, while superscripts denote distinct factors f, and outcome modalities g. Tensors and matrices are denoted by uppercase bold, while posterior expectations are in lowercase bold. The $\odot$ notation denotes a generalised inner (i.e., dot) product or tensor contraction, while $\otimes$ denotes an outer product. The updates in the lighter panels correspond to learning; i.e., updating Bayesian beliefs about parameters. ch(·) and pa(·) return the children and parents of latent variables. A version of this figure previously appeared in ref. ⁴¹ under a CC-BY license.

Here, the ascending messages from the likelihood factor are a linear mixture of expected states and observations, weighted by digamma functions of the Dirichlet counts that parameterise the likelihood model (c.f., connection weights). The $\odot$ notation implies a sum product operator; i.e., the dot or inner product that sums over one dimension of an array or tensor. These operators are applied to a vector a and tensor A where, $\mathbf{a}\odot \mathbf{A}$ implies the sum of products is taken over the leading dimension, while $\mathbf{A}\odot \mathbf{a}$ implies the sum is taken over a trailing dimension. For example, $\mathbf{1}\odot \mathbf{A}$ is the sum over rows and $\mathbf{A}\odot \mathbf{1}={\mathbf{A}}_0$ is the sum over columns, where $\mathbf{1}$ is a vector of ones and A is a matrix. If A is a matrix then $\mathbf{a}\odot \mathbf{A}={\mathbf{a}}^T\cdot \mathbf{A}$. Finally,${\mathbf{A}}_{\bullet ,i}$ and ${\mathbf{A}}_{i,\bullet }$ refer to the ith column and row of A, respectively. This notation replaces the Einstein summation notation to avoid visual clutter.

The digamma function—the logarithmic derivative of the gamma function—appears here as it affords a concise representation of the expectation of (log) parameters under a Dirichlet distribution. The expressions in Fig. 8 are effectively the fixed points (i.e., minima) of variational free energy. This means that message passing corresponds to a fixed-point iteration scheme^52,60,61. Figure 8 and Eq. 4 provide forward and backward messages in time. In practice—and in the examples below—it is sometimes simpler to omit backward messages; in which case the scheme becomes a Bayesian filter—as opposed to smoother. One can also replace the variational messages with those from sum-product belief-propagation schemes to achieve exact marginal posteriors.

Belief updating under active inference is sample efficient by design, in virtue of acting to maximise information gain—and minimise ambiguity—where expected free energy can be expressed in terms of sufficient statistics as follows:

$$G_h={\sum }_g{\underbrace{{\mathbf{o}}_{\tau +1}^{g,h}\odot \ell \left({\mathbf{o}}_{\tau +1}^{g,h}\right)-{\mathbf{o}}_{\tau +1}^{g,h}\odot \phi \left({\mathbf{c}}^g\right)}}_{\mathrm{Risk}}{\underbrace{-{\mathbf{H}}^g\odot {\mathbf{s}}_{\tau +1}^h}}_{\mathrm{Ambiguity}}-{\underbrace{{\mathbf{o}}_{\tau +1}^{g,h}\cdot {\mathbf{W}}^g\cdot {\mathbf{s}}_{\tau +1}^h}}_{\mathrm{Novelty}}{\mathbf{o}}_{\tau +1}^{g,h}=\mu \left({\mathbf{a}}^g\right)\odot {\mathbf{s}}_{\tau +1}^h{\mathbf{s}}_{\tau +1}^h=\mu \left(\mathbf{b}\odot {\pi }_h\right)\odot {\mathbf{s}}_{\tau }{\mathbf{H}}^g=\mathbf{1}\odot \left(\mu \left({\mathbf{a}}^g\right)\times \phi \left({\mathbf{a}}^g\right)\right){\mathbf{W}}_{ij}^g=\frac{1}{{\mathbf{a}}_{ij}^g}-\frac{1}{{\mathbf{a}}_{0j}^g}+\ln \frac{{\mathbf{a}}_{0j}^g}{{\mathbf{a}}_{ij}^g}+\psi \left({\mathbf{a}}_{ij}^g\right)-\psi \left({\mathbf{a}}_{0j}^g\right)\approx \frac{1}{2{\mathbf{a}}_{ij}^g}-\frac{1}{2{\mathbf{a}}_{0j}^g}$$ 5

Please see Figs. 7 and 8 for definitions of the operators used here. This expresses expected free energy in terms of risk, ambiguity and novelty for policy h; namely, the hth combination of actions over factors. This equation effectively gives specific implementational details—i.e., the way in which one can compute this quantity mathematically—of (3). Please see above for a conceptual interpretation.

Active learning

In the setting of discrete models, learning corresponds to updating model parameters by accumulating Dirichlet counts based upon posterior expectations. For example, for the likelihood tensors we have:

$${\mathbf{a}}^g\leftarrow {\mathbf{a}}^g+{\mathbf{o}}_{\tau }^g{\otimes }_{i\in pa\left(g\right)}{\mathbf{s}}_{\tau }^i$$ 6

Effectively, the likelihood parameters count the number of co-occurrences of outcomes and their latent causes. The expected information gain over parameters in Eqs. 2 and 5 ensures that outcomes are sampled to minimise uncertainty about the likelihood mapping. Intuitively, this implies seeking outcomes that inform columns of the likelihood tensor that have small Dirichlet counts. Conversely, when combinations of latent states and outcomes have been seen many times, there is less uncertainty to resolve and the outcomes offer less epistemic affordance or expected information gain.

Active selection

In contrast to learning—that optimises posteriors over parameters—Bayesian model selection or structure learning^30,88,89 can be framed as optimising the priors over model parameters. One can interpret Bayesian model comparison as the application of Bayesian inference to a random variable that effectively weights elements in a mixture model. Model selection is the special case in which the inference about the mixture coefficient approaches zero entropy (e.g., when one knows with confidence which cluster a datapoint is sampled from). It is this interpretation that allows one to directly incorporate model identity as a random variable into expected free energy. On this view, model selection can be implemented efficiently using Bayesian model reduction. Bayesian model reduction is a generalisation of ubiquitous procedures in statistics, such as the Savage–Dickey method⁹⁰. It gives the exact difference in model evidence—and reduced model parameters under specific conditions—when a reduced model differs from the full model in its priors. This holds when the posteriors under the full model are exact, and the priors are conjugate to the likelihood of the full and reduced models. When these things are approximately true, the estimates are themselves approximate. For readers interested in the details and applications of this technique, please see refs. ^21,91.

The use of Bayesian model reduction in this setting casts each alternative model as a reduced version of the Bayesian model average under uniform priors. In other words, each model is defined by removing (combinations of) parameters through suitable shrinkage priors; c.f.,^55,92,93. In the implementation adopted in this work, one simply applies shrinkage priors by setting Dirichlet counts to zero or a small value (e.g., 1/32). This provides a fast and efficient scheme for scoring the likelihood of different models, given just the original priors and a posterior that can be updated. Fast and efficient inference is important both in biological and artificial systems in conserving energy and ensuring timely action in a dynamic world. This is increasingly recognised as important for sustainability in the space of artificial intelligence^94–96.

By applying Bayes rules to full and reduced models, it is straightforward to show that the change in log marginal likelihood can be expressed in terms of posterior Dirichlet counts a, prior counts a and the prior counts that define a reduced model $a_m=:a\mid m$. Using $\mathrm{B}$ to denote the beta function, we have²¹ the following expression for the log Bayes factor (or difference between log marginal likelihoods) between a full and reduced model:

$$\mathrm{B}\left(\mathbf{a},a_m,a\right)=\ln P\left(o\mid a\right)-\ln P\left(o\mid a_m\right)=\ln \mathrm{B}\left(\mathbf{a}\right)\mathrm{B}\left(a_{\mathrm{m}}\right)-\ln \mathrm{B}\left(a\right)\mathrm{B}\left({\mathbf{a}}_{\mathrm{m}}\right){\mathbf{a}}_m=\mathbf{a}+a_m-a$$ 7

Here, ${\mathbf{a}}_m$ corresponds to the posterior that would obtain under the reduced model.

So far, we have not considered the priors over parameters. By appeal to Eq. 3, we can assume a set of models $m\mathrm{\in }\mathrm{M}$, all of which entail a high mutual information (between latent states and observations), under suitable constraints. Each model within this space differs in that it contains different priors over the parameters of the distributions within the model. These imply different model structures in that some priors set some parameters as precisely zero (or close to), abolishing some conditional dependencies. In this setting, the model m becomes a random variable, over which we can assume a uniform prior. This leads to a prior over parameters that can be read as a Bayesian model average:

$$a={\mathrm{E}}_{P\left(m\right)}\left[a_m\right]P\left(\mathbf{A}\right)=Dir\left(a\right)P\left(m\right)=1/\mid \mathrm{M}\mid$$ 8

This specification of priors admits a simple evaluation of posteriors over models in terms of the Bayesian model reduction of Dirichlet parameters that have accumulated outcomes in the past. Applying Bayes rule and dropping terms that do not depend upon m, we have:

$$\ln P\left(m\mid o\right)=\ln P\left(o\mid m\right)+\ln P\left(m\right)-\ln P\left(o\right)=\ln P\left(o\mid a\right)-\mathcal{B}\left(\mathbf{a},a_m,a\right)-\ln \mid \mathcal{M}\mid -\ln P\left(o\right)\Rightarrow P\left(m\mid o\right)=\sigma \left(-\mathcal{B}\right){\mathcal{B}}_m=\mathcal{B}\left(\mathbf{a},a_m,a\right)=\ln P\left(o\mid a\right)-\ln P\left(o\mid m\right)$$ 9

In summary, if we start with a set of models or hypothesis space $\mathcal{M}$, of equally plausible models (here, likelihood mappings), we have a principled way of specifying the prior over model parameters in terms of a Bayesian model average. One can then accumulate evidence for each of the models until one is sufficiently confident that one model supervenes over the remainder. One then selects the ‘last man standing’ for subsequent inference and learning; i.e., applies Occam’s razor. This constitutes Bayesian model selection by reduction. See⁹⁷ for a biomimetic example of this form of self-organisation, applied to dendritic neuronal processes and the sampling of presynaptic inputs. So, can we optimise evidence accumulation for subsequent model selection?

Using the expression for posteriors models in Eq. 9, one can generalise expected free energy to include the expected information gain over models. This rests upon equipping an agent with a repertoire of models or hypotheses, so that it can actively search for disambiguating evidence and identify the most plausible model through an application of Occam’s razor. The expected information gain is relatively straightforward to evaluate, given the evidence accumulated by the Dirichlet parameterisation of the full or parent model in Eq. 8. The expected information gain is the expected KL divergence between the posterior probability of a model with, and without, the outcomes under a particular policy.

$$G\left(u\right)=-{\underbrace{{\mathrm{E}}_Q\left[\ln Q\left(m\mid \mathbf{A},s_{\tau +1},o_{\tau +1},u\right)-\ln Q\left(m\mid \mathbf{A},s_{\tau +1},u\right)\right]}}_{\mathrm{Expected}\mathrm{information}\mathrm{gain}\left(\mathrm{Models}\right)}-{\underbrace{{\mathrm{E}}_Q\left[\ln Q\left(\mathbf{A}\mid s_{\tau +1},o_{\tau +1},u\right)-\ln Q\left(\mathbf{A}\mid s_{\tau +1},u\right)\right]}}_{\mathrm{Expected}\mathrm{information}\mathrm{gain}\left(\mathrm{Parameters}\right)}-{\underbrace{{\mathrm{E}}_Q\left[\ln Q\left(s_{\tau +1}\mid o_{\tau +1},u\right)-\ln Q\left(s_{\tau +1}\mid u\right)\right]}}_{\mathrm{Expected}\mathrm{information}\mathrm{gain}\left(\mathrm{States}\right)}{\underbrace{-{\mathrm{E}}_Q\left[\ln P\left(o_{\tau +1}\mid c\right)\right]}}_{\mathrm{Expected}\mathrm{cost}}$$ 10

The requisite predictive posteriors for the expected information gain over models follow from Eqs. 5, 6 and 9:

$$Q\left(m\mid \mathbf{A},s_{\tau +1},o_{\tau +1},u\right)=\sigma \left(-\mathrm{B}\right):{\mathrm{B}}_m=\mathrm{B}\left({\mathbf{a}}^g+{\mathrm{\Delta }}^g,a_m^g,a^g\right)Q\left(m\mid \mathbf{A},s_{\tau +1},u\right)=\sigma \left(-\mathrm{B}\right):{\mathrm{B}}_m=\mathrm{B}\left({\mathbf{a}}^g,a_m^g,a^g\right){\mathrm{\Delta }}^g={\mathbf{o}}_{\tau +1}^{g,u}{\otimes }_{i\in pa\left(g\right)}{\mathbf{s}}_{\tau +1}^u$$ 11

The Δ symbol is defined as in the final line, so that the first line can be expressed concisely. Note that Q here is a predictive posterior, as opposed to an approximate posterior over models. In other words, it is the predictive posterior under the model average in Eq. 8. This speaks to the distinction between the current post-hoc application of Bayesian model reduction⁵⁵ and inferring the posterior over models through continuous structure learning. Please see the subsection ‘Model posteriors and model averages’ below for a brief discussion.

For clarity, we have assumed only the likelihood mapping to modality g is unknown and that there is only one factor with a controllable path (i.e., h = u). When multiple mappings are unknown, one simply adds the accompanying expected information gains to expected free energy. Note that this expected information gain does not lend itself to evaluation over sequences of actions, unless the parameters are updated over successive time points in the future, as the expected information gain over models is a function of the parameter updates, while parameter updates depend upon the states inferred over time. Therefore, we will restrict the use of expected information gain over models to the next action—or combination thereof—and invoke planning over longer time horizons after the best model has been selected.

Including the expected information gain over models affords the evidence or data that best disambiguates hypotheses or models. It does not, in and of itself, commit to using a new model. To replace the full model with a reduced model requires an application of Occam’s razor when, and only when, the agent is sufficiently confident that a reduced model is the best explanation for the evidence accumulated thus far. One can operationalise this application of Occam’s razor in terms of a log Bayes factor (i.e., log odds ratio) scoring the probability that the best model is more likely than any other model:

$$\mathcal{R}=\ln \frac{P\left(m^{*}\mid o\right)}{1-P\left(m^{*}\mid o\right)}{m}^{*}=\text{arg}{\max }_{m}P\left(m\mid o\right)$$ 12

When Occam’s razor surpasses a threshold, the agent replaces its posterior over parameters with the Bayesian model average based upon its posterior over models. If we use a threshold of 16 natural units, the Bayesian model average becomes Bayesian model selection. This follows because the winning model is at least exp(16) times more likely than alternative models:

$$\mathbf{a}\leftarrow a_{m^{*}}={\mathrm{E}}_{P\left(m\mid o\right)}\left[a_m\right]\Rightarrow Q\left(\mathbf{A}\right)=Dir\left(a_{m^{*}}\right)$$ 13

While this thresholding can be easily motivated from a pragmatic perspective (there is no point continuing to evaluate alternative models when there is a clear winner), it is not fundamental to the underlying theory, and one could substitute this with alternative thresholds or omit the thresholding altogether. A Bayesian model average under a set of models in which only one is plausible would achieve a very similar outcome. In practice, one can multiply the Dirichlet counts in the selected model by some arbitrarily high number (e.g., 512), to suppress novelty seeking. This works because this high scaling implies high prior confidence and therefore little parameter uncertainty to resolve. Conversely, one can render the agent impressionable, and use the reduced model directly (whose Dirichlet counts usually sum to one over columns).

The ensuing active model reduction or reasoning recapitulates how a scientist would proceed when resolving uncertainty about her alternative hypotheses. She will acquire a sufficient amount of data—under an unbiased (flat) prior over her hypotheses—to inform her predictive posterior. After acquiring the data, she will commit to the best hypothesis, if the evidence in favour of this hypothesis is sufficiently large, relative to others. This analogy emphasises the separation of temporal scales that accompanies active inference, learning and selection. In short, inference is a fast process of belief updating about time-dependent latent states. Learning is a slow process of belief updating about the parameters of a generative model, and selection is a post-hoc process that is enacted after sufficient data have been accumulated.

In this post-hoc scheme, evidence is accumulated by the Dirichlet parameters that are learned; enabling Bayesian model selection based purely upon what has observed thus far. One could imagine invoking hyperpriors about the number of data points required before selecting a model. Indeed, scientists are often required to do power calculations to estimate the number of observations that should be collected before analysis. For simplicity, we will just use the log odds ratio in Eq. 12, to select the best model—i.e., apply Occam’s razor—when it surpasses a threshold. Clearly, this does not guarantee that the selected model is the best model, in the sense that the agent could jump to premature conclusions. However, these should be relatively rare events with a sufficiently high threshold.

An outstanding issue is the elaboration of alternative models; i.e., the specification of the model space. The requisite specification of $\mathcal{M}$ effectively implements hyperpriors over hypothetical rules, symmetries, invariances and the like. The problem chosen to illustrate the application of the above scheme speaks to a fairly straightforward procedure for automatically specifying the model space.

In summary, can understand the procedure outlined above as follows. One begins by learning the parameters of a single model constructed via a Bayesian model average of all plausible (‘reduced’) models. Bayesian model reduction proceeds by comparing the learned parameters with the reduced priors associated with each of the reduced models to efficiently compute their evidence, until a single model gains overwhelmingly more evidence than the others, enabling commitment to this model.

Model posteriors and model averages

It is important to distinguish between inference under a Bayesian model average and inferring the posterior over models subtending that average. In the post-hoc scheme considered in this paper, inference and learning rest upon minimising the variational free energy in Eq. 1.

$$F={\mathrm{E}}_Q\left[\ln {\underbrace{Q\left(s_{\tau }\right)Q\left(u_{\tau }\right)Q\left(\mathbf{A}\right)}}_{\mathrm{Posterior}}-\ln {\underbrace{P\left(o_{\tau }\mid s_{\tau },\mathbf{A}\right)}}_{\mathrm{Likelihood}}-\ln {\underbrace{P\left(s_{\tau }\mid s_{\tau -1},u_{\tau }\right)P\left(u_{\tau }\right)P\left(\mathbf{A}\right)}}_{\mathrm{Prior}}\right]P\left(\mathbf{A}\right)=Dir\left(a\right)a={\mathrm{E}}_{P\left(m\right)}\left[a_m\right]P\left(\mathbf{A}\mid m\right)=Dir\left(a_m\right)$$

This is distinct from inferring the model per se. Inference over models would require minimising a variational free energy that included an approximate posterior over models:

$$\mathbf{F}={\mathrm{E}}_Q\left[\ln {\underbrace{Q\left(s_{\tau }\right)Q\left(u_{\tau }\right)Q\left(\mathbf{A}\right)Q\left(m\right)}}_{\mathrm{Posterior}}-\ln {\underbrace{P\left(o_{\tau }\mid s_{\tau },\mathbf{A}\right)}}_{\mathrm{Likelihood}}-\ln {\underbrace{P\left(s_{\tau }\mid s_{\tau -1},u_{\tau }\right)P\left(u_{\tau }\right)P\left(\mathbf{A}\mid m\right)P\left(m\right)}}_{\mathrm{Prior}}\right]$$

However, inference and learning under the prior over models, i.e., $Q\left(m\right)=P\left(m\right)$, renders the variational free energy in Eq. 1 a lower bound approximation to the variational free energy required for inference over models.

$$\mathbf{F}=F+{\mathrm{E}}_Q\left[\ln Q\left(m\right)-\ln P\left(m\right)\right]-{\mathrm{E}}_Q\left[\ln P\left(\mathbf{A}\mid m\right)-\ln P\left(\mathbf{A}\right)\right]=F-{\mathrm{E}}_Q\left[\ln P\left(\mathbf{A}\mid m\right)-\ln P\left(\mathbf{A}\right)\right]=F-{\mathrm{E}}_{Q\left(\mathbf{A}\right)}\left[{\mathrm{E}}_{P\left(m\right)}\left[\phi \left(a_m\right)\right]-\phi \left({\mathrm{E}}_{P\left(m\right)}\left[a_m\right]\right)\right]\Rightarrow \mathbf{F}\ge F$$

This rests on the Jensen inequality, $\mathrm{E}\left[\phi \left(a\right)\right]\le \phi \left(\mathrm{E}\left[a\right]\right)$, given that the digamma function $\psi$ is strictly concave on (0,∞) and $\phi \left(a\right)=\psi \left(a\right)-\psi \left(a_0\right)$. On this view, updating the approximate posterior over models—by application of Occam’s razor—can be regarded as a discrete (post-hoc) approximation to continual structure learning; i.e., inference over models. One might conjecture that accumulating evidence under uninformative model priors—before updating posterior beliefs about hypotheses—ensures the above inequality holds. Practically, a post hoc scheme means one does not have to entertain multiple models during active inference and learning.

Author contributions

All authors contributed to the conception of the research reported here. KF completed the formal analysis and completion of the initial draft of the manuscript. LDC, AT, CH, CB, TV, and TP contributed substantially to the drafting and revisions of both text and figures, including creation of new figures, and the interpretation of results.

Peer review

Peer review information

Nature Communications thanks Peter Thestrup Waade, and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. A peer review file is available.

Funding

KF discloses support for the research of this work from Wellcome Trust (Ref: 226793/Z/22/Z). TP discloses support for the research of this work from NIHR [Academic Clinical Fellowship (ref: ACF-2023-13-013)].

Data availability

No empirical data were generated for this paper. The simulated data reported in the figures can be reproduced by the software detailed in the code availability statement.

Code availability

The MATLAB routines used to generate the simulations reported here is publicly available via the SPM 25 academic software package (https://www.fil.ion.ucl.ac.uk/spm/), in the DEM toolbox of SPM and is called DEM_reasoning.m. This is also available on GitHub at the following URL: https://github.com/spm/spm/blob/main/toolbox/DEM/DEM_reasoning.m. The version of the code used in this paper is compatible with that associated with the following SHA commit 28d1f103b021f729266cb7ea07baaf4d0285b22a.

Competing interests

The authors declare no competing interests.

Supplementary information

The online version contains supplementary material available at https://doi.org/10.1038/s41467-026-77209-5.