Embodied decisions as active inference

Abstract

Decision-making is often conceptualized as a serial process, during which sensory evidence is accumulated for the choice alternatives until a certain threshold is reached, at which point a decision is made and an action is executed. This decide-then-act perspective has successfully explained various facets of perceptual and economic decisions in the laboratory, in which action dynamics are usually irrelevant to the choice. However, living organisms often face another class of decisions—called embodied decisions—that require selecting between potential courses of actions to be executed timely in a dynamic environment, e.g., for a lion, deciding which gazelle to chase and how fast to do so. Studies of embodied decisions reveal two aspects of goal-directed behavior in stark contrast to the serial view. First, that decision and action processes can unfold in parallel; second, that action-related components, such as the motor costs associated with selecting a particular choice alternative or required to “change mind” between choice alternatives, exert a feedback effect on the decision taken. Here, we show that these signatures of embodied decisions emerge naturally in active inference—a framework that simultaneously optimizes perception and action, according to the same (free energy minimization) imperative. We show that optimizing embodied choices requires a continuous feedback loop between motor planning (where beliefs about choice alternatives guide action dynamics) and motor inference (where action dynamics finesse beliefs about choice alternatives). Furthermore, our active inference simulations reveal the normative character of embodied decisions in ecological settings – namely, achieving an effective balance between a high accuracy and a low risk of missing valid opportunities.

Author summary

In this study, we introduce a novel modeling approach to explore embodied decision-making, where decisions and actions occur simultaneously in dynamic environments. Unlike traditional models that treat decision and action as separate, our framework, based on active inference, reveals that crucial features of embodied decisions—such as feedback loops between decision and action dynamics—emerge naturally. By simulating real-time decision-making tasks, we show how organisms continuously refine their choices by integrating sensory information and motor dynamics. This allows them to strike a balance between decision accuracy and the need for fast, adaptive actions. Our model offers a new perspective on how decisions are influenced by the actions taken, highlighting the importance of considering motor control as an integral part of decision processes. This approach broadens the scope of decision-making research and provides new insights into behavior in ecologically valid, time-sensitive contexts, with potential implications for neuroscience, cognitive science, and fields involving human and animal behavior.

1. Introduction

Decision-making is traditionally conceptualized as a serial, decision-then-act process, in which sensory evidence is accumulated until a certain threshold, at which point a decision is made and an action is executed. This approach, as formalized by drift-diffusion and related models, has been very useful in analyzing behavioral and neural data from laboratory studies of perceptual and economic decisions [1–3]. In these studies, participants select between fixed choice alternatives (usually two) reflecting perceptual judgments (e.g., motion discrimination) or economic offers (e.g., lotteries).

However, animals often face a different class of embodied decisions, which imply the choice between courses of actions to be immediately executed in dynamic environments; for example, for a lion, the choice of which gazelle to chase or for a soccer player, the choice of which teammate to which passing the ball [4–10]. To face with the demands of these embodied decisions, animals often need to specify, prepare and sometimes execute actions in parallel to the decision process—as captured by the notion of affordance competition [5, 11].

These considerations motivate a series of experiments that use continuous measures of performance during perceptual and economic choices; for example, tracking hand kinematics—using a computer mouse—during the movement from the start position to a response button [12, 13]. Despite their simplicity, these experiments permit analyzing the dynamic processes leading to a decision and the reciprocal influences between the ongoing deliberation and movement (i.e., decide-while-acting or continuous decisions [14]). They reveal that participants move and deliberate simultaneously: they generally start moving very early, either toward a specific target or in the middle, if they are more uncertain; they often revisit their decisions in the middle of the trial, as apparent by the curvature of their movements; and they sometimes change mind between the targets, as evident by drastic changes of trajectory [15–18]. These findings eschew serial models and are better explained by parallel [19] or continuous flow models [20] in which unfolding perceptual and decision processes concurrently drive the preparation and possibly the overt execution of one or more responses in parallel—meaning that movements during the task provide a continuous readout of the ongoing deliberation. From a normative perspective, parallel models provide a way to realize decisions faster, which is crucial to survival as it avoids the risk of losing valued opportunity—although sometimes at the cost of reduced accuracy [21].

Crucially, some studies of embodied decisions reveal feedback effects of action dynamics to decision processes that were previously ignored in serial and even parallel decision-making models. For example, a recurring finding is that motor costs associated with different choice alternatives influence perceptual and economic decisions. During ambiguous perceptual decisions [22, 23] and value-based decisions [24], participants show a bias to select the response choice associated with the less costly movement. Changes of mind during a perceptual task are less frequent if the costs associated with changing movement direction are greater, such as when response buttons are farther apart [25]. Similarly, during an economic task, changes of mind following a perturbation of the movement trajectory are sensitive to the current state (position and velocity) of the motor system and are less frequent when counteracting the perturbation would be more costly [26].

These and other studies suggest not only that deliberation continues after movement onset (in accordance with parallel models) but also that it is affected by feedback from action dynamics (e.g., motor costs). This motivates a novel class of embodied decision models in which action is not the inert outcome of a decision process but influences it, forming a closed loop [21, 27]. These models are motivated by the fact that, from an embodied perspective, the goal of the agent is not just selecting between choice alternatives (as in classical setups) but also simultaneously selecting between potential courses of action to reach the targets (often within a deadline) and tracking the action itself—which means that both decision and action processes need to be jointly and continuously optimized. In turn, at the neural level, embodied decisions might require a distributed consensus across various brain networks that process outcome values and motor plans, rather than a centralized process as traditionally assumed [28].

Here, we show that the key signatures of embodied decisions emerge naturally in active inference, a framework that models perception and action selection as two aspects of the same objective of free energy minimization [29–33]. By simulating embodied decisions as an active inference process, we are able to reproduce various empirical findings about the parallel unfolding of actions and decisions in time, as well as feedback effects of movement dynamics in perception. Furthermore, we illustrate the normative advantages of embodied choices over serial choices under time pressure.

2. Results

We illustrate the functioning of an active inference model of embodied decisions by simulating a two-alternative forced choice (2AFC) decision task with time-varying information, i.e., in which evidence for one choice or the other, expressed in terms of sequentially provided cues, changes throughout each trial, as in [34, 35] (Fig 1A). The agent has to move the 3-DoF arm from a start position (small blue dot at the center) to reach a left (red circle) or right (green circle) target button, to report which target has or will have more cues in it. During the task, 15 cues appear, one after the other, either in the left or the right circle, and then disappear leaving only the last one visible. The agent can start moving at any moment and the cues continue to appear normally during movement. The trial ends when the agent reaches one of the two buttons (or within a deadline).

Fig 1. Embodied decision setup and active inference model.

(a) Experimental setup, during three consecutive discrete time steps $\tau$. The agent controls a 3-DoF arm (the three segments in blue), which starts at a home position (blue dot) at an equal distance from the two targets (red and green circles). The current cue is represented with a big purple dot, while the old cues are represented with smaller gray dots. For each trial, the agent has to reach with the hand the target it believes will contain more cues. The hand trajectory is represented with a thinner blue line. (b) Hybrid active inference model for embodied decisions. The model comprises four processes, numbered from 1 to 4. In the first process, discrete hidden states $\mathit{s}$, encoding the probability that each target is the correct choice for the current trial, are iteratively inferred by discrete cues ${\mathit{o}}_c$ by inverting the cue likelihood matrix ${\mathit{A}}_c$. In the second process, the hidden states $\mathit{s}$ generate a particular combination of discrete hand dynamics ${\mathit{o}}_h$ through the extrinsic likelihood matrix ${\mathit{A}}_h$. Each hand dynamics o_h,m is related to a continuous dynamics function ${\mathit{f}}_m$, where the target positions are defined (see [36, 37] for more details). A forward message imposes a prior over the hand velocity (in Cartesian coordinates) ${\mathit{x}}_h^{\prime }$, while a backward message infers the related Cartesian position of the hand ${\mathit{x}}_h$, ready for kinematic and dynamic inversions. In the third process, for each continuous time step t, the current position and velocity of the hand (in a visual domain) are inferred by continuous observations ${\mathit{y}}_v$ and ${\mathit{y}}_v^{\prime }$, via the corresponding likelihood functions (${\mathit{g}}_v$ and ${\mathit{g}}_v^{\prime }$). The inferred hand trajectory flows back toward the prior ${\mathit{\eta }}_h$ for action execution (see Section 4.2). For each discrete time step $\tau$, the target probabilities $\mathit{s}$ are also inferred by the current motor trajectory. Finally, in the fourth process, the prior $\mathit{D}$ over the correct target is updated across trials, implementing habitual learning. Of note, some readers may find the edge from $\mathit{s}$ (the agent’s belief about the latent state) to ${\mathit{o}}_c$ (an observation) somewhat unconventional. This is because, in many contexts, observations are typically assumed to be generated from the true latent states rather than the agent’s beliefs about them. However, this formulation is standard in active inference studies, even when representing the agent’s generative model as in this figure.

Crucially, by manipulating the sequence of cues, we compare the agent’s decision dynamics in three conditions (or trial types): congruent, in which a greater proportion of cues initially appears in the correct target; incongruent, in which a greater proportion of cues initially appears in the incorrect target; and neutral, in which the proportion of cues initially appearing in both targets is balanced.

Below, we show that a hybrid active inference model (i.e., a model composed of both discrete and continuous variables) that jointly optimizes decisions and actions reproduces these signatures of embodied choices. The model can be decomposed into four interacting processes—evidence accumulation, motor planning, motor inference, and statistical learning across trials (along with habit formation)—see Fig 1B for a schematic illustration and Section 4.1 for technical details. Below, we discuss these processes and present simulations about how they affect the agent’s decision and action processes.

2.1. The first process: evidence accumulation for the choice alternatives

The first process is responsible for the accumulation of sequential evidence for the choice alternatives. It includes discrete hidden states $\mathit{s}$, which encode the probability that each target is the correct choice for the current trial (i.e., the one that will contain the most cues). They are sampled from a categorical (here, binomial) distribution, i.e., $\mathit{s}=Cat(\mathit{D})=\left[\begin{array}{cc}{s}_{t1}& s_{t2}\end{array}\right]$, where $\mathit{D}$ are the parameters of a Dirichlet distribution and define the agent’s prior beliefs. In the following simulations, we initialize them with a uniform distribution for each trial. The discrete hidden states $\mathit{s}$ generate two discrete predictions in parallel. The former—computed through the likelihood matrix ${\mathit{A}}_c$—is a prediction of the cue that the agent will observe next, with ${\alpha }_c$ playing the role of a precision (i.e., a confidence or certainty in the likelihood mapping between cause and consequence), similar to the drift rate in drift diffusion models. The latter—computed through the likelihood matrix ${\mathit{A}}_h$—corresponds to the hand dynamics: it predicts whether the hand will move toward the left target, toward the right target, or not move at all (with probability $1-{\alpha }_h$). In short, precision or inverse uncertainty (i.e., ambiguity) of the two mappings affects how fast evidence accumulation unfolds, and which movement strategy the agent adopts:

$${\mathit{A}}_c=\left[\begin{array}{cc}{\alpha }_c& 1-{\alpha }_c\\ 1-{\alpha }_c& {\alpha }_c\end{array}\right]{\mathit{A}}_h=\left[\begin{array}{cc}{\alpha }_h& 0\\ 0& {\alpha }_h\\ 1-{\alpha }_h& 1-{\alpha }_h\end{array}\right]$$ (1)

Since the agent has to update its (Bayesian) beliefs based upon continuous observations, the requisite probability transition matrices $\mathit{B}$ (as defined in Section 4.1) reduce to the identity matrix; such that in the absence of any observations, posterior estimates of latent or hidden states do not change. At each discrete step $\tau$, a particular cue ${\mathit{o}}_c$ and a particular hand dynamics ${\mathit{o}}_h$ are observed and compared with the corresponding predictions. Hence, the inference of the discrete hidden states $\mathit{s}$ follows the equation:

$${\mathit{s}}_{\tau }={\sigma }_w(k_d\ln {\mathit{s}}_{\tau -1}+\ln {\mathit{A}}_c^T{\mathit{o}}_{c,\tau }+k_h\ln {\mathit{A}}_h^T{\mathit{o}}_{h,\tau })$$ (2)

where ${\sigma }_w$ is a weighted softmax function:

$${\sigma }_w(\mathit{x})=\frac{e^{w{x}_i}}{{\sum }_k{e}^{w{x}_k}}$$ (3)

with slope (or precision) w. A high value of w ensures fast transitions between discrete states, thus avoiding positions within the two targets. In short, the discrete update is a combination of a prior from the previous step (which is equal to $\mathit{D}$ at the beginning of the trial) and two likelihoods. The first contributes to the accumulation of sensory evidence and iteratively refines the choice based on the sensory cues. The second likelihood links target estimation and hand dynamics; in this way, the variable ${\mathit{o}}_h$ behaves as a sensory signal for the discrete model (similar to ${\mathit{o}}_c$) and permits accumulating evidence from the agent’s movements. We will unpack the role of the second likelihood in the next sections; here, to simulate the standard evidence accumulation, we let the third term depend on a parameter k_h, which we set here to 0 so that the inference of the correct choice only relies on sensory cues ${\mathit{o}}_c$. Finally, we include a parameter k_d acting as a forgetting factor (which might be useful to deal with non-stationary tasks), which we keep fixed at 1 in our simulations. Also, we set ${\alpha }_h=1.0$ (so that the agent starts moving at the beginning of the trial) and ${\alpha }_c=0.6$.

We test the active inference model with the three conditions explained above. In congruent trials, cues move toward the correct target with an initial probability of 80%, which then gradually increases and reaches 100% after 8 cues. In neutral and incongruent trials, the probabilities of the correct target are respectively initialized to 50% and 20% and then increase as in congruent trials. Each trial comprises 21 discrete time steps $\tau$, each in turn comprising 30 continuous steps t. At $\tau =0$, no cue is presented, but the agent can move. For the next 15 time steps, a cue per time step is presented. Finally, in the last 4 time steps, no cue is presented, but the agent can still move and reach the target.

Fig 2 shows that, under all conditions, when movement onset is immediate the agent moves in between the two targets. In the incongruent condition, the agent first moves toward the wrong target and eventually changes mind. These results match qualitatively key empirical findings discussed in the Introduction [15–18]. To assess whether the model generates statistically different trajectories under the three conditions, we simulated 100 trials per condition and considered a widely used index of choice uncertainty: the maximum deviation of the trajectories from an ideal, straight line between the start and the correct target [10, 12]. We found significantly larger maximum deviation in incongruent (M = 125.97, SD = 25.0) compared to neutral (M = 88.19, SD = 30.73) trials and neutral compared to congruent (M = 56.64, SD = 18.56) trials (for all tests, p$<$.001).

Fig 2. Evidence accumulation during (a) congruent trials, (b) neutral trials, and (c) incongruent trials.

The first row shows the dynamics of a sample trial for each condition: specifically, the first plot shows the discrete hidden states $\mathit{s}$ encoding the two target probabilities over continuous time; the second plot shows the cumulative sum of the cue observations ${\mathit{o}}_c$; the third plot shows the distances between the hand and the two targets. Note that the discrete signals are maintained for a whole discrete period $\tau$, generating a stepped behavior. The second row shows the agent’s average trajectory (in dark blue) across 100 trials for each condition. A dotted line of minimum distance between the initial hand position and the left target is also displayed.

2.2. The second process: motor planning and urgency

When facing the same task, different groups of participants might show different strategies; for example, a conservative strategy to postpone movement until they feel sufficiently confident, or a risky strategy to guess the correct choice and start moving immediately [38].

In our model, the selection of a conservative versus risky strategy depends on two things. First, the rate of evidence accumulation depends upon the precision parameter ${\alpha }_c$, such that a greater precision (i.e., less ambiguity) enables the agent to form more precise posterior beliefs about the correct target. Second, the movement urgency rests upon the precision parameter ${\alpha }_h$ (of the likelihood mapping to kinematics), which determines the confidence—the agent has about the correct target—required to initiate movement. Recall that ${\mathit{o}}_h$ (the discrete set of hand dynamics) encodes the probability that the two targets generate a movement toward the left, toward the right, or no movement (i.e., ${\mathit{o}}_h=[o_{h,t1},o_{h,t2},o_{h,s}]$).

Importantly, ${\mathit{o}}_h$ specifies potential action plans, as it communicates with continuous hidden states specifying the instantaneous trajectory (position ${\mathit{x}}_h$ and velocity ${\mathit{x}}_h^{\prime }$) of the agent’s hand in extrinsic (e.g., Cartesian) coordinates. Specifically, each element of ${\mathit{o}}_h$ is linked to a dynamics function:

$$\mathit{f}({\mathit{\mu }}_h)=\left[\begin{array}{c}{\mathit{f}}_{t1}\\ {\mathit{f}}_{t2}\\ {\mathit{f}}_s\end{array}\right]=\left[\begin{array}{c}\lambda ({\mathit{p}}_{t1}-{\mathit{\mu }}_h)\\ \lambda ({\mathit{p}}_{t2}-{\mathit{\mu }}_h)\\ \mathbf{0}\end{array}\right]$$ (4)

where ${\mathit{\mu }}_h$ is the belief over the hand position ${\mathit{x}}_h$, $\lambda$ is an attractor gain, while ${\mathit{p}}_{t1}$ and ${\mathit{p}}_{t2}$ are the positions of the two targets – assumed to be known and fixed. This mechanism implements the simultaneous preparation of competing motor plans, as also reported in monkey premotor cortex [39].

The mapping between discrete and continuous signals in these hybrid models—with mixed continuous and discrete states—rests on a form of Bayesian model averaging; namely, estimating the movement trajectory by averaging over some discrete models (i.e., hidden states) generating possible trajectories—as explained in Section 4.1. In particular, a prior ${\mathit{\eta }}_{x,h}^{\prime }$ over the hand velocity ${\mathit{x}}_h^{\prime }$ is computed by weighting the three potential trajectories (to reach the two targets and to stay) with their respective probabilities ${\mathit{o}}_h$ that the agent plans for a given discrete goal:

$${\mathit{\eta }}_{x,h}^{\prime }={\mathit{o}}_h·\mathit{f}({\mathit{\mu }}_h)$$ (5)

This desired velocity enters the update of the continuous hidden states as a dynamics prediction error ${\mathit{\epsilon }}_{x,h}={\mathit{\mu }}_h^{\prime }-{\mathit{\eta }}_{x,h}^{\prime }$, expressing a composite motion that the continuous model will realize. The inverse process, i.e., the inference of the current trajectory, is explained in the following section, and more details are found in Section 4.1.

Fig 3 illustrates the effect on movement onset and velocity with three levels of urgency, during an incongruent trial. Since k_h = 0, the evidence accumulation is the same for the three cases, but the trajectories change depending on the agent’s urgency to move, giving rise to risky, medium and conservative strategies. High and intermediate levels of urgency produce riskier strategies that initially move toward the wrong target and then manifest changes of mind. Low urgency produces a conservative strategy that moves directly toward the correct target, but has higher reaction times when the two target probabilities are too close, failing to complete the trial within the deadline. This is because the trajectories generated by the discrete model are constantly weighted by the stay dynamics, hence low precision ${\alpha }_h$ (which means low urgency) results not just in late movement onset but also in slower motion. This simulation illustrates that manipulating urgency provides flexibility in the link between evidence accumulation and movement dynamics. With high urgency, the agent moves earlier and takes the risk of failing, whereas with low urgency, the agent may wait until it accumulates sufficient evidence to reach very high confidence about the correct target. Interestingly, urgency and speed of evidence accumulation can interact, as shown in Fig 4. Finally, this example illustrates that when urgency is set to a very low level, the embodied model approximates – or even transforms into—a serial decide-then-act model, initiating movement only after complete evidence accumulation. This result highlights the critical role of urgency in shaping the interaction between decision-making and movement.

Fig 3. Motor planning with (a) a risky strategy (high urgency, or ${\mathit{\alpha }}_{\mathit{h}}\mathbf{=}\mathbf{1.0}$), (b) a medium strategy (medium urgency, or ${\mathit{\alpha }}_{\mathit{h}}\mathbf{=}\mathbf{0.55}$), and (c) a conservative strategy (low urgency, or ${\mathit{\alpha }}_{\mathit{h}}\mathbf{=}\mathbf{0.5}$). (d) First panel: dynamics of the discrete hidden state of the first target over discrete time $\mathit{\tau }$, which in this case are the same for every strategy.

Second panel: dynamics of the discrete variable o_t1 of the first target. Third panel: dynamics of the discrete variable o_s of staying in position. The fourth panel shows the L2-norm of the belief ${\mathit{\mu }}_h^{\prime }$ over the hand velocity in continuous time t. The vertical dashed lines represent the movement onset for each strategy (using $||{\mathit{\mu }}_h^{\prime }||>10$ as threshold).

Fig 4. Interaction between urgency and speed of evidence accumulation.

(a) Fast evidence accumulation (${\alpha }_c=0.9$) and low urgency to move (${\alpha }_h=0.5$). (b) Slow evidence accumulation (${\alpha }_c=0.51$) and high urgency to move (${\alpha }_h=0.8$). (c) While movement dynamics look similar, the evolution of the hidden states and hand dynamics are different.

2.3. The third process: motor inference and commitment

Various studies found that participants who initiate a movement show high commitment to the initially selected target—even in the face of contrasting evidence—when the costs required to reach the alternative target increase [25, 26]. Interestingly, in our model, this commitment emerges automatically during model inversion. As the agent jointly infers the correct target and the optimal discrete hand dynamics to reach it, there is a reciprocal interaction between the (top-down) process of motor planning and its dual (bottom-up) process of motor inference (Fig 1B). This is because in our model, not only the agent makes predictions over the cue that will be observed next (i.e., ${\mathit{A}}_c\mathit{s}$), but also over the hand trajectory (i.e., ${\mathit{A}}_h\mathit{s}$). The latter prediction entails a causal relation between discrete goals (in this case, the probability of the two targets) and the agent’s movements (here, the dynamics needed to reach the targets)—as explained in Section 4.1. The key implication of this perspective is that the probability of the two targets can be estimated from the hand trajectory itself (${\mathit{o}}_h$). As a consequence, a self-evidencing mechanism takes place, during which a target inferred by some cues produces a plan to reach it, which in turn confirms the initial agent’s estimate. In other words, via motor inference, movement stabilizes (i.e., reduces the uncertainty about) the decision and creates commitment to the initially selected target.

The rich interplay between evidence accumulation, motor planning and motor inference gives rise to three competitive processes. The first competition occurs when estimating the continuous hidden states of the hand trajectory, defined by the generalized beliefs ${\dot{\tilde{\mathit{\mu }}}}_h=[{\mathit{\mu }}_h,{\mathit{\mu }}_h^{\prime }]$:

$${\dot{\tilde{\mathit{\mu }}}}_h=𝒟{\tilde{\mathit{\mu }}}_h-{\partial }_{{\tilde{\mu }}_h}ℱ=\left[\begin{array}{c}{\mathit{\mu }}_h^{\prime }-{\mathit{\pi }}_{\eta ,h}{\mathit{\epsilon }}_{\eta ,h}+{\partial }_{{\mu }_h}{\mathit{g}}_v^T{\mathit{\pi }}_v{\mathit{\epsilon }}_v+{\partial }_{{\mu }_h}{\mathit{f}}^T{\mathit{\pi }}_{x,h}{\mathit{\epsilon }}_{x,h}\\ \\ {\partial }_{{\mu }_h^{\prime }}{\mathit{g}}_v^{\prime T}{\mathit{\pi }}_v^{\prime }{\mathit{\epsilon }}_v^{\prime }-{\mathit{\pi }}_{x,h}{\mathit{\epsilon }}_{x,h}\end{array}\right]$$ (6)

This update rule comprises three main components. First, a prior prediction error ${\mathit{\epsilon }}_{\eta ,h}$ (along with its precision ${\mathit{\pi }}_{\eta ,h}$) that biases the belief over the hand position. Although not shown in the model, the latter is linked, through forward kinematics, to an intrinsic continuous model encoding proprioceptive trajectories (e.g., expressed in joint angles). As a consequence, the backward message sent from the belief over the hand position to the intrinsic model performs inverse kinematics, eventually driving action. See Section 4.2 and [40, 41] for more details about kinematic inference. Second, visual prediction errors ${\mathit{\epsilon }}_v={\mathit{g}}_v({\mathit{\mu }}_h)-{\mathit{y}}_v$ and ${\mathit{\epsilon }}_v={\mathit{g}}_v^{\prime }({\mathit{\mu }}_h^{\prime })-{\mathit{y}}_v^{\prime }$ computed through likelihood functions ${\mathit{g}}_v$ and ${\mathit{g}}_v^{\prime }$, and expressing the difference between predicted and observed hand position/velocity. These terms are backpropagated for both orders, to keep the belief close to the actual hand trajectory. Third, the dynamics prediction error ${\mathit{\epsilon }}_{x,h}$ defined in the previous section, affecting both orders as a forward or backward message.

The second (and most interesting) competition happens when this continuous belief clashes with the desired hand dynamics needed to reach the selected target. While from a top-down perspective the dynamics defined in Eq 4 act as potential trajectories averaged by the probabilities ${\mathit{o}}_h$, from a bottom-up perspective they are used to infer the most likely explanation of the real trajectory. More formally, the discrete hand dynamics ${\mathit{o}}_h$ at time $\tau$ is found via Bayesian model comparison, i.e., by comparing the discrete prediction ${\mathit{A}}_h{\mathit{s}}_{\tau }$ with the continuous evidence ${\mathit{ℒ}}_h$ of the hand trajectory, accumulated over a time window T:

$${\mathit{o}}_{h,\tau }={\sigma }_w(\ln {\mathit{A}}_h{\mathit{s}}_{\tau }+{\int }_0^T{\mathit{ℒ}}_{h}dt)$$ (7)

As before, ${\sigma }_w$ is a weighted softmax whose slope w here controls how fast the transition between different dynamics occurs. High and low values of w correspond to abrupt and gradual movement onsets, respectively. In practice, evidence accumulation reduces to a sum over the continuous steps that comprise a single discrete step. For each mth dynamics of Eq 4, the log evidence ${ℒ}_{h,m}$ is computed by:

$${ℒ}_{h,m}=\frac{1}{2}({\mathit{\mu }}_{h,m}^{\prime T}{\mathit{p}}_{x,h}{\mathit{\mu }}_{h,m}^{\prime }-{\mathit{f}}_m({\mathit{\mu }}_h{)}^T{\mathit{\pi }}_{x,h}{\mathit{f}}_m({\mathit{\mu }}_h)-{\mathit{\mu }}_h^{\prime T}{\mathit{p}}_{x,h}{\mathit{\mu }}_h^{\prime }+{\mathit{\eta }}_{x,h}^{\prime T}{\mathit{\pi }}_{x,h}{\mathit{\eta }}_{x,h}^{\prime })$$ (8)

More details about the reduced posteriors ${\mathit{\mu }}_{h,m}^{\prime }$ and prior precisions ${\mathit{p}}_{x,h}$ can be found in Section 4.1, and in [36, 37, 42]. Here, we note that each potential dynamics function ${\mathit{f}}_m$ is compared to the current dynamics ${\mathit{\mu }}_h$ inferred via sensory observations. As a result, the log evidence ${\mathit{ℒ}}_h$ assigns higher values to the potential trajectories that better match the real one. For instance, if the arm moves toward the red target, the difference in magnitude between the real trajectory and the potential trajectory associated with reaching the red target will be lower, while the one associated with the opposite trajectory will be higher. Hence, the agent can understand which of the two targets is reaching. This evidence is further compared with the discrete prediction ${\mathit{A}}_h{\mathit{s}}_{\tau }$ encoding the agent’s desires (reaching the red target, reaching the green target, or staying), so the final hand dynamics will be a combination of the two contributions.

The third and final competition emerges at the intersection between motor inference and evidence accumulation, as highlighted in Eq 2. Since the discrete hand dynamics are constantly steered toward the current hand trajectory, the latter becomes a predictor for the correct target. This ultimately produces a commitment toward the chosen dynamics—expressed in the rightmost term of Eq 2.

One way to appreciate commitment is to consider that, as the distance between the two targets increases, human participants make fewer changes of mind [25]. In line with this, when simulating 100 neutral trials with targets at different distances, we found that changes of mind were more frequent with targets at low distance (n = 29) than at medium (n = 20) and high distance (n = 8)—see Fig 5 for some sample trials. The reason for this result lies in Eq 7. Since the discrete hand dynamics ${\mathit{o}}_h$ are used to infer the correct choice alternative, closer targets are scored with greater probabilities than farther targets. Furthermore, since the softmax function amplifies the differences between current and potential trajectories, a target is assigned an increasingly lower probability, the farther it is from the other one. This is evident in Fig 6, which shows the potential trajectories to reach both targets for the entire trial, the actual trajectory, and the log evidence accumulated over time. Another way to appreciate commitment is by comparing the agent’s behavior during congruent and incongruent conditions, with motor inference (k_h>0) and without it (k_h = 0)—see Fig 7. When motor inference is active in the congruent condition (Fig 7B), the hand dynamics stabilize the decision: the target is inferred faster than without motor inference (Fig 7A) and the second wrong cue (observed at $\tau =9$) is ignored. When motor inference is active under the incongruent condition (Fig 7D), the agent could commit to a wrong decision ignoring contrasting evidence—a behavior that is not observed without motor inference (Fig 7C).

Fig 5. Commitment during an incongruent trial, with (a) low, (b) medium, and (c) high distance between the two targets, with ${\mathit{k}}_{\mathit{h}}\mathbf{=}\mathbf{0.15}$ and ${\mathit{\alpha }}_{\mathit{h}}\mathbf{=}\mathbf{1.0}$.

For each condition, the first plot shows the discrete hidden states $\mathit{s}$; the second plot shows the L2-norms of the estimated and true hand velocities ${\mathit{\mu }}_e^{\prime }$ and ${\mathit{x}}_e^{\prime }$; the third plot shows the distances between the hand and the two targets. For each condition, the second row also shows the agent’s final trajectories, in dark blue.

Fig 6. Commitment during an incongruent trial, with (a) low, (b) medium, and (c) high distance between the two targets; continued from Fig 5.

(a–c) The panels show the direction and magnitude of the estimated velocity ${\mathit{\mu }}_h^{\prime }$ (in dark blue), the potential trajectory needed to reach the first target ${\mathit{f}}_{t1}$ (in red) and the second target ${\mathit{f}}_{t2}$ (in green), for each condition. These potential trajectories are used to estimate which of the two targets is more likely to have generated the current trajectory. (d) The top panel shows the discrete hand dynamics o_h,t1 of the first target over discrete time $\tau$; the middle panel shows the log evidence ${ℒ}_{h,t1}$ of the hand trajectory associated with the first target, on a logarithmic scale; the bottom panel shows the normalized log evidence of the first target.

Fig 7. Commitment to an initially selected target resulting from motor inference.

Panels a– b compare two agents—without motor inference (a, k_h = 0) and with motor inference (b, k_h = 0.2)—during a congruent trial. Panels c– d compare the same two agents—without motor inference (c, k_h = 0) and with motor inference (d, k_h = 0.2)—during an incongruent trial. In all conditions, ${\alpha }_h=0.8$. For each panel, the first plot shows the discrete hidden states $\mathit{s}$; the second plot shows the discrete hand dynamics for reaching both targets and staying in position; the third plot shows the L2-norms of the estimated hand velocity ${\mathit{\mu }}_h^{\prime }$, compared with the three potential trajectories ${\mathit{f}}_{t1}$, ${\mathit{f}}_{t2}$, and ${\mathit{f}}_s$. Note that although the stay dynamics function ${\mathit{f}}_s$ is (initially) the closest to the actual trajectory, the related probability o_h,s decreases rapidly, as soon as the predictions ${\mathit{A}}_h\mathit{s}$ shift toward one of the targets, showing the top-down influence from choice to movement. The fourth (bottom) plot shows the agent’s movement trajectories, in dark blue.

Finally, we considered the tradeoffs between various models: a serial, decide-only model that makes decisions between targets by considering the discrete hidden states $\mathit{s}$ and then moves instantaneously to the selected target; a serial decide-then-act model that is identical to the decide-only model, but requires a fixed movement time of 125 time steps (i.e., the average movement time in our setup) to reach the selected target; and two embodied choice models, one with motor inference (k_h = 0.1) and one without it (k_h = 0). For this comparison, we simulated 100 neutral trials, by varying the urgency ${\alpha }_h$ and the drift rate ${\alpha }_c$ to obtain a wide range of solutions. Fig 8A shows the results of the comparison. As expected, the speed-accuracy curve of the decide-only model appears to be the best (i.e., the leftmost). However, this advantage is misleading, as the model moves instantaneously. It is therefore inappropriate as a model of human decision-making behavior, but it is shown here to illustrate idealized performance. When movement time is accounted for, the decide-only model becomes the decide-then-act model, which is outperformed by both embodied models—whether without or with motor inference—the latter being the best overall. Furthermore, Fig 8B shows that across various levels of urgency ${\alpha }_h$, there is a significant correlation between the speed of embodied choice and confidence (i.e., probability of the correct choice) as reported empirically [43]. In summary, embodied choice models show a better speed-accuracy curve than serial decide-then-act models. The fact that they afford faster decisions might be particularly advantageous in ecological settings, where slow decisions imply missing valuable alternatives [4, 21]. The relative advantages of including or not including motor inference are likely to be task-dependent. The embodied model using motor inference appears to be advantageous when processing congruent (Fig 7) and neutral trials (Fig 8A), but its commitment to the initially selected target can lead to more incorrect decisions when processing incongruent trials (Fig 5).

Fig 8. (a) Speed-accuracy curves for four models: two serial models (decide-only, decide-then-act) and two active inference models (with and without motor inference).

The former two were sampled by varying the decision threshold in 500 trials, while the latter two were sampled by running 100 trials for different values of urgency (from medium-high urgency, i.e., ${\alpha }_h=0.7$, to low urgency, i.e., ${\alpha }_h=0.5$). The samples were then fitted into a curve. Motor inference was realized with k_h = 0.1. For all conditions, ${\alpha }_c=0.6$. To allow the agents to complete the trials on time with low levels of urgency, we set the maximum trial duration to 630 time steps. (b) Pearson product-moment correlation coefficient between the belief over the hand trajectory ${\mathit{\mu }}_h^{\prime }$ and the probability of the correct choice, computed for two conditions (with and without motor inference), with 500 neutral trials per condition, and with different levels of urgency.

2.4. The fourth process: statistical learning and habit formation

During various cognitive tasks such as the Flanker [44] and the Posner task [45], it is possible to learn statistical regularities, such as the probability of the correct response or the validity of cues across trials. In these tasks, trial sequence effects are often reported, indicating that participants form expectations across trials that influence their subsequent responses and movements [46]. The fourth process of our model implements this kind of statistical learning, which simply amounts to keeping count of the Dirichlet priors over the discrete hidden states $\mathit{s}$ across trials. After every trial, these counts are updated according to:

$${\mathit{d}}_n=\omega {\mathit{d}}_{n-1}+\eta \mathit{s}$$ (9)

where n is the trial number, $\omega$ is a forgetting factor of older trials, and $\eta$ is the learning rate of new trials (usually initialized with a reasonably high value, reflecting a high confidence over the prior belief, ${\mathit{d}}_0$). Then, the counts are normalized to compute the priors of the correct response for the next trial.

Fig 9 shows the effects of learning the prior over the correct response, during 50 incongruent trials. During the first 10 trials, the correct (left) response remains stable and then it is reversed. During the early trials of the first (learning) phase (dark blue in Fig 9A), the agent moves toward the wrong direction and then changes mind. However, in later trials (dark red in Fig 9A) it gradually begins to move early toward the correct target, anticipating the transition in the accumulation of wrong cues. In parallel, movement onset decreases (Fig 9B). These results show that a strong prior can overcome conflicting evidence. After the reversal at trial 10, the discrete prior for the first target slowly decreases, as the Dirichlet counts for the second target begin to increase (Fig 9B). In early trials, movement curvature increases and movement onset is slower, as the agent is uncertain about the correct distribution underneath the cue sampling. In late trials, movement curvature decreases and movement onset fastens, as the agent learns the novel contingencies.

Fig 9. Statistical learning of the prior for the correct choice over 50 incongruent trials.

The correct choice is fixed in the first 10 trials, but is reversed in the next 40 trials. (a) Hand trajectories in equally spaced trials, during learning (top) and reversal learning (bottom). Dark blue trajectories represent early trials, while dark red trajectories are late trials. Here, k_h = 0, $\omega =0.99$, $\eta =0.2$, ${\alpha }_c=0.6$, and ${\alpha }_h=0.6$. (b) The five panels show Dirichlet counts $\mathit{d}$; discrete priors $\mathit{D}$; time step of movement onset across trials; discrete hidden state s₁ in 5 equally spaced trials for learning; and for reversal learning over discrete time $\tau$. The vertical dashed lines indicate the time step when reversal occurs.

These results show that our model can incorporate sequence effects that emerge during cognitive tasks [46]. Note that while we focus on learning the prior probability of the correct choice, other model parameters, such as the uncertainties of the likelihood matrices ${\mathit{A}}_c$ and ${\mathit{A}}_h$, could be updated using the same approach – see also [47].

3. Discussion

For many years, the dominant view regarding human and animal behaviors has been that of a serial, decide-then-act strategy. However, various studies show that during embodied decisions that require simultaneously specifying and selecting between alternative action plans, the serial view is insufficient. These studies report early movement onset, changes of mind, and the influence of motor costs over decisions, suggesting that decision and action processes unfold in parallel and reciprocally influence each other [10, 12, 13, 21, 27]. Here, we show that these signatures of embodied decisions emerge naturally in active inference: a framework that jointly optimizes decisions and actions under a free energy minimization imperative [29, 31]. Our simulations highlight that four model processes—evidence accumulation, motor planning, motor inference, and statistical learning—form a closed loop, allowing decision and action processes to influence each other reciprocally. The resulting embodied models attain a better speed-accuracy tradeoff, compared to serial models (Fig 8A)—suggesting that they could confer ecological advantages [4, 21].

An innovative aspect of our model is the reciprocal interaction between motor planning and motor inference. During motor planning, the agent’s inference of the correct choice generates predictions about the next discrete hand dynamics, which are converted into a continuous motor plan to reach the associated target. In turn, during motor inference, the agent uses the action dynamics as evidence for the correct choice. In other words, the agent treats its own behavior as a source of information [48]. This mechanism implies that movement stabilizes decisions and creates commitment. Furthermore, it explains key aspects of embodied decisions, such as the fact that motor costs apparent before a task [22] or changing during it [26] influence decision outcomes.

Note that there are two alternative perspectives on (or interpretations of) motor costs in embodied decisions. According to the value-based perspective [21, 25, 27], during movement the agent continuously estimates the cost of the actions to make the alternative choices, then combines the estimated cost with the correct choice (inferred by sensory evidence) to decide the next move. Instead, the perspective offered here is based on (active) inference. Our model does not explicitly compute motor costs, but rather the probabilities of discrete hand dynamics ${\mathit{o}}_h$: in short, the agent just tries to infer the correct target from the information at its disposal (including its own movements), and a high motor cost only means a potential dynamics that poorly explains the present context. The two perspectives, active inference and value-based, are mathematically related given the duality of inferential formulations (that use probabilities) and control formulations (that use costs) [49, 50]. But since in active inference (motor) costs are absorbed into (prior) probabilities, its appeal lies in affording embodied decisions using the same inferential machinery required for action control, without the need to compute additional quantities (such as motor costs) on the fly. The same inferential machinery also permits considering other types of prior preferences (or motor costs). For example, including in the model a prior preference for biomechanically simpler movements would automatically favor targets that are closer or can be reached more easily, as empirically observed [22]. Furthermore, while this study focused on perceptual decision tasks, the same model could also be applied to value-based decisions, where different choice targets are associated with varying values or rewards. This could be achieved by assigning a stronger prior preference to targets with higher economic value. Such an extension would also enable the study of how economic value influences motivation and movement vigor in embodied choice tasks [51]. Another way prior preferences (over policies or outcomes) could be leveraged is to account for biases, such as repetition biases that develop across trials. Studies have demonstrated sequence effects in perceptual [52] and visuomotor tasks [53], showing that recent trial history can bias subsequent decisions. The simulations in Fig 9 illustrate how certain biases or habits can emerge from the statistical learning of previous responses. However, habits can also arise from other aspects of interaction beyond simple repetition. For instance, a reward could become associated with a target and influence movement even if the target changes position—an effect that cannot be explained solely by repeating the same movement. Accounting for these and other cognitive aspects of biases, beyond mere movement repetition, may require increasing prior preferences for abstract outcomes (e.g., reaching a rewarding target regardless of its spatial position). Investigating this possibility remains an open question for future research.

Another key aspect of our model is the fact that by modulating the urgency to move, it can model a range of strategies—from riskier to more conservative—observed in empirical studies [38]. Future work might explore how the generative model advanced here could be inverted, to identify personalized parameters (e.g., a person’s urgency) from behavioral data. While we covered several important aspects of embodied decisions in active inference, we kept the focus on the relationship between discrete decisions and continuous dynamics. A more realistic model would also consider discrete dynamics, which may be crucial for explaining how humans optimize the number of successes accumulated in a limited period—as analyzed in [34]. Furthermore, while our model explains how movement can stabilize decisions, it does not include other stabilization mechanisms, such as the modulation of sensory precision (i.e., Kalman gain) [54–56], which might be covered in future studies.

An important avenue for future research is the empirical validation of the embodied choice models introduced here. In this study, we provided normative arguments for the benefits of embodied models in terms of better speed-accuracy curves compared to serial strategies. Furthermore, we have shown that the model reproduces qualitative aspects of data, such as the increase in the curvature of trajectories and the frequency of “changes of mind” with choice uncertainty [15–18], as well as the correlation between the speed of embodied choice levels and confidence (i.e., probability of the correct choice) as reported empirically [43]. Future studies could use embodied choice models to fit empirical data, by exploiting the fact that different kinds of behavior can be elicited by changing the precision of various likelihood mappings—as shown in [47]. In principle, it would be possible to optimize the precisions of a generative model of a given experimental paradigm to match the behavior of an experimental subject. This would provide an opportunity for computational phenotyping of subjects (or patients) in terms of the (precision of) key components of their generative models [57]. This kind of computational phenotyping has been used to characterize the prior precisions and preferences of psychiatric subjects using choice behavior and, in principle, could be extended to cover the embodied decision-making considered in this work. This would allow for the systematic study of the sensitive dependence of behavior on the relative precision of various likelihood mappings as a potentially important aspect of embodied decision-making. Notably, while each of the four processes illustrated in this study operates in a relatively straightforward manner, their interactions are more complex. It remains an open question whether certain parameter combinations are particularly effective in specific embodied contexts, such as those characterized by varying levels of urgency or precision. For instance, our simulations suggest that the benefits of motor inference depend on both trial statistics (e.g., the proportion of congruent, neutral, and incongruent trials) and urgency, yet the interplay between these factors remains to be explored. Additionally, it is unclear whether individuals can flexibly select the most appropriate set of parameters for each context or whether they instead reuse suboptimal parameterizations across different contexts.

From a neural perspective, embodied choice models align with sensorimotor theories of decision-making and neural evidence from areas such as the posterior parietal cortex (PPC) and dorsal premotor cortex (PMd), where action plans are dynamically represented and compete for selection [58–60]. Importantly, in the embodied choice model, decision-making is a distributed process that integrates multiple (sub)processes or pathways, consistent with the concept of decision-making as a “distributed consensus” rather than a centralized process [28]. The hybrid architecture shown in Fig 1B could correspond to a (hierarchical) neural architecture involving recurrent dynamics and extensive cortico-basal ganglia-thalamo-cortical loops, which support the continuous interaction between perception, action, and decision processes [61]. For a more detailed discussion of the biological underpinnings of movement control in active inference, see also [62]. Furthermore, the model’s precision-weighting mechanism, which determines how likelihood mappings influence both decisions and movements, could correspond to neuromodulatory processes regulating synaptic efficacy, such as those mediated by dopamine and noradrenaline [63]. Future studies using neuroimaging and electrophysiology could further investigate the neural implementation of our framework and how the variables in Fig 1B correspond to neural activity during embodied decision-making tasks.

4. Methods

4.1. Dynamic hybrid models in active inference

Active inference is a computational theory that proposes a unifying paradigm to understand cognitive processing and behavior in living organisms. It is based upon the free energy principle which states that, in order to survive, every creature must actively minimize surprise [29, 31, 64].

Active inference models have been formulated both in discrete time and in continuous time. However, it is possible to argue that a comprehensive account of living organisms might require hybrid models, which combine both discrete and continuous time formulations [65, 66]. For example, in the human nervous system, the cerebral cortex might operate in a discrete state-space, while the low-level sensorimotor loops might be better understood in terms of continuous representations; the interface between the two is attributed to subcortical structures such as the thalamus or the superior colliculus [67]. Hybrid models have been used to simulate many scenarios such as pictographic reading [65], movements under neurological disorders [62], or interoceptive control [68]. Here, we briefly present a particular instance of such models useful to infer and act upon dynamic trajectories [36, 37, 42]. See [31] for more details about active inference and the free energy principle, and [69] about Bayesian model selection.

Discrete models share resemblances with Hidden Markov Models (HMMs) and are defined as partially observable Markov decision processes (POMDPs) [70, 71]. In particular, they are related to a subfield of machine learning known as planning as inference [72, 73]. Discrete models assume that organisms perceive the environment by optimizing an internal generative model, inferring how external causes lead to sensory signals within the environment (called generative process). Denoting by $\mathit{s}$ the discrete hidden states, by $\mathit{o}$ the discrete outcomes, by $\mathit{\pi }$ the policies (which in active inference are sequences of actions), the agent’s generative model after a discrete period T is factorized as:

$$p({\mathit{s}}_{1:T},{\mathit{o}}_{1:T},\mathit{\pi })=p({\mathit{s}}_1)·p(\mathit{\pi })·{\prod }_{\tau =1}^{T}p({\mathit{o}}_{\tau }|{\mathit{s}}_{\tau })·{\prod }_{\tau =2}^{T}p({\mathit{s}}_{\tau }|{\mathit{s}}_{\tau -1},\mathit{\pi })$$ (10)

where every distribution is assumed to be categorical:

$$\begin{gathered} p({\mathit{s}}_1)=Cat(\mathit{D}) \\ p(\mathit{\pi })=Cat(\mathit{E})p({\mathit{o}}_{\tau }|{\mathit{s}}_{\tau })=Cat(\mathit{A}) \\ p({\mathit{s}}_{\tau }|{\mathit{s}}_{\tau -1},\mathit{\pi })=Cat({\mathit{B}}_{\pi ,\tau }) \end{gathered}$$ (11)

Here, $\mathit{D}$ is the prior over the initial state, $\mathit{E}$ is the prior over policies, $\mathit{A}$ is the likelihood (or observation) matrix, and ${\mathit{B}}_{\pi ,\tau }$ is the transition matrix. In order to find the causes of their perceptions, organisms try to infer the posterior distribution:

$$p({\mathit{s}}_{1:T},\mathit{\pi }|{\mathit{o}}_{1:T})=\frac{p({\mathit{o}}_{1:T}|{\mathit{s}}_{1:T},\mathit{\pi })p({\mathit{s}}_{1:T},\mathit{\pi })}{p({\mathit{o}}_{1:T})}$$ (12)

However, this requires computing the intractable model evidence $p({\mathit{o}}_{1:T})$. Hence, organisms are supposed to implement some sort of approximate Bayesian inference by relying on an approximate posterior distribution $q({\mathit{s}}_{1:T},\mathit{\pi })$, and then minimizing the Kullback-Leibler (KL) divergence between the approximate and real posteriors:

$$\begin{gathered} 0\le =D_{KL}[q({\mathit{s}}_{1:T},\mathit{\pi })||p({\mathit{s}}_{1:T},\mathit{\pi }|{\mathit{o}}_{1:T})] \\ ={𝔼}_q[\ln q({\mathit{s}}_{1:T},\mathit{\pi })-\ln p({\mathit{o}}_{1:T},{\mathit{s}}_{1:T},\mathit{\pi })]+\ln p({\mathit{o}}_{1:T}) \\ =ℱ+\ln p({\mathit{o}}_{1:T}) \end{gathered}$$ (13)

The KL divergence still requires the computation of the log evidence $p({\mathit{o}}_{1:T})$; however, we can exploit its non-negativity to instead minimize the first RHS term of the second line of Eq 13 – called variational free energy (VFE) and known in machine learning as the negative ELBO – which ensures that surprise $-\ln p({\mathit{o}}_{1:T})$ is minimized. Then, expressing the approximate posterior by its sufficient statistics ${\mathit{s}}_{\pi ,\tau }$ and conditioning upon a specific policy:

$$\begin{gathered} p({\mathit{s}}_{1:T}|{\mathit{o}}_{1:T},\mathit{\pi })\approx q({\mathit{s}}_{1:T},\mathit{\pi })=q(\mathit{\pi }){\prod }_{\tau }^{T}q({\mathit{s}}_{\tau }|\mathit{\pi }) \\ q(\mathit{\pi })=Cat(\mathit{\pi }) \\ q({\mathit{s}}_{\tau }|\mathit{\pi })=Cat({\mathit{s}}_{\pi ,\tau }) \end{gathered}$$ (14)

We can infer the most likely discrete hidden state at time $\tau$ and under policy $\pi$ by computing the gradient of the VFE ${ℱ}_{\pi }$ of that policy and applying a softmax function to ensure that it is a proper probability distribution:

$${\mathit{s}}_{\pi ,\tau }=\sigma (\ln {\mathit{B}}_{\pi ,\tau -1}{\mathit{s}}_{\pi ,\tau -1}+\ln {\mathit{B}}_{\pi ,\tau }^T{\mathit{s}}_{\pi ,\tau +1}+\ln {\mathit{A}}^T{\mathit{o}}_{\tau })$$ (15)

Via VFE minimization, organisms are able to capture the optimal representation of the environment; however, they are unable to perform any sort of future planning. To do this, they additionally consider unobserved outcomes as random variables, and they infer the most likely policy or sequence of actions that will lead to their preferred outcomes. More formally, if we denote by $p({\mathit{o}}_{\tau }|\mathit{C})$ the probability distribution encoding the agent’s preferred outcomes at some future point $\tau$, the optimal policy is found by minimizing the free energy that the agent expects to perceive in the future – called expected free energy (EFE) and denoted by $𝒢$:

$$\begin{gathered} \mathit{\pi }=\sigma (\ln \mathit{E}-𝒢) \\ {𝒢}_{\pi }\approx {\sum }_{\tau }{D}_{KL}[q({\mathit{o}}_{\tau }|\mathit{\pi })||p({\mathit{o}}_{\tau }|\mathit{C})]+{𝔼}_{q({\mathit{s}}_{\tau }|{\mathit{s}}_{\tau -1},\mathit{\pi })}[H[p({\mathit{o}}_{\tau }|{\mathit{s}}_{\tau })]] \\ ={\sum }_{\tau }{\mathit{o}}_{\pi ,\tau }(\ln {\mathit{o}}_{\pi ,\tau }-{\mathit{C}}_{\tau })+{\mathit{s}}_{\pi ,\tau }\mathit{H} \end{gathered}$$ (16)

where:

$${\mathit{o}}_{\pi ,\tau }=\mathit{A}{\mathit{s}}_{\pi ,\tau }{\mathit{C}}_{\tau }=\ln p({\mathit{o}}_{\tau }|\mathit{C})\mathit{H}=-diag({\mathit{A}}^T\ln \mathit{A})$$ (17)

Eq 16 underlies a stark difference with theories of optimal control and reinforcement learning. These theories assume that hidden states have intrinsic values and that agents infer the optimal policy that maximizes the accumulation of future rewards obtained from the environment. Rather, active inference only assumes that each organism believes that the environment will evolve in a specific way, determined by its phenotype. Under this view, action is just another way, complementary to perception, to minimize free energy (hence, surprise), i.e., to reduce the difference between the agent’s prior beliefs and the actual generative process. In other words, by acting, they make future observations coherent with their internal model—a process known as self-evidencing. Furthermore, the two components of the second and third lines of Eq 16 entail the well-known tradeoff between exploitation and exploration, with the addition that the latter, also called ambiguity, is involved in itinerant and novelty-seeking behavior.

To get a discrete model working with the richness of the continuous input, it is linked to an active inference continuous model [65, 66]. The latter is highly similar to the discrete model described above, with the difference that now only the VFE is minimized (for this reason, a continuous model alone is not capable of advanced decision-making). The standard way to link the two models is by letting a discrete outcome generate a causal variable (e.g., the position of a target to reach) that in turn produces a continuous trajectory; in this way, inverting the model means to infer the target based on the continuous trajectory, and then finding the discrete outcome that best explains the inferred target by comparing it with some fixed positions that the agent knows a-priori. In order to render the agent’s representation more flexible, we can instead generate a continuous trajectory directly by a discrete outcome. Here, we briefly describe this alternative approach.

First, we model the continuous environment with the following non-linear stochastic equations:

$$\begin{gathered} \tilde{\mathit{y}}=\tilde{\mathit{g}}(\tilde{\mathit{x}})+{\mathit{w}}_y \\ 𝒟\tilde{\mathit{x}}=\tilde{\mathit{f}}(\tilde{\mathit{x}})+{\mathit{w}}_x \end{gathered}$$ (18)

where $\tilde{\mathit{x}}$ are continuous hidden states, $\tilde{\mathit{y}}$ are continuous observations, $\tilde{\mathit{g}}$ is a likelihood function defining how hidden states cause observations, $\tilde{\mathit{f}}$ is a dynamics function expressing how the hidden states evolve, and the letter w indicates Gaussian noise terms. Note that the symbol $~$ denotes generalized coordinates of motion encoding instantaneous trajectories (e.g., position, velocity, acceleration, and so on), which in a continuous formulation replace the future states. Also, $𝒟$ is a differential operator that shifts every coordinate of motion by one. The joint distribution of this hybrid generative model is factorized into the following:

$$p(\tilde{\mathit{x}},\tilde{\mathit{y}},{\mathit{o}}_{\tau })=p(\tilde{\mathit{y}}|\tilde{\mathit{x}})p(\tilde{\mathit{x}}|{\mathit{o}}_{\tau })p({\mathit{o}}_{\tau })$$ (19)

where the first two distributions are assumed to be Gaussian, while the last one is the categorical (likelihood) distribution defined before:

$$\begin{gathered} p(\tilde{\mathit{y}}|\tilde{\mathit{x}})=𝒩(\tilde{\mathit{g}}(\tilde{\mathit{x}}),{\tilde{\mathit{\Sigma }}}_y) \\ p(\tilde{\mathit{x}}|{\mathit{o}}_{\tau })=𝒩(\mathit{\eta },{\tilde{\mathit{\Sigma }}}_x) \\ p({\mathit{o}}_{\tau })=Cat(\mathit{A}) \end{gathered}$$ (20)

Here, $\mathit{\eta }$ is the mean (also called full prior) of a complex model that defines the actual evolution of the hidden states. We suppose that the agent maintains M probability distributions which are reduced versions of this full model. Each reduced distribution corresponds to a particular discrete dynamics $o_{\tau ,m}$ and encodes a specific way that the agent thinks the environment may evolve [36, 37]:

$$p(\tilde{\mathit{x}}|o_{\tau ,m})=𝒩({\tilde{\mathit{f}}}_m(\tilde{\mathit{x}}),{\tilde{\mathit{\Sigma }}}_{x,m})$$ (21)

where ${\tilde{\mathit{f}}}_m$ is the dynamics function of the mth reduced model, and ${\tilde{\mathit{\Sigma }}}_{x,m}$ the related precision. These alternative hypotheses act as empirical priors at the lower continuous level. Following the theory of Bayesian model reduction [69, 74], in order to infer the posterior of the full model we first have to compute the posterior probability of each reduced model. In fact, reduced means that the likelihood of some data is equal to that of the full model and the only difference rests upon the specification of the priors – hence, the posterior of a reduced model can be expressed in terms of the posterior of the full model:

$$p(\tilde{\mathit{x}}|\tilde{\mathit{y}},o_{\tau ,m})=p(\tilde{\mathit{x}}|\tilde{\mathit{y}},{\mathit{o}}_{\tau })\frac{p(\tilde{\mathit{x}}|o_{\tau ,m})p(\tilde{\mathit{y}}|{\mathit{o}}_{\tau })}{p(\tilde{\mathit{x}}|{\mathit{o}}_{\tau })p(\tilde{\mathit{y}}|o_{\tau ,m})}$$ (22)

As in the previous discrete formulation, we introduce a full and M approximate posteriors, here assumed to be Gaussian:

$$\begin{gathered} q(\tilde{\mathit{x}})=𝒩(\tilde{\mathit{\mu }},{\tilde{\mathit{C}}}_x) \\ q(\tilde{\mathit{x}}|m)=𝒩({\tilde{\mathit{\mu }}}_m,{\tilde{\mathit{C}}}_{x,m}) \end{gathered}$$ (23)

where $\tilde{\mathit{\mu }}$ and ${\tilde{\mathit{\mu }}}_m$ are the full and reduced posterior means (also called beliefs), while ${\tilde{\mathit{C}}}_x$ and ${\tilde{\mathit{C}}}_{x,m}$ are the full and reduced posterior precisions. Replacing the real posteriors with these approximate posteriors, we can write the free energy of each reduced model in terms of the full model. As before, maximizing each reduced free energy makes it approximate the log evidence:

$$ℱ(o_{\tau ,m})=ℱ({\mathit{o}}_{\tau })-\ln \int \frac{p(\tilde{\mathit{x}}|o_{\tau ,m})}{p(\tilde{\mathit{x}}|{\mathit{o}}_{\tau })}q(\tilde{\mathit{x}})d\tilde{\mathit{x}}\approx \ln p(\tilde{\mathit{y}}|o_{\tau ,m})$$ (24)

This relation implies that the free energy related to each discrete dynamics $o_{\tau ,m}$ can be found from the approximate posterior $q(\tilde{\mathit{x}})$ of the full model, without directly computing the reduced posteriors. With the Gaussian approximation of Eq 23, the mth reduced free energy breaks down to a simple formula:

$$\begin{gathered} ℱ(o_{\tau ,m})=\frac{1}{2}\ln |{\tilde{\mathit{\Pi }}}_{x,m}{\tilde{\mathit{P}}}_x{\tilde{\mathit{C}}}_{x,m}{\tilde{\mathit{\Sigma }}}_x| \\ -\frac{1}{2}({\tilde{\mathit{\mu }}}^T{\tilde{\mathit{P}}}_x\tilde{\mathit{\mu }}-{\tilde{\mathit{\mu }}}_m^T{\tilde{\mathit{P}}}_{x,m}{\tilde{\mathit{\mu }}}_m-{\tilde{\mathit{\eta }}}^T{\tilde{\mathit{\Pi }}}_x\tilde{\mathit{\eta }}+{\tilde{\mathit{f}}}_m(\tilde{\mathit{\mu }}{)}^T{\tilde{\mathit{\Pi }}}_{x,m}{\tilde{\mathit{f}}}_m(\tilde{\mathit{\mu }})) \end{gathered}$$ (25)

where the mean and precision of mth reduced model are:

$$\begin{gathered} {\tilde{\mathit{\mu }}}_m={\tilde{\mathit{C}}}_{x,m}({\tilde{\mathit{P}}}_x\tilde{\mathit{\mu }}-{\tilde{\mathit{\Pi }}}_x\tilde{\mathit{\eta }}+{\tilde{\mathit{\Pi }}}_{x,m}{\tilde{\mathit{f}}}_m(\tilde{\mathit{\mu }})) \\ {\tilde{\mathit{P}}}_{x,m}={\tilde{\mathit{P}}}_x-{\tilde{\mathit{\Pi }}}_x+{\tilde{\mathit{\Pi }}}_{x,m} \end{gathered}$$ (26)

and we wrote the covariances $\tilde{\mathit{\Sigma }}$ and $\tilde{\mathit{C}}$ in terms of precisions $\tilde{\mathit{\Pi }}$ and $\tilde{\mathit{P}}$. In short, the free energy assigns a score to each reduced model based on its fit with the full posterior, and two models i and j can be compared through a log-Bayes factor $ℱ(o_{\tau ,i})-ℱ(o_{\tau ,j})$. Crucially, this posterior is a continuous trajectory and the agent’s hypotheses are constantly updated via sensory observations. If a discrete step $\tau$ corresponds to a continuous period T, we integrate the reduced free energies over this period and compare them with the prior surprise $-\ln {\mathit{o}}_{\tau }$. This results in an ascending message that infers the most likely discrete dynamics ${\mathit{o}}_{\tau }$ that may have generated the current continuous observations:

$$\begin{gathered} E_{\tau ,m}=-\ln o_{\tau ,m}-{\int }_0^Tℱ(o_{\tau ,m})dt \\ {\mathit{o}}_{\tau }=\sigma (-{\mathit{E}}_{\tau }) \end{gathered}$$ (27)

Instead, the descending message is computed as a simple Bayesian model average of the agent’s hypotheses, e.g., by weighting the discrete dynamics $o_{\tau ,m}$ of a reduced model with the related dynamics function ${\tilde{\mathit{f}}}_m(\tilde{\mathit{\mu }})$:

$$\tilde{\mathit{\eta }}={\sum }_m{o}_{\tau ,m}{\tilde{\mathit{f}}}_m(\tilde{\mathit{\mu }})$$ (28)

Then, $\tilde{\mathit{\eta }}$ embeds a prior over a continuous trajectory, which steers the inference of the continuous hidden states toward preferred outcomes. The posterior over the hidden states is finally found by computing the gradient of the free energy of the full model with respect to the mean $\tilde{\mathit{\mu }}$ of the full approximate posterior, and updating the latter via gradient descent:

$$\dot{\tilde{\mathit{\mu }}}-𝒟\tilde{\mathit{\mu }}=-{\partial }_{\mu }ℱ({\mathit{o}}_{\tau })=\partial {\tilde{\mathit{g}}}^T{\tilde{\mathit{\Pi }}}_y{\tilde{\mathit{\epsilon }}}_y+{\partial }_{\mu }{\tilde{\mathit{f}}}^T{\tilde{\mathit{\Pi }}}_x{\tilde{\mathit{\epsilon }}}_x-{𝒟}^T{\tilde{\mathit{\Pi }}}_x{\tilde{\mathit{\epsilon }}}_x$$ (29)

This update is expressed in terms of message passing of the following prediction errors:

$$\begin{gathered} {\tilde{\mathit{\epsilon }}}_y=\tilde{\mathit{y}}-\tilde{\mathit{g}}(\tilde{\mathit{\mu }}) \\ {\tilde{\mathit{\epsilon }}}_x=𝒟\tilde{\mathit{\mu }}-\tilde{\mathit{\eta }} \end{gathered}$$ (30)

and has a specular form of the discrete update of Eq 15, except that now the inference is done over a continuous path—hence the additional term $𝒟\tilde{\mathit{\mu }}$. This minimization entails the process of perception, i.e., conforms the agent’s beliefs to the perceived sensations. In order to conform the environment to the agent’s beliefs (in short, to act), the free energy can be minimized with respect to the motor commands $\mathit{a}$. This additional mechanism reduces to a minimization of sensory prediction errors:

$$\dot{\mathit{a}}=-{\partial }_aℱ({\mathit{o}}_{\tau })=-{\partial }_a{\tilde{\mathit{g}}}^T{\tilde{\mathit{\Pi }}}_y{\tilde{\mathit{\epsilon }}}_y$$ (31)

where ${\partial }_a\tilde{\mathit{g}}$ is an inverse mapping from sensations to actions. This mapping implements a dynamics inversion and is thought to be realized by classical reflex arcs in the spinal cord [75, 76].

4.2. Inverse kinematics in active inference

Several methods have been proposed on how to realize inverse kinematics in active inference. The most common approach for simulating a reaching task is to encode a target location in extrinsic coordinates as a causal variable of a continuous model. This variable generates a prediction for the 1st-order (e.g., velocity) of the hidden states—encoding the agent’s joint angles in intrinsic coordinates—which therefore act as a dynamic attractor toward the desired location. In this simple representation, the link between the causal variable and the hidden states performs an inverse kinematics of the target location, whose product (i.e., a possible agent’s configuration with the hand at the target) is compared with the extrinsic position of the hand computed via forward kinematics. In this simple representation, both (intrinsic and extrinsic) reference frames are used in a single active inference level.

An alternative and more powerful approach exploits an aspect of the theory inherited from predictive coding, i.e., that the nervous system manages to approximate the real posterior by building a hierarchical architecture wherein a particular level acts as an observation for the level above and as a prior for the level below [40, 41]. In this way, higher levels can construct increasingly richer and more invariant representations of the environment, similar to deep generative models of neural networks. Since a level communicates only with the levels immediately above and below, the overall generative model can be factorized into independent distributions, and every level can be analyzed as in the general formulations of discrete and continuous active inference. Specifically, the hierarchical approach to inverse kinematics is to design a structure of two levels, wherein an intrinsic unit encoding the agent’s joint angles has a causal influence on an extrinsic unit encoding the agent’s hand. This approach follows the natural flow of the generative process, with an efficient decomposition between intrinsic and extrinsic dynamics. Here, we briefly describe this second approach.

The higher-level intrinsic unit ${𝒰}_i$ is governed by the following equations:

$$\begin{gathered} {\tilde{\mathit{y}}}_p={\tilde{\mathit{g}}}_p({\tilde{\mathit{x}}}_i)+{\mathit{w}}_{y,p} \\ 𝒟{\tilde{\mathit{x}}}_i={\tilde{\mathit{f}}}_i({\tilde{\mathit{x}}}_i,{\tilde{\mathit{v}}}_i)+{\mathit{w}}_{x,i} \end{gathered}$$ (32)

where ${\tilde{\mathit{y}}}_p$ are generalized (e.g., position, velocity, and so on) proprioceptive observations, ${\tilde{\mathit{g}}}_p$ are the generalized proprioceptive likelihoods, ${\tilde{\mathit{x}}}_i$ are intrinsic hidden states encoding the instantaneous trajectory of the agent (e.g., every joint angle of the arm), ${\tilde{\mathit{v}}}_i$ are intrinsic causal variables encoding the trajectory priors, and ${\tilde{\mathit{f}}}_i$ are the related generalized dynamics. Also, the letter w indicates Gaussian noises.

Instead, the lower-level extrinsic unit ${𝒰}_e$ follows the system:

$$\begin{gathered} {\tilde{\mathit{y}}}_v={\tilde{\mathit{g}}}_v({\tilde{\mathit{x}}}_e)+{\mathit{w}}_{y,v} \\ 𝒟{\tilde{\mathit{x}}}_e={\tilde{\mathit{f}}}_e({\tilde{\mathit{x}}}_e,{\tilde{\mathit{v}}}_e)+{\mathit{w}}_{x,e} \end{gathered}$$ (33)

where the letter v indicates the visual (or, more in general, exteroceptive) domain. The two units are related via connection between the 0th (i.e., position) temporal orders:

$${\mathit{x}}_e={\mathit{g}}_e({\mathit{x}}_i)=\mathit{T}({\mathit{x}}_i)+{\mathit{w}}_i$$ (34)

where $\mathit{T}$ performs a forward kinematics between the agent’s joint angles and its hand. As a consequence, the backward message from ${𝒰}_e$ to ${𝒰}_i$ realizes an inverse kinematics of an extrinsic prediction error conveying the difference between the prediction of ${𝒰}_i$ and the actual extrinsic belief of ${𝒰}_e$:

$$\partial {\mathit{g}}_e^T{\mathit{\epsilon }}_e={\mathit{J}}^T({\mathit{x}}_e-\mathit{T}({\mathit{x}}_i))$$ (35)

In this way, extrinsic trajectories (e.g., linear or circular motions) are easily realized by defining appropriate dynamics in ${\tilde{\mathit{f}}}_e$, which then travels back to infer the most likely agent’s configuration corresponding to that trajectory. The updates of the belief over intrinsic and extrinsic hidden states are the following:

$$\begin{gathered} {\dot{\tilde{\mathit{\mu }}}}_i-𝒟{\tilde{\mathit{\mu }}}_i=-{\partial }_{{\mu }_i}ℱ=\partial {\tilde{\mathit{g}}}_e^T{\tilde{\mathit{\pi }}}_e{\tilde{\mathit{\epsilon }}}_e+\partial {\tilde{\mathit{g}}}_p^T{\tilde{\mathit{\Pi }}}_p{\tilde{\mathit{\epsilon }}}_p+\partial {\tilde{\mathit{f}}}_i^T{\tilde{\mathit{\Pi }}}_{x,i}{\tilde{\mathit{\epsilon }}}_{x_i}-{𝒟}^T{\tilde{\mathit{\Pi }}}_{x,i}{\tilde{\mathit{\epsilon }}}_{x,i} \\ {\dot{\tilde{\mathit{\mu }}}}_e-𝒟{\tilde{\mathit{\mu }}}_e=-{\partial }_{{\mu }_e}ℱ=-{\tilde{\mathit{\Pi }}}_e{\tilde{\mathit{\epsilon }}}_e+\partial {\tilde{\mathit{g}}}_v^T{\tilde{\mathit{\Pi }}}_v{\tilde{\mathit{\epsilon }}}_v+\partial {\tilde{\mathit{f}}}_e^T{\tilde{\mathit{\Pi }}}_{x,e}{\tilde{\mathit{\epsilon }}}_{x_e}-{𝒟}^T{\tilde{\mathit{\Pi }}}_{x,e}{\tilde{\mathit{\epsilon }}}_{x,e} \end{gathered}$$ (36)

For both units, we note an extrinsic prediction error acting either as a prior or as an observation, a sensory-level observation (either proprioceptive or exteroceptive), and a dynamics prediction error from previous or successive temporal orders. This two-level architecture is highly effective in tasks that require dynamic constraints in both intrinsic and extrinsic domains (e.g., when moving the arm while keeping the hand’s palm up) but fails when simultaneous coordination of multiple limbs is needed. In this case, we can extend the model by designing an intrinsic-extrinsic module for every degree of freedom of the agent’s body. Then, a dynamic attractor at the last (e.g., hand) level results in a prediction error that is backpropagated throughout the whole hierarchy, eventually inferring an appropriate hierarchical configuration of the body.

Data Availability

The code is available here: https://github.com/priorelli/embodied-decisions.

Funding Statement

This research received funding from the European Research Council under the Grant Agreement No. 820213 (ThinkAhead) to GP, the Italian National Recovery and Resilience Plan (NRRP), M4C2, funded by the European Union – NextGenerationEU (Project IR0000011, CUP B51E22000150006, “EBRAINS-Italy”; Project PE0000013, “FAIR”; Project PE0000006, “MNESYS”) to GP, the European Union’s Horizon H2020-EIC-FETPROACT-2019 Programme for Research and Innovation under Grant Agreement 951910 to IPS, and the Ministry of University and Research, PRIN PNRR P20224FESY and PRIN 20229Z7M8N to GP. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.