Understanding the Brain as a Mentalizing Machine with a Symbol Grounding Function

Abstract

Social creatures can infer the mental states of others. This cognitive ability, called mentalizing, can be considered a process of inferring others’ hidden states behind their actions from partially observable sensory information. The purpose of this review is to propose the computational mechanisms of mentalizing and review the neural substrates that might underlie each computational process. In fact, inference about hidden states is a ubiquitous task in many sensory systems, and this can be achieved under the predictive coding framework in which the brain probabilistically estimates a latent state that most likely causes the observed sensory events by minimizing errors between the actual and predicted sensory signals. We argue that what might be unique to mentalizing is not merely the representation of others’ internal states in an arbitrary latent space but also the capacity to represent them in a mental space that one can experience subjectively. This function can solve the so-called symbol grounding problem. Further, the use of symbol grounding makes the inference system efficient and reliable by reducing the cost to learn de novo the latent representations of others’ mental states. On the basis of a preliminary simulation, we demonstrate that an artificial mentalizing system with a symbol grounding function performs better in predicting the actions of virtual agents than a pure Bayesian observer without the symbol grounding function. Emerging novel paradigms integrating artificial and biological neural networks will enable a better understanding of the neural algorithms and computational processes for complex social cognition including mentalizing.

Significance Statement

Mentalizing is the ability to infer the mental states of others, such as their beliefs, desires, and intentions, and thereby predict their upcoming behavior. Although mentalizing enriches daily social interactions, how the brain mediates this cognitive function remains unknown. Here, we propose the computational principles and mechanisms underlying mentalizing through the contrast with the operations of artificial neural networks. We argue that, to achieve mentalizing, the biological brain not only represents others’ internal states in an arbitrary latent space but also associates them with a mental space that one can experience subjectively (symbol grounding). Results of our simulation show that artificial neural networks with symbol grounding outperforms those without symbol grounding in predicting the actions of observed agents.

Introduction

In everyday life, people infer the mental states of others, such as their beliefs, desires, and intentions. This cognitive ability is called mentalizing, a critical brain function that enriches social interactions. Arguably, others’ mental states are not directly observable, and one can only experience one's own mental states (Premack and Woodruff, 1978). Then, how might mentalizing be accomplished? Accumulating evidence points to several brain regions and neural networks that are associated with mentalizing (Frith and Frith, 2006; Isoda, 2021; Michael and Ochsner, 2021); however, the computational principles and neural substrates of mentalizing remain largely unknown.

Remarkable progress has been made in computer vision and artificial intelligence, resulting in the development of state-of-the-art artificial neural networks (ANNs) with performance being functionally comparable or even superior to certain aspects of human vision and cognition (Silver et al., 2016; Krizhevsky et al., 2017; Vaswani et al., 2017). For example, pose estimation neural networks can extract the precise positions of major body parts of freely moving animals or humans filmed in a natural environment without the use of markers on their bodies (Mathis et al., 2018; Cao et al., 2019; Bala et al., 2020; Lauer et al., 2022; Matsumoto et al., 2023). The extracted pose data allow artificial systems not only to quantify the kinematics of actions but also to infer the subjects’ internal states driving their actions (Calhoun et al., 2019; Kaneko et al., 2024). Humans can also perceive others’ actions and intentions only from biological motion, i.e., from multiple dots placed on the moving joints (Vaina et al., 2001; Jastorff et al., 2012). These parallels suggest that the aforementioned artificial intelligence technology may share some critical aspects of human social cognition. These technological advancements have impelled researchers to create artificial systems that emulate complex cognitive processes such as mentalizing. Further, a careful examination of such artificial systems may help identify neuroscientifically plausible computational principles underlying social cognition (Hassabis et al., 2017).

This review aims to formulate a neurocognitive model of mentalizing that is consistent with neuroscientific findings and can be implemented in modern ANNs. We seek the computational mechanisms that might underlie the grasping of others’ mental states based on external sensory signals. For this purpose, we place a particular emphasis on the mechanism by which the latent representation of others’ mental states is connected via symbol grounding with one's own mental states, which are readily accessible. The symbol grounding, the term originally coined by Stevan Harnad (1990), refers to connecting abstract symbols, such as words and icons, to real-world entities and experiences. It is concerned with how otherwise meaningless symbols can acquire their meanings. The symbol grounding mechanism we propose in this review makes an inference system efficient and reliable by substantially reducing the need to learn de novo the latent representations of others’ mental states.

In the following sections, we first describe that mentalizing is a process for inferring others’ hidden states from observations, but such hidden state inferences are also common in general sensory processing. Next, we argue that a unique component inherent in mentalizing is a symbol grounding function whereby the latent representation of the inaccessible hidden states of others can be mapped onto one's own mental space, which is readily accessible. We then introduce a neuroscientifically plausible neurocognitive model that can implement mentalizing with a symbol grounding function. On the basis of a preliminary simulation, we demonstrate that an artificial mentalizing system with a symbol grounding function outperforms a pure Bayesian observer without symbol grounding in the prediction of virtual agent's actions. We then review possible neural substrates for key processes involved in mentalizing with a symbol grounding function. Finally, we propose experimental strategies to understand better the neural basis of mentalizing.

Mentalizing as an Estimation of Hidden States from Observation

Mentalizing is a psychological process for inferring others’ hidden states behind their actions based on partial observations (Koster-Hale and Saxe, 2013; Jara-Ettinger et al., 2016; Baker et al., 2017; Rabinowitz et al., 2018). This sort of inference is a general task not limited to social contexts but also occurs in various situations in daily life. For instance, on a morning, when you leave home and observe that it is dark, windy, and cold, you might anticipate rain, despite the lack of detailed knowledge about meteorology. Then, you might decide to carry an umbrella or consult the weather forecast. In this way, through sensory modalities shaped by biological evolution, we make inferences about the hidden states behind observations, that is, the state of the external milieu that has most likely caused sensory events.

These inference mechanisms are thought to be implemented across different neural systems in the brain. Visual perception is a good example to discuss how the brain interprets the external environment (Kawato et al., 1993; Rao and Ballard, 1999; Bastos et al., 2012). For instance, when viewing another's face, the angular subtense of its retinal image can vary depending on several factors. The visual system estimates the actual size of the face as the most plausible configuration that is consistent with not only the current retinal image but also various sources of information, such as a three-dimensional shape and distance to the observer derived from binocular cues (Khandhadia et al., 2023). However, the estimation of the hidden state becomes inaccurate if only feedforward computation from sensory input to the state estimation is relied upon. Reverse computation that predicts sensory signals given an estimated state enables more accurate and robust perception (Kar et al., 2019). The brain is thought to improve the accuracy of state inference by minimizing prediction errors, which are the differences between the actual sensory signals and the predicted sensory signals computed backward based on a given inference. This computation model is called the predictive coding model (Rao and Ballard, 1999; Friston, 2005; Isomura, 2022) or the Bayesian brain hypothesis, named after the underlying statistical principles (Knill and Pouget, 2004).

The inference of hidden states is not limited to readily interpretable representations, such as object size. Consider the ventral visual stream of the primate brain as an example. The ventral visual stream analyzes semantic information about complex objects, such as the identity of faces (Ungerleider and Mishkin, 1982; Kravitz et al., 2013). Cortical areas at the later stages of the ventral visual stream include neurons representing face direction (Freiwald and Tsao, 2010; Taubert et al., 2020) or the face of a specific person (Eifuku et al., 2004; Yang and Freiwald, 2021; Waidmann et al., 2022), corresponding well to our subjective recognition of object properties or categories. Conversely, neurons in the primary visual cortex represent simple geometric features, such as a Gabor patch, in which it is hard to find biological significance or semantic meaning, but an analytic interpretation of objects can be traced (Ohzawa et al., 1990; Olshausen and Field, 1996; Freeman and Simoncelli, 2011). What is most difficult to interpret is the representations in the intermediate cortical areas. The best-preferred stimulus of neurons in these areas can be complex and peculiar, as shown in Figure 1 (Bashivan et al., 2019; Ponce et al., 2019; Willeke et al., 2023). One can hardly determine what information is encoded by these neurons; yet, it can be assumed that they should encode, in some way, crucial aspects of real-world information. These response profiles in the intermediate cortical areas bear resemblance to latent representations in the context of artificial intelligence. A latent representation is the compressed representation of input data, i.e., a simplified summary of the most important features (variables) that are useful to reconstruct an output. Furthermore, a latent space refers to a low-dimensional vector space capturing the key features of the input, with each dimension corresponding to each feature.

Figure 1. Complex representations of neurons in mid-level visual areas. The images are the best-preferred stimuli for different neurons in the V4 and TEO. These images were optimized and synthesized by artificial neural networks. The synthesized images excited neurons more strongly than the best-preferred natural images chosen from thousands of candidates. Red crosses in the images for the TEO represent a fixation point. Images in each row were modified from Bashivan et al. (2019), Willeke et al. (2023), and Ponce et al. (2019), respectively.

For the design of the architecture and training of ANNs, substantial efforts have been made to develop sophisticated latent representations, which dramatically improve computational efficiency by reducing information load while preserving critical information about the external world (Ni et al., 2022; Zhao et al., 2023; Elharrouss et al., 2024). For instance, an ANN recognizing hand-drawn numbers (Fig. 2A) transforms an original image into a few-dimensional latent space, from which the original image can be reconstructed precisely (Kingma and Welling, 2013; Fig. 2B). The ANN consists of an encoder that maps input images onto the latent space and a decoder that restores the original images from the latent variables (Fig. 2B). The ANN adjusts the connections between artificial neurons by minimizing the errors between the original and reconstructed images to learn optimal probabilistic representations of latent variables (Fig. 2C) and acquires an optimal transformation function, i.e., encoder and decoder, which is essentially the same computational principle as used in the predictive coding model (Friston, 2005; Kingma and Welling, 2013; Alemi et al., 2016). Although the ANNs of this sort and the predictive coding model differ in that the implementations of the former is optimized through backpropagation (Amari, 1967; Rumelhart et al., 1986) and those of the latter through Hebbian learning (Hebb, 1949), the algorithm of the predictive coding model can converge with the backpropagation algorithm under certain conditions (Whittington and Bogacz, 2017).

Figure 2.

An artificial neural network (ANN) that obtains latent representations through unsupervised learning. A, Examples of the training dataset. Hand-drawn numbers with variations in their form. B, Architecture of the ANN. C, Artificial images generated by the trained decoder from the variables at arbitrary points on the latent space. D, Latent variables obtained by the trained encoder with several images of “6” or “9” as input. Note that the different numbers are spontaneously clustered in the latent space even though the training dataset includes only pixel information without any labels indicating which number an image corresponds to.

The latent representations based on these models are not obtained as strict solutions. Rather, they are the results of optimization that can vary according to several factors, such as the datasets used for training, model architecture, and hyperparameters. Researchers are usually unable to decipher what each latent variable represents, particularly when the model size of the ANN architecture becomes larger, i.e., the number of artificial neurons increases (Scherlis et al., 2022; Templeton et al., 2024). Latent representations can be learned in an unsupervised manner without any specific teaching signal or ground truth that guides optimization. The latent representations obtained in this way not only retain the capacity to reconstruct input images but are also useful for different purposes, such as categorical classifications (Zhuxi et al., 2017; Fig. 2D) or anomaly detection (An and Cho, 2015).

Predicting others’ behavior requires reference to a wide variety of information ranging from environmental information at hand to previous social interactions. There is no doubt that the reduction of such high-dimensional data into a compact representation reflecting others’ internal states facilitates accurate and prompt predictions of their behavior. Indeed, several studies indicate that the predictive coding model is useful for the recognition of others’ actions (Kilner et al., 2007; Koster-Hale and Saxe, 2013; Jara-Ettinger et al., 2016; Baker et al., 2017; Rabinowitz et al., 2018; Isoda, 2021).

A seminal study by Rabinowitz et al. (2018) demonstrated that, based on the predictive coding framework (or Bayesian inference), an artificial system can learn to represent the latent variables reflecting others’ internal states based only on the observation of others’ actions without any supervising signals. In their study, an artificial system observed and predicted the actions of an agent in a virtual environment, that is, a grid world with a particular arrangement of walls and objects (Fig. 3A). The agent could receive different amounts of rewards for reaching these objects and could see the local surroundings to different extents. Each agent was designed to possess different characteristics and states, such as skewness in the preference for each object or movement costs. Thus, an agent with a low movement cost and a strong bias in favor of a particular object would explore the grid world for a long distance until it found the best-preferred object (Fig. 3A), while an agent with the opposite parameters would approach the closest object. Animations consisting of these geometric shapes are often used especially in developmental psychological research, and human infants attribute animacy, desires, intentions, belief, or even social dispositions, e.g., cooperative or competitive, to these moving agents (Scholl and Tremoulet, 2000). The artificial observer, named the machine theory of mind (ToM), consisting of an encoder and decoder as mentioned above, not only successfully predicts the agent's actions (Fig. 3B,C) but also represents their internal parameters differently in learned latent space (Fig. 3D). Furthermore, the machine ToM could even represent the agent's false beliefs, which were introduced by occasionally switching the locations of objects that were out of the agent's view (Fig. 3E,F). In another study, an artificial observer implementing Bayesian inference was capable of estimating an agent's internal states to the same degree as a real human observer viewing the same animation (Baker et al., 2017). These studies demonstrate the competence of predictive coding as a core computational process of mentalizing.

Figure 3.

Inferences of an agent's actions and internal states by a machine theory of mind (ToM) designed using a predictive coding framework. A, An example of a grid world environment. A dotted black circle denotes the initial position of an agent. Colored squares denote target objects. Black squares denote walls. In each episode, the agent acquired one of the objects based on an internally determined preference and action cost by moving leftward, upward, rightward, or downward at each step. Red arrows indicate the trajectory of the agent (the intensity of red increases as the agent approaches the target object). The machine ToM learned to predict the agent's actions by observing different episodes by different agents. B, The machine ToM's prediction of the next action (top) and the finally acquired object (bottom) in a novel scene after the machine ToM observed the past actions of the same agent. C, The machine ToM's prediction of the finally acquired object. Top left, A single past episode by a particular agent. Top right, Prediction of the final object acquired by the same agent in a novel grid world after observing a single past episode. Color in each grid position indicates the machine ToM's prediction of the final object acquired, given that the agent is currently located in that position. Bottom, Examples for another agent. D, A two-dimensional latent representation of internal states for 100 different agents encoded by the machine ToM. Agents are color-coded by their ground truth preferred object. The agents were spontaneously clustered according to their best-preferred object. E, False belief task. Top, The agent is initially forced to move along a predefined path to acquire an intermediate goal (green star). Object swaps then occur with a predetermined probability of 0.1 when the agent has arrived at the intermediate goal. The agent can only see the extent of a 5 × 5 grid centered on itself. This agent has a preference for the blue object. When the agent has arrived at the intermediate goal after two actions (left), the optimal action for the next move is to go left if no swap occurs. In this example, swap outcomes can be seen, i.e., no false belief condition. When the agent has arrived at the intermediate goal after four actions (right), the grid position where the blue object was originally located is no longer observable due to the visibility constraint. Therefore, swap outcomes are also unobservable, and a false belief arises if a swap has occurred. F, Agents’ actions after object swaps (left) and the prediction by the machine ToM (right). Note that the machine ToM was not explicitly trained to infer the internal beliefs of the agent. Figures modified from Rabinowitz et al. (2018).

Symbol Grounding as a Key Component of Mentalizing

The inference of hidden states from partial observations, or more operationally, the transformation of sensory signals into latent representations is ubiquitous in numerous computational machineries. Then, what might be unique to the mentalizing process? We argue that the essence of mentalizing is not merely the representation of the internal states of others in an arbitrary latent space but also the capacity to represent them in a mental space that one can experience more directly.

While modern artificial intelligence may have no difficulty inferring the internal states of other from animations, this process is fundamentally different from what mentalizing does in humans. For example, a pose estimation network can extract the actions of animals (Mathis et al., 2018; Bala et al., 2020; Mathis and Mathis, 2020; Isoda, 2021; Lauer et al., 2022; Pereira et al., 2022; Matsumoto et al., 2023; Zhou et al., 2023). One can train neural networks, such as recurrent neural networks suited for the analysis of temporal data, to decode the internal state of a subject whose physiological states are controlled to be either sated or hungry before an experimental session (Kaneko et al., 2024; Fig. 4). However, these ANNs do not really know what satiety or hunger means, and they just learn to classify each pose sequence into one of two symbols, say, symbol A or B. Comparatively speaking, when people infer the internal states of others, whether consciously or unconsciously, they do not merely learn each state as an arbitrary symbol A or B; rather, people infer others’ internal states by associating them with their own subjective experiences. This association is known as symbol grounding in computer science, which refers to a process by which the meaning of otherwise meaningless symbolic representation is acquired by connecting it to real-world entities. We argue that it is this symbol grounding that is an essential component of mentalizing (Fig. 5).

Figure 4.

Artificial system decoding the internal states of marmosets, a nonhuman primate species. A three-dimensional (3D) pose estimation artificial neural network (ANN) extracted the animals’ key points, i.e., positions of the eyes, wrists, elbows, etc., from video recording data. A state inference ANN decoded the animals’ internal states based on the 3D pose time-series data. One marmoset, called partner (P), was experimentally placed in either a hungry or satiated state before the start of the experimental session, but the other marmoset, called subject (S), was not given an explicit cue about P's internal states. A food ball was given only to S, and P might try to take the food away from S. The state inference ANN was trained to decode the experimental conditions based on P's actions in a supervised manner. The line plot shows an example of ANN output. Scenes with higher scores featured P directly approaching S to try and take the food ball. Scenes with lower scores showed P moving around the cage with an apparent lack of interest in S or the food ball. As shown in the bar graph, the accuracy of the ANN in decoding P's internal state was markedly high. Figures modified from Kaneko et al. (2024).

Figure 5.

Mentalizing as an inference process of others’ hidden states under the predictive coding framework with a symbol grounding function. Monkey S (self) monitors the actions of monkey P (partner) through sensory channels, e.g., visual perception. These sensory signals are not merely transformed to S's latent representation of P's mental states but are also grounded in the S's accessible mental space. Note that this symbol grounding is achieved beforehand through one's own experiences in everyday life (see text and Fig. 6 for details). Colored dots depict the representations of P's hidden states with different levels of hunger, and the red arrows indicate those based on the current observation. In this example, the dynamics of hidden states are represented in the three-dimensional space in the top plot, and their representations are simplified to lower dimensions in the bottom plot, making information use easier in downstream tasks.

In the history of artificial intelligence development, Stevan Harnad (1990) suggested, as a concept of symbol grounding, that any symbol system acquired by a cognitive machine should be connected to actual real-world entities in the right way. For example, the statement “matter is made of quarks, leptons, and bosons” makes sense as a sentence, but most people are unable to connect it to the real world, as the symbols “quark,” “lepton,” and “boson” are not grounded. Conversely, in the case of the statement “apple pies are made of flour, sugar, and apples,” we can even imagine the taste and smell, as the symbols in the statement are all grounded. Of course, a quark is a concept created by physicists with substantial experimental support, and there should be no problem in considering that it captures a fundamental aspect of the world. The point here is that without symbol grounding, people cannot understand the rich meanings of symbols that are necessary for creating higher-level concepts.

Coming back to mentalizing, there is little need to acquire the latent representations de novo to express the internal states of others. The starting point for our argument is that others are inherently similar to the self. Thus, when estimating others’ hidden states under given sensory signals, it is not necessary to learn a latent representation in an unconstrained, arbitrary space. Rather, the estimation of others’ hidden states can be achieved more efficiently with greater predictive power by linking those states with a space that one can access as a reference (Asai et al., 2022; Migimatsu and Bohg, 2022; Li et al., 2024; Toussaint et al., 2024). A critical component of mentalizing is mapping an inaccessible latent representation obtained from a predictive coding schema onto a mental space that one can experience subjectively. In what follows, we propose a conceptual hypothesis on how such mapping might be achieved in the biological brain.

Neurocognitive Model of Mentalizing with Symbol Grounding

We have described that mentalizing is essentially a process of inference about others’ hidden states based on partial observations, and the inferred states are mapped via a grounding function to a similar mental space to which one can readily refer. Then, what might be the algorithms in the brain that can implement such computational principles? Isoda (2021) proposed a hypothetical neurocognitive mechanism of mentalizing based on findings in the fields of human neuroimaging and primate neurophysiology. Specifically, there is a “self-mental model” in the brain that simulates how “I” would feel and behave under a given context. This self-mental model can be diverted to a “general-other mental model” that is used for simulating how others in general would feel and behave during social interactions. As interactions progress with specific individuals or groups of individuals, the general-other mental model is differentiated to form a “specific-other mental model” that is tailored to specific individuals, such as close relatives and friends. These models are hypothesized to reside at least in the medial prefrontal cortex (MPFC), with the self-mental model in the ventral MPFC and the other mental model in the more dorsal MPFC (Isoda, 2021). Note that when the application of any model is not considered feasible in extreme conditions, such as novel situations that one has never encountered, an alternative cognitive process, such as general reasoning or inference, might play a role in predicting others’ behavior, as seen in people with autism spectrum disorder (Senju et al., 2009).

This neurocognitive model aligns well with the idea of symbol grounding for mentalizing, and its simplest form of implementation can be depicted as in Figure 6. Let us begin with the self-mental model (Fig. 6A). The input of the model, when considered as a kind of a mathematical function, is a sequence of one's own actions and sensory perception derived from the surrounding environment (functions f and g will be described later). The model then learns an encoder (Φ) that transforms the input into a probabilistic representation of latent variables (Z) in accordance with the predictive coding schema. The model further acquires a decoder (Ψ) that predicts a subsequent sequence of one's own actions from the latent representation (Z). Finally, the model is optimized by minimizing the prediction error (La) between the decoder's prediction and the actual action outputs. Now, we place an additional constraint to implement a symbol grounding function that the error Lm be minimized between the latent representation Z and the mental state M representing one's own internal state. It is not realistic to assume that M reflects every detail of one's own internal state; rather, M should be as compact a representation as possible, but should carry sufficient information to predict one's own action sequences. Via these processes, an inherently latent self-mental model is grounded in subjectively experienceable self-mental states (Fig. 6A, bottom).

Figure 6.

Simplified design of the mentalizing artificial neural network. The basic architecture consists of an encoder (Φ) and decoder (Ψ) with additional components for implementing the symbol grounding function. A, Mentalizing about the self (self-mental model). The direction of arrows indicates that learning progresses so that the variable at the root has the same value as the variable at the tip. Z/M means that the symbol grounding is done. B, Mentalizing about general others (general-other mental model). See the main text for details.

To mentalize general others, the same set of the encoder (Φ) and decoder (Ψ) as is can be utilized (Fig. 6B). This is feasible, because as mentioned above, people generally assume that others think, feel, and behave as they do (Malloy, 2022). Thus, the symbol grounding problem can be solved here. It should be noted, however, that inputs, i.e., others’ actions and environment, require some extra processing. Specifically, their actions need to be treated as agent-invariant, identifying the actions of others with those of one's own. This can be achieved by the mirror-neuron system (see the next section for details). Moreover, since the perception of the surrounding environment can differ between the self and others, some mechanisms of minimizing this difference must come into play [see “Perspective taking” in the next section]. The models dedicated to specific others can be acquired via transfer learning of the encoder and decoder developed for general others. With these mechanisms, the hidden states of others that are otherwise estimated solely from observable information can be associated with a mental space that one can experience personally.

Although any model can be implemented in different ways, the algorithm described above is surely a plausible one that can implement a symbol grounding function in mentalizing. Moreover, the algorithm can be implemented using modern ANNs. The development of theoretical and computational approaches has been relatively rare in social neuroscience, as opposed to the cases of sensory perception and motor control, as social cognition involves more complex processing and abstract conceptions. However, the advancement of ANNs has enabled computation that addresses such complicated issues. By applying these technological innovations, one can test the functional advantages of biological mentalizing by comparing the performance of an artificial system with symbol grounding and the performance of a pure Bayesian observer that infers others’ states based only on the predictive coding schema.

We performed a preliminary test in a virtual environment and found that an artificial mentalizing system with a symbol grounding function performed better than a pure Bayesian observer in predicting others’ actions. The machine ToM described above can be considered as a pure Bayesian observer without referring to one's own mental space to represent others’ latent variables. To implement a symbol grounding function, we allowed the machine ToM to explore the grid world by itself as an actor governed by two internal parameters, i.e., object preference and movement cost. As the actor, the machine ToM could access these internal parameters (states) as subjective experiences. At the same time, the encoder and decoder were adjusted to minimize Lm and La. This mentalizing ANN outperformed the pure Bayesian observer. Its advantage became even more pronounced when the prediction of a forthcoming action was required under a limited number of observations (Fig. 7A) and when the prediction of the final object was required after more steps of actions (Fig. 7B). The results show that the model with the symbol grounding can achieve an optimal level of performance earlier than the pure Bayesian model. This is likely because Lm directly instructs how the latent representation should be in the mentalizing model during learning, thus limiting the search space explored by the model. However, a close inspection of the results also revealed that, with a large number of observations, the pure Bayesian model showed slightly, yet significantly, higher performance in some conditions when predicting immediate actions (Fig. 7A). This finding suggests that the pure Bayesian model could find an even more efficient form of latent representation that is specifically tuned to action prediction. A possible reason for this is that in the case of the mentalizing model, learning proceeds on the basis of both La and Lm (Fig. 6A, top) and, therefore, improving the prediction accuracy of f (SA_t+1) is not necessarily a priority. In contrast, in the case of the pure Bayesian model, learning proceeds based only on La, meaning that the model can be optimized solely for improving the predictions accuracy of f (SA_t+1). Note that these results were derived from experiments in certain virtual conditions, and thus the assessment of their generalizability requires caution. Nonetheless, these results provide unique insights into how and under what conditions the cognitive system in the biological brain might have evolved. It is of particular interest for future work to determine the functional properties of each model using real-world data and compare predictive performance between the two models.

Figure 7.

Performance of artificial neural networks with or without symbol grounding. A, Accuracy of the prediction of the forthcoming one-grid action plotted against the number of observed agents, i.e., learning opportunity. B, Accuracy of the prediction of the acquired object plotted against the number of observed agents. *p < 0.05, **p < 0.01, ***p < 0.001 for two-tailed Welch's t test after Holm–Bonferroni correction. The upward and downward arrows indicate whether the performance of the mentalizing model is higher or lower than that of the pure Bayesian model, respectively.

Neural Substrates for Mentalizing with a Symbol Grounding Function

In this section, we discuss the possible neural substrates for mentalizing processes with a symbol grounding function (Fig. 8).

Figure 8.

Schematic illustration of mentalizing-related brain regions. Each subprocess of mentalizing is thought to be mediated by distinct or shared brain regions.

Agent-invariant action representation (A → f(A))

The general-other mental model requires that the actions of others be identified with one's own actions. The discovery of mirror neurons in monkeys (di Pellegrino et al., 1992; Gallese et al., 1996) makes this requirement physiologically plausible. The visual area known as the superior temporal sulcus (STS) seems to have a leading role in the analysis of the bodily movements of the self and others (Hietanen and Perrett, 1996; Jellema et al., 2000; Ninomiya et al., 2018, 2021), and the ventral premotor cortex (PMv) and inferior parietal lobule (IPL) give rise to a mirror representation based on such movement information (Rizzolatti and Luppino, 2001). The STS region sends relevant signals indirectly to the PMv (Matelli et al., 1986; Gerbella et al., 2011; Bruni et al., 2018) via the IPL, especially areas PF, PFG, and AIP (Andersen et al., 1990; Rozzi et al., 2006). While there has been extensive research into the functions of the mirror system (Rizzolatti and Luppino, 2001; Arbib, 2002; de Waal and Preston, 2017; Bonini et al., 2022), it remains unknown how mirror neurons can map observed actions onto one's own motor representations. Although this issue is beyond the scope of the present work, several theories have been proposed, including coordinate transformation of the body schema (Chang, 2013), view-invariant representation of actions (Keysers and Perrett, 2004), and bounding based on affordance (Oztop and Arbib, 2002).

Perspective taking (E → g(E))

Perspective taking refers to a cognitive process by which one mentally rotates oneself into the position of others and views the world from their perspective. Perspective taking is most frequently studied in the context of visual perception. Seeing is the first step to knowing, and visual perspective-taking is essential for forming others’ false beliefs, a hallmark of mentalizing (Baron-Cohen et al., 1985). Psychophysical experiments in human subjects have shown that visual perspective-taking seems to be processed spontaneously and implicitly (Samson et al., 2010). The development of visual perspective-taking is preceded by the emergence of gaze following (Emery, 2000). A variety of primate species including humans, chimpanzees (Itakura, 1996, 2004), macaques (Tomasello et al., 1998; Ferrari et al., 2000), and marmosets (Burkart and Heschl, 2006; Spadacenta et al., 2019) are sensitive to others’ gaze direction. Neuroimaging studies on human subjects have shown that the STS region encodes others’ gaze direction (Puce et al., 1998; Hoffman and Haxby, 2000; Hooker et al., 2003). Meta-analysis of neuroimaging studies has revealed that the temporoparietal junction (TPJ) and precuneus, in addition to the STS region, are activated during visual perspective-taking tasks (Vogeley et al., 2004; Mazzarella et al., 2013; Schurz et al., 2013, 2015; Martin et al., 2017; Gunia et al., 2021). However, few studies have investigated the neural mechanism of visual perspective-taking at the cellular level. A recent study demonstrated that a subset of neurons in the MPFC and PMv dynamically change their response field between self actions and others’ actions in ways consistent with an actor-centered perspective-taking coding scheme (Ninomiya and Isoda, 2024).

Self-mental monitoring (brain state → M)

The neural systems supporting mentalizing are unlikely to monitor every detail of one's own internal states. Instead, it is more plausible that the mentalizing system can access a compact representation of internal states sufficient for predicting one's own behavioral sequences. Here, we refer to such a condensed representation as the mental state of the self, M, which is derived from complete internal states of the self using a condensation function, Ω. Access to M can be prereflective (implicit) or reflective (conscious or cognitive) (Northoff et al., 2006).

The medial part of the cerebral cortex, the so-called cortical midline structures, may play a significant role in the monitoring of M. A chemical intervention study in monkeys showed that the rostral cingulate motor area is causally involved in adjusting behavioral strategy internally, but not externally, according to one's own intrinsic requirements (Shima and Tanji, 1998). Electrical stimulation of the dorsal MPFC during neurosurgery evokes a conscious urge to perform a movement in humans (Fried et al., 1991). The dorsal MPFC and adjacent structures are activated when attention is directed to intention in humans (Lau et al., 2004), and neurons in those structures fire selectively when switching from automatic to intentional actions in monkeys (Isoda and Hikosaka, 2007; Hikosaka and Isoda, 2010). These results suggest that at least the dorsal MPFC is involved in the formation of intention.

The ventral MPFC is activated by self-relevant external stimuli regardless of sensory modalities (Johnson et al., 2002; Fossati et al., 2003; Northoff et al., 2006). In fact, the ventral MPFC can acquire all modalities of sensory information, i.e., olfactory, gustatory, somatosensory, auditory, and visual, through its connections with other cortical areas (Yeterian et al., 2012). In addition to exteroceptive information, cortical midline structures can access interoceptive information through dense connections to other cortical areas including the insula and related subcortical regions (Barbas et al., 2003; Damasio, 2003; Craig, 2004). In functional neuroimaging studies, cortical midline structures including the medial orbitofrontal cortex, ventromedial prefrontal cortex, and anterior cingulate cortex are activated when subjects pay attention to, or make evaluations of, their interoceptive signals, such as heartbeat and relaxation/arousal states (Damasio, 2003; Craig, 2004; Critchley et al., 2004; Nagai et al., 2004; Singer et al., 2004).

For practical purposes, we assume that M represents mental states; however, strictly speaking, M can be any form of information that represents one's own internal parameters. M may be deeply related to conscious experience, but the theoretical predictions of the model described here yield no such requirements. How M, Ω, and subjective (or conscious) experience are associated with each other is an intriguing question; however, it is beyond the scope of the present review.

Symbol grounding (Z, Lm, and M)

The latent variable Z is likely to be represented in the MPFC. Evidence for this claim comes from an electrophysiological study showing that others’ false beliefs are encoded by single neurons in the human MPFC (Jamali et al., 2021). Consistent with this report, single neurons in the macaque anterior cingulate cortex encode others’ intentions (Haroush and Williams, 2015). Whether these neurons encoding others’ false beliefs or intentions also encode one's own with mirror-like properties is an interesting topic for future work. Neuroimaging studies show distinct areas of activation during self-mentalizing and other-mentalizing (Denny et al., 2012; Bicks et al., 2015), but this does not necessarily indicate that self-mentalizing and other-mentalizing are processed by distinct populations of neurons. Future research should also address how others’ affective and physiological states (Bastiaansen et al., 2009) are encoded at the cellular level.

Although we assume that the computation of Lm and the acquisition of Φ are performed in the mentalizing networks including the MPFC and TPJ, direct evidence for this assumption is sparse, and the precise neural implementation mechanism is elusive. We have discussed that latent space to encode sensory events can be acquired through unsupervised or self-supervised learning in the predictive coding scheme. In contrast, symbol grounding by which latent variables are mapped to the mental space by minimizing Lm can be achieved by supervised learning. Doya (1999) predicted that the cerebral cortex, cerebellum, and basal ganglia mediate unsupervised learning, supervised learning, and reinforcement learning, respectively. The loss term La requires the network to update the weights to minimize prediction error and feedback, while this mechanism does not directly control the acquisition of latent space. In this sense, model updating of the latent representation is not directly supervised (i.e., unsupervised learning). Complementary to this process, the latent representation is directly instructed by the signal from the mental state of the self to minimize Lm. Thus, the weight update based on Lm is supervised learning. According to this view, the cerebellum may be a critical addition to neural structures for symbol grounding in the context of mentalizing.

Research into motor learning has advocated that the cerebellum acquires an internal model that could simulate a future state of the motor system without actually executing movements. The predictive nature of this model enables adaptive control of movements by complementing sensory feedback that is slightly delayed due to neural transmission. It has been argued that the internal model scheme also operates in non-motor contexts, such as when the action of others is recognized (Miall, 2003; Wolpert et al., 2003; Ito, 2008). In this light, self-mental models for mentalizing could be implemented in a similar framework or as an extension of the known internal model.

Recent studies have shown that the cerebellum contributes to social cognition and mentalizing. A number of neuroimaging studies demonstrate activation of the cerebellar lobule crus I/II during mentalizing tasks (Yarkoni et al., 2011; Van Overwalle et al., 2014; Nettekoven et al., 2024). The cerebellum has functional connections to the STS and TPJ (Igelström et al., 2016; Van Overwalle and Mariën, 2016). Transcranial magnetic stimulation applied to the lateral cerebellar lobules impairs the emotional perception of other's facial expressions (Ciricugno et al., 2024) and body language (Ferrari et al., 2019). Patients with cerebellar dysfunction show impairments of not only motor control but also affective and cognitive control in social contexts (Van Overwalle et al., 2020). Morphological abnormalities, such as unusual cerebellar volume (Carper and Courchesne, 2000) and reduced size of Purkinje cells (Fatemi et al., 2002), are often observed in people with autism spectrum disorder. In genetically modified animals that recapitulate the autistic phenotype, including altered social communications (Katayama et al., 2016; Matsumura et al., 2020), abnormal cerebellar development has been reported (Suliman-Lavie et al., 2020; Kawamura et al., 2021).

The cerebral cortex can communicate with the cerebellum through the pontine nucleus. Cortical projections to the pontine nucleus arise from not only cortical motor areas but also a wide area of the prefrontal cortex, including the MPFC (Brodal, 1978; Glickstein et al., 1985; Schmahmann and Pandya, 1997). Neurons in the cerebellar dentate nucleus project to the cerebral cortex through the thalamus. In the dentate nucleus, the area occupied by neurons projecting to the prefrontal cortex is comparable with the area occupied by neurons projecting to the cortical motor areas (Strick et al., 2009). Research using transneuronal tracers has suggested that the cerebellum and prefrontal cortex, including the MPFC, form closed-loop circuits, in which the main cortical target of each cerebellar output channel is, in turn, a major source of input to the channel (Middleton and Strick, 2001; Kelly and Strick, 2003; Lu et al., 2012). Such cerebro-cerebellar circuits may contribute to the computation of Lm and the acquisition of Φ and may adaptively modify the mentalizing process so that it can be achieved in a skilled and error-free manner (Ito, 2008).

Some theories, such as the hierarchical predictive coding theory (Kilner et al., 2007) and action simulation theory (Gallese and Goldman, 1998), suppose that the system mediating the generative process for one's own actions can also be used for understanding others’ mental states behind their actions. An action is performed to achieve a short-term goal, and setting that goal involves the underlying mental states, such as desires and intentions. If the system can track this process in an inverted manner, it can be adapted to infer others’ mental states from their observed actions. The mirror-neuron system is likely to be positioned at the earliest stage of this processing, as it could link an observed action to its underlying goal (Rizzolatti and Sinigaglia, 2010; Rizzolatti et al., 2014). It can be argued that similar architecture might exist at a more abstract level to recognize others. Our emphasis on symbol grounding in mentalizing and the theories discussed above are not mutually exclusive. The present review aims to shed light on unique aspects of mentalizing in contrast to general inference processes of latent variables that are ubiquitous in the sensory system. The mentalizing system is distinct from the sensory system in that latent variables are inferred under the constraint that they are tied to one's own mental state. How this is achieved and implemented in the relevant neural systems remains to be investigated.

The involvement of the cerebellum in the symbol grounding is still speculative, and this idea needs to be investigated in future research. A testable approach is to compute representational similarity of brain activity (Kriegeskorte et al., 2008) between self-mentalizing and other-mentalizing in novel contexts where subjects learn associations of contextual cues with their own mental states and then to compare it between with and without reversible blockade of cerebellar functions using, for example, transcranial magnetic stimulation applied during learning.

Action prediction (Ψ and La)

When mentalizing about others’ states is performed, it should allow predictions of upcoming actions. Neuroimaging studies have shown that the human TPJ and nearby regions exhibit greater activity in response to observed actions that deviate from prediction (Pelphrey et al., 2004; Brass et al., 2007; Wyk et al., 2009). Studies employing computational modeling revealed that activation in the TPJ and MPFC is associated with inference of opponents’ strategy and prediction of their choices during competitive games (Hampton et al., 2008; Coricelli and Nagel, 2009; Zhu et al., 2012 ; Seo et al., 2014; Hill et al., 2017; Konovalov et al., 2021). Furthermore, activity in the amygdala and parahippocampal gyrus correlates with aspects of trustworthiness judgments during repeated games with the same individual (Bhatt et al., 2012), which may help induce the derivation of a specific-other mental model from a general-other mental model. In the macaque, too, the mid-STS region, the homolog of the human TPJ (Mars et al., 2013), is activated when the prediction about others’ behavior is violated (Roumazeilles et al., 2021). These findings imply that prediction signals per se also exist in these areas. In the macaque, prediction errors about others’ actions are encoded by single neurons in the mid-STS region, MPFC, and PMv (Yoshida et al., 2012; Ninomiya et al., 2020, 2021). However, there are notable differences in response properties between these regions. Specifically, the proportion of neurons responding to other-action prediction errors is substantially higher in the MPFC. Moreover, approximately half of such MPFC neurons also respond to reward prediction errors (Yoshida et al., 2012), while those in the mid-STS region specifically respond to action prediction errors (Ninomiya et al., 2021). The ventral bank of the macaque mid-STS region may encode others’ predicted actions at a more abstract level (Ong et al., 2021). It is unknown how predictions and prediction errors of observed actions are computed in the MPFC, PMv, and mid-STS region. These areas might operate differently depending on their connections and cortical hierarchy (Kilner et al., 2007). The combined use of viral vector-mediated pathway-selective blocking and direct neural recording in macaques (Ninomiya et al., 2020; Noritake et al., 2023) will be a promising avenue for understanding how predictions and prediction errors are generated at the circuit level.

Toward a Better Understanding of Biological and Artificial Neural Networks

Artificial intelligence has developed remarkably and reached a stage where its performance surpasses human performance at least in some aspects of sensory processing and intellectual activities (Silver et al., 2016). Interestingly, the information representations of artificial neurons are often similar to those of real neurons (Yamins et al., 2014; Güçlü and van Gerven, 2015; Bao et al., 2020). One major approach in systems neuroscience has been to identify the neural correlates of behavioral parameters that are defined by the experimenter. However, real neurons sometimes exhibit complex response patterns that are beyond intuitive interpretation (Chang and Snyder, 2010; McGuire and Sabes, 2011; Chang, 2013; Livi et al., 2019; Okazawa et al., 2021). Recent work has successfully characterized neurons in the intermediate cortical areas in the auditory system by comparing their activity with complex latent representations of ANNs (Li et al., 2023). This approach of comparing the behavior of artificial and biological neurons will be even more significant in the future.

Of particular importance is that this approach goes beyond the verification of representations. ANNs can be designed to perform the same input–output tasks using different algorithms under different computational principles. By comparing the activity of biological neurons with that of ANNs designed using different architectures, one could determine the most plausible algorithm implemented in the real brain (Chang et al., 2021; Higgins et al., 2021; Aldarondo et al., 2024). At the same time, the functional advantages or characteristics of different algorithms can be clarified, which may provide insights into why the brain is equipped with specific computational strategies. The key to realizing such an approach is to measure a subject's behavior in multiple dimensions (Musall et al., 2019) and large-scale neural activity at the cellular level (Trautmann et al., 2023), in addition to computational flexibility empowered by ANNs.

The above approach that compares neurons’ behavior between biological and artificial neural networks will be particularly beneficial in social neuroscience. In contrast to vision, movement, and learning, the theoretical and computational approach to social neuroscience is still in its infancy because of the need for highly abstract and complex computations associated with social cognition. Recent advances in ANNs have made it possible to construct artificial systems that can emulate complex social cognitive functions such as mentalizing using real-world data. For instance, artificial mentalizing with a symbol grounding function can be constructed by combining behavioral quantifications in freely moving primates (Bala et al., 2020; Marks et al., 2022; Matsumoto et al., 2023; Kaneko et al., 2024) with wireless neural recording technology (Courellis et al., 2019; Voloh et al., 2023; Manea et al., 2024; Shahidi et al., 2024; Testard et al., 2024). The modeling of cognitive mechanisms and validating them via comparison with ANNs will offer an important avenue for the development of social neuroscience, as it addresses subtle and complex aspects of cognition related to the self, others, and their interactions.

Concluding Remarks

David Marr (1982) proposed that any information processing system should be understood at three distinct, yet mutually complementary, levels, i.e., computation, algorithm, and implementation. Following this seminal framework, we conclude this review by summarizing the problem of mentalizing as follows. At the computational level, the mentalizing system infers the unobservable mental states of others based on partially observable information. To achieve this task, the mentalizing system must solve the problem of how it enables the observer to access the latent representation of others’ mental states in an interpretable way. A powerful approach to solving this problem at the algorithmic level is to install a symbol grounding function in the system, thereby connecting an inaccessible latent representation with one's own accessible mental states. The development of a self-mental model and its reuse for the comprehension and prediction of others’ actions is also of importance in solving this issue. We have described an ANN architecture that can accomplish such algorithmic operations for mentalizing with symbol grounding. Finally, at the implementation level, we have discussed possible neural substrates in the brain that are involved in each algorithmic aspect. Findings from human neuroimaging and nonhuman electrophysiological studies suggest candidate neural components that mediate mentalizing processes at the global, mesoscopic, and cellular levels. Emerging novel paradigms integrating artificial and biological neural networks will enable a better understanding of the neural algorithms and computational principles behind complex social cognition. Of course, real neural circuits are much more complex than the algorithms would require. A hallmark of Marr's framework is the use of computational and algorithmic constraints to understand this enormous complexity. More advanced cognitive functions, such as mentalizing, would benefit more from the use of his framework.