A brain-inspired theory of mind spiking neural network improves multi-agent cooperation and competition

Summary

During dynamic social interaction, inferring and predicting others’ behaviors through theory of mind (ToM) is crucial for obtaining benefits in cooperative and competitive tasks. Current multi-agent reinforcement learning (MARL) methods primarily rely on agent observations to select behaviors, but they lack inspiration from ToM, which limits performance. In this article, we propose a multi-agent ToM decision-making (MAToM-DM) model, which consists of a MAToM spiking neural network (MAToM-SNN) module and a decision-making module. We design two brain-inspired ToM modules (Self-MAToM and Other-MAToM) to predict others’ behaviors based on self-experience and observations of others, respectively. Each agent can adjust its behavior according to the predicted actions of others. The effectiveness of the proposed model has been demonstrated through experiments conducted in cooperative and competitive tasks. The results indicate that integrating the ToM mechanism can enhance cooperation and competition efficiency and lead to higher rewards compared with traditional MARL models.

Graphical abstract

Highlights

• •ToM helps agents infer others’ actions by self-modeling or modeling others
• •Agents with ToM optimize self-policy after considering others’ future actions in MARL
• •We build a brain-inspired SNN model to simulate the function and mechanism of ToM
• •The model exhibits superior performance on cooperative and competitive tasks

The bigger picture

Theory of mind (ToM), a kind of high-level social cognitive ability, enables individuals to infer others’ mental states and thus explain and predict others’ behavior. ToM plays a crucial role in human interaction. Currently, in multi-agent systems, agents without ToM make decisions based on observations of the environment, often ignoring the impact of other agents’ future behavior on current decisions. Inspired by the function and mechanism of ToM in the human brain, this article designed a spiking neural network (SNN) for ToM and validated it with multi-agent decision-making tasks. This method enables the effective facilitation of competition and cooperation among multiple agents. This work reveals the critical role of ToM in multi-agent interactions and lays the foundation for the future development of social intelligence for brain-inspired artificial intelligence.

In human social decision-making, theory of mind (ToM) optimizes decisions by inferring others’ mental states, based on self-experience or historical observations of others. We were inspired by the mechanism of ToM to construct a multi-agent ToM spiking neural network (MAToM-SNN). Agents with MAToM-SNN can predict other agents’ actions and make decisions while considering others’ future actions. These agents achieve better performance on various cooperative and competitive multi-agent reinforcement learning tasks than those without MAToM-SNN.

Introduction

Higher animals, such as humans, gradually derive behaviors such as cooperation and competition in social relationships, which are inseparable from the ability to infer others’ mental states, known as theory of mind (ToM). ToM often draws on information such as desires and beliefs to construct representations of others to predict others’ behavior in a given situation.^1,2,3 If we can learn to predict others’ behavior, we can avoid predictable trouble or take advantage of predicted opportunities. The critical role of ToM in social cognition inspired us to explore its role in multi-agent decision-making. An agent with ToM should be able to predict the actions of others depending on self-experience and observations of others and then adjust their policies to obtain more rewards overall based on predictions.

A simple example of using ToM to predict the behavior of others is shown in the scenario in Figure 1A, where there are three agents: B (Bob), G (Green), and O (Oli). B needs to predict the behavior of G and O. B’s self-experience is that he likes apples. B then finds that O likes oranges based on his observation of O (step 1 in Figure 1A). Since G’s behaviors have never been seen, B infers that G likes apples based on his own experience. From B’s observation of O, B would predict that O likes oranges. Imitation theory suggests that when an agent sees other agents exhibiting behavior similar to the one it has executed before, this similarity evokes empathy in the observing agent. Thus, the observing agent infers that the observed agents have the same intention⁴ (e.g., B’s inference about G in Figure 1A). The above process can be simulated by a hypothetical neural circuit as shown in Figure 1B. In the beginning, B observes the environment and can infer the observation of others. The inferior parietal lobule (IPL) and the posterior superior temporal sulcus (pSTS) store self-relevant and other-relevant observations, respectively.^5,6 The anterior cingulate cortex (ACC) is stimulated when B observes that their behavior is different from his expectations.^7,8 The ventral medial prefrontal cortex (vmPFC) stores information related to self, and the dorsal medial PFC (dmPFC) stores information related to others.⁹ Based on the stored information, the dorsolateral PFC (dlPFC) can simulate the decisions of others.¹⁰

Figure 1.

Introduction to ToM and related brain areas

(A) A case of ToM.

(B) A schematic hypothetical neural circuit of ToM.

Some existing studies focus on ToM mechanism-inspired decision-making models. A recursive ToM algorithm modeled by probabilistic methods can complete the rock-paper-scissors game.¹¹ Osten et al. introduced the idea of ToM in modeling multiple opponents.¹² Nguyen et al. used ToM to learn cooperative behavior with others quickly.¹³ Baker et al. modeled others’ beliefs through a Bayesian model, which enables simple inference.^14,15 In our past work, we have drawn on the neural circuits and learning mechanisms of ToM and proposed brain-inspired ToM models that incorporated multi-scale neural plasticity mechanisms and coordinated mechanisms in multiple brain areas.^16,17 These algorithms were separately applied to pass the false belief task and the artificial intelligence (AI) safety risks experiment. More complex multi-agent decision-making tasks would significantly challenge them. Rabinowitz et al. predicted the trajectories of others based on their behavioral data and constructed models of others’ goals.¹⁸ This work is limited to modeling others and does not use the predictions of others to help collaborate better. Wang et al.’s work predicted the goals and observations of others. The predicted results jointly helped choose their behavior, which led to better collaboration and communication on cooperative multi-agent tasks.¹⁹ In addition, the vast majority of multi-agent reinforcement learning (MARL) models focus on improvements in deep reinforcement learning (RL) methods, such as centralized learning and distributed decision-making, while lacking references to the mechanism of ToM.^20,21,22 In general, there is still a need to draw deeply on the mechanisms of ToM to improve the efficiency and collaborative effectiveness of multi-agent decision-making.

Considering the various limitations of the existing methods mentioned above, this article aims to build a brain-inspired multi-agent ToM spiking neural network (MAToM-SNN) model and explore its role in MARL. SNNs^23,24,25,26 have the advantages of simulating the brain’s structure and function, extracting spatiotemporal properties, and so on. More importantly, SNNs are more biologically plausible, energy efficient, and naturally more suitable for simulating various cognitive functions of the brain.^23,24,25,26 Information in SNNs is transmitted through non-continuous binary spikes. Thus, it is difficult to optimize SNNs with traditional backpropagation methods. Spike-timing-dependent plasticity (STDP)-based approaches^17,27,28 that draw on the characteristics of synaptic plasticity can solve simple decision-making or control tasks but are not competent for complex decision-making tasks. Some works^{17,29,30,31,32,33} adopted reward-modulated STDP to optimize SNNs to solve RL problems. However, these methods cannot be applied to the end-to-end optimization of deep SNN-based RL or MARL. The conversion of artificial neural networks (ANNs) into SNNs^34,35 and surrogate gradients for directly training deep SNNs³⁶ are feasible methods to improve the performance of deep SNNs. A hybrid learning framework in deep RL uses SNNs to build actor networks and ANNs to build critic networks that perform well in some complex tasks.³⁷ In multi-agent decision-making, our past work is based on reward-modulated SNNs for self-organized swarm unmanned aerial vehicle (UAV) obstacle avoidance,²⁸ which does not extend to MARL environments and lacks references to ToM. Little research on MARL has been implemented using SNNs: Saravanan et al.³⁸ enable SNNs to reduce time and data consumption when completing MARL tasks, and Ye et al.³⁹ combine mean-field MARL with SNNs to approximate device-to-device users and combine RL to optimize the convergence rate. The former explores the feasibility of SNNs in the field of MARL, while the latter solves a practical problem in a specific domain. In contrast, our work explores the significance of ToM in multi-agent cooperation and competition tasks with a combination of energy-efficient SNNs and MARL.

Inspired by the mechanism of ToM in social cognition, this paper proposes a MAToM-SNN model. The core mechanism of ToM is to use the self-experience or observation of others to infer others’ mental states and then optimize self-policy to gain more rewards. Motivated by this, the main contributions of this article can be summarized as follows.

• (1)MAToM-DM integrates MAToM-SNN and the decision-making module. The output of MAToM-SNN, the predicted behavior of others, is integrated into the decision-making module to help the individual and collective achieve more rewards and more efficient collaboration.
• (2)MAToM-SNN uses SNNs to model brain-inspired ToM networks, which integrate spiking neurons and surrogate gradients. MAToM-SNN incorporates Self-MAToM and Other-MAToM to infer others’ behavior, with one based on self-experience and the other based on observations of others’ behaviors.
• (3)MAToM-DM is validated on multiple cooperative (stag hunt game) and mixed cooperative-competitive (multi-agent particle environment) tasks. Experimental results demonstrate that the introduction of MAToM-SNN on MARL algorithms can improve the efficiency of group collaboration and help achieve higher rewards (compared with IQL and value-decomposition network [VDN] with recurrent neural networks [RNNs] and SNNs in cooperative environments and MADDPG with RNNs and SNNs in competitive environments).

Results

In MAToM-DM, each agent predicts the actions of others based on self-experience or observations of others. This predicted information helps to optimize its behavior and improve its own or the team’s reward. The environment contains multiple agents. Each agent contains a ToM module (MAToM-SNN) and a decision-making module, as shown in Figure 2. The inputs to MAToM-SNN are one’s observations of others. The multi-layer SNNs characterize the attributions of mental states to others. The output of MAToM-SNN is a prediction of others’ behavior. MAToM-DM concatenates the predicted actions of others obtained from MAToM-SNN with the observations of the environment as inputs to the decision-making module to learn the optimal policy. We model MAToM-SNN in two ways, Self-MAToM and Other-MAToM, and they have the same structure as shown on the right side of Figure 2. The Self-MAToM module is trained by self-experience. Nevertheless, we use historical observations of others to train the Other-MAToM module. This article compares the effects of self-modeling (Self-MAToM module) and others’ modeling (Other-MAToM module) on ToM. The interaction and collaboration between multiple agents with ToM could accomplish complex decision-making tasks such as cooperation and competition. The proposed MAToM-DM method is verified on cooperative stag hunt tasks⁴⁰ with discrete space and time and a cooperative-competitive multi-agent particle environment^22,41 with continuous space and discrete time. We specifically apply the neuron model and the surrogate gradient in the brain-inspired cognitive intelligence engine (BrainCog)²⁶ in the process of implementing the modules for this work.

Figure 2.

Overview of MAToM-DM and architecture implementation for MAToM-SNN

MAToM-ToM contains Self-MAToM and Other-MAToM, which are trained with the help of the self-trajectory and other-trajectory saved in the buffer, respectively. In addition, Other-MAToM can learn by using Self-MAToM’s parameters.

Stag hunt games

We evaluate the MAToM-SNN in cooperative tasks with discrete space and time in which two agents can act cooperatively to obtain more rewards or act independently to obtain less rewards. Because each agent learns the same policy, the experiences of the self and others are the same. In this environment, we do not distinguish between Self-MAToM and Other-MAToM. The environment of these tasks is a 5 ∗ 5 grid. All agents can take physical actions, including left, up, down, right, and stay. In each experiment, agents’ observations are one-dimensional vectors containing the agent’s position, others’ positions, the stag’s position, and plants’ positions. The detailed stag hunt task environments are depicted in Figure 3 and are introduced as follows.

Figure 3.

Illustrations of stag hunt tasks (left column) and results (right column)

The resulting figure shows the mean episode reward across the different random seeds.

Harvest

The environment contains two cooperative agents and randomly maturing plants. When an agent is on top of a plant, it harvests this plant. If an agent harvests a young plant, it will receive a young reward, which is 1, and the other agent will receive nothing. If an agent harvests a mature plant, both agents will receive the mature reward, which is 2.

Escalation

This scenario consists of two cooperative agents and a stag. At first, the stag is not moving. The stag will start moving when two agents walk together to the stag. Only when the two agents keep chasing the stag together will both receive a reward of 1. Otherwise, the game will end.

Hunt

This scenario contains two agents, two plants, and a randomly moving stag. Each agent can harvest a plant and obtain a plant reward, 1. Two agents can collaborate to hunt the stag and obtain a stag reward of 5. A single agent cannot get close to the stag or else it will be injured and get a penalty of −5.

In three tasks, both SNNs and RNNs are used to build the decision-making module, which is trained by the VDN²¹ method. We integrated MAToM-SNN into the decision-making module. During training of the network, the batch size is 250. The length of an episode is 50 steps. The learning rate of the decision-making module is 0.99. In the harvest task, MAToM-DM learns how to harvest mature plants. In the last two tasks, two agents learned to chase the stag together.

Comparative experiments

To validate the effectiveness of the MAToM-SNN model, we compare our methods with baselines, including DQN,⁴² IQL,²⁰ RNN-based VDN (RVDN),²¹ and SNN-based VDN (SVDN).

The comparative results on different baselines are shown in Figure 3 and Table 1. MAToM-RVDN and MAToM-SVDN imply that MAToM-SNN is combined with RVDN and SVDN, respectively. The experimental results show that RVDN and SVDN outperform the IQL and DQN models. The rewards of RVDN and SVDN in Table 1 imply that VDNs with different structures (such as RVDN and SVDN) perform differently in different tasks. Nevertheless, their total mean rewards have been improved after combining with MAToM-SNN. As shown in Figure 3 (MAToM-RVDN line and MAToM-SVDN line), MAToM-SNN can improve learning efficiency in most cases, especially when MAToM-SNN is added to RVDN (MAToM-RVDN), and the learning speed is dramatically improved. In addition, MAToM-SNN can guarantee almost no loss of performance and even improve performance. Similarly, MAToM-SNN also improves the decision-making ability of the agent with SVDN, especially in the hunt task, and MAToM-SVDN has achieved remarkably excellent performance. In the harvest task, the performance of SVDN is slightly better than that of MAToM-SVDN in the second half of the curve. The possible reason for this is that the two agents do not need to cooperate in hunting in this scenario, but both must work to collect mature plants. When both agents see a mature plant, both infer that the other will harvest the plant and so neither does so. Therefore, the result converges to the optimal local solution. Because RVDN performs better in this task, it causes the model to cross the local optimization and makes MAToM play its role.

Table 1.

Comparison results of stag hunt tasks

Method	Harvest	Escalation	Hunt
DQN42	$43.02\pm 7.06$	$1.89\pm 0$	$2.02\pm 0.15$
IQL20	$65.69\pm 5.22$	$52.09\pm 2.72$	$85.07\pm 10.02$
RVDN21	$77.79\pm 3.17$	$44.88\pm 9.35$	$111.53\pm 12.21$
MAToM-RVDN	100.06$\pm$3.43	83.47$\pm$4.19	106.07 $\pm$ 14.23
SVDN	$71.17\pm 3.33$	$68.62\pm 6.63$	$82.07\pm 16.27$
MAToM-SVDN	74.04$\pm$2.96	78.41$\pm$2.85	120.39$\pm$12.63

The table records the mean and standard deviation of raw rewards.

In summary, MAToM-SNN can improve the cooperation efficiency and total reward of MARL tasks. The advantage is mainly reflected in that the MAToM-SNN’s prediction of others’ future behavior can help the agents to expand the decision information and thus make more rational decisions.

Multi-agent particle environment

We choose multi-agent particle environment (MPE) for more complex environments where cooperation and competition coexist. The environment contains two competing teams (team A and team B). There is at least one agent within the team. If the team has more than one agent, then the agent within the team has the same reward function. The agents can take actions: up, down, left, right, and stay. Notably, the observation of the opposing agents is not consistent. We describe all scenarios (as shown in Figure 4) in detail below.

Figure 4.

Illustrations of the multi-agent particle environment (left column) and results (right column)

Each curve converges at the end of training. The shading region is a $95\%$ confidence interval across the different random seeds.

Physical deception

In this scenario, team A has one agent, and team B has two agents. The positive reward for team B is based on how close the closest one of them is to the target landmark. The smaller the distance, the greater the positive reward. The negative reward for team B is based on how close team A is to the target landmark. The smaller the distance, the smaller the negative reward. Team A is rewarded based on the distance to the target, but it does not know which landmark is the target landmark. Therefore, agents in team A need to infer agents in team B to predict the target landmark.

Predator-prey

Team A has one prey, and team B has three predators. When team B collides with team A, team B is rewarded, and A is penalized instead. Two landmarks impeded the agents’ way.

World communication

The environment contains team A (containing two agents), team B (containing four agents, who catch team A), two pieces of food that agents in team A can eat, two forests that can hide all the agents inside, and a landmark impeding the way. Only the leader of team B can know the coordinates of the agent in the forest and can choose to tell team members by communication.

In the MPE environment, the design of the decision-making module in MAToM-DM is based on MADDPG. We construct the SNN and RNN versions of MADDPG as the decision-making modules and use the output of MAToM-SNN as the input of the actor network of the decision-making module. During training of the MAToM-SNN, the batch size is 1,024. The length of an episode is 25 steps. The learning rate of the decision-making module is 0.99.

Comparative experiments

We compare our methods with DDPG,⁴² MADDPG,²² RMADDPG, and SMADDPG. Since there are different kinds of agents in mixed cooperative-competitive environments, MAToM-SNN can use both self-experience (the Self-MAToM module) and historical observations of others (the Other-MAToM module) to learn. Therefore, we also compare the experimental results of the Self-MAToM module and the Other-MAToM module.

To clearly explain the results, we record part of a learning curve in Figure 4 and the raw rewards in Table 2. As seen from the table, RMADDPG and SMADDPG outperform the DDPG and MADDPG models. Self-MAToM and Other-MAToM significantly outperform the baseline DDPG. Moreover, Other-MAToM even outperforms the baseline MADDPG model significantly, and in the predator-prey and physical deception scenarios, Self-MAToM also performs better than MADDPG. In summary, although MAToM-DM with RNN and SNN structures have different performances for different experimental tasks, the mean rewards have been improved after combining with Other-MAToM. We analyze the reason why Self-MAToM is slightly inferior to Other-MAToM. In a competitive task (the world communication scenario) where the opponents’ strategies are significantly different from self, using self-experience (Self-MAToM) to infer the opponent’s behavior would be more inaccurate and even lead to a decline in performance. Therefore, MAToM-DM with Other-MAToM performs better than Self-MAToM on mixed cooperative-competitive tasks.

Table 2.

Comparison results of multi-agent particle environments

Method	Physical deception	Predator-prey	World communication
DDPG42	$-8.07\pm 9.37$	$-3.34\pm 20.30$	$6.33\pm 8.00$
MADDPG22	$-0.29\pm 1.67$	$13.47\pm 5.42$	$13.36\pm 8.93$
RMADDPG	$0.03\pm 1.22$	$19.25\pm 4.42$	$31.89\pm 7.37$
Self-MAToM-RMADDPG	$0.12\pm 1.47$	$19.98\pm 4.88$	$31.49\pm 7.87$
Other-MAToM-RMADDPG	0.14$\pm$1.11	20.36$\pm$4.24	32.16$\pm$11.69
SMADDPG	$1.72\pm 1.11$	$11.72\pm 7.52$	$18.88\pm 5.20$
Self-MAToM-SMADDPG	$1.84\pm 0.9$	$15.15\pm 4.18$	$8.57\pm 8.76$
Other-MAToM-SMADDPG	2.43$\pm$1.06	20.70$\pm$4.63	19.19$\pm$4.68

The table records the mean and standard deviation of raw rewards.

Discussion

Taking inspiration from the ToM mechanism in social cognition, this article proposes a brain-inspired MAToM-SNN model that brings ToM into multi-agent decision-making. The proposed model uses more biologically plausible SNNs to construct MAToM-SNN and improves the efficiency and performance of multi-agent collaboration. Extensive experiments demonstrated that the MAToM-SNN model achieves outstanding performance in cooperative and competitive tasks. Compared with multiple baseline models, the proposed model is more stable, robust, and efficient and performs well in different stochastic scenarios.

At present, MARL mainly draws on the classical algorithms of RL to improve the objective function of the actor network and the loss function of the critic. Alternatively, from the perspective of multi-intelligent systems,^21,43,44 scholars have used the Q value of each agent to fit the Q value of the system. Most of these methods use the idea of centralized training decentralized execution (CTDE), which means that the agents are independent in the interaction process. The problem is that the local information observed by the agents would lead to biased decisions. Communication^45,46 methods allow agents to receive information directly from others and thus enrich the observation information to improve decision-making accuracy. These methods directly access the information of all other agents and are unsuitable for competitive tasks.

Unlike previous MARL models, MAToM-DM considers ToM in multi-agent interactions, which plays a crucial role in social cognition and is naturally suitable for solving multi-agent decision-making tasks. In addition, our main highlight is that we adopt full SNNs to implement two kinds of brain-inspired ToM models: Self-MAToM infers others’ behavior based on self-experience, and Other-MAToM is based on the observation of others. Then, the predicted behaviors of others are used to adjust the agent strategies to better interact with others. Therefore, our MAToM-SNN model is more biologically interpretable, plausible, and predictably effective, as well as convincingly illustrated by the superior results on different cooperative and competitive tasks.

Analysis of MAToM-SNN

To further analyze the role of MAToM-SNN, we conduct ablation study experiments on multiple competitive scenarios using Other-MAToM. For the opposing team A and team B, we grant the ability of ToM to team B (no ToM to team A at this point), as well as to both team A and team B. Table 3 depicts the mean episode reward for each team on three competitive tasks. The experimental results reveal that the team with the ToM ability (B-ToM) achieved a greater reward than the team without ToM (B). In addition, when only one of the opposing teams has ToM capability (B-ToM), the inferred team’s reward (A) is reduced. This suggests that ToM boosts one team’s rewards and suppresses the opposite team’s rewards. When both teams have ToM, the team with more agents (team B) can achieve more rewards and suppress the team with fewer agents from gaining rewards. This also validates that our MAToM-SNN model will help teams with larger numbers improve performance in competitive tasks.

Table 3.

Ablation study experiment results of multi-agent particle environments

Method	Team	Physical deception	Predator-prey	World communication
MADDPG22	A	$-5.96\pm 4.41$	$-11.45\pm 2.71$	$-7.53\pm 8.00$
	B	$5.81\pm 2.3$	$23.51\pm 7.38$	$20.89\pm 8.74$
MAToM-SMADDPG	A-ToM	$-9.65\pm 1.29$	$-17.27\pm 2.42$	$-13.82\pm 2.45$
	B-ToM	$12.06\pm 1.75$	$37.97\pm 6.62$	$33.01\pm 6.29$
MAToM-SMADDPG	A	$-12.59\pm 1.34$	$-17.32\pm 2.59$	$-12.25\pm 3.43$
	B-ToM	$13.9\pm 1.64$	$39.24\pm 7.07$	$33.47\pm 4.76$

The table records the mean and standard deviation of raw rewards.

Effects of Self-MAToM

To further analyze the role of Self-MAToM, we conduct ablation study experiments on multiple competitive scenarios. Intuitively, when we encounter an unseen person for the first time, we infer others by considering others in our place. Moreover, as we get to know others, we adjust our expectations of them based on their behaviors. Inspired by this, we designed Self-Other-MAToM as a combination of Self-MAToM and Other-MAToM. Self-Other-MAToM uses Self-MAToM’s parameters to initialize the network and learn from the observation of others. We can share parameters between Self-MAToM and Other-MAToM because they adopt the same network structure. At the same time, we proved that the number of parameters of both Self-MAToM and Other-MAToM is the same in the same experiment as shown in Table 4.

Table 4.

The number of parameters

	Physical deception	Predator-prey	World communication
Params	5.539K	5.803K	6.199K

Figure 5 depicts that, compared with Self-MAToM and Other-MAToM, Self-Other-MAToM has a higher starting reward because Self-Other-MAToM uses self-experience to predict others in the early stage of training so that it can model others more quickly. The starting reward of team A with Self-Other-MAToM is lower than that with Self-MAToM or Other-MAToM in the predator-prey scenario. One possible reason for this is that team B with Self-Other-MAToM is too strong, so team A is completely suppressed. Therefore, it is not a problem of Self-Other-MAToM that leads to the score decline but a lag in the competition. Based on the above analysis, the results prove that the initial Self-MAToM parameters will help agents participate faster in cooperative and competitive tasks at the beginning of the experiment.

Figure 5.

Learning curves of Self-MAToM, Other-MAToM, and Self-Other-MAToM

The left column shows the mean episode rewards of the system. The right column shows the mean episode rewards of each team.

In this article, we find that modeling and predicting others’ behavior based on others’ observations (Other-MAToM) is superior to using self-experience to predict others’ behavior (Self-MAToM). The possible reason for this is that self-experience and others’ experiences are different, especially in competitive tasks. The Other-MAToM model performs better in predicting others. In addition, Self-MAToM can help Other-MAToM learn quickly. The self is a prerequisite for inferring others. Thus, it is essential to infer about others based on self-experience when information about them is incomplete (e.g., B’s inference about G in Figure 1A). This has inspired us to explore better integration of self and others’ experiences into the ToM model in future studies and to delve further into more biologically plausible mechanisms of ToM.

With the development of AI, multi-agent systems are quickly entering human society, in ways such as autonomous driving and logistics delivery. ToM is an advanced cognitive function evolved in humans during social decision-making processes that allows individuals to predict the behavior of others by understanding their intentions. This provides valuable insights for building social intelligence, as ToM enables agents to act interpretably. In future research, we can explore how ToM can help multiple agents generate safe, trustworthy, and ethical behavior during interactions, as well as how high-order ToM performs in complex decision-making tasks. Like humans, the development of ToM in agents requires time and exposure to a variety of samples. In other words, when an agent with ToM encounters different types of agents, it needs to spend more time learning how to distinguish between them and what mental states each agent may produce. Therefore, it is worthwhile to explore how to improve the efficiency of the ToM model.

Experimental procedures

Resource availability

Lead contact

Further information and requests for resources should be directed to and will be fulfilled by the lead contact, Dr. Yi Zeng (yi.zeng@ia.ac.cn).

Materials availability

This study did not generate new unique materials.

Data and code availability

All original code has been deposited at GitHub under https://github.com/BrainCog-X/Brain-Cog/tree/main/examples/Social_Cognition/MAToM-SNN and at Zenodo under https://doi.org/10.5281/zenodo.7768874 and is publicly available as of the date of publication. The data used in this article are available from the code, and readers can also contact the lead contact for requests.

SNNs provide a better understanding of biological systems by simulating how information is transmitted through spikes in the brain. ToM, as a higher cognitive function of the human brain, plays an important role in the human decision-making process. In this article, we propose an MAToM-DM model consisting of a ToM module (MAToM-SNN) and a decision-making module. We designed the ToM module based on SNNs to infer the behavior of others. The decision-making module accepts MAToM-SNN’s predictions of others’ actions as input to guide self-decision-making and improve performance during cooperation or competition with others.

SNN.

This subsection introduces the neuron model of the SNN and the training method. The leaky integrate-and-fire (LIF) model^47,48 is used to describe the dynamic activity of neurons. The differential Equation 1 describes the dynamic process of voltage change caused by the postsynaptic potential (PSP), ${\tau }_m$ represents the time constant, $V_{rest}$ is the reset voltage, $V_t$ is the updated voltage, and $x$ is the PSP. When $V_t$ exceeds the threshold $V_{th}$, the neuron fires as shown in Equation 2. $s\left(V_t\right)$ is a spike function.

$${\tau }_m\frac{\mathrm{d}V\left(t\right)}{\mathrm{d}t}=-\left[V\left(t\right)-V_{rest}\right]+x$$ (Equation 1)

$$s\left(V_t\right)=\mathrm{\Theta }\left(V_t-V_{th}\right)\text{with}\mathrm{\Theta }\left(V_t-V_{th}\right)=\{\begin{array}{ccc}1\text{,}& & \text{if}{V}_t-V_{th}\ge 0\\ 0\text{,}& & \text{else}\end{array}$$ (Equation 2)

To make the model competent for both continuous and discrete control tasks, the network output is a continuous value. Therefore, we distinguish the last layer neurons in SNNs from the other layers, as shown in Figure 6. Information is passed between SNN layers by spikes. The output of the network is the voltage. We apply this setting to all modules in MAToM-SNN. The PSP represents the result of the spike-weighted accumulation of the $l-1$ layer as the input to the $l$ layer as shown in Equation 3.

$$x_t^l\left(t\right)=\sum _i{w}_i^l{s}_{t,i}^{l-1}$$ (Equation 3)

Figure 6.

Illustrations of LIF neurons

Since gradient backpropagation cannot optimize SNNs because the spike sequence is not differentiable in the derivation process, we trained SNNs with the help of the surrogate gradient instead as shown in Equation 4.⁴⁹ In the experiments, $\alpha$ is equal to 2.

$$\frac{\partial s\left(V_t\right)}{\partial V_t}=\frac{\alpha }{2\left(1+{\left(\frac{\pi }{2}\alpha V_t\right)}^2\right)}$$ (Equation 4)

MAToM-SNN

In this subsection, we describe the MAToM-SNN modeling process in detail. We applied a four-layer fully connected SNN to construct a ToM module. The inputs to the ToM module are observations of others (including environmental context and observations of others’ states and behaviors). The output is the prediction of others’ actions. The hidden layer encodes others’ mental states.

The ToM module learns the parameters in a supervised manner. We use Kullback-Leibler (KL) divergence (Equation 5) as the loss function (Equation 6).

$$\mathrm{L}=\text{KL}\left(\mu \Vert {\mu }^{\ast }\right)$$ (Equation 5)

$$L^{ToM}=\sum _j\left[{\mu }_j·\log \left({\mu }_j\right)-{\mu }_j·\log \left({\mu }_j^{\ast }\right)\right]$$ (Equation 6)

In this article, we consider two sources to infer others’ mental states and predict others’ actions. One is self-experience, and the other is observation of others. The Self-MAToM module and Other-MAToM module have the same structure, as shown in Figure 2.

Self-MAToM module

This module is considered to be coding memories. Self-experience is considered to be the training sample, which is the same as the history trajectory saved in the RL buffer. The Self-MAToM module is to infer others with self-experience. Therefore, the training sample should remove the quantities that are unobservable to others, such as others’ target landmarks. The continuous optimization of the loss function (Equation 6) indicates the continuous improvement of the Self-MAToM module.

Other-MAToM module

The training sample uses observations of the historical trajectories of others. This history trajectory is stored in the same way as RL. The difference is that the ToM module’s buffer also contains behavioral observations of others. In the loss function (Equation 6), $\mu$ is the target policy, and ${\mu }^{\ast }$ is the predicted policy.

MAToM-DM with different decision-making modules

We constructed the decision-making module based on the existing MARL algorithm. In the previous subsection, we described the ToM module. The purpose of the ToM module is to be integrated in front of the decision-making model. We integrate the ToM module on the critic-based VDN and actor-critic-based MADDPG algorithms as follows.

Markov games

We describe multi-agent tasks as a multi-agent extension of Markov decision processes (MDPs). A Markov game for $N$ agents contains the states, $S$; a set of actions, $A=\left\{A_1,\dots ,A_N\right\}$; and a set of observations, $O=\left\{O_1,\dots ,O_N\right\}$. Observations are related to states and actions. Each agent selects its actions according to the policy ${\pi }_{{\theta }_k}$ and generates the next state depending on the state transition function. At the same time, each agent will obtain rewards, $R=\left\{R_1,\dots ,R_N\right\}$. After each agent interacts with the environment, a trajectory ($o_{k,1}$, $a_{k,1},r_{k,1},o_{k,2},\cdots$) will be formed and stored in buffer $D$.

MAToM-DM with VDN

The cooperation tasks

We added ToM to the VDN method. VDN extends DQN to multi-agent tasks and optimizes the team reward. VDN learns to decompose the team value function into agentwise value functions.²¹ Each agent’s observations are composed of the agent history $\left(h_k\right)$ and its current action $\left(a_k\right)$. We denote $\overline{a_o^k}:=\left(a_k^1,a_k^2,\dots ,a_k^d\right),\left(k\ne d\right)$ as a tuple of predicted others’ actions, and $a_k^d$ means the prediction of agent $k$ to agent $d$. We concatenated the predicted actions of others $\overline{a_o^k}$ obtained by the ToM module with the original observations as the inputs. In Equation 7, we describe the value-decomposition process that integrates the ToM module. This means that the system’s total Q function can be approximately equal to the sum of all agents’ Q functions.

$$\begin{array}{l}Q\left(\left(h^1,h^2,\dots ,h^d\right),\left(a^1,a^2,\dots ,a^d\right),\left(\overline{a_o^1},\overline{a_o^2},\dots ,\overline{a_o^d}\right)\right)\approx \\ \sum _{k=1}^d{\tilde{Q}}_k\left(h^k,a^k,\overline{a_o^k}\right)\end{array}$$ (Equation 7)

MAToM-DM with MADDPG

The competitive tasks

We added the ToM module to the MADDPG method. The MADDPG algorithm endows each agent with an actor and a critic network. The actor network obtains the observations to generate actions, and the critic network evaluates the observations and other people’s behavior. The actions obtained from the actor network are considered part of the critic network inputs. The actor network obtains the best behavior through a gradient-based backpropagation algorithm. The objective function is shown in Equation 8, where ${\mu }_k$ is the policy of agent k and $Q_k^{\mathbf{\mu }}$ is the critic function of ${\mu }_k$. The critical network is updated with the temporal difference (TD) error. We concatenated the actions of others, $\overline{a_o^k}$, obtained by ToM with the original observations as part of the actor network inputs.

$${\nabla }_{{\theta }_k}J\left({\mathbf{\mu }}_k\right)={\mathbb{E}}_{\mathbf{o},a\sim \mathcal{D}}\left[{\nabla }_{{\theta }_k}{\mathbf{\mu }}_k\left(a_k|o_k,\overline{a_o^k}\right){\nabla }_{a_k}{Q}_k^{\mathbf{\mu }}\left(\mathbf{o},a_1,\dots ,a_N\right){|}_{a_k={\mathbf{\mu }}_k\left(o_k,\overline{a_o^k}\right)}\right]$$ (Equation 8)

Published: June 23, 2023