Imaging the Social Brain by Simultaneous Hyperscanning During Subject Interaction

Established evidence supports the idea that the social systems of primates, like their brains, are more complex than those of other species. These systems exhibit peculiar behavioral processes, including deception and coalition formation. The hypothesis that links the size of primate species’ brains to the complexity of their social system is called the social brain hypothesis.¹ One of the most relevant and characteristic traits of the social brain is cooperative behavior.

As humans, our daily life experiences testify to the fact that through cooperation with other human beings we can achieve goals that we could not reach as individuals. Despite the common occurrence and impact of these cooperation processes on our life, we still know little about the neural processes that underline the emergence of a cooperative (or competitive) interaction in a group of subjects. We know even less about how these social interactions simultaneously influence brain activities and thus the behavior of the subjects involved.

This article illustrates how modern brain-signal acquisition and processing technologies can help us investigate brain activity and connectivity related to social interactions in human subjects. We present an advanced methodology to measure simultaneously the neural activity of different brains during daily life interactions in order to understand the neural processes generating and generated by social cooperation or competition. We measure such activities by simultaneously recording electroencephalography (EEG) from couples of subjects playing the Prisoner’s Dilemma (PD) game and in a group of four subjects playing a card game. By simultaneously studying interacting subjects, we thus move from the physiology of the individual to the physiology of the group.

Imaging Brain Activity In Vivo

In the last 10 years, powerful brain-scanning techniques such as functional magnetic resonance imaging (fMRI) and high-resolution EEG (HR-EEG) have provided important insights into the neural basis of memory, decision-making, and other cognitive functions at the basis of social interactions. Such techniques can measure the hemodynamic or neuroelectric signatures of brain activity. The standard experimental paradigm employed consists of measuring brain activity in different individual subjects while they perform an identical sensory, cognitive, or motor task.

Researchers have also applied similar experimental paradigms to the study of brain activity during social interactions. However, because of technological limitations, especially for the fMRI studies, most studies on social interaction have used offline designs, in which scanned subjects did not actually take part in face-to-face social interactions with other humans. In such cases, the neural responses are measured in individual subjects during simulated situations, in which they interact with a computer or with another person outside the scanner.

Simultaneous Multisubjects’ Recordings

A major limitation of the approach used in the majority of the studies of “social interaction” is that they measure neural activity in only one of the brains participating in the task. Thus, the interaction between cooperating, competing, or communicating brains is not captured directly when it occurs. Instead, it is inferred by independent observations aggregated by cognitive models and assumptions that link behavior and neural activation. This approach is unsatisfactory when we want to measure the neural substrates underlying the interaction between individuals, which is a process involving different brains simultaneously. To reveal the neural substrates supporting the development of a cooperation/deception behavior between individuals, direct observation of the interaction established between the brains of different subjects is necessary.

The problem is that the brain does not necessarily (or even typically) perform the same task using exactly the same neural resources from one occasion to the next. Therefore, we cannot simply aggregate independent observations under generalizations that link types of behavior and their neural signatures. That is, if we first scan a subject playing one strategic position in a game, then another subject in the opposite strategic position, and so on iteratively, and then assemble a post hoc picture of the brains, we might lose most of the actual interaction.

The simultaneous, multisubject recording of different subjects engaged in interactive tasks lets us directly scan such interactions. This new approach is called hyperscanning, and researchers have performed it by using either fMRI or EEG devices.^2–7 Previously, the neural activity in the brains of a group of people during the formation of different strategic computations could only be conjectured. With the fMRI or EEG/MEG (magnetoencephalography) hyper-scanning, these activities could be tracked and appropriately quantified.

P. Read Montague and his colleagues pioneered fMRI hyperscanning.^2–4 Recently, the feasibility of EEG hyperscanning was demonstrated with experiments involving two or more subjects simultaneously⁵ during a card game or while subjects engaged in a task related to game theory.^6–7

The advantages of the use of neuroelectric hyperscanning when compared to the use of fMRI are in the ecological setup of the recordings. In fact, the use of EEG equipment allows the subjects to freely perform the social tasks without spatial constraints, while fMRI hyperscanning obliges the subject to lie within the device, staring at a monitor. Moreover, EEG profits from a higher temporal resolution, performing analysis of different oscillations and unveiling significant information that might be lost by the hemodynamic data. Additionally, it is simpler to record two, three, or four subjects simultaneously with EEG hyperscanning,⁵ as Figure 1 shows. Finally, the cost of an EEG device is about two orders of magnitude lower than an fMRI device.

Figure 1.

A typical setup of an electroencephalography (EEG) hyperscanning during a card game. (a) The brain activity is simultaneously recorded from four subjects during the different rounds of the game. (b) Realistic models of the individual subjects’ heads, including scalp and cortical envelopes, are presented for each subject. The arrows depict the statistically significant connections estimated between the cortical areas of two members of a team during a phase of the game. The arrows start from a cortical region and point toward another cortical region. The causal links mean that the activity in the target cortical region can be better modeled by including the source region’s activity.

Hyperscanning During Social Interactions

Game theory has proved useful in the investigation of the neural basis of social interaction. In fact, this discipline allows a formal definition of social situations in which the players might, from time to time, profit or lose from a cooperative or individualistic behavior. We performed an EEG hyperscanning experiment involving 52 healthy subjects, organized in 26 couples, who interacted by playing the PD game. In particular, we estimated the statistically significant brain activities related to the different phases of the game, divided according to the different strategies of cooperation or competition chosen by the players to get or share an economic resource.

The Iterated PD involves two players, each facing two possible choices: to cooperate with or defect from the opponent. The choice is blind—that is, players must make their decision without knowing what the other player is going to do. The outcome is not fixed; it depends on the combination of choices. If both players cooperate, they both win money (the pure-cooperation condition). If one player cooperates and the other defects, all the advantage goes to the defector. If both players defect, they have small losses (the pure-defect condition). A tit-for-tat condition occurs when each player adopts a certain behavior not to maximize his or her gain, but to react to his or her opponent’s behavior in the previous run of the game, by iterating the opponent’s past choice in the next move. The aim of the game is to reach the highest possible score.

The 52 subjects we recorded were all informed of the aim of the experiment, and they all approved the study. The subjects were beside each other with a screen displaying the necessary game information and the timing of the tasks in front of them. They expressed their choices (to cooperate or defect) using a keyboard, and the computer recorded each subject’s response and generated a mark on the subjects’ EEG traces for subsequent offline analysis. The choice was blind to the other player.

The trial started with the presentation of the payoff matrix related to the decision that a subject could make in the game. The players were then prompted to enter their choices. Afterward, we showed them the trial results for 4 seconds, reporting the other player’s cooperation/defection choice and each subject’s total score. The EEG analysis was performed within this 4-second period and considered significant for successive decisions. We used a 64-channel system (BrainAmp by Brainproducts) to record EEG electrical potentials using an electrode cap, while also recording electromyogram (EMG) and electrooculogram (EOG) signals. The sampling rate was 200 Hz. We then corrected EEG signals from eye movements and filtered muscular artifacts from all recordings. Only artifact-free trials were processed for all the subjects and submitted to the subsequent analysis.

HR-EEG technologies enhance the spatial information content of the EEG activity.^8–10 In this study, we reconstructed the cortical activity from HR-EEG recordings using the average head model available from McGill University. Estimation of the current density strength for each dipole was obtained by solving the linear inverse problem.^8–10 From the cortical estimated waveforms, the spectral activity during the task time interval was first estimated for each of the thousands of dipoles of the cortical model; then such spectral activity was statistically compared with those related to a rest period.

During the rest period, each subject was seated in front of the screen watching images similar to those used in the game, but without any relation with the game itself. T-test values between the power of the frequency spectra during the task and the rest were then mapped on the cortical model in the different frequency bands: theta 3–7 Hz, alpha 7–12 Hz, beta 13–29 Hz, and gamma 30–40 Hz. Because of multiple comparisons, we used the Bonferroni- corrected statistical threshold to reach a nominal value of p < 0.01.

Mapping the Statistically Significant Group Activations

Overall, the statistically significant cortical activity generated by the analyzed behaviors (cooperation, defect, and tit-for-tat) is greater than the cortical activity observed during the rest condition. Figure 2 shows the statistically significant spectral power distribution in the theta band (averaged for the group of subjects investigated) in the cooperation and defect conditions when compared to the rest period.

Figure 2.

Distribution of the statistically significant differences in power spectra for the (a) cooperation and (b) defect conditions in the theta frequency band. We depict results on the average cortex model used in the study, seen from six different perspectives. The first row (from left to right) presents the brain cut sagittally, view from the right, then the cortical surface view from the left and the brain view from above, frontal part down. The second row shows the sagittal view of the cut brain from the left, the cortical surface from the right, and the frontal cortical areas. The grey areas indicate no statistically significant difference with respect to the rest period. The areas in color (from dark red to yellow) indicate a significant activation with respect to the rest. We only show activations common to at least 36 subjects.

We depict the results on the average cortical model used in the study, seen from six different perspectives. The areas that showed no significant difference with respect to the baseline are in grey, and the areas that showed a statistically significant activation with respect to the baseline are in color. We represent the Bonferroni-corrected statistically significant differences with colored pixels. The color scale indicates the number of subjects sharing the statistically significant cortical activation in that pixel across population. (The yellow is associated with a higher number of subjects.) Only activations common to at least 36 subjects are reported.

Interestingly, relatively few cortical areas displayed a statistically significant difference in the power spectra when compared to the rest state. In particular, during the cooperation task (left panel, Figure 2), we can see an activation of the right inferior frontal gyrus cortex and the left orbitofrontal cortex. Several prefrontal activations are visible in the left and right hemispheres.

The right panel in Figure 2 shows the distribution of the statistically significant power spectra during the defect condition across the whole population in the theta frequency band. We noted an increase in brain activity again in the right supramarginal gyrus as well as in the dorsolateral prefrontal cortex and the orbitofrontal areas. In particular, the activation of the orbitofrontal cortical areas appears to be proper for the defect condition when compared to the cooperation or tit-for-tat (not reported here).

Because of space constraints, we do not report statistically significant cortical activities in the alpha, beta, and gamma frequency bands. Results in this respect show specific activations related to the different bands as well as a partial overlapping with the activations in the theta band.

The results we present here focus on the theta band because (based on the evidence we give in the article), the differences between the two conditions (cooperation and defect) were stronger in this frequency range. Finally, the statistically significant activity elicited for the tit-for-tat condition is characterized by a cortical activity different from the rest condition in the prefrontal cortical areas of the right hemisphere.

The results obtained from EEG hyperscanning and presented here are related to increased activity in the dorsolateral prefrontal and orbitofrontal areas during the different task phases when compared to the baseline. These cortical activities are specifically larger during the defect conditions than in other situations characterizing the social interaction between the subjects investigated (that is, cooperation or tit-for-tat).

Functional Connectivity Hyperlinks

As we described earlier in this article, we estimated cortical activity from HR-EEG recordings by solving the associated linear inverse problem.^8–11 We then considered the cortical activity related to particular cortical areas, called regions of interest (ROIs), depicted on the realistic reconstruction of the cortical model. In particular, the considered ROIs are related to the frontal, prefrontal, and parietal cortical areas, including the Brodmann areas (BA) 8, 9, 10, 6 lateral, 7, as well as the Anterior Cingulate Cortex (ACC) and Cingulate Motor Area (CMA).¹²

From the cortical estimated waveforms, we computed the functional links between ROIs using the Partial Directed Coherence (PDC) technique.¹³ PDC is a full multivariate spectral measure used to determine the directional influences between any given pair of signals in a multivariate data set. It is computed on a Multivariate Autoregressive (MVAR) model that simultaneously models the whole set of signals. Researchers have used MVAR processes to study the functional connectivity between cortical waveforms.^11,13 To be used for EEG hyperscanning, this approach was improved by a statistical normalization to adapt MVAR processes to the brain activities estimated on different brains. We then estimated the functional links between different brains (hyperconnectivity) by taking into account the activity generated simultaneously in ROIs belonging to different subjects engaged in the cooperation games.

When used to estimate the causality relationships between different areas of the same brain, MVAR modeling assumes the same “generating system” for each waveform representing the ROI’s activity. However, because the EEG can show large deviations from subject to subject, a normalization procedure is needed to estimate multisubject causality. For this reason, we z-transformed the data recorded from each subject before estimating a general MVAR model to extract the causality relationships between the subjects.

Data from each subject were z-transformed, after the linear inverse estimation, in the time domain, with respect to the EEG during a baseline condition that was recorded before the start of each round of the PD game. During the baseline condition, the subjects were sitting in front of the same screen and were exposed to the same stimuli; they were pressing keystrokes as during the game but were not interacting. In this way, we can assure that the connectivity estimation we got was not caused by the simultaneous exposure of the subjects to the same stimuli.

With the z-transformation, we kept the information about the variations in the two data sets without keeping the information related to the different power spectral densities, which could lead to spurious results. In this way, we also solved the problem of different power spectra between different subjects.

Finally, the MVAR estimation was applied to the sets of data related to the two subjects playing together. We made an a priori selection of six cortical areas for each subject (12 areas total) to be included in the hyperconnectivity estimation. In fact, to ensure accuracy in the MVAR modeling, the number of nodes that it is possible to consider strongly depends on the amount of data used for the parameters estimation. Given the number of trials of this study, 12 areas was the best compromise between the need to include relevant sources of information in the model and the necessity to obtain a sufficiently accurate estimation. We chose the selected areas both from the activation maps obtained in this study and on the basis of previous studies performed with the aid of the fMRI.

Figure 3 shows the average results obtained in the population of 52 subjects. We reported the statistically significant (after Bonferroni correction for multiple comparisons) functional connectivity links obtained for the different conditions (pure cooperation and pure defection). The statistically significant PDC links reported here describe the existence of a functional connection between the brain signals estimated in cortical areas of different subjects—such functional links are called hyperconnectivity links, or hyperlinks.

Figure 3.

Common patterns of the statistically significant functional connectivity links in the population of subjects when they decided to adopt (a) a pure-cooperative or (b) a pure-defect behavior. Such connectivity links describe parts of the brain in which the signals shared similar properties. All the represented links are statistically significant at p < 0.001, Bonferroni corrected for multiple comparisons. During the cooperative behavior, the pattern of functional connectivity links is much denser than in the defect situation.

Figure 3 depicts the hyperlinks between cortical areas in the two couples of subjects performing the PD game. Such hyperlinks describe that the activity in the two brains is linked by a causal relation during the execution of the PD game.

The pattern of functional connectivity links during the cooperative behavior differs significantly from the defect situation. We analyzed the time interval between the presentation of the previous game’s results and the moment the subjects were cued to express their choice to cooperate or defect. During this time window, the subjects were evaluating the outcome of the previous round and making their choice for the next one. The cortical activity and connectivity patterns described in Figures 2 and 3 are thus related to the interval preceding the overt expression of the subjects’ decisions. It was demonstrated that, by extracting particular properties related to such hyperconnectivity links, it is possible to achieve an accurate prediction of the decisions to be expressed by the subjects, with an accuracy of 91 percent in the theta band.⁶ We can thus say that Figures 2 and 3 represent images of “social brains.”

When estimating cortical hyper-links during the card game scenario shown in Figure 1, we used two couples of subjects recorded simultaneously with four EEG systems to study the “spirit of the group” between partners in the game. Figure 1 shows the hyperlinks existing between different players of a team during such a paradigm. In Figure 1b, we represented, in front of each player, the realistic head model built from his MRI images. The colored areas on the reconstructed cerebral cortex indicate brain regions that showed similar activity. The existence of particular hyperlinks between members of the same team (which don’t exist between players of opposite teams) can be observed during the development of the different tasks’ phases.

Such neuroelectric hyperlinks do not mean that a signal is physically transmitted from one brain to another. Instead, they highlight which areas of the different brains are active in a related way during the common performance of the task. Interestingly, such functional hyperlinks, found between couples of partners in the game, are not present between players belonging to opposite teams, and can thus be considered to be an expression of the “spirit of the group.”

Toward Imaging Social Interactions

An analysis of statistical power spectra for the Prisoner’s Dilemma data suggests that during the decision to defect, the subjects elicited significantly increased cortical activity in the theta frequency band when compared to the cooperation conditions. We can hypothesize that this higher power spectra activity might reflect the major penalty and risky conditions implied in the generation of such decisions, when compared to the cooperation. This consideration is also supported by a strong activation of cortical areas that are compatible with neural circuits mediating negative and adversive emotional feelings, such as the insula and the orbitofrontal cortices.

We can also hypothesize that a large involvement of the frontal areas during the defect condition is generated by the effort of the decision system, which previous studies also located in the orbitofrontal regions. This is consistent with the interpretation of the orbitofrontal regions as the site in which decisions are made on the basis of rational and emotional conditions.¹⁴

The patterns of spectral cortical activity obtained from the 52 subjects during the PD game are presented here for the first time. The cortical connectivity patterns (hyperlinks) were presented in earlier research for a representative subject,⁶ but not for the entire population, as in this work. Taken together, these results illustrate the possibilities offered by EEG hyperscanning and estimation of the hyperlinks.

The functional connectivity results suggest a clear difference between the cortical patterns that occurred during the cooperation and the defect decisions. In particular, the pattern of intersubject connectivity in the cooperation condition is denser than in the defect case. The defect behavior, an individualistic one, eliciting a stronger cortical response produced a lower synchronization between brains, while on the contrary, the cooperation condition elicited a weaker activation but a denser synchronization between the two brains.

The results we present here suggest that the EEG hyperscanning address the analysis of brain functions, which can be used by any scientist interested in studying the neural processes that underline the emergence of social behaviors in subjects involved in real-life interactions. The hypermethods developed ad hoc for this kind of analysis can help to estimate synchronizations and causality relations between cortical areas of different subjects’ brains.

Future developments of this research will focus on ecological paradigms reproducing daily life situations and possible clinical applications of this methodology. A big challenge will be the necessity to give a physical interpretation of the correlations found between the activity in the brains of interacting subjects.

In many daily life situations, we often observe that components of a successful group are recognized to possess a shared feeling—a “spirit of cooperation” or “spirit of the group”—that characterizes and promotes their collaboration and the success of social interactions. Such a “spirit” might be investigated by looking at the patterns of functional connectivity between the neuroelectric activity in the members of the group during the performance of the cooperative task.

Biographies

Laura Astolfi is an appointed researcher in the Department of Computer Science and Systems at the Sapienza University of Rome and a research scientist at Fondazione Santa Lucia Hospital, Italy. Her research interests include functional connectivity, HR-EEG source reconstruction, simultaneous recordings and connectivity analysis from multiple subjects (hyperscanning), consciousness, cognition, and social neuroscience. Astolfi has a PhD in biomedical engineering from the University of Bologna. She is an associate editor for the International Journal of Bioelectromagnetism. Contact her at laura.astolfi@uniroma1.it.

Jlenia Toppi is a doctoral student in the School of Biomedical Engineering, Department of Computer Science and Systems at the Sapienza University of Rome and is collaborating with the Neuro-Imaging and BCI Laboratory at Fondazione Santa Lucia Hospital, Italy. Her research interests include processing EEG signals, the estimation of the cortical activity and connectivity in the human brain from noninvasive neuroelectric measurements, and the estimation of hyperconnectivity between cerebral areas of different brains. Toppi has a MS in biomedical engineering from the Sapienza University of Rome. Contact her at jlenia.toppi@uniroma1.it.

Fabrizio De Vico Fallani is a postdoctoral fellow at the Sapienza University of Rome and a research scientist at Fondazione Santa Lucia Hospital, Italy. His research interests are in neuroimaging, brain-computer interfaces, linear inverse solutions, and graph theory. De Vico Fallani has a PhD in biophysics from the Sapienza University of Rome. He is a member of the Italian society of biophysics (SIBPA) and IEEE. Contact him at fabrizio.devicofallani@uniroma1.it.

Giovanni Vecchiato is a doctoral fellow in the Department of Physiology and Pharmacology at the Sapienza University of Rome and in the Neuroimaging and BCI Laboratory at the Fondazione Santa Lucia Hospital, Italy. His research interests include investigating cognitive and emotional correlates of the EEG with observation of advertisements and real faces. Vecchiato has an MS in telecommunication engineering from the University “Federico II” of Naples, Italy. Contact him at giovanni.vecchiato@uniroma1.it.

Febo Cincotti is a research scientist in the Neurophysopathology Unit at the Fondazione Santa Lucia Hospital, Italy. His research interests include neuroelectrical inverse problem, brain–computer interfaces, and the development of algorithms for HR-EEG processing. Cincotti has a PhD in biophysics from the Sapienza University of Rome. Contact him at f.cincotti@hsantalucia.it.

Christopher T. Wilke is a resident physician in the Department of Radiation Oncology at the University of Minnesota. His research interests include functional brain mapping and image-guided therapy. Wilke has a PhD in biomedical engineering from the University of Minnesota, Minneapolis. Contact him at wilk0242@umn.edu.

Han Yuan is a postdoctoral associate at the Laureate Institute for Brain Research, Tulsa, Oklahoma. Her research interests include functional neuroimaging and brain–computer interfaces. Yuan has a PhD in biomedical engineering from the University of Minnesota, Minneapolis. Contact her at hyuan@laureateinstitute.org.

Donatella Mattia is the director of the Laboratory of Neuroelectrical Imaging and Brain-Computer Interface (NEILab) at Fondazione Santa Lucia Hospital, Italy. She promoted the development of clinical protocols for assessing the application of brain-computer interfaces and the development of the first BCI-operated assistive device. Mattia has a PhD in organization and pathology of movement from the Department of Neuroscience at the Sapienza University of Rome. Contact her at d.mattia@hsantalucia.it.

Serenella Salinari is a professor of modeling of biomedical systems in the Department of Systems and Informatics at the Sapienza University of Rome. Her research interests include mathematical modeling and analysis of biological data, mathematical models applied to metabolic systems, and the modeling and signal analysis of scalp-recorded potentials. Salinari has a PhD in engineering from the Sapienza University of Rome. Contact her at serenella. salinari@dis.uniroma1.it.

Bin He is a Distinguished McKnight University Professor, the director of Center for Neuroengineering, and the associate director for research of the Institute for Engineering in Medicine at the University of Minnesota. His major research interests include neuroengineering, functional biomedical imaging, and bioelectromagnetism. He has a PhD from the Tokyo Institute of Technology, Japan. He is a recipient of the NSF Career Award and American Heart Association Established Investigator Award, and he is a fellow of IEEE, the American Institute of Medical and Biological Engineering, the Institute of Physics, and the International Society for Functional Source Imaging. He is also the past president of the IEEE Engineering in Medicine and Biology Society. Contact him at binhe@umn.edu.

Fabio Babiloni is an associate professor of human physiology in the Faculty of Medicine at the Sapienza University of Rome. His research interests are in multimodal integration of EEG, MEG and fMRI data, cortical connectivity estimation, and brain-computer interfaces. Babiloni has a PhD in computational engineering from the Helsinki University of Technology. He is an associate editor of Frontiers in Neuroscience, International Journal of Bioelectromagnetism, IEEE Transactions on Neural System and Rehabilitation Engineering, and Computational Intelligence and Neuroscience. Contact him at fabio.babiloni@uniroma1.it.