Concurrent mapping of brain activation from multiple subjects during social interaction by hyperscanning: a mini-review

Abstract

Social interaction plays an essential role in acquiring knowledge and developing our own personalities in our daily life. Meanwhile, functional magnetic resonance imaging (fMRI)-, electroencephalograph (EEG)-, and functional near inferred spectroscopy (fNIRS)-hyperscanning, enables us to concurrently map brain activation from two or more participants who are engaged in social interaction simultaneously. In this review, we first highlight the recent technologies advances and the most significant findings towards social interaction by using the hyperscanning method. In addition, we also illustrate several well-designed hyperscanning tasks that have been extensively adopted for the study of social interaction. Basically, hyperscanning contains six categories of experimental paradigms that can track the interactive neural process of interest. Furthermore, it contains two main elucidated neural systems which are involved in social interaction, including the mirror neuron system (MNS) and mentalizing system (MS). Finally, future research directions and clinical implications that are associated with hyperscanning are also highlighted and discussed.

Introduction

Our communication or social interactions with one another is essential for us to acquire knowledge and to develop our own personalities. Although the forms of social interactions are very broad including imitations, exchanges, completions and cooperation, as well as making decisions (1), we basically exchange our thoughts and ideas in two different manners. The dominant manner is through our sophisticated languages, which is also a characteristic that distinguishing ourselves from other creatures (1,2). The other way is by using non-verbal signs, such as our gestures and facial expressions, which can provide us with additional auxiliary information for social interactions (1,2).

Interestingly enough, although our social nature has been shaped for hundreds and thousands of years, neuroscience studies only just began to shed light on social interaction in the recent years (1,3,4). More importantly, previous neuroimaging studies have exhibited two basic limitations in elucidating neural correlates of social interactions. The first restriction is from the low ecological validity, since most of the previous experiments were performed in an enclosed room, in which individuals were instructed by computer programs, to complete the test tasks (2). However, this is not the case for social interactions in real life, in which individuals need to talk and act with each other simultaneously in a more natural way. Therefore, further neuroscience or neuroimaging studies should be performed by using a more realistic experimental paradigm which can duplicate a real-life situation. The other limitation is that previous studies can only acquire brain data from a single participant each time (5). However, as two or more individuals are engaged in social interactions, it is essential to conduct a concurrent recording from multiple subjects with multiple setups rather than to perform it in isolation (1,2,5).

Recently, a new strategy that had combined two functional magnetic resonance imaging (fMRI) machines together for simultaneously measuring two participants’ brain activity was adopted, which was coined as “hyperscanning” method (6). Since then, extensive hyperscanning studies have been performed, which improves our understanding of brain-to-brain synchronization during a social interaction (7). To date, hyperscanning has enabled the inspection of social interaction by using various neuroimaging techniques such as electroencephalograph (EEG) (8-30), functional near inferred spectroscopy (fNIRS) (31-48) and fMRI (49-55). Meanwhile, the experimental paradigms involved in hyperscanning studies can be categorized into six types of tasks: (I) imitation tasks; (II) coordination/joint tasks; (III) eye contact/gaze tasks; (IV) economic games/exchanges; (V) cooperation and competition tasks; and (VI) interactions under natural scenario. In particular, it is noted that during the performance of all those tasks, two major neural systems are largely involved (1,2,5). One is the mirror neuron system (MNS), which plays an important role in tasks involving movements, such as imitation and coordination/joint tasks (56). The other is a mentalizing system (MS), which is engaged in tasks pertaining to the inferences of yourself or others’ intentions or thoughts (57), such as the economic game (58) and natural social interactions (33,36).

In this review, fMRI, EEG and fNIRS hyperscanning neuroimaging technologies which have engaged in social interaction are first introduced. Then, the representative experimental paradigms that were extensively adopted in hyperscanning, are also summarized in detail. Subsequently, two core neural systems involved in social interactions are carefully demonstrated. One is MNS, which consists of the primary motor, sensory cortex and parietal cortex, and is responsible for the imitation process; the second one is MS comprising the TPJ (temporal-parietal cortex) and PFC (prefrontal cortex), which is in charge of a more complex cognitive process. More importantly, the future of research perspectives and clinical implications of hyperscanning, are stated clearly in the final section.

Hyperscanning neuroimaging techniques

fMRI hyperscanning

As it is hard to place two or more participants into one fMRI tube, two or more fMRI machines should be utilized for an fMRI hyperscanning method to simultaneously record multiple participants’ brain signals. In that circumstance, two or more remote fMRI apparatus can be connected by an intranet, while the data sets are stored in a host client (Figure 1A) (6). To date, several fMRI hyperscanning studies (49-55) have been conducted to inspect the inter-brain synchrony (Table 1). For example, neural correlates of trust between two individuals, had been examined by fMRI hyperscanning. They had discovered that trust is an essential social process, involved in all human interaction (54). Inarguably, fMRI hyperscanning has exhibited its advantages in mapping the coherence of brain regions which were associated with social interaction with high structural accuracy and excellent imaging depth. However, it is not accessible and available for everyone, because of the high cost of multiple fMRI setups. More importantly, the ecological validity is also relatively low, since the lab was under the controlled circumstances for fMRI, and is significantly different from real life.

Figure 1.

Configurations of hyperscanning studies. (A) fMRI hyperscanning; (B) EEG hyperscanning; and (C) fNIRS hyperscanning. (A) was adapted from reference (52) with permission from John Wiley and Sons. (B) and (C) were adopted from reference (26) and (44), respectively, under a Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/). fMRI, functional magnetic resonance imaging; EEG, electroencephalograph; fNIRS, functional near inferred spectroscopy.

Table 1. fMRI hyperscanning.

Authors	Neuroimaging methods	Subjects	Paradigms	Main discoveries
Montague et al., 2002, Neuoimage	Two 1.5 T fMRI	3 pairs; gender: N/A; relationship: N/A	Game theory (a deceive game)	The very first hyperscanning study, where the term ‘hyperscanning’ was coined in this study
King-Casas et al., 2005, Science	Two 3 T fMRI	48 pairs; gender: N/A; relationship: strangers	Game theory (trust game)	The study extends previous model-based fMRI studies into the social domain and broaden our view of the spectrum of functions implemented by the dorsal striatum
Fliessbach et al., 2007, Science	One 3 T fMRI, one 1.5 T fMRI	19 pairs (5 subs were excluded); gender: all male; relationship: N/A	A simple estimation task that entailed monetary rewards for correct answers	A variation in the comparison subject’s payment affects BOLD responses in the ventral striatum
Krueger et al., 2007, PNAS	Two 3 T fMRI	22 pairs; gender: 11 F-F; 11 M-M; relationship: strangers	Game theory (trust game)	The paracingulate cortex is critically involved in building a trust relationship by inferring another person’s intentions to predict subsequent behavior. Conditional trust selectively activated the ventral tegmental area, a region linked to the evaluation of expected and realized reward, whereas unconditional trust selectively activated the septal area, a region linked to social attachment behavior
Stolk et al., 2014, PNAS	One 3 T fMRI, one 1.5 T fMRI	27 pairs; gender: all male; relationship: N/A	Cooperation task (jointly create a goal configuration of two geometrical tokens)	Establishing mutual understanding of novel signals synchronizes cerebral dynamics across communicators’ right temporal lobes
Spiegelhalder et al., 2014, BBR	Two 3 T fMRI	11 pairs; gender: all female; relationship: good friends	Natural scenario (live dialog)	The time course of neural activity in areas associated with speech production was coupled with the time course of neural activity in the interlocutor’s auditory cortex
Koike et al., 2016, Neuroimage	Two 3 T fMRI	Gender: same gender; relationship: stranger; Exp. 1: 17 pairs (9 M-M;8 F-F); Exp. 2: 15 pairs (8 M-M;7 F-F); Exp. 3: 16 pairs (6 M-M;10 F-F)	Eye contact/gaze tasks (mutual gaze task: eye to eye) (joint attention task: eyes on other stuff together)	The right inferior frontal gyrus had been activated both by initiating and responding to joint attention
Shaw et al., 2018, Sci Rep	Two 3 T fMRI	19 pairs; gender: all male; relationship: strangers	Game theory (ultimatum game)	Brain signals implicated in social decision making are modulated by the estimates of expected utility and become correlated more strongly between interacting players who reciprocate one another

Main discoveries were directly extracted or adapted from the article’s abstracts, which is also applied to the Tables 2,3. The situation of subjects in fMRI hyperscanning is two subjects lying in fMRI tubes separately. PNAS, Proceedings of the National Academy of Sciences of the United States of America; BBR, Behavioural Brain Research; N/A, not available means the authors did not explicitly depict their subjects’ relationship or the exact numbers of gender pairs even though some studies addressed the overall numbers of genders.

EEG hyperscanning

Since the electrical activity of human brain was firstly recorded by Hans Berger in 1924, EEG has become a core neuroimaging tool in the study of cognition and diseases (59,60). More importantly, EEG is also one of the most powerful techniques for noninvasively exploring neural oscillations (61), in which the EEG signals are originated from the synchronized synaptic activity in populations of cortical neurons (62). Although EEG has been extensively utilized for mapping single individual’s brain dynamics underlying specific cognitive tasks, the potential of EEG in exploring the inter-brain interactions or inter-brain connections has not been fully exploited.

Recently, a number of EEG hyperscanning studies (Table 2) were conducted (8-30), aiming to reveal the complex brain interactions between two or multiple participants, as illustrated in Figure 1B (26). These studies exhibited that EEG hyperscanning can map the moment-to-moment interactions between two or more individuals simultaneously, which can elucidate how co-variations of the tested individuals’ brain activations are correlated with their social interactions. However, despite EEG being suitable to inspect inter-brain synchronization due to its high time resolution, it is still very challenging for EEG to capture the neural activity from deep brain structures.

Table 2. EEG hyperscanning.

Authors	Neuroimaging methods	Subjects	Paradigms	Analytic method	Main discoveries/contributions
Tognoli et al., 2007, PNAS	Two 60-channel EEG	8 pairs; gender: 4 gender-mixed; 3 M-M; 1 F-F; relationship: N/A; situation: face to face	Coordination/joint tasks (self-paced rhythmic finger movements)	Power comparison	A pair of oscillatory components located above right centroparietal cortex distinguished effective from ineffective coordination: increase of phi1 favored independent behavior and increase of phi2 favored coordinated behavior. Phi (9.2–11.5 Hz)
Fallani et al., 2010, PLos One	Two 64-channel EEG	26 pairs; gender: N/A; relationship: N/A; situation: N/A	Game theory (prisoner’s dilemma game)	Partial directed coherence, graph theory	The hyper-brain networks of two defector couples have significantly less inter-brain links and overall higher modularity than couples playing cooperative or tit-for-tat strategies. The decision to defect can be ‘‘read’’ in advance by evaluating the changes of connectivity pattern in the hyper-brain network
Dumas et al., 2010, PLos One	Two 32-channel EEG	9 pairs; gender: 5 F-F; 6M-M (3 pairs were excluded, but not know which pair); relationship: N/A; situation: separated into two room	Imitation tasks (imitate counterparts’ hands movements)	Phase locking value (PLV)	States of interactional synchrony correlate with the emergence of an interbrain synchronizing network in the alpha-mu band between the right centroparietal regions
Babiloni et al., 2011, Cortex	Four 30-channel EEG	One quartet (four men) of professional saxophonists; situation: side by side	Natural scenario (music performance)	Power comparison	During the resting state, dominant EEG power density values were observed at alpha band (8-12 Hz) in posterior cortex. During the music performance, alpha power density values decreased in amplitude in several cortical regions, whereas power density values enhanced within narrow high-frequency bands
Babiloni et al., 2012, Neuroimage	Four 30-channel EEG	Three quartets (12 men) of professional saxophonists; situation: side by side	Natural scenario (music performance)	Power comparison	The higher the empathy quotient test score, the higher the alpha desynchronization in right BA 44/45 during the OBSERVATION referenced to RESTING condition
Naeem et al., 2012, Neuroimage	Two 60-channel EEG	6 pairs; gender: 3 mixed; 2 M-M; 1 F-F; relationship: N/A; situation: face to face	Coordination/joint tasks (adopted from Tognoli et al., 2007)	Power comparison; PLV	Clear and systematic modulation of mu band activity in the 10–12 Hz range as a function of coordination context
Yun et al., 2012, Sci Rep	Two 128-channel EEG	10 pairs; gender: all male; relationship: N/A; situation: face to face	Coordination/joint tasks (hand movement task)	PLV; source localization	Synchrony of both fingertip movement and neural activity between the two participants increased after cooperative interaction
Konvalinka et al., 2014, Neuroimage	Two 32-channel EEG	9 pairs; gender: N/A; relationship: N/A; situation: back to back	Coordination/joint tasks (a synchronized finger-tapping task)	Power comparison Multivariate classification analysis	The interactive condition was characterized by a stronger suppression of alpha and low-beta oscillations over motor and frontal areas in contrast to the non-interactive computer condition. Leaders invest more resources in prospective planning and control
Menoret et al., 2014, Neuropsychologia	Two 32-channel EEG	20 pairs; gender: 6 mixed, 7 M-M, 7 F-F; relationship: N/A; situation: face to face	Coordination/joint tasks (complete a goal with human or robot)		Acting in a social context induced analogous modulations of motor and sensorimotor regions in observer and actor
Toppi et al., 2016, PLos One	Two 16-channel EEG	6 pairs civil pilots; gender: 5 M-M; 1 M-F; relationship: all from the national Italian airline (Alitalia)	Cooperation task (a simulated flight)	Event related potentials (ERPs), power comparison	During the most cooperative flight phases pilots showed, in fact, dense patterns of interbrain connectivity, mainly linking frontal and parietal brain areas. On the contrary, the amount of interbrain connections went close to zero in the non-cooperative phase
Mu et al., 2016, SCAN	Two 32-channel EEG	Exp. 1: 34 pairs; gender: 17 M-M; 17 F-F; relationship: stranger; Exp. 2: 30 pairs; gender: all male; relationship: stranger; situation: separated by two monitors (Figure 1B)	Coordination/joint task (a dyad to synchronize with a partner by counting in mind rhythmically)	PLV	First evidence that oxytocin enhances inter-brain synchrony in male adults to facilitate social coordination
Mu et al., 2017, SCAN	Two 32-channel EEG	Exp. 2: 45 pairs; gender: same gender; relationship: N/A; situation: separated by two monitors (Figure 1B)	Coordination task (same as Mu et al., 2016)	PLV	Interbrain synchrony of gamma band oscillations is enhanced when people are under high threat, and increased gamma interbrain synchrony is associated with lower dyadic interpersonal time lag (i.e., higher coordination)
Jahng et al., 2017, Neuroimage	Two 64-channel EEG	10 pairs; gender: all male; relationship: stranger; situation: face to face; face-blocked	Game theory (prisoner’s dilemma game)	Power comparison, PLV	The power of the alpha frequency band (8–13 Hz) in the right temporoparietal region immediately after seeing a round outcome significantly differed between face-to-face and face-blocked conditions and predicted whether an individual would adopt a ‘cooperation’ or ‘defection’ strategy
Szymanski et al., 2017, Neuroimage	Two 64-channel EEG	25 pairs; gender: 12 M-M; 13 F-F; relationship: stranger; situation: side by side	Cooperation game (a visual search task)	PLV	The inter-team differences in behavioral performance gain in the visual search task were reliably associated with inter-team differences in local and inter-brain phase synchronization
Dikker et al., 2017, Current Biology	Twelve 14-channel wireless EEG	A group [12] of high school students; gender: 9 F; 3 M; relationship: classmates; situation: sit as a circle	Natural scenario (taking class)	Spectral coherence	They find that students’ brainwaves are more in sync with each other when they are more engaged during class. Brain-to-brain synchrony is also reflective of how much students like the teacher and each other
Kinreich et al., 2017, Sci Rep	Two 32-channel EEG	24 pairs romantic partners; 25 pairs strangers; situation: face to face with 45 degree	Natural scenario (talk with each other)	Power	Neural synchrony was found for couples, but not for strangers, localized to temporal-parietal structures and expressed in gamma rhythms
Perez et al., 2017, Sci Rep	Two 32-channel EEG	15 pairs; gender: 8 M-M;7 F-F; relationship: stranger; situation: side by side with a board	Natural scenario (talk with each other)	PLV	interpersonal synchronization is mediated in part by a lower-level sensory mechanism of speech-to-brain synchronization, but also by the interactive process that takes place in the situation per se
Leong et al., 2017, PNAS	Two 2-channel (C3 C4) EEG	Exp. 2: one adult (F) with 29 infants; gender: 29 infants (15 M; 14 F); relationship: N/A; situation: face to face	Natural scenario (infants viewed an adult in a live context, singing with direct or indirect gaze)	General partial directed coherence, G-causality	During live interactions, infants also influenced the adult more during direct than indirect gaze. Further, infants vocalized more frequently during live direct gaze, and individual infants who vocalized longer also elicited stronger synchronization from the adult
Hu et al., 2018, Biological Psychology	Two 64-channel EEG	15 pairs; gender: all female; relationship: stranger; situation: face to face	Game theory (prisoner’s dilemma game)	PLV	The results showed a higher cooperation rate and larger theta/alpha-band inter-brain synchrony in condition human-human (H-H) than in human-machine. In the condition H-H, there were larger centrofrontal theta band and centroparietal alpha-band inter-brain synchrony in tasks set for high cooperation
Ahn et al., 2018, Human Brain Mapping	Two 19-channel EEG with 146-channel MEG	5 pairs; gender: 1 Mixed; 4 M-M; relationship: stranger; situation: separated in two rooms communicating through cameras	Natural scenario (live dialog)	Power spectral density (PSD), weighted PLV	This hyperscanning study using simultaneous EEG/MEG is the first to identify the oscillations and interbrain phase synchronization involved in turn-taking verbal interactions
Kawasaki et al., 2018, Neuropsychologia	Two 27-channel EEG	17 pairs; gender: 8 F-F; 7 M-M; 2 mixed; relationship: 6 pairs were strangers and 11 pairs were acquaintances; situation: back to back	Coordination task (match their partners’ tapping intervals using visual feedback)	PLV	Alpha-(approximately 12 Hz) and beta-(approximately 20 Hz) amplitude modulation in the left motor areas
Ciaramidaro et al., 2018, Sci Rep	One 128-channel EEG was separated into two 64-channel EEG	21 pairs; gender: all male; relationship: N/A; situation: N/A	Third party punishment game	G-causality (partial directed coherence), graph theory	To their knowledge, this report is the first multiple-brain connectivity study to investigate empathic compassion and altruistic punishment
Goldstein et al., 2018, PNAS	One 64-channel EEG was separated into two 32-channel EEG	22 couples (4 were married); situation: side by side with face to face	Natural scenario (perceiving pain under touch/no-touch condition)	Circular correlation coefficients	Hand-holding during pain administration increases brain-to-brain coupling in a network that mainly involves the central regions of the pain target and the right hemisphere of the pain observer. Moreover, brain-to-brain coupling in this network was found to correlate with analgesia magnitude and observer’s empathic accuracy

Main discoveries/contributions were directly extracted or adapted from the articles’ abstracts. PNAS, Proceedings of the National Academy of Sciences of the United States of America; SCAN, Social Cognitive and Affective Neuroscience; N/A, not available means the authors did not explicitly depict their subjects’ relationship or the exact number of gender pairs even though some studies addressed the overall numbers of genders.

fNIRS hyperscanning

fNIRS is also a noninvasive and affordable neuroimaging technique, which utilizes the near-infrared light to image brain activation, by measuring the concentration changes of oxyhemoglobin (HbO) and deoxyhemoglobin (HbR) (63-65). In addition, fNIRS has exhibited its unbeatable advantages in inspecting infants or children’s brain activation (66) since it is relatively more tolerant with movement artifacts. More importantly, fNIRS hyperscanning (Table 3) can ideally be applied to a natural scenario (33,35,36,39), as illustrated in Figure 1C. Although fNIRS has a better temporal resolution when compared to fMRI, it has the low spatial resolution and limited capability to detect deep brain structures.

Table 3. fNIRS hyperscanning.

Authors	Neuroimaging methods	Subjects	Paradigms	Analytic method	Main discoveries/contributions
Cui et al., 2012, Neuroimage	22-channel for each subject within one fNIRS system (frontal lobe)	11 pairs; gender: 8 mixed; 2 F-F; 1 M-M; relationship: 3 pairs were strangers and 8 pairs were acquaintances; situation: side by side	Cooperation and competition task	WTC	This work represents the first use of a single NIRS instrument for simultaneous measurements of brain activity in two people
Holper et al., 2012, Neuroimage	Two 4-channel wireless fNIRS sensors (premotor cortices)	8 pairs; gender: N/A (7 F; 9 M); relationship: N/A; situation: face to face	Coordination/joint tasks (a paced finger-tapping task)	WTC; GC	The signal of the model G-caused that of the imitator to a greater extent as compared to vice versa
Jiang et al., 2012, JN	20-channel for each subject within one fNIRS system (left frontal, temporal, and parietal cortices)	10 pairs; gender: 4 M-M; 6 F-F; relationship: acquaintance; situation: face to face; back to back	Natural scenario (live dialog)	WTC; Fisher linear discrimination analysis	Face-to-face communication, particularly dialog, has special neural features that other types of communication do not have and that the neural synchronization between partners may underlie successful face-to-face communication
Cheng et al., 2015, Human Brain Mapping	22-channel for each subject within one fNIRS system (frontal lobe)	45 pairs; gender: 16 mixed; 14 M-M; 15 F-F; relationship: stranger; situation: side by side	Cooperation and competition task (task from Cui et al., 2012)	WTC	Partners with opposite gender showed significant task-related cross-brain coherence in frontal region whereas the cooperation in same gender dyads was not associated with such synchronization
Jiang et al., 2015, PNAS	10-channel for each subject within one fNIRS system (left IFC and TPJ)	12 three-person groups; gender: 6 female groups; 6 male groups; relationship: stranger; situation of pairs: face to face	Natural scenario (group discussion)	WTC; GC; Fisher linear discrimination analysis	These results suggest that leaders emerge because they are able to say the right things at the right time
Liu et al., 2015, Brain & Cognition	19-channel for each subject within one fNIRS system (the bilateral frontoparietal)	10 pairs; gender: 7 M-M; 3 F-F; relationship: N/A; situation: side by side with no board	Cooperation and competition task (a turn-taking game)	Pearson correlation (time domain)	The competitor may actively trace the builder’s disk manipulation, leading to deeper mind-set synchronization in the competition condition, while the cooperator may passively follow the builder’s move, leading to shallower mind-set synchronization in the cooperation condition
Osaka et al., 2015, Frontiers in Psychology	34-channel for each subject within one fNIRS system (bilateral hemisphere)	29 pairs; gender: 17 M-M; 12 F-F; relationship: stranger; situation: face to face; separated by a board	Natural scenario (sing)	WTC	A significant increase in the neural synchronization of the left inferior frontal cortex compared with singing or humming alone
Tang et al., 2016, SCAN	19-channel for each subject within one fNIRS system (right dlPFC and TPJ)	97 pairs; gender: 52 F-F; 45 M-M; relationship: stranger; situation: face to face; separated by a board	Game theory (ultimatum game)	WTC	FNIRS results indicated increased interpersonal brain synchronizations during face-to-face interactions in rTPJ (but not in rDLPFC) with greater shared intentionality between partners
Nozawa et al., 2016, Neuroimage	Four 2-channel wireless fNIRS devices (frontopolar)	12 groups of four subjects; gender: 5 male groups; 4 female groups; 3 mixed group; relationship: 2 groups were strangers and 10 groups were acquaintances; situation: face to each other	Cooperation task (a modified Japanese cooperative word-chain game)	WTC	This study provides a prospective technical basis for future hyperscanning studies during daily communicative activities
Pan et al., 2016, Human Brain Mapping	22-channel for each subject within one fNIRS system (right frontoparietal region)	All male-female pairs; 17 lover pairs; 16 friend pairs; 16 stranger pairs; situation: pairs were separated by a board	Cooperation and competition task (task from Cui et al., 2012)	WTC; GC	Lover dyads demonstrated increased IBS in right superior frontal cortex. Lover dyads revealed stronger directional synchronization from females to males than from males to females
Liu et al., 2016, Frontiers in Human Neuroscience	19-channel for each subject within one fNIRS system (rPFC; rSTS)	9 pairs; gender: 5 mixed; 2 M-M; 2 F-F; relationship: stranger; situation: face to face	Natural scenario (playing the JengaTM)	WTC	BA9 may be particularly engaged when theory-of-mind (ToM) is required for cooperative social interaction
Baker et al., 2016, Sci Rep	19-channel for each subject within one fNIRS system (rPFC; r-temporal cortex)	111 pairs; gender: 34 mixed; 39 M-M; 38 F-F; relationship: stranger; situation: side by side	Cooperation task (task from Cui et al., 2012)	WTC	Female/female dyad’s exhibited significant inter-brain coherence within the right temporal cortex, while significant coherence in male/male dyads occurred in the right inferior prefrontal cortex
Hirsch et al., 2017, Neuroimage	42-channel for each subject within one fNIRS system with eye-tracking (bilateral hemisphere)	19 pairs; gender: 10 mixed; 6 F-F; 3M-M; relationship: participants were either strangers prior to the experiment or casually acquainted as classmates; situation: face to face	Eye contact/gaze tasks (eye to eye contact)	WTC; PPI	A left frontal, temporal, and parietal long-range network mediates neural responses during eye-to-eye contact between dyads
Zhang et al., 2017, Sci Rep	19-channel for each subject within one fNIRS system (frontal and left temporal cortices)	30 pairs; gender: 13 M-M; 17 F-F; relationship: stranger; situation: face to face	Natural scenario (a card game)	WTC; GC	This study was the first to investigate such inter-brain correlates of deception in real face-to-face interactions
Liu et al., 2017, Sci Rep	48-channel for each subject within one fNIRS system (bilateral hemisphere)	22 pairs; gender: all male; relationship: level of friendship was assessed by using a self-report questionnaire; situation: side by side	Cooperation and competition task (a turn-taking game, same as Liu et al., 2015)	linear regression analysis (time domain)	The right pSTS may be commonly involved in both cooperation and competition tasks while the right IPL may be more important for competition task
Xue et al., 2018, Neuroimage	46-channel for each subject within one fNIRS system (prefrontal cortex and rTPJ)	30 pairs; gender: N/A; relationship: stranger; situation of pairs: face to face	Natural scenario (solve a realistic presented problem)	WTC	When two less-creative individuals worked on a creativity problem together, they tended to cooperate with each other (indicated by both behaviour index and increased IBS at rDLPFC and rTPJ), which benefited their creative performance
Reindl et al., 2018, Neuroimage	22-channel for each subject within one fNIRS system (prefrontal cortex)	30 pairs; gender: 13 mother-daughter pairs; 17 mother-son pairs; 1 father-daughter pair; 2 father-son pairs; relationship: parents with their own kids; situation: side by side; situation of pairs: side by side with no board	Cooperation and competition task (adopted from Cui et al., 2012)	WTC	Brain-to-brain synchrony may represent an underlying neural mechanism of the emotional connection between parent and child, which is linked to the child’s development of adaptive emotion regulation
Dai et al., 2018, Nature Communications	11-channel for each subject within one fNIRS system (left frontal, temporal, and parietal cortices)	21 groups of three subjects; gender: 11 male groups; 10 female groups; relationship: stranger; situation: face to face; back to back	Natural scenario (group discussion)	WTC	Selectively enhanced interpersonal neural synchronization (INS) between the listener and the attended speaker at left temporal–parietal junction, compared with that between the listener and the unattended speaker across different multi-speaker situations

Main discoveries/contributions were directly extracted or adapted from the articles’ abstracts. PNAS, Proceedings of the National Academy of Sciences of the United States of America; SCAN, Social Cognitive and Affective Neuroscience; N/A, not available, meaning the authors did not explicitly depict their subjects’ relationship or the exact numbers of gender pairs even though some studies addressed the overall numbers of genders; WTC, wavelet transform coherence; GC, granger causality; PPI, psychophysiological interaction.

Hyperscanning paradigms adopted in social interaction

Altogether, there were about six categories of experimental paradigms that were routinely used by the hyperscanning method in the investigation of social interaction.

Imitation tasks

The first category is the imitation tasks, during which one participant imitates the others’ movements or behaviors. Although we cannot request one participant to teach the others how to perform specific tasks in the laboratory, we can still instruct one participant to simulate the other individual’s actions or behaviors (Figure 2A). For example, in one EEG hyperscanning study (17), the participant was instructed to imitate the counterpart’s meaningless hand movements. The results showed that inter-brain synchronization of right centroparietal regions at alpha-mu band was strongly correlated with the interactional synchrony (Figure 2B).

Figure 2.

An example study of imitation tasks. (A) Schematic of the imitation task. One participant imitated the second one’s movements through cameras. (B) EEG hyperscanning results based on the imitation task. Inter-brain synchronization of the right centroparietal regions at alpha-mu band was associated with the interactional synchrony. (A) and (B) were adopted from reference (17) under the terms of the Creative Commons Attribution License. EEG, electroencephalograph.

Coordination tasks

The second category is the coordination tasks, in which two or more participants need to try their best to act in a synchronized manner. Interestingly, behavioral synchronization in our daily life is one mechanism through which we coordinate our behaviors during social interaction. For example, when we are walking together, our footsteps might be unconsciously synchronized with one another even though our foot lengths and our intrinsic cycles are totally different (22). In addition, coordination/joint movements can also be synchronized, such as self-paced rhythmic finger movements (15,16). In particular, a number of EEG or fNIRS hyperscanning studies have been performed to examine the neural synchronizations in coordination/joint movements (9,15,16,20,22,24,29). One example was illustrated in the previous reports (20,24), in which dyads were instructed to synchronize with each other by counting in their mind rhythmically. This study also examined how the social context such as threats, or oxytocin, affected the coordinated movements (20,24), which showed that oxytocin can enhance inter-brain synchronization to facilitate social coordination (20).

Eye contact/gaze tasks

The third category was the eye contact or gaze tasks, in which dyads are instructed to look in each other’s eyes, or look towards the third object. Interestingly, the mutual gaze or eye-to-eye contact has the functions that offer pivotal social cues in social interaction and communication. In particular, a universally recognized social link or a pipeline can be established during a non-verbal communication through eye contact or a mutual gaze (35). Importantly, we can infer the others’ intentions as well using eye-to-eye contact (50). Further, eye-to-eye contact, through which reciprocal information between individuals are dynamically exchanged, provides a great opportunity to model the neural mechanisms of human interpersonal communication by hyperscanning neuroimaging techniques (35,50). For example, an interesting study was performed, in which the dyads were instructed to look at each other’s eyes or eyes in portraits (35). And they discovered that the inter-brain coherence of the left superior temporal, middle temporal and supramarginal gyri as well as the pre- and supplementary motor cortices were significantly increased in the eye-to-eye contact case when compared to the data from the eye-to-picture gaze case (35).

Economic games involving game theory/exchange tasks

The fourth category is playing economic games/exchange tasks, in which one participant provided an economic offer while the counterpart need to make a decision on whether they wanted to take it or not. Game theory can offer a rich collection of both behavioral tasks and well-specified models aiming to articulate social interactions where decision-makers have to interact with one another (58). By contrast, exchange is the most basic type of social interaction, which involves a social process whereby social behavior is exchanged for some type of reward for equal or greater value. One instantiation of game theory/exchange is the trust game, in which one participant need decide how much money should be returned to your opponent (52), as illustrated in Figure 3A. One hyperscanning study illustrated that the paracingulate cortex is critically involved in building a trustworthy relationship (54). In addition, the prisoner’s dilemma game was also utilized as a task for the design of hyperscanning. This required the two participants to make their own decisions simultaneously (Figure 3B). The prisoner’s dilemma game usually consists of three experimental conditions: win-win, lose-lose, and a tit for tat case (14,21,27). Interestingly, previous reports have demonstrated that the decision to defect can be decoded in advance by monitoring the changes of connectivity patterns, as shown in Figure 3C (21). Further, the ultimatum game is also applied to the paradigm design for hyperscanning (Figure 3D), in which one participant need decide to take your opponent’s offer or not (34,55).

Figure 3.

Procedures of three economic tasks involving game theory. (A) Schematic of the trust game. Two participants are denoted as “investor” or “trustee”. The investor is assigned with amount of money ($20) and then decided how much to give to the trustee as an investment. After the decision made by the investor with amount of money ($14), the investment income would be tripled ($42). At this time, the trustee needs decide how much to be returned to the investor ($13). The results would be that the investor and trustee get $19 and $29, respectively (52). (B) Schematic of prisoner’s dilemma game. Win-win condition denotes that the two participants trust each other and they both win the rewards. Lose-lose condition represents that the two individuals deceive each other, and they both lose the money. Tit for tat condition denotes that if your partner deceives you, you might do the same in the next round as a counterattack. (C) The brain synchronization at alpha band under different conditions (21). (D) Schematic of the ultimatum game. Two participants were randomly assigned to a ‘proposer’ who gave the offer or ‘responder’ who decided whether to accept the offer or not. (A) was adapted from (52) with permission from The American Association for the Advancement of Science. (C) was adapted from (21) under the terms of the Creative Commons Attribution License.

Cooperation and competition tasks

The fifth category is cooperation and competition tasks, in which participants need to achieve a goal cooperatively or competitively. Cooperation and competition tasks are ubiquitous, in which goals should be obtained efficiently. One representative paradigm used in the hyperscanning studies was to explore the brain synchronization’s underlying cooperation or competition, as plotted in Figure 4A, which consisted of three conditions: the cooperate, competitive, and control conditions (31). Interestingly, this paradigm was first initiated in 2012, and later was utilized to examine the brain coherence differences between groups of the same sex and of mixed sexes (Figure 4B) (40,44), groups with lovers and strangers (Figure 4C) (42), or groups with parents verse the child and the stranger verse the child (38). In addition, other paradigm designs were also formulated to inspect the inter-brain synchronization engaged in cooperation and competition (13,32,41,46).

Figure 4.

An instantiation of cooperation and competition tasks. (A) Schematic of the cooperation and competition tasks. In cooperate condition, both participants needed press a button as soon as possible after seeing blue circles. If their respond time difference was smaller than the threshold, both of them got the rewards. However, if the difference was larger than the threshold, they should get nothing. In competition condition, after seeing the blue circle, the one who responded faster won the game. In control condition, one participant reacted to blue circles and the other one just watched it (42). (B) Inter-brain coherence underlying the cooperation condition for different gender groups. F-F represented female-female, M-M denoted male-male, and F-M denoted female-male. (C) Inter-brain synchronization underlying cooperation condition associated with different relationships. (A-C) were adapted from (40,42) with permission from John Wiley and Sons.

Natural scenario

The paradigms mentioned above do offer great opportunities in inspecting the inter-brain dynamics during social interaction. However, only social interaction through a natural scenario can reflect the real situations in our daily life, which is also the dominated way of communication and thought exchange. An interesting test has been performed, in which two participants were instructed to have conversations with each other while their neural data was concurrently recorded (25,26,28,33,36,39,51). Intriguingly, their findings showed that inter-brain synchronization was higher for a face-to-face talk case, as compared to that of the back-to-back talk case (39). In addition, neural synchronization under other circumstances was also explored, such as music playing (18,19), singing together (43), playing games (47,48) or taking a class (10). For example, one study showed that during class time, students’ brainwaves are more in sync with each other while they are highly engaged in the teaching (10).

Neural systems involved in hyperscanning during social interaction

Two main neural systems are involved in inter-brain connections (1,2,5). One is the MNS, which includes the primary motor cortex and posterior parietal cortex. The second one is the MS, which consists of the temporal-parietal junction (TPJ), precuneus and prefrontal cortex (PFC).

MNS

When we imitate or even just see the others’ actions or movements, neurons in the MNS are fired. This phenomenon was discovered in both monkey and human brains (56). In human brains, the MNS (Figure 5) consists of the inferior frontal gyrus (IFG) and inferior parietal lobule (IPL), which is related to language, motor and sensory detection. In addition, the superior temporal gyrus (STG) also plays an essential role in imitations, which can provide additional visual information inputs (56), in which the encoded information of imitated actions is first transformed into a more sophisticated visual representation through STG and then is delivered to the IPL. Once the IPL is activated, potential movements are able to be executed. In addition, the IFG is also activated to manipulate the potential action, which can provide additional supplemental information, such as the goal of the action.

Figure 5.

Two main brain systems involved in social interaction. This picture was adapted from reference (67) under a Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/).

The present hyperscanning studies associated with imitation show empirical evidence that MNS is involved in dual participant imitation (8,17). For example, one study demonstrated that when two participants were synchronized in behaviors, their brains were also tuned to the same frequency. Consequently, an inter-brain sychornizing network in the alpha-mu band between the right centroparietal regions was produced (Figure 2B).

MS

Besides imitating others’ actions, we might as well try to understand others’ intentions or emotions by their gestures, behaviors and facial expressions, which is termed as mentalizing (57,68). The TPJ and PFC particularly, and the dorsomedial PFC (DMPFC) are the two main brain regions associated with the mentalizing process (68).

The TPJ is the boundary brain region between the temporal and parietal cortex, which is labelled in a red circled area in Figure 5 (67). As depicted in a previous study (69), the mentalizing process contains two steps. In the first step, the static social images are coded as a neural representation from the extrastriate body area. For step two, the encoded representations are constructed to generate moving social entities, and are then incorporated into a context for interpreting the intention. Interestingly, several fNIRS hyperscanning studies have highlighted the TPJ as their region of interest (33,34,39). For example, in an adapted version of the ultimatum game, the interpersonal brain coherence for the right TPJ was higher for underlying the face-to-face condition, than that of the face blocked condition. This indicated the functions of right TPJ, is collaborative in social interactions (34).

The PFC, likes a commander, is also involved in the mentalizing processes. It is responsible for the planning, regulation, integrating of information, and other high cognitive functions. Accumulated neuroimaging evidences have shown that the PFC was related to interpersonal brain synchronization (31-34,39,41). For example, the inter-brain coherence in left inferior frontal cortex was significantly higher in face-to-face dialogues, than those from back-to-back dialogues, face-to-face monologue, or back-to-back monologues (39).

In summary, both MNS and MS play vital roles in social interactions, although the relationship between them is still unclear. A few studies demonstrated that they are collaborated (70), whereas additional reports also stated that MNS is inferior to MS (57).

Future perspectives and clinical implications of hyperscanning

Multimodality hyperscanning

Further investigation should be performed by using EEG-fNIRS, fNIRS-fMRI or EEG-fMRI hyperscanning techniques, since the multimodality neuroimaging methods can take advantage of the high temporal resolution of the EEG/fNIRS and the high spatial resolutions of an fMRI. To date, hyperscanning studies that utilize two or three neuroimaging modalities (e.g., EEG & fNIRS fusion) have not been extensively examined. Interestingly, multimodality can provide us new perspectives that a single modality cannot offer, because each neuroimaging method possesses its own advantages. For example, our group recently discovered that a combed EEG and fNIRS can enhance the sensitivity of lie detections (71). Although this is not a hyperscanning study, it enlightens us to more intriguing results or findings about the inter-brain dynamics which can be identified by applying a multimodality hyperscanning method for testing social interaction. In particular, more linked neural information can be revealed, based on the fused measures from neurovascular and neuroelectrical signals, which enable us to gain a more full understanding of the inter-brain effects during social interactions in our daily life.

Applications of hyperscanning in education and interrelationships

For most of the hyperscanning studies, neural data were recorded with two participants simultaneously, although several studies were also conducted by acquiring the brain signals from three or multiple participants (10,33,36). However, inspecting multiple individuals’ brain dynamics is crucial in some circumstances such as for the teaching and education settings. For example, one study demonstrated that students’ brain-to-brain group synchrony can track not only classroom engagement but also classroom social dynamics (10). But they did not explore the neural dynamics between teachers and students. In addition, the teaching style that can stimulate students’ inter-brain synchronization by inspecting the neural dynamics between teachers and students should be further investigated.

Interestingly, hyperscanning can also be applied to examining the interactions between an adult and a child (11,38) and interpersonal relationships, such as lovers (12,25,42). For example, lovers who held their hands together exhibited their capability in alleviating their pain perception (12).

Clinical implications

The hyperscanning method has exhibited a potential for the study of inter-brain synchronization of normal individuals during social interaction. In contrast, hyperscanning of abnormal individuals might manifest an aberrant, or null interpersonal dynamics for disorder detection, particularly those in social deficiencies such as autism and schizophrenia. For example, a previous hyperscanning study showed that autism patients have the ability in recognizing their counterparty’s intentions, but they cannot convey this information (72). To date, inspecting the interpersonal neural synchronizations among aberrant populations is still lacking. As a result, it is urgent for us to elucidate the neural mechanisms underlying those social deficits disorders by hyperscanning (1), which can pave a new avenue for improving the detection and treatment of neurological or psychiatric disorders.