Beyond synchrony: the capacity of fMRI hyperscanning for the study of human social interaction

Abstract

Hyperscanning—simultaneous brain scanning of two or more individuals—holds great promise in elucidating the neurobiological underpinnings of social cognitive functions. This article focuses on functional magnetic resonance imaging (fMRI) hyperscanning and identifies promising targets for studying the neuroscience of social interaction with fMRI hyperscanning. Specifically, we present applications of fMRI hyperscanning in the study of social interaction along with promising analysis approaches for fMRI hyperscanning, with its high spatial and low temporal resolution. We first review fMRI hyperscanning studies in social neuroscience and evaluate the premise of using this costly neuroimaging paradigm. Many second-person social neuroscience studies are possible without fMRI hyperscanning. However, certain fundamental aspects of social cognition in real-life social interactions, including different roles of interactors, shared intention emerging through interaction and history of interaction, can be addressed only with hyperscanning. We argue that these fundamental aspects have not often been investigated in fMRI hyperscanning studies. We then discuss the implication of the signal coupling found in fMRI hyperscanning and consider analysis approaches that make fair use of it. With fMRI hyperscanning, we can explore not only synchronous brain activations but whole-brain asymmetric activation patterns with a lagged association between interacting individuals.

Introduction

Social interactions are part of daily life, essential to survival and a key factor in human evolutionary success. Human brain development, higher-order cognitive functions and self-awareness evolved through our ability to engage in social interaction. Understanding the neurobiological processes that support social interactions may provide insight into what facilitates productive interactions and could inform the development and implementation of socioemotional interventions. Despite such needs, the neural processes underlying social interactions remain relatively unknown. Hyperscanning—simultaneous neuroimaging of two or more individuals—is a useful tool for studying the neural underpinnings of social interaction.

Hyperscanning is an important methodological advancement in our ability to study participation in interactive, social processes rather than passive observation of social stimuli (Schilbach et al., 2013; Hari et al., 2015; Redcay and Schilbach, 2019). Since the first hyperscanning study was reported by Montague et al. (2002), this technology has incorporated a variety of tasks and expanded to different imaging modalities such as electroencephalogram (EEG; e.g. Babiloni et al., 2007; Dumas et al., 2010; Astolfi et al., 2011), functional near-infrared spectroscopy (fNIRS; e.g. Cui et al., 2012; Baker et al., 2016; Reindl et al., 2018) and magnetoencephalography (e.g. Hirata et al., 2014; Hasegawa et al., 2016).

Hyperscanning has been mostly used for investigating interbrain synchrony, a temporally and spatially symmetric brain activation in two or more individuals across time (Hoehl et al., in press). However, limiting the focus to interbrain synchrony may result in missing other ways individuals’ brains might relate to one another during interactions. Hasson and Frith (2016) argue that social interactions are often characterized by asymmetric complementary actions, rather than mirroring or aligned ones. In such asymmetric actions, the brain-to-brain relationships might be temporally or spatially asynchronous, such that effects are delayed, or activity in a region of one person’s brain may be related to activity in a different brain region in the other person. Functional magnetic resonance imaging (fMRI) hyperscanning is particularly well suited to examine such effects, with its ability to measure activation in the whole brain with high spatial resolution.

This article aims to consider the premise of using fMRI hyperscanning in social neuroscience, discuss the advantages and limitations of fMRI hyperscanning for revealing interbrain systems of social interaction and present suitable experimental paradigms and possible analysis methods. We start with a brief review of findings from previous fMRI hyperscanning work in order to provide context for the current discussion. This review does not aim at complete coverage of the surging number of studies in this field; a more comprehensive recent review can be found in Redcay and Schilbach (2019). We instead focus on what types of study questions are best suited for fMRI hyperscanning experiments and the advantages and limitations of addressing such questions with fMRI hyperscanning. We then consider the implications of synchronization found in fMRI hyperscanning and its difference from EEG and behavioral synchronization analyses. Through these considerations, we propose that the true potential of fMRI hyperscanning is realized not only in analyses of interbrain synchrony but also by examining asymmetric brain activation with whole-brain mapping. We conclude by providing recommendations for future directions in the field of fMRI hyperscanning research.

fMRI hyperscanning in the study of human social interaction

A general goal of hyperscanning studies is to reveal neural underpinnings of social cognitive functions during real-time social interaction. Several experimental tasks have been used for creating real-time interactions while participants are being scanned. Here, to elucidate why fMRI hyperscanning is required for such studies, we review previous social interaction studies using fMRI hyperscanning and their analysis approaches.

One popular paradigm in fMRI hyperscanning is the cooperative or competitive game. Krill and Platek (2012) used fMRI hyperscanning to examine neural correlates of cooperation during a maze task. They used a general linear model (GLM) analysis to identify brain activation associated with each condition in the task and found activation in reward centers of the brain, namely the caudate and putamen. This demonstrates the benefit of using fMRI in identifying activation in subcortical areas that cannot be seen with EEG and fNIRS. However, a single-brain approach, in which only one participant is scanned while the other is outside the scanner (Redcay and Schilbach, 2019), might be able to elucidate the same activation found with the GLM analysis.

Shaw et al. (2018) investigated brain activation during a modified interactive ultimatum game, an iterated cooperative/competitive task with monetary reward. They identified brain regions activated according to the expected payoff and then calculated the correlation of signal time-courses in the identified regions between participants as a measure of interbrain synchrony. They found synchrony in regions including the right anterior insula and the anterior cingulate cortex, and greater synchrony in the anterior cingulate cortex was related to reciprocity (adapting to their partner’s behavior in order to achieve a desired outcome). Similarly, Abe et al. (2019) investigated an interbrain correlation in an online joint force-production task (i.e. participants tried to produce the same average grip strength with visual feedback from both their own and their partner’s grip). They identified individual-level activation in regions of the mentalizing system, including the medial prefrontal cortex, precuneus and bilateral posterior subdivision of the temporoparietal junction (TPJ). They found that between-participant correlation (cross-brain functional connectivity) of the right anterior TPJ was high during the joint task. These studies demonstrate the advantage of simultaneous recording of brain activation in identifying a signal association between brains that can emerge only in online reciprocal interaction.

Špiláková et al. (2019) examined dyadic social interactions during the pattern game, in which players cooperate or compete to build a target pattern with concurrent or alternate action. They introduced interpersonal brain-behavior dependencies (iBBDs) analysis, which models one player’s brain response to another player’s action using GLM. They reported greater activation in the medial prefrontal cortex, precuneus and temporal cortices in response to the other’s actions in the cooperative task as compared to the competitive one. While this task included an online reciprocal interaction, a sequential dual-brain approach, in which one participant’s brain is scanned at a time alternately during an online reciprocal interaction (Redcay and Schilbach, 2019), might also allow for the examination of iBBDs without hyperscanning.

The experimental paradigms of Špiláková et al. (2019), Shaw et al. (2018) and Abe et al. (2019) all involved dynamic, real-time interactions between participant dyads. However, if the measure of interest is a common response pattern occurring in multiple trials (as is the basis of the GLM analysis), a single-brain or sequential dual-brain approach could be used. In contrast, analyses focused on interbrain relationships in the context of a dynamic interaction typically require hyperscanning, as we discuss in more detail in the next section.

Another popular experimental paradigm in fMRI hyperscanning is the mutual gaze or joint attention task. In mutual gaze paradigms, participants are instructed to gaze into the other person’s eyes shown by a live video feed. In joint attention tasks, participants are instructed to follow the other person’s gaze direction as a simple form of information sending and receiving. While mutual gaze and joint attention tasks have been used in social neuroscience without hyperscanning (Redcay et al., 2010, 2012; Anders et al., 2011), hyperscanning allows the investigation of the correlated signal fluctuation between individuals during a real-time interaction. fMRI hyperscanning studies have primarily found interbrain synchronization in middle occipital gyrus during mutual gaze in same-sex, previously unacquainted dyads (Koike et al., 2016, 2019). Bilek et al. (2015) used cross-correlation analysis to identify concurrent or time-shifted correlations of brain activation between participants in a joint attention task with fMRI hyperscanning. They found a significant zero-lag correlation in the right TPJ. Activation in the right TPJ has previously been reported with a single-brain scan and joint attention task (Redcay et al., 2010), but cross-brain correlation in its spontaneous temporal fluctuation might be identified only with hyperscanning.

While most fMRI hyperscanning studies use correlation between the brain activation signals as a measure of synchrony, Goelman et al. (2019) employed a more sophisticated analysis for fMRI hyperscanning in a joint attention task. Their analysis, multivariate functional connectivity analysis, modeled multiple regions’ directed paths with lagged association measured by wavelet analysis. They indicated that initiation and responding to joint attention evolved with composite neural systems, including the dorsomedial prefrontal cortex, the ventromedial prefrontal cortex, the superior temporal sulcus, the TPJ and the posterior cingulate cortex. This result indicates that a single region-to-region synchronization analysis might be insufficient to describe the neural underpinnings of cognitive function in the joint attention task, and whole-brain mapping with fMRI can reveal a coordinated network.

These fMRI hyperscanning studies highlight its advantage in identifying temporal association in signal fluctuations between interacting individuals. However, we also note that, in most cases, similar results could have been achieved with alternate and less costly approaches. Indeed, hyperscanning is not the only scanning paradigm in social neuroscience. Redcay and Schilbach (2019) presented four types of scanning paradigms in social neuroscience: (i) the third-person single-brain approach, a conventional study paradigm that scans a single participant who is merely observing a social stimulus but not interacting with it; (ii) the second-person single-brain approach, which scans a single participant engaged in an interactive task, and the ‘partner’ in the interactive task may be outside the scanner or computerized responses; (iii) the second-person sequential dual-brain approach, which scans a single participant engaged in an interactive task and repeats this with the interaction partner to investigate the different brain activations for asymmetric roles of interaction (e.g. sender/receiver); and (iv) the second-person dual-brain approach, which employs hyperscanning to examine brain activations of two or more participants simultaneously during a reciprocal interaction task. Single-brain or sequential dual-brain approaches could have been used rather than fMRI hyperscanning in many of the studies described earlier.

These approaches have been used to address important social cognition questions, and hyperscanning is certainly not the only approach in the study of social interaction. In light of this, we consider the fundamental questions of social neuroscience that truly require hyperscanning by revisiting arguments raised by Montague et al. (2002), Przyrembel et al. (2012) and Schilbach et al. (2013) in the next section.

Study questions requiring fMRI hyperscanning

Montague et al. (2002) claimed that the study of social neuroscience requires simultaneous brain scanning of multiple persons during social interaction. The premise underlying this claim was that social cognition is different when we interact with others rather than observe them, and only real human-to-human interaction could illuminate this social cognition. Schilbach et al. (2013) also argued that social cognition under a reciprocal interaction is different from observation and applied the term ‘second-person neuroscience’ to an approach focused on cognition during engaged interaction. However, Schilbach et al. (2013) did not consider hyperscanning to be the only way to address this issue. While they praised the potential of hyperscanning, they also proposed several single-brain approaches that could work in second-person social neuroscience. Przyrembel et al. (2012) further divided the social interaction by the first-, second-, third-person perspectives, and online and offline interaction. They noted that while an online interaction demands a second-person perspective, the second-person perspective could also be utilized in an offline interaction. Thus, the second-person perspective and online/offline interaction are independent factors in social neuroscience.

Considering that hyperscanning is not the only method in second-person social neuroscience, when is it necessary to use this costly experimental paradigm? Human-to-human real-time reciprocal interaction is possible with the single-brain approach by placing one person in a scanner and another outside the scanner; thus, second-person social neuroscience is possible with the single-brain approach. Online reciprocal interaction is not sufficient to validate the use of hyperscanning. Indeed, Schilbach et al. (2013) proposed several single-brain approaches for second-person social neuroscience, including an adapted joint attention task.

Measuring brain-to-brain synchronization (correlation) app-ears to require hyperscanning. However, if the interaction time-course is highly-structured and fixed, brain-to-brain correlation can be observed without hyperscanning. Correlated brain activation between individuals engaged in the same time-course of stimulus has been reported with the single-brain paradigm. Dikker et al. (2014) indicated a correlated signal time-course between the brains of a listener and a speaker when seeing a common visual scene in the posterior superior temporal gyrus. Schippers et al. (2010) also showed a signal time-course association between a gesturer’s and a guesser’s brain activation in areas of the mirror neuron system using Granger causality mapping. These fMRI experiments were conducted without the use of hyperscanning. Thus, hyperscanning is not the only paradigm for a synchronization analysis.

Even when an interaction is asymmetric and we want to measure both participants’ brain activation during the interaction, a sequential dual-brain approach can be used if the interaction time-course is fixed or the quality of interaction does not change with repeating sessions. While hyperscanning still has a benefit in saving time by scanning multiple participants at once, it is not necessarily essential for this study paradigm. Thus, many second-person social neuroscience studies are possible with the single-brain approach.

According to the arguments by Schilbach et al. (2013) and Przyrembel et al. (2012), we argue that fMRI hyperscanning is needed only under the following conditions: (i) the study design includes online reciprocal interaction, (ii) the interaction sequence is unpredictable and evolves during the interaction, (iii) the correlation between the brain responses is transient and depends on the unpredictable interaction and (iv) it is difficult to reproduce the same interaction in repeated sessions (see Figure 1 for a visual depiction of these criteria). The critical point of the study paradigm that necessitates fMRI hyperscanning is the unpredictability of the interaction sequence, typical in social interactions. Przyrembel et al. (2012) claimed that a study paradigm is needed that demonstrates ‘a pattern of actions and reactions in which living and uncontrolled partners engage in behavior that leads to reciprocal impact on each other’s behavior’ to capture “real” social interaction. fMRI hyperscanning can address this issue, and only for addressing this issue is fMRI hyperscanning truly required.

Fig. 1.

Decision tree for determining suitability of fMRI hyperscanning based on the study design and research question. Third-person approaches refer to designs where the participant is observing a social stimulus and not interacting with it, while second-person approaches involve a social interaction. Single-brain designs include neuroimaging of one participant. Dual-brain designs scan both participants in a dyad. Note that this figure represents general guidelines, is not meant to cover every consideration or possibility and is only applicable to fMRI hyperscanning. Different considerations exist for other hyperscanning methods (e.g. EEG and fNIRS) based on their own advantages and disadvantages.

Characteristics of real social interaction often emerge as an asymmetric, complementary process. While a successful interaction may partly depend on aligned actions and shared cognitions between individuals, and brain-to-brain symmetric synchronization could illuminate such alignment, this is only one of many possible brain-to-brain associations (Hasson and Frith, 2016). fMRI hyperscanning can address the richer patterns of brain activation and coupling in multiple regions. In the next section, we review fMRI signals’ characteristics, with its advantages and limitations, to consider its best uses for hyperscanning in social neuroscience studies.

Characteristics of fMRI hyperscanning

A significant limitation of fMRI hyperscanning is the scanning environment. The interaction between individuals in separate MRI scanners is inevitably hindered as compared to direct in-person interactions that are possible with EEG and fNIRS hyperscanning. fMRI hyperscanning with a dual-head coil (Lee et al., 2012; Lee, 2015; Renvall et al., 2020) has been introduced to overcome this limitation. This approach has the advantage of allowing for physical touch between participants. However, we think that the fundamental aspect of social interaction that can be addressed with fMRI hyperscanning does not necessarily need direct physical interaction. For example, conversations in remote places using videoconferencing or audio-only interaction can be natural enough and meet the conditions requiring hyperscanning, as we discuss below. Thus, this limitation (lack of physical touch) would likely not be fatal for fMRI hyperscanning in a social interaction study, depending on the study question.

Another point to note about fMRI hyperscanning is the synchronization measure. Most fMRI hyperscanning studies use a correlation of the blood-oxygenation-level-dependent (BOLD) signals between brains. The functional implication of this synchronization in fMRI could be different than EEG and behavioral measurements. The temporal resolution of the BOLD signal is limited by the hemodynamic response, whose major temporal frequency resides in the very low range (<0.1 Hz). In contrast, EEG and physiological studies often measure a much higher frequency range and use the cyclic signal’s coherent phase association to measure synchronization (Hoehl et al., in press, SCAN).

Natural dynamic social interaction usually occurs in higher temporal frequency than the cyclic oscillation of the BOLD signal, and thus the synchronization or correlation in fMRI signals would not reflect the phase association of the cyclic oscillations. Instead, the correlation of BOLD signals could be due to the similarity of signal amplitudes for transient neural activation rather than signal-to-signal entrainment of cyclic signals. Indeed, the BOLD signals’ correlation could be driven by activity at only a few critical time points (Liu and Duyn, 2013). Thus, the correlation between the BOLD signals might implicate a transient activation of the common cognitive process. If the interacting individuals have a similar cognitive response or emotional state, such as an alignment of high-level cognition with a mentalizing or mirroring cognitive system (Hasson and Frith, 2016), their brain activation patterns could become similar.

Also, if such transient activation patterns drove the fMRI signal synchronization, its temporal association should not be limited to simultaneous occurrence. For example, considering an association of brain activations between a message sender and receiver in a conversation, this process inevitably induces a delay of a few seconds or more. The process could include compiling a message (building a sentence) in the sender’s brain; transferring the message (speech production, which could take several seconds depending on the length of the sentence); and comprehending the message in the receiver’s brain. The association of transient neural activations throughout this interaction could be seen in a lagged correlation. We must also be cautious when interpreting the concurrent correlation with a transient neural activation, as it could reflect a common external stimulus or environment regardless of the shared cognitive state.

While a search for temporal synchrony has heavily dominated current hyperscanning analyses (Redcay and Schilbach, 2019), we argue that fMRI is not ideal for elucidating a detailed timing association between neural activations of different brains. The hemodynamic signal change is slow, and its shape could be variable between individuals (Handwerker et al., 2004), which inhibits a comparison of exact activation timing between brains. The primary advantage of fMRI over other neuroimaging modalities is its high spatial resolution and whole-brain coverage in mapping functional activations. While the BOLD signal cannot elucidate the detailed temporal association like EEG can, with fMRI we can investigate the whole-brain social cognition system in mutual interaction. To make best use of this advantage, we propose that future studies with fMRI hyperscanning address the mapping of inter-individual brain associations between asymmetric brain regions, rather than a detailed temporal association in limited brain regions.

Study design and analysis methods for fMRI hyperscanning

Taken together with the conditions of a study paradigm requiring hyperscanning and the characteristics of the fMRI signal, we consider general issues to optimize study design and analysis approaches for future fMRI hyperscanning. The social interaction issues that can be addressed only with hyperscanning should include the conditions of (i) online reciprocal interaction, (ii) unpredictable and evolving interaction, (iii) transient cognitive (and cross-brain) coupling and (iv) quality of the interaction affected by history (irreproducible interaction). While there may be several social interaction situations meeting these conditions, a simple conversation is one of the most promising study paradigms for fMRI hyperscanning.

Conversation is possible in the fMRI hyperscanning environment using noise-canceling headphones and a microphone. Natural conversation is an online reciprocal process. How conversations progress is unpredictable and evolves during the interaction. Brain activation and interbrain associations during conversation are dynamic, and unscripted conversations are different every time. While the noise resulting from head motion due to speech production might limit fMRI signal quality, Xu et al. (2014) have demonstrated that an advanced noise reduction technique utilizing independent component analysis (ICA) can eliminate the significant motion effect on the signal during speech. The early study by Spiegelhalder et al. (2014) also demonstrated the feasibility of fMRI measurement during speech. Thus, motion-induced noise does not prohibit the use of conversation in fMRI hyperscanning. While the conversation is not the only credible study paradigm for fMRI hyperscanning, here we discuss this possible study design and analysis approaches with a conversation experiment, as an example to consider the concrete issues in the implementation of fMRI hyperscanning.

Highly structured interactions without freedom of response timing are not optimal targets for hyperscanning. The conversation should be a timing-free and spontaneous one. However, this spontaneity and lack of experimental control may not result in scientifically meaningful data. For example, if the interaction lacks context and flows too quickly, the functional implication of an asymmetric cross-brain association, even if it is identified, may not be obvious. To avoid such an unorganized interaction, minimal constraints such as limiting the context or topic of a conversation to a specific one in a block of a short period and repeating it several times might be useful for the fMRI experiment. With this constraint, we can evaluate the cross-brain association within a block and relate it to a specific interaction context.

Regarding the context of a conversation, Schilbach et al. (2013) suggested that for the second-person perspective and a fundamentally different social cognition to emerge in the interaction, emotional content needs to be included. We have identified a conflict discussion task as an example of such a study paradigm. Conflict discussions have been used in measuring behavioral and emotional characteristics of individuals (Melby and Conger, 2001) and have been implemented in multiple studies of parent–adolescent interactions (Hofer et al., 2013; Cui et al., 2015a, 2015b). The conflict discussion’s ideal characteristic is that it involves all critical aspects of fundamental social interaction raised by Schilbach et al. (2013). These aspects include (i) the emotional characteristics of the participants, (ii) different roles (parent and a child), (iii) possible shared intentions and motivations to solve the conflict, (iv) the opportunity to create a solution through discussion, and (v) the historicity of the relationship. While a conflict discussion is not the only possible study paradigm for fMRI hyperscanning, researchers should always consider what social interaction issues are being addressed and whether fMRI hyperscanning is required in a proposed social paradigm.

A possible analysis target of conversation hyperscanning is cross-brain coupling that emerges through interaction. A common way to explore such associations is functional connectivity analysis between the same regions in the two brains (a synchronization analysis) for fMRI hyperscanning. For the analysis of neuroimaging data with an unstructured task time-course, intersubject correlation (ISC) analysis (Nastase et al., 2019) is also possible. In ISC, the time series of corresponding voxels in the paired participants are correlated with each other. This analysis evaluates common brain activation with a similar response time-course across participants engaged in the same task. We, however, argue that these analyses are insufficient for conversation fMRI hyperscanning. Considering the different roles for the interactors, which is the fundamental aspect of real-life social interaction, measuring the correlation between the same brain region with similar timing might not elucidate an asymmetric social cognition. The analysis for conversation fMRI hyperscanning must address the associations between different brain regions with different temporal delays.

One such analysis method is Granger causality mapping between the brains (Schippers et al., 2010). Schippers et al. (2010) evaluated the Granger causality between the signal in a seed area in one brain and the signal in another brain. The analysis was repeated for all voxels in the other brain to make a map of Granger causality. The Granger causality analysis models a seed signal time-course with a linear combination of its own (autocorrelation) and another signal’s past values. While their study was a sequential dual-brain paradigm, this analysis can be used in fMRI hyperscanning as well.

Another approach to examining a time-shifted correlation is cross-correlation analysis between the brains. Bilek et al. (2015) used this analysis for their fMRI hyperscanning study with the joint attention task. They applied ICA to extract task-related components and evaluated cross-correlation for the component’s signal time-course between the brains. While they examined the independent component’s cross-correlation, this approach can also be used with seed-based mapping, as in Schippers et al. (2010). Both Granger causality analysis and cross-correlation analysis examine the time-shifted association between the signals. The difference is that Granger causality evaluates the effect of own and other’s signal effect of multiple time points in one model, while cross-correlation analysis calculates the covariance between respective time-shifted signals separately.

Goelman et al. (2019) introduced a sophisticated analysis method, multivariate functional connectivity analysis, for the fMRI hyperscanning study of a joint attention task. This analysis makes a model of directed associations between multiple regions with a lagged association measured by wavelet analysis. This approach can examine the multiple regions’ mutual lagged effects in one model. However, since the model complexity increases factorially with the number of modeled regions, this analysis is applicable only when the target areas are predefined with a specific hypothesis. Due to this limitation, Goelman et al. (2019) chose four regions at a time and explored the multiple quad sets of regions to extend the evaluation to more regions.

While an advanced synchronization or coherence analysis might also be applied, a synchronization analysis seeking a detailed temporal association is not well suited for fMRI hyperscanning due to its low temporal resolution. For example, in an fMRI hyperscanning study using a coherence spectral-density analysis (Stolk et al., 2014), the result indicated a coherence in very low frequency. The study found a significant zero-lag coherence between the brains in the right superior temporal gyrus at 0.01–0.04 Hz (25–100-s cycle). The coherence in this long cycle would not reflect a dynamic social interaction but a sustained state throughout the session. The study also found 7 s-lagged coherence between the brains in the left central sulcus at 0.05 Hz (20 s). This 7-s lag, however, matched to the task time-course. As 0.05 Hz is in a major frequency of the event-related hemodynamic response, this coherence should reflect a transient neural activation time-locked to a task event. These results suggest that the phase coherence in cyclic oscillations of the low-frequency BOLD signal has limited utility to elucidate the activation associations between brains in a dynamic and evolving social interaction.

Other advanced tools for measuring asymmetric coupled dynamics have been proposed, such as intersubject mutual information and transfer entropy analyses (Jantzen and Kelso, 2007; Kostrubiec and Kelso, 2014). Transfer entropy (Gourevitch and Eggermont, 2007) can measure the amount of information transferred from a sender to a receiver in an interaction. However, calculating high-order statistics like transfer entropy requires the assumption of a probabilistically stationary process and a large sample size for robust estimation. Stationarity could be violated in a dynamic interaction with emerging interaction like a discussion. The low temporal resolution of fMRI also limits the number of samples. Therefore, applying this method for whole-brain mapping analysis with fMRI hyperscanning might be difficult in natural social interactions.

We argue that the choice of analysis method should depend on the study research question and hypothesis. If there is a specific hypothesis about what regions should be included in the evaluation, the detailed modeling of their association with multivariate functional connectivity analysis (Goelman et al., 2019) is possible. However, to map a cross-brain association between many candidate areas or even in whole-brain voxels, Granger causality mapping or cross-correlation connectivity analysis is appropriate. While these analyses still need an a priori definition of the seed region in one brain, this limitation might be relaxed using a connectome-wide approach previously employed in single-brain analyses (Shehzad et al., 2014; Misaki et al., 2018). The connectome-wide approach uses multivariate regression to evaluate the association between the multivariate whole-brain connectivity pattern and multiple experimental variables (e.g. group, conditions, etc.) without a priori seed definition. While this analysis can be combined with any connectivity measure, cross-correlation analysis might be the best choice, as it can reduce the burden of heavy computation in this extensive analysis.

For lagged correlations, one must determine how long a time shift to evaluate in the analysis. Kuhlen et al. (2012) examined the synchronization between the speaker’s and listener’s brain activations during storytelling with an EEG experiment using a sequential dual-brain paradigm. They reported that canonical correlation between the topological patterns of the brains was maximum at 12.5 s delay from a speaker to a listener. This delay corresponded to the temporal range of the semantic unit of the story, suggesting that a shared semantic understanding might have driven the correlation. This result suggests that the timing shift of the cross-brain association depends on the context of the conversation. In a natural conversation, semantic content will be transferred with a shorter speech unit than storytelling, so that shorter lags than Kuhlen et al. (2012) may be examined (Schippers et al., 2010; Stephens et al., 2010).

Also, as indicated in Stolk et al. (2014), lagged associations may emerge due to an experimental setting regardless of interaction. A change of common external environment, such as a condition block transition and/or an alteration of roles, could induce correlated signals between the brains. The effect of such a common external event, when it is irrelevant to the mutual interaction, must be excluded when analyzing the correlated activity between brains. Also, one must be careful about the existence of a trivial environmental factor (e.g. scanner noises) that could drive the cross-brain correlation when interpreting the results.

We mentioned previously that a minimum constraint on interaction would allow for ease of interpretation. This constraint might be relaxed if combined with other methods that can help specify the context of the interactions and implication of the cross-brain correlation. For example, in a conversation, classifying each utterance with its emotional valence, evaluating the event-related response for each of them, and then the association between the cross-brain responses using a beta series correlation (Rissman et al., 2004) could possibly elucidate a cross-brain coupling specific to certain emotional content. The level of emotional valence can also be evaluated by heart rate, blood pressure, or other physiological indices within specified time periods. Combining these measures can help interpret cross-brain associations and may help relax the interaction constraint and improve the ecological validity of the interaction in the experiment.

Last, we note that the quality of the dyadic relationship and roles of the dyad members should be considered in the analysis. The interaction quality, such as emotional context and the history of interaction, which are the fundamental aspects of real-life social interaction, may be variable across participants and for each interaction. When researchers want to identify the effect of these properties, we need to include a measure of these qualities as an effect of interest in the group analysis. Connectome-wide association analysis (Shehzad et al., 2014; Misaki et al., 2018) can be used to examine the association of these factors with whole-brain cross-brain connectivity.

Possible analysis approaches are not limited to what we have reviewed here. For example, the dynamic functional connectivity analysis (Hutchison et al., 2013) may be useful to identify the transient association between the brains. Summarizing the cross-brain correlation with a higher-level abstract measure using a graph theory approach (Bullmore and Sporns, 2009) may also help elucidate the association between the individual character of interaction and brain-to-brain correlation. The possibilities are numerous, with many approaches likely yet to be developed, and we thus refrain from discussing these extensive possibilities here. We instead hope that this discussion has illuminated the abundant opportunities for fMRI hyperscanning that have not been well explored.

Conclusions

In conclusion, we reviewed the fundamental aspects of real-life social interaction, according to Schilbach et al. (2013) and Przyrembel et al. (2012), and proposed that conversation fMRI hyperscanning is a promising paradigm that can assess these aspects. Specifically, we introduced the conflict discussion as an example experimental paradigm and discussed its potential in addressing social cognition in real-life social interaction. Given the asymmetric complementary actions in natural social interaction and the characteristics of the BOLD signal, we proposed that fMRI hyperscanning should map the whole-brain asymmetric activation patterns with a lagged association between interacting individuals. We presented several possible analysis options for this approach. Although developing a new mathematical toolkit for elucidating complex asymmetric dynamic coupling is warranted (Hasson and Frith, 2016), employing the existing analysis methods developed for intrabrain connectivity is also promising for investigating unexplored fundamental aspects of social interaction.

It should be noted that fMRI hyperscanning is a complex and relatively costly experimental paradigm. However, it can provide a unique insight into the neural basis of real-life human interactions. It requires the concurrent collection of two fMRI datasets, orchestrated control of experimental procedures by synchronizing the task, scanning two or more individuals and using advanced analysis methods to investigate the two-in-one brain system. Many second-person social neuroscience questions can be addressed without fMRI hyperscanning. To validate the use of this costly neuroimaging paradigm, researchers should focus their efforts on specific social neuroscience targets and questions that can be addressed only with fMRI hyperscanning.

While we have presented considerations for deciding if fMRI hyperscanning is necessary, these are general guidelines and may not apply to all of the experimental paradigms that may exist or be developed. Certain situations may not fit the above criteria wherein fMRI hyperscanning may still be necessary and/or appropriate. For example, fMRI hyperscanning could save time and reduce participant burden as compared to a sequential design. It should also be noted that our considerations here are specific to fMRI hyperscanning. Other hyperscanning methods that are less costly and can be used for face-to-face interactions (e.g. EEG, fNIRS) have many advantages and may be employed for a broad range of study questions. Our purpose is to highlight the ideal use of fMRI hyperscanning given its associated advantages and limitations, rather than to say that other uses and paradigms lack empirical validity or do not provide meaningful contributions.

The strength of fMRI hyperscanning is its spatial resolution and whole-brain coverage. While the temporal synchronization between the brains of interacting individuals is an attractive target, we argue that researchers look not only for symmetric temporal synchrony but also spatial mapping of asynchronous activation coupling to elucidate a complex two-in-one system of social cognition. Considering the limited temporal resolution in fMRI, we propose investigating correlations between multiple brain regions with multiple temporal lags across brains, (i.e. cross-brain connectivity analysis) as promising for such mapping. One critical aspect of social interaction is the emotional context (Schilbach et al., 2013), and the brain regions implicated in emotion reactivity and emotion regulation are primarily found in the medial and limbic brain regions, which are inaccessible to all other hyperscanning modalities aside from fMRI. With an optimized study design qualifying the fundamental aspects of second-person social neuroscience and an analysis method addressing asymmetric cross-brain connectivity, fMRI hyperscanning has considerable potential to advance our understanding of previously untapped processes and to discover fundamental neural mechanisms of social interactions.

Funding

This work was supported by the National Institute for General Medical Sciences, National Institutes of Health [P20GM109097, P20GM121312]; Laureate Institute for Brain Research; and the William K. Warren Foundation.

Conflict of interest

None declared.