Attending to and neglecting people: bridging neuroscience, psychology and sociology

Abstract

Human behaviour is context-dependent—based on predictions and influenced by the environment and other people. We live in a dynamic world where both the social stimuli and their context are constantly changing. Similar dynamic, natural stimuli should, in the future, be increasingly used to study social brain functions, with parallel development of appropriate signal-analysis methods. Understanding dynamic neural processes also requires accurate time-sensitive characterization of the behaviour. To go beyond the traditional stimulus–response approaches, brain activity should be recorded simultaneously from two interacting subjects to reveal why human social interaction is critically different from just reacting to each other. This theme issue on Attending to and neglecting people contains original work and review papers on person perception and social interaction. The articles cover research from neuroscience, psychology, robotics, animal interaction research and microsociology. Some of the papers are co-authored by scientists who presented their own, independent views in the recent Attention and Performance XXVI conference but were brave enough to join forces with a colleague having a different background and views. In the future, information needs to converge across disciplines to provide us a more holistic view of human behaviour, its interactive nature, as well as the temporal dynamics of our social world.

1. Introduction

Other people form our most important environment and the stimuli that shape our brains, minds and behaviour throughout the lifespan. Recent advances in neuroimaging and brain-signal analysis have drastically increased the possibilities for studying the brain mechanisms of social interaction, even simultaneously from two persons engaged in natural communication. These studies have revealed that networks of brain areas support perception of self and others, interpretation of non-verbal and verbal social cues, mutual understanding, and social bonding (see reviews in e.g. [1–3]). However, to really understand the brain basis of social cognition and interaction, we should search for fresh approaches and dialogues between different disciplines studying human social behaviour. This issue aims at multidisciplinary integration of neuroscience, psychology, interactional sociology, and behavioural animal research to advance our understanding of social interaction and person perception.

Social stimuli, such as the faces and bodies of other persons, are dynamic and complex combinations of a multitude of physical features. However, our reactions to such stimuli go far beyond the sensory information given. We pay radically different amounts of attention to physically very similar humans and their actions in different contexts, depending on whether the persons are our significant ones, colleagues, dentists, bus drivers, shop keepers or refugees. Naturally, how we attend to and neglect other people, and thereby look, listen, align and synchronize with them during social encounters, strongly affects the information we receive from them, and how we consequently understand their intentions and behaviour.

While the knowledge of the neural underpinnings of social behaviour has rapidly accumulated, most experimental set-ups for behavioural and brain imaging work are still very far from real-life-like. Another apparent limitation in advancing understanding of human behaviour is the gap between disciplines that are interested in human behaviour, social interaction and their determinants. Social behaviour occurs at many different levels, from the processing of sensory social cues studied by cognitive neuroscientists, to large-scale culturally shared conventions and automatized rituals studied by sociologists and cultural anthropologists. Owing to this multilevel nature of social behaviour, a comprehensive picture of its brain basis cannot be composed without integrating different levels of analysis.

The articles in this theme issue derive from a recent multidisciplinary meeting on ‘Attending to and neglecting people’ (Attention and Performance XXVI). Several invited speakers agreed to write joint articles with other speakers with whom they had never collaborated before, thereby crossing borders of disciplines and views [4–7]. These and other papers in the issue discuss how we should study and conceptualize the neurocognitive mechanisms of person perception, social cognition and social interaction. A central question is the role of social interaction in cognition: does the brain have a specialized machinery for processing sensory cues from conspecifics (the classical view); or could the interaction with others constitute even the ‘default mode’ of human brain function that enables human social interaction (see the debate in [5]). In this special issue, research on person perception in healthy humans [8] is complemented by studies of autistic individuals [9,10] and patients with brain lesions [11], development of social brain functions in young infants [12], social learning strategies (SLSs) [13], synchrony between individuals in musical ensembles [14], analysis of different aspects of on-going musical therapy [15], human–robot interaction [16], and vocal turn-taking in marmoset monkeys [17]. Other articles discuss alignment of people beyond mirroring [7], new data-analysis techniques for quantifying neural and behavioural data related to social cognition [4], and the contributions of interactional sociology to human cognitive and social neuroscience, and vice versa [6].

The questions discussed in this issue might contribute to solving significant societal problems: How can we diminish the detrimental effects of loneliness throughout the lifespan [18]? How can new social technologies, such as interacting robots, be tailored to help to entertain and assist the lonely, or to help children to learn? Or how can scientific knowledge help to mitigate xenophobia, the irrational dislike of people from other cultures? Addressing these questions requires detailed understanding of the determinants of social interaction that shape the brain and mind during the whole lifetime.

2. Social perception and predictive coding

We constantly monitor numerous social cues in other people, such as identity, emotional state, trustworthiness, and intentions [19,20] to predict others' actions in the short and long term—an essential prerequisite for social behaviour. Coordinated activity of specific facial muscles, associated with different emotional states, signals about the individuals' feelings, and tiny facial cues, such as skin colour and adiposity, affect the perception of the health of the other person, which is critical when evaluating both the reproductive fitness of a potential mating partner and the possible risk of a contagious disease. In this theme issue, Henderson et al. [8] discuss the inferences people make about another person's health on the basis of facial cues that change at different timescales: symmetry and sexual dimorphism of the face alter slowly across the lifespan, facial adiposity changes over a medium time course, and skin colour can alter over a short time.

In human behaviour, previous experience and expectations affect the analysis and interpretation of new information. This dependence can be formulated in a Bayesian predictive-coding framework that has turned out to be widely useful for understanding of both behaviour and brain function. Interestingly, a somewhat related integrative theory of the functional systems that organize neural activity for predicted results of a future action was already introduced in the 1930s by Anohkin [21]. Predictions work at several levels of brain hierarchy: higher-level cortical areas generate predictions of forthcoming events for lower-level areas, and the error signals that inform about discrepancies between the expected and received sensory feedback are evaluated to adjust the model in the higher-level areas. Predictive coding has been used to explain both mirroring, [22] which we discuss below, and mental-state attributions [23]. Recent work suggests that the predictive-coding framework may also be extended to two-person interactions: an observer is modelling the behaviour of another person, who is reciprocally modelling the observer [24].

Several psychiatric disorders, such as psychosis and autism, can also be viewed as phenotypes involving false inferences or abnormal predictive coding about the world and other people [25], associated with aberrant—either too high or too low—precision of the corresponding representations [26]. In this theme issue, von der Lühe et al. [10] address the roots of abnormal interpersonal behaviour in individuals with high-functioning autism (HFA). By showing subjects point-light displays of two agents performing either communicative or non-communicative actions, the authors suggest that individuals with HFA are impaired in the implicit processing of social information, possibly reflecting their impaired capabilities for social predictive coding.

3. Contextual effects in social situations

All social interaction and perception is inherently contextualized. Most human-to-human interactions are so automatic that effortful mind reading—interpretation of another's beliefs and intentions—is not needed all the time; often just a glance of an eye is enough to find out that the other is still ‘tuned in’ [1,27]. Anthropologist Gregory Bateson used a metaphor of binocular vision to characterize social interaction: the two eyes receive slightly different information because they view the world from different positions, but the combined picture is better than that obtained by either eye alone. Accordingly, two heads are better than one only if the perspectives of the two persons complement each other, but still comprise enough overlap to improve mutual understanding.

One sign of the similarity across individuals in interpreting the physical and social aspects of the world is the similarity of people's brain functions in complex real-life-like situations. Neuroimaging studies have established that the more similar views of the external world two individuals have, the more alike their spatio-temporal brain-activation patterns are [28]. Accordingly, enhanced speaker–listener neural synchronization is associated with successful comprehension of the monologue [29], and non-verbal communication via hand gestures and facial expression enhances region-specific neural ‘coupling’ between the individuals [30,31].

Depending on a person's experiences, background, and goals, exactly the same physical stimuli can trigger very different processing chains: think for example of the meaning of a beautiful rare flower for a botanist and a poet. Hortensius et al. [11] demonstrate in this theme issue that lesions of basolateral amygdala (BLA) can compromise context-dependent social processing: in individuals suffering from focal BLA damage, neutral faces seen in a threatening context triggered increased activity in frontal and parietal cortices. However, BLA damage did not influence responses in the ventral face-processing regions or in the limbic parts of the emotion circuits. The result was interpreted as a failure to discriminate relevant and irrelevant threats, associated with increased reflexive reactions to threat. Context thus seems to have a strong bottom-up influence on brain processes, and projections of the limbic circuits (here amygdala) encoding the saliency or affective value of a stimulus significantly contribute to the contextual effects [11].

4. At which level should social interaction be studied?

Neuroscientists typically aim at very strict stimulus control and thus apply stimuli where all parameters can be exactly specified and varied. When moving from artificial to natural stimuli, one loses stimulus control but may benefit from strengthened brain responses. For example, videos showing real-world events trigger more reliable and robust neural responses than do artificial stimuli [32,33]. Such immersive stimuli also keep the subjects motivated and alert. Importantly, results obtained using naturalistic stimuli can be more readily generalized to real-world events, whereas statistical combinations of neural responses to simple stimuli do not necessarily predict the responses to complex stimuli [34].

Other humans are such special ‘stimuli’ that reducing them to photographs or line drawings may markedly alter the associated neural processes. Importantly, the mere presence or even assumed presence of other people changes the way the human brain processes sensory input. For example, early electrophysiological responses to faces are larger when the observers see the face of a real person, rather than the face of a dummy [35]. Similarly, interaction with ‘live’ rather than recorded persons elicits stronger brain responses [36]. These findings highlight the importance of co-presence with other people, and the consequent changes in the way the brain processes both internal and external cues. However, whether the other persons trigger our genuine interest and attention or whether we neglect them as much as possible as long as we do not directly bump into them typically depends on contextual and background information going beyond the persons' physical properties during the encounter.

5. Data-driven analysis in social neuroscience

Social cognition and interaction can be successfully studied at different levels of realism. Many social processes span over several overlapping and hierarchical time intervals, and real-life social interaction is inherently high-dimensional. Thus, it may be difficult to apply hypothesis-driven experimentation and data analyses. Instead, recent advances in data-driven signal analysis, and especially in machine learning and the associated classification approaches, have provided new tools for capturing the high-dimensional stimulus spaces that arise in such naturalistic experiments. The case-study examples by Adolphs et al. [4] in this theme issue nicely illustrate how these data-driven techniques can be used for dimension reduction in many complex social-perception and interaction tasks. Importantly, the organization of the resultant dimensions may inform about the structures of the applied stimuli, and these techniques can also be used for inferring the optimal stimulation parameters for activation of certain cortical regions. These data-driven techniques, including e.g. principal component analysis and independent component analysis, provide powerful means for generating new, focused hypotheses that can be tested in subsequent tailored experiments.

6. Mirroring and beyond

Social interaction is expected and needed at all stages of human life. For example, in ‘still-face experiments’ the infant becomes anxious soon after the mother freezes her face in front of the infant. Adults automatically align their actions and feelings with the partner [37–40], but the neurobehavioural studies of interaction are few because of a lack of conceptual and experimental frameworks.

Currently, one of the best-understood mechanisms of interaction, in addition to speech and conversation that we discuss later, is mirroring. In its basic form, mirroring refers to responses of motor mirror neurons to seen actions, first discovered in monkey premotor area F5 [41]. Mirroring is thought to trigger in the viewer similar, although weaker, motor-system activity as in the performer of the action. Although simultaneous inhibition prevents inappropriate automatic imitation [42,43], action observation may be beneficial in facilitating imitation of actions, such as knitting, that are difficult to explain verbally, but can easily be performed while viewing another person doing them. Importantly, however, the mirroring systems—which extend far beyond the motor mirror neurons in the prefrontal cortex—are under top-down control, so that viewing another person's actions can lead to complementary actions and coordination mechanism, such as in playing tennis or playing in a musical ensemble [14,44]

Along similar lines, Hasson & Frith [7] argue in this theme issue that successful dyadic interaction goes far beyond mirroring. When people interact, mirroring each other is useful, but the participants actually have to adapt to each other's behaviour if they want to communicate and to exchange abstract ideas. Because alignment of the interaction partners at multiple physical and mental levels acts as a crucial precursor for successful social interaction, Hasson & Frith [7] suggest that the dyad should be treated as a dynamically coupled system. They consider the observed similarity of brain activity in early sensory areas of viewers of a film [45,46] a sign of low-level alignment; whereas shared neural signatures at higher-order brain areas are related to more conceptual issues, such as meaning, context, and rewards. The challenge is to develop experimental set-ups and advanced signal-analysis methods to characterize active exchange of ideas and concepts between two interactive persons.

Mirroring—which demonstrates that motor functions are at the core of human social cognition—may provide the elementary means for social learning. Heyes [13] discusses in this issue two different SLSs that enable humans, non-human animals, and artificial agents to make adaptive decisions about when and whom they should copy when trying to cope with an unfamiliar situation. In an evolutionarily sense, copying others seems to be the best strategy to survive whenever one is in doubt what to do. Most SLSs are widely present (‘planetary’) in the animal kingdom, but flexible (‘cook-like’) SLSs are found only in humans [13]. The latter SLSs depend on explicit, metacognitive rules and allow copying certain persons or person groups whom the agent knows to be skilful in the tasks of interest, even if they are younger and/or in other respects less experienced (such as for example digital natives). These metacognitive SLSs contribute to cultural evolution as they foster the development of processes that enhance the exclusivity, specificity, and accuracy of social learning [13].

Adult humans smoothly adjust their actions, taking into account the state of their interaction partner, e.g. by effortlessly passing a cup to the partner's free rather than their occupied hand. In their contribution, Mayer et al. [12] show that during the first years of life, children gradually become more engaged in joint actions, but that they start to take their partner into account in their action plans only at 3.5 years of age. Even at age 5, children still show minimal adjustments to their action partner. The child's action planning capabilities develop at the same pace as the child's cognitive flexibility and inhibitory control increase. Coordinated joint actions have the potential of increasing social bonds (see also [47]), and more generally, both mutual mimicry and engaging in joint actions makes humans feel more connected [37,38].

Accumulating evidence shows that over-learned own movement patterns are important for understanding the actions of others. In this issue, Cook [9] reviews literature showing that movement kinematics is significantly altered in autistic individuals, who consequently have difficulties in understanding the actions of healthy individuals. Importantly, the misunderstanding of actions is reciprocal, as the neurotypical subjects do not understand well the movements of autistic individuals. Such mutual disability may compromise smooth social communication. Importantly, lack of synchronous movements and reciprocal imitation may also lead to diminished bonding and mutual liking. Another factor hampering the smooth social interaction of autistic individuals is the increased saliency of low-level visual features, so that irrelevant features effectively capture the attention. At the same time, the subjects pay less attention to social stimuli, such as faces [26].

7. Behavioural synchrony in dyads and ensembles

People march, dance, play, sing and express emotions together. Such synchronous collective activities enhance rapport and liking between people and the feeling of being a member of a group [47,48], and may explain the ubiquity of different, synchronized social rituals across cultures. As suggested by Volpe et al. [14] in this theme issue, musicians playing in ensembles are experts in non-verbal social interaction: they play in the same tempo, listen to and react to others, and occasionally find the groove. In small ensembles, all musicians have to co-regulate their performances, whereas in big orchestras, the musicians follow the conductor (leader), who, on the other hand, has to be very sensitive to both visual and acoustic cues from the orchestra.

Small musical ensembles thus provide exciting possibilities for combining cognitive neuroscience and computational approaches to the study of cooperative goal-directed actions. It is possible to record the physiological reactions of the players, measure their brain activity, and analyse the kinematics of their movements. Importantly, the sound of the music itself provides accurate information about the output of the whole performance.

Music-induced synchrony and turn-taking may also have therapeutic effects. Previous studies imply that music therapy can improve communicative behaviours and joint attention in children with autism. In this issue, Spiro & Himberg [15] discuss methods to quantify video recordings of improvisational music-therapy sessions. By focusing on straightforward behavioural units—shared movement and facing behaviours, joint rhythmic activity and musical structures, and the relationships between them—they show how to trace aspects of interaction during music therapy. In the context of the reciprocal difficulty of action understanding in autistic and neurotypical individuals [9], it is interesting to speculate that one of the major mechanisms underlying positive music therapy would be the facilitation of both synchrony and proper turn-taking in the autist–therapist dyad by the accentuated musical cues.

8. Basic principles of non-verbal and verbal interaction

Social interaction is surprisingly easy although we do not yet know why. People align their styles of speaking, rhythm, and dialect spontaneously and without any instructions [49]. Communication can work rather well between very imbalanced participants, such as two people with different language skills, an adult and a child, and even a human and a pet animal. Mistakes of course happen, making the interaction vulnerable, but they are collectively repaired all the time.

A common approach to study brain function is to isolate different processes and study separately their inputs, outputs and the inner structure. However, how can one study interaction without both parts/partners being present?

In this theme issue, de Jaegher et al. [6], combining the views of interactional sociology and enactive cognitive science, discuss how people co-create meaningful actions. They consider the interaction as an autonomous process that self-organizes during the course of the interaction and thereby becomes clearly distinct, although not isolated, from the environment, being most strongly determined by factors internal rather than external to the ‘interaction unit’. The self-organizing interaction thus has systemic properties that cannot be reduced to the sum of the participants' properties and intentions. As stated by de Jaegher et al. [6], it is ultimately necessary to understand both the interactive and the individual contributions to the (co-)regulation and coordination of behaviours that form the social interaction.

The most sophisticated interaction studies so far are probably those on dialogues with spoken language. Garrod & Pickering [50] consider that conversation is so easy because humans are designed for dialogue rather than monologue. This is a very interesting statement as, in principle, dialogue should be rather difficult as the expressions are fragmentary, and the other person's utterances are unpredictable, needing lots of repairs and repetition. The speakers also have to take into account what and whom the other knows and refer to those items appropriately. Moreover, they have to take turns in expressing themselves.

Dialogues with spoken language start in early infancy as proto-conversations—structured interchanges between the carer and the infant in the form of isolated words, utterances and gestures—in attempts to convey meanings before the onset of language in the child. Proto-conversation already has clear forms of turn-taking, alternation of active and inactive phases.

Turn-taking is evolutionarily old and it happens in animals not using language. It therefore cannot be considered to be determined (only) by cultural norms and conventions as suggested in conventional conversation analysis [6]. In this volume, Takahashi et al. [17] show that marmoset monkeys display vocal turn-taking which develops in infancy at the same rate as the skills of self-monitoring. Apparently, marmoset vocal turn-taking reflects convergent evolution as the species is evolutionarily far from humans.

In all languages and cultures, turn-taking occurs with an average lag of 250 ms, which is far too short a time for the interlocutors to react to the end of the previous turn [51]. Instead, the interlocutors have to be aligned to the other person's utterance content and rate, and start the planning of their own turn several hundred milliseconds before the other person ends the turn. In other words, the preparation for the new turn and listening to the message must overlap [52]. During conversation, however, one does not need to start the planning of the next turn from scratch, as the previous interaction and turns have primed certain utterances and thoughts. Thus, the resulting multitasking challenge is easier than, for example, when trying to do your homework while someone is talking to you.

During conversation, both partners are involved in a joint task. Aptly, conversation and other smooth turn-taking interactions can be compared to a three-legged race [53], where both partners are aiming at the same goal, in part bound to each other, which is very different from a tennis match, where the participants just react to each other's shots (which they will try to make as unpredictable and difficult to return as possible). Another fitting analogue for the two persons' actions during a conversation would be a bimanual task [7] where—to build something sensible—both hands do have their own role but both are controlled by the same processor and guided by a common goal.

9. From one-person to two-person neuroscience

We have already suggested that social interaction cannot be reduced to sequential and partially parallel processing of the input by two independent brains, but that social interaction actually emerges only when a two-brain network is established. Hari et al. [54] recently dissected the different levels of brain imaging of social cognition and interaction into single-person studies (‘one-person neuroscience’, 1PN) and two-person set-ups (‘two-person neuroscience’, 2PN [2,55,56]). The 1PN set-ups have evolved from presentation of well-defined artificial stimuli (such as chessboard patterns and isolated tones), to the use of complex social stimuli, such as faces or body postures, and then finally to presentation of dynamic stimuli, such as movies. However, all these set-ups can be criticized as examples of ‘spectator science’, because the brains under study are assumed not to change their operating state during the experiments, but just generate responses directly related to the stimuli or the task at hand, even if those are highly emotional [54].

The spectator view contrasts with the 2PN set-ups, where people function as engaged interaction partners and the forthcoming stimuli (e.g. the facial expression of an interaction partner) are influenced by the participant's own previous reactions [54]; thus the stimuli cannot be accurately predicted in advance. Whether these two-person settings should be used despite their complexity depends on the timing of the studied interaction [54]: all two-person studies where the interaction is slow, such as text messaging or playing an economical-decision game, can be replaced by clever pseudo-hyperscanning set-ups where the two persons are alternatingly subjected to brain scanning. However, during conversation, for example, the turn-taking takes such a short time that the interaction can be captured only in simultaneous time-sensitive 2PN recordings [56].

Behavioural evidence very strongly suggests that during joint tasks people may enter into states of ‘togetherness’, characterized by two-person flow in which neither of them is consciously leading or following [57]. Similarly, the smooth turn-taking occurring during conversation [52] strongly speaks for an autonomous and self-organizing interactive state, as discussed above [6].

An important empirical question is whether social interaction emerges from lower-level perceptual, motor, and cognitive functions—as is usually assumed in neuroscience—or whether it could be the default mode governing other brain functions. In this issue, de Jaegher et al. [5] initiate an interesting dialogue between cognitive neuroscience and enactive views of social cognition discussing the interactive brain hypothesis (IBH), which in its strong form would claim that social interaction has an enabling or even constitutive role for cognitive functions. Such a primacy of social interaction would challenge many current ideas about human brain function [54]. Whether this view is correct is ultimately an experimental question that might benefit from (or even need) brain imaging with two-person settings.

10. Interacting with robots

Robots are adept in performing laborious, repetitive, and dangerous tasks, but they are increasingly also used for interacting with humans. Because robots can be controlled accurately, human–robot interaction provides a test-bed for the naturalness of social interaction and joint attention as different parameters can be accurately varied, and the effects on the interaction partners can be studied. People sense the engagement of another person by means of mutual adjustments of timing in the interaction, and anecdotal evidence suggests that it is just the unnatural timing that people easily get annoyed with during human–robot interactions.

Wykowska et al. [16] discuss, in this theme issue, artificial agents, in particular embodied humanoid robots. Such carefully controlled agents—with changing appearance, expressiveness, gaze cuing, joint attention, and timing—may provide an attractive experimental model for studying neurocognitive mechanisms of ‘real’ social interaction. Interestingly, many requirements for interaction are roughly similar for humans [6] and for robots: (i) co-presence (with the possibility to monitor others and oneself) is needed to know whether other persons/agents are present in the same space. (ii) Engagement as revealed by mutual reactivity and synchrony informs whether the others hear, see, or feel the interactor. Finally, (iii) turn-taking as the strongest form of engagement and (iv) sequentiality of actions are characteristics of any smooth interaction, be it with humans or robots. It is this sequence of joint actions and turns that finally forms the fabric of the successful interaction.

11. Current pitfalls

We can understand brain functions underlying behaviour only by binding them to the phylo- and ontogenesis of humans and the processing demands of the environment. The current studies of social cognition and interaction seem to suffer from at least three major pitfalls: (i) studies are mostly limited to the spectator view; (ii) neurodynamical information about social interaction is scarce; and (iii) the behaviour of the interaction partners is typically characterized at a very crude level.

• (1) Spectator view. As described above, social neuroscience has so far mainly targeted reactions and actions of individual persons who are presented with well-defined social stimuli. However, this kind of ‘spectator view’ is not representative of real-life brain function that has to support engaged participants in dynamic, interactive settings. Somewhat surprisingly, social interaction, such as dialogue with fast turn-takings, often unfolds more easily than the corresponding individualistic action (e.g. monologue) despite the fragmentary, incomplete, and unpredictable information on which the interaction has to rely [52]. We need a leap from the spectator view and individualistic stance, as with proceeding from monologues to dialogues in language research.

• (2) Focus on neurodynamics. Timing is quintessential for human behaviour and the nested timescales of interest range from the submillisecond scale to seconds, minutes, and even lifetimes [58]. While the neuroimaging community has previously focused on detailed characterization of the static connectome, it is becoming increasingly interested in timing information, and the same should happen in the context of social neuroscience, where the joint timing of the social interaction is critical for smooth interplay. Basically, no social interaction—such as shaking hands, discussing, or walking together—can be accomplished if one or another partner is out of time. Thus, in addition to characterizing the timing of individual social behaviours, we also need to quantify the timeframes of dyadic social interactions [54,59].

• (3) Quantification of behaviour. Our most widely used tools for quantifying human movement are surprisingly crude, clearly below the resolution we have obtained for the corresponding brain functions. Although we have access to accurate motion-capture systems and although computer vision now allows recognition of human faces and facial expressions, accurate online tracking of human movements and facial communication is rarely employed in social neuroscience and psychological studies. One reason may be that such natural movements are too fast for people to note and are thus neglected by experimenters. However, the human brain reacts markedly to eye blinks of a conversation partner, even though the blinks typically do not capture the viewer's attention [60,61]. The same is apparently true for other fast facial expressions that exceed the capacity limits of human awareness, but can be captured with, for example, facial electromyography [62].

Similarly, ‘forms of vitality’ [63], diverse facial and bodily expressions that cannot be verbally explained and that rapidly change on the faces of even newborn infants, clearly carry much information about the state of the person, but really cannot be classified or described with the current methods. Here, we may search for inspiration from recent machine-learning-based classification of mouse behaviour, indicating that movements are combinations of different elementary movements, subsecond postures of mouse body language which form ‘syllables’, like those in language [64].

12. Conclusion

Recent methodological developments now allow studies of brain mechanisms of social interaction in highly naturalistic settings, and even quantification of the brain basis of ‘live’ interactions between humans. These methodological developments have the potential to result in a major paradigm shift in the social sciences and neurosciences, and they have already challenged the conventional ways of thinking about the ‘social’ brains of humans and other animals. Thus, we might be getting a little bit closer to a complete description of human behaviour where, so far, the verbal communication has been studied much more than non-verbal communication involving tiny expressions, gestures, and eye movements.

The multidisciplinary interactions and new concepts presented in this theme issue will hopefully change the ways we view and study social interaction and what questions we dare to ask. For example, to what extent do we need to move beyond single-person neuroimaging to the study of two persons at the same time? How can we take both neuroscience and behavioural factors into account in understanding others during social interaction, be they in-group or out-group members? Can we dampen our prejudices? Which methodological and data-analytical approaches are best suited for quantifying the brain dynamics of complex, natural social interaction? Mutual understanding is, for sure, getting more and more important in our increasingly unstable world. We need evidence converging from different disciplines to form a more holistic view of human behaviour and brain function, and to finally understand why and how we attend to some people and neglect the others.

Biographies

Guest editor biographies

Riitta Hari is Distinguished Professor (emerita) in systems neuroscience and neuroimaging at the Aalto University, Finland. She was trained in medicine and clinical neurophysiology at the University of Helsinki. Since the early 1980s, she has led a multidisciplinary Brain Research Unit, with a focus on systems-level human neuroscience, especially development of time-sensitive human brain imaging with magnetoencephalography (MEG). Prof. Hari's work has provided fundamental insights into human sensory, motor, and cognitive functions, with applications both in basic research and in clinical diagnostics and follow-up. More recently, Hari has introduced ‘two-person neuroscience’ into studies of the brain basis of social interaction. Hari was nominated Academician of Science in Finland in 2010, and has been a member of the National Academy of Sciences USA since 2004.

Mikko Sams received his PhD from the University of Helsinki in 1985. In his thesis, he studied neural mechanisms of auditory discrimination with electroencephalogram (EEG) and MEG. Before his present position as a Full Professor of Cognitive Neuroscience in Aalto University (Finland), he was a Full Professor of psychology at the University of Tampere, Finland. He was nominated as an Academy Professor for 2001–2007. He has studied feature processing in the human auditory system and how it is modified by attention. He is one of the pioneers in the study of neural mechanisms of audiovisual speech perception. He is currently studying neural basis of shared reality, mutual understanding, and emotions using naturalistic research paradigms. His teaching duties have included setting up a new human neuroscience and technology major in the Master's Programme in Life Science Technologies.

Lauri Nummenmaa did his PhD on neurocognitive mechanisms of social attention at the University of Turku in 2006. After a post-doc period at the Medical Research Council Cognition and Brain Sciences Unit in Cambridge, UK, focusing on the neural mechanisms of face perception in Andy Calder's group, Prof. Nummenmaa returned to Finland in 2008. He has worked as Academy of Finland junior and senior fellow at Turku PET Centre and Aalto University. Currently, Nummenmaa is a tenure-track assistant professor in cognitive neuroscience at the Aalto University, Finland, with joint part-time appointments at the Turku PET Centre and the Department of Psychology, University of Turku. His group studies functional and molecular neural mechanisms of human emotions and social interaction using positron emission tomography, magnetic resonance imaging, EEG and MEG, and behavioural techniques.

Competing interests

The authors declare no competing financial interests.

Funding

This work was supported by European Research Council (advanced grant no. 232946 Brain2Brain to R.H. and starting grant no. 313000 Fitting the world to minds: brain basis of sharing and transmitting representations of the social world to L.N.) and the Academy of Finland (grants nos. 265917 and 259952) and by the European Union Seventh Framework Programme (FP7/2007–2013) under grant agreement no. 604102 (Human Brain Project).