Enhanced emotional responses during social coordination with a virtual partner

Abstract

Emotion and motion, though seldom studied in tandem, are complementary aspects of social experience. This study investigates variations in emotional responses during movement coordination between a human and a Virtual Partner (VP), an agent whose virtual finger movements are driven by the Haken-Kelso-Bunz (HKB) equations of Coordination Dynamics. Twenty-one subjects were instructed to coordinate finger movements with the VP in either inphase or antiphase patterns. By adjusting model parameters, we manipulated the ‘intention’ of VP as cooperative or competitive with the human's instructed goal. Skin potential responses (SPR) were recorded to quantify the intensity of emotional response. At the end of each trial, subjects rated the VP's intention and whether they thought their partner was another human being or a machine. We found greater emotional responses when subjects reported that their partner was human and when coordination was stable. That emotional responses are strongly influenced by dynamic features of the VP's behavior, has implications for mental health, brain disorders and the design of socially cooperative machines.

1. Introduction

Emotion and motion are complementary sides of human social interaction that shape the way we associate with ourselves and other human beings (Kelso and Engstrøm, 2006; Markus and Kitayama, 1991; Strayer, 2002). The study of emotion engagement during dynamic and reciprocal social interaction (second–person perspective) is a key to understanding social cognition and its neural mechanisms (Schilbach et al., 2013). The present study aims to probe the dynamic relationship between emotion and coordinated movement during continuous social interaction. Synchronization of movement is an important mechanism for social coordination, as for example in emotional contagion or counter-contagion (Hatfield et al., 1994). Socially synchronized rhythmic movement has been associated with trust, liking, affiliation, and compassion toward a synchronized social partner (e.g. Hove and Risen, 2009; Valdesolo and DeSteno, 2011; Launay et al., 2013, 2014). Moreover, disruptions of social synchrony are associated with negative affect and antagonistic social interactions (Tschacher et al., 2014; Paxton and Dale, 2013). In the present study, we investigate dyadic social interaction as a coupled dynamical system. In such a perspective, synchronization or phase-locking between interacting components re-flects attracting states of their collective dynamics (Strogatz, 1994; Kelso, 1995). By studying the stability of attracting states, one can learn more about the underlying dynamic landscape of the coupled pair (Fuchs, 2013). Therefore, a key aspect of our investigation is the relationship between stability of the social Coordination Dynamics and physiological measures of the participant's emotional state. Depending on the goals and expectations (e.g. of a competitive situation) that one brings into the social interaction, social partners’ spontaneous emotional and behavioral coordination may not always be symmetrical (e.g. you smile, I smile) but can also be counterempathetic (Englis et al., 1982; Lanzetta and Englis, 1989; Hatfield et al., 1994), meaning behavioral synchronization or matching might not always be a person's preferred response, or congruent with his or her emotional state. If one's intention is to not synchronize with others, might there be emotional responses associated with unwanted synchronization? In the present research, we manipulated subjects’ intention by instructing them to coordinate either inphase (i.e. synchronization) or antiphase (i.e. syncopation) with a computational surrogate of a social partner, a Virtual Partner or VP.

In previous work, Kelso et al. (2009) investigated the behavioral patterns of human subjects when they tried to coordinate rhythmic movement with a VP. When the VP was parameterized to syncopate in contradiction with the subject's goal to synchronize, subjects reported in post-experiment interviews that their partner was “messing” with them. Such verbalizations suggested that subjects may have experienced negative emotions and attributed agency/humanness to the computational model. This enticed us to try to quantify the emotional experience of the subjects when they behaviorally coordinated with a social partner. How is the attribution of humanness related to emotional experience? As a first step in this direction, in the present study we asked subjects to explicitly judge whether they were interacting with a human being rather than a computer program. The idea was to assess the contribution of both the perceived sociality and dynamics of the movement coordination to emotion. Since subjects did not have direct access to their partner's intention other than through their experience of the collective behavioral outcome, we were curious to identify which characteristics of the behavioral outcome, if any, might connect to subjects’ emotional experience. In the present research, we created a situation as in Kelso et al. (2009) in which the VP did not always share the same “intention” with the subject (i.e. sometimes cooperative, sometimes competitive), but had stronger capability to dominate the behavioral outcome regardless of the subject's intention. Thus we were able to treat the behavioral outcomes of coordination, the underlying intentions of the partners and their cooperative-/competitiveness as separable dimensions, in principle allowing us to study their interrelationship.

Why have we chosen to investigate such issues using a VP, a computational surrogate, interacting with humans as a model for social coordination? The main reason is that it is possible to manipulate coordination between self and other quantitatively by changing model parameters of the VP (e.g. coupling strength and preferred phase relationship) in a realistic manner and in ways that might not be possible in ordinary experiments (Kelso et al., 2009; Dumas et al., 2014). The VP is the key component of an experimental paradigm – the Human Dynamic Clamp (HDC) – recently proposed as a new tool to study human social interaction (Dumas et al., 2014; Kelso et al., 2009; Kostrubiec et al., 2015). The HDC allows a human being to interact in real-time with a VP driven by well-established models of Coordination Dynamics. People coordinate hand movements with the visually observed movements of a virtual hand, the parameters of which depend on input from the subject's own movements. Thereby a perceptuo-motor coupling is created between a human being and his/her mathematically modeled partner. In the present study, VP's behavior was governed by the Haken-Kelso-Bunz (HKB) equations (Haken et al., 1985) – a theoretical model that has been demonstrated to capture critical features of intrapersonal, sensorimotor and social coordination (Kelso, 1995). Coordination Dynamics, the theoretical and empirical basis of this paradigm, uses the language of nonlinear dynamical systems to address how collective patterns form in a self-organized system of reciprocally coupled components across different scales (see Kelso et al., 2013 for recent review). Such collective patterns are defined by relational quantities that are appropriate for the system under study (e.g. relative phase between oscillatory components). The stability of observable collective patterns reveals the underlying landscape of the dynamics. Besides the observable behavioral patterns, another relational quantity is the discrepancy between the partner's intentions, which is termed cooperative-competitiveness. In the present study, the cooperative-competitiveness dimension of coordination and the stability of relative phase are key variables in characterizing social interaction from the perspective of Coordination Dynamics.

In order to probe the dynamic relationship between emotion and interpersonal coordination, a measurement of emotional responses was chosen that could be recorded continuously and non-obtrusively during the experiment. Skin Potential Responses (SPR) were recorded during and after social movement coordination, and later analyzed to quantify subjects’ level of emotional arousal. Amongst other measures of human electrodermal activity (EDA), SPR reflects the sweat gland activities controlled by the autonomic nervous system (ANS), known to be associated with emotional experiences (Critchley, 2005; Sequeira et al., 2009; Boucsein, 2012). In general, measurements of EDA are more sensitive to changes in the arousal dimension of emotion but not to specific types of emotion (Kreibig, 2010). Here we used SPR as an indicator of persistence and change of emotional patterns during and after social coordination.

Three main questions are addressed in the present research: (1) whether perceiving oneself interacting with a human or a computer provokes different levels of emotional arousal; (2) whether competitive or cooperative interaction with a human-like partner leads to more or less emotional arousal; and (3) whether the level of emotional arousal is linked to the stability of coordination. Although our approach is discovery based, given the literature alluded to above, we hypothesized that all three dimensions – interacting with a VP that is perceived as human, competitive interaction, and stable coordination – would lead to greater emotional arousal.

2. Method

2.1. Subjects

21 subjects (9 female and 12 male) were recruited at Florida Atlantic University aged between 18 and 48 (average 25) years. All subjects were right-hand based on the Edinburgh Handedness Inventory and had no reported sensorimotor impairment. The protocol was approved by Florida Atlantic University Institutional Review Board and met the requirements of the World Medical Association Declaration of Helsinki. Informed consent was obtained from all subjects. Two subjects’ data were excluded from the analyses of emotional arousal during coordination and during subjective report, due to their lack of compliance to experimental instructions (see section 3.2).

2.2. The Virtual Partner system

The experimental setup is shown in Fig. 1. VP (Fig. 1B) is a computational surrogate of a social partner who coordinates with a human in real time. It “senses” human movement (sensory unit, B-1), the information from which it combines with its own ongoing movement to compute the next movement state (computation unit, B-2) and update the image on the computer screen (display unit, B-3).

Fig. 1.

Experimental setup: human-VP interaction and SPR measurement of emotion. The experiment involves the coordination of finger movement between a human subject (A) and VP (B). The subject's right index finger motion is received by VP sensory unit (B-1). VP computes behavioral updates based on a model of social coordination (B-2) that includes the subject's position (y) and velocity (ẏ) and its own (x,ẋ). VP's online behavior is returned to the subject through vision of an avatar displayed on screen (B-3). In parallel, a pair of electrodes placed on the left arm (C) provides the subject's SPR in a continuous and non-obtrusive manner. See details in Sections 2.2 and 2.3.

2.2.1. The sensory unit

VP “sees” the human partner's movement through a goniometer attached to a manipulandum (Fig. 1B-1). The manipulandum limits the movement of the subject's right index finger on the horizontal plane. The displacement of the manipulandum is transduced into a voltage by the goniometer, digitized by a National Instruments analog-to-digital converter at 500 Hz, and then sent to the computational unit.

2.2.2. The computational unit

To update its behavior continuously, VP computes the Haken et al. (1985) equations in real-time (500 Hz) with an interactive program implemented in C++ (Fig. 1B-2; Dumas et al., 2014). The program receives information on subjects’ current finger position (y in Eq. (1)) from the sensory unit and computes the velocity (ẏ). Based on the current human movement state (y, ẏ) and VP movement state (VP finger position x and velocity ẋ), the core routine of the program computes VP's next movement state by integrating the HKB equations expressed at the oscillator level (Eq. (1)), using a Runge-Kutta fourth-order algorithm.

$$\stackrel{\widehat{A}}{x}+\left(\alpha x^2+\beta {\stackrel{.}{x}}^2-\gamma \right)\stackrel{.}{x}+{\omega }^2\ddot{x}=\left(A+B{(x-\mu y)}^2\right)(\stackrel{.}{x}-\mu \stackrel{.}{y}).$$ (1)

On the left hand side, α, β, and γ are parameters that modulate the characteristics of the movement trajectories of VP's finger, such as velocity-dependent damping and are chosen to approach the kinematic shape of human movement (Kay et al., 1987, 1991). For this experiment their respective values are 0.641, 0.00709, and 12.457. ω controls VP's natural movement frequency (the frequency at which the VP moves without a partner), chosen to be 1.6 Hz (10 rad/s) – a comfortable and familiar rate for human finger movement, and one at which humans are typically able to coordinate both at inphase and antiphase (Kelso, 1984). The right hand side specifies the VP-to-human coupling. Parameters A and B modulate coupling strength (chosen to be −3 and −0.025 respectively) and μ determines the VP's intentions (Kelso et al., 2009). In this experiment, the intention parameter μ was fixed at either +1 (VP “intending” inphase coordination, strongest attractor for a relative phase with the partner at π rad) or −1 (VP “intending” antiphase coordination with a subject, strongest attractor at π rad). The VP-to-human coupling strength was set to be relatively high so that when a human subject and VP have different intentions, VP's intention dominates the outcome of the coordination.

Each updated VP state is sent to the display unit (Fig. 1B-3), and both VP and human states are also appended to a file stored on a local hard drive for offline analysis. The computational unit also manages event markers (e.g. start/end of different periods within a trial, see Section 2.4), which are sent to a Neuroscan bioamplifier (Synamp2, TX, El Paso) to synchronize the partners’ movement data with the SPR recording.

2.2.3. The display unit

The human partner sees VP's behavior thanks to the graphic routine of the real-time interactive program whose information is returned to a computer monitor placed in front of him/her (Fig. 1-C). At each video refresh (120 Hz refresh rate), the program selects out of 119 position-indexed photos of a hand, one with a position matching the computational unit's current position x. The high refresh rate guarantees that the animation looks like a video of a hand whose index finger moves continuously.

2.3. Skin potential response measurement

Two Ag/AgCl electrodes filled with isotonic recording electrode gel (BioPac, GEL101) were used to acquire subjects’ SPRs. One was placed on the palmar area of the left hand (active site rich in sweat glands) and the other was placed at the lateral epicondyle of the left elbow (reference site). The electric potential signal was acquired (in DC mode with no high pass filter), digitized at 1000 Hz with the Neuroscan amplifier, and then recorded on a dedicated computer (along with the subject's EEG of which the analysis is beyond the scope of the present study). The choice of Skin Potential Response over Skin Conductance Response was dictated by requirements to avoid EEG artifacts, since SPR can be recorded without applying a voltage on the skin (Fowles et al., 1981).

2.4. Experimental procedures

2.4.1. Control conditions

In order to extract a valid metric of SPR that quantifies emotional arousal and obtain a baseline for each subject, two control conditions were used. In the first condition, subjects’ SPRs were evoked by a startling sound. The stimulus was repeated for 5 times with inter-stimuli intervals that varied depending on the relaxation time of each evoked SPR. The evoked SPRs were used for selecting a metric that best represents the magnitude of a whole response without being biased by variations in shape (e.g. the number of waves, amplitude ratio between waves). In the second condition, subjects rested, with their eyes fixating upon the middle of the screen for 2 min, with no stimuli presented. The comparison between the intensity of SPR responses in these two conditions was later used to verify that the selected metric reflects arousal level (higher SPR intensity when evoked than at rest). Further details are presented in Section 2.5.

2.4.2. Experimental conditions

The experimental design included 3 factors: the human's intention (inphase or antiphase), the VP's initial intention (inphase or antiphase), and the VP's intention change (with or without change midway through the trial). A key variable generated by this design is whether human and VP shared the same intention or not, here termed as “cooperativeness” of the coordination task. Recall that the “intention” of the VP was manipulated by parameter μ in Eq. (1) and the human's intention by instruction. We define coordination to be “cooperative” if human and VP shared the same intention, and “competitive” if not. In the present paper, “cooperativeness” refers to this experimental manipulation, unless specified otherwise. Before the experiment started, the basic coordination patterns inphase and antiphase were demonstrated to all subjects. Subjects were told that, for each trial, their partner on the screen may or may not intend to achieve the same coordination pattern, and that the partner would be controlled either by the computer or an experimenter (to lend plausibility to the possibility of experimenter engagement, an additional manipulandum was set up outside of the recording room; although the Virtual Partner was always controlled by the HKB equations during the experiment, the ability of the experimenters to control the avatar in real time had been demonstrated to the subject during the instructional phase of the experiment). Subjects’ perception of VP's cooperativeness and humanness were reported at the end of each trial by pointing finger to the left or right to answers to binary questions.

Each trial (Fig. 2) began with a screen display indicating subjects’ assigned intention (inphase or antiphase, Fig. 2 bottom left), presented simultaneously with a 3 s metronome (Fig. 2 top left) at 1.6 Hz, during which subjects matched their finger movement to VP's intrinsic movement frequency (ω in Eq. (1)). After the metronome and a 1 s transient, VP's animated hand appeared on the screen for 8.2 s, and both subjects and VP were able to coordinate with each other. VP's intention (inphase, antiphase) was randomly assigned for each half of the 8.2 s period of interaction (“Coordination Period 1” and “Coordination Period 2” thereafter). As a result, there was a 50% chance that VP changed its intention midtrial. Since the subject's intention remained the same throughout the trial, when VP changed its intention, cooperativeness changed accordingly. On completion of the coordination periods, three successive questions were presented to the subject to assess his/her subjective experience of the interaction: (1) whether the partner was cooperative or competitive in Coordination Period 1; (2) likewise in Coordination Period 2; and (3) whether the subject thought the VP was controlled by a human or the computer.

Fig. 2.

The structure of an experimental trial. The timeline is shown on top; shaded area for transients between phases of the trial, when a plain dark screen appeared for 1 s. Screen displays illustrative of those seen by subjects are on the bottom. See details in text.

In total, each subject completed 80 experimental trials, which were divided in 8 blocks of 10 trials. Subjects’ intention was switched between inphase or antiphase at each block with the help of written and verbal instructions.

2.5. Analysis strategy

2.5.1. Selection of representative metric for SPRs and its validation

SPR has a more complex and variable waveform than the Skin Conductance Response even though they reflect the same sweat gland activity and subsequent changes in skin electrical properties (see Boucsein et al., 2012 for a summary of mechanism). Whereas a typical Skin Conductance Response only has one positive component (a monophasic wave), SPR usually has a multiphasic waveform, with both positive and negative components (Fig. 3A). Given SPR's multi-phasic nature, evaluation of response intensity is more difficult than Skin Conductance Response (Fowles et al., 1981).

Fig. 3.

Identification of SPR latency and the computation of SPR Magnitude. (A) SPR is a multiphasic wave, the onset of which is estimated retrospectively from the peak of a-wave (pink box; for details see text); (B–C) SPR Magnitude is computed as the average of rectified SPR within 4 s from SPR onset.

Although amplitude of the first two waves of SPR (i.e. a- and b-wave in Fig. 3A) both increase monotonically with emotion (Burstein et al., 1965), there is a lack of consensus about which SPR metric best characterizes the intensity of subject's emotion (Boucsein, 2012), suggesting that both waves carry some meaningful information. In an effort to select a single measure that is most representative of the whole response, we developed a new metric, SPR Magnitude, which is the average of the rectified SPR signal within a 4 s window from the onset of the a-wave (Fig. 3B-C). Window size was chosen to match the temporal properties of our trial structure. To test whether SPR Magnitude is more representative than a- or b-wave alone, we studied the correlation between SPR Magnitude and the amplitude of a- and b-wave with subjects’ evoked SPRs to startling sounds. Further, to validate SPR Magnitude as a measure of emotional arousal, we tested whether the average SPR Magnitude of evoked responses exceeded that of the baseline condition (subjects at rest).

2.5.2. Calculation of SPR latency

In order to connect SPR Magnitude to an emotional event, we must take into account the time it takes for emotional information from the central nervous system to reach peripheral effectors (sweat glands) and give rise to SPRs. This delay will be referred to as SPR latency (Fig. 3A). Latency is unknown in the case of endogenous emotional responses elicited during experimental conditions, but is observable during exogenous responses to startling sounds (control conditions). Since latency has more intra-subject consistency than inter-subject consistency (Levinson and Edelberg, 1985), we used subjects’ SPRs to startling sounds to estimate subject-specific SPR latencies, that is, the average delay from sound onset to SPR onset within each subject. SPR onset was not directly identified from the rise of the a-wave from baseline, because background noise causes this event to have low Signal-to-Noise Ratio (SNR). Instead, SPR onset (denoted as t₀ in Eq. (2) below) was estimated by subtracting double the peak/half-peak interval from when the a-wave reaches its peak (t_a) (Eq. (2)):

$$t_0=t_a-2(t_a-t_{a∕2})$$ (2)

where t_a is when an a-wave reaches its peak, and t_a/2 is when the a-wave rises to half of its amplitude (Fig. 3A). Estimation of SPR onset with t₀ was robust since peaks and half-peaks have enhanced SNR as compared to the baseline SPR signal, and is legitimate because the rise of the a-wave is quasi-linear (see Fig. 3A for a representative example).

2.5.3. Quantification of the level of emotional arousal in experimental conditions

During coordination, SPR Magnitude was calculated following the onset of Coordination Period 1 and Coordination Period 2. During subjective report, SPR Magnitude was calculated following the onset of each question. For cross-subject statistics, SPR magnitudes in the experimental task were normalized to the subject-wise average SPR magnitude in the sound-evoked SPRs.

2.5.4. Characterizing the stability of behavioral coordination

The stability of the coordination was defined as the stability (γ_cv, Eq. (3)) of the relative phase (ϕ, Eq. (4)) between subject and VP finger movement.

$${\gamma }_{cv}=\frac{1}{N}\mid \sum _{t=1}^N{e}^{i\varphi \left(t\right)}\mid$$ (3)

$$\varphi ={\phi }_{vp}-{\phi }_{human}.$$ (4)

Movement phase, denoted as φ_vp and φ_human, was obtained from the Hilbert Transform of the detrended movement trajectories x and y. ϕ is the relative phase wrapped in the interval (−π, π]. γ_cv is derived from the circular variance ( CV=1-γ_cv), and characterizes the stability of social coordination (e.g. Tognoli et al., 2007; Kelso et al., 2009). The value of γ_cv ranges from 0 to 1, with 1 indicating a perfectly stable relative phase. Following (Tognoli et al., 2007), a cutoff value of 0.9 was used to dichotomize coordination periods into stable ( γ_cv>0.9), and unstable (γ_cv≤0.9). We define a change of stability if the coordination changes between stable and unstable from Coordination Period 1 to Coordination Period 2.

2.5.5. Characterizing the actual pattern of behavioral coordination

The actual coordination pattern is defined as the average distance to inphase (ϕ, Eq. (5)) within a coordination period:

$$\varphi =\frac{1}{N}{\sum }_{t=1}^N\mid \varphi \left(t\right)\mid$$ (5)

ϕ was dichotomized into inphase if $\varphi <\frac{\pi }{2}$, and antiphase if $\varphi \ge \frac{\pi }{2}$. This means that if the relative phase of the dyad's finger movements was closer to inphase on average, we called the actual coordination pattern inphase, otherwise, antiphase.

Here ϕ was used to verify the success of the experimental manipulation, to wit whether strong VP-to-human coupling (terms A and B, Eq. (1), chosen to examine subjects’ ability to detect change in cooperation) resulted in actual coordination patterns that were more highly correlated with VP's intention than with the human's.

2.5.6. Statistical analysis of the relationship between the level of emotional arousal and coordination

MANOVA models were used to study how different characteristics of coordination collectively contribute to variations in emotional arousal, along with all the two-way interactions. Type III sum of squares was used to study the significance of each factor while controlling for others (Yates, 1934). Emotional arousal during human-VP coordination (Coordination Periods 1 and 2, see Fig. 2) and during post-coordination subjective reports were studied under two separate statistical models (Table 1). Tukey's HSD test (Hochberg and Tamhane, 1987) was used for post-hoc analyses of the main and interaction effects found in the MANOVAs.

Table. 1.

MANOVA models of emotional arousal.

Emotional arousal during:		Coordination	Subjective report
Factors (Levels)
Humanness attribution	(Human, computer)	✓	✓
Cooperativeness	(Cooperative, competitive)	✓	✓
Stability	(Stable, unstable)	✓	✓
Human intention	(Inphase, antiphase)	✓	✓
Change of cooperativeness	(Change, persistence)		✓
Change of stability	(Change, persistence)		✓

3. Results

3.1. SPR characteristics and validation

SPRs to startling sounds (control condition) were used to test whether SPR Magnitude was representative of both the a-wave and b-wave, and whether it varied according to change of arousal level. Ninety-four SPRs with identifiable a-waves and b-waves were extracted following 105 startling sounds (11 out of 105 responses were excluded due to a lack of response). Descriptive statistics and correlation coefficients are listed in Table 2. The amplitudes of the a-wave and b-wave are only moderately correlated with each other but are both highly correlated with SPR Magnitude.

Table. 2.

Correlations between different metrics of the intensity of SPRs (descriptive statistics are shown in the first row, in μV).

		a-Wave amplitude	b-Wave amplitude	SPR magnitude
	Mean (SD)	449 (377)	549 (471)	286 (209)
a-Wave amplitude	Correlation coefficient	1
b-Wave amplitude		0.523***	1
SPR magnitude		0.739***	0.913***	1

^***p < 0.001.

The average latency of these SPRs was 1.72 s (±0.46 s). Between-individual variations were significantly larger than within-individual variations (one-way ANOVA: F (20.73) = 2.87, p < 0.001). Subject-wise average latency was used to determine the window for SPR Magnitude calculation in experimental conditions, in order to reduce the bias due to individual differences in SPR latency.

Significant individual differences were also found in SPR Magnitude (one-way ANOVA, F (20.73) = 7.57, p < 0.001), therefore we used normalized SPR Magnitude throughout. SPR Magnitude was higher in response to startling sounds (Mean = 1, SD = 0.61) than at rest (Mean = 0.305, SD = 0.383; t = 9.35, p < 0.001), suggesting that normalized SPR Magnitude is a valid measurement of the level of emotional arousal.

3.2. Summary of the characteristics of behavioral coordination

In roughly half (50.1%) of the coordination periods, the dyads coordinated closer to inphase, and 49.9% antiphase. By our criteria, coordination was stable in 52.9% of the coordination periods, and unstable 47.1% of the time. Fig. 4 shows a typical example of human-VP coordination, with their movement trajectories (A) and a measure of their coordinative stability using relative phase (B). Two subjects had a very low number of stable coordination periods (<10%), while other subjects ranged from 31.8% to 75.6%. Visual inspection revealed that in many trials the 2 subjects were not moving rhythmically according to the experimenter's instruction. Their data, therefore, were excluded from further analysis.

Fig. 4.

Behavioral outcomes of an example trial. (A) Amplitude-normalized movement trajectory of human and VP; (B) corresponding relative phase time series, which shows that coordination was unstable in Coordination Period 1 when human and VP didn't share the same intention (competitive), and became stable after VP changed its intention to adopt the same goal as the human (cooperative).

Correlation coefficients between the characteristics of the coordination and experimentally-manipulated factors are shown in Table 3. The actual coordination pattern is highly correlated with VP's intention, but not with the human subjects’, which verifies that the choice of strong coupling was successful (for reminder, see Section 2.2). Coordination stability is positively correlated with cooperativeness, and human subjects’ intention to coordinate inphase. The actual coordination pattern and the stability of the coordination are not correlated (r < 0.01, n.s.).

Table. 4.

Occurrence of SPRs by condition.

		During coordination	During subjective report
Humanness attribution	Human	286	413
	Computer	257	391
Cooperativeness	Cooperative	275	412
	Competitive	268	392
Stability	Stable	304	328
	Unstable	239	476
Human intention	Inphase	263	400
	Antiphase	280	404
Change of cooperativeness	Change	283	417
	Persistence	260	387
Change of stability	Change	225	347
	Persistence	318	457

3.3. Summary of the subjective report

In 51.8% trials, the subjects judged their partner as human, and in 48.2% trials as a computer program. Humanness is very weakly correlated with cooperativeness of the coordination (Phi = 0.057, p < 0.01), and not significantly correlated with other factors (p > 0.10). The subjects judged VP to be cooperative in 54.8% trials, and competitive in 45.2% trials. Subjects correctly judged VP as cooperative 84.3% of the time; correct judgements of competitiveness were 77.6%.

3.4. Factors affecting emotional arousal

In total, we found 543 SPRs during coordination, and 804 SPRs during subjective report (see Table 4 for details). We computed SPR Magnitude and applied MANOVA, the results of which are reported below.

3.4.1. Humanness, stable coordination associated with higher emotional arousal

During both coordination and subjective report, emotional arousal was higher when subjects thought that VP was controlled by a human (F(1,3029) = 13.7, p < 0.001; F(1,4536) = 7.62, p < 0.01), and when the coordination was stable (F(1,3029) = 16.3, p < 0.001; F (1,4536) = 10.7, p < 0.01) (Fig. 5). Other factors did not show a significant main effect on our measure of SPR magnitude (p > 0.05).

Fig. 5.

Main effects of MANOVA. Emotional arousal during coordination (A) and after coordination (during subjective report) (B) was significantly greater when subjects thought that VP was controlled by a human, and when the coordination was stable (shaded/colored in the figure). (**p < 0.01; ***p < 0.001.)

3.4.2. Factors modulating the effect of humanness attribution

A human-like partner elicited higher emotional arousal during subjective report than a computer-like partner when there was a change of cooperativeness during a trial (F(1,4536) = 10.7, p < 0.01; Fig. 6A), and when the coordination was persistently stable or persistently unstable across two periods of interaction (F(1,4536) = 4.74, p < 0.05; Fig. 6B).

Fig. 6.

Modulation of the effect of humanness attribution on emotional arousal during subjective report. (A) Interaction effect between change of cooperativeness and humanness attribution; (B) interaction effect between change of stability and humanness attribution. (*p < 0.05; **p < 0.01; ***p < 0.001.)

3.4.3. Factors modulating the effect of stability on emotions

Greater SPR magnitude was observed when subjects intended to coordinate inphase resulting in a stable coordination pattern (during coordination, F(1,3029) = 5.26, p < 0.05; during subjective report, F (1,4536) = 14.0, p < 0.001). Fig. 7A shows the corresponding multiple-comparison for emotional arousal during subjective report; similarly, for emotional arousal during coordination, “inphase + stable” related to greater emotional arousal than “inphase + unstable”, and “antiphase + unstable” at a significance level of p < 0.001. Greater SPR magnitude during subjective report was also observed for persistently stable coordination across the two periods of interaction (F(1,4536) = 25.4, p < 0.001; Fig. 7B).

Fig. 7.

Effect of stability of coordination on emotional arousal modulated by human intention and change of stability. (A) Interaction effect between human intention and stability of the coordination on emotional arousal during subjective report reveals that stable inphase (dash bar, right) aroused emotions more than the three other conditions. Similar interaction effects were also found for emotion during coordination (not shown, see section 3.4.3 text). (B) Interaction effect between change of stability and stability of the coordination in Coordination Period 2 during subjective report shows that persistently stable trials (dash bar, left) elicited greater emotional arousal than the three other conditions. (**p < 0.01; ***p < 0.001.)

4. Discussion

4.1. Emotional arousal and humanness attribution

In support of our initial hypothesis, subjects’ emotional arousal was greater when they perceived themselves interacting with a human rather than a computer, both during real-time interaction and afterwards when they reported their subjective experience. This suggests a tight connection between the perceived social nature of an interaction and human emotional responses. For example, in Pfeiffer et al. (2014), human participants engaged in gaze coordination with an anthropomorphic virtual agent, while fMRI was recorded. Similar to our paradigm, the participants were told that the gaze of a virtual agent could be controlled either by a computer algorithm or by a confederate sitting outside the scanner room. The participants’ gaze was reflected by a mirror and monitored by an eye-tracking device situated outside the scanner. The participant was led to think that his gaze was directed to the confederate via a virtual agent similar to his, so that they could interact reciprocally. In fact, the virtual agent displayed to participants was always controlled by an algorithm, and the number of joint-attention vs. non-joint attention responses of the virtual agent within each block was systematically manipulated. After each block, the participants were asked whether they were interacting with the other person or with a computer algorithm. When participants thought they were interacting with another human being, there was enhanced activation of reward-related neurocircuitry including ventral tegmental area (VTA), ventral striatum (VS) and the medial orbitofrontal cortex (mOFC). That is, social interaction per se can serve as reward, a highly plausible pathway to emotional experience. Emotion has social functions, e.g., as a means of communicating intention and feelings of individual members as well as signifying critical events in the environment (see Keltner and Haidt, 1999, for a review). Evidence suggests that emotional sensitivity toward a social (say conspecific) partner's behavior is fostered by accumulating interactions during development (Fogel et al., 1992; Camras and Witherington, 2005; Camras, 2011), and evolution (Darwin, 1872; Ekman, 1992; Berntson et al., 2012). The dynamics of movement, e.g. facial expressions and gestures, can serve as a medium for the detection and communication of social and emotional information (Blakemore and Decety, 2001; Meltzoff, 2005; Fogel et al., 1992).

Perceived humanness comes in several forms, e.g. anthropomorphic appearance, movement kinematics, coordination of movement within and between individuals. Attribution of humanness toward a computer-animated agent may originate in part from the dynamic features observed in the movement of the agent and its relationship to the perceiver's own movement. Dynamic figures with little anthropomorphic appearance can be perceived as human movement when certain kinematic features (speed, stiffness, and phase etc.) approach those of natural human movements (Morewedge et al., 2007; Thompson et al., 2011; Blake and Shiffrar, 2007; Johansson, 1973; Heider and Simmel, 1944). Similarity, the contingencies between a human and a virtual agent's movement also lead to humanness attribution to the virtual agent following their interaction (Pfeiffer et al., 2011; Ventrella et al., 2010). In the current study, VP's hand has anthropomorphic appearance—it looks human. VP's movement was based on a theoretical model that has been demonstrated to capture the kinematics of natural human movement. Central to the present paradigm, the reciprocal coupling of the VP with a human resembles human social coordination – the sociality of humanness. All of these factors contribute to subjects’ perception of the VP and VP motion as being produced by a human being. Robotics and artificial intelligence may benefit from kinematic and coordinative attributes that resemble natural human movement and interactivity in the design of socially efficient human-machine interfaces (Duffy, 2003).

Having a human-like partner is not the whole story about social interaction and emotion. Our results show that emotional responses toward a human-like partner was modulated by whether a change of cooperativeness occurred in a trial, which by experimental design was induced by a change of intention on the part of the VP. Specifically, when change of cooperativeness was initiated by a human-like partner, emotional arousal was greater than in all other conditions, i.e. whenever the VP was perceived as a computer and whenever there was no change of cooperativeness. Attribution of agency is associated with the lack of predictability and controllability of external events (Maselli and Altrocchi, 1969; Gilbert et al., 2000). Perceiving an event as a consequence of an act produced by a mindful human being intensifies the perceiver's psychological experience (Waytz et al., 2010; Heider, 1958), while emotion facilitates understanding of others’ intentionality during social interaction (Schilbach et al., 2013). Thus, in order for VP's parametrically controlled “intention” change to be perceived by a human as an intentional act and to affect his/her emotion, both the change of VP's intention and the humanness attribution are required.

Emotional arousal was also greater toward a human-like partner than toward a computer partner when the stability of the actual coordination did not change within a trial. Note that the stability of actual coordination may not change even when the cooperativeness of coordination changes. For example, a subject may engage in stable antiphase coordination with the VP, then after a change of cooperativeness, switch to stable inphase coordination. This means that persistently unstable or persistently stable coordination with a human-like partner elevates subjects’ emotional responses. Stability of social coordination may be considered a “trait-like” property that interacts with the perceived humanness of a partner; and its persistence over time leads to greater emotion. Schilbach et al. (2010) found that subjects, who were engaging in gaze coordination with a virtual agent, showed greater arousal when the subjects and the agent consistently looked at different objects on the screen (no joint attention, NOJA) than when they consistently looked the same object (joint attention, JA). Consistent gaze coordination (or consistent NOJA) are instances of persistently stable gaze-patterns, but in the case of Schilbach et al. (2010), there was no comparison with conditions in which gaze patterns did not persist. It would be interesting to conduct further experiments to ascertain the role of persistent coordination on emotion in various contexts.

4.2. Emotional arousal and cooperative vs. competitive interaction

How cooperativeness and competitiveness of social interaction relate to emotion was one of the chief motivations behind this study. In earlier studies (Kelso et al., 2009), subjects expressed frustration at VP's “messing with them” when subjects’ and VP's goals contradicted each other. With the techniques and parameters employed here to assess emotional arousal during social interactions (i.e. SPR and a “strongly willed” VP), we were not able to detect a direct, statistically significant, contribution of cooperative-competitiveness. Variations in the experimental protocols may have contributed to this result. In previous research, the VP-to-human coupling was rather weak, which introduced more intermittency into the coordination. In contrast, VP in the present study was strongly coupled to subjects, resulting in more stable, less intermittent coordination. Previous work relied on open-ended verbal report to assess subjects’ experience of the coordination and did not specifically evaluate emotion. SPR was employed in the present study to systematically record emotional arousal. The lack of significant contribution of cooperative-/competitiveness to emotion may be due to the combination of protocol and measurement differences between the previous and the present study. However, we did find that the changes of cooperativeness played a modulatory role in the relationship between humanness attribution and emotional arousal. It may well be that such changes played a role in the spontaneous verbal reports collected from previous work.

4.3. Emotional arousal, stable coordination and human intention

The present results demonstrate a strong relationship between emotional arousal and stability of the coordination. Consistent with our hypothesis, emotional arousal was greater when the human-VP coordination was stable. From the perspective of dynamic coordination (Kelso et al., 2013), this relationship may be bidirectional: stable coordination with a VP leads to greater emotional arousal, and in return, emotional arousal stabilizes social coordination. In general, stable interpersonal coordination or synchrony of movement has been associated with positive social affects (e.g. Miles et al., 2009; Lakens and Stel, 2011; Hove and Risen, 2009; Valdesolo and DeSteno, 2011; Launay et al., 2013, 2014). Between emotion and social interaction, both directions of influence have been studied. Temporal coordination of movement during social interaction has been shown to have a positive influence on the subsequent affective state of persons (Tschacher et al., 2014). Conversely, stable emotion can lead to stable social interaction patterns (Connell and Thompson, 1986), whereas changes in mood may contribute to the fluctuations in perceived social relationships (Hlebec and Ferligoj, 2001). Neural imaging studies also imply a bidirectional story: stable behavioral coordination promotes emotion (Fairhurst et al., 2013) and emotion promotes social interaction and interpersonal understanding (Nummenmaa et al., 2012). From a communication perspective, stable movement coordination or synchrony facilitates information exchange between persons (Marsh et al., 2009; Macrae et al., 2008; Oullier et al., 2008). It seems that emotion, reciprocating with social behavioral coordination, serves as a higher order facilitation or constraint on social information exchange. Healthy and effective social interaction and communication may depend on the reciprocal influence and the “co-development” of emotion and social behavioral coordination (Fogel et al., 1992; Camras and Witherington, 2005; Camras, 2011). Mental disorders, like schizophrenia or autism spectrum disorder (ASD; American Psychiatric Association, 2013), which implicate socio-emotional deficits, are also associated with unstable social and/or intrapersonal motor coordination (e.g. Teitelbaum et al., 1998; Varlet et al., 2012; Fitzpatrick et al., 2013; Marsh et al., 2009, 2013). Our results are consistent with previous research and suggest that the connection between social motor coordination and emotion is an important path to understanding social emotion and the treatment of socio-emotional disorders.

This link between emotional arousal and stable coordination is further modulated by people's intentions. Affective changes may occur during unintentional motor coordination (e.g. Hove and Risen, 2009), but here we found inphase but not antiphase intention enhanced the emotional arousal for stable behavioral coordination. Studies (e.g. Launay et al., 2013, 2014) showed that prosocial affects occur when subjects intentionally coordinate inphase with partners, but not when they intended antiphase. However, the intention to coordinate inphase versus actual inphase behavioral outcome is difficult to separate without a strongly coupled partner pursuing a different goal. In our experiments, the behavioral outcome was dominated by VP's intention due to strong coupling. Such parameter setting created a situation in which subjects intended inphase but the actual outcome of coordination was stably antiphase. In our study, actual inphase behavioral outcome alone showed no statistically significant contribution to emotional arousal. The intention of subjects to coordinate inphase with the VP, rather than the actual inphase behavioral outcome, seems to be the predominant factor modulating the relationship between emotional arousal and stability of the coordination. Emotional arousal seems to emerge from the interaction between what you want (inphase intention) and what you get (stable coordination) in a social interaction. Emotions are known to be deeply associated with intentions or action tendencies (Freeman, 2000; Ciompi, 2003) and are affected by their outcomes (Lewis and Todd, 2005; Higgins et al., 1997). Our results show that inphase intention alone (sans stable outcome) might not be sufficient to translate into emotion, which requires interaction with the ongoing behavior.

4.4. Time scales of the coordination-emotion relationship

The connection between emotional arousal and stable coordination, as well as the modulatory role of intention, appeared both during and following the coordination, i.e., when subjects reported their perception of the interaction based on the memory of what transpired. This suggests that the influence of stable social coordination on emotional arousal may extend beyond the ongoing interaction (Tschacher et al., 2014; Connell and Thompson, 1986). Emotion entails phenomena on multiple time scales. Emotions may last from a few seconds to more than a week, spanning from emotional episodes, to mood, and to personality development (Frijda et al., 1991; Lewis, 2000). Comparably, real life human social interactions also span different time scales ranging from moment-to-moment to developmental and cultural scales (Boiger and Mesquita, 2012). Our results showed that persistent stable coordination throughout the entire trial relates to high emotional arousal after the interaction, compared to all other conditions (i.e. the where coordination was either temporarily stable or always unstable). The time scale on which stable social coordination occurs was found to influence the level of emotional arousal. Such interplay between the time scale of social coordination and the time scale of emotion may be a key to understanding the long-term development of socio-emotional communication and social relationships. The present protocol can easily be extended to greater time scales in order to explore this question.

5. Conclusion

In this study we have developed a dynamical framework for the simultaneous and continuous study of emotion and movement coordination in dyadic social interaction. We were able to characterize some important features of the subtle interplay between motion and emotion in social coordination, bridging two important but often separated domains of study. The magnitude of emotional arousal is linked to the human-like kinematic and coordinative features of VP, and the stability of dyadic coordination. Links between emotion and motion manifest themselves on various timescales. We propose that a deeper investigation of emotion-motion coupling is a key to understanding human social behavior and socio-emotional development on both individual and collective levels. In future research, it would be valuable to incorporate measures of emotional valence into the current experimental paradigm and even consider demonstrable effects of attachment style (Vrtička and Vuilleumier, 2012). Different emotions can have the same arousal level but very different valence and behavioral effects (Faith and Thayer, 2001; Kreibig, 2010). With the additional dimension of valence, one can better characterize the dynamics of emotion, which may lead to further discrimination of various behavioral coordination patterns in relation to their emotional relevance. Electromyography (EMG) recordings may be a good candidate for obtaining information on emotional valence without interrupting continuous behavioral coordination (Cacioppo et al., 1986, 1988).

The Virtual Partner or Human Dynamic Clamp paradigm (Dumas et al., 2014; Kelso et al., 2009) allowed us to separate subjects’ intention from actual behavioral outcome, enabling us to study how their interaction influences emotion. Alteration of VP's ‘intention’ allowed us to parametrically control ongoing coordination and study its effects on emotional arousal. Such experimental manipulations are mainly due to the real time, quantitative control of coupling strength and intention (Dumas et al., 2014; Tognoli et al., in press) and constitute a first step to explore the “emotional landscape”.

Combining the Human Dynamic Clamp paradigm with measurements of emotion like electrodermal activity and EMG might also bring new insights to the concurrent dysregulation of emotion and social behaviors in schizophrenia (Brüne, 2005; Williams et al., 2004), autism (Kennedy et al., 2006), and patients with psychopathic personality (Herpertz et al., 2001). Emotional along with behavioral measurements can be fed back to VP so it may adjust its own parameters, thereby offering a means to stabilize or destabilize behavioral and emotional patterns. This avenue may lead to new therapeutic approaches to social emotional disorders through interaction between humans and machines. Beyond therapeutic purposes, our findings may also provide interesting information for developing human-machine interfaces—based on Coordination Dynamics—that allows affective communication and human-like, socially competent, artificial intelligence.

Table. 3.

Correlations (Phi Coefficients) between behavioral outcomes and experimental manipulations.

	Actual coordination pattern (1 = inphase, 0 = antiphase)	Stability (1 = stable, 0 = unstable)
VP's intention (1 = inphase, 0 = antiphase)	0.99**	0.01
Human's intention (1 = inphase, 0 = antiphase)	0.03	0.13**
Cooperativeness (1 = cooperative, 0 = competitive)	–0.01	0.28**

Text in the parentheses indicate the labels in the contingency tables (not shown), i.e. categories with the same label are on the diagonal of each contingency table. Significant differences are in bold.

^**p < 0.001.