Live interaction distinctively shapes social gaze dynamics in rhesus macaques

We examined social gaze patterns when dyads of monkeys explored the face of a conspecific and compared these results to picture and movie conditions. Interacting with a real partner evoked the strongest focal attention to the eyes and exclusively induced socially meaningful divergences in gaze dynamics. Our results indicate that certain aspects of social cognition may be exclusively engaged in interactive settings.

Abstract

The dynamic interaction of gaze between individuals is a hallmark of social cognition. However, very few studies have examined social gaze dynamics after mutual eye contact during real-time interactions. We used a highly quantifiable paradigm to assess social gaze dynamics between pairs of monkeys and modeled these dynamics using an exponential decay function to investigate sustained attention after mutual eye contact. When monkeys were interacting with real partners compared with static images and movies of the same monkeys, we found a significant increase in the proportion of fixations to the eyes and a smaller dispersion of fixations around the eyes, indicating enhanced focal attention to the eye region. Notably, dominance and familiarity between the interacting pairs induced separable components of gaze dynamics that were unique to live interactions. Gaze dynamics of dominant monkeys after mutual eye contact were associated with a greater number of fixations to the eyes, whereas those of familiar pairs were associated with a faster rate of decrease in this eye-directed attention. Our findings endorse the notion that certain key aspects of social cognition are only captured during interactive social contexts and dependent on the elapsed time relative to socially meaningful events.

NEW & NOTEWORTHY

We examined social gaze patterns when dyads of monkeys explored the face of a conspecific and compared these results to picture and movie conditions. Interacting with a real partner evoked the strongest focal attention to the eyes and exclusively induced socially meaningful divergences in gaze dynamics. Our results indicate that certain aspects of social cognition may be exclusively engaged in interactive settings.

social behaviors require dynamic and contingent processing of another's behaviors with respect to one's own behaviors, inherently implying that the action or reaction of an animal in response to an action of another agent changes constantly over time. In human and nonhuman primates, a substantial amount of social interaction takes place in the form of gaze interactions. When we encounter a person in daily life, we not only react to the other individual's gaze patterns but also have the opportunity to initiate social interaction and to observe reactions using our own gaze behavior (Emery 2000; Schilbach et al. 2013; Tomasello 1995). The important and unique qualities of social gaze patterns emerge through the reciprocal interplay of two or more agents whose unpredictable or partially predictable behaviors trigger continuous updates of cognitive processing (Baron-Cohen et al. 2013; Emery 2000; Shepherd 2010). In particular, social gaze patterns effectively capture underlying cognitive processes that are being engaged during social interactions (Emery 2000), and, accordingly, social gaze has been considered to reflect the state of social cognition or a “window into social cognition” (Baron-Cohen et al. 2013; Shepherd 2010). In human and nonhuman primates, processing gaze information of others is a principal component of social communication and interaction that allows identification of group members and social status, interpretation of facial signals, and formulation of appropriate behavioral responses (Emery 2000; Tomasello 1995). Furthermore, social gaze behaviors are atypical in many social dysfunctions, including autism (Dawson et al. 2004), psychopathy (Dadds et al. 2006), schizophrenia (Langdon et al. 2006), depression (Shean and Heefner 1995), and social phobia (Horley et al. 2003), endorsing the notion that social gaze processing lies at the core of intact social cognition.

To date, social gaze has predominantly been studied from an observational perspective with artificial stimuli, which has led to the paradoxical situation in which social cognition is studied without social interaction (Becchio et al. 2010). The stimuli used typically consist of controlled face images displayed on a computer monitor in the form of pictures (Dal Monte et al. 2014; Gothard et al. 2007; Guastella et al. 2008; Riby and Hancock 2009) or movies (Mosher et al. 2014). When presented with a picture or video of a face, both human and nonhuman primates indeed show a strong preference for attending to the eyes, the most socially informative feature of a face (Emery 2000). However, such presentations often cannot sufficiently capture rich behavioral contingencies between the observer and the observed subjects, challenging the capacity to generalize beyond simplistic laboratory paradigms to complex real-life social behavior.

Thus our understanding of social attention and social deficits has so far strongly depended on how individuals process two-dimensional social stimuli, like faces, that are presented on a monitor (Schilbach et al. 2013). Although most of these works are important and have increased our knowledge of social cognition, it is important to question whether such stimuli can fully capture the interactive nature of realistic social information processing. This view has been supported recently not only by neuroscientists but also by clinicians, who are beginning to characterize psychiatric disorders based on impairments in the dynamics of social interaction rather than in the perception and evaluation of artificial social stimuli (Schilbach 2016).

In recent years, several studies have begun to explore social attention in the context of real-life interactions (Freeth et al. 2013; Pfeiffer et al. 2013; Risko et al. 2012; Schilbach et al. 2013), motivated by the notion that artificial stimuli do not always accurately reflect social attention at work in natural settings (Foulsham et al. 2011; Laidlaw et al. 2011). For example, Laidlaw and colleagues (2011) reported that humans look less often at another individual physically present in the same room compared with the same person shown on a video, possibly because of a social norm dictating that one should not engage a stranger, who in this case was preoccupied with filling out a fake consent form. Gallup and colleagues (2012) showed that gaze-following behavior is strongly reduced by the presence of other potential social partners with whom one could interact. Furthermore, turn-taking behavior during conversations has been examined extensively to study social attention in the context of real-life interactions (Duncan and Fiske 2015). However, very few studies have recorded the eye movements of two interacting participants simultaneously during such conversations (Holler and Kendrick 2015; Macdonald and Tatler 2013; Sandgren et al. 2012). When eye positions were monitored during conversations from both participants simultaneously, the temporal aspects of social gaze emerged as a critical modulator of the ongoing exchanges (Ho et al. 2015). Nonetheless, no studies to date have examined at millisecond resolution the real-time interindividual gaze dynamics that occur spontaneously upon mutual eye contact. How these gaze dynamics are modulated by social relationship variables such as dominance and familiarity also remains unexplored.

Here we investigated social gaze dynamics between pairs of rhesus macaques while tracking eye positions from both monkeys simultaneously. We imposed no behavioral constraints, in order to capture spontaneously occurring gaze behaviors. Our first aim was to quantify interindividual gaze dynamics upon mutual eye contact, one of the most socially relevant events during gaze interactions. We then modeled these gaze patterns as an exponential decay process to characterize the changes in sustained attention over time. Moreover, we investigated whether and how social relationship variables such as dominance, familiarity, and sex drive differential interindividual gaze dynamics. Finally, informed by previous studies in human subjects (Foulsham et al. 2011; Laidlaw et al. 2011), we examined whether and how gaze contingency uniquely influences gaze behaviors compared with viewing pictures or prerecorded videos of the same other individuals. We demonstrate that social gaze patterns show unique features during live interaction and that social relationships impact dynamic gaze parameters after mutual eye contact only when interacting with a real partner. Overall, the present findings indicate that certain aspects of social gaze behaviors, and perhaps underlying social cognition, are distinctively manifested after interactive social situations in a time-dependent manner.

MATERIALS AND METHODS

Animals

Five adult rhesus monkeys (5–8 yr old) served as subjects. Animals (3 males, 2 females) weighed 5–13.5 kg and were housed together in a colony in either pairs or triads. Dominance relationships among all possible same-sex pairs were quantitatively determined by two independent measures, including the food-grab (Hosokawa and Watanabe 2012) and controlled-confrontation (Deaner et al. 2005) tasks. These measurements were additionally verified based on in-cage observations by experimenters blind to the outcomes. More vs. less familiar pairs were defined by whether or not the monkeys shared the same cage. All procedures were approved by the Yale Institutional Animal Care and Use Committee and were in compliance with the National Institutes of Health Guide for the Care and Use of Laboratory Animals.

Surgical Procedures

Before testing began, monkeys received a surgically implanted headpost (Crist Instruments, Grey Matter Research) for restraining their head while tracking eye positions. At the time of surgery, anesthesia was induced with ketamine hydrochloride (10 mg/kg im) and maintained with isoflurane (1.0–3.0%, to effect). Aseptic procedures were employed, and heart rate, respiration rate, blood pressure, expired CO₂, and body temperature were monitored throughout the procedure. Monkeys were allowed an additional 30–40 days of recovery after the implant surgery.

Behavioral Conditions

At each testing, animals sat in primate chairs (Precision Engineering) inside a testing room, head-restrained, and were randomly assigned to one of three conditions: picture, movie, or live. Each day included six sessions in which monkeys viewed a real conspecific, a movie, or one of a set of acquired still frames (3 direct and 3 averted gaze per day) for 3 min with a 3-min break between sessions in which monkeys could only see a black screen. Monkeys were free to look at the other monkey's face as well as to close their eyes or saccade away from the conspecific's face. For the picture and movie conditions, monkeys were exposed to unique videos or pictures in each session. Prior to data collection, we recorded videos from all animals, with their head restrained and with the same configuration, light, and background as in the live condition. We recorded 12 videos of 3 min for each animal. From those videos, we extracted 12 still frames (6 with direct and 6 with averted gaze). We cropped videos and pictures to match the size of each monkey's face. Before testing, each animal underwent a systematic calibration procedure. Horizontal and vertical eye positions were sampled at 1,000 Hz with an infrared eye camera (EyeLink, SR Research) mounted on the primate chair (located at the top of the chair pointing downward) to allow clear exploration of the faces of the conspecifics. In the live condition, two animals sat in front of each other while the gaze of each animal was recorded continually and simultaneously. Each EyeLink output contained a data stream of eye locations with additional information about the calibration, blinks, fixations, and saccades. In addition to the data originating from each eyetracker itself, the eyetracker outputs contained time-synching messages that were sent every 2 s to the eyetracker from the experimental software (MATLAB) stamped with the time they were received. We used these time stamps from the two EyeLinks to synchronize the eye position data from the two tested monkeys in the live condition. During the calibration and up until the beginning of each session, the two animals had no visual access to each other, with a screen fully separating the view of both animals. The screen was lifted at the start of each session, marking the beginning of live viewing.

Determining Dominance Within Pairs

We assessed the dominant-subordinate relationship between every possible pair of same-sex monkeys that participated in our behavioral tasks, using two independent measures taken on independent days: food grab (Hosokawa and Watanabe 2012) and controlled confrontation (Deaner et al. 2005). For the food-grabbing measures, we arranged two monkeys sitting in chairs face to face at 62 cm from each other with a table between them. We put a piece of fruit on the table at approximately the same distance from each monkey 20–30 times in a session for a total of two sessions for a given pair and counted the number of times that each monkey got the fruit. We then calculated a ratio of successful food retrievals between each monkey and identified the dominant monkey as the one that scored higher.

The configurations of the monkeys in the food-grab measures were identical to those in the controlled-confrontation measures. Monkeys were free to interact visually and vocally without the presence of an experimenter in the room while a camera recorded their head orientations for 10 min. We used Tinbergen (Geoffrey 2014), a custom-made software package, to quantitatively analyze the recorded video data. An ethogram approach was used to guide a frame-by-frame image analysis to score the time and frequency that each subject turned toward or away from the other monkey. Higher scores of orienting toward the conspecific identified the dominant monkey, while higher scores of orienting away identified the subordinate monkey.

While our monkeys were housed in the same colony room, some were kept in separate cages. This precluded us from using additional observational measures of dominance to determine a continuous linear relationship between all monkeys tested. Thus we decided to divide monkeys in a binary fashion on a pair-by-pair basis, defining one monkey as subordinate and the other dominant in each given pair. Importantly, for animals tested that were housed in the same cage, our food-grabbing and controlled-confrontation measures were consistent with data collected from observers blind to the outcomes of our behavioral tests.

Calibration Procedures

Before the start of recording of gaze behavior, each animal first underwent a systematic calibration procedure. A temporarily placed screen, 36 cm away from the subject's eyes, displayed five stimuli in different locations. Subjects were required to fixate on a specific point at a specific time to estimate the viewing angle. Stimuli were controlled by Psychtoolbox and Eyelink toolbox in MATLAB (Brainard 1997). The same procedure was repeated for the second animal (in the live condition) before testing began. For analysis purposes we then determined where the animal was looking on a theoretical screen further back from the actual screen on which the animal was calibrated. To accomplish this, we applied a correction to the locations of our regions of interest within the face of the conspecific rather than to the coordinates of every fixation. With a simple trigonometric approach, the regions of interest were identified and matched based on individual measurements of each monkey's face to the dimension of fixations as measured from the calibration screen. Fixations could then be mapped in detail across the face of the conspecific. We used the identical calibration correction for the picture and movie conditions to ensure that the differences we observed between the conditions were not due to artifacts of our correction procedure.

Data Analysis

Calculation of dominance indexes.

Dominance indexes were calculated from the food-grab test and controlled-confrontation test in order to determine pairwise dominance relationships among all possible same-sex pairs. First, individual indexes were calculated for each test. For the food-grab test, indexes were constructed as the ratio of trials in which a given monkey successfully obtained food at the cost of the other animal (Hosokawa and Watanabe 2012). For the confrontation test, the amount of time that each individual monkey oriented toward the conspecific was summed for each pair (Deaner et al. 2005). The proportion of time that each individual monkey in the pair oriented toward the conspecific was then divided by this value as an additional index of dominance. These two independently obtained indexes were averaged over 2 days of testing and then averaged with each other to give a single dominance index for each monkey in each pair such that the indexes of two monkeys in any given pair always summed to 1. The median dominance index for monkeys classified as dominant was 0.64, while the median dominance index for monkeys classified as subordinate was 0.36. Additionally, these two measures of dominance were significantly correlated [R² = 0.45, t(8) = 2.58, P = 0.03].

Dynamic gaze analyses.

To measure gaze dynamics after mutual eye contact, we identified mutual eye contact events in which both animals initiated eye contact within a window measuring 7.7° × 3.8° of visual angle, allowing a lag time of ±500 ms to account for the onset not being exactly simultaneous for a given pair. This temporal window accounted for the typical time lag for shifting gaze to the eyes across the two monkeys. That is, if one monkey was looking at the eyes of the partner monkey and the partner monkey in turn looked at the eyes of the other monkey within 500 ms, this instance would be counted as an instance of mutual eye contact. A 3-s window after each instance of mutual eye contact was examined to identify when and for how long the monkey looked back into the same eye region of the conspecific. The 3-s window was chosen empirically based on the observation that looking behavior returns to baseline for all groups of monkeys tested (dominant, subordinate, etc.), with no significant differences observed after the 3-s periods (see Figs. 3 and 5) in all conditions. For the picture condition, we identified mutual eye contact events in which the tested animals looked within the eye region of a picture that depicted the conspecific as gazing directly at the monkey. For the movie condition, we first used Tinbergen (Geoffrey 2014), a custom-made software package, to quantitatively analyze the recorded video data. Frame by frame, we first identified the times at which the monkey in the video looked toward or away from the camera. We then identified mutual eye contact events in which the tested animal looked within the eye region of the conspecific in the video when the latter was looking toward the camera. As in the live data, we allowed a lag time of ±500 ms. A control analysis was performed in which a 3-s window after all fixations to the eye region was isolated, regardless of the gaze of the conspecific (i.e., nonmutual eye contact). For both analyses, we used 10-ms bins and created binary data sets to characterize whether the animal was looking at the eye region within each bin. These data were summed across all instances of mutual eye contact for each individual within a unique pair on a given day. Data were then normalized for each monkey by dividing the values from each bin by the maximum value. Data were then averaged across given groups. Days in which pairs in the picture, movie, and live conditions were observed to make mutual eye contact in fewer than five instances were excluded from the dynamic gaze analysis.

Fig. 3.

Dominance-induced differences in gaze dynamics after mutual eye contact. A: proportion of looking at the eyes after mutual eye contact in the live condition for dominant (green) vs. subordinate (orange) monkeys (mean ± SE). Horizontal marks indicate 10-ms bins with significant differences (P < 0.05, green; P < 0.01, black, paired t-tests). B: proportion of looking at the eyes after nonmutual eye contact in the live condition for dominant vs. subordinate monkeys (mean ± SE). None of the 10-ms bins showed significance (P > 0.05, paired t-tests). C: proportion of looking at the eyes after mutual eye contact in the picture and movie conditions for dominant vs. subordinate monkeys (mean ± SE). None of the 10-ms bins showed significance (P > 0.05, paired t-tests). D: significance level of dominance-induced differences in looking at the eyes as the size of the eye region of interest (ROI) was progressively increased. Values indicate the maximum difference between dominant and subordinate monkeys after mutual eye contact subtracted by the P < 0.05 threshold value determined via a permutation test.

Fig. 5.

Sex-related variables do not drive differential social gaze dynamics after mutual eye contact. A: proportion of eye contact after mutual eye contact in the live condition for opposite (pink)- vs. same (yellow)-sex pairs (mean ± SE). B: proportion of eye contact after mutual eye contact in the movie condition for opposite- vs. same-sex pairs (mean ± SE). C: proportion of looking at the eyes after mutual eye contact in the live condition for male-male (blue) vs. female-female (red) pairs (mean ± SE). D: proportion of looking at the eyes after mutual eye contact in the movie condition for male-male vs. female-female pairs (mean ± SE). None of the curves showed significance in A–D (all P > 0.05, paired t-tests and permutation tests).

To analyze the dynamics based on dominance, familiarity, and sex, we first performed t-tests to compare values at all bins within the 3-s window after mutual eye contact. Windows of activity were defined in which gaze behavior between the two groups diverged (750-1,250 ms for dominance and 300–500 ms for familiarity) or took place over the entire 3-s window if no divergence was observed (0–3,000 ms for sex). Significant differences within these windows were determined with a permutation test by shuffling the data 1,000 times and randomly assigning the data from each monkey an identity in one of the two groups being compared in each condition. The data were then averaged and compared, with a discrete value being obtained as the maximum difference between groups in some bin within the defined window. The permutated values were then sequentially ordered for determining threshold values for significance. To test the sensitivity of the gaze dynamics to a narrowly defined eye region, we progressively increased the region to larger sizes (step size of 0.69° × 1.11° of visual angle for 7 steps) only for defining whether or not animals looked back to the eyes of a conspecific (the definition of mutual eye contact remained identical). We then repeated the same analyses mentioned above for every discrete size of the eye region.

Other discrete measures were also used to compare groups around the time of mutual eye contact. First, the initial time it took for each monkey to look back to the eye region of the conspecific for the first time after mutual eye contact (i.e., first-returning) was averaged for each monkey in a unique pair on a given day. As a control, the average time between the immediately previous fixation to the conspecific's eyes and an instance of mutual eye contact (i.e., before-contact) was also determined. The average dispersion of fixations around the eyes of the first-returning fixations and before-contact fixations were also calculated. These values were then compared across groups with paired-sample or two-sample t-tests.

Fixation proportion, dispersion of fixations, and duration analyses.

We first collapsed across all social variables (dominance, familiarity, and sex) to assess overall differences between the picture, movie, and live conditions. Fixations to the defined eye region were first obtained for each monkey, as were fixations to the face and total fixations in general. Fixations to the eye region were normalized by dividing the total number of fixations to the eyes by the total number of fixations to the face (Patton and Lefton 1985). This was done for each monkey in each individual pair and then averaged. The normalization process was identical for the picture, movie, and live conditions. These normalized values were compared with a one-way ANOVA (with condition as a factor) and subsequent multiple-comparison tests (Tukey-Kramer post hoc test). To determine the dispersion of fixations around the eyes, we first defined the dispersion of fixations as the average two-dimensional distance of all fixations to the face of the conspecific from the center of the eye region (Coull et al. 1992). This value was obtained for each monkey in each pair and compared across conditions with an ANOVA (with condition as a factor) and subsequent multiple-comparison tests (Tukey-Kramer post hoc test). Finally, the average duration of fixations to the eyes of a conspecific was calculated for each monkey in each pair, and these values were also compared with a one-way ANOVA (with condition as a factor) and subsequent multiple-comparison tests (Tukey-Kramer post hoc test). We used paired and two-sample t-tests to compare proportion, dispersion of fixations, and duration measures for monkeys separated on the basis of dominance, familiarity, and sex. Fixation proportion, dispersion of fixations around the eyes, and duration were all also analyzed over time within sessions and over sessions within days. Specifically, fixations were broken into 30-s bins within each session and averaged or averaged based on session number across days, then further averaged for comparison across the picture, movie, and live conditions. The data were then analyzed with a two-way ANOVA with condition and time as factors.

Data visualization using heat maps.

Fixation coordinates to the faces of conspecific monkeys were determined with the correction procedure described above. Heat maps were then plotted for each session based on fixations specifically to the face of the conspecific with the EyeMMV toolbox in MATLAB. Heat maps were based on average number of fixations, normalized within each condition to arbitrary values ranging from 0 to 255. Image dimensions were defined as 340 × 340 arbitrary units to align to the fixation data and binned at 20-unit intervals. Smoothing was accomplished with a Gaussian filter function in MATLAB with a kernel size of 5 and standard deviation 3. Fixations to the face of the conspecific were then averaged for all monkeys in a given condition and overlaid on a representative image of the face of one particular monkey. Monkeys were also separated into groups based on dominance index and whether they were more or less familiar with a given partner. Fixations were averaged within these groups for the picture, movie, and live conditions and plotted on a representative image of the monkey face. Movies of fixations (Supplemental Movies S1–S3) were constructed by breaking fixations into 500-ms time bins over the 3-min session period and averaging for all monkeys across all sessions for the picture, movie, and live conditions.¹ Fixations from each bin were plotted and displayed for 0.1 s sequentially.

Modeling of dynamic gaze curves.

To determine whether the differences in observed gaze dynamics across dominance and familiarity could be distinguished by modeling the data on both the individual and group levels, we utilized MATLAB to fit curves with the appropriate model, focusing on three similar but distinct possibilities based on our evaluation of the overall shape of the peristimulus time histogram (PSTH) dynamic gaze curves (see Fig. 2D).

Fig. 2.

Gaze dynamics across the picture, movie, and live conditions. A: raster plots of fixations to the eyes of a conspecific from a single session for 2 female monkeys in the live condition. Red marks show points of mutual eye contact (vertical lines) and their durations (lengths of the bar extending from the vertical lines) between the 2 monkeys. Green and orange marks denote the times and durations of looking at the eyes of the partner for each monkey. The sums of fixations to the eyes for the given session are shown on right. B: proportion of looking back to the eye region after mutual eye contact in the live condition averaged over all sessions in a given day for the same pair shown in A. The dominant (green) and subordinate (orange) animal in the pair are labeled appropriately. C: heat maps showing the amount of fixations to various regions of the face after mutual eye contact in the live condition; overlaid on an example monkey face. Fixations are plotted based on their normalized frequency. D: mean (±SE) proportion of eye contact after mutual eye contact in the picture, movie, and live conditions averaged over all monkeys.

First, we considered a single-term exponential (Exp₁) decay function:

$$\text{Social}\hspace{0.17em}\text{Attention}\hspace{0.17em}(t)=a{e}^{bt}$$ (1)

where t is time after mutual eye contact, Social Attention(t) is the attention directed at the eyes at time t, a is a parameter describing the gain of the function, and b is a parameter describing the rate of decay of the function. This model conceptualizes the decay of social attention after mutual eye contact as a time-consistent process occurring at one constant decay rate. Thus, although this model could explain the initially high rate of decay, our PSTH curves displayed two relatively discrete decay rates, the first relatively high and the second relatively low. Thus we also subjected the data to the two-term exponential (Exp₂) decay function to attempt to characterize both potentially discrete steps of decay:

$$\text{Social}\hspace{0.17em}\text{Attention}\hspace{0.17em}(t)=a_1{e}^{b_{1}t}+a_2{e}^{b_{2}t}$$ (2)

where a₁ and a₂ are gain parameters and b₁ and b₂ are decay parameters. In our model, a₁ describes the relative gain of the rapid decrease in looking in the first ∼500 ms after mutual eye contact while a₂ describes the gain over the entire curve. These values interact such that when a₁ is decreased and a₂ is increased the gain of the entire function is shifted upward while the initial decay becomes shallower. The decay parameters b₁ (defined as the rate constant with the larger absolute value) and b₂ (defined as the rate constant with the smaller absolute value) more intuitively describe the initial high rate of decay and later low rate of decay, respectively. This model conceptualizes decay of social attention after mutual eye contact as two time-consistent steps with discrete rates of decay. Finally, we considered a hyperbolic decay (Hyp) function:

$$\text{Social}\hspace{0.17em}\text{Attention}\hspace{0.17em}(t)=a\left(\frac{1}{1+bt}\right)$$ (3)

where a is again the gain parameter and b is again the rate parameter. This model conceptualizes the decay of social attention after mutual eye contact as a time-inconsistent process.

Cross-validation procedures were used to determine goodness of fit and parsimony for each of the three models. The data were randomly permutated 5,000 times into two equal groups. The data from the first half of each group were averaged and fit with each of the three potential models. Each of the three fitted models was then applied to the second half of the data, and the root-mean-square error (RMSE) was calculated. The average RMSE values over the 5,000 permutations were compared with a paired t-test, and the model with the lowest RMSE was determined to be superior. This procedure allowed us to compare the RMSE of our three competing models independent of their relative complexities (Hampton et al. 2008; Stone 1974). These findings were then independently validated with a more traditional leave-one-out cross-validation approach, which was found not to change the results.

The fitted parameters among dominant and subordinate animals as well as among more and less familiar animals were compared to one another depending on their calculated 95% confidence intervals. We then applied the model individually for all monkeys in both the dominance and familiarity comparisons. All parameters from fits with R² > 0.4 were then averaged and compared for both dominant vs. subordinate and more vs. less familiar monkeys with a two-sample t-test and a permutation test for validation, in which the data were scrambled relative to the relevant social variable.

RESULTS

Dominance- and Familiarity-Induced Differences in Social Gaze Dynamics Are Unique to Live Gaze Interaction

We continuously tracked gaze from 10 pairs of behaving rhesus macaques facing each other without imposing any task constraints (live condition). Since each pair consisted of two behaving monkeys with gaze positions from both recorded continuously and simultaneously, this allowed for an effective sample size of 20 unique behavioral perspectives. These unique perspectives consisted of 8 dominance-related, 20 familiarity-related, and 20 sex-related pairs. Two days of data were collected for each pair, with six sessions of 3 min/day. This resulted in a total of 120 sessions of data. To compare gaze patterns in this live context with gaze patterns from more traditional paradigms, we also included picture and movie conditions, again collecting 120 sessions of 3 min for each condition. In the picture and movie conditions monkeys viewed the same partners, but these partners were presented instead in picture or movie form (Fig. 1).

Fig. 1.

Experimental design. A: setups for live, picture, and movie conditions. In all cases, monkeys were calibrated to a screen in front of the plane of interest. Monkeys looked at a real conspecific, a picture of the same conspecific, or a movie of the conspecific. B: schematics of the food-grabbing and controlled-confrontation tests used to determine dominance. More vs. less familiar monkeys were identified based on sharing a cage within the same colony (right).

Dynamic and contingent gaze exchanges are a hallmark of social interactions (Jarick and Kingstone 2015). Here we compared interindividual gaze dynamics across the live interaction condition and more traditional picture and movie conditions. To focus on arguably the most socially relevant time periods during gaze interaction, we identified instances of mutual eye contact (Fig. 2A). These events were identified in all three conditions in which monkeys made mutual eye contact with a live partner, a movie partner, or a picture partner. We then aligned the data to these events, taking a 3-s window after each instance of mutual eye contact and characterizing whether or not the animal looked back to the eye region of the conspecific. A PSTH analysis was used to characterize average gaze behavior in this time period over a given day of testing (see materials and methods). The PSTH approach is ideal for a high-resolution temporal analysis aligned to specific behavioral events. This method can depict changes over time and allows a higher level of granularity in the analyses of social gaze interactions, providing novel insight into the relationship between two subjects' mutual eye contact in a real-life setting. An example PSTH from a single pair of monkeys over 2 days of testing is shown in Fig. 2B. This plot demonstrates that meaningful data can be obtained by averaging over all instances of mutual eye contact in a given day and that these data are reproducible over multiple days. Figure 2C shows fixations after mutual eye contact averaged for all monkeys in the live condition. Qualitatively, fixations were highly clustered around the eye region of the conspecific relatively early in the 3-s time window but seemed to diverge to other areas of the face as time progressed. Correspondingly, the shape of the PSTHs followed a stereotypical decay function, with higher values for eye fixations appearing quite early that dissipated nonlinearly over the course of the 3 s. The general shape of this curve was remarkably conserved across conditions (Fig. 2D), with an initial (200–600 ms after mutual eye contact) average decay rate of 0.33 fix/ms² across all conditions, describing a common decay process of social attention following a specific social gaze event. Note that the selected 3-s window was informed by these average curves, as fixations to the eye region seemed to reach a stable baseline by 3 s after mutual eye contact in all conditions.

We next asked whether differences due to social variables could be determined in this analysis and whether these differences were specific to the live condition after mutual eye contact. Since our data set consisted of 10 pairs of monkeys and 20 unique behavioral perspectives for all experimental conditions, we were able to separate monkeys within our sample based on dominance and familiarity (see materials and methods). We then performed our interactive analyses for each monkey in a given group, produced an average PSTH curve for each group, and compared curves across groups to determine whether significant differences were observed. As a control, we repeated the same analyses by aligning the data in the live condition based on nonmutual eye contact (i.e., only one monkey looked at the eyes at a given time; materials and methods).

For dominance analyses, we grouped animals based on whether they were dominant or subordinate in a given same-sex pair (4 unique pairs, 4 subordinate perspectives, 4 dominant perspectives, a total of 48 sessions). In the live condition, dominant monkeys over time reoriented their attention back to the eyes more frequently than subordinate monkeys after mutual eye contact at various time points throughout the 3-s window (Fig. 3A; both P < 0.05, paired t-tests and permutation test). By contrast, after nonmutual eye contact, differences between dominant and subordinate were not significant at any time point (Fig. 3B; all P > 0.05, paired t-tests and permutation test), indicating that the dominance-induced differences in gaze dynamics were evoked by mutual eye contact and manifested over various time points after this event. Notably, we did not observe such differences due to mutual eye contact in the picture and movie conditions, perhaps because of the absence of true behavioral contingency (Fig. 3C; all P > 0.05, paired t-tests and permutation tests). To more rigorously determine the level of specificity of the dominance effects on gaze dynamics to the eye region, we progressively increased the size of the eye region when defining whether or not animals looked back at the conspecific's eyes while keeping the size constant for defining instances of mutual eye contact. The dominance-driven differences were highly sensitive to the size of the eye region, as differences were lost at middling eye region sizes (Fig. 3D; P < 0.05, permutation test). These results indicate that the differences observed in social gaze dynamics were specific to a narrowly defined eye region bounded by the width of the eyes themselves, suggesting that dominance shapes gaze dynamics with respect to focal attention to the eyes distributed over time relative to the time of mutual eye contact.

For familiarity, we grouped more and less familiar monkeys as evaluated by whether or not they shared the same cage (10 unique pairs, 4 more familiar perspectives, 16 less familiar perspectives, a total of 120 sessions). Familiarity-driven differences in social gaze dynamics were also unique to the live condition. Less familiar monkeys looked more frequently at the eyes in the live condition during an early period (300–500 ms) after mutual eye contact (Fig. 4A; both P < 0.05, 2-sample t-tests and permutation test). Similar to the dominance effect, these differences were specific to mutual eye contact, as no differences were found at any time point after nonmutual eye contact, and we did not observe such differences due to mutual eye contact in the picture and movie conditions (Fig. 4, B and C; all P > 0.05, 2-sample t-tests and permutation tests). These familiarity effects were again highly specific to the narrowly defined eye region and were lost for even slight increases in eye region size (Fig. 4D; P < 0.05, permutation test), again indicating that social variables shape gaze dynamics through focal attention to the eyes. Although significant differences in our analyses were verified with permutation tests known to be robust for imbalanced sample size between compared groups, we were concerned that the imbalance in our sample could be driving our familiarity results. Opposite-sex pairs were enriched in the less familiar group, which could also contribute to our results. To control for these factors, we performed the same analyses independently using only same-sex pairs, as was done for the dominance analyses. Not only did this eliminate all opposite-sex pairs, but it also balanced our sample (4 unique pairs, 4 more familiar perspectives, 4 less familiar perspectives, a total of 48 sessions). Crucially, a significant difference between more and less familiar monkeys persisted in the live condition (P < 0.05, permutation test), while nonmutual eye contact, picture, and movie conditions still did not exceed significance (all P > 0.05, permutation tests). In addition, we grouped monkeys based on whether they were in same- or opposite-sex pairs (10 unique pairs, 8 same-sex perspectives, 12 opposite-sex perspectives, a total of 120 sessions). These groups did not differ in their gaze interaction in any condition (Fig. 5, A and B; all P > 0.05, 2-sample t-tests and permutation tests). We also compared same-sex male and female pairs (4 unique pairs, 2 female perspectives, 6 male perspectives, a total of 48 sessions), again finding no significant differences (Fig. 5, C and D; all P > 0.05, 2-sample t-tests and permutation tests). These additional analyses indicate that our findings were not dependent on imbalanced sample size or enrichment of opposite-sex pairs among less familiar monkeys. Together, these results indicate that less familiar monkeys looked at the eyes of the conspecific to a greater extent than more familiar monkeys, but these familiarity effects on gaze dynamics were restricted to a relatively early period after mutual eye contact.

Fig. 4.

Familiarity-induced differences in gaze dynamics after mutual eye contact. A: proportion of looking at the eyes after mutual eye contact in the live condition for more (blue) vs. less (purple) familiar monkeys (mean ± SE). Same format as Fig. 3. B: proportion of looking at the eyes after nonmutual eye contact in the live condition for more vs. less familiar monkeys. C: proportion of looking at the eyes after mutual eye contact in the picture and movie conditions for more vs. less familiar monkeys (mean ± SE). No time points showed significance in B and C (P > 0.05, paired t-tests). D: significance level of familiarity-induced differences in looking at the eyes as the size of the eye region of interest is progressively increased. Values indicate the maximum difference between more vs. less familiar monkeys after mutual eye contact subtracted by the P < 0.05 threshold value determined via a permutation test.

We used two additional metrics of social attention to further quantify gaze dynamics after mutual eye contact in the live condition, including the initial time to make a saccade to the eyes after mutual eye contact and the distance of the first fixation after mutual eye contact from the eyes of the other monkey. Dominant monkeys took a significantly lower amount of time to return to the eyes after mutual eye contact (“first-returning”; materials and methods) [Fig. 6A; t(14) = 2.14, P = 0.05, paired-sample t-test]. Moreover, the average dispersion of fixations around the eyes during the first-returning period was significantly lower in dominant compared with subordinate monkeys [Fig. 6B; t(14) = 2.34, P = 0.03, paired t-test]. By contrast, we did not observe any differences in these measures for the time from the last fixation to the eyes immediately before mutual eye contact (“before-contact”; materials and methods) [Fig. 6, A and B; time, t(14) = 1.12, P = 0.28, paired-sample t-test; dispersion of fixations around the eyes, t(14) = 0.37, P = 0.72, paired t-test]. While no familiarity-driven difference in the live condition was observed for the time it took until the first-returning fixation after mutual eye contact [Fig. 6C; t(34) = 0.59, P = 0.56, 2-sample t-test], these fixations were more clustered around the eyes for less familiar monkeys [Fig. 6D; t(34) = 2.31, P = 0.03, 2-sample t-test]. Corresponding to earlier analyses, no significant differences were observed between male-male and female-female pairs in any of these measures (Fig. 6, E and F; all P > 0.05, 2-sample t-tests). Together, these results indicate differences in social gaze dynamics due to distinct social variables that were specific to the period following, but not preceding, mutual eye contact when interacting with a real partner.

Fig. 6.

Time to look back at the eyes and dispersion of fixations around the eyes immediately after mutual eye contact differ based on dominance and familiarity but not sex. A, C, and E: time to return to eyes after mutual eye contact as well as time between the last fixation to the eyes and mutual eye contact for dominant vs. subordinate (A), more vs. less familiar (C), and male vs. female (E) monkeys (mean ± SE). B, D, and F: dispersion of fixations around the eyes immediately after and immediately before mutual eye contact for dominant vs. subordinate (B), more vs. less familiar (D), and male vs. female (F) monkeys (mean ± SE). *P < 0.05, paired or 2-sample t-tests.

Live Gaze Interaction Elicits Most Focal Attention to the Eyes

We next explored overall differences in looking behavior between the three experimental conditions (picture, movie, live). Supplemental Movies S1–S3 show the averaged fixation locations over time of all monkeys in the three conditions. Visual inspection of the movies suggests that the live condition was associated with a higher number of fixations clustered around the eye region compared with the picture and movie conditions. Heat maps showing the averaged fixation locations on the conspecific's face also faithfully reflected this description (Fig. 7A). To quantify these effects, we calculated the proportion of fixations to the eyes by normalizing it to the number of fixations to the entire face. We then compared the proportion of fixations to the eyes within each condition. This proportion was significantly higher in the live compared with picture or movie condition [Fig. 7B; F(2,38) = 14.77, P < 0.001 for both picture and movie comparisons, 1-way ANOVA with Tukey-Kramer post hoc test]. Furthermore, the dispersion of fixations from the eyes averaged over all fixations was significantly lower in the live than in the picture or movie condition [Fig. 7B; F(2,38) = 14.66, P < 0.001 for both picture and movie comparisons, 1-way ANOVA with Tukey-Kramer post hoc test], indicating more focally distributed attention to the eyes when interacting with a real partner. Importantly, this clustering pattern of eye-directed attention could not be explained by a simple convergence on a Gaussian distribution, as the number of fixations to the overall face was actually highest in the movie condition [Fig. 7C; F(2,38) = 5.76, P < 0.05 for both picture and live comparisons, 1-way ANOVA with Tukey-Kramer post hoc tests]. These results indicate that the differences in the dispersion of fixations from the eyes between conditions were not driven by a general increase in saliency associated with the overall face of a real-life partner but rather related to a specific increase in the saliency of the eyes. The average duration of fixations was also sensitive to interacting with a real-life partner, in that the live condition was associated with the lowest average fixation duration to the eyes [Fig. 7B; F(2,38) = 14.01, P < 0.001 for both picture and movie comparisons, 1-way ANOVA with Tukey-Kramer post hoc tests].

Fig. 7.

Live condition is associated with the strongest focal attention to the eyes. A: heat maps showing the average and normalized fixation locations on the conspecific's face of all monkeys in the 3 conditions; overlaid on an example monkey face. B: differences in proportion of fixations to the eyes normalized by dividing with total fixations to the entire face, dispersion of fixations around the eyes as measured by the 2-dimensional distance from the eye region of interest, and average duration of fixations to the eye region of interest (right) for the picture, movie, and live conditions (mean ± SE). C: raw fixation counts to the eyes, face, and any location (right) for the picture, movie, and live conditions (mean ± SE). *P < 0.05, 1-way ANOVA with Tukey-Kramer post hoc tests; **P < 0.01, 1-way ANOVA with Tukey-Kramer post hoc tests.

The proportion of fixations to the eyes as well as the dispersion of fixations and the duration of these fixations were qualitatively similar across both times within a given session and sessions over a given day for all conditions (Fig. 8). However, while the proportion of fixations and dispersion of fixations around the eyes of the conspecific remained relatively constant over time within sessions and across sessions (Fig. 8, A, B, D, and E; all P > 0.05, effect of time in 2-way ANOVA within sessions and across sessions over a given day for proportion of fixations and dispersion of fixations around the eyes), the average duration of fixations to the eyes of a conspecific increased over time within individual sessions and across sessions within a given day of testing for all conditions [Fig. 8, C and F; F(5,35) = 2.48, P = 0.03, effect of time in 2-way ANOVA within sessions; F(5,35) = 3.29, P = 0.006, effect of time in 2-way ANOVA over sessions within a given day]. Corresponding to our previous analyses, a significant main effect of condition was observed for proportion, dispersion of fixations, and duration of fixations to the eyes of a conspecific (Fig. 8; all P < 0.005, effect of condition in 2-way ANOVA). Together, these results indicate that attention to the eyes of the conspecific was sustained both within and across sessions in all three conditions. However, the overall magnitude of attention to socially relevant areas was consistently highest in the context of live interaction.

Fig. 8.

Proportion, dispersion, and duration of fixations to the eyes of a conspecific are sustained within and across sessions in the picture, movie, and live conditions: proportion (A), dispersion (B), and duration (C) of fixations to the eyes averaged over 30-s time bins as well as proportion (D), dispersion (E), and duration (F) of fixations to the eyes averaged over each of 6 sessions in a given day in the picture (green), movie (red), and live (blue) conditions (means ± SE).

Differences in Gaze Interaction After Mutual Eye Contact Are Only Partially Recapitulated in a Time-Averaged Analysis

Importantly, these time-dependent results concerning social variables can only be partially recapitulated in static (i.e., time averaged) analyses using overall proportion, dispersion of fixations, and duration of fixations to the face without taking into account an interactive event such as mutual eye contact. In our time-averaged analyses dominant monkeys looked at the eyes of subordinate monkeys more frequently, and this stereotypical behavior was most dramatic in the live condition [Fig. 9, A and C; t(14) = 2.14, P = 0.05, paired-sample t-test]. Fixations from dominant monkeys were also more clustered around the eyes compared with those from subordinate monkeys in the live condition, as indicated by a significantly lower dispersion of fixations from the eyes in dominant monkeys [Fig. 9, A and C; t(14) = 2.19, P = 0.05, paired-sample t-test]. Finally, eye fixation duration was also higher in dominant animals in the live condition [Fig. 9C; t(14) = 2.41, P = 0.03, paired-sample t-test]. These differences were largely absent in the picture and movie conditions [Fig. 9, A and C; all P > 0.05, paired-sample t-tests], indicating that the differences in dominance-driven social gaze were most robust, or at least most reliably detected, in the live condition.

Fig. 9.

Time-averaged dominance effects of social gaze are most pronounced during live interaction. A: heat map of fixations (normalized) taken from a single day in the live condition from dominant and subordinate female monkeys; overlaid on an example monkey face. B: heat maps showing the averaged fixation location of dominant and subordinate monkeys in the picture, movie, and live conditions. C: differences in proportion of fixations to the eyes normalized by dividing with total fixations to the entire face, dispersion of fixations around the eyes as measured by the 2-dimensional distance from the eye region of interest, and average duration of fixations to the eye region of interest (right) between dominant (green) and subordinate (orange) monkeys for the picture, movie, and live conditions (means ± SE). *P < 0.05, paired t-test.

Although these trends seem to correspond to those observed in the dynamic analyses of social gaze after mutual eye contact, trends diverged for other social variables including familiarity and sex for the time-averaged data. Familiarity generally reduced exploration of the eyes, possibly reflecting more efficient social information foraging when interacting with more familiar partners. However, these effects were neither significant when analyzed individually within each condition nor specific to the live context (Fig. 10, A and C; P > 0.05 for comparisons of proportion and dispersion of fixations and duration for all conditions, 2-sample t-tests). In the live condition, female monkeys looked substantially more at the eyes of a same-sex conspecific than male monkeys in terms of the proportion of fixations to the eyes [Fig. 11, A and C; t(14) = 4.20, P = 0.0009, two-sample t-test] and a smaller dispersion of fixations around the eyes [Fig. 11, A and C; t(14) = 3.59, P = 0.003, 2-sample t-tests]. Notably, the sex-related differences in social gaze patterns were present but less pronounced in the movie condition [Fig. 11, A and C; t(14) = 2.94, P = 0.01, 2-sample t-test for proportion; t(14) = 2.33, P = 0.04, 2-sample t-test for dispersion of fixations around the eyes] and absent in the picture condition [Fig. 11, A and C; t(14) = 1.07, P = 0.30, 2-sample t-test for proportion; t(14) = 0.08, P = 0.94, 2-sample t-test for dispersion of fixations around the eyes]. However, no differences were observed for duration of fixations for female vs. male comparisons for any condition (Fig. 11C; P > 0.05 for comparisons of duration for all conditions, 2-sample t-tests). Additionally, opposite-sex pairs trended to display a lower proportion and a greater dispersion of fixations around the eyes similarly in all conditions relative to same-sex pairs, although these values were neither significant nor specific to the live condition (Fig. 11D; P > 0.05 for comparisons of proportion and dispersion of fixations around the eyes and duration for all conditions, 2-sample t-tests). Together, these results indicate that our previous dynamic analysis of mutual eye contact captured gaze patterns that could not be detected with more traditional analytical techniques measuring average gaze behavior across entire sessions. Still, data from these more traditional analyses suggest that the live condition elicited the most modulation of overall gaze behavior based on social variables compared with picture and movie conditions.

Fig. 10.

Time-averaged familiarity effects of social gaze are not specific to any condition. A: heat map of fixations (normalized) taken from a single day in the live condition from more and less familiar monkeys; overlaid on an example monkey face. B: heat maps showing the averaged fixation location of more and less familiar monkeys in the picture, movie, and live conditions. C: differences in proportion of fixations to the eyes normalized by dividing with total fixations to the entire face, dispersion of fixations around the eyes as measured by the 2-dimensional distance from the eye region of interest, and average duration of fixations to the eye region of interest (right) between more (blue) and less (purple) familiar monkeys for the picture, movie, and live conditions (means ± SE). Difference between more and less familiar monkeys were neither significant nor specific to the live condition (P > 0.05 for comparisons of proportion, dispersion, and duration for all conditions, 2-sample t-tests).

Fig. 11.

Time-averaged sex effect of social attention is enhanced in females and in the live condition. A: heat map of fixations taken from a single day in the live condition of a female looking at another female and of a male looking at another male; overlaid on an example monkey face. Fixations are plotted based on their normalized frequency. B: heat maps showing the averaged fixation location of males looking at other male monkeys and females looking at other female monkeys in the picture, movie, and live conditions. C: differences in proportion of fixations to the eyes normalized by dividing by total fixations to the entire face, dispersion of fixations around the eyes as measured by the 2-dimensional distance from the eye region of interest, and average duration of fixations to the eye region of interest (right) between males looking at male monkeys (blue) and females looking at female monkeys (red) for the picture, movie, and live conditions (mean ± SE). D: differences in proportion of fixations to the eyes, dispersion of fixations around the eyes, and average duration of fixations to the eye region of interest (right) between opposite (pink)- vs. same (yellow)-sex pairs for the picture, movie, and live conditions (mean ± SE). *P < 0.05, 2-sample t-test; **P < 0.01, 2-sample t-test.

Decay Patterns of Sustained Social Attention After Mutual Eye Contact Follow an Exponential Decay Process

The gaze patterns after mutual eye contact strongly resembled an exponential decay process (Fig. 2, B and D). Furthermore, dominance and familiarity differentially impacted the shape of these after mutual eye contact (Figs. 3 and 4). We therefore recapitulated the gaze patterns using three models describing a decay process in order to first confirm the fits to the data as well as to determine how the dominance- and familiarity-induced differences in gaze dynamics were manifested in the model after mutual eye contact. We modeled the decay of social attention to the eyes as a single-term exponential decay (Exp₁) function (Eq. 1), a two-term exponential decay (Exp₂) function (Eq. 2), and a hyperbolic decay (Hyp) function (Eq. 3), informed by the common shapes of the PSTHs after mutual eye contact (Fig. 2D; see materials and methods for models and analyses). When these models were tested against each other with a cross-validation procedure, the Exp₂ model was determined to be superior relative to Exp₁ and Hyp models (Fig. 12A; both P < 0.0001, paired t-tests). Notably, when the Exp₂ model was applied to the population, differences in dominance vs. familiarity drove robustly diverging effects (Fig. 12, B and C). Using 95% confidence intervals (Exp₂, least squares), the gain parameters a₁ and a₂ were found to be different in dominant vs. subordinate monkeys, such that dominant monkeys showed stronger sustained attention throughout an extended period after mutual eye contact. By contrast, differences in familiarity exclusively impacted the early decay parameter b₁, such that sustained attention decayed more rapidly immediately after mutual eye contact for the familiar pairs.

Fig. 12.

Modeling of social gaze dynamics after mutual eye contact. A: root-mean-square error (RMSE) values from the cross-validation procedure (materials and methods) for the Exp₁, Exp₂, and Hyp models. Significantly lower RMSE values were obtained with the Exp₂ compared with the Exp₁ model (***P < 0.00001, paired t-test) and the Hyp model (**P < 0.0001, paired t-test). B and C: averaged population values for gaze dynamics of dominant (green) vs. subordinate (orange) animals (B) and more (blue) vs. less (purple) familiar animals (C) fitted with the Exp₂ function. D and E: gain parameters (a₁ and a₂) and decay parameters (b₁ and b₂) of the Exp₂ model, averaged from individual fits of dominant and subordinate animals (D) and more and less familiar animals (E). *P < 0.05, ⁺P < 0.1, paired t-test validated with permutation test.

We next applied the Exp₂ model to the data obtained from individual monkeys on a given day of testing. Corresponding to the population fit, we found significant differences in both a₁ and a₂ gain parameters for dominance-related curves (Fig. 12D; P < 0.05 for both a₁ and a₂ gain parameters, 2-sample t-tests validated with permutation tests). The pattern of gain terms a₁ and a₂ was consistent with the observation that dominant monkeys consistently looked more at the eyes of the conspecific throughout and displayed a shallower decrease in looking after mutual eye contact. Critically, dominance did not drive differences in either decay parameter (P > 0.5 for both b₁ and b₂ decay parameters, 2-sample t-tests validated with permutation tests). Conversely, neither a₁ and a₂ gain parameters nor the b₂ decay parameter were observed to differ based on familiarity (P > 0.6 for both a₁ and a₂ gain parameters, P > 0.3 for the b₂ decay parameter, 2-sample t-tests validated with permutation tests). However, the early decay parameter b₁ trended to be higher for more familiar monkeys, corresponding to the overall shape of the population average (Fig. 12E; P < 0.1 for b₁ early decay parameter, 2-sample t-test validated with permutation test). Thus dominance and familiarity effects were functionally and temporally separable in a simple model describing a natural decay process.

DISCUSSION

Our findings support the importance of gaze patterns after bidirectional, or mutually engaged, social events in shaping social cognition that is specifically enacted when interacting with a real-life partner. We observed several notable differences in key aspects of social gaze patterns that were either unique or most pronounced when interacting with a real partner compared with interacting with abstract depictions of another. These effects were manifested as a smaller dispersion of fixations around the eyes as well as the time-dependent specificity of dominance- and familiarity-induced gaze patterns after mutual eye contact, but not nonmutual eye contact, that could be described by separable model parameters.

The “duality of gaze” concept describes how we look at others to both extract and send information (Jarick and Kingstone 2015). Mutual gaze signals communicative intent and promotes social interaction (Cary 1978). Efforts have been made in recent years to study people's gaze behavior during face-to-face interaction. However, such studies have focused on examining mutual gaze based on either filming how two people interact (Jarick and Kingstone 2015; Wu et al. 2014) or examining the eye positions from only the tested subject (Falck-Ytter 2015; Foulsham et al. 2011; Freeth et al. 2013; Gallup et al. 2012; Laidlaw et al. 2011), thus underemphasizing the interactive component of social gaze. In our study, we quantified the gaze dynamics upon mutual eye contact. In the live condition, dominant monkeys looked significantly more at the eyes than subordinate monkeys after mutual eye contact at various time points throughout the 3-s window. By contrast, after nonmutual eye contact these differences were completely absent. Familiarity-driven differences in social gaze dynamics were also unique to the live condition. Less familiar monkeys looked more frequently at the eyes in the live condition during the early period immediately (300–500 ms) after mutual eye contact. Similar to the dominance effect, these differences were specific to the live condition after mutual eye contact.

Social gaze patterns are highly sensitive to the relationships between individuals (Broz et al. 2012; Coutts and Schneider 1975; Jarick and Kingstone 2015), a phenomenon that cannot be conclusively studied from a single subject's perspective. Social behaviors and interindividual relationships are also dynamic and demand flexible updating over time based on previous interactions. Quantitatively capturing these features in a controlled laboratory setting is an important step toward understanding the neurobiology of social behaviors and social deficits. Rhesus macaques, like humans, live in complex societies where dominance and familiarity regulate social interactions (Smuts et al. 1987). Within a group, dominance is directly related to the capacity of an individual to influence the behavior of another by claiming his priority and social status, whereas familiarity is related to in-group and out-group protection (Mitani et al. 2012). Our gaze interaction setup in rhesus macaques, whose social interactions are regulated by dominance and familiarity (Mitani et al. 2012), provides a novel platform for elucidating neural mechanisms underlying social dynamics that may inform how humans integrate ongoing social information and adjust behaviors. We were able to extract dominance- and familiarity-induced differences in gaze dynamics by capturing temporally distinct gaze patterns impacting either a gain or a rate parameter of an exponential decay process of sustained attention after mutual eye contact. Notably, these differences were found exclusively after mutually engaged social attention.

The results outlined so far indicate that interacting with a real partner promotes differential dynamics and amplifies the attention directed to the eyes. When monkeys interacted with a real conspecific compared with viewing the pictures and prerecorded videos of the same partner, we observed a significant increase in the proportion of fixations to the eyes as well as a smaller dispersion of fixations around the eyes, indicating an enhanced focal attention to the eye region during live interaction. Furthermore, we observed a trade-off between the number of fixations and the duration of these fixations to the eyes in the live but not the picture and movie conditions (Fig. 7B). Such a trade-off in the social attention domain may be a signature of real-time interactions. Critically, the enhanced social gaze patterns in the live condition were not driven by a nonspecific increase in arousal or saliency of the conspecific's face. The total count of fixations in the live condition was not significantly higher than in the movie condition, and the number of fixations to the overall face was in fact highest in the movie condition. To date, the majority of social gaze studies have largely overlooked the interactive nature of real-world social information (Schilbach et al. 2013), and it has become increasingly clear that studying how people attend to mere representations of others is failing to sufficiently capture social attention in the real world (Kingstone et al. 2003, 2008; Myllyneva and Hietanen 2015). Several studies have started to show the importance of live interaction in capturing critical aspects of social cognition. Electroencephalogram (EEG) activity and galvanic skin responses in humans are strongly modulated by another's gaze in a live context but not when stimuli are presented as pictures (Hietanen et al. 2008). There are also notable differences in eye contact patterns when having a conversation with a live interviewer compared with a video-recorded interviewer (Freeth et al. 2013), and the act of averting gaze while approaching a real person is distinct from that while approaching a video of the same person (Foulsham et al. 2011). Furthermore, viewing a live face with direct gaze results in more pronounced neural processing (based on EEG) than viewing photographs of the same face (Pönkänen et al. 2011a, 2011b), and brain areas associated with social cognition and attention show differential activation patterns between live-video and recorded-video interactions (Redcay et al. 2010). Moreover, recent methodological advances for improving ecological validity for studying social cognition include interactions with virtual avatar agents in gaze-contingent eye-tracking paradigms (Wilms et al. 2010), live interactions via video feeds while the participants' gaze behaviors are monitored by an eyetracker (Auyeung et al. 2015; Redcay et al. 2010), and dual eye-tracking in two-person functional neuroimaging or EEG hyperscanning setups for studying the neural mechanisms of joint attention (Lachat et al. 2012; Saito et al. 2010).

Our results in monkeys indicate that interacting with a real partner amplifies the modulatory effects of social relationship variables on attention directed to the eyes. Human gaze cuing effects are also modulated by facial features, signaling social attributes such as dominance (Jones et al. 2010), familiarity (Deaner et al. 2007), status (Dalmaso et al. 2011), and group membership (Pavan et al. 2011). Similarly, macaques show enhanced joint attention, a phenomenon largely influenced by head and face orientation (Marciniak et al. 2014), especially when triggered by the gaze direction of dominant compared with subordinate monkeys (Shepherd et al. 2006). We found that dominant monkeys looked at the eyes of subordinate monkeys more frequently, and the fixations from dominant monkeys were also more clustered around the eyes compared with those from subordinate monkeys in the live condition. In a previous study in which monkeys sacrificed a small amount of juice reward to view faces, it has been shown that low-status monkeys tend to avert their gaze from the pictures of high-status monkeys whereas high-status monkeys tend to look directly at the pictures of low-status monkeys (Deaner et al. 2005). Our results are consistent with recent studies in humans showing that social gaze patterns between two individuals carry information about social rank (Foulsham et al. 2010; Gobel et al. 2015) and dominance (Jarick and Kingstone 2015). Moreover, we found that familiarity overall reduced exploration of the eyes, possibly reflecting more efficient (e.g., through more exposure and learning) social information acquisition when interacting with more familiar partners. Furthermore, female monkeys looked substantially more at the eyes of a same-sex conspecific than male monkeys in the live condition. These sex-related differences in the magnitude of attention to the eyes are consistent with previous studies reporting that human females show greater memory for faces (Guillem and Mograss 2005), stronger eye gaze (Bayliss et al. 2005), and longer duration of gaze at their partners (Frances 1979; Hall 1984) compared with males. Furthermore, female neonates spend more time looking at a human face than at a phone image, whereas male neonates display the opposite preference (Connellan et al. 2000). Interestingly, female infants make more eye contact than male infants already at 12 mo of age (Lutchmaya et al. 2002). Collectively, these findings suggest that natural social interaction is a bidirectional process, in which each individual both signals and reads gaze information (Wu et al. 2014), and that the nature of this gaze signaling changes with the relationship between individuals (Gobel et al. 2015).

It has been also shown that social context can dramatically influence social gaze patterns (Gobel et al. 2015). Laidlaw and colleagues (2011) have reported that human participants spend less time looking at a live confederate completing a questionnaire when sitting in the same room than the same confederate shown on a monitor. Looking at another person may serve as a signal to initiate interaction (Cary 1978), which could have been undesirable in that particular context. By contrast, during an active interview where participants were encouraged to interact with an experimenter, participants looked more at the experimenter's face when the experimenter looked at the participant's eyes in person compared with in a video (Freeth et al. 2013). The context and the configuration of our study were similar to that described by Freeth and colleagues (2013). In the present study, monkeys sat face to face in close proximity in order to encourage social interactions between them. We found an enhancement of attention directed to the eyes when interacting with real partners compared with static images and movies of the same monkeys. Taken together, these findings converge on the conclusion that not only social variables but also social contexts gate social attention.

It is worthwhile to note that our results are based on natural, spontaneous gaze behaviors without any task or environmental constraints. Specific goals of social interactions would likely require individuals to flexibly update their social actions by, for example, enhancing or reducing the sensitivity for reacting to others and sending cues. Understanding how different environmental contingencies would shape social context and how they consequently impact social dynamics would begin to reveal more comprehensive insight into social cognition. In one of the first highly quantitative studies of live social interactions in monkeys, Livneh and colleagues (2012) explored communication through facial expressions. In our study, we did not observe extensive facial expressions, perhaps because of the difference in the time that we allowed our monkeys to visually interact with one another and possibly because of the necessity of restraining head movements of the monkeys during eye-tracking, precluding directed head and facial gestures to the partner monkeys. Future studies should investigate how facial expressions might modulate gaze behavior if those were depicted in a live agent vs. a movie or a picture of the same individual. Additionally, we need to better understand how sensory stimuli associated with specific social variables could influence gaze dynamics. Although auditory vocalization was largely absent in our experiments, similar to the study by Livneh and colleagues (2012) in which essentially no vocal calls were emitted, olfactory cues might have nevertheless contributed to some differences in social gaze. However, the observation that dominance- and familiarity-induced differences in gaze dynamics were specific to mutual gaze, but not nonmutual gaze, strongly mitigates this possibility.

There are several advantages in first establishing our basic paradigm in nonhuman primates. Dominance and even familiarity in humans is very likely to be highly context dependent. This is perhaps less of a problem with monkeys, which may allow for more robust measurement of social variables without the added complexity found in humans. Situational influences are also perhaps less of an issue when using monkeys, as humans could certainly change their typical behavior when they are told that their gaze is being tracked continuously. Additionally, the validation of this paradigm in nonhuman primates allows for the eventual use of neurophysiological and pharmacological techniques that are simply unavailable in healthy humans at this time, including single-unit and local field potential neural recordings as well as brain region-specific focal administration of novel therapeutics to assist in our understanding and treatment of social deficits. That being said, our paradigm could be easily translatable to humans, and particularly to younger children and infants, for testing social deficits often present across a broad spectrum of neuropsychiatric disorders.

Behavioral contingency or statistical dependence over time is required for social interactions across at least two individuals. It is easy to conceive that a failure in the interindividual contingency would lead to atypical social behaviors in humans and nonhuman animals. Our findings demonstrate the importance of gaze dynamics in shaping social cognition that is specifically engaged when interacting with a real-life partner compared with interacting with abstract depictions of another. These effects include a more focal attention around the eyes and the specificity of dominance- and familiarity-induced gaze patterns after mutual eye contact with separable dynamic components. Quantitatively capturing these features in a controlled laboratory setting is an important step toward understanding the neurobiology of social behaviors and social deficits. Our dual gaze-tracking paradigm in rhesus macaques provides a platform for elucidating neural mechanisms underlying social dynamics that may inform how humans compute ongoing social information and adjust behaviors.

GRANTS

This study was supported by the Alfred P. Sloan Foundation (S. W. C. Chang), the National Institute of Mental Health (R00-MH-099093, R21-MH-107853, O. Dal Monte, M. Piva, S. W. C. Chang), the Simons Foundation (365029, O. Dal Monte, S. W. C. Chang), a NSERC PGSD Fellowship (471313, M. Piva), and a Timothy Dwight Richter Fellowship (J. A. Morris).

DISCLOSURES

No conflicts of interest, financial or otherwise, are declared by the author(s).

AUTHOR CONTRIBUTIONS

O.D.M., M.P., and S.W.C.C. conception and design of research; O.D.M., M.P., and J.A.M. performed experiments; O.D.M., M.P., and S.W.C.C. analyzed data; O.D.M., M.P., and S.W.C.C. interpreted results of experiments; O.D.M., M.P., and S.W.C.C. prepared figures; O.D.M., M.P., and S.W.C.C. drafted manuscript; O.D.M., M.P., J.A.M., and S.W.C.C. edited and revised manuscript; O.D.M., M.P., J.A.M., and S.W.C.C. approved final version of manuscript.