The Influence of Joint Attention and Partner Trustworthiness on Cross-Modal Sensory Cueing

Abstract

Joint attention refers to the coordinated attention between social partners to an object of shared interest, usually involving shared gaze toward the object. In the laboratory, however, joint attention is often investigated using computerized gaze cueing tasks that do not allow shared gaze. Instead, these computerized tasks require the participant to maintain fixation on the virtual partner’s face, while the partner gazes to the left or right. Here we designed a modified gaze cueing task that better simulates a natural joint attention episode by allowing shared gaze, while still maintaining tight experimental control. In our computerized task the participant’s gaze and the gaze of a virtual partner were manipulated independently, resulting in shared or unshared gaze. Following each gaze shift of the virtual partner a touch stimulus was delivered on one of the cheeks of the participant. We analyzed behavioral and neural (electro-encephalography) responses to the touch. Faster reaction-times and stronger lateralization of alpha power were observed when the touched cheek was in a jointly attended hemispace compared with a singly attended or unattended hemispace. Importantly, these effects were unique to joint attention and could not be explained as the additive effects of own gaze and gaze cue direction. Underlining its social nature, we found that the behavioral effect was absent when we repeated our experiment with nonsocial cues (arrows) instead of gaze cues. Furthermore, when we compared trustworthy with untrustworthy virtual partners (trustworthiness judgements based on facial appearance) we found the effect only for trustworthy and not for untrustworthy virtual partners. We conclude that joint attention based on shared gaze influences attentional orienting such that cross-modal sensory processing at the jointly attended location is facilitated, particularly when the partner is trustworthy. This indicates that social interactions and trustworthiness judgements affect cortical and behavioral responses to sensory information.

Introduction

Joint attention refers to a situation in which two individuals have a common focus of attention on the same object (Reddy, 2003; Schilbach et al, 2013). The importance of joint attention as a social skill and a prerequisite for typical human social cognition is well-established. Infants engage in joint attention approximately by the end of the first year of life and their ability to do so is predictive for further development of linguistic and socio-emotional skills during childhood (Carpenter et al, 1998; Reddy, 2003). Furthermore, joint attention is thought to be associated with “Theory of Mind”, the ability to infer what someone else is thinking (Baron-Cohen, 1994; Schilbach et al, 2013), and reduced establishment of joint attention is an early predictor of autism spectrum disorders (Baron-Cohen, 1994; Dawson et al, 2004).

It is inherently difficult to investigate social phenomena like joint attention in an ecologically valid setting and still maintain tight experimental control. Therefore, many investigations in the laboratory have focused on a pre-stage of joint attention known as ‘gaze following’, which is a shift of spatial attention in the direction of someone else’s gaze. Using so-called ‘gaze cueing tasks’, it has been shown that gaze following occurs involuntarily, even when the observed gaze direction is detrimental to task performance (Emery, 2000; Frischen et al, 2007). In a typical gaze cueing task the image of a face that looks either to the left or right is presented on a computer screen and participants respond to a subsequent target stimulus appearing left or right of the face. Reaction times are faster when the face gazes in the direction of the target than when the face gazes in the opposite direction.

In a typical gaze cue task participants are not allowed to gaze at the same location as a virtual partner, as strict fixation on the virtual partner’s face is required. Thereby, there is no shared gaze between the participant and the virtual partner. Even when we consider joint attention in its most basic form, i.e., as sharing gaze on a common object without requiring knowledge or awareness of the other’s direction of attention (Kingstone et al., 2000; Mundy & Jarrold, 2010; cf. Tomasello & Hamann, 2012), this lack of shared gaze poses a cardinal difference between joint attention and the situation tested in a gaze cueing task. It therefore remains unclear whether joint attention, here narrowly defined as ‘sharing gaze’, elicits involuntary orienting by itself, or whether the involuntary part of joint attention is restricted to the pre-stage of mere gaze following. We know that a participant’s own gaze direction also elicits involuntary orienting (Bisley, 2011; Itti & Koch, 2001; Shephard et al, 1986/2009). Our question is whether the influences of own gaze and partner’s gaze (gaze cueing) operate independently, or whether they interact, in which case it would make a difference whether own gaze and the partner’s gaze are directed to the same location or not.

To answer this question we designed a modified gaze cueing task in which both the participant’s gaze and the partner’s gaze were manipulated independently and we tested whether their influences interact. If they do not, then own gaze and partner’s gaze elicit orienting by themselves, irrespective of whether they are shared or not. However, if they do interact, then neural processes dedicated to own gaze and partner’s gaze are integrated, providing a ground for an elemental form of joint attention based on shared gaze and a basis for the development of reciprocal social interactions (Caruana et al, 2017; Mundy & Jarrold, 2010; Pfeiffer et al, 2013; Schilbach et al, 2013). Hypothesizing that they interact, we anticipate that shared gaze elicits additional speeding of reaction times, beyond an extent that can be explained by the added effects of own gaze and partner’s gaze.

We instructed participants to fixate on a stimulus left or right of a virtual partner’s face and thus view the face peripherally, in the corner of their eyes. Following a gaze shift of the virtual partner a touch stimulus was delivered to one of the cheeks of the participant. Participants responded to the touch by pressing a button and reaction times and electro-encephalography (EEG) were recorded. The aim of measuring neural activity is to assess whether behavioral influences of joint attention are reflected in – and may be attributed to – modulations in sensory processing. We expected sensory processing of the touch stimulus to be reflected in modulations at fronto-central sites over the face-representation in sensory cortex in the theta and alpha frequency band (Genna et al, 2017; Hu et al, 2013). The alpha band is particularly interesting, as it has been attributed to attention and extended processing of touch (Genna et al, 2017; Klimesch, 2012).

We used a touch rather than a visual target to circumvent the misalignment of the spatiotopic field with the participant’s retinotopic field (due to their averted gaze). Moreover, we considered a touch feasible and ecologically valid, considering existing knowledge about visuo-tactile cueing (Soto-Faraco et al, 2005; Spence et al, 2004) and visuo-tactile integration near the face (Brozzoli et al, 2012; Graziano & Cooke, 2006).

To assess the social nature of possible effects of shared gaze, we repeated our experiment with a nonsocial stimulus, i.e., an arrow (Frischen et al, 2007), hypothesizing that processes related to joint attention are not activated when own gaze direction is shared with an arrow’s pointing direction. Arrows are often used as a control stimulus for gaze cues, because they are omnipresent in modern society, they do not contain eye-like features (minimizing the risk of pareidolia; Ichikawa et al., 2011; Takahashi & Watanabe, 2013) and they orient attention in a comparable way (Kuhn & Kingstone, 2009; Ristic et al., 2002; Tipples, 2002).

Additionally, we were interested to see if effects of shared gaze are influenced by the perceived trustworthiness of the virtual partner (trustworthiness judgements based on first impression). Trustworthiness inferences are made within a split second (Todorov, 2008; Willis & Todorov, 2006) and could, thereby, potentially influence fast social processes like the effects of shared gaze or gaze following tested here. Consistent with earlier work showing that partners who tend to look away from a target stimulus are rated more untrustworthy (Bayliss & Tipper, 2006; Manssuer et al., 2015), we anticipated that processes related to joint attention are attenuated when a partner is untrustworthy.

Materials and Methods

Participants

Experiments were carried out in accordance with the Declaration of Helsinki (World Medical Association 2000) and approved by the ethical committee of the Faculty of Social and Behavioral Sciences at Utrecht University. Participants gave written informed consent prior to participation, were right-handed, had normal or corrected to normal vision, normal touch sensation and no psychiatric or neurological history. A priori power analysis indicated a recommended sample size of 24 participants (estimated effect-size: moderate to small, f= 0.25; power= 80%; alpha= 0.05). We measured more participants in anticipation of technical failures or auditory transfer of the vibrotactile stimulus (i.e., using audition rather than touch to determine the touched location). All inclusion/exclusion criteria used in this study were established prior to data collection and we report all manipulations and all measures in the study. The data analysis included 24 participants (12 female, 12 male; mean age 24.1 years ±0.9 SEM). Four additional participants were excluded because of auditory transfer of the vibrotactile stimulus, as assessed by a questionnaire (more information in ‘Touch stimuli’-paragraph below; including these participants yielded similar behavioral results). Three additional participants were excluded because of technical failures. The study procedures and analyses were not preregistered in a time-stamped, institutional registry prior to the research being conducted. The ethical committee of the Faculty of Social and Behavioral Sciences at Utrecht University does not grant permission for anonymized public archiving of the data, but data will be shared upon request with all requestors (requestors do not need to meet certain criteria; all participants gave written informed consent for data sharing). The data are stored at the Faculty of Social and Behavioural Sciences of Utrecht University and can be obtained by contacting H.C. Dijkerman. Stimuli, questionnaires and analysis code are publicly archived at the Open Science Framework (https://osf.io/4n9yx; 10.17605/OSF.IO/4N9YX).

Procedure and task

Participants were seated in an electronically and acoustically shielded room, wearing a head-cap for electro-encephalography (EEG), using a chin-rest and looking at a computer screen (refresh-rate: 60 Hz; size: 302 x 378 mm). We placed the computer screen relatively close to the face of the participant (viewing distance was 50 cm), because visuo-tactile integration around the face is stronger inside than outside peri-personal space (Brozzoli et al, 2012; Iachini et al, 2014). In addition, we ensured that this viewing distance was not further away than the inter-personal distance that our participants prefer during a social interaction. Specifically, preferred inter-personal distance when asking a stranger for directions was measured after the experiment for each participant and averaged to 73.4 cm ±4.2 SEM when they interacted with the experimenter and 76.7 cm ±4.9 SEM when they interacted with a life-sized poster of an unknown standing individual.

Participants completed 576 experimental trials (48 of which were pseudo-catch trials, as described below) and 48 catch trials, divided over 4 experimental sessions that were separated by short breaks. Four participants completed extra trials to compensate for data loss due to technical problems (more information about data loss below in ‘Behavioral analysis’-section). Fixation stimuli (black and white asterisk, each 1.0°) were placed 11.9° apart (center to center) at eye-height of the to-be-presented virtual partner and alternated their position between sessions (Figure 1B). A session was subdivided in 6 blocks of 24 trials, each lasting 68 seconds. At the start of a block an instruction in text indicated which of the two fixation stimuli should be fixated. The instructed fixation position alternated between blocks to avoid tiring of the eye muscles and remained fixed within blocks to eliminate oculomotor effects (Schutz et al, 2011; Wardak et al, 2011). The first trial of each block was excluded from the analysis, because it was preceded by eye movements (related to reading the instruction in text and initiating fixation of the instructed asterisk). Details of the face stimuli and the touch stimulus are described in the following paragraphs. Importantly, the face stimuli were presented in the center of the screen (between the fixation stimuli) and there were 4 locations on the cheeks of the participant where the touch stimulus could be delivered: left up, left down, right up and right down (Figure 1).

Figure 1. Paradigm.

A) Schematic illustration of the averted gaze of the participant and the 4 possible touch locations on the cheek (white dots). Per trial only 1 of the touch locations vibrated (illustrated as black star). The participant indicated whether the touch was high or low on the cheek, irrespective of which cheek. Gaze direction of the participant was fixed across trials to eliminate oculomotor effects (but alternated between experimental blocks to avoid tiring of the eye muscles).

B) Fixation stimuli were presented throughout the trial. A variable delay after trial onset the virtual partner appeared with straight gaze. Then gaze direction of the virtual partner gradually changed to one of the fixation stimuli during a short movie. As a result, the participant and the virtual partner then fixated the same fixation stimulus (joint attention) or opposite fixation stimuli (disjoint attention). The tactile stimulator was activated a variable delay after that (illustrated by black star). The participants reported the vertical location of the touch (up/down) with a button press.

C) Vibration amplitude over time relative to onset of the stimulator. The stimulator was activated for 200 ms (illustrated by black star). After activation the rotating mass inside the stimulator needed some time to accelerate and produce a perceivable vibration. Then, when activation has stopped, the rotating mass needed some time to decelerate and thereby stop the vibration. The vibration was perceivable approximately 80-280 ms after onset of the stimulator, when it was >50% of its maximal amplitude (see Methods). The delay of approximately 80 ms between activation of the stimulator and delivery of the vibration does not differ between conditions, but should be taken into account when interpreting absolute reaction times and neural activity time-locked to onset of the tactile stimulator.

Participants used their right thumb and index finger to operate two buttons on a button box. The button operated by the thumb was positioned to the lower left of the button operated by the index finger. On experimental trials, participants indicated perceived vertical location of the touch (thumb= low; index finger= high). The horizontal location (i.e., which cheek) was task-irrelevant to eliminate cueing effects on responding. On 48 catch trials the asterisk that the participant was fixating turned red and the participant was instructed to indicate gaze direction of the virtual partner (thumb= left; index finger= right). Correct perception (91.6% correct responses) of the gaze of the peripherally-viewed virtual partner was validated (in line with Canadas & Lupianez, 2012; here we analyzed catch trials for which the experimental task would have required a different response than the catch task). Additionally, catch trials prevented participants from ignoring the visual modality. One participant mistakenly reported horizontal touch location instead of the partner’s gaze direction on catch trials. This participant was included in the analysis, because the change in color of the asterisk that indicated a catch trial was correctly used by this participant to switch to an alternative task, indicating that the visual modality was not ignored. On 48 pseudo-catch trials the non-fixated asterisk colored red. Confirming that participants followed fixation instructions, they treated these trials as experimental trials (94.3% correct responses).

To assess whether effects of joint attention were associated with certain personality traits participants completed 3 questionnaires after they completed the experiment, namely the Autism spectrum Quotient (AQ; Baron-Cohen et al., 2001; Hoekstra et al., 2008), the State and Trait Anxiety Inventory (STAI; Spielberger et al., 1983) and the Behavioral Inhibition and Behavioral Activation Scales (BIS/BAS; Carver & White, 1994). Obtained scores were not significantly associated with effects of joint attention reported below, possibly due to very small variability in the scores of our non-clinical participant group.

Face stimuli

A virtual partner appeared 700-1300 ms after trial onset, in the center of the screen, displaying straight gaze and subtending 9.0° horizontally. After 250 ms its gaze direction gradually changed during a short movie to either the left or the right (randomly) and was fully averted 233 ms later. A cheek stimulator was activated 150-350 ms later and the trial ended 1100 ms after that (Figure 1). The 9-frame movies of the gaze change were used in previous publications (e.g. de Jong et al., 2008) and contain face stimuli of 9 different actors with neutral expression taken from the validated stimulus set of MacArthur Foundation Research Network on Early Experience and Brain Development. Development of the MacBrain Face Stimulus Set was overseen by Nim Tottenham and supported by the John D. and Catherine T. (contact Nim Tottenham at tott0006@tc.umn.edu for more information concerning the stimulus set). Using Photoshop (Adobe) straight and averted eyes were edited, images were converted to grayscale and faces were matched on size, shape, contrast and luminance (mean luminance equaled the mid-gray background luminance). A gamma correction was applied using Matlab (MathWorks Inc.). The movies were generated by morphing straight-gaze images to averted-gaze images of the same actor using Smartmorph (Meesoft).

Touch stimuli

Stimulators on the cheeks looked like small button batteries, were attached using adhesive tape and produced a touch like the vibration of a mobile phone (eccentric rotating mass in cylindrical casing, 8 mm diameter, 3.4 mm thick, Precision Microdrives, Model: 308–100). Stimulators were placed high on the left cheek (below zygomatic bone), low on the left cheek (above mandible bone) and at equivalent positions on the right cheek (Figure 1A). Per trial 1 stimulator was activated (for 200 ms). Note that after activation of the stimulator the rotating mass inside the stimulator needed some time to accelerate and produce a perceivable vibration. Also, when activation of the stimulator has stopped the rotating mass needed some time to decelerate and thereby stop the vibration. The time-course of this acceleration and deceleration is provided in Figure 1C, showing that vibration amplitude was perceivable approximately 80-280 ms after stimulator onset (we here estimated an amplitude >50% of the maximum amplitude to be perceivable, based on information provided by Precision Microdrives). In the 80-280 ms interval the vibration frequency was ~100 Hz, as measured with a Bosh Sensortec BST-BMI160 inertial measuring unit. The delay of approximately 80 ms between activation of the stimulator and delivery of the vibration does not differ between conditions, but should be taken into account when interpreting absolute reaction times and neural activity time-locked to onset of the tactile stimulator.

The stimulators produced negligible sound, but facial bones/tissue or legs of eyeglasses may transmit the vibration to the ear. To minimize auditory transfer participants listened to white noise sound through ear-phones. After the experiment participants indicated whether they used audition or touch to determine the horizontal and vertical location of the vibration on a visual analogue scale that was labeled ‘touch only’ on the left end and ‘audition only’ on the right end. For analysis the positions marked on the scale were converted to a numerical scale ranging from 0 for audition only to +100 for touch only. As mentioned earlier, we excluded 4 participants who mainly used audition to determine the horizontal and/or vertical location (i.e., scores ≤ 50%; more extreme cutoffs of 75% and 0% yielded equivalent results, data not shown). Participants also rated the pleasantness of the tactile stimulus on a visual analogue scale, assuring it was not aversive (‘very unpleasant’-label on the left end, ‘very pleasant’-label on the right end; judgements converted to numerical scores between -100 and +100 for very unpleasant and very pleasant, respectively; average score: +22.7 ±6.8 SEM).

Trustworthiness

Before and after the experiment participants rated their first impression of the trustworthiness of each virtual partner as based on facial appearance, using a visual analogue scale (labeled ‘untrustworthy’ on the left end and ‘trustworthy’ on the right end; partners were presented with straight gaze). For analysis the judgements were converted to a numerical scale ranging from -100 for untrustworthy to +100 for trustworthy (as in Manssuer et al., 2015). One participant only completed the rating before the experiment. This participant was included in the analysis.

Experiment with nonsocial cues

To assess the social nature of observed effects, we repeated our experiment showing arrows instead of virtual partners. Durations of the initial non-directional image (arrow-shaft only; subtending 4.8°, equaling center-to-center distance between the actors’ pupils), the movie and the directional image (arrow with head/vanes; subtending 6.4°, equaling distance between outer corners of the actors’ eyes; illustration in Figure 2D) were the same as in the main experiment, meaning the arrow head and vanes appeared gradually during the movie. Each of 9 images of an arrow was matched to one of the 9 actors (vertical position and size matched to actors’ eyes). The arrows ranged 0-100% in luminance with mean luminance equaling background luminance. On catch trials participants were instructed to report arrow direction.

Figure 2. The influence of joint attention on reaction times.

A) Joint attention toward the touch hemispace resulted in exceptionally fast reaction times, i.e., faster than could be explained by the added effects of the participant’s gaze and the virtual partner’s gaze per se (referred to as own and partner’s gaze, respectively). Check marks and X-marks indicate gaze to touched and untouched hemispace, respectively. Grey and black symbols refer to own and partner’s gaze, respectively. Error bars indicate ±SEM.

B) Reaction-time distributions per condition shown for illustrational purposes. The distribution for jointly attended (black line) compared with singly attended (solid light grey and dark grey lines) and unattended (dashed grey line) targets was shifted in time, but not markedly different in shape.

C) The amount of additional speeding due to joint attention, i.e., shared gaze, for the individual participants (difference in reaction time between congruent joint attention and the average of the two conditions with disjoint attention, minus the difference between this average and incongruent joint attention).

This experiment was performed by another group of participants and did not include EEG recordings. Analysis included 19 participants (14 female, 5 male; age 29.1 years ±2.9 SEM). Three participants mistakenly reported horizontal touch location on catch trials. These participants were included in the analysis, because they correctly used the visual signal that indicated a catch trial to switch to an alternative task, showing that they did not ignore the visual modality. Six additional participants were excluded: 3 participants mainly used audition to determine touch location (including these participants yielded equivalent behavioral results), 1 participant performed the experimental task on catch trials and 2 participants performed the catch task on pseudo-catch trials.

Electro-encephalography recording and preprocessing

Electro-encephalography (EEG) was sampled at 2048 Hz and recorded with 64 silver/silver-chloride (Ag/AgCl) flat type active electrodes (Active Two system, Biosemi) positioned at standard locations on an elastic cap (Quickcap, Neuromedical supplies of Neurosoft inc.) and referenced to an additional active electrode (Common Mode Sense) during recording. Two electrodes in the cap provided an active ground. Horizontal and vertical electro-oculograms were measured. Offline preprocessing was performed using BrainVision Analyzer 2 (Brainproducts GmbH). A 0.1-200 Hz band-pass filter was applied and signals were referenced to the average of all scalp electrodes. Artifacts due to eye movements were removed (algorithm by Gratton et al. 1983) and epochs time-locked to the onset of the tactile stimulator were extracted from the continuous data. Artifact rejection was performed on individual channels and included removal of epochs with excessive amplitude changes (>100 μV within 200 ms). We also checked for recording failure (<1 μV differences within 200 ms and steps per sample point >50 μV), but these seldom occurred.

Further analysis of electro-encephalography

Time-frequency transformation was performed over a frequency range of 5-13 Hz (5-60 Hz in Supplementary Materials, Figure S2) using in-house-developed Matlab code (Mathworks Inc.) and the Matlab toolbox EEGlab (Delorme & Makeig, 2004). We used a 3-18 cycle Morlet wavelet tapered with a Hanning window, taking frequency steps of 1 Hz and time steps of 10 ms. To isolate induced activity, the event-related potential per participant, per condition, was subtracted from the single trial data prior to the transformation. Normalization was performed per epoch, per frequency (normalized power = power / mean power). Mean normalized power over a 100-ms pre-stimulus baseline interval was subtracted. We analyzed a fronto-central electrode site associated with tactile processing (site FC5/FC6; Figure 4A; Genna et al, 2017; Hu et al, 2013). We analyzed contralateral and ipsilateral power as well as power lateralization according to: lateralization = (contralateral-ipsilateral) / (contralateral+ipsilateral).

Figure 4. The influence of joint attention on touch processing.

A) Modulation in theta and alpha power at the site-of-interest. Top: contralateral increase in alpha power was observed for joint but not disjoint attention, possibly reflecting enhanced processing of joint compared with disjoint attention (as seen prior to the touch, Supplementary Figure S1). Middle: ipsilateral alpha power increase was present only for joint attention toward the untouched hemispace, possibly reflecting allocation of spatial attention to this hemispace. Bottom: lateralization of alpha power ((contralateral-ipsilateral) / (contralateral+ipsilateral)) was observed for jointly attended, but not for singly attended or unattended touch, in line with the idea of facilitated tactile processing for jointly attended touch locations. Conventions as in Figure 2. Small head in panel B shows location of fronto-central site-of-interest.

B) Map of alpha lateralization across the scalp (frontal sites on top, lateral sites on the left) averaged for all conditions (left) and for jointly attended touch only (right), showing alpha lateralization for jointly attended touch was maximal at the site-of-interest (indicated by black square). Small head: location of fronto-central site-of-interest.

Behavioral analysis

Experimental trials in which no button press was recorded were excluded from the analysis (7.6% of trials; data loss was mostly due to technical problems with the button box or the tactile stimulator). Using Matlab (Mathworks) the mean reaction time per participant, per touch location was subtracted from the data and outliers were removed (2.0% of experimental trials). Outliers included reaction times <100 ms and >1100 ms (before normalization) and reaction times >3 standard deviations from the participant’s mean reaction time (following e.g. Hietanen et al., 2008; Manssuer et al., 2015). Trials with an incorrect response (3.4%) and trials for which no EEG was recorded (6.6%; due to technical failures) were also excluded. These exclusion criteria were also applied to the EEG analysis. The final number of analyzed trials was 459.6 on average ±10.6 SEM (114.9 per condition).

Statistical analysis

Unless indicated otherwise, we performed a repeated-measures analysis of variance (ANOVA) over participants, with own gaze and partner’s gaze as within factors. The within factors had two levels: congruent and incongruent with target laterality.

Results

Reaction times

Mean reaction time was 599 ms ±17 SEM (this reaction time includes approximately 80 ms that is needed for the tactile stimulator to accelerate and produce a perceivable vibration, see Methods and Figure 1C). Reaction times were faster when the cheek in the gazed-at hemispace was touched than when the other cheek was touched, both regarding the participant’s gaze (own gaze: F _1,23= 29.2, p= 0.000017) and the partner’s gaze (partner’s gaze: F _1,23= 23.1, p= 0.00008; Figure 2; mean normalized reaction times: congruent joint attention: -13.4 ±1.7 SEM; only own gaze congruent: 1.2 ±1.9 SEM; only partner’s gaze congruent: 4.5 ±1.5 SEM; incongruent joint attention: 7.7 ±1.8 SEM). Importantly, there was an interaction between these main effects (own gaze x partner’s gaze: F _1,23= 8.8, p< 0.007), which revealed that, when the touched cheek was in a jointly attended hemispace, there was a speeding of reaction times that exceeded the added effects of own gaze and partner’s gaze per se (i.e., joint attention to touched hemispace differed from all other conditions: all t₍₂₃₎> 5.2, all p< 0.00003, while the other conditions did not differ from each other: all t₍₂₃₎≤ 2.0, all p≥ 0.06; Figure 2). This additional speeding was more robust for long compared with short delays between partner’s gaze shift and touch (>250-ms compared with ≤250-ms between final face and touch onset; delay as additional within factor: delay x own gaze x partner’s gaze: F _1,23= 4.6, p= 0.04; own gaze x partner’s gaze, long delays: F _1,23= 10.2, p= 0.004; short delays: F _1,23= 0.1, p= 0.7). The main effects of own gaze and partner’s gaze were not influenced by the duration of this delay.

Trustworthiness of the virtual partner

Trustworthiness ratings were similar over time (repeated-measures ANOVA over participants with time and actor as within factors; time: F _1,20≈ 0.0, p≈ 1.0, actor: F _8,160= 5.6, p= 0.000003, time x actor: F _8,160= 1.0, p= 0.4), so we averaged ratings obtained before and after the experiment. Faces with a positive and negative averaged rating were labeled ‘trustworthy’ (5 faces; score: +13.6 ±2.9 SEM) and ‘untrustworthy’ (4 faces; score: -5.6 ±4.6 SEM), respectively. The difference between these sets of faces was highly significant (t₍₂₃₎= 4.8, p= 0.00007), reflecting the consistency of the ratings across participants. Trustworthy faces differed from untrustworthy faces with respect to the interaction between own gaze and partner’s gaze (trustworthiness as additional within factor: trustworthiness x own gaze x partner’s gaze: F _1,23= 7.3, p= 0.01; repeating the analysis with 4 faces in each group, leaving out the trustworthy face with the lowest score, yielded equivalent results: trustworthiness x own gaze x partner’s gaze: F _1,23= 5.9, p= 0.02). There were no other effects of trustworthiness. The interaction between own gaze and partner’s gaze was significant for trustworthy but not untrustworthy faces (Figure 3A; own gaze x partner’s gaze, trustworthy faces: F _1,23= 18.7, p= 0.0002; untrustworthy faces: F _1,23≈ 0.0, p≈ 1.0). Partial analyses showed that trustworthy faces resulted in faster responses than untrustworthy faces when the touch was in a jointly attended hemispace, with no or less robust influences of trustworthiness in the other conditions.

Figure 3.

A) The same data as in Figure 2A, here plotted separately for trustworthy and untrustworthy virtual partners. The additional speeding of responses with jointly attended touch was present for trustworthy but not untrustworthy virtual partners. Reaction times for untrustworthy virtual partners showed a modulation by own gaze and partner’s gaze per se, but these effects did not interact, thus revealing no additional modulation by joint attention. Conventions as in Figure 2A.

B) When we used arrows as a spatial cue instead of gaze shifts by a virtual partner there was no additional speeding of responses when the participant’s gaze was shared with the arrows pointing direction toward the touch hemispace. Conventions as in panel A.

Social vs. nonsocial cues

Mean reaction time was 587 ms ±20 SEM (this reaction time includes approximately 80 ms that is needed for the tactile stimulator to accelerate and produce a perceivable vibration, see Methods and Figure 1C). The arrows were effective spatial cues (F _1,19= 5.4, p= 0.03), but contrary to gaze shifts, the influence of arrows did not interact with the influence of own gaze (Figure 3B; own gaze x arrow: F _1,18= 0.0, p= 0.9), although this difference between arrows and gaze cues was only marginally significant in an overall analysis (cue type as additional between factor: cue type x own gaze x cue direction: F _1,41= 3.2, p= 0.08).

Neural processing of touch

Two participants were excluded from these analyses because of recording artifacts at the fronto-central site-of-interest. There was a bilateral increase in theta power for all conditions. For joint but not disjoint attention contralateral alpha power was also increased (Figure 4A; 7-14 Hz, 175-250 ms time-interval, own gaze x partner’s gaze: F _1,21= 15.8, p= 0.0007). This difference in contralateral alpha power was not present before onset of the target (Supplementary Materials, Figure S1), validating its post-target onset. When the touched cheek was in a jointly unattended hemispace there was a concomitant ipsilateral increase in alpha power. Alpha power was thus lateralized to the contralateral side only when the touched cheek was in a jointly attended hemispace (Figure 4A; 175-375 ms time-interval: t₍₂₁₎= 3.9, p= 0.0007) and not in the other conditions (all t₍₂₁₎< 1.2, all p≥ 0.3; own gaze x partner’s gaze: F _1,21= 12.6, p= 0.002).

Across conditions more alpha lateralization was associated with faster reaction times (slope of regression line: t₍₈₆₎= -2.4, p= 0.02), while there was no relation with contralateral or ipsilateral alpha power. The above-reported alpha lateralization for touch in a jointly attended hemispace was narrowly localized to the site-of-interest for tactile stimulation of the face, corroborating it reflected facilitated touch processing (Figure 4B). There was no pre-target alpha lateralization (Supplementary Materials, Figure S1).

An overall analysis yielded no significant influences of trustworthiness on alpha lateralization. In separate analyses for trustworthy and untrustworthy faces, however, we observed a pattern in line with the behavioral results for trustworthiness (Supplementary Materials, Figure S2).

Discussion

Considering the important role of shared gaze in joint attention, we designed a modified gaze cue task that allowed shared gaze between the participant and the virtual partner and investigated interactions between own gaze and partner’s gaze direction. We found speeded responses to touch in a jointly attended compared with a singly attended or unattended hemispace. Importantly, this speeding reflected an effect unique to joint attention, as it exceeded the added effects of own gaze and partner’s gaze per se (Figure 2). To our knowledge no such automatic effect of joint attention based on shared gaze has yet been reported. The additional speeding was significant for long but not short delays between the partner’s gaze shift and the touch, whereas the effects of own gaze and partner’s gaze per se were not influenced by this delay. This suggests joint attention involves more extended neural processing than mere gaze following.

Underlining its social nature, the additional speeding was not observed when we presented arrows instead of gaze cues (Figure 3B; interpretation in line with: Becchio et al., 2008; Greene et al., 2009; Ristic et al., 2002), indicating that sharing own gaze direction with the pointing direction of an arrow does not elicit additional attentional orienting. Our results are in line with the idea that merely observing someone else’s gaze does not elicit socio-cognitive processes associated with reciprocal social interactions, while interacting through gaze shifts does (Caruana et al, 2017; Mundy & Jarrold, 2010; Schilbach et al, 2013; Shephard et al, 2009). Hence, they suggest that typical gaze cueing tasks are well-suited to investigate gaze following as a pre-stage of joint attention (Frishen et al., 2007), but are less suited to investigate joint attention as a social interaction, i.e., a more advanced social skill associated with the synergy between own gaze and partner’s gaze direction (Caruana et al., 2017; Mundy & Jarrold, 2010; Pfeiffer et al., 2013; Schilbach et al., 2013).

Neural processing of touch

Electro-encephalography (EEG) revealed that alpha power was lateralized to the contralateral site only when the touched cheek was in the jointly attended hemispace and not in the other conditions (Figure 4A), suggesting our behavioral results regarding response times were associated with enhanced sensory processing of the touch. Indeed, this alpha lateralization was found specifically at the site-of-interest for tactile stimulation of the face, corroborating its association with tactile processing. It occurred roughly 175-375 ms after touch onset, a latency at which bottom-up activation of primary and secondary somatosensory cortex has already occurred and extended tactile processing is ongoing (Nguyen et al., 2004/2005; Suzuki et al., 2004). It could thus reflect an enhancement of extended rather than initial touch processing, in line with the modulation of reaction times by joint attention, which existed for long but not short delays between the partner’s gaze shift and touch. Reaction times correlated with alpha lateralization but not with contralateral or ipsilateral alpha power, corroborating that more alpha lateralization was associated with faster responding.

Regarding contralateral alpha, we observed a power increase for joint but not disjoint attention (Figure 4A). This effect resembled previously reported contralateral alpha with nociceptive tactile stimulation, particularly when stimuli were attended (Hu et al., 2013; Ohara et al., 2004). It might reflect a general enhancement/prioritization of sensory processing when gaze direction is shared. In line with this interpretation, we found enhanced neural modulations prior to the touch that we attribute to increased hedonic value and/or priority of joint attention compared with disjoint attention (Supplementary Materials, Figure S1B). A concomitant ipsilateral alpha increase was found only when the touched cheek was in the jointly unattended hemispace (Figure 4A), possibly related to spatial attention being allocated to the untouched cheek.

Trustworthiness of the virtual partner

Even though all virtual partners were non-predictive and had a validated neutral expression, the modulation of reaction times by joint attention was present for virtual partners who were judged to be trustworthy but not for those who were judged to be untrustworthy (Figure 3A). Trustworthiness inferences are made reliably and automatically within a split second, even by young children (Cogsdill et al., 2014; Willis & Todorov, 2006), possibly reflecting an evolutionary advantage of detecting trustworthiness from facial appearance (Todorov, 2008; Willis & Todorov, 2006). Our results show that trustworthy partners elicited greater speeding of responses to jointly attended touch than untrustworthy partners, suggesting our first impression of someone influences the way we process and respond to jointly attended stimuli.

The above-reported alpha lateralization that was unique for jointly attended touch was not significantly modulated by trustworthiness, although there was a trend suggesting it was more robust for trustworthy than for untrustworthy faces (Supplementary Materials, Figure S2F).

We cannot exclude the possibility that the observed effects of trustworthiness reflect differences in visual characteristics between the trustworthy and untrustworthy faces rather than the trait trustworthiness. However, if so, it would be expected that gaze following per se is also affected by trustworthiness, an effect we did not observe, as only the interaction between own gaze and partner’s gaze direction was influenced by trustworthiness. Further supporting the idea that our results relate to trait trustworthiness, the visual characteristics of the faces were well-matched and we found no systematical differences between the features of trustworthy and untrustworthy faces.

Limitations

We here tested our modified gaze cue task for the first time, showing there is an interaction between own gaze and partner’s gaze regarding behavioral as well as neural responses. Future investigations with larger sample sizes could extend our knowledge of individual differences. For example, they could investigate whether individual differences or gender differences in gaze following (Frishen et al., 2007) are correlated with differences in the interaction between own gaze and partner’s gaze direction. Also, clinical populations with Autism Spectrum Disorders often perform normal in typical gaze cueing tasks, but may show abnormalities on our task that involves joint attention based on shared gaze (de Jong et al., 2008).

As we recorded neural activity from the scalp, we do not know precisely which brain regions the observed neural modulations originated from. Importantly, note that it is unlikely that our neural findings reflect motor preparation, because 1) sensorimotor activity associated with the hands is expected at sites located more medial, 2) response-related effects were averaged out when averaging across touch locations and 3) participants responded to the vertical location of the touch while gaze was manipulated in the horizontal dimension.

Given that one cannot see his/her own cheek, the present paradigm took advantage of the spatially unfocused nature of visuo-tactile cueing (visual cue and tactile target do not need to be at the exact same location; Graziano & Cooke, 2006; Spence et al., 2004; Wiese et al., 2014). The spatial precision of joint attention is, however, unknown. Would our results hold when the participant and the partner fixated two different stimuli on the same side, or when fixation stimuli were omitted and merely the fixation direction was instructed? Also, are our findings specific for the defensive zone around the face (Iachini et al., 2014; Brozzoli et al., 2012; Graziano & Cooke, 2006) or do they extend to other body parts (for example the hands; Soto-Faraco et al., 2005) or to further distances (away from the body and outside the range of normal social interactions)? These questions need further investigation. Another direction for future investigations could be to test whether results are the same for intentional rather than incidental joint attention, for responding to rather than initiating a joint attention episode and for real-life partners rather than virtual partners.

Although the arrow cues and gaze cues both gradually changed into a directional cue during a short movie, we cannot rule out that differences in motion, or other low-level visual features, contributed to the observed differences between gaze and arrow cues reported here. Notably, it has been shown that gaze cueing does not rely on motion (no difference between sudden onset gaze cue and implied motion gaze cue; Canadas et al., 2012; Kuhn & Kingstone, 2009).

Conclusion

We found that effects of gaze following are modified by the gaze direction of the participant. Specifically, when the touched cheek was in a jointly attended hemispace reaction times were faster than could be explained by the added main effects of own gaze and the virtual partner’s gaze direction. These behavioral effects were accompanied by neural modulations that indicated more elaborate tactile processing of jointly attended touch. Furthermore, the behavioral effects were absent when nonsocial cues (arrows) were used and when the virtual partner was considered untrustworthy at first impression. We conclude that joint attention influences the way we process and respond to stimuli at a jointly attended location, particularly when the partner is trustworthy. Our findings indicate that social interactions and trustworthiness judgements modify our behavioral and cortical responses to cross-modal sensory stimulation.