Social attraction in video-mediated communication: The role of nonverbal affiliative behavior

Abstract

The first aim of this study was to analyze video-mediated communication (VMC), in comparison to face-to-face (FTF) communication, and the effect it has on how communicators express nonverbal affiliative behaviors relevant for social attraction. Second, this study aimed to discover whether these nonverbal expressions relate to communicators’ social attraction. An experiment with 93 cross-sex dyads was conducted, with a get-acquainted exercise in a VMC or a FTF condition. Our findings revealed that communicators in VMC smiled more and spoke louder. In addition, VMC interactants displayed less facial touching than FTF interactants. Finally, more gaze aversion and a higher speech rate were found to influence social attraction. These findings have implications for research on cue-rich computer-mediated communication (CMC) and the way in which communicators nonverbally express themselves in comparison to copresent FTF communication. Additionally, this study has implications for social information processing theory which may be extended to include cue-rich forms of CMC.

Introduction

Nonverbal communication in face-to-face (FTF) interpersonal interactions has been studied extensively regarding how communicators express affection and form relationships (e.g., Coker & Burgoon, 1987; Ray & Floyd, 2006). However, research has not yet examined how nonverbal communication in video-mediated communication (VMC) relates to social attraction and relationship initiation. VMC platforms allow users to communicate with others by a combination of voice, video and text which means nonverbal cues can be transmitted (e.g., Manstead, Lea, & Goh, 2011). Although VMC more closely resembles FTF communication than previous forms of computer-mediated communication (CMC), there are two fundamental differences. First, VMC is mediated which implies an interaction between communicators who are not in each other’s direct physical environment; so there is a lack of physical copresence. Second, VMC is lower in information richness as the quality and speed of the video and audio signal can diminish the transmission of nonverbal cues (Bohannon, Herbert, Pelz, & Rantanen, 2013). Due to bandwidth limitations, it is difficult for VMC technologies to transmit all the sources of information available and at the same speed as in natural FTF communication (van der Kleij, Schraagen, Werkhoven, & de Dreu, 2009). Even in cue-rich VMC, communicators are typically only visible from the chest up which means they have a limited frame to communicate in. Consequentially, communicators are unable to transmit all of the cues that are available FTF, such as postural cues and social context cues.

These differences raise the question as to what extent the expression of nonverbal cues in VMC differs from FTF communication in the context of relationship initiation and development. Previous studies focusing on the display of nonverbal behavior in VMC, in comparison to FTF communication, did so in a variety of contexts, such as gameplay (Shahid, Krahmer, & Swerts, 2012), or gesture research (Mol, Krahmer, Maes, & Swerts, 2011). However, these studies did not investigate relationship initiation in VMC. Given the increasing use of communication technologies in interpersonal interactions (Ruppel, 2015), understanding how individuals in VMC use nonverbal behavior to communicate affection is an important question. Therefore, the first aim of this study is to analyze the effect of VMC on how communicators express nonverbal affiliative behaviors in initial interactions, by comparing it to FTF communication.

The second aim of this research is to discover whether nonverbal affiliative behavior in VMC and FTF communication relates to communicators’ social attraction. Although recent studies found that social attraction can be achieved in CMC just as well as in FTF communication, these studies mainly focused on reduced-cues CMC (e.g., Antheunis, Schouten, Valkenburg, & Peter, 2012; Burgoon & Le Poire, 1999). Previous research related the development of social attraction to visual anonymity, the presence of text, or the absence of nonverbal or social context cues (Antheunis, Valkenburg, & Peter, 2007; Walther, Slovacek, & Tidwell, 2001). However, central to these studies and research on CMC in general is the notion of reduced cues and these factors are unlikely explanations for attraction in VMC, which has both a visual and an auditory channel. Moreover, relationship initiation is likely to influence the process of nonverbal affiliative expression, as self-consciousness and concerns about self-presentation are high in initial interactions (Ruppel, 2015). At low levels of relationship development, individuals may be concerned with their self-image and the threat of rejection may be high. Hence, people may deliberately and strategically monitor their own and their partner’s (nonverbal) behavior, both consciously and unconsciously. Moreover, ineffective nonverbal communication may hinder the achievement of social attraction and reduce motivations for future interactions (Babin, 2013). Thus studying nonverbal behavior is vital in understanding how communicators in cue-rich CMC achieve social attraction and whether this differs from FTF communication in the context of relationship initiation.

This study combines research on relationship initiation, communication technologies, and nonverbal communication to propose an integrated perspective on nonverbal communication in VMC, compared to FTF communication.

Theoretical background

VMC versus FTF communication

VMC interactions closely mimic FTF interactions and function in much the same way (e.g., Fullwood, 2007). The addition of both a visual and an audio channel offers more possibilities compared to leaner CMC channels regarding the transmission of nonverbal cues, such as smiles, pitch variation, and vocal intensity. However, two important differences remain. First, in VMC, communicators are not physically copresent. When people are copresent, they can use the full range of linguistic, paralinguistic, and nonverbal cues to communicate (van der Kleij et al., 2009). Physical copresence influences the transmission of nonverbal immediacy behaviors, such as touch, body orientation, and body lean (Mehrabian, 1969), that create psychological closeness between people in initial interactions. Nonverbal immediacy behaviors are believed to communicate interest and warmth between communicators which may be easier to achieve FTF. Moreover, being physically copresent makes it easier to communicate information to one another and interpret the communication of the other person (Fullwood, 2007). Additionally, VMC technologies are believed to weaken the transmission of social context cues, such as features of the physical environment, available in FTF communication (van der Kleij et al., 2009).

Second, although VMC resembles FTF communication in terms of the number of cues that can be transmitted, the medium imposes technical constraints due to limited bandwidth. In VMC, a camera and monitor are used to communicate which means the environment participants are in as well as the information they transmit is distorted (Heath & Luff, 1993). Hence, the nonverbal behavior produced may not be similar to the behavior perceived (Heath & Luff, 1993). As a result, certain nonverbal movements may not have the same impact in VMC as they do in FTF. A communicator’s view of the other person is from one specific angle and a large part of the environment as well as the body of the interactant is not visible. Moreover, the audio channel in VMC differs from FTF, as there may be a delay in the audio signal and audio can be out of sync with video (Sanford, Anderson, & Mullin, 2004).

Thus, it may be that nonverbal cues are used and perceived differently in VMC and FTF as a means to compensate for the lack of physical presence and the limited bandwidth. Social information processing (SIP) theory poses that when fewer cues are available, people adapt to the remaining cues (Walther, 1992). Therefore, it may well be possible that people utilize certain nonverbal cues differently in VMC than FTF to compensate for a reduction in presence.

Although the limited bandwidth in VMC and the absence of physical presence may be viewed as restrictions to a conversation, in initial “getting to know each other” interactions, VMC may be a safer platform for social interactions than physically copresent communication. A lack of physical presence means fewer social skills are required and individuals feel an increased sense of control over the interaction (Philippot & Douilliez, 2011). Copresent FTF communication may more easily embarrass communicators, and cause tension in the form of distress and unease among communicators (Manstead et al., 2011). Additionally, in VMC, participants may feel less identifiable as they are only partially visible. Hence, they may feel less exposed and more anonymous in VMC, and less concerned with how they are coming across or are being evaluated by their interaction partner. This suggests that, for some people, VMC may be a safer environment than FTF communication, especially in initial interactions (Lapidot-Lefler & Barak, 2012).

Nonverbal cues: Multiple relational meanings

Thus, initial interactions in VMC are likely to differ from copresent FTF interactions. More specifically, the absence of physical copresence in VMC may protect communicators from certain social constraints, such as the perception and impact of (facial) rejection cues, normally present in FTF interactions (Manstead et al., 2011). As noted above, although nonverbal behaviors can be communicated through VMC, technological constraints may distort them so that they cannot be picked up (Fullwood, 2007). This distortion may result in communication which has less of a human feel to it. Since nonverbal cues are essential in the process of relationship initiation and impression formation, it may be that they do not have the same impact over a mediated system like VMC, compared to copresent FTF communication. If nonverbal cues are indeed perceived differently in VMC, compared to FTF, we may also react differently to the communication of these cues in initial social interactions.

The messages that nonverbal cues relay are not unambiguous (Burgoon & Le Poire, 1999). According to the social meaning model, nonverbal behaviors have connotative meanings that are culturally and socially constructed (e.g., Knapp, Hall, & Horgan, 2013; Ray & Floyd, 2006). Therefore, nonverbal cues have various social meanings in different contexts and cultures; eye contact, for instance, is often seen as a cue for affection (Gore, 2009), but it can also indicate anger or intimidation (Burgoon & Le Poire, 1999). The social meaning model attempts to show that nonverbal behaviors convey multiple social meanings that vary among senders and receivers. Therefore, when studying nonverbal behavior it is important not to focus on the social meaning of single cues but on the occurrence of multiple cues and how these may come across in interactions.

The present study focuses on the expression of the following visual cues: smiling, forward lean, postural matching, head nods, mutual gaze, and facial touching (e.g., Farley, Hughes, & LaFayette, 2013). Additionally, this study analyzes the following auditory cues: pitch variation, vocal intensity, speech rate, and vocal tension. It is likely that these nonverbal behaviors are less pronounced in VMC compared to FTF communication, due to the physical distance and the technical constraints the medium imposes in terms of bandwidth and potential delays in transmission (Manstead et al., 2011). Aspects of body orientation and postural cues may be best displayed and reciprocated in FTF communication where the entire body of the communicators is visible as well as the communicative environment (Burgoon & Le Poire, 1999; Mehrabian, 1969).

Nonverbal cues can have both positive and negative meanings. People can communicate positive affect through nonverbal involvement or immediacy cues. When individuals actively participate with each other in a conversation they show kinesic and vocal expressiveness as well as physical and vocal energy (Coker & Burgoon, 1987). Communicators involved in an interaction are believed to integrate their feelings and thoughts into an interaction, while uninvolved communicators come across as distracted, withdrawn and/or psychologically removed from the interaction (Burgoon, Buller, White, Afifi, & Buslig, 1999). Specifically, involvement often manifests itself in cues that signal vocal activity, speech rate, and postural relaxation (Dunbar & Burgoon, 2005). Additionally, nonverbal involvement in an interaction is marked by the occurrence of more relational gestures, head nods, facial expressiveness, and a direct body orientation (Coker & Burgoon, 1987). Finally, gaze signals a desire to communicate, while gaze aversion has primarily negative connotations and can reveal disinterest, detachment, and dislike toward an interaction partner (e.g., Burgoon et al., 1999).

Similar to involvement, immediacy is believed to enhance direct communication and closeness between communicators and lead to positive outcomes such as communication satisfaction, intimacy, and social attraction (Babin, 2013; Madlock, 2008; Manstead et al., 2011). Unlike involvement, immediacy involves the expression of implicit indicators of liking of a communicator toward an addressee (Walther, Loh, & Granka, 2005). Nonverbally, greater immediacy is communicated through variations in posture and proximity which enhance psychological closeness between two communicators (Burgoon & Le Poire, 1999; Mehrabian, 1969). Previous studies analyzed immediacy nonverbally in the form of postural lean, touch, smiling, and postural mirroring (Burgoon et al., 1999; Walther et al., 2005). Additionally, immediacy can be communicated through vocal cues like vocal pitch variation and vocal intensity (Ray & Floyd, 2006).

Although the aforementioned cues signal positive affect, communicators can also show negative affect nonverbally through postural cues, such as random head or trunk movements and postural shifting, and other cues such as self-touching (Coker & Burgoon, 1987; Mehrabian, 1969; Walther et al., 2005). Tense communicators often smile less and in addition express more vocal tension and a higher pitch (Coker & Burgoon, 1987). Nonverbal behaviors that suggest tension can come across as unpleasant and correspond to body positions which involve muscle tension and nervous activity (Mehrabian, 1969).

Hence, the expression of nonverbal affiliative behaviors differs depending on what medium communicators use and the constraints that medium imposes. However, it remains unclear whether and how communicators in VMC express nonverbal affiliative behaviors in comparison to their FTF counterparts, as most previous studies have not yet analyzed VMC with eye contact. How communicators in VMC nonverbally express themselves when they can transmit many of the same cues as in FTF interactions has to the best of our knowledge not been studied yet in the context of social attraction and relationship initiation. This is why a research question is posed:

• RQ1: To what extent do VMC and FTF communication differ regarding the expression of nonverbal affiliative behaviors?

Nonverbal cues and social attraction

The second aim of this research is to determine whether nonverbal cues in VMC and FTF communication influence social attraction. Previous CMC research shows that communicators adapt to a constrained communication situation (e.g., Jiang & Hancock, 2013), but those studies did not analyze VMC. The nonverbal affiliative behaviors outlined above may enhance or reduce social attraction. As said, the meaning nonverbal cues carry and the effect they may have is ambiguous and may vary according to sender and context (Burgoon & Le Poire, 1999) which is why we pose a second and a third research question:

• RQ2: To what extent do nonverbal affiliative behaviors result in the enhancement or impairment of social attraction?

• RQ3: To what extent is this effect more pronounced in VMC or FTF communication?

Method

Sample and procedure

An experiment was conducted among 186 undergraduate students (50.0% female) with an age ranging from 18 to 32 (M = 21.54; SD = 2.50). Most of the students were recruited from courses in communication (59.7%), while the remaining 40.3% came from various other undergraduate courses. The students were randomly assigned to a cross-sex dyad with an unacquainted individual. We chose cross-sex dyads because adding gender composition (same-sex and cross-sex dyads) to our experimental design could have led to power problems, as adding both male-only and female-only dyads to our design would mean collecting thrice as much data. For this reason, we focused on cross-sex dyads only. Moreover, we chose cross-sex dyads as previous research shows that opposite-sex interactions are livelier and elicit more arousal (e.g., Argyle & Dean, 1965). For this reason, we decided to include only cross-sex dyads as a means to provoke nonverbal affiliative behavior.

Participants were randomly assigned to one of two experimental conditions: FTF communication (n = 92) or VMC (n = 94). Both participants of a dyad were asked to report to different rooms to ensure they did not meet prior to the start of the experiment. A get-acquainted exercise was used for the experiment. Participants were instructed to get to know each other by means of a conversation during which they are free to talk about anything. Participants interacted for 12 min and were not allowed to end the conversation before this time limit. After the experiment, participants were asked to indicate whether they knew the other person prior to the interaction and none indicated they knew the other beforehand.

In the VMC condition, participants were seated in separate cubicles in the same building, but on different floors to ensure they would not run into each other prior to the experiment. In the FTF condition, the participants were led to an office where the conversation took place. During the conversation, they sat across from each other in a chair with a table in between them. In the VMC condition, participants communicated via an Eye-Catcher, a type of VMC with a high-resolution camera set up behind the screen; invisible to the participant. Because of the camera’s center screen location, participants were able to look at the screen and make eye contact at the same time (Shahid et al., 2012) without being able to see themselves. In the VMC condition, the Eye-Catcher was placed on a table in front of them. This mimicked the FTF condition, where participants sat across from each other at a table. Hence, in both conditions, participants sat at roughly equal distance from each other. This is not possible with webcams which are usually placed above the screen, making it seem as if the participants are looking down while communicating.

Both the VMC and the FTF conversations were recorded. After the experiment, participants were asked to fill in a short questionnaire and they did so individually in separate rooms, without having any further contact. During the experiment, the participants were unaware that they were being filmed, so this would not influence their conversation. After the experiment, participants were asked for permission to use their recordings for analyses to which none of them objected. Finally, participants were debriefed and either paid a small fee or given course credit for their participation.

Coding of visual nonverbal cues

Nonverbal visual behavior was coded on the video-only portion of the recorded conversations with the video annotation software program ELAN (version 4.6.2) (Wittenburg, Brugman, Russel, Klassmann, & Sloetjes, 2006). ELAN integrates the video image with a timeline on which one can place annotations, making it possible to analyze frequencies and the duration of particular cues. Two trained coders coded the video-only portion of the recordings which consisted of one video file for each participant. Nonverbal behavior was coded at a microbehavioral level by coding the occurrence of cues (Burgoon & Le Poire, 1999; Ray & Floyd, 2006). Microscopic measurement of nonverbal behavior involves single, concrete behaviors or cues (Burgoon & Baesler, 1991). Based on the recommendations given by Burgoon and Baesler (1991), nonverbal behaviour was coded by dividing the recordings into 2-min segments. Minutes 2:00–4:00, 6:00–8:00, and 10:00–12:00 were analyzed. These intervals were long enough for the coders to code the occurrence of multiple nonverbal behaviors and to establish the dynamics of behavioral change. The size of these intervals is appropriate to measure the occurrence of sequential (time-based) nonverbal behaviors and achieve high coder agreement (Burgoon & Baesler, 1991).

To establish intercoder reliability, two coders analyzed 40 (20%) separate video files, which were randomly selected. At the start of the coding process, the coders sat together to discuss the coding process and codebook. After that, they analyzed the same video file, after which any discrepancies and disagreements were discussed. For the 40 files, the intraclass correlation coefficient (ICC) was calculated for each variable in the codebook to determine the consistency of the relative frequency of the cues across both coders. An ICC between .41 and .60 is moderate, substantial between .60 and .80, and near perfect beyond .80 (Landis & Koch, 1977). All analyses were conducted by aggregating the individual scores of the participants of the three time intervals and subsequently collapsing those scores.

Head nods

The first cue was the frequency of head nods. The number of head nods per minute was tallied, both during listening and speaking of the communicator. The ICC was .90.

Gaze aversion

Coders tallied the frequency and duration of gaze aversion per minute, with the focus on the position of the communicator’s face. Coders needed to indicate how often communicators changed the position of their face for longer than 1 s. A communicator can avert gaze either through a left or right profile view or by looking upward or downward in an attempt to avoid eye contact. The ICC for the frequency of gaze aversion was .79. For the duration of gaze aversion, the ICC was .82. The final gaze aversion variable was formed by multiplying the mean frequency with the mean duration.

Smiling

To code a smile, a change in a communicator’s smiling behavior needed to be observed; when communicators smiled continuously, a new smile was only coded when they stopped smiling before starting to smile again. The number of smiles per minute was tallied and the intensity of the smiles was rated on a 7-point scale ranging from (1) no intensity to (7) high intensity. The ICC for the frequency of smiling was .86 and .71 for the intensity of smiling. A new variable was formed by multiplying the mean frequency of smiles with the mean intensity score.

Forward lean

Forward lean is a cue related to the position of the torso of a communicator. A forward lean was coded from the moment communicators changed their body position to accomplish forward lean, to the moment they changed their body position again. The frequency of forward leans per minute was coded and only coded when communicators changed their body position in order to achieve forward lean. In the VMC condition, this meant leaning toward the Eye-Catcher. The ICC for the frequency of forward lean was .56, which is moderate but acceptable.

Facial touching

To code facial touching per minute, coders indicated the frequency of facial touches where a new facial touch was coded when the communicator stopped facial touching or did something different (i.e., first touches his/her mouth and then his/her cheek). The ICC for the frequency of facial touching was .86.

Postural matching

The next cue was postural matching, which was coded according to a procedure by La France and Ickes (1981). Posture judgments were made from the video records by two coders. The first coder coded communicator A (female) and the second coder coded communicator B (male) of a dyad. These two coders indicated whether something happened to a communicator’s body position or hand movement above chest height. More specifically, the coding system consisted of the following cues: head nods, forward lean, and facial touching. The procedure required the coders to indicate whether and when the behavior occurred within a specific time interval. Additionally, for facial touching, the coders had to specify what the behavior entailed (i.e., face rubbing, touching the hair, chinrest/resting the head on one’s hands). Any alternative touch of the face was disregarded by the coders. On the timetable in ELAN, the coders indicated whether they observed one of the aforementioned cues, which behavior they observed and when it occurred. The final step of the coding process consisted of a third person examining the findings by comparing both timetables of the two coders for each time interval and for each variable. Subsequently, this person determined whether postural matching occurred. The following time intervals were set for each cue: head nods (5 s), forward lean (30 s), and facial touching (30 s; La France & Ickes, 1981). If two behaviors occurred outside of this interval, or at the exact same time, postural matching could not be established. Postural matching was measured as a frequency count, where (1) was the value given when postural matching occurred (so when two behaviors occurred within the same set time interval) and (0) was the value given when postural matching did not occur. Furthermore, the total number of occurrences in a conversation/dyad was summed and divided by the intervals in which the matching was coded. Like with all other nonverbal cues, we measured postural matching at three 2-min time intervals.

Coding of auditory nonverbal cues

In order to analyze vocal cues, the audio track of the conversations was analyzed with a software program called Praat (http://www.fon.hum.uva.nl/praat/). The audio recordings of all conversations for both experimental conditions were edited into 1-min segments of which only time intervals 2:00–4:00, 6:00–8:00, and 10:00–12:00 were analyzed. Those time intervals were subsequently evaluated in Praat which automatically analyzes speech and speech processes from both an acoustic and a physiological point of view. Although the coding of the visual nonverbal cues was based on a video recording for each individual participant, the coding of the auditory cues was based on a single audio recording of the conversation as a whole. Hence, we did not sum the individual measurements of the auditory cues, as the analyses in Praat were based on recordings of the conversations both participants had together, and not on individual recordings of the participants of a dyad.

Speech rate

Speech rate is used as a measure of fluency (de Jong & Wempe, 2009). To measure speech rate, we used a script in Praat, developed by de Jong and Wempe (2009), which detects syllable nuclei in running speech and has been found to correlate high with human-measured speech rate. Peaks in intensity following by dips in intensity are considered syllable nuclei and are measured in decibels. Speech rate is calculated by dividing the total number of syllables by the total speaking time (Ray & Floyd, 2006). All other vocal cues were measured in semitones. The semitone scale is a logarithmic transformation of the Hertz scale and is preferred when the purpose of the analysis is to measure and compare pitch intervals occurring around different mean frequencies. The Hertz scale is linear, while our production and perception of pitch usually is not (Nolan, 2003).

Pitch variation

Pitch variation was measured by taking the average variance of the total pitch and was measured in semitones (Boersma, 2004). More specifically, an average (statistical mean) was calculated for pitch, with the distribution about the mean (slope) indicating pitch variation (Ray & Floyd, 2006). Vocal expressiveness in the form of increased vocal pitch variation has been studied before as a measure of communicators’ positive evaluations of one another (e.g., Coker & Burgoon, 1987; Ray & Floyd, 2006).

Vocal intensity

We also analyzed vocal intensity (loudness) which is derived from the total pitch of the conversation. The pitch file displays dips and peaks in intensity and calculates a mean intensity value for the specified time intervals (Boersma, 2004). For vocal intensity, an average (statistical mean) was first calculated, after which a distribution about the mean was taken to show the average intensity (loudness) of the voice throughout a conversation (Ray & Floyd, 2006). This measure of vocal intensity was also measured in semitones.

Vocal tension

Finally, we measured vocal tension in Praat, in semitones, by analyzing the jitter in the participants’ voices which are deviations from or irregularities in the periodicity of a human voice (Boersma, 2004). Jitter occurs during vowel phonation and is defined as perturbations of the glottal source signal that affect the glottal pitch period (Vasilakis & Stylianou, 2007). Jitter is a fundamental metric of the quality of the voice and increased jitter (i.e., pitch perturbations) has been found to be associated with anxiety (Fuller, Horii, & Conner, 1992).

Self-report measures

Social attraction

To measure social attraction between members of a dyad, participants had to fill out a short questionnaire after the experiment with three items derived from the measurement of interpersonal attraction by McCroskey, McCroskey, and Richmond (2006). One additional item was added to the scale: “I would like to meet up with my conversation partner again.” The other items were “I think my conversation partner could be a friend of mine”, “I think my conversation partner is pleasant to be with”, and “I think my conversation partner would fit into my circle of friends.” The response categories ranged from 1 (completely disagree) to 5 (completely agree). The items formed a one-dimensional scale with an acceptable Cronbach’s α of .70 (M = 3.28; SD = .59). For the analyses, the social attraction score for each participant was recoded, so that the score did not indicate the rating they gave their interaction partner, but the rating of attraction they were given by their interaction partner. We switched the scores because we were interested to discover whether an individual’s nonverbal expressions would influence their interaction partner’s attraction toward them.

Results

In order to answer our research questions, we analyzed our data using multilevel modeling (MLM). All analyses employed restricted maximum likelihood estimation procedures and models were fitted with random intercepts and random slopes for all fixed effects (Barr, Levy, Scheepers, & Tily, 2013). In the present study, MLM was performed using the SPSS MIXED procedure (version 22). In all analyses, participants were the Level 1 measures that were nested within the dyads, which were Level 2 measures.

The first research question asked as to what extent VMC and FTF communication differ regarding the expression of nonverbal affiliative behaviors. In order to answer our first research question, multilevel analyses were performed for each of the 10 nonverbal cues separately. Therefore, we applied the strict Bonferroni correction which set the appropriate α level to .005 (.05/10). The analyses were performed with condition (two levels: FTF and VMC) as a fixed factor and the nonverbal cues as dependent variables. In each analysis, the experimental condition was the Level 2 predictor. In addition, the nonverbal cues postural matching, vocal intensity, pitch variation, speech rate, and vocal tension were modeled at Level 2, whereas head nods, forward lean, facial touching, gaze aversion, and smiling were modeled at Level 1.

The analysis revealed that the effect of condition on head nods, F(1, 184) = 1.00, p = .319, and gaze aversion, F(1, 184) = 2.56, p = .111,¹ was not significant. The effect of condition on speech rate was also not significant, F(1, 184) = 7.42, p = .007. For smiling, the analysis revealed a main effect of condition, F(1, 184) = 14.38, p < .001. In VMC, participants expressed more animated smiles (M = 16.95; SD = 7.95) compared to participants in the FTF condition (M = 12.67; SD = 7.44). The effect of condition on forward lean was not significant, F(1, 184) = 7.31, p = .008. The effect of condition on postural matching was not significant either, F(1, 184) = 2.10, p = .149. Next, the analysis revealed an effect of condition on pitch variation, F(1, 184) = 22.86, p < .001. As compared to VMC (M = 5.81; SD = .96), FTF communicators expressed more pitch variation (M = 6.52; SD = 1.07). The effect of condition on vocal intensity was also significant, F(1, 184) = 114.13, p < .001. In VMC, communicators expressed more vocal intensity (M = 61.95; SD = 4.45) compared to FTF (M = 53.74; SD = 5.94). Facial touching was also found to differ between the conditions, F(1, 184) = 23.80, p < .001. Communicators in FTF touched their faces more often (M = 1.70; SD = 1.59), compared to their VMC counterparts (M = .78; SD = .87). Finally, the effect of condition on vocal tension was not significant, F(1, 184) = .01, p = .905. All regression results, parameter estimates and effect sizes of the fixed effects in the multilevel analysis with the confidence intervals are displayed in Table 1, along with the means and standard deviations for both experimental conditions. As our first research question concerned the difference between two groups, namely FTF and VMC, we calculated Cohen’s d for the effect sizes of all nonverbal cues, which is the difference between the two group means divided by the within-group standard deviation (see Table 1 for the calculations; Snijders & Bosker, 2012).

Table 1.

Parameter estimates and effect sizes of all fixed effects and the means and standard deviations for the analysis of the effect of experimental condition on the 10 nonverbal cues.

	b (SE)	95% CI	F	p	Cohen’s d	M (SD) FTF	M (SD) VMC
Head nods	−.34 (.34)	[−1.01, .33]	.91	.341	−.15	4.72 (2.31)	4.38 (2.33)
Gaze aversion	−.05 (.03)	[−.11, .01]	2.56	.111	−.24	.16 (.23)	.11 (.19)
Speech rate	.16 (.06)	[.04, .28]	7.42	.007	.40	3.39 (.44)	3.55 (.36)
Smiling	4.28 (1.13)	[2.05, 6.51]	14.38	.000	.76	12.67 (7.44)	16.95 (7.95)
Forward lean	.22 (.08)	[.06, .38]	7.31	.008	.40	.34 (.46)	.56 (.62)
Postural matching	.37 (.26)	[−.13, .88]	2.10	.149	.21	1.25 (1.49)	1.62 (1.98)
Pitch variation	−.71 (.15)	[−1.01, −.42]	22.86	.000	−.70	6.52 (1.07)	5.81 (.96)
Vocal intensity	8.21 (.77)	[6.69, 9.73]	114.13	.000	1.56	53.74 (5.94)	61.95 (4.45)
Facial touching	−.91 (.19)	[−1.28, −.54]	23.80	.000	−.72	1.70 (1.59)	0.78 (0.87)
Vocal tension	−1.71 (.00)	[−.00, .00]	.01	.905	.00	.01 (.00)	.01 (.00)

Note. df for all nonverbal cues was 184. Significant predictors are in boldface. Since multiple tests were conducted for RQ1, a Bonferroni correction was applied using the following formula: α_{{per comparison}} = α/k, where k is the number of analyses conducted. The critical value for these analyses was α = .005. VMC = video-mediated communication, FTF = face-to-face.

Our second research question set out to determine which of the nonverbal cues would influence social attraction. To answer this question, MLM was performed using the SPSS MIXED procedure, where the nonverbal cues were the independent variables and social attraction was the dependent variable. For this analysis, the Level 2 predictors were postural matching, vocal intensity, pitch variation, speech rate, and vocal tension. Additionally, the Level 1 variables were head nods, forward lean, facial touching, gaze aversion, smiling, and social attraction. The analysis was run with the 10 nonverbal cues and social attraction as fixed factors. Table 2 gives an overview of the correlations between all nonverbal cues and social attraction.

Table 2.

Correlations between the nonverbal cues and social attraction.

	1	2	3	4	5	6	7	8	9	10
1. Head nods	1
2. Forward lean	−.05	1
3. Postural  matching	.54**	.11	1
4. Facial touching	−.04	.13	−.19*	1
5. Vocal intensity	.20**	.13	.50**	−.21**	1
6. Pitch variation	−.05	−.15*	−.10	.04	−.33**	1
7. Speech rate	.05	−.04	.02	−.09	.30**	−.24**	1
8. Vocal tension	.08	−.01	.20**	−.11	.22**	.24**	.35**	1
9. Gaze aversion	−.09	.08	.04	.13	−.10	.05	−.21**	−.03	1
10. Smiling	−.01	.15*	.06	−.08	.23**	−.13	.00	.00	−.11	1
Social attraction	−.01	−.08	.10	−.01	.12	.01	.16*	.08	.12	.06

*p < .05; **p < .01.

The analysis revealed no significant effect of head nods, smiling, forward lean, postural matching, pitch variation, vocal intensity, facial touching, or vocal tension on social attraction (all p’s >.11).² We did find a significant effect of speech rate, F(1, 175) = 5.50, p = .020, and gaze aversion, F(1, 175) = 4.72, p = .031, on social attraction. A higher speech rate of the participants and more gaze aversion¹ was found to result in a higher rating of social attraction by their interaction partner. The regression results, parameter estimates and effect sizes of the fixed effects in the MLM and the confidence intervals are displayed in Table 3. To calculate the effect sizes of the individual variables on social attraction, we calculated the proportional reduction in variance (PRV) statistic, which is a local effect size estimate frequently used in MLM analyses (Peugh, 2010). We used the following equation to calculate the PRV:

Table 3.

Parameter estimates and effect sizes of all fixed effects for the analysis of the effect of the 10 nonverbal cues on social attraction.

	Df	b (SE)	95% CI	F	p	PRVa
Head nods	175	−.02 (.02)	[−.07, .02]	1.05	.306	.003
Gaze aversion	175	.47 (.22)	[.04, .89]	4.72	.031	.074
Speech rate	175	.29 (.12)	[.05, .54]	5.50	.020	.002
Smiling	175	.01 (.01)	[−.00, .02]	1.44	.231	.058
Forward lean	175	−.13 (.08)	[−.29, .03]	2.60	.109	.023
Postural matching	175	.05 (.03)	[−.02, .11]	1.72	.191	.023
Pitch variation	175	.04 (.05)	[−.05, .13]	.70	.404	.026
Vocal intensity	175	.01 (.01)	[−.01, .02]	.36	.551	.026
Facial touching	175	.02 (.03)	[−.05, .08]	.26	.614	.020
Vocal tension	175	−19.54 (52.58)	[−123.31, 84.22]	.14	.711	.027

Note. Significant predictors are in boldface.

^aPRV is the proportional reduction in variance.

$$\text{PRV }=({\text{var}}_{\text{NoPredictor}}-{\text{var}}_{\text{Predictor}})/{\text{var}}_{\text{NoPredictor}}$$

In this equation, “var” presents the Level 1 variance, “NoPredictor” stands for the variance estimate from the model prior to adding a predictor, and “Predictor” represents the variance from the model that contains the predictor variable (Peugh, 2010).

Furthermore, we conducted additional analyses to answer RQ3, which asked whether the impact of the nonverbal cues on social attraction would be more pronounced in either the VMC or the FTF condition. To see whether the experimental condition may have a moderating effect, we split our dataset and reran our original analyses, but now for the FTF and VMC condition separately. These analyses revealed that for the FTF condition, gaze aversion was the only cue to come close to significance, F(1, 81) = 3.94, p = .051. None of the other cues significantly impacted social attraction in the FTF condition (all p’s >.090). For the VMC condition, we found a significant effect of speech rate on social attraction, F(1, 83) = 9.62, p = .003. A higher speech rate was found to lead to more social attraction in the VMC condition. None of the other cues significantly influenced social attraction in this condition (all p’s >.121).

Discussion

Our first research question asked as to what extent VMC and FTF communication differ regarding the expression of nonverbal affiliative behaviors. First, no significant differences between the conditions were observed regarding the expression of head nods, gaze aversion, and speech rate which are believed to signal conversational involvement (Coker & Burgoon, 1987). Higher speech rate signals responsiveness to one’s addressee in an interaction; which suggests that in both the VMC and the FTF condition participants were equally responsive. Moreover, it seems transmitting subtle eye and/or head movements like gaze and head nods is not a problem in VMC, as such small movements are easily transmitted on a monitor and do not require aspects of the environment or the communicator’s body to be visible. Moreover, because VMC allowed for eye contact in this study, both VMC and FTF communication were very similar regarding the expression of cues like mutual gaze and head nods.

Second, our findings revealed that in VMC participants smiled more animatedly and spoke louder. These cues are often employed when interactants wish to reduce the distance between them (Mehrabian, 1969; Walther, 1992). This suggests that the limited and distorted access in VMC may result in communicators emphasizing their own behaviors to achieve a certain impact (Heath & Luff, 1993). It may be that the expression of more animated smiles and vocal loudness are ways to decrease the physical distance between communicators in VMC. Hence, we conclude that in VMC, communicators attempt to compensate for the lack of physical presence as well as the technical constraints the medium imposes by being more expressive in their communication in the form of more facial and vocal expressiveness.

Third, FTF interactants displayed more facial touching and more pitch variation than VMC interactants. Although facial touching may signal tension or anxiety, pitch variation has been found to be associated with greater immediacy and affection (e.g., Ray & Floyd, 2006). However, research also shows that higher pitch may be an indicator of tension as well (Coker & Burgoon, 1987). Thus, the meaning of the expression of these cues is ambiguous. More specifically, communicators may express nonverbal cues related to nervousness or tension in initial interactions for many reasons. For instance, they may feel uncomfortable communicating FTF, as they are in close proximity to someone they do not know, or they could experience social anxiety. Moreover, they may even be nervous because they are attracted to their interaction partner and want to make a good impression. In fact, research shows that initial interactions may be uncomfortable, especially for individuals who have a difficulty initiating a conversation with a stranger (e.g., Finkel, Eastwick, & Matthews, 2007). From our findings, it is unclear why individuals expressed these cues and whether or not they actually felt nervous; or for what reason. Thus, it is difficult to conclude whether or not FTF communication in itself resulted in more tense communicators, or whether other factors played a role.

The effect of nonverbal cues on social attraction

Our second research question asked as to what extent nonverbal affiliative behaviors affect social attraction. Speech rate and gaze aversion were the only cues to influence social attraction. Moreover, our third research question asked whether the impact of nonverbal affiliative behaviors would be more pronounced in either the VMC or the FTF condition. The results showed that a higher speech rate positively influenced social attraction in the VMC condition only. Our findings suggest that the faster people spoke and the more and the longer they averted their gaze, the more attracted their interaction partners were toward them. Speech rate signals involvement in an interaction and is indicative of communication ease, efficiency and fluency (Coker & Burgoon, 1987). Moreover, faster speech is believed to show affinity (Coker & Burgoon, 1987). As a result, communicators who speak faster often appear as more pleasant and friendlier compared to uninvolved communicators (Burgoon et al., 1999). Our results show that this is indeed the case; but only in the VMC condition.

Gaze aversion positively influencing social attraction is in contrast with many studies that highlight the importance of gaze in interpersonal interactions (e.g., Bohannon et al., 2013; Burgoon et al., 1999; Walther et al., 2005). However, a distinction can be made between gaze and mutual gaze. It is possible to gaze at your communication partner without him/her matching your gaze. In this study, we analyzed the frequency and the duration of averted gazes, but we did not examine whether the interaction partners were matching each other’s gaze. Hence, we do not know how often mutual gaze was established and it is possible that whenever an individual gazed at their interaction partner it went by unnoticed. Studies show that interaction partners are generally evaluated more positively, the longer and less frequent their gazes are. Since we measured gaze as a combination of frequency and duration, this may explain our results. To determine whether this is in fact the case, we ran our analyses again with gaze frequency and duration as separate variables.¹ The analysis revealed that neither frequency nor duration of averted gazes alone impacted social attraction, which suggests that it is in fact only these two variables combined that influences social attraction.

Furthermore, previous research showed that there are sex differences when it comes to the use of gaze (Burgoon, Coker, & Coker, 1986). For this reason, we ran additional analyses for the effect of gaze on social attraction, for males and females separately, to control for a possible influence of sex. These analyses revealed that the effect of gaze aversion on social attraction was only significant for female participants.² This is in line with previous studies on cross-sex dyads and courtship/flirtation, which showed that females tend to take control in initial interactions through subtle nonverbal signals such as short glances at their partners (Grammer, Kruck, Juette, & Fink, 2000). Moreover, Scherwitz and Helmreich (1973) found that higher levels of mutual gaze result in less attraction, but only in same-sex dyads. As our study focused only on cross-sex interactions, it may be interesting for future studies to include both same- and cross-sex dyads to see if there is indeed a sex difference in both the expression and effect of nonverbal affiliative behavior in initial interactions.

Besides speech rate and gaze aversion, none of the other nonverbal cues directly affected social attraction. Hence, we did not find a conclusive answer to our second research question with regard to whether nonverbal affiliative behaviors result in the enhancement or impairment of social attraction. Thus, although we found that communicators in VMC spoke louder and expressed more animated smiles compared to FTF communicators, who displayed more facial touching and vocal pitch variation, none of these cues impacted social attraction. Moreover, speech rate was found to only enhance social attraction in the VMC condition and gaze aversion did not differ between VMC and FTF communication. Since these cues were the only cues to enhance attraction, this suggests social attraction developed largely independent of the communication mode and the expression of the majority of the nonverbal affiliative cues we examined in this study.

We propose two explanations for the finding that most of the nonverbal affiliative cues examined in this research did not impact social attraction. First, the study of nonverbal behavior is complex and multidimensional, as individuals express a substantial amount of nonverbal information, both consciously and subconsciously, which includes single cues and combinations of cues that may convey more than one meaning. According to the social meaning model, nonverbal behaviors can carry multiple social meanings which are different for both senders and receivers in an interaction (Ray & Floyd, 2006). Thus, in the context of this study, the nonverbal cues analyzed may have been interpreted by the participants as something other than affection or attraction toward them.

Second, studies suggest that nonverbal cues may be used differently and do not have the same effects in VMC, compared to FTF communication (e.g., Heath & Luff, 1993). More specifically, our results revealed that speech rate enhanced social attraction in VMC only. In addition, the effect of gaze aversion on social attraction was close to significance in the FTF condition. This suggests that the effects of these two nonverbal behaviors on social attraction varied based on the communication condition. Certain subtle nonverbal behaviors may be less noticeable in VMC or not come across at all (Sanford et al., 2004). Moreover, studies on VMC show that nonverbal signals are often distorted and hence not picked up by others (Heath & Luff, 1993). Even when video quality is high, nonverbal signals are still found to have less of an impact in VMC, compared to FTF communication (Fullwood, 2007), which may explain why gaze aversion only enhanced social attraction in FTF communication. However, our results also show that an auditory cue like speech rate does have an impact in VMC, which suggests that different cues enhance social attraction in different communication modes.

Theoretical implications

A first implication is that when studying new CMC technologies like VMC, we need to move away from discussing the number of cues various media have and move toward creating a more coherent overview of the differences between mediated and non-mediated technologies. When the transmission of both visual and auditory cues is possible, people still express themselves differently depending on whether or not a conversation is mediated. Hence, it may be that the aspect of mediation itself, or the absence of physical presence, influences the way people express themselves in an interaction. VMC may be a more attractive medium as people are not in the same room. They may be less easily embarrassed and more comfortable compared to a copresent FTF interaction.

Second, the role of nonverbal cues in the achievement of social attraction may be less significant than previous studies claim. The present study demonstrates that in comparing FTF communication and VMC, with the exception of speech rate and gaze aversion, these cues do not explain why individuals are attracted to one another, at least not in an open conversation in which people have to get to know each other. This is in contrast with ample research that highlights the importance of nonverbal cues in relationship initiation and development (e.g., Babin, 2013; Burgoon et al., 1999; Ray & Floyd, 2006). However, a recent study by Ta, Babcock, and Ickes (2016), that also examined nonverbal behavior in initial interactions, found that nonverbal behaviors like smiling, gaze, and head nods did not impact the outcomes of the interactions. The authors concluded that while these behaviours are an important part of communication, the key to establishing common-ground understanding in initial interactions is verbal, and not nonverbal, communication. Moreover, previous studies on CMC and social attraction demonstrated that when people get to know each other, they use the cues they have at their disposal to express affection and become attracted toward each other, for example, verbal statements of liking (e.g., Walther et al., 2005). In fact, previous research found that explicit statements of liking are more common in CMC compared to FTF interactions (e.g., Antheunis et al., 2012). Thus, it may be that in initial interactions, nonverbal cues have a relatively small effect on social attraction compared to, for instance, verbal cues.

Finally, this study has important implications for SIP theory (Walther, 1992). The findings of this research show that in VMC, communicators display more immediate nonverbal behaviors compared to their FTF counterparts. This suggests that despite the availability of nonverbal cues in CMC, communicators still feel the need to compensate for the physical distance and do so through the expression of nonverbal immediacy cues. Hence, even in cue-rich VMC, communicators still adapt to the medium. SIP currently mainly focuses on text-based CMC and thus identifies different strategies in the adaptation process, such as the use of emoticons and chronemics. We believe the results of the current study show that SIP can be extended to include cue-richer forms of CMC, such as nonverbal immediacy cues to compensate for the limitations of mediated interactions.

Limitations and suggestions for future research

Although our study provides new insights into how communicators express nonverbal affiliative behaviors in initial interactions in VMC and FTF communication and how these behaviors relate social attraction in the context of relationship development, our study is not without limitations. First, although we coded every dyadic conversation for visual nonverbal cues for each participant separately, we could not do so for the auditory cues as we were unable to isolate the individual voices. Therefore, we were unable to code variables like speech rate and pitch variation for each participant individually. Although we did try to include auditory cues identified in previous research as elements of the communication of liking (e.g., Ray & Floyd, 2006), males and females may differ in the use of these cues. Hence, it would be interesting for follow-up research to analyze nonverbal auditory affiliate behavior for males and females separately to see if there are sex differences in the expression of these cues.

Furthermore, in the present research, nonverbal behavior was coded at a microlevel by coding the occurrence of single cues. Another way of coding nonverbal behavior is by coding each conversation as a whole on the basis of global gestalt-type measures on perceptions of, for instance, involvement, warmth, friendliness, and pleasantness (Burgoon & Le Poire, 1999). Research shows that apart from individual visual and auditory cues, nonverbal affiliative behavior has also been linked to the encoding of global perceptions of behaviors (Ray & Floyd, 2006). This is because individuals are believed to process individual cues at a more global level. Although the purpose of the present research was to discover sequences of individual behaviors that may signal attraction, future research could extend these findings by including global gestalt-type attributes as well. Individual nonverbal affiliative cues may be related to more proximal interaction qualities, such as warmth and friendliness, which may, in turn, influence social attraction in initial interactions (Burgoon & Baesler, 1991). Thus, future research on nonverbal affiliative behavior could include both concrete nonverbal behaviors and proximal perceptions to create a more coherent overview of nonverbal affiliative behavior in initial interactions, as well as a possible relationship between the occurrence of individual behaviors and global interaction attributes (Burgoon & Baesler, 1991).

Moreover, in the present research, we investigated both the occurrence of single cues and the duration of specific cues, such as gaze aversion. We coded the duration of gaze aversion as research shows that high levels of direct gaze (90% of the time) are related to more immediacy and more social and physical attraction (Burgoon, Manusov, Mineo, & Hale, 1985). However, coding the duration of other cues, such as smiling or forward lean, may have resulted in more significant findings. More specifically, a smile was only coded after a change in an individual’s facial expression occurred; so if participants smiled continually, we did not code this as a smile. Thus, if a participant was continuously smiling, the coding process did not capture this and this is a limitation of the present research. Including both the frequency and duration of single cues may have led to more significant findings, which is something future research might explore.

In addition, in this study, we analyzed whether an individual’s use of specific nonverbal cues, like gaze, would influence the other’s attraction toward them. Hence, we analyzed whether the use of a cue like gaze by interaction partner A in a dyad would influence partner B’s attraction toward partner A. In contrast, it could be that the use of these nonverbal affiliative cues communicates an underlying message. More specifically, it could be that partner A’s use of gaze, or another nonverbal cue, signals their attraction toward partner B. Thus, the more attracted partner A is to partner B in an interaction, the more he or she uses specific nonverbal cues to signal this attraction, whether consciously or not. Hence, just as specific nonverbal cues may reflect immediacy or involvement (Coker & Burgoon, 1987; Ray & Floyd, 2006), there may be certain cues that individuals use in initial interactions as their attraction for their conversation partner grows. Few studies have analyzed the effects of nonverbal cues, like gaze, during an interaction. An early study by Williams and Kleinke (1993) found that mutual gaze in initial, cross-sex interactions elevates blood pressure and heart rate which may suggest attraction. It would be interesting for future research to analyze whether the use of specific nonverbal cues, like gaze, are in fact employed more frequently by individuals as a means to express their attraction toward their conversation partner in initial interactions. This could be done by measuring the heart rate during an interaction, or by analyzing verbal statements of affection throughout a conversation to see if an increase of mutual gaze is accompanied by an increase in the use of these statements.

The effect nonverbal cues have in the context of relationship development may also differ cross-culturally. Research suggests that the ways in which communicators adopt levels of gaze, proximity, and other nonverbal cues are derived in part from cultural norms (Argyle & Dean, 1965). For example, while long periods of eye contact can be uncomfortable for most Americans, in Japan this can be seen as a sign of understanding and respect (Barnlund, 1989). Additionally, greater nonverbal immediacy behaviors are believed to correspond to more positive attitudes only if those behaviors do not exceed those expected by the norms of a certain (sub)culture (Mehrabian, 1969). Moreover, the communication of emotions is also believed to be influenced by cultural norms for the expression of emotions (Manstead et al., 2011). These findings show that cultural norms may play an important role in the expression of nonverbal affiliative behaviors.

Finally, as noted above, in our study we focus exclusively on the role of nonverbal behavior in the context of relationship development, while previous research also highlights the importance of verbal cues, such as self-disclosure or verbal statements of liking, in the development of close relationships (e.g., Ruppel, 2015). As our findings suggest that nonverbal cues have a relatively small effect on social attraction in initial interactions, it may be interesting for future research to analyze both verbal and nonverbal affiliative behaviors to discover whether there is a difference in how individuals express themselves both verbally and nonverbally in VMC and FTF communication in other relationship contexts.

Conclusion

The goal of this study was to analyze VMC in comparison to FTF communication in terms of how nonverbal affiliative behavior is expressed and the effect these behaviors have on social attraction in the context of relationship initiation. Our results revealed that in VMC, communicators attempt to compensate for the absence of physical presence and the technical constraints by smiling more and speaking louder. Furthermore, our findings show that FTF communicators expressed more facial touching, which may be a cue for tension, but also expressed more varied pitch which is believed to communicate liking. This suggests that FTF communicators may have been slightly more nervous. Finally, even though there were differences in how communicators nonverbally expressed themselves in VMC and FTF communication, only gaze aversion and speech rate were found to influence social attraction. Hence, although this study revealed differences in how individuals express themselves in a get-acquainted interaction in VMC, as compared to FTF, for the most part these differences do not appear to impact how much participants like each other.