Comparison of Deliberate and Spontaneous Facial Movement in Smiles and Eyebrow Raises

Abstract

We investigated movement differences between deliberately posed and spontaneously occurring smiles and eyebrow raises during a videotaped interview that included a facial movement assessment. Using automated facial image analysis, we quantified lip corner and eyebrow movement during periods of visible smiles and eyebrow raises and compared facial movement within participants. As in an earlier study, maximum speed of movement onset was greater in deliberate smiles. Maximum speed and amplitude were greater and duration shorter in deliberate compared to spontaneous eyebrow raises. Asymmetry of movement did not differ within participants. Similar patterns contrasting deliberate and spontaneous movement in both smiles and eyebrow raises suggest a common pattern of signaling for spontaneous facial displays.

Introduction

An emerging body of empirical research on spontaneous facial movement has reported basic dynamic characteristics of common spontaneous facial displays, particularly smiles (Frank et al. 1993; Schmidt et al. 2003; Schmidt et al. 2006; Tarantili et al. 2005). Current research has indicated that there is likely an underlying consistency to the dynamic characteristics of spontaneous smiles, including speed, amplitude, and duration of movement and possibly asymmetry of movement. As compared to deliberate smiles, spontaneous smiles are relatively slower and smaller in onset amplitude than posed smiles of the same subjects (Schmidt et al. 2006).

It is unknown whether these movement patterns can also be observed in other spontaneous facial movements, such as eyebrow raises. Automated movement analysis has been extended to the measurement of eyebrow movement, with success in recognizing frequent facial movement combinations such as the simultaneous raising of inner and outer parts of the eyebrow, although specific facial movement measurements were not reported (Cohn et al. 2004). A preliminary study attempting to classify deliberate (volitional) and spontaneous eyebrow raises in different sets of subjects using automated measures showed that movement factors were important in distinguishing spontaneous actions (Valstar et al. 2006). Amplitude, maximum speed, and duration contributed to correct classification of eyebrow raises, although these results must be interpreted with caution as deliberate and spontaneous movements were obtained from individuals in different data sets (Valstar et al. 2006). Observed contrast between deliberate and spontaneous movements and the high frequency of eye brow raises in interaction (Cohn et al. 2004) suggest that eyebrow raises may make an interesting comparison with spontaneously produced smile movements. In this study, we compare deliberate and spontaneous facial movements within individuals, focusing on two frequently observed facial displays: the smile and the eyebrow raise.

The average onset period duration of spontaneous smiles has recently been reported as ranging between .5 and .75 seconds. Solitary spontaneous smiles had an average onset duration of .52s (Schmidt et al. 2003), and spontaneous smiles produced in social contexts had average durations of .50 s, .59s, and .67s (Schmidt et al. 2006; Schmidt et al. 2003; Tarantili et al. 2005). Smile onset duration in each case is brief enough to be consistent with the overall timing originally reported for spontaneous smiles (Ekman and Friesen 1982; Hess and Kleck 1997). Spontaneous smiles have been described as signals with a brief (<.75s) onset and relatively slower and smaller movement at onset than deliberate smiles¹. Although there is little quantitative data available for eyebrow movements, an observational study indicated a relatively short duration of spontaneous onset (.1 s) (Grammer et al. 1988).

The case has also been made that spontaneous expressions of emotion are less asymmetric than deliberate ones (Hager and Ekman 1997; Skinner and Mullen 1991). Studies of asymmetry using measured rather than coded movement have suggested that this is not the case; lip corner movement in both is actually fairly symmetric (Schmidt et al. 2006). Asymmetry also did not distinguish deliberate and spontaneous eyebrow movements (Valstar et al. 2006). Asymmetry of onset timing was found in a preliminary study (Cohn and Schmidt 2004), but this finding was not confirmed in a subsequent study employing the same methods with a larger sample and within subject design (Schmidt et al. 2006). The question of the amount of asymmetry in spontaneous facial movement is also complicated by the possibility that asymmetry results from asymmetry of underlying facial structure rather than any inherent asymmetry in facial movement itself (Schmidt, et al. 2006).

Comparing spontaneous smiles and eyebrow raises allows us to consider the differences in two facial displays commonly observed in social interactions. Because both are socially oriented facial displays, it is possible that both will exhibit similar movement characteristics (Schmidt and Cohn 2001). Muscles of facial expression in the upper face, however, show a more bilateral pattern of motor control. Voluntary control of these muscles is not as great as those in the lower face (Rinn 1984). Given these contrasts in motor control, it is also possible that the quality and dynamics of spontaneous facial movements will vary across the face, resulting in eyebrow raises and smiles that exhibit contrasting movement characteristics.

In the current study we aim to replicate earlier findings on facial movement in smiling and to extend this approach to the study of spontaneous eyebrow raises, investigating maximum speed, amplitude, duration, and asymmetry of facial movement during movement onset. We predicted that deliberateness of movement would affect maximum speed, amplitude, and duration of onset (spontaneous smiles would be slower, lower, and longer, as would spontaneous eyebrow raises). We also predicted that deliberate and spontaneous eyebrow raises would both be asymmetric, as individual differences in anatomical facial asymmetry have previously been shown to account for much of the asymmetry of the display (Schmidt et al. 2006).

Method

Material

The facial movements in this study were collected during a videotaped interview between a single experimenter and each participant, conducted as part of a study of facial displays by the first author². Participants were 57 adults (16 men and 41 women) recruited among staff, faculty, and students associated with the University of Pittsburgh. Participants provided informed consent and rights of the participants were protected according to the approved human subjects protocol.

Deliberate movements in the current study are defined as those produced volitionally in response to a specific interviewer request. They were collected from a facial movement assessment in which participants were instructed to “raise your brows” and also to “smile with your usual smile”. The first deliberate smile and eyebrow raise produced in response to interviewer instructions were digitally captured for analysis. Smiles and eyebrow raises were sampled from the tape, with the guidance of a certified Facial Action Coding System (FACS) coder (Ekman et al. 2002).

Spontaneous movements in the current study are defined as those produced during the interview but outside of an explicit request to smile or to raise the eyebrows. The first observations of spontaneous smiling and of eyebrow raise in the interview were sampled, without regard to the immediate social or emotional context. Spontaneous movements were obtained from either the initial portion of the interview or from subsequent portions in which participants responded to verbally administered questionnaires about their social contacts and their experiences of past events. Movement sampling proceeded throughout the interview until spontaneous displays were found, or until the end of the interview was reached. Utilizing spontaneous displays sampled from various portions of the interview allowed us to maximize the number of within subject comparisons in the study, although we acknowledge that slight differences in context may have introduced variation into the spontaneous movement sample.

Presence of specific facial movements accompanying smiles and eyebrow raises was recorded. Orbicularis oculi activity (AU6; Ekman et al. 2002)) during smiles was recorded by a trained and certified FACS coder. Dampening movements during smiling (AU 14, 15, 17, 23, 24) were also recorded, as were smiles (AU12) and speech (AU 50) occurring during eyebrow raises. Inter rater agreement for these action units was .87. Spontaneous smiles were sampled first. In the event that the first spontaneous brow raise occurred during the first spontaneous smile, the next spontaneously occurring eyebrow raise was sampled. Nine participants (16%) did not display spontaneous eyebrow raises over the course of the entire interview, reducing the number of individuals with coded eyebrow movement to 48.

Automatic Facial Image Analysis (AFIA) and Measurement of Facial Movements

Automated facial image analysis (AFIA) was performed on all smiles to measure movement parameters, using digitized video frames as described previously (Cohn and Kanade 2004; Schmidt et al. 2006). Individual facial features (lip corners and eye brow corners) were delineated manually in the initial frame (Fig. 1.) and then tracked by the program through successive frames (Lien, et al. 2000). Head motion was also tracked automatically and used to stabilize sequences so that movement of facial features due to head motion was effectively removed.

Figure 1.

Facial Feature Points Measured by Automated Facial Image Analysis (Points 1, 4: Outer Brow; Points 2,3: Inner Brow; Points 5,6: Lip Corner)

Following an earlier study (Schmidt et al. 2003), movement of facial features was represented as the mean displacement of the lip corner points, computed as $d=\sqrt{x^2+y^2}$. These values were standardized by dividing each value by the initial distance between the lip corners (for smiles) or by the distance between inner eyebrow corners (for eyebrow raises). Data series were smoothed. The periods of longest continuous increasing value of d were designated as onset of movement (smile or eyebrow raise) ³. Onsets were identified for left and right lip corners and left and right eyebrow inner corners in each sequence. Maximum speed was measured as the maximum frame to frame difference in d occurring during onset, and amplitude was measured as the total change in d from beginning to end of onset. Maximum speed and amplitude were reported as proportional values, standardized on the initial sequence video frame. Duration of onset was measured in seconds. Several eyebrow raises were not tracked successfully, reducing the number of participants with data from both deliberate and spontaneous eyebrow raise data to n = 42.

Data Reduction

Measures of lip corner movement were obtained for each smile on both left and right sides (Fig. 1). Measures of inner eyebrow point movement from both left and right sides of the face were obtained for each eyebrow raise (Fig. 1.). Movement variables for smiles and eyebrow raises were calculated by averaging these values. Movement asymmetry variables were calculated by taking the difference of right and left values of duration, maximum speed, and amplitude for smile and eyebrow raise onsets.

Both movement and asymmetry measures for smiles and eyebrow raises were then transformed to yield Z scores. Original values for movement variables were retained and reported for comparison with earlier studies.

Analysis and Results

Descriptive Movement Analysis of Deliberate and Spontaneous Smiles and Eyebrow Raises

Values of maximum speed, amplitude, and duration for deliberate and spontaneous smiles (Table 1) and eyebrow raises (Table 2) were obtained. Asymmetry of maximum speed, amplitude, and duration of movement onset were also obtained for deliberate and spontaneous movements (Tables 1 and 2.).

Table 1. Maximum Speed, Amplitude, Duration of Movement, and Movement Asymmetry in Smiles.

	Deliberate (n = 57) M (SD)	Spontaneous (n = 57) M (SD)	F	ηp2
Max. speed	.023(.013)	.016(.009)	12.95**	.19
Amplitude	.143(.065)	.116(.064)	2.29	.04
Duration (s)	.544(.174)	.600(.245)	2.65	.05
Asym max speed	.025(.027)	.025(.022)	.03	.001
Asym amplitude	.161(.142)	.211(.164)	1.40	.03
Asym duration (s)	.121(.132)	.169(.163)	1.16	.02

Note. Max = Maximum. Asym = Asymmetry.

^**p < .01

Table 2. Maximum Speed, Amplitude, Duration of Movement and Movement Asymmetry in Eyebrow Raises.

	Deliberate (n = 57) M (SD)	Spontaneous (n = 57) M (SD)	F	ηp2
Max. speed	.094(.035)	.077(.044)	6.65*	.14
Amplitude	.437(.145)	.314(.129)	25.61**	.35
Duration (s)	.489(.178)	.738(.543)	12.71**	.24
Asym max speed	.011(.010)	.015(.029)	.38	.01
Asym amplitude	.048(.049)	.049(.042)	0	.00
Asym duration (s)	.134(.173)	.074(.142)	1.20	.03

Note. Max = Maximum. Asym = asymmetry.

^*p <.05.

^**p < .01.

Effects of Deliberateness and Participant Gender on Facial Movement

Repeated mixed multivariate analysis of variance was conducted to test the effects of facial action type (deliberate or spontaneous) on movement. Separate analyses were conducted for smiles and eyebrow raises. Deliberateness was designated a within subjects factor. Gender of participant was included as a between subjects factor to test for the effects of the interaction of deliberateness and gender on facial movement. Each analysis included only individuals that had movement data available for both deliberate and spontaneous movement: smiles (n = 57) and eyebrow raises (n = 42). Z transformed measures of speed, amplitude, and duration were used in both analyses and significance and effect size of test statistics was reported. Effect sizes were reported and significance levels were set at p = .05; significance was reported with values corrected for multiple tests (Bonferroni correction).

There was a main effect of deliberateness on movement in smiles and eyebrow raises (F (3,53) = 4.89, p <.01; F (3,38) = 12.11, p < .01, for smiles and eyebrow raises, respectively; Table 1 and 2.). The interaction of deliberateness and gender did not affect movement for either smiles or eyebrow raises (all F values < 2, p > .10). Results of univariate tests in the model showed that deliberate smile onsets were faster (F (1,55) = 12.95, p <.01, for maximum speed), although amplitude and duration did not differ significantly (all F values <2.7, p > .10). Deliberate eyebrow raise onsets were faster, higher amplitude, and shorter in duration than spontaneous eyebrow raises (F (1,40) = 6.65, p < .05; F (1,40) = 25.61, p < .01; F (1,40) = 12.71, p < .01, for maximum speed, amplitude, and duration of onset, respectively).

A separate repeated mixed multivariate analysis of variance was conducted for onset asymmetry variables (Z score transformed asymmetry values) in smiles and in eyebrow raises. There was no main effect of deliberateness on movement asymmetry and no effect of the interaction of deliberateness and gender on movement asymmetry (F values < 2.00, p > .10).

Additional Facial Movement in Smiles and Eyebrow Raises

Half of the participants (.56 [n = 32]) showed evidence of O. oculi activity (the duchenne marker, AU6) in their deliberate smiles (Table 3). Most participants also showed O. oculi activity in their spontaneous smiles (.97 [n = 55]). Some deliberate (.26 [n = 15]) and spontaneous smiles (.19 [n = 11]) had dampening of lip corner movement (movements that affect lip corner raising in smiles, Table 3).

Table 3.

Frequency of accompanying facial movements during smiles and eyebrow raises (Facial Action Coding System (FACS) (Ekman et al., 2002)

	Proportion (frequency) with accompanying movement/speech
	Movement type
	Smile		Eyebrow raise
	D (n=57)	S (n=57)	D (n=57)	S (n=48)
Duchenne marker O. oculi (AU6)	.56(32)	.97(55)
Smile dampening1	.26(15)	.19(11)
Smile Z. Major (AU12)			.09(5)	.29(14)
Speech (AU50)			0(0)	.58(28)

¹Note. A combination of AU12 and any of the following actions with dampening effect: AU14, AU15, AU17, AU23, AU24 (Ekman et al., 2002). D = Deliberate. S = Spontaneous

Of those deliberate eyebrow raises coded (N = 57), 9% showed Z. major action (smile [AU12], Table 3.). There was no speech during deliberate eyebrow raises. Coded spontaneous eyebrow raises (N = 48) showed a greater frequency of Z. major action (.29 [n =14]), and were accompanied by speech in half of the participants (.58 [n = 28]).

Discussion

Differences Between Deliberate and Spontaneous Movement in Facial Displays

As expected, we found that deliberate smiles showed higher maximum speed than spontaneous ones. Deliberate eyebrow raises followed the same pattern, with higher amplitude and maximum speed, in addition to shorter duration compared to spontaneous eyebrow raise onsets. These results are consistent with an earlier study which found that deliberate smiles were significantly faster and higher amplitude than spontaneous (Schmidt et al. 2006). The finding that deliberateness is a significant factor in the pattern of movement of smiles and eyebrow raises indicates that control of these movements may have some consistency as a result of their social signaling functions.

Facial movement in the current study was congruent with movement patterns found in observational studies of facial displays. A study of children's smiles found similar onset duration of spontaneous smiles, although deliberate smiles were not investigated (Tarantili et al. 2005). Hager and Ekman (1997) originally suggested that spontaneous smile onset would be under .75 seconds and our results are within this limit. These studies lend validation to our interpretation of current findings, despite the fact that they are based on different methods and conceptualization of facial display onsets.

In a study employing facial coding (FACS [Ekman et al. 2002]), however, displays of posed happy emotion were found to be longer than spontaneous emotion elicited happy smiles (Hess and Kleck 1997). Previously reported duration for coded observations of spontaneous eyebrow raises (Grammer et al. 1988) was shorter than we observed. These contrasts are possibly due to the way in which duration was measured and differences in the definition and sampling of deliberate and spontaneous displays.

The potential future implications of the current study for future research lie in the recognition of the unique nature of spontaneous facial displays; spontaneous displays are smaller, subtler, and more complex than deliberate ones (Schmidt and Cohn 2001). Spontaneous facial displays, inasmuch as they function as signals between individuals in close proximity, are expected to be smaller and this was confirmed in the current study. Similar contrasts in timing of spontaneous compared with deliberate movement in both eyebrows and lip corners suggest that there may be a common spontaneous facial movement control mechanism that coordinates spontaneous facial signaling.

Facial Movements Accompanying Deliberate and Spontaneous Smiles and Eyebrow Raises

The presence of O. oculi activity (Duchenne marker) in spontaneous smiling is consistent with an interpretation of spontaneous smiles as more likely to signal positive emotion, although we did not assess emotion here. As was previously found, however, a relatively high proportion of participants (.56) also showed the Duchenne marker in their deliberate smiles (Table 3; Schmidt et al. 2006). The presence of the Duchenne marker in deliberately produced smiles suggests that this action can be produced voluntarily and is part of the repertoire of typical deliberate smiles in some individuals.

Similar proportions of deliberate and spontaneous smiles showed dampening of the lip corner movement by the presence of additional facial movements around the mouth (Table 3.). Dampening has been associated with mixed emotion, or with non joyful feelings (Ekman and Friesen 1982). The presence of dampening in smiles elicited by the deliberate instruction to “smile with your usual smile”, however, is less easily interpreted. The low frequency of dampened smiles in the current study precluded testing of the effects of this variant of smiling.

Spontaneous eyebrow raises showed a greater degree of accompanying Z. major action (.25) and were accompanied by speech in half of all participants (.58). The combination of spontaneous eyebrow raise with smiling and speech was not unexpected, given the well documented role of eyebrow raises in both greetings and in conversational speech (Ekman 1979; Grammer et al. 1988).

Asymmetry of Deliberate and Spontaneous of Facial Movement

A main effect of deliberateness on asymmetry of facial movement variables was not observed. It is somewhat difficult to interpret the absence of differences in asymmetry between deliberate and spontaneous movements, particularly at the low levels of asymmetry observed. These results however, suggest that asymmetry of facial movements may play a much smaller role in distinguishing deliberateness of a facial display than previously suggested, whether the display is deliberate or a spontaneous social or emotional signal.

Limitations of the Current Study

The current study is based on a relatively limited sample of spontaneous displays given the requirement that participants show both deliberate and spontaneous movements to be included in the primary within subject analyses. Additionally, only the onset of each display was examined in the current study. A prior study provided the justification for restricting the study to onsets, given the similarity of results for onset and offset movement (Schmidt et al. 2006). An additional consideration was our view that ecologically speaking, the movement of offsets is not independent of onset. However, we acknowledge that the social impact of a facial display, once initiated, can be modified through variation in offset (see for example, Krumhuber and Kappas 2005). Future work should focus not only on movement parameters during offset, but also on the variation in apex and offset timing that characterizes spontaneous as compared to deliberate displays (Valstar et al. 2006).

Research on deliberate and spontaneous facial movements across ages and health status informs our understanding of the patterns of neural control and facial signaling dynamics in general. Although variation inherent in spontaneous movement has hindered the investigation of facial displays in the past, newer results support the premise that spontaneous facial movements can be studied in a systematic fashion.

Figure 2.

Effects of Deliberateness on Amplitude of Facial Movement in Smiles and Eyebrow Raises.