You look stressed: A pilot study on facial action unit activity in the context of psychosocial stress

Abstract

Quality and quantity of the human stress response are highly individual. Not only are there differences in terms of psychological and physiological stress reactivity, but also with regard to facial muscle stress reactivity. In a first correlative pilot study to decipher the signature of stress as it presents in the physiognomy of a stressed individual, we investigated how stress-induced muscle movement activity in the face is associated with stress marker activation during a standardized laboratory stress test. Female and male participants (N = 62) completed the Trier Social Stress Test and provided multiple measurements of salivary cortisol, subjective experience, heart rate, and high-frequency heart rate variability. In addition, participants were filmed during stress induction to derive the activation of 13 individual muscles or muscle groups, also termed action units (AU). Mean AU intensity and occurrence rates were measured using the opensource software OpenDBM. We found that facial AU activity correlated with different aspects of the psychosocial stress response. Higher stress-induced cortisol release was associated with more frequent upper eyelid raiser (AU05) and upper lip raiser (AU10) occurrences, while more lip corner pulling (AU12) went along with lower cortisol reactivity. More frequent eyelid tightener (AU07) occurrences were linked to higher subjective stress reactivity but decreased heart rate and HF-HRV reactivity. Last, women showed greater stress-induced smiling intensity and occurrence rates than men. We conclude that psychosocial stress reactivity is systematically linked to facial muscle activity, with distinct facial stress profiles emerging for different stress markers. From all the AUs studied, eyelid tightening (AU07) seems to provide the strongest potential for future attempts of diagnosing phases of acute stress via facial activity.

Highlights

• •There are differential associations of facial activity with psychological and physiological stress reactivity.
• •Eyelid tightening (AU07) was linked to higher subjective stress reactivity but decreased heart rate and HF-HRV reactivity.
• •During stress, women show significantly more smiling behavior than men.

Author note

This study was preregistered at https://doi.org/10.17605/OSF.IO/Y5SXG.

Data is accessible upon reasonable request.

1. Introduction

Our ability to infer others’ mental and affective states is a prerequisite to navigate the complexities of society. Thus, we are able to deduce the intentions and feelings of others using cues such as gaze direction, body posture, voice features, and facial expressions from a very young age [1]. This is possible, because specific intentions or affective states are characterized by specific behaviors, such as facial expressions [2,3], granting us the possibility to quickly understand our social surroundings and behave accordingly [4]. Psychosocial stress, although not necessarily defined as a discrete emotion, is a physiological and psychological state that, like many emotions, is implicitly understood, and shared when passively observed [5]. It is characterized by physiological changes driven via autonomic nervous system and hypothalamic-pituitary-adrenal axis (HPA-axis) activity [6]. While stress is a healthy and necessary reaction to the challenges of everyday life, extreme or persisting stress exposure can cause or exacerbate numerous psychological and somatic health disorders [6]. Aim of the current pilot study was to explore the associations of facial muscle movement (action unit; AU [3]) activity with subjective-psychological and physiological stress reactivity during a potent psychosocial laboratory stressor, the Trier Social Stress Test (TSST; [7]).

Other than the primary emotions (anger, fear, sadness, disgust, and enjoyment), which were shown to have a universal facial signature, at least in western societies [[8], [9], [10]], both physiological and psychological stress reactions are highly individual, depending on trait characteristics such as age, sex [11,12], and the quality and quantity of the experienced stress load. Likewise, stressed facial expressions appear to differ inter-individually [[13], [14], [15]], possibly because individuals react with a range of physiological, psychological, and cognitive changes to the experience of stress [16,17]. While a growing body of literature is investigating effects of psychosocial stress on facial recognition abilities (e.g. Refs. [18,19]), only few studies have focused on associations of psychosocial stress reactivity with facial muscle movement activity [13,15,[20], [21], [22]]. In other words, the facial signature of stress that is at the basis of our capacity to express stress ourselves and to recognize stress in others, remains unknown.

Responses to psychosocial stress have been theorized to fall into one of two categories: Fight-or-flight [23] or tend-and-befriend responses [24,25]. While the former describes the allocation of resources via increases in heart rate and glucocorticoid release to either engage with (fight) or disengage from (flight) the stressor at hand, energy mobilization in the latter is proposed to trigger positive social behavior [25]. Thus, behavioral responses to psychosocial stressors, such as facial expressions, which are behavioral actions used to convey intentions and information about subjective experience in a real or imagined social context [2], will differ depending on the subjective experience (i.e., valence and arousal) and intentions of the stressed individual (i.e., engage, disengage, befriend).

As a psychosocial stressor consisting of a mock-job talk and an arithmetic task completed in front of an unemphatic evaluation committee, the TSST [7] is well suited to illustrate these different responses. An individual engaged in a tend-and-befriend-like response would likely try to win the committee's sympathy by providing prosocial smiling behavior. In contrast, when showing a confrontational fight response, frowning or squinting behavior would be more dominant. In accordance with Taylor's assumption that men are more likely to show fight-or-flight, while women engage in tend-and-befriend behavior when stressed [24,25], Mayo and colleagues (2019) found gender-specific differences in participants' facial muscle reactivity in the musculus corrugator supercilia when confronted with a cold-pressor task. While men demonstrated stress-induced increases in corrugator activity measured via electromyography (EMG), women showed decreases. These findings corroborated an earlier study on anger and fear expressions and their relationship with cortisol and heart rate reactivity in the TSST [13]. In this study, angry and fearful expressions during the stress test were coded using the Facial Action Coding System (FACS; [3]), which provides a standardized categorization for discrete emotional expressions. The authors found elevated cortisol and heart rate levels given higher occurrences of anger and fear expressions in men, but not in women.

Three machine-learning studies have suggested a wide range of facial AUs to predict stressed vs. non-stressed affective states [[20], [21], [22]]. Importantly, only the study by Gavrilescu and colleagues [20] could actually predict self-reported subjective stress experience via AU activity. However, none of the studies collected physiological stress markers, making inferences on the quantitiy and quality of the underlying physiological stress-induced activation impossible. Given that subjective-psychological and physiological stress responses are frequently unrelated [26] and not all challenge is stress, the machine learning algorithms employed may have categorized rest vs. challenge (or arousal) rather than acute stress.

In the current study, subjects (N = 62) were confronted with the TSST and provided multiple samples of saliva cortisol, subjective stress, heart rate and high-frequency heart rate variability (HF-HRV) data across the 2.5-h testing session. Facial AU activity was derived from 15-min videos of the participants’ faces recorded during the anticipation and stress induction period of the TSST.

Based on previous studies, which investigated either an index of activational strength (or intensity) [15], or the amount of occurrences of a discrete emotion [13], we focused on how both, AU intensity and occurrence, related to stress reactivity across different stress markers. Facial muscle intensity (activational strength) and occurrence (activated vs. not activated) are associated measures of muscle activity, because intensity can only be measured when occurrence is given. They are not interchangeable, however, because, while some participants might provide a high number of low intensities, other participants might show few but very intense activations. Due to a) the exploratory nature of this study, b) the theoretical assumption of highly inter-individual facial expressions when confronted with a psychosocial stressor [15], and c) the complexity of the human stress response [16], we had no a priori hypotheses on the associations of specific AUs with specific stress markers. We did, however, expect certain AUs to have a higher probability to show an association with psychosocial stress. These included AUs relevant for fearful and angry facial expressions and therefore confrontational behavior (AU04: brow lowerer; AU05: upper lid raiser; AU07: lid tightener) [13,15], and AUs relevant for smiling and therefore prosocial behavior (AU06: cheek raiser; AU12: lip corner puller) [25]. In addition, we attempted to replicate findings of sex differences in corrugator activity [14], with men showing greater corrugator reactivity than women. To extend these sex specific effects, we hypothesized that women would show greater cheek raising (AU06) and lip corner pulling (AU12) than men.

2. Methods

2.1. Transparency and openness

The current study was preregistered before data analysis; the preregistration can be accessed at https://doi.org/10.17605/OSF.IO/Y5SXG. Data was collected before preregistration. We report how all data exclusions, manipulations, and study measures were determined. Because this is a secondary study, sample-size was determined by the initial study in the context of which the data was collected. Analyses were performed in R 4.2.1 [27]. Data is accessible upon reasonable request.

2.2. Participants

The primary research question of this dataset was concerned with empathic stress responses within romantic partner dyads, with a total of N = 85 dyads (170 participants) tested. The current secondary study only investigates stress reactivity and facial activity in one of the two dyad members, namely the dyad member confronted with a psychosocial stress task, not the dyad member passively observing the situation. In total, N = 79 adults were confronted with a psychosocial stressor, the TSST. N = 15 were confronted virtually (rather than in real-life) due to Covid-19 restrictions at the time of data collection and N = 2 participants did not provide useable video data. These 17 participants were excluded from the current study.

Our final sample consisted of N = 62 adults (48.39% female, age: M = 26.48, SD = 4.58, age range = 20–38, 86.7% of women in the luteal phase). The study was advertised in and around Leipzig, Germany. Interested heterosexual romantic dyads were screened via telephone interviews. Dyads were excluded if participants reported non-fluency in German, smoking (>5 cigarettes a week), recreational drug abuse, inability to abstain from alcohol intake for at least a week, previous experience with psychosocial stress testing, BMI (<18.5/>30), or dyslexia. Furthermore, dyads were excluded in case of significant health problems, recent stressful life events (e.g., death of a family member), diagnosed mental disorders in the last 2 years, or intake of medication affecting HPA axis activity (e.g., steroids). Regarding female hormonal status, women (and therefore the dyad) were excluded if they reported using hormonal contraceptives, were pregnant or in menopause. If possible, female participants completed the testing session during their luteal cycle phase (N = 26; 86.7%). Due to scheduling difficulties, two participants were tested during their follicular phase and one participant during her period. One participant did not provide hormonal status data at all.

The study was approved by the Ethics Board at the Faculty of Medicine, Leipzig University (ethic number: 285/19-ek). Participants gave written informed consent and were financially compensated upon study completion. Participants were informed that they could withdraw from the study at any point in time.

2.3. Stress task

Participants completed the TSST, a standardized psychosocial laboratory stressor comprised of an anticipation phase (5 min), a speaking task resembling a job interview (5 min), and a difficult mental arithmetic task (5 min) [7], during which they were audio- and video-taped. The task was completed in front of a gender-mixed committee. The committee members were introduced as behavioral analysists and remained unresponsive to the struggles of the participants by showing a neutral demeanor throughout the testing session. Due to Covid-19 restrictions in Germany during data collection, committee members were wearing a face mask in the majority of testing sessions (90%).

2.4. Measures of acute stress reactivity

2.4.1. Cortisol

Adrenocortical activity was measured from salivary cortisol and collected using Salivettes (Sarstedt, Nümbrecht, Germany). For each sample, subjects placed the Salivette collection swab in their mouth for 2 minutes and refrained from chewing. Until analysis, Salivettes were stored at −20 ^◦C. A time-resolved fluorescence immunoassay was used to determine cortisol activity (nmol/l) [28]. Intra- and interassay variabilities were less than 10% and 12%.

2.4.2. Autonomic activity

Participants were equipped with a portable ECG-device, the Zephyr Bioharness 3 chest belt (Zephyr Technology, Annapolis, Maryland, USA). ECG data was recorded at 250 Hz from −65 min to +52 min relative to stressor onset (altogether 117 min). The full recording was split into thirteen timeframes in accordance with the testing timeline and specific phases of the TSST. In detail, the first two timeframes acted as baseline phases before stressor onset (from −62 min to −52 min). During timeframe 3, participants were instructed to read aloud for 5 minutes (from −13 min to −8 min); during timeframe 4, they anticipated the stress test (from −7 min to −2 min). Timeframe 5 reflected the speaking task (from 0 min to 5 min) and timeframe 6 the arithmetic task of the TSST (from +5 min to +10 min). Timeframes 7 to 13 reflected 35 min of stress recovery. For each timeframe in each participant, heart rate, and heart rate variability averages were calculated. While beats per minutes (bpm) were used to calculate heart rate as a proxy of sympathetic nervous system activity, the variance in the length of RR-intervals in the high frequency spectral range of heart rate variability (0.15–0.4 Hz) were used to calculate high frequency heart rate variability as a measure of parasympathetic activity. In a first analysis step, in-house software was used to automatically identify R-peaks for heart rate and HF-HRV calculation. Second, two independent research assistants manually corrected raw ECG recordings and R-peak identification. Artifacts (e.g., movement artifacts, ectopic beats) were cut from raw recordings if R-peak identification proved difficult. If more than 10% of timeframes needed cutting due to excessive artifacts, the respective phase was dropped from analysis. After ECG data preparation, average timeframe heart rate and HF-HRV were calculated using the hrv-analysis python package [29].

2.4.3. Subjective stress

Subjective stress experience was measured with the state version of the State-Trait-Anxiety Inventory [30]. It consists of 20 items concerned with feelings of tension, nervousness and worry measured on a 4-point scale (ranging from not at all to very much) that are condensed to one summary score.

2.4.4. Facial action unit activity

Participants were filmed throughout the 20 min TSST procedure, including listening to the instructions, the anticipation phase, speaking and arithmetic tasks using AIPTEK SeeMe Pocket Camcorder cameras at 30 frames per second (fps) resulting in approximately 27,000 frames per participant (Median = 26,912, range = 26,789–27,138, 1st quartile: 26,893, 3rd quartile: 26,961). AU occurrence (active vs. not active) and intensity (ranging from 0 to 5) were derived from each frame using OpenDBM, an opensource digital biomarker tool [31]. For each frame, a measure of AU detection accuracy was reported (ranging from 0 to 100%). Frames with less than 80% accuracy were excluded from analysis (Median = 0, range = 0–3393, 1st quartile: 0, 3rd quartile: 38.25). OpenDBM provides information for 16 distinct AUs (AU01, AU02, AU04, AU05, AU06, AU07, AU09, AU10, AU12, AU14, AU15, AU17, AU20, AU23, AU26 and AU45; see Fig. 1 for an overview). Unfortunately, because participants were wearing a functional near infrared spectroscopy cap on their heads during the TSST, participants’ foreheads were obstructed. Thus, all AUs indexing activity on the forehead (AU01, AU02 and AU04) were excluded from analysis.

Fig. 1.

Overview of Action Units analyzed via openDBM

Note. Blinking (AU45) is not depicted.

2.5. Experimental design and procedure

Testing was scheduled between 12 p.m. and 5 p.m. to minimize circadian fluctuations in cortisol release rhythm [32]. Due to the dyadic nature of the primary study on empathic stress, participants arrived at the laboratory together with their romantic partner. Upon arrival, dyads were seperated and offered a snack and a glass of juice to normalize blood sugar levels. For the remainder of the testing session, participants were only allowed to drink water. After their snack, participants were instructed about their roles in the upcoming task. Targets were subjected to the TSST while observers took on the role of passive onlookers. Both dyad members were instructed to refrain from interacting both verbally and non-verbally during the actual testing procedure. The current study focuses only on target data, that is, data collected in the actively stressed dyad partners. Targets rested for approximately 30 min before the first sample of saliva and subjective stress (STAI) was collected (at −50 min before stressor onset). They were brought to the stress testing room 40 min before stressor onset, and electrodes for functional near-infrared spectroscopy (fNIRS) assessments were installed (fNIRS data is not subject to the current study). At approximately 8 min before stressor onset, participants received detailed instructions for the upcoming stress test and started the anticipation phase (from −7 to −2 min before stressor onset). Before starting the speaking task (−2 min before stressor onset), participants provided the second cortisol and subjective stress samples. Participants provided further samples of cortisol and subjective stress immediately after TSST (+10 min after stressor onset) and at +20, +25, +30, +40, +50, +60 and + 70 min to capture complete stress recovery. A continuous electrocardiogram (ECG) was recorded from −65 min until +52 min to measure autonomic system activity on both the sympathetic and the parasympathetic level.

2.6. Statistical analysis

2.6.1. Data processing and estimation of stress reactivity

Only data from directly stressed targets was analyzed. Due to their left-skewed distribution, raw cortisol data as well as heart rate and HF-HRV time-phase means were logarithmized using the natural logarithm to approach normal distribution. Subsequently, cortisol, heart rate, and HF-HRV data were winsorized to 3 standard deviations.

For all stress markers, the area under the curve with respect to increase (AUCi) [33] was calculated. The AUCi incorporates both overall stress reactivity and recovery in one measure of individual stress-sensitivity. Please note that HF-HRV decreases during stress, resulting in negative HF-HRV AUCi values. Because AUCi calculation requires full data sets, single heart rate and HF-HRV missing values due to poor data quality were imputed via predictive mean matching. In detail, two participants were missing phase 1, and six participants were missing either phases 3, 4, 5, 7, 8, or 13, respectively. Analyses excluding participants with imputed data yielded the same results as the full, imputed dataset (see Supplementary Results). N = 10 participants were missing more than one heart rate and HF-HRV phase mean and were therefore excluded from further analysis via list-wise deletion.

2.6.2. Action unit intensity and occurrence

For each of the utilized 13 AUs per participant (AU05, AU06, AU07, AU09, AU10, AU12, AU14, AU15, AU17, AU20, AU23, AU26 and AU45), mean AU intensity and occurrence rates were calculated for the anticipation and the 10-min stress phases (combining interview and mental arithmetic) of the TSST. Per participant, AU intensity was averaged across all valid frames (Median = 26,895, range = 23,460–27,066, 1st quartile: 26,853, 3rd quartile: 26,938). AU occurrence was operationalized as the rate of activated to non-activated AUs. Although AU intensity and occurrence during the anticipation phase were not interpreted in the main data analysis, they were calculated to provide a pre-stress AU activity baseline. Because a number of AUs were not engaged across the entire testing session (i.e., revealed a mean and variance close to zero), they were dropped from subsequent linear regression modeling. In terms of intensity, this included AU05, AU09, AU20 and AU23 (see Table 1). In terms of occurrence, this included AU09 (see Table 1).

Table 1.

Descriptive statistics of AUs.

Action Unit (AU)	Mean intensity (SD)	Occurrence rate (SD)
AU05 (Upper lid raiser)	0.07 (0.03)	0.54 (0.33)
AU06 (Cheek raiser)	0.63 (0.41)	0.34 (0.28)
AU07 (Lid tightener)	0.35 (0.38)	0.16 (0.25)
AU09 (Nose wrinkler)	0.06 (0.02)	0.01 (0.02)
AU10 (Upper lip raiser)	0.84 (0.50)	0.54 (0.31)
AU12 (Lip corner puller)	0.81 (0.51)	0.40 (0.29)
AU14 (Dimpler)	0.88 (0.47)	0.62 (0.28)
AU15 (Lip corner depressor)	0.22 (0.14)	0.17 (0.10)
AU17 (Chin raiser)	0.52 (0.12)	0.27 (0.07)
AU20 (Lip stretcher)	0.13 (0.04)	0.14 (0.08)
AU23 (Lip tightener)	0.16 (0.05)	0.18 (0.24)
AU26 (Jaw drop)	0.46 (0.09)	0.15 (0.04)
AU45 (Eye blink)	0.32 (0.10)	0.27 (0.07)
Overall	0.37 (0.39)	0.26 (0.27)

Note. Mean intensities and occurrence rates with standard deviation (SD).

2.6.3. Modeling

Associations of AU occurrence and intensity during the stress phase were investigated with a backward step-wise multiple linear regression approach. With regard to associations of AU intensity and occurrence with the collected stress markers (cortisol, heart rate, HF-HRV, subjective stress), separate models were calculated for occurrence and intensity because including them in the same model revealed high variance inflation factors (VIF >5), suggesting multicollinearity. This resulted in two models for each stress marker.

In our preregistration, we initially planned 16 step-wise regression models, calculating separate models for heart rate and HF-HRV in the anticipation, speaking and arithmetic phases of the TSST, for both AU intensity and occurrence. We chose not to follow this approach due to the excessive number of statistical tests. With two models for each stress marker, we achieved a much clearer presentation of our results while limiting the probability of alpha-error cumulation due to multiple testing.

Regarding control variables, sex, age, and time of testing were added to the cortisol models due to their known influence [34,35]. For heart rate and HF-HRV models, participant age, sex, and BMI were added as controlling variables, again due to their know and reliable influence [36,37]. In the STAI models, only age and sex were added as controlling variables.

Using a backward stepwise regression approach, control variables and all AUs were included into the full model. AU model coefficients were then removed iteratively until best model fit was achieved. Standardized beta coefficients (β), 95% confidence intervals (CI), and p-values (p) are reported.

Because models using either AU intensities or occurrences as predictor variables were not nested and could not be compared with ANOVAs, the Akaike Information Criterion (AIC) was used to descriptively compare model fit.

2.6.4. Sex differences

Because participants were wearing an fNIRS cap, AU04 activity could not be reliably derived from the videos. Thus, unlike originally planned, sex differences in frowning (AU04) behavior could not be investigated. Sex differences in smiling (AU06 and AU12) behavior were tested across each phase of the TSST (anticipation, talking task, arithmetic task) for both AU intensity and AU occurrence using a repeated-measures ANCOVA with sex as a covariate. F-values (F), p-values (p) and eta-partial squared (η_p²) are reported.

2.6.5. Power analysis

Since this is a secondary study, we are limited to the sample size determined for the primary research question.

3. Results

3.1. Manipulation check

To verify whether TSST stress induction was successful, the percentage of participants exceeding a stress-induced cortisol increase of >1.5 nmol/l from baseline levels was calculated [38]. 91.93% of participants fulfilled this criterion (see Fig. 2A). For an overview of stress markers trajectories across the testing session, see Fig. 2.

Fig. 2.

Trajectories of stress markers.

Note. Means with standard errors for cortisol (A), STAI (B), heart rate (C) and HF-HRV (D) across the stress testing session.

3.2. Action Unit Intensity and Occurrence

Our data revealed an overall mean AU intensity of M = 0.37 (SD = 0.39), and an overall AU occurrence rate of M = 0.26 (SD = 0.27). Mean intensity and occurrence rates across the 10 min stress phase are presented for all AUs in Table 1. Several AU mean intensities and standard deviations were very small and showed almost no variation (SD < 0.05), providing very little useable information (e.g. AU05, AU09, AU20, AU23). With respect to AU occurrence rates, only AU09 provided little to no information.

AU intensity means and occurrence rates by TSST phase (anticipation, speaking task, arithmetic task) are presented in Fig. 3. To give a short summary, AU14 (M = 0.87, SD = 0.55) showed the highest overall intensity during the anticipation phase, followed by AU10, AU12, AU06, and AU07 (see Fig. 2A). During the speaking task, AU10 (M = 0.87, SD = 0.54), AU12 (M = 0.78, SD = 0.49), and AU06 (M = 0.60, SD = 0.43) increased in mean intensity (see Fig. 3B). During the arithmetic task, AU10, AU12, and AU14 continued to show the highest mean intensities (see Fig. 2C).

Fig. 3.

Mean AU intensity during anticipation, speaking and mental arithmetic tasks of the TSST.

Note. AU mean intensity during the anticipation phase (A), speaking task (B) and arithmetic task (C) of the TSST. AU intensity varies between 0 and 5. AU05: upper lid raiser; AU06: cheek raiser; AU07: lid tightener; AU09: nose wrinkler; AU10: upper lip raiser; AU12: lip corner puller; AU14: dimpler; AU15: lip corner depressor; AU17: chin raiser; AU20: lip stretcher; AU26: jaw drop; AU23 lip tigthener.

Regarding AU occurrence, a similar pattern was observed, with AU14 (M = 0.69, SD = 0.34) showing the highest occurrence rate during anticipation and AU06, AU10, and AU12 showing the highest occurrence rates during the speaking and arithmetic tasks in comparison to anticipation (see Fig. 4). Please see Table S3 in the Supplementary Results for a full correlation matrix of all AUs.

Fig. 4.

AU occurrence rate (%) during anticipation, speaking and mental arithmetic tasks of the TSST.

Note. AU occurrence rate (%) ranging from 0 to 1 during the anticipation phase (A), speaking task (B) and arithmetic task (C) of the TSST. AU05: upper lid raiser; AU06: cheek raiser; AU07: lid tightener; AU09: nose wrinkler; AU10: upper lip raiser; AU12: lip corner puller; AU14: dimpler; AU15: lip corner depressor; AU17: chin raiser; AU20: lip stretcher; AU23 lip tigthener; AU26: jaw drop.

3.3. Associations between stress reactivity and facial activity

To assess associations between stress markers and facial AU activity we calculated backward stepwise linear regression models using AUs as our independent variables. Cortisol, heart rate, HF-HRV, and STAI stress-sensitivity indexed by AUCis acted as dependent variables. Models were calculated with AU intensity means and AU occurrence rates separately, resulting in two models per stress-marker. For bi-variate scatter plots of all primary results please see Fig. S1 in the Supplementary Results.

3.3.1. Cortisol

There was a significant association of cortisol reactivity and both cheek raising (AU06) (β = 0.33, CI [0.00–0.66], p = 0.048) as well as lip corner pulling intensity (AU12) (β = −0.43, CI [−0.82 to −0.05], p = 0.028), showing contrasting effects. In detail, while higher AU06 intensity predicted greater cortisol reactivity, AU12 was associated with lower cortisol levels.

Regarding cortisol reactivity associations with AU occurrence rates, we found significant effects of the upper lid raiser (AU05) (β = 0.31, CI [0.05–0.58], p = 0.023), the upper lip raiser (AU10) (β = 0.38, CI [0.07, 0.68], p = 0.016) and the lip corner puller (AU12) (β = −0.39, CI [−0.74, −0.04], p = 0.031). Both more frequent upper lid raising (AU05) and upper lip raising (AU10) predicted higher cortisol reactivity. A higher rate of lip corner pulling (AU12) occurrences contrarily predicted lower cortisol reactivity (see Table 2). The intensity model provided comparable fit (AIC = 689) as the occurrence model (AIC = 690).

Table 2.

Regressions coefficients predicting cortisol, STAI, heartrate and HF-HRV reactivity.

Cortisol	Intensity Model			Occurrence Model
Coefficient	B	CI	p	B	CI	p
Sex [Female]	−0.09	−0.69–0.50	0.752	−0.06	−0.64–0.52	0.836
Time of day	−0.00	−0.25–0.24	0.980	−0.10	−0.35–0.15	0.418
Age	−0.12	−0.41–0.18	0.437	−0.14	−0.40–0.13	0.305
AU06	0.33	0.00–0.66	0.048
AU12	−0.43	−0.82–−0.05	0.028
AU15	−0.33	−0.68–0.03	0.070
AU17	0.23	−0.05–0.52	0.110
AU26	−0.21	−0.48–0.06	0.125
AU05				0.31	0.05–0.58	0.023
AU07				0.22	−0.04–0.48	0.093
AU10				0.38	0.07–0.68	0.016
AU12				−0.39	−0.74–−0.04	0.031
AU23				−0.18	−0.45–0.08	0.177
Observations	62			62
R2/R2 adjusted	0.246/0.132			0.233/0.117
AIC	689.200			690.243
STAI	Intensity Model			Occurrence Model
Coefficient	B	CI	p	B	CI	p
Sex [Female]	0.47	−0.04–0.98	0.072	0.45	−0.02–0.91	0.058
Age	−0.03	−0.29–0.22	0.802	−0.04	−0.28–0.20	0.745
AU07				0.36	0.12–0.60	0.004
AU23				0.20	−0.04–0.44	0.102
Observations	61			61
R2/R2 adjusted	0.058/0.026			0.252/0.198
AIC	975.732			965.698
Heart rate	Intensity Model			Occurrence Model
Coefficient	B	CI	p	B	CI	p
Sex [Female]]	−0.41	−0.97–0.16	0.153	−0.52	−1.06–0.01	0.056
Age	−0.02	−0.30–0.26	0.891	−0.04	−0.30–0.22	0.756
BMI	0.07	−0.22–0.37	0.616	0.10	−0.17–0.36	0.464
AU07	−0.30	−0.59–−0.01	0.041
AU07				−0.39	−0.64–−0.13	0.004
AU23				0.27	0.00–0.53	0.050
AU26				0.29	0.02–0.57	0.036
Observations	52			52
R2/R2 adjusted	0.124/0.050			0.312/0.220
AIC	330.911			322.375
HF-HRV	Intensity Model			Occurrence Model
Coefficient	B	CI	p	B	CI	p
Sex [Female]	0.29	−0.35–0.94	0.364	0.59	0.05–1.12	0.033
Age	0.20	−0.09–0.49	0.170	0.15	−0.12–0.42	0.265
BMI	0.04	−0.24–0.32	0.784	−0.10	−0.38–0.18	0.470
AU12	0.34	−0.01–0.68	0.057
AU17	−0.19	−0.49–0.10	0.185
AU26	0.21	−0.08–0.49	0.152
AU07				0.29	0.02–0.56	0.037
AU23				−0.23	−0.51–0.04	0.094
Observations	52			52
R2/R2 adjusted	0.215/0.111			0.218/0.133
AIC	489.943			487.747

Note. Coefficient estimates (Β) with 95% confidence intervals (CI) and p-values (p).

3.3.2. Heart rate

There was a significant association of heart rate reactivity and lid tightening (AU07) intensity (β = −0.30, CI [−0.59 to −0.01], p = 0.041), showing that higher lid tightening (AU07) intensity predicted decreased heart rate reactivity.

With regard to occurrence rates, we found significant effects of lid tightening (AU07) (β = −0.39, CI [−0.64 to −0.13], p = 0.004), lip tightening (AU23) (β = 0.27, CI [0.00–0.53], p = 0.050) and jaw drop (AU26) (β = 0.29, CI [0.02–0.57], p = 0.036) with more frequent lid tightening (AU07) predicting reduced heart rate reactivity and more frequent lip tightening (AU23) as well as jaw dropping (AU26) predicting increased heart rate reactivity (see Table 2). The occurrence model provided better fit (AIC = 322) than the intensity model (AIC = 330).

3.3.3. HF-HRV

Because HF-HRV decreases under stress, positive effects of covariates have to be interpreted as stress-buffering. There were no significant associations of AU intensity with HF-HRV reactivity. Regarding occurrence rates, there was a significant interaction with lid tightening (AU07) (β = 0.29, CI [0.02–0.56], p = 0.037), showing that, parallel to the heart rate findings, higher AU07 activation predicted decreased HF-HRV reactivity (see Table 2). Again, the occurrence model provided better fit (AIC = 487) than the intensity model (AIC = 489).

3.3.4. STAI

There were no significant associations of STAI reactivity with AU intensity for any AU. Regarding AU occurrence rates, we found a significant effect of lid tightening (AU07) (β = 0.48, CI [0.19, 0.76], p = 0.001), with more frequent lid tightening (AU07) occurrences predicting greater subjective stress reactivity (see Table 2). Again, the occurrence model provided better fit (AIC = 964) than the intensity model (AIC = 974).

Importantly, bivariate scatter plots revealed heavily skewed associations of AU07 and AU23 with the respective stress markers. Therefore, heart rate, HF-HRV and STAI effects on these two AUs are clearly driven by the minority of participants showing any variation, and have to be taken with caution (see Fig. S1).

3.4. Sex differences

For cheek raiser (AU06) and lip corner puller (AU12) we conducted repeated measures ANCOVAs to compare mean AU intensity and mean AU occurrence rates between men and women across the three TSST phases (anticipation, speaking task, arithmetic task).

Regarding cheek raiser (AU06) intensity, we found a significant main effect of phase (F (2, 120) = 28.27, p < 0.001, η_p² = 0.32) and a significant interaction of phase and sex (F (2, 120) = 7.70, p = 0.001, η_p² = 0.11), suggesting an overall increase in cheek raiser activity from the anticipation phase to the speaking and arithmetic tasks that was driven by women. In detail, while cheek raiser intensity in men did not change, women showed elevated cheek raiser intensity in both the speaking and the arithmetic tasks compared to the anticipation phase (see Fig. 5A). Regarding cheek raiser occurrence, we found a significant main effect of phase (F (2, 120) = 31.13, p < 0.001, η_p² = 0.34) but only a marginally significant interaction of phase and sex (F (2, 120) = 2.85, p = 0.06, η_p² = 0.05). In detail, cheek raiser occurrences increased from the anticipation to the stress phase (particularly the speaking task) overall, but the increase was marginally stronger in women than in men (see Fig. 5B).

Fig. 5.

Sex differences in cheek raiser (AU06) and lip corner puller (AU12) intensity as well as occurrence across the testing session.

Note. Mean cheek raiser (AU06) and lip corner puller (AU12) intensities and occurrences for male (green) and female (blue) participants across the three phases of the TSST (anticipation, speaking, arithmetic). AU06 (A) and AU12 intensities (C) as well as AU12 occurrence rates (D) were significantly higher in female subjects during the speaking task and the arithmetic task in comparison to the anticipation phase. AU intensity varies between 0 and 5. (For interpretation of the references to colour in this figure legend, the reader is referred to the Web version of this article.)

Regarding lip corner puller (AU12) intensity, there was a significant main effect of phase (F (2, 120) = 28.30, p < 0.001, η_p² = 0.32) and a significant interaction of phase and sex (F (2, 120) = 12.96, p < 0.001, η_p² = 0.18), suggesting an overall increase in lip corner puller activity from the anticipation phase to the speaking and arithmetic tasks that was driven by women. In detail, while lip corner puller intensity did not change in men, women showed elevated lip corner puller intensity in both the speaking and the arithmetic tasks (see Fig. 5C). For lip corner puller (AU12) occurrences, there was a significant main effect of phase (F (2, 120) = 28.01, p < 0.001, η_p² = 0.32) and a significant interaction of phase and sex (F (2, 120) = 5.18, p < 0.001, η_p² = 0.08), suggesting an overall increase in lip corner puller occurrences from the anticipation phase to the speaking and arithmetic tasks, which was again driven by women. In detail, while lip corner puller occurrence rates did not change in men, women showed elevated lip corner puller occurrence rates during both speaking and arithmetic task components (see Fig. 5D).

4. Discussion

In the current pilot study, we explored the facial expressions of psychosocial stress experience. Facial expressions were operationalized through muscle activity, coded as AU intensity and occurrence. Psychosocial stress was induced with a standardized laboratory paradigm, the Trier Social Stress Test [7], and multimodally quantified in terms of cortisol, sympathetic, parasympathetic and subjective-psychological activation. Based on Taylor's tend-and-befried theory [[24], [25]], we expected greater stress-induced frowning reactivity in men [15], and greater smiling reactivity in women. Muscle activity was coded from videos over a 15-min timeframe, spanning a stress anticipation phase and two different stress-inducing tasks (public speaking and difficult metal arithmetic).

Depending on the underlying stress marker, distinct facial stress profiles emerged. Overall, upper eyelid raiser (AU05), eyelid tightener (AU07), upper lip raiser (AU10), lip corner puller (AU12) and lip tightener (AU23) were significantly involved in the stress response, independent of stress marker. Specifically, more frequent upper eyelid raiser (AU05) and upper lip raiser (AU10) occurrences were associated with higher cortisol reactivity; more frequent lip tightener (AU23) occurrences were associated with greater heart rate reactivity, and more frequent eyelid tightener (AU07) occurrences with greater subjective stress reactivity. Contrarily, more frequent lip corner puller (AU12) occurrences were linked to lower cortisol release, higher eyelid tightener intensity to lower heart rate reactivity, and, lastly, more eyelid tightener occurrences to lower heart rate and HF-HRV reactivity. For all stress markers except cortisol, occurrence models provided better model fit than intensity models, suggesting that it is primarily the amount of facial muscle activation rather than the intensity of activation that is involved in psychosocial stress reactivity.

Regarding sex differences in smiling behavior, women exhibited greater cheek raising (AU06) and lip corner pulling (AU12) intensity as well as lip corner puller (AU12) occurrence rates than men during the TSST stress phase (speaking and arithmetic tasks). There was a marginal sex difference for changes in cheek raiser (AU06) occurrence rates, again with women exhibiting higher occurrence rates than men.

4.1. Associations of stress reactivity and facial activity

As expected, no consistent pattern of facial activity was found across all stress markers. This comes as no surprise given that stress markers themselves (except for heart rate and HF-HRV) tend to associate poorly [26], emphasising the complexity of the human stress response [16]. Zooming in on the specific stress markers, however, distinct facial stress patterns emerged.

For cortisol release, more frequent and intense zygomaticus major activity (AU12), which is usually shown when smiling, showed stress-buffering associations. Indexing positive affect, smiling during stress may be attributed to intentions of pro-sociality and a tend-and-befriend reaction to the TSST [25]. Both may be coupled with increased oxytocin release, which has prosocial and stress buffering effects [39,40]. An alternative explanation for the stress-buffering effects of smiling behavior could be its association with, and induction of, positive affect in the context of the facial feedback hypothesis [41]. The theory proposes a reciprocal relationship between positive valence and smiling, such that positive affect would lead to smiling behavior and, vice versa, smiling behavior to positive affect. Toward that end, a previous study by Kraft et al. [42] found reduced heart rate reactivity during stress if participants were either smiling normally or with a forced Duchenne smile in comparison to a neutral expression control group. More frequent upper lid raiser (AU05) and upper lip raiser (AU10) occurrences were associated with higher stress-induced cortisol secretion. While the former muscle is involved in the discrete emotions of anger and fear [3], which have been shown to increase cortisol reactivity in men [13], the latter indicates feelings of physiological [43] and moral disgust [44]. Interestingly, cheek raising (AU06) intensity showed a contrasting effect with increased intensity suggesting greater cortisol reactivity, although both AU06 and AU12 are necessary prerequisites for a genuine smile. This finding illustrates a very important limitation of our pilot study, that is, the lack of a control group. Because facial expressivity and muscle movement is highly context dependent and cannot be directly interpreted without distinguishing spurious facial activity and actual facial expression, a non-stressed control group would greatly improve the interpretability and replicability of our results. We further discuss this in our limitation section.

More frequent lid-tightening or squinting (AU07) behavior was associated with decreased autonomic and increased subjective stress reactivity. Although these results were likely driven by a minority of participants, we would like to discuss possible reasons for the associations found. AU07 indexes activity of the pars palpebral part of the musculus orbicularis oculi, an annular skeletal muscle surrounding the eye. Lid-tightening can be observed during anger and fear expressions [3], during states of clinical and experimental pain [45] but also during laughter [3]. Thus, increased subjective stress reactivity in association with greater AU07 activity can be explained by a higher tendency to convey acute negative subjective affect. But why would participants exhibiting high lid-tightening activity show parallel decreases in heart rate and HF-HRV reactivity? First, although stress-markers such as cortisol, alpha-amylase, and subjective stress often show only low correlations [46], heart rate and HF-HRV reactivity are highly correlated, because heart rate variability necessarily decreases given a higher amount of heart beats per minute. Thus, finding associations of both heart rate and HF-HRV with AU07 is not surprising. Second, and contrasting the relationship between orbicularis oculi activity and negative affect or pain, a study by Heponiemi and colleagues (2006) reported increases in EMG-derived lid-tightening given high positive affect during challenge. This relationship was also found while perceiving positive visual and auditory stimuli, albeit only in a pilot study including 16 participants [47]. Third, lid-tightening or squinting is also found while experiencing distrust or suspicion [48]. The TSST is a psychosocial stressor created through deception by introducing the committee as behavioral psychologists allegedly judging the participants every move. Highly distrusting subjects might be better able to deduce the goals and intentions of the committee (i.e., purposely stressing the target) through social capacities such as mentalizing. Last, AU07 activity might have partially indexed the amount of smiling during the TSST, which would tie into the previously mentioned associations of AU12 with decreased cortisol reactivity.

Contrarily, increasing lip tightener (AU23) and jaw dropping (AU26) occurrences were linked to higher heart rate reactivity. Like the upper lid raiser (AU05), AU23 is involved in the expression of anger [3], which has shown a stress-inducing effect on cortisol reactivity in men [13]. AU26, on the other hand, indexes both surprise and fear but is more importantly an action unit highly influenced by the amount of speaking. In general, all AU measuring facial muscle activity around the mouth have been shown to be influenced by the amount of speaking [31], which is a confounder impossible to control for in this pilot study. Thus, all results regarding AU10, AU12, AU23 and AU26 should be considered with caution.

In sum, the predictive value of facial activity using distinct AUs to predict psychosocial stress reactivity across several stress markers was expectably low, with only eyelid tightener (AU07) performing well overall. However, distinct activation patterns emerged with specific stress markers, with lid-tightening or squinting behavior (musculus orbicularis oculi; AU07) being associated with decreases in autonomic activity and increases in subjective stress. Overall, it seems to be AU occurrence rather than intensity that is linked to physiological and psychological stress reactivity.

4.2. Sex differences in smiling and frowning behavior

While we could not investigate relatively increased frowning behavior in men compared to women [14,15] due to assessing fNIRS and therefore obstructing participants' foreheads, women did exhibit more smiling behavior during stress, indexed as higher cheek raiser (AU06) and lip corner puller (AU12) activity. This finding is in line with the “tend and befriend” hypothesis [25], which originally stated that, compared to men, women would react with positive affiliative social behavior to situations of psychosocial stress. We suggest that women's greater tendency to smile during stress can be explained by gender-specific societal expectations. The TSST confronts individuals with a threat to their ego or socio-hierarchal standing. While, in most societies, men are stereotypically portrayed as stoic or aggressive in the face of adversity, women are rather expected to show compassion and foster pro-sociality [49]. In other words, although the feeling of stress is negative in valence for both sexes, their facial reactions tell different stories: Men show a more frowning demeanour [15], likely aiming to demonstrate superiority and frighten their opponent, women react with a smile. According to Taylor and colleagues . [24,25], the adaptive evolutionary value of this behavior would be to mobilize social support, especially from other women, in times of challenge and need.

4.3. Limitations

The current pilot study has a number of limitations. First, the nature of the data only provides exploratory and correlational inferences, as it was collected in the context of a different main hypothesis and no experimental manipulation or control group was employed. This lack of control group would have to be addressed to exclude possible confounding effects of speaking and allow direct interpretations of facial expression context dependency. Second, although research assistants are instructed to show a neutral facial expression during a TSST, absolute neutrality can never be achieved. However, the distinction of facial activity induced by the TSST committee via changes in their facial expressions (pulled effects) and activity actually provided by the participants (pushing effects) would be necessary to interpret changes in facial expression during stress due to actual physiological stress reactivity or merely the stressful environment. Again, a stress-free control condition would be helpful. However, due to Covid-19 restrictions in Germany, the majority of TSSTs (90%) were performed while wearing a face mask, standardizing committee facial behaviour to a degree. Third, only aggregated measures of AU intensity or occurrence were used as dependent measures. Human facial activity is a highly complex and quickly fluctuating phenomenon. Time-series analyses with large amounts of samples assessed via EMG recordings could provide a more fine-grained investigation into facial activity in the context of psychosocial stress. Last, the simultaneous acquisition of fNIRS data in our study obstructed participants’ foreheads. Consequently, data on inner brow raiser (AU01), outer brow raiser (AU02) and brow lowerer (AU04) could not be acquired.

5. Conclusion

We conclude that psychosocial stress reactivity is systematically linked to facial muscle activity, with distinct facial stress profiles emerging for different stress markers. Only eye lid-tightening behavior showed associations across different stress markers, providing an indicator for multi-modal stress detection in situations of psychosocial threat. In line with the classical interpretation of the tend-and-befriend hypothesis [24,25], women showed stronger smiling behavior than men. The overall pattern of results indicates that, although acute psychosocial stress is a highly negative affective and physiological state, facial expressions do not simply reflect this adversity. Rather, they seem to encompass immediate coping strategies aiming to reduce the impact of the stressor. In the current stress-ridden times, diagnosing states of acute stress via facial expressions might develop into a useful therapeutic tool, helping individuals identify states of stress early on, and automatically prompting stress-reducing strategies to prevent the accumulation of allostatic load.

Funding

Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany.

Author contributions

V.E. and M.G. designed the experiment. M.G. planned the study and was responsible for study execution. J.U.B. analyzed the data and drafted the manuscript. M.D. and F.B. supported data preparation and data analysis. V.E. supported data analysis and drafting of the manuscript. All authors critically revised the manuscript.

Declaration of competing interest

None.

Appendix A. Supplementary data

The following is the Supplementary data to this article.