Automated measurement of infant and mother Duchenne facial expressions in the Face-to-Face/Still-Face

Abstract

Although still-face effects are well-studied, little is known about the degree to which the Face-to-Face/Still-Face (FFSF) is associated with the production of intense affective displays. Duchenne smiling expresses more intense positive affect than non-Duchenne smiling, while Duchenne cry-faces express more intense negative affect than non-Duchenne cry-faces. Forty 4-month-old infants and their mothers completed the FFSF, and key affect-indexing facial Action Units (AUs) were coded by expert Facial Action Coding System (FACS) coders for the first 30 seconds of each FFSF episode. Computer vision software, automated facial affect recognition (AFAR), identified AUs for the entire 2-minute episodes. Expert coding and AFAR produced similar infant and mother Duchenne and non-Duchenne FFSF effects, highlighting the convergent validity of automated measurement. Substantive AFAR analyses indicated that both infant Duchenne and non-Duchenne smiling declined from the FF to the SF, but only Duchenne smiling increased from the SF to the RE. In similar fashion, the magnitude of mother Duchenne smiling changes over the FFSF were 2–4 times greater than non-Duchenne smiling changes. Duchenne expressions appear to be a sensitive index of intense infant and mother affective valence that are accessible to automated measurement and may be a target for future FFSF research.

The Face-to-Face/Still-Face (FFSF) assesses infant responses to parent unresponsiveness, an age-appropriate stressor. Although still-face effects are well-studied, little is known about the degree to which the FFSF is associated with the production of intense affective displays, such as Duchenne smiles and Duchenne cry-faces that involve pronounced eye constriction. Duchenne smiles appear to express more intense positive affect than non-Duchenne smiles, while Duchenne cry-faces appear to express more intense negative affect than non-Duchenne cry-faces. The current study investigated whether levels of Duchenne and non-Duchenne smiling and cry-faces differ across FF, SF, and RE episodes, using computer vision-based automated measurement.

The Face-to-Face/Still-Face (FFSF; Tronick et al., 1978) has been used to examine how infants and parents communicate during naturalistic face-to-face interaction (Cohn & Tronick, 1987), how infants respond to parent emotional unavailability (Ekas et al., 2013), and how infant-parent dyads repair their interaction after the still-face (Feldman et al., 2010). In the FFSF, the parent is instructed to play with the infant normally (face-to-face) and to then stop responding to the infant and hold a neutral expression (i.e., “poker face”) while maintaining eye contact (still-face), and to resume interaction with the infant (reunion). The FFSF is typically conducted in a laboratory context with the infant seated in an infant seat and the parent seated facing the infant. The prototypical still-face effect is characterized by reduced infant positive affect and gaze at the parent and increased negative affect during the still-face episode from the previous face-to-face episode (Mesman et al., 2009). The still-face effect tends to be followed by a partial carry-over effect into the reunion episode consisting of decreased levels of positive and increased levels of negative affect compared to the face-to-face.

The FFSF assesses infant response to the unresponsiveness, cessation of facial cues, and absence of contingent interaction in the still-face, and addresses whether infants return to positive interaction after the still-face, an index of self-regulation (Mesman et al., 2009; Tronick et al., 1978). The FFSF has been used to investigate developmental trajectories of infants’ emotional regulation (Yato et al., 2008), episodic memory (Montirosso et al., 2013, 2014), physiological reactivity (Provenzi et al., 2017), genetic differences in infant self-regulation (Montirosso et al., 2016), self-regulation in infants of depressed and anxious mothers (Weinberg et al., 2006; Reck et al., 2013), and associations of early self-regulation with later externalizing and internalizing behaviors (Ekas et al., 2013; Moore et al., 2001). The still-face effect is robust to procedural variations (Mesman et al., 2009) and is associated with later attachment security (Ekas, et al., 2013).

FFSF research typically relies on manual coding of infants’ positive (e.g., smiling, joy, positive vocalizations) and negative affect (e.g., frowning, crying, anger, sadness, fussing, protesting, as well as mothers’ positive affect (e.g., smiling). The laborious quality of manual coding represents a limitation to sample sizes and the detail with which facial expressions are characterized (Cohn & Kanade, 2007; Messinger et al., 2012; Mitsven et al., 2020). Automated measurement approaches have the potential to address these limitations. The current study examined patterns of infant and mother Duchenne and non-Duchenne expressions in the FFSF and ascertained the convergent validity of expert coding and automated measurement of those patterns.

Automated Measurement

A growing body of research on emotional facial expressions has employed automated measurement using computer vision approaches, based on the Facial Action Coding System (FACS; Ekman et al. 2002) and its application to infants (BabyFACS; Oster, 2006). FACS is a comprehensive manual system for recording anatomically based appearance changes in the form of facial action units (AUs; Lucey et al., 2007), which has been used to intense positive affect and negative affect (Bolzani-Dinehart et al., 2005; Messinger et al., 2009; Messinger et al., 2012; Mattson et al., 2013a). Smiling (indexed by AU12) is the prototypical expression of positive emotion. Smiling combined with eye constriction (AU6; with associated raising of the cheeks), which is known as Duchenne smiling, indexes intense joy in adults (Ekman et al., 1990) and infants (Fox & Davidson, 1988). The cry-face (indexed by AU20), on the other hand, is the prototypical expression of negative emotion in infants (but not adults). Infant cry-faces involving stronger eye constriction, referred to as Duchenne cry-faces, are perceived to be more affectively negative than cry-faces with lower levels of eye constriction (Bolzani-Dinehart et al., 2005; Messinger, 2002; Oster, 2003).

A small body of research has examined infant Duchenne expression in the FFSF. Messinger et al. (2012) found that automated measurement of eye constriction (AU6; the Duchenne marker) indexed both the positive emotional intensity of smiling (AU12) and the negative emotional intensity of cry-faces (AU20) in twelve 6-month-olds. In the same sample, Mattson et al. (2013a) showed that the proportion of smiling accompanied by eye constriction was higher during the positive-emotion eliciting face-to-face episode of the FFSF than during the still-face, and the proportion of cry-faces with eye constriction (stronger cry-faces) was higher during the negative-emotion eliciting still-face episode than during the face-to-face. These findings indicate that smiling with eye constriction (Duchenne smiling) indexes strong positive emotion in 6-month-old infants while cry-faces with eye constriction (Duchenne cry-faces) index strong negative emotion.

It remains unclear, however, whether infant Duchenne expressions are especially impacted by the FFSF procedure. One possibility, for example, is that the FFSF is associated with changes in levels of infant Duchenne expressions but has little or no bearing on non-Duchenne expressions. We are not aware of investigations of studies of mothers’ Duchenne smiles in the FFSF. In adults, Duchenne smiles are perceived as intensified positive affective state and more genuine than non-Duchenne smiles, especially if they are naturally elicited, rather than posed (Gunnery & Ruben, 2016). It is unknown, however, whether mother Duchenne smiling is especially impacted by the FFSF in comparison with non-Duchenne smiling. To our knowledge, the current study is the first to document mother Duchenne smiling in the FFSF.

Automated detection of early facial expressions has yielded insights into the objective measurement of emotion expression during naturalistic interactions (Messinger et al., 2009; Messinger et al., 2012). To date, however, the few applications to automated measurement to infant face-to-face interaction have used relatively small sample sizes. Moreover, no studies have reported convergent validity in which two measurements show the same changes in facial expressions in a specific standardized protocol. Assessing convergent validity of automated measurement of Duchenne and non-Duchenne expressions during the standardized FFSF protocol is a step toward reducing the human resource burden involved in making precise measurements of facial expressions in a large number of infants and mothers.

Current Study

The current study examined whether Duchenne and non-Duchenne expressions in 4-month-old infants are differentially distributed in the face-to-face, reunion, and still-face episodes of the FFSF. We investigated whether expert manual coding and automated measurement (using computer vision approach) showed concordant changes in infant Duchenne and non-Duchenne smiling (AU12 with and without AU6, respectively) and in infant Duchenne and non-Duchenne cry-faces (AU4+AU20 with and without AU6, respectively) during the FFSF. Likewise, we examined whether expert coding and automated measurement of mother Duchenne and non-Duchenne smiling produce identical patterns of changes between the face-to-face and the reunion episodes. Finally, we assessed whether automated measurement of infant and mother Duchenne and non-Duchenne smiling–and infant Duchenne and non-Duchenne cry-faces–differed across the face-to-face, still-face, and reunion episodes. A pattern of results of particular interest would suggest that Duchenne smiling and Duchenne cry-faces are more sensitive indices of affective reactions to the FFSF than are non-Duchenne expressions.

Methods

Participants

Forty 4-month-old (M_age = 4.07 months, SD = 0.31) full-term infants and their mothers (M_age = 29.38 years, SD = 5.15) participated in the study. Twenty-two infants (55%) were male, and 27 infants (68%) were Hispanic/Latinx. Infants’ racial distribution composed of 28 White (70%), 7 Black/African American (17.5%), 4 multiracial (10%), and 1 unknown (2.5%). Twenty-three mothers (57.5%) were Hispanic/Latinx. Mothers’ racial distribution consisted of 27 White (67.5%), 7 Black/African American (17.5%), 1 multiracial (2.5%), and 5 unknown (12.5%). Sample demographics are reported in Table 1. Data were collected between 2011 and 2014.

Table 1.

Demographics of Infants and Mothers

	N (%)
Infant age (in months)	M=4.07, SD=0.31, Range=3.4–4.7
Infant sex
Male	22 (55%)
Female	18 (45%)
Infant ethnicity
Hispanic/Latinx	27 (68%)
Other	13 (32%)
Infant race
White	28 (70%)
Black/African-American	7 (17.5%)
Multiracial	4 (10%)
Unknown	1 (2.5%)
Mother age (in years)	M=29.38, SD=5.15, Range=18–41
Mother ethnicity
Hispanic/Latinx	23 (57.5%)
Other	17 (42.5%)
Mother race
White	27 (67.5%)
Black/African-American	7 (17.5%)
Multiracial	1 (2.5%)
Unknown	5 (12.5%)
Mother education
High school	7 (17.5%)
Some college	5 (12.5%)
2-year college	11 (27.5%)
4-year college	10 (25%)
Advanced/professional degree	6 (15%)
Unknown	1 (2.5%)

The present study was conducted according to guidelines laid down in the Declaration of Helsinki, with written informed consent obtained from a parent or guardian for each child before any assessment or data collection. All procedures involving human subjects in this study were approved by the Institutional Review Board at the University of Miami and conducted in accordance with the ethical standards of the American Psychological Association. Families were compensated for their participation.

Procedure

Infants and their mothers completed the Face-to-Face/Still-Face (FFSF), which consisted of a 2-minute face-to-face (FF), a 2-minute still-face (SF), and a 2-minute reunion (RE). Infants were placed in an elevated infant seat designed in-house, and mothers sat facing the infants. Mothers received instructions for each episode before the procedure (“This will be a 6 minute activity, where you play with your baby for 2 minutes, put on a still-face and not interact with your baby for 2 minutes, then play with your baby again for 2 minutes.”). They were instructed not to move out of the seat, to touch the infant’s face, to lift infant up from the chair, or to shake infant from side to side or back and forth. They were informed that there will be audio prompting each episode.

The FF episode began with an audio recording, Please play with your baby now, followed by a 2-second tone. After 2 minutes, an audio recording, Please stop interacting with your baby now, and a 2-second tone prompted the SF episode. After 2 minutes, mothers heard another audio recording, Please resume playing with your baby now, followed by a 2-second tone, which indicated the beginning of RE episode. The interaction was recorded with one video camera directed at the infant’s face and another at the mother’s face and upper torso. Infant and mother videos were synchronized with a common time code.

Measures

Manual Coding

The Facial Action Coding System (FACS; Ekman et al., 2002) is an anatomically-based gold standard for measuring facial action units that has been adapted for use with infants (BabyFACS; Oster, 2006). Frame-by-frame anatomically based coding of mother and infant facial movement was conducted independently by one of three FACS-certified, Baby-FACS-trained coders. Smiling was indexed by the action of zygomaticus major (AU12), which pulls the lip corners laterally and upward; cry-faces were indexed by the combination of the action of risorius (AU20), which pulls the lip corners laterally, and the actions of depressor glabellae, depressor supercilia, and corrugator supercilli (AU4), which lowers the eyebrows. Eye constriction was indexed by the action of orbicularis oculi, pars orbitalis (AU6), which draws the cheeks and skin around the temples toward the eyes. Duchenne smiling involved the presence of both AU6 and AU12. Duchenne cry-faces involved the presence of AU4, AU6, and AU20. The individual AUs were coded as 0 (absent) or 1 (present). Duchenne smiling (AU6+AU12), non-Duchenne smiling (AU12 without AU6), Duchenne cry-faces (AU4+AU6+AU20), non-Duchenne cry-faces (AU4+AU20 without AU6) were coded as 1 (present) when all of their respective AUs were coded as 1.

Frame-by-frame expert coding of infant Duchenne and non-Duchenne smiling, and Duchenne and non-Duchenne cry-faces was obtained for the first 900 frames (30 seconds) of each episode (FF, SF, and RE). Expert coding of mother Duchenne and non-Duchenne smiling was obtained for the first 900 frames (30 seconds) of the FF and RE episodes. Expert coding of mother smiling was not obtained for the SF episode, based on the parental instructions to maintain a neutral facial expression during the SF. Sixteen infant videos (16 episodes) and 7 mother videos (7 episodes) were randomly selected to be coded by the other two FACS-certified, BabyFACS-trained coders. Intercoder reliability was examined using free-margin kappa (Brennan & Prediger, 1981) for each AU (AU4, AU6, AU12, and AU20). The mean free-margin kappa for infant AUs (κ_free = .85) were reported in Onal Ertuğrul et al. (2022). All free-margin kappas were above .80 (see first two rows of Table 2).

Table 2.

Free-margin kappas (S-scores)

	AU4	AU6	AU12	AU20
Expert-Expert Infant	.94	.80	.85	.88
Expert-Expert Mother		.82	.92
Expert-Automated Infant	.88	.75	.77	.81
Expert-Automated Mother		.63	.84

Note. Overall free-margin kappas (S-scores) assessing reliability between expert FACS coders’ manual coding (expert-expert) and between manual coding and automated measurement (expert-automated). Statistics were calculated over the first 30 seconds of FFSF episodes. N=40.

Automated Measurements

Infants’ and mothers’ synchronized video-recordings were submitted to the Automated Facial Affect Recognition (AFAR; Onal Ertuğrul et al., 2019a; 2019b; 2022) computer vision software, an automated tool for detecting FACS AUs in the infant and mother faces. First, the Zface module (Jeni et al., 2017) of AFAR toolbox was used for automatic face tracking and registration. Zface used a fast-cascade regression framework to accomplish dense 3D registration of the face from the 2D video without person-specific training. The tracked facial images were normalized and centered in regard to rotation and scale to the inter-ocular distance (IOD) of 80 pixels, yielding 224 × 224 pixel images of faces with 80 pixels IOD. Mothers’ facial images were then input to the Adult AFAR network, which is a convolutional neural network that was trained on adult faces (Onal Ertuğrul et al., 2019b). Infants’ facial images were input to the Infant AFAR network that was trained on infant faces (Onal Ertuğrul et al., 2022). The AFAR network output the probability of the facial AUs (i.e., AU4, AU6, AU12, AU20) for each video frame. A probability score of 0.5 or higher indicated the presence of an AU (recoded as 1). Using 25% of the frame-by-frame data (the first 30 seconds of each episode for each dyad), we obtained free-margin kappas above .63 for all reported infant and mother AUs between objective AFAR measurement and expert coding (see the last two rows of Table 2). These represent a moderate level of agreement corrected for chance. The kappa values of infant AUs were reported in Onal Ertuğrul et al. (2022).

In analyses, the proportions of expert-coded Duchenne and non-Duchenne smiling and Duchenne and non-Duchenne cry-faces in the first 30 seconds of each episode were compared to AFAR (automated) measurement. The proportions were computed as the number of frames of facial expression in the first 30 seconds of an episode divided by 900 frames, where facial expression was either Duchenne smiling (AU6+AU12), non-Duchenne smiling (AU12 without AU6), Duchenne cry-face (AU4+AU6+AU20), or non-Duchenne cry-face (AU4+AU20 without AU6).

For analyses of automated measurement data from the full 2-minute episodes, the proportions were computed as the number of frames of facial expression in each episode divided by the total number of frames in each episode, where facial expression was either Duchenne smiling (AU6+AU12), non-Duchenne smiling (AU12 without AU6), Duchenne cry-face (AU4+AU6+AU20), or non-Duchenne cry-face (AU4+AU20 without AU6).

Analytic Approach

Analyses were conducted in IBM SPSS Statistics version 26.0.0 on a Windows computer. Using data from the first 30-seconds of each episode, we conducted a series of repeated measures analysis of variances (ANOVAs) to assess the validity of automated measurement. The patterns of changes in expert coding and automated measurement of infant Duchenne and non-Duchenne smiling, Duchenne and non-Duchenne cry-faces, and mother Duchenne and non-Duchenne smiling were compared. Following comparisons of expert and automated coding, we used repeated measures ANOVAs to assess the changes in the proportions of Duchenne and non-Duchenne smiling using the automated measurement in the full FFSF (i.e., 2-minute FF, 2-minute SF, and 2-minute RE) and to compare those changes in Duchenne and non-Duchenne smiling across episodes. Identical ANOVA models were used for comparing infant Duchenne and non-Duchenne cry-faces and comparing mother Duchenne and non-Duchenne smiling.

Results

Convergent Validity of Automated Measurement

Missing data were comparable for expert coding and automated measurement. Data for infants included the first 30 seconds of the FF, SF, and RE episodes while data for mothers included only the FF and RE. Experts used an un-codable category for 30,523 frames (28.3%) of infant videos and 2,699 frames (3.7%) of mother videos, due to occlusion (e.g., the infant or mother’s hands blocked a view of their faces). In the same reliability sample, the automated Zface approach did not identify the face (and hence could not identify facial actions) in a total of 18,283 frames (16.9%) of infant videos and 1,099 frames (1.5%) of mother videos. See Table S1 in Supplementary Materials for missing data information for the full FFSF.

Table 3 includes descriptive statistics of the proportions of infant Duchenne and non-Duchenne smiling and cry-faces and mother Duchenne and non-Duchenne smiling, obtained by expert coding and automated measurement during the first 30 seconds of each episode of the FFSF. To assess the convergent validity of automated measurement, we conducted a series of 3 (Episode: FF, SF, RE) x 2 (Coding: Expert, Automated) mixed-model ANOVAs. Expert coding and automated measurement were compared in the changes in mean proportions of infant Duchenne and non-Duchenne smiling and Duchenne and non-Duchenne cry-faces across episodes (the first 30 seconds). These were followed by pairwise comparisons between episodes. The Least Significant Difference comparisons are reported in Supplementary Materials (Tables S2 and S3). Results were identical when Bonferroni corrections were applied.

Table 3.

Proportions of infant and mother Duchenne and non-Duchenne expressions

	FF		SF		RE
Infant; Mean (SD)	Expert	Automated	Expert	Automated	Expert	Automated
Duchenne smiling	.18 (.18)	.22 (.20)	.07 (.08)	.09 (.09)	.14 (.15)	.20 (.19)
Non-Duchenne smiling	.06 (.07)	.07 (.08)	.03 (.06)	.03 (.06)	.05 (.09)	.05 (.06)
Duchenne cry-face	.01 (.02)	.01 (.04)	.02 (.08)	.03 (.11)	.08 (.18)	.09 (.20)
Non-Duchenne cry-face	.00 (.00)	.03 (.06)	.01 (.03)	.06 (.13)	.005 (.01)	.13(.21)
Mother; Mean (SD)
Duchenne smiling	.63 (.23)	.67 (.22)	-	.11 (.16)	.47 (.29)	.51 (.26)
Non-Duchenne smiling	.26 (.20)	.22 (.18)	-	.18 (.21)	.27 (.23)	.27 (.19)

Note. Means and (standard deviations) of the proportions of infant Duchenne and non-Duchenne smiling and cry-faces and mother Duchenne and non-Duchenne smiling during the first 30 seconds of each episode of the FFSF. N = 40.

Infant Smiling

There were Episode effects for infant Duchenne, F(1.73, 67.63) = 12.06, p < .001, η_p² = .24, and non-Duchenne smiling, F(2, 78) = 5.94, p = .004, η_p² = .13 (see Table 4). For Duchenne smiling, there was a Coding effect that indicated a greater proportion of automated measure of Duchenne smiling than expert-coded Duchenne smiling, F(1, 39) = 13.22, p = .001, η_p² = .25. There was no Coding effect for non-Duchenne smiling, F(1, 39) = .001, p = .98, η_p² < .001. There were no interactions between Episode and Coding for either Duchenne smiling, F(2, 78) = 1.51, p = .23, η_p² = .04, or non-Duchenne smiling, F(2, 78) = .09, p = .92, η_p² = .002.

Table 4.

ANOVA comparisons of expert coding and automated measurement over the FFSF

Measures	Test of sphericity	Adjustment method and ε level	Effect	F	df1	df2	p	ηp2	Power (1- β error prob.)
Infant measures
Duchenne smiling	χ2(2)=8.42, p=.01	Huynh-Feldt correction (ε=.87)	Episode	12.06	1.73	67.63	<.001	.24	1.00
			Coding	13.22	1	39	.001	.25	1.00
	χ2(2)=1.01, p=.61	Sphericity assumed (ε=.97)	Episode x Coding	1.51	2	78	.23	.04	.79
Non-Duchenne smiling	χ2(2)=3.23, p=.20	Sphericity assumed (ε=.92)	Episode	5.94	2	78	.004	.13	1.00
			Coding	.001	1	39	.98	<.001	.05
	χ2(2)=.94, p=.63	Sphericity assumed (ε=.98)	Episode x Coding	.09	2	78	.92	.002	.09
Duchenne cry-face	χ2(2)=20.65, p<.001	Greenhouse-Geisser correction (ε=.70)	Episode	6.34	1.41	54.96	.01	.14	1.00
			Coding	3.32	1	39	.08	.08	.99
	χ2(2)=9.46, p=.01	Huynh-Feldt correction (ε=.85)	Episode x Coding	.17	1.70	66.34	.81	.004	.12
Non-Duchenne cry-face	χ2(2)=6.22, p=.04	Huynh-Feldt correction (ε=.91)	Episode	5.59	1.81	70.65	.01	.13	1.00
			Coding	19.84	1	39	<.001	.34	1.00
	χ2(2)=8.50, p=.01	Huynh-Feldt correction (ε=.87)	Episode x Coding	6.07	1.73	67.53	.01	.13	1.00
Mother measures
Duchenne smiling			Episode	22.47	1	39	<.001	.37	1.00
			Coding	2.58	1	39	.12	.06	.88
			Episode x Coding	.02	1	39	.88	.001	.07
Non-Duchenne smiling			Episode	1.70	1	39	.20	.04	.71
			Coding	.46	1	39	.50	.01	.24
			Episode x Coding	1.95	1	39	.17	.05	.81

Note. Analysis of Variance (ANOVA) comparisons of the proportions of infant and mother facial expressions during the first 30 seconds of each episode of the FFSF. Episode: main effect of episode (FF, SF, RE for infants; FF, RE for mothers); Coding: main effect of coding (expert & automated); Episode x Coding: interaction effect of Episode by Coding. The assumption of sphericity was violated for the main effect of episode on infant Duchenne smiling, Duchenne and non-Duchenne cry-faces, as well as the interaction effect of episode by coding on infant Duchenne and non-Duchenne cry-faces. The degrees of freedom were corrected using a Huynh-Feldt correction if ε is greater than .75 and a Greenhouse-Geisser correction if ε is less than .75 (Field, 2013). The Episode effect result for infant Duchenne cry-faces is not altered when its degrees of freedom are corrected using a Huynh-Feldt correction. Post-hoc power analysis was performed using G*Power 3.1. Linear mixed-effects analyses yielded conclusions identical to those based on the current results. η_p² = Partial eta squared.

Follow-up Least Significant Difference post-hoc comparisons indicated that expert coding and automated measurement both showed declines in Duchenne and non-Duchenne smiling from the FF to the SF (ps < .05). Both expert coding and automated measurement indicated a rise in Duchenne smiling from the SF to the RE (ps < .01). Neither expert coding nor automated measurement indicated mean difference in the proportions of non-Duchenne smiling between the SF and RE episodes (see Table S2). That is, expert coding and automated measurement yielded identical significant infant still-face effects involving Duchenne and non-Duchenne smiling (see Figure 1).

Figure 1.

Expert coding and automated measurement of infant and mother Duchenne and non-Duchenne expressions in the first 30 seconds of each episode. Line plots contain estimated marginal means of the proportions of infant Duchenne and non-Duchenne smiling (first row), infant Duchenne and non-Duchenne cry-faces (second row), and mother Duchenne and non-Duchenne smiling (third row). The first column shows the Duchenne expressions, and the second column shows the non-Duchenne expressions. The x-axes indicate the FFSF episodes (FF, SF, and RE for infants; FF and RE for mothers). The y-axes indicate mean proportions. The blue line represents AFAR coding, and the red line represents expert FACS coding. Error bars indicate 95% confidence intervals.

Infant Cry-Faces

There was an Episode effect for infant Duchenne cry-faces, F(1.41, 54.96) = 6.34, p = .01, η_p² = .14, and non-Duchenne cry-faces, F(1.81, 70.65) = 5.59, p = .01, η_p² = .13 (Table 4). There was no Coding effect for Duchenne cry-faces, F(1, 39) = 3.32, p = .08, η_p² = .08. However, there was a Coding effect for non-Duchenne cry-faces, F(1, 39) = 19.84, p < .001, η_p² = .34, such that there was a greater proportion of automated measure of non-Duchenne cry-faces than expert-coded non-Duchenne cry-faces. There was no interaction between Episode and Coding for infant Duchenne cry-faces, F(1.70, 66.34) = .17, p = .81, η_p² = .004. However, there was a significant interaction between Episode and Coding for non-Duchenne cry-faces, F(1.73, 67.53) = 6.07, p = .01, η_p² = .13.

For both expert coding and automated measurement, post-hoc pairwise comparisons indicated that the proportion of Duchenne cry-faces did not differ between the FF and the SF (p = .27) but increased from the SF to the RE (p = .02). Non-Duchenne cry-faces did not differ between the FF and the SF for either expert coding or automated measurement (ps > .05). Automated measurement, but not expert coding, yielded a significant rise in non-Duchenne cry-faces from the SF to the RE (p = .04). Post-hoc pairwise comparisons are reported in Table S2.

Mother Smiling

For mothers, 2 (Episode: FF, RE) x 2 (Coding: Expert, Automated) ANOVAs were used to compare the changes from the FF to RE in expert coding and automated measurement of Duchenne and non-Duchenne smiling. There was an Episode effect for mother Duchenne smiling, F(1, 39) = 22.47, p < .001, η_p² = .37, but not for non-Duchenne smiling, F(1, 39) = 1.70, p = .20, η_p² = .04. Specifically, for both expert coding and automated measurement, there was a decline in mother Duchenne smiling, but not in non-Duchenne smiling, from the FF to the RE (Figure 1). There was no Coding effect for Duchenne, F(1, 39) = 2.58, p = .12, η_p² = .06, and non-Duchenne smiling, F(1, 39) = .46, p = .50, η_p² = .01. There were no interactions between Episode and Coding for Duchenne, F(1, 39) = .02, p = .88, η_p² = .001, and non-Duchenne smiling, F(1, 39) = 1.95, p = .17, η_p² = .005.

Associations Between Infant and Mother Smiling

We examined associations between levels of infant and mother Duchenne and non-Duchenne smiling in the first 30 seconds of the FF and RE episodes to ascertain whether expert and automated coding produced similar associations. Levels of infant Duchenne and non-Duchenne smiling were, respectively, not associated with mother Duchenne and non-Duchenne smiling in the first 30 seconds of the FF and RE episodes. The absence of significant associations was observed for both automated measurement and expert coding (automated Duchenne smiling: r(78) = .09, p = .43; expert-coded Duchenne smiling: r(78) = .13, p = .24; automated non-Duchenne smiling: r(78) = −.16, p = .17; expert-coded non-Duchenne smiling: r(78) = −.11, p = .31).

Full Automated Measurement of FFSF Effects

Table 5 includes means and standard deviations of the proportions of infant smiling and cry-faces, as well as mother smiling, during each of the full 2-minute episodes of the FFSF, which were obtained via automated measurement. A series of 3 (Episode: FF, SF, RE) x 2 (Type: Duchenne, non-Duchenne) mixed-model ANOVAs were conducted.

Table 5.

ANOVA and descriptive statistics of AFAR-identified facial expressions over the entire FFSF

Measures	Effect		M	SD	F	df1	df2	p	ηp2	Power (1- β error prob.)
Infant smiling	Episode	FF	.12	.13	22.46	2	78	<.001	.37	1.00
		SF	.06	.09
		RE	.10	.12
	Type	Duchenne	.15	.14	44.43	1	39	<.001	.53	1.00
		Non-Duchenne	.04	.05
	Episode x Type	FF Duchenne	.20	.15	11.19	2	78	<.001	.22	1.00
		FF Non-Duchenne	.05	.06
		SF Duchenne	.09	.11
		SF Non-Duchenne	.02	.04
		RE Duchenne	.17	.14
		RE Non-Duchenne	.03	.04
Infant cry-faces	Episode	FF	.02	.05	6.15	2	78	.003	.14	1.00
		SF	.05	.13
		RE	.06	.13
	Type	Duchenne	.06	.14	1.56	1	39	.219	.04	.80
		Non-Duchenne	.03	.07
	Episode x Type	FF Duchenne	.01	.03	5.00	2	78	.009	.11	1.00
		FF Non-Duchenne	.03	.06
		SF Duchenne	.07	.17
		SF Non-Duchenne	.04	.07
		RE Duchenne	.10	.17
		RE Non-Duchenne	.03	.06
Mother smiling	Episode a	FF	.42	.26	167.10	1.44	56.28	<.001	.81	1.00
		SF	.10	.15
		RE	.36	.22
	Type	Duchenne	.36	.30	12.20	1	39	.001	.24	1.00
		Non-Duchenne	.22	.18
	Episode x Type	FF Duchenne	.58	.23	31.36	2	78	<.001	.45	1.00
		FF Non-Duchenne	.25	.17
		SF Duchenne	.06	.09
		SF Non-Duchenne	.14	.18
		RE Duchenne	.45	.24
		RE Non-Duchenne	.28	.16

Note: Statistics were computed using the proportions of infant smiling and cry-faces and mother smiling during the full 2-minute episodes of the FFSF. Raw means and standard deviations are reported. Episode: main effect of episode (FF vs. SF vs. RE); Type: main effect of type of expression (Duchenne vs. Non-Duchenne expressions); Episode x Type: interaction effect of episode by type of expression. Post-hoc power analysis was performed using G*Power 3.1. η_p² = Partial eta squared.

^aAs the assumption of sphericity was violated for the main effect of episode on mother smiling (Mauchly’s W = .61, p < .001), degrees of freedom were corrected using a Greenhouse-Geisser correction (ε = .72). Assumption of sphericity was met for the main effects and interaction effect on infant smiling and cry-faces, and for the interaction effect on mother smiling (Mauchly’s Ws > .90, ps > .05). Linear mixed-effects analyses yielded conclusions identical to those based on the current results.

Infant Smiling

There was an Episode effect for infant smiling, F(2, 78) = 22.46, p < .001, η_p² = .37, which indicated that there was an overall change in total smiling (Duchenne and non-Duchenne) across episodes. There was a Type effect for infant smiling, F(1, 39) = 44.43, p < .001, η_p² = .53, which indicated a greater proportion of Duchenne smiling than non-Duchenne smiling across episodes. Moreover, there was a significant interaction between Episode and Type, F(2, 78) = 11.19, p < .001, η_p² = .22, which indicated that the pattern of changes across the FFSF episodes differed between Duchenne and non-Duchenne smiling (Figure 2).

Figure 2.

Full-episode automated measurement over the full FFSF. Estimated marginal means of the proportions of AFAR-identified infant Duchenne and non-Duchenne smiling (first plot), infant Duchenne and non-Duchenne cry-faces (second plot), and mother Duchenne and non-Duchenne smiling (third plot) in each full 2-minute episode of the FFSF. Time in infant smiling, infant cry-faces, and mother smiling as a proportion of time in each episode of the FFSF. The blue line represents Duchenne expressions, and the red line represents non-Duchenne expressions. The first plot demonstrates significant declines in both infant Duchenne and non-Duchenne smiling from the FF to the SF, and a rise in infant Duchenne smiling from the SF to the RE. The second plot illustrates a significant rise in infant Duchenne cry-faces from the FF to the SF and no change from the SF to the RE. There was no difference in infant non-Duchenne cry-faces between the episodes. The third plot shows significant declines in mother Duchenne and non-Duchenne smiling from the FF to the SF, and a rise from the SF to the RE. Error bars indicate 95% confidence intervals. Note that the y-axis scales are different for infant smiling, infant cry-faces, and mother smiling.

Pairwise comparisons indicated that total smiling declined from the FF to the SF (p < .001) and increased from the SF to the RE (p < .001). Both infant Duchenne and non-Duchenne smiling declined from the FF to the SF (ps < .001), but only Duchenne smiling rose significantly from the SF to the RE (p < .001). Post-hoc pairwise comparisons are reported in Table S3.

Infant Cry-Faces

There was an Episode effect for infant cry-faces (Duchenne and non-Duchenne), F(2, 78) = 6.15, p = .003, η_p² = .14. There was no Type effect for cry-faces, which indicated that the proportions of infant Duchenne and non-Duchenne cry-faces did not differ across episodes, F(1, 39) = 1.56, p = .22, η_p² = .04. There was a significant interaction between Episode and Type for infant cry-faces, F(2, 78) = 5.00, p = .01, η_p² = .11, which indicated that the pattern of changes across the FFSF episodes differed between Duchenne and non-Duchenne cry-faces (Figure 2).

Pairwise comparisons indicated that total cry-faces (Duchenne and non-Duchenne) increased from the FF to the SF (p = .02) but did not change significantly between the SF and RE (p = .40), which appears to be driven by Duchenne cry-faces, as Duchenne cry-faces increased from the FF to the SF (p = .03) and did not change between the SF and the RE (p = .27; Table S3). Non-Duchenne cry-faces did not change significantly between the FF and the SF (p = .28), or between the SF and the RE (p = .19).

Mother Smiling

There was an Episode effect for mother smiling (Duchenne and non-Duchenne), F(1.44, 56.28) = 167.10, p < .001, η_p² = .81. Similar to infant smiling, there was a Type effect, F(1, 39) = 12.20, p = .001, η_p² = .24, which indicated a greater proportion of Duchenne smiling than non-Duchenne smiling across episodes. Moreover, there was a significant interaction between Episode and Type, F(2, 78) = 31.36, p < .001, η_p² = .45, which indicated that the pattern of changes across the episodes differed between Duchenne smiling and non-Duchenne smiling (Figure 2).

Pairwise comparisons indicated that total smiling (Duchenne and non-Duchenne) declined from the FF to the SF (p < .001), then rose from the SF to the RE (p < .001). Both Duchenne and non-Duchenne smiling significantly declined from the FF to the SF (p < .001 and p = .002, respectively) and rose from the SF to the RE (ps < .001). However, the mean differences between the episodes were 2 – 4 times greater for Duchenne smiling than for non-Duchenne smiling (see Table S3).

Discussion

The FFSF is widely used to examine infant affective responses to parental unresponsiveness. However, we have little understanding of the role of Duchenne expressions, which index strong affective states, in infants and parents during the FFSF. This may be due to the resources required for manual coding of facial AUs such as AU6 (cheek raising or eye constriction) that characterizes Duchenne expressions. The current study examined infant and mother Duchenne and non-Duchenne expressions in the FFSF, using both manual expert coding and an automated measurement procedure based on machine learning detection to detect AUs. Expert coding and automated measurement of FACS AUs yielded similar patterns of change in infant and mother Duchenne and non-Duchenne expressions across the FFSF. These results highlight the convergent validity of automated measurement of facial expressions, particularly smiling, in the FFSF.

Full-episode AFAR data indicated that both infants and mothers demonstrated more Duchenne smiling during the interactive FF and RE episodes than the SF episode. Infants demonstrated more Duchenne cry-faces during the SF than the interactive FF and RE episodes. However, these patterns were not evident for infant non-Duchenne cry-faces. These findings suggest that Duchenne expressions may be an especially sensitive index of infant and mother affective valence during the FFSF. It is possible, then, that standard finding of reductions of infant smiling and increases in cry-faces during the FFSF are specifically dependent on changes in the Duchenne forms of these expressions.

Expert Coding and Automated Measurement

We ascertained the convergent validity of infant and mother Duchenne and non-Duchenne smiling—and infant Duchenne and non-Duchenne cry-faces—by testing for differences between expert and automated (AFAR) levels of these expressions (main effects of measurement approach). There were no differences between expert coding and AFAR in the overall proportion of infant non-Duchenne smiling and Duchenne cry-faces, which suggests convergent validity for these expressions. However, AFAR detected more infant Duchenne smiling and non-Duchenne cry-faces than did expert coders. In fact, expert coding indicated almost no non-Duchenne cry-faces in the FFSF. It is noteworthy, however, that there was substantial agreement between expert coding and AFAR at an individual AU level (Kappas above .75). One possibility is that AFAR over-detects AU6 during smiles and under-detects the absence of AU6 during cry-faces. This suggests AFAR does not reliably detect all facial expressions formed by different AU combinations. It may be that data augmentation of training data for machine learning is necessary to more accurately measure rarely occurring combinations of AUs, such as those involved in the non-Duchenne cry-face. AU6, termed cheek raiser in FACS, constricts the eyes, which may be a relatively subtle change in infants who have relatively high levels of subcutaneous fat. Another possibility is that AFAR is more accurate at detecting these changes than human coders. Further investigation employing automated measurement appropriately trained on combinations of AUs is warranted.

We also examined patterns of change in infant and mother Duchenne and non-Duchenne expressions across FFSF episodes (the interaction between measurement approach and episode). The convergent validity of AFAR automated measurement was supported by non-significant interaction effects for infant Duchenne and non-Duchenne smiling, and Duchenne cry-faces. This suggests that expert coding and automated measurement yielded substantively identical FFSF effects for these infant expressions. However, a significant interaction effect for infant non-Duchenne cry-faces indicated that while automated measurement indicated an increase in infant non-Duchenne cry-faces from the SF to the RE, expert coding indicated no change between the two episodes. A conservative interpretation suggests caution in interpreting FFSF effects based on automated measurement of non-Duchenne cry-faces.

For mothers, there were no overall differences in the levels of Duchenne and non-Duchenne smiling between expert coding and AFAR. In addition, interaction effects were not significant. Reliability between expert and AFAR coding for mother facial expressions were somewhat variable (κ = .63 for AU6 and κ =.84 for AU12). Nevertheless, results from both expert coding and AFAR indicated declines in Duchenne smiling in the RE compared to the FF for expert coding and automated measurement. For both expert coding and automated measurement, mother non-Duchenne smiling did not differ between the FF and the RE. This suggests a concordance between expert coding and automated measurement for mother smiling in the FFSF. Levels of infant and mother Duchenne and non-Duchenne smiling were not associated during the first 30 seconds of the FFSF. Although this finding may reflect the brief sample window examined, it was consistent between experts and AFAR. It appears that expert coding and automated measurement may produce broadly comparable results when applied to one of the most common interactive protocols used in early infancy.

Automated Duchenne and Non-Duchenne Expression

Infant Smiling

Here we report on substantive results from automated facial expression detection (AFAR) in the full FFSF episodes. Infant total smiling (Duchenne and non-Duchenne) declined from the FF to the SF. This finding is consistent with reports of declines in overall infant smiling from the FF to the SF (Ekas et al., 2013; Mattson et al., 2013a; 2013b) and with more general reports of declines in of positive affect indicators from the FF to the SF (Mesman et al., 2009; Stockdale et al., 2020; Yaari et al., 2018). We found that both infant Duchenne smiling and non-Duchenne smiling declined from the FF to the SF, which is consistent with Mattson et al.’s (2013a) report on 12 infants. However, Mattson et al. reported on Duchenne smiling as a proportion of overall smiling, while we examined Duchenne smiling as a proportion of the overall episode. We believe the overall proportion to be a relatively convenient, well-validated metric of infant positive affect, which is well suited to the exploration of individual differences in the FFSF.

Automated measurement of the FF, SF, and RE episodes indicated accentuated patterns of changes across episodes for infant Duchenne expressions. Only Duchenne smiling (not non-Duchenne smiling) increased from the SF to the RE. Thus, infants demonstrated more Duchenne smiling in the interactive FF and RE episodes than in the SF. In other words, when infants interacted with their parents, infant smiles were more emotionally positive than when reacting to a still-faced parent. In addition, a significant interaction effect indicated that there was a steeper decline in Duchenne smiling than in non-Duchenne smiling from the FF to the SF and a steeper increase from the SF to the RE. The results suggest that infant Duchenne smiling is an especially sensitive metric of response to the cessation and resumption of playful interaction with the parent. To index infant positive affect in the FFSF future studies might efficiently focus on Duchenne smiling—rather than all infant smiling.

Infant Cry-faces

The current study extended Mattson et al.’s (2013a) investigation of infant Duchenne and non-Duchenne cry-faces in the FF and SF to the RE episode. Automated measurement indicated a rise in infant Duchenne cry-faces, but not non-Duchenne cry-faces, from the FF to the SF. This is consistent with Mattson et al.’s (2013a) reports of greater levels of the Duchenne marker in cry-faces in the negative affect-eliciting SF episode than the positive affect-eliciting play (FF) episode, as well as with more general reports of increases in infant cry-faces from the FF to the SF (Ekas et al., 2013; Mattson et al., 2013a; 2013b). Infant Duchenne cry-faces persisted into the RE episode of the current FFSF. In the RE episode, infants typically show a partial reduction in affectively negative behaviors from the previous SF episode (Mesman et al., 2009). These results suggest that the infant Duchenne cry-face is a particularly sensitive metric of the impact of the cessation of playful interaction with the parent on subsequent interaction. Future research might focus on Duchenne cry-faces – rather than all infant cry-faces – to index infant negative facial affect in the FFSF.

Mother Smiling

Validating adherence to instructions to hold a still-face, we found both greater mother Duchenne and non-Duchenne smiling in the FF and the RE than in the SF. Following up on a significant interaction effect, mothers showed proportionally more Duchenne smiling than non-Duchenne smiling in the FF and the RE than the SF. Naturally-elicited Duchenne smiling appears to index more intense positive affect than non-Duchenne smiling in adults (Gunnery & Ruben, 2016). The current findings of accentuated levels of mother Duchenne smiling during the interactive episodes of the FFSF suggest that Duchenne smiling may function as a primary expression of positive affect for mothers in infant-mother face-to-face interaction (Messinger et al., 2009).

Limitations and Future Directions

The current study compared expert and automated approaches to investigate infant and mother Duchenne and non-Duchenne expressions in a standard interaction protocol. Accentuated patterns of changes in Duchenne expressions over the FFSF suggest that still-face effects may be driven by Duchenne expressions that index intense positive and negative affect for both infants and mothers in infant-mother face-to-face affective communication. While the Duchenne marker was coded as present or absent, it is also a continuous action produced by the muscle encircling the eye (Messinger et al., 2009; 2012). Future investigations employing continuous measurements of the intensity of the Duchenne marker and other facial actions may further our understanding of facial expressions during infant-mother face-to-face interactions and dynamic changes within episodes of the FFSF (Ekas et al., 2013).

We used experts who were trained in and employed the comprehensive, anatomically-based FACS to code facial actions (Ekman et al., 2002). Based on that standard, we conducted reliable, automated measurement of facial expressions in the largest FFSF sample to date. Levels of missing data in which facial actions could not be coded were comparable for the expert and automated approaches. For both approaches, missing data levels were higher for infants who exhibit high levels of head and hand movement—than for mothers. Nevertheless, there are limitations to the current study’s automated measurement approach.

Although automated measurement reliably measured facial expressions, the machine learning algorithms required expert coders’ manual measurement for training, which may replicate subjectivity and biases involved in manual coding. Moreover, limitations in manually coded training data may constrain the performance and function of automated measurement when applied to different samples in different contexts. For example, mothers, but not fathers or other caregivers, were participants in the current study. Likewise, the FFSF physically constrains the infant and mother, and toys were not permitted. Thus, the current results may differ from those in which infants and parents play on the floor, or play with objects. The development of automated measurement methods trained with diverse samples in diverse settings is imperative for improving the objectivity and generalizability of automated measurement methods. Potential applications include infant temperament assessments and clinical protocols for assessing behavioral indicators of disorders such as autism spectrum disorder (Carpenter et al., 2021; Ahn et al., 2023).

In sum, the current study united research on intense affective valence communicated by Duchenne expressions with research on the widely used FFSF assessment of socioemotional functioning competence (Granat et al., 2016; Mitsven et al., 2020). Results highlight the potential of computer vision approaches to enable objective understanding of early interactive facial expressions. They suggest that affective changes produced by the FFSF may be uniquely reflected in infant and mother Duchenne expressions, and that these expressions are indices of intense positive and negative emotion.

Data Availability Statement:

The data that support the findings of this study are openly available in Databrary at https://nyu.databrary.org/volume/1474.