Face models (combined appearance models, Cootes, 2001) were used to study synchrony in the facial expressions of several pairs of people in previously-recorded two to four minute dyadic conversations. In each conversation, one person was chosen to be the speaker. Speakers described a memory where they felt happiness, sadness, surprise, or disappointment.
Each face model consisted of 99 points that represent the major features of the face. Point matrices from the models were constructed such that the columns of these matrices represented the X and Y point coordinates in the model, and the rows represented each frame in the original tracked video. To ensure that facial expressions were analyzed, the non-rigid head movement components of these matrices were extracted by taking the singular value decomposition of the models. Reconstructed matrices from the models based on people within the same conversation were then horizontally concatenated to form paired matrices. To test synchrony of the movement of the face in addition to static facial expressions, the unsigned root mean square (RMS) velocities of the points were also calculated. The RMS velocity matrices were constructed similarly to the point matrices.
Factor analyses were performed on the paired matrices to see whether factors were shared between participants within each pair. Analyses were performed with no time delay and with a constant phase shift. In order to include the time delay in the model, the matrices of each person in the conversation were shifted by a pre-specified number of rows before being concatenated. These lags were chosen to best represent the lag between speaker and listener interaction. A variable time delay was also examined by performing windowed cross-correlations on the root mean square of the point coordinates. This process split the data from the pairs into five-second sections and found lags that corresponded to the highest correlation between each pair of sections (Boker et al., 2002).
All factor analyses included factors that were shared by both the speaker and listener. The number of factors shared by participants increased after incorporating a time delay. Overall, the constant time delays that produced the greatest number of factors shared between participants were between 20 and 30 frames (2/3 to 1 second delay) apart. This delay was also reflected in the windowed cross-correlations. In conclusion, there was some degree of synchrony between the facial expressions of speakers and listeners in these conversations, although this synchrony could be dependent on the topic of conversation.
Acknowledgments
The author would like to thank Steven Boker and Stephen West for their comments on prior versions of this manuscript. The ideas and opinions expressed herein are those of the author alone, and endorsement by the author’s institution is not intended and should not be inferred.
Funding: Funding for this work was provided in part by the National Institute on Aging (1R21-AG041035) and National Science Foundation (BCS--1030806). Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation or the National Institutes of Health.
Role of the Funders/Sponsors: None of the funders or sponsors of this research had any role in the design and conduct of the study; collection, management, analysis, and interpretation of data; preparation, review, or approval of the manuscript; or decision to submit the manuscript for publication.
Footnotes
Conflict of Interest Disclosures: The author signed a form for disclosure of potential conflicts of interest. The author did not report any financial or other conflicts of interest in relation to the work described.
Ethical Principles: The author affirms having followed professional ethical guidelines in preparing this work. These guidelines include obtaining informed consent from human participants, maintaining ethical treatment and respect for the rights of human or animal participants, and ensuring the privacy of participants and their data, such as ensuring that individual participants cannot be identified in reported results or from publicly available original or archival data.
References
- Boker SM, Rotondo JL, Xu M, King K. Windowed cross-correlation and peak picking for the analysis of variability in the association between behavioral time series. Psychological Methods. 2002;7(3):338. doi: 10.1037/1082-989x.7.3.338. [DOI] [PubMed] [Google Scholar]
- Cootes TF, Edwards GJ, Taylor CJ. Active appearance models. IEEE Transactions on Pattern Analysis & Machine Intelligence. 2001;6:681–685. [Google Scholar]