Universal facial expressions uncovered in art of the ancient Americas: A computational approach

Analyses of facial poses in ancient American sculpture reveal parallels with modern Western conceptions of emotional behavior.

Abstract

Central to the study of emotion is evidence concerning its universality, particularly the degree to which emotional expressions are similar across cultures. Here, we present an approach to studying the universality of emotional expression that rules out cultural contact and circumvents potential biases in survey-based methods: A computational analysis of apparent facial expressions portrayed in artwork created by members of cultures isolated from Western civilization. Using data-driven methods, we find that facial expressions depicted in 63 sculptures from the ancient Americas tend to accord with Western expectations for emotions that unfold in specific social contexts. Ancient American sculptures tend to portray at least five facial expressions in contexts predicted by Westerners, including “pain” in torture, “determination”/“strain” in heavy lifting, “anger” in combat, “elation” in social touch, and “sadness” in defeat-supporting the universality of these expressions.

INTRODUCTION

Poets, playwrights, painters, and sculptors have long portrayed emotional responses—laughter during play or tears following tragedy—as signs of our humanity. Scientists’ views have been more divergent. One view, consistent with artists’ insights, is that humans have evolved specific patterns of nonverbal behavior, such as laughter, tears, embraces, and sighs, to convey specific meanings, including subjective experiences such as “awe” and “love,” appraisals such as certainty or behavioral dispositions such as avoidance (1–4). A competing view is that the meanings of expressions such as laughter are socially constructed and highly variable across cultures (5–7). Expressive behavior is widely assumed to be a core component of emotion. Given this, the debate over whether emotional expression is biologically prepared or culturally learned speaks more generally to the question of whether human responses to experiences deemed to be among the most important in life (8)—pain, pleasure, triumph, defeat, love, loss, and so on—should be understood as consequences of human nature or as cultural constructions (1, 3–6, 9, 10).

Investigations of the cultural universality or relativity of emotional expression have focused on whether people in remote cultures with minimal Western contact, ranging from the Himba in northern Namibia (11–13) to remote villagers in Bhutan (14), recognize Western expressive signals, including joyful smiles, angry scowls, or sympathetic sighs (4, 6). Typically, participants in a remote culture are asked to match depictions of Western facial, bodily, or vocal expressions to situations or words in their native language (14–22). Results have varied. Whereas many studies have reported strong evidence of universality in the recognition of emotional expression in remote cultures (13, 14, 20–22), other studies have reported little to no evidence of universality (11, 12, 23). Within an ongoing debate over the universality of expressive behavior, proponents of extensive cultural relativity have argued that the evidence for universality in expression has been confounded by the inadvertent communication of Western expectations to participants (5, 7). By contrast, advocates of moderate universality in expressive behavior have reasoned that differences in the language, values, beliefs, and daily experiences of people in remote cultures have caused many survey-based studies to underestimate the degree of universality of the meaning of expressive signals (1, 24). Given these ambiguities, the same data from emotion recognition studies have spawned diametrically opposed positions regarding the cultural universality and relativity of emotional expression (1, 4, 6, 25).

What is needed is an understanding of whether people completely isolated from Western civilization still exhibit evidence of universality in the production or recognition of emotional expression. This is a question that survey-based studies, which rely on some form of contact, are poorly suited to answer. Fortunately, clues to the nature of emotional expression in cultures uncontacted by the West may lie elsewhere. In particular, artistic portrayals of emotional expression in the West have parallels within the ancient Americas, long preceding contact between Western and ancient American civilizations. One example is found in traditions of sculpture, represented in geographically disparate cultures of the ancient Americas dating as far back as 1500 BCE (26, 27). While depictions of emotion are relatively rare in pictorial art forms—which often rely on posed subjects—ancient American figurines, reliefs, and sculptures sometimes portray discernible facial expressions (27). Although ancient American artifacts have largely been removed from their original contexts, it is still sometimes possible to identify aspects of the situation in which the subject of a sculpture is portrayed, be it childbirth, playing a sport, or being held captive (27). Thus, it is possible to investigate whether apparent facial expressions depicted in ancient American sculpture co-occur with portrayals of contexts such as sport in a manner that accords with contemporary Western expectations, reflected in everyday intuition (28) and scientific theory (9).

RESULTS

To investigate the expression of emotion in art of the ancient Americas, we first scoured tens of thousands of artifacts archived by reputable museums in search of Mesoamerican figurines, sculptures, and reliefs that (i) portray subjects within identifiable contexts, (ii) include discernible depictions of faces, and (iii) were deemed credibly authentic upon expert review [by R. R. Stone (R.R.S.); see Materials and Methods for details], given concerns regarding the authenticity of many putative ancient American sculptures (29, 30). While artworks that meet all three of these criteria were quite rare, we were able to compile 63 in total, portraying subjects in eight readily identifiable contexts: being held captive, being tortured, carrying a heavy object, embracing someone, holding a baby, in a fighting stance, playing a ball sport, and playing music (see Fig. 1 for examples). We then isolated the face depictions from images of each artwork, removing signs of their broader context, and gathered judgments from Western participants (N = 325, 148 female, mean age = 35.7) of the 63 artistically portrayed facial poses in terms of 30 emotion categories such as “awe” and “anger” and 13 broader affective features such as valence and arousal, ratings that are broadly representative of the meanings conveyed by expression in present-day cultures (1, 17). Independently, we gathered judgments of the emotions that Westerners (N = 114, 69 female, mean age = 35.6) would expect someone to express in each of the eight contexts portrayed by the 63 sculptures, based on verbal description alone, in terms of the same emotion categories and affective features. Figure 2 shows the correlations, for each emotion category and affective feature, between emotions perceived in face depictions extracted from the ancient American sculptures and emotions predicted for each kind of context portrayed in the sculptures by contemporary individuals. Last, we applied a recently developed principal preserved components analysis (PPCA) method (15, 31) to determine the number of dimensions, or varieties of emotion, required to explain the similarity between the emotions perceived in faces depicted in each artwork and Westerners’ expectations of the emotions that someone would be likely to express in the context that it portrays.

Fig. 1. Ancient American sculptures with discernible faces and contexts.

(A) Captive from Tonina archeological site (Mexico, 690–700 CE). Photo credit: Mauricio Marat, Instituto Nacional de Antropología e Historia. https://www.inah.gob.mx/images/boletines/2016_215/demo/#img/foto5.png (1 July 2019). (B) Tortured, scalped prisoner from Campeche (Mexico, 700–900 CE). Baltimore Museum of Art, Kerr Portfolio 2868, photo by J. Kerr. (C) Maya man carrying large stone (Mexico, 600–1200 CE). Kerr Portfolio 8237, photo by J. Kerr. (D) Joined couple (Mexico, 200–500 CE). Los Angeles County Museum of Art (LACMA) AC1996.146.21, gift of C. M. Fearing. (E) Maya woman holding child (600–800 CE). Princeton University Art Museum 2003-26, gift of G. G. Griffin. (F) Kneeling Maya warrior with facial tattoos and shield (Mexico, 600–800 CE), detail. Earthenware and pigment, 15.9 cm by 10.8 cm. Fine Arts Museums of San Francisco 2009.38.2, gift of G. Merriam and J. A. Merriam. (G) Maya ballplayer (Mexico, 700–900 CE). University of Maine HM646, William P. Palmer Collection. (H) Colima drummer (Mexico, 200 BCE–500 CE). LACMA, Proctor Stafford Collection, purchased with funds provided by Mr. and Mrs. Allan C. Balch.

Fig. 2. Accordance between emotions perceived in sculptures’ isolated face depictions and Western expectations for the emotions that unfold in eight portrayed contexts.

To calculate the accordance between sculptures’ expressions and Westerners’ expectations, we correlated the participants’ average judgments of the emotions and affective features associated with each isolated face and each context across the eight contexts and divided by the maximum attainable correlation given sampling error (see Materials and Methods). Correlations are generally positive, indicating that facial muscle configurations portrayed in ancient American sculptures align, in terms of the emotions they communicate to Westerners, with Western participants’ expectations for the emotions that unfold in different contexts. Error bars represent SEs. Here, we excluded 10 emotions and 1 affective feature used seldom enough that <1/3 of the covariance in judgments was explainable, as a result of which SEs were very large.

Ancient American sculpture was found to portray at least three dimensions, or varieties, of facial expression that accord, in terms of the emotions they communicate to Westerners, with Western expectations for the emotions that might unfold in the eight contexts portrayed (P ≤ 0.0066, q[FDR, false discovery rate] < 0.02, cross-validated PPCA; see Fig. 3A and Materials and Methods for details). The second and third dimensions had interpretable positive loadings on one set of emotions and negative loadings on another set of emotions (Fig. 3B); the first dimension had interpretable loadings only in the positive direction. Thus, faces depicted in ancient American artwork convey to Westerners at least five distinct, interpretable kinds of emotion that are expected to occur in the portrayed contexts. In particular, our findings reveal that ancient American artwork portrays facial muscle configurations that Westerners recognize as expressions of (i) “pain,” often in the context of torture; (ii) “determination”/“strain,” often in the context of heavy lifting; (iii) “anger,” often in the context of combat; (iv) “elation,” often in contexts of familial or social touch; and (v) “sadness”, often in the context of being held captive (defeat).

Fig. 3. Dimensions shared between sculptures’ expressions and Western expectations for the emotions that unfold in eight contexts.

(A) To extract dimensions, or varieties, of perceived facial expression in sculptures that accord with Western expectations for the emotions that might unfold in the eight portrayed contexts, we applied PPCA in a leave-one-subject-out fashion. PPCA extracts shared dimensions between two parallel datasets—in this case, judgments of isolated faces from each of the eight contexts and judgments of emotions associated with descriptions of each context. This analysis revealed three dimensions (P ≤ 0.0066, q[FDR] < 0.02; Wilcoxon signed rank tests). Principal component analysis (PCA) applied to mean expression judgments (blue) also revealed three dimensions of covariance (P = ≤0.0021, q[FDR] < 0.005), although they were not the first three PCs. This misordering, or improper rotation, of dimensions by PCA is expected, given that PCA does not optimize for reliability across datasets but for variance within a dataset (see the “PPCA versus PCA” section in Materials and Methods and movie S1 for illustration). Error bars represent SE. See fig. S1 for analysis including 13 affective features. (B) Positive and negative loadings of perceived emotions in the face on each PPCA dimension. The second and third dimensions were found to have substantial negative and positive loadings, representing two kinds of perceived facial expression. (C) To determine the contexts corresponding to the facial expressions represented by the dimensions, we projected the average ratings from each context onto each dimension. The greatest average rating for each context is outlined in black, each significantly greater than 0 (P < 0.0002; q[FDR] < 0.0002, bootstrap test). Together, (B) and (C) reveal the emotions captured by each dimension. The first dimension primarily represents expressions of pain in one direction (1+), which often occurs in the context of torture, and a number of positive emotions in the other (1−), which tend to occur in contexts of holding a baby, playing a ball game, and playing music. The second dimension most strongly represents expressions of determination and strain in one direction (2+), which occurs more often in the contexts of heavy lifting, and elation in the other (2−), which occurs in contexts of social touch. Last, the third dimension most strongly represents anger in one direction (3+), which occurs in the contexts of combat, and sadness in the other (3−), which occurs in the context of captivity. Expression categories and contexts are ordered according to maximally loading dimension.

To explore the distribution of ancient American artwork along the three dimensions of perceived facial expression we extracted, we created an interactive map (https://s3.amazonaws.com/precolumbian/map.html and Fig. 4). Within the map, faces from sculpture, represented as letters, can be perused alongside the artworks from which they have been extracted. The letters representing each face are located near each other if the faces were evaluated similarly along the three dimensions, as determined by a nonlinear manifold embedding method (32). We also represent the dimensions as distinct colors—two distinct colors for each dimension, representing its positive and negative loadings, blended together according to their scores on each face. Hovering over each letter within the interactive map reveals its exact emotion ratings.

Fig. 4. Mapping sculptures along dimensions of perceived facial expression that tend to accord with predicted emotions.

To explore the distribution of ancient American artwork along the dimensions of perceived facial expression that accorded with predicted responses to each context, projections of each face onto the dimensions were subjected to t-SNE (32), which positioned each face near faces with similar projections. The dimensions are also represented with five distinct colors—one for the first dimension, which was unipolar, and two distinct colors for each the two dimensions that were bipolar. Colors assigned to individual faces are weighted averages of their loadings on each dimension. Eight example sculptures are shown. To explore all 63 sculptures, see online map: https://s3.amazonaws.com/precolumbian/map.html. Credit, from top left down: (i) Metropolitan Museum of Art 2005.91.12, gift of the Andrall and Joanne Pearson Collection, 2005; (ii) Princeton University Art Museum 2003-26, gift of G. G. Griffin; (iii) Metropolitan Museum of Art 1979.206.578, Michael C. Rockefeller Memorial Collection, Bequest of Nelson A. Rockefeller, 1979; (iv) Kerr Portfolio 342, Jaina Figure, photo by J. Kerr; (v) Kimbell Art Museum, Fort Worth, Texas, AP 1971.07, Presentation of Captives to a Maya Ruler (detail) (39); and (vi) Photograph: Museum of Fine Arts, Boston 1983.288, gift of L. T. Clay (40).

Not all faces from each context conveyed identical emotions, just as not all people who are being held captive, holding a baby, or playing a sport would be expected to feel the same way at all times (1, 4, 9). Rather, what the map in Fig. 4 illustrates is that the average expression shifts for certain contexts (Fig. 4). Within the context of being tortured, for example (letter B), the average expression shifts radically toward pain and “distress.” For other contexts, there is often a less radical, but still appreciable, shift in expression. For example, sculptures of people being held captive (letter A) fall in different places within the map—near perceptions of anger, pain, and determination, for example. However, a disproportionate number of sculptures of people being held captive are concentrated near the extremes of the dimension that represents sadness, consistent with how present-day Westerners expect someone being held captive to feel on average (4, 9).

Furthermore, note that the discovery of three of seven possible dimensions of accordance between expression and context (given eight mutually exclusive contexts) does not imply that four dimensions were culture specific. To study cultural differences in emotional expression portrayed in art, it will be critical to analyze Western sculpture, which likely also portrays only a limited number of dimensions of expression in accordance with lay expectations.

DISCUSSION

The findings presented here reveal parallels between the meanings present-day Westerners attribute to facial muscle configurations and the contexts with which these configurations are associated in ancient American artwork. These associations predate contact between Western civilizations and the civilizations of the ancient Americas and therefore cannot be explained by Western cultural influence. This overcomes confounds of survey-based studies that have been the foci of debate over the universality of emotion (1, 4–7, 24).

More generally, our findings demonstrate how clues to the origins of psychological behavior can be found in ancient artifacts. Another recent study uncovered parallels between present-day moral decision-making behaviors and intuitions captured in ancient Chinese and Sumerian legal texts (33). Together, these studies establish how documenting psychological behavior in ancient people can rule out influence from present-day Western civilization.

The study of sculpture from the ancient Americas comes with several methodological concerns. First, it is notable that there have been concerns of authenticity in ancient American sculpture (29, 30). We have mitigated this concern within the present study by applying conservative criteria for inclusion of each sculpture (see the “Gauging authenticity” section in Materials and Methods), leaving it to future studies to analyze expressive behavior in contexts portrayed by sculptures that are often more difficult to authenticate, such as birth and sex. Second, there may be concerns of bias in Western experts’ evaluations of the contexts portrayed in each sculpture. Thus, we only studied sculptures in which the context was readily apparent, marked by clear physical attributes such as visible bodily injury, weaponry being brandished, or babies being held. Last, there may be concerns regarding selection bias in the sculptures included in the study. To mitigate this concern, we scoured thousands of sculptures in museum archives and compiled all that had discernible faces and contexts (see Materials and Methods). Future inquiry into a broader array of sculptures and contexts could be facilitated by future archeological discoveries, along with advances in methods of authentication.

Note that in studying any artwork, there are limitations: We cannot know for certain whether its portrayals are faithful to the everyday lives of the people it depicts. We have no direct insight into the feelings of people from the ancient Americas. What we can conclude is that ancient American artists shared some of present-day Westerners’ associations between facial muscle configurations and social contexts in which they might occur, associations that predate any known contact between the West and the ancient Americas. One parsimonious explanation for these associations is that people actually produced the facial muscle configurations more often in the depicted contexts.

The present results thus provide support for the universality of at least five kinds of facial expression: those associated with pain (17, 34), anger (20), determination/strain, elation (17), and sadness (20). These findings support the notion that we are biologically prepared to express certain emotional states with particular behaviors, shedding light on the nature of our responses to experiences thought to bring meaning to our lives (8).

MATERIALS AND METHODS

Collecting ancient American sculptures

An initial set of ancient American sculptures was gathered by analyzing thousands of sculptures in seven academic and museum databases (the Kerr Portfolio, the Art Institute of Chicago, the Los Angeles County Museum of Art, the Metropolitan Museum of Art, the Princeton University Art Museum, the Boston Museum of Fine Arts, and the Harvard Peabody Museum of Archeology and Ethnology). We compiled sculptures that included discernible faces and contexts. After an initial set of 14 contexts depicted in one or more sculptures was identified (six contexts were later excluded, as detailed below), further examples in museums and galleries were gathered by searching Google images and Pinterest, with queries constructed as follows: <Mesoamerican OR pre-Columbian OR ancient American> <description of context> <figure OR figurine OR sculpture OR relief>. This resulted in an initial set of 161 sculptures.

Gauging authenticity

An expert in ancient American art, R.R.S., conducted a detailed review of the 161 sculptures to gauge their authenticity. R.R.S. served as curator of a major collection of ancient American art at the Emory University’s Michael C. Carlos Museum for 30 years. She has authenticated more than 2500 pieces using her own expertise and by conducting scientific tests. The criteria she used to authenticate the present sculptures were highly conservative, given past concerns regarding the authenticity of many ancient American sculptures (29, 30). As a result, the majority of sculptures were excluded during this second stage of review because of a wide range of potential concerns, summarized in the list below. 1) Construction from materials that cannot be scientifically dated, such as greenstone or goldwork. 2) Idiosyncrasies in the subject matter of the sculpture relative to other known sculptures of the same style. For example, a Chupicuaro birthing mother and Chimu sex vessel were excluded, given that these contexts have not been seen in other sculptures from the same provenance. More generally, birthing and sex scenes were excluded given the frequency of forgeries. 3) Idiosyncrasies in the details of a sculpture relative to other known sculptures of the same style or provenance. For example, a navel chipped into a finished effigy vessel, which is not a known feature of other ancient American sculptures, or lines of body paint vertically traversing the torso and legs in a manner highly unusual for Galo Polychrome female figurines. 4) Iconographic idiosyncrasies, such as a loincloth “scoop” on the back rather than over the genitals in West Mexican figures. 5) Inability to locate sculpture within a reputable archive. Sculptures found in many auction houses were excluded, given past doubts regarding the authenticity of sculptures sold in such venues. 6) Sculptures of putative Western Mexican provenance that included a splattering on of black paint, which is often added by dealers in imitation of a real surface deposit of manganese from ancient times.

Only the remaining sculptures in which R.R.S. expressed a high degree of confidence of authenticity were included in our analysis. Furthermore, we only retained 63 sculptures from eight contexts for which we had three or more examples, listed in the main body of the paper. The contexts that were ultimately excluded were “giving birth,” “having sex,” “ingesting alcohol,” “psychedelic mushrooms,” “holding someone captive,” and “dancing,” because these ultimately had fewer than three credible exemplars.

Gathering judgments of emotions perceived in face depictions extracted from the 63 sculptures

To characterize the perceived facial expressions of each sculpture, we isolated each depicted face and gathered ratings from separate U.S. subjects on Amazon Mechanical Turk of the perceived expressions in terms of 30 emotion categories (N = 125, 64 female, mean age = 35.8) and 13 affective features (N = 200, 84 female, mean age = 35.7) drawn from extensive literature reviews and previous research described by Cowen and Keltner (17) [with added categories of strain and determination (35)]. Survey questions are given in table S1. The experimental procedures were approved by the Institutional Review Board at the University of California, Berkeley. All participants gave their informed consent.

Participants in the emotion category survey were asked to select all emotions that they felt described the feelings expressed by each sculpture, from a list of 30, and assign a 1 to 100 intensity score to each selection. Participants in the affective feature survey rated were asked to rate each face in terms of 13 affective features (listed in Fig. 3C) by answering questions using 1 to 9 Likert scales. For each face, at least 20 emotion category ratings were collected, and at least 12 ratings were collected for each of the 13 affective features (sample sizes were based on the explainable variance obtained in mean judgments in previous studies using similar methods) (15–17, 28). With these methods, 84.0 and 85.7% of the variance across the eight contexts in mean emotion and affective feature judgments, respectively, of the emotions perceived in the face depictions was found to be explainable (see the “Explainable variance and maximum attainable correlation” section).

Gathering emotion predictions from descriptions of the eight contexts

To capture predictions from Western participants of the emotions that would be likely to be expressed in each context, we gathered ratings from separate U.S. subjects on Amazon Mechanical Turk of verbal descriptions of each context in terms of each of the 30 emotion categories (N = 84, 49 female, mean age = 36.7) and 13 affective features (N = 30, 20 female, mean age = 32.4). Survey questions are given in table S1. The experimental procedures were approved by the Institutional Review Board at the University of California, Berkeley. All participants gave their informed consent.

Again, participants in the emotion category survey were asked to select all emotions that applied from a list of 30 and to assign a 1 to 100 intensity score to each selection. Participants in the affective features survey answered 13 questions about each context description using a 1 to 9 Likert scale. For each context, 30 emotion category ratings were collected. Thirty ratings were collected for each of the 13 affective features. With these methods, 94.5 and 96.4% of the variances across the eight contexts in mean emotion and affective feature judgments, respectively, of predicted emotions were found to be explainable (see the “Explainable variance and maximum attainable correlation” section).

Averaging emotion judgments

For each context, we averaged all emotion category and affective feature ratings of each face from sculptures depicting that context and all category and affective feature ratings of descriptions of that context. This resulted in: (i) an 8 × 30 matrix of 1 to 100 values for each emotion category judgment of faces in each context; (ii) an 8 × 13 matrix of 1 to 9 values for each affective feature of faces in each context; (iii) an 8 × 30 matrix of 1 to 100 values for each emotion category judgment of descriptions of each context; and (iv) an 8 × 13 matrix of 1 to 9 values for each affective feature of descriptions of each context. Correlations across the eight contexts were computed between corresponding columns of the face judgment and context description judgment matrices (Fig. 2).

Explainable variance and maximum attainable correlation

When two sets of sample means are correlated, the correlation is biased downward by sampling error (36, 37). Thus, more ratings would lower the SE and yield a higher correlation, although “unbiased” estimates should not be biased by sample size. To correct for this bias, we can divide the sample correlation by the approximate maximum correlation that could be obtained given the sampling error. The maximum attainable correlation is the square root of the proportion of the variance that is not attributable to sampling error (the explainable variance).

To calculate explainable variance (38), we note that the variance of a given rating across stimuli is equal to the explainable variance plus the unexplainable variance. The unexplainable variance can be estimated as the mean of the squared SEs across stimuli. Hence, the proportion of explainable variance can be estimated by simply dividing the mean of the squared SEs by the total variance and subtracting this quantity from 1.

More formally, let ${\overline{Y}}_j$ be the mean judgment of stimulus j, ${\mathrm{\sigma }}_j^2$ be the SE of the mean judgment ${\overline{Y}}_j$, and σ² be the variance of ${\overline{Y}}_j$ over all stimuli j. Note that the actual proportion of explainable variance in the mean is given by

$$r_{\text{exp}}^2=1-\frac{\frac{1}{J}{\displaystyle {\sum }_{j=1}^J}{\mathrm{\sigma }}_j^2}{{\mathrm{\sigma }}^2}$$

Now, if $\overline{Y}$ is the observed mean over all ${\overline{Y}}_j$, then we estimate σ² with $s^2=\frac{1}{J} {\sum }_{j=1}^J{({\overline{Y}}_j-\overline{Y})}^2$. We estimate the SE for each stimulus, ${\mathrm{\sigma }}_j^2$ with $s_j^2$, the sample SE. The maximum attainable correlation can be estimated as the square root of r²_exp. See Cowen et al. (15) for results of repeated Monte Carlo simulations, further validating these methods.

To calculate the explainable variance and maximum attainable correlation in facial expression judgments across the eight contexts (rather than across the individual faces extracted from the 63 sculptures), we estimated SEs s²_c using the formula for the SD of the mean of random variables, $s_c^2=\frac{1}{J} {\sum }_{j=1}^J{s}_j^2$ across the J faces in each context. The proportion of explainable variance and maximum attainable correlation across the eight contexts was then calculated, as above, replacing s²_j with s²_c and taking $s^2=\frac{1}{8} {\sum }_{c=1}^8{({\overline{Y}}_c-\overline{Y})}^2$ for the eight contexts.

Principal preserved components analysis

PPCA extracts shared dimensions that maximize the covariance between two parallel datasets—in this case, the 8 × 30 matrix of average facial expression ratings of isolated faces from each of the eight contexts and the 8 × 30 matrix of emotions participants would, on average, expect someone to express in each context (see also fig. S1 for results of analysis, including the 13 affective features, which captured an additional, difficult-to-interpret dimension). To do so, PPCA first seeks a unit vector α₁ that maximizes the objective function

$$\text{Cov}(\mathbf{X}{\mathbf{\alpha }}_1,\mathbf{Y}{\mathbf{\alpha }}_1)$$

We call α₁ the first principal preserved component. Subsequent components are obtained by seeking additional unit vectors α_i that maximize the objective function Cov(Xα_i, Yα_i) subject to the constraint that α₁ is orthogonal to the previous components, α₁,…, α_i–1.

In the special case that X = Y, PPCA is equivalent to principal components analysis (PCA), given that the latter method maximizes the objective function

$$\text{Var}(\mathbf{X}{\mathbf{\alpha }}_i)=\text{Cov}(\mathbf{X}{\mathbf{\alpha }}_i,\mathbf{X}{\mathbf{\alpha }}_i)$$

(substituting another X for Y in Cov[Xα₁, Yα₁]). Also note the similarity to the Partial Least Squares Correlation objective, which seeks to find two separate bases α and β to maximize

$$\text{Cov}(\mathbf{X}{\mathbf{\alpha }}_i,\mathbf{Y}{\mathrm{ß}}_i)$$

and the CCA objective, which seeks to maximize

$$\text{Corr}(\mathbf{X}{\mathbf{\alpha }}_i,\mathbf{Y}{\mathrm{ß}}_i)$$

However, given our aim of finding preserved dimensions of emotion across sculptures’ perceived facial configurations and Westerners’ expectations based on each context, PPCA derives only one basis, α, that applies to both datasets. In PPCA, therefore, the data matrices must be commensurate: Observations in both datasets must be of the same dimension; i.e., the number of rows in X and Y must be equal.

To solve the PPCA objective and find an α₁, we apply eigendecomposition to the addition of the cross-covariance matrix between datasets and its transpose: Cov(X,Y)/2 + Cov(Y,X)/2. We claim that the principal eigenvector of this symmetric matrix maximizes Cov(Xα₁, Yα₁). To derive this, first recall a general property of cross-covariance, Cov(Xa, Yb) = b^TCov(X, Y)a. Thus

$$\text{Cov}(\mathbf{X}{\mathbf{\alpha }}_1,\mathbf{Y}{\mathbf{\alpha }}_1)={\mathbf{\alpha }}_1^T\text{Cov}(\mathbf{X},\mathbf{Y}){\mathbf{\alpha }}_1$$ (property 1)

In addition, because both Xα₁ and Yα₁ are vectors, Cov(Xα₁, Yα₁) = Cov(Yα₁, Xα₁). Thus

$$\text{Cov}(\mathbf{X}{\mathbf{\alpha }}_1,\mathbf{Y}{\mathbf{\alpha }}_1)=\text{Cov}(\mathbf{X}{\mathbf{\alpha }}_1,\mathbf{Y}{\mathbf{\alpha }}_1)/2+\text{Cov}(\mathbf{Y}{\mathbf{\alpha }}_1,\mathbf{X}{\mathbf{\alpha }}_1)/2$$ (property 2)

Combining these two properties, we can see that

$$\text{Cov}(\mathbf{X}{\mathbf{\alpha }}_1,\mathbf{Y}{\mathbf{\alpha }}_1)=\text{Cov}(\mathbf{X}{\mathbf{\alpha }}_1,\mathbf{Y}{\mathbf{\alpha }}_1)/2+\text{Cov}(\mathbf{Y}{\mathbf{\alpha }}_1,\mathbf{X}{\mathbf{\alpha }}_1)/2$$ (by property 2)

$$={\mathbf{\alpha }}_1^T\text{Cov}(\mathbf{X},\mathbf{Y}){\mathbf{\alpha }}_1/2+{\mathbf{\alpha }}_1^T\text{Cov}(\mathbf{Y},\mathbf{X}){\mathbf{\alpha }}_1/2$$ (by property 1)

$$={\mathbf{\alpha }}_1^T[\text{Cov}(\mathbf{X},\mathbf{Y})/2+\text{Cov}(\mathbf{Y},\mathbf{X})/2]{\mathbf{\alpha }}_1$$

Now, letting R = [Cov(X,Y)/2 + Cov(Y,X)/2], we see that maximizing α₁^TRα₁ is equivalent to maximizing Cov(Xα₁, Yα₁), the originally stated PPCA objective. Note that if X = Y, then we are applying eigendecomposition to Var[Xα_i] = Cov[Xα_i, Xα_i], which performs PCA.

Last, the min-max theorem dictates that the principal eigenvector of R maximizes α₁^TRα₁ subject to α₁ being a unit vector (|α₁| = 1).

We have thus found a unit vector α₁ that maximizes Cov(Xα₁, Yα₁)—the covariance between the projections of X and Y projected onto the first component. On the basis of the min-max theorem, subsequent eigenvectors α_i will maximize Cov(Xα_i, Yα_i) subject to their orthogonality with previous components α₁ through α_i–1 and to each α_i also being a unit vector (|α_i| = 1).

We note that the min-max theorem also provides that the last eigenvector, α_n, will minimize Cov(Xα_n, Yα_n), equivalent to maximizing −Cov(Xα_n, Yα_n). Hence, if there are dimensions of negative covariance between the two datasets, then some eigenvectors will maximize the negative covariance.

With respect to the corresponding eigenvalues, each eigenvalue λ_i will be equal to Cov(Xα_i, Yα_i). To see this, note that

$$[\text{Cov}(\mathbf{X},\mathbf{Y})/2+\text{Cov}(\mathbf{Y},\mathbf{X})/2]{\mathbf{\alpha }}_i={\mathrm{\lambda }}_i{\mathbf{\alpha }}_i$$ (eigenvalue equation)

$${\mathbf{\alpha }}_i^T[\text{Cov}(\mathbf{X},\mathbf{Y})/2+\text{Cov}(\mathbf{Y},\mathbf{X})/2]{\mathbf{\alpha }}_i={\mathbf{\alpha }}_i^T{\mathrm{\lambda }}_i{\mathbf{\alpha }}_i$$

$$\text{Cov}(\mathbf{X}{\mathbf{\alpha }}_i,\mathbf{Y}{\mathbf{\alpha }}_i)={\mathrm{\lambda }}_i{\mathbf{\alpha }}_i^T{\mathbf{\alpha }}_i$$ (by property 1)

Now, α_i^Tα_i = 1, because the α_is are orthonormal. Hence

$$\text{Cov}(\mathbf{X}{\mathbf{\alpha }}_i,\mathbf{Y}{\mathbf{\alpha }}_i)={\mathrm{\lambda }}_i$$

This also entails that there will be negative eigenvalues corresponding to negative covariance. To ascertain the number of significant dimensions of covariance between perceived facial expression and expected emotions, we performed a leave-one-subject-out analysis, in which PPCA was iteratively performed on data from all but one rater from the facial expression judgment task, and then the held-out ratings were projected onto the extracted dimensions and correlated with projections of the expected emotions in each context onto the extracted dimensions. Partial Spearman correlations were used, controlling for projections onto previous dimensions, to account for possible curvilinear relationships. Wilcoxon signed rank tests were then applied to the correlations for held out raters to test each dimension for significance. See Cowen et al. (15) for results of repeated Monte Carlo simulations, further validating these methods.

To perform the leave-one-subject-out analysis, including the 13 affective feature judgments (fig. S1), which were rated by a separate set of participants, we arbitrarily paired each of the 125 category judgment participants with an affective feature judgment participant. These participant pairs were then treated as a single participant during the leave-one-subject-out procedure.

PPCA versus PCA

We also performed the analysis replacing PPCA with PCA applied only to the average facial expression judgments, not incorporating their relationship with the expected emotions. This analysis established that the dimensions we uncovered were present in the ratings even if we did not explicitly seek them out with PPCA and ensured that the PPCA dimensions were not overfit to the particular sculptures included in our study.

However, PCA extracted a different order, or rotation, of dimensions, which no longer reflected the degree of accordance between perceived expression and predicted emotion (Fig. 3A). This is expected, given that PCA does not optimize for covariance across datasets (it maximizes Var[Xα_i] rather than Cov[Xα_i, Yα_i]; see movie S1 for illustration). To understand the degree of alignment between Western perceptions of emotion in the depicted faces and Western expectations of emotion in the depicted contexts, we therefore analyzed the PPCA dimensions and not the PCA dimensions.

Visualizing the distribution of perceived expressions

To generate the spatial coordinates of each sculpture sample within the map, we applied a method called t-distributed stochastic neighbor embedding (t-SNE) (32), among the most popular techniques for visualizing high-dimensional data. To visualize the data along just two dimensions, t-SNE attempts to preserve shorter distances between data points (average judgments of sculpture facial configurations) while sacrificing the accuracy of its representation of longer distances. As a result, t-SNE naturally groups together sculptures that convey similar emotions and is able to capture smooth, continuous variations within the space, despite being limited to two dimensions. Of course, some information is lost in this process—this is why it is important to simultaneously view a second, independent channel of information, conveyed through the color assigned to each sculpture (see Fig. 4). Given that t-SNE will generate different results each time it is run, we ran t-SNE 10 times and verified that the relative locations of sculptures within the map were qualitatively similar each time.

SUPPLEMENTARY MATERIALS

Supplementary material for this article is available at http://advances.sciencemag.org/cgi/content/full/6/34/eabb1005/DC1