The Importance of Context: Three Corrections to Cordaro, Keltner, Tshering, Wangchuk, & Flynn (2016)

Abstract

In their recently published paper, The Voice Conveys Emotion in Ten Globalized Cultures and One Remote Village in Bhutan, Cordaro and colleagues conclude that certain emotion categories are universally recognized by people around the world, barring illness and measurement error. The impact of Cordaro et al.’s paper, like the impact of all empirical papers, is determined not only by it’s research findings, but also by how the research findings are situated. Accuracy in characterizing the scientific context of new findings is as important as maintaining the highest standards for other aspects of the scientific method. In this regard, we point out three areas of concern in Cordaro et al.’s discussion of past research on remote samples, the use of more discovery-oriented (and less confirmatory) experimental methods in past research, and the use of manipulation checks in past research. Ultimately, a paper’s contribution to scientific progress is limited when ambiguities and oversights obscure the real value of its findings.

It is widely believed that certain emotions are elicited, expressed, and recognized by everyone in the world, barring illness and measurement error. Both Western and traditional Buddhist theories of human nature share a belief in universal emotions. It’s a pedagogical cornerstone in many scientific disciplines, including psychology, affective computing, and neuroscience. Young children are even encouraged to recognize the presumed universal emotions with games, puzzles, posters, and television programming, such as episodes of Sesame Street. The belief in universal emotions is big business: The U.S. government spent 900 million dollars training agents in the U.S. Transportation Security Administration to detect potential terrorists using a method that is grounded in the belief that certain emotions are universally expressed (Weinberger, 2010); this screening method was used on the approximately 1.8 million travelers that pass through U.S. airports every day between 2007 and 2013 (The American Civil Liberties Union v. the U.S. Transportation Security Administration; http://www.gao.gov/products/GAO-14-159). And companies like Apple and Google are betting that the belief in universal emotions is true by investing considerable resources (i.e., millions of dollars and person hours) to develop “emotion reading” devices for a variety of purposes.

In this milieu, Cordaro and colleagues’ recent paper, The Voice Conveys Emotion in Ten Globalized Cultures and One Remote Village in Bhutan, is particularly valuable, because it has the potential to broaden our knowledge of emotion perception across a range of cultures, including within a remote cultural context. People who have little exposure to Western cultural practices and norms provide an opportunity for the strongest scientific test of whether some emotions are truly universal (Norenzayan & Heine, 2005). Cultural differences in learning and language are less able to explain emotion perception if people from relatively more isolated locations in the world are similar to people who live in more urban, Westernized settings in perceiving certain communicative gestures (such as a scowling face or a growl) as emotional expressions (such as anger). To date, emotion perception has been studied in only a handful of remote cultural contexts. Cordaro, Keltner, Tshering, Wangchuk, & Flynn (2016) appear to add to this small list. They report that people who live in an isolated northeastern sector of the Himalayas (in the country of Bhutan) are similar to people who live in more Western industrialized settings in perceiving sadness, disgust, fear, awe, amusement, and several other emotions.

A paper’s impact is not only determined by the research it details, however. Also important is the way that the authors situate their findings in the existing literature. In this regard, it is worth discussing three concerns regarding how Cordaro et al. handled previously published scientific findings that are inconsistent with the universality hypothesis.

Concern #1: Did Cordaro et al. accurately describe previous research on remote samples?

In 2014, we published a paper that reports evidence of meaningful cultural variation in emotion perception in a remote, rural sector of Northwestern Namibia. People of the Himba cultural group did not perceive emotion in vocal bursts in a way that is broadly consistent with the hypothesis of universal emotions (i.e., their perceptions differed in substantial ways from those of Western perceivers; Gendron, Roberson, van der Vyver & Barrett, 2014a). In our experiment, we asked one group of participants to categorize vocal bursts using the methods described in Sauter et al. (2010), which reports evidence in support of universal emotion perception from a sample of Himba participants. Sauter et al. used a procedure where a participant is first presented with a conceptual context (e.g., a brief emotional story) followed by a selection of vocalizations (a target and either one or two foils), after which the participant chooses the vocalization that best matched the story; Cordaro et al. refer to as the “story matching method” (it is also called the “Dashiell method,” after the scientist who first introduced it in 1927). In Gendron et al. (2014a), we also asked a second sample of Himba participants to offer their own labels for the vocalizations (called the “free labeling method”). We also published a companion piece, a conceptual replication reporting similar results for another sample of Himba participants who were asked to judge stereotyped, facial poses for emotion (Gendron Roberson, van der Vyver, & Barrett, 2014b). The findings from both papers build on a series of published laboratory experiments from our lab (e.g., Gendron, Lindquist, Barsalou, & Barrett, 2012; Lindquist, Barrett, Bliss-Moreau, & Russell, 2006; Lindquist, Gendron, Dickerson, & Barrett, 2014) and other labs (e.g., Carroll & Russell, 1996; Knudsen & Muzekari, 1983; Roberson, Damjanovik, & Pilling, 2007; Russell, 1993, 1994; Widen, Christy, Hewett, & Russell 2011), showing that the standard experimental paradigms (such as the story matching method) appear to, encourage false positives, yielding evidence of universal emotion perception that might not otherwise be present (such as when using a free labeling method). It is necessary to accurately describe Gendron et al. (2014a), as well as its companion piece, to properly understand the scientific value of the studies reported in Cordaro et al. (who used a version of the story matching method). Unfortunately, this description was difficult to find in Cordaro et al.

For a number of years now, the story matching method (and similar classification methods) have been criticized because they contain conceptual elements that influence participants’ responses in a confirmatory way, encouraging them to produce responses that have the appearance of universality. Not only do these methods constrain the test of emotion perception to the options preferred by the experimenter (cf. Russell, 1994; Nelson & Russell, 2013),¹ but the words and stories offered to participants across the trials of the experiment serve to prime concept knowledge that serves as context (e.g., Lindquist and Barrett, 2008; Lebois, Wilson-Mendenhall & Barsalou, 2015; Barsalou, 2008), even though experimenters consider them psychologically inert. As such, these methods provide a poor test of the hypothesis that certain emotions are “recognized” in an obligatory way by everyone in the world, without the benefit of language or learning or context (see Izard, 2004 for a discussion). Classification methods are “confirmation” oriented, while free labeling methods are “discovery” oriented (cf. Russell, 1994) and assess the emotions that participants spontaneously perceive. The studies we report in Gendron et al. (2014a, b) are perhaps the first to demonstrate these observations in a remote culture using well-controlled experimental methods. Knowing this background would help readers evaluate the Cordaro et al. findings for themselves.

Cordaro et al.’s discussion of Gendron et al. (2014a) will likely confuse readers. The paper is first mentioned on page 118, where it is cited as a justification for using the story matching method; specifically, Cordaro et al. write that they used a story matching methodology developed in previous studies, including Gendron et al. in that list ² and they again cite Gendron et al. in a similar way on page 120. On page 124, Cordaro et al. present their findings with those from similar studies of vocal bursts, including Gendron et al. (2014), and write, “When taken together, these studies provide strong evidence for universal vocal bursts” of some emotion categories, while “moderate evidence of universality for vocal bursts” of other categories. This statement misleading; Gendron et al (2014a) does not support any such conclusion.

Concern #2: Did Cordaro et al. accurately describe previous research that is freer from the artificial constraints of standard classification methods?

In discussing the future research that is stimulated by their findings, Cordaro et al. draw readers’ attention to the fact that tests of the universal emotion hypothesis would benefit from using discovery-oriented methods. On page 126, they wrote, “future research would benefit from submitting this extensive array of vocal bursts to even more stringent testing, such as free response, free labeling, and no-story contexts (see LeDoux, 2015).” This statement is misleading, however, because it implies that such studies do not yet exist, when in fact this is exactly what Gendron et al. (2014a) reports (also see it’s conceptual replication with posed faces, Gendron et al. 2014b). Furthermore, “LeDoux, 2015” citation refers to the afterword that LeDoux graciously composed for The Psychological Construction of Emotion, edited by one of us (LFB) and James Russell (Barrett & Russell, 2015). Nowhere in his comments does he suggest that emotion perception research would benefit from using less concept laden methods. There are many other papers that Cordaro et al. could have cited to make this point (e.g., Barrett, 2011: Barrett, Lindquist & Gendron, 2007; Barrett, Mesquita & Gendron, 2011; Fernandez-Dols & Crivelli, 2015; Nelson & Russell, 2013; Russell, 1994; Russell, Bachorowski, & Fernandez-Dols, 2003).

Concern #3: Did Cordaro et al. accurately describe the use of manipulation checks in prior research?

On page 124, Cordaro et al.’s statement that “no comprehension validation check was used” in the Gendron et al. (2014a) research is not, in fact, correct. We agree with the authors that manipulation checks are crucial – and we routinely use them in our studies, including the experiment reported in Gendron et al.³ This apparent misunderstanding is masking an important conceptual issue: there is disagreement over whether certain methods constitute a validation check (as Cordaro et al. seem to suggest) or whether they are a method for teaching emotion concepts that ensure evidence for, rather than test for, the existence of universal emotion perception (as we have suggested; Gendron et al., 2015).

In a recent exchange published in Psychological Science (Sauter et al., 2015 and Gendron et al. 2015), it became clear to us that Sauter et al. (2010) appears to have unwittingly encouraged Western emotion concept and category learning in their experiment. After hearing the story for a given emotion, each Himba participant was required to explain how the target person was feeling, despite the fact that individuals in the Himba cultural group do not make mental state inferences as frequently as more Westernized perceivers (e.g., see findings from Gendron et al., 2014a, b, and also H. C. Barrett et al., 2016; in the anthropology literature, this is called “opacity of mind”, Robbins & Rumsey, 2008). Only once it was clear that a Himba participant understood the intended (Westernized) emotion concept was he or she allowed to proceed with a block of experimental trials for that emotion category. That is, Sauter et al. did not allow their Himba participants to hear any vocalizations for a given story until they demonstrated knowledge of the concept that would allow them to categorize the vocalizations in a manner consistent with the Western cultural expectations that are embedded in the “universal” solution.⁴ In contrast, we only ensured that our Himba participants understood how to perform the emotion perception task, which included their comprehension of the stories (from their own cultural perspective). In our view, if it is necessary to ensure that Himba participants possess Western emotion concepts before they can perceive emotion in a universal way, then emotion perception is not all that universal in the first place.

Summary

We heartily agree with Cordaro et al. that future research on the hypothesis of universal emotion perception requires the use of more theory-neutral experimental methods, along with an acknowledgement that certain aspects of the story matching paradigm (and similar classification paradigms) are not as psychologically inert as is commonly assumed. There continues to be a growing body of evidence to testify that conceptual aids embedded in experimental procedures (i.e., emotional stories of the story matching method, emotion words in standard classification methods, etc.) encourage findings that support the hypothesis of universal emotion perception that are not in evidence when conceptual content is removed (for a review, see Barrett, 2011; Barrett et al., 2011).

Ultimately, a paper’s contribution to scientific progress is limited when ambiguities and oversights creep in and ultimately obscure the real value of its findings. The importance of accurately characterizing the scientific context in which studies are reported is magnified in leaner times when research funding is scarce; no one wants to waste what little money there is rediscovering findings that are already in print but are concealed by inexact reporting.