MaqFACS: A Muscle-Based Facial Movement Coding System for the Rhesus Macaque

Abstract

Over 125 years ago, Charles Darwin suggested that the only way to fully understand the form and function of human facial expression was to make comparisons to other species. Nevertheless, it has been only recently that facial expressions in humans and related primate species have been compared using systematic, anatomically-based techniques. Through this approach, large scale evolutionary and phylogenetic analyses of facial expressions, including their homology, can now be addressed. Here, the development of a muscular-based system for measuring facial movement in rhesus macaques (Macaca mulatta) is described based on the well-known FACS (Facial Action Coding System) and ChimpFACS. These systems describe facial movement according to the action of the underlying facial musculature, which is highly conserved across primates. The coding systems are standardized, so their use is comparable across laboratories and study populations. In the development of MaqFACS, several species differences in the facial movement repertoire of rhesus macaques were observed in comparison to chimpanzees and humans, particularly with regard to brow movements, puckering of the lips, and ear movements. These differences do not appear to be the result of constraints imposed by morphological differences in the facial structure of these three species. It is more likely that they reflect unique specializations in the communicative repertoire of each species.

Standardizing descriptions of facial movement

Hjortso (1970) was the first to utilize a numerical system to document the appearance of facial movements in humans with clear reference to the underlying physiology. Not long after, Ekman and colleagues published the Facial Action Coding System, or FACS, which is also an anatomically-based system that describes facial movement in humans according to contractions of the underlying facial musculature (Ekman & Friesen, 1978; Ekman et al., 2002). Following this, Vick and colleagues (2007) developed a FACS system for the chimpanzee, ChimpFACS, and Dobson (2009) adapted a FACS-methodology to compare facial mobility across a range of primate species. In FACS, each muscular-based facial movement is identified using a numerical code, referred to as an Action Unit, or AU, and the majority of these movements have been verified in humans using both surface (Duchenne, 1872) and intramuscular electrical stimulation (Waller et al., 2006). In this way, FACS is able to describe the range of observable movements possible in the face. Although FACS is ultimately concerned with facial expressions and emotion, it is, first and foremost, a facial movement coding system. Using FACS, researchers are able to identify the individual component movements that comprise facial expressions in a bottom-up approach. In total, FACS describes 58 component movements in the human face including 33 AUs for which the muscular basis is specified, and an additional 25 action descriptors (ADs) where the movements are more general, e.g., head movements, eye movements. Each AU is given a number and descriptive name, e.g., AU4=brow lowerer, and then describes the basic appearance changes that are commonly observed in the majority of people when this movement (AU) is made. These basic appearance changes are used as minimal criteria for identifying and coding the presence of an AU. Thus, FACS is able to compare facial expressions across individuals regardless of the inherent variability in the surface morphology of faces, e.g., bone structure, fatty deposits and skin texture. The strength of FACS, however, lies in its standardization: researchers must learn the system and pass a test for certification and, as a result, FACS has become the gold standard for studies of facial movement in humans with applications ranging from basic emotion assessment to clinical diagnosis (Ekman & Rosenberg, 1997).

Modifying FACS for comparative primate research

In order to apply a FACS-based system for use in nonhuman species, one would first need to demonstrate that the basic facial musculature is comparable. Despite some early reports that humans have a more complex and well-differentiated facial musculature compared to “lower” nonhuman primates (Huber, 1931), recent studies using modern dissection techniques have confirmed strong phylogenetic continuity in the facial musculature across primate groups (Figure 1, Burrows & Smith, 2003; Burrows et al., 2006; Burrows et al., 2009). Therefore, a logical extension of the FACS is that it be adapted for use in nonhuman primates in order to make direct comparisons of facial movement, specifically facial expressions, across related species (Preuschoft & van Hooff, 1995). FACS is premised on the principle that the actions of similar facial muscles leads to similar facial movements that can be compared between individuals even if there are significant differences in facial morphology. These issues were considered during the first modification of FACS for use with another species, the chimpanzee, Pan troglodytes (ChimpFACS, Vick et al., 2007) as well as for its modification for use in rhesus monkeys, described here.

Figure 1.

Lateral views of right side of a macaque (top), chimpanzee (left), and human (right) illustrating similarities and differences in the presence and form of the mimetic facial muscles. Non-colored muscles are common to each species, e.g., platysma, OOM. Muscles colored red are present in chimpanzees and humans but not rhesus macaques. Muscles colored in blue are present in each species, but are far greater in size in rhesus macaques. The zygomaticus major muscle in the rhesus macaque splits around the DAO, completely enveloping the superior attachment, unlike the condition in chimpanzees and humans where the ZM is more independent of the DAO. Abbreviations: AA – anterior auricularis muscle; DAO – depressor anguli oris muscle; DS – depressor septi muscle; R – risorius muscle; SA – superior auricularis muscle; ZM – zygomaticus major muscle. A color version of this figure is available online.

The MaqFACS: Rhesus Macaque Facial Action Coding System

FACS was modified for use with rhesus macaques following a three-step methodology(anatomical, physiological, behavioral), similar to that used for the creation of FACS (Ekman & Friesen, 1978) and ChimpFACS (Vick et al., 2007 www.chimpfacs.com). First, facial muscle dissections were performed on six deceased rhesus macaques in order to document the presence, variability, and general morphology of each muscle in comparison to humans and chimpanzees (Burrows et al., 2009; Burrows et al., 2006). The presence and inter-species variability in the facial muscles are noted in Table 1. Second, based on the information obtained during the dissections, an effort was made to identify how each muscle functioned in changing the surface appearance of the face by performing intramuscular stimulations of the facial muscles (Waller et al., 2008). These appearance changes were described with reference to the basic morphological features/landmarks of the face (see Figure 2). Finally, the spontaneous occurrence of each facial movement in rhesus monkeys was identified from video footage of naturally occurring behavior. These movements were then assigned the appropriate Action Unit numerical code, corresponding to the same muscle movement-based codes used in FACS and ChimpFACS. It should be noted that several AUs were unable to be identified, despite the corresponding muscle being present on dissection (see Table 1, e.g., AU13, AU15). Reasons for this could be that those movements are simply not present in the rhesus macaque, e.g., the muscular contribution did not lead to an identifiable appearance change, or that they simply occur with low frequency or in conjunction with other movements that make independent classification difficult. For example, independent movement of several muscles could not be identified although it could be determined that they were active in collaboration with other movements, e.g., AU17 + AU18i, AU6 + AU27.

Table 1.

Comparison of facial movements (Action Units) in humans, chimpanzees and macaques.

Action Unit	Action Descriptor Name	Muscle (see Fig 1)	Human	Chimpanzee	Macaque	AU In MaqFACS?
AU1	inner brow raiser	medial frontalis	Y*	NO*	NO*	no
AU2	outer brow raiser	lateral frontalis	Y*	NO*	NO*	no
AU1+2	brow raiser	frontalis	Y*	Y*	Y*	yes
AU4	brow lowerer	1CS, 2DS, 3Proc	Y*	NO*3	NO*1,3	no
AU41	glabella lowerer	procerus	Y*	NO*	Y*	yes
AU5	upper lid raiser	orb. oculi	Y^	NO^	NO^	no
AU6	cheek raiser	orb. oculi (orbital)	Y*	Y*	Y*	yes
AU7	lid tightener	orb. oculi (palpebral)	Y^	Y^	Y^	no
AU8	lips towards each other	orb. oris	Y^	NO^	Y^	yes
AU9	nose wrinkle	llsan	Y*	Y*	Y*	yes
AU10	upper lip raiser	lev. labii sup	Y^	Y*	Y*	yes
AU11	nasiolabial furrow deepener	zyg. minor	Y	NO	NO^	no
AU12	lip corner puller	zyg. major	Y*	Y*	Y*	yes
AU13	cheek puffer	caninus4	Y*	NO	NO	no
AU14	dimpler	buccinator	Y	NO^	NO^	no
AU15	lip corner depressor	dep. anguli oris6	Y*	NO*	NO*	no
AU16	lower lip depressor	depressorlabii inf	Y*	Y*	Y*	yes
AU17	chin raiser	mentalis	Y*	Y*	Y*	yes
AU18	lip pucker	orb. oris	Y^	NO	Y*	yes5
AU20	lip stretcher	risorius	Y*	NO	NA	no
AU22	lip funneler	orb. oris	Y^	Y*	NO	no
AU25	lips parted	various	Y^	Y^	Y^	yes
AU26	jaw drop	various	Y^	Y^	Y^	yes
AU27	mouth stretch	various	Y^	Y^	Y^	yes
AU28	lip suck	orb. oris	Y^	Y^	NO^	no
EAU1	ears forward	ant. auricularis	NO^	NO^	Y*	yes
EAU2	ear elevator	sup. auricularis	NO^	NO^	Y*	yes
EAU3	ear flattener	post. auricularis	NO^	NO^	Y*	yes

⁴The caninus is also referred to as the levator anguli oris in humans

⁵In macaques, we have divided AU18 into two separate AU codes, AU18i-true pucker and AU18ii-outer pucker (see text)

⁶Also referred to as triangularis

Observed movements are noted with ‘Y’ and those not observed are noted with ‘NO.’ Independent stimulation of these movements is marked by an asterisk (*) for humans and chimpanzees (Waller et al., 2006) and macaques (Waller et al., 2008). If no stimulation was attempted, these are marked by a carrot (^{^}). No * or ^{^} listed means that stimulation was attempted but was not successful. If the muscle was not present on dissection, ‘NA’ is noted.

Figure 2.

An illustration of key facial landmarks for the rhesus macaque. A color version of this figure is available online.

What follows is a brief description of the specific facial appearance changes associated with the majority of AUs identified in the rhesus monkey, with direct comparison to humans and chimpanzees. A complete description of each AU, including video and still frame photographic examples can be found at the manual website, and readers are referred here (http://userwww.service.emory.edu/~lparr/index.html).

Movements of the upper face

AU 1+2 (inner and outer brow raiser)

FACS describes independent movements of the inner and outer portion of the brow; however, clear independent movements were not observed in either the chimpanzee or the rhesus monkey. Thus, for both primate species, we have combined the action units AU1 and AU2 into a single combined movement AU1+2 describing the raising and lifting of the browline (see Figure 2). This movement reveals greater surface area in the underbrow region, which can be lighter in color in the rhesus monkey than chimpanzee or humans, so visibility of the underbrow is a particularly salient appearance change for identifying this movement in MaqFACS. Moreover, rhesus monkeys do not have as pronounced a brow ridge as the chimpanzee. Thus, when AU1+2 is extreme in the rhesus monkey, the brow region may appear to flatten in the vertical plane, smoothing any wrinkles along the vertical nose ridge. Also, depending on the curvature of the brow in the monkey’s neutral state, AU1+2 can function to curve the brow into a smooth arc. A final appearance change of AU1+2 is that it can create a bulging of the hair superior to the brow region.

AU41 (glabella lowerer)

One of the most conspicuous movements in the FACS is AU4, the brow lowerer (used primarily in human frowning/anger). This is achieved by the contraction of three muscles, the corrugator supercilii, depressor supercilii and procerus, which results in both a lowering and medial contraction, e.g., knitting, of the brow. FACS also reports individual AUs for the contraction of each muscle independently, AU41 (procerus), AU42 (depressor supercilii) and AU44 (corrugator), although it is very rare that these can be differentiated. Although clear brow lowering movements were observed in both chimpanzees (Vick et al., 2007) and macaques, in neither species did we observe the medial contraction, or knitting, characteristic of AU4 in humans. In ChimpFACS, the appearance changes of AU4 are noted with the exception of the medial contraction. In MaqFACS, however, we report that the brow lowering movement appears to consist mostly of a medial bulging in the glabellar region due to the action of the procerus. Because of this, brow lowering in MaqFACS is specifically identified as AU41 (glabella lowerer). This movement pulls the brow downward, reducing the visibility of the underbrow and changes the curvature of the brow such that it becomes lowered at the midpoint.

Movements of the lower face

AU9 (nose wrinkle), AU10 (upper lip raiser)

In FACS, the movements of AU9 and AU10 can be tightly coupled. Extreme nose wrinkling, for example, functions to raise the upper lip slightly, and many expressions in humans and chimpanzees combine AU9+AU10, as in disgust. In both the chimpanzee and the rhesus monkey, the AU9, by itself, can be difficult to detect. However, in both species, a combined AU9+AU10 has been observed, in addition to the independent action of AU10. Therefore, in describing these two movements, ChimpFACS and MaqFACS have attempted to describe the appearance changes of each, although in most cases AU9 would be reported in combination with an AU10. In addition to the action of AU10, which pulls the upper lip upward in a smooth arc causing wrinkles and furrows in the infra-orbital triangle, the AU9 in MaqFACS functions to pull the nose upward, causing oblique nose wrinkles to deepen. The action of AU9 alone pulls the lateral aspect of the nostril wings upward and medially towards the root of the nose which causes the nasal groove to deepen.

AU12 (lip corner puller)

AU12 represents one of the most robust movements seen in all three species studied using the FACS systems. The function of AU12 is to pull the lip corners back and slightly upward in a movement that produces the homologous expressions of human smiling and the chimpanzee and macaque bared-teeth displays (Preuschoft & van Hooff, 1997; Parr et al., 2007). Among chimpanzees, AU12 also retracts the lips, but caution should be exercised when coding the AU12 as often the mouth corners of the chimpanzee appear slightly elevated, producing the appearance of an AU12 in the neutral state. The mouth shape of the rhesus macaque is more horizontal than the chimpanzee in the neutral state so this is less of a problem when coding AU12 in the rhesus macaques. AU12 functions to retract the lips laterally and upwards towards the ears. It narrows and slightly bulges the upper lip, reducing the visibility of the vertical lip ridge and deepening the furrows at the mouth corners. AU12 also creates oblique wrinkles and deepens the furrows of the infraorbital triangle, one of its most prominent appearance changes.

AU16 (lower lip depressor)

The appearance changes associated with AU16 are common to humans, chimpanzees, and rhesus macaques, despite considerable differences in the morphology of the lips, e.g., thick red everted lips in humans, prognathic mobile lips in the chimpanzee, and thin lips (appearing to invert) in macaques. Appearance changes associated with AU16 include lowering the bottom lip to expose the teeth and lower gum. In macaques, AU16 also causes a slight eversion of the lower lip which may cause the inner portion of the lip to appear to thicken slightly. AU16 can also increase the curvature of the lower lip by pulling the medial aspect downwards towards the chin, in contrast to the resting shape of the mouth, in which the medial portion of the lower lip can appear to turn upward.

AU18i/AU18ii (true pucker and outer pucker)

In FACS, there are two main movements responsible for protruding the lips, AU18- the lip pucker, e.g. kiss, and AU22-the lip funneler, which pushes the lips outward as if saying the word “flew.” In chimpanzees, the action of lip protrusion was attributed to the AU22 and the ChimpFACS does not contain the AU18, lip pucker (see Table 1). In the rhesus macaque, however, lip puckering was attributed to two movements. The first is a pucker similar to AU18 in humans which, in MaqFACS, is referred to as AU18i, the true pucker (see Figure 3). The AU18i purses the lips medially forward towards each other, narrowing the mouth corners medially, protruding the lips and reducing the mouth aperture in both the horizontal and vertical directions. This movement causes distinct oblique wrinkles to appear extending from the cheek along the length of the upper lip. Because the rhesus macaque does not have everted lips, this movement causes the medial portion of the lip to take a scalloped appearance on either side of the midline as the lips are pursed forward. AU18i causes the philtal region to deepen and produces a depression in the medial portion of the lower lip causing it to appear slightly curved.

Figure 3.

An illustration of a neutral face and two pucker faces, AU18i, the true pucker, and AU18ii, the outer pucker. A color version of this figure is available online.

The second lip protrusion movement observed in the rhesus macaque contained distinct appearance changes from AU18i, but was also insufficient to be labeled AU22. This movement is instead described as AU18ii, the outer pucker, in which the lips are pushed forward so as to protrude slightly, causing oblique wrinkles to extend from the cheek along the upper lip (see Figure 3). What distinguishes this movement from AU18i is that the furrow between the nose and upper lip (philtral region) is reduced and the lips cinch together at a point distal to the midline causing them to part and appear inflated. We speculate that the AU18i is produced by the action of orbicularis oris, incisivii labii superioris and inferioris while the movement of AU18ii is produced by contraction of the incisivii portions specifically. In addition to these, MaqFACS includes an action descriptor, AD181 (lip smacking), to denote tightening of the lips together followed by a rapid opening and parting motion, which is a common facial movement associated with AU18i and the lip-smacking expression of the rhesus macaque.

Ear movements

EAU1, EAU2, EAU3, EAD

Neither FACS, nor ChimpFACS, contains descriptions of ear movements as humans and chimpanzees have lost the independent control of ear musculature common to many other mammals. Among macaques, three prominent and independent ear movements are described. EAU1 (ears forward) functions to push the ears forward towards the face, increasing the visibility of the ear if viewed from a frontal orientation, but reducing the visibility of the ear if viewed in profile. EAU2 (ear elevator) pulls the ears superiorly towards the top of the head, and EAU3 (ear flattener) pulls the ears towards the back of the head, flattening them against the skull. This may reduce the visibility of the ears if viewed from a frontal orientation, but increase the visibility of the ears if viewed in profile. It should be noted that coding specific EAUs can be very difficult if the neutral position of the ears is unknown, e.g., an EAU1 may actually be the release of an EAU3. Moreover, fighting among macaques often injures the ears, and these injuries can reduce the visibility of the pinnae which are required to denote the appearance changes described above. Thus, in the MaqFACS, it is recommended that users code an EAD, Ear Action Descriptor, to denote movement of the ears without specifying its muscular basis unless there is clear sufficient evidence about the neutral position of the ears to justify a specific EAU code.

In summary, adapting the FACS systems for use in several species of nonhuman primates has led to the identification of both similarities and differences in the morphology and movement of the primate face (Dobson, 2009; Vick et al., 2007). The basic facial muscle plan among primates is far more conserved than has been previously reported (Huber, 1931; Burrows & Smith, 2003; Burrows et al., 2009), leading to remarkable similarity in the facial expression repertoires across related species (Burrows, 2008). Several important species differences, or specializations, have also been revealed. Strikingly, brow movements are more variable and under greater independent control in humans than macaques or chimpanzees, and these movements have important signaling functions when used during social interactions (Ekman, 1977). Future studies should examine the functional significance of these reported differences in brow movements across species. Several distinct ear movements appear to be under independent muscular control in macaques, but this independence has been lost in humans and chimpanzees. Ear movements are a prominent feature of many rhesus monkey facial expressions including the lipsmack and bared-teeth displays and, therefore, are presumed to play an important role in social communication (van Hooff, 1962, 1967; Partan, 2002).