Continuous automated analysis of facial dynamics of brachycephalic and normocephalic dogs in different contexts

Abstract

This study develops a novel automated method for measuring continuous dynamics of dog facial behavior based on video-based tracking of 46 facial landmarks grounded in the Dog Facial Action Coding System. This method is applied for comparing the facial behavior of ($n_1=7$) brachycephalic (Boston Terrier) and ($n_2=7$) normocephalic (Jack Russel Terrier) dogs in four different contexts, eliciting various inner states: positive (play and called by name) and negative (separation and stranger). Having objectively quantified facial dynamics in brachycephalic and normocephalic dogs, we have found that brachycephalic dogs exhibited consistently lower facial dynamics across all four tested contexts and facial regions (eyes, mouth, and ears) compared to normocephalic dogs. They further demonstrated relatively higher dynamics in positive play and negative stranger conditions than in the other two conditions. In contrast, normocephalic dogs showed elevated dynamics exclusively in the positive play condition, with significantly reduced dynamics in the negative stranger condition. These findings highlight distinct patterns of facial expressivity between the two morphological groups, suggesting decreased facial expression in brachycephalic dogs and demonstrating our method’s value in providing novel insights into canine communication.

Introduction

Emotions are complex states that involve physiological, cognitive, and behavioral components [1]. Today, it is widely recognized that facial expressions provide significant visual cues for identifying emotional states. In humans, facial expressions are a primary form of nonverbal communication that helps regulate social interactions [2]. The link between facial expressions and emotions has been extensively established in human psychology [3, 4].

Not only humans but also non-human animals, especially mammalian species, produce facial expressions, which are believed to convey information about their emotional states [5, 6]. Therefore, there is an increasing interest in facial expressions as potential indicators of emotions and welfare. For example, assessing pain can be difficult in species that otherwise give no or very weak behavioral signs of painful states. However, based on a validated facial assessment of pain-related expressions, veterinary science can now rely on objective methods, among others, in horses [7] and cats [8].

Domestic dogs are of growing popularity in science, being used as valuable clinical models for numerous human disorders [9], research on domestication [10], attachment [11, 12], human-animal interactions [13, 14] and more. However, the scientific understanding of canine visual signaling is still somewhat limited. While animal facial expressions were once considered purely involuntary, it has been shown recently that dogs actively use facial expressions, particularly when interacting with an attentive human audience [15], highlighting the role of the human partner in canine communication. Some studies have focused on dog-human relationships, addressing both the perception by dogs of human emotions [16] and dog emotions as perceived by humans [17, 18]. The remarkable diversity of dog facial phenotypes due to the selective breeding processes [19] leads to fascinating questions on the impact of dog facial features on dogs’ ability to produce visual cues in their communication with humans and humans’ ability to understand them [20].

Sexton et al. [21] investigated dogs’ facial complexity and behaviors and found that dogs with less complex facial coloring (fewer markings) were objectively scored as more behaviorally expressive than those with colored markings on their faces. A recent study by Eretová et al. [22] investigated the effect of brachycephaly on humans’ ability to interpret visual signals in dogs exposed to different situations. Participants more often mistakenly attributed positive emotions in negative situations to brachycephalic dogs (Boston Terrier) than to normocephalic dogs (Jack Russel Terrier). The study provided more evidence that brachycephalic dogs may have a reduced capacity to signal emotions through facial expressions, as has been previously suggested [23]. This is due to the facial musculature, particularly the m. zygomaticus (which pulls back the corner of the mouth) are malpositioned in brachycephalic dogs, while other muscles (like m. nasolabialis) are folded and malfunctioning [24]. The decreased facial activity seems to negatively affect human understanding of canine facial expression and thus can potentially negatively affect the animal’s welfare. However, the above-mentioned studies on canine signals’ interpretation may indicate that changes in facial coloration or skull shape also may distort a human’s ability to perceive dogs’ facial expressions.

The most commonly used method for objectively assessing changes in dog facial expressions is an adaptation from the Facial Action Coding System (FACS), widely used in human emotion studies [25]. Recently, it has been extended to many species, including dogs [26]. Indeed, DogFACS has been applied in several studies [21, 26–29] to measure facial changes in dogs objectively. However, this method has some serious limitations. First, it depends on laborious manual annotation, which also requires extensive human training and certification and may still be prone to at least some level of human error or bias [30]. Some first steps to automating facial movements (action units) detection in dogs were taken by Boneh-Shitrit et al. [31], but much more data is needed for substantial progress in this direction. Another limitation of the DogFACS coding method is its discrete nature. Indeed, the method captures discrete facial action units and descriptors based on the underlying facial musculature. However, they may not necessarily capture dynamics that are not directly induced by muscle movement (e.g., muscle tightening).

Approaches based on facial landmarks may offer an alternative for facial analysis that can capture a continuous stream of information from animals’ facial visual cues. An important advantage of landmark-based approaches is their ease of application to video data, producing time series of multiple landmark coordinates. Such uses of landmark time-series data are just beginning to be explored in the domain of animal behavior, primarily for body landmarks (see, e.g., Labuguen et al. [32] and Wiltshire et al. [33]), while facial analysis remains largely underexplored.

Martvel et al. recently addressed this gap for cat facial analysis by introducing a dataset and the Ensemble Landmark Detector (ELD) model for 48 anatomy-based cat facial landmarks [34]. The detector performed well in tasks requiring subtle facial analysis, such as breed, cephalic type, and pain recognition [35, 36], as well as classification of social interactions [37]. Martvel, Abele, et al. [38] extended the ELD model to dogs using a landmark scheme grounded in DogFACS and creating the Dog Facial Landmarks in the Wild (DogFLW) dataset. Figure 1 shows an example of a landmark scheme containing 46 facial landmarks from the DogFLW.

Fig. 1.

Forty-six dog facial landmarks based on DogFACS from Martvel, Abele, et al. [38]

This paper applies the ELD model to develop a novel approach for quantifying continuous facial behavior changes in dogs. We do so by introducing a rigorously defined mathematical metric for analyzing the dynamics of facial visual cues, taking into account multiple scales and camera distances, and separating relative facial movement from the absolute movement of the dog in the video. This approach utilizes a continuous stream of information obtained from automated tracking of dog facial landmarks and converts the relative movement into facial dynamics. Notably, the method can be used with any automated landmark tracking approach and generalized to other species. Another benefit of the approach is that it supports a separate analysis of dynamics associated with various facial parts (e.g., eyes, muzzle, ears, etc.) and is resistant to noise (occlusions, camera movement, etc.).

To demonstrate the usefulness of the proposed novel measurement tool, we use it on the dataset previously collected by Eretová et al. [22], in which video footage was collected of brachycephalic dogs (Boston Terriers) and normocephalic dogs (Jack Russell Terriers) in four different original contexts: play, separation from the owner (separation), called by the name (called) and threatened by a stranger (stranger). These contexts were induced to elicit visual cues that could be related to different emotional states. We use the proposed metrics to study whether there are differences in facial dynamics between the Boston Terrier and Jack Russell Terrier individuals across contexts and within the contexts.

Methods

Dataset

We filtered all videos from the original study [22] to balance the number of dogs of brachycephalic and normocephalic types. The final dataset comprised 230 videos in 50 fps with a length of $8.55\pm 2.56$ seconds, containing 14 dogs: 7 brachycephalic (Boston Terrier; 4 females, 3 males; mean age: $2.71\pm 1.88$ years; 97 videos, $14\pm 6$ videos per dog) and 7 normocephalic (Jack Russel Terrier; 5 females, 2 males; mean age: $2.64\pm 0.64$ years; 133 videos, $19\pm 3$ videos per dog). For a full breakdown of all dogs used in the study, please see Table 8 in the Supplementary Materials. We discarded videos with the stranger condition for two normocephalic and one brachycephalic dog and videos with the separation condition for one brachycephalic dog due to either dog’s hyperreaction or absence of automatically detected landmarks. Below, we briefly reproduce the experimental protocol from Eretova et al. for completeness. Further details can be found in the original study.

Conditions

Two Experimenters conducted the experiment in a laboratory environment at the Department of Ethology, ELTE, Budapest, Hungary. The recorded situations followed 5 minutes of free exploration, during which owners were present but instructed not to engage with the dogs. The experiment consisted of eight phases (four situations and four interludes between the situations), each lasting 1 minute. Before the first situation started, the dog was put on a leash that was attached to the handle in front of the chair on which the owner was sitting. Each used condition has been validated to non-invasively induce positive (Conditions 1 and 2) and negative (Conditions 3 and 4) affective states in dogs for a brief period of time. We did not inspect the dogs’ true affective states via other means (for detailed methodology, see Eretová et al. [22]).

Condition 1 — Called.

Experimenter 1 called the dog by its name while operating a tripod camera. Experimenter 2 filmed the dog’s reactions with a hand-held camera from an upper-front angle. The owner was sitting in a chair behind the dog, not engaging with it in any way. Interlude 1 followed, in which the dog was released from the leash and played with a tennis ball with Experimenter 1. For the validity of the situation, see [39, 40].

Condition 2 — Play.

The dog, put on the leash loosely, was teased with the tennis ball by Experimenter 1 without receiving the ball. The reactions were recorded by a tripod camera (operated by Experimenter 1) and a hand-held camera (operated by Experimenter 2) from different angles. The owner was sitting in a chair behind the dog, not engaging with it in any way. Interlude 2 followed, in which the owner and Experimenter 2 left the room, leaving Experimenter 1 operating the handheld camera. The tripod camera was left unattended, filming the dog continuously. For the validity of the situation, see [41, 42].

Condition 3 — Separation.

The owner left the room, leaving the dog leashed to a floor handle with Experimenter 1 present. Experimenter 1 did not interact with the dog and maintained a minimum distance of 2 meters while operating the handheld camera. The dog’s behavior was filmed by the tripod camera freely and unoperated. Interude 3 followed, in which the owner returned to the room, standing behind the dog, not engaging with it in any manner. Experimenter 2 returned to the room after that and closed the door behind herself. For the validity of the situation, see [43, 44].

Condition 4 — Stranger.

Experimenter 2 approached the dog slowly in a threatening manner (hunched posture, arms clasped behind her back, sliding her feet slowly on the floor and maintaining direct eye contact with the dog). The approach continued until Experimenter 2 was 1.5 meters away from the dog. The dog’s reactions were filmed from a front-side view by Experimenter 1 using the hand-held camera, keeping Experimenter 2 out of the camera’s field. The tripod camera was filming the dog freely and unoperated. At a distance of 1.5 metres, Experimenter 2 broke the threatening posture and direct eye contact, called the dog by its name, greeted it and petted it. Interlude 4 followed, in which the dog was let to play with the tennis ball freely with the owner and both Experimenters to release any residual tension. When the dog was sufficiently relaxed, the owner left, and the experiment ended. For the validity of the situation, see [45–47].

For the purposes of the study, the raw videos were edited — the soundtracks were removed, and the timestamps of each filmed situation were cut into separate videos, which were coded as per situation and individual dog identity. A team of experts in canine behavior affiliated with the Department of Ethology and Companion Animal Science at CZU Prague and the Department of Ethology at ELTE reviewed the resulting videos for clarity of signals (such as body posture, facial expressions, ear and tail movement, eye contact, pacing or standing still, etc.) and approved them for use in the original study [22]. For the purpose of this study, videos were only cut to isolate individual contexts (conditions) while maintaining each dog’s unedited expressions.

Figure 2 shows dog faces cut from the original videos in four conditions.

Fig. 2.

Boston Terrier and Jack Russel Terrier dogs in different conditions. Facial images are cropped from original video frames

Landmark detection

We processed all videos from the dataset with the Ensemble Landmark Detector (ELD) [34], trained on the DogFLW dataset [38]. The output time series contained the frame’s timestamp, coordinates of 46 facial landmarks, and the detection model’s confidence, representing the quality of detected landmarks (from 0 to 1). We limited the fps in the obtained time series to 25, discarding each second frame to avoid too similar data points (since the dog’s movement in 0.02 seconds is neglectable in most cases).

Dynamic metrics

For each processed frame, we obtained 46 facial landmarks. Since the dogs in the videos are moving and we’re interested in subtle facial movements, we divided the movement into two components: general and relative. General movement is the movement of the dog as a whole or the movement of a camera. Most of the landmarks move by approximately the same trajectory during this movement. Relative movement is a movement of facial parts separate from the general movement. During this movement, specific landmarks have a distinct trajectory, different in direction from the general one. To capture this relative movement, we defined stable landmarks as landmarks less prone to relative movements, such as eye corners, ear bases, and nose landmarks, as shown in Fig. 3. In other words, we selected landmarks that generally don’t have a relative component in their movement to be able then to subtract this general movement from other landmarks’ movement.

Fig. 3.

Facial landmarks grouped in facial regions and a center (centroid) of selected stable landmarks

For each landmark set, we calculated the position of the center (centroid) of stable landmarks by the formula:

$$\begin{array}{c}\hfill {\overline{r}}_c(x,y,t)=\frac{1}{n}\sum _{i=1}^n{\overline{r}}_i(x,y,t),\end{array}$$

where ${\overline{r}}_c(x,y,t)$ is a center coordinate vector, ${\overline{r}}_i(x,y,t)$ is a coordinate vector of an i-th landmark from the set of $n=8$ stable landmarks. We imply the centroid coordinates to represent the general movement of the dog’s head.

Using the center coordinates ${\overline{r}}_c$, we translated all $k=46$ landmarks’ coordinates into the center’s coordinate system:

$$\begin{array}{c}\hfill {\overline{r}}_{i,rel}={\overline{r}}_i-{\overline{r}}_c,i\in (1,k),\end{array}$$

where ${\overline{r}}_{i,rel}$ — relative coordinates of an i-th landmark in the center’s coordinate system, ${\overline{r}}_i$ — absolute coordinates of an i-th landmark in the frame’s coordinate system. Further, we refer to obtained relative coordinates simply as coordinates unless stated otherwise.

Obtained relative coordinates represent separate facial movements (blinks, ear movements, yawns, etc.), but this representation is limited in its accuracy. Since the detected landmarks have only two dimensions (x and y in a frame’s plain), some movements, such as head rotation, can’t be described with two dimensions without distortion and proportion changes in landmark coordinates. Nevertheless, we negated a part of the general movement “leak” into the relative part by selecting stable landmarks and transitioning into their centroid’s coordinate system.

Having coordinates for the whole video, we computed the dynamics of landmark movement. Since, for different movements, different landmarks could have various trajectories and velocities, we averaged the amount of movement per landmark. For that, we calculated the absolute difference in coordinates between two consecutive frames:

$$\begin{array}{c}\hfill d_i(t_j)=|x_i(t_j)-x_i(t_{j-1})|+|(y_i(t_j)-y_i(t_{j-1})|,\end{array}$$

where $d_i(t_j)$ is a dynamic of an i-th landmark between a j-th and a j-1-th frames, $x_i(t_j)$ and $y_i(t_j)$ — coordinates of an i-th landmark on a jth frame. Taking the average on all $k=46$ landmarks, we obtained the overall dynamic per landmark $d(t_j)$:

$$\begin{array}{c}\hfill d(t_j)=\frac{1}{k}\sum _{i=1}^k{d}_i(t_j).\end{array}$$

Intuitively, the dynamic metric introduced above reflects the total relative movement of all facial parts between two frames, averaged per landmark. In this sense, the higher the metric is in a frame, the more facial movement is detected on the dog’s face, captured by the movement of the facial landmarks.

To distinguish between the movement types, we additionally divided all landmarks into morphological regions: left and right ears, nose, and left and right eyes (from the dog’s perspective). Computing landmark dynamics in a specific region allowed us to track potential lateralization and focus on different facial parts. The dynamic in the region M containing m landmarks is defined as follows:

$$\begin{array}{c}\hfill d_M(t_j)=\frac{1}{m}\sum _{i=1}^m{d}_i(t_j),\end{array}$$

where an i-th landmark belongs to the region M, as shown in Fig. 3.

The videos in the dataset have frames where landmarks are not detected or detected with a low model’s confidence. Usually, the dog in such frames has its head heavily rotated, and its face isn’t partially or at all visible. The landmark dynamic is not defined in those cases since the landmarks are absent or poorly detected. During the preprocessing, we noticed that such time intervals are usually “surrounded” by landmarks detected with low confidence. It happens due to the extreme head rotation, where the model can still detect the dog’s face but cannot detect landmarks with high enough certainty. As a result, the dynamic metric may behave randomly and contain misleading values. To mitigate that, we introduced masking padding around the missing landmarks by computing the number of consecutive frames where the model’s confidence is below a certain threshold (the value of 0.6 was chosen empirically) and masking a sequence of frames of length equal to $\frac{1}{4}$ of this number (minimum of 1 frame) before and after the missing landmarks. By this, we discarded potential bias related to the limitations of the landmark-based approach. Figure 4 shows the masked d(t) metric graph from a random video from the dataset and the model’s confidence level (scaled).

Fig. 4.

Facial dynamics metric d(t) in a random video from the dataset (blue) masked by the landmark detector’s confidence level (scaled, red)

After aggregating dynamics for a video v with l frames, we compute the average dynamic value per video across all frames. To negate the differences between the scale of the face across different frames, we normalize the dynamic metric by dividing $d(t_j)$ by the distance between outer eye corners (inter-ocular distance, IOD) in each frame. The total dynamic D is defined as:

$$\begin{array}{c}\hfill D(v)=\frac{1}{l}\sum _{j=1}^l\frac{d(t_j)}{IO{D}_j}.\end{array}$$

Statistical analysis

Due to the hierarchical nature of the dataset (videos are grouped by dogs, each dog having several trials), we opted for using the mixed-effects ANOVA model with the dog as a random factor to compare the dynamic metrics D (looking at the face holistically) and $D_M$ for each facial region M across cephalic types (normocephalic vs. brachycephalic) and conditions (called, play, separation, stranger). The statistical analyses were performed using the statsmodels python library [48].

Results

Novel landmark-based approach for measuring continuous facial dynamics

Our landmark-based approach supports the quantitative measuring of subtle behavioral facial changes. It has the following novel features:

• The metric uses a continuous stream of visual information (as opposed to discrete events captured by the DogFACS system).

• It supports both a holistic analysis (whole face) and a separate analysis based on different facial parts.

• It is practical in the sense that it can handle noisy data (frames with insufficiently detected landmarks are automatically discarded).

Figure 5 demonstrates the pipeline suggested in this study.

Fig. 5.

Pipeline overview. The approach takes video as an input, which is then passed to the ELD model, following pre-processing steps. The dynamics metrics are then calculated based on the time series, representing values that quantify the degree of facial movement in the video

Facial dynamics comparison across cephalic types and conditions

We denoted the set of videos featuring normocephalic dogs by JRT and the set of videos featuring brachycephalic dogs by BT. For condition c (called, play, separation, or stranger), we denoted the subset of JRT/BT in condition c by JRT(c)/BT(c), respectively. The average dynamic value per video is denoted as D, and such dynamic for a separate facial region is denoted as $D_{\mathit{Region}}$.

Comparing across breeds (brachycephalic vs. normocephalic)

Table 1 shows that overall dynamics and dynamic metrics for all facial regions were significantly higher for JRT than BT.

Table 1.

Mean dynamics values D for different facial regions aggregated over Boston Terriers (BT) and Jack Russel Terriers (JRT) in all experiments. The intercept is JRT

	D	$D_{lefteye}$	$D_{righteye}$	$D_{nose\&mouth}$	$D_{leftear}$	$D_{rightear}$
JRT	3.55	2.29	2.38	3.16	4.96	4.9
BT	1.79	1.77	1.67	1.65	1.99	2.02
p	$<0.001$	0.002	0.003	$<0.001$	$<0.001$	$<0.001$

Across-context comparisons within the cephalic type

On average, the dynamics $D^{\mathit{BT}}$ did not differ significantly across conditions $(df=3,F=2.6,p=0.058)$. All regional dynamics, except for the nose and mouth region, also did not differ significantly overall $(D_{lefteye}^{\mathit{BT}}:df=3,F=1.99,p=0.121;D_{righteye}^{\mathit{BT}}:df=3,F=2.37,p=0.077;D_{nose\&mouth}^{\mathit{BT}}:df=3,F=4.13,p=0.009;D_{leftear}^{\mathit{BT}}:df=3,F=1.19,p=0.32;D_{rightear}^{\mathit{BT}}:df=3,F=2.27,p=0.087)$.

However, we observed significant differences when comparing dynamics between conditions, as shown in Table 2. For BT, the stranger and play conditions had higher dynamics than called and separation. The called condition had a significantly lower overall dynamic and dynamics in all facial regions, except for ears, than the stranger condition. In the separation condition, we observed significantly lower overall dynamic and dynamic for the muzzle area, as well as right-side asymmetry, with $D_{righteye}^{\mathit{BT}}(sep)$ and $D_{rightear}^{\mathit{BT}}(sep)$ being significantly lower than $D_{righteye}^{\mathit{BT}}(str)$ and $D_{rightear}^{\mathit{BT}}(str)$. The play condition did not differ significantly in dynamics compared to the stranger condition.

Table 2.

Mean dynamics values D for different facial regions in Boston Terriers (BT) videos subset in different conditions. The intercept is a stranger condition. Significantly different dynamic values are highlighted in bold

	$D^{\mathit{BT}}$	$D_{lefteye}^{\mathit{BT}}$	$D_{righteye}^{\mathit{BT}}$	$D_{nose\&mouth}^{\mathit{BT}}$	$D_{leftear}^{\mathit{BT}}$	$D_{rightear}^{\mathit{BT}}$
Stranger	2.29	1.97	2.13	2.1	2.54	2.57
	$\mathbf{(}\mathit{p}\mathbf{<}\mathbf{0.001}\mathbf{)}$	$\mathbf{(}\mathit{p}\mathbf{=}\mathbf{0.001}\mathbf{)}$	$\mathbf{(}\mathit{p}\mathbf{<}\mathbf{0.001}\mathbf{)}$	$\mathbf{(}\mathit{p}\mathbf{=}\mathbf{0.001}\mathbf{)}$	$\mathbf{(}\mathit{p}\mathbf{=}\mathbf{0.001}\mathbf{)}$	$\mathbf{(}\mathit{p}\mathbf{<}\mathbf{0.001}\mathbf{)}$
Called	1.6	1.42	1.61	1.43	1.88	1.9
	$\mathbf{(}\mathit{p}\mathbf{=}\mathbf{0.028}\mathbf{)}$	$\mathbf{(}\mathit{p}\mathbf{=}\mathbf{0.029}\mathbf{)}$	$\mathbf{(}\mathit{p}\mathbf{=}\mathbf{0.035}\mathbf{)}$	$\mathbf{(}\mathit{p}\mathbf{=}\mathbf{0.016}\mathbf{)}$	($p=0.112$)	($p=0.085$)
Play	1.94	1.79	1.86	1.9	2.01	2.07
	($p=0.254$)	($p=0.493$)	($p=0.282$)	($p=0.494$)	($p=0.178$)	($p=0.178$)
Separation	1.59	1.59	1.59	1.38	1.89	1.7
	$\mathbf{(}\mathit{p}\mathbf{=}\mathbf{0.017}\mathbf{)}$	($p=0.114$)	$\mathbf{(}\mathit{p}\mathbf{=}\mathbf{0.023}\mathbf{)}$	$\mathbf{(}\mathit{p}\mathbf{=}\mathbf{0.005}\mathbf{)}$	($p=0.099$)	$\mathbf{(}\mathit{p}\mathbf{=}\mathbf{0.013}\mathbf{)}$

In contrast to BT, dynamics in JRT differed significantly in general across conditions, except for the left ear dynamics, where the difference was marginally significant ($D^{\mathit{JRT}}:df=3,F=4.58,p=0.006;D_{lefteye}^{\mathit{JRT}}:df=3,F=5.34,p=0.002;D_{righteye}^{\mathit{JRT}}:df=3,F=4.67,p=0.005;D_{nose\&mouth}^{\mathit{JRT}}:df=3,F=5.7,p=0.001;D_{leftear}^{\mathit{JRT}}:df=3,F=2.56,p=0.062;D_{rightear}^{\mathit{JRT}}:df=3,F=2.93,p=0.04)$.

When comparing conditions in JRT, we observed higher average dynamic metrics in play than in called, stranger, and separation conditions: the dynamics in the play condition were significantly higher than in the stranger condition, except for the left eye dynamic $D_{lefteye}^{\mathit{JRT}}$. The called condition, on the other hand, significantly differed from the stranger only in $D_{lefteye}^{\mathit{JRT}}$. The separation condition did not differ significantly in dynamics compared to the stranger condition. Table 3 shows the mean dynamic metric values for different conditions in JRT.

Table 3.

Mean dynamics values D for different facial regions in Jack Russell Terriers (JRT) videos subset in different conditions. The intercept is a stranger condition. Significantly different dynamic values are highlighted in bold

	$D^{\mathit{JRT}}$	$D_{lefteye}^{\mathit{JRT}}$	$D_{righteye}^{\mathit{JRT}}$	$D_{nose\&mouth}^{\mathit{JRT}}$	$D_{leftear}^{\mathit{JRT}}$	$D_{rightear}^{\mathit{JRT}}$
Stranger	3.45	2.38	2.45	3.12	4.62	4.76
	$\mathbf{(}\mathit{p}\mathbf{<}\mathbf{0.001}\mathbf{)}$	$\mathbf{(}\mathit{p}\mathbf{<}\mathbf{0.001}\mathbf{)}$	$\mathbf{(}\mathit{p}\mathbf{<}\mathbf{0.001}\mathbf{)}$	$\mathbf{(}\mathit{p}\mathbf{<}\mathbf{0.001}\mathbf{)}$	$\mathbf{(}\mathit{p}\mathbf{<}\mathbf{0.001}\mathbf{)}$	$\mathbf{(}\mathit{p}\mathbf{<}\mathbf{0.001}\mathbf{)}$
Called	3.06	1.99	2.12	2.58	4.68	4.34
	($p=0.257$)	$\mathbf{(}\mathit{p}\mathbf{=}\mathbf{0.049}\mathbf{)}$	($p=0.098$)	($p=0.113$)	($p=0.928$)	($p=0.420$)
Play	4.24	2.77	2.85	3.93	5.74	5.78
	$\mathbf{(}\mathit{p}\mathbf{=}\mathbf{0.025}\mathbf{)}$	($p=0.056$)	$\mathbf{(}\mathit{p}\mathbf{=}\mathbf{0.046}\mathbf{)}$	$\mathbf{(}\mathit{p}\mathbf{=}\mathbf{0.026}\mathbf{)}$	$\mathbf{(}\mathit{p}\mathbf{=}\mathbf{0.044}\mathbf{)}$	$\mathbf{(}\mathit{p}\mathbf{=}\mathbf{0.049}\mathbf{)}$
Separation	3.27	2.26	2.39	2.85	4.38	4.72
	($p=0.597$)	($p=0.569$p=)	($p=0.755$)	($p=0.441$)	($p=0.648$)	($p=0.941$)

Comparing contexts between cephalic types

Play.

JRT had significantly higher dynamics in all facial regions in the play condition (see Table 4).

Table 4.

Mean dynamics values D for different facial regions aggregated over Boston Terriers (BT) and Jack Russell Terriers (JRT) in a play condition. The intercept is JRT

	D(play)	$D_{lefteye}(play)$	$D_{righteye}(play)$	$D_{nose\&mouth}(play)$	$D_{leftear}(play)$	$D_{rightear}(play)$
JRT	4.24	2.77	2.85	3.93	5.74	5.78
BT	1.94	1.79	1.86	1.90	2.01	2.07
p	$<0.001$	0.002	0.001	$<0.001$	$<0.001$	$<0.001$

Separation.

In the separation condition, JRT had significantly higher dynamics in all facial regions except the left eye (see Table 5).

Table 5.

Mean dynamics values D for different facial regions aggregated over Boston Terriers (BT) and Jack Russell Terriers (JRT) in a separation condition. The intercept is JRT

	D(sep)	$D_{lefteye}(sep)$	$D_{righteye}(sep)$	$D_{nose\&mouth}(sep)$	$D_{leftear}(sep)$	$D_{rightear}(sep)$
JRT	3.27	2.26	2.39	2.85	4.38	4.72
BT	1.59	1.59	1.59	1.39	1.89	1.70
p	0.002	0.072	0.042	0.003	0.007	$<0.001$

Stranger.

In the stranger condition, only ear dynamics were significantly lower in BT than in JRT (see Table 6).

Table 6.

Mean dynamics values D for different facial regions aggregated over Boston Terriers (BT) and Jack Russell Terriers (JRT) in a stranger condition. The intercept is JRT

	D(str)	$D_{lefteye}(str)$	$D_{righteye}(str)$	$D_{nose\&mouth}(str)$	$D_{leftear}(str)$	$D_{rightear}(str)$
JRT	3.45	2.38	2.45	3.12	4.62	4.76
BT	2.29	1.97	2.13	2.10	2.54	2.57
p	0.115	0.345	0.416	0.195	0.031	0.029

Called.

JRT had significantly higher dynamics in all facial regions in the called condition (see Table 7).

Table 7.

Mean dynamics values D for different facial regions aggregated over Boston Terriers (BT) and Jack Russell Terriers (JRT) in a called condition. The intercept is JRT

	D(call)	$D_{lefteye}(call)$	$D_{righteye}(call)$	$D_{nose\&mouth}(call)$	$D_{leftear}(call)$	$D_{rightear}(call)$
JRT	3.06	1.99	2.12	2.58	4.68	4.34
BT	1.60	1.42	1.61	1.43	1.88	1.90
p	$<0.001$	0.015	0.019	0.006	$<0.001$	$<0.001$

Discussion

The use of AI methods for facial analysis in animals is attracting great interest, as highlighted in the comprehensive review by Broomé et al. [49]. Studies focusing on dogs (e.g., Boneh-Shitrit et al. [31], Franzoni et al. [50], and Hernández-Luquin et al. [51]) predominantly employ deep learning approaches to tasks such as classification of particular emotional states. While these models achieve good accuracy and offer extensive practical applications, they function as “black-box” systems, i.e., the reasoning behind their decisions is not comprehensible to humans. This lack of explainability limits their utility as tools for scientific discovery in behavioral research, which traditionally relies on rigorous measurement of behavior and the use of statistical methods on these measurements for hypothesis testing.

Facial behavior measurements in dogs have primarily been conducted using the DogFACS annotation system [21, 27]. While being as objective as possible, this system has significant drawbacks, including the labor-intensive process of manual coding and the requirement for specialized certification of coders. Furthermore, it relies on discrete categories, such as action units and descriptors, a common practice in behavioral measurement. However, this type of categorical abstraction may overlook subtle, continuous visual signals in facial behavior, limiting its ability to capture the full complexity of dynamic facial expressions.

Landmark-based approaches present an appealing alternative for measuring facial behavior, offering the “best of both worlds”. By developing landmark schemes based on DogFACS (or other AnimalFACS systems), we can maintain a connection to the underlying facial musculature while enabling continuous tracking of behavioral visual signals. The use of these approaches for facial behavior analysis has already been demonstrated useful for recognizing pain [35], classification of social interactions [37], and analyzing morphological differences in cephalic types [36] in cats.

This study is the first to utilize a computer vision model for facial landmark detection in dogs based on the scheme developed by Martvel, Abele, et al. [38] to create a fully automated method for measuring continuous facial dynamics in dogs. To achieve this, we have proposed a dynamic metric that quantifies the relative movement of facial landmarks throughout a video, enabling also a separate analysis for different facial regions, such as the ears or eyes. In this metric, we have addressed the normalization of landmark positions to account for camera angles and distance variations. Most importantly, we explicitly addressed the isolation of the relative movement of facial landmarks from the general motion of the dog within the frame, leading to better capturing the subtle visual cues of facial behavior. An additional technical advantage of our method over automated deep learning approaches is its robustness under noisy conditions. By leveraging the model’s confidence threshold, we can automatically exclude noisy frames where facial landmarks are not clearly identified, eliminating the need for labor-intensive manual or automated preprocessing.

The limitation of the proposed method is that it operates with two-dimensional facial landmarks, introducing errors connected with three-dimensional movement since it is impossible to fully consider the distortions and perspective changes. One possible solution is using 3D animal face approximation models constructed from detected (x,y) coordinates [52]. By such approximation, it becomes possible to operate with three-dimensional coordinates, but with a cost of their placement accuracy. We plan to explore this method in our future studies.

Another important note about the landmark-based method is that it captures only a part of a dog’s activity, primarily working with dog faces oriented toward the camera. When the dog’s face is heavily rotated, the computer vision model cannot detect landmarks with high confidence. Such limitation applies to both other automated and manual motion coding methods to some degree, but landmarks tend to be more acute in this sense. However, for this study, it would be reasonable to assume that the dog was oriented toward the experimenters most of the time and displayed at least not more facial dynamics when oriented away from the camera.

Using the novel measurement method described above, we quantified the facial dynamics of brachycephalic and normocephalic dogs across four contexts, leading to some new insights into the differences. Brachycephaly is particularly intriguing in this context due to its distinctive physical traits, including a large forehead and eyes, a shortened rostrum, a small-appearing mouth, and a reduced chin [53]. These features not only alter the dog’s appearance but may also impact its communication.

Our results indicate that the facial dynamics of BT dogs, as objectively quantified by our method, are significantly lower compared to JRT dogs across all contexts and facial regions. This could be related to the communication limitations attributed to paedomorphic traits of brachycephalic breeds [23]. Additionally, it aligns with the finding that humans had more difficulty perceiving negative signals in BT rather than in JRT. They even attributed positive emotional charge to negative situations [22], although our analysis showed that facial dynamics are the highest in the case of the stranger condition. The misinterpretation by humans may be influenced by the “baby schema” effect, described by Lorenz [54], wherein features such as a round head and large eyes serve as optical signals that elicit caretaking behaviors and feelings of cuteness [55]. Unlike subjective human assessments, our method provides a more objective evaluation of these signals.

The difference in facial dynamics across all contexts between the Boston Terrier and the Jack Russell Terrier can be related to the findings of Paul et al. [56], who compared their craniofacial ratio (CFR). Indeed, brachycephalic breeds possess relatively larger (taller) foreheads and a larger eye area than their normocephalic counterparts. This, along with significant differences in facial muscle distribution between the two cephalic types of dogs [24], would explain the differences between the two breeds in all conditions, as revealed in our study.

An additional interesting finding is that the play (positive) condition has significantly higher facial dynamics across both breeds than the separation (negative) condition. Freezing and passiveness have been described in the literature as separation-related behaviors [57]. Interestingly, the Boston Terriers also had the highest dynamics in the stranger (negative) condition, while significantly lower dynamics characterized both negative conditions for JRT.

The tendency of human viewers to focus on the eyes and nasal/mouth region when looking for affective cues has been documented [58]. Similarly, participants of the study of Eretová et al. [22] self-reported the eyes and ears of dogs, regardless of their skull shape, as the most informative features of the face, followed closely by the snout. Our findings show that ears have the highest dynamics across all contexts for both breeds, which may explain the viewers’ attention to them. On the other hand, the measured eye dynamics are lower than ears in both breeds. It should be noted in this context that in brachycephalic dogs, the eyes typically appear large as they protrude from the orbit more than in normocephalic dogs [59], so humans may consider them a vital communication cue and look at them intently, despite the objective relatively low dynamics.

The obtained results demonstrated a difference in dynamic between paired parts of the face (eyes and ears) in some cases. We further tested the results with mixed-effects regression tests for different conditions; the asymmetry between left and right facial dynamics across conditions or the two breeds was insignificant (see Table 9 in Supplementary Materials). Regardless, the presented materials were not suitable for the proper inspection of left-right asymmetry in facial expressions due to the fact that the Experimenters’ positions and angles of the camera used in this study varied and were not designed to inspect symmetry properly. In future studies, we plan to further emphasize the study of asymmetry, addressing it fully in experiment design and data collection.

A limitation of the present study is that only two breeds represent brachycephalic and normocephalic morphological types, respectively. Some dogs’ reactions could be partly influenced by the different temperaments of the two breeds studied; therefore, it is necessary to test other representatives of dogs that differ in nose length. Additionally, the snout length and overall head shape somewhat varied among the brachycephalic dogs. Future research should incorporate a greater diversity of breeds to better represent the spectrum of morphological variations. Additionally, expanding the sample size to include a larger number of individual dogs will enhance the generalizability and robustness of the findings.

This study represents an initial step in systematically and objectively quantifying canine visual cues across different morphological types and situational contexts. Future research should involve a broader range of canine phenotypes to fully understand how facial dynamics interact with the effectiveness of canine visual cues. Further expanding, validating, and calibrating the computational approach demonstrated here, which has proven effective in capturing subtle facial dynamics, could provide deeper insights into this complex topic.

Appendix

1 Sex and age information for dogs participated in the study

Table 8.

Sex and age of Jack Russell (JRT) and Boston Terriers (BT) participated in the study

Code	Sex	Age (years)
JRT01	Female	2.5
JRT02	Male	4
JRT03	Female	2.5
JRT04	Female	2
JRT05	Male	3
JRT06	Female	2.5
JRT07	Female	2
BT01	Female	1.25
BT02	Male	1.25
BT03	Male	5
BT04	Female	3
BT05	Male	1.5
BT06	Female	1
BT07	Female	6

2 Dynamics asymmetry

Table 9.

Regression coefficient $\beta$ and p-value for left and right eye and ear dynamics aggregated over Boston Terriers (BT) and Jack Russell Terriers (JRT) in all conditions. The left dynamic is the dependent variable, and the right dynamic is the independent variable

Condition	$D_{lefteye}^{\mathit{BT}}\sim D_{righteye}^{\mathit{BT}}$	$D_{lefteye}^{\mathit{JRT}}\sim D_{righteye}^{\mathit{JRT}}$	$D_{leftear}^{\mathit{BT}}\sim D_{rightear}^{\mathit{BT}}$	$D_{leftear}^{\mathit{JRT}}\sim D_{rightear}^{\mathit{JRT}}$
Called	0.80 ($p<0.001$)	0.93 ($p=0.002$)	0.91 ($p<0.001$)	0.63 ($p=0.034$)
Play	0.77 ($p<0.001$)	0.82 ($p<0.001$)	0.88 ($p<0.001$)	0.75 ($p<0.001$)
Separation	0.85 ($p<0.001$)	0.99 ($p<0.001$)	0.73 ($p<0.001$)	0.50 ($p=0.008$)
Stranger	0.88 ($p<0.001$)	0.96 ($p<0.001$)	1.04 ($p<0.001$)	0.71 ($p<0.001$)

Authors' contributions

PE, LP, HC, and PP acquired the data. GM, IS, YE, DG, and AZ conceived the experiment(s). GM conducted the experiment(s). GM, PE, LP, HC, PP, IS, YE, DG, and AZ analyzed and/or interpreted the results. All authors reviewed the manuscript.

Funding

The research was partially supported by the Data Science Research Center at the University of Haifa. PE, LP, and HC were supported by the Grant Agency of the Czech University of Life Sciences Prague (grant no. SV22-18-21370).

Data availability

Videos from the original study are available from the corresponding author on reasonable request. The landmark time series dataset generated during the current study is available at https://github.com/martvelge/dog_dynamics.

Code availability

The code generated during the current study is available at https://github.com/martvelge/dog_dynamics.

Declarations

Ethics approval and consent to participate

The data collection of the dogs’ behavior was assessed as noninvasive experimentation and approved by the Animal Welfare Committee of Eötvös Loránd University (Certificate number PEI/001/1056-4/2015).

All dog owners who participated in video recording with their dogs signed a written informed consent form.

Consent for publication

No personal data was published in the current study.

Competing interests

The authors declare no competing interests.