GeSTICS: A Multimodal Corpus for Studying Gesture Synthesis in Two-party Interactions with Contextualized Speech

Abstract

Generating natural co-speech gestures and facial expressions for effective human-agent interactions requires modeling the intricate interplay between verbal, non-verbal, and contextual cues observed in dyadic human communication. Two types of contextual cues are of particular interest: (1) individual factors of the interlocutors, such as their demographic attributes, and (2) situational factors, like the outcome of a preceding event. To facilitate their study, we introduce the GeSTICS Dataset, a novel multimodal corpus comprising 9,853 questions and 10,460 answers from audiovisual recordings of post-game sports interviews by 147 interviewees. The dataset contains speech data, including textual transcriptions, lexical descriptors, and acoustic features, as well as visual data encompassing the interviewee’s body pose and facial expressions, with an emphasis on capturing these modalities during both the question-listening and answering phases of the interview. Furthermore, GeSTICS incorporates metadata about individual factors, such as the age and cultural background of the interviewees, and situational factors, like the results of the games, which are often overlooked in existing multimodal datasets. Our preliminary analysis of GeSTICS reveals that the effects of speech features, such as loudness and lexical choice, on the production of co-speech gestures in both speaking and listening phases are moderated by situational factors and the interviewee’s individual factors. GeSTICS is designed to enhance the generation of realistic nonverbal behaviors in virtual agents, animated characters, and human-robot interaction systems, thus contributing to more engaging and effective human-agent communication. The analysis code and the dataset are available at https://gkebe.github.io/gestics/.

1. INTRODUCTION

Recent advancements in deep learning have significantly improved the generation of natural and semantically relevant co-speech gestures. Most approaches focus on direct speech-gesture mapping [33, 53]. While these approaches have shown promising results, they do not fully capture the complexity of human gesticulation.

In this paper, we use the term contextual factors to refer to the various elements beyond speech content that influence an individual’s gestures, including their inherent traits, background, and the specific circumstances they encounter. We further categorize these contextual factors into individual factors and situational factors. Individual factors, such as personality traits [28], disorders [32], and gesturing style [2, 35, 52, 54], are more stable over time and have been extensively studied, serving as the focus of numerous computational approaches and datasets.

In contrast, situational factors, including social context, mood, and previous events, are more transient and subject to change between different interactions. Despite their well-documented impact on speech and gestures in human communication [4, 14, 25], situational factors have received less attention in the literature, particularly in terms of their specific role in shaping co-speech gestures. This gap in research highlights the need for further investigation, as situational factors play a crucial role in natural human interaction. A major cause of this gap is the inherent challenge of collecting a well-controlled dataset that effectively accounts for these variables during spontaneous interactions. By addressing this gap and developing a more comprehensive understanding of the complex interplay between speech, gestures, and the contextual factors that shape them, we can enhance the naturalness and contextual appropriateness of generated gestures, leading to more engaging and effective human-agent interactions.

To contribute to this research trajectory, we present the GeSTICS Dataset, a novel multimodal corpus that captures the complex interactions between speech, gestures, and contextual factors, encompassing both individual factors and situational factors, in dyadic human communication. GeSTICS consists of 9,852 questions and 10,460 answers extracted from audiovisual recordings of post-game basketball interviews featuring 147 individuals. The dataset includes a wide range of speech data (textual transcriptions, lexical descriptors, acoustic features) and visual data (interviewee’s body pose, facial expressions), captured during both the question listening and answering phases.

We hypothesize that post-game sports interviews provide a suitable setting for studying the influence of situational factors on co-speech gestures. The interviews occur immediately after games and involve a range of contextual factors that may shape the interviewees’ verbal and non-verbal expressions, such as the game outcome, game importance, public opinion of the team, the interviewee’s experience level, and their role on the team (player or coach). Additionally, the interview format also allows for an analysis of gesture production in both speaking and listening phases.

The GeSTICS Dataset aims to advance the development of more naturalistic and context-aware co-speech gesture generation models for virtual agents, animated characters, and human-robot interaction systems. By providing a rich, multimodal corpus that captures the complex interplay between verbal, non-verbal, and contextual factors, with an emphasis on both individual factors and situational factors, GeSTICS could support the training of personalized models capable of generating affectively appropriate gestures based on situational factors, enhancing the realism and contextual sensitivity of virtual agents.

Finally, we employ mixed-effects regression models to examine two key relationships: the influence of the interviewer’s speech features on the interviewee’s gestures during the listening phase, and the connection between the interviewee’s own speech features and gestures in the answering phase. Additionally, we investigate the impact of situational factors, such as game score difference, and individual factors, including age and team role, on these relation-ships.

2. RELATED WORK

2.1. Datasets and Approaches for Co-speech Gesture Generation and Listener Feedback

Previous research on co-speech gesture generation has focused mainly on the direct relationship between speech and gesticulation. This work aims to enhance the naturalness of virtual agents and animated characters [24, 33, 45]. Datasets like the TED Gesture Dataset [13, 53] have been instrumental in supporting these efforts. Similarly, researchers have explored modeling listener feedback during conversations [5, 8, 29, 38–41, 50], with datasets like ALICO [11] and ViCo [55] focusing on this aspect of dyadic interactions. Moreover, some studies have pursued the joint modeling of gestures and behaviors considering both the speaker’s and listener’s perspectives within such interactions [22, 26], particularly in contexts involving human interactions with virtual agents.

2.2. Individual Factors in Co-speech Gestures and Listener Feedback

As research in nonverbal behavior generation progressed, the importance of individual factors such as styles, cultural differences, and personality traits became increasingly apparent [1–3, 19–21, 34, 52, 54]. Studies have shown that incorporating personality information can lead to more diverse and realistic co-speech gestures [28] and that cultural differences influence facial expressions during co-speech gestures [37]. Datasets like BEAT [35] have enabled the generation of better stylized gestures by including emotional and semantic annotations. Similarly, individual characteristics have been found to influence listener feedback, with personality traits and conflict handling styles affecting the use of non-verbal cues in phone conversations [49] and person-specific factors influencing feedback expressiveness during backchanneling opportunity points [6].

2.3. Other Contextual Factors

While individual factors have received considerable attention in recent works, the impact of other contextual factors on co-speech gestures and listener feedback remains relatively underexplored. Some researchers have investigated the role of contextual factors related to affect, with the JESTKOD database [9] including emotion annotations, agreement/disagreement labels, and other affective factors and Yang and Narayanan [51] showing that emotion modulates the relationship between speech and gesture. Interlocutor context has also been explored [11, 42, 48], with Buschmeier et al. [11] investigating the impact of listening distraction on speaker behavior.

In contrast to these transient contextual factors, our work highlights the importance of considering broader situational influences that may shape an individual’s communication patterns over extended periods. Situational factors like social settings, interpersonal relationships, and previous events can induce more prolonged emotional/mental states and behaviors. These overarching circumstances likely have longer-lasting effects on verbal and nonverbal expression compared to momentary contextual variables within a single interaction. Additionally, the situational factors in GeSTICS arise organically from spontaneous real-life interactions.

3. GESTICS DATASET

The dataset is curated from publicly available post-game interviews from the 2022–2023 NBA season, meticulously selected to encompass a diverse array of interviewees and situational factors. The videos were specifically chosen due to their consistent recording conditions, comprising stable camera angles, high-fidelity audio and minimal ambient noise. To preserve the anonymity of the participants, all identifiable information, including the names of players, coaches, and teams, was removed from the dataset. Consequently, the original video and audio recordings are not included. Instead, the dataset contains anonymized interview transcripts and interpretable audio-visual features, such as pose vectors, facilitating comprehensive analyses of verbal and nonverbal communication dynamics. The cumulative duration of the answer segments is approximately 104.53 hours (M=29.41 minutes per answer), while the total duration of question segments is approximately 32.29 hours (M=9.59 seconds per question).

3.1. Audio-Visual Data

The GeSTICS dataset employs several models to extract rich and interpretable multimodal features from the raw interview data: Speech Features: Automatic Speech Recognition (ASR) is performed using Whisper [44] to generate transcripts of the interview audio. Speaker diarization is then applied using pyannote.audio [10] to label speaker turns in the interviews. The diarization segments and whisper transcriptions are used to identify question and answer segments in a semi-automatic way based on the fact that the longest speaking track should belong to the interviewee. Sentiment analysis is performed on the transcripts using the VADER (Valence Aware Dictionary and sEntiment Reasoner) model [27] to quantify the positive, negative, and neutral sentiment expressed. Additionally, acoustic features are extracted using the Opensmile toolkit [16], and lexical features are computed using LIWC (Linguistic Inquiry and Word Count) [43]. Visual Features: To track the interviewee’s face throughout the videos, face detection is first performed using RetinaFace [15]. The detected faces are then recognized and clustered using FaceNet [46]. Once the interviewee’s face is identified, the corresponding body region is cropped from the selected frames. Mediapipe [36] is then applied to the cropped frames to extract body poses, hand keypoints, facial landmarks, and facial action units (blendshapes). All pose frames are centered and normalized based on the distance from the left shoulder to the right hip to ensure smooth frame transitions and body pose size consistency. Table 1 summarizes the full set of features included in the GeSTICS dataset.

Table 1:

Features included in the GeSTICS dataset (number of features in parentheses)

Speech
	Whisper Transcriptions
	eGeMAPS Acoustic Features (88)
	LIWC (93) & VADER Lexical Features
Visual
	Body Pose Landmarks (33)
	Hand Keypoints (21 per hand)
	Face Landmarks (468) & Action Units (52)

3.2. Contextual Metadata

The GeSTICS dataset incorporates a rich array of contextual metadata, enabling researchers to study how various factors may influence the interaction dynamics in post-game interviews. These metadata can be broadly categorized into two types: situational factors and individual factors. Situational Factors: These factors provide a measure of the competitive context surrounding each post-game interview question and answer segment. The data was collected from the reputable sports analytics site FiveThirtyEight [17]. The key metrics, summarized in Table 2, include game score differentials, game quality indices that capture team strengths via the harmonic mean of Elo ratings [31], game importance ratings that quantify a result’s impact on postseason projections on a 0–100 scale, and team strength disparities calculated using FiveThirtyEight’s RAPTOR metric [18].

Table 2:

Summary statistics of situational factors

Attribute	Win	Loss
Game Score Difference	12.15 ± 9.36	−12.22 ± 8.6
Game Quality Index	28.74 ± 68.51	26.36 ± 70.86
Game Importance	39.5 ± 61.46	40.52 ± 58.21
Team Strength Difference	39.33 ± 126.21	−24.1 ± 119.29
Games	308	338
Total Games	505

Individual Factors: These factors encompass essential demographic attributes, including race, nationality, team role, and age. We employ a rigorous anonymization process to protect the identity of the interviewees while preserving the richness of the demographic and professional data. Each interviewee is assigned a unique identifier unlinked to their name or other directly identifying information. In Table 3, the mean and standard deviation of actual ages are reported in the table, but in the dataset, only the age percentile is provided, representing the interviewee’s relative age within the entire pool. Race, origin, and team role are recorded as categorical variables, providing insights into cultural, demographic, and professional factors without compromising anonymity.

Table 3:

Summary statistics of individual factors

Attribute	Type of Data	Statistics Summary
Race	Categorical	Black: 75.0% White: 22.3% Asian: 1.4% Latino: 0.7%
Origin	Categorical	Local: 79.7% International: 20.3%
Team Role	Categorical	Player: 82.9% Coach: 17.1%
Age	Continuous	Mean: 31.89 Std Dev: 9.78
Total Interviewees		147

4. EXPERIMENTAL APPROACH

We explore three main research questions:

1. What relationships, if any, exist between interviewer’s speech features and interviewee’s gestures during questions and interviewee’s own speech and their gestures while answering?

2. Do situational factors, specifically the score difference between teams, influence the relationships between speech features and gestures?

3. Do individual factors, such as the interviewee’s role on the team (coach vs. player), affect the relationships between speech features and nonverbal behaviors?

To investigate these research questions, we first extract relevant multimodal features as described in Section 4.1, and then conduct the experimental analyses explained in Sections 4.2, 4.3, 4.4.

4.1. Speech and Gesture Features

Gesture Features.

Gestural rate, a measure of the frequency and amplitude of movements, was calculated using the pose keypoints provided by MediaPipe, considering only keypoints with a confidence score above 0.5 [13]. The gestural rate was computed as the average Euclidean distance between the same keypoint across adjacent frames. Mean frowning, smiling, and downward gaze scores were obtained using the blendshapes extracted from MediaPipe. These scores represent the intensity of the respective facial expressions, ranging from 0 to 1.

Speech Features.

Linguistic features, specifically the count of positive emotion words and negative emotion words, were extracted using the Linguistic Inquiry and Word Count (LIWC) tool [43]. LIWC compares the words in the transcribed speech to predefined dictionaries and provides the frequency of words associated with positive emotions and negative emotions. Prosodic features, namely voice pitch (fundamental frequency, F0) and loudness (intensity), were extracted using the openSMILE toolkit [16]. The mean values of these features were calculated over the duration of each speech segment to capture the overall prosodic characteristics.

4.2. Associations of Speech Features and Contextual Factors with Co-occuring Gestures

We investigate how various aspects of interviewer and interviewee speech features, contextual factors (game score), and individual factors (team role) relate to specific nonverbal cues in question-listening and answer-speaking phases. To achieve this, we used mixed-effects regression models for each feature and outcome pair. Speech feature variables and contextual factor variables are standardized and mean centered within each interviewee, allowing us to separate effects within and between interviews. Individual characteristic variables are included without standardization or mean centering. The models account for individual variation among interviewees by including as fixed effects, the mean values of each interviewee’s speech features and contextual factors, as well as random intercepts. We control for the interviewee’s role in all models, except when role itself is the feature being investigated. The mathematical formulation is as follows:

$$\begin{gathered} y_{ij}={\beta }_{0i}+{\beta }_1\left(x_{ij}-{\bar{x}}_i\right)+e_{ij} \\ {\beta }_{0i}={\gamma }_{00}+{\gamma }_{01}{\bar{x}}_i+{\gamma }_{02}ROL{E}_i+u_{0i} \\ {e}_{ij}\sim \mathrm{N}\mathrm{o}\mathrm{r}\mathrm{m}\mathrm{a}\mathrm{l}\left(0,{\sigma }_e^2\right) \\ {u}_{0i}\sim \mathrm{N}\mathrm{o}\mathrm{r}\mathrm{m}\mathrm{a}\mathrm{l}\left(0,{\sigma }_{u0}^2\right) \end{gathered}$$ (1)

where:

• $y_{ij}$ is the outcome variable for instance $j$ of interviewee $i$,

• ${\beta }_{0i}$ is the interviewee-specific intercept,

• ${\beta }_1$ is the fixed slope for the within-interviewee speech feature component,

• $x_{ij}$ is the speech feature variable for instance $j$ of interviewee $i$, which is decomposed into within- and between-interviewee components,

• ${\bar{x}}_i$ is the interviewee-specific mean of the speech feature variable, capturing the between-interviewee component,

• $\left(x_{ij}-{\bar{x}}_i\right)$ is the within-interviewee component (i.e., deviations from their own mean in a given observation),

• $e_{ij}$ is the residual error for instance $j$ of interviewee $i$,

• ${\gamma }_{00}$ is the fixed intercept,

• ${\gamma }_{01}$ is the fixed slope for the between-interviewee speech feature component,

• ${\gamma }_{02}$ is the fixed slope for the interviewee role,

• ${\mathrm{R}\mathrm{O}\mathrm{L}\mathrm{E}}_i$ is a categorical variable (i.e., dummy code) representing the team role of interviewee $i$,

• $u_{0i}$ represents the random deviation of interviewee $i$’s intercept from the fixed/average intercept ${\gamma }_{00}$.

4.3. Influence of Game Score Difference

We investigate how the game outcome, quantified by the score difference between the interviewee’s team and their opponent, may moderate the relationships between speech features and interviewee gestures during the post-game interview. To explore this, we extend the mixed-effects models that examine the associations between speech features and gestures by incorporating the score difference variable and its multiplicative interactions with the within-interviewee speech features. We also control for the interviewee’s role to account for its potential impact on the relationships. This approach enables us to determine whether the game outcome modulates the connections between speech and gestures while considering the possible effect of the interviewee’s role within the team. The presence of a significant ${\beta }_3$ coefficient in Equation (2) would indicate that the game outcome moderates the relationship between speech features and gestures.

$$\begin{gathered} y_{ij}={\beta }_{0i}+{\beta }_1{x}_{cj}+{\beta }_2{SCORE}_{ij}+{\beta }_3{x}_{cj}{SCORE}_{ij}+e_{ij} \\ {x}_{cj}=x_{ij}-{\bar{x}}_i \\ {\beta }_{0i}={\gamma }_{00}+{\gamma }_{01}{\bar{x}}_i+{\gamma }_{02}{ROLE}_i+u_{0i} \\ {e}_{ij}\sim \mathrm{N}\mathrm{o}\mathrm{r}\mathrm{m}\mathrm{a}\mathrm{l}\left(0,{\sigma }_e^2\right) \\ {u}_{0i}\sim \mathrm{N}\mathrm{o}\mathrm{r}\mathrm{m}\mathrm{a}\mathrm{l}\left(0,{\sigma }_{u0}^2\right) \end{gathered}$$ (2)

where:

• $SCOR{E}_{ij}$ represents the game score difference for instance $j$ of interviewee $i$,

• ${\beta }_2$ and ${\beta }_3$ are the fixed slopes for score difference and its interaction with the within-interviewee speech feature.

4.4. Influence of Interviewee Team Role

We explore how the interviewee’s role within the team may moderate the relationships between speech features and interviewee gestures during the post-game interview. To investigate this, we employ a mixed-effects model that includes the interviewee’s role as a predictor variable and examines its interaction with the speech features. The model allows the intercept and the slope of the speech feature to vary by individual, and it also incorporates the interviewee’s role as a fixed effect on both the intercept and the slope. By including the interaction term between the interviewee’s role and the speech features, we can assess whether the associations between speech features and gestures differ depending on the interviewee’s role (coach vs. player) while accounting for individual variability in the relationships. In Equation (3), ${\gamma }_{11}$ represents the fixed effect coefficient for the interaction between the interviewee’s role and the within-interviewee speech feature variable. The presence of a significant ${\gamma }_{11}$ coefficient would indicate that the interviewee’s role moderates the relationship between speech features and gestures.

$$\begin{gathered} y_{ij}={\beta }_{0i}+{\beta }_{1i}\left(x_{ij}-{\bar{x}}_i\right)+e_{ij} \\ {\beta }_{0i}={\gamma }_{00}+{\gamma }_{01}{\bar{x}}_i+{\gamma }_{02}ROL{E}_i+u_{0i} \\ {\beta }_{1i}={\gamma }_{10}+{\gamma }_{11}ROL{E}_i+u_{1i} \\ {e}_{ij}\sim \mathrm{N}\mathrm{o}\mathrm{r}\mathrm{m}\mathrm{a}\mathrm{l}\left(0,{\sigma }_e^2\right) \\ \left(\genfrac{}{}{0pt}{}{u_{0i}}{u_{1i}}\right)\sim \mathrm{M}\mathrm{V}\mathrm{N}\mathrm{o}\mathrm{r}\mathrm{m}\mathrm{a}\mathrm{l}\left(\left[\begin{array}{c}0\\ 0\end{array}\right],\left[\begin{array}{cc}{\sigma }_{u0}^2& {\sigma }_{u10}^2\\ {\sigma }_{u01}^2& {\sigma }_{u1}^2\end{array}\right]\right) \end{gathered}$$ (3)

where:

• ${\gamma }_{02}$ and ${\gamma }_{11}$ are the fixed effect coefficients for the interviewee’s age and its interaction with the within-interviewee speech feature variable,

• $u_{01i}$ represents the random slope for the within-interviewee speech feature variable, allowing the effect of the speech feature to vary across interviewees.

5. RESULTS

In this section, we present the findings from our mixed-effects regression models examining the associations between speech features, co-speech gestures, and contextual factors in post-game sport interviews. The coefficients are presented and significant associations (p < 0.05) are shaded and bolded in the tables.

5.1. During Questions: Interviewer Speech Features Influence Interviewee Gestures

The mixed-effects regression models revealed several significant associations between interviewer speech features and interviewee gestures during the question-listening phase (see Table 4a). Specifically, higher interviewer loudness was associated with increased interviewee gestural rate (${\beta }_1=0.026,p=.041$). In contrast, higher interviewer voice pitch was related to decreased interviewee gestural rate (${\beta }_1=-0.031,p=.003$). Furthermore, a higher frequency of negative words in interviewer questions was associated with reduced interviewee smiling (${\beta }_1=-0.031,p=.001$). Interviewee smiling was also positively related to the score difference between the teams, with more smiling when their team won by a larger margin (${\beta }_1=0.058,p<.001$). Interviewee frowning showed a significant positive association with their team role (${\beta }_1=0.198,p<.001$), indicating that coaches were more likely to frown compared to players. Additionally, interviewer questions with higher voice pitch were linked to increased downward gaze by the interviewee (${\beta }_1=0.051,p<.001$). These findings suggest that interviewer speech features and contextual factors can influence the nonverbal behaviors of interviewees during question-listening phases in post-game sport interviews.

Table 4:

Within-interviewee associations (β₁) between question speech features (green), answer speech features (purple), situational factors, and individual characteristics with co-speech gestures.

Question	Pos. Word Freq.	Neg. Word Freq.	Loudness	Voice Pitch	Score Difference	Team Role 1
Gestural Rate	−0.010	0.002	0.026 *	−0.031 **	0.002	−0.100
Smile	−0.004	−0.031 **	0.006	0.011	0.058 ***	−0.019
Frown	0.015	0.001	−0.011	0.011	−0.011	0.198 ***
Downward Gaze	0.004	−0.012	0.003	0.051 ***	0.007	0.038
(a) Question features
Answer	Pos. Word Freq.	Neg. Word Freq.	Loudness	Voice Pitch	Score Difference	Team Role 1
Gestural Rate	0.019 *	−0.004	0.059 ***	0.016	0.004	−0.066
Smile	0.022 **	−0.021 **	0.016	−0.007	0.035 ***	−0.020
Frown	0.013	0.009	0.014	−0.030 ***	−0.012	0.340 ***
Downward Gaze	0.016	−0.017 *	−0.041 ***	−0.019	−0.013	0.039
(b) Answer features

Cells display direct-effect estimates from the mixed-effect models in eq. (1), with significant associations indicated by asterisks (*) and bolded. Significance levels:

^***p < 0.001,

^**p < 0.01,

^*p < 0.05.

5.2. While Answering: Interviewee Gestures Correlate with Own Speech Features

The mixed-effects regression models also revealed significant associations between interviewee speech features and their own gestures during the answer-speaking phase (see Table 4b). Interviewee gestural rate was positively related to their own positive word frequency (${\beta }_1=0.019,p=.045$) and loudness (${\beta }_1=0.059,p<.001$). Interviewee smiling showed a positive association with their own positive word frequency (${\beta }_1=0.022,p=.005$) and a negative association with their negative word frequency (${\beta }_1=-0.021,p=.009$). Additionally, interviewee smiling was positively related to the score difference between the teams, with more smiling when their team won by a larger margin (${\beta }_1=0.035,p<.001$). Interviewee frowning was negatively associated with their own voice pitch (${\beta }_1=-0.030,p<.001$) and positively associated with their team role (${\beta }_1=0.340,p<.001$), indicating that coaches were more likely to frown compared to players. Interviewee downward gaze was negatively related to their own negative word frequency (${\beta }_1=-0.017,p=.035$) and loudness (${\beta }_1=-0.041,p<.001$). These findings suggest that interviewee speech features and contextual factors are associated with their own nonverbal behaviors during answer-speaking phases in post-game sport interviews. Our experiments also yielded intriguing results regarding the relationship between an interviewee’s general speaking style and their gestures during the answering phase. By examining the interviewee’s mean speech features ($\bar{x}i$), we could directly assess how their typical speaking patterns relate to their own gestures and expressions, as both the speech and nonverbal behaviors originate from the same individual. We discovered that interviewees who generally spoke with higher vocal intensity tended to exhibit a higher rate of gesturing ($\gamma 01=0.228,p=.002$), while those who habitually used more negative language displayed more downward gaze (${\gamma }_{01}=0.432,p=.027$). Finally, interviewees whose teams won by larger margins on average tended to smile more during their responses (${\gamma }_{01}=0.197,p=.032$).

5.3. Game Score Difference Moderates Speech and Gesture Correlations

Our analysis also investigated how the game outcome, quantified by the score difference between the interviewee’s team and their opponent, moderated the relationships between speech features and interviewee gestures during post-game interviews (see Table 5a and Table 5c). The impact of the outcome of the game on these relationships is also depicted in Figure 2. For instance, Figure 2a shows that the negative correlation between the interviewer’s use of positive emotional language and the interviewee’s downward gaze diminishes as the victory margin increases (${\beta }_3=-0.019,p=.027$). Conversely, Figure 2b shows that the positive correlation between positive emotional language use by the interviewee and their smiling intensifies with larger victory margins (${\beta }_3=0.021,p=.007$). Additionally, we found that a larger victory margin also weakens the association between the pitch of the interviewer’s voice and the smile of the interviewee (${\beta }_3=-0.025,p=.021$) when listening to questions. These findings suggest that when the interviewee’s team won by a larger margin, the interviewer’s speech features had less influence on the interviewee’s nonverbal behaviors. However, when answering, a larger margin notably strengthens the positive correlations between interviewee prosodic features—namely loudness (${\beta }_3=0.026,p=.010$) and pitch (${\beta }_3=0.032,p=.005$)—and their gestural rate. However, consistent with the listening phase, the relationship between interviewee voice pitch and smiling was also attenuated when the victory margin was larger (${\beta }_3=-0.025,p=.014$). Furthermore, our analysis revealed that a larger win amplified the positive relationships between interviewee negative language use (${\beta }_3=0.020,p=.007$), loudness (${\beta }_3=0.018,p=.015$), and frowning behavior. These findings indicate that a situational factor such as the magnitude of the victory differentially moderated the associations between the interviewee’s speech features and their gestures, some relationships being amplified and others attenuated.

Table 5:

Associations between question speech features (a,b) and answer speech features (c,d) with co-occurring interviewee gestures, moderated by score difference and team role (coach or player).

Score Difference	Pos. Word Freq.	Neg. Word Freq.	Loudness	Voice Pitch	Team Role	Pos. Word Freq.	Neg. Word Freq.	Loudness	Voice Pitch
Gestural Rate	0.002	0.012	0.006	0.001	Gestural Rate	0.001	0.005	−0.028	-
Smile	−0.014	−0.006	0.018	−0.025 *	Smile	−0.016	-	−0.040 *	−0.005
Frown	−0.004	−0.002	0.003	0.014	Frown	0.023	-	0.032	−0.013
Downward Gaze	−0.019 *	0.005	0.018	−0.014	Downward Gaze	−0.013	0.005	−0.006	0.026
(a) Questions moderated by game score difference					(b) Questions moderated by interviewee role (coach or player)
Score Difference	Pos. Word Freq.	Neg. Word Freq.	Loudness	Voice Pitch	Team Role	Pos. Word Freq.	Neg. Word Freq.	Loudness	Voice Pitch
Gestural Rate	−0.016	0.006	0.026 **	0.032 **	Gestural Rate	−0.011	−0.001	−0.100 *	−0.005
Smile	0.021 **	0.010	0.003	−0.025 *	Smile	−0.014	0.002	−0.007	−0.018
Frown	−0.002	0.020 **	0.018 *	0.000	Frown	0.029 **	−0.021	0.050	−0.025
Downward Gaze	−0.002	0.011	0.006	−0.001	Downward Gaze	−0.012	−0.002	−0.020	0.019
(c) Answers moderated by game score difference					(d) Answers moderated by interviewee role (coach or player)

Cells display fixed effect estimates, with significant associations indicated by asterisks (*) and bolded. Significance levels:

^***p < 0.001,

^**p < 0.01,

^*p < 0.05.

These results indicate that the relationship between speech and gesture is contingent upon situational and individual factors.

Figure 2:

Spotlight analyses showing how score difference moderates associations between (a) interviewer’s positive words and interviewee’s downward gaze, and (b) interviewee’s positive words and smiling. Plots show estimated effects at −1 standard deviation (SD), mean, and +1 SD of score difference. Crosses (x) represent observed data points; shaded regions indicate ±1 standard error.

5.4. Interviewee’s Role in their Team Moderates Speech and Gesture Correlations

We also examined how the interviewee’s role within their team (coach vs. player) moderated the correlations between speech features and gestures during post-game interviews (see Table 5b and Table 5d). Figure 3a shows a negative moderation of the correlation between the loudness of the question and the smiling of the interviewee for coaches compared to players (${\gamma }_{11}=-0.040,p=.017$), indicating that coaches had a more pronounced decrease in smiling when their interlocutor increased vocal intensity compared to players. Similarly, Figure 3b shows that the relationship between interviewee loudness and gestural rate was more negative for coaches compared to players (${\gamma }_{11}=-0.100,p=.040$), implying that coaches are less likely to pair loud speech with expressive gestures compared to players. Furthermore, the results showed that coaches exhibited a stronger propensity to frown when using positive language than players (${\gamma }_{11}=0.029,p=.005$). The moderation effects suggest that associations between speech and gestures can differ depending on whether the interviewee is a coach or a player, and some relationships are stronger for one group than the other, highlighting that individual factors, such as the role of the interviewee within his team, significantly influence the dynamics between speech features and gestures during post-game interviews.

Figure 3:

Spotlight analyses showing how interviewee team role moderates the associations between (a) interviewer question pitch and interviewee frowning, and (b) interviewee positive words and interviewee gestural rate.

6. DISCUSSION

In this section, we analyze our experiments and results from the GeSTICS dataset to explore how various factors influence the interaction between speech and gestures.

6.1. Key Findings

Our results indicate that situational factors, such as the outcome of the game, can modify the way speech features and gestures are associated. Specifically, we observed that a larger victory margin diminishes the impact of interviewer vocal and linguistic characteristics on the nonverbal behaviors of interviewees during the question-listening phase. This reduction in influence might be due to the interviewee’s positive emotional state or an increased sense of confidence and control after a positive experience [23, 47]. On the contrary, during the response phase, a larger victory margin appeared to strengthen the connections between the vocal characteristics of the interviewees and their gestural rate. This observation is consistent with previous findings that emotional states can influence the interaction between facial gestures and speech [12].

Furthermore, our analysis suggests that individual characteristics, such as the interviewee’s role on the team, influence the interaction between speech features and gestures. Specifically, coaches demonstrated a more pronounced negative speech-gesture coupling compared to players, especially in response to louder questions. This may reflect a greater need for coaches to control their nonverbal expressions to maintain a leadership posture. This distinction could also be attributed to variations in age and experience.

Notably, our analysis of the relationship between an interviewee’s general speaking style and their nonverbal expressions during the answer-speaking phase yielded intriguing findings. We found that interviewees who spoke with a higher average volume tended to also exhibit a higher rate of gestures (${\gamma }_{01}=0.228,p=.002$), possibly reflecting individual differences in personality traits such as extraversion or confidence [7]. Furthermore, interviewees who habitually used more negative language showed more frequent downward gaze (${\gamma }_{01}=0.432,p=.027$), which could be linked to individual differences in emotional regulation strategies or self-consciousness [30]. These findings highlight the value of the GeSTICS dataset in examining how individual differences in speaking style relate to non-verbal expressive patterns, offering a more comprehensive understanding of the factors that shape multimodal communication.

6.2. Limitations and Future Directions

While our study provides valuable insights into the context-dependent nature of speech-gesture relationships, it is important to recognize its limitations and identify avenues for future research. As an exploratory analysis, we focused on a subset of speech features and nonverbal behaviors, but there may be additional dimensions of vocal expression (e.g., intonation contours, spectral features) and bodily communication (e.g., posture shifts, self-adaptors) that play important roles in shaping the dynamics of interpersonal interaction. Future research could leverage the GeSTICS dataset to explore a broader range of features, nonlinear relationships, and temporal dynamics in multimodal communication. Time series analyses could reveal how speech-gesture coordination evolves over the course of an interaction, accounting for complex interactions among different communication channels. While post-game sports interviews provide a rich setting for studying situational factors, it’s important to note that the observed multimodal patterns may be partially shaped by the specific domain. Future work should collect and analyze data from other interaction domains to assess the generalizability of these findings. Finally, the GeSTICS dataset could be used to develop and test computational models for predicting nonverbal behaviors from speech features, or for jointly analyzing and generating realistic multimodal behaviour.

7. CONCLUSION

In this paper, we introduced GeSTICS, a new large-scale dataset of multimodal communication behaviors in post-game sports interviews. Our exploratory analysis of GeSTICS revealed significant associations between speech features, gestures, and contextual factors, highlighting the importance of considering the interaction of contextual factors in co-speech gestures. Specifically, our findings suggest that the context in which interviews take place can affect the interactions between speech and gesture in both phases of a dyadic interaction. By making GeSTICS publicly available and sharing our initial findings, we aim to stimulate interdisciplinary research on the dynamics of natural human communication.

Figure 1:

Example of a Q&A pair in the GeSTICS dataset. Left: The interviewee, depicted as a stick figure, assumes a listening pose while reacting to the interviewer’s question shown in a speech bubble. Right: The interviewee is portrayed in a speaking pose as they provide their response, with their speech represented in a corresponding speech bubble. The dataset also includes metadata on the interviewee’s age, cultural background, and the game outcome.

CCS CONCEPTS.

• Human-centered computing → Human computer interaction (HCI); Gestural input; Natural language interfaces; Collaborative interaction.