The GaMMA corpus of Danish polyadic conversations with gaze speech and motion data in quiet and noise

Abstract

The GaMMA (Gaze, Motion, and Multi-talker Audio) corpus captures the behavior of polyadic conversations among native Danish speakers under both normal and cocktail party conditions. Eleven groups of four normal-hearing participants are recorded while engaged in natural and spontaneous interactions. All conversations were conducted without conversational tasks. Each group was intentionally composed of participants with prior intragroup and interpersonal relations. Gaze and motion data were collected using an optical tracking system and eye-tracking glasses, while speech was recorded via omnidirectional head-worn microphones and binaural hearing aid microphones with low occlusion. Calibrations were conducted before trials and compensation filters were created to account for differences in microphone placements. Processed versions of the audio signals, with background noise attenuated and crosstalk removed, were used to compute speech activity for all participants. The corpus, including both raw and processed gaze and audio data, as well as filters, calibration signals, and speech activity output, is publicly available.

Background & Summary

Research on communication in challenging auditory environments, notably the “cocktail party effect”^1,2, has explored how listeners focus on specific sounds amid background noise, in speech-adverse conditions. Extending this concept, it’s important to examine how conversational dynamics and communicative capabilities shift in multi-speaker environments, particularly under noisy conditions. This understanding is essential for grasping the broader impact of such challenging conditions and applying strategies for auditory compensation or enhancement.

The context of polyadic conversations, involving multiple participants engaged in spontaneous interactions and turn-taking behavior³, presents a particularly rich, yet challenging domain for study. The nuances of such interactions, especially in environments mimicking real-life scenarios, such as those found under cocktail party conditions, offer valuable insights into both verbal- and non-verbal communication patterns, intent, and social behavior^4–8.

Recent advances in multimodal signal processing have leveraged multiple modalities to enhance auditory scene analysis and augmentation. Many researchers have explored areas such as audiovisual stream segregation⁹, speaker diarisation and categorisation¹⁰, and the prediction of turn-taking events^11–16. These studies emphasize the necessity of ecologically valid, multimodal datasets to improve communicative capabilities in adverse auditory environments^17–19.

Related Works

The increasing availability of polyadic conversation datasets has facilitated research into speech interactions in noisy environments. However, these datasets differ in scope, structure, and modality, making each suited for distinct types of analysis.

• DiPCo & CHiME-5: The Amazon Dinner Party Corpus (DiPCo) consists of 39-channel audio recordings of unscripted, interactive conversations in a simulated home environment²⁰. The dataset includes 32 speakers across 10 sessions, featuring both native and non-native English speakers. CHiME-5, which inspired DiPCo, features 20 dinner parties in real home environments with four close friends per session, captured using Kinect devices and binaural microphones²¹. While both datasets capture natural, unconstrained conversations, they include repeated participant groups and external stimuli (e.g., music) at various intervals. Additionally, all recordings are in English, and the background noise conditions are not systematically varied.

• EasyCom: The EasyCom dataset²² offers a highly relevant multimodal corpus, including video, pose, and microphone recordings of talkers in speech-adverse conditions. It features annotated voice activity segments and speech category labels, allowing researchers to examine conversational dynamics. However, EasyCom’s fixed background noise level (71 dB SPL) does not allow for the analysis of adaptive conversational behavior across different noise intensities. Furthermore, its participant sampling strategy does not account for prior relationships between speakers, which can significantly influence conversational patterns.

Dataset Overview

To address these gaps, we introduce a novel, publicly available dataset²³ comprising over 9 hours of polyadic conversations between 44 unique individuals, spread over 11 groups, recorded under varying levels of speech-adverse stimuli. The dataset includes:

• 13 calibrated microphones capturing binaural and close-mic recordings

• Advanced gaze- and head-tracking technology for detailed analysis of head orientation and visual attention

• Raw and processed multimodal signals supporting research in conversational dynamics in noisy environments

• Binary speech activity signals for each talker

A key methodological decision was the deliberate omission of scripted conversational tasks^24–26 ensuring spontaneous interactions without artificial constraints. While structured tasks can enhance participation, they often bias turn-taking dynamics²⁷ and shift gaze behavior from interaction coordination to task management^28,29.

To preserve natural conversational behavior, participants were exclusively native Danish speakers, engaging in unscripted discussions with close friends or family members. The dataset provides individualized binaural voice recordings, egocentric gaze tracking, and relative angle data, allowing for a detailed group- and interlocutor-level analysis of conversational adaptation in challenging auditory environments.

Special emphasis was placed on maintaining the ability to assess the dynamics of naturally occuring conversations on both group- and interlocutor levels. This was done by sampling all participants equally through individualised binaural- and voice recordings, as well as egocentric gaze information and relative angle to each interlocutor within a scene. Furthermore, the restriction of including only native Danish speakers and limiting conversations to solely be between circles of intragroup friends and families was also put in place to facilitate the naturalness of the different conversations.

Methods

Ethics Statement

The study was approved by the Aalborg University Research Ethics Committee (Case No. 2025-505-00609). All participants gave their informed consent prior to participation, and the study adhered to ethical guidelines set by both the university and the collaborating company’s legal department. For the two minors included in the dataset (siblings within a single group), special consent procedures were followed. Both parents were present as active participants throughout the entire trial and provided written informed consent and signed all participation contracts on behalf of the children. Participant data were anonymized, and all procedures complied with relevant data protection regulations. In the event of a consent withdrawal post-publication, the relevant files will be removed from the repository.

Hardware

Four Tobii Pro Glasses 3 were used to track the gaze behavior and record egocentric video footage through the camera embedded in the frames, at the bridge of the nose. Eight infrared Vicon Vero cameras, which were situated on the inside of an elevated truss-frame close to the ceiling, were used to track the position and orientation of marker sets attached to each of the Tobii glasses. The Vicon cameras and Tobii glasses were connected to the same Linksys PoE switch, which was connected to a designated lab desktop computer. A wooden clapperboard with retro-reflective markers attached was used to create cues for aligning audio, motion-, and gaze-tracking. Four Genelec 8010A loudspeakers were placed at fixed positions. Four DPA 4066 omnidirectional headset microphones (close-microphones) were used to record the speech from each participant in a trial. Four pairs of GN ReSound ONE M&RIE (Microphone- and Receiver-in-ear) hearing aids were reconfigured with a wired connection to the microphones. Each pair were fed to a battery-powered preamplifier that supplied power to the microphones. No domes were attached to the receivers and the hearing aid casings were fully left out. Figure 1a shows the resulting set of low-occlusion binaural microphones, and what they look like when worn by a participant, which can be seen in Fig. 1b. The full setup as placed on an individual, including the binaural microphones, can be seen in Fig. 1c.

Fig. 1.

Reconfigured Microphone- and Receiver-in-ear microphones used for binaural recordings and the full user setup. (a) Set of binaural microphones. (b) Microphone placement in ear canal. (c) Full setup with Tobii glasses, close-mic and binaural microphones. The depicted individual consented to have their likeness published.

All of the aforementioned loudspeakers and microphones were routed into an RME Fireface 802 audio interface that was connected to the designated lab desktop computer. A DPA 4061 microphone was fixed to the centre of the table and was used as a reference microphone throughout the recordings. A Zoom H6 audio recorder was used to capture audio from this DPA 4061 microphone. All signals from the binaural- and DPA-microphones were recorded as 32-bit audio, at a sampling rate of 48 kHz. For the reference microphone, everything was recorded as 24-bit audio at 48 kHz. See section Audio Hardware Configuration and Technical Justifications, for more reasoning behind this configuration.

Setup and Room Acoustics

The dimensions of the room used for the experiment were: L: 7.1 m, W: 4.8 m, H: 2.6 m, with a total volume of 88.6 m³. The setup used for the experiment, and the position of the talkers were all confined within one half of the room throughout all recordings and calibrations. A table with a diameter of 1m was set up in the middle of the Vicon area, and four chairs were placed evenly around the table. Figure 2a shows a top view of the layout of the experimental setup with the table and participant positions, as well as microphone- and loudspeaker placements at 1.5 m from the center of the table. Figure 2b shows an example of four people having a conversation during a hardware test. See section Loudspeaker Placement for specific loudspeaker positions.

Fig. 2.

Room Setup. (a) Visualisation of layout and participant placement. (b) Four talkers during a pilot test. Loudspeaker position was not final, during the depicted conversation. All depicted individuals consented to have their likeness published.

The reverberation time of the room was measured at each participant position around the table, with no one present in the room during the measurements. The measuring procedure consisted of playing an interrupted pink noise signal from a Genelec 8010A loudspeaker, at a given participant position (S), towards an adjacent or opposing position (R). A B&K hand-held analyzer type 2250 was used to record the signal at the position R. Figure 3 shows the reverberation time 60 (RT60) according to the standard: ISO 3382-2:2008³⁰.

Fig. 3.

Reverberation time 60 at each location R for all participant position permutations (green) and mean value for all measurements (red, horizontal), with standard deviations (red, vertical). Individual measurement values are slightly, visually offset on the x-axis for legibility.

The RT60 values for the experimental setting fall within the range of 0.35 seconds (minimum) to 0.82 seconds (maximum) between 125 Hz to 4 kHz. In compliance with the DIN 18041: 2016 standard for acoustic quality in small- and middle-sized rooms. The target reverberation time for frequencies 250 Hz and above lie between the A2 (speech/lecture) and A3 tolerance (teaching/communication). Frequencies below 200 Hz exceed the A2 tolerances and fall within the A1 thresholds, as described in³¹.

Audio Recordings and Calibration

To capture what people would hear during the conversations, binaural recordings were a desired implementation. A key consideration, during the planning phase, was to not reduce the verbal self-monitoring capabilities of participants by fully- or partially occluding their ear canals during the conversations³² when adding binaural microphones to the setup. The binaural M&RIE microphones were used as a low-occlusion alternative to other binaural microphones tested and were thus reconfigured to act as in-ear microphones for binaural recordings.

Binaural Calibration

In order to measure the effect of having the binaural microphones placed in the ear canal, the impulse responses from a loudspeaker to the microphones position on a human head were measured. A Brüel & Kjær (B&K) High Frequency Head and Torso Simulator (HATS) Type 5128, with Ear Simulator, type 4620 (ear simulator type 4.3) was used. The binaural microphones were calibrated using an Interacoustics TBS25 test box, and the HATS coupler was calibrated by using a B&K sound calibrator type 4231. A single binaural microphone was inserted in the left ear canal of the mannequin, after which a logarithmic sine sweep ranging from 20 Hz to 20 kHz was played back by a Genelec 8010a loudspeaker at a distance of 1 m from the HATS mannequin, on-axis with the ear of said mannequin. Although the calibration sweep spanned the entire audible frequency range, the loudspeaker used for the measurements has a documented lower frequency limit of 74 Hz (-2.5 dB). To account for this hardware constraint and prevent the eardrum-compensation filter from introducing noise-driven artifacts or artificial gain in the sub-bass range, the filter’s magnitude response was manually pinned to 0 dB (unity gain) for all frequencies below 74 Hz. This strategy ensures the filter remains acoustically transparent in the range where the loudspeaker cannot produce a reliable signal. The sweep was captured by the coupler- and binaural microphone. The insertions and measurements were repeated ten times by a single person. Subsequently, an initial coupler measurement was collected without the binaural microphone present, in order to capture the unobstructed ear canal response of the HATS.

Early reflections were suppressed by multiplying each impulse response with a one-sided raised-cosine window (N = 100 samples,  ≈ 2.1 ms at 48 kHz), starting 150 samples ( ≈ 3.1 ms at 48 kHz) after the direct-sound arrival. For k = 0, …, N − 1, the window is defined as

$$w\left[k\right]=0.5\left(1+\cos \left(\frac{\pi k}{N-1}\right)\right).$$

Figure 4 shows the difference in magnitude response between the resulting windowed impulse responses for an unobstructed ear and an ear with a binaural microphone, as measured by the coupler.

Fig. 4.

Difference between occluded HATS ear-canal response and unoccluded HATS ear-canal response for ten insertions. The figure shows the mean values (red) and  ± 2 standard deviations (dashed black).

This demonstrates the amount of occlusion introduced by inserting the binaural microphones.

Results showed that the mean values do not exceed  ±2 dB in the frequency range of speech. This suggests that the amount of attenuation is negligible for the purpose of the study. Furthermore, 95% of observations (or two standard deviations) were shown to be within  ±1 dB of the mean, for all frequencies. This suggests that the binaural microphone placement had no considerable effect on the amount of occlusion.

The sound pressure at the entrance of the ear canal was deemed less salient for applications such as loudness modeling or for assessing listener intent. Therefore a different reference point was selected, specifically the position of the ear drum, where the acoustic energy is transformed into mechanical vibrations in the middle ear.

Figure 5 shows the difference in magnitude response between the resulting windowed impulse responses for an ear canal obstructed by a binaural microphone, as measured by the binaural microphone and by the coupler. The ear canal resonance at around 4 kHz is clearly visible. The mean values of the differences between the binaural microphone response and the coupler response, in the obstructed ear canal, serve as the eardrum-compensation filter.

Fig. 5.

The magnitude of the eardrum-compensation filter as a function of frequency. The filter (red) adjusts the reference point of the binaural microphones to match that of a HATS coupler, emulating a generalized response of an eardrum. The figure shows the mean values (red) and  ± 2 standard deviations (dashed black).

The eardrum-compensation filter is stored as part of the corpus, as a minimum-phase filter, along with filtered and unfiltered versions of the binaural recordings.

Close-mic Calibration

To account for variation in microphone placement between participants and to estimate the level of each participant, the close-mics were calibrated such that the sound-pressure-level (SPL) of the recordings was related to a reference point 80 cm in front of each test participant. The corresponding calibration was done in three steps. 1) A reference microphone was placed in a B&K anechoic test box playing a 1 kHz tone at 94 dB SPL. The corresponding recording was multiplied with 0.1 to have set the full scale at 114 dB SPL. The RMS value of this signal is denoted as ${\mathrm{Puretone}}_{114dB}^{RMS}$. 2) The same reference microphone was placed 80 cm in front of each participant. Participant p was fitted with a close-mic and asked to read out six Hagerman sentences each, which were recorded with the close-mic and the reference microphone. The corresponding RMS value of the close-mic for participant p is denoted ${\mathrm{Closemic}}_p^{RMS}$ and the RMS value of the reference microphone for participant p is denoted ${\mathrm{Reference}}_p^{RMS}$. This procedure was carried out for each of the 44 participants before the recordings of the conversations. 3) The conversational recording of participant p done with the close-mic was multiplied with the Calibrationfactor_p to determine the sound pressure level at a distance of 80 cm. The calibration factor is given by:

$${\mathrm{Calibrationfactor}}_p=\frac{{\mathrm{Reference}}_p^{RMS}}{{\mathrm{Closemic}}_p^{RMS}{\mathrm{Puretone}}_{114dB}^{RMS}}$$ 1

Audio Hardware Configuration and Technical Justifications

To ensure temporal consistency across all primary speech data, the four close-headset microphones and eight binaural hearing aid microphones were routed into a single RME Fireface 802 audio interface. This configuration ensured that these 12 primary channels shared a common hardware clock, eliminating inter-channel drift between the participants’ individual speech and their respective binaural microphones. This setup reached the physical 12-channel capacity of the interface (3 microphones per participant x 4 participants). Consequently, the DPA 4061 reference table microphone (the 13th channel) was recorded using a separate Zoom H6 recorder. This was an acceptable trade-off because the reference microphone’s primary function was pre-trial calibration and the calculation of gain coefficients for level normalization, rather than the primary conversational analysis where sample-accurate synchronization is critical.

The close-mic recordings were captured using DPA 4066 omnidirectional microphones. Since omnidirectional pressure transducers are not subject to near-field proximity effects, and no high-pass filtering or spectral compensation was applied to the close-microphone signals. To further preserve signal integrity, microphones were positioned off-axis from the participants’ mouths, effectively mitigating plosive-induced artifacts and breath noise at the source. These design choices allowed for a transparent signal chain without the use of physical pop-shields or digital filtering, providing users of the corpus with raw, full-spectrum speech data.

Gaze and Positional Data

Gaze data were captured using Tobii eye-tracking glasses, while head position and orientation were obtained by tracking unique marker-set constellations attached to each pair of glasses for the four individual talkers. Head orientation, position, and gaze vectors were streamed from Nexus to Matlab via the Vicon Datastream SDK. This hardware-level integration ensured that all tracking modalities shared both a unified global coordinate system and a common master clock.

The global coordinate system was defined with its origin (0, 0, 0) fixed at the center of the table, and all positional data are reported in millimeters (mm). The positive Y-axis is oriented between participants P1 and P2, the positive X-axis between P2 and P3, and the Z-axis extends vertically upward from the table surface. Head positional data are reported as absolute coordinates within this global reference frame, where head position corresponds to a centroid of the six-marker set mounted on each pair of Tobii glasses, that is located between-, and slightly in front of the eyes of the respective wearer, as can be seen in Fig. 6.

Fig. 6.

Centroid position for a set of markers on a pair of glasses, worn by a participant. (a) Shows the front view, which clearly displays the eye position, object (centroid) position, and the unique marker set constellation. (b) Shows the same participant, but from an angled perspective. Not the local offset of the centroid in the direction of the blue axis.

Gaze data are provided as normalized unit vectors (0-1 range), representing the direction of visual attention relative to the head position. During preprocessing, gaze vectors were re-normalized following independent spatial interpolation and smoothing to ensure that directional integrity was preserved.

Gaze data were captured at a sampling rate of 50 Hz, while head positional data were explicitly sampled at 200 Hz. The 200 Hz rate was selected because the Vicon camera clocks provide true integer sampling intervals at this frequency (e.g., a nominal setting of 150 Hz would result in non-integer frame timing). This choice additionally ensured a precise 4:1 ratio for subsequent temporal alignment between the 50 Hz gaze data and the positional data.

Temporal alignment between gaze, head position, and audio signals was achieved using a tracked clapperboard with a fully calibrated range of motion. Alignment was performed by measuring the distance and velocity between the two arms of the clapperboard during a clap sequence, where the arms moved toward contact and returned to a closed position. The sample (or frame) at which contact occurred was used to determine the amount of initial data to truncate from the gaze and head-positional recordings. For the audio signals, clap peaks were detected in the close-microphone recordings. Due to manual placement of the clapperboard at the center of the table, between the four talkers, their close microphones, and directly above the reference microphone, small deviations of up to a few milliseconds were observed between peak sample indices across the four channels. To account for this, peak indices were averaged, and the resulting mean sample index was used to define time zero consistently across all audio signals.

Because both gaze and motion data were integrated through the Tobii-Vicon pipeline using the same PC, Nexus software, and Vicon Datastream SDK, the two modalities were inherently synchronized on a shared hardware clock. To facilitate multimodal analysis at a unified temporal resolution, the 50 Hz gaze data were upsampled to 200 Hz using linear interpolation, resulting in a one-to-one sample correspondence between gaze and motion streams.

Drift

Due to the nature of the recording setup and the use of a single desktop PC with an external soundcard running two separate clocks, some drift was assumed to be introduced into the system. For the purpose of determining the drift in the setup used for the data collection, a series of measurements were done. The measurements involved attaching a DPA 4066 omnidirectional microphone to the clapperboard, to ensure that the distance between the two would remain constant throughout all measurements. For each trial, a test-conductor would initiate a clap, leave the room, and then come back at a set time and initiate another clap, before ending the recording. For each recording, the clapper frame zero from the first tracked clap was estimated and the associated audio peak captured by the microphone during the clap were aligned and samples recorded after the second peak were truncated for both signals. Accounting for the different sampling rates (200H˙z and 48000H˙z respectively), absolute value of the total length of the two signals were subtracted from each other. The results of the tests showed an average drift of 5.998m˙s (n = 6, SD = 0.28) after 12 minutes of recording (approximate average length of a conversation per trial). This results in a drift and subsequent desynchronization of one tracked sample (5m˙s) between gaze- and positional- data, and audio in the corpus.

Participants and Procedure

Participants

To facilitate natural conversational dynamics, a purposive sampling method was employed to identify pre-existing social groups. Participants were recruited through personal outreach and social-network-targeted invitations to identify prospective group leads. These leads were asked to propose three additional candidates (friends, family members, or partners) with whom they shared established interpersonal relations and prior conversational familiarity as a group.

All proposed candidates were vetted by the researchers to ensure they met the inclusion criteria. Participants were required to be native Danish speakers (L1), have no diagnosed hearing loss, and have no significant or uncorrected visual impairments. Only candidates fulfilling all criteria received a formal invitation and a comprehensive explanation of the project prior to participation.

A total of 44 participants in groups of 4, totaling 11 groups were included in the experiments. The genders were distributed accordingly: 23 Male (52.3%), 21 female (47.7%) and 0% other. The ages of the participants ranged from 12 to 64 (M = 35.3, SD = 13.54).

Regarding institutional affiliation, the cohort was largely independent of the research facilities. Only 2 of the 44 participants had any professional connection to the institutions involved. One participant was a researcher from an unrelated domain at GN Store Nord who participated together with their family, and one participant was an intern from a different research area. The remaining 42 participants were external volunteers with no connection to the research staff, facilities or company.

Group Constellations

Three nominal categories were established for participants to choose from when asked to best describe their affiliation with the other participants within a group. These categories were: “friends”, “family” or “partner”. A fourth condition labeled “colleague” was present but was never selected and is thus omitted from the data. The individual group constellations can be seen in Table 1.

Table 1.

Group constellations and the overlapping nominal categories for each group.

Group Number	G01	G02	G03	G04	G05	G06	G07	G08	G09	G10	G11
Friends	X	X	X		X				X	X
Family			X	X		X	X	X			X
Partner				X					X

Design and Stimuli

A within-subject test design was applied for the experiment. There were four conditions presented to each group throughout the trials. Each condition introduced a change in the level of external babble noise introduced to the auditory scene, via the loudspeaker array. The babble noise used was generated through the use of the Cafe Restaurant noise generator from mynoise.net, with the Cocktail Party setting applied³³. The signals from all output channels were uncorrelated. An overview of the four conditions are shown in Table 2.

Table 2.

Overview of the external stimuli that all groups were subjected to.

External Stimuli	None	Babble	Babble	Babble
Noise Level	47 dBA	55 dBA	65 dBA	0–65 dBA
Type	Ambient	~Constant	~Constant	Varying (iLombard)

Ambient noise was present in all conditions. The total measured level at the centre of the table separating participants amounted to the respective levels reported in the table.

The noise levels for all relevant conditions were estimated at the centre of the table, by using a sound level meter measuring the sound pressure level at the position. The condition featuring no external stimuli had an ambient level of 47 dBA, with two other conditions featuring levels of 55 dBA and 65 dBA respectively, with a variance of approximately  ±2 dBA. The varying-noise condition applied a piecewise gain envelope to the background babble noise. The stimulus had a total duration of 10 minutes and consisted of two identical modulation cycles. Each cycle comprised gradual level ramps between plateau regions at 0, 45, and 65 dBA. The low-level (0 dBA) plateau between cycles lasted approximately 45 s. Level transitions were implemented as nominal 1 dB increments, resulting in smooth perceptual ramps between plateau levels. This condition is identified as “iLombard” within the dataset’s file-naming convention and data structs, but for the purpose of the documentation, is interchangeable with the “Varying” condition.

Procedure

Before the start of every trial, each group would receive a brief introduction to the study and the setting, in which they were informed what type of equipment they would be wearing during the ensuing conversations. They were also told how long the experiment would last and that they were not allowed to get up and leave, unless they wanted to abort the trial. During the introduction, participants were asked to pick a seat around the table (see previous, Fig. 2 and section “Setup and Room Acoustics”).

Participants were then given a nameplate with a unique number that they were asked to wear, in a manner where it would be visible to the other participants in the group. The number was based on the seating position of the respective participant and was used to denote group- and participant number, as well as to map gaze- and audio correctly. Participants were asked to remain at their original seating choice throughout the entire experiment. The participants were then introduced to the first part of a two-part questionnaire.

Each participant was fitted with a close-mic, a set of binaural microphones, and a pair of Tobii glasses. The fitting was done by the same two co-present test-conductors for all recordings. After the seating and fitting, in order to calibrate the close-mics, each participant was given a random selection of Danish Hagerman sentences from the Danish Dantale II material³⁴. The participants were then asked to take turns reading the given sentences out loud at a distance of 80 cm from the reference microphone, which was placed at the centre of the table. The distance was measured and participants placed accordingly by the test-conductors. No external stimuli was present during the sentence readings. This calibration was only done once, prior to starting the experiment and was performed in order to calibrate the close-mics described in the section Audio Recordings and Calibration. Participants were informed that the test-conductors would leave the room and re-enter approximately every 15 minutes to initiate a change in the auditory scenery. A test-conductor would generate a clap using the tracked clapperboard, before leaving the room. The participants would then be alone in the room with said test-conductors monitoring camera tracking, and audio input- and output levels throughout the entire segment of a condition. The clap generated by the clapperboard would be repeated for every change of conditions and the auditory scenery. After conversations had concluded for all the conditions, the participants were asked to fill out the second half of the questionnaire.

Data Processing

Gaze

Outliers in the gaze vectors were eliminated using a Hampel filter with a window size encompassing 16 samples on either side of the targeted sample, creating a sliding window of 33 samples. Any samples that deviated more than three standard deviations from the window’s median were replaced by the window’s median value. Next, any missing samples were filled in through linear interpolation based on the two adjacent samples. Finally, the gaze vector was smoothed using a one-pole bi-directional recursive filter with a forgetting factor of 0.9. These steps, i.e. outlier removal, interpolation, and smoothing, were performed in Euclidean space for each dimension separately. Due to independent interpolation and smoothing across spatial dimensions, the gaze vectors were then re-normalised as unit vectors, to account for scaling caused by these independent processes. The relative gaze angle to other participants was then computed using the participant’s processed gaze vector and the vector from their head position to the other participants³⁵.

Audio

For the three conditions featuring external babble noise, a speech enhancement model based on the DEMUCS framework³⁶ was used to remove the added noise from each microphone signal. An adaptive filtering technique, namely a normalized least mean squares (NLMS) algorithm³⁷, was employed in order to attenuate cross-talk from other interlocutors in the vicinity, for each individual close-mic and binaural microphone. This approach resulted in speech signals with reduced audio bleed from other undesired talkers, for segments with both single- or multiple talkers acoustically overlapping simultaneously. A voice-activation detection (VAD) algorithm was used to identify and label segments of active speech for each talker, producing a binary representation of voice production onset and offset³⁸.

Data Records

The data is stored in structured formats for ease of access:

• Audio: Stored as .wav files, available in both raw and processed forms.

• Annotations and Tracking Data: Provided in structured .csv and .json formats.

• Calibration Data: Contains constants for each participant, allowing for precise alignment.

Further details on accessing and working with the dataset, including software recommendations and file handling tips, can be found in section: Usage Notes.

The corpus consists of multimodal recordings from polyadic conversations among native Danish speakers under both quiet and noisy conditions. The dataset includes audio, gaze, and head motion data, providing both raw and processed signals for analysis, model development, and training purposes. It is structured into the main categories found in Table 3.

Table 3.

Overview of GaMMA corpus data categories.

Category	Description
Binaural Audio Raw	Stereo recordings from binaural hearing aids, sampled at 48 kHz.
Closemic Audio Raw	Individual omnidirectional microphone recordings at 48 kHz.
Closemic Audio Processed	Denoised and crosstalk-suppressed versions of close-mic recordings at 16 kHz.
Gaze and Head Position	Motion capture and gaze tracking data at 200 Hz.
Voice Activity Detection (VAD)	Speech activity annotations including onset and offset times.
Calibration Data	Audio calibration signals and compensation filters used to normalize recording conditions.

Dataset Composition

The dataset consists of more than 9 hours of recorded polyadic conversations from 44 unique participants across 11 groups of four. Each group participated in four different levels of speech-adverse background noise, allowing the study of conversational dynamics in diverse acoustic conditions.

Multimodal Data Collection

The dataset captures various aspects of conversation dynamics, including speech, gaze, head orientation, and the surrounding acoustic environment. The recorded data consists of both raw and processed signals for different research applications.

Speech and Acoustic Data

• 13 calibrated microphones:

  • 4 individual close-talk microphones
  • 8 binaural microphones
  • 1 reference microphone
• Speech recordings are available as both raw and processed signals, with processed versions including:

  • Crosstalk reduction and background noise attenuation
  • Segmented speech regions identified by voice activity detection (VAD)
  • Filtered binaural recordings mimicking generalized eardrum responses

Gaze and Head Orientation Data

• Gaze and head orientation were captured using an optical tracking system and four gaze-tracking glasses (one per participant).

• The dataset includes both raw tracked gaze data and processed relative gaze vectors, providing information on gaze angles and head movements throughout the recorded conversations.

Gaze and Head Position

Each participant’s gaze and head-position data are provided in MATLAB .mat files (one struct per condition) and as. csv files (one file per participant, per condition). Each file contains synchronized time series sampled at 200Hz, representing head kinematics from Vicon motion capture and gaze direction from Tobii Pro Glasses 3.

Variables in the Gaze.mat files

Coordinate system and units

All positional and directional quantities are expressed in a global, right-handed coordinate system defined by the Vicon capture volume:

• Origin (0, 0, 0): centre of the table.

• + x: direction from the P1–P4 axis toward the P2–P3 axis.

• + y: direction from the P4–P3 axis toward the P1–P2 axis.

• + z: vertical upward.

All positions and distances are in millimetres. All angular quantities are expressed in radians. Head position corresponds to the rigid-body origin of the Tobii glasses, located near the midpoint between the eyes. Gaze direction vectors are expressed in the global coordinate frame after outlier removal, interpolation, smoothing, and renormalisation. See Table 4 for an overview.

Table 4.

Variables contained in each Gaze.mat file.

Field	Dimensions	Units	Description
position	3 × T	mm	3D head position of the participant’s Tobii glasses in global coordinates.
quaternion	4 × T	—	Unit quaternion describing head orientation.
gazevector	3 × T	—	Unit 3D gaze direction vector in global coordinates.
gazeAzimuth	1 × T	rad	Horizontal gaze direction, atan2(y, x) from gazevector.
gazeElevation	1 × T	rad	Vertical gaze direction relative to the horizontal plane.
gazeDistance	1 × T	—	Magnitude of the interpolated gaze vector prior to renormalisation; ideally  ≈ 1, not a physical distance.
toP#vector	3 × T	mm	Vector from this participant’s head position to participant #.
toP#distance	1 × T	mm	Euclidean distance to participant #: ∥toP#vector∥.
toP#azimuth	1 × T	rad	Horizontal direction to participant #.
toP#elevation	1 × T	rad	Vertical direction to participant #.
toP#relativeGazeAzimuth	1 × T	rad	Signed horizontal angular offset between gaze and target direction, wrapToPi(gazeAzimuth − toP#azimuth).
toP#relativeGazeAngle	1 × T	rad	Full 3D angular separation between gaze and the direction to participant #, $\arccos \left(\mathbf{g}\cdot \mathbf{d}\right)$.

The symbol # refers to the three other participants in the group.

Processing notes

Gaze data were originally sampled at 50Hz and were aligned to the 200Hz Vicon head-position timeline through linear interpolation. Because interpolation and smoothing were performed independently on each axis, the intermediate gaze vector may temporarily deviate from unit length; this deviation is captured in gazeDistance and corrected during renormalisation before angular measures are computed.

Loudspeaker Placement

Four loudspeakers were positioned around the conversational space and used to reproduce external babble noise. In the coordinate system defined above, the table center corresponds to (0, 0, 0). The loudspeakers were arranged at the cardinal directions in the horizontal plane, at a nominal radius of approximately 1.5 m from the table center and at approximately table height (≈0.5 m).

Nominal loudspeaker positions relative to the table, (with all values in mm) were therefore:

• West: (-1500, 0, 500)

• East: (1500, 0, 500)

• North: (0, 1500, 500)

• South: (0, -1500, 500)

Because the local coordinate origin was defined manually for each recording, small session-specific deviations in absolute loudspeaker coordinates exist; This is in the order of a few centimeters.

Technical Validation

Audio Quality

To evaluate the quality of the individual unprocessed and processed speech signals, the objective speech quality metric Deep Noise Suppression Mean Opinion Score (DNSMOS) was utilised³⁹. The open-sourced DNN model, DNSMOS P.835 was used to generate the DNSMOS scores. The model is a modified version of ITU-T P.835⁴⁰, that yields three scores on different parameters of a signal; These scores are: speech quality (SIG), background noise quality (BAK) and overall quality (OVRL).

Given that certain polyadic speech activity patterns may pose greater challenges for audio processing algorithms, we extended our evaluation to assess the performance across different classifications of speech activity. VAD envelopes from the denoised and debled signals were used to assess speech activity at the group level. Four different states of speech activity were defined, i.e.: No activity, where no speech from any talker is detected. “Wearer Only”, where the person wearing the corresponding close-mic is the only person speaking. Interlocutor only, where one or more talkers, other than the respective close-mic wearer, is speaking. “Overlapping”, where both the respective close-mic wearer and one or more other talkers are speaking. Segments were created and then sorted by their states and then concatenated, splitting them into smaller signals consisting only of input matching the criteria of a respective state. Three versions of each segmented signal were saved, namely an unprocessed signal, a denoised signal using the DEMUCS framework, and then a debled version of the denoised signal using the previously mentioned NLMS algorithm.

Figure 7 shows the DNSMOS scores for segments of speech where only the wearer is talking, as well as segments of overlapping speech, for the two conditions with the most- and least amount of added external background noise, namely the Quiet (a) and 65 dBA (b) conditions.

Fig. 7.

(a) DNSMOS scores for all talkers at the different stages of the post-processing pipeline, for the quiet condition without external noise added, separated by speech activity state. (b) DNSMOS scores for all talkers at the different stages of the post-processing pipeline, for the condition with 65 dBA of external babble noise, separated by speech activity state. The three scores represented are: background noise quality (BAK), speech quality (SIG) and overall quality (OVRL).

The DNSMOS scores seen in Fig. 7a revealed that for the segments of speech in the Quiet condition, both the denoising and debleeding processes increased the overall perceived quality of the signal. However, for the wearer only segments, where only the close-mic wearer is talking under quiet conditions, only a negligible effect is present. This suggests that the background noise did not have a significant impact to offset any potential artifacts or speech degradation introduced by the processing. Upon manual inspection of multiple samples by several evaluators, no artifacts were perceived, and no significant change in quality was experienced. In the 65 dBA condition, shown in Fig. 7b, the scores showed a larger effect on the perceived audio quality from denoising, with debleeding primarily providing decreased variability, suggesting a more uniform perception of quality across various samples. The smaller impact of debleeding on the DNSMOS scores can likely be attributed to the fact that the model does not distinguish between wanted- and unwanted speech, thus not crediting the removal of interfering speech as an improvement in quality. The same reasoning applies to the wearer only segments, where only small differences were incurred by the debleeding process.

Audio Post-Processing Benchmarks

To evaluate the specific impact of each post-processing stage, four distinct speech-activity scenarios were selected as benchmark cases. These scenarios were chosen to isolate the performance of the denoising (DEMUCS) and debleeding (NLMS) algorithms in their ideal operational environments:

1. Noise-Only Validation (65 dBA): Segments with no active speech in the highest noise condition were used to measure the maximum efficacy of the denoising algorithm.

2. System Baseline (Quiet): Segments with no speech in the quiet condition were used to gauge the impact on baseline hardware noise (e.g., PC and camera equipment).

3. Crosstalk Isolation (Quiet): “Interlocutor Only” segments were selected to specifically evaluate the removal of acoustic bleed from other talkers (debleeding).

4. Signal Integrity (Quiet): “Wearer Only” segments were analyzed to ensure that the primary speech signal remains uncorrupted when minimal processing intervention is expected.

Table 5 shows the amount of signal attenuation due to the different levels of processing for these segments.

Table 5.

Aggregated mean RMS values for all signals of four segments (columns), for all three versions of the signal both pre- and post-processing (rows).

Condition	Post-Processing	RMS - Raw	RMS - Denoised	RMS - Denoised & Debled
No Activity, 65 dBA		−59 dBFS	−70 dBFS	−73 dBFS
No Activity, Quiet		−63 dBFS	−71 dBFS	−74 dBFS
Interlocutor Only, Quiet		−60 dBFS	−69 dBFS	−77 dBFS
Wearer Only, Quiet		−48 dBFS	−52 dBFS	−52 dBFS

Values are measured in dB relative to full-scale.

The segments with no activity and 65 dBA of background babble noise primarily served to assess the efficacy of the denoising algorithm. As expected, these segments exhibited the largest amount of attenuation from the denoising process.

The segments with no activity in quiet conditions were used to gauge the effect of post-processing on baseline acoustical noise from sources such as the desktop PC controlling the experiment, Vicon cameras, and the network switch. These segments also functioned as reference signals, where minimal effect from post-processing was anticipated. However, an attenuation of 3 dB was observed in these segments, after debleeding.

The interlocutor only segments in quiet conditions were mainly selected to evaluate the denoising algorithm, where the largest effect was expected. The results aligned with these expectations, as an attenuation of the signal by 9 dB could be seen.

The wearer only segments in quiet conditions were expected to show the smallest amount of attenuation from post-processing, as the only unwanted energy would originate from ambient acoustical noise present during the recording. Again, the results met expectations: a small amount of attenuation (4 dB) was observed after removing background noise, and no changes were detected after debleeding, as no bleed was expected during these segments.

Behavioral Data Quality and Missing Data Ratios

To evaluate the integrity of the multimodal corpus, missing data ratios for gaze and positional modalities were calculated for all 176 individual data streams prior to interpolation. For motion tracking, the data is exceptionally robust, with a global mean error ratio of 0.28%. This high reliability is attributed to the redundancy of the six-marker sets attached to the tracking glasses; the Vicon Nexus system was able to maintain 100% object labeling even during intermittent sensor occlusion, resulting in 0.00% error for 92% of the recorded streams. The gaze tracking modality exhibited a global mean missing data ratio (NaNs) of 11.9%. These NaNs represent a combined metric of physiological eye blinks and technical tracking interruptions (e.g., from use of corrective lenses, or extreme head angles). One notable outlier, participant P1 in Group 04, showed a significantly higher mean error of 42.5% due to being afflicted with wet age-related-macular-degeneration, which hindered the eye-tracker’s ability to maintain a pupil lock, which was only discovered after the trials had concluded. Because individual differences in blinking frequency and physiology contribute to these ratios, researchers are advised to consult the per-trial quality metrics. A comprehensive error table for all participants and recordings is provided as Supplementary Table S1.

Questionnaire Data

The purpose of the questionnaire data was to quantitatively verify that the recording equipment and laboratory environment did not significantly impede the natural conversational behavior that the corpus intends to capture. These metrics serve as a behavioral validation of the study’s ecological validity, ensuring that the methodological focus on unscripted interactions and low-occlusion hardware effectively preserved spontaneous group dynamics. The dataset consisted of responses to two constructs measured by a Likert-type questionnaire on a 7-point scale: perceived intrusiveness (the effect of being in a controlled environment and wearing equipment) and perceived naturalness (how natural the participant feels in the environment). Each construct was measured using 4 items, with responses collected from all 44 participants (see Supplementary Table S2 for construct items and response anchors). The means suggest generally high levels of perceived intrusiveness and naturalness. Skewness values reveal that both distributions are negatively skewed, indicating a tendency toward higher scores. The kurtosis values show that the distribution of Perceived Naturalness is more peaked, while Perceived Intrusiveness has a flatter distribution compared to a normal distribution. See Table 6 for descriptive statistics.

Table 6.

Descriptive statistics for Perceived Intrusiveness and Perceived Naturalness.

Construct	Mean (M)	SD	Min	Max	Skewness	Kurtosis
Perceived Intrusiveness	5.32	0.82	3.50	6.75	−0.37	−0.66
Perceived Naturalness	6.02	0.62	4.25	6.50	−1.35	0.77

To guide the choice of appropriate statistical analyses for evaluating the two constructs we performed three widely recognized tests for normality: the Shapiro-Wilk (SW), Kolmogorov-Smirnov (KS), and Anderson-Darling (AD) tests. For each construct, we calculated the mean response for each participant across all items within the construct, resulting in 44 mean values (one per participant). These mean values were then used in the normality tests for each construct.

Perceived Intrusiveness

The SW test results for perceived intrusiveness did not reject the null hypothesis of normality (W = 0.97, p = 0.21), suggesting that the distribution of scores is consistent with a normal distribution. This finding was supported by the AD test, which yielded a statistic of A² = 0.53, well below the critical values at conventional significance levels, indicating no significant departure from normality. In contrast, the KS test showed a deviation (D = 0.11, p = 0.65), suggesting there is no significant difference from a normal distribution at commonly used significance levels. The overall evidence from the SW, AD and KS tests indicates that the data for perceived intrusiveness is normally distributed.

Perceived Naturalness

Conversely, the normality tests for perceived naturalness consistently indicated significant deviations from a normal distribution across all three assessments. The SW test revealed a substantial departure (W = 0.778, p < 0.001), and the AD test statistic (A² = 3.64) exceeded all critical thresholds, underscoring this conclusion. Similarly, the KS test affirmed these findings with a statistic of D = 0.24 and p = 0.01. Based on these results, further analysis was conducted using non-parametric methods to account for potential non-normality in the data, particularly in the tails.

To assess the internal consistency of the items measuring perceived intrusiveness, we calculated Cronbach’s alpha and obtained a value of 0.72. This statistical measure of reliability is typically considered acceptable, indicating that the internal consistency is sufficiently reliable. The Cronbach’s alpha for the perceived naturalness construct was found to be 0.62. This value is typically considered to be in the “questionable” range for reliability, indicating that the internal consistency of the measure could potentially be improved⁴¹.

Spearman’s rank correlation analysis revealed a moderate negative correlation between perceived naturalness and perceived intrusiveness (ρ = −0.39, p = 0.01). This suggests that higher perceived naturalness is associated with lower perceived intrusiveness. In other words, as users perceive the system as more natural, they experience it as less intrusive.

Usage Notes

The GaMMA corpus²³ provides a rich multimodal dataset for analyzing polyadic conversations among Danish speakers under quiet and adverse speech conditions. This section provides practical guidelines for effective use of the dataset.

Accessing the Data

The dataset is structured in well-organized directories, with separate folders for different data modalities. Users should familiarize themselves with the folder structure and naming conventions to efficiently locate required files.

Each dataset file follows a standardized naming convention that includes:

Group#_P#_Binaural_LR_Condition#_Signal, where:

• Group# refers to the participant group (e.g., Group01 to Group11).

• P# identifies the participant (P1 to P4).

• Binaural_LR indicates left/right order for binaural recordings.

• Condition# represents the recording condition (e.g., Quiet, 55dB, 65dB, iLombard/Varying).

• Signal signifies whether it is the raw- or processed versions of the signal.

File Formats

• Audio Files: Available in .wav format (single- and multi-channel).

• Gaze and Motion Data: Provided in .csv and .mat formats, requiring either standard CSV processing tools or MATLAB/Python with the scipy.io library.

• Calibration Data: Stored in .csv and .mat formats for ease of use across different platforms.

Recommended Software

To process and analyze the data effectively, users are advised to use:

• Audio Analysis: Python libraries such as librosa or MATLAB.

• Data Handling: Pandas and NumPy for CSV files.

• MAT File Processing: SciPy in Python or MATLAB.

• Playback and analysis: any Digital Audio Workstation, such as Audacity (free) or Adobe Audition.

Calibration and Compensation

Users can apply the provided binaural compensation filters to adjust microphone placements and ensure consistency across recordings. The calibration signals and gain factors (available in the calibration folder) are already applied to both raw and processed signals but can be used to revert signals to their original levels if needed.

Known Limitations

• Incomplete Calibration for Group09: The calibration signals for Group09 are incomplete due to missing reference microphone data. As a result, these signals are retained at their originally recorded levels without post-hoc adjustments.

• Background Noise in Raw Audio: The raw audio files contain background noise, which may necessitate additional preprocessing depending on research objectives. Processed versions aimed at reducing crosstalk and background noise are included, but some audible artifacts remain, particularly in periods when multiple talkers were active. Researchers should evaluate the suitability of these processed files for their specific analyses.

Temporal Alignment Considerations

• Clock Drift Between Audio and Positional Data: The recording setup, which relied on separate clocks, introduced a drift between audio and positional data, averaging  ~ 6 ms over 12 minutes. While this desynchronization is relatively small, it could impact analyses requiring precise temporal alignment, especially for high-speed events.

  • The smallest relevant time constant for planned analyses falls between 20–40 ms, corresponding to saccadic gaze behavior. The observed drift is unlikely to significantly interfere with most analyses.
  • However, for measurements involving precise saccade timing, even a 5 ms discrepancy could influence the interpretation of rapid eye movements relative to the audio or positional data.
  • While this drift is largely inconsequential for broader timing windows, researchers should consider post-hoc corrections for analyses requiring precise synchronization of saccadic events.

Conversational Dynamics and Participation Imbalance

• Uneven Participation: Observations indicate that participants engaged freely in conversation, as evidenced by the inclusion of personal topics and explicit language in the dataset. However, due to the purposive sampling strategy and the lack of structured conversation tasks, participation was uneven in certain conditions.

  • This imbalance was often topic-dependent and, in some cases, spanned across conditions. For example, when a participant told a personal story, it often led to a decrease in speech activity from others in the group, regardless of the acoustic environment.
  • The effect was more pronounced in casual conversations, where participants did not feel the need to perform, compared to formal settings (e.g., business meetings), where efficiency was a higher priority.

While this natural variability reflects ecological conversation dynamics, researchers analyzing speaker turn-taking or participation balance should account for topic-driven influences. Future studies may explore non-intrusive conversation tasks that promote more balanced participation while maintaining ecological validity in polyadic settings.

Author contributions

The development of the GaMMA corpus was a collaborative effort between Aalborg University and GN Store Nord. The contributions of each author are outlined below:• Mark Dourado - Conceptualized the study, designed the data collection protocols, supervised the data acquisition process, and contributed to the development of the analysis framework. Led the manuscript writing and project coordination, as well as implementing software and building the lab for the study.• Jesper Udesen - Provided expertise in audio signal processing, assisted in developing calibration and compensation methods. He also played a key role in ensuring the technical quality of the recordings.• Henrik G. Hassager - Contributed to the experimental design and data validation, Assisted in the review and revision of the manuscript. He also implemented software, as well as identifying and procuring the necessary hardware for the study.• Stefania Serafin - Provided guidance on multimodal data integration and supported the interpretation of gaze and motion data. Reviewed and provided critical feedback on the manuscript, particularly in the discussion and broader impact sections. She also ensured compliance with ethical and technical academic standards.

All authors reviewed and approved the final manuscript. For further inquiries or technical support, please contact the corresponding authors.

Data availability

The GaMMA corpus is available via Zenodo at: 10.5281/zenodo.18343509²³.

Code availability

The custom code used to generate and process the data described in this manuscript is proprietary and owned by GN. Due to company policy, it cannot be publicly released.

Competing interests

The authors declare no competing interests.

Supplementary information

The online version contains supplementary material available at 10.1038/s41597-026-06851-x.