Strength in Numbers, Power in Subjectivity: Scalable Modeling of Individual Annotators for Emotion Recognition Within and Across Corpora

Abstract

Emotion expression and perception are nuanced, complex, and highly subjective processes. As a result, when multiple annotators label emotional data, the resulting labels generally contain high degrees of variability. This variability is often addressed by averaging annotator labels into a single averaged or majority-voted ground-truth label. However, an averaged label may mask variation among the perception of individuals, which is at the core of subjective perception. This is an issue because inter-annotator variability contains valuable information about both the sample being evaluated and the annotators themselves. Previous works have proposed methods to retain this variability by predicting distributions of annotator perception, rather than a single label. However, distributions lose the information about the individual annotators. Other approaches have instead learned models of individual annotators. Yet, computational complexity limited these approaches to considering only small sets of annotators. In this paper, we present a new approach that learns to efficiently model arbitrarily large numbers of individual annotators. We demonstrate the efficacy of this approach, compared to previous state-of-the-art approaches, and the insights that can be gleaned once annotator variability is retained. For the first time, it is possible to ask when it is beneficial to model every single annotator, as an individual, vs. when it is beneficial instead to model a more limited number of annotators who have provided ample data.

I. Introduction

Expressions of emotion are nuanced, complex, and subject to individual differences in perception. When multiple people are asked to evaluate the same emotional display, they often will report different labels. Standard approaches in speech emotion recognition (SER) mitigate label variability by simplifying the emotion prediction problem by learning to estimate a single averaged or majority-voted ground-truth label, often referred to as the consensus label. However, this simplification overlooks the nuance inherent in emotion perception [1]. As a result, the variability in the labels is lost [2].

Yet, variability has been shown to be a useful signal [3]-[6]. Specifically, inter-annotator variation has been shown to provide useful information for predicting the consensus opinion of the emotion in speech [2], [3]. Disagreement between annotators does not make any annotator necessarily wrong [2], but combining their annotation with other annotators may suppress minority opinions [7]-[9]. These issues lead to challenges with generalization, a difficult task for SER, with many pre-trained models exhibiting degraded performance on new datasets [10]. Previous works have identified many reasons for this such as recording condition, label schema, rater bias [11], [12], but it is often overlooked that the annotators have changed when comparing across datasets. This is because the study of individual annotators’ perception have been limited to small groups of the annotators present in a given dataset who labeled the most data, due to computational complexity [8], and generally have not considered cross-corpus perception [8], [13]-[17]. Deploying an individual annotator model in a cross-corpus setting provides a unique way to investigate the impact of annotator-shift on the generalizability of SER models.

In this work, we present a novel approach that enables the efficient modeling of a large number of individual annotators (10k+ annotators). If needed, these predictions can be accurately aggregated to create a single ground-truth, mirroring standard SER approaches, or to create a distribution of annotator perception, mirroring distribution-focused SER approaches [2]. We compare our approach in which we learn annotator-specific prediction heads for all annotators to several baselines. Our baselines cover models that learn prediction heads for only a subset of annotators, make consensus predictions, or learn a shared underlying distribution using soft labels. We then frame cross-corpus generalizability in SER as an annotator-shift problem, where changes in annotator populations induce systematic shifts in labeling across datasets. Fundamentally, our approach enables the modeling of individual variation while still retaining the ability to summarize annotator variance and uncertainty.

We find that annotators can be accurately modeled as individuals. We demonstrate that it is important to consider and adapt to the individual annotators in a new database (annotator adaptation) when deploying models cross-corpus rather than just adapting to changes in the acoustics (dataset adaptation). Our work enables, for the first time, the learning of every annotator in very large groups of annotators (e.g., 10k+ annotators in MSP-Podcast [18]) regardless of the number of annotations provided without excessive computational resources, and demonstrates that the change in annotators plays a significant role in the generalizability problem of SER.

II. Related Work

A. Dimensional Emotion Recognition

Emotion is commonly discussed as felt, expressed, or perceived emotion. Felt emotion refers to the internal affective state experienced by an individual person [19]. Expressed emotion refers to the external behavior/physiological attributes of an individual that are manifested due to their affective state (e.g., facial expressions) [20], [21]. The perceived emotion is how these external behaviors/attributes of an individual are interpreted by a third-party observer [22]. Prior work in emotion recognition indicates that annotations of felt and perceived emotion diverge [19]. Generally, emotion recognition datasets contain perceived emotion annotations, which are annotated by outside observers [19], [23]. In this work, we focus on the prediction of multiple annotators’s ratings of their perception of the emotion present in utterances of speech.

Two popular emotion theories are the Basic Emotion Theory, and Dimensional Emotion Theory. In Basic Emotion Theory categories (e.g., anger) are used to describe emotion [24], while in Dimensional Emotion Theory emotion is described in a multi-dimensional continuous space [25]. Categorical emotion prediction presents additional difficulties in annotation due to the ambiguity of label meanings (e.g., irritated and angry) [26]. Additionally, complex emotions become hard to reduce to a single category [27].

For these reasons we focus on Dimensional Emotion Theory, where emotion is measured in a continuous, multi-dimensional affective space. Commonly this is represented as a two-dimensional space made up of valence and activation (arousal) [25], [28]. Valence represents the positive or negative sentiment of the emotion, while activation is a measure of how energized or calm the emotion is [24], [28]. since annotators rate observable emotional features on a scale for each dimension [27].

B. Speech Emotion Recognition

The traditional approach is to predict consensus ground-truth values from speech [29], [30]. In this setting, individual annotations over a single sample are averaged, and the average itself becomes the label. The number of annotations on a sample varies significantly over different SER datasets. For example, large crowd-sourced datasets can have upwards of 30 annotations on some samples [18], [31], while others may only have a maximum of 6 annotations on each sample [32].

Dimensional SER models are often trained in a multi-task manner for regression to predict the dimensions of activation, valence, and dominance [33]-[35]. This multi-task approach is often taken in this case as it has been shown to provide regularization and improve generalization [34]. Fusion of text and acoustic information into a shared space before activation and valence prediction in a multi-task fashion has further been shown to improve dimensional emotion prediction [36]. Our model is based on a bimodal multi-task approach.

The use of large pre-trained transformer models such as Wav2Vec2 [37] and WavLM [38] has become prevalent in SER [29], [39], [40]. Additionally, correlation-based metrics and loss functions have become common in dimensional SER regression tasks [29], [30], [41], [42] due to correlation-based loss functions demonstrating better ability to model the dimensional space compared to other common regression loss functions in SER [43]. For this reason we make use of correlation-based metrics and loss functions in this work.

C. Capturing Label Variability in SER

Previous research has demonstrated that predicting the perception uncertainty (arising from the subjectivity of emotion annotation) can improve model performance [3]. Han et al. explored the introduction of inter-annotator standard deviation as a multi-task target to improve emotion recognition performance [3]. This approach highlights the importance of retaining at least some information from all annotators during SER.

Previous research has also investigated prediction of label distributions. The benefit is that the distribution inherently conveys variability and can be collapsed to a single label (e.g., the average). The challenge is that there are often insufficient numbers of evaluations, critical for creating these distributions. There have been empirical solutions proposed [2], while other research efforts have proposed that given an audio sample, each annotator’s perception is drawn from a sample-specific underlying distribution [5], [6]. However, in all cases, individual annotator labels are not retained.

Zhang et al. proposed an approach to capture distributions in the valence-activation space [2]. They first up-sampled the data and then smoothed the resulting annotator distribution using Kernel Density Estimation (KDE) via Diffusion. The smoothed KDE distribution was then divided into a variable number of bins. The models were trained to predict these binned distributions.

Sridhar et al. proposed a semi-supervised learning approach implemented using variational autoencoders pre-trained on unlabeled speech [5]. The goal was to learn a latent space representation of the speech input. They sampled the latent vector and used these samples as input to a regression model. They predicted both the consensus label alongside a soft-label prediction.

Wu et al. proposed an approach entitled Deep Evidential Emotion Regression (DEER). DEER leverages information from all annotators to model uncertainty, estimating emotion predictions alongside data/model-related uncertainty [6]. Annotations for each sample are assumed to come from an unknown Gaussian distribution. The model predicts the parameters of a normal-inverse gamma prior, trained to maximize the likelihood of each annotation. The predictions for the parameters for each sample can be used to calculate the sample mean (emotional attribute prediction), the variance (total uncertainty), the data-related uncertainty, and the model-related uncertainty. The model makes predictions for activation, valence, and dominance. Additionally, the authors incorporated a reject option using the model’s predicted uncertainty values, highlighting how important it can be to leverage information contained within the variation of the original annotations. The authors found state-of-the-art results for MSP-Podcast (release 1.8) and IEMOCAP. In this work we compare against the 2D KDE approach [2] and DEER [6].

D. Individual Annotator Prediction

Alternatively, subjectivity and variability of annotators can be incorporated by learning annotator-specific models/predictions. Kocoń et al. demonstrated that annotator-specific embeddings could be used to personalize model predictions and capture the bias of individual annotators. They introduced four methods for encoding annotator information into the model, including a one-hot annotator embedding. This one-hot embedding was an annotator ID vector concatenated to the model input. They found that this led to improved textbased emotion predictions [44]. Davani et al. used an encoderbased model trained for a binary categorical text emotion recognition task using a dataset containing 82 annotators. The estimates of the annotator-specific model were aggregated to predict the majority ground-truth labels. They found that this approach outperformed a baseline that was trained to predict the consensus ground-truth [8]. Stappen et al. demonstrated that the fusion of individual annotator predictions can perform similarly to traditional consensus-only training [13].

Hayat et al. trained a multimodal model to make individual annotator predictions, alongside a consensus prediction. They experimented with this model on three tasks with no more than seven individual annotators, most relevantly on four-class emotion recognition on IEMOCAP [14]. They found that learning the consensus label alongside the individual labels improved both the individual predictions and the consensus prediction.

Chou et al. demonstrated that categorical emotion recognition performance could be improved by incorporating inter-annotator information. They trained separate models on a target annotator, the consensus label, and the soft-label of the other (non-target) annotators. These separate models are frozen and their final representations are concatenated through additional linear layers to learn the target annotator’s perception [15]. Ando et al. modeled up to 155 annotators for categorical emotion classification on MSP-Podcast. The authors modeled the 154 annotators who had labeled more than 100 samples. Then, they grouped the remaining annotators into a “rest” annotator [16]. However, the accuracy of the “rest” annotator was not reported. The authors then investigated prediction of unseen annotators within-corpus using held out annotators, and found that given an enrollment set for the annotator they could perform adaptation to estimate the annotator-specific parameters to outperform conventional consensus approaches.

The central challenge is that annotator-specific training is resource-intensive and becomes increasingly infeasible when extended to large numbers of annotators [8]. As a result, studies that predict individual annotators tend to be restricted to populations of annotators that number at most 155 [16]. While this is a large number of annotators modeled, it is substantially smaller than the complete set of over 10,000 annotators on MSP-Podcast and requires grouping sparse annotators that may have diverging perceptions into one group.

An alternative approach is to first identify consistent groups of annotators and to learn models that accurately capture the group. This reduces the number of ensemble models needed. Upadhyay et al. introduced Rater-Perception Coherency-based (RPC) learning. RPC first clusters annotators based on how their categorical labels compare with the majority vote label, producing groups that are majority-perception (majority-vote-aligned) and minority-perception (diverge from the majority-vote) [17]. The multi-task model then jointly learns to estimate the perception for each group, in addition to a consensus label for all annotators. While this work addresses the computational efficiency challenges of ensemble models, annotator predictions are limited to a small number of perception groups and the perception of individual annotators is not retained. Further, we note that (RPC) is not directly applicable to dimensional emotion recognition (which we focus on) as this approach is designed for clustering over categorical emotion labels, rather than dimensional labels.

The models discussed in this overview do not individually model the annotators that have annotated a small number of samples (highly related works are summarized in Table I). In this work, we introduce an efficient method that allows us to model large numbers of annotators, evaluating for the first time the efficacy of directly modeling annotators (even sparse annotators) vs. using “similar” annotator-specific models trained on well-represented annotators to capture the less-well-represented annotators.

TABLE I:

Comparison of previous approaches and our proposed approach. A/V is arousal and valence, A/V/D is arousal, valence, and dominance. ✓and ✗ mean “Yes” and “No” respectively. “Consensus” indicates a primary focus of the model was consensus modeling. “Soft-label” indicates the models use/provide more than just a single (e.g., averaged) label.

Method	Modality	Target	Prediction Types		Annotator-Specific Prediction		Baseline
			Consensus	Soft-Label	Annotators Learned	Restrictions
Han et al. [3]	Speech	Dimensional A/V	✓	✓	N/A	N/A	✓*
Zhang et al. [2]	Speech	Dimensional A/V	✗	✓	N/A	N/A	✓
Sridhar et al. [5]	Speech	Dimensional A/V/D	✓	✓	N/A	N/A	✗
Wu et al. (DEER) [6]	Speech	Dimensional A/V/D	✓	✓	N/A	N/A	✓
Kocoń et al. [44]	Text	Categorical	✓	✓	All	Not directly applicable to speech.	✗
Davani et al. [8]	Text	Categorical	✓	✓	≤ 82	Computationally expensive.	✓*
Stappen et al. [13]	Speech & Video	Dimensional A/V/D	✓	✓	5	Uses continuous annotations.	✗
Hayat et al. [14]	Speech & Video	Categorical	✓	✓	≤7	Computationally expensive.	✗
Chou et al. [15]	Speech	Dimensional A/V	✓	✓	4	Computationally expensive.	✗
Ando et al. [16]	Speech	Categorical	✓	✓	≤ 1551	Computationally expensive.	✓*
Upadhyay et al. [17]	Speech	Categorical	✓	✓	N/A	N/A	✗
Ours	Speech	Dimensional A/V	✓	✓	12,913

¹Including 1 “Rest” annotator.

^*We apply similar limitations/approach to our proposed model. For Han et al. [3] we learn variance as an additional target. For Davani et al. [8] and Ando et al. [16] we use our more efficient model with the same restrictions they define (only annotators with a minimum number of annotations are learned).

III. Research Questions

A. Q1: Can a model learn to predict labels for every annotator present in a large crowd-sourced dataset?

Previous work has demonstrated the feasibility and efficacy of modeling relatively small groups of annotators [14]-[16]. However, it is not known whether such a large number of annotators can be accurately modeled nor whether it is beneficial to do so. In our first research question, we compare the performance of learning all annotators to only learning the more common annotators (labeled many samples), as in prior work [8], [16]. We will answer if such models are possible, and identify when/if such models should be learned rather than learning only the most common annotators.

B. Q2: Can models trained to predict individual annotators be aggregated to produce consensus predictions as accurately as models trained using consensus labels?

Multi-annotator modeling approaches in SER have often augmented the learning task with a consensus ground-truth [14], [17], [45] or alternatively aggregate individual-specific output to predict the consensus ground-truth [13]. In our second research question we investigate whether large-scale multi-annotator modeling can match or improve consensus-only predictions. It is also currently unknown if augmenting the learning task with a consensus prediction will still be useful when learning every annotator. We will use the output of multi-annotator models to create a consensus ground-truth. We will compare the consensus ground-truth to other consensus-prediction methods. This includes the prediction of a single consensus label or a distribution over annotations.

C. Q3: Can multi-annotator models be used in cross-corpus settings?

Cross-corpus generalizability is one of the persistent challenges of SER. We suggest that the challenge is due in part to differences between the populations who evaluate the different datasets. Since the annotators are not consistent in a cross-corpus setting, the perception differences that exist between annotator populations is likely negatively impacting cross-corpus model performance. We posit that by learning many annotator-specific prediction heads, we will be able to identify similar annotators to those in a new target dataset. We can then use selected pre-trained annotator prediction heads in the target dataset, without additional training. We investigate methods to select annotators and the efficacy of deploying pre-trained annotator-specific prediction heads.

However, there are use cases where it may be advantageous to adapt a pre-trained annotator prediction head to a target dataset. This would allow for both enhanced personalization and the ability to account for acoustic or lexical changes between the source and target datasets. We additionally investigate adaptation of pre-trained annotator prediction heads.

IV. Data and Preprocessing

A. Datasets

We select datasets that have been annotated with utterance-level valence and activation labels, focusing on Likert-scale labeling. We use the largest dataset to train the models and perform adaptation to the other datasets.

MSP-Podcast is a dataset containing naturalistic speech sourced from podcasts [18]. We use MSP-Podcast release 1.11 for our experiments as the training dataset. We use the predefined splits and evaluate on test set 1. We use the predefined speaker-independent split and filter out annotators who appear only in the test set, but not in the training set. This results in 84,030 training utterances and 30,647 testing utterances with 12,913 training annotators, of which 10,750 are also testing annotators, with an average of 5.67 and 5.88 dimensional annotations per utterance for the training and testing sets, respectively. We recalculate the valence and activation annotations over the filtered annotator set (the removal of annotators may change the consensus label). Additionally, we report results on MSP-Podcast 1.8 for backwards compatibility. MSP-Podcast 1.11 provides files to reconstruct previous MSP-Podcast versions. However, transcripts are not provided for utterances removed from previous versions, in these cases we use Whisper¹ [46] to transcribe the speech. There are a total of 903 utterances present in release 1.8 that have since been removed by release 1.11. Of these removed utterances, release 1.11 provides 612 in a “previous releases” folder (the remaining 291 audio files are completely missing and so we do not use them).

MuSE is a dataset of unscripted audio-visual monologues recorded from 28 college students in two 45-minute sessions each, as they react to emotional stimuli. One session was recorded when the students were affected by an external stressor, and the other was without the stressor [47]. Students were recorded using a lapel microphone. Utterances were evaluated by crowdsourced annotators for both categorical and dimensional labels. There are 2,648 utterances comprised of 1,422 stressed and 1,226 non-stressed samples. MuSE contains both contextual annotations, where annotators see and label utterances in the actual order they were recorded, and no-context annotations, in which utterances are seen in a random order during labeling. There are 160 individual annotators who labeled the utterances without context. We use these annotators because no-context annotation is consistent with the majority of the other datasets. Each sample was evaluated by between 7 and 9 annotators, with 8.03 on average. For our adaptation and testing folds there are 8.03 ± 0.00 and 8.03 ± 0.01 annotators per sample. We use Whisper² [46] to transcribe samples.

MSP-Improv is an acted dataset using both scripted and improvised sessions of dialogue between two speakers, designed to elicit particular emotions [31]. Participants were recorded using collar-worn microphones across six sessions with a total of twelve speakers. Crowdsourced annotators evaluated each utterance, comprising of 1,495 total unique annotators across the dataset, across categorical and dimensional labels. There are 8,438 utterances with 7.31 annotations per utterance on average, with 7.31 ± 0.07 and 7.28 ± 0.25 annotators per utterance on average across the adaptation and testing fold respectively.

IEMOCAP is a highly scripted, yet partially improvised, acted dataset comprising of five sessions of dialogue between two speakers [32]. There are 10,039 total samples in the dataset. Annotations were completed in-house using a total of six unique annotators, across both categorical and dimensional labels. Four of the six annotators provided between 1,748 and 8,251 dimensional annotations, while the remaining two annotators only labeled 58 samples for dimensional emotion. There are 2.14 dimensional emotion annotations per sample on average across the dataset, with 2.14 ± 0.03 and 2.13 ± 0.05 annotators per utterance on average across the adaptation and testing fold respectively.

B. Dataset Label Preprocessing

We use three sources of ground-truth for all datasets: 1) valence/activation annotations of the individual annotators, 2) consensus valence/activation labels, and 3) 2D distributions over the valence-activation space.

We first min-max scale the valence and activation dimensions to the [−1, 1] range. Our prior work demonstrated the importance of this scaling for effective learning [41]. We apply this scaling to the individual annotations and to the consensus labels provided by the datasets using dataset-level statistics (not per annotator).

We next create 2D distributions over the ground-truth individual annotations. We smooth the distributions using KDE as in [2]. We use a KDE grid size of 512 as in our previous work [41].

C. Training, Validation, and Testing

For IEMOCAP and MSP-Improv, we create session-independent folds to ensure speaker-independence and remove any risk of potential crosstalk. For MuSE, we randomly generate speaker-independent validation folds³. For MSP-Podcast we used the pre-defined training-test split.

V. Multi-annotator Modeling

In this section, we describe our novel approach that enables the learning of the dimensional annotations of over 10,000 individual annotators.

A. Features

WavLM⁴ [38] and BERT⁵ [48] are large pre-trained transformer models. These models output contextual features for speech and language and perform well when finetuned for downstream tasks, such as SER [29]. Previous work has demonstrated that BERT and Wav2Vec2 [37] (a similar model to WavLM) is effective for SER prediction [10], [29] and prediction of multiple annotators [41]. WavLM has demonstrated improved performance on SER compared to Wav2Vec2 [49], and our previous work has demonstrated effectiveness as predictions of multiple annotators [42].

B. Model Architecture

The general architecture of the model is shown in Figure 1, in line with previous work [41], [42]. The input to the model is the frozen mean-pooled final layers of WavLM and BERT CLS embeddings. We apply dropout with probability 0.2 and concatenate the embeddings. The concatenated embedding is passed through a single linear layer of size 256 with ReLU activation. There is a unique prediction head for each annotator, across valence and activation, that includes two linear layers of size 256 with ReLU activations (Figure 1).

Fig. 1:

Individual Annotator Model Architecture. Gray layers are shared between all annotators. Layer output size is marked in brackets. Unshaded layers are individual to each learned annotator.

VI. Experimental Settings

A. Consensus Baseline

We implement a consensus model that learns the averaged annotation over all annotators of a sample, with one prediction head for activation and one for valence. We use the architecture described in Section V-B.

B. Distribution Baselines

1). 2D KDE Model:

The first baseline is the 2D KDE model discussed in Section II-C and presented in [2]. It has 16 output logits resulting in a distribution that is of size 4x4 over a space defined by valence and activation.

2). DEER Model:

To the best of our knowledge DEER represents the current state-of-the-art performance on distributional uncertainty modeling in SER [6]. One major difference between DEER and the other approaches discussed in this manuscript: DEER fully finetunes the pretrained audio encoder. We present the results from the DEER publication on MSP-Podcast 1.8 [6]. Results for 1.11 are not available.

C. Multi-annotator models

1). IA Model:

The model that predicts the annotations of individual annotators is denoted as IA. It has separate prediction layers for each annotator as in previous multitask approaches [8]. We use a multi-task approach as it has shown to improve performance over methods such as ensemble and one-hot encoding [8], [41]. For each sample, the model makes predictions for all annotators who have a corresponding ground-truth annotation of that sample (i.e., predictions are only made for annotators who have labeled the current sample).

2). IA_μ Model:

Previous work has demonstrated the benefit of predicting the consensus label, in addition to the annotations of the individual annotators [14], [17], [45]. We augment the IA model with a consensus prediction and denote the model, IA_μ. Like IA, it has separate prediction layers for each annotator and one more prediction layer for the consensus label, the average label.

3). IA_σ² Model:

Previous work has demonstrated the benefit of directly considering the individual-specific variability [3]. We augment the IA model with annotator-specific prediction heads that predict the individual’s variance over activation and valence, in addition to the that annotator’s annotation prediction heads. We denote the model, IA_σ².

4). IA_μσ² Model:

We anticipate that additional benefit comes from modeling the individual annotators, the consensus annotator, and the variance associated with each annotator. We denote this combined model, IA_μσ².

D. Prediction of Consensus Labels

The 2D KDE baseline does not produce continuous ratings. We create a consensus label by summing along the activation/valence dimensions and then multiplying this sum by [−1, −0.5, 0.5, 1]. There was no consensus label provided in the original DEER paper and as a result we do not present consensus results for DEER [6].

The only IA models that predict the consensus labels alongside the individual annotator labels is IA_μ and IA_μσ². In all other cases, we create consensus labels by averaging the output of the model over valence and activation, separately.

E. Prediction of Distribution

We create distributions using the output of the IA models. We follow the 2D KDE approach’s label generation technique [2] to foster comparability between the baseline and IA approaches. We draw a random number of IA predictions from the annotations of a given sample and take the average. For example, if there were 16 annotations from sample in MSP-Podcast, at each iteration, we would choose between 2 and 16 of these annotations and take an average - this would form a new sample. We repeat this process 200 times, resulting in 200 new averaged samples. We use the new samples to perform 2D KDE at a higher resolution (such as 512x512). The resulting KDE is converted into a 4x4 distribution by placing a 4x4 grid over the 2D KDE and averaging the density in each grid (for a 2D KDE at 512x512 resolution, each grid would be of size 128x128). The 4x4 grid with average density in each point is then converted to a probability by dividing each value in the 4x4 grid with the total sum of the values in the 4x4 grid (see [2] for more details). The final distribution for all models is 4x4 across valence and activation.

F. Metrics

1). Individual Annotator Metrics:

We use Lin’s Concordance Correlation Coefficient (CCC) [50] as our main metric. This metric is common in the field of SER [10], [29], [41]-[43]. We define and motivate two varieties of CCC: 1) CCC_ind, and 2) CCC_con.

Our first CCC metric, CCC_ind, captures how well a model predicts each annotator. We filter to consider all annotators who annotated at least two samples in the test set. For each annotator (e.g., annotator j), we compare the annotator’s labels and the system prediction. This results in CCC_ind,j. We average this value over all annotators (all are weighted equally).

Our second metric, CCC_con, captures the ability of the model to estimate the consensus label. In this case, there is one prediction per sample. We calculate CCC by comparing the consensus label and the predicted label across the full test set.

2). Distribution Metrics:

We measure the differences between the learned and ground-truth 4x4 probability distributions using Total Variation Distance (TVD) and Jensen-Shannon Divergence (JSD) [2]. We use log₂ for JSD.

We also calculate the Unweighted Average Recall (UAR). This requires the creation of activation and valence classes. First consider activation, we marginalize the 4x4 distribution over valence to get the resulting one-dimensional (1x4) distribution over only activation. The activation class is the argmax of this one-dimensional distribution. We repeat this process for valence.

G. Annotator Filtering

We replicate previous work by reducing the number of annotators, focusing only on annotators who annotated at least 100 samples [16]. In the original work, the authors retained 155 annotators and also modeled a “Rest” annotator [16]. In our work, we optionally model the consensus annotation in the μ variant of the model. Our preliminary results indicate that the two approaches perform comparably. We will be using the consensus annotator for clarity and simplicity. We perform additional filtering, retaining annotators who annotated at least 1,000 samples, as in [8]. We train and test each model on the full-dataset, the 100+ annotations dataset, and the 1000+ annotations dataset. In the two filtered datasets, we filter based on the number of training annotations provided, e.g., in the 100+ case all annotators provided at least 100 training annotations. We report results on all three versions of the dataset. We include μ, σ², μσ² variants. When testing a model trained on the filtered datasets on a less filtered dataset (e.g., trained on 1000+ tested on 100+ or full dataset), we predict unseen annotators using the approach outlined in Section VIII-A and denote the model with subscript “Sim” to indicate this approach was used for prediction of unseen annotators (e.g., IA_Sim).

VII. Model Training

We train the models using stochastic gradient descent with a learning rate of 0.001. We use a learning rate scheduler that adjusts the learning rate by a factor of 0.1 after five epochs of no reduction in the validation loss. We train models until early stopping triggers with a patience of 10. We train with a batch size of 32 and for all methods we use the relevant task’s validation losses.

We jointly train all annotator prediction heads using a CCC metric that treats individual annotator labels as the target, yet is robust to the nature of randomly shuffled training batches. This is needed because our main metric, CCC_ind, cannot be used directly as a loss function when a particular annotator only appears once in a training batch (CCC_ind would be undefined). We define CCC_flat, which captures the performance of the individual annotator predictions, flattened, over all annotators. The flattened CCC allows us to handle sparse annotators. For example, the set of annotations A and predictions $\widehat{\mathbf{A}}$. We denote an individual annotation for sample i, annotator j as A_i,j and the prediction for that sample-annotator combination as ${\widehat{A}}_{i,j}$. Then, $CC{C}_{flat}(A,\widehat{A})=CCC(\{a\mid a\in {\cup }_{i,j}{A}_{i,j}\},\{\widehat{a}\mid \widehat{a}\in {\cup }_{i,j}{\widehat{A}}_{i,j}\})$. Our primary loss function is defined by subtracting this metric from $1:ℒ=(1-CC{C}_{flat}(Y_{act},{\widehat{Y}}_{act}))+(1-CC{C}_{flat}(Y_{val},{\widehat{Y}}_{val}))$. CCC_flat benchmarks how well a model is performing across all annotators and samples. We note that the metric is weighted more heavily in favor of annotators who have annotated more samples. For models that predict the consensus in addition to the individual annotators (IA_μ, IA_μσ²) we add an artificial consensus annotator with a ground truth value of the average label and include them in the CCC_flat calculations.

We pre-train the IA models for consensus label prediction with standard CCC loss for five epochs before initializing the individual annotator prediction head with the consensus label prediction weights and training with CCC_flat.

Our secondary loss function is only for models that model annotator variance (IA_σ², IA_μσ²). The challenge is that we do not have individual annotator variance ground-truth labels to train this prediction. We only have the sample-level inter-annotator variance. In order to facilitate learning an individual annotator’s variance, we must first convert the individual annotator variance predictions into a sample-level inter-annotator variance prediction. Given a uniformly chosen random annotator, j, who annotated the sample with label A_i,j. By the law of total variance:

$$\text{Var}(A_{i,j})=\underset{j}{\mathbb{E}}[\text{Var}(A_{i,j}\mid j)]+{\text{Var}}_j(\mathbb{E}[A_{i,j}\mid j])$$

That is, the sample-level variance is the expected variance of a random annotator, and the variance of the expected label of a random annotator. Therefore ${\widehat{\sigma }}^2≔\text{Var}(A_{i,j})≔\frac{1}{n}{\sum }_i{\widehat{\sigma }}_i^2+\text{Var}(\{{\widehat{\mu }}_i\mid i\in \text{Number of annotators on sample}\})$, where ${\widehat{\mu }}_i$ is the expected label of annotator j on the given sample.

We calculate $\widehat{\mu }$ by calculating the mean over all individual annotator mean predictions on the given sample. We assume that the label is given by a Gaussian distribution as in [6], and we use the definition of Kullback-Leibler Divergence (KL-Div) for univariate Gaussian distributions [51], [52] as our loss function. We minimize the KL divergence from the predicted Gaussian to the ground-truth Gaussian ($\text{KL-Div}(𝒩(\widehat{\mu },{\widehat{\sigma }}^2)\parallel 𝒩(\mu ,{\sigma }^2)$). That is $ℒ=\frac{1}{2}[log\frac{{\sigma }^2}{{\widehat{\sigma }}^2}+\frac{{\widehat{\sigma }}^2+{(\widehat{\mu }-\mu )}^2}{{\sigma }^2}-1]$, where μ, σ² are the mean and variance of the sample’s annotations, and $\widehat{\mu }$, ${\widehat{\sigma }}^2$ are defined as above. We apply this loss function for both activation and valence and sum them to get our variance KL-Div loss. We clamp the value of the KL-Div loss to be no larger than 10 to prevent large loss values when variances are small. We interleave KL-Div and CCC for these two models. We use the CCC_flat validation loss for early stopping as individual annotator label prediction is the primary goal of the model.

VIII. Cross-Corpus Prediction

We experiment with two approaches in cross-corpus prediction. In both cases, we start with the IA models described in VI-C. In the first, we select individual pre-trained annotator models to apply to the target datasets (Section VIII-A). In the second, we adapt these pre-trained models to the new dataset (Section VIII-B). The first case can be thought of as annotator adaptation, while the second case can be thought of as both annotator and dataset adaptation. In all cases, the source dataset on which the pre-trained annotator models are learned is MSP-Podcast. The target datasets on which these models are applied are MuSE, IEMOCAP, and MSP-Improv.

A. Annotator Selection

We describe two types of similarity. In the first type, a given annotator is similar to another if they annotate both valence and activation similarly, as in our previous work [42]. In this case, we identify one source annotator for every target annotator (one-to-one). In the second type, a given annotator is similar to another if they annotate either valence or activation similarly. In this case, we identify two source annotators (note that the two annotators may end up being the same annotator), one for valence and one for activation, for every target annotator (one-to-two).

We identify similar annotators using the training partition of the target datasets. Our goal is to identify the most accurate pre-trained prediction head on the target training set, either for the dimensions together (one-to-one) or separately (one-to-two). As in Section VI-G, we choose the pre-trained prediction head with the highest CCC (or lowest absolute error if the target annotator annotated a single sample in the target dataset’s training data). We then use this pre-trained prediction head to make predictions for the annotators on the test partition of the target dataset. We again refer to this approach as IA_Sim. We also compare against IA_Rand where annotators are assigned randomly rather than based on similarity.

B. Annotator Adaptation

We investigate an upper-limit for cross-corpus performance by performing annotator adaptation given the training data of the target datasets. We start by fully finetuning the IA model on the target dataset’s training data. We first take the IA_Sim one-to-one and IA_Rand (Section VIII-A), we then finetune these models following the approach in Section VII.

C. Baselines

We experiment with two baselines. The first is a zero-shot application of the consensus baseline model (Section VI-A). The second is the consensus model after finetuning to the target datasets. In this case we take the pre-trained consensus model, and continue training on the new target dataset’s consensus labels until early-stopping triggers. We train the same as in pre-training (Section VII).

IX. Results

We analyze each of the three questions reporting test results over five seeds. Significance is asserted at a 5% confidence using a paired two-sided t-test.

A. Comparison of Full and Partial Annotator Modeling

We find that it is possible to learn all annotators in large crowdsourced datasets, including the annotators who have labeled few samples. However, we also find that it is often not necessary to learn every single annotator. Instead, we can use the annotators that have labeled more data (applying the proposed similarity mapping approach to predict all annotators) instead.

We first compare the IA approaches using CCC_ind, and CCC_con. We present results over three test partitions: 1) all annotators - 12,913 annotators (of which 10,750 appear in the test set), 2) annotators who annotated at least 100 samples in the training set - 461 annotators (of which 450 appear in the test set), and 3) annotators who annotated at least 1000 samples in the training set - 38 annotators (all of whom appear in the test set). Previously, the maximum number of annotators considered was 155.

We first consider the models trained on the full-dataset containing all annotators (Table II). We find that IA and IA_μ perform similarly in eleven of the twelve tasks (Table II row one and two of each of the three sections). The only exception is CCC_ind for valence when considering all 10,750 test annotators. IA and IA_σ² are similar in ten of the twelve tasks, excepting CCC_ind activation on the 10,750 test set, and CCC_ind valence on the 450 test set. IA and IA_μσ² is similar in eleven of twelve tasks, showing significant decrease in CCC_ind valence on the 450 test set.

TABLE II:

Individual Annotator Results trained on the full set of 12,913 MSP-Podcast annotators. * and † represent significant increase and decrease calculated against the first row of each IA block (e.g., for 10,750, 450, and 38 test sets), respectively. Bold denotes the best result in a section (i.e. 10,750 test annotators). CCC is defined for 2D KDE in Section VI-D

Model	# Test Annotators		Activation		Valence
	(# Annotations)		CCCind	CCCcon	CCCind	CCCcon
Consensus		N/A	0.306±0.001†	0.633±0.003†	0.335±0.001	0.616±0.001†
2D KDE		N/A	0.276±0.008†	0.593±0.024†	0.286±0.005†	0.561±0.011†
IA	(Any #)	10750	0.326±0.002	0.668±0.003	0.331±0.003	0.658±0.004
IAμ		10750	0.325±0.002	0.665±0.008	0.335±0.002*	0.659±0.002
IAσ2		10750	0.321±0.002†	0.656±0.012	0.333±0.004	0.650±0.006
IA μσ2		10750	0.324±0.001	0.668±0.007	0.334±0.004	0.655±0.004
IA	(100+)	450	0.389±0.008	0.652±0.001	0.392±0.002	0.642±0.004
IAμ		450	0.391±0.006	0.651±0.006	0.393±0.003	0.643±0.003
IAσ2		450	0.384±0.002	0.641±0.009	0.386±0.005†	0.633±0.005
IAμσ2		450	0.388±0.002	0.649±0.004	0.385±0.003†	0.637±0.003
IA	(1000+)	38	0.414±0.004	0.684±0.005	0.478±0.005	0.629±0.007
IAμ		38	0.415±0.006	0.684±0.005	0.480±0.005	0.631±0.005
IAσ2		38	0.406±0.007	0.671±0.010	0.471±0.005	0.626±0.008
IAμσ2		38	0.415±0.005	0.681±0.007	0.476±0.004	0.628±0.008

Now we consider the models trained on the 100+ annotations training set as in [16] (Table III). We compare these models against the IA model trained on the full dataset (e.g., see the row marked (Table II) in each of the three blocks of Table III). We first consider the full set of test annotators (10,750 in total). In the overwhelming majority of these cases (10,300 annotators), no annotator-specific prediction head was trained. Instead, the most similar prediction head was selected and used to predict their annotations (Section VIII-A). We find that in all cases, using the proposed similarity approach (Section VIII-A), IA_Sim was comparable to or statistically significantly better than training an IA model over all annotators. We also find significant improvement on CCC_con for activation and CCC_ind for valence for IA_Sim,μ and IA_Sim,μσ² compared to IA on the full-dataset. We additionally find IA_Sim,σ² to significantly improve CCC_ind for valence. When testing on the 100+ annotations and 1000+ annotations test sets, we generally see either comparable or statistically significantly improved performance over training on the full set of annotators.

TABLE III:

Individual Annotator Results trained on the 461 annotators with at least 100 annotations. * and † represent significant increase and decrease calculated against IA (first row in each block), respectively. Bold denotes the best result in a section (i.e. 10,750 test annotators). In the case that an annotator was not learned (i.e., not in the set of the 461 learned annotators), we choose the most similar annotator (Sim).

Model	# Test Annotators		Activation		Valence
	(# Annotations)		CCCind	CCCcon	CCCind	CCCcon
IA (Table II)	(Any #)	10750	0.326±0.002	0.668±0.003	0.331±0.003	0.658±0.004
IASim		10750	0.327±0.001	0.673±0.001*	0.339±0.001*	0.659±0.002
IASim,μ		10750	0.329±0.001	0.674±0.001*	0.342±0.001*	0.659±0.001
IASim,σ2		10750	0.327±0.001	0.673±0.001	0.336±0.001*	0.659±0.003
IASim,μσ2		10750	0.329±0.002	0.672±0.002*	0.339±0.001*	0.656±0.004
IA (Table II)	(100+)	450	0.389±0.008	0.652±0.001	0.392±0.002	0.642±0.004
IA		450	0.388±0.001	0.656±0.001*	0.396±0.001*	0.648±0.001*
IAμ		450	0.389±0.001	0.656±0.001*	0.398±0.002*	0.649±0.001*
IAσ2		450	0.388±0.002	0.656±0.001*	0.394±0.003	0.646±0.002
IAμσ2		450	0.387±0.002	0.654±0.002	0.394±0.004	0.645±0.003
IA (Table II)	(1000+)	38	0.414±0.004	0.684±0.005	0.478±0.005	0.629±0.007
IA		38	0.431±0.003*	0.691±0.000*	0.484±0.002*	0.637±0.002
IAμ		38	0.425±0.004*	0.691±0.002*	0.482±0.003	0.637±0.001
IAσ2		38	0.434±0.003*	0.691±0.002	0.483±0.002	0.638±0.001*
IAμσ2		38	0.426±0.004*	0.689±0.003	0.481±0.004	0.636±0.002

Finally, we consider the models trained on the 1000+ annotations set as in [8] (Table IV). We find that over the full set of annotators, the reduction in training annotators results in significantly lower performance in all activation tasks and consensus valence tasks. Conversely, we see statistically significantly improved performance for CCC_ind valence tasks. The results for the set of annotators with 100+ annotations is more mixed. We again see consistent CCC_ind performance improvements and CCC_con performance decreases for valence. The results for activation are largely comparable to those obtained using models trained on all annotators. Finally, we observe the performance of the model for the annotators who have annotated at least 1000 samples. We find that the models trained on the reduced set of annotators (38 vs. 12,913) either improves or does not harm performance on this population, notably improving CCC_ind for activation. We remind that, in this case, all test annotators have been seen at training time.

TABLE IV:

Individual Annotator Results trained on the 38 annotators with at least 1000 annotations. * and † represent significant increase and decrease calculated against IA (first row in each block), respectively. Bold denotes the best result in a section (i.e. 10,750 test annotators). In the case that an annotator was not learned (i.e., not in the set of the 38 learned annotators), we choose the most similar annotator (Sim).

Model	# Test Annotators		Activation		Valence
	(# Annotations)		CCCind	CCCcon	CCCind	CCCcon
IA (Table II)	(Any #)	10750	0.326±0.002	0.668±0.003	0.331±0.003	0.658±0.004
IASim		10750	0.315±0.001†	0.654±0.001†	0.337±0.001*	0.627±0.002†
IASim,μ		10750	0.317±0.000†	0.659±0.001†	0.342±0.001*	0.628±0.002†
IASim,σ2		10750	0.313±0.002†	0.655±0.002†	0.337±0.001*	0.628±0.003†
IASim,μσ2		10750	0.316±0.001†	0.660±0.003†	0.341±0.001*	0.630±0.002†
IA (Table II)	(100+)	450	0.389±0.008	0.652±0.001	0.392±0.002	0.642±0.004
IASim		450	0.402±0.002	0.652±0.001	0.409±0.001*	0.628±0.003†
IASim,μ		450	0.410±0.001*	0.654±0.002	0.417±0.001*	0.627±0.002†
IASim,σ2		450	0.395±0.003	0.650±0.002	0.410±0.003*	0.628±0.003†
IASim,μσ2		450	0.405±0.001*	0.655±0.003	0.418±0.003*	0.630±0.003†
IA (Table II)	(1000+)	38	0.414±0.004	0.684±0.005	0.478±0.005	0.629±0.007
IA		38	0.422±0.001*	0.689±0.001	0.477±0.003	0.633±0.001
IAμ		38	0.425±0.002*	0.693±0.001*	0.481±0.002	0.637±0.001
IAσ2		38	0.426±0.004*	0.688±0.002	0.477±0.003	0.634±0.001
IAμσ2		38	0.429±0.004*	0.691±0.002*	0.483±0.003	0.637±0.001

The competitiveness of the 100+ model suggests that there is little benefit to using the least well-represented annotators. Our previous work demonstrated that as little as ~ 15 – 20 annotations is sufficient for the similarity approach to perform well [42]. A similar value has been found in other works [53].

B. Multi-annotator and Consensus-only Performance

Our experiments suggest that the individual annotator models are at least as effective as consensus-only models for performing consensus prediction.

We find that the IA models significantly improve consensus label prediction compared to the DEER, Consensus, and 2D KDE models for consensus predictions (Tables II&V). The consensus results between the various IA models are not statistically different.

TABLE V:

Aggregate metric performance on MSP-Podcast 1.8. * and † Represent significant increase and decrease calculated against IA_μ respectively. Bold denotes the best result.

Model	Activation CCCcon	Valence CCCcon
IA	0.734±0.002	0.685±0.001
IAμ	0.732±0.002	0.684±0.003
IAσ2	0.731±0.003	0.683±0.002
IAμσ2	0.733±0.004	0.684±0.001
Consensus	0.643±0.001†	0.588±0.000†
DEER [6]	0.698↓	0.506↓

We find that for probability measures such as TVD and JSD, directly learning the distributions (2D KDE) does not outperform the creation of a 2D distribution using the IA model output. In addition, we find the UAR metric is significantly higher for the IA models, compared to the 2D KDE model in all cases (Table VI).

TABLE VI:

Probability Distribution Results. * and † Represent significant increase and decrease calculated against 2D KDE respectively. Bold denotes the best result.

Model	Activation UAR↑	Valence UAR↑	TVD↓	JSD↓
2D KDE	0.377±0.014	0.408±0.007	0.489±0.004	0.291±0.004
IA	0.564±0.002*	0.558±0.003*	0.489±0.004	0.295±0.004
IAμ	0.557±0.009*	0.558±0.003*	0.488±0.002	0.295±0.002
IAσ2	0.551±0.008*	0.550±0.004*	0.492±0.005	0.299±0.005
IAμσ2	0.558±0.007*	0.551±0.003*	0.487±0.005	0.294±0.005

C. Cross-Corpus Results

We find that individual annotator models can be applied in cross-corpus settings effectively by identifying similar annotators between the learned annotators and new annotators from the target dataset using an enrollment set. We find that all IA models, after identifying similar annotators, outperform zero-shot consensus model predictions on almost all metrics and target datasets (Table VII).

TABLE VII:

Cross-corpus results. Bold denotes the best result in a section (i.e. one-to-one). * and † Represent significant increase and decrease calculated against IA (Any #) respectively. Underline denotes the best result in each column.

Model	Activation CCCcon	Valence CCCcon	Activation CCCcon	Valence CCCcon	Activation CCCcon	Valence CCCcon
	MuSE		IEMOCAP		MSP-Improv
Consensus	0.635±0.090	0.557±0.066	0.636±0.018	0.487±0.047	0.526±0.065	0.471±0.052
one-to-one
IASim (Any #)	0.729±0.076	0.618±0.039	0.636±0.029	0.517±0.042	0.633±0.062	0.536±0.053
IASim (100+)	0.730±0.083	0.626±0.039*	0.629±0.029†	0.521±0.040	0.631±0.057	0.540±0.054*
IASim (1000+)	0.710±0.094†	0.607±0.043†	0.623±0.041†	0.498±0.036†	0.623±0.071†	0.522±0.057†
one-to-two
IASim (Any #)	0.726±0.078	0.634±0.041	0.640±0.033	0.530±0.038	0.656±0.067	0.536±0.051
IASim (100+)	0.731±0.084	0.651±0.040*	0.637±0.037	0.530±0.035	0.654±0.064	0.543±0.049*
IASim (1000+)	0.720±0.092	0.640±0.038	0.642±0.042	0.506±0.038†	0.638±0.068†	0.522±0.055†
	CCCind	CCCind	CCCind	CCCind	CCCind	CCCind
Consensus	0.233±0.037	0.393±0.027	0.471±0.030	0.403±0.049	0.274±0.054	0.250±0.026
one-to-one
IASim (Any #)	0.334±0.047	0.439±0.027	0.524±0.031	0.452±0.046	0.354±0.064	0.276±0.031
IASim (100+)	0.333±0.047	0.447±0.023*	0.519±0.032	0.452±0.042	0.349±0.062†	0.278±0.031
IASim (1000+)	0.314±0.053†	0.428±0.023†	0.508±0.035†	0.425±0.043†	0.332±0.067†	0.268±0.032†
one-to-two
IASim (Any #)	0.333±0.047	0.443±0.029	0.526±0.027	0.462±0.043	0.359±0.065	0.277±0.030
IASim (100+)	0.334±0.049	0.454±0.029*	0.525±0.029	0.456±0.042†	0.355±0.063†	0.280±0.029*
IASim (1000+)	0.325±0.051†	0.442±0.028	0.522±0.030	0.439±0.043†	0.342±0.066†	0.271±0.032†

We find that, in general, it is more effective to select similar valence and activation annotators separately (one-to-two, IA), compared to selecting a single similar annotator for both (one-to-one, IA). Please see Table VII for more details.

Additionally, we find that it is crucial to identify similar annotators (IA_Sim) for cross-corpus predictions (Table VIII) compared to IA_Rand. However, when finetuning on the cross-corpus dataset it is not necessary to identify similar annotators before finetuning as IA_Sim and IA_Rand perform similarly after finetuning (Table VIII).

TABLE VIII:

Cross-corpus results for zero-shot and finetuned in the one-to-one setting. Bold denotes the best result in a section (i.e. Zero-Shot). * and † Represent significant increase and decrease calculated against IA (Any #) respectively. Underline denotes the best result in each column.

Model	Annotator Selection	Activation CCCcon	Valence CCCcon	Activation CCCcon	Valence CCCcon	Activation CCCcon	Valence CCCcon
		MuSE		IEMOCAP		MSP-Improv
Zero-Shot
Consensus	N/A	0.635±0.090	0.557±0.066	0.636±0.018	0.487±0.047	0.526±0.065	0.471±0.052
IASim (Any #)	Similar	0.729±0.076*	0.618±0.039*	0.636±0.029	0.517±0.042*	0.633±0.062*	0.536±0.053*
IARand (Any #)	Random	0.548±0.096†	0.525±0.088†	0.544±0.080†	0.475±0.052	0.518±0.068	0.470±0.058
Finetuned
Consensus	N/A	0.641±0.078	0.651±0.042	0.697±0.016	0.676±0.045	0.636±0.070	0.608±0.052
IASim (Any #)	Similar	0.784±0.055*	0.682±0.039*	0.725±0.027*	0.662±0.030†	0.724±0.051*	0.621±0.043*
IARand (Any #)	Random	0.780±0.053*	0.684±0.034*	0.725±0.027*	0.660±0.033†	0.718±0.050*	0.625±0.045*
		CCCind	CCCind	CCCind	CCCind	CCCind	CCCind
Zero-Shot
Consensus	N/A	0.233±0.037	0.393±0.027	0.471±0.030	0.403±0.049	0.274±0.054	0.250±0.026
IASim (Any #)	Similar	0.334±0.047*	0.439±0.027*	0.524±0.031*	0.452±0.046*	0.354±0.064*	0.276±0.031*
IARand (Any #)	Random	0.197±0.037†	0.302±0.024†	0.424±0.055†	0.368±0.063†	0.263±0.056†	0.221±0.030†
Finetuned
Consensus	N/A	0.228±0.041	0.394±0.032	0.491±0.031	0.566±0.074	0.306±0.060	0.341±0.035
IASim (Any #)	Similar	0.296±0.053*	0.442±0.032*	0.573±0.026*	0.573±0.051	0.364±0.060*	0.322±0.031†
IARand (Any #)	Random	0.292±0.048*	0.444±0.029*	0.572±0.026*	0.571±0.054	0.359±0.058*	0.342±0.035

Finally, we return to the traditional SER task of consensus label prediction in a cross-corpus setting. We study the relative impact of: (1) applying a consensus model off-the-shelf in a new domain naively, i.e., zero-shot (2) finetuning that model, i.e., dataset adaptation at the consensus-level (3) selecting similar annotators and predicting the consensus label by aggregating the result, i.e., annotator adaptation and (4) finetuning the annotator-specific models, i.e., annotator and dataset adaptation.

We first compare zero-shot and dataset adaptation at the consensus level. This isolates the impact of finetuning on cross-corpus datasets. We find that finetuning the consensus model outperforms the zero-shot consensus model in all cases for CCC_con. We find overall performance gains for CCC_ind as well. However, this is not true on MuSE where we actually see a decrease in the activation CCC_ind performance.

We next consider the impact of annotator adaptation by comparing the performance of the zero-shot consensus model and the model after annotator adaptation (IA_Sim). We find that annotator adaptation shows statistically significant improvement over zero-shot in nearly all cases for both CCC_con and CCC_ind. The only exception is IEMOCAP activation on CCC_con where performance is similar.

Finally, we compare the three types of adaptation: dataset adaptation at the consensus level (finetuning the consensus-model), annotator adaptation (IA_Sim), and annotator and dataset adaptation (finetuning IA_Sim). We first compare the performance of IA_Sim to the finetuned consensus model. We find that IA_Sim outperforms the finetuned consensus model in most cases for CCC_ind, but the finetuned consensus model is generally better at CCC_con. We next compare the performance of the finetuned consensus model to the finetuned IA_Sim model. We find that finetuned IA_Sim outperforms the finetuned consensus model in all cases, except for valence on IEMOCAP and CCC_ind valence on MSP-Improv. Both dataset adaptation and annotator adaptation should be performed to maximize performance on cross-corpus datasets.

For valence we find that adaptation to the dataset (i.e. finetuning on new data), rather than annotator perception (mapping similar annotators), improves performance for the most scripted dataset (IEMOCAP). We find an increase of 0.089 and 0.145 for activation and valence, respectively, when finetuning the IA_Sim model. However, for the least scripted dataset (MuSE), we find a much smaller increase upon finetuning (0.055 and 0.064 for activation and valence, respectively). When we consider the impact of the annotator-specific adaptation (comparing IA_Sim and IA_Rand), the opposite is true: we find a performance increase on MuSE of 0.181 and 0.093 compared to 0.115 and 0.066 on IEMOCAP for activation and valence respectively. These results suggest that annotator-specific adaptation is important for unscripted data.

D. Efficiency and Complexity

The traditional annotator-specific model, IA Model (with loops), has the same computational complexity as our improved approach. Yet, our approach is much faster in practice because it replaces loops (many small GPU operations) with a single batched matrix multiplication. Using the same hardware (RTX 2060) and seed we compare the computation time and peak memory usage between consensus-only prediction, our multi-annotator approach, and an identical multi-annotator approach that replaces only the single matrix multiplication with a for loop. In each case we run 20 warmup batches and then begin recording the time and memory usage over 200 batches, this is similar to a previous approach [54]. However, we increase the number of warmup batches from 10 to 20, and the number of benchmarking batches from 10 to 200 because we consider that the number of annotators in a batch may have a significant impact and want to ensure a broad range of realistic batch compositions are covered.

We benchmark performance by creating a synthetic dataset simulating between 5,000 and 30,000 unique annotators on MSP-Podcast (in steps of 5,000).There are two conditions: the synthetic dataset has fewer annotators than MSP-Podcast or it has more. In the case that the dataset has fewer annotators we randomly subsample annotators, otherwise we duplicate annotators by evenly sampling them until we have the desired amount of annotators. We also report performance on the actual MSP-Podcast dataset (Represented as ⋆ on the plot in Figure 2 and Figure 3). We also track the total number of annotators in each batch.

Fig. 2:

Average time taken during model forward method by number of annotators at both dataset-level and batch-level.

Fig. 3:

Average peak memory usage during model forward method by number of annotators at both dataset-level and batch-level.

The training time for our approach only narrowly increases the computation time compared to a consensus model (Figure 2), while the overhead of looping takes several times longer. The increase in training time for our approach is minimal even up to the simulated 30,000 annotators. We find a similar trend when plotting against the total number of annotators in each batch.

Our approach also improves the memory usage compared to the conventional approach. However, large numbers of annotators significantly increases the memory usage over a consensus-only model. We found that the memory usage was impacted most heavily by the total number of annotators (Figure 3). This is likely because our model uses frozen representations as input rather than raw audio, if the batch-duplication step was applied before generating audio representations we expect memory usage to be much higher. Since this is still relatively low (on the order of a couple hundred mebibytes - including all 200 benchmarking batches loaded into memory). It is likely that thousands more annotators may be used before this approach becomes infeasible on modern hardware.

We find a relatively constant and negligible increase (on the order of a few miliseconds) in the time taken for batch collation to generate the inputs our model requires compared to the consensus baseline.

X. Discussion

A. Summary

In this work we have highlighted a novel method that enables the modeling of individual annotators at an unprecedented scale. In addition, we have proposed using annotator-specific models to identify unseen annotators with similar perceptions to seen annotators. We have then used pre-trained annotator-specific models as a proxy to make predictions for unseen, but similar, annotators. We have demonstrated our approach to be effective and efficient compared, and we have also highlighted the impact of annotator-perception when applying these models in a cross-corpus setting.

By overcoming the infeasibility of learning all annotators in large crowdsourced datasets we can answer many open questions. We are able to demonstrate that learning only the annotators that provided more annotations, as in previous works [8], [16], is a good strategy. We find that using only the annotators with 100+ annotations, as in [16], improved over learning all annotators when employing our IA_Sim approach (Section VIII-A). However, it seems that restricting to 1000+ annotations as in [8] is too restrictive and slightly harms model performance.

Our experiments highlight that variation in annotations is an important signal and not simply noise. Previous work has found that augmenting multi-annotator models by including a consensus-prediction task has improved model performance [14], [17], [45]. We have found generally similar results with IAμ, however, we have found that the improvement is only notable when using a restricted annotator training set (100+/1000+). This suggests that the extra signal provided by the consensus label is already captured when learning all annotators.

We have demonstrated that pretrained similar annotator prediction heads can be used for unseen annotator adaptation in cross-corpus settings. Our cross-corpus results highlight the importance of considering annotator-perception during cross-corpus experiments. Our results also suggest that annotator-specific cross-corpus adaptation is particularly important for unscripted data. This may be because scripted data tends to be less ambiguous (less inter-annotator variation) since it is lab collected with more consistent environments and less noise [55].

B. Limitations

MSP-Podcast often generalizes well to the datasets used during adaptation, particularly IEMOCAP [10], which may also explain the reduced importance of annotator-specific adaptation for this dataset.

One restriction of our approach is that it requires knowledge of which annotators provided which annotator labels. If a dataset provides multiple ratings per utterance, but no information about which annotator provided the label it would not be applicable. Similarly, during inference target annotators must be specified.

During cross-corpus application it is also possible that no similar annotator can be identified from the pretrained annotator-specific prediction heads. In this case, inference with the most similar annotator may perform very inaccurately.

We also note that our loss function is based on the CCC across the individual annotator’s ratings, which has two possible problems. Since the CCC is calculated across the annotator-specific results, it encourages learning the trends across annotators rather than specifically encouraging the within-annotator CCC to be good. This could result in poor within-annotator CCC, but across-annotator CCC to perform well (similarly to Simpson’s paradox [56]). Additionally, CCC is an ideal loss function for dimensional emotion regression, but these labels at the annotator-level are ordinal not continuous, and it is possible that the ordinal structure of the individual annotator labels could be leveraged to improve performance over a regression loss function. In addition our approach is limited to dimensional emotion prediction, and is not directly applicable to categorical emotion prediction without changes to the loss function.

C. Future Work

Future work should investigate improving the loss function and early stopping of the model. The model may have reduced performance when learning all annotators as the annotators that are very well represented vs. the annotators that are poorly represented in the validation set will be contributing much more heavily towards the early stopping of model training. In addition, due to the ordinality of the individual annotator labels, future work should investigate using an ordinal loss that addresses the within-annotator problem of the CCC loss function. Finally, we believe that the effectiveness of identifying annotators with similar perception suggests that many annotators should be sharing weights. We suggest that the individual annotator model could likely be distilled to a much smaller model of perceptions that can be used to construct all annotator perceptions, while sharing weights to reduce the negative affects of sparse annotators during early-stopping.

XI. Conclusion

Learning to predict the label of all individual annotators of a dataset is challenging. The model must learn a very large number of annotators across both the dimensions of activation and valence. When considering the most extensive, most generalizable datasets, there are many annotators who annotated few samples, introducing further challenges of limited resources for some annotators and the computational cost of tracking thousands of annotators. In this work, we have introduced a novel efficiency trick to enable the modeling of all individual annotators present in a dataset without the requirement to cluster or leave out annotators from the training set. We have demonstrated significant performance improvement over soft-label baselines by modeling all annotators. Further, we have quantified the impact of annotator perception on the cross-corpus generalizability problem and demonstrated the impact and performance of annotator-specific adaptation in cross-corpus settings.

Biographies

James Tavernor is a fifth-year Computer Science and Engineering Ph.D. Candidate at the University of Michigan. In 2020, he received a Master of Engineering in Joint Honours Mathematics and Computer Science from Imperial College London, London, United Kingdom. His research focuses on cross-corpus domain adaptation and annotator modeling for speech emotion recognition, building on modern pretrained speech representations to improve generalization and robustness in real-world affective computing systems via lightweight adaptation.

Emily Mower Provost (M’11, SM’17) is a Professor and Senior Associate Chair in Computer Science and Engineering and Professor of Psychiatry (by courtesy) at the University of Michigan. She received her Ph.D. in Electrical Engineering from the University of Southern California (USC), Los Angeles, CA in 2010. She has been awarded a Toyota Faculty Scholar Award (2020), National Science Foundation CAREER Award (2017), the Oscar Stern Award for Depression Research (2015), a National Science Foundation Graduate Research Fellowship (2004-2007). She is a co-author of multiple award-winning papers in the field of automatic emotion recognition. Her research interests are in human-centered speech and video processing, multimodal interfaces design, and speech-based assistive technology. The goals of her research are motivated by the complexities of the perception and expression of human behavior.