A Privacy-Preserving Unsupervised Speaker Disentanglement Method for Depression Detection from Speech

Abstract

The proposed method focuses on speaker disentanglement in the context of depression detection from speech signals. Previous approaches require patient/speaker labels, encounter instability due to loss maximization, and introduce unnecessary parameters for adversarial domain prediction. In contrast, the proposed unsupervised approach reduces cosine similarity between latent spaces of depression and pre-trained speaker classification models. This method outperforms baseline models, matches or exceeds adversarial methods in performance, and does so without relying on speaker labels or introducing additional model parameters, leading to a reduction in model complexity. The higher the speaker de-identification score (DeID), the better the depression detection system is in masking a patient’s identity thereby enhancing the privacy attributes of depression detection systems. On the DAIC-WOZ dataset with ComparE16 features and an LSTM-only model, our method achieves an F1-Score of 0.776 and a DeID score of 92.87%, outperforming its adversarial counterpart which has an F1Score of 0.762 and 68.37% DeID, respectively. Furthermore, we demonstrate that speaker-disentanglement methods are complementary to text-based approaches, and a score-level fusion with a Word2vec-based depression detection model further enhances the overall performance to an F1-Score of 0.830.

1. Introduction

Depression is anticipated to become the second leading cause of disability globally, revealing significant diagnostic accessibility gaps [1]. Recent advancements in speech-based automatic detection have proven invaluable in tackling the challenges posed by this formidable illness [2]. The evolution of speech-based depression detection encompasses diverse acoustic features [3, 4, 5], sophisticated backend modeling techniques [6, 7, 8], and innovative data augmentation frameworks [9, 10]. While the efficacy of depression detection systems has seen notable improvements, safeguarding patient privacy remains a paramount concern in digital healthcare systems [11], particularly within the realm of mental health, where societal stigma persists as a formidable challenge [12].

Given the pivotal importance of privacy preservation in speech-based depression detection, numerous studies have attempted to address this issue. Approaches such as federated learning [13] and sine wave speech [14] have been explored to safeguard patient identity; however, these methods often incur a performance degradation in depression detection. More recently, adversarial learning (ADV), introduced in [15, 16], has demonstrated an enhancement in depression detection performance at the cost of a reduction in speaker classification accuracy. In the work by [17], non-uniform adversarial weights (NUSD) were identified as superior to vanilla adversarial methods in the context of raw audio signals. Additionally, in [18], the utilization of reconstruction loss in conjunction with an autoencoder was found effective in achieving speaker disentanglement, consequently leading to improved depression detection performance.

Despite the notable progress achieved by the aforementioned studies in enhancing depression detection performance while reducing dependency on a patient’s identity, there are significant drawbacks. Firstly, the training of these systems still necessitates speaker labels from patient datasets, posing a challenge to the privacy-preserving aspect of depression detection systems. Secondly, many prior methods rely on an adversarial loss maximization training procedure for speaker disentanglement. While effective in achieving good performance, it is acknowledged that loss maximization is inherently unstable due to the absence of upper bounds for the ad-versarial domain objective function [19]. Thirdly, all the aforementioned methods introduce additional parameters, such as adversarial domain prediction layers or reconstruction decoders, to the model training framework, which are extraneous for the primary task.

Driven by the widespread adoption of unsupervised methods, of unsupervised learning approaches [20], this paper introduces a novel speaker disentanglement method to address the above-mentioned challenges. The proposed method focuses on reducing the cosine similarity between the latent spaces of a depression detection model and a speaker classification model. Operating at the embedding level, this approach eliminates the need for speaker labels from the patient dataset. By reformulating the training process into a loss minimization framework, we overcome the issues of unboundedness associated with adversarial methods. Since the speaker classification models serve as embedding extractors and undergo neither retraining nor fine-tuning, our method achieves efficiency by not requiring domain prediction or reconstruction, resulting in fewer model parameters compared to previous approaches.

Extensive experiments are conducted to validate the efficacy of the proposed method, showcasing its superiority over baseline models (without speaker disentanglement) in terms of depression detection. Furthermore, the method demonstrates performance that is either better than or comparable to adversarial methods. Evaluation across multiple input features and backend models establishes the generalizability of the proposed framework to diverse architectures. The complementary nature of speaker disentanglement methods is highlighted through score-level fusion with text-based models, resulting in an enhanced overall performance when the models are combined.

Subsequent sections of this paper are: Section 2, which describes the proposed method, Section 3, which outlines experimental details, Section 4, which presents and discusses the results, and Section 5, which discusses future research directions.

2. Proposed Method

In conventional speaker disentanglement methods [21, 22], the loss function for the adversarial domain (speaker-prediction) is maximized. Consider the depression prediction loss $L_{MDD}$ and the speaker prediction loss for the adversarial method $L_{SPK-ADV}$. The total loss for the model training can be written as -

$$L_{total-ADV}=L_{MDD}-\alpha \cdot L_{SPK-ADV},$$ (1)

where $\alpha$ is a hyperparameter controlling the contribution of the adversarial loss to the main loss function where the negative sign indicates that the speaker prediction loss is maximized thereby forcing the model to learn more depression discriminatory features and less speaker discriminatory features. The speaker prediction loss $L_{SPK-ADV}$ is usually the Cross-Entropy loss defined as -

$$L_{SPK-ADV}(y,\widehat{y})=-\frac{1}{N}\sum _{i=1}^N\sum _{j=1}^C{y}_{ij}\cdot \log \left({\widehat{y}}_{ij}\right),$$ (2)

$y$ is the ground-truth speaker label and $\widehat{y}$ is the predicted speaker probabilities for $N$ samples and $C$ speakers.

As discussed earlier, this approach has three major issues: 1) this method requires the ground-truth speaker label $y$ to achieve disentanglement, 2) the disentanglement of speaker identity is based on loss maximization ($-\alpha \cdot L_{SPK-ADV}$ which does not have an upper bound, resulting in degraded stability during training and 3) the speaker prediction branch in the model, to obtain $\widehat{y}$, adds additional model parameters that are not useful for depression detection making this approach inefficient. In [18], along with speaker labels, feature reconstruction is used for speaker disentanglement which adds even more unnecessary parameters. In contrast, we propose an unsupervised method of speaker disentanglement that does not need any patient dataset speaker labels and neither involves loss maximization nor adds additional model parameters. The proposed method is depicted in Figure 1.

Figure 1:

The unsupervised speaker disentanglement method (USSD) aims to minimize cosine similarity between latent spaces of depression classification and speaker classification models.

Consider a depression classification model $\left({\theta }_{MDD}\right)$ and a speaker classification model $\left({\theta }_{SPK}\right)$. For a given speech input $X\in {\mathbb{\text{R}}}^{\mathbb{\text{NXF}}}$ ($N$ is the batch size and $F$ is the number of features) the latent embeddings of these models are:

$$H_{MD{D}_X}={\theta }_{MDD}(X)$$ (3)

$$H_{SP{K}_X}={\theta }_{SPK}(X)$$ (4)

$H_{MD{D}_X}$ and $H_{SP{K}_X}\in {\mathbb{\text{R}}}^{\mathbb{\text{NXD}}}$ where $D$ is embedding size. Next, we compute the predicted cosine similarity matrix between the two latent space embeddings by computing the cosine similarity between every pair of embeddings as follows -

$$Y_{pre{d}_{(i,j)}}=\frac{H_{MD{D}_{X_i}}\cdot H_{SP{K}_{X_j}}}{\Vert H_{MD{D}_{X_i}}\Vert \cdot \Vert H_{SP{K}_{X_j}}\Vert }$$ (5)

where $1<=i,j<=N$ and $Y_{pred}\in {ℝ}^{\mathbb{\text{NXN}}}$. The objective of the disentanglement process is to minimize the cosine similarity between the two embedding spaces by enforcing orthogonality between the depression and speaker latent spaces. To achieve this, we specifically set $Y_{target}$ to 0, instead of −1. To enhance convergence during implementation, a small noise value, denoted as $ϵ$ is incorporated [23].

$$Y_{targe{t}_{(i,j)}}=0+ϵ,$$ (6)

$$ϵ\in U(0,1e-8)$$ (7)

We define the proposed speaker disentanglement loss function $L_{USSD}$ as follows -

$$L_{USSD}=MSE\left(Y_{pred},Y_{target}\right)$$ (8)

and the total loss as:

$$L_{total-USSD}=L_{MDD}+\alpha \cdot L_{USSD},$$ (9)

Minimizing the loss function described in Eq. 9 compels the model to emphasize learning more discriminatory information related to depression while reducing its focus on speaker-related distinctions. In contrast to ADV (Eq. 1), speaker disentanglement in the proposed method is achieved via loss minimization.

It is important to note that embeddings from ${\theta }_{SPK}$ can be extracted without the necessity of speaker labels, rendering the proposed speaker disentanglement method unsupervised. Moreover, only the parameters of ${\theta }_{MDD}$ require updating, as the ${\theta }_{SPK}$ model does not need finetuning and can remain a pre-trained model with frozen weights. Lastly, experiments where is set to zero, meaning the squared cosine similarity is directly minimized, yielded subpar performance compared to those with a non-zero $ϵ$. Consequently, results from experiments with $ϵ=0$ are not included in this paper.

3. Experimental Details

3.1. Dataset: DAIC-WoZ

The dataset [24], comprises audio-visual interviews conducted in English with 189 participants experiencing psychological distress, including male and female speakers. For our experiments, 107 speakers were employed for training, while an additional 35 speakers were designated for evaluation purposes, aligning with the dataset specifications. The audio data only from the patients were extracted based on the provided time labels. For text-based experiments, the transcripts provided with the database were used. Results are reported using the validation set in line with previous research [25, 26, 27, 28].

3.2. Input Features

For the audio, four input features are evaluated to show that the proposed framework is independent of the acoustic features used. Mel-Spectrograms, raw-audio signals, ComparE16 features from the OpenSmile library [29], and the last hidden state of the Wav2Vec2 [30] model are used. Mel-Spectrograms are 40 and 80 dimensional, raw-audio features are 1-dimensional, ComparE16 features are 130-dimensional and Wav2vec2 features are 768 dimensional. For the text, a Word2vec model [31] is used to extract word-level embeddings from the transcripts of the patient’s audio. The embeddings are 200 dimensional. Audio and text feature processing is based on publicly available code repository [26]. Since there is an imbalance in the dataset, similar to [25, 26], random cropping and segmentation are applied. To negate the bias effects of randomness, 5 models are trained with different random seeds, and performances are obtained via majority voting (MV).

3.3. Models

Similar to input features, multiple model architectures are designed for the audio modality to show that the proposed method generalizes to different model architectures. Mel-spectrogram features and Raw-Audio signals are used with two model configurations – CNN-LSTM and ECAPA-TDNN [32, 33]. The other two features, ComparE16 and Wav2vec2 are used with an LSTM-only configuration. For the speaker classification model, two pre-trained models are used - ECAPA-TDNN (128-dimensional embedding) and the X-Vector model [34] (256-dimensional embedding) from the hugging face speechbrain library [35]. Note that the number of parameters reported for each experiment does not include off-the-shelf speaker classification models that have not undergone re-training or fine-tuning. For the text model, a simple CNN-LSTM framework was used. In the interest of space and since this paper does not propose any new neural network architecture but rather uses previously established models, we do not explain the model architecture in detail. However, the model weights and code repository will be publicly available here¹.

3.4. Evaluation Metrics

3.4.1. Depression Detection

As is common in the depression detection literature, to measure system performance, the F1 scores [36] for the two classes (Depressed: D and Non-Depressed: ND) F1-D and F1-ND as well as their macro-average, F1-AVG were reported.

3.4.2. Privacy Preservation

To assess the privacy-preserving capabilities of the models, we employ the De-Identification score (DeID[37]), a metric inspired by the voice privacy literature[38]. The DeID score calculation begins with a voice similarity matrix denoted as $M_{AB}$, computed for a set of N speakers. This matrix is derived from the log-likelihood ratio (LLR) of two segments—one from model A and the other from model B—considered to be from the same speaker. The LLR computation uses a Probabilistic Linear Discriminant Analysis (PLDA) model [39].

Subsequently, voice similarity matrices, $M_{oo}$ and $M_{od}$, are calculated. $M_{oo}$ utilizes embeddings solely from the baseline model $(o)$, while $M_{od}$ incorporates embeddings from both the baseline model $(o)$ and the speaker-disentangled model $(d)$.. The next step involves calculating the diagonal dominance $D_{diag}(M)$ for both $M_{oo}$ and $M_{od}$. This measure is determined as the absolute difference between the average diagonal and off-diagonal elements in the matrices. The diagonal dominance value serves as an indicator of how identifiable individual speakers are within a given embedding space, ranging from 0 to 1.

When $D_{diag}\left(M_{oo}\right)$ equals 1, speakers are completely identifiability in the original embedding space, whereas if $D_{diag}\left(M_{od}\right)$ equals 0, speakers are unidentifiable after disentanglement. To measure how good the anonymization (disentanglement) process is, the DeID score is formulated as -

$$\text{DeID}=1-\frac{D_{diag}\left(M_{od}\right)}{D_{diag}\left(M_{oo}\right)}$$ (10)

DeID is expressed as a percentage, where 0% signifies poor anonymization, and 100% denotes fully successful anonymization. As DeID relies on voice similarity matrices constructed from embeddings pre and post-disentanglement, it is exclusively reported for the experiments involving speaker disentanglement.

4. Results and Discussion

4.1. Speaker Disentanglement versus Baseline

Table 1 shows enhanced depression detection performance (F1-AVG) across all experiments when applying speaker disentanglement, either in the form of ADV or USSD. On average, a notable improvement of 8.3% and 8.2% over the baseline was observed for ADV and USSD, respectively, for the six experiments. The highest improvement with ADV, 13.8%, occurred when utilizing Raw-Audio features with the ECAPA-TDNN model, while the lowest improvement, 5.3%, was observed with MelSpectrograms features and the ECAPA-TDNN model. In the case of USSD, the highest improvement was 11.7% with ComparE16 features and the LSTM-only model, and the lowest improvement was 3.8% with Mel-Spectrogram features and the CNN-LSTM model. This highlights the advantage of USSD over ADV in scenarios where speaker labels for the training set are unavailable.

Table 1.

F1-scores using majority voting (MV) and DeID, for speaker disentanglement through ADV and USSD using the DAIC-WOZ dataset. Recall that, unlike ADV, USSD does not use speaker labels for disentanglement. The parameter count for USSD does not include speaker ID models, as they are neither retrained nor fine-tuned. The best results are bold-faced.

Feature	Model	Disentanglement	Number of Parameters ↓	F1-AVG (MV) ↑	F1-ND ↑	F1-D ↑	DelD ↑
Mel-Spectrogram		No	280 k	0.658	0.756	0.560	NA
	CNN-LSTM	ADV	293k	0.694	0.773	0.615	14.01%
		USSD	280 k	0.683	0.783	0.583	10.29%
		No	515k	0.709	0.809	0.609	NA
	ECAPA-TDNN	ADV	529k	0.746	0.826	0.667	3.69%
		USSD	515k	0.746	0.826	0.667	5.97%
Raw-Audio		No	445 k	0.669	0.792	0.546	NA
	CNN-LSTM	ADV	459 k	0.709	0.809	0.609	55.83%
		USSD	445 k	0.746+	0.826	0.667	45.35%
		No	595k	0.694	0.773	0.615	NA
	ECAPA-TDNN	ADV	609k	0.790	0.880	0.700	22.32%
		USSD	595k	0.773+	0.851	0.696	19.90%
		No	1.15M	0.694	0.773	0.615	NA
ComparE16	LSTM-only	ADV	1.18M	0.762+	0.857	0.667	68.37%
		USSD	1.15M	0.776	0.885	0.667	92.87%
		No	3.6M	0.683	0.783	0.583	NA
Wav2vec2	LSTM-only	ADV	3.7M	0.747	0.863	0.632	52.43%
		USSD	3.6M	0.720	0.840	0.600	58.65%

The symbols

^{‘↑’ and ‘↓’}indicate a higher or lower value is better, respectively.

⁺indicates improvements are not statistically significant.

4.2. USSD versus ADV

Comparing USSD to its adversarial counterpart, ADV, we observe that the proposed method outperforms ADV in 2 out of 6 experiments: Raw-Audio with CNN-LSTM (0.746 for USSD vs. 0.709 for ADV) and ComparE16 with LSTM-only (0.776 for USSD vs. 0.762 for ADV). Conversely, ADV exhibits better performance than USSD in 3 out of 6 experiments, with both methods yielding the same results in 1 out of 6 experiments. In the aggregate, ADV achieves the best overall results with an F1-Score of 0.79, whereas the corresponding USSD model achieves 0.773—a slight decrease of 2.15%, despite using 15k fewer parameters and not relying on speaker labels. Even without utilizing speaker labels or additional parameters for predicting speakers, USSD showcases comparable or superior performance to ADV. This highlights the advantage of USSD over ADV in scenarios where speaker labels for the training set are unavailable .

4.3. Privacy Preservation - DeID

Privacy is a crucial aspect of speech-based depression detection, and Table 1 demonstrates positive DeID results for both USSD and ADV across all models. Notably, ComparE16 features with USSD achieve the highest DeID at 92.87%. Despite a marginal depression detection performance drop in USSD compared to ADV, USSD excels in privacy preservation. An intriguing finding is that USSD’s effectiveness is independent of the type or dimension of speaker embeddings used. Mel-spectrogram and Raw-Audio experiments employed ECAPA-TDNN speaker embeddings, while ComparE16 and Wav2Vec2 experiments used X-vector embeddings with dimension reduction. USSD’s reliance on a pre-trained speaker classification model may contribute to leveraging pre-trained speaker embeddings, enhancing the masking of depression embeddings, and resulting in a higher DeID.

4.4. Text Fusion

Fusing speaker-disentangled audio models with Word2vec-based text models yields a notable improvement in depression F1-score, particularly for the top 2 audio-only models, as shown in Table 2. Specifically, when the ECAPA-TDNN model trained on Raw-Audio is combined with Word2vec, the depression detection F1-Score reaches 0.860. This result compares favorably to the state-of-the-art (SOTA) depression detection F1Score of 0.89 (F1-Max) reported in [28], which involves a four-model ensemble, including parameter-heavy models like RoBERTa [40] and WavLM [41]. In contrast, our approach utilizes only Raw-Audio/ECAPA-TDNN for audio classification and Word2vec/CNN-LSTM for text classification. Similar to ADV, the USSD model demonstrates a significant improvement in F1-Score when fused with text models. These findings underscore the complementarity of speaker-disentangled audio-based depression classification with text-based methods. Contrary to the assumption that speaker-disentanglement models shift focus from non-linguistic features to content-related features [42], our results suggest that the information learned by speaker-disentanglement models can be complementary to content-related features.

Table 2.

F1-AVG scores (MV) with and without score-level fusion with the Word2vec text model. Results are shown for the top 2 audio-only models together with their DeIDs that illustrate the privacy-preserving feature of USSD.

Audio-Model	Disent.	Audio-only	Word2vec Fusion (Text-only)	DeID (Audio-only)
Raw-Audio ECAPA-TDNN	ADV	0.790	0.860 (0.762)	22.32%
ComparE16 LSTM-only	USSD	0.776	0.830 (0.762)	92.87%

5. Conclusion and Future Work

The proposed unsupervised method for speaker disentanglement in depression detection is a promising approach for improving model efficiency and privacy attributes. By reducing reliance on speaker labels and streamlining the model through the minimization of squared cosine similarity between latent spaces, we achieve superior performance compared to both baseline models and adversarial methods. A higher DeID indicates better masking of speaker identity, contributing to the algorithm’s enhanced privacy. The compatibility of speaker-disentanglement methods with text-based approaches further solidifies the versatility of the method. Future work will study dimension mismatch between speaker and depression embeddings, speaker-embedding extractors from SSL models such as Instance Discrimination Learning [43] which are trained without supervision and capture significant speaker information, as well as understanding the nature of information learned through speaker disentanglement methods.