DPD (DePression Detection) Net: a deep neural network for multimodal depression detection

Abstract

Depression is one of the most prevalent mental conditions which could impair people’s productivity and lead to severe consequences. The diagnosis of this disease is complex as it often relies on a physician’s subjective interview-based screening. The aim of our work is to propose deep learning models for automatic depression detection by using different data modalities, which could assist in the diagnosis of depression. Current works on automatic depression detection mostly are tested on a single dataset, which might lack robustness, flexibility and scalability. To alleviate this problem, we design a novel Graph Neural Network-enhanced Transformer model named DePressionDetect Net (DPD Net) that leverages textual, audio and visual features and can work under two different application settings: the clinical setting and the social media setting. The model consists of a unimodal encoder module for encoding single modality, a multimodal encoder module for integrating the multimodal information, and a detection module for producing the final prediction. We also propose a model named DePressionDetect-with-EEG Net (DPD-E Net) to incorporate Electroencephalography (EEG) signals and speech data for depression detection. Experiments across four benchmark datasets show that DPD Net and DPD-E Net can outperform the state-of-the-art models on three datasets (i.e., E-DAIC dataset, Twitter depression dataset and MODMA dataset), and achieve competitive performance on the fourth one (i.e., D-vlog dataset). Ablation studies demonstrate the advantages of the proposed modules and the effectiveness of combining diverse modalities for automatic depression detection.

Introduction

Depression (also known as major depressive disorder) is a prevalent mental disorder with serious impact on people’s personal life and the society. According to the World Health Organization (WHO), there are approximately 280 million people in the world that suffer from depression and it is ranked as the fourth leading cause of death among people aged 15–29 in 2023.¹ Researchers predict it to be the second leading cause of burden of disease by 2030 [1]. The economic impact of depression was estimated at $51.5 billion annually in the United States [2]. Additionally, the COVID-19 pandemic caused a 27.6% increase in cases of major depressive disorders globally [3].

The most common approach of depression screening is based on physician-administered interviews using questionnaires such as the Physical Health Questionnaire Depression Scale (PHQ) [4]. This type of screening highly relies on a physicians’ subjective interpretation [5]. Also, some patients are reluctant to share their honest thoughts and talk about their symptoms during the screening interviews as they are ashamed of the stigma attached to depression.

Researchers have shown increasing interest in machine learning-based automatic depression detection using behavioural cues such as facial activity, gesturing, head movements and speech, as studies have shown that they are strongly correlated with depression [4]. By integrating different cues, information from single modalities can complement each other, and the fused information has the potential of revealing underlining depression-related patterns. Automatic depression detection remains a difficult task for researchers given the following challenges. Firstly, public available multi-modal data of depressed individuals in clinical settings is limited. Secondly, a thorough depression interview session is usually long, so extracting useful context information from long sequences of audiovisual data for depression detection can be challenging [6]. Furthermore, existing studies solve the problem of multi-modal depression detection in one single setting, e.g., the clinical setting or the social media setting, and mostly are validated on a single dataset, which might lack robustness, flexibility and scalability.

Based on the work of Joshi et al. [7], where the Transformer architecture and Graph Neural Networks (GNNs) are leveraged for modeling global dependencies and local dependencies respectively, we introduce a GNN-enhanced Transformer model named DePressionDetect Net (DPD Net) with novel convolution modules for automatic depression detection, which can be applied to the audio, speech and textual modality in different settings and work across different datasets while addressing the challenges mentioned above. Also, we design an extended version of the DPD Net called DePressionDetect-with-EEG Net (DPD-E Net) to incorporate Electroencephalography (EEG) data for depression detection.

In this study, we investigate the problem of automatic depression detection in a multi-modal setting. Given audio, visual cues, textual cues and EEG signals, our proposed model should be able to predict the severity of depression in terms of Physical Health Questionnaire Depression Scale (PHQ) scores or predict if the subject is experiencing depression. Our main contributions are:

1. A novel GNN-enhanced Transformer model named DePressionDetect Net (DPD Net) is proposed based on the work of [7], which is originally adopted for muti-modal emotion recognition, we upgrade the model so it can be better adapted to muti-modal depression detection using audio, visual and textual cues. To be more specific, we change the structure of the previous proposed model and use it as a multi-modal encoder module, and design a novel unimodal encoder module which consists of conv-based sub-modules for encoding each single modality along with a detection module.

2. An ensemble model named DePressionDetect-with-EEG Net (DPD-E Net) is proposed for depression detection in EEG signal, speech and text data, which takes both the spatial and temporal information of the EEG signals into consideration.

3. We perform feature engineering and evaluate our proposed models on four depression datasets, which are under two different application settings: the clinical setting and the social media setting. Comparisons to other baseline methods show that DPD Net and DPD-E Net can outperform the state-of-the-art models on three datasets (i.e., E-DAIC dataset, Twitter depression dataset and MODMA dataset), and achieve competitive results on the rest one (i.e., D-vlog dataset).

4. Extensive experiments together with ablation studies are conducted to investigate the impact of representation methods, different modules of our proposed model, and the effect of modalities.

Related work

In current literature, many studies in the domain of automatic depression detection from speech utilize Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) as the backbone of their proposed models. Adrián et al. [8] design an ensemble model which consists of three stacked CNN base learners to balance bias and variance to perform depression detection from speech on the Distress Analysis Interview Corpus-Wizard of Oz database (DAIC-WOZ) [9]. This database is considered as one of the benchmark databases in the domain of multi-modal depression detection under the clinical setting, and its newly extended version called E-DAIC dataset [4] that contains more subjects is used in our study. Zhao et al. [10] introduce a bidirectional long short term memory (BiLSTM) model with a hierarchical attention mechanism to differentiate depressed individuals from healthy controls using various frame-level audio features.

Even though head movements and gesturing are also considered as discriminative cues for depression detection [4], most of the research focus on the utilization of facial activities. Zhou et al. [11] propose a model named Multi-Region DepressNet to predict depression levels using sequential images of facial regions cropped from interview videos. Each single DepressNet module is based on a pre-trained ResNet50 [12] network with modifications on the prediction block. By stacking four DepressNet modules aimed for different facial regions, features learnt from each region are combined to make the final prediction. Given that the dynamics of facial expression could play an essential role in depression detection, Melo et al. [13] design a novel encoding approach that performs temporal pooling on sequential video images. The raw images along with the encoded images are input to two separate CNN-based regressors for prediction, and scores are fused by taking the average. This model is able to produce state-of-the-art results on two video-based depression datasets.

As textual data is convenient to be collected and normally could be accessible after proper de-identification, it is widely used in the domain of automatic depression detection. Ansari et al. [14] develop a hybrid model to recognize depressed users on social media platforms using the user-generated textual content. The model consists of a logistic regression classifier that takes four types of lexicon as input, and an attention-enhanced LSTM that takes word embeddings as input. Zhang et al. [15] design a model that integrates a modified Robustly optimized BERT approach (RoBERTa) model [16] with a BiLSTM model. Evaluations on two clinical datasets of different languages indicate the model’s potential.

Since depression is a mental condition of complex patterns, most of the studies utilize multi-modal information to fully exploit each modality’s predictive power. To deal with the long-interview problem we mentioned in the Introduction, Gong et al. [17] propose a novel approach for multi-modal depression detection based on the topics of the clinical interview recording. They extract the topic of each answer of the interview questions and obtain the corresponding textual, visual and acoustic features of the answer. Different features of the same topic are aggregated to construct new feature vectors so the sequence length of each sample is decreased to the total number of topics [17]. Transformer-based models gained their popularity in multi-modal depression detection given their high performance in the field of Natural Language Processing (NLP) and Computer Vision (CV). Audibert [18] designed by Ermal et al. is a depression detection model that takes two streams of input. A textual stream is sent to pre-trained BERT while an audio stream is fed into pre-trained audio networks such as Wav2vec [19] and SincNet [20], both of which are followed by a BiLSTM and their output are aggregated for depression prediction.

Even though text, visual cues and audio cues are the most commonly used modalities for depression detection, estimating depression from Electroencephalography (EEG) data has become a promising research direction, as EEG signals of high temporal resolution can well capture complex brain activities and turn out to be suitable for depression-related research [21]. The Multi-modal Open Dataset for Mental-disorder Analysis (MODMA) is introduced by [22] to boost research in depression detection using physiological data. The dataset comprises both speech data and EEG data collected under clinical setting, but most current studies only use one modality.

As the depression data under clinical settings is limited, depression datasets curated by collecting social media contents are investigated by researchers to discover potential depression-related patterns in daily life. The D-vlog Dataset [23] is an audio-visual depression dataset collected from daily video blogs, and the Twitter Depression Dataset [24] is build by extracting user posts which contain tweets accompanied with images. In the work of [25], a Chinese social media depression dataset is created by extracting user-generated textual and image content from a depression forum.

Our work differs from all the previous works in terms of the method and the scope of our research. By introducing a novel unimodal encoder module, a detection module, and modifying the architecture of the network [7] as a multi-modal encoder module, our proposed model is the first work to our knowledge that uses a GNNs-enhanced transformer in multi-modal depression detection. Also, we use four aforementioned benchmark datasets including the E-DAIC dataset [4], the D-vlog Dataset [23], the Twitter depression dataset [24] and the MODMA dataset [22], which cover different settings for a comprehensive study in this domain, where the proposed models have the potential to be applied to real scenes.

Methodology

In this section, we present our two proposed deep learning-based models DPD Net and DPD-E Net together with their implementation details.

DPD Net

We introduce our novel model DePressionDetect Net (DPD Net) in detail. Figure 1 presents the overall structure of the DPD Net, which consists of three components: a unimodal encoder module for each of the modalities, a multi-modal encoder module and a detection module.

Fig. 1.

The overall architecture of the proposed DePressionDetect Net (DPD Net), which consists of three main modules: a unimodal encoder module, a multi-modal encoder module and a detection module

Given the text input denoted as $X_T$, audio descriptors as $X_A$ and visual descriptors as $X_V$, DPD Net starts by encoding each single modality. The proposed unimodal encoder module consists of a pre-trained MpNET model [26] for encoding the text, and two novel convolution-based submodules that encode the audio and visual descriptors.

After processing the input of each modality through their corresponding encoders, the features obtained from each modality are concatenated as a fused feature and then fed into the multi-modal encoder module, which is based on the work of COGMEN model [7] originally proposed for multi-modal emotion recognition. The architecture of [7] is presented in “Appendix A”. To the best of our knowledge, there is no work in the domain of multi-modal depression detection that uses GNNs along with transformers, so our proposed model can be considered as a new attempt in this domain. The Multi-modal encoder module begins with a transformer submodule including a transformer encoder and a linear layer, to produce a multi-modal embedding. Based on this multi-modal representation, two types of homogeneous graphs are formed using past and future information, and are sent into a GNN submodule for further encoding. This newly obtained embedding is then concatenated with the embedding from the transformer submodule to make the final prediction using the detection module.

Unimodal encoder module

The unimodal encoder module is designed to learn representations for each modality before the fusion of the modalities. Two conv-based submodules are proposed for encoding the audio and visual modality, and a pre-trained language model called MpNET [26] is adopted for encoding the textual content.

To encode audio information $X_A$, we design a conv-based submodule for the audio modality shown in Fig. 2a. It starts with a 1D convolutional layer with 256 filters with kernel size 3 and stride 1, followed by a batch normalization, a rectified linear unit (ReLU) activation and a dropout layer. The convolutional layer is designed for capturing local patterns of the input data by leveraging convolutional filters. In case of a 1D convolutional layer, each filter of certain kernel size slides over the input data in one axis with a certain stride, and performs linear operations to produce feature maps in which essential patterns of the input data are encoded. Then, the output of the dropout layer is fed into a 2-layer LSTM to capture longer dependencies of the audio modality.

Fig. 2.

The conv-based submodule for audio and visual modality

For the visual encoder, we adopt another conv-based submodule inspired by the work of [27]. As can be seen from Fig. 2b, video descriptors $X_V$ are firstly normalized by layer normalization and are encoded by a 1D convolutional layer with 256 filters of kernel size 1 and stride 1. Then, a Gated Linear Unit (GLU) activation is used followed by another 1D convolutional layer with 128 filters of kernel size 3, stride 1 and number of groups 128. Then, a batch normalization and a ReLU activation is used followed by a 1D convolutional layer with 128 filters of kernel size 1, stride 1. Finally, a dropout layer is used and a 4-layer LSTM is adopted to compliment the convolutional layers so both local information and longer-term relations can be obtained for the visual modality.

We adopt MpNET [26] for encoding the texts. By experimenting with pre-training settings such as the batch size, the masking method and the choice of input, MpNET is proposed as an upgraded version of BERT [28] using the best settings found in their experiments [26].

Multimodal encoder module

After obtaining the representations of each modality from the unimodal encoder module, these representations are fused by concatenation and fed into the multi-modal encoder module to further explore the local and global dependencies. This module starts with a transformer submodule that encodes the input into attention values as a new representation, followed by a GNN submodule that learns potential new patterns through graph information.

The transformer submodule begins with the classic transformer encoder from the work of [29]. The fused multi-modal representation input $X_{\mathit{input}}$ is sent to the multi-head attention layers after being added to the positional encoding used for encoding the position of each element of the input sequence. X is then added to the obtained multi-head attention value using a residual connection and normalized by layer normalization, and passed to a two-layer feed-forward block with another residual connection followed by another layer normalization. Finally, it is sent to a fully connected layer for downscaling.

Since the transformer submodule is responsible for modeling global dependencies of the multi-modal representation, a GNN Submodule, which is composed of a Relational Graph Convolutional Network (RGCN) [30] and a Graph Transformer [31], is added in charge of modeling local dependencies.

Firstly, we model the output feature vectors obtained from the transformer submodule as a directed graph G $=$ $($ $\mathcal{V}$ $,$ $\mathcal{E}$ $,$ $\mathcal{R}$ $)$ so it can be leveraged in the GNN submodule. Each feature vector is considered as a node $v_i\in$ $\mathcal{V}$, with its labeled edges $(v_i,r,v_j)\in$ $\mathcal{E}$ and an edge type $r\in$ $\mathcal{R}$. Two types of edge relations are constructed by taking the past and future 6 utterances or time steps into consideration. Then, nodes along with graphs are sent to the RGCN. The idea behind RGCN is to update the center node feature by taking its neighbor information into consideration in the form of aggregation using the following Eq. 1:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill z_i^{\prime }=\sum _{r\in \mathcal{R}}\sum _{j\in {\mathcal{N}}_i^r}\frac{1}{|{\mathcal{N}}_i^r|}{W}_r{z}_j+W_0{z}_i\end{array}\end{array}$$ 1

Here $z_i\in {\mathbb{R}}^{d_m^{\prime }}$ denotes the node feature obtained from the previous transformer submodule, $W_r$ and $W_0$ are trainable parameters, ${\mathcal{N}}_i^r$ is the set of neighbors of node i with relation r. Then, the updated node features $z_i^{\prime }$ are sent to a graph transformer and then fed into a batch normalization layer followed by Leaky ReLU activation.

Detection module

The detection module shown in Fig. 1 contains a LSTM followed by a fully-connected output layer, which takes as input the obtained multi-modal representations from the multi-modal encoder module to estimate $\widehat{Y}$, the depression scores or labels.

DPD-E Net

In this section, we introduce the DePressionDetect-with-EEG Net (DPD-E Net) in detail. Figure 3 presents the overall structure of the DPD-E Net, which comprises four individual networks, each making their own predictions, and the results are integrated to yield a final prediction.

Fig. 3.

The proposed DPD-E Net’s overall architecture

DPD-E Net is an extended version of the DPD Net which is designed to handle EEG, speech and text modalities. In Fig. 3, $X_{E1}$, $X_{E2}$ and $X_{E3}$ denote three streams of EEG input while $X_A$ and $X_T$ represent the input data from audio and text. $X_{E1}$, $X_{E2}$ are sent to a GAT-based encoder for further encoding, and the obtained new embeddings are then passed to the classifier to produce the probability of the subject belonging to the depressed class. $X_{E3}$ is sent to a conv-based encoder followed by another classifier. $X_A$ and $X_T$ are processed by the same architecture as in DPD Net using the audio-text modality to produce the depression probabilities. Finally, predictions from each modality $Y_{E1}$, $Y_{E2}$, $Y_{E3}$ and $Y_{\mathit{AT}}$ are fused to generate the decision $\widehat{Y}$.

EEG modality

Our DPD-E Net leverages EEG data in a 3-stream fashion. The first stream $X_{E1}$ is a local spatial stream represented by Brain Functional Networks (BFNs) constructed using brain sub-regions right temporal (RT) and left central (LC) while the second stream $X_{E2}$ is a global spatial stream using BFNs constructed from 16 main electrodes. $X_{E3}$ is the third stream which represents temporal features extracted from EEG signal.

From the 128 available electrodes of the EEG signal, we select 16 main electrodes in Fig. 4, which are highlighted in bold and in blue. Because they are the most representative electrodes of EEG signal and have been proven to be effective in depression detection [33]. For each epoch of the EEG signal, we extract 29 linear and non-linear features listed in “Appendix C” for each channel, which are finally flattened to a 464-dimensional input feature vector $X_{E3}$ of the temporal stream classifier. For this stream, DPD Net is leveraged in a unimodal way for the classification with the GNN Submodule removed.

Fig. 4.

128-channel EEG electrode positions and segmentation of brain regions: left frontal (LF), right frontal(RF), left temporal (LT), right temporal (RT), left central (LC), right central (RC), left parietal occipital (LPO) and right parietal occipital (RPO) [32]

Brain Functional Networks (BFNs) are a type of networks that are constructed based on the the correlations among EEG channels. They describe the brain functional connectivity and have been widely used for brain disorder diagnosis [21]. We construct BFNs by considering channels as nodes and using their Pearson Correlation Coefficient (PCC) to define the edges of the networks. For channels x and y, the PCC is calculated as follows:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill PCC=\frac{\sum (x-\overline{x})(y-\overline{y})}{\sqrt{\sum {(x-\overline{x})}^2\sum {(y-\overline{y})}^2}}\end{array}\end{array}$$ 2

where $\overline{x}$ is the mean of the x signal and $\overline{y}$ is the mean of the y signal. The PCC value ranges between – 1 and 1, with 0 meaning no correlation between these two signals. We take the absolute value, and set the threshold as 0.6 to define an edge. For the global spatial stream, we use the before-mentioned 16 main electrodes.

Brain regions can be divided into 8 sub-regions shown in Fig. 4. We choose right temporal (RT), left central (LC) as the main sub-regions for exploring the local spatial information, as it is found that channels belonging to these two sub-regions exhibit the most discriminative features between depressed and non-depressed subjects [34]. Then, all electrodes within each of the two sub-regions are leveraged to construct two separate BFNs as input to the local spatial streams.

For EEG data of each subject, channels are considered as nodes and channels’ correlations are used for determining edges. Let C denote the number of channels (electrodes) of the brain sub-regions or the number of selected main electrodes, node features of shape (C, 29) along with an edge matrix are the input BFN $X_{E1}$ or $X_{E2}$ to the base classifier shown in Fig. 5. BFNs are firstly passed to the Graph Attention Network (GAT) [35] which updates the node features by aggregating information from first-order neighborhood nodes with different attention coefficients using the following equations:

$$\begin{array}{cc}& \begin{array}{c}\hfill e_{\mathit{ij}}=a^T[W{h}_i\Vert W{h}_j],j\in {\mathcal{N}}_i\end{array}\hfill \end{array}$$ 3

$$\begin{array}{cc}& \begin{array}{c}\hfill {\alpha }_{\mathit{ij}}=\frac{exp(LeakyReLU(e_{\mathit{ij}}))}{{\sum }_{k\in {\mathcal{N}}_i}exp(LeakyReLU(e_{\mathit{ik}}))}\end{array}\hfill \end{array}$$ 4

$$\begin{array}{cc}& \begin{array}{c}\hfill h_i^{\prime }=\underset{k=1}{\stackrel{K}{||}}\sigma \left(\sum _{j\in {\mathcal{N}}_i},{\alpha }_{\mathit{ij}}^k,W^k,h_j\right)\end{array}\hfill \end{array}$$ 5

Let i be the current node, and j be a node in the set of first-order neighbors ${\mathcal{N}}_i$. $h_i$ denotes the current node features, and $h_j$ are its first-order neighbor node features. W is a learnable parameter which performs a linear transformation to both $h_i$ and $h_j$, and || is the concatenation operation. a is another learnable parameter to map the concatenated vector to a real number space. The obtained $e_{\mathit{ij}}$ are then normalized using Eq. 4. The final updated node features $h_i^{\prime }$ are computed using Eq. 5 by concatenating each newly obtained node features from k heads. Then, the updated node features $h_i^{\prime }$ are sent to a fully connected layer followed by a ReLU activation. Finally, DPD Net’s detection module consisting of a LSTM and a fully connected layer is used as the classifier to make the prediction.

Fig. 5.

The EEG-based base classifier of the proposed DPD-E Net for the local spatial stream $X_{E1}$ and the global spatial stream $X_{E2}$

Speech-Text modality

For audio cues and textual information, we leverage these two modalities in a multi-modal way by reusing the proposed DPD Net which can be seen from Fig. 3. Audio and textual cues are firstly encoded by separate unimodal encoders, and then fused into multi-modal embeddings that are further encoded by the multi-modal encoder for the final prediction using the detection module.

Both DPD Net and DPD-E Net are implemented using PyTorch, and the pretrained models are provided by the Hugging Face library [36]. “Appendix D” presents the specifid models used in this study.

Experiments

We conduct extensive experiments to evaluate DPD Net and DPD-E Net. Firstly, we conduct experiments to explore multiple features obtained from pre-trained models and the fusion of different features. Subsequently, the best models are compared to several baseline methods. Ablation studies are presented to investigate the contribution of the respective modalities and proposed modules.

Datasets

Experiments are conducted on four distinct multi-modal depression datasets. The well-established E-DAIC dataset is used for evaluating the depression detection performance of DPD Net in the clinical setting; the Twitter depression dataset and the D-vlog dataset are used to further evaluate DPD Net’s effectiveness in the social media setting; the MODMA dataset is used for validating DPD-E Net’s prediction ability.

• E-DAIC: The Extended DAIC Database (E-DAIC) [4] is a multi-modal depression dataset of clinical interview recordings. Audio, video and the transcript of each participant’s interview are recorded in the dataset along with their final PHQ scores. The dataset consists of 275 samples of interview sessions with 40.5 h recordings in total, and is split into a training, validation and test set, with 163, 56 and 56 samples, respectively. E-DAIC is the only public available depression dataset with data of three modality and is considered as the benchmark dataset in this domain [4].

• Twitter Depression Dataset: The Twitter Depression Dataset [24] is build upon another dataset which only contains textual modality. Authors curate the new dataset by extracting extra tweets accompanied with images using user IDs obtained from the text-based dataset [24]. This dataset is the most widely used multi-modal depression detection dataset in the social media setting. In total, it includes 2804 user samples with approximately 110 M tweets.

• D-vlog: The D-vlog Dataset [23] is an audio-visual depression dataset consisting of daily video blogs from Youtube. The selected videos have the subject speaking directly to the camera to make sure each subject’s facial activities can be leveraged for depression estimation [23]. The dataset contains 961 samples of 816 subjects, with each sample having an average length of around 10 min, and is split into the training set, validation set and test set with a ratio of approximately 7:1:2. No raw audio or video data are provided for privacy concerns.

• MODMA: The Multi-modal Open Dataset for Mental-disorder Analysis (MODMA) [22] is used in this project, which is the only available open dataset which comprises both speech data and EEG data. The speech data are recorded during the interviews in which each subject is asked 18 depression-related questions, and only the audio of the answering part are kept. All the interviews are conducted in Chinese. EEG signals are collected under resting-state for 5 min using a 128-channel HydroCel Geodesic Sensor Net with a sampling frequency of 250Hz [22]. In total, 53 subjects are included in the speech data and 52 participants’ EEG signals are collected.

Feature extraction

Features from different modalities are used as the input to out proposed models.

• Audio Features: Low-level audio features including Mel-Frequency Cepstrum Coeffiecient (MFCCs) and Extended Geneva Minimalistic Acoustic Parameter Set (eGeMAPS) are extracted for the E-DAIC and the MODMA dataset. We extract 40 MFCCs using Librosa² and 88 eGeMAPS features using Opensmile³ for each utterance (answer), which are then concatenated into a 128-dimensional audio feature. All the 88 parameters are given and explained in the original eGeMAPS paper [37]. For the D-vlog dataset, the provided 25 low-level acoustic descriptors including loudness, MFCCs, spectral flux, etc., are used as audio features.

• Visual Features: For the E-DAIC dataset, we use the provided Facial Action Units (FAUs), the Gaze position, and pose position as visual features. For the D-vlog dataset, 68 provided facial landmarks are used as visual features. For the Twitter depression dataset, instead of learning image representations from scratch, we leverage the pre-trained CLIP [38] model as our visual feature extractor.

• Textual Features: We adopt MpNET [26] to encode the texts of the E-DAIC dataset, Twitter depression dataset, and the MODMA dataset. This pre-trained language model is discussed in Sect. “Unimodal encoder module”.

• EEG Features: Two types of EEG features are used in this study, which are described in details in Sect. “EEG modality”, including temporal features presented by a combination of different linear and non-linear features, and spatial features presented by Brain Functional Networks (BFNs).

Baselines

Different baseline methods are selected for comparison on the aforementioned four datasets. All of them are transformer-based models and produce the state-of-the-art results on each of the datasets, respectively. For the E-DAIC dataset, TensorFormer [39] is used as the baseline method. The main component of this model is the proposed TensorFormer block, which performs global cross-modality interaction through the attention tensor to further encode the text features obtained from a pre-trained BERT model, and the audio-visual features obtained from two bidirectional long short term memory (BiLSTM) models. The TM Transformer [40] is selected as baseline for the Twitter Depression dataset. It consists of three pre-trained models to encode each single modality, a cross-modality attention module capturing informative patterns through modalities’ interaction and a classic transformer encoder followed by a fully connected layer making the final prediction [40]. Three different models based on the used positional embeddings and the post sampling strategies are proposed in this work. For the D-vlog dataset, we use the spatio-temporal squeezed transformer (STST) [41] as the baseline, which adopts a novel cross-attention mechanism by a module named STST block that extracts spatio-temporal features using an additional convolutional-based block applied to the obtained values (V). We use the ES vision transformer [42] as the baseline for the MODMA dataset, which is comprised of a CNN-LSTM module for encoding the raw EEG signal, a vision transformer module to learn features from EEG spectrogram, another vision transformer module for extracting features from speech spectrogram, and a classifying module consisting of fully connected layers.

Experimental setup

To evaluate and validate our proposed models in a more comprehensive manner, we design four experiments: pre-trained models experiments, modality experiments, comparison to state-of-the-art models and ablation studies of our proposed models. The DPD Net framework is implemented with PyTorch. Implementation details are given in “Appendix D”.

Results and analysis

In this work, our proposed models tackle two types of problems: the classification problem and the regression problem. For regression tasks on the E-DAIC dataset, Concordance Correlation Coefficient (CCC) score and Root Mean Square Error (RMSE) are leveraged as the evaluation metrics. For classification tasks on the Twitter Depression dataset, D-vlog dataset and MODMA dataset, F1 score, precision, and recall are used as metrics. The detailed description and calculation of all these metrics are presented in “Appendix E”.

Pre-trained models experiments

Firstly, we experiment with features obtained from multiple pre-trained models, as the quality of the initial features obtained from these pre-trained models plays an essential role in the depression detection task. In this setting, at least two modalities are used for each dataset, since the main scope of this research is to study multi-modal depression detection.

The intuition behind the design of these experiments is from the work of [7], where we observe that the textual modality is vital in boosting their model’s performance. As our model is based on their work, we suppose that the potential of the textual modality should be explored by experimenting with features obtained from multiple pre-trained language models. Then, we extend this idea to experiment features obtained from different pre-trained vision models to reuse useful patterns exhibited by these pre-trained models. Hence, in general this experiment is designed to investigate what representation methods we should utilize for encoding texts and images.

Experimental results for E-DAIC dataset are presented in Table 1. ‘A’ stands for the audio modality, ‘T’ is the textual modality and ’V’ denotes the visual modality. When fusing all the three modalities, the model using MpNET features considerably outperforms its counterpart with the highest CCC score 0.682, and the RMSE is reduced to 4.79. Based on this experiment, we choose DPD Net using MpNET as the best model for further experiments on E-DAIC dataset.

Table 1.

The results on E-DAIC testset using different pre-trained language models and different modality combination

	Modality	CCC	RMSE
DistilRoBERTa	T + A + V	0.617	5.51
MpNET	T + A + V	0.682	4.79

Bold value indicates the best-performing values within each metric category

We experiment with multiple pre-trained vision models and language models on the Twitter depression dataset. As shown in Table 2, the model using the combination of MpNET as text encoder and CLIP as image encoder outperforms all the other models. With the same visual encoder, models that utilize MpNET achieve better results than models using DistilRoBERTain in terms of F1 score. F1 score is computed by taking both the precision and recall into consideration, also here the experiment used suggests that in general MpNET-based model is the best candidate for this twitter depression detection task. Since the superiority of MpNET can also be observed in the experiments on E-DAIC dataset, we could conclude that MpNET should be chosen over DistilRoBERT as the text encoder of the DPD Net, so the network can work properly on both clinical setting and social media setting. As for the visual encoders, models using CLIP are always be able to produce better F1 score, and for the same reason stated above, we choose CLIP over EfficientNet and FLAVA as the visual encoder to extract initial features from the images.

Table 2.

The results on Twitter depression dataset using different pre-trained vision and language models

Text modality	Viusal modality	Precision	Recall	F1
DistilRoBERTa	CLIP	0.815	0.810	0.812
DistilRoBERTa	EfficientNet	0.796	0.811	0.802
DistilRoBERTa	FLAVA	0.794	0.829	0.809
MpNET	CLIP	0.839	0.874	0.855
MpNET	EfficientNet	0.787	0.836	0.811
MpNET	FLAVA	0.839	0.794	0.813

Bold value indicates the best-performing values within each metric category

Modality experiments

Using the best models obtained from the previous experiments, we experiment combinations of each modality to investigate the impact of different modalities on the models’ performance, and to verify that different modalities can complement each other in our proposed model.

Table 3 presents the results on the E-DAIC dataset. We can see that for unimodal cases, the model using textual modality outperforms the other two modalities by a significant margin in terms of CCC score and RMSE, which is consistent with the finding that we mentioned at the beginning of the previous Sect. “Pre-trained models experiments” that the textual modality plays an essential role in detecting depression. With three modalities, we achieve a CCC score of 0.682 and a RMSE of 4.79 on the E-DAIC dataset, which is the best results obtained from this study on this dataset.

Table 3.

The results on E-DAIC testset using unimodal and multi-modal models

Modality	CCC	RMSE
T	0.475	5.88
A	0.161	7.42
V	0.241	6.19
T + A	0.596	5.16
T + V	0.563	5.21
T + A + V	0.682	4.79

Bold value indicates the best-performing values within each metric category

As for results on the Twitter depression dataset in Table 4, DPD Net using both modalities outperforms the unimodal models in all classification metrics, achieving a performance gain of around 13.5% in terms of F1 score and 23.7% in terms of recall.

Table 4.

The results on Twitter depression dataset using unimodal and multi-modal models

Modality	Precision	Recall	F1
T	0.824	0.742	0.777
V	0.831	0.721	0.771
T + V	0.834	0.918	0.874

Bold value indicates the best-performing values within each metric category

From Table 5, we can see that on the D-vlog dataset, DPD Net using visual features and audio features outperforms the models using only the audio or visual modality in F1 score, precision and recall.

Table 5.

The results on D-vlog dataset using unimodal and multi-modal models

Modality	Precision	Recall	F1
V	0.681	0.504	0.579
A	0.748	0.650	0.696
A + V	0.759	0.715	0.736

Bold value indicates the best-performing values within each metric category

Table 6 lists the results of DPD-E Net on the MODMA dataset. Here ’E1’ denotes the global stream using BFN features constructed from the 16 main electrodes, ’E2’ denotes the local stream using BFN features constructed from electrodes of two prominent brain sub-regions, and ’E3’ is the temporal stream using EEG features obtained from raw signals. ‘A’ is audio modality, and ‘T’ is the textual modality. The MODMA dataset has limited number of subjects and has no train-development-test data split, so experiments on this dataset are conducted using 5-fold cross-validation. Note that the F1 score, precision, and recall are reported as averages over the five folds. Results show that most of the single-modality models perform poorly in terms of all metrics except for DPD-E Net using textual features. When combining 3-stream EEG features, DPD-E Net achieves a F1 score of around 0.620 and recall of 0.767 but adding audio modality slightly degrades results in these two metrics. With all the listed features, best results in F1 score, precision and recall on the MODMA dataset are produced by DPD-E Net.

Table 6.

The results on MODMA dataset using unimodal and multi-modal models

Modality	Precision	Recall	F1
E1	0.400	0.333	0.313
E2	0.600	0.300	0.393
E3	0.633	0.599	0.567
A	0.310	0.467	0.364
T	0.853	0.833	0.816
E1 + E2 + E3	0.599	0.767	0.620
E1 + E2 + E3 + A	0.633	0.600	0.567
E1 + E2 + E3 + A + T	0.900	0.900	0.876

Bold value indicates the best-performing values within each metric category

Metrics are reported as averages over the five folds

Based on the experimental results of this section, it is clear that the fusion of different modalities can bring performance gain on all the four datasets, indicating that both DPD Net and DPD-E Net are able to integrating useful information from each modality.

Comparison to state-of-the-art models

Table 7 suggests that with three modality, the CCC score of DPD Net shows considerable advantages, which is approximately 38.3% higher than the TensorFormer on the E-DAIC dataset. We rank the performance using CCC score as this metric is used as the only criteria for the depression detection challenge of the E-DAIC dataset paper.

Table 7.

Comparison to baseline methods on E-DAIC dataset

Models	CCC	RMSE
TensorFormer (T + A + V)	0.493	4.31
DPD Net (T + A)	0.596	5.16
DPD Net (T + A + V)	0.682	4.79

Bold value indicates the best-performing values within each metric category

The comparison on the Twitter depression dataset is presented in Table 8. Since this dataset is released without train-development-test data split, we split the dataset with ratio of 7:1:2, and re-implement three variants of the baseline method mentioned in Sect. “Baselines” using hyperparameters from the TM Transformer paper. It is obvious that DPD Net outperforms all the other baseline results in terms of F1 score and recall, precision.

Table 8.

Comparison to baseline methods on Twitter depression dataset

Models	Precision	Recall	F1
Vanilla TM Transformer	0.828	0.904	0.864
Set TM Transformer	0.819	0.827	0.823
Time2vec TM Transformer	0.824	0.856	0.840
DPD Net	0.834	0.918	0.874

Bold value indicates the best-performing values within each metric category

According to Table 9, DPD Net outperforms its counterpart on the D-vlog dataset in terms of precision but STST transformer [41] achieves higher F1 score and recall according to their published results. The performance of the comparison experiment on these three datasets has proven that DPD Net can achieve sound performance for depression detection in both the clinical setting and the social media setting.

Table 9.

Comparison to baseline methods on D-vlog dataset

Models	Precision	Recall	F1
STST Transformer	0.725	0.776	0.750
DPD Net	0.759	0.715	0.736

Bold value indicates the best-performing values within each metric category

For a fair comparison on the MODMA dataset, we re-implement the baseline method ES Vision Transformer [42] with hyperparameter tuning since the model using their reported hyperparameter values performed poorly on our five folds. The F1 score, precision, and recall are reported as averages over the five folds. As can be seen from Table 10, DPD-E Net performs significantly better than the ES Vision Transformer in all the three metrics, which demonstrates the potential of incorporating EEG signals with speech data for depression detection.

Table 10.

Comparison to baseline methods on MODMA dataset

Models	Precision	Recall	F1
ES Vision Transformer	0.566	0.800	0.651
DPD-E Net	0.900	0.900	0.876

Bold value indicates the best-performing values within each metric category

Metrics are reported as averages over the five folds

Ablation study: contribution of modules experiments

We conduct ablation experiments to confirm the effectiveness of the modules of DPD Net and the base classifiers of DPD-E Net.

For DPD Net, we conduct an ablation study to investigate the contribution of each of the sub-modules. Experiments are only conducted on the E-DAIC dataset as it is currently the most used benchmark depression dataset.

Firstly, we explore the effect of the Conv-based Submodule which is the core component of DPD Net’s unimodal encoder module. The Conv-based Submodule consists of Conv blocks and LSTM, and to study their effectiveness, we remove each of them or omit both of them. Table 11 presents the results after changing these blocks. It is obvious that only using the Conv block or LSTM block, DPD Net performs to a certain extent with a CCC score around to 0.4. However, after removing both the Conv block and LSTM block, the model performs poorly with the CCC score dropping from 0.682 to 0.141, which suggests that the proposed unimodal encoder module is able to significantly improve the efficiency of DPD Net and its inclusion is important to the DPD Net’s performance. This is because the Conv block discovers the local patterns through convolution operations while the LSTM block captures the long-term relations through the sequential processing, and the combination of these two types of information reveals potential key depression-related cues from each modality.

Table 11.

Ablation study on Conv-based submodule of DPD Net on the E-DAIC dataset

Conv-based submodule
Conv block	LSTM	CCC	RMSE
–	–	0.141	7.72
–	$✓$	0.441	6.22
$✓$	–	0.453	5.91
$✓$	$✓$	0.682	4.79

Next, we investigate the impact of the Transformer Submodule and GNN Submodule by removing one of them separately or removing them both while keeping other components of DPD Net unchanged. Results shown in Table 12 reveal that both the Transformer Submodule and GNN Submodule play an essential role in multi-modal encoding as deleting one of them leads to low CCC scores of 0.106 and 0.206, respectively. Also, it is worth noting that using only one of submodules is almost equivalent to removing the whole multi-modal encoder module, which means the information obtained from the Transformer Submodule and GNN Submodule can complement each other to produce more informative embeddings for depression detection. Looking at the performance variation without the GNN Submodule and Transformer Submodule, we can conclude that DPD Net benefits from the GNNs and the Transformer, which jointly model local and global dependencies from the multimodal representations.

Table 12.

Ablation study on transformer submodule and GNN submodule of DPD Net on the E-DAIC dataset

Transformer submodule	GNN submodule	CCC	RMSE
-	-	0.160	7.59
-	$✓$	0.106	7.61
$✓$	-	0.206	7.36
$✓$	$✓$	0.682	4.79

As for the ensemble model DPD-E Net, we study the effects of the base classifiers on the MODMA dataset. In Table 13, the last row presents the performance of the complete DPD-E Net and the first four rows are the results of removing each of the base classifiers. Note that again all the metrics are reported as averages over the five folds. The F1 score after deleting the classifier based on local spatial stream drops from 0.876 to 0.748. Thus including this base classifier could bring a performance gain of around 15%. The deletion of the global spatial stream classifier or the EEG temporal features stream classifier leads to an overall decreased in performance, with F1 scores decreasing around 7%. These results suggest that EEG-based base classifiers improve DPD-E Net’s performance. Further, we find that DPD-E Net without the Speech-Text stream classifier gives the most significant decline in performance, with a F1 score, precision, recall decreasing from 0.876 to 0.620, from 0.900 to 0.767, from 0.900 to 0.599, respectively. This approximately 29% decline in F1 score indicates that the Speech-Text stream classifier is the most important classifier, which also confirms the finding that we mentioned above that the involvement of textual features plays an essential role in detecting depression.

Table 13.

Ablation study on the base classifiers of DPD-E Net

Classifiers	Precision	Recall	F1
– Local spatial stream	0.850	0.700	0.748
– Global spatial stream	0.870	0.833	0.815
– EEG temporal features stream	0.920	0.800	0.817
– Speech-Text stream	0.599	0.767	0.620
DPD-E Net	0.900	0.900	0.876

Metrics are reported as averages over the five folds

Ethical considerations

Even though AI-driven automatic depression detection systems have shown promising performance and potential in assisting with diagnosis, there are concerns that lie in the ethical side. For the data used in this study, the clinical-based datasets are collected with participants’ informed consent and approved by relevant ethics committees. However, the datasets curated from social media content lack attention from the general public and oversight from ethics councils [43]. The users are not aware that the content they create is used to undertake automatic depression detection. Also, some predictive results attributed to the bias in the training data could aggravate unintended stereotypes if the data is not provided as a representative sample for the population [44].

Conclusion

In this work, a GNN-enhanced Transformer model named DPD-Net is proposed to solve the problem of automatic depression detection using multi-modal data. This is the first attempt to our knowledge to employ GNNs along with transformers for automatic depression detection. Also, unlike previous works which mostly are tested on a single dataset, DPD Net can work across different datasets under two different application settings, including the clinical setting and the social media setting. Furthermore, its extended version DPD-E Net can be applied to an additional EEG modality.

Our ablation studies demonstrate the advantages of the proposed sub-modules and base classifiers, and the effectiveness of combining diverse data modalities for automatic depression detection. Comparisons to other baseline methods show that DPD Net and DPD-E Net can outperform the state-of-the-art models on three benchmark datasets: E-DAIC dataset, Twitter depression dataset and MODMA dataset, and achieve competitive performance on the D-vlog dataset.

Based on the modality experimental results, we observe that even though the involvement of audio, visual and EEG modality can improve the prediction results, the performance of both DPD Net and DPD-E Net highly rely on the text modality. DPD Net only produces comparable results but is not able to beat the baseline method in all metrics on the D-vlog dataset, which is the only dataset in this work where the textual modality is not included. In the future, we hope to combine textual data to improve the performance on the D-vlog dataset and add more explainability to the proposed models. A potential limitation of our proposed models is that we do not take the gender difference into consideration. Designing gender-independent models could help to reveal more potential insights in automatic depression detection.

Appendix A COGMEN architecture

The multi-modal encoder module of DPD Net is based on the COGMEN model [7]. Here we present the architecture of this model in Fig. 6.

Fig. 6.

The architecture of the COGMEN

Appendix B Multi-head attention

Let Q, K and V stands for Query, Key, and Value, respectively, obtained by performing linear projections to the multi-modal representation input, which is used for constructing the output embeddings. Let X of shape $(T,d_m)$ as the multi-modal representation input (after addition with positional encoding) of sequence length T, the Q, K and V are computed using Eq. B1.

$$\begin{array}{c}\hfill \begin{array}{c}\hfill Q_h=X{W}^{h,Q}\\ \hfill K_h=X{W}^{h,K}\\ \hfill V_h=X{W}^{h,V}\end{array}\end{array}$$ B1

here $W^{h,Q}$, $W^{h,K}$ and $W^{h,V}$ are trainable parameters, with $W^{h,Q}\in {\mathbb{R}}^{d_m\times d_q}$, $W^{h,K}\in {\mathbb{R}}^{d_m\times d_k}$, $W^{h,V}\in {\mathbb{R}}^{d_m\times d_v}$, and $d_q=d_k=d_v=d_m$. Then, the output attention matrix of a single head self attention is computed using Eq. B2:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill hea{d}_h=softmax\left(\frac{Q_h{K_h}^T}{\sqrt{d^k}}\right)V_h\end{array}\end{array}$$ B2

A multi-head attention with h heads is defined in Eq. B3, which is simply done by concatenating the attention value obtained from each head and performing another linear operation with a trainable parameter $W^o\in {\mathbb{R}}^{d_m\times h{d}_v}$ and $d_q=d_k=d_v=d_m/h$:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill MultiHead(Q,K,V)=Concat(hea{d}_1,...,hea{d}_h)W^o\end{array}\end{array}$$ B3

Appendix C EEG features

Table 14 presents the 29 linear and non-linear EEG features used for both the temporal stream and the spatial stream.

Table 14.

29 linear and non-linear EEG features

Feature name	Feature name
Kurtosis	Activity
Maximum of the Second Order Difference	Mobility
Mean of the Second Order Difference	Complexity
Maximum of the First Order Difference	Mean FFT Amplitude
Mean of the First Order Difference	Median FFT Amplitude
Coeffiecient of Variation	Min FFT Amplitude
Skewness	Max FFT Amplitude
Wavelet Approximate Mean	Mean FFT Amplitude for Alpha Band
Wavelet Approximate Standard Deviation	Mean FFT Amplitude for Beta Band
Wavelet Detailed Mean	Mean FFT Amplitude for Delta Band
Wavelet Detailed Standard Deviation	Mean FFT Amplitude for Theta Band
Wavelet Approximate Energy	Singular Value Decomposition Entropy
Wavelet Detailed Energy	Spectral Entropy
Wavelet Approximate Entropy	Permutation Entropy
Wavelet Detailed Entropy

Appendix D Implementation details

For the E-DAIC dataset, we use Adam optimizer with cosine annealing with warm restarts scheduler during training. Each model is trained with 1000 epochs using a learning rate of 0.0001, a batch size of 32 and a dropout rate of 0.2. According to the E-DAIC dataset paper, the loss function is designed by leveraging the CCC score, which is simply computed as $1-CC{C}_{\mathit{score}}$. For the Twitter depression dataset, Adam optimizer and exponentialLR scheduler are used along with negative log likelihood loss. Each model is trained with 30 epochs using a learning rate of 0.00001, a batch size of 32, a dropout rate of 0.2. For the D-vlog dataset, we use the same settings as we used for the Twitter depression dataset except that each model is trained with 60 epochs. For the MODMA dataset, Adam optimizer and reduceLR scheduler are used during a 20-epochs training with a learning rate of 0.0001, a batch size of 4 and a dropout rate of 0.2.

Hugging Face toolkit [36] is used to extract features from raw tweet images of the Twitter depression dataset. Firstly, we use ’AutoProcessor’ to convert images to the correct input format, then 768-dimensional visual features are obtained from Hugging Face pre-trained models ’clip-vit-base-patch16’. To encode the English text from E-DAIC dataset and Twitter depression dataset, we use the Hugging Face pre-trained model ’all-mpnet-base-v2’. As for the Chinese text in MODMA dataset, the multilingual-model ’paraphrase-multilingual-mpnet-base-v2’ is utilized. These two models are selected as text encoders based on our preliminary experiments.

Appendix E Evaluation metrics

We leverage the following metrics to evaluate the performance of the models. For the E-DAIC dataset, we use the Concordance Correlation Coefficient (CCC) score to evaluate the model’s regression performance. This metric is considered as the only criteria for ranking participants’ methods of the depression detection challenge proposed by the original E-DAIC dataset paper, and is the most widely used metric for depression detection research on the E-DAIC dataset, as it is a metric which perfectly takes both precision and accuracy into consideration in one formula and is robust to scale variance [4]. The CCC score measures the correlation between the prediction and the ground truth using the following equation:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill CC{C}_{\mathit{score}}=\frac{2C_{\mathit{py}}}{C_p^2+C_y^2+{(\overline{p}-\overline{y})}^2}\end{array}\end{array}$$ E4

Note that a depression score ranges from 0 to 24. Here $\overline{p}$ is the mean value of the predicted depression scores and $\overline{y}$ is the mean value of the true depression scores. $C_p$ and $C_y$ are the variance of the predicted and the real depression scores, respectively. $C_{\mathit{py}}$ is the covariance of the predicted and the real depression scores. The $CC{C}_{\mathit{score}}$ ranges from – 1 to 1, where 1 indicates the highest correlation between the predictions and the ground truths. Also, we report Root Mean Square Error (RMSE) as an auxiliary metric, which is calculated using Eq. E5. Here N is the number of samples, $p_i$ and $y_i$ denote the predicted depression score and the real depression score of sample i, respectively.

$$\begin{array}{c}\hfill \begin{array}{c}\hfill RMSE=\sqrt{\frac{\sum _{i=1}^N{(p_i-y_i)}^2}{N}},\end{array}\end{array}$$ E5

For the classification tasks on the Twitter depression dataset, D-vlog dataset and MODMA dataset, we use precision, recall and F1-score as the metrics. Precision represents the percentage of all the assigned positive labels are correctly assigned by the model. Let TP denote the number of true positives, and FP denote the number of false positives, then precision is computed as follows:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill precision=\frac{\mathit{TP}}{TP+FP}\end{array}\end{array}$$ E6

Recall is a metric to measure, for all the true positive samples, what percentage of them are actually found by the model, and is formulated as:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill recall=\frac{\mathit{TP}}{TP+FN}\end{array}\end{array}$$ E7

The F1-score takes both the precision and recall into consideration using the following equation:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill F1=2\times \frac{precision\times recall}{precision+recall}\end{array}\end{array}$$ E8

Appendix F Reported results of the baselines

Note that in Sect. “Comparison to state-of-the-art models”, for the comparison with Tensorformer on the E-DAIC dataset and the comparison with STST Transformer on the D-vlog dataset, we present their reported results from the original papers. For the comparison with TM Transformer on the Twitter depression dataset and the comparison with ES Vision Transformer on the MODMA dataset, we only present experimental results from our implementations of these two baselines. Here, the results published in the original papers using their own data splits are shown in the following Tables 15 and 16.

Table 15.

The reported results of TM Transformer on the Twitter depression dataset

Models	Precision	Recall	F1
Vanilla TM Transformer	0.868	0.905	0.886
Set TM Transformer	0.921	0.934	0.927
Time2vec TM Transformer	0.931	0.931	0.931

Table 16.

The reported results of ES vision transformer on the MODMA dataset

Models	Precision	Recall	F1
ES vision transformer	0.977	0.973	0.973

Funding

Not applicable.

Data availability

The four datasets used in this work are available at: https://dcapswoz.ict.usc.edu/, https://drive.google.com/file/d/11ye00sHFY5re2NOBRKreg-tVbDNrc7Xd/view, https://sites.google.com/view/jeewoo-yoon/dataset, https://modma.lzu.edu.cn/data/index/.

Materials availability

Not applicable.

Declarations

Conflict of interest

The authors have no competing interests to declare that are relevant to the content of this article.

Consent to participate

Not applicable.

Consent for publication

Not applicable.

Ethical approval

Not applicable.