Reply to Wang: Clarifying model performance and language markers of depression across races

We thank Wang for commenting on our recent paper investigating the relationship between race and language markers of depression (1). The letter has two critiques of the prediction analysis.

First, the letter suggests that reviewing details of the analyses could shed light on “why [the] models trained on the language of Black individuals perform better in the White test set than in the Black test set” (2).

The details of our analyses, which the letter requests, are included in SI Appendix. We report sample sizes at each step of dataset processing (Section 1A), validation metrics, and training and test set sizes (Section 1C). While it seems counterintuitive that a model trained on Black individuals would perform better on White individuals, this is mathematically possible if M_Black has a larger error term than M_White. Said otherwise, the language of Black individuals explains less of the variance in depression than the language of White individuals (Table 1). Notwithstanding platform effects, why that is the case is an open clinical question. Prior work also reported attenuated model performance when testing on data from Persons of Color compared to data from White individuals, even with the same amount of training data (3). It is plausible that among Black individuals, depression is associated with language effects that have too rare a frequency to be included as features in our models or that Black individuals use Facebook differently than White individuals. See Data, Materials, and Software Availability for mean features scores for Linguistic Inquiry and Word Count (LIWC) (4) and Topic distributions; we did not share raw social media data to protect participants’ privacy. We hope these details are helpful to the reader in interpreting our results.

Table 1.

Comparison of performance of different feature sets at predicting depression scores reported as Pearson r with p values

1-3grams, LIWC, and Topic distributions
	White test set	Black test set
MWhite	0.392 (<0.001)	0.132 (0.006)
MBlack	0.204 (<0.001)	0.126 (0.009)
Pretrained BERT embeddings (L11 and L12)
MWhite	0.347 (<0.001)	0.104 (0.031)
MBlack	0.161 (0.001)	0.058 (0.225)
Pretrained RoBERTa embeddings (L11 and L12)
MWhite	0.345 (<0.001)	0.142 (0.003)
MBlack	0.235 (<0.001)	0.128 (0.007)
RoBERTa embeddings fine-tuned on Facebook
MWhite	0.353 (<0.001)	0.179 (<0.001)
MBlack	0.288 (<0.001)	0.095 (0.048)

Second, rather than using Bidirectional Encoder Representations from Transformers (BERT) (5) as a frozen embedding model, the letter suggests fine-tuning BERT or using a model such as Robustly Optimized BERT Pre-training Approach (RoBERTa).

To study whether and how known language markers of depression (i.e., first-person pronouns and negative emotions) vary by race, we prioritized interpretable feature sets (i.e., LIWC and topics). We considered BERT-base to evaluate the predictive performance of one of the classes of large language models. In (Table 1), we report the evaluation of pretrained RoBERTa embeddings L11 and L12 and a fine-tuned RoBERTa model where all 12 layers were unfrozen to minimize Masked Language Modelling loss (6) with masking probability set to 0.15 on a dataset of 36 million Facebook messages from over 60 thousand individuals (7) for one epoch. While such domain adaptive training aligned the model to Facebook language, the impact of racial distribution in the training data on the finetuned model is an open question.

RoBERTa-base outperformed BERT-base, and finetuned RoBERTa obtained results comparable to the pretrained models (Table 1), as also noted by other work (8, 9). All models performed relatively poorly on data from Black individuals than White individuals, which aligns with our paper‘s results (1).