Large language models for identifying depression concerns in cancer patients

Introduction

Cancer patients frequently experience depression due to the severity of the disease and its poor prognosis. To improve the identification of depression, researchers recently developed classifiers using logistic regression, support vector machines (SVMs), and 2 Bidirectional Encoder Representations from Transformers (BERT) models—one original and one further pretrained on Reddit data—fine-tuned on 6600 patient messages from a cancer center.¹ While the authors rightly highlight “the potential of BERT and RedditBERT in identifying depression concerns in messages from cancer patients,” we believe their models may not have been fully optimized. In this paper, we explain why this is the case, propose solutions through hyperparameter tuning, and suggest further improvements using alternative transformer models within the BERT family. We hope our insights will help researchers and practitioners enhance patient care and improve quality of life.

Classical methods vs transformer models

For depression identification, existing work trained 4 models: logistic regression, an SVM, BERT-base, and RedditBERT.¹ Among them, RedditBERT is a BERT-base model further pretrained on the authors’ own Reddit data. In comparing model performance, the SVM achieved an area under the ROC curve (AUROC) of 0.83 (95% CI: 0.78–0.87). In contrast, BERT-base reached an AUROC of 0.86 (95% CI: 0.82–0.90), while RedditBERT outperformed both with an AUROC of 0.88 (95% CI: 0.85–0.91). Regarding F1 scores, the fine-tuned BERT-base achieved the highest score at 0.48, followed by RedditBERT at 0.46 and SVM at 0.45. What is noteworthy is not that the BERT-based models have higher mean AUROCs (0.86 and 0.88) or higher F1 scores than the classical models—that is expected. Rather, it is that their performance gain over the classical models is so small. Statistically, the overlapping in confidence intervals indicates that there is no significant difference between the SVM models and the BERT-based models. In terms of F1 scores, the performance gap between SVM (F1: 0.45) and RedditBERT (F1: 0.46) is also negligible. Objectively, an F1 score of 0.46 with a recall of 0.74 and a precision of 0.33 is quite low. It means that only 74% of concerning messages are correctly identified, while only 33% of the flagged messages are actually concerning. In practical terms, considering that BERT models typically require GPUs to run and contain tens of millions of parameters, the current results do not provide sufficient justification for deploying BERT over SVM models.

How to (potentially) fix it

Fine-tuned BERT models have been known to consistently outperform earlier approaches, including SVMs and recurrent neural networks, and remain state-of-the-art for text classification, even amid the rise of generative artificial intelligence models like GPT-4o and DeepSeek.² The fact that they do not outperform SVM models by a substantial margin on patient messages is intriguing and strongly suggests suboptimal fine-tuning.³ Unfortunately, the authors provide little detail on the fine-tuning process or hyperparameter selection. The only available information indicates the use of bert-base-uncased as the tokenizer, a maximum input length of 512 tokens, a learning rate of 1e−5, and 5 training epochs. Among these, the learning rate is arguably the most critical hyperparameter in fine-tuning.⁴ If the learning rate is set too high, the training loss may increase instead of decrease, a problem easily detectable from training logs. Conversely, if it is set too low, learning becomes slow and ineffective—while the training error decreases, the rate of improvement is insufficient, especially in the context of 5 training epochs. We believe that adjusting the learning rate in conjunction with the number of training epochs could lead to substantial performance gains.

How to further improve it

To develop RedditBERT, the authors scraped 1 000 000 Reddit posts and continued pretraining the BERT-base model for 20 epochs using 4 GPUs on the Google Cloud Platform.¹ For most researchers, this represents a significant computational investment. While continuing pretraining on in-domain data is a valuable approach, we note that more advanced models in the BERT family are already available and may offer improved performance with lower computational demands than building RedditBERT. In particular, we encourage researchers to consider using RoBERTa and RoBERTa-large models.⁵

Conclusion

Researchers have made a significant contribution to the literature on depression identification among cancer patients. Building on their work, we note that the reported performance gain of BERT-based models over classical models is unexpectedly small. In this article, we have explored the reasons behind this limited improvement, proposed solutions through more thorough hyperparameter tuning, and suggested further enhancements by leveraging alternative transformer models within the BERT family. Given that the earlier work represents a proof-of-concept study, we hope our analysis will further inspire researchers and practitioners to refine depression identification methods for cancer patients, ultimately contributing to improved patient care and quality of life.

Author contributions

Yu Wang (Conceptualization, Investigation, Methodology, Writing – original draft), Xin Ye (Investigation, Writing – original draft, Funding acquisition), Huiping Luo (Investigation, Writing – original draft, Funding acquisition), Wei Feng (Investigation, Writing – original draft, Funding acquisition). All authors reviewed and approved the final manuscript.

Funding

This work was supported by the National Natural Science Foundation of China (72304070), the Key Project of the National Natural Science Foundation of China (72234001), Funds for High-quality Discipline Construction during the 14th Five-Year Plan period, Beijing Xisike Clinical Oncology Research Foundation (Y-HS202202-0165), and Three-Year Action Plan for Strengthening of the Construction of the Public Health System in Shanghai (GWVI-11.2-XD23).

Conflicts of interest

Authors have no competing interests to declare.

Data availability

No new data were generated or analysed in support of this research.