Predictive value of biomarker signatures for suicide risk in hospitalised patients with major depressive disorders: a multicentre study in Shanghai

Abstract

Background

Biomarkers for predicting suicide risk in hospitalised patients with mental disorders have been understudied. Currently, suicide risk assessment tools based on objective indicators are limited in China.

Aims

To examine the value of various biomarkers in suicide risk prediction and develop a risk assessment model with clinical utility using machine learning.

Methods

This cohort study analysed patients with major depressive disorder (MDD) who were hospitalised for the first time between January 2016 and March 2023 from four specialised mental health institutions. A total of 139 features, including biomarker measurements, medical orders and psychological scales, were assessed for analysis. Their suicide risk was evaluated by qualified nurses using Nurse’s Global Assessment of Suicide Risk within 1 week after admission. Five machine learning models were trained with 10-fold cross-validation across three hospitals and were externally validated in an independent cohort. The primary performance was assessed using the area under the receiver operating characteristic curve (AUROC). The model was interpreted using the SHapley Additive exPlanations (SHAP) analysis. Biomarker importance was evaluated by comparing model performance with and without these biomarkers.

Results

Of 3143 patients with MDD included in this study, the incidence of high suicide risk within 1 week after first admission was 660 (21.0%). Among all models, the Extreme Gradient Boosting can more effectively predict future risks, with an AUROC higher than 0.8 (p<0.001). The SHAP values identified the 10 most important features, including five biomarkers. After clustering analysis, electroconvulsive therapy, physical restraint, β2-microglobulin and triiodothyronine were found to have heterogeneous effects on suicide risk. Combining biomarkers with other data from electronic health records significantly improved the performance and clinical utility of machine learning models based on demographics, diagnosis, laboratory tests, medical orders and psychological scales.

Conclusions

This study demonstrates the potential for a biomarker-based suicide risk assessment for patients with MDD, emphasising the interaction between biomarkers and therapeutic interventions.

WHAT IS ALREADY KNOWN ON THIS TOPIC

• Existing suicide risk assessment tools have certain limitations. Biomarkers that can predict suicide risk have been rarely studied.

WHAT THIS STUDY ADDS

• Machine learning models based on biomarkers and other patient data can effectively predict future risk of suicide. Incorporating biomarker measurements into prognostic models significantly improved clinical utility in scenarios when symptom data were limited.

HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY

• Biomarkers have the potential to improve the predictive performance of suicide risk in clinically relevant scenarios.

Introduction

Suicide is a leading public health problem worldwide, accounting for more than 1.4% of the total mortality globally.¹ Many individuals who died by suicide suffered from mental disorders, with major depressive disorder (MDD) present in half to two-thirds of these cases.² In patients diagnosed with MDD, suicidal ideation and attempts are often the driving factors necessitating hospitalisation.³ During their hospitalisation, the prevalence of suicide attempts was 17.3% (95% confidence interval (CI) 12.4% to 23.7%), whereas the incidence of suicidal ideation during an acute episode within 1 month was 27.7% (95% CI 15.4% to 44.5%).⁴ Furthermore, compared with patients with MDD without a suicide attempt, those with a suicide attempt had a higher likelihood of hospitalisation within 3 years.⁵ Therefore, identifying individuals with MDD in hospitalisation who are at high risk of suicide represents a significant public health challenge and carries important clinical implications.

In China, the most common risk assessment tools for psychiatric inpatients are the Tool for Assessment of Suicide Risk and the Nurse’s Global Assessment of Suicide Risk (NGASR), which effectively assist professionals in identifying individuals at risk of suicide. Several risk factors have been examined, including psychological, familial, genetic and other biological factors.⁶ However, suicide assessments based on self-report or other-rated scales in clinical practice may be subject to inconsistency or unreliability of the evaluation.^{7 8} More importantly, these scales can only assess a patient’s current suicide risk. The biomarkers predicting potential future suicide risk were rarely studied.^{6 9} Therefore, the construction of a reliable and accurate screening model for future suicide risk based on objective data can compensate for the shortcomings of existing clinical assessment tools and provide evidence for the predictive value of biomarkers. This study aimed to examine the clinical value of various routinely tested biomarkers in suicide risk prediction and develop a prognostic model with clinical utility using machine learning.

Methods

This study followed the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis reporting guideline.¹⁰

Study participants and data source

Patient data were collected from four specialised mental health institutions (Shanghai Mental Health Center, Tongji University Mental Health Center, Shanghai Putuo District Mental Health Center and Shanghai Changning District Mental Health Center). This database includes admission information, diagnoses, scale tests, medical orders, laboratory tests and risk assessments for more than 20 000 patients. As a multicentre database, with Shanghai Mental Health Center being one of the four National Medical Centers for Mental Diseases, the data are highly representative and broadly reflect the inpatient depression population in China.

This study included detailed and comprehensive data on patients with MDD first hospitalised from 2016 to 2023. The aim of this study was to use machine learning to assess the predictive value of laboratory tests for the future risk of suicide in patients with MDD based on biomarkers and clinical data measured at or prior to admission. The eligibility criteria (figure 1A) were (1) 18–65 years old, (2) primary diagnosis of MDD on admission according to the International Classification of Diseases 10th Revision (ICD-10), (3) first hospitalisation and (4) resident in China. Exclusion criteria were (1) missing demographic information, (2) excessive missing of hospital records, (3) hospitalisation shorter than 7 days, (4) diagnosis of mental retardation, personality disorder or organic brain disorder, (5) severe physical illness, (6) long-term psychotropic drug use, (7) pregnancy or lactation and (8) no suicide risk assessment conducted. We specifically excluded patients who were already assessed as high risk for suicide at admission to ensure that the biomarkers were evaluated prior to any escalation in suicide risk.

Figure 1. The flowchart of the participants’ selection. ICD-10, International Classification of Diseases 10th Revision; MDD, major depressive disorder; NGASR, Nurse’s Global Assessment of Suicide Risk.

Procedure and outcome assessment

This retrospective cohort study simulated the prospective time sequence of the research procedures post hoc (figure 1B). Each patient was assigned a unique hospitalisation code. The patient’s first admission was identified by this code across the database. Demographic information included age and sex at first admission. We extracted whether the patient had concomitant psychotic symptoms based on ICD-10. From the medical order data, we extracted commonly used drug usage records, immediate enforcement of physical restraint, electroconvulsive therapy (ECT) and repetitive transcranial magnetic stimulation (TMS). The psychiatric scales included Zung Self-rating Depression Scale (SDS), Zung Self-rating Anxiety Scale (SAS), Life Event Scale (LES), and Symptom Checklist 90 (SCL-90), which are widely used in the study hospitals. The SCL-90 was recorded as factor scores and the other scales were recorded as total scores. Commonly used laboratory and other auxiliary examinations were collected, including complete blood count, liver function, renal function, blood lipid and glucose, coagulation function, hepatitis virus, immunological function, electrolyte, routine urinalysis, anaemia index, vitamin, thyroid function and myocardial infarction index. A total of 139 features, including 108 biomarkers, were assessed for analysis. See online supplemental table 1 for the detailed definition and explanation of each included variable.

The time of the included tests was recorded on the day of admission or within the previous week, as some patients may have had these tests conducted in outpatient clinics prior to admission. Considering that psychiatric drugs require time to take effect, medical order record data were extracted from 1 month before admission to the day of admission. As some patients may seek medical treatment in more than one hospital, medical order records of all patients were obtained across the study hospitals during that time period to reduce the impact of prescriptions from multiple places. The procedure to confirm the time frame of the data was as follows: we first extracted the testing/execution times for each variable, and then identified each patient’s unique hospitalisation number and admission date. We only included data collected on or prior to admission, ensuring a consistent time window across all variables. Furthermore, to ensure that the collected biomarkers and clinical data were assessed before any escalation in suicide risk, patients who were already identified as high risk for suicide at admission were excluded from the analysis. The timing of these measurements ensured that the biomarkers and other variables were assessed before any escalation in suicide risk, reinforcing their potential predictive role.

The outcome variable of this study was the risk of suicide assessed by the NGASR recorded in risk assessment records. The NGASR consists of 15 questions, and a total score of ≥9 indicates high suicide risk.^{11 12} The time of the outcome variables was set to be within 7 days of hospitalisation. According to institutional requirements and clinical pathways, the risk assessments were conducted weekly by qualified nurses.

Models

Models were built in Python V.3.8 with code packages of scikit-learn V.1.1.3, pytorch V.2.0.0, xgboost V.1.7.2, pykan V.0.0.5 and lightgbm V.3.3.3. Five machine learning and deep learning models were developed and compared, which included the logistic regression model (LRM), Extreme Gradient Boosting (XGBoost),¹³ Light Gradient Boosting Machine (LightGBM),¹⁴ multilayer perceptron (MLP) and Kolmogorov-Arnold Network (KAN).¹⁵ The LRM, XGBoost, LightGBM and MLP were commonly used in medical prediction modelling and achieved acceptable performance in previous studies.^{16 17} KAN is a recently proposed deep learning model and has received a lot of attention due to its superior performance over MLP in several tests.^{18 19} The Shanghai Mental Health Center was randomly chosen as an external testing cohort, while the remaining data source hospitals constituted a cross-validated cohort. The models were trained based on 10-fold cross-validation repeated 10 times in the cross-validated cohort for the best replicability.²⁰ Random parameter search was used to turn the hyperparameters during cross-validation. Tuned hyperparameters for MLP and KAN included the number of layers and nodes, learning rate, the percentage of dropout and the mini-batch size, while the number of trees, tree depth and learning rate were tuned for XGBoost and LightGBM. To account for label imbalance, the training samples were oversampled to ensure that the negative and positive samples were equally distributed. Patients with more than 20% of missing values were excluded. For models that do not allow missing values, missing information was filled with the median within the cohort.

The model was interpreted using the SHapley Additive exPlanations (SHAP) analysis,²¹ which can explain the output of any machine learning model with a game theoretic approach. In this study, the average SHAP value for each feature is greater than zero, indicating that the presence or increase of this feature results in an added positive probability (predicted suicide risk probability) that the model outputs.

Statistical analyses

Initial data processing was conducted in PostgreSQL V.4.2. Model performance metrics were calculated in Python V.3.8 with the following packages: scikit-learn, pandas and numpy. Further data description and analysis were conducted in R V.4.1.3. Continuous variables were reported as mean (standard deviation (SD)) or median (interquartile range (IQRs)), depending on the normality of their distribution. The Anderson-Darling test was used to assess the normality of distributions. Categorical variables were presented as frequencies/numbers and percentages (%). Models were evaluated in 10-fold cross-validation with averaged performance and verified for generalisability in the external testing cohort. Primary model performance was evaluated using the area under the receiver operating characteristic curve (AUROC). Secondary metrics included area under the precision-recall curve (AUPRC), accuracy, sensitivity, specificity, positive predictive value (PPV), negative predictive value and F1 score. Reported values are averages and 95% CI obtained from bootstrapping with 1000 iterations. Feature importance was assessed by measuring gain, that is, the average increase in F1 score brought by a feature to the branches it appeared on, and cumulative importance was calculated as the sum of individual feature importance within feature categories. The impact of features on model predictions was determined using SHAP values. For better visualisation and interpretation, the K-means clustering algorithm is used to downscale the SHAP values. Silhouette score was used to evaluate the clustering effect. Decision curve analysis (DCA) was used to estimate the clinical utility (net benefit) of predictive models with and without biomarkers at varying diagnostic thresholds. The blank model did not include any features except for baseline prevalence. Model metrics were compared using Nadeau and Bengio’s corrected resampled t-test,²² which adjusts for the non-independence of resampled statistics. The association of biomarker measurements with patient symptom data was evaluated using Pearson correlation. The p values less than 0.05 were considered statistically significant. Holm-Bonferroni adjustment was used to adjust for multiple testing.

Results

Patients

A total of 3143 first-admitted patients with MDD who met the inclusion criteria were included in this study. Of these, 1912 were assigned to the cross-validation cohort, and 1231 were assigned to the external testing cohort. The mean (SD) age was 42 (12.818) years; 1574 (50.1%) patients were male, and 456 (14.5%) had concomitant psychotic symptoms. The overall incidence of high suicide risk within 1 week after first admission was 660 (21.0%). See online supplemental table 2 in the supplementary file for a detailed overview of patient characteristics.

Model performance

Detailed model performance metrics in the cross-validation and the external testing cohorts are presented in table 1 and online supplemental table 3 in the supplementary file. The right-sided p values test the hypothesis that the AUROC of the model is better than that of the blank model. The AUROCs of all trained models were significantly superior to a random guess in both the cross-validated cohort and the external testing cohort (p<0.001). In the external testing cohort, XGBoost achieved the best AUROC (0.828, 95% CI 0.824 to 0.833), AUPRC (0.605, 95% CI 0.595 to 0.617), accuracy (0.819, 95% CI 0.814 to 0.823), PPV (0.660, 95% CI 0.649 to 0.674) and F1 score (0.552, 95% CI 0.541 to 0.561), consistent with its performance in the cross-validated cohort. Thus, XGBoost was determined as the best model in this study based on the primary performance metric.

Table 1. Summary of model performance in the external testing cohort.

Model	AUROC (95% CI)	AUPRC (95% CI)	F1 score (95% CI)	P value
LRM	0.484 (0.476 to 0.490)	0.223 (0.216 to 0.229)	0.371 (0.365 to 0.377)	0.960
LightGBM	0.774 (0.769 to 0.780)	0.530 (0.517 to 0.543)	0.390 (0.379 to 0.400)	<0.001***
XGBoost	0.828 (0.824 to 0.833)	0.605 (0.595 to 0.617)	0.552 (0.541 to 0.561)	<0.001***
MLP	0.517 (0.511 to 0.521)	0.241 (0.235 to 0.246)	0.384 (0.378 to 0.388)	0.036*
KAN	0.505 (0.504 to 0.506)	0.236 (0.232 to 0.240)	0.381 (0.375 to 0.386)	<0.001***

This table summarises several core performance metrics of different models in the external testing cohort, with the reported values representing point estimates and their corresponding 95% CIs.

*p<0.05; ***p<0.001.

The 95% CI was calculated using bootstrap with 1000 iterations. The p value was calculated using right-sided Nadeau and Bengio’s corrected resampled t-test, testing the hypothesis that the AUROC is better than the blank model.

AUPRC, area under the precision-recall curve; AUROC, area under the receiver operating characteristic curve; CI, confidence interval; KAN, Kolmogorov-Arnold network; LightGBM, Light Gradient Boosting Machine; LRM, logistic regression model; MLP, multilayer perceptron; XGBoost, Extreme Gradient Boosting.

The ROC curves are presented in figure 2A, 2B, indicating good classifier performance (p<0.001).²³ The calibration curves are demonstrated in figure 2C, 2D, with a mean (SD) Brier score loss of 0.139 (0.008) in the external testing cohort, significantly lower than that of the blank model (p<0.001). The DCA is shown in Figure 2E, 2F, with the optimal cut-off threshold determined by Youden’s index ranging from 7.685% to 9.753%.

Figure 2. Model performance and clinical utility. (A) The receiver operating characteristic curve in the cross-validated cohort. (B) The receiver operating characteristic curve in the external testing cohort. (C) The calibration curve in the cross-validated cohort. (D) The calibration curve in the external testing cohort. (E) Decision curve analysis in the cross-validated cohort. (F) Decision curve analysis in the external testing cohort. SD in the bootstrapping process. The p value was calculated using right-sided Nadeau and Bengio’s corrected resampled t-test, testing the hypothesis that the score is better than the blank model. None: no patients are treated, assuming no one is at risk. All: all patients are treated, assuming everyone is at risk. XGBoost, Extreme Gradient Boosting. These lines show the outcomes of treating none or all patients, helping to see if the model adds benefit. AUROC, area under the receiver operating characteristic curve; SD, standard deviation.

Feature importance

The ten most important variables determined by the average gain of the cross-validated cohort within XGBoost model included TMS, urine glucose positive, mean corpuscular haemoglobin, SDS, psychotic symptom, ECT, urine glucose negative, mean corpuscular haemoglobin concentration, SAS and age (online supplemental figure 1). Subsequently, in the external testing cohort, a SHAP summary plot (online supplemental figure 2) identified the ten features that have the greatest impact on the model outputs. The ranking according to importance was ECT, olanzapine (OLA), thyroid-stimulating hormone (TSH), physical restraint, age, β2-microglobulin (BMG), psychotic symptoms, uric acid (UA), cystatin C (CYS-C), and triiodothyronine (T3).

The individual correlation between each feature and SHAP values with linear interpolation is presented in online supplemental figure 3, showing the contribution of each feature to the outcome. The SHAP feature interaction plot illustrates the interaction of each feature with the one exhibiting the strongest potential interaction (online supplemental figure 4). Based on the SHAP values, ECT had the strongest interaction with five characteristics (TSH, UA, physical restraint, age and psychotic symptoms), including two biomarkers. BMG interacted with age and OLA. T3 and CYS-C interacted with the psychotic symptoms.

To provide a group-level feature contribution while considering the potential interaction, individual SHAP values of each feature were clustered using the K-means clustering algorithm. The optimal number of clusters was two groups, determined by the silhouette score (online supplemental figure 5). The clustered SHAP heatmap is presented in figure 3, which demonstrates a quantitative heterogeneous response of ECT, physical restraint, BMG and T3 to the predicted suicide risk. The clustered SHAP decision curves (online supplemental figure 6) showed the average contribution of each feature on model outputs within clusters. The Pearson correlation between the five most important biomarkers with psychopathology (online supplemental figure 7) indicated that UA was positively associated with paranoid and psychotic symptoms; T3 was related to somatisation symptoms, psychotic symptoms and the total score of SCL-90.

Figure 3. Clustered SHAP values heatmap. BGM, β2-microglobulin; CYS-C, cystatin C; ECT, electroconvulsive therapy; OLA, olanzapine; SHAP, SHapley Additive exPlanations; T3, triiodothyronine; TSH, thyroid-stimulating hormone; UA, uric acid.

Added benefits of biomarkers

Combining biomarkers with treatment data (0.039, 95% CI 0.021 to 0.058, p<0.001), SAS (0.054, 95% CI 0.035 to 0.074, p<0.001), SDS (0.066, 95% CI 0.044 to 0.085, p<0.001), SCL-90 (0.074, 95% CI 0.056 to 0.091, p<0.001) and LES (0.053, 95% CI 0.036 to 0.069, p<0.001) led to significant improvements in the AUROC of prognostic models (figure 4A). Regarding clinical utility under optimal decision threshold (figure 4B), biomarkers significantly enhanced performance within seven categories (demographic: 0.408, 95% CI 0.390 to 0.425, p<0.001; psychotic symptom: 0.315, 95% CI 0.297 to 0.331, p<0.001; treatment: 0.102, 95% CI 0.088 to 0.117, p<0.001; SAS: 0.225, 95% CI 0.212 to 0.237, p<0.001; SDS: 0.200, 95% CI 0.186 to 0.214, p<0.001; SCL-90: 0.126, 95% CI 0.118 to 0.135, p<0.001; LES: 0.134, 95% CI 0.119 to 0.143, p<0.001).

Figure 4. Added benefits of biomarker measurements. (A) Added predictive value of biomarker measurements; (B) Added clinical utility of biomarker measurements. The p value was calculated using right-sided Nadeau and Bengio’s corrected resampled t-test, testing the hypothesis that the score is better than blank model. Blank, blank model; LES, Life Event Scale; SAS, Zung Self-rating Anxiety Scale; SCL-90, Symptom Checklist 90; SDS, Zung Self-rating Depression Scale. ***p < 0.001.

Discussion

Main findings

Our findings suggest that incorporating biomarker measurements into prognostic models based on self-reported, demographic or symptomatic data enhances their ability to predict future suicide risk and may lead to clinically relevant improvements. The improved clinical utility was particularly evident in scenarios where information on psychiatric symptoms or treatment variables was unavailable, suggesting that biomarker tests may be especially beneficial to patients who do not accurately report their symptoms or whose treatment histories are unclear.

Among our models, XGBoost has the best predictive accuracy and stability. The AUROC ranged from 0.828 to 0.861, indicating a good classifier performance²³ and significantly better than chance. The calibration analysis²⁴ and DCA analysis²⁵ also indicate a significant clinical utility of XGBoost in the screening of future suicide risk in psychiatry.

Several important features are identified through average gain in cross-validation and SHAP values. The average gain is a measure of the variable’s contribution to accuracy, while the SHAP value evaluates the impact on the predicted probability of suicide risk. We found that the 10 most important variables included five biomarkers: TSH, BMG, UA, CYS-C and T3. Our findings reaffirm the significant associations between TSH,²⁶ UA,²⁷ T3⁹ and CYS-C²⁸ levels and suicide risk, consistent with prior research. Notably, we also identify BMG as an additional biomarker, providing novel insights beyond existing studies, which largely focused on individuals with depression. These results underscore the potential utility of a broader biomarker profile, particularly thyroid and renal function indicators, in understanding and assessing suicide risk across diverse populations.

Moreover, the K-means algorithm identifies two clusters. Most clustered SHAP values showed similar trends to the univariate values. However, within one of the clusters, ECT, physical restraint, BMG and T3 showed opposite effects. As evidenced by the interaction plot, these features with heterogeneous effects typically interacted with others. Our results reveal a strong interaction between ECT and the biomarkers TSH and UA, consistent with the significant roles of TSH and UA in suicide risk prediction discussed earlier. A previous study found that the hypothalamic–pituitary–thyroid axis stands out in the effect of the ECT mechanism. An acute increase in serum TSH levels and a decrease in the TSH response to thyroid-releasing hormone after ECT treatment have been reported.²⁹ The relationship between ECT and UA is indirect but mediated through metabolic and endocrine pathways. Research suggested that in males with normal thyroid function, higher thyroid hormone was associated with a lower risk of hyperuricaemia.³⁰ Additionally, patients with bipolar disorder exhibit reduced adenosine triphosphate and adenosine levels, leading to increased UA.³¹ Given that ECT can influence both thyroid hormones and adenosine availability, it may indirectly modulate UA levels.

The impact of ECT on suicide risk itself has been widely studied, though findings are mixed: while most studies suggest ECT reduces suicide risk,^{32 33} some indicate no effect³⁴ and a few suggest an increased risk in certain populations.³⁵ In our study, the cluster-based SHAP analysis indicates that ECT may elevate suicide risk. The inconsistency in previous findings on the association between ECT and suicide risk may be attributed to differences in depression severity or subtype (such as mild depression, MDD or treatment-resistant depression), treatment course and the timing of suicide risk assessment.³³ Given the complex interactions between ECT and multiple variables, further research is warranted to clarify these effects.

Strengths and limitations

The analysis benefited from the extensively characterised, multicentred and chronological real-world cohort. This study therefore provides real-world evidence based on a large sample size and rigorous design. Limitations include that the data comes from only hospitalised patients in a single city, which may lead to potential selection bias. Missing information, as well as the reliability and consistency of self-reported data, is also likely to introduce confounders.

Conclusions

This cohort study identifies a reproducible biomarker signature in patients with MDD that effectively predicts future risk of suicide and enhances the predictive value and clinical utility of prognostic models based on self-reported and other commonly accessible patient information. Our results suggest potential interactions between features, which may present heterogeneous effects on suicide risk.

Biography

Enzhao Zhu graduated from Tongji University School of Medicine, China in 2021 with a Bachelor of Science degree. He has been pursuing a PhD in the Department of Medical Statistics at Tongji University since 2021. His research focuses on digital mental health, leveraging advanced methodologies in medical statistics, deep learning, and large language models to enhance diagnostic and prognostic capabilities. His supervisor is Professor Zisheng Ai. Enzhao Zhu is adept in individual treatment effect estimation, causal inference techniques, general psychiatry-specific large language models, and automatic mental examination systems. His work aims to integrate artificial intelligence with clinical applications to optimise decision support systems in mental healthcare.

Data availability statement

Data are available upon reasonable request.