Modern views of machine learning for precision psychiatry

Summary

In light of the National Institute of Mental Health (NIMH)’s Research Domain Criteria (RDoC), the advent of functional neuroimaging, novel technologies and methods provide new opportunities to develop precise and personalized prognosis and diagnosis of mental disorders. Machine learning (ML) and artificial intelligence (AI) technologies are playing an increasingly critical role in the new era of precision psychiatry. Combining ML/AI with neuromodulation technologies can potentially provide explainable solutions in clinical practice and effective therapeutic treatment. Advanced wearable and mobile technologies also call for the new role of ML/AI for digital phenotyping in mobile mental health. In this review, we provide a comprehensive review of ML methodologies and applications by combining neuroimaging, neuromodulation, and advanced mobile technologies in psychiatry practice. We further review the role of ML in molecular phenotyping and cross-species biomarker identification in precision psychiatry. We also discuss explainable AI (XAI) and neuromodulation in a closed human-in-the-loop manner and highlight the ML potential in multi-media information extraction and multi-modal data fusion. Finally, we discuss conceptual and practical challenges in precision psychiatry and highlight ML opportunities in future research.

The bigger picture

Mental health issues are an epidemic in the United States and the world and have imposed a tremendous burden to the healthcare system and society. To date, there is still a lack of biomarkers and individualized treatment guidelines for mental illnesses. In recent years, machine learning (ML) and artificial intelligence (AI) have become increasingly popular in analyzing complex patterns of neural and behavioral data for psychiatry. We provide a comprehensive review of ML methodologies and applications in precision psychiatry. We argue that advances in ML-powered modern technologies will create a paradigm shift in the current practice in diagnosis, prognosis, monitoring, and treatment of mental illnesses. We discuss conceptual and practical challenges in precision psychiatry and highlight future research opportunities in ML.

Managing and analyzing a large amount of neuroimaging and behavioral data related to mental health have become increasingly important in both precision psychiatry and data science research. Chen et al. present a comprehensive review of machine-learning methodologies, artificial intelligence (AI)-powered technologies and applications by combining neuroimaging, neuromodulation, and mobile devices in psychiatry practice. The review highlights the machine-learning potential in multi-media information extraction and multi-modal data fusion and discusses the challenges and opportunities in future computational psychiatry research.

Introduction

Mental health issues are an epidemic in the United States and the world. According to the National Institute of Mental Health (NIMH), nearly one in five American adults suffer from a form of mental illness or psychiatric disorder (www.nimh.nih.gov/health/statistics/). According to the Centers for Disease Control and Prevention (CDC), the COVID-19 pandemic has witnessed a significant impact on our lifestyle and considerably elevated adverse mental health conditions caused by fear, worry, and uncertainty.¹ Increased suicide rates, opioid abuse, and anti-depressant usage have been observed in both adults and teenagers. The diagnosis and treatment of mental health has imposed a burden to the healthcare system and society. In the United States alone, the economic burden of depression alone is estimated to be at least $210 billion annually.² Precision medicine (or personalized medicine) is an innovative approach to tailoring disease prevention, diagnosis, and treatment that accounts for the differences in subjects’ genes, environments, and lifestyles. The goal of precision medicine is to target timely and accurate diagnosis/prognosis/therapeutics for the individualized patient’s health problem and to further provide feedback information to patients and surrogate decision-makers. Recent decades have witnessed various degrees of successes in precision medicine, especially in oncology³. Traditional diagnoses of mental illnesses rely on physical exams, lab tests, and psychological and behavioral evaluations. Meanwhile, precision psychiatry has increasingly received its deserved attention.^4,5 Although psychiatry has not yet benefited fully from the advanced diagnostic and therapeutic technologies that have an impact on other clinical specialties, these technologies have the potential to transform the future psychiatric landscape.

The NIMH’s Research Domain Criteria (RDoC) initiative aims to address the heterogeneity of mental illness and provide a biology-based (as opposed to symptom-based) framework for understanding these mental illnesses in terms of varying degrees of dysfunction in psychological or neurobiological systems; it attempts to bridge the power of multi-disciplinary (such as the genetics, neuroscience, and behavioral science) research approaches.^6,7 The current gold standard for diagnosis and treatment outcome in mental disorders—the Diagnostic and Statistical Manual of Mental Disorders (DSM), maintained by the American Psychiatric Association (APA)—is often based on the clinician’s observations, behavioral symptoms, and patient reporting, which are all susceptible to a high degree of variability. Therefore, it is imperative to develop quantitative neurobiological markers for mental disorders while accounting for their heterogeneity and comorbidity.

One important goal in neuropsychiatry research is to identify the relationship between neurobiological/neurophysiological findings and clinical behavioral/self-report observations. Machine learning (ML) and artificial intelligence (AI) have generated growing interests in psychiatry because of their strong predictive power and generalization ability for prognosis and diagnosis applications.^8,9,10 The interest of applying ML/AI in psychiatry has grown steadily in the past two decades, as reflected in the number of PubMed publications (Figure 1A). To improve mental health outcomes with digital technologies, the so-called “digital psychiatry” focuses on developing ML/AI methods for assessing, diagnosing, and treating mental health issues.¹¹ A recent global survey has indicated that psychiatrists were somewhat skeptical that AI could replace human empathy, but many predicted that “man and machine” would increasingly collaborate in undertaking clinical decisions, and psychiatrists were optimistic that AI might improve efficiencies and access to mental care and reduce costs.¹²

Figure 1.

ML research in mental health and categorization of neuroimaging

(A) The number of PubMed publications with keywords “machine learning or AI” and “psychiatry or mental health” in the title or abstract (years 2000–2021).

(B) Growth of mental health tech funding in the US market (years 2017–2021; data source: https://www.cbinsights.com).

(C) Human neuroimaging at various spatial and temporal resolution (copyright IEEE; figure reproduced from Thukral et al.¹³ with permission).

The past two decades have witnessed substantial growth of ML applications for psychiatry in the literature, reflected in many applications and reviews.^{17,18,19,20,21,22,23,24,25,26,27} Although multiple reviews of ML for psychiatry are available, the majority of reviews are restricted to relatively narrow scopes. In this paper, we try to provide a comprehensive review of ML and ML-powered technologies in mental health applications. Our view is “modern” in the sense that the development of new technologies, consumer market demand, and public health crises (such as COVID-19) have constantly redefined the role of ML and reshaped our thinking in precision psychiatry. Specifically, we will cover state-of-the-art methodological developments in ML, multi-modal neuroimaging, large-scale circuit modeling, neuromodulation, and human-machine interface. Due to space limitations, our reviewed literature is by no means exhaustive. To distinguish our review from others, we will focus on several issues central to the ML applications for psychiatry: generalizability, interpretability, causality, and clinical and behavioral integration.

Our view about this emerging field is cautiously optimistic for several reasons. First, with an increasing amount of data and computational power, there is a growing demand for psychiatrists to use ML to reevaluate clinical, behavioral, and neuroimaging data. The interests in mental health funding from the industry have also grown substantially (Figure 1B). Second, it is becoming increasingly important to leverage the power of ML and develop explainable AI (XAI) tools for unbiased risk diagnosis, personalized medicine recommendation, and precise neurostimulation. The integration of ML with neuroimaging can potentially help us identify and validate biomarkers in diagnosis and treatment of mental illnesses. Third, there is a growing demand for psychiatrists in the United States, and the shortage is even more acute in poorer countries.²⁸ ML/AI technologies may change the practice of psychiatry for both clinicians and patients. Finally, advanced technologies such as social media, multi-media, and mobile and wearable devices also call for the development of ML/AI tools to assist the assessment, diagnosis, or treatment of individuals who are mentally ill or at risk. The meanings of ML and AI are relatively broad in our current review and generally cover a wide range of analytic or predictive tools that are designed for finding structures or regularity of data; therefore, ML under our discussion also includes data mining and knowledge discovery. From now on, we will use ML and AI interchangeably throughout the paper.

Background of neuroimaging

Advances in neuroimaging

Neuroimaging provides a window to probe human brains in terms of both structural and functional forms and offers various resolutions to examine brain activity at macroscopic, mesoscopic, and microscopic scales across spatial and temporal domains (Figure 1C).

Our understanding of brain and behavior relationships has expanded exponentially over the last few decades. While this improvement may be attributed to a multitude of factors, advancement in neuroimaging has played a prominent role.²⁹ Ranging from increased utilization of structural neuroimaging techniques to the significant scientific advancements brought about by the increased availability of functional neuroimaging, these technologies have provided significant benefits to improved understanding of neural correlates and discovery of biomarkers in psychiatric disorders.³⁰ Some of the most common neuroimaging methods³¹ for probing brain function include the utilization of magnetic resonance imaging (MRI), diffusion MRI (dMRI), functional MRI (fMRI), electroencephalography (EEG), magnetoencephalography (MEG), electrocorticography (ECoG), functional near-infrared spectroscopy (fNIRS), and positron emission tomography (PET). To date, EEG and fMRI are two most commonly used imaging modalities for precision psychiatry. Specifically, EEG is low cost and easy to operate, making it more appealing for clinical practice or home use.

Neuroimaging analysis

These rich neuroimaging modalities allow us to comprehensively probe brain functions. Numerous research efforts have been devoted to revealing the neurobiological basis of various psychiatric disorders using advanced neuroimaging analyses. Under specifically designed cognitive paradigms, task-related neuroimaging allows us to examine the relationship between brain activities (e.g., event-related potential and spectral perturbation and reward or emotional processing-related functional activation^32,33) and cognitive dysfunctions. A promising direction for probing brain function using neuroimaging is to investigate brain connectivity (or connectome).³⁴ Studying the resting-state brain connectome provides an elegant way to characterize the complex brain architecture and uncover brain dysfunctions in intrinsic brain networks.³⁵ Increasing neuroimaging studies suggest that functional connectivity may fluctuate rather than being stationary during an entire session of data collection.³⁶ Studies examining spatiotemporal dynamics of brain networks have recently received growing attention and may reveal meaningful brain states associated with different psychiatric conditions.³⁷ Another promising approach to establish robust biomarkers for psychiatry is to combine multiple neuroimaging modalities in a data-driven manner, which offers opportunities to exploit cross-modality complementary information that a single modality approach may not capture.³⁸

Feature engineering

The input data comprise features that are fed to ML algorithms. All ML methods will benefit from proper feature engineering (including but not limited to imputation, scaling, standardization, normalization, transformation, and one-hot encoding).³⁹ Incorporating knowledge-driven feature engineering into the analysis of specific neuroimaging modalities has shown significant promise in enhancing the model performance and its physiological interpretability. For instance, spatial filtering (or source localization) followed by orthogonalizing the resulting time series and calculating their power envelope correlations can mitigate the effects of volume conduction and source leakage, which may lead to more accurate quantification of functional connectivity in EEG or MEG.^40,41,42

To fully understand the brain structural and functional organization, we argue that neuroimaging, when combined with modern ML and other ML-powered technologies, can provide powerful tools in advancing diagnosis, prognosis, and intervention of psychiatric disorders.

How can ML help psychiatry?

Psychiatry versus other medicine disciplines

The nature and etiology of mental illnesses remain unclear and challenging to study. Traditional studies for the neurobiology of psychiatric disorders have followed a categorical classification framework using a case-control design whereby all patients with a given diagnosis are compared with healthy individuals. The symptom-based diagnosis covered hundreds of thousands of different symptom combinations, which has caused extensive clinical heterogeneity.^43,44 It is increasingly recognized that existing clinical diagnostic categories could misrepresent the causes underlying mental disturbance. The conventional case-control design often fails to match a clinically useful decision process in the absence of differential diagnostic specificity, which is due to its limited strengths in delineating the significant clinical and neurobiological heterogeneity of psychiatric disorders. On the other hand, previous studies have broadly explored the group effects of neurobiology to explain its connection to behavior and disease. However, such group-level analyses cannot fully capture individual-level brain abnormality that is crucial for developing personalized medicine.

In addition, many psychiatric disorders may be considered as falling along multiple dimensions. Cooccurrence of multiple psychiatric disorders might reflect different patterns of symptoms resulting from shared risk factors and perhaps the same underlying disease processes. The high comorbidity in these disorders significantly affects the characterization of psychopathology according to the traditional diagnostic categories. Conventional studies focusing on a single diagnostic domain are therefore insufficient in uncovering the neural correlates of comorbidity among multiple disorders or identifying the dimensions of neural circuits and behavioral phenotypes.

Clinical need driving ML applications in mental health

Despite the rapid progress in psychiatric studies, several areas appear highly underexplored but may carry substantial potentials for achieving major breakthroughs toward precision psychiatry. First, the capacity to dissect inter- and intra-individual variability is crucial for understanding the neural basis of variation in human cognition and behavior.⁴⁵ Studies focusing on the level of individuals may find greater success over conventional group-level analyses. Translational study-orientated approaches for psychiatric neuroimaging may further enhance the ability to find statistically significant effect sizes that can be used in individuals.⁴⁶

Second, identifying subgroups (i.e., subtypes) in psychiatric disorders may delineate disease heterogeneity. Increasing evidence suggests that data-driven subtyping may drive novel neurobiological phenotypes associated with distinctive behavior and cognitive functioning.¹⁵ These stratified phenotypes may help improve the predictability of clinical outcomes and serve as potential biomarkers for treatment selection.⁴² However, subtyping analysis is widely viewed as hypothesis generating and poses significant challenges related to reproducibility and physiological interpretability.^43,47 Linking subtype identification to a particular outcome or question using well-designed ML approaches is vital to address these challenges.⁴⁸

Third, another promising area focuses on transdiagnostic approaches to uncover neural correlates of specific domains (such as cognition, arousal, and emotion regulation), which have been implicated in psychopathology across the diagnostic spectrum.⁴⁹ Recent ML efforts have been dedicated to identifying transdiagnostic brain dysfunctions and dimensions of psychopathology to improve understanding of comorbidity among psychiatric disorders.^50,51,52,53 Importantly, leveraging “big data” from a longitudinal perspective offers a promising way to track the neurobiological and phenotypic trajectories that have been rarely examined in previous cross-sectional psychiatric studies.^54,55,56 Ultimately, such longitudinal studies may help reveal the neural mechanism underlying the disease progression and provide new insights for the development of timely interventions.

It should be noted that the presence of confounding effects is one of the most critical challenges in psychiatry studies.^57,58 For example, the site effects or unmatched phenotypic information (e.g., demographics and clinical measures) may distort the apparent relationship between input features and output. Medications were also found to strongly alter brain activity and connectivity.^59,60 Therefore, inappropriately modeling those confounders can lead to erroneous findings. To date, mental health studies have been done to control the impact of confounders on biomarker quantification.^61,62

These new frontiers in studying psychiatric disorders can be substantially empowered by ML methodologies summarized in Table 1. The applications include stratifying patients into clinically meaningful subtypes, discovering novel transdiagnostic disease dimensions, and tailoring treatment decisions to individual patients. Together, these research outcomes can deliver a significant promise in promoting the development of objective biomarker-based precision psychiatry.

Table 1.

Categories of ML, concepts, typical methods, and their representative applications

Learning category	Concepts	Representative methods	Applications
Supervised	learning from labeled data to predict class/clinical measures	SVM, random forest, sparse learning, ensemble learning	Disease diagnosis, prognosis, treatment outcome prediction
Unsupervised	learning from unlabeled data to uncover structure and identify subgroups	Hierarchical clustering, K-means, PCA, CCA	Disease subtyping, normative modeling, identify behavioral and neurobiological dimension
Semi-supervised	learning from both labeled and unlabeled data to perform supervised or unsupervised tasks	multi-view learning, Laplacian regularization, semi-supervised clustering	multi-modal analysis, joint disease subtyping and diagnosis, prediction with incomplete data
Deep	learning hierarchies and non-linear mappings of features for higher-level representations, can be either supervised or unsupervised	CNN, deep autoencoder, GCN, RNN, LSTM, GAN	a large class of generic learning problems
Reinforcement	solving temporal credit assignment problems, optimal control, trial-and-error learning	temporal difference learning, Q-learning, actor-critic model, dynamic programming	online control, modeling of decision-making and choiced behaviors

The applications of ML in psychiatry can be mainly categorized according to their clinical purposes: diagnosis, prognosis, treatment, and readmission. In contrast to most medical disciplines, traditional diagnoses in psychiatry remain restricted to subjective symptoms and observable signs and therefore call for a paradigm shift. ML offers a new paradigm to achieve automated and more objective assessments for various psychiatric conditions. For disease diagnosis, supervised classification can be used to identify discriminative biomarkers that distinguish a specific disorder from healthy condition or other mental illnesses.⁶³ Unsupervised clustering is useful in identifying disease subtypes for dissect clinical and biological heterogeneity, thus offering new ways of defining psychiatric conditions.^42,64 For disease prognosis, classification models can be built to distinguish different course trajectories (e.g., progressor versus non-progressor), whereas regression models are useful for predicting symptom development during the course of the disease.⁶⁵ For treatment studies, individual responses to the treatment can be predicted using classification methods that distinguish responder from non-responder. Regression-based approaches can also be utilized to predict changes in post-treatment symptoms.⁶⁶ For readmission evaluation, supervised classification can be used to predict whether an individual would be rehospitalized or to detect the relapse trajectory.^67,68 With a properly designed strategy, feature reduction/mapping approaches and knowledge-driven feature engineering can be integrated into the ML model training to identify more informative and interpretable biomarker patterns.⁴⁶

It should be noted that ML is an ever-growing big discipline and covers many categories and emergent topics, each with a different technical focus. A standard taxonomy of ML typically includes supervised, unsupervised, and reinforcement learning paradigms.^69,70 There are also various extensions or special treatment of each category or combined (e.g., semi-supervised learning, kernel learning, ensemble learning, deep learning; see Figure 2). Here, our rationale for reviewing specific ML methods is simply based on their applicability in existing mental health applications.

Figure 2.

Various ML models for mental health applications

(A) Left: multi-modal supervised classification scheme. Three modality-specific factors are optimized on the training data: classifier types, parameters, and weights. The final diagnostic classification is based on a weighted sum of decision values, where weights correspond to those estimated during training. Right: feature importance maps of functional neuroimaging modalities.¹⁴

(B) Unsupervised learning. Left: whole-brain functional-connectivity matrix averaged across all subjects. z = Fisher-transformed correlation coefficient. Right: hierarchical clustering analysis (copyright Springer Nature;¹⁵ figure reproduced with permission).

(C) Semi-supervised learning pipeline for phenotype stratification based on EHRs (Beaulieu-Jones and Greene;¹⁶ figure reproduced with permission).

(D) Deep neural networks (DNNs) for group-level and individualized treatment predictions. Future data points could then be used to forecast symptom onset, treatment response, or other mental health-related variables (Koppe et al.;¹⁷ Creative Commons licenses 4.0).

In the following subsections, we will review several key ML paradigms in mental health applications based on neuroimaging, behavioral, and clinical measurements. A tabular review of representative applications is shown in Table 2. Specifically, we focus on the review of neuroimaging-based psychiatric studies, and detailed reviews of the other data domains (such as genetic, clinical, behavioral, and social media data) will be presented in later sections. For the sake of space limitation, we will not include exhaustive reviews of all ML paradigms (such as reinforcement learning, active learning, and transfer learning) in this section but rather provide some reference pointers whenever necessary.

Table 2.

Representative ML applications in psychiatry based on neuroimaging and clinical data

Application	Learning category	ML method	Mental disorder	Data type	Reference
Diagnosis	supervised classification, deep	dynamic GCN	ADHD	rs-fMRI + phenotypic data	Zhao et al.71
	supervised classification	ensemble learning	ADHD	multi-modal	Luo et al.72
	supervised classification, deep	GCN	ASD	task fMRI	Li et al.73
	supervised classification, deep	ensemble learning + GCN	ASD	rs-fMRI	Khosla et al.74
	supervised classification	PCA + LASSO	bipolar	dMRI + cognitive data	Wu et al.75
	supervised classification	RVM	PTSD	rs-fMRI	Zhu et al.76
	supervised classification, deep	ICA + LSTM	schizophrenia	fMRI	Yan et al.77
	supervised classification	SVM	schizophrenia	sMRI	Mikolas et al.78
	supervised classification, deep	CNN	depression	rs-EEG	Uyulan et al.79
	supervised classification, deep	autoencoder + DNN, SVM, random forest	ASD	rs-fMRI	Heinsfeld et al.80
	semi-supervised classification	GNN	ASD	rs-fMRI + phenotypic data	Parisot et al.81
	unsupervised, subtyping	normative modeling + clustering	PTSD	rs-fMRI	Maron-Katz et al.82
	unsupervised, subtyping	CCA + hierarchical clustering	depression	rs-fMRI	Drysdale et al.15
	unsupervised, subtyping	sparse K-means	PTSD, depression	rs-EEG	Zhang et al.42
	unsupervised, subtyping	latent class analysis	ADHD	task fMRI	Lecei et al.83
	supervised, transdiagnostic	normative modeling + GP regression	multiple disorders	rs-fMRI	Parkes et al.84
	unsupervised, transdiagnostic	sparse CCA	multiple disorders	rs-fMRI	Xia et al.51
	supervised, transdiagnostic	PLS	multiple disorders	rs-fMRI	Kebets et al.52
Prognosis	supervised classification	GP classifier	depression	task fMRI	Schmaal et al.85
	supervised classification	LASSO	psychosis	rs-EEG	Ramyead et al.86
	supervised classification	SVM	psychosis, depression	multi-modal	Koutsouleris et al.87
	supervised classification, deep	DNN	PTSD	rs-fMRI/task fMRI	Sheynin et al.88
	supervised classification	SVM	schizophrenia	sMRI	Nieuwenhuis et al.89
	supervised classification, deep	SVM, random forest, DNN	schizophrenia	task fMRI	Smucny et al.90
	supervised regression	LASSO	substance use	MRI/task fMRI	Bertocci et al.91
	supervised regression, deep	SVR + LSTM	PTSD	MEG	Zhang et al.92
Treatment prediction	supervised classification	SVM	ADHD	sMRI	Chang et al.93
	supervised classification	SVM	psychosis	sMRI	Koutsouleris et al.94
	supervised classification	GP classifier	PTSD	MRI/rs-fMRI	Zhutovsky et al.95
	supervised classification	SVM	schizophrenia	rs-fMRI	Cao et al.96
	supervised classification	SVM	depression	rs-EEG	Zhdanov et al.97
	supervised classification	SVM + GP classifier	depression	sMRI	Redlich et al.98
	supervised regression	latent space learning	depression	rs-EEG	Wu et al.99
	supervised regression	RVM	depression	task fMRI	Fonzo et al.100
	supervised regression	MVPA	ASD	task fMRI	Yang et al.101
	supervised regression	LASSO	anxiety	rs-fMRI	Reggente et al.102
Readmission	supervised classification	SVM	depression	multi-modal	Cearns et al.67
	supervised classification	growth mixture modeling	depression	clinical data	Gueorguieva et al.68
	supervised classification	decision tree	bipolar	EHR	Edgcomb et al.103
	supervised classification	ensemble learning	substance use	phenotypic data	Morel et al.104

Supervised and unsupervised learning

ML holds substantial promise in promoting research from small case-control studies to those with large transdiagnostic samples and from prior specified brain regions to whole-brain circuit dysfunction for individual-level precision medicine.^29,105,106 In a new era of evidence-based psychiatry tailored to individual patients, objectively measurable endophenotypes could allow for early disease detection, personalized treatment selection, and dosage adjustment to reduce the burden of disease.^18,107,108 These promising applications in psychiatric disorders have been enabled by leveraging the powerful strength of modern ML techniques.^{21,63,109,110}

Supervised learning

Supervised learning, being the most popularly used ML category, has been widely applied to neuroimaging-based predictive modeling tasks for psychiatric disorders.¹¹¹ Classic supervised-learning algorithms include logistic regression, support vector machine (SVM), and random forest. Given the high-dimensional nature of neuroimaging data, these approaches are commonly accompanied by a feature selection step to obtain low-dimensional representations. Connectome-based predictive modeling^112,113 is one of such approaches that combine simple linear regression and feature selection to predict individual differences in traits and behavior from connectivity data. Least absolute shrinkage and selection operator (LASSO) provides an alternative approach that performs simultaneous feature selection and prediction to learn a compact feature pattern for accurate prediction of a specific disorder or clinical outcome.¹⁰² Relevance vector machine (RVM) builds upon a probabilistic framework by leveraging automatic relevance determination to learn a sparse solution and penalize unnecessary model complexity.^114,115 RVM has recently demonstrated its strength in quantifying neuroimaging biomarkers for post-traumatic stress disorder (PTSD) diagnosis⁷⁶ as well as for treatment outcome prediction in depression.¹⁰⁰ As an extension of the conventional single-task methods, multi-task learning (MTL) approaches have been increasingly employed to exploit complementary features jointly from multiple views of neuroimaging data.^116,117,118

Due to the complex nature of the brain’s function, informative features may not be observable in the raw high-dimensional feature space. To address this challenge, latent space-based supervised learning has been developed to uncover latent dimensions of neural circuits in psychiatric disorders. For example, a sparse latent space regression algorithm tailored for EEG data was developed to identify anti-depressant-responsive brain signatures in depression.⁹⁹ By jointly estimating the spatial filters and regression weights under a convex optimization framework, the ML model was able to successfully reveal treatment-predictive signatures in a low-dimensional latent space (see case study 1 below). To address comorbidities among psychiatric disorders, dimensional approaches have been developed using statistical models capable of discovering the complex linear relationship between high-dimensional datasets. For instance, low-dimensional representations of depression-related connectivity features have been successfully identified by applying canonical correlation analysis (CCA) to resting-state fMRI (rs-fMRI) connectivity and clinical symptoms.¹⁵ The discovered representations defined two disease dimensions corresponding to an anhedonia-related component and an anxiety-related component, respectively. A similar dimensional analysis was also utilized to examine the neural correlates of neuropsychiatric symptoms in dementia. Using CCA, two latent modes were identified with distinct neuroanatomical bases of common and mood-specific factors of the symptoms.¹¹⁹ A sparse CCA approach has been applied to reveal linked dimensions of psychopathology and functional connectivity in brain networks for psychiatric disorders.⁵¹ This approach successfully identified interpretable dimensions, involving mood, psychosis, fear, and externalizing behavior, all of which guided neural circuit patterns across the clinical diagnostic spectrum. The partial least squares (PLS) approach was also applied to identify latent components linking a broad set of behavioral measures to functional connectivity.⁵² The latent components defined distinct dimensions with dissociable brain functional signatures and provided potential intermediate phenotypes spanning diagnostic categories. These dimensional analytics hold great promise in uncovering novel transdiagnostic phenotypes for developing targeted interventions.

Ensemble learning

Although ML approaches have been extensively designed for supervised learning, using a single model may not produce the optimal generalization performance for a complex prediction task. By combining multiple ML models to reduce variance, ensemble learning outperforms a single model in prediction and has proven successful in discovering robust biomarkers for psychiatric disorders. For instance, multi-atlas ensemble-learning algorithms have been proposed for improved schizophrenia detection¹²⁰ and autism spectrum disorder (ASD) diagnosis.⁷⁴ By utilizing multi-modal neuroimaging including sMRI, fMRI, and DTI, a bagging-based SVM produced significant improvement in prediction of adult outcomes in childhood-onset attention-deficit/hyperactivity disorder (ADHD).⁷² Based on the selective ensemble algorithm, a sparse multi-view prediction model has been designed with rs-fMRI connectivity for ASD diagnosis;¹²¹ this model combined multiple classifiers under a bootstrap framework and significantly outperformed other single-model approaches.

Although sophisticated models of supervised learning often produce better classification or prediction performance, their interpretability decreases at the cost of increasing model complexity. We will discuss the interpretable ML methods in more detail later (XAI in psychiatry). Additionally, labeled data require ground-truth knowledge, which is not always accurate or reliable in the case of mental disorders. For instance, a skin cancer diagnosis may rely on training samples that have been biopsied and cataloged, leaving no doubt as to whether they are malignant or not; however, there is no equivalent of the biopsy in mental disorder.

Unsupervised learning

Unsupervised learning relaxes the assumption of labeled samples and can be useful, e.g., for exploratory data analysis, feature engineering, or cluster analysis. Unsupervised learning aims to uncover the intrinsic data structure by either identifying potential clusters (e.g., using latent class analysis or K-means clustering) or learning a feature mapping that satisfies certain criteria (e.g., using principal-component analysis [PCA]). Identifying patient subtypes offers a promising strategy to delineate neurobiological heterogeneity in psychiatric disorders.⁴³ With rs-fMRI, hierarchical clustering was applied to successfully identify four subtypes of functional connectivity in depression.¹⁵ These subtypes were found to correlate with differing clinical-symptom profiles and predict responsiveness to brain stimulation therapy. From rs-EEG, two transdiagnostic subtypes were identified using sparse K-means clustering with distinct power envelope connectivity patterns and found to respond differentially to anti-depressant medication and psychotherapy.⁴² As a non-distance probability-based clustering approach, latent class analysis has also been applied to discover subgroups in psychiatric disorders. A proof-of-concept study was conducted using latent class analysis to identify ADHD subtypes from fMRI activation profiles⁸³ and revealed that the subtype with attenuated brain activity showed fewer behavior problems in daily life. By leveraging data resources from multiple time points, psychiatric studies have been shifting from cross-sectional analysis to longitudinal modeling.²⁹ Finite mixture modeling became increasingly popular for the analysis of longitudinally repeated-measure data, which can identify latent classes following similar paths of temporal development.^122,123 Typical finite mixture models include growth mixture modeling, group-based trajectory modeling, and latent transition analysis. The use of latent growth mixture modeling (LGMM) and group-based trajectory modeling has become increasingly popular in studying psychiatric disorders, such as depression, anxiety, and ASD. They offer flexible ways to identify latent subpopulations that manifest heterogeneous symptom trajectories.^124,125,126 LGMM approaches have also been successfully used to predict the PTSD course among the population at risk.¹²⁷ As an extension of latent class analysis (LCA) to longitudinal data, latent transition analysis (LTA) may predict the longitudinal service use for individuals with substance use disorder (SUD).¹²⁸ Together, these approaches provide powerful tools to delineate longitudinal heterogeneity and the corresponding distinctive phenotypes during the course of psychiatric disorders.

Semi-supervised learning

Semi-supervised learning is an ML approach that combines supervised learning and unsupervised learning. Popular semi-supervised-learning techniques include self-training, mixture models, cotraining and multi-view learning, graph-based methods, and semi-supervised clustering.¹²⁹ These methods have been increasingly applied to psychiatric studies. By unifying autoencoder and classification, a semi-supervised model was developed for ASD diagnosis.¹³⁰ A semi-supervised classification has been devised using graph convolutional networks and applied to the population graph-based diagnosis of ASD.⁸¹ A semi-supervised clustering has also been designed by extending SVM with implicit clustering driven by a convex polytope to form a method called heterogeneity through discriminative analysis, which can achieve joint disease subtyping and diagnosis.¹³¹ This approach has shown strength in delineating neurostructural heterogeneity in bipolar and major depressive disorders (MDDs)¹³² and schizophrenia,¹³³ as well as in youth with internalizing symptoms.¹³⁴ Additionally, semi-supervised learning has gained increasing mental health applications in digital data from electronic health records (EHRs), social media, and mobile phones.^16,135,136 See ML-powered technologies for psychiatry for a detailed discussion.

Normative modeling

Normative modeling is an emerging and innovative framework for mapping individual differences at the level of a single subject with respect to a reference model.¹³⁷ It has been increasingly used in mental health¹³⁸ to parse the substantial neurobiological heterogeneity by quantifying individual deviations. By building a normative model of neuroimaging data on a large-scale healthy population, brain abnormalities of individual patients can be quantified by examining their statistical differences from the distribution of the norm. Gaussian process (GP) regression-based normative modeling has been applied to quantify individual deviations and dissect neurobiological heterogeneity in various psychiatric disorders. With this tool, an association was successfully discovered between transdiagnostic dimensions of psychopathology and an individual’s unique deviations from normative neurodevelopment in brain structure.⁸⁴ By combining tolerance interval-based normative modeling and clustering analysis, individual abnormalities in rs-fMRI were accurately quantified to define two stable subtypes in patients with PTSD.⁸² The two subtypes showed distinct patterns of functional connectivity with respect to the healthy population and differed clinically on levels of reexperiencing symptoms. These novel data-driven approaches provide useful techniques to identify “abnormal” subtypes in patients, thereby advancing clinical and mechanistic investigations in psychiatric disorders. More recently, an autoencoder model has been utilized to realize normative modeling for detecting microstructural deviations in Chamberland et al.¹³⁹

Deep learning

Deep learning consists of a collection of methods that use multi-layered-architecture ($\ge$2 hidden layers) artificial neural networks for ML tasks. Through a specifically designed deep neural network structure, high-level feature representations can be learned from raw features. Deep learning thus holds promise in offering an end-to-end analytic framework for disease diagnosis and prediction. With the advancement in neuroimaging technologies, an increasing number of large-scale multi-center datasets have been established for building powerful ML models to fully explore the informative feature representations from complex brain and genomic data. By training on these large-scale datasets, deep learning can learn robust neuroimaging representations and outperform standard ML methods in a variety of application scenarios in mental health.^{17,140,141,142}

Deep autoencoder

The deep autoencoder, also known as stacked autoencoder (SAE), aims to learn latent representations of input data through an encoder and uses these representations to reconstruct output data through a decoder. By stacking multiple layers of autoencoders, a deep autoencoder is formed to discover more complicated and potentially non-linear feature patterns. Deep autoencoder has been applied to extract low-dimensional features from the amplitude of low-frequency fluctuations in fMRI.¹⁴³ Clustering analysis with the latent features uncovered by deep autoencoder further identified two subtypes within major psychiatric disorders including schizophrenia, bipolar disorder, and MDD. A deep-learning model was also designed based on a sparse SAE and applied to lower the dimensionality of fMRI connectivity. The sparsity constraint used in this model yielded interpretable neural patterns for improved ASD diagnosis.¹⁴⁴ Deep autoencoder has also been applied to implement normative modeling with structural MRI for the quantification of individual abnormalities in neuropsychiatric disorders, including schizophrenia and ASD.¹⁴⁵ The abnormal features extracted using the normative model led to improved diagnosis performance compared with the traditional case-control analysis. Recently, a deep contrast variational autoencoder was used to extract neuroanatomical features from MRI data to identify brain dysfunction that can be attributed to ASD and not to other causes of individual variation.¹⁴⁶

Convolutional neural networks (CNNs)

Different from conventional multi-layer perceptron or autoencoder assigning a different weight to each input feature, CNNs were designed to better capture the spatial and local structure information from pixels or voxels.^110,147 Due to its strength in utilizing neighborhood information to learn hierarchies of features,¹⁴⁸ CNNs have been one of the most successful deep-learning models applied in various medical applications. A diagnosis model was established through EEG-based image construction coupled with the CNN for accurate detection of MDD.⁷⁹ This model provided an end-to-end framework to successfully identify translational biomarkers from rs-EEG in distinguishing depressive patients from healthy people. With whole-brain structure MRI, a three-dimensional (3D) CNN model has also been designed to automatically extract multi-layer high-dimensional features for the diagnosis of conduct disorder.¹⁴⁹

Graph neural networks (GNNs)

Though deep-learning models have shown strengths in capturing complex neuroimaging patterns, they may not generalize well to non-Euclidean data types (e.g., brain networks). In contrast, GNNs provide a clever way of learning the deep graph structure of non-Euclidean data, leading to enhanced performance in various network neuroscience tasks.¹⁵⁰ For instance, a framework based on graph convolutional networks (GCNs) has been designed for the diagnosis of ASD.⁸¹ By building a population graph that integrates rs-fMRI data as node features and phenotypic measures as edges, the designed model outperformed other state-of-the-art methods. An inductive GNN model was also devised to embed the graphs containing different properties of task fMRI and drive interpretable connectome biomarkers for ASD detection.⁷³ More recently, a novel GNN model was developed to incorporate dynamic graph computation and feature aggregation of 2-hop neighbor nodes into graph convolution for brain network modeling.⁷¹ This dynamic GNN significantly improved the performance in ADHD diagnosis and revealed the circuit-level association between connectomic abnormalities and symptom severity.

Recurrent neural networks (RNNs)

As a specific extension of the feedforward neural network, RNNs have the ability to learn features and long-term dependencies from sequential and time-series data. Long-short-term memory (LSTM) models are the most popular RNN and have shown advantages in capturing temporal dynamic information of neuroimaging data for various psychiatric disorder studies.¹⁵¹ An LSTM-based RNN architecture was built with the time course of fMRI-independent components to exploit the temporal information, which yielded an improved diagnosis of schizophrenia.⁷⁷ By combining RNNs with other deep neural networks, novel ML models have also been proposed to model the spatiotemporal dynamics in neuroimaging data. A spatiotemporal CNN model was proposed for 4D modeling of fMRI, with confirmed robustness in identifying key features in the default mode network.¹⁵² LSTMs have also been applied to incorporate multi-stage neuroimaging data into longitudinal analytic frameworks for modeling the trajectories of psychopathology development in various psychiatric disorders. A recent LSTM-based model was built with MEG data to achieve accurate longitudinal tracking of pathological brain states and prediction of clinical outcomes in PTSD.⁹²

Generative adversarial networks (GANs)

As one type of generative model, GANs have gained considerable attention in computer vision and natural language processing and also have become increasingly popular in neuroimaging analysis.¹¹⁰ GANs consist of two competing neural networks (one as generator and the other as discriminator) and can learn deep feature representations without extensive labeled data. Due to this unique advantage, GANs have been increasingly applied in data augmentation to enhance the sample size for model training.¹⁵³ Moreover, GANs have been used to impute missing values in multi-modal datasets, a common problem in psychiatric studies, rather than discarding an entire multi-variate data point.¹⁵⁴ The adversarial model has also been incorporated into other ML models for specific applications in psychiatric studies. For instance, the discriminative and generative components were incorporated in LSTM to form a MTL approach for fMRI-based classification, which resulted in an improved diagnosis of ASD compared with the standard LSTM.¹⁵⁵ By integrating GANs with group independent component analysis (ICA), a functional connectivity-based deep-learning model was developed for the diagnosis of MDD and schizophrenia.¹⁵⁶ Specifically, the generator with fake connectivity was trained to match the discriminator with real connectivity in the intermediate layers, whereas a new objective loss was determined for the generator to improve the diagnosis accuracy. More recently, a confounder-free deep-learning framework was designed by incorporating the concept of GAN into the model training.⁶² This end-to-end approach is capable of simultaneously learning informative features and controlling for confounder effects to improve model performance.

The strength of deep-learning algorithms is that they can learn complex predictor-response mappings, but the power also comes at the cost of requiring a very large sample size for model optimization. This poses potential overfitting and interpretability challenges in psychiatric applications.¹⁷

Key ML concepts for precision psychiatry

Regardless of the ML paradigms in psychiatric applications, there are some common themes that distinguish human intelligence from automated or human-in-the-loop machine intelligence. In a recently published white paper, “Machine intelligence for healthcare,” four important features are emphasized for ML systems.¹⁵⁷ These concepts are broadly applicable to precision psychiatry.¹⁹

• •Trustworthiness: the ability to access the validity and reliability of an ML-derived output across varying inputs and environments. In other words, psychiatrists need to be able to evaluate the limitations of an ML system and confidently apply system-derived information for psychiatric evaluation.
• •Explainability: the ability to understand and evaluate the internal mechanism of a machine. The development of ML systems will need to account for data quality, quality metrics for the system’s functioning and impact, standards for applications in the environment, and future updates to the system.
• •Usability: the extent to which an ML system can be used to achieve specified goals with effectiveness, efficiency, and patient satisfaction in multiple environments. These applications need to be scalable across multiple settings while preventing additional burdens on providers and patients.
• •Transparency and fairness: the right to know and understand the aspects of an input that could influence outputs (clinical decision support) from the system. Such factors should be available to the people who use, regulate, and are affected by any type of care decision that employs the ML system. The potential bias in the data or the system needs to be identified and informed prior to decision-making.

The first two features are related to interpretability, which we will discuss in more detail in XAI in psychiatry. The other two features will be discussed in discussion and conclusion.

Case studies

To help the reader get a concrete idea of the reviewed ML techniques in psychiatric applications, here we present several case studies to illustrate the strengths in prediction/classification diagnosis analytics. These representative case studies employ different ML strategies and cover different data modalities, including rs-EEG, task fMRI, and ECoG.

Case study 1: Sparse latent space learning for EEG-based treatment prediction in depression

Anti-depressants have shown only modest superiority over placebo, which is partly because the clinical diagnosis of MDD encompasses biologically heterogeneous conditions that relate differentially to treatment outcomes. It is important to develop a robust neurobiological signature for an anti-depressant-responsive phenotype that determines which patients will benefit from medications. To address the challenge, Wu et al.⁹⁹ developed a sparse EEG latent space regression (SELSER) model to predict the treatment outcome. Specifically, SELSER optimizes the spatial filters and regression weights in conjunction under a convex optimization framework and identifies an anti-depressant-responsive EEG signature for MDD (Figure 3A). The identified signature accurately predicts anti-depressant outcomes (n = 228). A neurophysiologically interpretable cortical pattern was further observed through a source mapping from the scalp spatial pattern, mainly contributed by the right parietal-occipital regions and the lateral prefrontal regions (Figure 3B). The validation on an independent cohort showed that the treatment outcomes predicted by the brain signature are significantly higher in a partial responder group versus a treatment-resistant group, demonstrating its further clinical utility in the broader construct of treatment resistance in depression.

Figure 3.

Concepts and major findings in case studies 1 and 2

(A) Illustration of the sparse EEG latent space regression (SELSER) framework in case study 1 for treatment outcome prediction.

(B) Interpretable cortical pattern derived from the scalp pattern (copyright Springer Nature; figures are modified from Wu et al.⁹⁹ with permission).

(C) Distinctive EEG connectivity profiles were identified by sparse K-means for defining psychiatric subtypes in case study 2 on PTSD and MDD. The two identified subtypes were further found to predict treatment responsiveness to psychotherapy and antidepressant medication.

(D) The EEG connectivity-defined subtypes are distinguishable by rs-fMRI connectivity patterns derived from an RVM-based classifier (copyright Springer Nature; figures are modified from Zhang et al.⁴² with permission).

Case study 2: Unsupervised learning-based identification of neurophysiological subtypes in psychiatric disorders

Neurobiological heterogeneity has a substantial impact on treatment outcome independent of pre-treatment clinical symptoms. For example, although psychotherapy is currently the most effective treatment for PTSD, many patients are nonetheless non-responsive and display differences in brain function relative to responsive patients. Using sparse K-means clustering, Zhang et al.⁴² developed a data-driven framework to achieve simultaneous feature selection and subtyping on the high-dimensional power envelope connectivity of rs-EEG source-reconstructed signals. This approach successfully identified two transdiagnostic subtypes with distinct functional connectivity patterns in PTSD and MDD (n = 648), which were prominently within the frontoparietal control network and default mode network (Figure 3C). Importantly, linear mixed models in an intent-to-treat analysis on symptom severity revealed that the two subtypes differentially responded to psychotherapy and anti-depressant versus placebo. An RVM-based classification analysis further confirmed that the EEG connectivity-driven subtypes were distinguishable using rs-fMRI connectivity. The discriminative pattern identified from fMRI was also consistent with the EEG connectivity pattern (Figure 3D).

Case study 3: Classification of anxious versus non-anxious brains from fear extinction learning task-based fMRI

Using a neuroimaging cohort study (n = 304 adults, 92 patients with anxiety, 74 trauma-exposed individuals, 138 matched controls), Wen et al.¹⁵⁸ examined how the fMRI activations of 10 brain regions that were commonly activated during fear conditioning and extinction (Figure 4A) might distinguish anxious or trauma-exposed brains from controls. They proposed a CNN classifier (Figure 4B) to map fear-induced fMRI activities in space and time to a prediction probability score indicating that the subject belongs to the anxious group. The CNN achieved an area under the receiver operating characteristic curve (AUC) of 0.84 ± 0.01, 0.75 ± 0.03 sensitivity, and 0.77 ± 0.02 specificity in 5-fold cross-validation (Figure 4C), outperforming other ML methods (e.g., SVM and random forest). The prediction score was also found to correlate with the anxiety sensitivity index (ASI) in the control group (Figure 4D). Furthermore, control analyses were performed to demonstrate the specificity of the fear network in discrimination (Figure 4E).

Figure 4.

Illustrations of concepts and major findings in case study 3

(A) Experimental paradigm.

(B) Schematic of the CNN.

(C) AUC curves produced by CNN versus chance level.

(D) The prediction score positively correlated with the anxiety sensitivity index (ASI) for the control group (r = 0.41, p < 0.001) but at the chance level for anxious brains (r = −0.05, p = 0.65).

(E) Distribution of AUCs based on brain activations within the 10-node fear randomly selected brain regions (figures were adapted from Wen et al.¹⁵⁸ with permission).

Case study 4: Decoding mood state from multi-site intracranial brain activity

From intracranial ECoG signals and simultaneously collected self-reported mood-state measurements over multiple days in seven patients with epilepsy, Sani et al.¹⁵⁹ developed a dynamic state-space model (SSM) framework to track the patients’ mood-state variations over time (Figure 5A). The modeling framework consists of unsupervised- and supervised-learning components (Figure 5B). The spectro-spatial features were extracted from the mood-predictive network within the limbic brain region. The neural decoders were also highly predictive of the immediate mood scaler (IMS) points at the population level. Furthermore, the same trained decoder could be used for mood-state prediction across hours and days and generalized across a wide range of IMS. In cross-validation, the decoders could predict IMS variations that covered 73% and 33% ± 7.2% of the total possible IMS range across all seven subjects and within individuals, respectively (Figure 5C). These results suggest that ML-based decoders can predict mood-state variations from brain activity across multiple days of recordings in patients.

Figure 5.

Illustrations of concepts and major findings in case study 4

(A) Schematic of cross-validation. An IMS point (e.g., $S_N$) is left out as the test IMS to be predicted. The other IMS points (i.e., training IMS, using $S_1$ to S_N–1) and the associated neural activity are used within the modeling framework to train a neural encoding model.

(B) Main components of the modeling framework based on both unsupervised and supervised learning.

(C) Cross-validated prediction of the mood state is shown against the true measured mood state (copyright Springer Nature; figures were modified from Sani et al.¹⁵⁹ with permission).

ML-powered technologies for psychiatry

ML can be applied to a wide range of digital platforms, including software (e.g., mobile apps), hardware (e.g., wearable devices, robots), social services (e.g., online chatbots), and clinical practice (e.g., EHRs). In this section, we will review various ML-powered technologies in the non-neuroimaging domains and highlight the emerging digital platforms for precision psychiatry.

A recent McKinsey study showed that use of telehealth has increased by 38-fold compared with the pre-COVID baseline.¹⁶⁰ With a steep increase in teletherapy demand and consumption, many companies (such as Talkspace and Headspace Health) provide services that include chat-based conversations with licensed mental health professionals. The definition of teletherapy today has expanded to include these newer modalities of care delivery. These advances in care delivery have enabled collecting massive amounts of text, audio, and video data on a regular basis, which was previously only available in controlled research settings. Furthermore, the recent advancements of natural language processing, speech, and video analysis technologies, combined with the ML tools, have generated numerous innovations in this emerging field. The global psychiatrist community is increasingly aware of these developments. For example, a recent survey among more than 700 psychiatrists showed that 49% believed that in the next 5–10 years, ML technology will help analyze patient information to establish prognosis, and 54% believed that this technology can help synthesize patient information to reach a diagnosis.¹²

ML can be applied across all stages of a patient’s journey:^161,162 risk assessment, diagnosis, prognosis, treatment, and relapse in a variety of disorders,¹⁶³ where the analysis can be applied to natural language, speech, facial expressions, body language, and social media, as well as traditional clinical surveys and neuroimaging data.^22,164 Table 3 summarizes recent representative studies that use ML to support various stages of patient journey. Applying ML can build personalized models that are optimized for each patient,⁸ as opposed to traditional models that are only optimized for group effects. Furthermore, given the inter- and intra-disorder variability between clinical diagnosis and symptoms, ML methods such as MTL can be used to model differential diagnoses between disease categories. All of these mentioned ML applications can be considered to be the first level of precision psychiatry.

Table 3.

Representative ML applications of multi-media data in mental disorders

Study	Data source(s)	Patient journey stage	ML approach	Test sample size
Depression spectrum
Vázquez-Romero et al.165	audio – clinical interviews	diagnosis	CNN ensemble	47 speakers
Harati et al.166	audio – answers to personal questions	diagnosis	transfer learning	3,078 speakers
Huang et al.167	audio – clinical interviews	diagnosis	SVM with speech landmark features	47 speakers
Zhu et al.168	face video – reading and personal questions	diagnosis	CNN	50 videos
Shao et al.169	gait-only video – casual walking in a corridor	diagnosis	LSTM + CNN weighted fusion	40 videos
Lu et al.170	language – answers to personal questions	diagnosis	LSTM fine-tuned with health forum data	2,425 subjects
Eichstaedt et al.171	language – Facebook posts	risk assessment	logistic regression	68 patients
Sun et al.172	audio, video – clinical interviews	diagnosis	transformer + multi-modal fusion	56 subjects
Bipolar spectrum
Weiner et al.173	audio – verbal fluency tasks	relapse	SVM	56 subjects
Palmius et al.174	sensor – GPS	diagnosis	linear regression	36 subjects
PTSD
Marmar et al.175	audio – clinical interviews	diagnosis	random forest	43 veterans
Mallol-Ragolta et al.176	audio, video, skin conductance	relapse	SVM	110 subjects
Schizophrenia spectrum
Tahir et al.177	audio – clinical interviews	diagnosis	SVM	70 subjects
Abbas et al.178	video – neutral open-ended questions	diagnosis	logistic regression	16 subjects
Birnbaum et al.179	language – internet search queries	relapse	random forest	23 subjects
Birnbaum et al.180	audio, video – clinical interviews	diagnosis	gradient boosting	17 subjects

However, the amount of precision that can be modeled using ML is far beyond the first level.^181,182 During psychiatric evaluation, psychiatrists may try to build a mental model of what is going on in the patient’s life in about 30 min. They aim to understand as much as possible about the patient’s history within a short time, define what “normal” looks like for the patient, and identify deviations from the normal. This is often done by asking questions to the patient and examining their speech, body language, and behavioral responses. It is very challenging and nearly unrealistic to expect psychiatrists to build an accurate baseline model of the patient’s entire life within such a short time span while interacting with the patient in a compromised psychological state. ML can help by building baseline models specific to each patient before their visit and present the bounds for various observations as a reference to psychiatrists during the exam.¹⁸³ This can be viewed as the second level of precision psychiatry (Figure 6A). Take MDD as an example: Figure 6B shows how ML can be applied at different stages of a patient’s journey.

Figure 6.

ML for precision psychiatry

(A) Two levels of precision in applying ML for mental health.

(B) Examples of ML applications at various stages of a patient’s journey in case of MDD.

In the following subsections, we describe how ML can be applied to clinically relevant data and to support one or more stages of the patient’s journey.

Mobile and sensing technologies

The development of smartphones, smart watches, and other wearable sensing devices have enabled us to access more information about our physical and mental health than ever.¹⁸⁴ Specifically, several types of signals are relevant for mental health monitoring and assessment (Figure 7A):

• •Behavioral and physical signals: location (e.g., GPS coordinates), mobility (e.g., accelerometer).
• •Multi-media signals: face expression, speech patterns.
• •Social signals: social interactions (e.g., call and text message logs), communication patterns, engagement, online gaming.
• •Physiological signals: skin conductance, heart rate variability (HRV), eye movement, electrodermal activity (EDA).
• •Sleep activity: sleep duration, sleep staging, phone on/off status.

Figure 7.

Illustrations of ML-powered technologies for mental health

(A) ML applications in mobile health.

(B) Different types of data collection strategies for digital measurement tools.

(C) A technological infrastructure for the integration of digital measurement tools. Independent platforms for measurement of health will have their own data repositories, depicted as clouds. These data could be safely transferred across platforms using transfer tools such as secure application program interfaces (APIs), depicted using dashed arrows. Such tools could allow for both unidirectional and bidirectional movement of data. ML can be applied to integrate all measures for clinical decision-making (B and C are reproduced from Vázquez-Romero and Gallardo-Antolín¹⁶⁴ with permission).

These signals have different implications and relevance to mental illnesses. Although a single signal may not be indicative of a mental disorder, combination of these physical/physiological/social cues may reveal important clues of individual mental health. In what follows, we will focus on the analysis of multi-media, language, and social media data for development of their mental health applications.

Speech and video analyses

To date, voice and visual (video of facial expressions and body language behaviors) data have gained increasing attention in the studies of mental disorders. ML technologies using speech samples obtained from the clinic or accessed remotely may help identify biomarkers to improve diagnosis and treatment. Since the early days of practice, psychologists have already used auditory and visual cues to assist with the diagnosis of mental illnesses.¹⁸⁵ Furthermore, speech and video are not only readily available in traditional teletherapy settings but are also easily interpretable as the most natural form of human communication.

Audio and speech features

Acoustic features derived from audio data have been found to be relevant in many mental disorders,^175,186 including speech analysis for patients with depression, bipolar, and schizophrenia. Table 4 lists some commonly used acoustic features in the analysis of mental illnesses.¹⁸⁷ These categories have enabled standardization and interpretation of ML-analyzed speech data in clinical applications.

Table 4.

Multi-modal data features and their uses in mental health

Features	Example(s)	Relevant in mental disorder(s)	Study
Acoustic
Source of sound features	jitter	increase with depression severity	Cummins et al.188
Filtering features by vocal and nasal tracks	first resonant peak in the spectrum	increase with bipolar severity	Weiner et al.173
Spectral features of speech	Mel-frequency cepstral coefficients	a variety of disorders	Low et al.186
Prosodic features of speech	pause duration	higher in schizophrenia	Low et al.186
Video
Facial	smile duration, eyebrow movement, disgust expression	increased disgust expression in suicidal ideation	Galatzer-Levy et al.189
Eyes	gaze angle	more non-mutual gazes in MDD	Song et al.190
Gait	arm swing and stride	reduced arm swing in MDD	Shao et al.169
Posture	head pitch variance, upper body movements	reduced head movement in schizophrenia	Abbas et al.191
		higher head movement in ASD	de Belen et al.192
Language
Grandiosity	unrealistic sense of superiority	increased in bipolar	Castro et al.193
Semantic coherence	flow of meaning	decreased in psychosis	Morgan et al.194
Rumination	repetitive thought patterns	increased in MDD	Rutowski et al.195
Self-focus	self-referent information	increased in stress	Kim et al.196

Models built from speech-based features may be effective in predicting the diagnosis of depression and suicidality.¹⁸⁸ Applications for depression include predicting the presence, severity, and score.^165,173 These models use prosodic, spectral, or other features computed from raw speech data to quantify flattered speech, slow speech, and other relevant markers. The target outcome variable is derived from a clinically valid scale such as a patient health questionnaire (PHQ-9). Furthermore, models for suicidality that explore similar features have been used in multi-class settings to differentiate among healthy, depressed, and suicidal speech.

One key challenge in applying speech-based models in clinical practice is the lack of longitudinal data validation in real-world settings. However, this issue is starting to get addressed in recent studies,¹⁹⁷ which detect manic and depressive speech from recordings of outgoing calls from phone conversations of consenting participants. Another challenge is the lack of large labeled datasets for evaluating performance across various methods. To this end, it is noted that companies like Ellipsis Health^166,198 have used deep learning and transfer learning to predict depression and anxiety scores with high accuracy based on a large labeled dataset of over 10,000 unique speakers. Human-level accuracy using only 20–30 s of an audio clip has been reported in some commercial applications to detect depression.^199,200

Visual features

Although body language and facial expressions have always formed a key part of a psychiatric exam, ML has only recently been applied to analyze such data objectively. To date, most work has focused on suicidal ideation,¹⁸⁹ depression,^168,169,190 schizophrenia,¹⁷⁸ and ASDs.¹⁹² Features derived from overall facial expression, eyes, gait, and posture (Table 4) have been shown to be relevant across many mental disorders.

Studies in suicidal ideation have mainly focused on using interpretable ML for characterizing the disorder. This makes the ML models more applicable in augmenting human caregivers by bringing up a specific insight that they would like to measure. In depression studies, some approaches have also involved fusion of video features derived from each frame that are used to train a sequential DNN,²⁰¹ and some have used pre-training to compensate a relatively small sample size of depression datasets.²⁰² While these models perform very well on the same held-out test set, their clinical applications remain limited due to a lack of interpretability. To improve interpretability, depression activation maps were developed to highlight the facial areas corresponding to the depression severity as learned by the ML model.²⁰³ Meanwhile, utilization of pre-defined features has been most effective in providing interpretable results.^202,204

Natural language processing (NLP)

NLP techniques enable computers to analyze, understand, and derive meaning from text and speech in a similar manner to humans. NLP techniques can enable mental health professionals to evaluate language patterns to help identify and predict psychiatric illness (Table 4). Language is not only one of the primary expressions of human behavior that carries a variety of implicit and explicit markers relevant to mental health^205,206 but is also more abundantly available compared with speech data. For example, social media platforms contain a large quantity of real-world language data, whereas speech data are rarely available at that scale. There are two types of NLP applications for detecting specific mental health symptoms. The first type of applications is directly applied to the patient data, varying from predicting the risk of suicide and early psychiatric readmission to identifying phenotypes and comorbidities. The second type of applications is applied to EHRs and clinical records (tests, transcripts), which can be used for automating chart reviews, clustering patients into phenotype subtypes, and predicting patient-specific outcomes. The EHRs (including pathology reports, lab results, clinical tests, and clinical session transcripts) are systematic collections of longitudinal, patient-centered clinical records. Patients’ EHRs consist of both structured and unstructured data. The structured data include information about a patient’s diagnosis, medications, and laboratory test results, and the unstructured data include information in the form of clinical notes.

Massive EHR datasets have provided opportunities to adapt ML approaches to track and identify target areas for quality improvement in mental health care. According to a 2015 national survey, 61.3% of US psychiatrists use EHRs.²⁰⁷ The EHR language is at least one level abstracted from the patient’s symptoms, consisting of clinical notes. However, the unique advantage of EHR data is the ease with which demographic and socioeconomic features can be combined with the language data. Symptoms derived from free texts in EHRs have been used for prediction of bipolar disorder,¹⁹³ situational aggression,²⁰⁸ and suicidal ideation,²⁰⁹ with achieved comparable performance to clinicians. Furthermore, discharge summaries from EHRs have also been used to predict relapse.²¹⁰ Aside from symptoms, a variety of relevant mental health data (such as the intervention status and physical health comorbidities) can be routinely extracted from EHRs using NLP methods.²¹¹ Privacy concerns around EHR data sharing remain one of the key challenges in validating generalization of NLP methods. Encouragingly, there has been a growing interest in using transformers for generating artificial mental health clinical notes to mitigate this issue.^211,212

The advances in text-based mental health interventions (e.g., Talkspace and CrisisTextLine) have made transcripts of clinical sessions easily amenable to NLP. Aside from developing models for detecting suicidal ideation,²¹³ NLP can also be applied to these datasets to identify the population-level trend, such as the increase in anxiety and decrease in quality of personal relationships during the COVID-19 pandemic.²¹⁴ Since language data are ubiquitous, one of the NLP challenges in mental health applications is data standardization. Depending on the task, different types of data may yield different levels of “signal.” For example, to predict first-episode psychosis, language data from clinical tests have higher performance compared with transcripts of free speech.¹⁹⁴ On the other hand, data collected from free-speech samples for diagnostic purposes can be highly effective for developing a language-based depression screening that generalizes well across various age groups.^170,195,198

Social media

To date, social media companies have collected a large amount of language data that may contain clinically relevant information. This information can not only be extracted on a population level, such as the notable rise in cognitive distortions over time,²¹⁵ but can also be attributed on an individual level,²¹⁶ all of which have made social media a powerful tool to support mental health risk assessment and diagnosis. Language from Facebook posts, for example, has been shown to contain markers for depression. Rumination and sadness can be detected in such data up to 6 months prior to a clinical diagnosis.¹⁷¹ Models applied to Facebook and other social platforms (e.g., Twitter and Reddit) have been successful in predicting the diagnosis of psychosis, anorexia, anxiety, and stress levels.^196,217,218 In addition to the language analysis for the user posts and comments, ML models may also process media data such as Instagram images²¹⁹ or integrate images and text to infer the user’s mental state.^219,220 Entries of online search also form a complementary and equally compelling dataset alongside social media activity.

Recent developments of transformer models, including those learning multi-lingual language representations, have enabled researchers to apply powerful NLP models to detect depression or self-harm from social media data.^221,222 Furthermore, specialized language representations that were trained on mental health-specific conversations and that became publicly available²²³ have been shown to improve performance compared with non-specific representations. Finally, an ML technique known as transformers can assess text responses via NLP and predict traditional subjective well-being measures approaching the theoretical upper limits in accuracy.²²⁴

While social media solves the scale issue with millions of samples available, most social media data lack clinically valid labels.²²⁵ Most reported studies have relied on using labels from self-disclosure of mental illness, which are not only inaccurate but also bring additional issues of defining a healthy control. Despite the challenges, the validity of social media data has been repeatedly proven to support mental health diagnosis and risk assessment.

Sensing technologies and mobile mental health

Smartphones, wearables, and other devices equipped with ambient sensors (Figure 7B) are increasingly capable of recording physiological measurements that are known to affect mental health.²²⁶ In addition, some of the less obvious measurements (such as keystroke usage patterns) have been shown to be implicated in mental health.^227,228 Additionally, online gaming behaviors, such as interaction patterns with non-player characters (NPCs) and other game behavior patterns, can be used to measure cognitive performance and their relationship with mental illness.^229,230

Measurements from mobile sensors may constitute valuable sources of mental health data (Table 5) and can be useful at various levels of granularity: from raw sensor data (e.g., the accelerometer) to derived high-level features (such as psychomotor activity). This has inspired many corporations to invent technologies for detecting depression and cognitive decline based on data collected from their wearable devices.²³¹ Sensor-based measurements are found to be correlated with high-stress levels and a variety of ailments including depression, anxiety, psychosis, and bipolar disorder.^232,233 Since sensor-based data are widespread and readily available, they offer an opportunity to build baseline models for individual users, which can be then used to identify significant physiological changes in users and further inform clinical interventions. Devices that collect various data streams from patients, such as surveys, cognitive tests, social medial interactions, GPS coordinates, and behavioral patterns (e.g., keyboard typing), have a great potential for monitoring, managing, and predicting the individual’s mental health.^234,235 Overall, longitudinal quantification of these data streams may result in clinically meaningful markers that can be used to refine diagnostic processes, tailor treatment choices, improve condition monitoring for actionable outcomes (such as early signs of relapse), and develop new intervention models.^{236,237,238,239}

Table 5.

Mobile sensor measurements and potential applications for mental health monitoring

Measurement	Feature	Effect in mental health
Movement	psychomotor agitation	increased in anxiety
Location	social avoidance	increased in MDD
Social activity	call/text volume	reduced in MDD
Keystroke	keystroke latency	impaired in ADHD
Heart rate	heart rate variability	impaired in stress
Gaming	NPC interactions	impaired in social anxiety

Commercial and research platforms and services

While studies have demonstrated promising results in ML applications for the individual’s journey in mental health, their broad applicability in clinical practice remains limited. Table 6 lists examples of platforms and services that use ML for mental health applications. While most platforms focus on risk assessment based on single modality, the initial commercial viability of these platforms is still promising for the success of using ML in mental health because they enable a collection of large amounts of data that can be used to further validate biomarkers.

Table 6.

Commercial and research platforms and services for mental health applications

Platform	Primary data source	Mental health appl.
WoeBot240	language	depression, anxiety
Mindstrong227	keystrokes	serious mental illness
Sonde Health167	voice	mental fitness
Ellipsis Health166	voice	stress
Amazon Halo241	voice	emotion detection
Apple Watch200	mobility, sleep	depression
Alphabet Fitbit242	skin conductance	stress
Kintsugi243	voice	depression, anxiety
Bewie234	raw data from smartphone	multiple
MindDoc	language (ask daily question)	depression
Clarigent Health	voice	suicide risk

There are between 10,000 and 20,000 smartphone apps that digitize mindfulness or cognitive behavioral therapy (CBT) techniques,²⁴⁴ allowing the user to engage in psychotherapy at a greatly reduced price compared with in-person therapy. However, the quality is highly variable, and the mechanisms used to validate them are often dubious. Moreover, since this area is relatively new, the industry and governmental standards to validate such a technology are still in the early phase. We will briefly outline two interrelated areas of development: digital measurements and digital interventions.

Digital measurements

We are entering a new era of digital psychiatry.^11,245 In 2016, the Harvard professor Jukka-Pekka Onnela coined the term “digital phenotype,”²⁴⁶ which refers to the use of mobile devices and other digital data sources to measure behavior and physiology for understanding brain activity that is relevant to pathological states. These techniques utilize measurement paradigms from translational neuroscience, which were developed in laboratory settings such as direct quantification of motor (i.e., movement, muscle activation) and physiological activity (i.e., heart rate, electro-dermal response) more than traditional clinical scales or self-report scales. The advantage of this approach is that it better aligns with emerging knowledge of rapid-acting biological processes and provides high measurement accuracy through direct rapid sampling, which is in contrast to traditional clinical measures that are taken sporadically over a long period.^164,247,248 These measurement approaches have relevance in multiple areas including treatment development, treatment selection, and ongoing monitoring.

Medications that target mental health conditions have a significant history of failure. Most psychiatric medications were discovered capriciously rather than being developed based on knowledge of the underlying biological mechanisms. As new medications emerge from basic and translational neuroscience research, both drug developers and clinicians struggle with how to measure the effects of new treatments and how to properly target old treatments. For example, traditional anti-depressant medications are designed to slowly titrate serotonin levels, resulting in slow global effects over a 2- to 4-week period. Correspondingly, measures of depression based on the DSM, query about the presence of depressive states over a 2-week period. New classes of anti-depressants such as ketamine and psilocybinpsylocibin affect specific depressive symptoms in minutes. Further, the mechanistic effects, and thus the need for measurement, is much more specific and granular. In fact, most classes of anti-depressants including serotonin reuptake inhibitors (SSRIs) and psilocybinpsylocibin, and ketamine, act on serotonin receptors that ultimately impact peripheral motor and physiological activity.^249,250 Serotonin regulation will likely have a direct effect on depression symptoms such as psychomotor retardation, but the direct effect on feelings of guilt is minimal. As such, methods used to directly measure motor output have a higher likelihood of capturing both pathology and treatment effects.

As an example, research effort has been dedicated to using computer vision and voice to directly quantify motor activity. Some recent work has demonstrated that digital phenotyping parameters that reflect gross motor activity including speech characteristics (rate of speech, tone) and facial/head movements are associated with suicidal risk,¹⁸⁹ SSRI response in MDD,²⁵¹ negative symptomatology in schizophrenia,¹⁹¹ and Parkinsonian tremor.²⁵² Such approaches are now being commercialized for all phases of drug development from proof of concept to direct measurement in order to make decisions about ongoing treatment needs. Such measures solve many of the current problems in clinical measurement since they can be captured remotely in an automated way. These measures can also be captured at a much higher frequency and provide a sensitive numeric value.

Meanwhile, these new approaches to measurement have significant challenges. First, methods that are adapted from the laboratory often lack the tight experimental control necessary to interpret the data correctly. For example, a rapid change in physiological responses (such as adrenaline and cortisol, or HRV) can indicate stress but also exercise or other forms of exertion. Second, while the scientific basis of these measurement paradigms may be sound, commercial approaches are rarely validated to the extent required to be of clinical utility, or they are rarely sufficiently transparent in their approach to be used for regulatory approval.

Digital interventions

The other rapidly emerging area of mental health technology are digital approaches to clinical care. We will briefly outline some of the leading approaches. Importantly, digital approaches to clinical care are often aligned with a digital measurement approach as these approaches are “blind” without some sort of remote data. A number of companies such as Mindstrong Health,²²⁷ IesoTrigger Health,²⁵³ and Headspace Health^254,255 have attempted to integrate digital phenotyping to identify when patients are in acute clinical need. However, it is unclear how accurate these methods are, as they are typically unpublished. This has led to the development of models that can identify patterns in patient’s and clinician’s language that are markers of improved outcomes;²⁵⁶ these can be further used to measure success of various therapy modalities and treatment design, as well as to improve care quality.²⁵⁷

Digital therapeutics proposes the use of mobile devices to offer CBT, mindfulness, or other validated psychotherapy in an automated fashion. These app-based approaches undergo the same clinical validation process as traditional medications and are often developed in collaboration with large drug developers. Examples that are in development or have received US Food and Drug Administration (FDA) approval include treatments for SUD, ADHD, schizophrenia, ASD, MDD, PTSD, and generalized anxiety disorder (GAD).²⁵⁸

While digital therapeutics attempt to scale treatment, telehealth aims to scale the treatment provider network. Mental health treatment is an area where there are many effective treatments but little access to treatment providers.²⁵⁹ The issue of access became most acute during the emergence of COVID-19, when significant wide-spread mental health needs emerged along with greatly decreased access to care. To address this need, a large array of options have emerged, many reinforced by the emergency COVID-19 Telehealth Act of 2021 that enabled remote patient care.²⁶⁰ These services, which are directly accessible to consumers or, more often, provided by a third-party payer, provide access to professionals based on different mobile platforms including text, voice, and video communication. Services vary from mental health coaching provided by certified professionals to mental health assessment provided by psychiatrists and psychologists. While still in their infancy, early evidence has shown that telehealth services can perform at parity with traditional in-person therapy.²⁶¹

Limitations

Although digital measurements have been rapidly adopted in mental health applications, most of current commercial AI applications for mental health have not focused on neuroimaging data. This is partially because of several reasons. First, the data acquisition, quality, privacy, and security issues have not been sufficiently or satisfactorily addressed (see discussion and conclusion), creating a gap between commercial and laboratory settings. Second, the scalability of real-time neuroimaging technology (such as EEG) is not yet quite ready. Third, the lack of an automatic and efficient (e.g., cloud-based) ML-powered neuroimaging data analysis pipeline also creates a barrier for brain-state monitoring. Future integration of portable EEG recordings and ML-powered platforms would expand the horizon of mobile mental health and teletherapy.

Multi-modal data fusion in diagnostic analytics

A central goal of precision psychiatry is to integrate all clinical, physiological, neuroimaging, and behavioral data to derive reliable individualized diagnosis and therapeutics. Importantly, health-related data are produced daily, especially from personal devices. The most essential effort in multi-modal data analysis tasks is to explore the relationship between modalities, complementarity, shared versus modality-specific information, and other mutual properties. Multi-modal data fusion techniques present a framework to infer information on how different data modalities interact and can be integrated for improved disease prediction.^38,262,263 In this section, we will review several data fusion methods in diagnostic analytics (popular ML methods for multi-modal fusion). We will focus on multi-modal neuroimaging data (multi-modal neuroimaging studies), and then extend the discussion to other modalities including vocal and visual expression data (multi-modal fusion of non-imaging data).

Popular ML methods for multi-modal fusion

In the past decades, numerous research efforts have been dedicated to developing powerful ML methods for multi-modal data fusion.^{38,264,265,266,267} Some commonly used approaches are summarized below (Figure 8).

Figure 8.

Summary of typical approaches for multi-modal data fusion in psychiatry studies

Multi-variate correlation analysis

CCA is a standard statistical method based on second-order statistics for data fusion. It aims at finding a pair of linear transformations to drive latent variables (also known as [aka] canonical variates) that have maximized correlation between two different data modalities.²⁶⁸ For a more general setting, multi-set/multi-way CCA (mCCA) has been developed as an extension of the standard CCA to multi-modal fusion by maximizing the overall correlation among latent variables from more than two sets of modalities.^38,269 Similar to CCA, PLS and its extensions, i.e., multiway PLS (N-PLS), provide alternative approaches to integrating multi-modal data by maximizing the covariance between latent variables from different modalities.^270,271

Matrix and tensor factorization

Based on matrix and tensor factorization techniques, joint blind source separation (BSS) approaches have been developed and successfully applied to multi-modal fusion of biomedical data.^265,272 As a typical example, joint ICA (jICA) aims to maximize the independence among jointly estimated components from multiple modalities that are assumed to share the same mixing matrix.²⁷³ The jICA approach involves concatenating modality features alongside each other and then performing ICA on the composite feature matrix.²⁷⁴ Independent vector analysis (IVA) is another extension of ICA to multiple datasets. IVA makes use of dependence across datasets by defining source component vectors concatenating a specific source estimated from multiple modalities.^264,265 Coupled matrix and tensor factorization (CMTF) was also developed to simultaneously factorize multiple datasets in the form of matrices and high-order tensors using tensor decomposition,²⁷⁵ showing strength in capturing the potential multi-linear structure for multi-modal fusion. Besides extracting shared common components, some multi-modal fusion tasks are also interested in deriving individual components that are modality specific. Common and individual feature analysis (CIFA)²⁷⁶ and joint and individual variation explained (JIVE)²⁷⁷ models have been proposed to achieve this goal. By jointly decomposing multiple feature matrices, CIFA and JIVE are able to simultaneously estimate common and individual feature subspaces. A further extension of CIFA has been achieved by leveraging high-order tensor factorization,²⁷² which provides an efficient way to perform a multi-dimensional fusion of multiple data modalities.

Multi-kernel learning

Multi-kernel learning (MKL) has won many successful applications in multi-modal data fusion due to the full utilization of multiple kernels that enable simultaneous learning from various modalities with heterogeneous data.^278,279 Different kernels naturally correspond to different modalities, such as neuroimaging, clinical, behavior, speech features, etc., which may provide complementary information to drive improved modal learning performance. The MKL problem can be set as a linear combination of kernel matrices or a non-linear function with specified forms of regularization. MKL may be designed under different ML models including SVM, GP, and clustering. Among them, MKL-SVM has been most popularly applied to integrate heterogeneous data modalities in studies of mental health.^280,281,282

Deep learning-based fusion

Cutting-edge deep-learning techniques have become increasingly popular for deep multi-modal fusion.²⁸³ Data fusion through deep learning allows integration of multiple modalities based on learned high-level feature representations that are theoretically more comparable with each other and more informative for predicting the targets.²⁸⁴ By exploiting cross-modal manifolds as a feature graph, a deep manifold-regularized learning model was recently designed to integrate transcriptomics and electrophysiology data from neuronal cells, and yielded promising performance for phenotype prediction.²⁸⁵ GNNs show capability in information fusion for multi-modal causability by defining casual links between features with graph structures, thereby enhancing the explainability of the derived multi-modal feature representation.²⁸⁶ By extending GNNs to multi-modal structures, deep-representation approaches have also been designed for integrating brain networks constructed from diverse modalities.^287,288,289

Multi-modal neuroimaging studies

Neuroimaging data types are intrinsically dissimilar in nature, having different spatial and temporal resolutions.²⁶⁶ Instead of feeding the entire dataset into a combined analysis, an alternate approach is to reduce each modality to low-dimensional (latent) features of selected brain activity or structure and then explore associations across these feature sets through variations across individuals. Exploiting such latent feature representations from multiple neuroimaging modalities for diagnosis has generally been shown to improve performance compared with using a single modality alone.²⁹⁰ Multi-modal fusion allows for integration of neuroimaging data modalities from different scales of spatial and temporal resolutions. Combining multi-modal neuroimaging offers an elegant way to exploit complementary information for more accurate and robust characterization of brain dysfunctions and hence is instrumental in optimal decisions for diagnosis, prognosis, and treatment in psychiatry.

A combination of mCCA and jICA was successfully applied to fMRI and DTI fusion in the diagnosis of schizophrenia and bipolar disorder.²⁹¹ CMTF has been applied to identify diagnostic biomarkers of schizophrenia by integrating sMRI, fMRI, and EEG.²⁹² MKL-SVM has been successfully applied to integrate multi-modal structural neuroimaging for predicting differential diagnosis between bipolar and unipolar depression²⁹³ and to combine sMRI and fMRI for improved classification of trauma survivors with and without PTSD.²⁹⁴ More recently, it also showed efficacy in the diagnosis of early adolescent ADHD by integrating sMRI, fMRI, and DTI.²⁹⁵ In learning low-dimensional representations of fMRI and sMRI,²⁹⁶ the fMRI can be split into several independent component networks, each treated as a separate modality along with the structural scan for learning using autoencoders. Furthermore, MKL methods have been used for diagnosing schizophrenia by combining markers from MRI and DTI.²⁹⁷ A multi-modal GCN was designed to integrate functional and structural connectomics data for an improved prediction of phenotypic characterizations in ASD.²⁸⁷ By combining multiple typical neural network structures, multi-modal deep-learning models have also been developed to effectively integrate fMRI connectivity and sMRI features²⁹⁸ and also genomic data²⁹⁹ for discovering schizophrenia-associated brain dysfunction. Methods for learning joint representations from neuroimaging and non-neuroimaging data are still in early development,³⁰⁰ and there is an opportunity for ML methods to evolve for this task. For example, transformer networks with late fusion can be used to learn joint representations from various modalities such as EEG and eye movement signals.³⁰¹

Multi-modal fusion of non-imaging data

Multi-modal approaches consist of combining data from various sources to jointly arrive at an answer. Given how little is conclusively known about which type of data, neuroimaging, social media, speech, video, and sensor data carry the most phenotypes for mental illnesses, naturally it makes sense to combine the information from these data sources. In addition, this enables modeling the inter-dependencies between these data that may not be observable by a human expert at the same time. For example, many features listed in Table 4 are known to be relevant in depression; however, observing them simultaneously can be very challenging for a clinician. MKL and MTL methods have been used to jointly learn from sensor and smartphone usage data to predict subjective well-being.³⁰² The success of transformer networks in jointly modeling video, speech, and language data has catalyzed multi-modal modeling in mental health.³⁰³ Multi-modal modeling techniques can also be used in characterizing symptoms such as emotion dysregulation,³⁰⁴ loneliness,³⁰⁵ and sentiment analysis.³⁰⁶ In a prognostic study, an SVM-based multi-modal ML approach was developed to integrate clinical, neurocognitive, neuroimaging, and genetic information to predict psychosis in patients with clinical high-risk states.⁸⁷ Deep autoencoder-based fusion approaches have been designed to integrate dynamics of facial and head movement and vocalization and successfully applied to the prediction of depression severity.³⁰⁷

ML for molecular phenotyping in psychiatry

Molecular phenotyping is referred to as the technique of quantifying pathway reporter genes (i.e., pre-selected genes that are modulated specifically by metabolic and signaling pathways) in order to infer activity of these pathways. Mapping genes and genomics to behaviors can identify risk factors and biomarkers in mental disorders. The brain is the central organ exposed to stressors and external behavioral interventions and is therefore vulnerable to changes in multiple interacting biological networks at the systems level. ML methods may play an active role in capturing the complexities of interacting variables within and across multiple levels (Figure 9A). For instance, at the molecular level, ML may help identify mechanistic-based phenotyping models as new targets for prevention and treatment of mood and cognitive disorders. The advent of unbiased next-generation sequencing (NGS) has prompted the development of bioinformatics and ML tools to profile and decode large molecular datasets (e.g., transcriptomics, epigenomics, metabolomics) at the genome-wide level in health and disease states (Figure 9B). To date, increasing applications of ML methods have integrated these multi-level molecular datasets with clinical characteristics to map specific neurobiological substrates into the complexity of symptom clusters, which may further help the classification of diseases, prediction of treatment outcomes, and selection of personalized treatment.

Figure 9.

ML for molecular phenotyping in psychiatry

(A) Different levels of interacting variables (genes, cells, circuits) to behaviors in mental illnesses.

(B) Combining ML with novel molecular biology technologies (for deep molecular phenotyping of brain plasticity) creates opportunities to develop new mechanistic models for prevention and treatment of clinical endophenotypes of mood and cognitive disorders.

Animal models have been playing a vital role in precision psychiatry for understanding disease mechanisms and predicting treatment responses.^308,309 Gene expression studies that integrate neuroscience, ML, and bioinformatics approaches can contribute to advancing understanding of the molecular basis of MDD and bridge the knowledge gap between animals and humans. Using RNA sequencing assays and gene coexpression network analyses (based on hierarchical clustering to identify gene modules), differential gene expression profiles have been shown across six key brain regions in post-mortem tissues of patients with MDD compared with age- and sex-matched controls, along with remarkable sex differences in these molecular pathways.³¹⁰ Recent work using RNA sequencing assays at single-nucleus resolution (snRNA-seq) and t-distributed stochastic neighbor embedding (t-SNE) analyses showed cell-type-specific transcriptomic profiles in the post-mortem dorsolateral prefrontal cortex (PFC) that are differentially regulated in MDD cases.³¹¹ Importantly, these gene expression studies in humans were supported by findings in rodents showing a brain that continually changes with experience.³¹² Several studies based on RNA-seq assays and bioinformatic analyses have showed striking transcriptomic differences in the ventral and dorsal hippocampus in the responses to stress—a primary risk factor for multiple psychiatric diseases—with the ventral hippocampus being sensitive not only to the effects of stress³¹³ but also a target for the responses to next-generation anti-depressants.^314,315

The expansion of NGS to single-cell resolution assays provides opportunities for advanced bioinformatics and ML approaches to analyze large datasets, which include denoising and dimensionality reduction, cell-type classification, gene regulatory network inference, and multi-modal data integration.^316,317 For instance, the software toolkit for single-cell genomics, Seurat (https://satijalab.org/seurat/), combines unsupervised non-linear dimensionality reduction, K-nearest neighbor graph analysis for cell-type clustering, and weighted nearest-neighbor analysis for multi-modal data integration.³¹⁸ Deep-learning approaches, such as the deep autoencoder, provide analytic tools for denoising and dimensionality reduction.^319,320,321 Autoencoders can also be used in a supervised manner for transfer learning across datasets, such as to learn the embedding from a larger previously annotated dataset and to transfer this knowledge to cluster new datasets.³²² The combination of multi-modal data generated from the simultaneous assessment of transcriptomic profiles with regulatory landscape or spatial location in the same single cell^{323,324,325,326} will allow a deeper molecular characterization of discrete cellular states.³¹⁸

Integration of multidimensional factors for new mechanistic treatment models

It has been increasingly recognized that mood and cognitive disorders are unlikely to be only contributed by the brain. Instead, growing evidence has suggested that they are system-level disorders affecting multiple interacting biological pathways,³²⁷ involving dynamic cross-talk between the brain and the body. Using hierarchical clustering to integrate in vivo molecular measures of brain metabolism with clinical symptoms in patients with MDD, recent work has showed that the specific neurobiological substrates map into discrete clinical symptoms, including anhedonia.³²⁸ Furthermore, the integration of multi-dimensional factors spanning mitochondrial metabolism, cellular aging, metabolic function, and childhood trauma may provide more detailed signatures than individual factors to predict longitudinal changes in depression severity in response to the metabolic agents used as anti-depressant treatment.³²⁹ Furthermore, deployment of multi-omics approaches and a random forest classifier has achieved 85% sensitivity and 77% specificity in prediction of the PTSD status. This system-level diagnostic panel of multiple molecular and physiological measures outperformed separate panels composed of each individual data type, showing certain mitochondrial metabolites as the most important predictors.^330,331

Another example of ML applications includes the integration of multi-dimensional phenotypic measures to identify those mechanisms that pre-dispose apparently healthy individuals to develop maladaptive coping strategies from those that confer resilience. A recent study has used a high-throughput unbiased automated phenotyping platform to collects more than 2,000 behavioral features and applied supervised ML to minimize Bayesian misclassification probability. The results have demonstrated that such a rich set of behavioral alterations can distinguish susceptible versus resilient phenotypes after exposure to social defeat stress (SDS) in rodents.^332,333 Furthermore, the ML classifier may integrate a priori constructs (such as the measures of anxiety and immune system function) and predict whether a given animal developed SDS-induced social withdrawal or remained resilient. Additionally, integration of features can improve the classification sensitivity (∼80%), which is better than the performance derived from either individual measure alone.³³⁴

The development of personalized psychiatry strategies for diagnosis and treatment will benefit from meeting the demand enforced by the recent advent of molecular biology protocols, which may provide opportunities to capture CNS nanovesicles (known as exosomes) and examine specific neurobiological substrates (e.g., transcriptomic profiles). ML-based dynamic network analyses will also enable us to link brain molecular targets and signaling pathways with other levels of analyses and to incorporate the brain-body relationship to redefine thinking about the mechanisms throughout the complex disease course.

XAI in psychiatry

XAI aims to provide strong predictive values along with a mechanistic understanding of AI by combining ML techniques with effective explanatory techniques. XAI has found emergent applications in medicine, finance, economy, security, and defense.^335,336 In psychiatry, XAI can help clarify the link between neural circuits and behavior and improve our understanding of therapeutic strategies to enhance cognitive, affective, and social functions.^337,338 XAI distinguishes the standard AI in two important ways: (1) it promotes transparency, interpretability, and generalizability and (2) transforms classical “black box” ML models into “glass box” models while achieving comparable or improved performance. From the diagnosis or prognosis perspective, it is crucial to know whether the ML solutions are explainable to the point of providing mechanistic insights into the way brains execute a particular function or complex behaviors. For instance, an ML-produced classification function to predict a disease outcome would need to not only report a probability outcome but also address additional questions for the end user: why is it this outcome instead of the alternative? How reliable is the outcome? When will it fail if something is missing or misrepresented? When and why is the prediction wrong? Accordingly, a model with high interpretability is often accompanied with parameter/structure/connectivity constraints or some prior domain knowledge. These explainable models can be continuously adapted such that an iterative process may be required to force ML methods to fit models with specific interpretations.

Interpretability and interpretable ML models

A model is interpretable if its outcome and operation that leads to the outcome can be understood by human users. In terms of taxonomy, intrinsic interpretability of ML models is attributed to their simple structures, such as short decision trees or sparse linear models (Figure 10A). Post-hoc interpretability is referred to the application of interpretation methods after model training³³⁹ (online resource: https://christophm.github.io/interpretable-ml-book/). Interpretation may appear in different forms: (1) finite feature summary statistics, (2) meaningful model parameters, or (3) easy visualization of the model outcome (e.g., feature summary or decision boundary). Interpretability and explainability are two similar concepts and are sometimes used interchangeably. Interpretability often, but not always, implies explainability. A model is highly explainable if it shares at least one of the following properties of explainability:³³⁹ high portability (regarding the range where the model can be applied), high expressive power (regarding the model strength in explaining the outcome), low translucency (regarding the model dependency on specific conditions), low algorithmic complexity, and informative constraints. Generally, there is a trade-off between model explainablity and performance. For instance, a constrained linear or bilinear model may fit many of these criteria, but the linear model does not warrant a good performance. Additionally, a model that is potentially explainable does not guarantee explainability. For example, codependence of input variables may make explanations ambiguous; latent variables of probabilistic generative models may face the problem of “explaining away.”³⁴⁰ Here, we briefly mention several classes of interpretable ML models.

Figure 10.

ML model interpretability and closed-loop brain-behavior intervention

(A) A wide spectrum of interpretability in representative ML models.

(B) Schematic of the closed loop of neuroimaging/modeling/neurostimulation.

Hybrid rule-based ML models

This type of ML models can be used for generating rules, such as a decision rule set: IF (condition) THEN (outcome 1) ELSE (outcome 2) statement, where the conditional clause will be learned from data.³⁴¹ This type of model has more expressive power but less portability.

Constrained ML models

This type of ML models imposes parameter constraints to avoid overfitting and enhance interpretability. Examples of such include the constrained convolutional filters in the CNN model³⁴² or constrained mixture models used for clustering.³⁴³ As a result, these constrained models have low translucency.

Feedback ML models

ML models can be provided with user feedback in the human-in-the-loop system, where the user feedback is treated as a constraint in the optimization problem.^344,345 The feedback may appear as a form of rule sets that are either known or unknown in advance. Feedback can also help interpolate missing data and constrain the solution. Iterating feedback-rule optimization steps can generate more accurate rule sets. This type of model has good expressive power and high portability.

Circuit-level modeling for computational psychiatry

Rooted in ML, computational psychiatry shares a similar goal with XAI and tries to combine multiple levels and types of computation with behavioral and neuroimaging data in an effort to improve understanding, prediction, and treatment of mental illness.³⁴⁶ The levels of computation range from short to long timescales (min/h/days/weeks) and focus on the changes in brain activity and behavior. The types of computation include circuit-level modeling, data-driven analytics, and theory-driven algorithmic development. Two complementary approaches have been proposed in computational psychiatry: (1) data-driven approaches, which apply ML methods to high-dimensional multi-modal data to tackle classification and prediction problems (how can ML help psychiatry?), and (2) theory-driven approaches (such as reinforcement learning), which develop algorithmic or mechanistic models to test hypotheses.

In the second approach, an important research topic is circuit-level computational modeling of macroscopic or mesoscopic brain dynamics for mentally ill brains in task or resting-state conditions.^347,348,349 A common strategy is to first use a biologically inspired model to simulate neural activity based on a network of interacting neural masses and next, within each brain area, to model the neuronal population activity as the Wilson-Cowan neural mass model, with each consisting of excitatory and inhibitory populations.³⁵⁰ Furthermore, individual brain nodes are coupled together according to the empirically derived anatomical network.³⁵¹ The computational model can be driven by an empirical hypothesis or EEG/fMRI data.

One data-driven macroscopic-level modeling approach is dynamic causal modeling (DCM). DCM has been widely used in characterizing the effective connectivity of a functional network based on task or rs-fMRI,^352,353 where the model parameters are inferred from unsupervised learning. By incorporating prior knowledge or hypotheses of network connections, DCM may reveal important brain mechanisms and offer experimental predictions. One potential application of DCM is to characterize the neural plasticity in human brains, especially the change in functional connectivity informed by neuroimaging studies. The functional connectivity can either change gradually during the course of tasks or be induced by neurostimulation. These changes are often, but not always, associated with changes in functional activation of specific brain regions.

Closing the loop for testing causality through neurostimulation

From the treatment perspective, it is critical to obtain an improved understanding of brain dynamics that are responsible for dysfunctional cognitive functions and maladaptive behaviors in mental illnesses. To find the hidden cause, the concept of causality requires special attention in perturbing the brain activity. Because of the complexity of the human brain and brain-behavior relationship, adaptive closed-loop neurostimulation provides a critical step to understand healthy and diseased brains.^354,355

Neuroimaging provides a passive sensing approach to observe the (correlational) brain-behavior relationship. However, correlation is different from causation. Correlational dependencies describe associations of measurements that experiments do not control, whereas causal dependencies link a dependent variable to an experimentally controlled variable. The key concept in causal inference is to introduce randomization to perturb the mapping. The relationship between every dependable variable and the randomized variable is causal, whereas the relationship between non-randomized variables and behavior remains correlational.³⁵⁶ A closed-loop experimental design would help to test the potential causality.³⁵⁴ In human experiments, we classify closed-loop testing into two categories: one being fully automated, and the other being closed human in the loop.

One big challenge in human psychiatric neuroscience is the causality gap.³⁵⁷ Statistical causality or Granger causality between two variables is not equivalent to brain-behavioral causality. To identify an effective treatment strategy for mental illnesses, it is critical to causally modulate neural circuitry that is responsible for maladaptive behaviors. Human neuroimaging alone only demonstrates correlations but not causation. To understand the causal mechanisms, it is imperative to close the loop in experiments by perturbing the brain circuits and measuring its outcome, as commonly done in animal experiments.^354,356 Unfortunately, a rigorous and causal grounding of clinical symptoms and behavior is still missing. Since the clinical symptoms are diverse, how to define the dimension of brain function that defines one or few clinical symptoms and how to effectively manipulate them remains unknown. Furthermore, closed-loop neurostimulation has a conceptual link to XAI for mental health studies and can be seen as an extension in the design of human brain-machine interface (BMI) to causally test brain-behavior mapping.³⁵⁸

Neuromodulation therapies have become increasingly popular in treating various neuropsychiatric and neurological disorders. Temporally precise neurostimulation tools provide a plausible means to perturb or stimulate the brain. Clinically used neuromodulation methods include invasive deep brain stimulation (DBS), non-invasive transcranial magnetic stimulation (TMS), non-invasive transcranial direct/alternating current stimulation (tDCS/tACS), and transcranial focused ultrasound stimulation (tFUS). A review of advances in neuromodulation technologies for treating mental disorders can be found in the literature.^359,360,361 To date, repetitive TMS (rTMS) has been cleared by the FDA for the treatment of depression and has recently been used in studies of neural functioning and behavior.^362,363 Along this research line, ML may potentially help addressing three important questions (where/when/how) to achieve precision neuromodulation in psychiatry.

For the where question, depending on the neuromodualtion techniques, delivery of target-specific stimulation requires active and scheduled stimulation strategies to identify behaviorally activated targets. In the case of depression, neurostimulation can have multiple potential targets or modes of action,³⁶⁴ but how to identify the optimal target to achieve effective treatment outcomes remains unexplored. For the when and how questions, compared with open-loop stimulation, closed-loop stimulation can deliver temporally precise stimulation triggered by detected features, symptoms, or user demand. Traditional neurostimulation strategies are designed in an on/off stimulation fashion, where the stimulation is determined by pre-selected parameters. However, these stimulation parameters may not be optimal. To accommodate an adaptive subject-specific stimulation strategy, adaptive stimulation uses neurofeedback to adjust the stimulation parameters or control policy to achieve various optimality criteria. Therefore, ML can play a guiding role in online adaptive stimulation.^365,366,367 For instance, the feedback loop can analyze the neural signal’s oscillatory patterns or other reliably detectable biosignals (e.g., biochemcial, electromyographic, and mechanical signals) to classify or detect the critical brain state for delivery of closed-loop neurostimulation.³⁶⁸ Additionally, reinforcement learning can be applied to learn a state-action value function to identify the best excitability brain state, where the state corresponds to the neural activity (e.g., the amplitude of evoked potentials, characteristics of brain connectivity) and the action corresponds to on/off stimulation mode.^367,369

Neurostimulation can not only induce changes in behavior but can also induce plasticity in brain connectivity. Simultaneous or post-neurostimulation neuroimaging provides a window of examining the change in brain network connectivity patterns. Brain connectivity and dynamics can be studied from a network communication and control perspective.^370,371 The distinction between a healthy and a pathological brain can be characterized by their different efficiency to route information between distributed brain nodes, to control or modulate the target node under specific constraints, or to influence its behavior for performing specific tasks (“cognitive control”).³⁷² Therefore, the well-established network and graph theories can be used to study the change in brain connectivity by a controller (neurostimulator). Specifically, the control-theoretic models have also been applied to quantify the response of brain networks to exogenous and endogenous perturbations. Several important research questions can be studied along this line: (1) can a target node stimulation rewire brain connectivity in evoked and steady-state conditions? (2) Can the neurostimulation-induced change of evoked or resting-state brain connectivity distinguish a pathological from a healthy brain? (3) Given a controller’s energy constraint, what is the optimal neurostimulation policy? Will alternate or simultaneous neurostimulations at multiple sites influence the network connectivity more effectively or bring additional benefit in treatment³⁷³? (4) Can the induced brain patterns or changes in network connectivity predict the treatment outcome? ML may address these questions by providing individualized treatment-response likelihood in precision psychiatry.³⁷⁴

Finally, we suggest that combining multiple efforts of XAI and neurostimulation in the loop (“neuroimaging $\to$ circuit modeling $\to$ neurostimulation $\to$ observing behaviors $\to$ revising models”; Figure 10B) will provide an effective pathway toward a better understanding of brain-behavior causation as well as individualized precision treatment in psychiatry. For instance, looping neuromodulation and DCM may provide a way to test the impact of neurostimulation on neural plasticity that underlies the change in adaptive or maladaptive behaviors.

Discussion and conclusion

Challenge and opportunities

The past few decades have witnessed growing interests and rapid developments in ML methods for precision psychiatry. However, caution was also raised regarding the unrealistic hope for ML applications in clinical practice,^182,375 and the field is still facing both conceptual and practical challenges.

At the conceptual level, first, the term disorder was used to specifically avoid the term disease, implying that precise mechanistic understanding is still missing in psychiatry; this further makes it very difficult to build clinical inference models for mental disorders. As a result, it is still infeasible to develop treatments that target underlying physiological risk factors in a similar manner as other medical disciplines (e.g., treating hypertension in heart diseases). Furthermore, each mental disorder has overlapping symptoms with varying degrees, bringing difficulties to uniquely define the psychiatric disorder. Second, many disorders are presented as a spectrum (e.g., ASD, generalized anxiety spectrum, and schizophrenia spectrum) and vary across different patients, creating a wide range of subtypes and subject variability within the same type of mental disorder. Third, due to various genetic, biochemical, and neuropathological factors, the same mental disorder may have different causes and symptoms in different age/gender/race populations. Fourth, overlapping symptoms can be found in many mental disorders, making the diagnosis less precise or more error prone.³⁷⁶ For example, changes in sleep and energy level, often found in depression and generally measured using the PHQ-9 questionnaire, are very common across many other disorders. One goal in precision psychiatry is to fully dissect the mechanisms and causally reveal the many-to-one relationship. This can be catalyzed by rigorous measurements and quantification of neural and behavioral data relevant to mental health.

At the practical level, many challenges also remain in effective applications of ML for mental health.

Sample size

Datasets used in many ML applications have a small sample size, especially by the standard of ML-based speech/image/video applications. Neuroimaging data collection from mental health patients is limited to one-shot examples, which creates large data variability in addition to the intrinsic heterogeneity and disease comorbidity. Recent developments in foundation models and their mental health applications may help overcome this challenge—for example, by sharing the pre-trained language model.²²³ However, further caution is needed to ensure appropriate validation methods on the problem-specific data. Reproducibility is the main bottleneck to biomarker discovery for any mental disorder.

Data quality

The lack of standardization in data acquisition and varying degrees of data quality present a challenge in rigor and reproducibility. For example, studies using social media data often rely on the mental disorder labels based on users’ self-identification instead of rigorous clinical diagnosis. This can lead to a post containing “I am depressed” being labeled as a depression sample regardless of the underlying clinical symptoms. Terms such as depressed or anxious have colloquial uses that can differ from clinical criteria, leading to inaccurately labeled samples. Furthermore, collection of properly matched control samples remains difficult given the diversity of age, gender, race, education, family history, and lifestyle among users. Furthermore, there are also intrinsic dataset biases during data collection.³⁷⁷

Data privacy and security

Advances in sensing technologies enable us to collect a large amount of personal data, including location, face images, speech conversations, and social interactions. However, how to store and process these data without the leakage risk of privacy information remains an important challenge. While research studies have rules and regulations (e.g., internal review board) to ensure ethical use, social media data are collected on a massive scale by companies. Due to the lack of sufficient regulation, such data have not been treated as personally identifiable information (PII) that can be used to inform the user’s health, creating a major obstacle in securing identifiable user data. To help alleviate this issue, regulations in the United States such as the Health Insurance Portability and Accountability Act (HIPAA) can be used to govern PII acquired by all social media companies and commercial entities.

Social implications and environmental factors

Gender and race are critical factors in mental health. According to WHO, mental disorders have a long history of gender bias. In terms of the gender risk factor, females are more likely to suffer from depression and anxiety, whereas there is more prevalence of autism in males. In terms of gender treatment bias, women are more likely to be diagnosed with depression compared with men, and women are more likely to be prescribed with mood-altering psychotropic drugs. ML may play a role in uncovering the gender or race risk factor and minimize the diagnosis or treatment bias related to these social factors.

Generalizability

The standard ML generalization issue becomes even more pronounced in mental health applications, especially due to the poor data quality and small sample size. Most ML studies use cross-validation to report the performance but lack independent validation datasets to assess generalizability. Furthermore, very few studies test generalization across data sources and experimental conditions. For example, it is important to test how well ML models trained from speech data from clinical interviews will perform on non-clinical speech data.

Algorithmic bias

Digital mental health inherits a long history of bias in psychiatry, which can be found at all stages of a patient journey.³⁷⁸ In addition to biological underpinnings, the domains of data (such as language) also represent social underpinnings,³⁷⁹ and therefore it is important to consider how socioeconomic factors may influence measurements. Using training and validation sets that are representative across all demographics can not only help address some of these issues but also uncover new symptom expressions in various groups. This is even more important for ML approaches that inherit biases from other ML models.

Interpretability

The ability to understand which latent factors contribute most to the outcome is the key for advancing clinical understanding of mental disorders for mental health professionals as well as for establishing the trust for the users of mental health technology. This is also an important dimension to improve precision in mental health. The choice of the interpretation method,³³⁸ either model specific (such as analyzing attention weights of a transformer) or model agnostic (such as local interpretable model-agnostic explanations [LIME]), is very specific to the nature of the problem. While various interpretation methods can be used to identify model functioning, it is important to note that the interpretation results can only be trusted as long as the issues of generalizability and data quality are addressed. In other words, model interpretation methods may produce erratic results with insufficient or poor-quality data. Furthermore, there is always an “explainability-performance” trade-off. For instance, deep learning may outperform traditional methods at the cost of reduced interpretability. Despite the rapid progress in the development of XAI and interpretable ML techniques, the challenge of ML explainability still remains in psychiatric applications, especially when dealing with data of mixed modality, data of high dimensionality, and data measured at varying timescales.

Causal inference

Most ML applications in mental health have focused on integrating information from multiple data sources and reaching a diagnosis decision faster. However, diagnosing a mental disorder, even with a highly interpretable model, neither speaks to the underlying causes nor has limited implications on treating the causes. ML-based causal inference methods³⁸⁰ may help with precision treatment design.³⁸¹ Recent developments in ultra-high-field neuroimaging with sufficient temporal and spatial resolution³⁸² may provide a means for developing inference models for mental disorders.

Clinical integration

It is important to consider the clinical need from a user experience perspective, varying from the mental health professionals using the application³⁸³ to patients who are mentally ill.³⁸⁴ Part of this work, such as conducting user research in various demographics, lies outside of the ML domain; however, such cross-functional research can inform the best practice in ML model development. This type of thinking with the end goal in mind is important for successful translation of precision psychiatry research to widespread clinical practice.³⁸⁵ Additionally, considering a possible disparity of interests between the clinical and ML communities,³⁷⁷ cross-disciplinary dialogues and collaborations between two communities would help the deployment of ML solutions in clinical practice.

Ethical considerations

ML applications in mental health also raise important ethical considerations. For example, ML models for risk assessment can lead to early screening that may help with early treatment.¹¹ However, when screening techniques are available outside clinical settings, it can create the risk of misinterpretation by patients, which may negatively affect treatment-seeking behavior or trigger self-harming thoughts in patients. Other ethical questions related to increasing the risk of self-harm arise inherently from ML that use foundation models like Generative Pre-trained Transformer 3 (GPT-3), which should be fully considered before deployment in clinical settings.³⁸⁶

Applications of new ML technologies

In addition to the opportunities arising from addressing the above-mentioned challenges, precision psychiatry is accompanied by plenty of opportunities in future ML applications.

Data-centric approach

In the data-driven ML view (“ML system = model/algorithm + data”), data are powerful. However, medical data are costly to collect and noisy. Currently, there is an ML paradigm shift from model centric to data centric (https://datacentricai.org/), which advocates using good “small” data instead of simply collecting from big, but possibly noisy, data. The good-quality criteria include (1) consistency, (2) coverage of important cases, and (3) inclusion of timely feedback from user or production data. Unlike the model-centric ML approach that focuses on modifying the model/algorithm (while fixing the data) to improve the performance, a data-centric ML approach involves building ML systems with quality data, with a goal to systematically process the data (while fixing the model) to improve the ML performance.³⁸⁷ The modification of the available data may include data regeneration, data augmentation, and label refinement strategies to improve data consistency. For instance, confident learning has been proposed to estimate label uncertainty and identify label errors, based on the principles of pruning noisy data, counting with probabilistic thresholds to estimate noise, and ranking examples to train with confidence.³⁸⁸ The iterative process of two approaches can bootstrap the system performance.

Data augmentation approach

To deal with the small sample size issue in patient data collection, one ML approach is to create synthetic data (as a data augmentation strategy) or increase the number of training instances.³⁸⁹ Deep-learning methods such as GAN and its variants have served as a powerful tool to generate synthetic brain scan images, speech, video, physiological data, and EHRs.^390,391,392 However, unlike traditional ML/computer vision studies, the value of synthetic training samples remains unclear in psychiatric applications. Whether ML-augmented samples can generate clinically meaningful and diversified samples that match psychiatric heterogeneity would require future in-depth investigations.

Automated-learning approach

In contrast to the human-in-the-loop solutions, automated ML (autoML) and automated deep learning (autoDL) represent a new paradigm that aims to automate the data analysis pipeline while minimizing the need of human intervention during the course of modeling and training.³⁹³ This has become increasingly important since the volume of social media and multi-media data streams is so overwhelming that even a small effort of human involvement would make the task prohibitive.

Data-integration approach

Integration of multi-modal data is critical for psychiatric diagnostics and monitoring. Therefore, it is urgently needed to develop weakly supervised, interpretable, multi-modal deep-learning pipelines to fuse histopathology, genomics, neuroimaging, and behavioral data, as well as to develop multi-modal fusion algorithms for speech, video, and EHRs, to assist both psychiatrists and patients. Because of the nature of multi-modality, not all data can be quantified in the Euclidean space. Graph and geometric deep learning may play a role in this research direction.^394,395 Finally, expert-augmented ML (EAML) methods that automatically extract problem-specific human expert knowledge and integrate it with ML to build robust, dependable, and data-efficient predictive models will also have great potentials for psychiatric applications.³⁹⁶

Conclusion

To date, there is still a lack of biomarkers and individualized treatment guidelines for mental illnesses. In this review, we have shown that ML technologies and data analytics can be used for various stages of a patient journey: detection/diagnosis, prognosis, treatment selection/optimization, outcome monitoring/tracking, and relapse prevention. We predict that the multi-modal integration of neuroimaging, ML, genetics, behavioral neuroscience, and mobile health will open doors for new method developments and technology inventions. First, making brain scans more accessible will be the key to clinical applications of neuroimaging techniques. Using real-time fMRI, ML can guide neurofeedback-based intervention and provide closed-loop treatment or rehabilitation. As a “real-time mirror” of psychiatry, mind-control intervention can improve behavioral outcomes. Second, data-driven ML methods can identify subtypes of symptoms and cognitive deficits and develop model-based phenotyping.³⁹⁷ Third, combination of ML methods with large EHR databases may accommodate a personalized psychiatry. Fourth, when developing ML-powered technologies for psychiatry, it is imperative to consider concerns and feedback from various stakeholders, including knowledgeable experts (clinical and ML experts, technology or engineer experts), decision-makers (hospital administrators, institutional leaders, state and federal governments), and end users (physicians, nurses, patients, friends, and family).³⁹⁸ Finally, an integration of medications, wearable devices, mobile health apps, social support, and online education will be essential to improve mental health and assist therapeutic outcomes in the new era of digital psychiatry. Future precision psychiatry will leverage ML and all technologies to provide individualized custom packages that are built upon the patient’s needs and specific pathology.

Biography

Zhe Sage Chen is an associate professor at the New York University School of Medicine, with joint appointment at the Departments of Psychiatry, Neuroscience, and Physiology and the Department of Biomedical Engineering at the NYU Tandon School of Engineering. Prof. Chen is the codirector of Computational Psychiatry Program at NYU and the director of the Computational Neuroscience, Neuroengineering, and Neuropsychiatry Laboratory. His research interests include computational neuroscience, computational psychiatry, neural engineering, brain-machine interface, and machine learning, with more than 120 peer-reviewed articles in these areas. More details about his lab can be found on his website: http://cn3laboratory.org/.

Prathamesh (Param) Kulkarni received a PhD degree in electrical engineering from the University of Houston. He is currently leading a machine-learning team at Headspace Health, Inc. Dr. Kulkarni is an expert in the digital health space with a specialization in AI and behavioral health. He has successfully built digital health companies from the ground up. Previously, as the cofounder of an innovative behavioral health company AwareHealth, he led the company from inception to product-market fit leading product, sales, and tech.

Isaac R. Galatzer-Levy received a PhD degree in Clinical Psychology from Columbia University. He completed post-doctoral training in neurobiology of stress and bioinformatics at NYU School of Medicine. He served on the research faculty at NYU School of Medicine Department of Psychiatry from 2015 to 2019. Subsequently, Dr. Galatzer-Levy led clinical and algorithm development across multiple successful startups in the AI-driven digital health space. He is currently working at the Meta Reality Lab. Dr. Galatzer-Levy has published more than 100 articles in the fields of neuroscience, psychiatry, psychology, medicine, and bioinformatics.

Benedetta Bigio is a Computational Engineer and Research Assistant Professor at NYU Grossman School of Medicine. Her work utilizes integrative computational and bioinformatic approaches applied to rodent and human models of main diseases, including mood and cognitive disorders. Using genome-wide assays and bioinformatic and machine-learning algorithms, her work identified novel molecular networks of mitochondrial metabolism in resilience versus susceptibility to stress.

Carla Nasca is an assistant professor at NYU Grossman School of Medicine. Her research focuses on developing an innovative model of epigenetic mechanisms of neuroplasticity to stress with a new angle on the communication between mitochondria and cell nuclei. Her laboratory also uses computational approaches together with emerging technologies to isolate patient-derived exosomes to identify in vivo neural mechanisms of main CNS diseases (mood and cognitive disorders) and potential biomarkers of responses to novel anti-depressant treatment and transmagnetic brain stimulation.

Yu Zhang is currently an assistant professor of bioengineering at Lehigh University. He was a post-doctoral research fellow at the Wu Tsai Neurosciences Institute and the Department of Psychiatry and Behavioral Sciences at Stanford University Medical School and in the Biomedical Research Imaging Center at the University of North Carolina, Chapel Hill. He also worked as an assistant professor at the School of Information Science and Engineering at East China University of Science and Technology, China. His research interests include computational neuroscience, pattern recognition, machine learning, signal processing, brain-computer interface, and medical imaging computing.