The promise of machine learning in predicting treatment outcomes in psychiatry

Abstract

For many years, psychiatrists have tried to understand factors involved in response to medications or psychotherapies, in order to personalize their treatment choices. There is now a broad and growing interest in the idea that we can develop models to personalize treatment decisions using new statistical approaches from the field of machine learning and applying them to larger volumes of data. In this pursuit, there has been a paradigm shift away from experimental studies to confirm or refute specific hypotheses towards a focus on the overall explanatory power of a predictive model when tested on new, unseen datasets. In this paper, we review key studies using machine learning to predict treatment outcomes in psychiatry, ranging from medications and psychotherapies to digital interventions and neurobiological treatments. Next, we focus on some new sources of data that are being used for the development of predictive models based on machine learning, such as electronic health records, smartphone and social media data, and on the potential utility of data from genetics, electrophysiology, neuroimaging and cognitive testing. Finally, we discuss how far the field has come towards implementing prediction tools in real‐world clinical practice. Relatively few retrospective studies to‐date include appropriate external validation procedures, and there are even fewer prospective studies testing the clinical feasibility and effectiveness of predictive models. Applications of machine learning in psychiatry face some of the same ethical challenges posed by these techniques in other areas of medicine or computer science, which we discuss here. In short, machine learning is a nascent but important approach to improve the effectiveness of mental health care, and several prospective clinical studies suggest that it may be working already.

Treatment interventions in psychiatry are far from being effective in all cases in which they are indicated. In depression, for example, only 30‐50% of individuals achieve remission after whatever initial treatment they receive, even in the context of a well‐conducted clinical trial ¹ . Eventually, after trying some number or combination of treatments, most patients do attain remission. What if, rather than iterating through the available treatments that a patient might benefit from, we could predict the right treatment for each individual from the start?

Researchers have wanted this for decades. Historically, they have tried to understand specific factors involved in treatment response based on theoretical groundings, leading to many studies focusing on single variables such as early childhood stress, suicidality, major life events, or comorbid diagnoses. Since then, the ongoing search for one (or a few) true explanatory variables has included many levels of analysis, including: the patient (clinical characteristics, blood marker levels), his/her brain (structural and functional neuroimaging, cerebral blood flow, scalp electrical recordings), his/her genes (single nucleotide polymorphisms, mutations/rare genetic variants, copy number variations, gene expression), and intervention characteristics (the medication or psychotherapy selected, the way it was delivered, the provider, the therapeutic alliance). If one variable alone could accurately predict treatment response, our field would probably have found it by now. Instead, most characteristics identified so far have shown small explanatory power over treatment outcomes, and researchers' attention naturally turned towards multivariable models that can incorporate many smaller effects.

Machine learning is a collection of statistical tools and approaches that are extremely well suited to this goal of detecting and aggregating small effects in order to predict an outcome of interest ² . It allows researchers to go from evaluating a small number (~10) of predictor variables to many hundreds or thousands of variables or variable combinations. There are many potential pitfalls when applying these techniques, but, when implemented well, they afford many opportunities for psychiatric research3, 4. They allow us to examine many variables, even correlated ones, simultaneously. They move away from exclusively additive models and allow us to identify more complex non‐linear patterns in data. They more naturally bridge disparate data types, potentially incorporating clinical assessments, geospatial information, and biological findings into a single analysis. By unlocking powerful hypothesis‐free approaches, they enable us to discover factors that are less intuitive but nonetheless predictive of outcomes.

The introduction of machine learning in psychiatry is more than just adding an analysis tool for combining and exploring bigger data sets – it marks a paradigm shift ⁵ . For years, we used classical statistical approaches to confirm or refute specific hypotheses. Now, machine learning studies shift the focus toward the overall predictive power of a model, particularly how ­accurately it predicts the desired outcome in a new, unseen dataset. Studies in this field are evaluated primarily by their potential clinical impact: what our model can reliably tell us about the prognosis of new patients in the future, and what we can do with that information to improve clinical practice.

With this in mind, this paper explores the promise of machine learning in predicting treatment outcomes in psychiatry. There are many things that we do not focus on. This is not a primer on machine learning ⁶ , an explanation of how it works ² , or a debate about what counts as machine learning versus traditional statistics or “non‐machine‐learning”. We do not explain how to build predictive models ⁷ or how to validate them. We are not formally comparing different algorithmic approaches, how each one works, or circumstances where one may be more appropriate than another. We also avoid a distinction between moderators versus mediators of treatment outcomes, or whether a model predicts outcomes specifically for a treatment versus others or predicts outcomes more generically for multiple treatments ⁸ . Finally, we do not aim to review the many sociodemographic and clinical variables that have been or can be used for prediction of treatment response in psychiatry, which generally have the most predictive power and are cheapest to collect9, 10.

We begin by discussing machine learning methods, how they compare to traditional statistical approaches, and to what extent is machine learning specifically adding value. Next, we provide an overview of the interventions for which researchers have tried to use machine learning methods to predict outcomes, ranging from medications and psychotherapy to digital interventions and neurobiological treatments. In doing so, we highlight characteristics that made them gold standard examples, and discuss the different goals that can be achieved in each context. Next, we focus on the potential utility of electronic health records, smartphone and social media data, and of data from genetics, electrophysiology, neuroimaging and cognitive testing for the development of predictive models based on machine learning. Finally, we help the reader understand the broader context: how close have we come to implementing these prediction tools in real‐world clinical practice; and what are the ethical challenges that these tools carry. The intent of this paper is to review studies throughout psychiatry; any emphasis on depression is not intentional, but it does reflect the fact that the majority of research in this field has been conducted in people with that mental disorder.

IS MACHINE LEARNING ADDING VALUE OVER TRADITIONAL STATISTICS?

Machine learning studies generally differ from traditional research in two ways. The first is a focus on prediction (explanatory power of the model) rather than inference (hypothesis testing). The second is a shift towards model flexibility, with the ability to handle large numbers of predictors simultaneously.

Prediction can be performed without machine learning algorithms, and many studies still use traditional statistical techniques such as logistic regression. In fact, when assumptions and sample size requirements are reasonably met, the number of predictors is small (≤25), and non‐linear effects are relatively weak, traditional parametric models will likely predict well. Several studies found no benefit of machine learning over traditional logistic regression, for example in predicting treatment resistance in major depression ¹¹ , brain injury outcomes ¹² , or major chronic diseases ¹³ .

One recent systematic review of clinical prediction models found no difference in performance between machine learning and logistic regression ¹⁴ , although the authors considered in the category of logistic regression some advanced frameworks that could be included within machine learning, such as penalization (e.g., lasso, ridge or elastic net) and splines (which capture non‐linearities). In areas of medicine such as diabetes and heart failure, simple logistic models have performed well and have been externally validated more than machine learning models15, 16.

The added value of machine learning approaches emerges when the number of potential predictors is large and/or their effects are non‐linear. Many machine learning algorithms are capable of handling large numbers of predictors, even in cases where there are more predictor variables than observations, due to built‐in overfitting control. For example, ridge, lasso and elastic net regression ¹⁷ include penalization, which forces the regression coefficients to be closer to zero than in the traditional linear or logistic regression models. Machine learning approaches are also good at capturing complex, interactive, or non‐linear effects. For example, tree‐based models are able to evaluate many possible variables and variable combinations to identify subgroups that could not be captured by traditional linear models. Another common technique adopted by machine learning approaches is “ensembling”. Here, several models are fitted on random samples of the original dataset, and then an average is taken amongst the predictions from each model. This approach is a key element of many popular machine learning techniques today, especially gradient boosting machines and random forests18, 19, 20.

Several recent treatment outcome prediction studies in psychiatry demonstrated the added value of machine learning. Random forests and/or elastic net regression21, 22, 23, 24, as well as support vector machines ²⁵ , were found to outperform traditional regression methods. Large‐scale comparisons on benchmark datasets consistently found machine learning to outperform traditional methods26, 27, 28, 29. Overall, boosted trees (random forests and gradient boosting machines), regularized regression, support vector machines, and artificial neural networks can all perform well, but no one method will have the best performance across all situations.

While researchers typically aim to maximize predictive performance, practical aspects such as explainability or the cost of including more variables should also be considered. In some cases, simpler models with slightly lower predictive accuracy or higher generalizability might be preferred, because they already capture most of the effects30, 31. There is no silver bullet in statistics, and all prediction algorithms face the so‐called bias‐variance tradeoff2, 32, 33, where flexibility needs to be balanced with the risk of overfitting. For machine learning methods to capture increasingly complex effects, much larger sample sizes are still needed. Although these methods can deal with large numbers of potential predictor variables, careful pre‐selection of variables likely improves predictive accuracy.

While traditional research approaches focused on p values for specific coefficients in a model, prediction studies focus on the overall explanatory power of the model, often in terms of R ² , balanced accuracy, or area under the receiver operating characteristic curve (AUC). Predictive studies require a keen focus on validation approaches, to examine whether the model is learning patterns that are substantive and consistent from one dataset to another, or whether the model has simply learned idiosyncrasies of the initial training data. Table 1 discusses various kinds of validation that are conducted in predictive studies, from internal approaches that use just one dataset, to external validation approaches that use data from independent sites, studies, trials, countries, or consortia to test model generalizability. Validation frameworks, especially external validation, are critical for developing models that are reliable and useful, and understanding whether the fitted model is likely to generalize to unseen data in the future34, 35, 36.

Table 1.

Common validation approaches used in clinical prediction studies

Generalizability test	Description
None, p value testing	The entire sample is used to predict an outcome, and a p value indicates the probability of obtaining the result in the absence of a true effect. The study cannot make any claims concerning translation or generalizability because they have not been tested.
Leave‐one‐out cross‐validation	One subject is randomly chosen and left out. A model is trained on the remaining subjects and applied to the left‐out subject to assess generalizability. This procedure is repeated for every subject in the dataset. This is the simplest form of cross‐validation. It produces optimistic biased results.
K‐fold cross‐validation	The sample is randomly divided into subsamples (called “folds”). One fold is left out and statistical models are trained on the ­remaining subjects. The models are applied to the subjects in the left‐out fold to assess generalizability. This is a common technique to reduce overfitting. However, when the data are from one sample (even if collected at multiple sites), generalizability claims need to be ­tempered.
Leave‐one‐site‐out cross‐validation	Instead of randomly leaving out subjects, sites are now randomly left out. Models are fitted on the remaining sites, and applied to the left‐out site. This assesses cross‐site generalizability, and the same technique can be extended to any other group definition, such as blocks of time, gender or ethnicity. Generalizability and translational claims still need to be tempered.
External validation	A model is created in one study and applied to a completely separate sample. This approach reflects a high degree of generalizability capacity. Demonstrations can be increasingly close to real‐life circumstances, which strengthens the evidence of generalizability and translational potential (but does not guarantee it). The approach may still be subject to poor sociodemographic representation, ­sampling biases, or study designs that do not reflect clinical reality.
Prospective validation	A previously‐created model is evaluated in a prospective study that is ideally randomized and in conditions as close to clinical reality as possible, in order to test whether the tool is safe and effective in practice. Prospective validation studies are still susceptible to the same concerns around external validity as all other clinical trials (e.g., participant compensation and meaningful endpoints), and require large sample sizes, a broad and unbiased recruitment process, and good clinical practices. As with other clinical trials, a phased process may be necessary to first evaluate feasibility and safety in a smaller sample before proceeding to broad evaluation of effectiveness.

PREDICTING TREATMENT OUTCOMES IN PSYCHIATRY BY USE OF MACHINE LEARNING

Medications

Predicting treatment outcomes for psychiatric medications is the most active area of research in the field, primarily because they were the easiest place to start. Machine learning studies require large volumes of data to build predictive models, ideally with clearly labelled outcomes, control over the intervention, and relevant data about the patients before treatment. Since this describes most large clinical trials, and most large clinical trials in psychiatry are conducted to evaluate efficacy of a medication, most machine learning efforts began by investigating treatment responses to medications treating depression, schizophrenia or bipolar disorder.

These studies mostly used information from demographic intake forms and clinical symptom scales common in clinical trials, although more recently genetic and neuroimaging data have also been incorporated (discussed later in this paper). Despite being the most active area of research, most resulting models have not yet been validated in external samples. Relatively few prediction tools generated by mental health researchers so far have advanced through implementation studies and into clinical practice37, 38, 39. Here we focus on examples of studies that were adequately powered, underwent external validation, or are notable for other reasons.

Most treatment prediction studies have focused on antidepressants commonly used in the acute phase of depression. For example, Chekroud et al ⁴⁰ determined a small group of 25 pre‐treatment variables that were most predictive of remission with citalopram in the Sequenced Alternative Treatments for Depression (STAR*D) trial. This model achieved an accuracy of 64.6%. The model was then applied to data from another clinical trial to examine whether it can generalize to patients from an entirely independent population. The model was able to predict response to two similar antidepressant regimens (escitalopram plus placebo, and escitalopram plus bupropion, each with an accuracy of around 60%), but the model did not predict remission better than chance for patients who took venlafaxine plus mirtazapine (51%).

The five most important variables identified by the model in predicting remission were baseline depression severity, employment status, feeling restless during the past seven days (psychomotor agitation), reduced energy level during the past seven days, and Black or African American ethnicity. The study was later replicated by Nie et al ⁴¹ , who similarly trained a model to predict citalopram treatment outcomes using information easily obtainable at baseline. The team trained and tested the model in the STAR*D dataset and validated it in data from a different open‐label citalopram trial, using 22 predictor variables that overlapped between the two trials. Despite minor differences depending on the specific algorithm used, the balanced accuracy of the models was roughly 64‐67%.

An earlier study by Perlis ¹¹ showed that eventual treatment resistance might also be predictable from the outset. The author developed a model using STAR*D data that was able to predict at baseline whether an individual would not reach remission after two antidepressant treatment trials, with an AUC of 0.71. Early proofs of concept like the Perlis study did not include external validation, at least partly due to the lack of independent datasets with similar trial designs that could be used for that validation.

The above antidepressant studies selected predictors in a purely data‐driven way, including all data that could be extracted at baseline and then using machine learning methods that discard irrelevant information or are amenable to including many variables at once. However, the choice of predictors is not always hypothesis‐free, and a priori knowledge from scientific literature can also guide the choice of variables and yield useful results. Iniesta et al ⁴² aimed to predict remission of depression in patients treated with escitalopram or nortriptyline using only variables that had previously been confirmed as individual predictors or moderators of response to treatment. Their models predicted overall response to medication with an AUC of 0.74 and response to escitalopram with an AUC of 0.75, but prediction of nortriptyline outcomes was not statistically significant. In subsequent work incorporating genetic data to the models ⁴³ , these authors predicted response to escitalopram and nortriptyline with an AUC of 0.77.

A second use of machine learning to predict medication outcomes is to better define subgroups of patients, symptoms, or symptom trajectories, and then use these subgroups to make more nuanced predictions. Drysdale et al ⁴⁴ used clustering to identify four “subtypes”, or groups, amongst 1,188 depressed patients based on patterns of dysfunctional connectivity in limbic and frontostriatal networks. They developed classifiers for each depressive subtype using support vector machines and later tested these models on an independent dataset, accurately classifying 86.2% of the testing sample. As a next step, the team used the subtypes to predict response to transcranial magnetic stimulation, but did not validate these predictions in any independent sample. Although the biotypes approach is interesting, subsequent methodological research has highlighted concerns and limitations ⁴⁵ .

Chekroud et al ⁴⁶ used clustering to identify groups of symptoms and mixed‐effects regression to determine if they had different response trajectories. Three symptom clusters (core emotional, sleep and atypical) emerged consistently from two independent medication trials – STAR*D and Combining Medications to Enhance Depression Outcomes (COMED) – across two commonly used symptom scales. The authors subsequently used data from STAR*D to train gradient boosting machines (one for each combination of cluster and medication arm), finding modest improvements in the ability of clusters of symptoms to predict total severity outcomes. The same symptom clustering approach was also effective in a study of treatments for adolescents ⁴⁷ .

Other researchers first used techniques like growth mixture modeling ⁴⁸ or finite mixture modeling ⁴⁹ to identify trajectories of symptom response such as “fast and stable remitter”, “sustained response”, or “late relapse”. Machine learning models were then developed to try and predict the specific response trajectory a patient will have for a given treatment. This approach is potentially more robust to the noise that is naturally present amongst individual patient trajectories and less affected by the way that outcomes are defined in trials – e.g., whether remission is defined as a score of 5 on the Patient Health Questionnaire‐9 (PHQ‐9) or a score of 5 or 6 on the Quick Inventory of Depressive Symptomatology (QIDS)48, 49. However, the approach relies on the availability of repeated measures.

Medication treatment outcomes have been most widely studied in depression, due to the prevalence of the condition and extant available data, but the approach has also been proven in other psychiatric conditions. For schizophrenia, Koutsouleris et al ²⁵ used data from the European First Episode Schizophrenia Trial (EUFEST, N=344) to predict good and bad outcomes based on global functioning scores over time using a support vector machine, and validated the ten most predictive features on an unseen sample of 108 patients with a balanced accuracy of 71.7%. The most valuable predictors identified were largely psychosocial variables, rather than symptom data: unemployment, poor education, functional deficits, and unmet psychosocial needs.

Again in schizophrenia, Leighton et al ⁵⁰ were not only successful in predicting response to medication treatment in first episode psychosis, but also in validating findings in two independent samples. They first identified predictors that were available across three studies – the Evaluating the Development and Impact of Early Intervention Services (EDEN) study in England, two cohorts recruited from the National Health Service (NHS) in Scotland, and the Danish clinical trial called OPUS. This allowed them to build and test harmonized models across the three studies to predict four outcomes capturing different aspects of recovery: symptom remission, social recovery, vocational recovery and quality of life. Next, they used logistic regression with elastic net regularization to identify the most relevant predictors in the EDEN study (N=1027) – much like Chekroud et al ⁴⁰ – to determine a smaller subset of variables that could still predict outcomes but require less effort for future data collection and improve clinical applicability. These regularized models trained in the EDEN sample reached internal validation AUCs of 0.70 to 0.74 (depending on the outcome measure). When tested in the second Scottish cohort, the AUC ranged from 0.68 to 0.87. In the OPUS trial, it ranged from 0.57 to 0.68.

Predicting medication response in other mental disorders is still in early stages. Two studies51, 52 used baseline sociodemographic, clinical and family history information to predict response to medications commonly used in bipolar disorder: lithium and quetiapine. Although both obtained models with performance above chance, neither was validated in independent samples, and one used 180 variables for prediction ⁵¹ , which limits its clinical applicability.

Psychotherapies

Historically, efforts to predict treatment outcomes in psychotherapies have focused on theoretically‐motivated single variables that might moderate treatment outcomes. Only relatively recently have psychotherapy researchers applied machine learning approaches to predict treatment outcomes ⁵³ . Even amongst these studies, the historical focus on moderators of psychotherapeutic effects has persisted, leading researchers to distinguish between “prognostic” and “prescriptive” models. Prognostic models are those that predict whether a patient will recover with a given treatment. Prescriptive models instead predict which of two (or more) treatments is best suited for a particular patient ⁵⁴ . Both kinds of model can clearly have clinical utility, even if they answer slightly different questions. The differences continue to blur further with more recent attempts to build prescriptive models by developing multiple prognostic models for different treatments and then comparing their outputs ⁵⁵ .

In an early effort, Lutz et al ⁵³ used nearest neighbor modeling to predict rate of symptom change and session‐by‐session variability. Models were based on age, gender and baseline symptom scores. Compared to non‐machine learning models, the nearest neighbor predictions were more highly correlated with actual values of rate of change, but not session‐by‐session variability.

Since then, other approaches to prediction in psychotherapy proliferated. DeRubeis et al ⁵⁶ developed a multivariable modeling method, known as the “personalized advantage index” (PAI), that uses interaction effects between baseline variables and treatment condition, to predict whether a patient will respond better to antidepressants versus cognitive behavioral therapy (CBT). Amongst their small sample of 154 individuals, a clinically meaningful advantage (PAI ≥3), favoring one of the treatments relative to the other, was predicted for 60% of the patients. When these patients were divided into those randomly assigned to their “optimal” treatment versus those assigned to their “non‐optimal” treatment, outcomes in the former group were better (d = 0.58, 95% CI: 0.17‐1.01). Similar approaches have been developed by other groups55, 58, and more recently improved further by the use of machine learning approaches ⁵⁹ to generate better predictions and incorporate more variables.

Several studies since then have tried to predict which evidence‐based psychotherapy is most likely to benefit a specific patient55, 59, including efforts to identify which of two (or more) psychotherapies may be most effective60, 61, and whether a given patient is predicted to respond better to psychotherapy or medications ⁵⁶ . A recent scoping review ⁶² identified a total of 44 studies that developed and tested a machine learning model in psychotherapy, but only seven of them reported on the feasibility of the tool. Since psychotherapy trials are often expensive and rarely have large sample sizes, some have argued that predictive models may need to be developed initially with large observational datasets ⁶³ .

PAI‐style approaches that calculate treatment by variable interactions quickly lead to high‐dimensionality prediction analyses that are prone to overfitting (or require very large sample sizes). Using data from two Dutch randomized trials, van Bronswijk et al ⁶⁰ examined whether PAI models developed in one clinical trial dataset were able to successfully generalize to an independent dataset. Although the models performed statistically above chance in the trial used to train them, they did not generalize to the other clinical trial when predicting benefit for CBT versus interpersonal therapy (IPT) for depression.

The psychotherapy literature has generated several other pre­diction models, potentially optimizing significant aspects of ­patient care. For example, models have been developed64, 65 that would enable mental health providers to select low‐ or high‐intensity treatments for patients on the basis of their expected prognosis. Other studies have tried to deconstruct the content that is traditionally combined to form a course of psychotherapy treatment, in order to predict which treatment components should be delivered within a given intervention, as well as the order in which the components should be implemented66, 67, 68. Other novel directions include using machine learning to match patients to specific therapists ⁶⁹ , replicating human ratings and judgements70, 71, and using natural language processing techniques to discover patterns of therapist‐patient interactions that predict treatment response72, 73.

In general, many machine learning approaches to predict responses to psychotherapies are in the early stages of development ⁶² . However, a notable exception is found in the well‐developed literature on routine outcome monitoring and “progress feedback”. This involves tracking a patient's response to treatment in real time by entering his/her self‐reported outcome/symptom measures into a computerized system that compares his/her response to predicted trajectories of improvement derived from clinical data using conventional statistical analyses (e.g., longitudinal multilevel/mixed models and growth curve modelling). There are now over 20 randomized controlled trials and several meta‐analyses indicating that such clinical prediction models can help to improve treatment outcomes ⁷⁴ .

In addition to models investigating differential response to treatment and treatment optimization, the psychotherapy literature also includes adequately powered studies predicting overall response to treatment based on sociodemographic and clinical variables, much like the literature on response to medication. Buckman et al ⁷⁵ built nine different models, using depression and anxiety symptoms, social support, alcohol use, and life events to predict depressive symptom response after 3‐4 months of treatment in primary care settings. Models were trained on data from three clinical trials (N=1,722) and tested on three independent trials (N=1,136). All models predicted remission better than a null model using only one post‐baseline depression severity measurement. Green et al ⁷⁶ also predicted depressive symptom response to psychotherapy in 4,393 patients from community health services. They found that a model with only five pre‐treatment variables (initial severity of anxiety and depression, ethnicity, deprivation and gender) predicted reduction of anxiety and depression symptoms with an accuracy of 74.9%. The number of sessions attended/missed was also an important factor affecting treatment response.

Digital CBT

In recent years, online delivery of mental health interventions has been seen as a promising approach to reducing barriers to care, with growing evidence for the effectiveness of both guided and unguided delivery77, 78. Interventions such as internet‐based CBT (iCBT) may be particularly amenable to the use of machine learning techniques, due to the possibility of longitudinal standardized collection of outcome data at scale, and the potential to directly incorporate machine learning outputs into online or app‐based interventions. For example, in guided treatments, machine learning tools could provide feedback to therapists or alerts regarding risk. They could also be used to drive just‐in‐time adaptive interventions ⁷⁹ . Smartphone delivery also opens up the possibility of automated collection of sensor data to derive behavioral markers ⁸⁰ , which would open up many possibilities for tailored interventions, while also raising a number of privacy and ethical concerns.

Machine learning‐derived outcome predictions for iCBT may have advantages with regard to ease of deployment, for example by providing integrated decision support for case management. However, most existing work focused on predicting outcomes has been exploratory in nature and based on modest sample sizes. A key distinction is between approaches that use only baseline pre‐treatment data, and hence may be applied to direct the choice of treatment, and approaches which use data gathered during the course of treatment, such as regular outcome measures or ecological momentary assessment (EMA).

As an example of the former, Lenhard et al ⁸¹ examined how clinical baseline variables can be used to predict post‐treatment outcomes for 61 adolescents in a trial of iCBT for obsessive‐compulsive disorder. Whereas multivariable logistic regression detected no significant predictors, the four machine learning algorithms investigated were able to predict treatment response with a 75 to 83% accuracy.

In a study which included, in addition to demographic and clinical data, therapy‐related predictors of treatment credibility and working alliance, assessed at week 2 of treatment, Flygare et al ⁸² used a random forest algorithm to predict remission from body dysmorphic disorder after iCBT in a sample of 88 patients, comparing the results to logistic regression. Random forests achieved a prediction accuracy of 78% at post‐treatment, with lower accuracy in subsequent follow‐ups. The most important predictors were depressive symptoms, treatment credibility, working alliance, and initial severity of the disorder.

van Breda et al ⁸³ added EMA data to models using baseline mea­sures in a study predicting outcomes for patients who were ran­domized to blended therapy (face‐to‐face CBT and iCBT) or treatment as usual. This approach did not improve prediction accuracy.

The effectiveness of digital CBT interventions is mediated by patient engagement ⁸⁴ . Detailed patient engagement data can be gathered automatically in online or app‐based interventions; this may include data such as content views, completion of exercises, and interactions with clinical supporters ⁸⁵ . Engagement data may be used within predictive models, providing interpretable and actionable outputs (e.g., the need for more frequent therapist contact in order to motivate greater engagement). Chien et al ⁸⁶ analyzed engagement data from 54,604 patients using a supported online intervention for depression and anxiety. A hidden Markov model was used to identify five engagement subtypes, based on patient interactions with sections of the intervention. Interestingly, while in general patients who engaged more achieved better outcomes, the best outcomes were found in those who were more likely to complete content belonging to key components of CBT (i.e., cognitive restructuring and behavioral activation) within the first two weeks on the program, despite not spending the highest amount of time using the intervention. This work demonstrates the feasibility of gathering detailed engagement and outcome data at scale.

Interactions between patient and therapist, and the content of text in patient exercises, may also be analyzed using sentiment analysis techniques ⁸⁷ . Analysis of patient texts might be embedded in therapist feedback tools for guided interventions, or as features within predictive models. Ewbank et al ⁷³ conducted an analysis of 90,934 session transcripts (specifically, CBT via real‐time text messages). Deep learning was used to automatically categorize utterances from the transcripts into feature categories related to CBT competences, and then multivariable logistic regression was applied to assess the association with treatment outcomes. A number of session features, such as “therapeutic praise”, were associated with greater odds of improvement.

Chikersal et al ⁸⁸ analyzed 234,735 messages sent from clinical supporters to clients within an iCBT platform, examining how support strategies correlate with clinical outcomes. They used k‐means clustering to identify supporters whose messages were linked with “high”, “medium” or “low” improvements in client outcomes, as measured by PHQ‐9 and Generalized Anxiety Disorder‐7 (GAD‐7). The messages of more successful supporters were more positively phrased, more encouraging, more often used first person plural pronouns, were less abstract, and referenced more social behaviors. Association rule mining was then applied to linguistic features in the messages in order to identify contexts in which particular support strategies were more effective. For less engaged patients, longer, more positive and more supportive messages were linked with better outcomes. For more engaged clients, messages with less negative words, less abstraction, and more references to social behaviors were associated to better outcomes. Such results could ultimately be used in the design of supporter training materials.

One could also try to predict whether a patient engages or drops out of care. Wallert et al ⁸⁹ aimed to predict adherence to an online intervention targeting symptoms of depression and anxiety in people who had experienced a myocardial infarction. The analysis included linguistic features of the homework texts as well as demographic and clinical characteristics. The strongest predictors of adherence were cardiac‐related fear, gender, and the number of words in the first homework assignment.

Neurobiological treatments

Numerous neurobiological options have emerged as potential treatments for severe and treatment‐resistant depression, such as transcranial magnetic stimulation (TMS) and electroconvulsive therapy (ECT). Given the potential risks and side effects of these treatments, as well as their higher financial costs, there is an especially strong interest in identifying for whom they are safe and effective90, 91, 92.

Recent reviews have examined predictors of treatment response and relapse among depressed patients receiving TMS92, 93, 94. TMS studies with more female patients tend to have higher effect sizes, suggesting that gender may be a predictor of TMS outcomes ⁹⁵ . Although several studies have attempted to examine neurobiological predictors of response to TMS, the findings are currently inconsistent ⁹² . Small sample size generally means that machine learning study designs are likely to overfit and produce results that will not replicate later.

Efforts to predict treatment outcomes for ECT are still primarily traditional association studies. Some of them identified a few variables that appear to replicate across studies. Better outcomes have been found for older patients, those with psychotic depression, those with high suicidal intent, and those who exhibit early symptom changes90, 96. However, due to the small sample size in most ECT trials, and the typically non‐randomized study designs, this area has not seen much progress. These are also obstacles to the application of machine learning techniques.

THE UTILITY OF ELECTRONIC HEALTH RECORDS, SMARTPHONE AND SOCIAL MEDIA DATA

Electronic health records (EHR) are increasingly widely adopted in health care systems. They comprise data routinely collected and maintained for individual patients over the course of their clinical care. As such, these data may be particularly useful for building predictive models in psychiatry that could be readily integrated into points of care within clinical settings ⁹⁷ . EHR data can be divided into two major types: coded structured data, including diagnostic codes, procedure codes, laboratory and medication prescription codes; and unstructured data, including clinical notes and other text‐based documentation, which can be mined using natural language processing.

Recent studies have tested the potential of EHR data to predict treatment outcomes in psychiatry, with the bulk of efforts to date focused on depression, though examples exist for bipolar disorder ⁹⁸ and schizophrenia ⁹⁹ . Machine learning‐based efforts using EHR data have sought to identify those individuals who are likely to drop out after initiating antidepressants ¹⁰⁰ , those who will show a stable treatment response to antidepressants ¹⁰¹ , and those who may transition to a bipolar diagnosis after starting antidepressants for depression ¹⁰² . Such applications have shown promising, though still modest and not yet clinically actionable, results.

Applying logistic regression and random forest approaches, Pradier et al ¹⁰² used demographic and structured EHR data (i.e., diagnostic, medication and procedure codes) available at the time of initial prescription to predict treatment dropout after initiating one of nine most common antidepressants. Although mean AUC was below 0.70, they found that incorporating EHR data significantly improved prediction of treatment dropout compared to demographic information alone, and that predictive performance varied by type of antidepressant (AUC as high as 0.80 for escitalopram) and provider type (higher accuracy among psychiatrist‐treated individuals).

Hughes et al ¹⁰¹ applied logistic regression and extremely randomized trees with demographic and structured EHR data to predict general and drug‐specific treatment continuity in patients receiving any of 11 antidepressants, observing a mean AUC of 0.63‐0.66 and similar performance when evaluated at a separate site.

Where symptom score (e.g., PHQ‐9) data have been available for smaller EHR cohorts (e.g., N<2,500) ¹⁰³ , LASSO models incorporating demographic information, structured and unstructured EHR data, and baseline symptom scores have shown modest‐to‐adequate performance in predicting improvements in depressive symptom severity, for both medication treatment (AUC=0.66) and psychotherapy (AUC=0.75). However, the most important predictor in these models was baseline symptom scores. Only when symptom scores are routinely integrated into EHR treatment workflows will such models be relevant for outcome prediction in large‐scale health systems.

When using EHR data for predicting treatment outcomes in psychiatry, a key challenge is how to operationalize the outcome of interest using available clinical information. This usually involves establishing a set of rules around which relevant EHR features are observed, or not observed, in a cohort of patients over a defined period. For example, treatment dropout was defined by Pradier et al ¹⁰⁰ as less than 90 days of prescription availability after index antidepressant initiation, with no evidence of alternative psychiatric treatment procedures. Antidepressant treatment stability, on the other hand, has been defined as two or more antidepressant medication prescription codes at least 30 days apart over a period of at least 90 days, with additional rules about the maximum time gap between adjacent prescription codes, and other medication possession indicators ¹⁰¹ .

EHR data are also highly dimensional, with tens of thousands of possible diagnostic codes in addition to possible medication and procedure codes. Machine learning methods may be particularly suitable for modeling complex signals across a diverse set of EHR‐based predictors, but also for reducing their dimensions prior to modeling. In their study of antidepressant treatment stability, Hughes et al ¹⁰¹ applied supervised topic modeling using latent Dirichlet allocation to reduce 9,256 coded EHR features into 10 interpretable empirically derived topics, finding that a classifier for continuous treatment based on this lower‐dimensional set of predictors showed comparable performance to a logistic regression based on a higher‐dimensional set of features. Simpler methods, such as selecting only diagnostic codes that meet a frequency threshold in the patient population, have also been used ¹⁰⁰ .

Smartphones can provide various kinds of data that are difficult to acquire through other means. Their first and biggest feature is that they contain many sensors that can passively collect data across a variety of domains. Passive smartphone data include dynamic measures of sleep quality, exercise, heart rate, geospatial locations, language use, and communication patterns80, 104. Machine learning methods are indispensable for dealing with complex patterns in these sensor data ¹⁰⁵ . Currently available studies applying machine learning to predict mental health outcomes using sensor data have generally employed modest samples of 7 to 70 participants, yielding proofs‐of‐principle more than generalizable results80, 106, 107, 108. Mobile phones also facilitate the collection of EMA data, allowing investigators to perform measurements at frequent intervals (e.g., several times a day). Furthermore, smartphone‐based neurocognitive assessments appear to be a promising way to scalably collect cognitive data109, 110.

Few studies have used smartphone data to predict treatment outcomes. These include studies using text data from emails to predict treatment response in patients with social anxiety ¹¹¹ , EMA data to predict changes in self‐esteem from an online intervention ¹¹² , and EMA data to predict treatment response in patients with depression ⁸³ . In the study predicting depression outcomes, a model including EMA data did not outperform a model using baseline characteristics ⁸³ , showing that the former data do not always provide incremental value.

Social media allow investigators to access large amounts of data relating to language use and online activity. However, to our knowledge, these data have not yet been used to predict treatment responses. One of the tradeoffs between incorporating different types of data is the cost and quantity versus quality of data: very often these data present with noise which may hinder the ability to identify meaningful patterns and signals. Novel methods of topological machine learning are robust to noise, and allow to extract descriptors of the shape and structure of data that can augment performance for the analysis of intensive time‐point measurements ¹¹³ . Such data with repeated measures may be useful for testing hypotheses, since sample size may compensate for the increased noise of data ¹¹⁴ .

THE USE OF DATA FROM GENETICS, ELECTROPHYSIOLOGY, NEUROIMAGING AND COGNITIVE TESTING

Genetics

Machine learning methods are an appealing analytical approach for bridging genetic data with the prediction of treatment response in psychiatry. They put the focus on prediction rather than association, are able to detect interactions between loci, wisely handle correlation, and do not assume a pre‐defined statistical model or additivity ¹¹⁵ .

Machine learning has been used with the objective to improve prediction of treatment outcomes from genetics alone in many diseases, including cancer116, 117 and hypertension ¹¹⁸ .

The question of whether an individual's genetic background could affect how he/she responds to medication treatment has been investigated in pharmacogenomics. An earlier study applying genome‐wide complex trait analysis in a sample of roughly 3,000 depressed patients suggested that common genetic variation could explain up to 42% of observed individual differences in antidepressant treatment response ¹¹⁹ , suggesting that modeling common genetic variation could be useful for prediction. However, results of pharmacogenomic studies have so far, in general, been underwhelming ¹²⁰ .

Polygenic scores are a common method for quantifying the overall contribution of common genetic variation to particular traits ¹²¹ . Polygenic associations with treatment response have been investigated in relatively small patient cohorts (most N<1000) to date, with mixed findings122, 123, 124, 125. For example, polygenic scores for major depression and schizophrenia did not significantly predict antidepressant efficacy (based on symptom improvement) in classic treatment studies such as Genome‐Based Therapeutic Drugs for Depression (GENDEP) and STAR*D ¹²³ . However, these scores were built on earlier genome wide association studies (GWAS) and were likely underpowered. Well‐powered GWAS of antidepressant response have produced mixed results, with one study identifying gene sets of relevance for bupropion response ¹²⁶ and another observing no significant findings for antidepressant resistance ¹²⁷ . Larger‐scale GWAS meta‐analysis efforts are needed and ongoing. Even fewer studies have examined common genetic variation associated with responses to other treatment modalities such as psychotherapy ¹²⁵ or ECT ¹²⁸ .

DNA methylation and gene expression data have been explored in combination with phenotypic datasets of demographic and clinical variables on their ability to predict response to multiple medications. A recent review ¹²⁹ pointed out genetic prediction of therapeutic outcomes in depression as the most promising43, 130, 131, 132, 133, with an overall accuracy of 0.82 (95% CI: 0.77‐0.87) ¹³⁴ . Models combining multiple data types, such as peripheral gene expression data, neuroimaging and clinical variables, achieved significantly higher accuracy ¹³⁴ .

Tree‐based approaches were the most popular machine learning methods, followed by penalized regression, support vector machines and deep learning ¹²⁹ . Studies were quite heterogeneous in design, methods, implementation and validation, limiting our capacity to elucidate the extent to which machine learning integrated with genetics can predict antidepressant drug response.

Evidence for polygenic risk scores versus support vector machines for the prediction of treatment‐resistant schizophrenia from GWAS data have been reviewed ¹³⁵ . Although support vector machines might be more suitable to take into account complex genetic interactions, the traditional polygenic risk score approach showed higher accuracy for classifying treatment‐resistant individuals ¹¹⁵ .

Despite many efforts to use many kinds of genetic information in many different ways, results so far have not been sufficiently compelling or accurate to support the use of these approaches to guide clinical care136, 137. In the future, until novel analytic techniques become available to extract signal from the genome, or a better understanding of the genetic basis for mental illness emerges, the most promising avenue in this context is to integrate genetic information into multivariable analyses to potentially improve broader model performance133, 137.

Electrophysiology and neuroimaging

Tailoring treatment decisions based on brain measures is intuitively appealing and empirically well‐justified. Systematic reviews and meta‐analyses indicate that therapeutic outcomes are often related to pre‐treatment brain differences and that the brain changes as a result of therapy138, 139, 140, 141, 142, 143, 144, 145. However, in previous research using traditional statistical methods, effect sizes were too low to make the jump from statistical significance to clinical relevance, external validation was rare, sample sizes were small, methodological and site‐related variance was high, and in many cases the techniques were not suited to an integration into clinical routine due to their cost‐benefit ratio (e.g., positron emission tomography) or reliance on experimental protocols that are unavailable in most clinical settings138, 139, 143, 145, 146. Machine learning approaches offer hope in overcoming these barriers to clinical implementation. Preliminary reviews comparing accuracies support this optimism by suggesting superiority for treatment prediction with respect to traditional statistical methods ¹³⁴ .

Early studies in this area applied machine learning to detect outcomes such as response to clozapine in psychosis ¹⁴⁷ and to selective serotonin reuptake inhibitors (SSRIs) in depression148, 149, 150, but the majority of research has focused on predicting brain stimulation outcomes for depression148, 151, 152, 153, 154, 155. For example, Corlier et al ¹⁵⁶ found that alpha spectral correlation could be used to measure EEG connectivity, which then predicted response to repetitive TMS (rTMS), using cross‐validated logistic regression, with an accuracy of 77% in a subgroup of depressed individuals. This increased to 81% when adding clinical symptoms of depression. Most studies report predictive accuracies of >80% on the basis of pilot samples consisting of approximately 50 cases or less ¹⁵⁵ , reflecting the strong likelihood of bias and overfitting that is also seen with magnetic resonance imaging (MRI) ¹⁵⁷ .

Task‐related functional MRI (fMRI) has been used for treatment prediction ¹⁵⁸ : for example, by modelling amygdala engagement interactions with early life stress during an experimental task to predict antidepressant outcome ¹⁵⁹ or by using fear conditioning responses to predict panic disorder treatment outcome160, 161. Similar task‐related predictive models have been built in a number of studies of CBT ¹⁶² or antidepressant responses162, 163, 164. In task‐based fMRI, however, the translational potential is limited due to the use of lengthy and methodologically complicated experimental paradigms. Resting‐state fMRI is a popular alternative, because it measures behaviourally‐relevant, synchronized brain network activity at rest, and the imaging protocols can be more easily harmonized across scanners ¹⁶⁵ . Studies in this field have demonstrated similar accuracies for CBT ¹⁶⁶ , trauma‐focused psychotherapy ¹⁶⁷ , antidepressant treatment ¹⁶⁸ , and antipsychotic therapy ¹⁶⁹ , while also showing predictive accuracy for ECT165, 170.

A challenge of functional imaging is reliability across scanners, especially in non‐experimental clinical settings. Structural neuroimaging may provide an opportunity for faster implementation into existing clinical routines. Most studies have involved grey matter measurements, and ECT treatment prediction has been a frequent focus, with studies using whole‐brain approaches ¹⁷¹ , regional measurements ¹⁷² , and combinations of neuroimaging modalities ¹⁷³ . White matter measurements (e.g., with diffusion tensor imaging) have been relatively less commonly considered.

Overall, the lack of multi‐site studies and external validation reflects the pilot‐study stage of research in this area, where results can be interpreted as promising but highly experimental. Whether the machine learning results will ultimately agree with the low effect sizes found with classical statistical approaches remains an open question143, 145.

Cognitive testing

Cognitive testing is a straightforward method to indirectly assess brain functioning that has been historically linked to treatment outcomes. Although such testing can be time‐consuming and costly when performed by a trained neuropsychologist, more recent computerized methods can facilitate efficient digital assessments that lend themselves especially well to machine learning, including from passively collecting smartphone mea­surements as described above80, 114, 174.

Etkin et al ¹⁷⁵ conducted an early study in this area, as part of the international Study to Predict Optimized Treatment in Depression (iSPOT‐D), aimed to predict response to antidepressant treatment using a battery of computerized cognitive tasks assessing attention, processing speed, memory, and executive and emotional functions. In order to obtain accurate predictive estimates, they first classified depressed individuals into a subgroup with particularly poor cognition before training a supervised discriminant function to predict remission. Results demonstrated that remission following escitalopram could be predicted with 72% accuracy, but this was not confirmed with sertraline or venlafaxine.

Subtyping or unsupervised learning approaches have also been helpful to identify response trajectories to cognitive training. A recent study found that self‐organizing maps detecting multivariate relationships between cognitive functions associated with working memory task performance could identify individuals who differentially responded to the training ¹⁷⁶ .

HOW CLOSE WE HAVE COME TO REAL‐WORLD IMPLEMENTATION

Not all prediction models will translate readily for use in clinical or other real‐world settings. In evaluating the readiness of predictive models for real‐world implementation, key criteria include external validation, empirical support from implementation trials, and acceptability to users (e.g., clinicians).

External cross‐validation remains the gold standard for evaluating real‐world performance, as it quantifies performance loss when a trained model is applied to a completely independent sample. In addition, it guards against increased researcher degrees‐of‐freedom that may result from the many tuning parameters of more complex machine learning methods. A review focusing on machine learning in psychotherapy research reported that only 3 of 51 studies had performed external validation ⁶² .

Studies without external validation are at high risk of overconfidence, as demonstrated by Van Bronswijk et al ⁶⁰ , who developed and then tested a treatment selection model across two randomized controlled trials comparing CBT and IPT. They found that the estimated effect size for the benefit of receiving the model‐recommended treatment (generated through internal cross‐validation) shrunk by 77% when the model was tested using the second study's data (external validation).

Some prediction efforts using large naturalistic samples have reported positive results following external validation65, 177, 178.

When a model undergoes external validation and successfully predicts outcomes, the next step towards real‐world use is an implementation trial. These trials provide the most compelling evidence for the value of a decision support tool. Here, patients are usually allocated to algorithm‐guided treatment (generally within a shared decision‐making framework) or treatment as usual.

Trial‐based efforts to evaluate the efficacy of treatment personalization tools have begun to emerge. One example is a multi‐service cluster randomized trial ¹⁷⁹ , in which patients (N=951) were referred to either high‐ or low‐intensity psychotherapy. In one arm, the choice of intensity was informed by an algorithm previously developed in a naturalistic dataset. In the other arm, most patients started on low‐intensity psychotherapy and were later referred to high‐intensity treatment in the case of non‐response, as per usual stepped care. The study found higher depression remission rates in patients whose initial treatment was recommended by the algorithm compared to usual stepped care (52.3% vs. 45.1%, odds ratio, OR=1.40, p=0.025).

Another recent example comes from Lutz et al ¹⁸⁰ , who used archival data from an outpatient CBT clinic to develop a predictive decision support system providing therapists with treatment strategy recommendations and psychometric feedback enhanced with clinical problem‐solving tools. They randomized therapist‐patient dyads (N=538) to treatment as usual or to algorithm‐informed treatment. They reported that, overall, outcomes for those who were randomized to the intervention did not differ from those who received usual care. However, there was significant variability in the extent to which therapists in the intervention condition followed the recommendations provided by the decision support tool. When the authors analyzed outcomes for patients whose therapists had followed the recommendations, significant benefits emerged.

Browning et al ¹⁸¹ conducted another trial randomizing depressed patients to either algorithm‐informed care or usual care for depression. Their algorithm, called PReDicT, used information from symptom scales and behavioral tests of affective cognition to predict non‐response to treatment with citalopram. After eight weeks of treatment, the rate of depressive symptom response in the PReDicT arm was 55.9%, versus 51.8% in the usual care arm (not significant, OR=1.18, p=0.25). Of all instances where the algorithm predicted non‐response, only 65% prompted a change in treatment regimen, and most consisted of an increase in dosage only.

In combination, the above findings highlight that accurate algorithms are not enough to ensure the success of a decision support system for precision treatment ³⁹ . When randomizing patients to algorithm‐informed care or usual care, clinicians may override algorithm recommendations and choose alternative treatments. Patients may refuse the algorithm‐recommended treatment, or have restrictions to its use that were not contemplated by the decision support tool (e.g., prohibitive cost of therapy). In light of this, effect sizes for these interventions will often vary when applied in different settings.

The use of predictive models may be uniquely challenging in psychotherapy research and practice. One challenge is that a given therapist is only trained to provide a limited subset of psychotherapies. Whereas a psychiatrist may be qualified to prescribe a large number of different medications or medication combinations, a psychotherapist is less likely to be able to competently provide many different psychotherapies. Another consideration is that predictions from a model may lead to self‐fulfilling prophecies, in which clinicians treat “easy” patients (those with good prognoses) differently than “difficult” patients ¹⁸² .

For both medications and psychotherapies, in real‐world, treatment decisions are rarely going to be made solely based on model recommendations. Rather, these decisions will involve the preferences of patients, the recommendations of clinicians, the availability and costs of treatments, and several other considerations ¹⁸³ . As such, the development of data‐driven decision tools should be informed by extensive consultation and co‐production with the intended users, in order to implement models that maximize acceptability and compatibility with other clinical guidelines (i.e., risk management procedures, norms about safe dosage or titration of medications).

Another crucial barrier to implementation is the interpretability of machine learning models. As algorithms become increasingly complex, sometimes called “black box” algorithms, they can become very difficult to interpret, and therefore unlikely to be acceptable to clinical users. Methods for explaining predictions of complex models have therefore been developed183, 184, but there is currently no agreed‐upon measure for assessing the quality or accuracy of these explanations. In addition, black‐box predictive models combined with (similarly complex) explanatory methods may yield complicated decision pathways that increase the likelihood of human error ¹⁸⁶ .

In order to ensure that algorithm recommendations are used in trials, additional thought and effort must be devoted to issues of dissemination and implementation, with the goal of making the recommendations simple to generate, easy to understand, trustworthy, ethical, cost‐effective, and compelling enough to influence the decision‐maker(s) ¹⁸⁷ .

A recent experiment was conducted with 220 antidepressant‐prescribing clinicians to assess the impact of providing machine learning recommendations and accompanying explanations ¹⁸⁸ . It was found that recommendations did not improve accurate selection of antidepressants in hypothetical patient scenarios, and that accuracy was even lower when incorrect recommendations were presented than when standard information was available. Prospective field‐tests182, 189 are one method for identifying the myriad institutional, cultural and contextual factors that could affect the uptake and sustained use of a precision psychiatry tool, aiming to co‐produce acceptable and interpretable decision tools with the intended users.

ETHICAL CHALLENGES

From the development of machine learning tools to their potential deployment into clinical care, we can identify several ethical challenges190, 191, 192, 193.

The first challenge concerns responsibility. With the implementation of machine learning programs into clinical practice, physicians and machine learning‐based tools would become “teammates” that collaborate in selecting an optimal treatment194, 195. In such a scenario, who will hold authority and ethical responsibility over the decision made? We believe that a competent human agent should check and take final responsibility on the machine learning‐based suggestions ¹⁹⁶ , as only he/she is equipped with empathy, a good understanding of the contextual environment and, most uniquely, consciousness.

The second challenge is to avoid dehumanization ¹⁹⁷ . Machine learning can incorporate a great variety of psychological, environmental and social variables, and there is some progress towards including subjective patient experience into machine learning models ¹⁹⁸ . However, giving a patient the space to articulate his/her concerns is essential to ensure accurate diagnosis, health outcomes, and humane care ¹⁹⁹ .

Third, making decisions is an intricate part of physicians' activity. The non‐expert tends to act as a “technician” and more likely relies on protocols, whilst the expert, after the observation of many cases, is more prone to making decisions based on tacit knowledge200, 201, 202. The ethical mandate is that practitioners use all of their capabilities, including those based on self‐experience and observation, even if this is in discordance with a statistical model. Disagreements between physicians and machine learning‐based decisions may lead to consultations with other clinicians ¹⁹³ . However, in the context of modern health care systems, respecting clinicians' judgement is vital193, 203, and they should not be forced to act against their own criteria (freedom of action) ²⁰⁴ .

Practitioners (especially those with less expertise) might be in danger of not developing/losing their own clinical judgement and become dependent on automatically deployed machine learning outcomes ²⁰⁵ , particularly for those complex cases that they fear they are not competent enough to solve. This would risk disempowerment of clinicians. On the other hand, it is a physicians' duty to train themselves in the use, understanding and interpretation of machine learning applications, so that they can trust the system and its outputs, and contribute to patients' acceptance ²⁰⁶ .

Machine learning tools need to be transparent to the human teammates to facilitate understanding194, 207. The idea of transparency is opposite to that of “black‐box” machine learning algorithms, in which the patterns the algorithm follows to make a decision for a given patient are opaque to the person and even to the developer, making very challenging (if not impossible) for the affected person to understand how the system worked out an output for him/her ¹⁹⁰ . This risks not only increasing clinicians' resistance to use the tool, but also disempowering patients and disrespecting their autonomy. Developers should consider simpler algorithms that balance interpretability with accuracy ¹⁹¹ .

Furthermore, a central issue in fair machine learning development arises when the training dataset is not a good representation of the phenomenon being studied192, 208. A model trained in such data will predict erroneous outcomes for groups that were underrepresented ²⁰⁹ . For example, a widely used machine learning algorithm assigned the same level of disease risk to Black and White patients, even if Black patients were sicker than White patients ²¹⁰ . As a consequence, the system was actively causing harm to Black patients by leading to allocation of fewer resources to them. Potentially discriminatory predictors should be left out of the model, but developers should be aware that surrogate variables correlated with the excluded set might still become relevant for prediction. Objective unbiased applications might help reduce discrimination in machine learning211, 212.

Finally, the risk of misuse of personal and sensitive data exchanged in machine learning is high ²¹³ . For this reason, machine learning tools can be only used when data security and privacy are guaranteed.

CONCLUSIONS

This paper reviews several studies suggesting that it is possible to predict outcomes and personalize psychiatric treatment by using machine learning. Several gold standard prediction studies have shown that we can predict whether a depressed patient will respond to specific antidepressants40, 41, to specific psychotherapeutic techniques ¹⁷⁷ , and whether patients with first episode psychosis will have good prognosis after one year with certain antipsychotic medications25, 50. At least three predictive models have even been tested in prospective clinical trials.

Despite this progress, the potential for machine learning in psychiatry has just begun to be explored. Predicting treatment response is just one relatively narrow use case where machine learning can add value and improve mental health care. Prediction can help with so many more clinical decisions and clinical processes. We could predict barriers that prevent an individual from engaging in care initially, or non‐adherence or dropout from care after initiation. We could streamline patients to the appropriate level of care, such as self‐guided programs vs. outpatient care, or intensive outpatient versus inpatient care, to maximize scarce health care resources. In selecting a specific treatment approach, we could optimize dosing or predict side effect profiles in order to improve symptoms but minimize impact on patient quality of life. Some psychiatric treatments carry high cost (e.g., ketamine, ECT) or unwanted side effects (e.g., metabolic disruption and weight gain for antipsychotics). Doing no harm is arguably more important than improving the probability of recovery, and so precision mental health efforts could be especially important in identifying which treatments are safest and most tolerable.

Machine learning could even help sequence treatments over time, or design specific treatment protocols for an individual. For example, modular psychological interventions can be personalized66, 68, or tailored health behavior change interventions can be customized for an individual. This form of personalization and customization has proven effective in contexts like smoking cessation, breast cancer screening, and physical activity180, 214.

Techniques like natural language processing, often using machine learning algorithms, give us the ability to draw insights from text‐based data – e.g., social media posts, peer‐support conversations, or conversation transcriptions – that might inform the content that is offered to an individual as part of his/her treatment to maximize future outcomes. In addition, the same analytic techniques may form the basis of interventions, such as chatbots, that could provide scalable support for loneliness, stress, or other subclinical psychological issues when human support is unavailable or not clinically warranted. This personalization of iCBT treatment may be particularly necessary for unguided interventions, where non‐adherence is widespread and undermines the potential for symptom relief.

Machine learning is a powerful tool that can help sift through multi‐modal predictors and model their complex/non‐linear contributions. And it can identify specific subtypes of patients, e.g., through clustering, for more nuanced prediction of treatment outcomes. Machine learning techniques are allowing us to extract more knowledge from bigger datasets in a more efficient way – which is a good and promising thing.

However, the ultimate goal of psychiatry is to better treat men­tal illness. The path toward machine learning improving psychiatric care in real‐life settings is not only governed by statistical, but also by implementation considerations. Recent seminal findings181, 182 highlight that accurate algorithms alone are not enough to ensure the success of a decision support system for precision treatment. This is because many things change in the transition from a research setting into real patient care ³⁹ . In practice, clinicians may override algorithm recommendations and choose alternative treatments. Patients may refuse the algorithm‐recommended treatment, or have restrictions to its use that were not contemplated by the decision support tool. Recommendations may be provided in a poorly‐designed user interface, and thus may go unseen or be actively ignored. All of these factors contribute to a general phenomenon of reduced effect sizes when an algorithm is implemented in clinical practice.

In our own personal experience, patient concerns around privacy are a very real problem. Because mental health is particularly sensitive, capturing personal data can be challenging and we need to innovate ways of collecting these data so that we do not have a biased perspective of the landscape due to a poor sampling within certain groups. Data needs to be collected in such a way that participants are aware of how and for what purposes those data will be used ²¹⁶ .

Technology systems must implement careful logging processes to examine concept or data drift, where the underlying distribution of a predictor or an outcome changes over time, and to ensure that the inputs and outputs of the system are auditable. This is a collective exercise of building trust in predictive models and how these will be potentially used to enhance patient outcomes, and can avoid the introduction of harm or biases in decision‐making processes.

This paper reviews many kinds of data that have been used to predict treatment outcomes in psychiatry. Ultimately, treatment responses emerge from multiple interacting biological, psychological and social factors. Therefore, in theory, multi‐modal approaches using demographic, clinical and brain variables should result in the most accurate predictions ²¹⁷ . However, to this date, it is clear that certain kinds of data – specifically sociodemographic, self‐report, psychosocial and clinical data – consistently offer more meaningful and generalizable predictions. Other types of data that might be more scientifically appealing – such as neuroimaging and genetic data – have not yet shown compelling results in a large external sample, let alone in prospective implementation studies.

Ultimately, data types that can be easily integrated into clinical care in a cost‐effective and ethical way, which is appropriate for the prevalence and invasiveness of the therapy, are most likely to show favorable return on investment for ultimate decision makers in health systems and health payers.