Predicting Dropout in Psychotherapy for Major Depressive Disorder: A Machine Learning Approach to Identifying At‐Risk Patients

ABSTRACT

Introduction

Dropout from psychotherapy for major depressive disorder (MDD) undermines treatment effectiveness for both patients and clinicians. Early identification of individuals at risk of discontinuation is therefore essential. Machine learning methods, particularly those incorporating resampling techniques to address imbalanced clinical data, may offer improved predictive capacity. This study evaluates several machine‐learning approaches to determine their potential in identifying patients at elevated risk of dropout.

Methods

Data were drawn from a randomised controlled trial of outpatients with MDD (n = 290) receiving short‐term psychodynamic supportive psychotherapy (SPSP) or cognitive behavioural therapy (CBT). Candidate predictors were restricted to variables available in both the development and external validation datasets, resulting in a limited number of overlapping predictors. Multiple machine learning models and resampling strategies were applied to predict dropout, and the best‐performing models were subsequently externally validated using an independent clinical dataset (n = 96).

Results

Models demonstrated low predictive performance overall. Resampling strategies influenced outcomes, with SMOTE consistently improving F1‐scores compared with no resampling. The combination of random forest with SMOTE achieved the highest F1‐score (0.303). Five key predictors of dropout emerged: younger age, unemployment, a higher number of comorbid Axis I disorders, absence of a partner, and moderate depression severity. All models showed reduced performance during external validation.

Discussion

The models showed very limited predictive performance, which may in part reflect the restricted predictor set resulting from the requirement that variables be assessed in both trials. While clinical application is premature at this stage, the findings provide a valuable proof of concept and highlight key methodological challenges inherent in cross‐trial dropout prediction. Despite this, the identified predictors are routinely available in clinical practice, suggesting that future models, if developed with richer predictor sets and validated in larger samples, could ultimately support therapists in the early identification of patients at risk of dropout and in proactively addressing disengagement.

Summary

• Machine learning models may offer a useful preliminary indication of which patients are at heightened risk of treatment dropout, although their predictive accuracy appears low.

• The key predictors identified—younger age, unemployment, multiple comorbid Axis I disorders, absence of a partner and moderate depression severity—are factors that tend to be routinely assessed in clinical practice, which could support practical application of the model.

• Resampling techniques, particularly SMOTE, seem to enhance model performance when working with imbalanced clinical datasets and may provide methodological guidance for similar clinical prediction efforts.

• Incorporating such predictive tools might help therapists to anticipate potential disengagement and consider early interventions aimed at supporting treatment continuity.

• If further validated and refined, predictive models could contribute to more efficient treatment planning and more tailored care for patients receiving psychotherapy for major depressive disorder.

1. Introduction

The effectiveness of psychotherapies for major depressive disorder (MDD) has been extensively demonstrated. For instance, in a recent network meta‐analysis encompassing 331 studies, Cuijpers et al. (2021) demonstrated that all psychotherapies are efficacious in the treatment of depression. However, ensuring that patients complete their prescribed treatment sessions is a challenge. An estimated 20% of patients terminate treatment prematurely while clinically significant symptoms remain (Cooper and Conklin 2015). These so‐called dropouts can have serious consequences. First, at the patient level, patients show higher symptom severity after dropout compared to those who complete treatment (Cahill et al. 2003). Second, on a therapist level, therapists can feel unsatisfied and demoralised, which could contribute to therapist burnout (Piselli et al. 2011). Third, from a societal perspective, it exacerbates the growing issue of untreated mental health conditions, placing a significant strain on the healthcare system (Barrett et al. 2008).

Given the high occurrence of dropout and its considerable impact on patients, therapists, and society, early identification of individuals at risk is essential (Swift and Greenberg 2012). Research indicates that predicting dropout can inform targeted interventions, which may reduce premature termination and improve treatment engagement and outcomes for depression (Lutz et al. 2018; Waumans et al. 2023).

In order to prevent dropout, it is important to understand factors associated with dropout across different domains. Previous research on patient‐specific predictors found that higher dropout was predicted by (a) patient variables, such as a low level of education, comorbid personality disorder, lower socioeconomic status, unemployment and younger age; (b) therapist variables, such as lower experience level of the therapist; and (c) therapy variables, such as unlimited treatment duration and receiving non‐preferred therapy (Cooper and Conklin 2015; Garfield 1994; Reis and Brown 1999; Swift and Greenberg 2012; Swift et al. 2017; Wierzbicki and Pekarik 1993; Windle et al. 2020). However, across studies, not all predictors were consistently associated with dropout, and they were often examined separately rather than in combination.

The introduction of machine learning (ML) approaches offers significant potential for improving predictions compared to traditional statistical methods. A key advantage of ML is its ability to capture complex interactions between variables (Makridakis et al. 2018). Common ML approaches include tree‐based models, which resemble flowcharts that recursively partition data into increasingly homogeneous groups based on relevant predictors. Boosted algorithms, such as gradient boosting machines and XGBoost, combine many weak learners into a strong predictive model by iteratively focusing on the errors made in previous rounds. Through this process, the algorithm reduces bias and improves predictive accuracy (e.g., Kuhn and Johnson 2013).

An important methodological challenge in predicting dropout is that datasets are typically imbalanced: Substantially more patients complete treatment than drop out. This imbalance can cause ML models to become biased toward the majority class. To mitigate this problem, researchers use resampling methods such as down‐sampling, which reduces the size of the larger group to create a more balanced dataset. Studies show that these resampling methods significantly improve dropout prediction (Bennemann et al. 2022; Giesemann et al. 2023).

This study aims to investigate the potential value of different ML algorithms and resampling methods for predicting dropout in a randomised controlled trial (RCT) comparing two provenly effective forms of psychotherapy for MDD. To this end, we tested the following models, which were developed based on the work of Bennemann et al. (2022): extreme gradient boosting (XGB), random forest (RF) and elastic net regularisation (ENR). Additionally, resampling strategies to address class imbalance, including up‐ and down‐sampling and the synthetic minority oversampling technique (SMOTE), were applied in line with recommendations by Giesemann et al. (2023).

The predictive utility of the developed dropout prediction model was then tested by external validation using a second independent dataset from a multicentre randomised trial of the two forms of psychotherapy. External validation is essential to assess generalizability and avoid overfitting, as models often perform optimistically on their development data but fail in new populations (Bleeker et al. 2003).

2. Methods

2.1. Trial Design

This study uses data from two Dutch multicentre randomised trials designed to evaluate the effectiveness of psychotherapeutic interventions for patients with MDD. The D*Phase trial (Miggiels et al. 2021) was used to train and internally validate the machine learning models, while the FreqMech trial (Bruijniks et al. 2015; Bruijniks et al. 2020) served as an independent dataset for external validation. Both studies employ a parallel‐group design with participants randomly assigned to different treatment conditions.

The D*Phase trial randomly assigned participants to short‐term psychodynamic supportive psychotherapy (SPSP, N = 148) or cognitive behavioural therapy (CBT, N = 142), each delivered in 16 sessions over 8 weeks (twice weekly). The FreqMech trial randomising depressed patients into different session frequencies of CBT and interpersonal therapy (IPT) (i.e., CBT weekly [n = 49], CBT twice weekly [n = 49], IPT weekly [n = 55], IPT twice weekly [n = 47]). All conditions included a maximum of 20 sessions, delivered over 16 to 24 weeks depending on session frequency.

2.2. Participants

The participants were adult outpatients (aged 18–65) with a primary diagnosis of MDD, confirmed by the Structured Clinical Interview for DSM‐IV Axis I disorders (SCID‐I; First et al. 1997) or Mini‐International Neuropsychiatric Interview (Sheehan et al. 1998). D*Phase participants were recruited at Dimence, a specialised mental healthcare institution in the north‐east of the Netherlands. FreqMech participants were recruited from nine specialised mental healthcare centres across the Netherlands (Bruijniks et al. 2020). The joint inclusion criteria were sufficient knowledge of the Dutch language and the allowance of a stable dosage of antidepressant medication at the start of the treatment (at least 6 weeks for D*Phase and at least 3 months for FreqMech).

2.3. Exclusion Criteria

Joint exclusion criteria were high suicide risk, substance dependence (excluding nicotine), and psychotherapeutic treatment in specialised mental health care within the past year.

Exclusion criteria for the two studies differed on the following aspects: The D*Phase trial excluded patients with psychotic symptoms and those unable to adhere to treatment requirements (such as following treatment sessions twice weekly, in the following 8 weeks of therapy), while in the FreqMech trial, patients were excluded who met the criteria for cluster A or B personality disorders and who had no access to internet facilities.

2.4. Therapists

The sessions were provided by qualified therapists who received additional training. In the D*Phase trial, the therapists had an average of 10.3 years' clinical experience, and in the FreqMech trial, the therapists had an average of 9.55 years' experience. In the D*Phase trial, therapists completed a 3‐day course in both treatments, facilitated by qualified and experienced experts. For the FreqMech trial, therapists received 2 days of training from qualified and experienced experts.

The trial protocols were approved by the Medical Ethics Committee (i.e., Medical Centre Leeuwarden for the D*Phase trial, VU Medical Centre for the FreqMech trial), and written informed consent was provided by all participants. The trials are registered with the Netherlands Trial Register (NL5753 and NTR4856). Further details regarding the study design can be found in the study protocols (Miggiels et al. 2021; Bruijniks et al. 2015).

2.5. Measures

2.5.1. Dropout

In both datasets, dropout was defined as the unexpected termination of treatment by individuals who were randomised to a treatment but attended fewer than eight sessions and did not achieve a response or remission. This is in line with the definition in the D*Phase study protocol and the per‐protocol analysis (Miggiels et al. 2025). The choice of eight sessions is based on it being considered the minimum recommended number for effective depression treatment (Miggiels et al. 2025). In the event of a dropout, the therapist would have asked about the reason for the dropout.

2.5.2. Pre‐Treatment Variables

Different potential predictors were measured at baseline. Table 1 provides an overview of the instruments at intake and baseline per dataset. As the table shows, only a few questionnaires were used in both studies. The following areas were covered by these questionnaires: social demographics, depression and clinical characteristics, and comorbidities. Table 2 provides an overview of the overlapping variables. This external dataset was chosen because the unique situation that its session frequency matches that of the current study.

TABLE 1.

Overview of instruments by dataset.

Instruments at intake and baseline	D*Phase	FreqMech
MINI	X	X
Inventory of Depressive Symptomatology (IDS‐SR)	X
Beck Depression Inventory II (BDI‐II)		X
Remission of Depression Questionnaire (RDQ)		X
Sheehan Disability Scale (SDS)	X
RAND‐36		X
Mental Health Continuum‐Short Form (MHC‐SF)	X
EQ‐5D		X
Longitudinal Interview Follow‐up Evaluation (LIFE)		X
Behavioral Activation for Depression Scale (BADS)		X
Cognition Checklist (CCL)		X
Action Control Scale (ACS)		X
List of Threatening Experiences (LTE)	X
Checklist Life Events for DSM‐5 (LEC‐5)	X
Childhood Trauma Questionnaire (CTQ)		X
Competencies of Cognitive Therapy Scale (CCTS)		X
IPT Skill Inventory		X
Brief Symptom Inventory (BSI)	X
Agoraphobia Scale (AGO)	X
Rosenberg Self‐Esteem Scale (RSES)	X
Rumination Response Scale (RRS)	X
Defense Style Questionnaire (DSQ 40)	X
Inventory of Personality Organisation (IPO)	X
Medication	X	X
Demographics	X	X

TABLE 2.

Demographical and clinical characteristics for each dataset.

	D*Phase (n = 290)	FreqMech (n = 96)
Social demographics
Age, M (SD)	35.17 (13.17)	39.55 (12.27)
Female, n (%)	174 (60)	57 (59)
Partner, yes, n (%)	148 (51)	41 (43)
Education level
Low, n (%)	26 (9)	5 (5.2)
Intermediate, n (%)	184 (63)	58 (60)
High, n (%)	69 (24)	33 (34)
Unknown, n (%)	11 (3.8)	0 (0)
Employment, yes, n (%)	170 (59)	64 (67)
Depression and clinical characteristics
Pharmacotherapy at start treatment, n (%)	116 (40)	25 (26)
Episode duration (> 2 years), n (%)	122 (42)	35 (36)
Depression symptoms
Minimal, n (%)	2 (0.7)	0 (0)
Mild, n (%)	19 (6.6)	10 (10)
Moderate, n (%)	104 (36)	16 (17)
Severe, n (%)	165 (57)	70 (73)
Comorbidity
Number of comorbid Axis I disorders assessed by the MINI, M (SD)	0.56 (0.80)	0.88 (1.10)

3. Statistical Analysis

3.1. Data Preparation

All data preprocessing and analyses were conducted using RStudio (RStudio Team 2020) and the R code can be found on Open Science Framework: OSF | Dropout_RCode.rmd.

The tidymodels framework was used for model building and analysis (Kuhn and Silge 2022). The tidymodels framework is a collection of R modelling and machine learning packages using tidyverse principles for consistent and flexible workflows.

First, missing values were checked. No variables with missing data were found in either dataset, and no patients were excluded from the analyses due to having too many missing values. Second, a standardised preprocessing pipeline was implemented using the tidymodels framework in R. The steps included the following: (1) conversion of categorical predictors into dummy variables; (2) removal of predictors with near‐zero or zero variance to reduce noise; (3) normalisation of all numeric predictors to ensure comparability across scales; and (4) elimination of highly correlated predictors (Spearman's ρ ≥ 0.70) to mitigate multicollinearity. All transformations were estimated on the training set and applied consistently to validation and test data. No variables were excluded due to near‐zero variance, nor were any removed due to high correlation.

3.2. Resampling Methods

The dropout rate was unbalanced in the dataset, which can affect the performance of ML models (Japkowicz and Stephen 2002; Kumar et al. 2022). Therefore, the models were run using different approaches: (1) no imbalance handling, (2) up‐ and down‐sampling and (3) SMOTE. Up‐sampling involves increasing the number of instances in the minority class by duplicating existing instances, whereas down‐sampling reduces the number of instances in the majority class by randomly removing some of them (Japkowicz and Stephen 2002). SMOTE combines up‐ and down‐sampling and generates new examples for the minority class by interpolating existing instances (Chawla et al. 2002).

3.3. Machine Learning Algorithms

XGB) and RF were selected based on Bennemann et al. (2022), who demonstrated the effectiveness of tree‐based and boosted algorithms in predicting psychotherapy dropout. ENR was additionally included as a penalised regression method, given its suitability for datasets with limited predictors and small sample sizes (Kuhn and Johnson 2013).

For these models, hyperparameters were optimised using the standard grid search procedure implemented in the tidymodels framework. And for all models tenfold cross‐validation was applied. In 10‐fold cross‐validation, in each of the 10 iterations, nine folds were used for training the model, whereas the remaining fold was used for validation. This process was repeated 10 times, with each fold serving as the validation set exactly once.

XGB is an advanced implementation of gradient boosting that uses decision trees as base learners. Its hyperparameters include the number of trees (boosting rounds), tree depth (maximum depth of individual trees), minimum observations per node (to prevent overly specific splits), learning rate (shrinkage to mitigate overfitting), loss reduction (minimum gain required to split a node), sample size (fraction of data used per tree to introduce randomness) and early stopping iterations (to halt training when validation performance plateaus). These hyperparameters control complexity and prevent overfitting (Chen and Guestrin 2016).

RF is an ensemble method that builds many decision trees on bootstrapped samples and aggregates their predictions to improve accuracy and stability. Each tree uses a random subset of predictors at each split to reduce correlation and overfitting. The main tuned hyperparameters were mtry (number of predictors considered at each split) and min_n (minimum observations per terminal node), while the number of trees was fixed at 10,000 for stable estimates (Kuhn and Johnson 2013).

ENR combines Lasso (L1) and Ridge (L2) penalties, allowing simultaneous variable selection and coefficient shrinkage, which improves generalisation and addresses multicollinearity (Kuhn and Johnson 2013). Two hyperparameters were tuned: the penalty (λ), which controls the overall strength of regularisation, and the mixture (α), which determines the balance between L1 and L2 penalties (α = 1 for pure Lasso, α = 0 for pure Ridge).

3.4. Model Evaluation

For imbalanced classification datasets, selecting an appropriate evaluation metric is crucial, since high accuracy can be achieved even when the detection of the minority class is poor (Sun et al. 2011). The F1‐score is commonly preferred because it gives equal weighting to precision and recall, providing a balanced measure ranging from 0 to 1, where 1 indicates perfect precision and recall (Weiss 2013).

All machine learning models were trained and internally validated on data of the D*Phase trial (Miggiels et al. 2021). All nine models, comprising three algorithms each combined with three resampling strategies, were subsequently evaluated on the independent FreqMech trial to assess generalisability (Bruijniks et al. 2015; Bruijniks et al. 2020). In addition to the F1‐score, the Brier score and area under the curve (AUC) were computed. The Brier score quantifies the accuracy of probabilistic predictions on a scale from 0 to 1, with lower values indicating better calibration and overall predictive accuracy. Conversely, the AUC reflects the model's discriminative ability, also ranging from 0 to 1, where values closer to 1 indicate excellent separation between classes and values near 0.5 represent performance no better than chance. Additionally, variable importance scores were examined to identify the most influential predictors contributing to dropout risk across models.

4. Results

4.1. Data Pre‐Processing

Descriptive analyses of the pre‐treatment variables of the D*Phase and FreqMech trial are presented in Table 2 and Table 3. The external validation dataset (FreqMech) differed from the development dataset (D*Phase) in several respects. The D*Phase trial included 290 participants, of whom 37 (12.75%) were classified as dropouts, whereas FreqMech comprised 96 participants receiving twice‐weekly sessions, with 7 (7.3%) classified as dropouts. Compared with D*Phase, FreqMech participants were generally older (completers: 39.3 vs. 35.8 years; dropouts: 42.3 vs. 31.2 years), more frequently employed (67% vs. 61% among completers; 57% vs. 47% among dropouts), and exhibited higher baseline depression severity among completers (severe: 75% vs. 58%). Notably, the direction of the age difference between dropouts and completers was reversed across the two datasets: in D*Phase, dropouts were on average younger than completers (31.2 vs. 35.8 years), whereas in FreqMech, dropouts were older than completers (42.3 vs. 39.3 years). This reversal suggests that the relationship between age and dropout may not be consistent across samples, which has direct implications for the generalizability of age as a predictor. The FreqMech dataset was selected for external validation on the basis of its session frequency, which corresponds with that of the current study. Nevertheless, the datasets differed in several other meaningful respects. These differences represent meaningful heterogeneity between the development and validation datasets, which is known to challenge the generalizability of machine learning models (Tait et al. 2025; van Bronswijk et al. 2021).

TABLE 3.

Demographical and clinical characteristics for completers and dropouts for each dataset.

	D*Phase		FreqMech
	Completer (n = 253)	Dropout (n = 37)	Completer (n = 89)	Dropout (n = 7)
Social demographics
Age, M (SD)	35.79 (13.27)	30.92 (11.79)	39.34 (12.13)	42.29 (14.68)
Female, n (%)	153 (60)	21 (57)	53 (60)	4 (57)
Partner, yes, n (%)	130 (51)	18 (49)	39 (44)	2 (29)
Education level
Low, n (%)	21 (8)	5 (14)	5 (6)	0 (0)
Intermediate, n (%)	160 (63)	24 (65)	52 (58)	6 (86)
High, n (%)	62 (25)	7 (19)	32 (36)	1 (14)
Unknown, n (%)	10 (4)	1 (3)
Employment, yes, n (%)	153 (60)	17 (46)	60 (67)	4 (57)
Depression and clinical characteristics
Pharmacotherapy at start treatment, n (%)	105 (42)	11 (30)	22 (25)	3 (43)
Episode Duration (> 2 years), n (%)	105 (42)	17 (46)	31 (35)	4 (57)
Depression symptoms
Minimal, n (%)	1 (0)	1 (3)	0 (0)	0 (0)
Mild, n (%)	16 (6)	3 (8)	9 (10)	1 (14)
Moderate, n (%)	89 (35)	15 (41)	13 (15)	3 (43)
Severe, n (%)	147 (58)	18 (49)	67 (75)	3 (43)
Comorbidity
Number of comorbid Axis I disorders assessed by the MINI, M (SD)	1.07 (1.16)	1.19 (1.22)	0.90 (1.12)	0.71 (0.95)

4.2. Model Development and Internal Validation

All variables were included in the machine learning algorithms. The internal validation results are presented in Table 4. Overall, RF achieved the highest F1‐score (0.303) and a low AUC (0.550) when combined with SMOTE. Among the XGB variants, the best performance was observed with SMOTE (F1‐score = 0.285, AUC = 0.563), whereas ENR demonstrated relatively stable but lower performance across resampling strategies, with F1‐scores ranging from 0.256 to 0.267.

TABLE 4.

Model performances.

		D*Phase (internal validation set)			FreqMech (external validation set)
Model	Resampling method	F1‐Score	Brier	AUC	F1‐Score	Brier	AUC
XGB	Up‐ and down	0.264	0.157	0.501	0.133	0.116	0.547
XGB	SMOTE	0.285	0.116	0.563	0.211	0.151	0.694
XGB	None	0.267	0.215	0.488	0.130	0.071	0.549
RF	Up‐ and down	0.300	0.125	0.527	0.188	0.101	0.642
RF	SMOTE	0.303	0.156	0.550	0.200	0.124	0.554
RF	None	0.258	0.187	0.417	0.150	0.073	0.509
ENR	Up‐ and down	0.267	0.176	0.484	0.111	0.121	0.398
ENR	SMOTE	0.267	0.125	0.494	0.119	0.174	0.483
ENR	None	0.256	0.230	0.503	0.000	0.078	0.491

Abbreviations: ENR, elastic net regularisation; RF, random forest; SMOTE, synthetic minority oversampling technique; XGB, extreme gradient boosting.

Resampling strategies influenced model performance. SMOTE consistently improved F1‐scores compared to no resampling, while up‐ and down‐sampling yielded competitive results. Despite these improvements, F1‐scores were relatively low across all models. Nevertheless, the top‐performing models surpassed the predefined minimum threshold of 0.20, indicating performance above chance level. This threshold was derived from the base dropout rate of 0.1275 and assuming a chance level of 0.50 ($2*\frac{0.1275*0.5}{0.1275+0.5}$) (Giesemann et al. 2023).

In the Random Forest models using SMOTE and up‐ and down‐sampling, younger age, unemployment, a higher number of comorbid Axis I disorders, absence of a partner, and moderate depression severity emerged as the strongest predictors of dropout.

4.3. External Validation

All models were evaluated in the external validation dataset, the FreqMech trial. The predictive performance of all models is shown in Table 4. Overall, XGB with SMOTE achieved the best results, with an F1‐score of 0.211, marginally above the predefined minimum threshold, and an AUC of 0.694. Although predictive performance was low, the models demonstrated some ability to discriminate between dropouts and completers. Compared to internal validation, performance decreased across all models.

5. Discussion

The present study set out to develop predictive models capable of identifying individuals at increased risk of dropping out of psychotherapy for major depressive disorder, using data from a large randomised controlled trial in which patients received either short‐term psychodynamic supportive psychotherapy (SPSP; N = 148) or cognitive behavioural therapy (CBT; N = 142). In addition, the study aimed to evaluate the performance of several machine learning algorithms and resampling techniques in predicting treatment discontinuation.

This study revealed a dropout rate of 12.75%. This finding is relatively low compared to previous studies that documented dropout rates of 20%–25% (Cooper and Conklin 2015; Swift and Greenberg 2012). One possible explanation is the relatively high number of sessions in this study offered in a relatively short time span of 8 weeks of treatment duration, as a long treatment duration is known for its increased risk of dropout (e.g., Cooper and Conklin 2015).

The results indicate that tree‐based machine learning models, particularly random forest and XGB, combined with resampling techniques such as SMOTE, delivered the strongest predictive performance in internal and external validation. Nevertheless, F1‐scores were low across all models. Internally, RF with SMOTE achieved the highest F1‐score (0.303), while externally XGB with SMOTE performed best (F1 = 0.211), the latter only marginally exceeding the predefined minimum threshold of 0.20, reflecting performance close to chance level. Internal AUC values around 0.55 likewise reflect limited discriminative ability. These results suggest that the models, in their current form, are not yet suitable for clinical application, though they do demonstrate that some meaningful signal can be extracted from routinely collected clinical data under challenging conditions, a small training sample, severe class imbalance, and a restricted predictor set. The superior performance of models using SMOTE is consistent with previous studies (Giesemann et al. 2023). These findings emphasise the importance of addressing class imbalance in dropout prediction and highlight the need for careful handling of imbalanced data in psychotherapy research.

Five predictors were identified as being associated with dropout in the final prediction model, with varying degrees of consistency with previous research.

First, a higher number of comorbid Axis I disorders was linked to increased odds of dropout. Some other studies have also found a relationship between comorbidities such as personality disorders and anxiety disorders (Arnow et al. 2007; Cooper and Conklin 2015; Swift and Greenberg 2012).

Second, moderate depression severity was associated with higher dropout risk. Prior studies have reported mixed results regarding baseline depression severity, with some identifying high depression severity as a predictor of dropout, others finding no effect, and others finding low depression severity as a predictor of dropout (e.g., Binnie and Boden 2016; Fernández et al. 2021; Simon and Ludman 2010; van Dijk et al. 2023). These inconsistencies may reflect a mismatch between treatment intensity and symptom burden. In this study, the relatively high treatment frequency (twice weekly) may have seemed less necessary for individuals with milder symptoms, prompting early discontinuation.

Third, unemployment emerged as a predictor of dropout. While some studies support this association (Hanevik et al. 2023), others do not (Swift and Greenberg 2012). Unemployment may contribute to a lack of daily structure, which is critical for maintaining treatment adherence, or indicate broader vulnerabilities such as difficulty managing responsibilities or financial constraints affecting travel to sessions.

Fourth, younger age was associated with dropout in D*Phase, consistent with previous research (Swift and Greenberg 2012; Bennemann et al. 2022; Hanevik et al. 2023). One possible explanation is that younger individuals may experience greater challenges with engagement, potentially due to less life experience or difficulties in sustaining long‐term commitment (Fenger et al. 2010). However, this pattern was not replicated across datasets: in the FreqMech validation sample, dropouts were on average older than completers, indicating the opposite relationship. This reversal likely contributed to the reduced generalisability of age as a predictor during external validation. More broadly, this finding illustrates that predictor–outcome relationships identified in one sample may not consistently generalise to others, particularly when there are substantial differences in sample characteristics.

Finally, absence of a partner was identified as a predictor of dropout. This finding aligns with evidence that social support and interpersonal relationships play a key role in treatment engagement and retention (Hanevik et al. 2023; Swift and Greenberg 2012).

5.1. Strengths and Limitations

A key strength of this study is its use of external validation with an independent dataset from a different centre (the FreqMech trial). This crucial step assesses generalisability and mitigates overfitting, as models often perform well on development data but less well on new populations. External validation is uncommon in prediction research: For example, only 4.6% of studies included in a recent systematic review by Salazar de Pablo et al. (2021) incorporated it. By incorporating external validation, the study provides robust evidence that the models identify meaningful patterns, thereby enhancing their credibility and clinical applicability. A noteworthy strength of the developed prediction model is also that the variables contributing to its final structure consist almost entirely of information that is routinely registered or assessed in everyday clinical practice. Because these predictors, such as demographic characteristics, diagnostic indicators and baseline symptom severity, are already standard components of routine intake procedures, the model can be readily applied without requiring additional measurement burden. This enhances its practical clinical utility, increasing the likelihood that the model can be integrated into existing workflows to support early identification of patients at elevated risk of dropout.

However, this study has several limitations. The most important limitation concerns the predictor set used in these models. Because the prediction models were developed on D*Phase data and externally validated on FreqMech data, only variables assessed in both trials could be considered candidate predictors. This resulted in a limited number of overlapping variables, many of which were not strongly associated with treatment outcomes in prior research. This restricted predictor set may have constrained the models' predictive capacity from the outset, and it is possible that a broader set of variables would have yielded stronger predictions because different predictors might have been relevant for the different datasets. Future studies should prioritise the use of comprehensive and harmonised data collection across trials to enable more robust predictor selection, but also investigate whether dropout is differentially predicted for different groups of patients.

Second, while the training sample of 290 participants is close to the recommended minimum of 300 cases for robust prediction, a somewhat larger sample might have further strengthened the model's stability (Giesemann et al. 2023). Third, the external validation set comprised 96 cases, which provides a reasonable basis for assessing generalisability; however, there is a limited number of dropout events in this dataset. Fourth, meaningful differences existed between the development (D*Phase) and validation (FreqMech) datasets, including differences in age, employment status, baseline severity and dropout rate. Recent research has demonstrated that machine learning prediction models frequently struggle to generalise across datasets with differing characteristics (Tait et al. 2025; van Bronswijk et al. 2021). Based on this literature, the decline in performance during external validation was to be expected: van Bronswijk et al. (2021) found substantial drops in predictive accuracy when models were externally validated across RCTs, even within the same diagnostic population. Similarly, Tait et al. (2025) reported limited external validity for a personalised advantage index for PTSD treatment selection. These findings underscore that generalisability of ML models in psychotherapy research cannot be assumed, and external validation should therefore be interpreted with caution when dataset characteristics differ substantially. Finally, the study lacks therapist‐related variables and patient preferences, both of which have been linked to dropout in prior research (Swift et al. 2011; Zimmermann et al. 2017). Incorporating these data in future work could substantially enhance predictive accuracy and clinical applicability.

5.2. Clinical Implications

Given the low predictive performance observed, the current models are not yet ready for direct clinical application. Nevertheless, the predictors identified such as younger age, unemployment and absence of a partner are routinely available in clinical practice and may help clinicians identify patients at elevated dropout risk. Future work refining these models with richer predictor sets and larger samples may ultimately support their integration into clinical decision‐making.

6. Conclusions

In conclusion, this study examined the potential of machine learning algorithms combined with resampling methods to predict psychotherapy dropout in individuals with major depressive disorder. Although predictive performance was low and the findings do not yet support clinical implementation, the study provides a methodological foundation for future work and underscores the importance of harmonised data collection, larger samples and broader predictor sets in developing viable dropout prediction models.

Author Contributions

SBre: conceptualisation, data curation, formal analysis, investigation, methodology, project administration, validation, visualisation, writing – original draft; SBro: conceptualisation, data curation, formal analysis, methodology, supervision, validation, writing – review and editing; AB: supervision, writing – review and editing; MM: funding acquisition, data collection, project administration, validation, writing – review and editing; MD: conceptualisation, funding acquisition, investigation, project administration, supervision, validation, writing – review and editing; SBru: data collection of FreqMech data, revising the manuscript. All authors contributed to the article and approved the final manuscript.

Funding

This study was funded by Dimence Mental Health Care and the VCVGZ association.

Ethics Statement

The D*Phase study has been approved by the medical ethical board of Medical Centre Leeuwarden (number: METC: RTPO970, registration number NL 56047.099.16, July 21, 2016). The study design was registered with the Dutch trial register, registration number: NL5753. The FreqMech study has been approved by the Medical Ethical Committee of VU Medical Centre Amsterdam (registration number 2014.337). The study design was registered with the Dutch trial register, registration number NTR4856.

Conflicts of Interest

The authors declare no conflicts of interest.

Data Availability Statement

The data that support the findings of this study are available on request. Supplemental R code for this article can be accessed https://osf.io/6qevw/files/tkbu8?view_only=55f848e67e6e4be58c5838e79e69ce71.