Which client with generalized anxiety disorder benefits from a mindfulness ecological momentary intervention versus a self-monitoring app? Developing a multivariable machine learning predictive model

Abstract

Precision medicine methods (machine learning; ML) can identify which clients with generalized anxiety disorder (GAD) benefit from mindfulness ecological momentary intervention (MEMI) vs. self-monitoring app (SM). We used randomized controlled trial data of MEMI vs. SM for GAD (N = 110) and tested three ML models to predict one-month follow-up reliable improvement in GAD severity, perseverative cognitions (PC), trait mindfulness (TM), and executive function (EF). Eleven baseline predictors were tested regarding differential reliable change from MEMI vs. SM (age, sex, race, EF errors, inhibitory dyscontrol, set-shifting deficits, verbal fluency, working memory, GAD severity, TM, PC). The final top five prescriptive predictor models of all outcomes performed well (AUC = .752 .886). The following variables predicted better outcome from MEMI vs. SM: Higher GAD severity predicted more GAD improvement but less EF improvement. Elevated PC, inhibitory dyscontrol, and verbal dysfluency predicted better improvement in most outcomes. Greater set-shifting and TM predicted stronger improvements in GAD symptoms and TM. Older age predicted more alleviation of GAD and PC symptoms. Women exhibited more enhancements in trait mindfulness and EF than men. White individuals benefitted more than non-White. PC, TM, EF, and sociodemographic data might help predictive models optimize intervention selection for GAD.

1. Introduction

Generalized anxiety disorder (GAD) is a mental disorder linked to compromised socio-occupational functioning, physical health, and quality of life (Newman et al., 2017; Zainal & Newman, 2022b; Zhou et al., 2017). Fortunately, effective psychosocial interventions exist. These include face-to-face cognitive-behavioral therapies (CBT; Newman et al., 2020), mindfulness-based interventions (MBIs; Zainal & Newman, In Press), and guided and self-guided digital mental health interventions (DMHIs) that can treat GAD and comorbid symptoms (e.g., panic; Apolinário-Hagen et al., 2020). However, providing high-intensity face-to-face psychotherapy to all clients with GAD is infeasible in many resource-limited treatment settings where demand outstrips supply (Rebello et al., 2014). Self-guided DMHIs can aid dissemination when logistical barriers prevent seeking face-to-face psychotherapies, such as shame, stigma, and financial constraints (Goetter et al., 2020). Persons with GAD also reported high levels of acceptability toward DMHIs, such as mindfulness ecological momentary interventions (MEMIs) and self-monitoring apps (SMs) to manage symptoms alone (Newman et al., 2021). EMI represents a growing field of keen interest, holding substantial promise in the context of GAD and related mental health problems. A recent review evidenced that users generally found EMIs beneficial as long as disturbances, such as measurement burden, did not interfere with participants’ everyday routines (Dao et al., 2021). Thus, determining which clients with GAD differentially benefit from unique scalable self-guided DMHIs is essential.

Heterogeneity of treatment effects (i.e., the fact that more efficacious interventions at the group level may not be the most effective for a specific patient) is intrinsic to all psychotherapies (Kaiser et al., 2022), including low-intensity self-guided DMHIs. For example, evidence suggested that for a particular subgroup, SM alone helped to alleviate anxiety, depression, and related symptoms (Gruszka et al., 2019). A meta-analysis also indicated that MEMIs, more than other DMHIs, were suitable for depression and anxiety during pregnancy in patients with specific symptoms and demographic profiles (Silang et al., 2022). However, most DMHI randomized controlled trials (RCTs) thus far solely examined aggregate treatment effects for GAD and related disorders (Carl et al., 2020b). Doing so precludes understanding heterogeneity of treatment effects (Goldberg et al., 2022). Thus, from a resource-management perspective, evaluating which DMHI yields the most optimal treatment effect for which specific client with GAD would be profitable for patients, clinicians, and public health administrators.

Additionally, there remains room for improvement when determining what specific DMHIs work best for which clients with GAD. Previous studies tended to use approaches based on classical ordinary least squares regression (OLS; Faaland et al., 2022). Traditional OLS is suboptimal as it necessitates manual specification of non-linearity and interactions and can lead to underfitting of the data (Wallert et al., 2022). Machine learning (ML) may facilitate the building of precise predictive models. ML comprises data-driven methods that empower computer algorithms to determine and iteratively improve the best parameters to fit complex predictor patterns (Jordan & Mitchell, 2015). Despite compromising model interpretability to some extent, fitting flexible nonlinear and higher-order interaction ML algorithms instead can enhance model derivation and performance and optimize model prediction (Pearson et al., 2019). Further, methods exist to determine significant treatment predictors and delineate or characterize how each predictor correlates with the outcome (Archer & Kimes, 2008). By managing collinearity and carefully regulating overfitting, the accuracy of ML algorithms can be evaluated (Christodoulou et al., 2019).

Harnessing ML for prediction using DMHI datasets is a nascent yet rapidly growing endeavor in clinical science (Jacobson & Nemesure, 2021). Combining ML with DMHIs could be valuable since self-guided DMHIs are easily implementable and have promising preliminary results. However, previous DMHI studies that used ML to predict intervention outcomes for adults with anxiety, depression, and obsessive-compulsive and related disorders (e.g., Hornstein et al., 2021; Lenhard et al., 2018; Pearson et al., 2019) had the shortcoming of testing a restricted set of possible ML models. Also, the only digitally delivered MBI that used ML to date (Lekkas et al., 2021) focused narrowly on compliance features as predictors of post-treatment stress reduction. Compliance features are not optimal because they rely on information collected during treatment as opposed to using information collected at baseline that may inform immediate prescriptive treatment assignment. Testing a broader predictor set comprising baseline symptom severity, demographic variables, and theory-based predictors with ML models (Canby et al., 2021; Elhai & Montag, 2020) can help to optimally determine which DMHI app works best for which person with GAD.

Drawing from the compensation model (Cheavens et al., 2012), treatment efficacy is contingent upon effectively addressing the specific deficiencies within patients’ disorder-pertinent vulnerabilities. Individuals with elevated perseverative cognitions might experience more gains in utilizing MEMIs than SMs, given deficits in understanding and applying mindfulness techniques. Higher trait perseverative cognition could afford individuals with GAD an expanded opportunity to cultivate mindfulness skills over time. Consequently, this could increase their chances of experiencing symptom relief through MEMIs more than SMs (Spinhoven et al., 2018). Such distinct effects would probably arise since individuals grappling with excessive worry and self-focused repetitive thinking tend to derive the greatest advantage from breaking free from those unproductive habits (Hallion et al., 2022). They stand to gain by redirecting their attention toward the present task and fostering a sense of personal agency (Gallagher et al., 2014) via mindfulness exercises. These patterns may manifest as individuals with GAD possess greater potential to nurture present-focused awareness and positive emotions through mindfulness practices (Perestelo-Perez et al., 2017).

Mindfulness theories propose that although sustaining mindfulness can be challenging, staying mindful by optimally using cognitive resources can foster more robust insight, cognitive flexibility, and self-regulation abilities (Hofmann & Gomez, 2017; Spinhoven et al., 2022). Lower levels of trait mindfulness could prevent participants from reaping optimal benefits from a MEMI vs. an SM. More robust levels of mindfulness traits likely promote a better response to a MEMI than an SM because they help to encourage prefrontal cortex-mediated regulation and reduce the inclination to experience or avoid negative emotions (Mizera et al., 2015; Spinhoven et al., 2017). Collectively, heightened trait mindfulness could facilitate being optimized by MEMI vs. SM to improve global trait mindfulness (cf. capitalization model; Murphy et al., 2021).

In addition, the self-regulatory executive function (S-REF; Matthews & Wells, 2004) model and cognitive model of pathological worry (Hirsch & Mathews, 2012) can be extended to suggest that weaker executive function (EF) skills may hamper assistance provided by MEMI vs. SM. EF refers to higher-order cognitive control skills that govern thought and action repertoires (Keller et al., 2023). Empirical data showed that EF deficits correlated consistently with attentional dyscontrol, metacognitive beliefs (e.g., thoughts of the “helpfulness” of rumination), and maladaptive coping across time (Hoffart et al., 2022). Therefore, inhibition (abstaining from autopilot responses) and working memory (WM; mentally attending to and altering information in real-time) problems might contribute to worse responses to MEMI than SM. Issues with set-shifting (fluently switching between distinct cognitive modes) and verbal fluency (producing apt thematic words under a time limit; Renna et al., 2018; Zainal & Newman, 2022a) could likewise prevent individuals from strengthening psychological flexibility and related processes that could enhance emotion regulation during treatment.

Accordingly, the goal of the present study was to use various ML algorithms to evaluate which DMHI app (i.e., MEMI vs. SM) would work best for which person with GAD and which theory-based predictors predicted a better outcome with the MEMI vs. SM. Our study offers novel contributions in several ways. First, inquiring into which GAD sufferers benefitted from a specific DMHI has been a neglected research question (McDevitt-Murphy et al., 2018). Relatedly, by determining prescriptive predictors that informed optimal intervention assignment across different DMHIs, we extended the precision psychiatry literature that primarily examined prognostic predictors within and between specific face-to-face psychotherapies (Aafjes-van Doorn et al., 2021). Prescriptive predictors refer to variables that contribute to heterogeneity of treatment effects (Wester et al., 2022). Second, our mostly self-report pre-treatment predictor set could be practical to implement in routine care settings. Our method is an alternative approach to most prior ML-based treatment predictor studies intended to build clinical decision-making support tools with biomarker or neuroimaging data (e. g., Tymofiyeva et al., 2019). Although the potential of biomarker and neuroimaging techniques is undeniable, their practical application in outpatient and hospital environments is constrained by limited availability and high expense.

Our study was a secondary analysis of a DMHI RCT for GAD that primarily evaluated efficacy using an intention-to-treat approach (Zainal & Newman, 2023). The primary RCT findings most relevant for the current study were that the MEMI significantly outperformed the SM in reducing pre-treatment to one-month follow-up (pre-1MFU) GAD severity, perseverative cognitions (Cohen’s |d| = 0.393–0.394), trait mindfulness (d = 0.303), and EF (|d| = 0.280) measured dimensionally. Herein, we evaluated the following pre-treatment variables as potential prescriptive predictors: EF, GAD severity, trait mindfulness, perseverative cognitions, and sociodemographic variables. Building on the primary RCT that determined differential treatment efficacy on continuous outcomes, we examined how these variables would function as prescriptive predictors of pre-1MFU clinically reliable improvement (a categorical outcome; Blampied, 2022) in GAD severity, trait perseverative cognition, mindfulness, and EF. First, we expected that the MEMI but not SM would significantly yield pre-1MFU clinically reliable improvement in GAD severity, global perseverative cognition, trait mindfulness, and EF. Second, we hypothesized that ML models would produce moderate-to-high-accuracy predictive models of prescriptive predictors (Hypothesis 2). Lastly, we tested the prediction that higher pre-treatment GAD severity, trait perseverative cognition, mindfulness, and EF would predict more reliable change in all of these pre-1MFU outcomes for the MEMI than SM (Hypothesis 3).

2. Method

2.1. Participants

The present RCT was preregistered (NCT04846777 on ClinicalTrials.gov). Table 1 summarizes sociodemographic and clinical descriptive statistics. Participants (N = 110) averaged 20.80 years of age (SD = 5.41, range = 18–52). Further, 86.67% were women, 13.63% were men, and one person declined to disclose their gender identity. Also, 64.55% identified as White, and the rest as African American (5.45%), American/Pacific Islander (1.82%), Asian or Asian American (13.63%), Hispanic (7.27%), Native American/Pacific Islander (1.82%), another race (5.45%), or declined to disclose (0.91%). Although 16% attained a college or postgraduate degree, the remaining 84% of participants achieved a high school education.

Table 1.

Comparison of sociodemographic and clinical variables across intervention arms

	MA (n = 68)		SM (n = 42)		p
Continuous variables	M	(SD)	M	(SD)
Age (in years)	20.53	(3.91)	21.24	(7.24)	.51
14-item GAD-Q-IV score	9.52	(2.10)	9.94	(1.96)	.30
Perseverative cognitions
Total score	17.07	(4.36)	18.02	(4.27)	.26
Loss of controllability	2.20	(0.89)	2.41	(1.05)	.26
Preparing for the future	3.32	(1.04)	3.52	(1.03)	.32
Expecting the worst	2.78	(1.32)	2.88	(1.07)	.66
Searching for causes and meaning	3.14	(1.27)	3.48	(1.24)	.16
Dwelling on the past	3.80	(0.87)	3.84	(0.84)	.78
Thoughts discordant with ideal self	1.84	(1.00)	1.88	(0.98)	.84
Trait mindfulness
Total score	14.25	(2.30)	14.11	(2.11)	.75
Observing	3.22	(0.67)	3.28	(0.65)	.63
Describing	2.70	(0.80)	2.56	(0.82)	.39
Awareness	2.75	(0.68)	2.66	(0.69)	.49
Nonjudgment	3.00	(0.89)	3.01	(0.87)	.94
Nonreactivity of inner experiences	2.57	(0.62)	2.59	(0.61)	.88
Executive functioning
Inhibitory dyscontrol (s)	43.50	(9.64)	43.87	(7.37)	.83
Set-shifting deficits (s)	49.59	(12.61)	51.02	(11.42)	.55
Verbal fluency	112.87	(18.59)	115.14	(22.99)	.57
Treatment expectations
Credibility	6.00	(1.39)	5.72	(1.58)	.34
Expectancy	43.46	(17.33)	44.29	(18.13)	.31
Categorical variables	n	(%)	n	(%)
Gender orientation					.85
Women	10	(14.71)	5	(11.90)
Men	57	(83.82)	37	(88.10)
Declined to disclose	1	(1.47)	–	–
Race					.99
White Caucasian	44	(64.71)	27	(64.29)
Asian or Asian American	11	(16.18)	4	(9.52)
Hispanic	3	(4.41)	5	(11.91)
African American	5	(7.35)	1	(2.38)
Another race	4	(5.88)	2	(4.76)
Declined to disclose	1	(1.47)	0	(0.00)

Note. GAD-Q-IV = Generalized Anxiety Disorder Questionnaire–Fourth version; MA = mindfulness app; SM = self-monitoring app. Univariate linear regressions were used for continuous variables. Univariate logistic regressions were used for categorical variables.

Participants diagnosed with GAD based on the Anxiety and Related Disorders Interview Schedule for DSM-5 (ADIS-5; American Psychiatric Association, 2013; Brown & Barlow, 2014) were randomized to MEMI (n = 68) vs. SM (n = 42). Randomization was carried out with the Microsoft Excel randomization function programmed into Qualtrics with the insertion of the appropriate treatment video played toward the end of the baseline visit after completing all pre-treatment assessments. Intervention arms were assessor-blinded, i.e., the intervention assignment was concealed from assessors at all study visits. During the baseline visit, the experimenter left the physical room (pre-COVID-19) or asked participants to switch off/mute their Zoom audio/video before they clicked on the Qualtrics link to play the appropriate intervention arm video (during COVID-19). Table 1 shows no statistically significant differences in age, race, GAD severity, trait perseverative cognition, mindfulness, EF, and treatment expectation between intervention arms at baseline (p values =.26 to.95). These variables and treatment condition also did not significantly predict attrition (p = .10 to .99). This non-significant difference indicated the success of the randomization and class balance of baseline scores of potential prescriptive predictors between arms. Fig. 1 illustrates the CONSORT flowchart depicting participant enrollment and advancement.

Figure 1.

CONSORT diagram

Note. CONSORT = Consolidated Standards of Reporting Trials.

2.2. Measures

2.2.1. Intervention outcome

We evaluated reliable change from pre-1MFU (Jacobson & Truax, 1991; McGlinchey et al., 2002) in the following measures: (1) GAD severity using the Generalized Anxiety Disorder Questionnaire–Fourth version (GAD-Q-IV; Newman et al., 2002) total score; (2) trait perseverative cognition using the total scores of the Perseverative Cognitions Questionnaire (PCQ; Szkodny & Newman, 2019); (3) trait mindfulness using the total scores of the Five Factor Mindfulness Questionnaire (FFMQ; Baer, 2006); and (4) EF composite using average standardized scores of inhibitory dyscontrol (Stroop, 1935), set-shifting (Delis et al., 2001), verbal fluency (Borkowski et al., 1967), WM (Wechsler, 2008), and EF errors on inhibition, set-shifting, and verbal fluency tests detailed in the next paragraph. In addition, the domains/facets, example items, and psychometric properties of the outcome and potential predictor measures are detailed in Appendix A of the online supplemental materials (OSM). The current study used binary outcomes (i.e., reliable improvement) on these measures to replicate and extend the primary analyses that examined continuous outcomes (Zainal & Newman, 2023) and to generate area under the receiver operating characteristic curve (AUC) and related metrics when evaluating ML model performance. Further, reliable improvement is a more stringent criterion than dimensional symptom severity that is essential to examine as the outcome to understand predictors of clinically significant change.

2.2.2. Potential prescriptive predictors

All potential prescriptive predictors were variables measured at pre-treatment. The 14-item GAD-Q-IV (Newman et al., 2002) measured GAD severity. The 45-item PCQ (Szkodny & Newman, 2019) assessed six factors of repetitive negative thinking (dwelling on the past, expecting the worst, lack of controllability, thoughts discrepant with the ideal self, preparing for the future, and searching for causes and meanings). The 39-item FFMQ (Goldberg et al., 2016) measured five trait-level mindfulness domains (attention to inner and outer events, non-judgment, non-reactivity to inner experiences, describing/labeling of internal events, and acting with awareness). The color-word interference test from the Delis-Kaplan Executive Functioning System (Delis et al., 2001; Stroop, 1935) measured inhibition and set-shifting EF domains. The composite score of the digit span (DS) forward, DS backward, DS sequencing, and letter-number sequencing (Egeland, 2015) on the Wechsler Adult Intelligence Scale–Fourth Edition (Wechsler, 2008) indexed WM. The Controlled Oral Word Association Test (Borkowski et al., 1967; Delis et al., 2001) measured verbal fluency ability. We also included demographic variables (i.e., age, gender, and race/ethnicity) as possible important prescriptive predictors.

2.3. Procedures

Participants who met GAD criteria on the ADIS-5 (Brown & Barlow, 2014) semi-structured interview received the MEMI or SM at baseline. As detailed in the primary study (Zainal & Newman, 2023), participants received prompts five times daily (about 9 am, noon, 3 pm, 6 pm, and 9 pm) for the next 14 days to engage in mindfulness or self-monitoring strategies depending on their condition (detailed below and in Appendix A). MEMI participants regularly practiced these mindfulness skills during the intervention: open monitoring, attending to small moments, slowed, rhythmic, diaphragmatic breathing, and non-judgmental acceptance. SM participants did not receive exposure to or learn any mindfulness techniques but completed instructions to notice their thoughts and how distressing they were (detailed below). These prompts directed participants to engage in brief, one-minute mindfulness or self-monitoring exercises tailored to their assigned group. Participants had the flexibility to customize these prompts within a two-hour time frame (e.g., 8 am to 10 am, 11 am to 1 pm, etc.) to accommodate their individual schedules to minimize disruption and user burden. These adjustments were generally consistent among all participants. Similar to prior EMI research (e.g., Folkersma et al., 2021; Rauschenberg et al., 2021), no participants in our study spontaneously complained that the one-minute self-assessments and exercises disturbed their daily routine.

2.4. Mindfulness ecological momentary intervention (MEMI)

In the MEMI, participants were presented with a standardized video explaining the principles of evidence-based MBI protocols (Kabat-Zinn, 1990). MEMI participants were introduced to mindfulness to empower chronic worriers with the skills of open monitoring and attending to subtle moments. Participants also learned slow, rhythmic, diaphragmatic breathing. Subsequently, participants learned the practice of non-judgmental acceptance, mindful observing, and non-reactivity, paralleling components in mindfulness-based cognitive therapy (MBCT; Segal et al., 2002). Each MEMI participant received positive reinforcement on the significance and advantages of cultivating a regular mindfulness practice.

MEMI encouraged participants to engage in mindfulness exercises five times/day for 14 days. MEMI participants were given the following directives at each prompt: “Pay attention to your breathing. Breathe in a slow, steady, and rhythmic manner. Stay focused on the sensations of the air coming into your lungs and then letting it out. As you’re breathing, observe your experience as it is. Let go of judgments that do not serve you. Focus on the here and now. Attend to the small moments right now (e.g., reading a chapter, having a cool glass of water), as that is where enjoyment, peace, and serenity in life happen.” Participants self-assessed their levels of anxiety (“To what degree do you feel keyed up or on edge right now?”), depression (“To what degree do you feel depressed right now?”), and mindfulness (“To what extent are you experiencing the present moment fully?”) on a 9-point Likert scale, ranging from 1 (Not At All) to 9 (Extremely) before and following the reception of these prompts. Every MEMI prompt concluded with motivation to instill these abilities in the long run: “Remember that the cultivation of mindfulness is lifelong. The goal of therapy is to be your own therapist. Practice mindfulness between the prompts and after you have completed this study.”.

2.5. Self-monitoring app (SM)

In the SM, participants watched a video that defined self-monitoring as the heightened awareness of one’s feelings and thoughts. They received the message that simply monitoring feelings and thoughts and noting associated distress can support more constructive thinking patterns, and self-monitoring, by itself, might alleviate anxiety symptoms. The SM rationale drew inspiration from the approach employed in a recent, concise ecological momentary intervention (LaFreniere & Newman, 2016). Its development aimed to mirror the MEMI while removing hypothesized active mindfulness components, such as acceptance, attending to small moments, diaphragmatic breathing retraining, ongoing mindfulness exercises, and open monitoring. It hence did not refer to mindfulness. It did not guide participants to heighten their present-moment awareness; instead, it centered on monitoring their feelings and thoughts. Participants were not instructed to concentrate solely on their current activities, which could influence their emotional states. Whereas SM participants were directed to observe their feelings and thoughts, there was no guidance on accepting these thoughts and feelings as they emerged. It purposefully offered no advice on diaphragmatic breathing retraining. Its function did not involve inducing the calming sensations associated with diaphragmatic breathing. SM participants were not instructed to engage in self-monitoring between and after the prompts. The SM control differed from the notion that continual practice of mindfulness is meant for lifelong cultivation.

Rather than receiving the more extensive messages for continuous mindfulness practice, SM participants were given the following concise directive on five occasions daily: “Notice your thoughts and how distressing they may be.” Levels of state anxiety, depression, and mindfulness were assessed using identical 9-point Likert scale questions both before and after each SM prompt.

2.6. Data analyses

The Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD; Moons et al., 2015) guidelines were followed in the current study. All analyses were conducted with the nestedcv R package (Lewis et al., 2023). Multiple imputation with the predictive mean matching algorithm was used to impute missing values (< 10% of the data set; van Buuren, 2007). We evaluated three ML algorithms (logistic regression, support vector machine (SVM)– radial kernel, and random forest (RF)) with all stated predictor variables. The logistic regression algorithm using the logistic link function with maximum likelihood estimation was employed as a benchmark ML algorithm. SVM and RF have several benefits that make them appropriate for the current study. SVM tries to maximize generalizability (i.e., model performance accuracy), is suitable for data sets with a relatively high number of predictors, and is robust to anomalies (Dreiseitl et al., 2001). Likewise, the RF algorithm, an ensemble of decision trees with stopping rules (Cawley & Talbot, 2010), is efficient computationally at managing diverse potential predictors (Kern et al., 2019) and maximizes differentiation of significant predictors between classes (cf. more details in Appendix B and Loh, 2011). Moreover, the RF and SVM algorithms are appropriate for small sample sizes. Studies have shown that using class balancing (detailed below) enhances RF prediction with small sample sizes (Han et al., 2021) and a high-dimensional predictor set relative to the total number of observations (Qi, 2012). A simulation study also showed that small sample sizes (n = 50–100) with 10–30 potential predictors could attain high cross-validated accuracy estimates with SVM (e.g., ranging from 93–95%; Jiang et al., 2008). RF and SVM are collectively suitable and widely accepted for intervention outcome prediction purposes with comparatively small data sets (Kokol et al., 2022).

The present study focused on ML models that included prescriptive predictors of optimization to MEMI or SM. We identified prescriptive predictors of optimization based on good practice recommendations by simulation and empirical studies of logistic regression and other more complex ML models (Petkova et al., 2019), such as RF (Laber & Zhao, 2015) and SVM (Song et al., 2015; Zhao et al., 2012; refer to Appendices B to C). Continuous predictors were standardized to have a mean of 0 and standard deviation of 1. Categorical predictors (race, gender) were dummy-coded using one-hot encoding.

We tested all ML models using nested 10-fold cross-validation (10 F-CV), which minimizes biased estimates of the true error (Varma & Simon, 2006). Nested 10 F-CV divides the dataset into outer and inner folds (Cawley & Talbot, 2010), using the configuration of 10 outer and 10 inner folds. The inner fold cross-validation is employed to fine-tune optimal hyperparameters for the models. Subsequently, the model was trained on the entirety of the outer training fold and evaluated using the data that was withheld from the outer fold. This process was iterated for all outer folds, and the aggregated predictions from the unseen test data across the outer folds were compared with the actual outcomes for the outer test folds. We also employed nested leave-one-out cross-validation (LOOCV) to test each model. In this procedure, we trained the model using all participants except one, reserved as the testing dataset in the outer fold (Cristianini & Shawe-Taylor, 1999). LOOCV could offer a more impartial estimation, particularly in light of the limited sample size. Further, nested 10 F-CV (including CV and LOOCV) has benefits over split-sample validation, including decreasing probability of model overfitting (due to dependence on a single or arbitrary sample split) and bigger sample size (due to using all observations as both training and test cases; James et al., 2013). All ML analyses were adapted from published online tutorials (e.g., https://tinyurl.com/nestedcv; https://tinyurl.com/shapxai).

The AUC was the decision threshold to evaluate classification performance. Whereas AUC =− .5 indicates discrimination between groups at no better than chance, AUC ≥ .5 reflects adequate classification (i.e., minimizes the false positive rate and maximizes the false negative rate). Interpreting the AUC requires considering the question of interest. We used conventional heuristics to facilitate our interpretation: AUC = .50 means zero discrimination, .70–.79 indicates adequate discrimination, and AUC ≥ .80 denotes good discrimination. These guidelines also applied to two other decision thresholds: accuracy (proportion of correctly classified cases) and balanced accuracy (BAC; average of the sensitivity [model’s ability to identify a case] and specificity [model’s ability to identify a non-case]; Swets, 1988). Their values ranged from 0 to 1, wherein higher values reflected better performance.

Class imbalances in the outcome variable are essential to take into account when predicting binary outcomes. A high level of class imbalance may bias ML outcomes by weighting them toward the majority class (i.e., higher frequency outcome). As there was a class imbalance in the intervention outcomes, we applied the synthetic minority over-sampling technique (SMOTE; Chawla et al., 2002). The SMOTE approach undersamples the high-frequency class and oversamples the low-frequency class to generate a more balanced structure for the clinical outcomes, improving the prediction accuracy (Zulfiker et al., 2021).

Since the principal aim of ML is to optimize the performance of predictive models, identifying the subset of predictors that yields good classification performance may be beneficial. We originally had a total of 21 possible predictors that included the subfacets of mindfulness and perseverative cognition. However, given our sample size (N = 110) and established heuristics to have a minimum of 10 observations per candidate predictor (Jenkins & Quintana-Ascencio, 2020), we instead used the total scores on the mindfulness and perseverative cognition measures, which resulted in an initial 11 (age, sex, race, EF errors, inhibitory dyscontrol, set-shifting deficits, verbal fluency, WM, GAD severity, trait mindfulness, trait perseverative cognitions) theory-driven predictors, and a final model with the top five most important predictors. Model performance metrics were calculated for both models using a recursive feature elimination method within a nested CV framework (Kuhn & Johnson, 2018). Feature importance analyses were carried out to identify the essential prescriptive predictors ranked in descending order of their predictive importance (Lewis et al., 2023). We determined that all final predictors were stable, appearing in all 10 outer folds (Appendix C Figs. S1–4).

Last, we examined predictor importance based on the model-agnostic kernel Shapley Additive Explanations (SHAP) values to facilitate interpretability to visualize the associations between predictors and differential intervention outcomes (Lundberg et al., 2020). The SHAP approach estimated the effect of participant-level predicted probability of reliable improvement of modifying a single predictor from its observed value to the sample average across all logical permutations of other potential predictors. The average SHAP value across all participants is 0, but the average absolute SHAP value informs relative predictor importance. We generated SHAP-based bee swarm plots, a vital step because RF and SVM are considered “black box” models that do not provide a straightforward interpretation of the predictor coefficients. SHAP plots communicate the possibly complex relationship between each value of a predictor and the probability of optimization to MEMI (vs. SM) to predict each clinical outcome. We created SHAP-based bee swarm plots for the top five prescriptive predictors of each distinct outcome in each final multivariable model.

3. Results

3.1. Differential treatment effects on reliable improvement in pre-1MFU outcomes

The MEMI yielded substantially higher reliable improvement in GAD severity (n = 49, 72.06% vs. n = 24, 58.54%; χ²(df = 1) = 9.30, p = .002) and trait perseverative cognitions (n = 39, 57.4% vs. n = 11, 26.2%; χ²(df = 1) = 10.17, p = .001) from pre-treatment to 1MFU than the SM. However, the MEMI (n = 24, 35.5%) vs. SM (n = 10, 23.8%) yielded no significant differential treatment efficacy on reliable improvement in trait mindfulness (χ²(df = 1) = 1.60, p = .205). Likewise, MEMI (n = 25, 59.5%) vs. SM (n = 29, 42.6%) had no significant differential treatment efficacy on reliable improvement in EF (χ²(df = 1) = 2.96, p = .085). Therefore, we found inconsistent support for Hypothesis 1.

3.2. Prescriptive predictors of pre-1MFU reliable improvement in GAD severity

SVM LOOCV was the best-performing initial model (AUC = .817, accuracy = .800, BAC = .795; Table 2 Part A). The final model also performed well (AUC = .834, accuracy = .795, BAC = .775). Predictors of optimization to the MEMI were higher GAD severity (#1), higher trait perseverative cognition (#2), lower set-shifting deficits (#3), older age (#4), and stronger trait mindfulness (#5; Fig. 2).

Table 2.

Model performance of each predictive machine learning (ML) algorithm to predict optimization to the mindfulness app with the initial 11 predictors

ML Model	AUC	Accuracy	Balanced accuracy	Sensitivity	Specificity
A. Outcome: Generalized anxiety disorder (GAD) remission
Logistic regression - CV	0.806	0.786	0.764	0.651	0.876
Logistic regression - LOOCV	0.816	0.772	0.746	0.616	0.876
Random forest - CV	0.797	0.786	0.781	0.756	0.806
Random forest - LOOCV	0.817	0.819	0.814	0.791	0.837
Support vector machine - CV	0.817	0.772	0.762	0.709	0.814
Support vector machine - LOOCV	0.817	0.800	0.795	0.767	0.822
B. Outcome: Reliable improvement in trait perseverative cognitions
Logistic regression - CV	0.799	0.719	0.703	0.586	0.819
Logistic regression - LOOCV	0.808	0.719	0.707	0.621	0.793
Random forest - CV	0.816	0.744	0.741	0.724	0.759
Random forest - LOOCV	0.807	0.729	0.726	0.701	0.750
Support vector machine - CV	0.808	0.724	0.721	0.701	0.741
Support vector machine - LOOCV	0.803	0.729	0.727	0.713	0.741
C. Outcome: Reliable improvement in trait mindfulness
Logistic regression - CV	0.736	0.644	0.633	0.750	0.517
Logistic regression - LOOCV	0.735	0.652	0.639	0.778	0.500
Random forest - CV	0.740	0.644	0.639	0.694	0.583
Random forest - LOOCV	0.738	0.629	0.624	0.681	0.567
Support vector machine - CV	0.752	0.674	0.671	0.708	0.633
Support vector machine - LOOCV	0.737	0.644	0.642	0.667	0.617
D. Outcome: Reliable improvement in executive functioning composite
Logistic regression - CV	0.878	0.797	0.790	0.859	0.721
Logistic regression - LOOCV	0.877	0.797	0.791	0.847	0.735
Random forest - CV	0.880	0.830	0.826	0.859	0.794
Random forest - LOOCV	0.879	0.843	0.843	0.847	0.838
Support vector machine - CV	0.886	0.843	0.843	0.847	0.838
Support vector machine - LOOCV	0.873	0.837	0.838	0.824	0.853

AUC = area under the receiver operating characteristic curve; PPV = positive predictive value; NPV = negative predictive value; CV = nested cross-validation; LOOCV = leave-one-out nested cross-validation.

Figure 2.

Top predictors of being optimized to mindfulness ecological momentary intervention (MEMI) for increased probability of pre-1MFU generalized anxiety disorder (GAD) remission in the final multivariable model

Note. Pre-1MFU = pretreatment to one-month follow-up; SHAP = SHapley Additive exPlanations. On the y-axis, the number denotes the relative importance of the prescriptive predictor in relation to the clinical outcome of interest. For each predictor, if each SHAP-based bee swarm plot turns darker in color from left to right on the x-axis, higher values of that predictor indicate a stronger positive correlation with the outcome and vice versa. On the x-axis, the dot’s position denotes the effect the predictor has on the model’s prediction for a specific participant. When multiple dots emerge at the same x position, they accumulate to display density.

3.3. Prescriptive predictors of pre-1MFU reliable improvement in perseverative cognitions

RF CV was the best-performing initial model (AUC = .816, accuracy = .744, BAC = .741; Table 2 Part B). The final model also performed well (AUC = .821, accuracy = .749, BAC = .739). The predictors of optimization to the MEMI were higher trait perseverative cognitions (#1), being White (#2), older age (#3), more inhibitory dyscontrol (#4), and lower verbal fluency (#5; Fig. 3).

Figure 3.

Top predictors of being optimized to mindfulness ecological momentary intervention (MEMI) for increased probability of pre-1MFU reliable improvement in global perseverative cognitions in the final multivariable model

Note. Pre-1MFU = pretreatment to one-month follow-up; SHAP = SHapley Additive exPlanations. On the y-axis, the number denotes the relative importance of the prescriptive predictor in relation to the clinical outcome of interest. For each predictor, if each SHAP-based bee swarm plot turns darker in color from left to right on the x-axis, higher values of that predictor indicate a stronger positive correlation with the outcome and vice versa. On the x-axis, the dot’s position denotes the effect the predictor has on the model’s prediction for a specific participant. When multiple dots emerge at the same x position, they accumulate to display density.

3.4. Prescriptive predictors of pre-1MFU reliable improvement in trait mindfulness

SVM CV was the best-performing initial model (AUC = .752, accuracy = .674, BAC = .671; Table 2 Part C). The final model had moderate-to-good performance (AUC = .770, accuracy = .667, BAC = .657). The predictors of optimization to the MEMI were higher trait mindfulness (#1), being White (#2), lower set-shifting deficits (#3), being women (#4), and lower verbal fluency (#5; Fig. 4).

Figure 4.

Top predictors of being optimized to mindfulness ecological momentary intervention (MEMI) for increased probability of pre-1MFU reliable improvement in global trait mindfulness in the final multivariable model

Note. Pre-1MFU = pretreatment to one-month follow-up; SHAP = SHapley Additive exPlanations. On the y-axis, the number denotes the relative importance of the prescriptive predictor in relation to the clinical outcome of interest. For each predictor, if each SHAP-based bee swarm plot turns darker in color from left to right on the x-axis, higher values of that predictor indicate a stronger positive correlation with the outcome and vice versa. On the x-axis, the dot’s position denotes the effect the predictor has on the model’s prediction for a specific participant. When multiple dots emerge at the same x position, they accumulate to display density.

3.5. Prescriptive predictors of pre-1MFU reliable improvement in EF composite

SVM CV was the best-performing initial model (AUC = .886, accuracy = .843, BAC = .843; Table 2 Part D). The final model also performed well (AUC = .886, accuracy = .804, BAC = .797). The predictors of optimization to the MEMI were higher trait perseverative cognitions (#1), stronger trait mindfulness (#2), more inhibitory dyscontrol (#3), lower GAD severity (#4), and being women (#5; Fig. 5). Whereas Hypothesis 2 received full support in examining ML models across all four outcomes, Hypothesis 3 had partial support.

Figure 5.

Top predictors of being optimized to mindfulness ecological momentary intervention (MEMI) for increased probability of pre-1MFU reliable improvement in executive functioning composite in the final multivariable model

Note. Pre-1MFU = pretreatment to one-month follow-up; SHAP = SHapley Additive exPlanations. On the y-axis, the number denotes the relative importance of the prescriptive predictor in relation to the clinical outcome of interest. For each predictor, if each SHAP-based bee swarm plot turns darker in color from left to right on the x-axis, higher values of that predictor indicate a stronger positive correlation with the outcome and vice versa. On the x-axis, the dot’s position denotes the effect the predictor has on the model’s prediction for a specific participant. When multiple dots emerge at the same x position, they accumulate to display density.

4. Discussion

Overall, MEMI outperformed SM to generate pre-1MFU reliable improvement in GAD severity and perseverative cognitions, but not trait mindfulness and global EF for participants with GAD. The more remarkable pre-1MFU reliable improvement rate in GAD severity and perseverative cognitions in the MEMI vs. SM was comparable to other DMHIs for GAD, such as a CBT app provided across six weeks (Miller et al., 2021). Plausibly, the non-significant differential treatment efficacy on reliable improvement in trait mindfulness and global EF might be due to the brief nature of the MEMI. Higher-intensity MBIs (Zainal & Newman, In Press) and alternative interventions that explicitly targeted EF, such as EF training (Kenworthy et al., 2014, pp. 12161) and metacognitive therapy (Doig et al., 2021), could be necessary to produce reliable improvement in trait mindfulness and global EF.

Consistent with Hypothesis 2, our strong AUC values (i.e., >0.70) for the final multivariate models were noteworthy. They aligned with other ML studies predicting therapy outcomes for diverse mental illnesses, such as body dysmorphic disorder (Curtiss et al., 2022), depression (Pearson et al., 2019), and social anxiety disorder (Hoogendoorn et al., 2017). We extended the limited but increasing studies that evaluated for whom MEMIs and other DMHI apps were appropriate (e.g., for highly ruminative adolescents; Webb et al., 2022).

Noteworthy was how GAD severity consistently emerged as a top predictor, but in different directions: Although higher GAD severity predicted better chances of GAD improvement, lower GAD severity predicted more EF improvement. However, it is unclear why these contradictory findings occurred, and coefficient direction differences could be due to variations in the targeted outcomes. The former finding concurred with evidence that higher initial GAD severity predicted greater improvement magnitudes following CBT among adults with GAD (Newman & Fisher, 2010; Wetherell et al., 2005). Perhaps adults with higher GAD severity exhibited increased treatment motivation, which promoted more symptom alleviation. At the same time, it is possible that the latter outcome occurred because less symptomatic patients with GAD were naturally more adept and versatile in their thought and action repertoires (Zainal & Newman, 2022a), and exploiting such assets might have encouraged more EF enhancements. These findings offer hope to clinicians treating severely anxious GAD patients by suggesting the value of a self-guided MEMI to augment treatment effects based on precision psychiatry methods. Harnessing optimal precision psychiatry methods is critical because about 10, 000–20,000 DMHIs exist (Bautista & Schueller, 2022), and those with GAD gravitate toward apps emphasizing self-monitoring and mindfulness skills (Nardi et al., 2022). Further, findings partially supported Hypothesis 3 (i.e., those with higher perseverative cognitions, trait mindfulness, and EF may have gained more from the MEMI vs. SM).

Why did higher trait perseverative cognitions, inhibitory dyscontrol, and verbal dysfluency predict optimization to the MEMI for most, if not all, outcomes (i.e., alleviating GAD severity, trait perseverative cognitions, improving mindfulness, and EF)? Our findings aligned with meta-analyses indicating that greater baseline severity correlated with an enhanced likelihood of treatment response in various psychiatric disorders (Andersson et al., 2019). Compromised inhibition and verbal fluency (Zainal & Newman, 2021a; Zainal & Newman, 2021b, 2022b) and excessive rumination (Kim & Newman, 2023) have also been shown to predict, correlate, and be a consequence of GAD symptoms. These pre-treatment attributes may thus have provided individuals with GAD an increased opportunity to enhance their mindfulness skills over time, potentially improving their chances of better intervention outcomes via the MEMI over SM (cf. compensation model; Cohen & DeRubeis, 2018). Individuals with weaker inhibitory control and verbal fluency skills may have benefitted further from MEMI’s structured approach, which included well-organized instructions, among other elements (Hybel et al., 2017). The MEMI repeatedly instructed attentional focus and acceptance of current life circumstances, which could have promoted EF by engendering better emotion regulation skills over time (Im et al., 2021; Schumer et al., 2018). Further research is essential for drawing definitive conclusions.

Of note, other aspects of our findings buttressed the capitalization model (Cheavens et al., 2012; Probst et al., 2022). Specifically, stronger set-shifting and trait mindfulness predicted higher odds of reliable improvement in GAD and trait mindfulness. Leveraging those preexisting behavioral strengths in DMHIs, including brief MEMIs for GAD, might have enhanced their efficacy. The MEMI may have further nurtured existing skills (e.g., capacity to switch thinking modes flexibly, detached observing, nonreactivity to inner experiences) that aligned well with patients’ aptitude.

Some demographic variables emerged as significant prescriptive predictors. Older age predicted being optimized by the MEMI vs. SM for reliable improvements in GAD and perseverative cognitions. This outcome may have been partly due to evidence that younger (vs. older) adults with GAD benefitted more from higher-intensity and more extensive face-to-face psychotherapies or pharmacotherapies (refer to a recent meta-analysis by Carl et al., 2020a). Women were likelier to experience reliable improvement in trait mindfulness and EF than men with MEMI. These findings could have been due to our consistent result that patients with higher tendencies toward perseverative cognition reaped more benefits regarding improvement in various outcomes and prior findings that women displayed increased propensities toward repetitive thinking (Nolen-Hoeksema et al., 2008). Our trial enrolled adults experiencing GAD, which is characterized mainly by persistent negative and future-oriented thoughts. The MEMI, emphasizing present-moment attention and metacognitive stance, could help adults grappling with GAD to shift to adopt alternative adaptive and flexible thinking habits (Zainal & Newman, In Press). Increased versatility via the MEMI might have enhanced organizational, planning, and other EF skills more for women. The MEMI was also more likely to improve trait perseverative cognitions and mindfulness for White individuals than non-White individuals. Cultural differences, such as more innate mindfulness philosophies and practices embedded in other cultures, might account for this difference. Such findings concurred with meta-analytic data that although MBIs were statistically more efficacious than active controls, those comparative effect sizes were modest for ethnic minorities when contrasted with cultural majority populations (Sun et al., 2022). Cultural adaptations of MEMIs may serve to better cater to the requirements of people of color. However, those adaptations based on lived experiences (e.g., structural racism) require rigorous evaluation given complex intersectionality factors (e.g., economic circumstances) and inconsistencies in the literature about the value of culturally adapting psychotherapies (Cougle & Grubaugh, 2022).

Our study had several limitations that merit attention. First, our prescriptive predictor models need larger follow-up replication studies to assess their applicability to different samples with various anxiety disorders (Luedtke et al., 2019). A larger sample size has the potential for more precise scalable and efficacious smartphone DMHIs targeting GAD, leveraging ML techniques, and tailoring treatments to individual characteristics. However, ML-based multivariable predictor models using RF and SVM algorithms with nested CV permit small sample sizes such as ours for building treatment prediction models, as they can address overfitting and class imbalance concerns (Chen & Jeong, 2007; Guo et al., 2010; Han et al., 2021). Future DMHI studies should explore applying ML to prospective observational designs, aggregated RCT data, and pragmatic trial designs (Kessler, 2018). Second, subsequent studies should examine the replicability of our ML models without SMOTE. Third, future studies should use a broader predictor set to determine unique DMHI outcome predictors. Fourth, readers should be cautious when drawing causal inferences when interpreting predictor-outcome relations (Leeuwenberg et al., 2022). Fifth, future studies should recruit more culturally diverse samples regarding race/ethnicity and sex. Sixth, we originally hoped to have 1:1 equipoise randomization, but unfortunately, we had uneven randomization. Lastly, ML models should be used to guide treatment planning instead of providing strict recommendations.

However, study strengths include novelty, RF and SVM suitability for small, imbalanced datasets, identification of differential predictors for two DMHIs (Boehmke & Greenwell, 2019), and good model performance. Also, ML could reduce biases due to multicollinearity (Hawes et al., 2022). Moreover, prescriptive predictors of differential DMHI outcomes were identified to determine persons with GAD most likely to benefit from MEMI over SM. This approach offered more clinically actionable results than identifying prognostic predictors (variables predicting outcomes regardless of treatment type). Together, these prescriptive predictors may serve as a means to impartially identify which patients with GAD would be most likely to benefit from MEMI vs. SM before they use the MEMI. Providing the free MEMI to optimized clients on a waitlist for more expensive and intensive psychotherapies might also improve their clinical care. It might also augment intensive face-to-face psychotherapies (Newman et al., 2014).

Some ethical aspects warrant consideration. One of our study strengths was that we used the TRIPOD guidelines to facilitate transparency in our decision rules in how we built our prescriptive predictor models. Through open science initiatives, it is also crucial to conscientiously address data privacy concerns (Walsh et al., 2018). The current study obtained consent for the use of patients’ deidentified data for clinical research and secondary analyses. We also ensured that training and testing data differed to prevent data leakage. However, our 35% diversity in the sample likely created generalization issues to more diverse groups (Tay et al., 2022). Future studies should recruit more diverse samples.

If future studies replicate our pattern of results and with these ethical considerations in mind, some clinical implications merit attention. The MEMI could benefit clients diagnosed with principal or comorbid GAD seeking treatment in busy outpatient settings such as college counseling centers and academic medical centers with long waitlists and heavy client loads (Xiao et al., 2017). Clinicians could collect client data from self-report and brief neurocognitive tests, then input them into the algorithm using the top five predictive factors model to predict MEMI effectiveness as part of a prescriptive calculator (Webb et al., 2022; Webb et al., 2021). When gathering pre-treatment data, clinicians should omit assessing variables deemed unimportant as prescriptive predictors in the current algorithm (EF errors and WM). Whereas some GAD clients may benefit notably from the MEMI, others may not experience the same optimization level. Still, the MEMI could offer more tools in the repertoire of emotion regulation skills for others predicted not to receive similar optimization. Offering a low-intensity MEMI to clients likely to benefit from it could aid in their transition to more intensive therapist-led cognitive-behavioral and related therapies, which involve homework to enhance their skills and mindset. Earlier actuarial research indicated that more complex cases should begin with more intensive psychotherapies (Delgadillo et al., 2017). In summary, providing higher-intensity psychotherapies to clients unlikely to benefit from the MEMI within a stratified model can enhance patient-centered care by offering this less common and costlier intervention to those who would gain the most (Herzog & Kaiser, 2022).

Data availability

The authors do not have permission to share data.