Individual differences associated with treatment adherence and transfer effects following gamified web-based cognitive control training for repetitive negative thinking

Abstract

Recent research suggests beneficial effects of cognitive control training (CCT) on repetitive negative thinking (RNT), a key risk factor for internalizing symptomatology. However, relatively little is known regarding predictors of adherence to internet-delivered CCT as well as moderators of treatment effects for this intervention. Answering these questions could improve efficiency of clinical implementation of CCT as an eHealth intervention. The current pre-registered single-arm trial set-out to address these questions using a web-based gamified CCT procedure based on the adaptive Paced Auditory Serial Addition Task. Participants (N = 382) entered the internet-based study, where we observed considerable drop-out during the assessment phase and the first training sessions. Emotional stability and resilience emerged as predictors for deciding not to commence the intervention. Drop-out throughout the course of CCT was explained by age, emotion regulation-, and personality factors. We used latent profile analysis, a probabilistic modeling approach, to identify clusters of participants (User Profiles) based on indicators of baseline cognitive- and emotional functioning, training progress, and user experience. We obtained three User Profiles, reflecting low-, moderate-, and high-risk status. Effortful control, emotion regulation, internalizing symptomatology, resilience, and emotional stability played a central role in these User Profiles. Interestingly, User Profile predicted training related cognitive gains, as well as effects of CCT on anxiety- and stress symptoms, and reappraisal. Our findings suggest that CCT is most effective for the moderate- and high-risk groups. In addition, the high-risk group would likely benefit from a more intensive training procedure or repeated administration of the training procedure over time to foster long-term retention of training related gains.

Highlights

• •Web-based cognitive control training (CCT) is a promising intervention for RNT.
• •We identified multiple predictors of decision to not participate to CCT or drop-out.
• •We identified 3 distinct User Profiles reflecting risk for internalizing disorders.
• •User Profile predicted training related cognitive gains and emotional transfer.
• •Taking this into account will facilitate optimization of intervention administration.

1. Introduction

Recent years have shown an increase in cognitive remediation studies aimed at reducing risk for the development of internalizing symptomatology (e.g., Hagen et al., 2020; Siegle et al., 2007). Internalizing disorders have been conceptualized as having a focus on inner emotions, including symptoms of anxiety, stress and depression (Achenbach, 1991). One cognitive remediation strategy that has developed to become a particularly promising intervention in this context, is cognitive control training (CCT; for a review, see Koster et al., 2017). CCT refers to the use of computerized tasks aimed at remediating executive function impairments (e.g., difficulties updating information in working memory, shifting, or inhibition of prepotent responses; Miyake et al., 2000) to allow flexible adaptation of one's thoughts and behavior as a function of one's goals (Braver et al., 2002; Miller and Cohen, 2001).

Multiple studies suggest that cognitive control is causally involved in emotion regulation processes (e.g., Cohen and Mor, 2018; Schweizer et al., 2013). In particular, cognitive control impairments have been linked to repetitive negative thinking (RNT; for recent reviews, see Joormann and Stanton, 2016; Mor and Daches, 2015), which is considered a transdiagnostic risk factor for the occurrence and maintenance of anxiety- and mood disorders (Aldao et al., 2010; Nolen-Hoeksema, 2000). In this context, it has been suggested that difficulties disengaging from irrelevant negative information in working memory may result in perseveration of negative thought processes (Koster et al., 2011; Watkins and Roberts, 2020), which is likely to induce and prolong one's negative mood state, and has shown to further deteriorate executive functioning (e.g., Philippot and Brutoux, 2008). As such, cognitive remediation strategies, such as CCT, could help to recruit resources useful for interrupting processes of RNT, thus decreasing vulnerability for anxiety- and mood disorders.

1.1. Cognitive control training targeting RNT

Beneficial effects of CCT procedures have been reported for different forms of RNT. For instance, Course-Choi et al. (2017) observed beneficial effects of CCT on worrying, where compared to a mindfulness only and a control condition, the combination of CCT with a mindfulness intervention resulted in continued improvement in RNT post-intervention. Moreover, multiple studies suggest beneficial effects of CCT on rumination (e.g., Cohen et al., 2015; Hoorelbeke et al., 2015; Hoorelbeke et al., 2016a; Siegle et al., 2007, Siegle et al., 2014; Swainston and Derakshan, 2018), as well as anxiety- and depressive symptomatology (e.g., Beloe and Derakshan, 2020; Calkins et al., 2015; Iacoviello et al., 2014; Morimoto et al., 2014). For instance, Siegle et al. (2007) reported the combination of CCT and treatment as usual (TAU) to be more effective than TAU in reducing rumination and depressive symptomatology in a sample of clinically depressed patients. In addition, previous work suggests that cognitive training gains may result in more distal emotional transfer effects via immediate effects of CCT on RNT (Hoorelbeke and Koster, 2017). Importantly, recent studies also showed that CCT can induce long-lasting transfer effects, reflecting that for some individuals remediating cognitive control impairments may successfully alter one's at-risk state for the development of internalizing symptomatology. For instance, Siegle et al. (2014) reported beneficial effects of CCT on number of intensive outpatient day-treatment visits over a period of one year follow-up in patients suffering from major depressive disorder (MDD). In addition, CCT has recently shown to reduce risk for recurrence of depression over a period of one year follow-up in a sample consisting of remitted depressed individuals (Hoorelbeke et al., 2021).

At the same time, however, the CCT literature also contains a number of inconsistent findings (for recent reviews, see Koster et al., 2017; Van den Bergh et al., 2018). For instance, Hotton et al. (2018) failed to observe beneficial effects of CCT on RNT or anxiety in a sample of high worriers. In addition, Van den Bergh et al. (2020) observed no additive effects of CCT in terms of self-reported RNT, and level of anxiety- or depressive symptoms in the context of a cognitive behavior therapy program targeting fear of failure. Similarly, in a healthy student sample Vervaeke et al. (2020) observed emotional transfer for self-reported anxiety levels, in absence of effects of CCT on RNT or depressive symptomatology. Such inconsistencies may at least partially be due to sample heterogeneity (e.g., community samples, MDD/RMD patients), where strong individual differences have been observed in terms of baseline features (e.g., cognitive impairments; Pu et al., 2018; Vicent-Gil et al., 2020) and treatment response (e.g., Hoorelbeke et al., 2021). Individual differences may be distinct from other factors contributing to variation across studies such as differences in administration modality (lab vs. internet), number of required sessions (varies from 1 to 10 in the literature), software versions (e.g., lab versions typically use multiple training tasks whereas internet protocols have used a single task), and demand characteristics (e.g., high variability in the level of reimbursement for participation across studies). Understanding the extent to which individual differences specifically contribute to observed variability in effects of CCT will thus help to clarify the robustness of treatment effects within a given sample.

1.2. Towards a multifactorial approach to moderation of training effects

Increased understanding of mechanisms and moderators of treatment effects would allow to better tune CCT for individuals with specific vulnerabilities to non-response (Koster et al., 2017). For instance, Siegle et al. (2014) reported that individuals who had increased CCT task engagement had greater reductions in RNT, suggesting that increased emphasis on task engagement could improve outcomes. Moreover, baseline level of cognitive functioning (Moshier and Otto, 2017), training task progress (i.e., improvement in CCT task performance throughout the training procedure; Hotton et al., 2018; Vanderhasselt et al., 2015), and training related cognitive gains (Hoorelbeke et al., 2015; Hoorelbeke and Koster, 2017; Peckham and Johnson, 2018) have also been linked to more beneficial effects of CCT on RNT. In addition, in the context of late-life treatment resistant depression, cognitive impairments at baseline have shown to predict greater improvement in depressive symptoms following CCT (Morimoto et al., 2016). Furthermore, multiple studies point towards the moderating role of level of RNT for treatment effects following CCT, although the direction of effects reported in the literature shows considerable inconsistency. For instance, Daches et al. (2015) found rumination to moderate training effects, where – using median split – high trait ruminators were less responsive to a single-session CCT procedure compared to low ruminators. That is, in contrast to low ruminators in whom a training-congruent effect was observed, Daches et al. (2015) reported a training-incongruent effect on inhibition for individuals reporting relatively higher levels of rumination. Using a different CCT procedure, Quinn et al. (2014) observed an opposite pattern of results, where effects of CCT on cortisol reactivity were only observed for participants showing elevated levels of rumination (+1 SD). In contrast, at −1 SD below the centered mean score of rumination, participants undergoing CCT did not significantly differ from the control group, suggesting that high trait ruminators were more responsive to CCT.

Another factor that has been suggested to impact effects of CCT (in combination with other neuromodulation techniques) is age (Brunoni et al., 2014; Segrave et al., 2014). As meta-analysis suggests that effects of cognitive training in the context of depression decrease with increasing age (Motter et al., 2016), such adjunctive approaches may be warranted. In addition, Motter et al. (2016) demonstrated that effects of cognitive training on depressive symptoms are unaffected by gender or ongoing pharmacological treatment.

Together, these findings indicate that an integrative approach accounting for predictors and potential moderators could help to improve potential CCT outcomes. Moreover, prior research has shown the group of individuals who may potentially benefit from CCT to be highly heterogeneous (e.g., Koster et al., 2017). This is particularly the case when implemented online as an eHealth intervention. Towards this end, in this manuscript we aimed to identify multifactorial User Profiles predicting treatment response via latent profile analysis.

Latent profile analysis allows to detect latent heterogeneity in samples based on a set of selected indicator variables (Hagenaars and McCutcheon, 2002). This technique operates under the assumption that unobserved clusters exist which can explain patterns of observed responses. As such, this probabilistic technique allows to identify and describe complex patterns of user characteristics for web-based CCT in a data-driven manner. Based on the literature reviewed above and considering task characteristics of the training procedure under investigation, an empirically and theoretically informed set of indicator variables was selected (e.g., based on level of RNT, symptomatology, cognitive functioning, user engagement).

In the context of intervention studies, it has been proposed that latent profile analysis allows identification of subgroups of individuals who – due to their shared characteristics – could benefit more from an intervention (Weller et al., 2020). This requires investigation of the relation between cluster membership and (change in) mental health outcomes. Latent profile analysis may also be informative to how an intervention could be optimally administrated, by identification of shared characteristics of less responsive subgroups of the population, for which modifications to the intervention or method of delivery may be necessary (e.g., following further research focused on modeling mechanisms of change during and following CCT; Hoorelbeke and Koster, 2017; Hoorelbeke et al., in press; Lass et al., 2021). Indeed, the discussed inconsistencies in moderators of training effects suggest that there may be merit in a more integrative approach, taking into account unobserved User Profiles, where latent profile analysis allows to explore how membership of a given cluster relates to treatment effects, which in turn may allow to ideally optimize intervention administration to account for baseline features.

1.3. Adherence to internet-delivered cognitive training

Treatment adherence for eHealth administration may also be improved. That is, previous studies suggest considerable drop-out from interventions that are fully administered online (for a review, see Christensen et al., 2009). A considerable portion of drop-out in these studies can be attributed to individuals who decide not to start the intervention after having initiated participation to the study. For instance, in an app-based cognitive training study of Arean et al. (2016), 58% of participants that were assigned to the gamified cognitive training procedure or to another eHealth component did not download the intervention app. In contrast, attrition rates are typically very low in laboratory CCT studies (e.g., Brunoni et al., 2014 (N = 2/40 dropouts); Calkins et al., 2015 (N = 2/50 following trial beginning); Segrave et al., 2014 (N = 1/20 dropouts); Siegle et al., 2014 (N = 4/27 dropouts)), or studies in which CCT was delivered online while pre- and follow-up assessments took place in a laboratory setting, including delivery of psychoeducation (e.g., Hoorelbeke and Koster, 2017 (N = 5/34 dropouts); Hoorelbeke et al., 2021 (N = 5/47 dropouts); Vervaeke et al., 2021 (N = 2/34 dropouts)).

Although multiple efforts have been made to prevent drop-out from online administered cognitive training procedures (e.g., using psychoeducation and gamification techniques, based on focus groups with end users; Vervaeke et al., 2018, Vervaeke et al., 2020), relatively few studies have examined predictors of adherence to online CCT interventions targeting RNT or mood- and anxiety symptoms. Using a fully app-based eHealth intervention, Arean et al. (2016) found higher baseline depressive- and anxiety symptoms to be associated with less use of online CCT in a sample with mild to moderate depressive symptomatology. Moreover, high severity of depressive symptoms at baseline predicted a lower likelihood of participants engaging in the first session of the CCT intervention (Arean et al., 2016).

1.4. Current study

Given the state of the literature, this pre-registered exploratory study (osf.io/cw4k3) used a two-step strategy. First, we wanted to model patterns of drop-out and identify predictors of drop-out in the context of CCT for RNT (Aim 1). For this purpose, we relied on an online variant of the adaptive Paced Auditory Serial Addition Task (aPASAT; Siegle et al., 2007), one of the most frequently used and well-established CCT procedures in the context of RNT (Koster et al., 2017). As to date, trajectories of attrition during web-based aPASAT training, and related predictors, remain to be investigated. For this purpose, in addition to considering drop-out that occurred during the course of the ten session CCT procedure, we also consider predictors of which individuals decided to not start the intervention after having completed the baseline assessment and having received psychoeducation regarding CCT (i.e., participants who discontinued participation after hearing about the intervention and before starting it).

Increased understanding of patterns of attrition of web-based aPASAT training is crucial for evaluation of the implementation potential of the current intervention as an eHealth intervention and may in the long-term contribute to further optimization of training administration. This study was promoted as an online intervention study targeting RNT, likely attracting individuals showing elevated levels of RNT, where given the transdiagnostic nature of this study we do not aim to distinguish between different types of RNT, among which worry and rumination. However, because we use an unselected sample with no inclusion criteria, our hypothesis is that individuals who opted to not engage with the intervention after having received psychoeducation regarding CCT may not view themselves as having problems that this intervention is suited to fix.

The second aim, and main focus of this study, was to model patterns of user characteristics for completers of the intervention and investigate how these relate to individual differences in response to CCT. That is, our second aim was to investigate complex patterns of moderation. Towards this end, we (a) identify multifactorial User Profiles, and (b) evaluate whether User Profile predicts change in indicators of cognitive- and emotional functioning following aPASAT training. In particular, based on prior studies (e.g., Hoorelbeke and Koster, 2017; Siegle et al., 2007, Siegle et al., 2014) we expected participants to improve on primary outcome measures cognitive functioning, self-reported RNT, and depressive-, anxiety-, and stress symptoms over time. In addition, previous studies reported beneficial effects of aPASAT training on broader indicators of functioning, among which other forms of maladaptive- (Hoorelbeke and Koster, 2017) and adaptive emotion regulation (Peckham and Johnson, 2018), and – related to this – resilience (Hoorelbeke and Koster, 2017), the ability to cope effectively upon facing adversity (Wagnild, 2009). Therefore, as exploratory outcome measures we included reappraisal and suppression, as alternative indicators for (mal)adaptive emotion regulation, and resilience.

Based on the literature, we selected a broad range of predictors for the decision not to participate and attrition during the course of CCT, and transfer effects following CCT, among which age (Brunoni et al., 2014; Motter et al., 2016), indicators of baseline cognitive functioning (Moshier and Otto, 2017), internalizing symptomatology (Arean et al., 2016), emotion regulation (Quinn et al., 2014), and closely related to this, resilience (Hoorelbeke et al., 2019; Kalisch et al., 2015). Due to the arithmetic nature of the aPASAT, fear of math and perceived math ability were also considered. User Profile was also informed by user experience as a potential indicator of task engagement and motivation (Siegle et al., 2014), as well as training task progress (Vanderhasselt et al., 2015). Finally, given that personality traits have been shown to moderate treatment adherence to other interventions (e.g., pharmacological, behavioral; Jerant et al., 2011; Kruisdijk et al., 2020; Molloy et al., 2014; Stilley et al., 2004) and have shown to place one at risk for RNT (Barańczuk, 2019), we also assessed emotional stability, extraversion, agreeableness, openness, and conscientiousness.

2. Material and methods

2.1. Participants

Participants were recruited via traditional and social media. In particular, we advertised this study in major Flemish newspapers and shared information regarding the study via a local radio station and social media accounts of researchers working within the Psychopathology and Affective Neuroscience lab of Ghent University. Given the exploratory nature of this study, we did not conduct power analysis. Instead, respecting the one in ten rule for the regression models, and based on previous research in which (extensions of) latent class or latent profile analysis were used (e.g., Collins and Wugalter, 1992; Finch and Bronk, 2011; Nylund-Gibson and Choi, 2018; Spurk et al., 2020; Wurpts and Geiser, 2014), we set-out to recruit 300–500 participants (osf.io/cw4k3). Prior to the end date of the project, 382 participants entered the study among which 322 participants completed the online baseline assessment and were provided access to the intervention and follow-up assessments.

Based on prior research (e.g., Hoorelbeke et al., 2015; Siegle et al., 2007, Siegle et al., 2014), the study was framed as an experimental, web-based, and freely accessible cognitive intervention aimed at targeting RNT. As such, this sample is likely to consist of participants scoring elevated in terms of RNT. In line with this, for participants who completed the baseline assessment (for demographic information, see Table 1), elevated RNT levels were observed as indicated by scores on the Perseverative Thinking Questionnaire (M = 39.76, SD = 8.17) which were in the range of previously observed levels of RNT in patients diagnosed with depressive- (M = 37.56, SD = 9.99) or anxiety disorders (M = 35.93, SD = 13.60; Ehring et al., 2011). Participants were eligible for participation in the study if they were between the age of 18–70,2 Dutch speaking, and could access the online training platform via a computer or tablet. Electronic informed consent was obtained for all participants. Two shopping vouchers (€50) were raffled among completers of the protocol. This study was approved by the medical ethical committee of Ghent University Hospital and was preregistered on Open Science Framework (osf.io/cw4k3).

Table 1.

Group characteristics.

Variable	Ratio
Gender (Male:Female:Other)	85:236:1
Degree (No formal degree:Basic education:Secondary education:Bachelor degree (college):Master degree (university))	1:4:52:140:125
Job status (student:fulltime employed:part-time employed:unemployed:sick leave:retired)	29:135:80:20:38:20
Undergoing psychopharmacological treatment (yes: no)	99:223
	M	SD
Age	41.62	12.96
naPASAT task performance (Accuracy)	29.25	14.13
Effortful Control (ATQ)	4.24	0.75
Math ability	3.19	0.76
Fear of math	2.43	0.99
Resilience (CDRISC)	21.84	6.54
Reappraisal (ERQ)	3.86	1.09
Suppression (ERQ)	3.39	1.34
RNT Core features (PTQ)	25.41	5.11
RNT Load (PTQ)	7.00	2.26
RNT Unproductivity (PTQ)	7.35	2.01
Overall level of functioning (RDQ)	35.89	16.61
Depressive symptomatology (DASS)	13.24	9.81
Anxiety symptoms (DASS)	11.39	8.83
Stress symptoms (DASS)	19.63	9.55
Emotional stability (TIPI)	3.46	1.39
Agreeableness (TIPI)	5.34	1.03
Conscientiousness (TIPI)	5.09	1.35
Extraversion (TIPI)	4.10	1.64
Openness (TIPI)	4.96	1.50

2.2. Cognitive tasks

2.2.1. Cognitive control training

We relied on a gamified internet variant of the adaptive Paced Auditory Serial Addition Task (aPASAT; Siegle et al., 2007) to train cognitive control. All participants were instructed to complete ten sessions of the aPASAT task within a two-week period. Every session lasted for 15 min. In this task, participants are presented with a continuous stream of auditory digits ranging from one to nine. The stimuli were presented in random order, where participants needed to respond to the sum of every last heard digit and the one presented just before in a continuous manner. As such, after every stimulus a response was due. Throughout the task, participants were presented with a screen consisting of 18 response buttons (1–18). Following every stimulus, participants responded by clicking on the corresponding number on the screen. For training, this task was adaptive. That is, difficulty was set at an individualized level. Specifically, during each session the inter trial interval (ITI) changed based on level of task performance: following four consecutive correct responses the ITI decreased with 100 ms, increasing task difficulty. Following four consecutive incorrect responses (i.e., incorrect, or untimely response), the ITI increased with 100 ms. In addition, difficulty level of the first trial of the training task was also contingent on one's task performance. For the first CCT session, the start ITI was based on performance during the baseline assessment of cognitive functioning, which consisted of the non-adaptive PASAT. For each consecutive session, start ITI was based on level of performance during the previous training session.

Previous research showed that gamifying this task (i.e., adding game elements such as points and levels) was associated with increased user engagement and motivation without altering training mechanisms (Vervaeke et al., 2020). Therefore, we relied on the gamified training version described in Vervaeke et al., 2018, Vervaeke et al., 2020 for the current study, to promote a positive user experience and maximize task engagement, which is a predictor of treatment response (cf. Siegle et al., 2014). This meant that the task had a vivid and colorful look and that participants received additional performance feedback both during the task and following each session. In addition, motivating messages were presented during the task and a badge system was used to foster completion of the ten training sessions. For a full overview of gamification features used in this task, we refer to Vervaeke et al. (2020).

2.2.2. Task-specific cognitive transfer

Task-specific cognitive transfer or ‘near transfer’ is one of the primary outcome measures of this study and was assessed using a non-adaptive version of the PASAT (naPASAT; Gronwall, 1977; Siegle et al., 2007). During this task, participants are presented with digits ranging from one to nine and are instructed to continuously respond to the sum of the last two heard digits. In contrast to the aPASAT, ITI is not contingent on level of task performance in the naPASAT, allowing to assess level of cognitive functioning prior to the training procedure as well as task-specific cognitive transfer following training. During the naPASAT, participants completed three blocks consisting of 60 trials each. Within each block, the ITI was fixed. As such, the naPASAT measures cognitive control in a standardized manner. In line with previous studies (e.g., Hoorelbeke and Koster, 2017), task difficulty increased over blocks, where we used ITIs of 3000 ms, 2000 ms and 1500 ms for Blocks 1, 2 and 3 respectively. The outcome of interest is mean accuracy over the three blocks.

2.3. Questionnaires

At baseline, post-training and one month follow-up, participants completed multiple questionnaires. In addition to the assessment of user experience, which was only included post-training, all baseline measures were used to establish a User Profile which formed the basis for the moderation analyses. The Perseverative Thinking Questionnaire (PTQ), Depression Anxiety Stress Scales (DASS), Emotion Regulation Questionnaire (ERQ) and Connor-Davidson Resilience Scale (CD-RISC) were re-administered during the post-training and follow-up assessment to evaluate treatment effects (further referred to as outcome measures).

2.3.1. Outcome measures

In addition to cognitive task performance, RNT and severity of internalizing symptomatology were pre-registered as primary outcome measures. As exploratory outcome measures, emotion regulation (suppression, reappraisal) was assessed, in addition to resilience.

2.3.1.1. RNT

RNT as a transdiagnostic risk factor for internalizing disorders was assessed using the Perseverative Thinking Questionnaire (PTQ; Ehring et al., 2011; Ehring et al., 2012). This 15-item questionnaire contains three components relating to the characteristics of RNT: core properties, unproductiveness, and use of mental capacity. The core properties of RNT include the repetitive (e.g., “The same thoughts keep going through my mind again and again”) and intrusive nature of the thoughts (e.g., “Thoughts come to my mind without me wanting them to”), in addition to experienced difficulties disengaging from these thoughts (uncontrollability; e.g. “I can't stop dwelling on them”). We obtained a Cronbach's α = 0.91 for the Core properties component. The second and third component relate to unproductiveness of thoughts (Cronbach's α = 0.74; e.g., “I keep asking myself questions without finding an answer”), and use of mental capacity respectively (load; Cronbach's α = 0.82; e.g., “My thoughts prevent me from focusing on other things”). Each item was rated on a scale ranging from 0 (“Never”) to 4 (“Almost always”). The three subscales of the PTQ were used to predict attrition and to identify User Profiles, whereas the total PTQ score (Cronbach's α = 0.92) was used to evaluate effects of CCT on RNT. Previous research suggests high internal consistency and re-test reliability for the total scale and subscales (Ehring et al., 2011).

2.3.1.2. Internalizing symptomatology

Severity of internalizing symptomatology was assessed using the 21-item version of the Depression Anxiety Stress Scales (DASS; Lovibond and Lovibond, 1995; de Beurs et al., 2001). Each item was rated on a scale ranging from 0 (“Did not apply to me at all”) to 3 (“Applied to me very much or most of the time”). The DASS shows adequate psychometric properties (Lovibond and Lovibond, 1995) and consists of three scales, reflecting severity of depressive- (Cronbach's α = 0.88; e.g., “I couldn't seem to experience any positive feeling at all”), anxiety- (Cronbach's α = 0.80; e.g., “I was worried about situations in which I might panic and make a fool of myself”), and stress symptoms (Cronbach's α = 0.87; e.g., “I found it difficult to relax”).

2.3.1.3. (Mal)Adaptive emotion regulation

In addition to the PTQ, we relied on the widely used and well-validated Emotion Regulation Questionnaire (ERQ; Gross and John, 2003) to assess two specific indicators of (mal)adaptive emotion regulation: Cognitive reappraisal (further referred to as Reappraisal) and Emotional suppression (further referred to as Suppression). The ERQ contains 10 items. Each item was rated on a scale ranging from 1 (“Strongly disagree”) to 7 (“Strongly agree”). Reappraisal (Cronbach's α = 0.79; e.g., “I control my emotions by changing the way I think about the situation I'm in”) is an antecedent-focused strategy which refers to attempts to cognitively reconstruct an emotion-eliciting situation in a way that changes the emotional impact of the situation (Lazarus and Alfert, 1964). Suppression (Cronbach's α = 0.78; e.g., “I keep my emotions to myself”), on the other hand, refers to a response-focused strategy, entailing inhibition of ongoing emotion-expressive behavior (Gross, 1998).

2.3.1.4. Resilience

Resilience, reflecting greater stress coping ability, was measured using the 10-item version of the Connor-Davidson Resilience Scale (CD-RISC; Connor and Davidson, 2003). Each item (e.g., “I am able to adapt to change”) was rated on a scale ranging from 0 (“Not true at all”) to 4 (“True nearly all the time”). This measure shows adequate psychometric properties (Campbell-Sills and Stein, 2007). In line with this, we observed adequate internal consistency (Cronbach's α = 0.87).

2.3.2. Other baseline questionnaires

2.3.2.1. Math ability and fear of math

Math Ability and Fear of Math were assessed using one item each, where participants were instructed to estimate their level of math proficiency or fear of math on a 5-point scale. Higher values on these items correspond to higher math proficiency or higher fear of math respectively.

2.3.2.2. Big 5 personality traits

We also included the Ten Item Personality Inventory (TIPI; Gosling et al., 2003; Hofmans et al., 2008). Each of these items measure one pole of the five personality dimensions of the Big Five personality framework, and were rated on a scale ranging from 1 (“Disagree strongly”) to 7 (“Agree strongly”). This results in a score for each of the five dimensions: extraversion (e.g., “I see myself as extraverted, enthusiastic”; Spearman-Brown coefficient = 0.77),3 agreeableness (e.g., “I see myself as sympathetic, warm”; Spearman-Brown coefficient = 0.24), conscientiousness (e.g., “I see myself as dependable, self-disciplined”; Spearman-Brown coefficient = 0.54), emotional stability (e.g., “I see myself as calm, emotionally stable”; Spearman-Brown coefficient = 0.53) and openness to experience (e.g., “I see myself as open to new experiences, complex”; Spearman-Brown coefficient = 0.55).

2.3.2.3. Overall functioning

The well-validated Remission of Depression Questionnaire (RDQ; Peeters et al., 2013; Zimmerman et al., 2013; Zimmerman et al., 2014) was used as an indicator of overall functioning. For this purpose, the instruction was slightly altered given that the target sample did not necessarily consist of remitted depressed patients (i.e., in line with the original version of the RDQ, participants were instructed to rate the extent to which the different items occurred during the past week, with the exception that the instruction did not include the statement that the goal of this measure was to assess the extent to which participants benefitted from a treatment for depression). Items were rated on a scale ranging from 0 (“Not at all or rarely true”) to 2 (“Often or almost always true”). The RDQ contains 41 items and forms a broad and inclusive measure of functioning in several domains (e.g., “When I woke up I looked forward to the day”), among which internalizing symptomatology, coping, positive mental health, general functioning, life satisfaction, and sense of well-being. We relied on the total score of this measure, where we observed excellent internal consistency (Cronbach's α = 0.96).

2.3.2.4. Effortful control

We relied on the Effortful Control subscale (EC) of the Adult Temperament Questionnaire (ATQ; Rothbart et al., 2000; Hartman and Rothbart, 2001) as a measure of subjective executive functioning. The EC scale (Cronbach's α = 0.77) contains 19 items constituting three components: attentional control (referring to the capacity to focus and shift attention when desired; Cronbach's α = 0.67), activation control (referring to the capacity to suppress inappropriate approach behavior; Cronbach's α = 0.72), and inhibitory control (referring to the capacity to perform an action when there is a tendency to avoid it; Cronbach's α = 0.53; Evans and Rothbart, 2007).

2.3.3. Other post-training questionnaires

At post-training, in addition to the outcome measures mentioned in 2.3.1, user engagement was assessed as an indicator of task engagement and motivation. For this purpose, we relied on the 12-item User Engagement Scale (O'Brien and Toms, 2010), referred to as the Short Form UES (UES-SF; O’Brien et al., 2018). Items (e.g., “My experience was rewarding”) were rated on a scale ranging from 1 (“Strongly disagree”) to 5 (“Strongly agree”). We observed adequate internal consistency (Cronbach's α = 0.83).

2.4. Procedure

Interested participants received written information regarding the study design and rationale, after which a personal account was created on the cognitive training platform (Vervaeke et al., 2020). Upon activation of the account, participants were asked to provide electronic informed consent, after which participants completed following stages of the study on the cognitive training platform over a period of 1.5 months: (1) baseline assessment, (2) training procedure, (3) post-training assessment, and (4) one month follow-up assessment.

During the baseline phase, participants provided relevant demographic information, and completed several self-report questionnaires (PTQ, RDQ, ERQ, TIPI, DASS, CD-RISC, and EC [ATQ]), among which the PTQ, DASS, ERQ, and CD-RISC were re-assessed at later time points. Next, participants completed the naPASAT, which included a video tutorial, practice- and test phase. During the baseline phase, participants also received brief online psychoeducation, summarizing the information that was previously shared with interested participants regarding the study design and training procedure. To ensure that all participants read these instructions, they were prompted to complete several multiple-choice items regarding the study.

Participants were instructed to commence the training procedure within two weeks following the baseline assessment. The training procedure consisted of ten sessions of the gamified aPASAT, which were to be completed within a period of 14 days. Each session consisted of 15 min aPASAT training. Upon completion of a session, participants scheduled the next session on the platform. Upon reaching the scheduled date, automated reminder e-mails were sent.

Participants were requested to complete the post-training assessment within the first week following the training phase. The post-training assessment contained re-administration of the naPASAT, PTQ, and DASS, which form the primary outcome measures of this study. In addition, participants completed exploratory outcome measures ERQ and CD-RISC. The post-training assessment also included a measure of user experience (UES). One month following the intervention, participants received an invitation to complete the follow-up assessment, which again included the PTQ, DASS, ERQ, CD-RISC, and the naPASAT. Following completion of the follow-up assessment, participants received a written debriefing.

2.5. Statistical analysis

The analyses were conducted using IBM SPSS Statistics Version 25 and R version 3.6.1 (see supplemental materials for more detailed version information on R-packages used).

2.5.1. Prediction of decision to not participate

We modeled not commencing the intervention following the baseline assessment and psychoeducation module via a logistic regression model (see Section 2.5.2 for predictors).

2.5.2. Prediction of risk for drop-out during the course of CCT

To model drop-out during the entire training procedure, we used a time-to-event data analytical approach, predicting attrition of participants during the training procedure (i.e., time to drop-out), operationalized as number of CCT sessions completed on the platform, ranging from 0 to 10. For this purpose, we relied on a Cox regression model, a semiparametric regression model that allows examination of the relation between each of the predictor variables and survival time. Here, drop-out (i.e., occurrence of the event) was defined as completion of less than ten training sessions.

For both the prediction of decision to not participate (Section 2.5.1) and drop-out during the course of CCT we included following predictors: (a) age, (b) cognitive functioning (naPASAT performance, self-reported effortful control (ATQ)), (c) self-reported math ability and fear of math, (d) baseline resilience (CD-RISC), (e) indicators of emotion regulation (reappraisal (ERQ), suppression (ERQ)), and central aspects of RNT (core features (PTQ), load (PTQ), and unproductivity (PTQ)), (f) level of internalizing symptomatology (severity of depressive- (DASS), anxiety- (DASS), and stress symptoms (DASS)), overall level of functioning (RDQ), and (g) personality factors (emotional stability (TIPI), agreeableness (TIPI), conscientiousness (TIPI), extraversion (TIPI), and openness (TIPI)).

2.5.3. Identification of user profiles

We relied on latent profile analysis, a probabilistic modeling approach, to identify clusters of participants (latent user profiles) based on indicators of baseline cognitive- and emotional functioning, training progress, and user experience. In particular, in line with the pre-registered analyses, we included the following indicator variables in the analysis: all baseline measures (measured at a continuous level), among which all predictor variables used for the logistic- and Cox regression models reported in the prediction of drop-out section above (a – g), in addition to (h) training task progress (change in median ITI from the first to the last completed session), and (i) user experience (UES).4 To facilitate interpretation of the output, all indicator variables were standardized (centered and scaled) using the package effectsize. As a result, each indicator variable is distributed with M = 0 and SD = 1.

The latent profile analysis was conducted in R using the mclust package (Scrucca et al., 2016). Selection of the optimal model, including covariance parameterization and optimal number of components, was based on the Bayesian Information Criterion (BIC; Schwartz, 1978). In addition, to test for the robustness of our results we re-ran the process of model selection based on the Integrated Complete-data Likelihood criterion (ICL; Biernacki et al., 2000), which penalizes BIC via an entropy term which measures the extent to which clusters show overlap (for a more detailed discussion of this procedure, see Scrucca et al., 2016). We used the dimension reduction method described by Scrucca et al. (2016) to visualize the clustering structure and geometric features of the obtained Gaussian finite mixture model. We modeled the maximal separation among clusters in addition to uncertainty boundaries using tuning parameter lambda = 1. As a result, the dimension reduction subspace was based on variation in group means.

2.5.4. Evaluation of functioning

To explore the moderating role of User Profile for change in the primary outcome measures RNT, depressive-, anxiety-, and stress symptoms following CCT, and exploratory outcome measures suppression, reappraisal, and resilience (Aim 2), we relied on linear mixed-effects models using the nlme R package (Pinheiro et al., 2021). In particular, for each outcome variable, a linear mixed-effects model was constructed in which Time (baseline, post-training, follow-up) and User Profile were included as factors. Evaluation of the effect of Time allows to model change in the primary and exploratory outcome measures following CCT. In addition, we modeled the interaction between both factors, allowing for a random intercept and random slope for Time. In line with Theeuwes et al. (2018), reported p-values for Time, User Profile, and the Time x User Profile interaction were based on a Type III ANOVA using a χ²-distribution implemented in the R package car (Fox and Weisberg, 2011). Contrasts were calculated using dummy coding. This allowed to test whether change in the primary and exploratory outcome measures was contingent on User Profile. Given that the preregistered analyses represent different families of tests, each of which was examined as an a priori contrast, we did not control for multiple comparisons.

3. Results

3.1. Group characteristics

382 participants entered the study, of which 60 participants did not complete the baseline assessment, resulting in a sample of N = 3225 for the prediction of decision not to participate and observed attrition during the course of CCT (Aim 1). Descriptive information for this sample is presented in Table 1. Of the 322 participants completing the baseline assessment, 213 participants completed the training procedure, among which six participants did not complete the post-training assessment of user experience necessary for the latent profile analysis. As such, identification of User Profiles was based on N = 207. Similarly, prediction of change in primary and exploratory outcome measures following CCT by User Profile (Aim 2) was based on this final sample. Of the 207 participants included in these analyses, another six participants did not complete the follow-up assessment.

3.2. Predicting decision to not participate

322 participants entered the analysis of drop-out prior to the start of the intervention (decision to not participate). 6% of potential participants opted to not participate to CCT following psychoeducation, and prior to the start of the first training session. The logistic regression model predicting non-participation was significant at χ²(20) = 40.53, p = .004, R_L² = 0.29. Several baseline features emerged as significant predictors for risk of drop-out prior to the first CCT session. In particular, risk of not commencing the training procedure was lower for participants reporting relatively higher emotional stability (TIPI; Odds ratio [OR] = 0.498, p = .039, 95% CI = [0.257, 0.966]). In contrast, participants reporting higher levels of resilience were more likely to decide to not participate prior to the intervention (OR = 1.260, p = .001, 95% CI = [1.102, 1.440]; all other ps ≥ 0.061).

3.3. Predicting treatment attrition

Approximately 34% of this sample did not complete the training procedure. The Cox regression model was significant at χ²(20) = 49.66, p < .001. Fig. 1 depicts the obtained survival curve at the mean levels of the covariates in the Cox regression model, suggesting that – although drop-out occurred throughout the entire training procedure – highest drop-out can be expected during the first three training sessions. Several variables emerged as significant predictors for the observed pattern of drop-out throughout the training procedure. In particular, we observed significant effects of age (Hazard ratio [HR] = 0.975, p = .004, 95% CI = [0.959, 0.992]), suppression (ERQ; HR = 1.191, p = .034, 95% CI = [1.014, 1.399]), unproductivity of RNT (PTQ; HR = 0.843, p = .012, 95% CI = [0.739, 0.962]), conscientiousness (TIPI; HR = 0.796, p = .006, 95% CI = [0.675, 0.937]), and extraversion (TIPI; HR = 1.241, p = .002, 95% CI = [1.082, 1.423]). The obtained hazard ratios suggest that drop-out risk decreases as participant age increases. In addition, drop-out risk was lower for participants reporting high unproductivity of RNT, and for participants scoring high on conscientiousness. In contrast, high levels of extraversion and more frequent use of suppression predicted higher rates of drop-out. That is, for every unit of increase in extraversion or suppression, risk of drop-out increased with a factor of 1.241 and 1.191 respectively (all other ps ≥ 0.066).

Fig. 1.

Retention throughout the course of CCT at mean levels of covariates.

3.4. Identifying user profiles

Based on BIC, optimal fit was obtained using a model containing three User Profiles (clusters) with a diagonal distribution, variable volume, and equal shape (VEI, 3; see Supplemental Figs. 1–2 for the dimension reduction subspace). A similar solution was obtained using ICL (VEI, 3). Mean uncertainty of classification was 0.05 (for a more detailed overview of classification uncertainty for each User Profile, we refer to supplemental table 1). Among the 207 completers of the baseline assessment, training procedure, and post-training assessment of user engagement, 63 participants (30.43%) were categorized into the first User Profile. The second and third User Profile contained N = 96 (46.38%) and N = 48 (23.19%) respectively. As Fig. 2 illustrates, the first User Profile contains individuals who seem to be at low risk for internalizing psychopathology (‘Low-risk’ group). That is, this User Profile is characterized by relatively high levels of effortful control, resilience, and use of cognitive reappraisal, while reporting low levels of RNT, depressive-, anxiety- and stress symptoms, and a higher level of overall functioning (i.e., lower RDQ scores). In addition, members of the first User Profile reported relatively high levels of emotional stability. In contrast, User Profile three seems to reflect individuals at high-risk for internalizing psychopathology (’High-risk’ group), as shown by relatively low effortful control, resilience, reappraisal, and emotional stability, in addition to higher self-reported RNT and internalizing symptomatology. Related to this, this group also showed a lower level of functioning (i.e., higher RDQ scores). The second User Profile contains individuals scoring close to the sample mean on most measures, yet show slightly higher scores on RNT (further referred to as ‘Moderate’ risk group).

Fig. 2.

Identification of User Profiles based on baseline level of functioning, personality, training progress and user engagement.

Note: for fear of math, ERQ suppression, the PTQ scales, RDQ, and DASS, a high score is indicative of a lower level of functioning.

3.5. Predicting treatment response

3.5.1. Primary outcome measures

Results of the linear mixed-effects model analyses are reported in Table 2. For naPASAT performance, our indicator of task-specific cognitive transfer, we observed a significant main effect of Time, reflecting improvement in cognitive task performance, and a significant Time x User Profile interaction effect. These findings indicate that change in naPASAT performance over time is contingent on one's profile of functioning. In particular, participants in the moderate-risk group showed a stronger increase in cognitive task performance from baseline to post-training compared to the high-risk group (χ²(1) = 4.16, p = .04; for descriptives see Table 3). In addition, although we observed an overall decrease in cognitive task performance from post-training to follow-up, the moderate-risk group showed higher retention of training related cognitive gains compared to the low-risk group (χ²(1) = 5.22, p = .02; all other χ²s < 1.89).

Table 2.

Evaluation of change in functioning following CCT and the moderating role of User Profile.

Variable	Main effect of Time			Main effect of User Profile			Time x User Profile
	χ2	df	p	χ2	df	p	χ2	df	p
Primary outcome measures
Task-specific cognitive transfer (naPASAT)	2212.62	2	<0.001	5.79	2	0.06	13.96	4	<0.01
Repetitive negative thinking (PTQ)	185.00	2	<0.001	171.10	2	<0.001	7.55	4	0.11
Depressive symptomatology (DASS)	47.94	2	<0.001	228.59	2	<0.001	8.89	4	0.06
Anxiety symptoms (DASS)	87.19	2	<0.001	212.68	2	<0.001	46.67	4	<0.001
Stress symptoms (DASS)	78.77	2	<0.001	220.15	2	<0.001	16.66	4	<0.01
Exploratory outcome measures
Suppression (ERQ)	5.15	2	0.08	7.61	2	0.02	3.12	4	0.54
Reappraisal (ERQ)	68.05	2	<0.001	38.11	2	<0.001	13.10	4	0.01
Resilience (CD-RISC)	34.38	2	<0.001	104.00	2	<0.001	2.82	4	0.59

Table 3.

Cognitive and emotional functioning of User Profiles at Baseline, Post-training and Follow-up.

Variable	Low-risk group (N = 63)			Moderate-risk group (N = 96)			High-risk group (N = 48)
	Baseline M (SD)	Post M (SD)	Follow-up M (SD)	Baseline M (SD)	Post M (SD)	Follow-up M (SD)	Baseline M (SD)	Post M (SD)	Follow-up M (SD)
Primary outcome measures
Task-specific cognitive transfer (naPASAT)	26.45 (13.18)	67.73 (17.48)	62.17 (16.07)	29.74 (13.51)	72.36 (14.99)	70.58 (15.57)	31.07 (14.80)	68.90 (17.24)	64.73 (17.53)
Repetitive negative thinking (PTQ)	30.87 (6.10)	27.97 (6.89)	24.97 (6.75)	41.51 (5.17)	35.81 (7.87)	32.41 (7.27)	46.46 (6.61)	41.60 (8.20)	38.23 (9.86)
Depressive symptomatology (DASS)	4.79 (4.12)	3.02 (3.61)	3.81 (5.00)	12.06 (7.27)	8.31 (7.09)	8.40 (7.66)	23.29 (7.58)	18.29 (8.91)	18.23 (10.51)
Anxiety symptoms (DASS)	4.51 (4.54)	3.65 (4.48)	2.97 (3.74)	9.31 (6.56)	6.83 (5.83)	6.25 (5.97)	22.50 (7.19)	13.92 (7.33)	15.09 (8.34)
Stress symptoms (DASS)	10.38 (5.52)	9.08 (5.77)	8.23 (5.80)	18.44 (6.88)	14.23 (7.74)	13.73 (7.07)	29.33 (7.19)	22.88 (8.54)	21.68 (8.39)
Exploratory outcome measures
Suppression (ERQ)	2.92 (1.20)	2.96 (1.26)	2.89 (1.17)	3.42 (1.33)	3.24 (1.24)	3.14 (1.19)	3.63 (1.61)	3.52 (1.47)	3.49 (1.29)
Reappraisal (ERQ)	4.53 (0.90)	4.74 (1.07)	4.78 (1.00)	3.89 (0.94)	4.37 (1.02)	4.51 (1.06)	3.10 (0.93)	3.91 (1.19)	4.00 (1.27)
Resilience (CDRISC)	26.75 (4.73)	27.02 (4.27)	28.16 (4.51)	20.70 (5.48)	22.02 (5.13)	22.88 (5.35)	17.75 (5.94)	18.65 (5.97)	20.05 (5.01)

The significant main effect of Time for self-reported level of RNT, depressive-, anxiety-, and stress symptoms suggests improvements over time on the primary emotional transfer measures (Table 2; for descriptives see Table 3). In addition, for each of the primary outcome measures, a main effect of User Profile was observed. We also observed a significant Time x User Profile interaction for anxiety- and stress symptoms.

In particular, participants in the high-risk group reported a stronger reduction in anxiety from baseline to post-training than participants in the low-risk group (χ²(1) = 42.32, p < .001) and moderate-risk group (χ²(1) = 31.03, p < .001). Similarly, the high-risk group reported a stronger reduction in stress symptoms compared to the low-risk group (χ²(1) = 11.78, p < .001). In addition, the low-risk and moderate-risk group differed significantly in terms of change in stress symptoms (χ²(1) = 5.23, p = .02), but not anxiety symptoms (χ²(1) = 2.60, p = .11), from baseline to post-training. However, compared to the moderate-risk group (χ²(1) = 4.52, p = .03), participants in the high-risk group also showed poorer long-term retention of improvement in anxiety symptoms (all other χ²s < 3.73).

3.5.2. Exploratory outcome measures

We observed a significant increase in reappraisal and resilience over time (Table 2). Moreover, for each of the exploratory outcome measures we observed a significant main effect of User Profile. The Time x User Profile interaction was significant for reappraisal; participants in the high-risk group showed stronger improvements over time in terms of reappraisal than did participants in the low- (χ²(1) = 10.75, p = .001) or moderate-risk group (χ²(1) = 3.94, p = .047). User Profiles did not significantly differ in terms of change in reappraisal from post-training to follow-up (all other χ²s < 2.93).

4. Discussion

Previous study findings point towards the potential of CCT as an eHealth intervention for RNT, a key risk factor for the development and maintenance of internalizing symptomatology (Koster et al., 2017). In this context, experimental studies have mostly investigated effects of non-gamified CCT under highly controlled lab settings, whereas the efficacy of web-based gamified CCT as a broad-spectrum transdiagnostic intervention has received less attention. As a result, relatively little is known regarding which factors contribute to adherence to web-based CCT and who benefits most from it. Such knowledge may allow for more effective use of CCT as an eHealth intervention. For instance, by identification of individuals among the target population who are at-risk for not commencing the intervention or drop-out prior to completion of the training procedure.

Attrition rates have been observed up to 50% in randomized controlled trials exploring effects of eHealth interventions and are known to be even higher for interventions that are presented via open access sites (for a review, see Christensen et al., 2009). In line with this, in the current study 44% of participants who enrolled in the study, and 34% of the completers of the baseline assessment, did not complete the CCT procedure. Attrition was higher in the current study than in previous CCT studies in which the aPASAT was delivered in a more controlled setting (e.g., fully administered in a laboratory setting, or following an intensive screening procedure and lab session with an experimenter), where drop-out rates ranging from 4 to 15% have been observed in at-risk, MDD, and RMD samples (Brunoni et al., 2014; Calkins et al., 2015; Hoorelbeke and Koster, 2017; Hoorelbeke et al., 2021; Segrave et al., 2014; Siegle et al., 2007, Siegle et al., 2014; Vervaeke et al., 2021). This may be due to the nominal applicability of the intervention to participants' acknowledged issues, as well as the availability of face-to-face contact with a clinician / experimenter in previous CCT studies, the lack of which has been reported as a reason for drop-out in eHealth studies (Christensen et al., 2009; Lange et al., 2003). In addition, previous CCT studies have typically also included a financial incentive for completion of the training procedure which was not the case in the current study (where compensation was a raffle ticket), more closely meeting the circumstances under which gamified CCT could be performed online as an eHealth intervention. Our previous focus groups have suggested a mandatory requirement for cognitive training is a “calm and private environment” (Vervaeke et al., 2018). The extent to which online administration affords this basic pre-requisite is unclear, and without it, the intervention may have been considered undesirable or unusable.

Most attrition in eHealth interventions is typically observed early on (Waller and Gilbody, 2009), where participants may choose not to start the intervention after receiving more information regarding the type and purpose of the intervention, or drop out during the first sessions of the intervention. This is in line with the observation of Arean et al. (2016) that 58% of participants that were assigned to the gamified cognitive training procedure or to another eHealth component did not download the intervention app. In the current study, attrition was also most prevalent during the first CCT sessions, albeit relatively limited compared to Arean et al. (2016). In the current study, 6% of completers of the baseline assessment – 21% of participants that entered the study – did not complete the first training session.

Among predictors for attrition in eHealth interventions in the context of individuals seeking treatment for depression are baseline severity of symptomatology and older age (Christensen et al., 2009). In the context of generalized anxiety disorder, reporting higher baseline levels of anxiety has been related to attrition to eHealth interventions (Christensen et al., 2009; Kenardy et al., 2003). Similarly, Arean et al. (2016) found baseline level of depressive symptomatology to be predictive for not commencing gamified online training. Consistent with these data, we observed decision to not participate in the intervention to be most likely for participants scoring low on emotional stability. This is in line with previous studies suggesting poor treatment adherence for individuals scoring high on neuroticism (Jerant et al., 2011). In addition, participants reporting to be highly resilient were more likely to choose not to participate to the intervention. This is potentially indicative of beliefs that the current intervention was not necessary for their situation, that these participants did not view themselves as having problems that this intervention was suited to fix after receiving psychoeducation regarding CCT. That is, the decision not to participate is situated between the baseline assessment, including online psychoeducation regarding the intervention, and the first training session. Alternatively, given the stressful nature of the assessment task (the naPASAT; for a review, see Tombaugh, 2006) which was part of the baseline assessment, it is possible that the stress induced during the assessment procedure and the mild complaints experienced by this group, led to the decision to discontinue. However, due to the lack of qualitative information one can only speculate regarding reason(s) for drop-out, which should be addressed in future research exploring patterns of attrition of aPASAT training.

In line with this, low levels of unproductivity of RNT emerged as a predictor for drop-out throughout the course of the intervention, where higher levels of self-reported RNT unproductivity were related to completion of more CCT sessions. This suggests that experiencing repetitive negative thoughts as being unhelpful to reach one's goal may motivate one to continue to partake in the training procedure. This may also suggest that adding psychoeducation or other interventions targeting metacognitions regarding RNT may aid in increasing adherence to cognitive remediation strategies such as CCT.

In addition, higher levels of suppression predicted more drop-out throughout the intervention period. Age also emerged as a significant predictor for attrition during CCT, where young participants were more likely to discontinue the training procedure compared to relatively older participants (observed range within this study: 18–72 years). In contrast, Christensen et al. (2009) found younger people to show stronger adherence to eHealth interventions. Similarly, in older adults Jerant et al. (2011) also found age to be related to poorer adherence to a pharmacological intervention. Although gamified, based on focus groups (Vervaeke et al., 2018) the current intervention has a retro feel where game elements were kept to a minimum. This may be less appealing for young participants compared to older participants, which previously reported a preference for light rather than strong gamification of the training procedure.

Several personality factors also predicted attrition. In line with Molloy et al. (2014) and Stilley et al. (2004), participants scoring high on conscientiousness adhered better to the intervention procedure. In a similar vein, Jerant et al. (2009) showed that missing data is less likely to occur in randomized controlled trials for individuals scoring high on conscientiousness. In contrast, participants that scored high on extraversion were more likely to drop-out throughout the CCT procedure. Interestingly, previous studies suggest individual differences in how one responds to gamification elements, which have been linked to personality features such as conscientiousness and extraversion (e.g., Codish and Ravid, 2014; Ghaban and Hendley, 2020). For instance, in the context of online learning courses, participants scoring high on extraversion seem to benefit more from an intervention when social (game) elements are added which allow to interact and compete with others. Moreover, although most participants seem to benefit from gamification elements such as use of points and badges, previous studies suggest that this may particularly be the case for participants scoring low in terms of conscientiousness (Ghaban and Hendley, 2020). In contrast, highly conscientious individuals, showing a strong intrinsic motivation to conduct tasks in a careful and diligent manner, seem to benefit less from addition of gamification elements. Noteworthy, although we used game elements, the current intervention did not include social game elements, as social comparison may induce unwanted effects in this context. Overall, for eHealth interventions, the role of personality in how gamified interventions are perceived is a domain that requires further investigation.

Taken together, we identified multiple predictors for deciding not to participate in the eHealth intervention after having received online psychoeducation, as well as attrition throughout the course of aPASAT training. This allows for future studies to investigate reasons for (early) drop-out in these subpopulations using appropriate methods (e.g., focus groups, workshops). Such information would be informative to which efforts could be undertaken to motivate these individuals for completing the training procedure (e.g., modifications to the online psychoeducation module, the cognitive assessment- and training platform, the selected gamification elements, further personalization of content; following up on Vervaeke et al., 2018), via which we could further increase the impact of the eHealth intervention. This is key given that recent findings suggest preventive aPASAT training to reduce risk for recurrence of depression (Hoorelbeke et al., 2021).

The second aim of this single-arm trial was to identify different types of users of the eHealth intervention, referred to as User Profile, and investigate whether User Profile moderated change in the cognitive and emotional transfer measures following completion of CCT. We identified three User Profiles which seem to reflect increasing levels of cognitive vulnerability for or presence of internalizing symptomatology. A recent study exploring effects of CBT and the combination of fluoxetine and CBT for the treatment of depressive symptomatology obtained similar profiles which largely reflected severity, albeit using different indicators of functioning (Gunlicks-Stoessel et al., 2019). In the current study, effortful control, level of internalizing symptomatology, use of (mal)adaptive emotion regulation strategies, resilience, and emotional (in)stability were most characteristic for one's User Profile. As CCT is designed to address effortful control, which is associated with all of emotion regulation, emotional instability, and RNT (Van den Bergh et al., 2021), it seems reasonable for this dimension to be associated with CCT outcomes. As engaging in RNT is negatively associated with cognitive control (Philippot and Brutoux, 2008; Whitmer and Gotlib, 2012) it is understandable that CCT could affect RNT by increasing recruitment of neural mechanisms underlying cognitive control (Siegle et al., 2007). Moreover, given that self-reported cognitive functioning has been linked to adaptive emotion regulation and resilience (Hoorelbeke et al., 2016b, Hoorelbeke et al., 2019) it is reasonable that User Profiles would be predictive of having these vulnerabilities addressed by CCT.

On average, we observed an increase of 41.28% in performance on the measure for task-specific cognitive transfer from baseline to post-training. This increase is in line with previous studies, where performance on the naPASAT following training typically increased with 37–47% (Hoorelbeke and Koster, 2017; Van den Bergh et al., 2020; Vanderhasselt et al., 2020; Vervaeke et al., 2020). We also observed improvements over time on the emotional transfer measures, where participants reported a significant decrease in RNT and level of depressive-, anxiety- and stress symptoms. In addition, our findings suggest a significant increase in reappraisal and resilience over time. These findings are in line with previous studies in which beneficial effects of aPASAT training were observed on RNT and more broader indicators of emotion regulation (Hoorelbeke et al., 2015; Hoorelbeke and Koster, 2017; Peckham and Johnson, 2018; Siegle et al., 2007, Siegle et al., 2014), depressive symptomatology (Brunoni et al., 2014; Calkins et al., 2015; Hoorelbeke and Koster, 2017; Segrave et al., 2014; Siegle et al., 2007; Vervaeke et al., 2021), anxiety (Vervaeke et al., 2020), and resilience (Hoorelbeke and Koster, 2017). In addition, the finding that participants reported increased use of reappraisal may suggest that increasing cognitive control may facilitate the use of adaptive emotion regulation strategies, in addition to reducing the likelihood of engaging in habitual ruminative responses. However, it should be noted that the current study relied on a single-arm design. Without a control group, it is possible that the pattern of emotional transfer effects observed in the current study may be influenced by other factors for which we did not control.

Interestingly, participants in the high- and moderate-risk groups reported a stronger reduction in stress symptoms over time compared to the low-risk group, though this observation could be due to floor effects whereby those at low risk also had few symptoms to remediate, or regression to the mean. In addition, participants in the high-risk group reported a stronger reduction in anxiety from baseline to post-training compared to the low-risk and moderate risk-group, though again this could be due to floor effects or regression to the mean. To the extent that this association reflects true remediation, it could suggest that online CCT is most strongly indicated for participants exhibiting a risk profile, as shown by relatively lower effortful control, resilience, use of adaptive emotion regulation strategies, and overall level of functioning, in combination with heightened use of RNT, self-reported internalizing symptomatology, and emotional instability. These findings are in line with the focus of preventive aPASAT training as an eHealth intervention, which is aimed at remediating cognitive control impairments to reduce risk for (recurrence of) internalizing disorders (Hoorelbeke and Koster, 2017; Hoorelbeke et al., 2021).

The moderate-risk group showed stronger improvement in terms of task-specific cognitive transfer following training compared to the high-risk group. To the extent that these findings reflect true variability as a function of risk, they suggest that the highest risk individuals would benefit from an adjusted “dose”, to foster further change in the underlying cognitive processes. Alternatively, these findings may suggest that the high-risk group may benefit from combining aPASAT training with additional interventions targeting cognitive dysfunction. In addition, as often occurs for those making large gains, but also consistent with regression to the mean, the high-risk group also demonstrated less stable emotional transfer effects following training compared to the moderate-risk group. In particular, the high-risk group reported poor retention of gains in anxiety symptoms from post-training to follow-up. This suggests a need for a more intensive training procedure or more flexible administration of CCT, in order to prevent deterioration over time following initial gains (e.g., using booster sessions). This may also point towards the need for a combined treatment approach for the high-risk group, where aPASAT training may potentially augment effects of other interventions. In this context, it should be noted that previous multi-session aPASAT training studies show strong heterogeneity in training intensity used (Koster et al., 2017). Considering the observed pattern of results, future studies could experimentally investigate the impact of training intensity on training related gains using a wide range of training intensity and explore how individual differences relate to this. Similarly, future research could investigate the role of booster sessions. In the long-term, such information, in combination with the derived User Profiles, may allow for a more individually tailored training approach.

In the current data, compared to the subjective indicator of executive functioning (effortful control), the objective indicator of executive functioning (i.e., naPASAT task performance) was not indicative for at-risk status (i.e., this indicator did not differentiate well between the low-, moderate- and high-risk group based on User Profile). This finding is line with recent findings suggesting poor correspondence between subjective and objective indicators of cognitive functioning (Hoorelbeke et al., 2016b; Mohn and Rund, 2016; Potvin et al., 2016; Van den Bergh et al., 2021), which in the context of depression may be explained by factors such as rumination, self-efficacy (Hagen et al., in press), and level of symptomatology (Serra-Blasco et al., 2019).

In contrast to the predictive role of User Profile for change in cognitive task performance and severity of internalizing symptomatology over time, and anxiety- and stress symptoms in particular (i.e., for depression only a non-significant trend was observed), User Profile did not predict change in RNT. These findings suggest that CCT was equally effective in targeting RNT in the low-, moderate-, and high-risk groups. This may partially be due to the specific sample characteristics and the recruitment strategy used, where this study was advertised as an intervention targeting RNT, potentially mostly attracting participants scoring relatively high in terms of level of RNT. In the current study, RNT was assessed using the PTQ, where scores can range from 0 to 60. The low- (M = 30.87, SD = 6.10), moderate- (M = 41.51, SD = 5.17) and high-risk groups (M = 46.46, SD = 6.61) reported relatively high levels of RNT, which on average decreased with 5.9–9.10 points from baseline to follow-up. The observed PTQ levels in the moderate- and high-risk groups were in the range of levels of RNT that were previously observed in patients diagnosed with depressive- (M = 37.56, SD = 9.99) or anxiety disorders (M = 35.93, SD = 13.60; Ehring et al., 2011). Observed PTQ scores for the low-risk group were, however, within the range of previously observed PTQ scores in nonclinical samples (Ehring et al., 2011; Ehring et al., 2012). Nonetheless, User Profile did impact effects of CCT on reappraisal. Participants in the high-risk group reported a stronger increase in reappraisal from baseline to post-training than participants in the low-risk group. In addition, participants in the high-risk group demonstrated a stronger increase in reappraisal compared to participants in the moderate-risk group.

In summary, our findings suggest that several indicators of cognitive and emotional functioning, as well as personality and age predict treatment adherence for gamified online aPASAT training. Among users of the eHealth intervention, based on indicators of cognitive- and emotional functioning as well as personality factors, we distinguished three User Profiles, potentially reflecting increasing level of risk for developing internalizing disorders. CCT may be most useful for individuals showing a moderate to high-risk for internalizing disorders based on level of effortful control, emotion regulation, resilience, internalizing symptomatology, and emotional stability. For the high-risk group, an adjusted training approach may be needed to foster long-term retention of emotional transfer effects. Building on these User Profiles, which as a first step have shown merit in predicting change in cognitive- and emotional outcomes over time in the context of web-based aPASAT training, future studies could work towards a more tailored training approach. This requires further investigation of individual differences in optimal training administration, among which level of training intensity, use of booster sessions, and possibilities for augmentation of CCT (e.g., Brunoni et al., 2014; Moshier and Otto, 2017; Segrave et al., 2014; Van den Bergh et al., 2020; Vanderhasselt et al., 2015). In addition, given that the current User Profiles are mostly based on trait measures, it would be interesting for future studies to extend this approach investigating the temporal dynamics between the constructs which emerged as key determinants of User Profile (e.g., relying on idiographic vector autoregressive network models), further evolving towards a personalized medicine approach.

At the same time, these findings should be interpreted with caution, given several limitations of the current study. First, given the focus of our study on complex moderation of treatment effects and patterns of drop-out throughout the course of CCT we did not include a control condition. As such, this study does not allow to evaluate the efficacy of CCT, but rather focusses on the application of web-based gamified CCT in a real world setting, investigating predictors of (a) adherence to CCT and (b) linking complex user profiles to heterogeneity in response to CCT. In the current study, most maladaptive processes significantly decreased and measures of adaptive processes and wellbeing increased over time. Although these findings are in line with findings from previous randomized controlled trials showing beneficial effects of CCT on emotion regulation and internalizing symptomatology (e.g., Hoorelbeke and Koster, 2017), we cannot make strong claims about the specificity of CCT on these outcomes. In addition, the current study relied on latent profile analysis to investigate patterns of complex moderation of treatment effects. Although – informed by prior CCT literature – a broad set of indicators was included in this analysis, our selection of variables is not exhaustive. As such, the possibility exists that other unmeasured variables moderate effects such as chronic trauma or illness duration as identified in other studies of similar treatments (Listunova et al., 2020). Moreover, based on the literature we aimed to recruit 300–500 participants. Prior to the end date of the project 382 participants entered the study. However, due to drop-out the latent profiles were derived from a smaller sample size. Importantly, in this context research suggests that the inclusion of a high number of indicator variables may compensate for the use of relatively small sample sizes (Wurpts and Geiser, 2014). Related to this, the sample size did not allow validation of the obtained latent profile structure in hold-out samples. Importantly, the observation that similar latent profile solutions were obtained based on BIC and ICL add to the robustness of the findings of the current study. To address this question using a meta-analytic approach would require future training studies to by default include more broad indicators of baseline functioning, taking into account potential candidates for moderation of CCT among which some of the key variables identified in the current study. In addition, the extent to which findings pertaining treatment engagement and drop out from this unguided online gamified aPASAT based CCT intervention generalize to other cognitive remediation interventions is unclear. Indeed, other studies suggest that therapist assisted eHealth interventions are more effective than non-assisted interventions (Spek et al., 2007) though this may not be the case for CCT (e.g., based on augmentation studies; Moshier and Otto, 2017; Van den Bergh et al., 2020). Furthermore, this study was promoted as an online intervention study targeting RNT. Due to the use of a general measure for perseverative thinking, the current study cannot distinguish between different types of RNT based on content of the perseverative thoughts. In addition, given the focus on moderation of training effects, participants were not pre-selected based on level of RNT. Future training studies may wish to include more specific indicators of RNT, among which indicators of rumination and worry in particular, and/or pre-screen based on level of RNT (e.g., Hoorelbeke et al., 2015) in combination with other central features of moderate- and high-risk User Profile. Related to this, given the high intensive nature of this study, due to inclusion of a high amount of indicator- and outcome variables in combination with a training procedure, we relied on one-item solutions for Fear of Math and Math Ability. These constructs are of relevance given the arithmetic nature of the task. However, the lack of inclusion of a well-validated multi-item measure for Fear of Math and Math Ability may have contributed to the pattern of findings in the current manuscript, where both variables did not emerge as significant predictors for drop-out, nor played a central role in the obtained User Profiles. Finally, due to the two-item operationalization of Agreeableness (TIPI), the inter-item reliability of the Agreeableness factor was inadequate (but see Gosling et al., 2003).

4.1. Conclusion

The current pre-registered single-arm trial set-out to: (1) investigate attrition during a web-based gamified CCT intervention targeting RNT as a transdiagnostic risk factor for internalizing psychopathology, and (2) identify different type of users of the eHealth intervention and investigate how this relates to change in cognitive and emotional outcomes following completion of CCT. Emotional stability and resilience emerged as predictors for decision not to participate in the intervention, whereas age, emotion regulation-, and personality factors were predictive for drop-out during the course of the online CCT procedure. We identified three latent profiles, corresponding with individuals who were at low-, moderate-, and high-risk for internalizing psychopathology. User Profile moderated task-specific cognitive transfer and effects of CCT on anxiety- and stress symptoms, as well as use of reappraisal. CCT was most effective for the moderate- and high-risk group. The latter group may benefit from a more intensive training procedure or the use of booster sessions to prevent deterioration over time following initial gains.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Appendix A. Supplementary data

Supplementary material