Training engagement, baseline cognitive functioning, and cognitive gains with computerized cognitive training: A cross-diagnostic study

Abstract

Computerized cognitive training (CCT) interventions are increasing in their use in outpatient mental health settings. These interventions have demonstrated efficacy for improving functional outcomes when combined with rehabilitation interventions. It has recently been suggested that patients with more cognitive impairment have a greater therapeutic response and that reduced engagement in training can identify cases who manifest low levels of benefit from treatment. Participants were psychiatric rehabilitation clients, with diagnoses of major depression, bipolar disorder and schizophrenia. Newly admitted cases received CCT, delivered via Brain HQ, with cognitive functioning divided into groups on the basis of a BACS t-score of 40 or less vs. more. Training engagement was indexed by the number of training levels achieved per day trained. Forty-nine cases trained on average for 17 days and completed a mean of 150 levels. Overall, patients improved by an average of 4.4 points (0.44 SD) in BACS t-scores (p < .001). Improvements were positively correlated with training engagement (r = 0.30, p < .05), but not with days trained (r = 0.09) or levels earned (r = 0.03) alone. Patients with higher levels of baseline cognitive performance had reduced cognitive gains (p < .003), but did not have less training engagement (p = .97). Diagnoses did not predict cognitive gains (p = .93) or target engagement (p = .74). Poorer performance at baseline and higher levels of training engagement accounted for >10% in independent variance in cognitive gains. The mean level of cognitive improvement far exceeded practice effects. The index of engagement, levels achieved per training day, is easily extracted from the training records of patients, which would allow for early and continuous monitoring of treatment engagement in CCT activities and therapist intervention as needed to improve engagement.

1. Introduction

Computerized cognitive training (CCT), often delivered with concurrent psychosocial rehabilitation as cognitive remediation therapy (CRT) has been widely used in healthy older people and in patients with schizophrenia. The evidence base for these two populations is quite broad (Harvey et al., 2018) and CRT in particular has been shown to be associated with functional gains. Meta-analyses have suggested that CCT has cognitive benefits and when delivered as CRT also has consistent functional gains (Wykes et al., 2011). The evidence base in other psychiatric conditions is less substantial and there are many fewer CCT and CRT studies in mood disorders, including bipolar disorder and major depression.

Although supported by the results of meta-analyses, there have been negative results. For instance, several trials have shown negative results (Goff et al., 2007; Kantrowitz et al., 2016; Murthy et al., 2012; Rass et al., 2012), even when paired with pharmacological interventions. It has been suggested that one possibility for some of the negative results has been failure of the training participants to actively participate in the intervention, failing to manifest engagement with training and make progress on the training procedures. It had been suggested that it would be possible to increase training engagement by adding game-like features to make the training procedures compelling (Fleming et al., 2017; Lumsden et al., 2016). However, one study found that gamification elements specifically lowered learning rates, perhaps by distracting users from the cognitive tasks themselves (Katz et al., 2014).

Recently it was also reported that there may be a specific threshold for improvement on the central training task that is required to induce transfer to untrained cognitive tasks. A previous study quantitatively analyzed the relationship between gains in auditory processing during training and overall cognitive gains and concluded that the final level of auditory speed performance predicted the magnitude of cognitive gain, and participants who did not achieve a level faster than ~85 ms did not show generalized cognitive gains (Biagianti et al., 2016).

Another possible factor associated with training gains in CRT may be baseline levels of cognitive performance. In a simple face-valid way, it would appear that individuals with psychiatric conditions who do not manifest cognitive impairments at baseline might not be candidates for interventions aimed at cognitive enhancement. In the absence of cognitive impairments, other factors might be responsible for disability. Several studies have addressed this issue. For example, Strassnig et al. (2018) reported that for schizophrenia patients with MCCB t-scores >40, there was no significant correlation between MCCB performance and independently rated indices of functional disability; whereas, for the sample of patients with more severe cognitive deficits, disability in everyday functioning was correlated with cognitive test performance. In an aggregated set of data across four cognitive training trials, Detore et al. (2019) reported that, in the entire database and across all of the studies, patients with more severe cognitive impairments had a greater benefit from CRT than patients with less impairment.

In this study, we examine cognitive gains associated with CCT in a sample of patients with mood and psychotic disorders. All patients received treatment at a psychiatric rehabilitation facility and patients were trained with CCT with the Posit Science Brain HQ training program. This program has shown efficacy in single site (Ahmed et al., 2015; Fisher et al., 2009, Fisher et al., 2010; Loewy et al., 2016; Surti et al., 2011) and multi-site trials (Fisher et al., 2014; Keefe et al., 2012) targeting people with schizophrenia, with some negative trials as described above. Further, in a large-scale (n = 150), long term (26 week; 5 days per week) randomized clinical trial (Mahncke et al., in press), active training did not separate from placebo training. However, there was evidence of substantial failure to engage in training in the active treatment group. In contrast to previous successful studies (Fisher et al., 2009; Keefe et al., 2012), performance on the index of target engagement described above was at 121 ms in the active treatment group. This is much slower than the other two studies (70 and 71 ms respectively) and the threshold suggested by Biagianti et al. (2016) of 85 ms.

We examined cognitive gains for the sample as a whole, gains in cases with greater and lesser cognitive impairments, and in each diagnostic group, as well as measuring training engagement in CCT and its association with cognitive gains on an untrained cognitive assessment measure to index near transfer. Our hypothesis was that the extent of engagement in training would be associated with cognitive gains from pretreatment to post-test. We also expected that patients with more cognitive impairment would manifest greater gains. In order to assess these variables, we performed exploratory comparative analyses of the impact on cognitive change of training engagement and baseline cognitive performance.

2. Method

2.1. Treatment site and participants

Participants were clients admitted to care at Skyland Trail, a non-profit residential, partial hospitalization, and intensive out-patient psychiatric rehabilitation facility located in Atlanta, GA. The center has a continuum of care, where more symptomatic individuals were initially placed in residential facilities and with planned transitions into day treatment, intensive outpatient, and transitional treatment tracks as symptomology and functionality improves. Less symptomatic individuals are directly admitted into outpatient services. The average length of treatment for patients in all levels of care at Skyland Trail is about 4 months. The data collected in this study were extracted from medical records and there was no additional contact with clients and no modifications of their care. At admission, all clients signed a general consent form agreeing that data in their electronic medical records may be used for research and quality improvement projects without revealing their identities. As a result, these data analyses were not submitted as a research project to an institutional review board, because all of the data collection procedures were part of ongoing standard clinical care at the facility and these analyses were aimed at determining which cases should be referred to CCT in the future.

These data were collected from May 2016 to December 2017, during which time all admissions to treatment services were administered a battery of assessments as part of the standard admissions process. All patients received a diagnosis with a structured procedure that has been previously published (Kotwicki and Harvey, 2013). This procedure included a structured interview with the MINI International Neuropsychiatric Inventory (MINI; Sheehan et al., 1998). During this time, all admissions were also tested with a neuropsychological assessment. The distribution of all diagnoses for the consecutive admissions was 23% bipolar disorder, 47% major depression, and 15% schizophrenia, with other diagnoses less common. We did not analyze data from patients whose primary diagnoses were substance abuse or personality disorders. We also examined data only from cases between the ages of 18 and 50. Cases with incomplete assessment data and those who were referred to CCT, but refused to participate, were also excluded.

The eventual sample was 52% female and 53% young adult, ages 18 to 25 years old (Mean age = 28.6, SD = 11.6). A total of 49 cases met the diagnostic criteria, completed baseline and post treatment assessments with the BACS and completed at least one day of CCT training. The diagnostic distribution of the cases was major depression (39%), bipolar disorder (39%), and schizophrenia (22%). The Brief Assessment of Cognition for Schizophrenia, paper version, (BACS; Keefe et al., 2004) was used to measure cognitive ability. CCT was delivered to patients via the online computer program Posit Science Brain HQ. The participants were instructed to practice at least 3 h of training each week, available either through the CRT group which met for 45 min each day or as independent homework.

2.2. BACS

The following 6 tests constitute the BACS. All tests with alternative forms were administered with form A first and B second.

• List Learning (Verbal Memory): Patients are presented with 15 words and then asked to recall as many as possible. This procedure is repeated 5 times. There are 8 alternate forms, of which 2 were used in this study.

• Digit Sequencing Task (Working Memory): Patients are presented with strings of numbers of increasing length. They are asked to tell the experimenter the numbers in order, from lowest to highest.

• Token Motor Task (Motor Speed): Patients are given 100 plastic tokens and asked to place them into a container as quickly as possible for 60 s.

• Verbal Fluency, Category Instances (Semantic Fluency): Patients are given 60 s to name as many words as possible within the animal category.

• Controlled Oral Word Association Test (Letter Fluency): In two separate trials, patients are given 60 s each to generate as many words as possible that begin with the letters F and S.

• Tower of London Test (Executive Functions) Patients look at two pictures simultaneously. Each shows 3 different-colored balls arranged on 3 pegs, with the balls in a unique arrangement in each picture. The patient is required to accurately estimate the fewest number of times the balls in one picture would have to be moved in order to make the arrangement of balls identical to that of the other, opposing picture.

• Symbol Coding (Attention and Motor Speed) In this test, the numbers 1–9 are coded to symbols and drawn on a response sheet for 90 s.

A composite score using previously published procedures was the primary outcome variable because the sample size did not allow for analyses of the subtests. This composite is a t-score with a mean of 50 and a standard deviation of 10. Standard interpretations of this composite score would suggest that scores of 40 or less reflect “possible cognitive impairment.”

2.3. Computerized cognitive training

Participants self-administered the training after receiving instructions, with a proctor in the room to answer questions and encourage adherence. All training used the commercially available Posit Science Brain HQ system. Participants were asked to train at least 30 min per session three days per week and to prioritize training on the “double decision” training task for at least half of the time. The rest of the training was self-selected, but if the participants asked what they should train on, they were told to prioritize brain speed, working memory, and attention tasks. Training information was taken directly from the Brain HQ portal and consisted of days spent training and levels achieved, with all information collected after the participants had completed their training. There was no attempt to directly monitor adherence with these indices during training.

2.4. Statistical methods

Total scores on the BACS composite score at baseline and endpoint were the dependent measures. We created an engagement score which was the number of levels achieved per training day and a change score which was the difference of baseline and endpoint scores on the BACS composite. We divided the patients into subgroups on the basis of their baseline composite BACS score (40 or less vs. higher). We used a two-way analysis of variance (Diagnosis × cognitive status) to examine differences in BACS change scores and target engagement. We then correlated the BACS change scores with days trained, levels achieved, and the ratio of the two. Finally, regression models were used to identify independent contributions of all variables found to be correlated with changes in cognitive performance.

3. Results

Table 1 presents the results of the BACS Composite at baseline and endpoint, days trained, levels achieved, and target engagement as a function of diagnosis and cognitive status. A paired t-test found that the BACS scores improved significantly from baseline to endpoint, t(48) = 3.38, p < .001 in the sample as a whole. The effect size for the change was d = 0.44. The two-way ANOVA examining the effect of diagnosis × cognitive status on BACS change scores found a significant effect of baseline cognitive status, F(1,48) = 10.21, p < .005, but no significant effect of diagnosis, F(2,47) = 0.08, p > .90 or diagnosis × cognitive status interactions F(2,47) = 2.24, p > .10. The two-way ANOVA examining the effect of diagnosis × cognitive status on training engagement scores found no significant effect of cognitive status, F(1,48) = 0.00, p > .95, no significant effect of diagnosis, F(2,47) = 0.31, p > .70 and no diagnosis × cognitive status interactions F(2,47) = 1.39 p > .20.

Table 1.

Scores on the BACS and cognitive training process and engagement variables: Presented by diagnosis and baseline cognitive status.

	Major depression		Bipolar disorder		Schizophrenia		Overall
	N = 19		N = 19		N = 11		n = 49
	M	SD	M	SD	M	SD	M	SD
BACS baseline	47.15	11.08	38.79	8.92	41.40	6.79	42.00	9.71
BACS endpoint	50.38	8.25	44.74	10.88	45.60	5.97	46.69	9.28
Training days	23.85	20.58	11.74	8.92	15.60	9.17	16.40	14.32
Levels achieved	209.54	150.51	98.95	117.28	135.10	88.21	141.79	129.15
Levels/training day	9.53	3.91	8.44	4.41	8.44	3.70	8.78	4.03

	Baseline BACS <41		Baseline BACS >40
	M	SD	M	SD
BACS baseline	34.26	5.80	49.08	5.99
BACS endpoint	42.35	8.55	50.23	7.49
Training days	14.48	11.34	18.46	15.49
Levels achieved	137.91	138.50	160.31	128.04
Levels/training day	8.94	4.43	9.01	3.71

Pearson correlations between changes in BACS performance from baseline to endpoint, days trained, levels achieved, and the training engagement variable are presented in Table 2. Changes in the BACS from baseline to endpoint were significantly correlated with training engagement, p < .05, but not with either days trained or levels achieved. Days trained correlated with levels attained, but not with levels per day and levels per day correlated with levels attained but not with days trained.

Table 2.

Intercorrelations of change scores on the BACS and cognitive training process and engagement variables.

	Training days	Levels achieved	Levels per day
Change in BACS scores over the training period	−0.09	0.03	0.30⁎
Training days	–	0.79⁎⁎⁎	0.00
Levels achieved	–	–	0.47⁎⁎

Note.

^⁎p < .05.

^⁎⁎p < .01.

^⁎⁎⁎p < .001.

A final analysis used the two variables found to predict cognitive improvement with training, baseline cognitive status and levels achieved per day, in a regression model. Since the two variables were not related to each other, we used a stepwise entry procedure to predict changes in cognition (Difference of BACS from Baseline to endpoint). The overall analysis was significant, F(2,46) = 8.73, p < .001. Both predictors entered with equation, with baseline cognitive status entering first, t(46) = 3.42, p < .001, accounting for 18% of the variance, followed by levels achieved per day, t(46) = 2.47, p = .02, accounting for an additional 10% of the variance.

4. Discussion

In this study in a diverse sample of psychiatric patients receiving rehabilitation therapy, two separate predictors of cognitive gains with CCT were identified. Patients with higher levels of cognitive performance made fewer training gains than patients with lower baseline performance. Further, training engagement during training was a significant and independent predictor of cognitive gains with training. The amount of variance accounted for by these two predictors in treatment-related gains in cognitive performance was quite substantial and there were several other variables that did not predict treatment gains, including diagnosis and the number of days spent training. The effect size for change was training is considerably larger than expected with practice alone (about 0.1–0.2 SD with one retest; Keefe et al., 2017), suggesting that gains are associated with training and not reassessment or placebo effects. Further, the systematic correlations with training engagement, but not simple exposure to the training program as indexed by days trained alone, support the idea that this is a CCT related cognitive gain and not an artifact of a nonrandomized research design.

Both of these predictors had been identified previously in more homogenous samples of patients (Biagianti et al., 2016; Detore et al., 2019; Fisher et al., 2009; Keefe et al., 2012). However, these data suggest that both engagement and baseline impairments are simultaneously applicable to the prediction of treatment-related cognitive gains in CCT. Conveniently, both of these variables are quite easy to measure, both prior to and during treatment. Further, these findings also suggest that, at least for patients with psychiatric diagnoses and persistent disability evidenced by rehabilitation treatment, patients with higher levels of cognitive performance do not benefit from treatment even with exertion of adequate effort.

There are a couple of important clinical points from these data. First, baseline assessment seems critical in populations where cognitive impairments may not be ubiquitous and severe. Second, monitoring of engagement should begin immediately after treatment starts. Third, it should not be expected that engagement will covary with baseline cognitive performance. Finally, as there is considerable evidence of enhanced treatment gains associated with combined skills training and CCT (Bowie et al., 2012), as well as combined CCT and social cognition training (Lindenmayer et al., 2013, Lindenmayer et al., 2018), CCT in mental health populations should probably not be offered without other training services. Although subjective reports of motivation have also previously been found to relate to training gains in CCT (Saperstein and Medalia, 2015), the current study uses a direct measure of efficiency of training gains to predict cognitive improvements with training.

The limitations of the study include the small sample size and underpowered diagnostic-group comparisons. With larger samples, the patients with MDD would likely, as expected, be found to have significantly less cognitive impairment. We also did not examine the time course of engagement in treatment, so we cannot tell if patients with poor engagement can be identified immediately. In a previous study at this site, we found that lack of treatment engagement identified immediately after admission was not amenable to targeted interventions using tangible rewards aimed at increasing engagement (Kotwicki et al., 2017). Immediate lack of engagement also predicted worse treatment outcomes. We did not have patients train on a single training procedure, so we cannot isolate the specific training that led to gains. However, the fact that training engagement with a heterogeneous training procedure led to cognitive gains may actually be a positive feature of the study, because of the ease of calculation of general training engagement. The number of training sessions is less than some previous studies with Brain HQ (Mahncke et al., in press; Fisher et al., 2009), but the effect size for gains on untrained tasks was similar to several previous studies for both Brain HQ (Fisher et al., 2010) and other strategies, including the studies reviewed in the Detore et al. (2019) meta-analysis. Finally, a randomized design could be more definitive, but even in this open study there was considerable variance in treatment outcomes that was systematically predicted by previously identified predictors.

These data suggest that assessment of baseline cognitive performance should be considered as a practice standard before engaging in CCT and CRT interventions. Further, monitoring of treatment engagement, either with task-specific indices such as those used by Fisher et al. (2009), Keefe et al. (2012), Biagianti et al. (2016), and Mahncke et al. (in press) or with more general indicators of training-related engagement such as the current study, should commence early in treatment and continuation decisions should be made quickly. Our previous findings of failures of tangible rewards to improve general engagement in rehabilitation treatment are consistent with previous suggestions that CRT interventions are not facilitated by extrinsic rewards (Saperstein and Medalia, 2015). Anecdotally, leveraging social interactions through pairing CCT participants during training sessions, and providing a therapist-led process group after training each day seemed to bolster engagement. The results of this study suggest that intrinsic motivation may not be adequate to induce CCT-related gains in patients whose baseline scores are in the unimpaired range. Although these findings replicate those of Detore et al. (2019), at least one other study has reported that higher levels of performance in certain cognitive domains leads to better gains (Lindenmayer et al., 2017). However, these patients had much more severe cognitive impairments on average than the patients in this study, being institutionalized people with schizophrenia. For example, our baseline mean cognitive performance score was a t-score of 42 (21st percentile) and mean baseline t-score in Lindenmayer et al. (2017) was 16 (0.1st percentile). Given the distributions of scores in that study, it appears as few of the participants would be expected to have had a baseline t-score of 40 or more (baseline score = 16; SD = 13; baseline plus 2 SD = 42). The reduced benefits of training in patients with higher levels of cognitive performance could be examined through re-analysis of existing datasets, as this appears to be an important topic.

In conclusion, a treatment intervention that was successful overall, leading to cognitive gains that notably exceed the expectations based on retesting alone, was maximally effective in participants with baseline levels of cognitive performance in the impaired range and in those patients who exerted consistent effort across training sessions while receiving CCT. These findings also suggest a trans-diagnostic effect of CCT in cases whose cognitive impairments are substantial enough and suggest that, particularly in patients with MDD whose cognitive impairments may be both less common and less severe, baseline assessment may be productive. Previous studies of CCT in people with MDD have reported successes (Bowie et al., 2013), particularly in treatment resistant MDD where cognitive impairments are likely to be more significant.

This research was not supported by external funding. Dr. Kotwicki and Ms. Balzer are full time employees of Skyland Trail. The CCT software and Cognitive assessment materials were purchased from their suppliers.

Conflict of interest

In the last three years, Dr. Harvey has received consulting fees or travel reimbursements from Allergan, Alkermes, Akili, Biogen, Boehringer Ingelheim, Forum Pharma, Genentech, Intra-Cellular Therapies, Jazz Pharma, Lundbeck Pharma, Minerva Pharma, Otsuka America (Otsuka Digital Health), Roche Parma, Sanofi Pharma, Sunovion Pharma, Takeda Pharma, and Teva. He receives royalties from the Brief Assessment of Cognition in Schizophrenia and the MATRICS Consensus Battery. He has a research grant from Takeda and from the Stanley Medical Research Foundation.

Dr. Kotwicki and Ms. Balzer are full-time employees of Skyland Trail.