Skip to main content
PMC home page
Internet Interventions logoLink to Internet Interventions
. 2017 Apr 29;9:7–14. doi: 10.1016/j.invent.2017.04.003

Online neurocognitive remediation therapy to improve cognition in community-living individuals with a history of depression: A pilot study

Maria Semkovska 1,⁎, Elayne Ahern 1
PMCID: PMC6096308  PMID: 30135832

Abstract

Major depression is a highly prevalent psychopathology with high relapse rates. Following remission from a depressive episode, neurocognitive difficulties in attention, working memory and executive function often persist, preventing full clinical recovery. These neurocognitive deficits are often present since the first depressive episode and have been shown to predict relapse. The efficacy of computerised neurocognitive remediation therapy (NCRT) to improve attention, memory and executive function has been demonstrated in several clinical populations but randomised controlled trials (RCT) have not been conducted in depression. The present study aimed to conduct a pilot, randomised study, of computerised NCRT for individuals with past depression, currently in remission. Twenty two individuals remitted from depression were randomly assigned to receive 20 one-hour sessions over 5 week of ether computerised NCRT or a component-equivalent allocation (play online computer games). The NCRT group showed significantly larger improvements in performance relative to the Games group in the three targeted neurocognitive domains: divided attention, verbal working memory, and planning, but also in non-targeted domains of long-term verbal memory and switching abilities. No significant effect was observed in the NCRT-targeted domain visual working memory. These preliminary results suggest computerised NCRT efficacy to improve targeted neurocognitive processes during depression remission and support its potential value as preventative connected intervention tool.

Keywords: Computerised intervention, Neurocognitive remediation therapy, Major depression, Remission, Divided attention, Working memory, Executive function, Pilot study

Highlights

  • •

    First randomised pilot of targeted online neurocognitive remediation therapy to improve cognition in remitted depression

  • •

    Participants were recruited from general practitioners referrals

  • •

    Completion rates were high (95%)

  • •

    The online intervention (NCRT) led to improvements in verbal memory, divided attention and planning abilities

  • •

    Need for a full-scale randomised controlled trial of online NCRT to assess its effectiveness for depression prevention

1. Introduction

Depression is the most prevalent mental disorder affecting about 13.5–21.2% of people during their lifetime (Hammar and Ardal, 2009). Direct costs to Europe represent 1% of its total economy and the overall depression burden is estimated at €118 billion (Sobocki et al., 2006). Depression is also the second largest cause globally for years lived with disability and the leading cause of disability for ages 15–39 (Vos et al., 2012). This substantial overall disease burden is further compounded by depression high relapse rates. After experiencing a first lifetime depressive episode, about 50 to 60% of people would develop a second episode (Beshai et al., 2011, Monroe and Harkness, 2011). The risk of depression relapse increases with each consecutive episode: about 70% relapse after a second episode and 90% relapse after a third episode. Estimated median of lifetime recurrence is 6 episodes with affected individuals spending about 21% of their life being depressed (Biesheuvel-Leliefeld et al., 2015). Recurring episodes are also associated with increasing risk of chronic depression, occurring when an episode does not remit for two years, and with an increased risk for dementia (Ownby et al., 2006).

Beyond mental and physical health issues, depression is associated with significant neurocognitive dysfunction. Numerous meta-analyses have demonstrated deficits in alertness, processing speed, sustained attention, memory and executive functioning (e.g., Ahern and Semkovska, 2017, Elderkin-Thompson et al., 2004, Rock et al., 2014). Empirical evidence further suggest that these neurocognitive deficits may worsen with repeated depressive episodes (e.g., Gorwood et al., 2008, Vanderhasselt and De Raedt, 2009), represent a significant predictor of relapse (e.g., Reppermund et al., 2009), and lead to reduced quality of life (e.g., Jaeger et al., 2006). Recent systematic reviews and meta-analyses show that, despite successful treatment (antidepressant medication and/or psychotherapy), these neurocognitive difficulties often persist following depression remission (Bora et al., 2013, Hasselbalch et al., 2011). Moreover, for some functions, this persistence of deficits following remission is already observable after recovery from a first episode of depression (Ahern and Semkovska, 2017). There is a growing consensus that neurocognitive impairment in depression cannot be fully explained by the presence or severity of mood symptoms (Bora et al., 2013, Rock et al., 2014). Therefore, improving neurocognition should be considered a key treatment goal while aiming at clinical remission, optimised recovery likelihood and relapse prevention. One such approach to therapeutically target these deficits is neurocognitive remediation therapy (NCRT).

NCRT involves the behavioural application of structured exercises with the aim of improving targeted neurocognitive processes (e.g., attention, memory) by mobilising brain plasticity that is the brain's ability to adjust its functions and connections in response to environmental change (Robertson and Murre, 1999). NCRT has demonstrated efficacy to remediate neurocognitive impairment in diverse groups of patients with neurological (e.g., for stroke, see Cicerone et al., 2011) or psychiatric conditions (e.g., for schizophrenia, see Wykes et al., 2011; for bipolar disorder, see Preiss et al., 2013). When computerised and non-computersied versions are compared, no significant differences in efficacy are usually found (e.g., Schoenberg et al., 2008, Wykes et al., 2011).

In the last ten years, several research groups have explored computerised NCRT for improving cognitive function in depression (e.g., Elgamal et al., 2007, Naismith et al., 2011). A recent systematic review and meta-analysis on the effect of computerised NCRT in depression (Motter et al., 2016) evaluated 9 studies and concluded that this intervention was associated with improved depressed mood, which could not be explained by concurrent treatment. Furthermore, this computerised intervention was associated with significantly improved daily functioning. Relatively to the targeted neurocognitive domains, attention and working memory improved significantly following computerised NCRT, while the small improvements in long-term memory and executive function did not reach statistical significance (Motter et al., 2016). The main meta-analysis's limitation consists in the high heterogeneity of the reviewed studies' design. More specifically, only five were randomised. From these, two studies (Calkins et al., 2015, Owens et al., 2013) were not using a clinical population with confirmed diagnosis (when this was the case for the remaining 7 meta-analysed studies), but a student population presenting with depressed mood as measured with the Beck Depression Inventory – II. One further study (Segrave et al., 2014) used concomitant transcranial direct current stimulation (tDCS). For the remaining two randomised studies, the control group was simply continuing receiving treatment as usual (Bowie et al., 2013, Siegle et al., 2014). Among the non-randomised observational studies, all four used waiting-list groups as a control condition (see Motter et al., 2016 for details).

To demonstrate robustly computerised NCRT efficacy, studies that are both randomised and use a “component-equivalent” control condition are essential. In fact, observed cognitive improvement following computerised NCRT, relative to a “no-cognitive-activity” control condition, can hypothetically be explained by the individual's engagement in mental activities, such as the memory and concentration required to perform computer games or for the interactions with the assisting therapist (Kurtz et al., 2007). Among the studies meta-analysed by Motter et al. (2016), only two were both randomised and used a “component-equivalent” control condition. However, as detailed above, these were conducted in non-representative samples of the general depression population, namely: students with low mood (Calkins et al., 2015) or patients treated concomitantly with tDCS (Segrave et al., 2014).

A recent randomised study in hospitalised patients with depression comparing computerised NCRT with a “component-equivalent” control, that is, playing computer games (Semkovska et al., 2015), partially supported Motter et al.' (2016) meta-analysis. More specifically, inpatients receiving 20 one-hour sessions of NCRT showed improved attention, working memory, long-term memory and planning abilities relative to inpatients involved in playing computer games for the same amount of time sessions, but both groups showed similar performance on other executive function measures. However, the improved neurocognitive performance in the NCRT group was not associated with the observed mood improvement and both groups showed equivalent depression severity decrease following the sessions' end. Here, we aimed to replicate the latter research methodology through a pilot study of patients recovered from depression completing the interventions online in the comfort of their home using the same research protocol as Semkovska et al. (2015). Although numerous effective options exist for treating depression during the acute stage, preventing depression relapse remains one of the biggest therapeutic challenges in the field (Monroe and Harkness, 2011). It is recognised that following successful therapy, depression relapse rates range from 50 to 80% once acute treatment is discontinued and 23–51% with continued treatment (Biesheuvel-Leliefeld et al., 2015). Given that neurocognitive difficulties are important predictors of depression relapse, target online NCRT could be helpful as adjunct to existing preventative therapies if reliably proven to improve targeted neurocognitive functions.

This study was planned and conducted according to recommendations for good practice in high quality randomised trial piloting design (Lancaster et al., 2004). It aimed to pilot the use of online NCRT as a connected intervention tool to improve cognition in community-living individuals with a history of at least one major depressive episode and survey participants' attitudes towards this intervention. Specific objectives were:

  • 1.

    To collect feasibility data regarding recruitment rates, intervention adherence, retention rates and acceptance of randomisation in preparation for a full-scale randomised controlled rater-blinded trial assessing the effectiveness of online NCRT for improving cognition in recurrent depression;

  • 2.

    To obtain preliminary data on the effectiveness of online NCRT as a connected intervention tool to improve targeted neurocognitive function (i.e., divided attention, working memory and planning);

  • 3.

    To quantitatively explore participants' attitudes and expectations and experience relative to computerised NCRT as a connected intervention tool.

  • 4.

    To compare the NCRT outcomes obtained in depression remitters to the results from Semkovska et al. (2015) obtained in inpatients treated for acute depression.

2. Methods

Recommendations for good practice in randomised trials piloting design (Lancaster et al., 2004) were followed for the development and analysis of the present research. Published guidelines for reporting the results of pilot investigation in preparation of RCTs using the Consolidated Standards of Reporting Trials were used (Thabane et al., 2010).

2.1. Participants and procedure

This pilot has received ethical approval from the Irish College of General Practitioners. Forty four local general practitioners (GPs) were contacted for participation in this pilot. Seven GPs agreed to participate. Based on our inclusion/exclusion criteria (see below), they referred, over a period of four months (17 weeks), 132 potentially eligible participants. More specifically, each GP referred patients who s/he has treated for depression in the past four years but who the GP now considered clinically remitted from depression. All 132 referred patients were approached to participate in the study through postal invitation containing the Information Sheet and 34 of them contacted us back. These 34 patients were assessed at the University of Limerick against the inclusion/exclusion criteria detailed below, Twenty two patients met the inclusion criteria and were randomised to receive either cNCRT (n = 11) or complete online computer games (n = 11) for 20 one-hour sessions. See flow-chart of recruitment in Fig. 1.

Fig. 1.

Fig. 1

Open in a new tab

Flow chart of participants' screening, recruitment, treatment adherence and assessment completion.

In terms of inclusion criteria, the study targeted community-living participants aged 18 to 65, with a past history of at least one major depressive episode (MDE) as confirmed with the Structured Clinical Interview for DSM-IV disorders (First et al., 1994), but currently in remission. Remission was defined as participant having: 1) not fulfilled criteria for a MDE (First et al., 1994) for at least 8 weeks; and 2) obtained a score of 7 or less on the 17-item Hamilton Rating Scale for Depression (HDRS-17) upon recruitment. All participants provided informed written consent to enter the study. Exclusion criteria were: another current or past Axis I diagnosis; alcohol or other substance abuse in the previous six months; treatment with electroconvulsive therapy in the previous 12 months; neurological disorder; history of head injury with loss of consciousness, a major sensory or motor deficit preventing standard computer use; and lack of computer or the internet access at the participant's residence.

Following recruitment, a standardised battery including neurocognitive tests, depression severity assessments and questionnaires relative to computer expertise and attitudes towards the online interventions was administered face-to-face at the University of Limerick (UL, Department of Psychology) by an evaluator blinded to the treatment arms. The administration time of the battery varied between 40 and 60 min, depending on the participant's performance and/or need for breaks in-between the tests. Participants were then randomised to either receive online NCRT or complete online computer games for five weeks with 4 one-hour individual sessions weekly (20 sessions overall). Allocation to each intervention arm was stratified by gender and generated by pre-programmed command with the statistical package R (WU Wien, 2011). During the intervention period, each participant, regardless of group allocation, received four times a week, on different, random days/times a reminder to complete their four weekly one-hour sessions. This was performed through programmed online webtexts sent from the pilot's mobile phone web account. Within three days of the final session, the standardised battery was again administered face-to-face at the UL Department of Psychology by an evaluator blinded to the intervention condition.

2.2. Intervention arms

RehaCom (Hasomed, Germany), an online computerised intervention package was used for NCRT. Following allocation to the treatment arm, participants received an e-mail that contained a weblink for downloading RehaCom on their computer, along with their assigned username and personal password needed for accessing prescribed sessions. We used the same six RehaCom procedures as in Semkovska et al. (2015), namely, the Divided Attention (1&2), Verbal Memory, Figural memory, Shopping and Plan a day. These procedures were selected as targeting remediation of neurocognitive domains known to be impaired in depression, that is selective and divided attention, visual and verbal working memory, and everyday planning (Ahern and Semkovska, 2017, Hammar and Ardal, 2009). Each procedure has multiple levels of difficulty. The number of levels varied from 14 (Divided Attention 1) to 55 (Plan a day). Shopping had 18 levels, while Divided Attention 2, Figural memory and Verbal Memory had, respectively, 22, 27 and 30 levels.

Participants need to achieve specific success rates to progress to the next level within each procedure. Each individual progresses at her/his own pace and is brought back to a lower level if s/he fails to achieve the success requirements of a given level. Within each 60-minute session, the participant initially completes three of these procedures for twenty minutes each. The procedures' administration order and the initial start level within each procedure were standardised. For example, the first session always begins with 20 min of Verbal Memory, starting at level 3, followed by 20 min of Divided attention 1, starting at level 3 and followed by 20 min of Shopping, starting at level 1. However, the progression from one level to the next depends on each participant's pace. Thus, although the procedures' administration order and initial levels are standardised, subsequent levels for each procedure are determined by each individual's previous level of achievement. See Table 1 for details on the components and duration of each of the 20 RehaCom one-hour sessions prescribed for participants in the oNCRT group.

Table 1.

Components and duration of each of the 20 RehaCom one-hour sessions prescribed for participants in the oNCRT group.

Week 1 Week 2 Week 3 Week 4 Week 5
Session 1 Session 5 Session 9 Session 13 Session 17
Verbal Memory (20 min)
 start at level 3
Divided Attention 1 (20 min)
 start at level 3
Shopping (20 min)
 start at level 1		 Verbal Memory (20 min)a
Divided Attention 1 (20 min)a
Shopping (20 min)a 		Verbal Memory (20 min)a
Divided Attention 1 OR 2 (20 min)a
Shopping (20 min)a 		Verbal Memory (30 min)a
Divided Attention 1 OR 2 (30 min)a 		Verbal Memory (30 min)a
Shopping (30 min)a
Session 2 Session 6 Session 10 Session 14 Session 18
Figural memory (20 min)
 start at level 3
Divided Attention 1 (20 min)a
Plan a day (20 min)
 start at level 1		 Figural memory (20 min):a
Divided Attention 1 (20 min)a
Plan a day (20 min)a 		Figural memory (20 min)a
Divided Attention 1 OR 2 (20 min)a
Plan a day (20 min)a 		Shopping (20 min)a
Divided Attention 1 OR 2 (20 min)a
Plan a day (20 min)a 		Divided Attention 1 OR 2 (30 min)a
Plan a day (30 min)a
Session 3 Session 7 Session 11 Session 15 Session 19
Verbal Memory (20 min)a
Divided Attention 1 (20 min)a
Shopping (20 min)a 		Verbal Memory (20 min)a
Divided Attention 1 (20 min)a
OR Divided Attention 2
start at level 1
Shopping (20 min)a 		Verbal Memory (20 min)a
Divided Attention 1 OR 2 (20 min)a
Shopping (20 min)a 		Verbal Memory (30 min)a
Divided Attention 1 OR 2 (30 min)a 		Verbal Memory (30 min)a
Divided Attention 1 OR 2 (30 min)a
Session 4 Session 8 Session 12 Session 16 Session 20
Figural memory (20 min)a
Divided Attention 1 (20 min)a
Plan a day (20 min)a 		Figural memory (20 min)a
Divided Attention 1 OR 2 (20 min)a
Plan a day (20 min)a 		Figural memory (20 min)a
Divided Attention 1 OR 2 (20 min)a
Plan a day (20 min)a 		Divided Attention 1 OR 2 (30 min)a
Plan a day (30 min)a 		Divided Attention 1 OR 2 (30 min)a
Plan a day (30 min)a
Open in a new tab

Note: If constant progression with no errors, Divided Attention 1 can be completed in 6 sessions; once the 14th level of Divided Attention 1 is completed Divided Attention 2 is introduced.

a

Starting level on the session depends on the participant's level achieved at the end of the previous session involving this component.

RehaCom collects data relevant to participant adherence, including but not limited to length of time spent on each level of a procedure, number of errors, type of errors, and reaction times to targets. These data were collected but not analysed. Adherence was measured as a dichotomous variable: ether completed full session or did not complete full session. For example, if a participant started a session, but discontinued before completing the 60 min, this was coded as non completion. The participant could however start over this session the next day or later.

The computer game condition involved performing online games requiring selective attention, using strategy and remembering cues, such as puzzles or avoiding monsters in a labyrinth (e.g., Free Online Games, 2014, Word Games, 2014). Within each 60-minute session, the participants were engaged in at least two and no more than three different games of their choice among eight different online games. The link that was sent to the participants in the game condition did not allow them to play more than four different games per week, but they still had the choice among the original eight at the start of each week. These online games also have different levels of difficulty that varied depending on the participant's performance. The time spent on each game was left to the participant, but they were required to spend at least ten minutes on a given game. Each participant also kept a log of number of games played within each session and time spent at each game.

2.3. Outcome measures

Upon recruitment, the National Adult Reading Test (Nelson and Willison, 1991) was used to estimate premorbid intellectual functioning and the Computer Proficiency Scale (CPS, Boot et al., 2013) to quantify participant's level of computer use and expertise. The CPS contains 20 items, each scored on a 0 to 3 points Likert scale. Total score varies from 0 to 60, with a score of 0–29 suggesting little computer/internet proficiency; 30 to 45 suggesting regular use, but below average proficiency, 46 to 55: average computer/internet proficiency and above 55: above average computer/internet proficiency. Neurocognitive functioning was assessed by a standardised battery including the same validated neuropsychological tests as administered to inpatients' in the previous study (Semkovska et al., 2015). These included: the Digit Symbol Substitution (Wechsler, 1997a) to assess psychomotor processing speed; the d2 Selective attention test (Brickenkamp, 1966) for divided attention, the Digit Span Forward (Wechsler, 1997a) for auditory attention; the Digit Span Backward (Wechsler, 1997a) for verbal working memory; the Logical memory-I&II (Wechsler, 1997b) for, respectively, verbal learning and retention; the Rey-Osterrieth Complex Figures (ROCF, Osterrieth, 1944) for visual learning (ROCF Immediate Recall) and retention (ROCF Delayed Recall); and the following Delis-Kaplan Executive Function System's (DE-KEFS, Delis et al., 2001) subtests: for the assessment of the following executive functions: Verbal Fluency (three consecutive categories) for self-regulation under external constraints, Fluency Switching for mental flexibility, Towers for planning, and 20-Questions for abstract thinking. Alternative versions to control for practice effects were used at Post-Treatment for all neuropsychological tests. These tests are known to measure neurocognitive functions affected by a major depressive episode (Lezak et al., 2012). Severity of depressive symptomatology was assessed objectively with the 17-item Hamilton Depression Rating Scale (HRSD; Hamilton, 1967) and subjectively with the Beck Depression Inventory-II (BDI-II; Beck et al., 1996) to control for the contribution of mood on observed outcomes. Additionally, to control for the influence of participants' attitudes and expectations of online NCRT on the observed outcomes, the Psychotherapy Expectations, Concerns and Hopes Inventory (Moritz et al., 2013) was administered both before randomisation and at the end of the treatment period. More specifically, scores on the “Hope” (about daily life improvement attributable to the intervention), the “Skill Acquisition” (expectations of the intervention) and “Scepticism” (towards online NCRT) subscales were extracted.

3. Results

3.1. Feasibility study

As detailed in the procedure subsection of the Methods, 132 potentially eligible participants, as identified by their GP over 17 weeks, were invited to participate. Twenty six percent (n = 34) agreed to participate and from these, 65% (n = 22) met the inclusion criteria, corresponding to an average recruitment rates of 1.3 participants per week. None of the participants were receiving psychotherapy at the time of recruitment; however 50% were taking antidepressants (see Table 2 for details).

Table 2.

Samples' characteristics: demographic, pre-treatment neurocognitive, depressive symptom expression and attitudes/expectations towards online NCRT data presented as means and standard deviations.

NCRT group (n = 11) Games (n = 11) Difference between groups
t(20) p
Age 45.9 (6.7) 46.9 (9.3) 0.29 0.78
Gender ratio women/men (% women) 9/2 (82%) 9/2 (82%) – –
Education (years) 12.3 (1.5) 11.2 (2.2) 1.42 0.17
Number of episodes 2.81 (1.3) 2.9 (3.2) 0.088 0.93
Time in remission (months) 23.3 (14.0) 17.2 (14.4) 1.01 0.33
Current medication (number of participants)
 No medication 6 5 – –
 Selective serotonin reuptake inhibitors (escitalopram vs citalopram vs sertraline) 4
(3 vs 1 vs 0)		 6
(5 vs 0 vs 1)		 – –
 Selective serotonin and norepinephrine reuptake inhibitors (venlafaxine) 1 0 – –
Boot Computer Proficiency Scale 52.1 (6.7) 51.7 (10.7) 0.10 0.93
NART-IQ 105.6 (7.4) 103.4 (5.9) 0.80 0.43
Digit Symbol Substitution (WAIS-III) 71.4 (17.8) 60.8 (24.4) 1.16 0.26
d2 Selective Attention Test: Total correct 434.0 (77.9) 378.5 (87.5) 1.54 0.14
Digit Span Forward (WAIS-III) 10.5 (2.5) 9.8 (2.3) 0.62 0.54
Digit Span Backward (WAIS-III) 6.2 (2.0) 5.9 (2.0) 0.32 0.75
Logical Memory-I: Total (WMS-III) 45.0 (11.6) 36.8 (15.2) 1.42 0.17
Logical Memory-II: % retention (WMS-III) 84.9 (6.8) 81.1 (11.8) 0.93 0.37
ROCF immediate recall 20.3 (5.7) 16.0 (7.6) 1.52 0.15
ROCF delayed recall 18.7 (6.1) 14.8 (6.5) 1.45 0.16
Towers (D-KEFS) 16.0 (3.6) 15.1 (3.0) 0.65 0.53
Verbal fluency (D-KEFS) 39.0 (8.0) 38.5 (8.5) 0.16 0.88
Fluency Switching (D-KEFS) 14.3 (3.2) 11.8 (3.3) 1.78 0.091
20-questions (D-KEFS) 11.0 (1.8) 9.6 (3.7) 1.10 0.28
Hamilton Depression Rating Scale 4.5 (2.3) 4.0 (2.8) 0.50 0.62
Beck Depression Inventory-II 7.7 (3.2) 5.1 (4.1) 1.69 0.11
PECHI hope 29.6 (17.0) 26.8 (8.8) 0.47a 0.64
PECHI expectations of skill acquisition 11.6 (3.8) 12.2 (2.0) 0.42a 0.68
PECHI scepticism towards intervention 3.63 (3.4) 3.5 (3.9) 0.086a 0.93
Open in a new tab

D-KEFS Delis-Kaplan Executive Function System (Delis et al., 2001); NART-IQ National Adult Reading Test estimated Intellectual Quotient (Nelson and Willison, 1991); PECHI Psychotherapy Expectations, Concerns and Hopes Inventory (Moritz et al., 2013); ROCF Rey-Osterrieth Complex Figures (Osterrieth, 1944); WAIS-III Wechsler Adult Intelligence Scale-III (Wechsler, 1997a); WMS-III Wechsler Memory Scale-III (Wechsler, 1997b).

a

df = 19, one participant from the game group declined completing the questionnaire.

Twenty-two participants (Mean age = 46.4, Standard Deviation [SD] = 7.9, range 31–65; 18 women, 4 men) were thus randomised to receive either NCRT or complete computer games for 20 one-hour sessions. There was no difference between the two groups in terms of age or levels of computer and internet use (see Table 2). Acceptance of randomisation was excellent: no participant refused their assigned treatment allocation. The adherence to the prescribed sessions was excellent: participants completed 417 of 440 prescribed sessions; thus, overall, 95% of all sessions were completed as prescribed. More specifically, 17 participants (77%) completed all 20 sessions and 95% of participants completed at least 75% of the assigned sessions. See Fig. 1 for details on completion rates per group. Mean number of completed sessions was 18.95 (SD = 2.4) and there was no difference in adherence to the assigned session between the two groups: NCRT Mean = 19.0 (SD = 2.0); Games Mean = 18.91 (SD = 2.8); t(20) = 0.088, p = 0.93. Twenty one participants returned to complete the post-intervention outcome assessment leading to an overall 95% pilot completion rate.

3.2. Pilot online NCRT effectiveness assessment

Table 2 presents the Pre-Treatment results of the two groups. Both groups' average Pre-Treatment performance was inferior to published normative samples mean on all neurocognitive tests, with the exception of performance on Verbal Fluency and 20-questions. Post-Treatment outcome data were obtained for 21 of the 22 initially recruited participants.

Similarly to the study conducted in hospitalised patients with depression (Semkovska et al., 2015) and results from previous studies involving regular game playing in non-clinical samples (Green and Bavelier, 2012), it was expected that both the online NCRT and online games groups might improve on tests involving speed of processing and/or attention control. Furthermore, in line with the inpatient study, it was also expected that at the end of the intervention period, the NCRT group would have improved significantly more than the games group in their performance on neurocognitive tests measuring functions targeted by the NCRT (i.e., divided attention, visual and verbal working memory, and everyday planning). No significant between-groups differences in improvement on the other neurocognitive tests were expected.

Effects of treatment group (NCRT; Games) and time (Pre-Treatment; Post-Treatment) on neurocognitive performance were tested with 2 × 2 repeated measures analyses of variance (ANOVAs). Before conducting the between-groups ANOVAs, depression severity was investigated as possible covariate of neurocognitive performance. For this, at both time points, Pearson correlations explored the possible association between depression severity and neurocognitive performance. No significant correlations were found: at Pre-Treatment, all p-values were above 0.08, at Post-Treatment: p > 0.11. See Table 3 for results on both groups' Post-Treatment performance and between-groups ANOVAs.

Table 3.

Post-treatment neurocognitive and depressive symptom expression data presented as Means, Standard Deviations and repeated-measures ANOVAs results.

NCRT group (n = 11) Games (n = 10) Effect of time
 Effect of group
 Time × group interaction
F (1,19) p F (1, 19) p F (1,19) p
Digit Symbol Substitution 75.3 (14.9) 67.9 (22.0) 14.95 0.001 0.72 0.41 0.046 0.83
d2 Selective Attention 501.1 (66.1) 403.0 (86.6) 28.99 < 0.001 5.44 0.031 7.05 0.016
Digit Span Forward 11.3 (2.1) 9.8 (1.7) 1.77 0.19 1.45 0.24 1.77 0.20
Digit Span Backward 8.0 (2.9) 5.7 (1.7) 7.9 0.011 2.05 0.17 11.81 0.003
Logical Memory-I: Total 45.5 (8.9) 37.1 (10.6) 0.053 0.82 3.01 0.098 0.003 0.96
Logical Memory-II: % retention 94.5 (7.7) 78.4 (13.6) 4.46 0.047 5.87 0.025 14.36 0.001
ROCF immediate recall 22.2 (5.2) 18.7 (5.3) 10.51 0.004 1.82 0.19 0.46 0.51
ROCF delayed recall 22.1 (5.5) 16.9 (6.4) 7.06 0.016 2.80 0.11 1.42 0.25
Towers 22.0 (2.9) 15.9 (2.8) 46.64 < 0.001 8.24 0.009 26.9 < 0.001
Verbal Fluency 42.2 (6.0) 39.6 (4.4) 3.54 0.074 0.33 0.57 0.74 0.40
Fluency Switching 17.4 (4.3) 12.0 (3.0) 9.17 0.007 7.97 0.011 7.24 0.014
20-questions 11.2 (1.33) 10.4 (3.4) 1.98 0.17 0.93 0.35 0.71 0.41
Hamilton Depression Rating Scale 5.5 (4.0) 5.0 (4.8) 1.22 0.28 0.16 0.70 0.003 0.96
Beck Depression Inventory-II 8.3 (7.6) 7.3 (8.6) 0.85 0.37 0.66 0.43 0.31 0.59
Open in a new tab

Consistent with our hypotheses, the NCRT group improved significantly more than the Games group on the tests of divided attention (d2 Test), verbal working memory (Digit Span Backward) and planning (D-KEFS Towers). However, both groups improved equally on the tests of visual working memory (ROCF immediate recall). Furthermore, the NCRT group significantly out-performed the Games group in delayed verbal memory retention (WMS-III Logical Memory-II) and in switching organisation of own thinking (D-KEFS Category Switching fluency). Both groups improved equally in their performance on the tests of psychomotor speed (Digit Symbol Substitution), Verbal Category Fluency and delayed visual memory (ROCF delayed recall). Neither group showed a significant change in performance on abstract thinking (20 questions).

3.3. Effect of individual's attitudes, expectations and experience of online NCRT on neurocognitive performance

One participant from the Games group declined to complete this questionnaire. There were no significant pre-randomisation differences between the two groups in terms of hopes about daily life improvement, expectations of skill acquisition or scepticism towards online NCRT as a connected intervention tool (see Table 2). The possibility that higher expectations towards the NCRT might increase the difference in neurocognitive performance between the two groups was explored through Persons correlations between change in neurocognitive performance, and both Pre- and Post-Treatment scores on the three PECHI sub-scales. No significant correlations were observed with all p-values > 0.18.

3.4. Post-hoc assessment of the effect of clinical status on neurocognitive performance

The present pilot aimed using the same protocol administered by Semkovska et al. (2015, n = 24) in a pilot study of individuals recovered from depression who completed the interventions online in the comfort of their home. Therefore, it was possible to conduct post-hoc analyses comparing the two samples. To explore the effect of clinical status on change in neurocognitive performance (Post-Treatment–Pre-Treatment), 2 × 2 ANOVAs of the effects of clinical status (depressed vs remitted) and treatment group (NCRT; games) were conducted on all neurocognitive variables. The effect of the intervention group was overall consistent with the above-described Results' Section 3.2, namely a significant effect of group, with the NCRT games performing better than the Games group, was observed on divided attention: F(1, 35) = 13.88, p = 0.001, verbal working memory: F(1, 36) = 16.63, p < 0.001, delayed verbal memory: F(1, 36) = 18.08, p < 0.001, planning: F(1, 36) = 26.23, p < 0.001, and Switching Fluency: F(1, 36) = 20.03, p < 0.001. There was additionally a group effect on both working and delayed visual memory: F(1, 35) = 24.43, p < 0.001 and F(1, 35) = 21.23, p < 0.001, respectively. The latter was possibly partly due to the fact that currently depressed participants in the NCRT group improved significantly more than the Games group on these variables. Indeed, there was an effect of clinical status on working visual memory: F(1, 35) = 21.07, p < 0.001. Furthermore, there was an interaction between treatment group and clinical status on both these visual variables (F(1, 35) = 18.10, p < 0.001 for ROCF immediate recall and F(1, 35) = 9.58, p = 0.004 for ROCF delayed recall), with participants that were acutely depressed when receiving NCRT beneficiating significantly more from this intervention than participants who were in remission. There were no other treatment versus clinical status significant interactions (all other p-values > 0.26).

4. Discussion

The present study aimed to assess the feasibility of conducting a randomised study of online neurocognitive remediation therapy (NCRT) in community-living, currently remitted, individuals with a history of depression, collect preliminary data on its effectiveness to improve neurocognitive function in this population, explore the potential contribution of participants attitudes and expectations to the observed results, and compare these depression-remission outcomes to results previously obtained in acutely depressed individuals treated with NCRT.

Overall, the present pilot results are very satisfactory and add to the growing empirical support of the NCRT benefits as an adjunct to usual treatment in individuals with a history of depression. Recruitment rates for the study were very good and compliance rates: excellent. Compliance rates (95%) were better that the ones observed in the inpatients' study (62%), and appeared superior to adherence/retention rates reported in meta-analyses of similar studies ranged from 63% to 67% (Hunsley and Lee, 2007, Rutherford et al., 2012). This finding could suggest that providing the possibility to complete the intervention at home during times most convenient to the individual optimises adherence. Expectations towards the intervention in terms of acquiring new skill or daily life improvement varied across individuals, but did not appear to influence adherence or neurocognitive outcome results. Indeed, no significant correlations were found between the three PECHI variables, compliance rates and neurocognitive tests performance.

Neurocognitive outcomes are consistent with previous studies involving the assessment of computerised NCRT in individuals with depression. More specifically, in line with Motter et al. (2016) meta-analysis, improvement in targeted divided attention and verbal working memory were observed. The improved visual working memory that was observed in the parent study in inpatients was not observed in depression remitters, despite this function being also targeted by the intervention. Visual working memory deficits observed during a depressive episode are usually small (e.g., Rock et al., 2014). Existing meta-analyses in depression remission have not reported on this function due to insufficient studies measuring it following a depressive episode (e.g., Bora et al., 2013). From these reviews, we could speculate that, possibly, visual working memory is not adversely affected by depression once remission is achieved. A potential clinical implication of the current results in light of existing literature would be to only therapeutically target visual memory during a depressive episode or if specific impairments in this neurocognitive function were observed during remission. A full-scale randomised controlled trial could further test this hypothesis.

Our NCRT programme did not target delayed memory and yet this intervention was associated with better long-term retention of verbal material in the group receiving it relative to the Games group. The same pattern was observed in the inpatients' study (Semkovska et al., 2015), giving further support to the hypothesis of possible generalisability of the targeted NCRT effects. More specifically, improving, through a targeted intervention, verbal working memory may lead to a better verbal encoding and, consequently, better long-term retention of the material.

Contrary to Motter et al. (2016) meta-analytical results, the present study did not find an association between depressive symptoms and neurocognitive improvement or between depressive symptoms and NCRT. However, these findings observed during depression remission were in line with the previous randomised study in patients hospitalised for depression (Semkovska et al., 2015). Differences in research design between our protocol and most of the meta-analysed studies might explain these discrepancies. Indeed, most of the meta-analysed studies used a waiting list or treatment as usual as a control condition, that is, without any therapist contact or other equivalent in time commitment mental activities (e.g., Naismith et al., 2011, Siegle et al., 2014). Therefore, we can speculate that the gains observed in participants' mood symptoms following the intervention were possibly due to the interactions with the NCRT therapists and/or the simple activation obtained by engaging in additional purposeful mental activities that was not available to participants in the control group. In the present study, both group engaged for a similar amount of time in a mental activity involving a computer, which might have removed this non-specific to the intervention effect on participants' mood. A larger randomised controlled trial comparing NCRT to two independent other arms (i.e., treatment as usual and an active control) could further evaluate the usefulness of NCRT as mood enhancer. Another hypothesis would be that NCRT improves neurocognitive independently from mood variations.

The present pilot evaluated if targeted NCRT could improve neurocognitive functioning in individuals remitted from depression. This study did not include an assessment of daily functioning or a follow-up assessment. These represent the main limitations of the study's protocol, as it forebode evaluating if neurocognitive could improvements generalise to everyday function and if observed gains persisted. Furthermore, significant findings from pilot studies should be considered only as very preliminary and the temptation of over-interpreting them should be avoided (Lancaster et al., 2004). However, the study added to the growing body of evidence in favour of computerised NCRT for depression through a robust randomised, active-control, research design demonstrating its potential efficacy to improve specifically neurocognition during remission. An adequately powered full-scale randomised controlled trial with long-term follow-ups needs to establish if these neurocognitive benefits generalise to everyday function as suggested by Motter et al. (2016) and/or persist with the passage of time. Such golden-standard high level empirical evidence could lead to the implementation of computerised NCRT as a neurocognitive enhancer add-on to the usual mood-focused treatments, but also as a preventative connected intervention tool.

Funding

This work was supported by awards from the EHS Faculty University of Limerick (Principal Investigator Seed Funding) and the Irish Research Council (New Foundations 2015) to Maria Semkovska.

Conflict of interest

There is no conflict of interest concerning this article.

Acknowledgments

The authors are thankful to Emma Taylor and Diarmaid Ó Lonargáin for assistance in testing study participants.

References

  1. Ahern E., Semkovska M. Cognitive functioning in the first-episode of major depressive disorder: a systematic review and meta-analysis. Neuropsychology. 2017;31:52–72. doi: 10.1037/neu0000319. [DOI] [PubMed] [Google Scholar]
  2. Beck A.T., Steer R.A., Brown G.K. The Psychological Corporation; San Antonio TX: 1996. Beck Depression Inventory-Second Edition. [Google Scholar]
  3. Beshai S., Dobson K.S., Bockting C.L.H., Quigley L. Relapse and recurrence prevention in depression: current research and future prospects. Clin. Psychol. Rev. 2011;31:1349–1360. doi: 10.1016/j.cpr.2011.09.003. [DOI] [PubMed] [Google Scholar]
  4. Biesheuvel-Leliefeld K.E., Kok G.D., Bockting C.L., Cuijpers P., Hollon S.D., van Marwijk H.W., Smit F. Effectiveness of psychological interventions in preventing recurrence of depressive disorder: meta-analysis and meta-regression. J. Affect. Disord. 2015;174:400–410. doi: 10.1016/j.jad.2014.12.016. [DOI] [PubMed] [Google Scholar]
  5. Boot W.R., Charness N., Czaja S.J., Sharit J., Rogers W.A., Fisk A.D.…Nair S. 2013. Computer Proficiency Questionnaire: Assessing Low and High Computer Proficient Seniors, Gerontologist. [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Bora E., Harrison B.J., Yücel M., Pantelis C. Cognitive impairment in euthymic major depressive disorder: a meta-analysis. Psychol. Med. 2013;43:2017–2026. doi: 10.1017/S0033291712002085. [DOI] [PubMed] [Google Scholar]
  7. Bowie C.R., Gupta M., Holshausen K., Jokic R., Best M., Milev R. Cognitive remediation for treatment-resistant depression effects on cognition and functioning and the role of online homework. J. Nerv. Ment. Dis. 2013;201:680–685. doi: 10.1097/NMD.0b013e31829c5030. [DOI] [PubMed] [Google Scholar]
  8. Brickenkamp R. Editest; Paris: 1966. Le test d2 d'attention concentrée. [Google Scholar]
  9. Calkins A.W., McMorran K.E., Siegle G.J., Otto M.W. The Effects of Computerized Cognitive Control Training on Community Adults with Depressed Mood. Behav. Cogn. Psychother. 2015;43:578–589. doi: 10.1017/S1352465814000046. [DOI] [PubMed] [Google Scholar]
  10. Cicerone K.D., Langenbahn D.M., Braden C., Malec J.F., Kalmar K., Fraas M.…Ashman T. Evidence-based cognitive rehabilitation, updated review of the literature from 2003 through 2008. Arch. Phys. Med. Rehabil. 2011;92:519–530. doi: 10.1016/j.apmr.2010.11.015. [DOI] [PubMed] [Google Scholar]
  11. Delis D.C., Kaplan E., Kramer J.H. The Psychological Corporation; San Antonio TX: 2001. Delis-Kaplan Executive Function System (D-KEFS) [Google Scholar]
  12. Elderkin-Thompson V., Boone K.B., Hwang S., Kumar A. Neurocognitive profiles in elderly patients with frontotemporal degeneration or major depressive disorder. J. Int. Neuropsychol. Soc. 2004;10:753–771. doi: 10.1017/S1355617704105067. [DOI] [PubMed] [Google Scholar]
  13. Elgamal S., McKinnon M.C., Ramakrishnan K., Joffe R.T., MacQueen G. Successful computer-assisted cognitive remediation therapy in patients with unipolar depression: a proof of principle study. Psychol. Med. 2007;37:1229–1238. doi: 10.1017/S0033291707001110. [DOI] [PubMed] [Google Scholar]
  14. First M.B., Spitzer R.L., Gibbon M., Williams J.B.W. Biometrics Research; New York: 1994. Structured Clinical Interview for Axis I DSM-IV Disorders. [Google Scholar]
  15. Free Online Games 2014. http://www.freeonlinegames.com/game/pacman Pacman at.
  16. Gorwood P., Corruble E., Falissard B., Goodwin G.M. Toxic effects of depression on brain function: impairment of delayed recall and the cumulative length of depressive disorder in a large sample of depressed outpatients. Am. J. Psychiatry. 2008;165:731–739. doi: 10.1176/appi.ajp.2008.07040574. [DOI] [PubMed] [Google Scholar]
  17. Green C.S., Bavelier D. Learning, attentional control, and action video games. Curr. Biol. 2012;22:R197–R206. doi: 10.1016/j.cub.2012.02.012. [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Hamilton M. Development of a rating scale for primary depressive illness. Br. J. Soc. Clin. Psychol. 1967;6:278–296. doi: 10.1111/j.2044-8260.1967.tb00530.x. [DOI] [PubMed] [Google Scholar]
  19. Hammar A., Ardal G. Cognitive functioning in major depression – a summary. Front. Hum. Neurosci. 2009;3:26. doi: 10.3389/neuro.09.026.2009. [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Hasselbalch B.J., Knorr U., Kessing L.V. Cognitive impairment in the remitted state of unipolar depressive disorder: a systematic review. J. Affect. Disord. 2011;134:20–31. doi: 10.1016/j.jad.2010.11.011. [DOI] [PubMed] [Google Scholar]
  21. Hunsley J., Lee C.M. Research-informed benchmarks for psychological treatments: efficacy studies, effectiveness studies, and beyond. Prof. Psychol. Res. Pr. 2007;38:21–33. [Google Scholar]
  22. Jaeger J., Berns S., Uzelac S., Davis-Conway S. Neurocognitive deficits and disability in major depressive disorder. Psychiatry Res. 2006;145:39–48. doi: 10.1016/j.psychres.2005.11.011. [DOI] [PubMed] [Google Scholar]
  23. Kurtz M.M., Seltzer J.C., Shagan D.S., Thime W.R., Wexler B.E. Computer-assisted cognitive remediation in schizophrenia: what is the active ingredient? Schizophr. Res. 2007;89:251–260. doi: 10.1016/j.schres.2006.09.001. [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Lancaster G.A., Dodd S., Williamson P.R. Design and analysis of pilot studies. Recommendations for good practice. J. Eval. Clin. Pract. 2004;10:307–312. doi: 10.1111/j..2002.384.doc.x. [DOI] [PubMed] [Google Scholar]
  25. Lezak M.D., Howieson D.B., Bigler E.D., Tranel D. fifth ed. Oxford Univeristy Press; New York, NY: 2012. Neuropsychological Assessment. [Google Scholar]
  26. Monroe S.M., Harkness K.L. Recurrence in major depression: a conceptual analysis. Psychol. Rev. 2011;118:655–674. doi: 10.1037/a0025190. [DOI] [PubMed] [Google Scholar]
  27. Moritz S.R., Schroder J., Meyer B., Hauschildt M. The more it is needed, the less it is wanted: attitudes toward intervention among depressed patients undergoing online treatment. Depress. Anxiety. 2013;30:157–167. doi: 10.1002/da.21988. [DOI] [PubMed] [Google Scholar]
  28. Motter J.N., Pimontel M.A., Rindskopf D., Devanand D.P., Doraiswamy P.M., Sneed J.R. Computerized cognitive training and functional recovery in major depressive disorder: a meta-analysis. J. Affect. Disord. 2016;189:184–191. doi: 10.1016/j.jad.2015.09.022. [DOI] [PubMed] [Google Scholar]
  29. Naismith S.L., Diamond K., Carter P.E., Norrie L.M., Redoblado-Hodge M.A., Lewis S.J., Hickie I.B. Enhancing memory in late-life depression: the effects of a combined psychoeducation and cognitive training program. Am. J. Geriatr. Psychiatry. 2011;19:240–248. doi: 10.1097/JGP.0b013e3181dba587. [DOI] [PubMed] [Google Scholar]
  30. Nelson H.E., Willison I. NFER Nelson; Windsor: 1991. National Adult Reading Test (NART) [Google Scholar]
  31. Osterrieth P.A. Le test de copie d'une figure complexe. Arch. Psychol. 1944;30:286–356. [Google Scholar]
  32. Owens M., Koster E.H., Derakshan N. Improving attention control in dysphoria through cognitive training: transfer effects on working memory capacity and filtering efficiency and filtering efficiency. Psychophysiology. 2013;50:297–307. doi: 10.1111/psyp.12010. [DOI] [PubMed] [Google Scholar]
  33. Ownby R.L., Crocco E., Acevedo A., John V., Loewenstein D. Depression and risk for Alzheimer disease: systematic review, meta-analysis, and metaregression analysis. Arch. Gen. Psychiatry. 2006;63:530–538. doi: 10.1001/archpsyc.63.5.530. [DOI] [PMC free article] [PubMed] [Google Scholar]
  34. Preiss M., Shatil E., Cermakova R., Cimermanova D., Ram I. Personalized cognitive training in unipolar and bipolar disorder: a study of cognitive functioning. Front. Hum. Neurosci. 2013;7:108. doi: 10.3389/fnhum.2013.00108. [DOI] [PMC free article] [PubMed] [Google Scholar]
  35. Reppermund S., Ising M., Lucae S., Zihl J. Cognitive impairment in unipolar depression is persistent and non-specific: further evidence for the final common pathway disorder hypothesis. Psychol. Med. 2009;39:603–614. doi: 10.1017/S003329170800411X. [DOI] [PubMed] [Google Scholar]
  36. Robertson I.H., Murre J.M. Rehabilitation of brain damage: Brain plasticity and principles of guided recovery. Psychol. Bull. 1999;125:544–575. doi: 10.1037/0033-2909.125.5.544. [DOI] [PubMed] [Google Scholar]
  37. Rock P.L., Roiser J.P., Riedel W.J., Blackwell A.D. Cognitive impairment in depression: a systematic review and meta-analysis. Psychol. Med. 2014;44:2029–2040. doi: 10.1017/S0033291713002535. [DOI] [PubMed] [Google Scholar]
  38. Rutherford B.R., Sneed J.R., Roose S.P. Does differential drop-out explain the influence of study design on antidepressant response? A meta-analysis. J. Affect. Disord. 2012;140:57–65. doi: 10.1016/j.jad.2012.01.031. [DOI] [PMC free article] [PubMed] [Google Scholar]
  39. Schoenberg M.R., Ruwe W.D., Dawson K., McDonald N.B., Houston B., Forducey P.G. Comparison of functional outcomes and treatment cost between a computer-based cognitive rehabilitation teletherapy program and a face-to-face rehabilitation program. Prof. Psychol. Res. Pr. 2008;39:169–175. [Google Scholar]
  40. Segrave R.A., Arnold S., Hoy K., Fitzgerald P.B. Concurrent cognitive control training augments the antidepressant efficacy of tDCS: a pilot study. Brain Stimul. 2014;7:325–331. doi: 10.1016/j.brs.2013.12.008. [DOI] [PubMed] [Google Scholar]
  41. Semkovska M., Lambe S., Lonargáin D.Ó., McLoughlin D.M. Neurocognitive remediation therapy for depression: a feasibility study and randomised controlled pilot protocol testing. J. Nerv. Ment. Dis. 2015;203:609–616. doi: 10.1097/NMD.0000000000000337. [DOI] [PubMed] [Google Scholar]
  42. Siegle G.J., Price R.B., Jones N.P., Ghinassi F., Painter T., Thase M.E. You gotta work at it: pupillary indices of task focus are prognostic for response to a neurocognitive intervention for rumination in depression. Clin. Psychol. Sci. 2014;2:455–471. [Google Scholar]
  43. Sobocki P., Jonsson B., Angst J., Rehnberg C. Cost of depression in Europe. J. Ment. Health Policy Econ. 2006;9(2):87–98. [PubMed] [Google Scholar]
  44. Thabane L., Ma J., Chu R., Cheng J., Ismaila A., Rios L.P., Robson R., Thabane M., Giangregorio L., Goldsmith C.H. A tutorial on pilot studies: the what, why and how. BMC Med. Res. Methodol. 2010;10:1471–2288. doi: 10.1186/1471-2288-10-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
  45. Vanderhasselt M.A., De Raedt R. Impairments in cognitive control persist during remission from depression and are related to the number of past episodes: an event related potentials study. Biol. Psychol. 2009;81:169–176. doi: 10.1016/j.biopsycho.2009.03.009. [DOI] [PubMed] [Google Scholar]
  46. Vos T., Flaxman A.D., Naghavi M., Lozano R., Michaud C., Ezzati M.…Memish Z.A. Years lived with disability (YLDs) for 1160 sequelae of 289 diseases and injuries 1990–2010: a systematic analysis for the Global Burden of Disease Study. Lancet. 2012;380:2163–2196. doi: 10.1016/S0140-6736(12)61729-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
  47. Wechsler D. The Psychological Corporation; San Antonio: 1997. Wechsler Adult Intelligence Scale-Third Edition (WAIS-III) [Google Scholar]
  48. Wechsler D. Third edition manual. The Psychological Corporation; San Antonio: 1997. Wechsler Memory Scale. [Google Scholar]
  49. Word Games 2014. http://www.wordgames.com/find-words.html Find words at.
  50. WU Wien Institute for statistics and mathematics. 2011. http://www.r-project.org/ (Accessed 30 May 2014)
  51. Wykes T., Huddy V., Cellard C., McGurk S.R., Czobor P. A meta-analysis of cognitive remediation for schizophrenia. methodology and effect sizes. Am. J. Psychiatry. 2011;168:472–485. doi: 10.1176/appi.ajp.2010.10060855. [DOI] [PubMed] [Google Scholar]

Articles from Internet Interventions are provided here courtesy of Elsevier

RESOURCES