Abstract
Goal Management Training (GMT) improved self-reported executive functioning in a recent randomized, controlled trial in early intervention for psychosis participants. Little is known about the mechanism for this benefit, so this study investigates objectively measured executive function, the difference between subjective and objective executive function, independent living and employment status as potential moderators of efficacy of GMT.
Baseline scores from 81 participants (GMT n = 39 vs Treatment-as-usual; TAU n = 42) were analyzed in a linear mixed model analysis for repeated measures as predictors of improvement on the self-reported Behavior Rating Inventory of Executive Function–Adult version (BRIEF-A) immediately and 30 weeks after GMT. Potential moderators were scores from objective measures of executive functioning, discrepancy between subjective and objective measures, independent living and employment status. Discrepancy was assessed by comparing four clusters of participants with differing patterns of scores.
The effect of GMT remained significant regardless of initial objective executive functioning at baseline. Those with higher subjective complaints at baseline in two clusters with (i) both objective and subjective executive dysfunction, and (ii) mostly subjective executive dysfunction experienced greater change after treatment. Living arrangements or participation in education or work did not significantly moderate the effects of GMT.
Poor performance on neuropsychological tasks is not an obstacle to making use of GMT, but further knowledge is needed about the benefits of strategy training for individuals with a combination of poor performance with few subjective complaints.
Keywords: Cognitive remediation, Moderators, Executive function, Self-assessment
Abbreviations: GMT, Goal Management Training; TAU, Treatment-as-usual; CW, Color Word Interference Test; TMT, Trail Making Test; CPT, Conners Performance Test; LNS, Letter Number Sequencing Test; DS, Digit Span
1. Introduction
Executive functions are vital to everyday functioning in people with schizophrenia and psychosis risk (Kim et al., 2019; McGurk and Mueser, 2006; Santesteban-Echarri et al., 2017). Executive functions are the top-down controls of attention, emotion and action that ensure goal achievement. They include inhibition, shifting (or mental flexibility), updating of working memory, planning and problem solving (Diamond, 2013; Friedman and Miyake, 2017; Miyake et al., 2000). Executive dysfunction is noticeable prior to the onset of psychotic illness in people who later develop schizophrenia (Catalan et al., 2021). Goal Management Training (GMT; Levine et al., 2000) is a metacognitive strategy training that targets executive functioning through promoting awareness of executive challenges, teaching attention monitoring and control, as well as problem solving strategies (Levine et al., 2011). The ultimate goal of GMT is to improve executive functioning in real-world situations (Levine et al., 2000). Previous GMT studies have shown improved scores on both subjective (self-report) and objective measures (neuropsychological tests) of executive functioning (Stamenova and Levine, 2018). However, GMT is designed to promote transfer to everyday functioning, and executive deficits that create problems in everyday life are often opaque in structured neuropsychological testing (Jurado and Rosselli, 2007; Sbordone, 2014). Thus change after the intervention may be expected to be more easily detected by subjective assessment of executive function.
In a recent randomized, controlled trial GMT was effective in improving self-reported executive functioning in young people with early schizophrenia spectrum disorders or psychosis risk syndromes (Haugen et al., 2022). The improvement was a clinically reliable change for most participants and remained significant when clinical symptoms were controlled for. GMT was not superior to treatment as usual in improving objective executive function, except for attentional control. Nonetheless, improvement of subjective executive function has clinical importance as it is associated with better physical and psychological well-being and greater personal recovery from psychotic illness (Paudel et al., 2020; Van Aken et al., 2022). The present study explores potential predictors of change in subjective executive functioning following GMT.
The efficacy of cognitive remediation in schizophrenia is well documented (Allott et al., 2020b; Lejeune et al., 2021; Vita et al., 2021), but the identification of reliably replicated moderators is challenging due to heterogeneity across studies in sample characteristics and type of intervention (Seccomandi et al., 2020). There are currently few studies on the predictors of treatment outcome in therapies aimed at young people or that specify executive functioning as the key outcome. Since the recent trial was the first study of stand-alone GMT in early schizophrenia, the mechanisms of change are important to identify in order to develop more targeted and personalized rehabilitation (Bowie et al., 2020; Cella et al., 2015; Wykes and Spaulding, 2011).
Although people with more severe impairments seem to gain the most from cognitive remediation in schizophrenia (DeTore et al., 2019; Vita et al., 2021), we chose to investigate baseline objective executive function as severe impairment could potentially prevent participants from learning and using the GMT-strategies (Collins et al., 2014; Emmanouel et al., 2018).
Another potential moderator of treatment effect of GMT is the discrepancy between subjectively and objectively assessed executive function. A substantial portion of people with schizophrenia report fewer subjective cognitive complaints relative to their objectively measured difficulties (Harvey and Pinkham, 2015; Haugen et al., 2021; Potvin et al., 2014). Individuals with psychosis risk syndromes, on the other hand, are more likely to report greater subjective complaints compared to objective test performance (Glenthøj et al., 2020). Discrepancy in either direction could be an obstacle for GMT. Greater subjective complaints could prevent making use of the strategies taught, if the participants do not recognize their potential ability due to negative thought patterns (Allott et al., 2020a; Beck et al., 2018; Cella et al., 2014). Few subjective complaints combined with poorer test results could make it difficult to recognize situations where the GMT-strategies might be effective. This pattern of scores may reflect inaccurate self-assessment or lack of insight into cognitive difficulties (Harvey and Pinkham, 2015; Olsson et al., 2019). Subjective cognition has rarely been investigated as a moderator of cognitive remediation in schizophrenia and results are contradictory (Seccomandi et al., 2020). One study found that more subjective cognitive complaints were associated with larger benefits in objective cognition (Twamley et al., 2011). In two other remediation trials subjective complaints was not a prerequisite for gains in objective cognition, but was associated with better attendance (Burton and Twamley, 2015; Saperstein et al., 2020). A study in psychosis risk found that subjective cognition did not moderate outcome (Glenthøj et al., 2020).
In addition, for GMT to be effective it might require sufficient opportunities for rehearsing the strategies taught in everyday situations such as school, work, or independent living (Bell et al., 2007; Bowie et al., 2020; Holshausen et al., 2014).
In this study we will test: first that poor baseline objective executive function will reduce GMT benefits, and second that a discrepancy between subjective and objective measures of executive function at baseline will reduce benefit from GMT. Dividing participants into subgroups with different patterns of discrepancy, we hypothesized that i) a pattern of poor scores on objective measures coupled with few subjective complaints or ii) a pattern of adequate scores on objective measures coupled with high levels of subjective complaints, would be associated with less improvement after GMT. Furthermore, we hypothesized that living independently and attending school or work would be associated with more improvement following GMT.
2. Methods
2.1. Design and procedure
This is a moderation analysis using data from an RCT with measures collected at 0, 5 weeks (post-intervention) and 30 weeks (follow-up). Participants were independently randomized to GMT (n = 39) or Treatment-As-Usual (TAU; n = 42). See Haugen et al. (2022) for details on assessments and intervention. Assessments were collected by a clinical psychologist or psychiatric nurse under supervision from specialists in neuropsychology and psychiatry. All participants gave written informed consent (Trial registration: clinicaltrials.gov, NCT03048695, Ethical Approval: Regional Committee for Medical and Health Research Ethics, 2015/2118, Norway).
The inclusion criteria were: age 16 to 67 years, symptoms of broad schizophrenia spectrum disorder according to the criteria in the Diagnostic and Statistical Manual of Mental Disorders, DSM-IV-TR (American Psychiatric Association, 2000) or one of three psychosis risk syndromes (attenuated positive symptoms, brief intermittent psychotic symptoms, genetic risk combined with deteriorated functioning; Miller et al., 1999; Yung et al., 1998) and subjective complaints of executive dysfunction expressed during the intake interview or a total score above T55 on the Behavior Rating Inventory of Executive Function–Adult version, BRIEF-A (Roth and Gioia, 2005) – a cut off considered to be clinically relevant in the Norwegian cultural context (Løvstad et al., 2016). Since the BRIEF-A was first administered during baseline assessment seven participants scored below T55, despite having expressed executive difficulties during the intake interview. Participants were excluded if they had comorbid neurological conditions, ongoing substance abuse, intellectual impairment (estimated IQ < 70) or psychosis treatment for longer than five years.
2.2. Participants
Participants (n = 81, approximately 60 % males and 40 % females with a mean age of 25 years), were recruited at Innlandet Hospital in Norway. Table 1 shows sample characteristics. Most participants were recruited through the hospital's specialized early detection and intervention for psychosis clinics, resulting in a young sample between 16 and 44 years of age. The majority of participants, 94 %, were between 16 and 35 years old. Sixteen individuals with a mean age of 23 years, were diagnosed with psychosis risk syndromes. The remainder of the sample was diagnosed with a disorder in the schizophrenia spectrum. The majority of participants, 85 %, had two parents of Norwegian ethnicity and 88 % percent were Caucasian. Approximately half the participants (46.8 %) were not engaged in any form of work or study, and 18.5 % received a disability pension. Sixteen percent of participants lived in supported housing.
Table 1.
Demographical and clinical characteristics (n = 81).
| Sample characteristics | Frequency | Mean | SD | SE |
|---|---|---|---|---|
| Age | 24.90 | 6.35 | 0.71 | |
| Gender | ||||
| Female | 32 (39.50 %) | |||
| Male | 49 (60.50 %) | |||
| Education in years | 12.90 | 1.83 | 0.20 | |
| Estimated IQ | 98.82 | 14.05 | 1.61 | |
| Employment | ||||
| Full-time work/study | 16 (19.80 %) | |||
| Part-time work/study | 11 (13.60 %) | |||
| Supported employment | 16 (19.80 %) | |||
| Not working/studying | 38 (46.80 %) | |||
| Living arrangements | ||||
| Alone | 25 (30.90 %) | |||
| With partner and/or children | 16 (19.80 %) | |||
| With parent | 24 (29.60 %) | |||
| With friends/in shared house | 3 (3.70 %) | |||
| In supported housing | 13 (16 %) | |||
| In a relationship | 18 (22.20 %) | |||
| Diagnosis (DSM-IV): | ||||
| Schizophrenia spectrum disorder | 65 (80.20 %) | |||
| Schizophrenia | 29 (35.80 %) | |||
| Schizoaffective disorder | 14 (17.30 %) | |||
| Schizophreniform disorder | 6 (7.40 %) | |||
| Psychotic disorder not otherwise specified | 15 (18.50 %) | |||
| Delusional disorder | 1 (1.20 %) | |||
| Psychosis risk syndrome | 16 (19.80 %) | |||
| Positive attenuated symptoms | 9 (11.10 %) | |||
| Brief limited intermittent symptoms | 5 (6.20 %) | |||
| Genetic risk with deteriorated function | 2 (2.50 %) | |||
| Duration of untreated psychosis (weeks) | 195.32 | 237.75 | 26.42 | |
| Hospitalizations | 2.75 | 4.68 | 0.52 | |
| Months in hospital | 4.86 | 7.61 | 0.85 | |
| Drug therapy | 60 (74.10 %) | |||
| Antipsychotics | 50 (61.70 %) | |||
Note: IQ was estimated at baseline with Vocabulary and Matrix Reasoning subtests from the Wechsler Abbreviated Scale of Intelligence (WASI) (Wechsler, 1999). A few participants had GAI (General Ability Index) scores from Wechsler's Adult Intelligence Scale, 4th edition (WAIS-IV) in place of WASI scores (Wechsler, 2008).
2.3. Interventions
Goal Management Training (GMT) (Levine et al., 2000; Robertson, 1996) is a manualized meta-cognitive strategy training aimed at improving executive functioning. Eighteen hours (nine modules of approximately two hours duration) of GMT was administered by a clinical psychologist to groups of three to eight participants twice a week. The main ingredient of training is verbalization techniques (STOP – FOCUS on the present – STATE your goal – SPLIT goal into subtasks – CHECK progress) for goal achievement in everyday situations. Sessions included group discussions and practical exercises in present mindedness. Home assignments included rehearsing strategies for half an hour each day between sessions. GMT was provided in addition to usual care. Treatment-as-usual (TAU) group continued in their usual care according to national Norwegian guidelines, which often consist of a combination of psychotherapy, medication and family interventions (Norwegian Health Authority, 2013).
2.4. Measures
The outcome variable in this moderation analysis is subjective executive functioning - reported as total raw score on Behavior Rating Inventory of Executive Function–Adult version, BRIEF-A (Roth and Gioia, 2005). This is a 75-item questionnaire with nine subscales covering inhibition, shifting, emotional control, self-monitoring, initiating, working memory, planning/organizing, task-monitoring and organization of materials. It is reliable and valid in healthy and clinical populations, including people with schizophrenia (Bulzacka et al., 2013; Roth and Gioia, 2005; Van Aken et al., 2022).
Three normed T-scores for the Inhibit, Shift and Working Memory subscales were chosen as measures of subjective executive function that theoretically overlap with the objective measures (Friedman and Miyake, 2017; Roth and Gioia, 2005) and used in the calculation of the discrepancy score (see below).
2.5. Objective executive functioning
The first potential moderator is the mean z-score for baseline objective executive functioning, created from normed scores on several neuropsychological tests of inhibition, shifting and working memory according to contemporary theories of executive function (Friedman and Miyake, 2017). A central criticism of previous studies has been the use of a single test to draw conclusions about global executive functioning. The mean score across tests is considered more robust against the influence of measurement error (Hwang et al., 2019).
The tests used were
1. Color word interference test
The age normed scaled score for time spent on Color-Word Interference Test condition three (CW3) from DelisKaplan Executive Function System (D-KEFS) (Delis et al., 2001), was used as a measure of inhibition and condition four (CW4) was used as a measure of shifting.
2. Trail making test
The aged normed scaled score for time spent on condition four, Letter Number Switching, from the Trail Making Test (TMT4) in D-KEFS (Delis et al., 2001) was used as a measure of shifting.
3. Conners continuous performance test
The age normed T-score for commission errors from Conners Continuous Performance Test 3rd edition (CPT3) (Conners, 2014) was used as a measure of inhibition.
4. Working memory: Digit span and letter-number sequencing
Working memory was assessed by averaging the age normed scaled scores from the Digit Span (DS) and Letter-Number Sequencing (LNS) subtests in the Wechsler Adult Intelligence Scale–4th edition, WAIS-IV (Wechsler, 2008).
All measures have shown adequate test-retest reliability in normative samples (Conners, 2014; Delis et al., 2001; Wechsler, 2008).
2.6. Discrepancy between subjective and objective executive function
The discrepancy variable was created by dividing participants into clusters based on the normed scores from three subscales of BRIEF, the Inhibit, Shift and Working Memory subscales, and normed scores from six neuropsychological tasks: CW3, CW4, TMT4, CPT3 commission errors, DS and LNS. A two-step cluster analysis specified four clusters based on Schwartz's Bayesian criterion. The cluster solution was judged to be fair with adequate cohesion and separation. Eight participants were not assigned clusters due to missing scores.
The participants in Cluster A had poor scores on objective executive function but an average level of subjective complaints. The cluster was labelled Mostly objective executive dysfunction. Cluster B had high levels of subjective complaints, but average performance on objective tests. It was labelled Mostly subjective executive dysfunction. Cluster C had high levels of subjective complaints and poor performance on objective measures. It was labelled Both objective and subjective executive dysfunction. The participants in Cluster D had average performance on objective measures combined with relatively lower levels of cognitive complaints. This cluster was labelled Neither objective, nor subjective executive dysfunction. Table 2 shows the scores of the four clusters on subjective and objective measures.
Table 2.
Scores on subjective and objective executive function in the four clusters.
| Clusters | Inhibition Mean (SD) |
Shifting Mean (SD) |
Working memory Mean (SD) |
||||||
|---|---|---|---|---|---|---|---|---|---|
| BRIEF inhibit T | CW3 SS | CPT-3 T | BRIEF shift T | CW4 SS | TMT4 SS | BRIEF WM T | LNS SS | DS SS | |
| Cluster A (n16): Mostly objective executive dysfunction | 48.56 (9.04) | 6.25 (1.77) | 55.25 (10.36) | 53.81 (7.71) | 6.06 (3.36) | 5.06 (3.23) | 54.50 (7.40) | 8.88 (0.89) | 8.80 (2.27) |
| Cluster B (n23): Mostly subjective executive dysfunction | 62.13 (10.88) | 9.36 (2.40) | 57.30 (10.81) | 72.22 (10.72) | 9.95 (2.82) | 8.30 (2.86) | 77.35 (4.43) | 8.64 (2.63) | 9.86 (2.82) |
| Cluster C (n13): Both objective and subjective dysfunction | 67.00 (11.39) | 4.15 (3.93) | 62.77 (9.88) | 64.54 (7.85) | 2.69 (2.39) | 1.75 (1.54) | 72.62 (5.58) | 8.55 (1.69) | 8.85 (2.73) |
| Cluster D (n21): Neither objective, nor subjective dysfunction | 56.52 (11.80) | 8.20 (3.54) | 50.38 (8.73) | 58.48 (8.33) | 10.00 (2.32) | 9.50 (2.40) | 64.00 (5.82) | 11.91 (3.45) | 11.24 (2.36) |
| Average across clusters (n73) | 58.41 (11.80) | 8.21 (3.54) | 55.84 (10.66) | 62.86 (11.37) | 7.79 (3.94) | 6.74 (3.85) | 67.66 (10.43) | 9.66 (2.89) | 9.86 (2.70) |
Note: The normative mean of BRIEF-A is T50 (SD 10). The normative mean for the task measures is scaled score (SS) 10 (SD 3).
2.7. Opportunities for practice
Two binary variables were created. Living situation: Independent living (Living alone, with a partner and/or children or with friends) or not (supported housing or with parents). Occupation: Participation (Full time, part time and supported employment or education) or not.
2.8. Data analysis
Four potential moderators of change in total subjective executive function after GMT were tested: Baseline objective executive functioning score, Work/School, Independent living and Discrepancy between baseline subjective and objective executive functioning. Linear mixed effect models for repeated measures were fitted, with subjective cognitive complaints as the outcome (total BRIEF-A score). The time variable was coded 0 for baseline, 1 for post-intervention measurement at 5 weeks, and 2 for follow-up measurements at 30 weeks, as a linear effect of the intervention was expected. Random intercept and first-order autoregressive covariance matrix was used. Moderator variables were entered separately to test their influence independently, resulting in four different models described below. Maximum likelihood estimation was used since the goal was to compare nested models. Better model fit was defined as a significant reduction in minus twice the log likelihood (−2LL) for the nested model, exceeding the critical values in a chi-square distribution, p < 0.05.
Model 0: The null model without moderators included fixed effects of time and the treatment x time interaction as predictors. The main effect of treatment group was removed from the model to adjust for potential baseline differences (Twisk et al., 2018).
Model 1a: Included objective executive functioning as a main effect and an interaction effect with treatment and time (treatment × time × objective executive function).
Model 1b: The discrepancy between subjective and objective measures of executive functioning was added to Model 0 as a main effect and an interaction effect with treatment and time (treatment × time × discrepancy). Discrepancy clusters were added as categorical variables. C: Both objective and subjective executive dysfunction was the reference category.
Model 1c: The categorical variable occupation (yes/no) was added to Model 0 as a main effect and an interaction effect with treatment and time (treatment × time × occupation).
Model 1d: Living situation was added as a categorical variable (yes/no) to Model 0 as a main effect and an interaction effect with treatment and time (treatment × time × independent living).
All tests of p-values were two-sided and a 5 % significance level was used. Statistical analyses were conducted using IBM SPSS Statistics version 26 (IBM Corp., Armonk, NY, USA).
2.9. Statistical power
According to the principle of intention to treat, all partial data from the 81 participants who entered into the study were analyzed (Gupta, 2011). There was some missing questionnaire data for the outcome variable with completed questionnaires from 74 participants at baseline, 49 post-intervention and 43 at follow-up. All participants responded at least once. All 81 participants had scores for the predictor variable objective executive functioning at baseline and 73 had discrepancy cluster membership. To evaluate the representativeness of the available data, we compared baseline characteristics in those who completed one, two or three questionnaires. We found no statistically significant differences in treatment condition, cluster membership, demography (gender, age and education), clinical characteristics (diagnosis, symptoms, global function and treatment) or cognitive characteristics (estimated IQ, subjective and objective executive function). Missing data was assumed to be missing at random. Thus, imputation of missing outcome values was not performed as the linear mixed effect models provides unbiased estimates under the assumption of missing at random (Krueger and Tian, 2004; Muth et al., 2016). Degrees of freedom are listed for each effect in Table 3. Sample size was calculated for the primary endpoint of the RCT, which was to estimate the efficacy of GMT compared to treatment as usual (Haugen et al., 2022). Because of the exploratory nature of this study, no power calculations were performed for the measures in this study.
Table 3.
Linear mixed model analysis of repeated measures (ITT n = 81): Predictors of improved executive functioning following GMT.
| Estimated coefficients |
Df | Sig. | Information criteria |
||||||
|---|---|---|---|---|---|---|---|---|---|
| b (est) | b SE | 95 % CI | -2LL | Sig. of -2LL | AIC | BIC | |||
| Model 0: | 1441.24 | 1453.24 | 1471.91 | ||||||
| 6 parameters | |||||||||
| Intercept | 142.63 | 2.59 | 137.48, 147.77 | 100.54 | <0.001 | ||||
| Time | −2.67 | 1.78 | −6.23, 0.88 | 63.05 | 0.137 | ||||
| GMT × time interaction | −5.14 | 2.43 | −9.99, −0.28 | 67.42 | 0.038 | ||||
| Model 1a: Objective executive function | 1439.95 | 0.616 | 1457.95 | 1485.96 | |||||
| 8 parameters | |||||||||
| Intercept | 142.66 | 2.59 | 137.52, 147.79 | 100.38 | <0.001 | ||||
| Time | −7.86 | 1.80 | −11.46, −4.26 | 53.41 | <0.001 | ||||
| Objective executive function | −0.59 | 2.59 | −5.73, 4.55 | 99.50 | 0.820 | ||||
| GMT × time interaction | −5.24 | 2.44 | −10.11, −0.37 | 66.71 | 0.035 | ||||
| GMT × time × objective executive function | 1.64 | 1.72 | −1.80, 5.08 | 55.71 | 0.343 | ||||
| Model 1b: Discrepancy | 1285.93 | <0.001 | 1315.93 | 1361.68 | |||||
| 15 parameters | |||||||||
| Intercept | 151.60 | 4.46 | 142.74, 160.45 | 103.68 | <0.001 | ||||
| Time | −13.18 | 3.44 | −20.06, −6.31 | 65.61 | <0.001 | ||||
| Discrepancy | |||||||||
| A: Mostly objective dysfunction (in GMT) | −33.69 | 5.57 | −45.62, −21.77 | 103.97 | 0.115 | ||||
| B: Mostly subjective dysfunction (in GMT) | 8.86 | 6.01 | −2.19, 19.90 | 101.68 | <0.001 | ||||
| GMT × time interaction | −19.13 | 7.62 | −34.32, 3.95 | 70.72 | 0.014 | ||||
| GMT × time × discrepancy | |||||||||
| A: More objective dysfunction (in GMT) | 13.13 | 4.30 | −6.11, 11.07 | 62.17 | 0.036 | ||||
| B: More subjective dysfunction (in GMT) | 2.48 | 6.11 | 0.92, 25.33 | 64.97 | 0.566 | ||||
| Model 1c: Work/school | 1436.98 | 0.118 | 1452.97 | 1477.87 | |||||
| 8 parameters | |||||||||
| Intercept | 139.70 | 3.44 | 132.87, 146.53 | 93.68 | <0.001 | ||||
| Time | −2.74 | 1.75 | −6.23, 0.76 | 62.74 | 0.122 | ||||
| Work/school | −6.34 | 4.92 | −3.43, 16.12 | 87.09 | 0.201 | ||||
| GMT × time interaction | −6.94 | 2.85 | −12.64, −1.24 | 64.29 | 0.018 | ||||
| GMT × time × work/school | −4.48 | 3.53 | −3.43, 16.12 | 54.23 | 0.210 | ||||
| Model 1d: Independent living | 1438.35 | 0.236 | 1454.35 | 1479.24 | |||||
| 8 parameters | |||||||||
| Intercept | 141.32 | 3.36 | 134.64, 147.99 | 92.32 | <0.001 | ||||
| Time | −2.79 | 1.80 | −6.39, 0.81 | 63.27 | 0.126 | ||||
| Independent living | −2.55 | 4.93 | −7.25, 12.35 | 87.94 | 0.606 | ||||
| GMT × time interaction | −6.24 | 2.68 | −11.60, −0.88 | 61.59 | 0.023 | ||||
| GMT × time × independent living | −6.35 | 4.30 | −2.23, 14.94 | 65.17 | 0.144 | ||||
Note: The outcome variable was total raw score from the Behavior Rating Inventory of Executive Function – Adult version (BRIEF-A) (Roth and Gioia, 2005). Higher scores indicate greater executive difficulties in real-world situations. Coefficients for discrepancy clusters are comparisons with scores for the cluster with Both objective and subjective executive dysfunction in the GMT-group. Bold values indicate statistically significant fixed effects at alpha level p < 0.05.
3. Results
Fig. 1 is an illustration of the mean values of BRIEF-A total raw score for each cluster in the GMT-group. In the GMT condition, participants in cluster C experienced the greatest reduction in executive complaints after intervention, followed by participants in cluster B. Participants in cluster D experienced a reduction in subjective complaints immediately after GMT that had increased somewhat again at follow-up. Participants in cluster A experienced little to no change in self-reported executive function in real-world situations.
Fig. 1.
Subjective executive function after intervention in discrepancy clusters.
Mean values of BRIEF-A raw score over time in four clusters based on discrepancy between subjective and objective measures at baseline in the treatment group.
Table 3 displays the results of the linear mixed effect models analyses.
Model 1a shows that objective impairment did not moderate the effect of GMT since the interaction effect between treatment x time x objective executive functioning was not significant, F 0.65, p 0.526, and the treatment effect of GMT (group x time) remained of similar size as in the previous model, F 4.62, p 0.035. Model 1a did not show a significant improvement in statistical fit compared to Model 0.
Model 1b shows that the treatment effect of GMT remained significant when taking discrepancy cluster membership into account, F 6.41, p 0.031. There was a significant main effect of discrepancy, F 25.64, p < .001, but the interaction effect between discrepancy and treatment effect was not significant, F 1.37, p 0.241. Model 1b was superior in describing the data with significantly increased model fit compared to Model 0. The cluster with Mostly Objective Executive Dysfunction improved significantly less than the cluster with Both Objective and Subjective Executive Dysfunction, p 0.036.
Model 1c shows that participation in work or school did not moderate the effect of GMT since the interaction effect between treatment x time x objective executive functioning did not reach statistical significance, F 1.61, p 0.210. Model 1c did not show a significant improvement in statistical fit compared to Model 0.
Model 1d shows that living independently did not moderate the effect of GMT since the interaction effect between treatment × time × independent living did not reach statistical significance, F 2.18, p 0.144. Model 1d did not show a significant improvement in statistical fit compared to Model 0.
4. Discussion
We did not find support for our hypothesis that pronounced executive dysfunction measured with objective tasks could interfere with learning in the GMT-groups and prevent participants from making use of the strategies taught (Cicerone et al., 2019; Emmanouel et al., 2018). GMT was equally effective irrespective of performance on tasks of executive function. These results mirror a recent meta-analysis of moderators of cognitive training in schizophrenia that found that even people with severe cognitive dysfunction benefit from cognitive remediation (Vita et al., 2021).
We found partial support for our hypothesis that discrepancy between subjective and objective executive function at baseline would be an obstacle to successful strategy training with GMT (Allott et al., 2020a; Beck et al., 2018; Harvey and Pinkham, 2015). Treatment effect of GMT remained significant when considering discrepancy, but participants with mostly objective executive dysfunction unaccompanied by pronounced subjective complaints experienced less benefit from GMT compared to participants with both subjective and objective dysfunction. This finding is in contrast to studies where objective cognition was the outcome (Burton and Twamley, 2015; Saperstein et al., 2020). Note, however, that in our study participants in the Mostly objective executive dysfunction cluster reported few subjective complaints to start with, so the lack of change could be due to a floor effect. Having mostly subjective complaints without accompanying poor task performance did not serve as an obstacle to benefiting from strategy training with GMT.
We did not find support for our hypothesis that opportunities for practicing GMT strategies in sufficiently demanding everyday situations represented by living independently or participating in work or school would improve the effect of GMT. One reason for this may be that the focus in GMT is on individual goals which may include other situations than housework, work, or school (Haugen et al., 2022; Krasny-Pacini et al., 2014). Some participants raised concerns about not having opportunities to practice skills, so GMT therapists could advise on alternative situations. However, this was a limited sample size that may not have been sufficient to detect significant moderating effects.
4.1. Clinical implications
Clinicians may recommend GMT to people with schizophrenia spectrum disorders or psychosis risk syndromes with self-reported executive dysfunction. Poor performance on neuropsychological tests is not an obstacle to benefiting from GMT. However, individuals with more severe objective dysfunction unaccompanied by subjective complaints might not benefit. Measuring both subjective and objective cognition in clinical assessment is important because discrepancy issues may be discovered and discussed with patients. Assisting patients in recognizing relevant everyday examples of cognitive difficulties should be carried out while also supporting self-efficacy and self-esteem (Cella et al., 2014; Haugen et al., 2021; Saperstein et al., 2020). Developing individual goals using a structured approach such as Goal Attainment Scaling may help to identify motivating opportunities for practicing cognitive remediation strategies for patients regardless of living arrangements and employment status (Ashford and Turner-Stokes, 2006; Krasny-Pacini et al., 2014).
4.2. Implications for future research
Future research on cognitive remediation in schizophrenia may benefit from exploring the significance of discrepancy between subjective and objective cognition. So far the evidence that unawareness of cognitive difficulties is an obstacle for successful remediation is sparse and contradictory (Seccomandi et al., 2020).
4.3. Strengths and limitations
A strength of the study is the extensive assessment battery tapping different aspects of executive function. A limitation of self-report measures of cognition is that they may be influenced by emotional states and challenges with self-awareness in serious mental illness (Harvey, 2012; Shwartz et al., 2020; Toplak et al., 2013). Nonetheless, self-assessment may complement neuropsychological tests that are often too structured to capture the complexity of interacting components of executive function required in everyday life (Friedman and Banich, 2019; Isquith et al., 2013; Sbordone, 2014). In addition, questionnaire data was missing at follow-up and may raise a concern about bias, except that we were unable to detect significant differences between the participants who completed all questionnaires and those who did not. This reduces the likelihood of bias. We also used mixed model analysis because it can accommodate missing data points and provide unbiased estimates under the assumption of missing at random (Krueger and Tian, 2004; Muth et al., 2016; Schielzeth et al., 2020).
5. Conclusions
Goal Management Training does not seem to be affected by levels of objective executive functioning, or concomitant independent living or occupational status. The relevance of differences between subjective and objective assessments as a barrier to successful intervention should be further explored.
CRediT authorship contribution statement
Ingvild Haugen: Investigation, Formal analysis, Writing – original draft. Torill Ueland: Conceptualization, Writing – review & editing. Jan Stubberud: Conceptualization, Supervision, Writing – review & editing. Cathrine Brunborg: Supervision, Writing – review & editing. Til Wykes: Writing – review & editing. Merete Glenne Øie: Conceptualization, Funding acquisition, Project administration, Supervision, Writing – review & editing. Elisabeth Haug: Conceptualization, Supervision, Writing – review & editing.
Declaration of competing interest
The authors have no conflicting interests to declare.
Acknowledgements
The study was funded by the South-Eastern Norway Health Authority (grant number 2017012), Innlandet Hospital Trust (grant number 150602) and University of Oslo (grant number 353139). Funders had no involvement in the execution of the study.
References
- Allott K., Steele P., Boyer F., et al. Cognitive strengths-based assessment and intervention in first-episode psychosis: a complementary approach to addressing functional recovery? Clin. Psychol. Rev. 2020;79 doi: 10.1016/j.cpr.2020.101871. [DOI] [PubMed] [Google Scholar]
- Allott K., Bryce S., van-der-El K. Compensatory interventions for cognitive impairments in psychosis: a systematic review and meta-analysis. Schizophr. Bull. 2020;46(4):869–883. doi: 10.1093/schbul/sbz134. [DOI] [PMC free article] [PubMed] [Google Scholar]
- American Psychiatric Association . 4th ed. American Psychiatric Association; Washington, DC: 2000. Diagnostic and Statistical Manual of Mental Disorders: DSM-IV-TR. text revision. edn. [Google Scholar]
- Ashford S., Turner-Stokes L. Goal attainment for spasticity management using botulinum toxin. Physiother. Res. Int. 2006;11(1):24–34. doi: 10.1002/pri.36. [DOI] [PubMed] [Google Scholar]
- Beck A.T., Himelstein R., Bredemeier K., Silverstein S.M., Grant P. What accounts for poor functioning in people with schizophrenia: a re-evaluation of the contributions of neurocognitive v. attitudinal and motivational factors. Psychol. Med. 2018;48(16):2776–2785. doi: 10.1017/S0033291718000442. [DOI] [PubMed] [Google Scholar]
- Bell M., Fiszdon J., Greig T., Wexler B., Bryson G. Neurocognitive enhancement therapy with work therapy in schizophrenia: 6-month follow-up of neuropsychological performance. J. Rehabil. Res. Dev. 2007;44(5):761–770. doi: 10.1682/jrrd.2007.02.0032. [DOI] [PubMed] [Google Scholar]
- Bowie C.R., Bell M.D., Fiszdon J.M., et al. Cognitive remediation for schizophrenia: an expert working group white paper on core techniques. Schizophr. Res. 2020;215:49–53. doi: 10.1016/j.schres.2019.10.047. [DOI] [PubMed] [Google Scholar]
- Bulzacka E., Vilain J., Schurhoff F., Meary A., Leboyer M., Szoke A. A self administered executive functions ecological questionnaire (the behavior rating inventory of executive function - adult version) shows impaired scores in a sample of patients with schizophrenia. Ment. Illn. 2013;5(1) doi: 10.4081/mi.2013.e4. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Burton C.Z., Twamley E.W. Neurocognitive insight, treatment utilization, and cognitive training outcomes in schizophrenia. Schizophr. Res. 2015;161(2–3):399–402. doi: 10.1016/j.schres.2014.12.002. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Catalan A., Salazar de Pablo G., Aymerich C., et al. Neurocognitive functioning in individuals at clinical high risk for psychosis: a systematic review and meta-analysis. JAMA Psychiatry. 2021 doi: 10.1001/jamapsychiatry.2021.1290. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Cella M., Swan S., Medin E., Reeder C., Wykes T. Metacognitive awareness of cognitive problems in schizophrenia: exploring the role of symptoms and self-esteem. Psychol. Med. 2014;44(3):469–476. doi: 10.1017/S0033291713001189. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Cella M., Reeder C., Wykes T. Cognitive remediation in schizophrenia—now it is really getting personal. Curr. Opin. Behav. Sci. 2015;4:147–151. doi: 10.1016/j.cobeha.2015.05.005. [DOI] [Google Scholar]
- Cicerone K.D., Goldin Y., Ganci K., et al. Evidence-based cognitive rehabilitation: systematic review of the literature from 2009 through 2014. Arch. Phys. Med. Rehabil. 2019;100(8):1515–1533. doi: 10.1016/j.apmr.2019.02.011. [DOI] [PubMed] [Google Scholar]
- Collins A.G., Brown J.K., Gold J.M., Waltz J.A., Frank M.J. Working memory contributions to reinforcement learning impairments in schizophrenia. J. Neurosci. 2014;34(41):13747–13756. doi: 10.1523/JNEUROSCI.0989-14.2014. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Conners K.C. 3rd edn. Multi-Health Systems Inc; Toronto: 2014. Conners Continuous Performance Test. [Google Scholar]
- Delis D.C., Kaplan E., Kramer J.H. NCS Pearson, Inc; Bloomington: 2001. Delis–Kaplan Executive Function System. [Google Scholar]
- DeTore N.R., Mueser K.T., Byrd J.A., McGurk S.R. Cognitive functioning as a predictor of response to comprehensive cognitive remediation. J. Psychiatr. Res. 2019;113:117–124. doi: 10.1016/j.jpsychires.2019.03.012. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Diamond A. Executive functions. Annu. Rev. Psychol. 2013;64(1):135–168. doi: 10.1146/annurev-psych-113011-143750. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Emmanouel A., Kontrafouri E., Nikolaos P., Kessels R.P.C., Fasotti L. Incorporation of a working memory strategy in GMT to facilitate serial-order behaviour in brain-injured patients. Neuropsychol. Rehabil. 2018:1–27. doi: 10.1080/09602011.2018.1517369. [DOI] [PubMed] [Google Scholar]
- Friedman N.P., Banich M.T. Questionnaires and task-based measures assess different aspects of self-regulation: both are needed. Proc. Natl. Acad. Sci. U. S. A. 2019;116(49):24396–24397. doi: 10.1073/pnas.1915315116. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Friedman N.P., Miyake A. Unity and diversity of executive functions: individual differences as a window on cognitive structure. Cortex. 2017;86:186–204. doi: 10.1016/j.cortex.2016.04.023. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Glenthøj L.B., Mariegaard L., Kristensen T.D., Wenneberg C., Medalia A., Nordentoft M. Self-perceived cognitive impairments in psychosis ultra-high risk individuals: associations with objective cognitive deficits and functioning. NPJ Schizophr. 2020;6(1):31. doi: 10.1038/s41537-020-00124-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Gupta S.K. Intention-to-treat concept: a review. Perspect. Clin. Res. 2011;2(3):109–112. doi: 10.4103/2229-3485.83221. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Harvey P.D. Clinical applications of neuropsychological assessment. Dialogues Clin. Neurosci. 2012;14(1):91–99. doi: 10.31887/DCNS.2012.14.1/pharvey. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3341654/pdf/DialoguesClinNeurosci-14-91.pdf Available at: [DOI] [PMC free article] [PubMed] [Google Scholar]
- Harvey P.D., Pinkham A. Impaired self-assessment in schizophrenia: why patients misjudge their cognition and functioning. Curr. Psychiatr. 2015;14(4):53–59. https://www.mdedge.com/psychiatry/article/98117/schizophrenia-other-psychotic-disorders/impaired-self-assessment?sso=true Available at: [Google Scholar]
- Haugen I., Stubberud J., Ueland T., Haug E., Øie M.G. Executive dysfunction in schizophrenia: predictors of the discrepancy between subjective and objective measures. Schizophr. Res. Cogn. 2021;26 doi: 10.1016/j.scog.2021.100201. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Haugen I., Stubberud J., Haug E., et al. A randomized controlled trial of goal management training for executive functioning in schizophrenia spectrum disorders or psychosis risk syndromes. BMC Psychiatry. 2022;22(1):575. doi: 10.1186/s12888-022-04197-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Holshausen K., Bowie C.R., Mausbach B.T., Patterson T.L., Harvey P.D. Neurocognition, functional capacity, and functional outcomes: the cost of inexperience. Schizophr. Res. 2014;152(2):430–434. doi: 10.1016/j.schres.2013.08.004. [DOI] [PubMed] [Google Scholar]
- Hwang W.J., Lee T.Y., Shin W.G., et al. Global and specific profiles of executive functioning in prodromal and early psychosis. Front Psychiatry. 2019;10:356. doi: 10.3389/fpsyt.2019.00356. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Isquith P.K., Roth R.M., Gioia G. Contribution of rating scales to the assessment of executive functions. Appl. Neuropsychol. Child. 2013;2(2):125–132. doi: 10.1080/21622965.2013.748389. [DOI] [PubMed] [Google Scholar]
- Jurado M.B., Rosselli M. The elusive nature of executive functions: a review of our current understanding. Neuropsychol. Rev. 2007;17(3):213–233. doi: 10.1007/s11065-007-9040-z. [DOI] [PubMed] [Google Scholar]
- Kim H.K., Park H.Y., Seo E., et al. Factors associated with psychosocial functioning and outcome of individuals with recent-onset schizophrenia and at ultra-high risk for psychosis. Front Psychiatry. 2019;10:459. doi: 10.3389/fpsyt.2019.00459. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Krasny-Pacini A., Chevignard M., Evans J. Goal management training for rehabilitation of executive functions: a systematic review of effectivness in patients with acquired brain injury. Disabil. Rehabil. 2014;36(2):105–116. doi: 10.3109/09638288.2013.777807. [DOI] [PubMed] [Google Scholar]
- Krueger C., Tian L. A comparison of the general linear mixed model and repeated measures ANOVA using a dataset with multiple missing data points. Biol. Res. Nurs. 2004;6(2):151–157. doi: 10.1177/1099800404267682. [DOI] [PubMed] [Google Scholar]
- Lejeune J.A., Northrop A., Kurtz M.M. A meta-analysis of cognitive remediation for schizophrenia: efficacy and the role of participant and treatment factors. Schizophr. Bull. 2021;47(4):997–1006. doi: 10.1093/schbul/sbab022. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Levine B., Robertson I.H., Clare L., et al. Rehabilitation of executive functioning: an experimental-clinical validation of goal management training. J. Int. Neuropsychol. Soc. 2000;6(3):299. doi: 10.1017/S1355617700633052. [DOI] [PubMed] [Google Scholar]
- Levine B., Schweizer T.A., O'Connor C., et al. Rehabilitation of executive functioning in patients with frontal lobe brain damage with goal management training. Front. Hum. Neurosci. 2011;5:9. doi: 10.3389/fnhum.2011.00009. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Løvstad M., Sigurdardottir S., Andersson S., et al. Behavior rating inventory of executive function adult version in patients with neurological and neuropsychiatric conditions: symptom levels and relationship to emotional distress. J. Int. Neuropsychol. Soc. 2016;22(6):682–694. doi: 10.1017/S135561771600031X. [DOI] [PubMed] [Google Scholar]
- McGurk S.R., Mueser K.T. Cognitive and clinical predictors of work outcomes in clients with schizophrenia receiving supported employment services: 4-year follow-up. Adm. Policy Ment. Health Ment. Health Serv. Res. 2006;33(5):598–606. doi: 10.1007/s10488-006-0070-2. [DOI] [PubMed] [Google Scholar]
- Miller T., McGlashan T., Woods S., et al. Symptom assessment in schizophrenic prodromal states. Psychiatry Q. 1999;70(4):273–287. doi: 10.1023/A:1022034115078. [DOI] [PubMed] [Google Scholar]
- Miyake A., Friedman N.P., Emerson M.J., Witzki A.H., Howerter A., Wager T.D. The unity and diversity of executive functions and their contributions to complex "Frontal Lobe" tasks: a latent variable analysis. Cogn. Psychol. 2000;41(1):49–100. doi: 10.1006/cogp.1999.0734. [DOI] [PubMed] [Google Scholar]
- Muth C., Bales K.L., Hinde K., Maninger N., Mendoza S.P., Ferrer E. Alternative models for small samples in psychological research: applying linear mixed effects models and generalized estimating equations to repeated measures data. Educ. Psychol. Meas. 2016;76(1):64–87. doi: 10.1177/0013164415580432. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Norwegian Health Authority . Norwegian Health Authority [Helsedirektoratet]; Oslo: 2013. National guideline for assessment, treatment and follow-up care of persons with psychotic illness [Nasjonal faglig retningslinje for utredning, behandling og oppfølging av personer med psykoselidelser]https://www.helsedirektoratet.no Available at: [Google Scholar]
- Olsson A.-K., Hjärthag F., Helldin L. Overestimated function in patients with schizophrenia: a possible risk factor for inadequate support? Schizophr. Res. 2019;206:194–199. doi: 10.1016/j.schres.2018.11.027. [DOI] [PubMed] [Google Scholar]
- Paudel S., Coman D., Freudenreich O. Subjective experience of cognitive difficulties as an important attribute of quality of life among individuals with schizophrenia spectrum disorders. Schizophr. Res. 2020;215:476–478. doi: 10.1016/j.schres.2019.09.008. [DOI] [PubMed] [Google Scholar]
- Potvin S., Pelletier J., Stip E. La conscience des déficits neurocognitifs dans la schizophrénie : une méta-analyse [Neurocognitive insight in schizophrenia: a meta-analysis] Sante Ment. Que. 2014;39(2):183. doi: 10.7202/1027839ar. [DOI] [PubMed] [Google Scholar]
- Robertson I.H. PsyConsult; Cambridge: 1996. Goal Management Training: A Clinical Manual. [Google Scholar]
- Roth R.M., Gioia G.A. Psychological Assessment Resources; Lutz, Florida: 2005. Behavior Rating Inventory of Executive Function- Adult Version. [Google Scholar]
- Santesteban-Echarri O., Paino M., Rice S., et al. Predictors of functional recovery in first-episode psychosis: a systematic review and meta-analysis of longitudinal studies. Clin. Psychol. Rev. 2017;58:59–75. doi: 10.1016/j.cpr.2017.09.007. [DOI] [PubMed] [Google Scholar]
- Saperstein A.M., Lynch D.A., Qian M., Medalia A. How does awareness of cognitive impairment impact motivation and treatment outcomes during cognitive remediation for schizophrenia? Schizophr. Res. 2020;218:70–75. doi: 10.1016/j.schres.2020.02.014. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Sbordone R.J. The hazards of strict reliance on neuropsychological tests. Appl. Neuropsychol. Adult. 2014;21(2):98–107. doi: 10.1080/09084282.2012.762630. [DOI] [PubMed] [Google Scholar]
- Schielzeth H., Dingemanse N.J., Nakagawa S., et al. Robustness of linear mixed-effects models to violations of distributional assumptions. Methods Ecol. Evol. 2020;11(9):1141–1152. doi: 10.1111/2041-210x.13434. [DOI] [Google Scholar]
- Seccomandi B., Tsapekos D., Newbery K., Wykes T., Cella M. A systematic review of moderators of cognitive remediation response for people with schizophrenia. Schizophr. Res. Cogn. 2020;19 doi: 10.1016/j.scog.2019.100160. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Shwartz S.K., Roper B.L., Arentsen T.J., Crouse E.M., Adler M.C. The behavior rating inventory of executive Function®-Adult Version is related to emotional distress, not executive dysfunction, in a veteran sample. Arch. Clin. Neuropsychol. 2020;35(6):701–716. doi: 10.1093/arclin/acaa024. [DOI] [PubMed] [Google Scholar]
- Stamenova V., Levine B. Effectiveness of goal management training® in improving executive functions: a meta-analysis. Neuropsychol. Rehabil. 2018:1–31. doi: 10.1080/09602011.2018.1438294. [DOI] [PubMed] [Google Scholar]
- Toplak M.E., West R.F., Stanovich K.E. Practitioner review: do performance-based measures and ratings of executive function assess the same construct? J. Child Psychol. Psychiatry. 2013;54(2):131–143. doi: 10.1111/jcpp.12001. [DOI] [PubMed] [Google Scholar]
- Twamley E.W., Burton C.Z., Vella L. Compensatory cognitive training for psychosis: who benefits? Who stays in treatment? Schizophr. Bull. 2011;37(Suppl. 2):S55–S62. doi: 10.1093/schbul/sbr059. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Twisk J., Bosman L., Hoekstra T., Rijnhart J., Welten M.H., <check>M. </check> Different ways to estimate treatment effects in randomised controlled trials. Contemp. Clin. Trials Commun. 2018;10:80–85. doi: 10.1016/j.conctc.2018.03.008. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Van Aken B., Wierdsma A.I., Voskes Y., Pijnenborg G., Weeghel J., Mulder C. The association between executive functioning and personal recovery in people with psychotic disorders. Schizophr. Bull. Open. 2022 doi: 10.1093/schizbullopen/sgac023. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Vita A., Barlati S., Ceraso A. Effectiveness, core elements, and moderators of response of cognitive remediation for schizophrenia: a systematic review and meta-analysis of randomized clinical trials. JAMA Psychiatry. 2021;78(8):848–858. doi: 10.1001/jamapsychiatry.2021.0620. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Wechsler D. NCS Pearson, Inc; San Antonio, Texas: 1999. Wechsler Abbreviated Scale of Intelligence (WASI) [Google Scholar]
- Wechsler D. Pearson Assessment; San Antonio, Texas: 2008. Wechsler Adult Intelligence Scale - Fourth Edition (WAIS-IV) [Google Scholar]
- Wykes T., Spaulding W.D. Thinking about the future cognitive remediation therapy--what works and could we do better? Schizophr. Bull. 2011;37(Suppl. 2):S80–S90. doi: 10.1093/schbul/sbr064. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Yung A.R., Phillips L.J., McGorry P.D., et al. Prediction of psychosis: a step towards indicated prevention of schizophrenia. Br. J. Psychiatry Suppl. 1998;172(33):14–20. doi: 10.1192/S0007125000297602. [DOI] [PubMed] [Google Scholar]

