A Meta-analytic Review of Non-specific Effects in Randomized Controlled Trials of Cognitive Remediation for Schizophrenia

Abstract

Cognitive remediation (CR) has been found to improve cognitive performance among adults with schizophrenia in randomized controlled trials (RCTs). However, improvements in cognitive performance are often observed in the control groups of RCTs as well. There has been no comprehensive examination of change in control groups for CR, which may inform trial methodology and improve our understanding of measured outcomes for cognitive remediation. In this meta-analysis, we calculated pre–post change in cognitive test performance within control groups of RCTs in 32 CR trials (n = 794 participants) published between 1970 and 2011, and examined the association between pre–post change and sample size, duration of treatment, type of control group, and participants' age, intelligence, duration of illness, and psychiatric symptoms. Results showed that control groups in CR trials showed small effect size changes (Cohen's d = 0.12 ± 0.16) in cognitive test performance over the trial duration. Study characteristics associated with pre–post change included participant age and sample size. These findings suggest attention to change in control groups may help improve detection of cognitive remediation effects for schizophrenia.

Introduction

Randomized controlled trials (RCTs) employ control groups to isolate the treatment effects of interest from non-specific effects, such as placebo responses, natural recovery, external influences, and various design features of studies that may affect outcomes [1]. While comparison of change between active treatment and control groups allows for detection of treatment effects, examination of change in control groups and identifying variables that moderate this change may inform trial methodology and improve our understanding of measured outcomes. For example, examination of control groups in medication trials for schizophrenia has provided a better understanding of placebo responses and helped to develop tools to evaluate trial design and quantify drug effects [2]. Many researchers recognize the existence of non-specific effects and have tried to understand the size of these effects and factors that moderate them [3, 4].

There has been no comprehensive examination of change in control groups among individuals who receive cognitive remediation (CR). CR is a behavioral intervention that consists of specific drills and exercises using computers, paper-and-pencil, or discussion to improve neurocognitive abilities such as attention and working memory. A growing literature shows the beneficial effects of CR on adults with schizophrenia [5, 6]. Adults with schizophrenia often experience various cognitive deficits and difficulties with various aspects of functioning [7–9] so evaluation of CR interventions is important in working to improve the lives of those with schizophrenia.

A recent meta-analysis of CR interventions among adults with schizophrenia reported a small-to-moderate effect size (Cohen's d = 0.45) on global cognition and functioning compared to control groups [10]. The effect size of CR interventions was associated with better pretreatment cognition, fewer psychiatric symptoms, and receipt of adjunctive psychiatric rehabilitation. However, change in control groups of CR trials was not examined. The non-specific effects that occur in CR trials for schizophrenia are not well-understood as there has been no review of pre–post change in control groups in CR trials as existing reviews have focused on change in treatment groups (i.e., difference between treatment and control groups over time).

In this review, we conducted a meta-analysis on pre–post change in cognitive test performance within control groups of CR trials for schizophrenia and examined factors that may moderate this change. This information may inform CR trial methodology and contribute to our understanding of non-specific effects that occur in CR trials for schizophrenia.

Methods

Data Source

Randomized controlled trials (RCTs) of CR for schizophrenia were identified from a previous meta-analysis [10] of 38 independent studies published from 1970 to 2011 that examined the effects of CR on cognition among adults with schizophrenia. All studies were required to meet rigorous methodological criteria for inclusion (detailed in 10). The current review included 32 of the original 38 studies that provided adequate data on control groups to calculate the effect size of pre–post change on cognitive test performance. Authors of studies were contacted for pre–post data on control groups when published data were not available.

For this meta-analytic review, only data on the control groups were extracted from each study. This included sample size, pre- and post-mean test performance (and standard deviations), duration of the study, and participants' age, intelligence quotient (IQ), duration of illness, and change in mental health symptomatology (i.e., scores on the positive and negative syndrome scale). Since the focus of the review was on pre–post change in control groups during the active phase of the study, data from any follow-up phases were not included in the analysis.

Data Analysis

Effect sizes of pre–post change on cognitive test performance within control groups were calculated as Cohen's d [11, 12] and an unbiased estimate of the population effect-size (Cohen's d_p) [13]. Effect sizes from individual studies were used to compute sample-size weighted effect sizes and inverse-variance weighted effect sizes. The effect size for each study was computed as the average of effect sizes of individual test measures, irrespective of the cognitive domain assessed. Care was taken to ensure that the directionality of effect size changes were consistent so that higher values represented an improvement in performance.

Analyses were then conducted to identify study features associated with magnitude of change in control groups. Associations between pre–post effect size change and study and sample characteristics were examined using Spearman's rho. Type of control group was classified as either (a) computer-based training, which was defined as any non-CR intervention provided through a computerized modality such as a computer game; (b) intensive skills training, which was defined as any psychosocial skills-related training four or more times per week, or interventions that had three or more components, such as psychoeducation, basic skills training and stress management strategies; (c) less-intensive skills training, which was defined as psychosocial skills-related training less than four times per week, or interventions that had less than three components; and (d) treatment-as-usual or wait-list control groups. Differences in pre–post change between these four types of control groups were tested using one-way analysis of variance.

Results

The 32 studies included in this review comprised 35 data points (i.e., presented in 35 publications) and 98 cognitive measures for a total of 794 patients with a mean age of 35.98 ± 6.49 years and mean IQ of 83.89 ± 0.83. There was wide variability in pre–post change on cognitive test performance in control groups (Fig. 1). The mean effect size of pre–post change in the control group was Cohen's d = 0.12 ± 0.16 (95 % CI −0.20 + 0.48; range = −0.24 to +0.52; sample size weighted Cohen's d = 0.10; inverse-variance weighted Cohen's d = 0.10). The approximately unbiased estimate of the population effect-size (Cohen's d_p) was 0.12 ± 0.16 (95 % CI −0.20 + 0.48; range = −0.23 to +0.51; sample size weighted Cohen's d_p = 0.09; inverse-variance weighted Cohen's d_p = 0.10).

Fig. 1.

Effect size change across randomized controlled trials of cognitive remediation for schizophrenia. Data from 32 studies provided data for the 35 publications shown left

Table 1 shows that the effect size for change in control groups was negatively correlated with age and sample size, but was not significantly correlated with IQ, illness duration, change in schizophrenia symptoms, or study duration. Effect size for change in control groups also did not significantly differ between control groups categorized as treatment-as-usual/wait-list (n = 12), “less intensive psychosocial skills training” (n = 10), “intensive psychosocial skills training” (n = 7), and “computer-based training” (n = 6), F = 1.11, p = 0.36.

Table 1. Correlations between study features/participant characteristics and pre–post change in cognitive test performance within control groups.

Study characteristics/participant characteristics	# of studies	Spearman's rho	P value
Duration of study (weeks)	34	−0.10	0.56
Dose of intervention (hours)	22	−0.27	0.22
Sample size	35	−0.37	0.03
IQ	9	0.50	0.90
Years of education	19	−0.26	0.28
Age of control group	33	−0.39	0.02
Age at illness onset	6	0.31	0.54
Duration of illness	12	−0.21	0.51
Positive symptoms of schizophreniaa	12	0.48	0.11
Negative symptoms of schizophreniaa	13	0.37	0.21
General psychopathologya	7	0.61	0.15

^aPositive, negative, and general symptoms of schizophrenia were assessed with the Positive and Negative Syndrome Scale (PANSS)

Discussion

To our knowledge, this is the first systematic review focused on change in cognitive performance in control groups of CR trials for schizophrenia. We found that pre–post change in cognitive performance in control groups was common among 32 CR trials and ranged from small to medium effects. These changes observed in control groups may need to be controlled to maximize the detection of treatment effects in CR trials. The degree of change in cognitive performance in control groups was negatively associated with participant age and sample size. These findings demonstrate the non-specific effects that occur in CR trials, how study characteristics are associated with change in control groups, and the importance of attention to control groups in designing CR trials.

The finding that age was negatively associated with change in control groups is consistent with studies on CR treatment groups [14, 15], and suggests there may be an age range when cognitive performance may be most amenable to change. Future studies should examine the optimal age range for cognitive enhancement, as younger age groups may be more responsive to cognitive intervention. Our findings also emphasize the importance of sample size, as larger control groups were observed to have smaller pre–post changes. Researchers designing CR trials for schizophrenia should be encouraged to employ large samples to minimize variability in control groups.

Change in control groups was not significantly associated with any other participant characteristic, such as IQ or schizophrenia symptoms, or any other study feature, such as type of control group. However, given the wide variability in pre–post change in control groups found between studies, additional research is needed with larger sample of studies. In particular, there have been mixed findings about whether improvement in cognitive functioning correlates with improvement in schizophrenia symptoms [16–18], which deserves further study.

Our review has several limitations worth nothing. We relied on published data or data that could be obtained from authors, so there were missing data. We used a rough categorization method to characterize the control groups, and effect sizes for each study were computed as the average across individual cognitive tests instead of on specific cognitive tests or domains. Lastly, there may be other important variables that moderate change in control groups that were not examined in this review, such as participant motivation and expectations, and socioaffective stimulation [19].

In conclusion, our review calls attention to changes in cognitive performance in control groups of CR trials. Many CR interventions have been shown to improve the neurocognition of individuals with schizophrenia, but the market for CR interventions continues to expand requiring rigorous research to evaluate their efficacy. Attention and control of non-specific effects in future CR trials may improve the quality of research on CR for schizophrenia.

Biographies

Rajiv Radhakrishnan, MD is a Psychiatry Resident at Yale School of Medicine. He received his medical degree from St. John's Medical College in Bangalore, India and is currently the Senior Deputy Editor of the American Journal of Psychiatry-Residents' Journal.

Brian D. Kiluk, PhD is an Assistant Professor of Psychiatry and Director of Technology-Based Interventions at Yale School of Medicine. He is an expert on computer-based behavioral therapies and has published on computerized psychotherapies and substance abuse.

Jack Tsai, PhD is a Core Investigator for the Veterans Affairs New England Mental Illness Research, Education, and Clinical Center (MIRECC) and an Assistant Professor of Psychiatry at Yale School of Medicine. He has received federally funded grants and published extensively on schizophrenia and psychiatric rehabilitation.