EPICOG-SCH: A brief battery to screen cognitive impact of schizophrenia in stable outpatients

Abstract

Brief batteries in schizophrenia, are needed to screen for the cognitive impact of schizophrenia. We aimed to validate and co-norm the Epidemiological Study of Cognitive Impairment in Schizophrenia (EPICOG-SCH) derived brief cognitive battery. A cross-sectional outpatient evaluation was conducted of six-hundred-seventy-two patients recruited from 234 centers. The brief battery included well-known subtests available worldwide that cover cognitive domains related to functional outcomes: WAIS-III-Letter-Number-Sequencing-LNS, Category Fluency Test-CFT, Logical-Memory Immediate Recall-LM, and Digit-Symbol-Coding-DSC. CGI-SCH Severity and WHO-DAS-S were used to assess clinical severity and functional impairment, respectively. Unit Composite Score (UCS) and functional regression-weighted Composite Scores (FWCS) were obtained; discriminant properties of FWCS to identify patients with different levels of functional disability were analyzed using receiver-operating characteristic (ROC) technique. The battery showed good internal consistency, Cronbach's alpha = 0.78. The differences between cognitive performance across CGI-SCH severity level subscales ranged from 0.5 to 1 SD. Discriminant capacity of the battery in identifying patients with up to moderate disability levels showed fair discriminant accuracy with areas under the curve (AUC) > 0.70, p < 0.0001. An FWCS mean cut-off score ≥ 100 showed likelihood ratios (LR) up to 4.7, with an LR + of 2.3 and a LR − of 0.5. An FWCS cut-off ≥ 96 provided the best balance between sensitivity (0.74) and specificity (0.62).

The EPICOG-SCH proved to be a useful brief tool to screen for the cognitive impact of schizophrenia, and its regression-weighted Composite Score was an efficient complement to clinical interviews for confirming patients' potential functional outcomes and can be useful for monitoring cognition during routine outpatient follow-up visits.

1. Introduction

Cognitive impairment is one of the primary features of schizophrenia, and such impairment has an associated impact on patient functioning in daily life (Bowie and Harvey, 2006, Bowie et al., 2008, Green et al., 2000, Green et al., 2004a, Harvey et al., 2006a, Harvey et al., 2006b, Velligan et al., 1997, Green, 1996, Velligan et al., 1997). Overall, cognition accounts for 20–60% of the variance in patients' functional outcomes across studies (Galderisi et al., 2014, Velligan et al., 1997, Zaragoza Domingo et al., 2015). On a practical level, it is currently broadly accepted that functional capacity shows very strong and consistent associations with performance on neurocognitive test batteries (Green and Harvey, 2014, Harvey, 2012).

The cognitive impairment associated with schizophrenia is accepted, including its wide variability and heterogeneity both, in the domains that are affected and the degree to which these domains are involved (Fioravanti et al., 2012). A large proportion, but not all, of schizophrenia patients might develop significant, moderate-to-severe cognitive impairment (Montgomery and van Zwieten-Boot, 2007), however approximately 20–25% of patients might have normal scores on neuropsychological tests (Lennertz et al., 2016, Palmer et al., 2009, Wexler et al., 2009), consequently showing a substantial overlap of cognitive performance with healthy population (Harvey, 2012). In one of the first published meta-analyses, based on the average patient performance on 22 psychological tests, patient performance was between 0.46 and 1.41 SD below the performance of controls (Heinrichs and Zakzanis, 1998), and the deficit be as severe as 2–3 standard deviations below the mean (Keefe et al., 2006).

Based on studies with different patient groups and different strategies of analysis, the processing speed and, more recently, working memory domains have been recognized as valid and efficient indicators of overall cognitive functioning in schizophrenia that are also related to functional outcomes (Hurford et al., 2011, Wong et al., 2013, Fervaha et al., 2014a, Kern et al., 2011). In these published studies, subtests such as letter sequencing (LNS), digit-symbol coding (DSC) and category fluencies subtest CFT) were identified as key subtests to be used in schizophrenia. Somehow, these key subtests are linked to the most relevant pathophysiological findings in the disease (Joyce, 2013).

Regarding cognitive assessment, the instruments typically used to measure cognitive function in schizophrenia fall into three main categories, with a high variation in their testing times: performance-based assessment batteries comprising standard neuropsychological tests (mostly paper and pencil), computerized performed-based test batteries, and interview-based assessments (Keefe, 2012). Although performance-based measures are essential for measuring cognitive change, clinicians at the International Society for CNS Clinical Trials Methodology (ISCTM) consensus meeting noted their cost and time requirements as evidence of impracticality. Among the factors considered there were the complexity of test administration, sensitivity and reliability, time, cost, reimbursement, and training.

In this context, brief performance-based assessment batteries are considered a real solution to cover evaluation needs in clinical practice. Some brief (RBANS (Gold et al., 1999); BACS (Keefe et al., 2004), SCIP-S (Pino et al., 2007, Purdon, 2005)) or ultra-brief cognitive batteries with < 4 subtests, such as the BNA (Fervaha et al., 2014a), the BCA (Velligan et al., 2004), or the B-CATS (Hurford et al., 2011), have been developed using different construct models and are currently available for use in clinical contexts (Bakkour et al., 2014, Keefe et al., 2016). The existing ultra-brief performance-based cognitive measures were developed using data from studies with large patient samples, suggesting that shorter tests were equally sensitive. However, this approach offers little control for study sample heterogeneity because the selection criteria have various objectives; therefore, validation of this ultra brief measures was performed using a sample recruited for various purposes.

In the Epidemiological Study of Cognitive Impairment in Schizophrenia (EPICOG-SCH), we evaluated the performance of clinically stable patients on specific cognitive domains proven to be associated with functional status as reported in the literature, and we estimated the prevalence of cognitive impairment in those domains and its relationship to patients' clinical features and functional outcomes (Zaragoza Domingo et al., 2015). In this paper, we present the validation data for this new performance-based assessment battery, i.e., the EPICOG-SCH brief battery, co-norming the 4 different subtests and standardizing the battery across a large of clinically stable sample of outpatients undergoing stable drug treatment with second-generation antipsychotic drugs as their primary therapy. Battery summary scores are presented emphasizing a functional regression-weighted Composite Score (FWCS) with the capacity to confirm and predict functional disability related to a patient's mental health condition.

This study will provide information to clinicians about the validity and reliability of the new EPICOG-SCH brief battery, which is useful not only to screen the cognitive status of schizophrenia outpatients but also to monitor the cognitive impact of schizophrenia over time. Normative data based on a large representative population are also provided and ready to use as a reference for mental health specialists in clinical practice.

2. Materials and methods

We conducted a cross-sectional epidemiological study with a sample of schizophrenia outpatients diagnosed according to the DSM-IV-TR criteria (American Psychiatric Association, 2002) who were on maintenance treatment with at least one second-generation antipsychotic drug and for whom all drug treatments had remained stable during the previous six months. The patients attended a routine follow-up visit at one of the community-based mental health service centers, within the National Public Health System in Spain. A full description of the methods and procedures for the study is given elsewhere (Zaragoza Domingo et al., 2015). The participants completed informed consent to participate in the study. The study was approved by the Clinical Research Ethics Committee of one of the participating centers, and this approval extended to all the other sites.

2.1. Cognitive assessment battery

For the selection of the domains to be included, the MATRICS-RAND review work was used to define the relationship to functional outcomes (Nuechterlein et al., 2008, Green et al., 2004b). With the aims of creating an ad hoc brief instrument, not all 7 domains selected by MATRICS panel were selected, but those domains more related to schizophrenia as a minimal set and also aiming to capture the effect of common concomitant drug prescription practices (as it is frequently reported CNS drug poly-therapy as well the use of concomitant anticholinergic agents, among others). Subtests also were selected by considering a list of criteria defined a priori and sorted by priority, as described in Table 1, the final composition of the battery was based on these criteria. Four domains were identified as relevant in schizophrenia, including executive function (although it was not initially considered within the MATRICS review model), and cognitive battery comprised four subtests: Letter-Number Sequencing (LNS), Digit Symbol Coding (DSC), Logical Memory (LM), and the Category Fluency Test (CFT). A full description of the testing procedures is provided elsewhere (Zaragoza Domingo et al., 2015). The relationship between cognitive subtests and functional outcomes was based on a previously published review (Nuechterlein et al., 2008): the review reported that performance on LNS subtest showed an association with the capacity to have a work contract in two studies including patients with severe mental disease (Study 1, N = 33, r = 0.48, p < 0.001, and Study 2, N = 40, r = 0.38, p < 0.05); performance on LM was also associated with the capacity to have a work contract in patients diagnosed with severe mental disorder (r = 0.4, p < 0.05) and was related to quality of life in a sample of patients with schizophrenia (r = 0.37, p = 0.04); performance on the CFT (4 categories) was not related to having a work contract (r = 0.44, non-significant) but was associated with quality of life in patients with schizophrenia (r = 0.54, p < 0.001); and performance on the DSC subtest in a sample of healthy adults was associated with measures of vocational outcome (r = 0.42, p < 0.001).

Table 1.

The EPICOG-SCH brief cognitive battery: development and derived scores.

Set of Criteria Used to Select Subtests
For the final composition of the EPICOG-SCH battery, a list of criteria was elaborated to guide the selection of cognitive subtests. Subtest candidates should be in line with as many criteria as possible, with criteria 1 to 5 taken as priority and criteria 6 to 9 taken to guide the selection when there are several options. 1.Subtest evaluating domains related to functional measures in the disease.a 2.Subtests evaluating any of the domains relevant to the disease supported by literature. 3.Subtest easy to administer without the need for special training. 4.Demonstrated sensitivity in pharmacological studies.a 5.Demonstrated stability over time (very small test-retest variation).a, b 6.Availability of local versions and normative data from the general population, published whenever possible. 7.Subtests allowing extracting conclusions related to the disease (not simplistic). 8.Subtests with strategic interest for future research. 9.Potential to be part of a battery and co-normative data on the target population.

Ad hoc cognitive subtests selected and the corresponding classical domains [Mean estimated administration time: 20 to 30 minutes]
Subtestc	Domain Contribution	Average Administration Time (min.)
•LNS, Letter Number Sequencing - WAIS-III (Wechsler, 2001, Gold et al., 1997)	Working memory	5-15
•CFT, Category Fluency Test (3 categories: animals, fruits, and cities/villages) (Benton A.L and Hamscher., 1978)	Speed of Information Processing Executive functioning	5
•DSC, Symbol-Digit Coding - WAIS-III (Weschler D, 2001)	Speed of information processing Working memory	5
•LM, Logical Memory WMS-III - Text A (Wechsler D, 2001)	Verbal memory	6-10
Composite Scores
•Unit Total Composite Score (UCS)	•Raw data from subtests were first transformed into standard Z scores and then transformed into scalar scores, with mean 10 SD 3. Subtest scalar scores were added, and the result was transformed into Z scores, followed by transformation into the UCS, with mean 100 and SD 15, resulting in a 95% CI of 98.8 to 101.1 and ranging from 53.6 to 152.9; test for normality p < 0.005.
•Functional Weighted Composite Score (FWCS)	•Raw subtest scores were transformed into Z scores, and then each subtest score is weighted using β values and summed using a regression equationd. Resulting values were first standardized into Z scores and then transformed to the FWCS, with mean 100 and SD 15, resulting in a 95% CI of 98.9 to 101.1, ranging from 60.7 to 159.8; test for normality p = 0.070.
Cut-off Scores for composite scores
•Impact on Global Functioninge	Cut-off score based on Comopsite Scores able to predict patient functional outcome. For FWCS a mean cut-off score ≥ 100 increase the probability of correctly identifying patients with no or moderate functional impact by up to 4.7 times, and FWCS cut-off ≥ 96 provided the best balance between sensitivity (0.74) and specificity (0.62).
Normative Data
•Subtest and Composite Scores	Tabulated data from the normative study population (clinically stable), without adjustments, to facilitate the conversion from row to percentile scores as mean group values (see Supplementary tables 2-6).

The aim of developing the EPICOG-SCH was to create a briefly administered battery, i.e., requiring less than 30 minutes, to be included in clinical practice protocols for schizophrenia outpatient follow-up visits. Method for initial test selection by setting a priori criteria as a first step was similar to the MATRICS initiative process of developing a consensus battery (Green et al., 2004). EPICOG-SCH battery, based in classical well known neuropsychological tests, is easy to administer in a standardized procedure by mental health professionals and is suitable for evaluating any type of patient who is able to cooperate and understand.

^aBased on review data from the MATRICS group (Nuechterlein et al., 2008).

^bAccording to published rest-retest properties (McCaffrey R.J., 2000).

^cThe relationship between cognitive subtests and functional outcomes was based on previously published information (Nuechterlein et al., 2008).

^dThe FWCS was based on a regression analysis in which standard scores were weighted by their contribution to a patient´s functional status according to WHO-DAS-S From this regression model, cognitive performance alone explained 17.4% of the patients’ functional status (WHO-DAS-SF (p = 0.000)), including LNS p = 0.000, CFT, p = 0.001, DSC p = 0.037, LM Items p = 0.289 (ns) and LM Issues p = 0.810 (ns).

^eROC analysis was applied to test models of functionality or disability according to pre-defined categories based on WHO-DAS-S scores for each dimension (see Supplementary Table 1).

2.2. Data analysis

All forms were centrally collected, and the scores for cognitive tests were centrally monitored to check for accuracy and consistency among the raters.

For the impairment prevalence estimates, raw scores were transformed into standardized scores (scalar or centile depending on the test) based on published local normative data (Wechsler, 2001, Casals-Coll et al., 2013, Pena-Casanova et al., 2009), with age as the only stratification variable (CFT-Animals) and age and years of education for the other subtests. The prevalence of impairment was defined as the percentage of patients with scores below the cut-off, which was equal to or < 1.5 standard deviations (SD) from the mean of standardized scores (i.e., a − 1.5 SD cut-off, corresponding to a score < 5.5 for scalar percentiles); the prevalence estimates are given and differences at all cut-off values by gender are described.

The battery summary Composite Scores were calculated using unit- and regression-weighted based models, UCS and FWCS respectively (see Table 1 for exact standardization and calculation process). The FWCS was based on a regression analysis in which standard scores were weighted based on their contribution to a patient's functional status according to the WHO-DAS-S. To improve the percentage of variability explained only by cognitive performance, a new WHO-DAS-S Total Score termed the WHO-DAS-SF was calculated per patient as the total number of unaffected domains according to the WHO-DAS-S; i.e., for each patient, the number of domains on the WHO-DAS-S with a score < 2 corresponding to “Non-disability or minimal disability” was calculated. This new measure ranged from 0 to 4, 4 representing no disability (i.e., all 4 domains were unaffected), and 0 representing disability in all 4 domains (i.e., meaning 0 domains were unaffected). A linear regression analysis was built with WHO-DAS-SF where cognitive subtests were predictive variables, and a stepwise method for factor inclusion was used to determine weights (β) for each cognitive subtest.

The internal consistency and criterion validity of the battery were calculated. The criterion validity of the battery was based on the relationship between the Composite Scores and the patients' clinical and functional status. The Wilcoxon test was used to describe differences in performance among clinical subgroups, and Pearson and Spearman correlations were used to describe the association between cognitive performance and clinical and functional disability results. Patients at each level of symptom severity were compared following the categorization of severity, and patients in the symptom “Not Present” category were compared with those in the other ill-defined “categories” grouped into a single category. This method was used to smooth differences in cognitive performance on each symptom subscale between extreme severity levels.

For group differences, we also present Cohen's d as a measure of the standardized effect size (SES). A stepwise multiple linear regression analysis was used to explore the relationship between cognitive performance (each subtest and each Composite Score) and factors related to the natural course of the disease, using natural factors as dependent variables. The factors considered here were Years of Disease Evolution i.e., elapsed time since first episode, Age at First Episode, Number of Relapses during the Previous Year, Elapsed Time since Last Relapse, Gender was also included in all regression models.

To further explore the utility for both Composite Scores for predicting a patient's functional status, the Composite Score with better discriminative properties based on receiver operating characteristic (ROC) analysis was selected, and a cut-off score was calculated according to its capacity to discriminate between levels of the patient functionality-disability under different definitions (see Table 1 and Supplementary Table 1). All statistical tests were performed with the significance level set at 5%. The data were analyzed using SAS software (SAS Institute, Cary, NC).

3. Results

Of 848 patients from 234 centers, 672 were analyzed; 176 cases (20.8%) with protocol deviations were identified, primarily due to an insufficient maintenance treatment duration or a drug regimen change (a detailed sample description can be found elsewhere (Zaragoza Domingo et al., 2015). The prevalence of cognitive impact varied according to the method used, ranging from 78% in patients' responses to an open question, to 91% when considering clinician impression scales (Haro et al., 2003). Ultimately, cognitive impairment based on objective measurement varied, depending on the subtest, from a prevalence of 20.9% for DSC to 65.8% for CFT.

Men and women showed similar cognitive results across all subtests and Composite Scores, and only small differences were observed; men showed better performance on LNS and CFT, while women showed better performance on LM-Issues. In addition, an analysis of gender differences in the prevalence of impairment at a cut-off of 1.5 SD showed highly similar rates for men and women (see Table 2), and at a less severe impairment (cut-off − 1 SD), fewer women than men showed impairment on Verbal Memory and Information Processing Speed, i.e., LM Items (p = 0.019), Issues (p = 0.003) and DSC (p = 0.024). Also less women were at severe impairment group (cut-off of − 2 SD) for Verbal Memory, LM Item subtest (p = 0.034).

Table 2.

Cognitive performance results of the EPICOG-SCH brief battery subtests and the prevalence of cognitive impairment by total sample and by gender.

	Cognitive performancea								Prevalence of cognitive impairment									Gender differences
Cognitive Subtest and Derived Scores	Total sample (N = 672)		Male (N = 447)		Female (N = 218)		p	SES	Total sample (N = 672)			Male (N = 447)			Female (N = 218)			Prevalence of Cognitive Impairment at ≤ 1.5 SD Cut-off
	Mean	SD	Mean	SD	Mean	SD			≤ 1 SD (Score ≤ 7)	≤ 1.5 SD (Score ≤ 5.5)	≤ 2 SD (Score ≤ 4)	≤ 1 SD (Score ≤ 7)	≤ 1.5 SD (Score ≤ 5.5)	≤ 2 SD (Score ≤ 4)	≤ 1 SD (Score ≤ 7)	≤ 1,5 SD (Score ≤ 5.5)	≤ 2 SD (Score ≤ 4)	p
Letter-Number Sequencing (WAIS-III)	8.5	3.9	8.8	4.0	8.0	3.7	0.008	− 0.21	37.7	20.9	12.8	36.8	20.4	12.7	39.9	22.1	13.0	ns
Digit-Symbol Coding (WAIS-III)	43.6	21.5	43.6	21.0	43.4	22.6	ns	–	63.4	38.1	27.9	65.4	38.9	29.0	59.5	36.2	25.7	ns
Logical Memory (WMS-III-Text A)
Units	10.4	4.6	10.2	4.6	10.9	4.6	ns	–	38.0	24.8	12.2	40.3	25.5	13.7	33.8	23.8	9.2	ns
Issues	4.6	1.7	4.4	1.7	4.9	1.6	0.002	0.25	25.4	11.7	6.0	28.7	13.7	6.7	19.2	8.1	4.8	0.004
									≤ 25th percentile	≤ 10th percentile	≤ 5th percentile	≤ 25th percentile	≤ 10th percentile	≤ 5th percentile	≤ 25th percentile	≤ 10th percentile	≤ 5th percentile
Category fluency test	39.4	15.6	40.2	15.2	37.9	16.4	0.045	− 0.15	–	–	–	–	–	–	–	–	–	–
Animalsb	14.0	5.6	14.2	5.7	13.5	5.8	ns	–	82.5	65.8	57.8	81.9	64.6	56.3	83.7	67.8	60.9	ns
Fruitsc	10.0	3.6	10.0	3.4	10.0	3.8	ns	–	–	–	–	–	–	–	–	–	–	–
Cities-Villagesc	15.5	8.1	16.0	8.0	14.4	8.4	0.017	− 0.20	–	–	–	–	–	–	–	–	–	–

SD, standard deviation; SES, standardized effect size (Cohen's d), interpretation Small 0.20, Medium 0.50, Large 0.80.

Prevalence of cognitive impairment is based on population normative data available in the country for each individual subtest. In a normal distribution, the expected percentages of patients performing at ≤ 1 SD and of patients performing at ≤ 2 SD cut-offs are 15.7% and 2.1%, respectively. Disability in cognitive tests was set at ≤ 1.5 SD (10th percentile). Estimates for the cut-offs of 1.0 (25th percentile) and 2.0 SD (5th percentile) below the mean are also included, and they correspond to the lower limit of normal cognitive performance and to severe impairment, respectively (Harvey et al., 2006b, Taylor and Heaton, 2001); these cut-offs for scalar scores correspond to scores between < 7 and < 4 and for centile scores between < 25 and < 5, respectively.

^aRaw Scores.

^bSpanish normative value available CFT-Animals, from the Neuronorma project; for young population (Casals-Coll et al., 2013) and for older population (Pena-Casanova et al., 2009). Estimated prevalence adjusted by both age and years of education achieved (N = 640).

^cNormative data not available in Spain.

The descriptive statistics obtained following the calculation of the summary Composite Scores UCS and FWCS are included in Table 1. We found a significant overall effect of age, level of education and functional status, on all individual subtests and on the Composite Scores (Table 3). Higher education, younger age and an active functional status were associated with better cognitive performance.

Table 3.

EPICOG-SCH battery composite scores relationship to sociodemographic, clinical and functional factors.

Variable	Unit Composite Score (UCS)								Functional Weighted Composite Score (FWCS)
				Differencea	95% CI Difference						Differencea	95% CI Difference		.
	n	Mean	SD	Mean	Low	High	Sign.	SES	Mean	SD	Mean	Low	High	Sign	SES
Sociodemographic
Age range (years)
18–39	335	102.3	14.3	− 10.2	− 14.1	− 6.2	< 0.0001	− 0.66	102.3	14.2	− 9.8	− 13.7	− 5.8	< 0.0001	− 0.66
40–49	186	100.5	14.2						100.2	14.8
50–69	95	92.2	15.6						92.6	15.1
Gender
Male	446	99.8	14.7	0.2	− 2.2	2.3	0.8114	− 0.11	100.7	14.8	− 2.3	− 4.7	0.1	0.061 ns	− 0.15
Female	216	100.0	15.7						98.4	15.2
Level of education
No education completed	66	84.6	14.7	24.4	18.6	30.9	< 0.0001	1.76	84.8	12.7	23.8	17.6	30.1	< 0.0001	1.78
Primary	312	96.9	13.6						97.2	13.6
High school	226	106.1	12.6						106.0	13.4
University	60	109.4	13.5						108.6	14.2
Clinical profile
Deficit syndromeb
No	453	106.6	13.9	− 9.8	− 12.2	− 7.4	< 0.0001	− 0.69	106.7	14.6	− 9.9	− 12.3	− 7.6	< 0.0001	− 0.69
Yes	209	96.8	14.5						96.8	14.1
Treatment-related measures
Treatment adherence
Yes	542	100.8	14.7	− 4.7	− 7.8	− 1.6	0.0041	− 0.32	101.2	14.8	− 6.0	− 9.1	− 2.9	0.0002	− 0.41
No	103	96.1	15.1						95.2	14.6
Anticholinergic agents
No	567	100.9	14.8	− 6.3	− 9.4	− 3.1	0.0002	− 0.42	100.7	14.9	− 4.3	− 7.4	− 1.2	0.0059	0.29
Yes	105	94.7	15.2						96.4	15.3
Treatment satisfactionc
Not at all	7	97.6	23.1	13.1	5.4	20.5	< 0.0001	0.81	97.5	20.4	13.3	5.8	20.8	< 0.0001	0.84
Slightly satisfiedc	69	90.5	16.5						90.8	15.3
Moderately satisfied	172	98.6	12.6						97.9	12.9
Very satisfied	370	102.1	14.8						102.5	14.7
Extremely satisfiedc	48	103.5	15.7						104.1	16.2
Functional measures
Occupational status
Non-active	417	97.8	14.9	7.2	5.0	10.1	< 0.001	0.51	98.4	14.8	5.6	3.3	8.4	< 0.0001	0.39
Active	204	105.3	14.6						104.3	15.2
Currently a student
No	569	98.4	14.6	12.7	9.3	16.1	< 0.001	0.92	98.6	14.5	11.6	8.2	15.0	< 0.0001	0.80
Yes	80	111.0	12.8						110.1	14.4
Type of work
Non-qualified worker	121	102.0	14.3	7.2	0.0	14.3	0.071 ns	0.46	101.6	13.8	8.5	1.4	15.6	0.0115	0.59
Qualified worker	54	103.5	11.1						104.9	11.9
Qualified professional	25	109.1	16.2						110.1	15.7
WHO DAS-S Disability personal care
No	473	103.4	13.8	− 11.8	− 14.1	− 9.5	< 0.001	− 0.82	103.9	13.7	− 13.3	− 15.5	− 11.0	< 0.001	− 0.96
Yes	198	91.6	13.0						90.6	13.9
WHO DAS-S Disability occupational functioning
No	195	105.9	13.4	− 8.4	− 10.8	− 5.9	< 0.001	− 0.59	106.4	13.8	− 9.0	− 11.4	− 6.6	< 0.001	− 0.56
Yes	474	97.5	15.0						97.4	14.7
WHO DAS-S Disability familiar functioning
No	257	105.2	13.4	− 8.4	− 10.6	− 6.1	< 0.001	− 0.59	105.5	13.5	− 8.8	− 11.1	− 6.1	< 0.001	− 0.63
Yes	413	96.8	15.0						96.6	14.9
WHO DAS-S Disability broad social context
No	153	106.9	13.7	− 9.0	− 11.5	− 6.3	< 0.001	− 0.69	107.3	13.9	− 9.5	− 12.0	− 6.9	< 0.001	− 0.67
Yes	518	97.9	14.8						97.8	14.6
WHO-DAS-S Length of disability
Less than one year	48	106.5	16.0	− 8.0	− 12.8	− 3.5	< 0.001	− 0.51	106.2	16.7	− 7.6	− 12.0	− 3.2	0.0029	− 0.48
One year or longer	534	98.7	14.9						98.7	14.7
WHO DAS-S Specific skills
No	503	99.0	15.2	3.7	1.1	6.3	0.0032	0.25	99.1	15.2	3.4	0.8	6.0	0.0078	0.24
Yes	169	102.7	14.1						102.6	14.2

Statistical significance: *p < 0.05, **p < 0.01, ***p < 0.001, ****p < 0.0001, SES, standardized effect size (Cohen's d), effect size interpretation, Cohen's d, small 0.20, medium 0.50, large 0.80. WHO DAS-S, World Health Organization Disability Scale-Short Version (Janca et al., 1996, Sartorius et al., 1986).

^aFor multiple categories, overall significance is reported but the mean difference, 95% IC and SES for the extreme categories are reported.

^bAccording to specific criteria for deficit syndrome (Arango et al., 1998, Arango et al., 2004, Kirkpatrick et al., 2000)

^cFor treatment satisfaction, the lowest significant difference is reported, and the mean differences across categories ranged between 1.0 and 13.0 points.

3.1. Battery properties and criterion validity

The internal consistency of the battery was demonstrated by a Cronbach's alpha score of 0.78, indicating good internal consistency among subtests.

The criterion validity analysis showed a moderate and statistically significant relationship between the patients' performance on the cognitive battery results and their overall clinical status based on the clinical impression scales (Table 4), battery Composite Scores showed a modest inverted linear relationship with the severity of Cognitive Symptoms (r = − 0.44, p < 0.001) and also with severity of Global Disease (r = 0.40, p < 0.001) and Negative Symptoms (r = − 0.28, p < 0.001). This association was further explored by comparing the performance of patients within severity levels first at the subtests level and subsequently at the Composite Scores level (Fig. 1). First, as for the individual subtests, the severity of Negative Symptoms and Global Disease was largely associated with a decreased performance on Memory/Working Memory subtests: Negative Symptoms severity was associated with LM (p < 0.0001) and LNS (p < 0.0001) and Global Disease severity was associated with LM (p < 0.0021). Furthermore, the severity of Positive Symptoms was associated with decreased performance on the Information Processing Speed subtest, DSC (p < 0.0001). To a lesser extent but still significantly, we found that patients with associated Deficit Syndrome performed poorly, with moderate differences on Memory and Executive subtests; LNS (p < 0.0001), LM (p < 0.0001) and CFT (p < 0.0001). Finally, the severity of Depressive Symptoms, was associated with the performance in all subtests, but predominantly to the Information Processing Speed task DSC (p = 0.0008). It is important to highlight the strong association observed between DSC and CFT subtests (r = 0.57, p = 0.001), which suggests that the DSC makes a moderate contribution to executive functioning domain.

Table 4.

Relationship between clinical impression symptoms severity, disability and cognitive results in the EPICOG-SCH battery.

Measure (N = 672)	LNS	CFT	DSC	LM-Items	LM-Issues	EPICOG-SCH Battery Composite Scores
						Unit Sum (UCS)	Functional Weighted (FWCS)
	r	r	r	r	r	r	r
Clinical Evolution
Age of onset	− 0.01	− 0.05	− 0.09⁎	0.01	− 0.00	− 0.06	− 0.03
Time of evolution (years)	− 0.18⁎⁎⁎	− 0.08⁎	− 0.22⁎⁎⁎	− 0.15⁎⁎⁎	− 0.08⁎	− 0.16⁎⁎	− 0.17⁎⁎
Relapses during last year (#)	− 0.06	− 0.03	0.05	− 0.02	0.05	− 0.02	− 0.07
CGI – Severity
Global severity	− 0.36⁎⁎⁎	− 0.26⁎⁎⁎	− 0.32⁎⁎⁎	− 0.32⁎⁎⁎	− 0.25⁎⁎⁎	− 0.40⁎⁎⁎	− 0.40⁎⁎⁎
Cognitive symptoms	− 0.37⁎⁎⁎	− 0.34⁎⁎⁎	− 0.33⁎⁎⁎	− 0.32⁎⁎⁎	− 0.25⁎⁎⁎	− 0.44⁎⁎⁎	− 0.44⁎⁎⁎
Positive symptoms	− 0.21⁎⁎⁎	− 0.15⁎⁎	− 0.17⁎⁎⁎	− 0.12⁎⁎	− 0.10⁎	− 0.21⁎⁎⁎	− 0.23⁎⁎⁎
Negative symptoms	− 0.34⁎⁎⁎	− 0.26⁎⁎⁎	− 0.28⁎⁎⁎	− 0.33⁎⁎⁎	− 0.26⁎⁎⁎	− 0.40⁎⁎⁎	− 0.38⁎⁎⁎
Depressive symptoms	− 0.17⁎⁎⁎	− 0.14⁎⁎	− 0.17⁎⁎⁎	− 0.04	− 0.01	− 0.14⁎⁎	− 0.18⁎⁎
WHO-DAS-S Total Sum Score	− 0.38⁎⁎⁎	− 0.33⁎⁎⁎	− 0.31⁎⁎⁎	− 0.28⁎⁎⁎	− 0.20⁎⁎⁎	− 0.41⁎⁎⁎	− 0.44⁎⁎⁎
Personal care	− 0.35⁎⁎⁎	− 0.31⁎⁎⁎	− 0.27⁎⁎⁎	− 0.23⁎⁎⁎	− 0.14⁎⁎	− 0.36⁎⁎⁎	− 0.40⁎⁎⁎
Family and household	− 0.29⁎⁎⁎	− 0.27⁎⁎⁎	− 0.26⁎⁎⁎	− 0.22⁎⁎⁎	− 0.13⁎⁎	− 0.32⁎⁎⁎	− 0.35⁎⁎⁎
Occupational functioning	− 0.29⁎⁎⁎	− 0.25⁎⁎⁎	− 0.23⁎⁎⁎	− 0.23⁎⁎⁎	− 0.19⁎⁎⁎	− 0.33⁎⁎⁎	− 0.33⁎⁎⁎
Functioning on broader social context	− 0.31⁎⁎⁎	− 0.28⁎⁎⁎	− 0.22⁎⁎⁎	− 0.25⁎⁎⁎	− 0.19⁎⁎⁎	− 0.35⁎⁎⁎	− 0.36⁎⁎⁎

LNS, Letter-Number-Sequencing; CFT, Category Fluency Test; LM-Items, Logical-Memory Immediate Recall for Items; LM-Issues, Logical-Memory Immediate Recall for Issues; DSC, Digit-Symbol-Coding.

Stepwise multiple regression analysis included all variables related to disease course-related factors and gender, analyzing the contribution of these factors on subtypes and global composite scores for each patient's performance.

For FWCS, the resulting model for the multiple regression analysis (R² = 0.06 p < 0.0001) included years of disease evolution (β = − 0.35, p < 0.0001), number of relapses during previous year (β = − 1.64, p = 0.004), and age at first episode, which was not significant (β = − 0.154, p = 0.143). Similarly, for UCS, the resulting model (R² = 0.06, p < 0.001), included the same factors, i.e., years of disease evolution (β = − 0.31, p = 0.0001), number of relapses during previous year (β = − 1.43, p = 0.0100).

For the subtests, for LNS, the model (R² = 0.06, p < 0.0001) included years of disease evolution (β = − 0.06, p < 0.0001), number of relapses during previous year (β = − 0.30, p = 0.007), and gender, which was not significant (β = 0.49, p = 0.076).

For DSC, the model (R² = 0.06, p < 0.0001) included years of disease evolution (β = − 0.07, p < 0.0001), age at onset (β = − 0.06, p = 0.001), and number of relapses during previous year (β = − 0.18, p = 0.095). Overall, the correlation between age at onset and years of disease evolution was r = − 0.22, p < 0.0001.

For LM items, the model (R² = 0.05, p < 0.0001) included years of disease evolution (β = − 0.05, p = 0.0001), number of relapses during previous year (β = − 0.27, p = 0.020) and gender, which was not significant (β = − 0.50, p = 0.077).

For LM issues, the model (R² = 0.02, p = 0.005) included gender (β = − 0.73, p = 0.008) and number of relapses during previous year (β = − 0.01, p = 0.072), which was not significant.

For CFT, the model (R² = 0.02, p < 0.0001) included years of disease evolution (β = − 0.035, p = 0.015), also but not significant number of relapses during previous year (β = − 0.022, p = 0.061) and age at onset (β = − 0.038, p < 0.085).

^⁎p < 0.05.

^⁎⁎p < 0.01.

^⁎⁎⁎p < 0.001.

Fig. 1.

Standardized effect sizes (SES) by subtest and Composite Scores according to the patients' clinical profile symptom severity. For each EPICOG-SCH measure, patients at each level of symptom severity were compared following categorization of severity where those subjects at “Not Present” category were compared versus subjects at the all other ill-defined categories” grouped in a single category. SES values obtained from other published sources comparing healthy control groups and schizophrenia patient groups can be used as a reference to better understand the comparison of performance across schizophrenia patients having different clinical profiles. Commonly, negative or positive SES value depends on how the author has reported the comparison, but it consistently indicates impairment in the schizophrenia group compared to the healthy control group. The oldest meta-analysis reported a mean SES for Global Verbal Memory subtests d = 1.41 and for Word Fluency d = 1.15, reporting 22 SES values from 204 studies to index schizophrenia versus control groups (Heinrichs and Zakzanis, 1998). Subsequent results showed Hedge's g for DSC g = − 1.55, Category Fluency g = − 1.21, LNS g = − 1.02 and Story Memory g = − 1.41, with data from 100 studies including healthy controls and 9048 people with schizophrenia (Schaefer et al., 2013)). More recently, results have been reported for Letter-Number Sequencing d = − 0.95 and Verbal Capacity d = − 0.89, and the average SES for all Memory Tasks d = − 0.95 was reported by comparing the relative size of group effects with 1101 healthy controls and 58 schizophrenia patients (Haut et al., 2015).

Looking at the battery Composite Scores, for all symptom subscales the mean score differences of UCS and FWCS across severity categories ranged from 0.5 to 1 SD below the mean, with the exception of the Depressive Symptoms which showed a smaller effect. Similar to the results for the subtests, the Composite Scores, showed the largest effect for the performance and severity of Global Disease and Negative Symptom severity, followed by a medium to large effect for the presence of associated Deficit Syndrome, a medium effect for the severity of Positive Symptoms severity, and a small effect for severity of Depressive Symptoms.

For each symptom subscale the following differences in mean scoring were obtained; for the Negative Symptoms subscale, the UCS mean score difference was − 11.6, SD 14.7, 95% CI (− 15.5 to − 7.7), p < 0.0001, and the FWCS mean difference was − 11.2, SD 14.7, 95% CI (− 15.1 to − 7.2), p < 0.0001. For the Global Severity subscale, the UCS mean difference was − 11.4, SD 14.8, 95% CI (− 16.8 to − 6.0), p < 0.0001, and the FWCS mean difference was − 9.9, SD 14.9, 95% CI (− 15.3 to − 4.4), p < 0.0001. For the Positive Symptoms subscale, for UCS, the mean difference was − 7.9, SD 14.7, 95% CI (− 10.5 to − 5.3), p < 0.0001, and for FWCS, the mean difference was − 8.1, SD 14.6, 95% CI (− 10.7 to − 5.5), p < 0.0001. For the Depressive Symptoms subscale, the largest mean difference was for FWCS which showed a mean difference of − 4.0, SD 14.9, 95% CI (− 6.3 to − 1.6), p = 0.0010.

Fig. 2 shows performance results for FWCS for each individual severity category for clinical impression subscales, illustrating the observed associations. With the aim of exploring other categorization strategies to enhance differences, i.e., instead grouping categories, for each subscale comparing the patients within two extreme severity categories (those Normal vs Markedly Ill). With this strategy we found even stronger effects on FWCS results, i.e., the SES results for the severity of Cognitive Symptom it was 1.41, for severity of Global Disease it was 1.20, for severity of Negative Symptom it was 1.18, for severity of Depressive Symptom it was 0.76, and for severity of Positive Symptom it was 0.72.

Fig. 2.

Relationship in the EPICOG-SCH between FWCS results and CGI-SCH symptom severity subscales. Battery EPICOG-SCH, composite score FWCS mean and 95% CI for each category of severity on CGI-SCH subscales. FWCS scores are significantly related to symptom severity: more ill patients show lower cognitive performance as measured by FWCS scores.

For factors related to the natural course of the disease, we observed only one factor, Years of Disease Evolution, which was inversely related to cognitive performance and had weak correlations with subtest performance and being with Information Processing Speed (DSC) the greatest one (r = − 0.22, p < 0.001) (see Table 4). Regression analysis allowed us to further explore these relationship, and revealed for all measures an inverse linear relationship between cognitive performance and natural course factors (Table 4); for both Composite Scores the resulting regression models included Years of Disease Evolution and Number of Past Year Relapses, and for the individual subtests the same two factors were related to the performance on Memory/Working Memory subtests (LNS and LM Items). The same two factors plus Age at Onset were significantly related to the performance on Information Processing Speed (DST). For Memory (LM-Issues) significant contributors to the model where Number of Relapses during the Past Year and Gender; for CFT performance only Years of Disease Evolution contributed significantly.

3.2. Battery performance and discriminative capacity

Patients who were actively working or engaged in types of work that required greater qualifications obtained significantly higher EPICOG-SCH battery scores (see Table 3). Performance on the battery as measured by UCS and FWCS was inversely and moderately associated with the level of functional disability assessed with the WHO-DAS-S. The strongest relationship was found between cognitive performance and Disability on Personal Care, followed by Disability on Social Functioning. We also observed that patients with disability lasting longer than one year performed nearly − 0.5 SD worse than those with disability lasting less than one year.

Based on ROC analysis, compared with the UCS, the FWCS provided more accurate diagnostic results in distinguishing moderately functional patients under different models (see Supplementary Table 1), i.e. the FWCS showed fair discrimination accuracy values with areas under the curve (AUC) > 0.70 in identifying no functional-disability status under different models tested (see Fig. 3). In the Personal Care dimension, the model to identify no disability performed the best, resulting in an AUC = 0.75, 95% CI 0.71 to 0.79; followed by the model to identify patients defined as with low-moderate disability (none or disability up to 2 domains), with an AUC = 0.70, 95% CI 0.66 to 0.74; and finally the model to identify patients defined with low disability (none or disability in just 1 domain), with an AUC = 0.71, 95% CI 0.66 to 0.76.

Fig. 3.

ROC curves to identify functional-disability status based on different diagnostic tools. ROC Receiver Operating Characteristics, FWCS Functional regression-Weighted Composite Score, AUC, Area under the Curve.

Fig. 3a ROC curves to identify functional outcomes based on cognitive testing from EPICOG-SCH brief battery FWCS. Only models with AUC results ≥ 0.70 are presented. An area of 1 represents a perfect test; an area of 0.5 represents a worthless test. Classification of the accuracy of a diagnostic test according to AUC: 0.5 = No Discrimination, 0.6–0.7 Poor, 0.7–0.8 Acceptable (fair), 0.8–0.9 Excellent (good) > 0.9 Outstanding.

Fig. 3b ROC curves to identify functional outcome based on Clinician Impression scale CGI-SCH Cognitive Subscale (Haro et al., 2003) considering all information available from patient's open question and cognitive testing from individual subtests performance.

At an FWCS mean score of 100, the discrimination power represented by the likelihood ratio (i.e., diagnostic odds) ranged between 3.7 and 4.7 depending on the model tested, indicating that following the administration of the battery, a cut-off ≥ 100 would increase the probability of correctly identifying patients with no or moderate functional impact by up to 4.7 times. Moreover, the confirmatory power of the battery exhibited low but interesting discriminative properties, as shown by the positive and negative likelihood ratios of LR + ranging from 1.7 to 2.3 and LR − from 0.4 to 0.5. This result indicates that a positive cognitive test was found among moderately functional patients up to 2.3 times more often than among patients with more severe impaired functionality and that a negative result in the battery decreases the possibility of having only moderate functionality impact by up to 0.5 times (and increases the possibility of having more severe disability by up to 0.5 times).

Regarding the sensitivity and specificity, the results showed good sensitivity, but low specificity, at specific cut-off scores. The best balance between both indexes was found at an FWCS cut-off score of 96, i.e., < 0.5 SD below the mean, at which the sensitivity ranged from 0.72 to 0.74 and the specificity ranged from 0.56 to 0.62 depending on the model tested; as the obtained FWCS scores decreased, the specificity progressively decreased.

4. Discussion

The brief battery EPICOG-SCH is based on 4 subtests demonstrated to be key in schizophrenia, and it has proven useful for screening to determine for the cognitive impact of schizophrenia in clinically stable outpatients. This battery showed good internal consistency, and the derived Composite Scores showed a relationship with patient sociodemographic profile and as well as association with clinical features of schizophrenia, including the presence of Deficit Syndrome. Furthermore, the battery demonstrated an adequate relationship with patient functional status (i.e., work situation, global disability and specific disabilities in daily life) and to have an adequate discriminative capacity to identify patients with no or a low-moderate global impact on functional outcomes related to mental health and complementary to clinician's impression.

The EPICOG-SCH is not a comprehensive battery but a brief one, and for that reason, it is a good candidate to be included as a screening tool to explore and monitor cognitive status in regular outpatient follow-up visits. Although the length of a test battery for evaluating cognitive status or changes in clinical settings is not established, there is some consensus that small batteries can be as sensitive as longer ones (Fervaha et al., 2014a, Hurford et al., 2011, Keefe et al., 2016). It is important to mention that based on post hoc analysis of studies with large study samples, ultra-brief batteries covering the assessment of the processing speed and working memory domains are recognized valid and efficient indicators of overall cognitive functioning in schizophrenia; furthermore these domains are also related to patient functional outcomes (Fervaha et al., 2014a, Shelton et al., 2009). Subtest selection for the EPICOG-SCH was guided not only by well-defined assessment objectives but also by existing practical limitations such as the availability of country normative data and its usability features (in simplicity of administration and scoring directions, administration time required, a need for minimal material, the use of well-known neuropsychological tasks), among others. Hence, this battery includes key subtests for schizophrenia such as those mentioned in existing brief or ultra-brief batteries (i.e., processing speed and working memory such as in the BNA) and includes the measurement of Executive Functioning and Verbal Memory, based on evidence that these domain are related to a patient's occupational situation or quality of life (Nuechterlein et al., 2008); therefore, the EPICOG-SCH included the CFT and the Immediate LM subtest. In summary, the EPICOG-SCH was not based on a post-hoc analysis of existing data, but on an ad hoc innovation that addresses a need that was unmet, when the project started, and it was validated through a homogenous patient sample in terms of clinical and treatment stability.

4.1. Relationship to sociodemographic and clinical factors

Similar to the results of other batteries, the EPICOG-SCH showed an association to sociodemographic factors and, as expected, effects of age and education level. Regarding gender, differences found at the subtest level were small and not visible when the performance was combined in global Composite Scores. This observation in such a large sample supports the idea that if differences do exist, they are not large. However, in addition to exploring differences between genders, further data analysis is needed to explore gender differences in terms of domains rather than individual subtests or global scores. Past research concluded that women have specific advantage in executive functioning (Karilampi et al., 2011) or even no differences were found when compared to normal population (Bozikas et al., 2010).

Overall, the Composite Scores were sensitive to the severity of the patient's clinical symptom profile. When comparing subjects with different symptom severities, we observed that differences on battery performance in both battery Composite Scores ranged from an SES of − 0.56 to − 0.87 for larger differences, to an SES 1.20, for the Global Disease severity subscale when FWCS performance of subjects at the extreme severity categories were compared. It has been broadly accepted that schizophrenia is associated with cognitive impairment of approximately 1 SES compared with healthy subjects (Schaefer et al., 2013). In this regard, our study indicated that similar SES differences can be found within a sample of schizophrenia patients, with different degrees of clinical symptom severity. Schaefer et al., 2013 published a meta-analysis based on data from 100 studies including healthy controls and 9048 people with schizophrenia in which a grand mean effect size of Hedges g = − 1.03 was reported across all cognitive tests when schizophrenia patients and controls were compared; most measure-by-measure and domain-level SES values fell within the medium to large range (− 0.63 to − 1–11), and larger SES values observed for the DSC g = − 1.55. Other published reviews comparing patients with healthy subjects, reported SESs ranging from − 0.85 for Memory Tasks (Haut et al., 2015) to − 1.55 for the DSC (Heinrichs and Zakzanis, 1998) or g = − 1.21 for the CFT (Schaefer et al., 2013). In our study, SES calculations and a between-subjects design enabled us to confirm the relationship between studied factors but did not allow for a generalization of the results (Lakens, 2013). This is because Cohen's d effect size can be influenced by several factors related to the sample's features such as heterogeneity of the studied measures.

The severity of Negative Symptoms and Global Disease was strongly associated with Verbal Immediate Memory/Working Memory (LM. LNS) and the severity of Positive Symptoms to Information Processing Speed (DSC). The association between the severity of Negative Symptoms and memory is consistent with results reported by previous researchers (Faerden et al., 2009, Faerden et al., 2013, Roth et al., 2004) and recently, has been confirmed using reliable tools to assess apathy dimensions and negative symptoms (Raffard et al., 2016). Factors such as the emotion toward stimuli novelty during information processing have been analyzed and were shown to play an important role in intermediating this relationship. Additionally, the relationship between “amotivation” and cognition has been extensively explored, and “effort” has been identified as the mediator between motivation and cognition (Fervaha et al., 2014b, Foussias et al., 2015). Despite the existing body of research in this field, the relationship has not yet been described. As previously suggested, contemporary neuropsychological assessment requires the assessment of other factors that may contribute to low cognitive functioning (Harvey, 2012), therefore future research would benefit from including a self-measurement of subjective motivation and effort associated to each cognitive task and also a measure of reliability of the evaluation session made by the clinician administering the tests.

The presence of Deficit Syndrome (Kirkpatrick et al., 2000, Arango et al., 1998, Arango et al., 2004) showed a pattern similar to that of the severity of Negative Symptoms (Verbal Immediate Memory/Working Memory) but adding a medium effect on Executive performance (CFT). Overall, the identified differences were not large, a finding that is consistent with previous research indicating that the deficit subtype of schizophrenia is not markedly distinct from non-deficit schizophrenia in terms of neurocognitive performance (Fervaha et al., 2015). Along these lines, recent research showed that deficit patients tend to perform worse on cognitive tests; however the magnitude of this effect is relatively modest, translating to over 70% overlap in scores between groups. However, small differences related to working memory and executive functioning might have a substantial impact on functional outcomes (Zaragoza Domingo et al., 2015). Similar batteries such as RBANS did not find relationship between cognitive performance and patient clinical symptoms, as measured by BPRS ratings at a short-term follow-up (Gold et al., 1999) but at a long-term follow-up, greater psychiatric symptom severity as measured by the PANSS was associated with lower cognitive performance on specific indexes (Immediate Memory and Attention) (Dickerson et al., 2014).

The EPICOG-SCH battery has proven sensitive to clinical profile and symptom severity; however a causal relationship between these factors cannot be confirmed by a cross-sectional design study, and prospective cohort studies are needed. Given our results, we only can hypothesize that during stable periods of schizophrenia, cognition may not be a static feature of the disease, and might instead be linked to fluctuations as symptom severity varies naturally across short periods of time and therefore co-varies with clinical course. If this hypothesis is confirmed with prospective cohort studies, it could have important methodological implications for clinical trials targeting cognition in schizophrenia.

4.2. Relationship to functional status

The EPICOG-SCH Composite Scores showed a significant medium-to-large relationship to patient functional status in terms of work situation and disability in daily life as measured by the WHO-DAS-S. Additionally they showed a large effect on specific domains such as Personal Care and Broad Social Functioning and, a moderate effect for Occupational and Familial Functioning.

In terms of correlations EPICOG-SCH Composite Scores showed higher correlations with functional measures than individual subtests did (r = 0.44) as expected (Harvey, 2012). In addition to substantial differences in measurement methods used, other similar batteries also reported this relationship; higher performance on RBANS was related to being actively employed (employment time) after controlling for educational differences (p < 0.01) (Gold et al., 1999), BACS performance was strongly related to functional measures such as everyday living skills (r = 0.56) and independent living skills (r = 0.48) (Keefe et al., 2006); and also differences in MCCB Composite Score results were found between patients as a function of employment status (p < 0.05)(August et al., 2012).

The battery's summary global scores provided an overall reliable index of how well the patient performed in multiple domains affected in schizophrenia. Generally speaking, although unit-weight Composite Scores might work well in a number of situations, regression weighted scores have been showed to be a good solution, in other contexts, for calculating a battery Composite Score (Bobko et al., 2016). The latter is a more accurate approach based on maximizing the linear relationship between the predictors (the cognitive subtest scores) and the prediction of a patient's functional outcome (number of areas of disability/functionality). In our study, compared to the UCS, the FWCS provided a more useful summary and consequently performed better in the ROC analysis in its capacity to relate cognition with a patient's functionality; therefore, the FWCS has a potential application in clinical practice as complementary information to confirm or predict a patient's functional outcome. When the battery is used by clinicians, the accuracy to identify patients with only low-moderate functional disability would improve in a range of 3 to 7 times at a FWCS cut off ≥ 100. This accuracy can be understood as the capacity of the battery to help clinicians identify patients by using an objective measure with only low-moderate functional impact (potential functional daily life), thereby complementing the clinical assessment based on interviews. Actually, this result is not a large improvement per se on diagnostic power, but a small improvement that can be meaningful from the mental health standpoint. Along these lines, the information provided by the battery improves the clinician's accuracy to nearly “good” accuracy to identify those patients with a Fair Functional status (see Fig. 3).

Our results have some shortcomings arising from the observed imbalance between sensitivity and specificity. These shortcomings can only be compensated, when determining the best cut off score to use in clinical practice, by analyzing its consequences, for the patient, caregivers or health system, of an incorrect prognosis made with the battery. Methodologically, increase in specificity can be achieved by applying age and education corrections, mainly with adults who are relatively old or poorly educated. This method ensures that cognitive performance is isolated from contextual matters so that each individual is viewed in relation to others based on age, education and gender (Heaton et al., 1999). However, when using corrected norms, it should be noted that education in schizophrenia is not always completed on time due to the natural disease course, consequently correction factors need to be considered on an individual basis. EPICOG-SCH group is using regression-based methods to produce corrected norms (van der Elst et al., 2011, Guardia-Olmos et al., 2015) that will be made available to clinicians as the website based tools for calculation of exact Composite Scores.

4.3. Assessing cognition in clinical practice

In clinical settings, the use of a battery with available normative data from both the general population and a disease-specific population allows for the interpretation of results at two levels: first, understanding a patient's cognitive state compared with that of the reference population, and second, understanding cognitive impact of the disease in relation to other patients with same condition. Within this project, we have co-normed well-known cognitive subtests, improving the comparison of different tests administered at the same time and within the same normative population (Smith and Ivnik, 2003). There is a lack of consensus regarding the best methods for assessing cognitive change in clinical practice, although cognitive impairment is as important as functional disability as a treatment target. Cognitive measures to be valid need to have strong test-retest reliability, validity, correlations with functional outcomes, minimal practice effects, sensitivity to diagnostic differences, and sensitivity to treatment (intervention) effects. In addition they should be practical for testers to administer and tolerable for patients (Keefe et al., 2016).

Similar to other well validated cognitive batteries for schizophrenia, the EPICOG-SCH battery (1) covers key domains that are commonly reported to be affected in schizophrenia, including processing speed, executive functioning and working memory/verbal memory (other brief batteries such as RBANS are less specific while others such as B-BCATS, BCA or BNA are too brief and do not include all these domains) (2) can be administered within a medium evaluation time of ~ 20′ (not as long as the MCCB which requires 60–90′ and the RBANS or BACS which require nearly 30′ or more) (3) does not require any extra material other than paper and pencil (other batteries such as the BACS or MCCB require additional items) (4) includes only traditional neuropsychological tasks that are well known by Mental Health professionals (other batteries such as SCIP or MCCB include some less known tasks) (5) requires minimal training for evaluators and the guidance included in the test manual seems to be sufficient to allow experienced clinicians to administer and score the battery (other batteries such RBANS, MCCB require more intensive training).

The EPICOG-SCH approach allows clinicians first, to quantify cognitive status at the single subtests and Composite Scores levels useful to establish a baseline or follow up cognitive measurement; second, to identify the presence of cognitive impairment (provided that country normative data from general population exist for each subtest included in the battery); third, to locate and understand patient performance within the normative sample of schizophrenia patients (Supplementary Tables 2–6); and fourth, to use the FWCS to predict the potential impact of cognition on functional outcomes by considering the cut off as reference (cut-off < 96). Eventually, setting a baseline for patient's cognitive performance using the EPICOG-SCH would allow clinicians to monitor cognition across routine visits and fluctuations across the course of the disease or in response to therapeutic interventions or drug adjustments.

Future research should address additional features of this battery, such as test-retest stability over time for frequent measurement. It is important to highlight that the stability of subtests included in the EPICOG-SCH battery has been widely described and published elsewhere (Lezack et al., 2004, McCaffrey, 2000) and was one of the main criteria for its inclusion in this battery. Other features such as sensitivity to diagnostic differences and to intervention effects also need to be confirmed. Furthermore alternative forms should be developed for repeated measurements over short time intervals.

Regarding limitations, it is important to highlight that the EPICOG-SCH battery does not evaluate all the relevant domains in this condition, i.e., Attention-Vigilance, Visual Learning and Memory, Reasoning and Problem Solving and Social Cognition, which can be found included in more comprehensive tools as MCCB. Clinicians who aim to explore these dimensions need to use a more comprehensive battery because the EPICOG-SCH does not provide a full picture and covers some of the cognitive skills affected by this disease; indeed, it is limited to the above mentioned domains of processing speed, executive functioning and working memory/verbal memory. The EPICOG-SCH can be especially useful in clinical settings with time constraints; however it does not provide a complete picture of patient cognitive status or enough information to establish cognitive rehabilitation plans. These factors are important to consider when evaluating our results.

It should be noted that the patients studied in the present paper were clinically stable and had mild to moderate degrees of severity, thus compromising the external validity of the results in other populations of outpatients with more severe schizophrenia. Consequently, to increase the external validity, a sensitivity analysis that included all patients in the project (N = 848) will be completed in the future to confirm our findings and further investigate the effects, in terms of direction and magnitude, of our observed results when patient heterogeneity is increased.

4.4. Clinical implications and conclusions

Although other brief and ultra-brief batteries exist, the EPICOG-SCH successfully achieved its objectives of serving as a brief cognitive battery that captures cognitive performance related to patients' functional outcomes in daily living using well-known classical subtests currently available in a number of countries and languages. The battery has proven to be useful to screen for cognitive impact of schizophrenia with normative data, and its functional regression-weighted Composite Score is an efficient complement to routine clinical interviews that aim to confirm patients' potential functional outcomes. Furthermore, this battery is useful for monitoring cognition during routine outpatient follow-up visits. Because the calculation method is available and patient-normative data are presented, the battery is a clinical outcome assessment instrument ready to screen cognitive impact and monitor clinically stable outpatients during maintenance drug or behavioral therapy.

The following are the supplementary data related to this article.

Supplementary Table 1

Prevalence for functioning-disability status under different models and corresponding ROC-AUC results.

Supplementary Tables 2–6

Co-normative Data for Stable Patients Subsample.

Appendix 1

List of investigators for the EPICOG-SCH Study Group.

Supplementary data to this article can be found online at http://dx.doi.org/10.1016/j.scog.2017.03.001.

Contributors

Mrs. S. Zaragoza Domingo, Dr. J. Bobes and Dr. P. García-Portilla designed the study and wrote the protocol. Mrs. C. Morralla managed the literature searches (Mrs. Claudia Morralla died on October 2017). Statistical analysis was performed by Dr. García-Portilla. All authors contributed to and approved the final manuscript. See Appendix 1 for a complete list of participating investigators.

Role of funding source

Funding for this study was partially provided by Sanofi-Aventis and Neuropsychological Research Organization s.l. (PSYNCRO).