Validity and reliability of the Brazilian Portuguese version of the BACS (Brief Assessment of Cognition in Schizophrenia)

Abstract

OBJECTIVE:

To assess the validity and reliability of the Brazilian Portuguese version of the Brief Assessment of Cognition in Schizophrenia by examining its temporal stability, internal consistency, and discriminant and convergent validity.

METHODS:

The Brief Assessment of Cognition in Schizophrenia was administered to 116 stable patients with schizophrenia and 58 matched control subjects. To assess concurrent validity, a subset of patients underwent a traditional neuropsychological assessment.

RESULTS:

The patients with schizophrenia performed significantly worse than the controls (p<0.001) on all subtests of the Brief Assessment of Cognition in Schizophrenia and on the total score, which attests to the discriminant validity of the test. The global score of the Brief Assessment of Cognition in Schizophrenia was significantly correlated with all of the subtests and with the global score for the standard battery. The Brief Assessment of Cognition in Schizophrenia also had good test-retest reliability (rho>0.8). The internal consistency of the Brief Assessment of Cognition in Schizophrenia was high (Cronbach's α  ϝ 0.874).

CONCLUSION:

The Brazilian Portuguese version of the Brief Assessment of Cognition in Schizophrenia exhibits good reliability and discriminant and concurrent validity and is a promising tool for easily assessing cognitive impairment in schizophrenia and for comparing the performance of Brazilian patients with that of patients from other countries.

INTRODUCTION

In recent decades, several lines of evidence have shown that cognitive impairment is a core feature of schizophrenia and a major determinant of general functioning in patients with schizophrenia. However, advances in this field have been hampered by the lack of a specific instrument for assessing cognition in schizophrenia that provides reliable and comparable data. Recently, many efforts have helped fill this gap. For example, the MATRICS is a US government multi-site enterprise designed to elaborate and disseminate instruments to assess cognition in schizophrenia for use in drug trials (1,2). Other neuropsychological batteries specifically designed for schizophrenia have been proposed for more general purposes (3–5).

The Brief Assessment of Cognition in Schizophrenia (BACS) is a portable pen-and-paper battery that takes approximately 40 min to administer and reliably assesses the main cognitive domains affected in schizophrenia. The BACS has been translated into and validated in several languages, including French (6), Japanese (7), German (8), Spanish (9), Italian (10) and Persian (11), making it an appropriate instrument for internationally registering and comparing data for both clinical and research purposes.

We translated the BACS from English to Portuguese, and the back-translation was approved by the original author (RK). We also performed a preliminary assessment of the applicability and sensitivity of this Brazilian Portuguese version of the BACS in a small sample of patients (12). The aim of the present paper is to assess the validity and reliability of the BACS by examining its temporal stability, internal consistency, and discriminant and convergent validity.

METHOD

Subjects

One hundred sixteen (116) patients diagnosed with schizophrenia and 58 control subjects were enrolled in this study. The patients were recruited from the outpatient clinic at the Raul Soares Institute, Belo Horizonte, Brazil. The inclusion criteria for patients were as follows: meeting the DSM-IV criteria for schizophrenia, having no history of brain trauma, and not suffering from a current substance use disorder. The psychiatric diagnosis was made according to a structured clinical interview using the MINI-PLUS (13). To assure that patients were stabilized for acute psychotic symptoms, only patients who scored 4 or less on any item of the positive subscale of the Positive and Negative Symptoms Scale (PANSS) were enrolled in the study. Furthermore, the patients needed a PANSS positive subscale score of less than 19 (14). There were no specific medication criteria for inclusion in the patient group. The control subjects were people from the community and hospital staff members. They were required to not have a DSM-IV Axis I disorder based on a structured clinical interview (MINI-PLUS) or any relevant neurological illness.

Instruments

Brazilian BACS

The adaptation of the BACS to Brazilian Portuguese is described elsewhere (12). The constructs measured with the Brazilian version of the BACS, including the tests, procedures, and measures, are listed below in the order of administration.

Verbal memory

The patients were presented with 15 words and then asked to recall as many words as possible. This procedure was repeated five times. There were two lists of words (A and B), one for each version. Measure: The number of words recalled per trial in any order (range: 0–75).

Working memory

Digit sequencing task: The patients were presented with clusters of numbers of increasing length. They were asked to tell the experimenter the numbers in order, from lowest to highest. Measure: The number of correct responses (range: 0–28).

Motor speed

Token motor task: The patients were given 100 plastic tokens and asked to place them two at a time into a container as quickly as possible. A 60-s time limit was imposed. Measures: the number of tokens correctly placed into the container in the first half min, the second half min, and the 1 min total (range: 0–100).

Verbal fluency

Semantic or category fluency: The patients were given 60 s to name as many words as possible within a given category (names of animals).

Phonetic or letter fluency: In two separate trials, the patients were given 60 s to generate as many words as possible that begin with a given letter (F, S). Measure: The number of words generated per trial.

Attention and speed of information processing

Symbol coding: As quickly as possible, the patients wrote the numerals 1–9 as matches to symbols on a response sheet. The patients had up to 90 s to complete the task. Measure: The number of correct numerals (range: 0–110).

Reasoning and problem solving

Tower of London: The patients were simultaneously shown two pictures. Each picture showed three balls of different colors arranged on three pegs, with the balls in a distinct arrangement in each picture. The patients were asked to name the total number of times the balls in one picture needed to be moved to arrange them in the manner shown in the other picture. There were 20 trials. The items were of variable difficulty, with a general tendency for latter items to be more difficult. The test was discontinued if the patients gave five consecutive incorrect responses. If the patients answered correctly for all 20 trials, two additional trials of greater difficulty were administered. There were two alternate forms on versions A and B. Measure: The number of correct responses (range: 0–22).

Standard battery

The standard battery used for the traditional neuropsychological assessment was the same one that was used to validate the BACS in French (6). The battery was composed of widely used tests that are thought to assess the same constructs as the BACS. The tests and their respective constructs are listed below in the order administered: The Rey Auditory-Verbal Learning Test (verbal memory); the Wechsler Adult Intelligence Scale, third edition, Digit Inverse Coding and Digit Symbol-Coding subtests (working memory; attention and speed of information processing); Trail Making A (motor speed), the Controlled Oral Word Association Test; Category Instances (verbal fluency); and the Wisconsin Card Sort Test - 128 card version (reasoning and problem solving). All of the tests included in the concurrent battery had been previously validated in Brazilian Portuguese (15–17).

Design

To assess the discriminant validity of the BACS, all patients and controls were randomly assigned to take either Brazilian BACS version A or version B. To assess the concurrent validity of the BACS, 30 patients were randomly selected from the original sample and completed the concurrent validity battery. This subgroup of patients did not differ from the whole sample with regard to demographic and clinical characteristics. The interval between these two assessments was approximately three weeks. To assess the temporal stability of the BACS, 21 patients also underwent a retest of the BACS. The interval between the two assessments was approximately three months, and the retest always used the alternate form (A or B). None of the patients who took the concurrent test battery participated in the temporal stability assessment. We also examined the internal consistency of the BACS within individual patients.

Statistical analysis

We calculated the mean and standard deviation for each subtest of the Brazilian Portuguese version of the BACS (using a control group, n ϝ 58) and for the standard battery (n ϝ 30) to obtain subtest z-scores, which were calculated as the subtest z-score  ϝ  (subtest score - subtest mean) / (subtest standard deviation). The sum of the respective subtest z-scores of each battery was used as the global score. The variables were processed in a database, and statistical analyses were conducted using R software version 2.15.2 (18). We performed a principal component analysis using the correlation matrix, and the first components of the Brazilian Portuguese version of the BACS (116 patients) and the standard battery (30 patients) were used as global scores. Comparisons between the patient and control groups were performed using independent t-tests. To assess concurrent validity, the global scores of the Brazilian Portuguese version of the BACS and the standard battery were compared using the Spearman correlation. The test-retest analysis was performed using the paired t-test and the Spearman correlation. The internal consistency of the Brazilian Portuguese version of the BACS was assessed using Cronbach's α.

Ethical aspects

This study was approved to the local Ethics Committee and was conducted in accordance with the Code of Ethics of the World Medical Association (Declaration of Helsinki). All of the patients gave their informed consent, and their anonymity was preserved.

RESULTS

The BACS required approximately 40 min for application in both the patients and the controls. This administration time is similar to that of other language versions of BACS (3,6,19) and significantly shorter than that of the standard battery, which required approximately 62 min to administer. No patient or control dropped out of the test during the session, suggesting that the BACS is feasible for administering to Brazilian patients.

Table 1 shows the demographics data and BACS performance of the controls and patients. The BACS global score and the scores for all of the subtests were significantly worse in the patient group compared with the controls.

Table 1.

Comparison of the demographic variables and the performance on the Brazilian Portuguese version of the BACS between patients and controls

	Variable	Patient mean (N ϝ 116)	Control mean (N ϝ 58)	p-value (x2/t-test)
Demographics	Gender – Male (%)	51.7	46.6	0.52
	Age (years)	38.5	39.1	0.78
	Education (years)	8.02	7.62	0.52
BACS subtest	Test duration (min)	40.7	39.5	0.36
	Verbal memory	28.64	39.62	<0.001
	Digit sequencing	10.98	16.79	<0.001
	Token motor	51.00	72.19	<0.001
	Verbal fluency - Semantic	14.04	19.22	<0.001
	Verbal fluency - Phonetic	17.28	25.74	<0.001
	Symbol coding	24.12	40.98	<0.001
	Tower of London	10.10	13.65	<0.001
	Global score †	0.0	2.93	<0.001

^†Global score  ϝ  principal component analysis score (116 patients).

To assess concurrent validity, we compared the global BACS scores with the scores for the standard battery. The global scores of the two batteries were significantly correlated (Spearman's rho ϝ 0.625, p-value ϝ 0.0003, N ϝ 30). The BACS global scores were significantly correlated (p-value<10-10) with each of the test constructs. The results are shown in Table 2.

Table 2.

Correlations of the global scores for the Brazilian Portuguese version of the BACS^† with each construct and with the standard battery global scores.

	Spearman's correlation with BACS global scores†	p- value
BACS subtests (N ϝ 116)
Verbal memory	0.729	<10−10
Digit sequencing	0.761	<10−10
Token motor	0.558	<10−10
Verbal fluency - Semantic	0.742	<10−10
Verbal fluency – Phonetic	0.761	<10−10
Symbol coding	0.797	<10−10
Tower of London	0.739	<10−10
Standard battery global scores‡ (N ϝ 30)	0.625	0.0003

^†BACS global score  ϝ  Principal component analysis score (116 patients).

^‡Standard battery global score  ϝ  Principal component analysis score (30 patients).

In the test-retest analysis (Table 3), all of the BACS items for the first test were significantly correlated (Spearman's rho>0.6, p-value<0.002) with the same items on the second test. The intraclass correlation coefficients (two-way mixed single variables) were higher than 0.7 and were significant (p-value<0.002) for every BACS item and also for the global score. There was no significant change in the mean score of any item (change<0.31 standard deviation, p-value>0.15).

Table 3.

Temporal test-retest analysis.

BACS (Brazilian version)	Test 1 mean (N ϝ 21)	Test 2 mean (N ϝ 21)	p-value (paired t-test)	Spearman's correlation	Intraclass correlation coefficient‡
Verbal memory	25.38	24.62	0.65	0.69*	0.70*
Digit sequencing	10.09	11.43	0.15	0.64*	0.68*
Token motor	43.62	46.76	0.24	0.74*	0.80*
Verbal fluency - Semantic	12.95	13.19	0.69	0.77*	0.80*
Verbal fluency - Phonetic	17.09	16.95	0.90	0.82*	0.84*
Symbol coding	19.81	22.48	0.21	0.84*	0.75*
Tower of London	9.71	9.00	0.38	0.79*	0.79*
Global Score †	-9.83	-9.28	0.48	0.81*	0.85*

^*p-value<0.002

^†Global score  ϝ  Sum of each subtest z-score in relation to the control group.

^‡Two-way mixed single variables

The Cronbach's α of the BACS battery was 0.874, indicating a very good internal consistency. The principal component analysis showed that the first component explained 57.4% of the variance, while the second component explained only 11.3% of the variance. The first component also correlated significantly with each of the test constructs (Spearman's rho>0.55, p-value<10⁻¹⁰). In the standard battery, the first component explained 41.7% of the variance, while the second components explained 17.6% of the variance. These results suggest that a one-factor solution best fits our data.

DISCUSSION

The patients with schizophrenia performed significantly worse than the controls on all subtests of the BACS and on the total score. This result was similar to those of the original English version and other language versions of the test and attests to the discriminant validity of the test. Importantly, unlike the subjects in the English and Spanish studies, our controls did not differ from the patients in terms of age, gender or years of education (3,9).

This study also showed the concurrent validity of the BACS because its global score was significantly correlated with the scores of all subtests and the global score of a standard battery. All of the correlations were high (rho>0.7) except for those of the global score and the motor speed, which showed lower correlations (rho ϝ 0.625 and 0.558, respectively). Similar results for the motor speed subtest were found for our study and the English, French and German versions of the BACS (3,6,8).

The BACS also had good test-retest reliability because its global score showed a very high correlation between the test and retest sessions (rho>0.8). Most of the subtests also showed high correlations (rho>0.7) between test and retest, although working memory and verbal memory showed only moderate (rho>0.6) correlations. These results are similar to those of the original BACS (3) and of all of the versions (German, Japanese, and Persian) that have been tested for temporal stability (7,8,11); together, they confirm that the BACS scores seem to be stable over time.

The high internal consistency of the BACS is comparable to that of the original English and the Japanese and Persian versions (3,7,11).

Our principal component analysis showed that the first component explained 57.4% of the variance. The first component also correlated significantly with each of the test constructs, suggesting that a one-factor solution best fits our data. The same solution was obtained in our pilot study of the BACS (12) and in studies of the Spanish and Persian versions (9,11), while the French version found a two-factor solution (6), and the original test in English and the Japanese version found a three-factor solution (3,7). This variance in the component analysis among these studies may be related to the differences in their samples (9).

This study has some limitations. The sample size used in the test-retest analysis was small, and their results should be interpreted with caution despite the very high correlation observed between test and retest.

The BACS was able to differentiate schizophrenic patients from the control group and had good test-retest reliability, concurrent validity, and internal consistency. Additionally, a representative global score of the test was obtained using principal component analysis, suggesting that the test has a one-factor structure. The BACS seems to be an easy and fast neuropsychological tool for assessing global cognition in Brazilian patients with schizophrenia.