Validation of the Chinese version of Brief Assessment of Cognition in Schizophrenia

Abstract

Objective

A test battery that measures cognitive function impairment in patients with schizophrenia, the Brief Assessment of Cognition in Schizophrenia (BACS), has been translated into various languages and validated. This study aimed to test the reliability and validity of the Chinese version of the BACS in a Chinese-speaking population.

Methods

All participants in this study (66 patients with schizophrenia [mean age: 41.2 years, 57.6% male] and 66 age- and sex-matched healthy controls) were from Taiwan and assessed using the BACS and the University of California, San Diego (UCSD) Performance-Based Skills Assessment, Brief Version (UPSA-B). Thirty-eight of the 66 patients with schizophrenia received a reassessment using the BACS.

Results

The BACS had good test–retest reliability, and all BACS subtests had statistically insignificant practice effects. Principal components analysis demonstrated that a one-factor solution best fits our dataset (60.9% of the variance). In both patients and controls, the BACS composite scores were positively correlated with all BACS subscales (P<0.001) and UPSA-B scales (P<0.001). Furthermore, all BACS subtests (verbal memory, working memory, motor speed, verbal fluency, attention and processing speed, and executive function) significantly differentiated patients with schizophrenia from healthy controls (P<0.001), and the BACS composite score had the best discriminative validity (P<0.001).

Conclusion

The Chinese version of the BACS exhibits satisfactory psychometric properties, including high test–retest reliability, high internal consistency, acceptable concurrent validity, and good discriminant validity. We suggest that the BACS is a reliable and practical tool for assessing cognitive function in patients with schizophrenia.

Introduction

Patients with schizophrenia exhibit a wide range of cognitive dysfunctions,¹ including deficits in working memory, verbal and spatial memory, motor and processing speed, attention, and executive function.^2,3 Cognitive impairment determines the level of patients’ daily function and social-occupational outcome.^4,5 Therefore, a cognitive test battery that can be efficiently and feasibly used to assess cognitive deficits in patients with schizophrenia is essential for outcome measurement in both clinical settings and research environments.^6,7 The ideal test battery should contain a broad range of cognitive dimensions, satisfy reliability and validity criteria, and be repeatable to assess changes over long-term follow-up periods.⁸

Numerous neuropsychological testing batteries have been developed to determine the magnitude of cognitive impairment in patients with schizophrenia.^9,10 For example, the well-known Measurement and Treatment Research to Improve Cognition in Schizophrenia is used in clinical trials for assessing cognitive function in patients with schizophrenia.^11,12 However, most of the present cognitive assessment tools are time-consuming, costly, or inconvenient to use in practice. The Brief Assessment of Cognition in Schizophrenia (BACS) is a brief test battery which measures impairment of verbal memory, working memory, motor speed, verbal fluency, attention and processing speed, and executive function.¹³ These domains of cognitive deficits have been demonstrated to reliably reflect the functionality of patients with schizophrenia.^14,15 The BACS is easily performed in clinical settings and can be administered by medical professionals in 40 minutes. The original version of BACS was in English and has undergone a validation study.¹⁶ The BACS has been translated into over 30 languages, and the validities of the Japanese,¹⁷ French,¹⁸ Italian,¹⁹ Spanish,²⁰ German,²¹ Persian,²² and Brazilian Portuguese²³ versions have been proven.

The BACS has been adapted into Chinese version by the original authors. The normative data of the Chinese version of the BACS have been established using a representative community sample in Taiwan (unpublished data), and notable differences in verbal tests and executive function were found when compared to the norms developed in the US.¹³ However, the BACS has not been validated in a Chinese-speaking population. Since Chinese is one of the most commonly spoken languages in the world, adaptation and validation of the BACS could advance the knowledge about cognition in schizophrenia, facilitate the comparison of patients with schizophrenia in various countries, and help to determine the cross-cultural validity of the BACS. Therefore, the current study aimed to validate the BACS by comparing 66 patients with schizophrenia to an age- and sex-matched sample of healthy controls.

Methods

Study participants

This cross-sectional study was conducted in Chang Gung Memorial Hospital, and was approved by the Institutional Review Board of Chang Gung Hospital in Taiwan. A written informed consent was obtained from all the participants. Eligible patients with schizophrenia at an outpatient department in the Kaohsiung Chang Gung Memorial Hospital were selected for this study if they met the following inclusion criteria: 1) were aged 18–65 years; 2) were diagnosed with schizophrenia as defined by the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition (DSM-IV-TR)²⁴ after assessment using the Chinese version of the Mini International Neuropsychiatric Interview (MINI) (which was translated from English by the Taiwan Society of Psychiatry²⁵ and known to have good inter-rater reliability);²⁶ 3) had received a stable dose of an antipsychotic drug and had relatively stable psychotic symptoms; 4) had no history of major physical illnesses that would influence cognitive performance, and 5) could speak and read Chinese to perform the test. In total, 66 patients with schizophrenia were recruited in this study (mean age: 41.2 years, 57.6% male). Information regarding the antipsychotic usage by patients and daily dose were obtained through patients’ medical records. The dose of antipsychotic medication in use was calculated according to the defined daily dose (DDD) recommended by the WHO Collaborating Centre for Drug Statistics Methodology.²⁷

The healthy individuals were recruited from the staff of Kaohsiung Chang Gung Memorial Hospital and Keelung Chang Gung Memorial Hospital, and from the community volunteers in Kaohsiung City and Keelung City, Taiwan. The recruitment criteria were as follows: 1) absence of a history of major psychiatric disorders (eg, psychotic disorders, bipolar disorders, major depressive disorders, organic mental disorders, and substance user disorders) confirmed using DSM-IV criteria and the MINI; 2) absence of any known systemic or neurological diseases that would influence cognitive performance; 3) Han Chinese ethnicity; and 4) capacity to speak and read Chinese to perform the test. We recruited 382 healthy individuals to establish the norms of the BACS (unpublished data). Finally, 66 healthy individuals who were age- and sex-matched to our patients with schizophrenia were selected as the control group.

Cognitive assessment

The BACS

The cognitive functions of all participants were assessed using the BACS. A total of 38 patients with schizophrenia were willing to receive BACS retesting, and the rest 28 patients either dropped out or refused to receive the retest. The BACS test battery is used for neuropsychological assessment of patients with various psychotic disorders,²⁶ and the subtests of the BACS are listed in the following section.

List learning test (verbal memory): patients are presented with 15 phrases and then asked to recall as many as possible. This procedure is repeated five times. The outcome measure is the total number of phrases recalled. Eight alternative forms are designed to minimize the practice effect. In the current study, the first form was chosen for testing all participants, and the second form was chosen for retest purposes.

Digit sequencing task (working memory): patients are presented with clusters of numbers of increasing length and are asked to repeat the numbers in order, from lowest to highest. The trials are of increasing difficulty. The outcome measure is the total number of correct items.

Token motor task (motor speed): patients are given 100 plastic tokens and asked to place them into a container as quickly as possible for 60 seconds. The outcome measure is the total number of tokens placed in the container.

Verbal fluency: performance is evaluated based on the total number of words/phrases adequately responded to in the Category Instances Test and Controlled Oral Word Association Test. In the Category Instances Test (semantic fluency), patients are given 60 seconds to list as many words as possible relating to the category of animals. The outcome measure is the total number of unique words listed for animals. In the Controlled Oral Word Association Test (letter fluency), patients are given 60 seconds to list as many “phrases” as possible that begin with the Mandarin word “Kou” and “Jing” in two separate trials. The outcome measure is the total number of unique phrases listed.

Symbol coding test (attention and processing speed): as quickly as possible, patients are asked to write numerals 1–9 that matched the symbols on a response sheet for 90 seconds. The outcome measure is the total number of correct responses.

The Tower of London test (executive function): patients are asked to look at two pictures simultaneously. Each picture shows three differently colored balls arranged on three pegs, with the balls in a unique arrangement in each picture. The patients are asked to calculate the smallest number of times that the balls in one picture would have to be moved to arrive at an arrangement of balls as seen in another picture. The outcome measure is the number of trials needed to find the correct solution. There are two alternative forms. In the current study, the first form was used for testing all participants, and the second form was used for retesting.

Finally, a composite score was calculated by summing scores on each measure for each patient and healthy control. The Z-score compared the sum of each patient with that of the corresponding control.¹³ A Z-score of 0 for each scale indicates average functioning with reference to the normal population of the same age range and sex, and every 1 point represents one standard deviation (SD).

The UCSD Performance-Based Skills Assessment, Brief Version (UPSA-B)

The UPSA-B is modified from the UCSD Performance-Based Skills Assessment (UPSA) which is designed for assessing daily functioning in patients with severe mental illness.²⁸ The UPSA-B consists of two subscales: a financial portion and communication domain. In the financial portion, the participant would be asked to count specific amounts of real currency, make change, and request a bank check to pay a bill. The communication domain requires the participant to correctly call directory assistance to obtain a telephone number and then reschedule an appointment in a hospital. The raw scores of the UBSA-B were standardized to a 100-point scale.²⁹ The Chinese version of the UPSA-B was developed and has been validated in patients with mental illnesses among Chinese populations.³⁰ The UPSA-B served as the validity criterion for the BACS in this study.

Statistical analyses

Data were analyzed using the statistical software package SPSS, version 21.0 (SPSS Inc., Chicago, IL, USA). The data are presented as either mean (±SD) or frequency (%). Categorical variables were compared between patients and controls using either the chi-square test or Fisher’s exact test, depending on case numbers, and an independent t-test was used to compare continuous variables between groups. In a two-tailed test, P<0.05 was considered statistically significant.

The test–retest reliability of all subtests of the BACS was measured with intraclass correlations (ICC). The paired samples t-test was used to examine whether there were significant practice effects. The principal components analysis (PCA) with varimax rotation was applied for determining the factor structure of the BACS subtests. Pearson’s correlation was performed to analyze the relationships between each BACS subscale and the correlation between BACS composite scores and UPSA-B scale scores, separately in patients and controls.

Scaled Z-scores for a subset of variables and a composite Z-score were calculated based on the normative data of the BACS established in our previous work (unpublished data). The receiver operating characteristic (ROC) analyses and area under curve (AUC) were further applied to examine the validity of using the Z-score of each BACS subtest to differentiate patients with schizophrenia from healthy controls. Sensitivity and specificity of the best cutoff point of the BACS composite score were calculated.

A Kruskal–Wallis test was performed to examine the potential difference in BACS composite scores among schizophrenia patients with different antipsychotic use. Pearson’s correlation test was performed to analyze the correlation between DDD of antipsychotics and BACS composite scores.

Results

A total of 66 patients with schizophrenia (schizophrenia group) and 66 age- and sex-matched healthy control subjects (control group) were recruited in this study. The demographic characteristics and BACS performance of the groups are listed in Table 1. A higher proportion of the control group were married (P<0.001) and employed (P<0.001). Compared with the control group, patients in the schizophrenia group were impaired in all dimensions of cognitive function assessed by the BACS and UPSA-B (P<0.001).

Table 1.

Demographic characteristics of patients with schizophrenia and healthy control subjects

Variables	Patients (N=66)		Controls (N=66)		Statistical value
	N or mean	% or SD	N or mean	% or SD	χ2 or t	P-value
Age (years)	41.24	10.06	41.24	10.09	0.000	1.000
Sex	–	–	–	–	0.000	1.000
Male	38	57.6	38	57.6	–	–
Female	28	42.4	28	42.4	–	–
Marital status	–	–	–	–	60.045	<0.001*
Married	6	9.1	50	75.8	–	–
Not married	60	90.9	16	24.2	–	–
Occupation	–	–	–	–	46.964	<0.001*
Employed	18	27.3	57	86.4	–	–
Unemployed	48	72.7	9	13.6	–	–
Education (years)	12.50	3.26	14.88	2.68	−4.654	<0.001*
BACS performance	–	–	–	–	–	–
Verbal memory	26.21	11.09	41.36	9.50	−8.430	<0.001*
Working memory	15.62	4.59	23.17	9.70	−5.712	<0.001*
Motor speed	57.27	13.13	81.38	12.52	−10.754	<0.001*
Verbal fluency	23.97	7.41	34.35	7.68	−7.903	<0.001*
Attention and processing speed	38.45	13.53	61.35	12.67	−10.037	<0.001*
Executive function	12.32	5.85	17.05	3.37	−5.688	<0.001*
Composite score (Z-score)	−1.94	1.10	0.02	0.72	−12.088	<0.001*

Note:

^*P<0.001.

Abbreviations: BACS, Brief Assessment of Cognition in Schizophrenia; SD, standard deviation.

The test–retest reliability and potential practice effects of the BACS are summarized in Table 2. In the 38 patients with schizophrenia who were tested and retested, the ICCs of the BACS subtests ranged from 0.59 (verbal fluency) to 0.86 (attention and processing speed), and all were significant (P<0.001). The paired samples t-test revealed that none of the BACS subtests showed statistically significant practice effects.

Table 2.

Test and retest reliability of the BACS among 38 patients with schizophrenia

Subtest of the BACS	Test session 1		Test session 2		Correlation		Paired t-test
	Mean	SD	Mean	SD	ICC	P-value	t	P-value
Verbal memory	28.47	10.54	29.63	10.39	0.735	<0.001*	−0.937	0.355
Working memory	16.71	4.69	17.37	5.13	0.673	<0.001*	−1.020	0.314
Motor speed	56.95	11.23	58.21	15.68	0.699	<0.001*	−0.694	0.492
Verbal fluency	24.95	7.30	26.08	7.64	0.592	<0.001*	−1.033	0.308
Attention and processing speed	42.03	11.32	41.21	11.56	0.862	<0.001*	0.837	0.408
Executive function	13.55	5.30	13.61	6.25	0.720	<0.001*	−0.074	0.942

Note:

^*P<0.001.

Abbreviations: BACS, Brief Assessment of Cognition in Schizophrenia; ICC, intraclass correlations; SD, standard deviation.

PCA demonstrated that the first and second components accounted for 60.9% (eigenvalue: 3.66) and 14.9% (eigenvalue: 0.90) of the variance, respectively. This suggests that a one-factor solution is the best fit for our dataset.

Table 3 shows the correlation matrix of each BACS subtest of patients and controls. In patients with schizophrenia, the BACS composite score was positively correlated with all BACS subtest scores (P<0.001), and all BACS subtest scores were positively correlated with each other (P<0.05), with the exception of the correlation between working memory and motor speed. The composite score was significantly correlated with all subtest scores in controls (P<0.01) and with UPSA-B scores (Figure 1) in patients (r=0.70, P<0.001) and controls (r=0.44, P<0.001).

Table 3.

Correlation matrix of subtests of the BACS in patients with schizophrenia and healthy control subjects

Subtest of the BACS	VM	WM	MS	VF	APS	EF	CS
Verbal memory	–	0.075	0.246*	0.208	0.262*	0.230	0.627***
Working memory	0.617***	–	−0.489***	0.250*	0.137	0.045	0.324**
Motor speed	0.337**	0.112	–	−0.070	0.213	0.276*	0.351**
Verbal fluency	0.579***	0.521***	0.285*	–	0.366**	0.132	0.583***
Attention and processing speed	0.781***	0.623***	0.433***	0.498***	–	0.390**	0.732***
Executive function	0.606***	0.601***	0.343**	0.351**	0.630***	–	0.645***
Composite score	0.860***	0.713***	0.566***	0.671***	0.869***	0.830***	–

Notes: Correlations in patients with schizophrenia below the diagonal; correlations in healthy control subjects above the diagonal;

^*P<0.05,

^**P<0.01,

^***P<0.001.

Abbreviations: APS, attention and processing speed; BACS, Brief Assessment of Cognition in Schizophrenia; CS, composite score; EF, executive function; MS, motor speed; VF, verbal fluency; VM, verbal memory; WM, working memory.

Figure 1.

Scatter plot of the composite scores of the BACS and the UPSA-B scaled scores for patients with schizophrenia and healthy controls (patients: r=0.70, P<0.001; healthy controls: r=0.44, P<0.001).

Abbreviations: BACS, Brief Assessment of Cognition in Schizophrenia; UPSA-B, UCSD Performance-Based Skills Assessment, brief version.

The ROC analysis (Figure 2) showed that all BACS subtest scores could significantly differentiate patients with schizophrenia from healthy controls (P<0.001). The BACS composite score had the best discriminant validity (AUC: 0.94, P<0.001), followed by the scores for motor speed (AUC: 0.91, P<0.001), attention and processing speed (AUC: 0.90, P<0.001), working memory (AUC: 0.88, P<0.001), verbal memory (AUC: 0.84, P<0.001), verbal fluency (AUC: 0.84, P<0.001), and executive function (AUC: 0.77, P<0.001), respectively (Table 4). The ROC output indicated that a composite Z-score of −0.71 had the best discriminant validity, with a sensitivity of 0.91 and a specificity of 0.89.

Figure 2.

The receiver operating characteristic analysis for subtests of the Chinese version of the BACS in discriminating patients with schizophrenia and healthy controls.

Abbreviation: BACS, Brief Assessment of Cognition in Schizophrenia.

Table 4.

The discrimination validity of cognitive performance measured by the BACS in patients with schizophrenia and healthy controls, using the ROC analyses

Subtest of the BACS	AUC	95% CI	Standard error	P-value
Verbal memory	0.844	0.778–0.909	0.033	<0.001*
Working memory	0.877	0.816–0.938	0.031	<0.001*
Motor speed	0.912	0.857–0.967	0.028	<0.001*
Verbal fluency	0.836	0.768–0.904	0.035	<0.001*
Attention and processing speed	0.903	0.853–0.952	0.025	<0.001*
Executive function	0.765	0.683–0.847	0.042	<0.001*
Composite score	0.942	0.904–0.980	0.019	<0.001*

Note:

^*P<0.001.

Abbreviations: AUC, area under curve; BACS, Brief Assessment of Cognition in Schizophrenia; CI, confidence interval; ROC, receiver operating characteristic.

Among patients with schizophrenia, no significant differences in BACS composite scores were observed between different antipsychotic users (P=0.066). The BACS composite scores were not significantly correlated to the DDD of antipsychotics (r=−0.07, P=0.561).

Discussion

We found that all BACS subtests in patients with schizophrenia have good test–retest reliability, represented by ICC. The practice effects in all BACS domains estimated by paired samples t-test were nonsignificant. Consistent with previous findings regarding the validation of BACS in other language versions,^16,17,21,23 we suggest that the Chinese version of the BACS is a reliable assessment tool for understanding the cognitive function of patients with schizophrenia. Keefe et al¹⁶ suggested that forms of the list learning test and Tower of London test need to be changed to retest patients and assess changes in their cognitive abilities independent of practice effects (such as memorization resulting from repeated test taking). Therefore, all patients were given an alternative form of the list learning test and Tower of London test at the time of retesting in our study. Notably, the category instances test (verbal fluency) differs between the English and the Chinese versions of the BACS. There are two alternative forms of the category instances test (naming supermarket items or tools) in the English BACS and only one form (naming animals) in the Chinese BACS. Therefore, no alternative form of the verbal fluency test could be used in the current study.

The PCA performed in our study (similar to previous studies of the Spanish,²⁰ Persian,²² and Portuguese versions of the BACS)²³ revealed that one factor accounted for 60.9% of the total variance. Nevertheless, a significantly better model fit was also associated with a three-factor solution in the original English BACS¹⁶ and Japanese BACS,¹⁷ and a two-factor solution in the French BACS.¹⁸ The components extracted by PCA might be different between studies because of discrepancies in sample characteristics, sample size, language or culture, and clinical features.^31,32 Nevertheless, all aforementioned studies consistently indicated that a single factor could explain a large amount of the variance (general cognitive function). Taken together, these findings indicate that the BACS can be used to assess specific cognitive domains, as well as general cognitive function in schizophrenia.

In agreement with the previous validation studies,^16,20–22 our study showed strong correlation between BACS composite scores and all BACS subscales, in both patients and controls and between BACS composite scores and UPSA-B scores. Previous researchers also found strong correlation between the BACS composite score and standard neurocognitive assessments other than the BACS.^{16–18,20,22,23,33} These findings indicate that the BACS has satisfactory intercorrelations and concurrent validity. However, the magnitude of our correlation coefficients between BACS subtests (Table 3), as well as between BACS and UPSA-B scores (Figure 1), was lower in healthy controls than in patients with schizophrenia. These results may be explained by the global nature of cognitive decline in patients but not healthy controls, giving rise to strong correlation among specific domains of cognitive function. In addition, the UPSA-B score is a measure of basic real-world functioning in patients with mental illnesses,²⁹ and an obvious ceiling effect may be present in healthy individuals without cognitive impairments.

The ROC analysis (Figure 2) showed the high sensitivity and specificity of the BACS composite score to distinguish patients from controls (AUC =0.94). This finding is in line with German BACS²¹ and Persian BACS findings.²² This result suggests that the Chinese BACS has good discriminant validity for distinguishing patients from controls based on cognitive function. Among the BACS subtests, motor speed and attention and processing speed also individually had a high discriminant validity (AUC >0.9), while executive function, measured by the Tower of London test, had a relatively low discriminant validity (AUC =0.77). Likewise, Keefe et al¹⁶ previously found that discriminant validity between patients and controls was smaller for executive function than for other BACS subtests. This implies that the Tower of London test is less sensitive to cognitive deficit detection in schizophrenia. Furthermore, cognitive function has been associated with clinical symptoms or prescription of psychotropic drugs in patients with schizophrenia.^34–36 The large deficits in motor speed and attention of schizophrenia patients may possibly be due to disease course or antipsychotic drug therapy.

This study has several limitations: first, the sample size in this study was small. Further study with a larger sample size is warranted to examine the reliability and validity of the BACS across different clinical groups (eg, paranoid type vs disorganized type; first-episode psychosis vs chronic schizophrenia). Second, only patients with schizophrenia, but not healthy control subjects, were given BACS retests. In addition, the second form of the list learning test was constantly used for BACS retesting. Therefore, we could not determine ICCs and practice effects in healthy individuals, nor compare test–retest reliability across different versions of the list learning test. Third, this study used only one cognitive test (UPSA-B) as the validity criterion for the BACS. A standard battery with less ceiling effect is needed to certify the concurrent validity of the BACS in future study. Finally, Chinese contains a group of related language varieties, and the participants in this study were tested in Mandarin (the most widely used dialect in Taiwan and People’s Republic of China). It remains uncertain whether the findings in this study could be generalized to population who speak other dialects.

Conclusion

In summary, the Chinese version of the BACS provides satisfactory psychometric properties, including high test–retest reliability, high internal consistency, acceptable concurrent validity, and good overall and domain-specific discriminant validity. We suggest that the Chinese BACS is a reliable and practical tool for assessing cognitive function in patients with schizophrenia.