Performance based assessment of functional skills in severe mental illness: Results of a large-scale study in China

Abstract

Performance-based assessments of everyday living skills have been shown to be highly correlated with cognitive functioning in schizophrenia and bipolar disorder, as well as being predictive of deficits in real-world outcomes such as independent living and employment. In this study, we expand our assessments of impairments in everyday living skills to China, evaluating people with schizophrenia, bipolar disorder, and major depression, and comparing their performance to that of healthy controls. Samples of people with schizophrenia (N=272), bipolar disorder (n=61), major depression (n=50), and healthy controls (n=284) were examined with the Chinese version of the UCSD performance-based assessment, brief version (UPSA-B). Performance was compared across the groups and the association between age, gender, educational attainment, marital status, and UPSA-B scores was evaluated. When the performance on the UPSA was compared across the groups, with education as a covariate, significant effects of both diagnosis (F=86.3, p<.001) and education were found (F=228.3, p<.001). Sex and age did not contribute significantly when age and education were considered. Post-hoc comparisons revealed that total UPSA-B scores were lowest in the schizophrenia patients, followed by the patients with major depression. Patients with bipolar disorder did not differ from the healthy comparison subjects on overall performance. Scores for all groups were lower than previously reported in western samples (e.g., HC mean = 64). While diagnostic differences in UPSA-B scores are similar to those previously seen in western samples, the education effect is considerably more substantial. These data suggest that in developing countries educational attainment may be strongly associated with levels of adaptive outcomes and the utilization and interpretation of functional capacity measures be adjusted accordingly.

Introduction

Disability is associated with all major mental illnesses, including major depression, bipolar disorder, and schizophrenia. In schizophrenia, substantial deficits in everyday functioning are common and related to cognitive deficits (Green, et al., 2000). Global deficits as well as impairments in specific cognitive functioning domains have been reported (Dickinson, Ragland, et al., 2008); (Reichenberg & Harvey, 2007); (Leung et al. 2008). These impairments include deficits in episodic memory (Ranganath, Minzenberg, et al., 2008), processing speed (Dickinson, Ramsey, et al., 2007), and executive functioning, including working memory (Reichenberg, et al., 2007); (Lee & Park, 2005). In unipolar depression impairments in memory, visual-motor speed (Austin, Ross, et al., 1992), attentional set shifting, and sustained motor responses have been observed during acute episodes (Purcell, Maruff, et al., 1997), while during remission deficits in measures of visual-motor sequencing, executive function, immediate memory, and attention have been seen (Paradiso, Ohkubo, et al., 1997). In bipolar disorder impairments in verbal memory and executive functioning tasks have been reported in episodes of mania, depression and, euthymia (Martinez-Aran, Vieta, et al., 2004a); (Martinez-Aran, Vieta, et al., 2004b); (Zubieta, Huguelet, et al., 2001).

While there has been an increasing literature on cognitive impairments in these patient populations, fewer studies have focused on functional impairments and disability. Studies of functional impairment in unipolar depression reveal that real-world disability increases during episodes of depression but can return to a reduced level of impairment during remission. In one such study, functioning in everyday activities improved with remission, but social functioning remained impaired (Spijker, Graaf, et al., 2004). Others have observed that functional impairments are both severe and widespread, affecting marital quality, income, occupation, and education even in depressed patients who have sustained a two year remission (Coryell, Scheftner, et al., 1993). Severity of depression and anxiety symptoms (Naismith, Longley, et al., 2007); Spijker, et al., 2004), cumulative history of depression, time spent depressed, psychiatric comorbidities, personality traits, and perceived social support (Rytsala, Melartin, et al., 2006) have all been associated with real-world functional impairments in persons with unipolar depression. Studies of functional impairment in bipolar disorder reveal a similar extent of severe, enduring psychosocial disability (Huxley and Baldessarini, 2007). Predictors of functional impairment include poorer performance on tests of cognitive function (Wingo, Harvey, et al., 2009), total number of mood episodes (Martinez-Aran, et al., 2004a), extent of symptom remission, substance abuse (Dickerson, Origoni, et al., 2010), symptom severity, history of psychiatric hospitalizations, and maternal education (Dickerson, Boronow, et al., 2004).

Recent advances in the assessment of disability in persons with severe mental illness have separated the measurement of the ability to perform everyday adaptive skills (referred to herein as “functional capacity”) from real-world functional outcomes, yielding a more precise set of predictors of outcome (Harvey, et al. 2007). In two studies of large samples of schizophrenia and bipolar patients, deficits on performance-based measures of everyday functioning efficiently predicted real world functional impairments. In the first (Mausbach, Harvey, et al., 2010), performance on a brief version (UPSA-B; (Mausbach, Harvey, et al., 2007) of the UCSD performance-based skills assessment (UPSA; (Patterson, Goldman, et al., 2001) predicted the level of residential independence in the combined sample, with diagnosis (schizophrenia vs. bipolar) adding no incremental predictive power. The second study (Bowie, Depp, et al., 2010) found that performance on the UPSA-B was a strong predictor of real-world outcomes in both patient samples, with minimal differences in the functional, cognitive, and symptomatic determinants of real-world functioning across the two diagnoses. Interestingly, the severity of both schizophrenia and bipolar (mania and depression) symptoms was correlated with real-world functional deficits in a manner independent of performance on the UPSA.

Some evidence has suggested that substantial differences in environmental conditions may have a minimal impact on performance-based measures of functional capacity and cognition. In a cross national study of schizophrenia patients in the United States and Sweden (Harvey, Helldin, et al., 2009a), no differences were found in functional capacity, cognitive performance, and clinician ratings of the ability to perform everyday living skills. In the same sample, residential independence varied significantly between the two countries, suggesting that performance-based measures may be less influenced by environmental factors than real-world outcomes. The interaction between cultural differences and everyday functioning in schizophrenia has been studied previously. Outcome in schizophrenia has been suggested to vary significantly between “lower and middle income” countries when compared to “high income” Western countries (Isaac, Chand, et al., 2007). However, in a study of Chinese patients in Hong Kong, Au et al. (2005) found that when they translated a real-world functioning scale, the St Louis Inventory of Community Living skills into Chinese, the scale was able to discriminate between patients discharged to different levels of community care. Further, in a study in Taiwan Jiang (1999) reported that patients with bipolar disorder had substantial residual disability after clinical recovery, suggesting that the characteristics of disability in bipolar disorder in Chinese patients seems similar to those reported in Western Cultures.

There have been no studies to date on functional capacity measures in schizophrenia patients in China and certainly no studies comparing healthy individuals to multiple samples of patients with different variants of severe mental illness. While the studies cited above suggest that functional disability associated with severe mental illness have similar characteristics, there are possible reasons that functional capacity measures would require come careful consideration in Chinese samples. Individuals with low levels of educational and vocational attainment, even without any evidence of mental illness, would be expected to be less likely to have experience with banking and using the telephone, both of which are central features of the UPSA-B. These differences may impact on performance on functional capacity measures and require this issue requires direct study in order to determine how these measures should be adjusted for use across different countries and cultures.

In this paper we present the results of a study measuring functional capacity in patients with schizophrenia, bipolar disorder, and major depression living in Beijing, China using a translated and culturally adapted version of the UPSA-B. Further, we examined a large sample of healthy individuals in order to understand normative performance on these ability measures in Beijing. We hypothesized that impairments in functional capacity are a central feature of severe mental illness, as indexed by a relative impairment in performance for the patient groups compared to the healthy sample. We also took advantage of the wide variation in educational attainment in this sample to examine whether level of education, as well as other demographic factors, influenced performance on functional capacity measures. As performance-based ability measures are used as outcome measures in clinical treatment studies (Green, Nuechterlein, et al., 2008), understanding their characteristics across different countries and cultures may help inform treatment of schizophrenia and bipolar illness. We expected that education would also have a notable impact on UPSA-B scores and were interested in whether this impact interacted with the presence of serious mental illness.

Method

Participants

Patient participants in this study met criteria for schizophrenia, bipolar disorder, or major depression (DSM-IV). Physically healthy patients who met DSM-IV (American Psychiatric Association, 2000) for schizophrenia, bipolar disorder and major depression were compared with 284 Chinese normal controls. All patients were recruited from among the inpatients of Beijing Hui-Long-Guan Hospital, a Beijing City owned psychiatric hospital. Diagnoses were made for each patient by two independent experienced psychiatrists and utilized structured interviews (a translated version of the SCID; First et al., 1995), access to detailed medical records, and relied on the DSM-IV criteria. Healthy controls were recruited from the local community, and selected for similarity to the patients in age and gender. Current mental status and personal or family history of any mental disorder was assessed by a clinical psychiatrist. None of the healthy control subjects presented a personal or family history of psychiatric disorder. All subjects were Han Chinese being recruited at the same period from the Beijing area.

A complete medical history and physical examination were obtained from all subjects. Any subjects with physical abnormalities were excluded. Neither the patients nor the control subjects suffered from drug or alcohol abuse/dependence. All subjects gave signed, informed consent to participate in the study, which was approved by the Institutional Review Board, Beijing Hui LongGuan hospital. Three hundred patients with schizophrenia, 70 patients with bipolar disorder and 60 patients with major depression were approached on inpatient units at the hospital. As far as agreement to participate, twenty-eight patients (9.3%) with schizophrenia, 9 (12.9%) with bipolar disorder and 10 (16.7%) with major depression declined to take part in the study. Three hundred healthy controls were approached, and 16 (5.3%) declined. Among the healthy controls, 4 (1.3%) had to be excluded due to a psychiatric diagnosis.

Performance Based Measure of Functional Capacity

The UCSD Performance-Based Skills Assessment, Brief Version (UPSA-B) is designed to directly assess everyday functioning in persons with severe mental illness (Mausbach, et al., 2007). The UPSA-B is based on the original UCSD Performance Based Skills Assessment (UPSA) which was designed to assess the capacity of persons with schizophrenia to perform the tasks necessary for daily functioning (Patterson, et al., 2001). The UPSA-B consists of two subscales (financial and communication) from the original UPSA which correspond with total UPSA-B Score. In the financial portion of the skills assessment, the participant must count out given amounts from real currency, make change, and fill out a check to pay a utility bill. The communication domain requires that the participant make emergency calls, call directory assistance to obtain a telephone number, call the number, and then reschedule an appointment. The scores on the UBSA-B are standardized to a 100-point scale, consistent with scoring of the original UPSA, which allows for comparison with previous studies that used the original version of the UPSA.

The Chinese version of the UPSA- B was developed for use in research studies with Han Chinese populations. Tasks were modified in order to be congruent with local functional demands, including changing currency management tasks, telephone numbers, and appointment formats. Further, the check writing task was changed in order to reflect more realistic local demands, which included making a bank withdrawal and requesting a bank check to pay a bill. After the tasks were modified, the instructions were translated into Mandarin by Jin Fan, PhD (Mt. Sinai School of Medicine, New York, NY) and then back translated by one of the authors, XZ and then examined by another of the authors PDH.

The assessments were administered by trained testers who were native Mandarin speakers. Testers did not know the educational levels of the participants and were not informed of the diagnosis of the psychiatric patients in the study.

Demographic Factors

We used categorical ranking of milestone achievements derived from self report, informant, and clinical chart data. These included years of education and completion of educational milestones (primary school or less, middle school, secondary school, college or higher) and marital status (ever married, never married). Other demographic information including age and sex were also collected for use in these analyses.

Data Analyses

A general linear model analysis of variance (SPSS version 17) was used to compare total UPSA-B scores between subject groups: healthy control, schizophrenia, bipolar disorder, and major depression. Regression analyses were used to examine the correlations between the demographic variables of age, education, and marital status on UPSA-B scores. The performance on individual items was expressed as the average correct for the dichotomous items, leading to proportion scores that were examined parametrically. Although we present education categorically in figures for ease of viewing, all analyses were performed with education as a continuous variable.

Results

Demographic characteristics of the study participants are presented by diagnostic category in Table 1. Males were over-represented in the schizophrenia sample (65%) and females over-represented in the depressed sample (65%) when compared to the healthy control group. Across groups schizophrenia patients were the least likely to have ever been married (48%), followed by bipolar patients (43%), depressed patients (18%), and healthy controls (8%). Level of education was lowest in the schizophrenia sample though there was wide variability in level of education in all groups. Internal consistency of the UPSA-B was examined with coefficient alpha in the schizophrenia patients and healthy controls, with these reliabilities being quite suitable: .84 in the schizophrenia sample and .87 in the healthy controls.

Table 1.

Demographic Characteristics and Performance on UPSA-B

	Schizophrenia (n=272)		Bipolar (n=61)		Depressed (n=50)		Healthy Control (n=284)
% Male	65		59		35		44
% Never Married	48		43		14		8
Mean Age (SD)	45.47 (12.4)		38.31 (11.3)		53.35 (14.6)		47.92 (11.6)
Education
% Elementary	7		10		22		15
% Middle School	40		26		42		36
% High School	38		39		18		34
% College or more	15		25		18		15
Mean Education (SD)	9.43 (6.4)		10.21 (4.3)		9.36 (5.4)		10.50 (4.8)
	Mean	SD	Mean	SD	Mean	SD	Mean	SD
UPSA-B Scaled Score	37.59	26.12	55.24	20.35	47.90	22.77	64.31	23.33

Performance on the total score for the UPSA-B was lowest in patients with schizophrenia (see Table 1) followed by patients with major depression. Figure 1 presents UPSA-B Score as a function of categorical educational attainment in patients with schizophrenia, bipolar disorder, major depression, and healthy controls. In a simultaneous entry regression analysis examining the influence of age, education, sex, and marital status on UPSA-B scores, only education entered the equation, F=109.2, p<.001, while the other three demographic variables did not enter the equation, all F< 2.02, all p>.22.

Figure 1.

UPSA-B Scores as a Function of Education Completed

When performance on the UPSA-B was compared across the groups, using education as a covariate, a significant effect of diagnosis (F = 86.3, p < .001) was found and education manifested a substantial covariate effect (F = 228.3, p < .001). The interaction of Diagnosis × Education was not significant, F=2.24, p=0.23. When the analysis was repeated with categorical educational attainment instead of years of education, the results were very similar: F=197.6, p<.001. Post hoc follow-up tests found that patients with schizophrenia and major depression performed significantly more poorly on the total score than healthy control subjects (p<.05). Performance on UPSA-B total score did not differ significantly between patients with bipolar disorder and healthy controls and between bipolar and unipolar patients (but the affective patient samples are smaller than the schizophrenia healthy comparison groups). At all levels of educational attainment, total UPSA-B scores were significantly lower in the schizophrenia patients than in all other groups. While total score on UPSA-B revealed more impairment in everyday functioning among patients with schizophrenia, specific item scores suggested differential sensitivity to deficits in patients with severe mental illness.

In order to examine these differences, we performed a one-way ANOVA for each item, comparing the average scores for each of the items across the four patient groups. We used Scheffe follow-up tests to examine specific differences between the groups and applied the Bonferroni correction to the results of the 19 ANOVAs (.05/19=.002). Table 2 shows the differences in performance across all of the items between the groups of research participants. The overall between-groups analyses revealed statistically significant overall Bonferroni corrected differences in performance of all items on the UPSA-B except for writing the correct date on a bank check. The patients with schizophrenia were more impaired than the healthy controls on all but that one item, indicating that all items were sensitive to schizophrenia collapsed across educational attainment levels when the two largest samples were compared.

Table 2.

Individual Item Performance and between Group Differences on UPSA-B Items

Financial Skill 1: Counting money and making change								P	F
1a: Show me 1.03 in coins	Schizophrenia	=	Bipolar	=	Depression	<	HC	<0.001	21.25
Mean Correct responses	52		52		57		81
1b: Show me 6.74 in coins	Schizophrenia	<	Depression	=	Bipolar	=	HC	<0.001	30.79
Mean Correct responses	59		80		87		90
1c: Show me 12.18 in coins	Schizophrenia	<	Depression	=	Bipolar	=	HC	<0.001	18.74
Mean Correct responses	65		82		84		88
1d: Make change from 10.00	Schizophrenia	=	Depression	<	Bipolar	=	HC	<0.001	24.50
Mean Correct responses	46		43		66		78
Financial Skill 2: Paying a bill
2a: Date check	Schizophrenia	=	HC	=	Depression	=	Bipolar	0.067	2.41
Mean Correct responses	43		51		55		61
2b: Write full name of remitter	Schizophrenia	=	Depression	=	Bipolar	<	HC	<0.001	24.00
Mean Correct responses	37		43		62		70
2c: Write full name of payee	Schizophrenia1	=	Depression	=	HC	=	Bipolar	0.001	5.80
Mean Correct responses	25		29		40		41
2d: Write account number	Schizophrenia1	=	Depression	=	Bipolar	=	HC	<0.001	27.17
Mean Correct responses	43		59		73		77
2e: Write amount in words	Schizophrenia1	=	Depression	=	Bipolar	=	HC	<0.001	20.30
Mean Correct responses	21		35		39		51
2f: Write amount in numbers	Schizophrenia	<	Depression	=	HC	=	Bipolar	<0.001	14.26
Mean Correct responses	46		61		71		72
Communication Skill: Dialing telephone numbers and calling to reschedule an appointment
3a: Dial medical emergency number	Schizophrenia	<	Depression	=	HC	=	Bipolar	<0.001	44.99
Mean Correct responses	41		74		78		81
3b: Call directory assistance to obtain a telephone number	Schizophrenia	<	Bipolar	=	Depression	=	HC	<0.001	8.68
Mean Correct responses	46		59		66		66
3c: Check the telephone number	Schizophrenia	<	Depression	=	HC	=	Bipolar	<0.001	10.84
Mean Correct responses	49		63		73		79
3d: Dial number from memory	Schizophrenia	=	Depression	<	Bipolar	=	HC	<0.001	35.18
Mean Correct responses	45		37		66		81
3e: Find phone number needed to reschedule physical exam	Schizophrenia	=	Depression	=	Bipolar	<	HC	<0.001	22.27
Mean Correct responses	60		65		70		89
3f: Leave accurate message requesting to reschedule exam	Schizophrenia	=	Bipolar	=	Depression	<	HC	<0.001	33.13
Mean Correct responses	35		49		53		70
3g: Recall items to bring to physical exam: insurance card	Schizophrenia	<	Bipolar	=	Depression	=	HC	<0.001	26.70
Mean Correct responses	18		28		31		49
3h: Recall items to bring to physical exam: medication list	Schizophrenia	<	Depression	=	Bipolar	=	HC	<0.001	23.60
Mean Correct responses	21		28		31		49
3i: Recall preparation required: fast for blood work	Schizophrenia	=	Bipolar	=	Depression	<	HC	<0.001	86.57
Mean Correct responses	16		18		21		72

Note.

¹Schizophrenia worse than HC is the only significant difference between groups.

In contrast, patients with affective disorders were more variable in their impairments. There were 7 items on which at least one of the affective samples was more impaired than the HC group, even with the reduced power and conservative nature of the post hoc statistical tests.

Discussion

These results suggest that among patients with severe mental illness living in China, patients with schizophrenia have more significant deficits in everyday living skills than healthy individuals and, in some domains, patients with affective disorders. UPSA-B scores for all groups were lower than previously reported in western samples. For example, in the current study average UPSA-B total score for healthy controls was 64, which is comparable to the total score obtained by schizophrenia patients in recently studied Western samples (Harvey, et al., 2009a). In the Chinese sample, the highest educated subjects had scores consistent with those seen previously in US samples, for both schizophrenia patients (Mean=78) and healthy controls (Mean=83). In addition, in this study of people in China, education exerted a significant influence on functional capacity, an influence that is much more substantial than that observed in a recent study of patients living in the United States and in Sweden. This implies, pending replication, that in developing countries the level of education is a more influential determinant of functional outcome than in western nations. It is also the case that extremely low levels of education are not often found in healthy comparison subjects in US studies. Therefore, cross national approaches to studying and treating severe mental illness should include consideration of patients’ level of educational attainment, because education apparently exerted similar influences on performance in healthy individuals and patients with schizophrenia.

The mechanism of influence of education on functional capacity is likely channeled through opportunity and daily life complexity. Individuals with very low levels of education who reside in developing countries, even in large cities, are unlikely to use telephones, schedule appointments, and have bank accounts. It would not be reasonable to expect that in an assessment situation they would be able to perform skilled acts that they may have never even observed previously. Careful consideration of the extent of prior educational attainment and extent of relevant life experiences may be prudent when implementing such assessments in developing countries, particularly in clinical treatment studies. Given the challenges in implementing multi-site treatment studies under more optimal circumstances (Keefe and Harvey, 2008) performance-based functional outcomes measures may have to be adapted and validated prior to their implementation in such environments.

There are several limitations of the study. These limitations include the smaller sample size of the two affective disorder samples. This is likely the reason that the current bipolar sample did not differ from the healthy comparison group despite performing about 0.5 SD more poorly on the UPSA total score. Further, the item by item analyses were performed on a purely post hoc basis and would require replication. We did not collect detailed information on histories of residential status, so we cannot link these results into independence in residence. It is likely that independence in residential status has different implications in Western as opposed to developing nations and may have substantially different features in regions with larger extended families, more housing challenges, and differences in the cohesiveness of basic family structure. All patients were assessed as inpatients, so these results might be different for ambulatory patients. This status cannot account for the differences we detected between the patient groups. Inpatient treatment is more common in China than in Western Cultures. Thus, clinical treatment studies that use performance-based outcomes measures might also recruit from such treatment sites. We did not have systematically obtained ratings of clinical symptom severity for these patients, so we cannot examine the relationships, if any, between symptoms and functional capacity measures. As we note above, there were no such correlations detected in several different samples of patients in the past. Finally, we do not have clinical ratings of real-world functioning or performance on neuropsychological assessments available for the entire sample. As a result, our focus is entirely on functional capacity and this is the largest sample of healthy controls ever tested with a functional capacity assessment in a mental health study.

These data suggest that performance-based measures of everyday living skills are sensitive to the presence of major psychiatric conditions even when confounded by extreme variation in educational attainment. The effect of education was independent of diagnosis in this data set, suggesting that selection of participants for research could be stratified by education without biasing the performance of the subgroups. Further, the education signal was substantially greater than the influence of age, education, and marital status. At the same time, modification of performance-based task demands to local standards, in both everyday living and social skills domains, would be the optimal way to tailor functional assessments in a cross-cultural perspective. It will be important to separate life-long participant characteristics (i.e., education) from correlates of illness in order to effectively measure treatment effects on cognitive and functional deficits in severe mental illness in developing countries.