Performance-Based Measurement of Functional Disability in Schizophrenia: A Cross-National Study in the United States and Sweden

Abstract

Background

Recent advances in the assessment of disability in schizophrenia have separated the measurement of functional capacity (the ability to perform everyday functioning skills in structured assessments) from real-world functional outcomes. This study examined the similarity of performance-based assessments of everyday functioning, real-world disability, and achievement of milestones in people with schizophrenia in the United States and in Sweden.

Methods

Samples of schizophrenia patients living in rural areas in Sweden (n=146) and in New York (n=244)) performed the brief version of the UCSD Performance-based Skills Assessment (UPSA-B) and a neuropsychological assessment and were rated for functioning by their case managers. Information from records and case managers was used to determine the frequency of living independently, working, and having ever experienced a stable romantic relationship.

Results

Performance on the UPSA-B was essentially identical in the two samples (New York, M=13.84; Sweden, M=13.30). So were scores on the case manager ratings of everyday activities (New York: M=49.0; Sweden: M=48.8). The correlations between UPSA-B scores, NP test performance, and SLOF ratings did not differ across the two samples. The proportion of cases who had never had a close relationship and rates of vocational disability were also nearly identical. In contrast, 80% of the Swedish patients and 46% of the New York patients were living independently.

Conclusions

Scores on performance-based measures of everyday living skills were very similar in people with schizophrenia across cultures. In contrast, real-world residential outcomes were very different. These data suggest that cultural and social support systems can lead to divergent real-world outcomes in individuals who have evidence of the same levels of ability and potential.

Disability in multiple everyday domains is present in people with severe mental illnesses including schizophrenia. Recent studies on real-world disability have focused on the relationships between impaired performance on neuropsychological (NP) tests and functional skills (or functional capacity: FC)^1-3. FC measures are standardized tests of everyday activities such as financial management and medication management; they are clearly face valid predictors of everyday functioning but other environmental and experiential factors may cause discrepancies between what people can do and what they actually do in their everyday lives. For instance, a person with schizophrenia may be capable of independent medication management but may not have the opportunity to do so because of rules and operating procedures at his board and care facility.

The results of studies examining the correlation between FC and NP measures have been remarkably consistent in finding substantial correlations between these two domains of functioning^3-7. Correlations between these same two sets of measures and various real-world outcomes have been smaller and more variable, with the studies rating real-world outcomes with self-report methods generally leading to smaller correlations.

Despite the relationship between NP, FC, and real-world outcomes, in most studies only a minority of variance in real-world outcomes is accounted for by these variables. Even patients with normal-range neuropsychological functioning have substantial disability in certain domains of everyday functioning⁸. Several other variables clearly mediate the relationships between skills competence and performance in real-world settings, including the motivation to engage in real-world functional activities, other symptoms such as depression and negative symptoms^9-10, and other skills such as social cognitive abilities¹¹. Perhaps the most important of these influences is the societal context, which includes health insurance, disability policies, and cultural attitudes toward disability and mental illness. In the large CATIE study, the most potent predictor of current unemployment in individuals with schizophrenia was the receipt of disability compensation¹², which is likely correlated with receipt of health insurance as well. This is a finding consistent with previous research.¹³

Societal social service plans are quite divergent across countries, with some countries having national health insurance and others having much more haphazard systems. It has been found that prevalence of schizophrenia and the distributions of classical schizophrenia symptoms are quite consistent across countries.¹⁴ In addition, cross-national studies of cognitive performance have reported substantial similarity across different western countries.¹⁵ However, since elements of everyday functional disability are multiply determined, it may the case that these influences are different across countries. In line with this hypothesis, we previously reported that several elements of disability in everyday skills (i.e., social and self-care functions) were differentially impaired across institutionalized long-stay schizophrenia patients in the US and the UK¹⁶. Interestingly, cognitive impairment was consistent in severity across the patients in the two countries and, despite the severity differences in disability, the correlations between cognitive impairments and the severity of the different aspects of disability were essentially the same across countries. We suggested that environmental differences in the long-stay hospital system drove variations in the topography of impairments in everyday skills across the two countries and systems of care.

In this paper, we present the results of a cross-national study conducted in the US and in Sweden, examining the severity of impairments in performance of structured examinations of everyday living skills (i.e., functional capacity), the level of observed disability in everyday living, performance on neuropsychological (NP) tests, and real world functional milestones (e.g., independent living) and their correlations. We created a translated version of a functional capacity measure, the UCSD Performance-based skills assessment, brief version (UPSA-B¹⁷), and a translated version of a real-world outcomes clinical rating scale, the Specific Levels of Functioning (SLOF¹⁸) for use in patients in Sweden. Patients in both countries performed the functional capacity measure and their case-managers rated their functioning in several domains, including residential, vocational, and social functioning. Real-world functioning milestones, including independence in living situation, relationship history, and current employment status, were all collected from records, interviews, and case manager reports. A Of additional interest was the fact that the Clinical Long-term Investigation of Psychosis in Sweden (CLIPS) sample was from a largely rural area (Trollhättan) and the American sample was collected in New York City and its immediate suburbs. This is, to our knowledge, the first study that directly compared functional capacity, clinical ratings of disability, and functional outcomes milestones across two different countries whose health care systems and levels of social support for people with mental illness are very divergent.. Thus, this study provides a wide-ranging test of the cross-cultural generalizability and utility of Functional Capacity assessments.

Methods

Participants

Participants in this study were older, ambulatory schizophrenia patients. Exclusion criteria for this study included: a primary DSM-IV Axis I diagnosis other than schizophrenia or schizoaffective disorder; Mini-Mental Status Examination score below 18; or any medical illnesses that might interfere with the assessment. All American subjects were in outpatient treatment at the time of recruitment at a VA, New York state, or academic research site. Patients in Sweden were also ambulatory and were receiving care from the county council funded outpatient clinics at NU-Health Care Hospital. Outpatient status was defined as living outside of any institutional setting, including a nursing home. All subjects signed a written informed consent form approved by the institutional review board at each research site after the testing procedures were fully explained.

Patients in both samples were excluded if they had a history of head trauma with loss of consciousness, active substance abuse or a lifetime history of substance dependence, and any disease of the central nervous system, including a history of stroke, degenerative dementia such as Alzheimer's Disease, or Parkinson's disease.

All subjects met diagnostic criteria for schizophrenia or schizoaffective disorder (DSM-IV). For the US patients, the Comprehensive Assessment of Symptoms and History (CASH¹⁹) was completed by a trained research assistant and diagnosis was confirmed with a senior clinician. For the Swedish patients their diagnoses were generated according to the DSM-IV diagnostic criteria and the Decision Trees for Differential Diagnoses by their psychiatrists. Data for these analyses came only from patients who were receiving case management services and actively involved in psychiatric rehabilitation services. The case managers were used as the informants for the real-world functional status ratings. All patients were receiving treatment with antipsychotic medications, either first or second generation, at the time of their assessments.

Performance-based Measure of Functional Capacity

The UCSD Performance-Based Skills Assessment Battery (UPSA²⁰) is designed to directly assess functional skills competence among the severely mentally ill. This test was designed for outpatients and measures performance in a number of domains of everyday functioning through the use of props and standardized skills performance situations. In this study, the UPSA-B¹⁷ was used, which contains two of the original UPSA domains, based on two recent studies^17,21 suggesting that these two subscales alone correspond excellently with the total score. In the Finance domain, the patient must count out given amounts from real currency, make change and fill out a check to pay a utility bill. The Communication domain involves a series of role-play situations that require the patient make emergency calls, call directory assistance to request a telephone number, call the number, and then reschedule a medical appointment. We standardized the scores to a 100-point scale, like the original 5-subtest UPSA, thus allowing comparisons to previous results. This total score was used as our dependent measure. In Sweden occupational therapists performed the assessment.

Real-world Functional Outcomes

In order to examine everyday functioning in the real world, the Specific Level of Function Scale (SLOF) was employed. This scale is a 43-item observer-rated report of a patient's behavior and functioning on the following domains: Physical Functioning (e.g., vision, hearing), Personal Care (e.g., toileting, eating, grooming), Interpersonal Relationships (e.g., initiating, accepting and maintaining social contacts; effectively communicating), Socially Acceptability (e.g., verbal and physical abuse, repetitive behaviors), Participation in Community Activities (e.g., shopping, using telephone, paying bills, use of leisure time, use of public transportation), and Work Skills (e.g., follows verbal instructions, completes tasks with minimal supervision, completes tasks, punctuality). Ratings are made on a 5-point Likert scale by the third-party informant on the basis of the amount of assistance that the patient requires to perform real-world skills (Personal Care, Activities) effects on daily living (Physical Functioning), or frequency of the behavior (Interpersonal Relationships, Social Acceptability, Work Skills). For all subjects in this study, a case manager for the patient completed the SLOF in order to obtain information on real-world performance.. In both samples, the case managers who generated these ratings were unaware of the performance of the patients on cognitive and functional capacity measures.

All case managers indicated that they knew the patient at least “very well” (4 or more) on the SLOF's 5-point Likert scale. The has been previously shown to be related to NP performance and scores on functional capacity measures^5,8,9. The testers and interviewers who completed and scored all other aspects of assessment were unaware of the case managers’ SLOF ratings. Higher scores on the SLOF reflect less impairment/more independence.

Real-World Milestones

In addition to the dimensional approach to rating real world behavior with the SLOF, we used a categorical ranking of milestone achievements, derived from a combination of self-report, case manager, and chart data that we had employed in a previous study⁸. Independent living status was the product of whether the patient lived in restrictive or supported housing, as well as whether the patient was at least partially financially responsible for the residence. Thus, patients were classified as living in restricted housing, living independently but not financially supporting the residence, or living independently and financially supporting the residence. Current work status was classified as unemployed or employed at least part time.. Marital status was classified as married or widowed, divorced or separated, or never married.

Translation of UPSA-B and SLOF

The English version of the UPSA-B requires manipulation of currency, paying bills, and communication, including emergency communication. In order to have the Swedish version be as similar as possible, all currency amounts were retained and expressed in terms of Swedish Kronor, which is also a decimal currency system. The bill for payment was modified to resemble a local Swedish Bill and emergency communication items were also modified to be congruent with the local requirements.

Instructions and scoring criteria for the UPSA-B and SLOF were translated into Swedish. For the UPSA-B two of the Swedish authors performed the translation together with a professional translator and in contact with Christopher Bowie. For the SLOF, the same two authors and the professional translator completed the translation. The two instruments were then back-translated and checked by the Swedish authors to ensure comparability with the original instruments.

Neuropsychological Assessment

Both samples were examined with a neuropsychological assessment, which was somewhat different in each country. The tests that overlapped were: Trail making test parts A and B²², Letter Number sequencing from the Wechsler Adult Intelligence Scale (WAIS)²³, the Rey Auditory Verbal Learning Test²⁴, the Wisconsin Card Sorting Test 128 card version²⁵, and WAIS vocabulary. The Swedish versions were previously translated and in clinical use in Sweden. All raw scores on the NP tests were converted to age, education, and gender corrected standardized T-scores from published American norms. We used American norms because of the standard practice in the Nordic Countries of using American norms for clinical assessment (Personal Communication from Erik Hessen to Robert K. Heaton, November, 2008). We created a composite score for the NP measures by averaging these t-scores across the tests that were administered to both samples. Since these t-scores are demographically corrected, we did not perform analysis of covariance with demographic factors.

Sample Overlap

This is the same sample of American research participants who were reported on by Leung et al⁸. For that study, we reported all analyses of real-world outcomes and performance-based measures with patients subdivided as a function of whether or not they met criteria for normal neuropsychological functioning and symptomatic remission. As a part of the CLIPS study all Swedish patients were collected specifically to be assessed with the functional capacity and real world outcomes assessment for this research project.

Results

Demographic characteristics of the sample, including vocabulary performance are presented in Table 1. As would be expected in a sample from Sweden, 100% of the Swedish patients were Caucasian, while the American sample was more ethnically diverse. Swedish patients were somewhat younger, slightly better educated, and more likely to be female compared to the American sample.

Table 1.

Characteristics of Swedish and U.S. Samples of Schizophrenia Patients in a Study of the Measurement of Functional Disability

	Country
Characteristic	Sweden (N=146)		United States (N=244)
	Mean	SD	Mean	SD
Age (years)	48.35	11.90	55.77	9.38
Years of education	13.60	3.44	12.57	3.22
WAIS-III vocabulary subtest	39.44	11.55	34.34	14.89

	N	%	N	%
Ethnicity
Caucasian	146	100	129	53
African origin	0	0	71	29
Native American	0	0	5	2
More than one race	0	0	15	6
Unknown or unavailable	0	0	24	10
Male	92	63	178	73

Performance on the UPSA-B, the neuropsychological tests, and the clinical ratings on the SLOF are presented in table 2. As can be seen in the table, t-tests found that UPSA-B scores in the two countries were essentially identical, on both a raw score basis and when the scores were converted into the UPSA-B's 100-point scale. Further, impairments in physical functioning, acceptable behavior, and everyday activities were also essentially identical across the two countries, while the American patients were rated by their case managers as less impaired in basic ADLs, social activities, and work-related activities. In contrast, NP performance was poorer on some of the tests in the American patients, with the composite t score suggesting significantly poorer performance on the part of the American patients

Table 2.

Performance on the UCSD Performance-Based Skills Assessment-Brief Version (UPSA-B), Neuropsychological Assessments, and Ratings of Real-World Functional Outcomes in Swedish and U.S. Samples of Schizophrenia Patients

	Country
	Sweden		United States		Analyses
Measure	Mean	SD	Mean	SD	t	p	Cohen's d
UPSA-B (Sweden, N=146; U.S., N=244)
Finances, raw subscore	7.73	2.52	7.47	3.59	0.77	0.40	0.07
Communication, raw subscore	5.57	1.91	5.67	2.99	0.71	0.43	0.03
Total score, raw	13.30	3.75	13.84	3.98	–1.33	0.83	0.13
Total score, adjusteda	69.89	18.59	69.03	19.64	0.43	0.68	0.04
Specific Level of Functioning Scale domain subscores (Sweden, N=142; U.S., N=197)
Physical functioning	23.87	2.34	23.74	2.08	0.53	0.60	0.03
Personal care	32.52	3.47	34.22	4.70	3.66	0.001	0.36
Interpersonal	23.61	6.03	29.78	5.37	9.09	0.001	0.67
Acceptable behavior	33.71	2.06	33.87	2.03	0.72	0.47	0.08
Community activities	48.78	7.19	49.06	8.54	0.31	0.75	0.03
Work skills	20.62	4.62	24.23	8.54	6.22	0.001	0.58
Specific Level of Functioning Scale total score	183.23	18.76	196.63	18.86	5.87	0.001	0.71
Neuropsychological tests (presented as T scores) (Sweden, N=100; U.S., N=244)
Trail Making Test, Part A	32.45	12.64	29.79	10.16	1.87	0.08	0.26
Trail Making Test, Part B	30.60	15.27	24.51	18.92	3.14	0.003	0.32
Rey Auditory Verbal Learning Test, learning subscore	35.52	14.26	34.15	13.31	0.83	0.40	0.10
Letter-number sequencing	41.26	12.18	39.94	10.84	0.94	0.32	0.15
Wisconsin Card Sorting Test categories	38.13	10.06	32.79	9.43	4.56	0.001	0.45
Average	35.59	11.40	32.23	11.82	2.06	0.04	0.28

^aConverted to a 100-point scale for comparability with the full version of the UCSD Performance-Based Skills Assessment.

While the SLOF total scores reflected significantly more impairments in the Swedish patients, some of the individual subscales did not differ, including physical functioning, socially acceptable behavior, and everyday living skills, which measures readiness for independent living. The effect sizes of the differences for the subscale that did different across the countries were moderate for work and basic ADLs and large for social functioning. In contrast, the effect sizes for NP performance were large for two variables, very small on two, and small (and nonsignificant) on one individual variable and on the composite score..

Figure 1 presents the residential, social, and vocational outcomes for the two samples of patients. As can be seen in the figure, social and vocational outcomes were essentially identical in the two samples. Where the outcomes diverged considerably was in the area of residential status. The majority of the Swedish patients were living independently and financially responsible for their housing. In contrast, the rate of individuals who were living in restricted settings was considerably greater in the American Sample.

FIGURE 1.

Real-World Outcomes Across Three Functional Domains in Patients in Sweden and the United States

Table 3 presents the Pearson Product-Moment correlations between composite NP performance, total scores on the UPSA-B, and total scores on the SLOF for the two samples. As can be seen in the table, these correlations were also extremely similar across the two countries and none of the correlations between any of the pairs of variables in the two samples were statistically significantly different using the two-sample z-test for the significance of the difference between correlations (all z<1.21; all p>.35).

Table 3.

Pearson Product-Moment Correlations Between Score on the UCSD Perfromance-Based Sklls Assessment–Brief Version Score, Neuropsychological Performance, and Score ont he Specific Level of Functioning Scale in Swedish and U.S. Samples of Schizophrenia Patients

	Assessment
Sample and Assessment	Neuropsychological Performance	Specific Level of Functioning Scale Score
U.S. sample (N=197)
UPSA-B score	.58***	.38***
Neuropsychological performance		.16*
Swedish sample (N=100)
UPSA-B score	.54***	.44**
Neuropsychological performance		.24*

Finally, Table 4 shows the association between UPSA-B scores and independent living status for the Swedish and American patients. We used one-way ANOVAs, followed by Tukey post hoc tests within each sample to examine the differences in UPSA B scores associated with levels of independent living. We chose not to use a status × country ANOVA because of the very unbalanced cell sizes. For the Swedish patients, there was no statistically significant difference across the three residential status groups in UPSA-B scores: F(2,143)=2.00, p=.14. The lack of homogeneity of variance across the cells led us to use a non-parametric analysis, the Kruskall-Wallace H test. The results of this analysis were also nonsignificant, X²(2)=.73, p=.31 When the same ANOVA was performed in the American sample, the results were statistically significant, F(2,203)=4.94, p<002. The follow-up Tukey tests indicated that each of the three groups was significantly, p<.05, different from the groups immediately adjacent in terms of residential status. There was essentially no difference in the UPSA-B scores received by patients in the US and Sweden who were living in restricted housing environments, while the main differences in UPSA scores appears to be the close association between residential outcomes and UPSA-B scores in the American patients, which resulted in the independently residing American patients having the highest UPSA scores..

Table 4.

Score on the UCSD Performance-Based Skills Assessment–Brief Version as a Function of Residential Independence in Swedish and U.S. Samples of Schizophrenia Patients

	UPSA-B Score
Country and Residential Status Group	Mean	SD
Sweden
Independent, financially responsible (n=118)	13.89	10.25
Independent, not financially responsible (n=21)	13.27	3.43
Restricted (n=7)	10.22	4.95
United States
Independent, financially responsible (n=112)	15.46	3.37
Independent, not financially responsible (n=41)	12.65	5.35
Restricted (n=91)	10.13	4.67

Discussion

The results of this study include several potentially important findings. Performance on the part of people with schizophrenia on a well-validated scale of functional capacity in the domains of everyday living skills was essentially identical in rural Sweden and New York City. Further, case manager ratings of the ability of these patients to function in terms of everyday living skills were also essentially identical in these two samples. The correlation between the different domains of the functional outcomes construct: cognitive abilities, functional capacity, and case manager ratings of real-world functioning was essentially identical in these two samples as well. Differences in performance on NP tests were modest and the correlations between NP performance and measures of functional capacity and real-world outcomes were also essentially the same in the two countries. An additional important finding, which renders the similarities in these multiple abilities even more important, is that there were substantial differences in residential outcomes between the two samples. These differences in outcomes lead to a finding of no association between performance on structured tests designed to measure everyday living skills and residential outcomes the part of people with schizophrenia in Sweden.

While there are essentially no differences in the ability to live independently (both on a functional capacity and case manager ratings basis), outcomes appear markedly more favorable in the Swedish sample. Patients who lived in restricted settings in the US and Sweden did have essentially no differences in UPSA-B performance scores, suggesting that the complete inability to live independently may be not be influenced by social service systems. In contrast, there was a clear functional capacity performance gradient associated with residential outcomes in the US sample. This finding is interesting in the context that independent living ability was the only one of the three domains of real world functional milestones measured in this study (social outcome, employment, and residential status) that was strongly associated with the presence of cognitive performance in the American sample in our previous publication⁸. Clearly these results do not necessarily apply across all countries and systems of care and a comparison of rural vs. urban outcomes in either the US or Sweden could help to unconfound rural/urban living status and differences in national health care policies.

Considerable evidence for the validity of the UPSA-B accrues from these findings, which suggests that this instrument measures some performance-based abilities that are consistent across substantial differences in culture. In order to perform this study, the subtasks from this instrument were not only translated into Swedish, but they also required modification of the stimuli in accordance with differences in financial requirements and communication demands across the two countries. The results suggest that this process may well be practical in other Western Cultures as well. Differences in NP performance across the two countries were modest when corrected for demographic factors.

There are several implications of these findings for improvements in real-world outcomes following cognitive enhancing or skills developing treatments. Equivalent improvements in ability lead to very different changes in real-world functional outcomes depending on other characteristics of the real-world environment. As Rosenheck et a1.²¹ clearly demonstrated, employment outcomes in the US are strongly associated with disability compensation, and this is particularly true for disability compensation that is linked to health insurance coverage. Other factors are likely operative. The health care system in Trollhättan is more generous with compensation than the US health care systems, with disability compensation in this part of Sweden at approximately $1,000.00 per month. Further, the cost of living is markedly less in Trollhättan (where a two-bedroom apartment costs approximately $650 USD per month) than in New York City and the immediate environment. Also, the local health authority provides each disabled patient with schizophrenia 93% of their monthly rent, up to $800 USD per month. The requirements associated with finding and paying for an apartment in the New York City metropolitan area are daunting for healthy individuals and much more challenging for individuals with disabling psychiatric symptoms and chronic unemployment. These differences in social support for people with mental illness have a clear and strong signal in terms of real-world functional outcomes. In contrast, other social and vocational milestones appear to occur with similar frequency in the two countries.

These data are consistent with the notion that ability-based measures may give a less confounded signal of functional potential than aspects of real-world community functioning^26,27. As these data suggest negligible differences in the ability to perform everyday living skills in the assessment setting and marked differences in residential outcomes, it appears likely that that residential outcomes, like employment outcomes, may be driven by factors other than ability. These findings also suggest that changes in the ability to perform everyday living skills are a central feature of schizophrenia, much like neurocognitive impairments. Although NP and FC abilities are correlated with each other, much like negative symptoms and NP deficits are as well, they appear to be separable domains of functioning that should be considered independently of their correlates Previous research has shown quite clearly that interventions that modify level of social support lead to gains in functioning in people with schizophrenia²⁸. Future research will need to continue to address the impact of social and cultural support mechanisms on outcomes and multi-national treatment studies will need to carefully consider the potential impact of these differences in social support for individuals with schizophrenia, both for the outcomes of treatment and the selection of potential participants for these research studies.