Cognitive Performance and Functional Competence as Predictors of Community Independence in Schizophrenia

Abstract

Measures of functional competence have been introduced to supplement standard cognitive and neuropsychological evaluations in schizophrenia research and practice. Functional competence comprises skills and abilities that are more relevant to daily life and community adjustment. However, it is unclear whether relevance translates into significantly enhanced prediction of real-world outcomes. The aim of this study was to assess the specific contribution of functional competence in predicting a key aspect of real-world outcome in schizophrenia: community independence. Demographic, clinical, cognitive, and functional competence data were obtained from 127 patients with schizophrenia or schizoaffective disorder and used to predict community independence concurrently and longitudinally after 10 months. Hierarchical regression analyses indicated that demographic, clinical, and cognitive predictors accounted jointly for 35%–38% of the variance in community independence across assessment points. Functional competence data failed to add significantly to this validity. Considered separately from demographic and clinical predictors, cognitive and functional competence data accounted for significant amounts of outcome variance. However, the addition of functional competence to standard cognitive test data yielded a significant increase in validity only for concurrent and not for longitudinal prediction of community independence. The specific real-world validity of functional competence is modest, yielding information that is largely redundant with standard cognitive performance.

Introduction

Performance on standard tests of cognitive ability is cited widely as a key predictor and, possibly, determinant of community adjustment and outcome in schizophrenia. Most studies of the illness confirm a broadly based impairment across perceptual processing, attention and working memory, executive abilities, visuomotor processing speed, declarative memory, and verbal fluency.^1,2 These impairments in turn predict aspects of real-world functional outcome including social behavior, recreation, community independence, employment, and daily living skills.^3–7 Not surprisingly, therefore, cognitive performance has become a focus in the search for new therapies^8–11 and in the reevaluation of existing treatments.^12,13

Nonetheless, recent developments and research raise questions about the cognition-functional outcome relationship in schizophrenia and the best way to measure it. Tests of functional competence or capacity have been introduced as complements to standard cognitive and neuropsychological evaluations.^14,15 The logic underlying this introduction is that closer approximation to the demands of everyday living will yield more accurate, powerful, and specific information relevant to real-world outcomes. Functional competence assessment elicits the kind of practical thinking and skills potentially required in many life situations. For example, a patient may be asked to demonstrate writing a cheque to make a bill payment, role-play a telephone call to arrange an appointment, or identify items for a shopping list. Functional competence is thereby distinguished from the relatively abstract, less obviously meaningful constructs, and stimuli typically used in standard cognitive tests. However, the distinction may be more apparent than real, with similar abilities underlying both kinds of measure. Moreover, cognitive and functional competence tests share “method variance” insofar as both approaches involve behaviorally based performance indicators and criteria. It follows that scores on functional competence and cognitive tests are statistically related, with shared variance ranging up to 46%.¹⁶ The current momentum to improve cognition and functional status in schizophrenia has given rise to a need for efficient and valid evaluation tools that predict everyday functioning. But are functional competence tasks an advance and truly complementary with existing cognitive tests in this respect or simply redundant and possibly unnecessary?

Ecological validity or sensitivity to real-world behavior in residential, social, and occupational settings is, arguably, an acid test for any instrument used to evaluate functional status. Additional considerations, including theoretical value, tolerability, efficiency, sensitivity to short-term change, and suitability as a “coprimary measure” in clinical trials, are also relevant.^17,18 However, it is difficult to promote a putative indicator of functional status unless it predicts important aspects of patients’ actual behavior and adjustment in real-life contexts. In this regard, the purpose and practical content of functional competence instruments imply an advantage relative to standard tests of cognitive ability. Yet, it is noteworthy that both kinds of tasks associate significantly with real-world outcomes. For example, summary scores from the University of California San Diego Performance Skills Assessment (UPSA),¹⁴ the most commonly used functional competence measure, capture up to 37% of the variance in outcomes like independent community living and social behavior.^3,16,19 Cognitive and neuropsychological test batteries account for up to 29% of real-world variance.^3,5,16,20 These validities compare favorably with those achieved by a wide spectrum of diagnostic, indicator, and simulation tasks in the medical and psychological literatures.²¹ However, it remains uncertain whether functional competence measures are superior to standard cognitive tests in terms of real-world validity and whether they contribute substantial new validity to the assessment of schizophrenia.

To investigate this issue, we evaluated schizophrenia and schizoaffective disorder patients at index and 10 months later with a set of commonly used cognitive tests and with a measure of functional competence. Scores from these instruments, along with basic demographic and clinical data, were used to predict a key aspect of real-world outcome—overall independence in community living. Our basic questions were does measurement of functional competence add new predictive validity not obtained by cognitive measures on their own and is combined validity superior to the individual validities of these 2 kinds of instrument?

Methods

Participants

Patients were recruited from outpatient settings including the Hamilton Program for Schizophrenia, the Community Schizophrenia Service (St Joseph's Healthcare Hamilton), the Cleghorn Program (St Joseph's Healthcare Hamilton), the Canadian Mental Health Association (Toronto branch), and the Challenging Directions Program (Whitby Mental Health Center). Male and female subjects who met the following criteria were included: (1) diagnosis of schizophrenia or schizoaffective disorder by Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition (DSM-IV),²² criteria confirmed with the Structured Clinical Interview for DSM-IV Axis I Disorder-Patient version²³; (2) age 18–65 years; (3) no history of serious neurological or endocrine disorder, including head trauma, epilepsy, Cushing disease, or thyroid disorder; (4) no concurrent DSM-IV diagnosis of substance abuse; (5) no history of developmental disability; (6) willingness and ability to sign informed consent; and (7) normal or corrected vision.

A total of 157 patients met inclusion criteria and enrolled in the index assessment, and 127 patients, including 82 males and 45 females, completed the follow-up assessment to comprise the study sample. The mean interval between index and follow-up assessment was 10.5 months. Medical chart review indicated that 126 (99%) patients were receiving antipsychotic medication at the time of data collection, with 108 (85%) patients treated with second-generation drugs. Patients ranged from 21 to 65 years of age, with a mean of 41.50 (SD = 9.03). Age of illness-onset data, comprising either first psychiatric hospital admission or first psychiatric contact and report of psychotic symptoms, indicated a mean onset age in years of 20.90 (SD = 5.45). All patients were administered the Positive and Negative Syndrome Scale (PANSS)²⁴ as an index of symptom severity.

To establish a normal reference point for community independence, healthy comparison participants (n = 60) were recruited by postings and advertisements for paid research participation in community newspapers. Potential participants were screened for medical and psychiatric illness and history of drug abuse. This yielded 47 males and 13 females, ranging from 18 to 65 years of age, with mean of 40.40 (SD = 14.20). Patients and comparison groups did not differ significantly in age, sex composition, parental socioeconomic status, birth country (foreign vs domestic), or first language learned (English vs all others). However, healthy comparison subjects had higher levels of educational achievement (${\mathrm{\chi }}_7^2$ = 36.53, P < .001). All participants signed informed consent and were paid for their time. The project was approved by the institutional review board at each research site and by York University.

Neurocognitive Measures

Neurocognitive tests measuring several aspects of cognitive performance were administered to all patients at the index assessment. English language versions of all measures were used. They included the vocabulary, matrix reasoning, letter-number sequencing, and symbol search subtests of the Wechsler Adult Intelligence Scale (WAIS)-III.²⁵ Vocabulary indexed verbal ability, whereas matrix reasoning indexed nonverbal ability, letter-number sequencing measured auditory working memory, and symbol search tapped visual processing speed. These 4 tests represent the ability factors underlying the WAIS-III. Verbal memory acquisition and learning were measured with the total number of words recalled over 5 trials on the California Verbal Learning Test (CVLT)-II.²⁶ Visual attention/response inhibition was assessed with the “detectability” or d′ statistic from the Conners Continuous Performance Test (CPT)-II,²⁷ and word generation and fluency were indexed by phonemic and semantic trials of the Controlled Oral Word Association Test (COWAT).²⁸ The selection of tests was based on considerations of efficiency, validity and reliability, and the nature and number of separable ability factors underpinning cognitive performance in schizophrenia patients.

Measurement of Functional Competence

Functional competence was assessed at index with the UPSA¹⁴ as follows. Participants were asked to perform everyday tasks in 5 domains. In the first domain, recreation planning was evaluated by having the participant read an advertisement for a theme park and then answer questions about planning an outing. Financial ability was assessed by a change-counting task as well as by having the patient identify important features of a household bill. Role-playing tasks utilizing a telephone, including rescheduling a medical appointment, evaluated communication abilities. The transportation domain involved answering questions about use of a public transportation system based on stimulus material provided. Finally, in the household management domain, the participant was given a recipe and asked to examine a number of displayed pantry items. The task assignment was to prepare a shopping list of recipe items not available in the pantry. Each of these 5 domains generated a raw score that was converted to a domain score ranging from 0 to 20 points.

Measurement of Community Independence

Real-world outcome and independent functioning in the community were measured at index and 10-month follow-up with the overall global rating from the Multidimensional Scale of Independent Functioning [MSIF]).²⁹ The MSIF comprises a structured interview and self-report measure, with verification of information provided by history, chart and document review, proxy reports, and informant interviews. Subjects are rated on 7-point Likert scales with respect to role position, level of performance, and degree of assistance required in residential, work, and educational settings. The summary global rating reflects the level of functioning across all domains and settings. Thus, an aggregate rating of 1 indicates role functioning that approximates community norms for healthy people. In contrast, a rating of 4 indicates moderate disability whereby a person may have difficulty performing in mainstream environments despite receiving support and assistance. “Mainstream” environments are defined as role settings located in the community among the general population. A rating of 7 indicates total disability whereby a person receives complete care across domains and settings with no independent functioning. Ratings are based on detailed anchors provided for each environment. The overall global rating has been shown to have good interrater reliability (intraclass correlation = 0.90) and varies strongly with outcome measures like the Social Adjustment Scale.³⁰ Moreover, the MSIF and its component ratings vary significantly and in expected directions with differences in patients’ residential, work, and educational situations. Nonetheless, to confirm the overall global rating as a valid discriminator of real-world functioning in patients and healthy people, the MSIF was administered to the nonpsychiatric sample as well as to the patient sample for comparative purposes. Two global scores were obtained for patients: an index global score and a follow-up global score.

Results

Overall global MSIF scores in patients (M = 4.39, SD = 1.1) and healthy comparison samples (M = 1.82, SD = 1.13) differed significantly (t₁₈₅ = 14.74, P < .001) and correspond to an effect size (Cohen d) of 2.24. This confirmed the discriminant validity of the MSIF as a measure of community independence in the present context. Descriptive statistics for the patient sample are presented in table 1 and table 2. The majority of patients were moderately disabled in terms of community independence, although the full range from normative functioning to total disability was represented in the sample. A paired-difference test (t₁₂₅ = 0.26, P > .05) detected no change in mean global MSIF scores over the follow-up period. Accordingly, this was a relatively stable group of patients in terms of community functioning.

Table 1.

Demographic and Clinical Characteristics of Schizophrenia Patients (n = 127)

Age in years, M (SD)	41.49 (9.03)
High school completion, n (%)	90 (71)
Gender, n (% male)	82 (65)
First language, n, % English	106 (84)
Birth country, n (% foreign)	24 (19)
Parental socioeconomic status, n (%)
Wealthy	4 (3)
Professional	38 (30)
White collar	33 (26)
Semiskilled laborer	39 (31)
Unskilled worker	13 (10)
Positive and Negative Syndrome Scale
Positive subscale, M (SD)	19.93 (5.93)
Negative subscale, M (SD)	19.21 (5.87)
General subscale, M (SD)	41.72 (8.27)

Table 2.

Cognitive, Functional Competence, and Community Independence Scores

	M (SD)
Wechsler Adult Intelligence Scale-III age scaled scores
Vocabulary	9.81 (3.49)
Matrix reasoning	9.22 (3.35)
Letter-number sequencing	8.57 (3.17)
Symbol search	7.47 (3.01)
California Verbal Learning Test-II
Trials I–V recall	40.44 (11.72)
Controlled Oral Word Association Test
Total words	34.58 (11.61)
Continuous Performance Test-II
d′ T score	49.11 (9.62)
University of California San Diego Performance Skills Assessment
Planning and organization/20	16.74 (3.81)
Finance/20	17.52 (2.81)
Communication/20	13.56 (3.98)
Transportation/20	18.50 (2.87)
Household management/20	16.69 (3.46)
Multidimensional Scale of Independent Functioning
Index global/7	4.39 (1.12)
Follow-up global/7	4.36 (1.04)

Hierarchical multiple regression was used to predict index and follow-up global MSIF scores in separate analyses. All predictors were obtained at the index assessment and entered in successive blocks as follows. First, a demographic block comprising patient age, education, sex, birth country, parental socioeconomic status, and first language was entered. Second, a clinical block comprising symptom severity scores on the positive, negative, and general psychopathology subscales of the PANSS was entered. The third block consisted of vocabulary (WAIS-III), matrix reasoning (WAIS-III), letter-number sequencing (WAIS-III), and symbol search (WAIS-III) scaled scores as well as trials I–V recall (CVLT-II), total words (COWAT) and d′ T score (CPT-II). The final block comprised functional competence scores from the UPSA: planning and organization, finance, communication, transportation, and household management. The significance of each successive block's contribution to validity was assessed with F tests. These results are presented in table 3.

Table 3.

Hierarchical Regression for Multidimensional Scale of Independent Functioning Scores in Schizophrenia Patients (n = 127)

	Index			Follow-up
Model/block	R2	F Change	P	R2	F Change	P
Demographica	0.05	1.10	.37	0.08	1.76	.11
Demographic + clinicalb	0.19	6.58	<.001	0.22	6.88	<.001
Demographic + clinical + cognitivec	0.38	4.69	<.001	0.35	3.21	.004
Demographic + clinical + cognitive + functional competenced	0.43	2.18	.06	0.41	1.93	.10

^aAge, sex, education, parental socioeconomic status, birth country, and first language.

^bPositive, negative, and general psychopathology scores (Positive and Negative Syndrome Scale).

^cVocabulary, matrix reasoning, letter-number sequencing, symbol search (Wechsler Adult Intelligence Scale-III), trials I-V recall (California Verbal Learning Test-II), total words (Controlled Oral Word Association Test), and d′ T score (Continuous Performance Test-II).

^dPlanning and organization, finance, communication, transportation, and household management (University of California San Diego Performance Skills Assessment).

The regression analysis for the index assessment indicates that demographic variables failed to account for a significant portion of concurrent real-world outcome variance. However, the block of symptom severity variables (PANSS) accounted for 14% of explained variance. Within the clinical block of predictors, the general psychopathology subscale (PANSS) contributed uniquely to the model's validity (β = 0.26, t = 2.06, P < .05). The block of standard cognitive measures added significantly to validity with a further 19% of explained variance. Within the cognitive block, WAIS-III vocabulary (β = −0.30; t = −2.60; P < .001) and symbol search (β = −0.21; t = −2.06; P < .05) as well as the CPT-II d′ score (β = 0.30; t = 3.68; P < .001) contributed uniquely to validity. The addition of functional capacity measures (UPSA) to the equation yielded a validity increment of 5%, but this was not significant. Moreover, only the CPT-II d′ score contributed uniquely to validity (β = 0.30; t = 3.57; P < .01) when all predictors were included in the prediction equation.

The identical hierarchical procedure for the follow-up assessment data shows that demographic variables again failed to capture a significant amount of real-world outcome variance. The addition of the clinical block of symptom variables yielded a significant increase in validity. Within the clinical block, the PANSS positive subscale contributed uniquely to prediction (β = 0.23; t = 2.04; P < .05). Inclusion of the cognitive test variables resulted in a significant increase in validity, but only WAIS-III symbol search scores contributed uniquely to prediction (β = −0.23; t = −2.23; P < .05). Finally, the UPSA competence measures did not contribute significant new validity to the variables already in the equation. In addition, with all variables entered no individual measure contributed uniquely to prediction.

To evaluate the separate and joint predictive validity of cognitive and functional competence measures without the contribution of demographic and clinical variables, more specific regression analyses were undertaken (Table 4). First, overall global MSIF scores were regressed on the block of cognitive variables. Next, the same procedure was carried out separately for functional competence variables. Finally, joint and conditional validity were assessed by including both blocks of variables in the analysis. Results of these procedures indicate that cognitive and functional competence predictors accounted for significant amounts of real-world outcome variance separately and jointly. Moreover, at the index assessment, WAIS-III symbol search (β = −0.26; t = −2.67; P < .01), CPT-II d′ (β = 0.27; t = 3.40; P < .01), and the UPSA planning and organization score (β = −0.18; t = −2.17; P < .05) all contributed uniquely to the validity of the joint prediction equation. In contrast, only WAIS-III symbol search (β = −0.23; t = −2.34; P < .05) contributed selectively to prediction in the joint model for the follow-up analysis. Consideration of the conditional validity of functional competence shows that the increase in validity and explained variance obtained by adding UPSA scores to cognitive predictors was relatively small (5%–7%) and significant only for index global MSIF scores.

Table 4.

Selected Regressions for Multidimensional Scale of Independent Functioning Scores in Schizophrenia Patients (n = 127)

	Index			Follow-up
Model/Block	R2	F	P	R2	F	P
Cognitivea	0.28	6.66	<.001	0.26	5.88	<.001
Functional competenceb	0.20	6.04	<.001	0.18	5.25	<.001
Cognitive + functional competence	0.35	5.02	<.001	0.31	4.23	<.001
Functional competence/cognitive	0.07	2.33	<.05	0.05	1.69	.14

^aVocabulary, matrix reasoning, letter-number sequencing, symbol search (Wechsler Adult Intelligence Scale-III), trials I–V recall (California Verbal Learning Test-II), total words (Controlled Oral Word Association Test), and d′ T score (Continuous Performance Test-II).

^bPlanning and organization, finance, communication, transportation, and household management (University of California San Diego Performance Skills Assessment).

Discussion

The data presented above suggest that researchers and clinicians may have been too uncritical in applying functional competence measures to schizophrenia assessment, assuming that these measures contribute new information not available in standard cognitive and neuropsychological evaluations. We found that symptom severity and cognitive performance jointly account for significant amounts of real-world outcome variance longitudinally as well as concurrently. There was evidence suggesting a special validity for processing speed and attention measures within the set of cognitive measures. In addition, cognitive tests and a widely used measure of functional competence (UPSA) account separately and jointly for significant amounts of outcome variance. However, functional competence does not enhance substantially the predictive validity already achieved with standard cognitive tests. Functional competence measures improve the predictive validity obtained by cognitive tests very modestly and only in relation to concurrent community functioning. At best, the inclusion of these measures adds marginally to the accuracy of inferences about a patient's current functional status. At worst, functional competence tests are redundant with standard cognitive evaluations and imply, but fail to deliver, a unique sensitivity to real-world behavior and adjustment.

Research on the cognition-functional outcome relationship has treated functional competence as a construct that mediates between standard cognitive performance and real-world outcomes.^3,16 In addition, both cognitive and functional competence tasks have direct relations with aspects of outcome.³¹ At the same time, instruments like the UPSA seem to possess “face” validity by incorporating obviously meaningful stimuli and role-play activities. Hence, from these perspectives, evaluation with functional competence measures seems an essential and, arguably, irreplaceable component in schizophrenia assessment. However, there are few clear comparisons of the relative and joint predictive validity of cognitive and functional competence measures in terms of real-world outcome. There is evidence of approximately equal real-world validities for neuropsychological test batteries and the UPSA.¹⁶ In contrast, Mausbach et al³² reported that a brief version of the UPSA was superior to the Dementia Rating Scale (DRS) in predicting real-world outcome. Nevertheless, although DRS validity was modest, the UPSA. Moreover, the DRS was not designed for schizophrenia patients and should not be regarded as equivalent to a battery of cognitive performance tests.³³ On balance, the putative superiority and specific real-world value of functional competence tasks remain doubtful, especially in light of our own findings.

These considerations bear on the role of cognitive and functional competence measures across a variety of clinical and scientific applications in the field of schizophrenia. It is difficult to argue for the inclusion of both kinds of measures in the evaluation of treatment if their sensitivity to real-world outcomes is about the same. In such a situation, the psychometric properties, theoretical value, neurological validity, and extensive research base support the selection of standard cognitive tasks over functional competence measures. Against this, the primary advantages of functional competence measures are their inherently meaningful, practical content, and brief administration time. Activities like making telephone calls and checking bus schedules seem to have immediate relevance to community functioning, whereas reasoning with abstract patterns or learning word lists seems to lack relevance. Yet, while training patients to improve their skills on the kind of tasks represented in the UPSA may increase their repertoire of adaptive behaviors, the tie with enhanced independence in community living remains to be demonstrated. There is also recent evidence that UPSA scores vary in tandem with behavioral interventions focused on life skills.³² Thus, there may be applications in program evaluation that favor the use of functional competence over standard cognitive measures. Moreover, drug approval and evaluation policies may require the use of simulated life skills in clinical trials even in the absence of scientific support for the unique validity of measures like the UPSA. Overall, researchers and clinicians should exercise caution in assigning a special significance to functional competence data, at least in relation to outcome variables like community independence.

It is noteworthy that recent reports from the Measurement And Treatment Research to Improve Cognition in Schizophrenia project indicate modest real-world validities for both cognitive and functional competence instruments.^34,35 These low validities have been attributed, in part, to the use of outcomes estimated exclusively by patient self-report and without the corroboration of informants or objective sources. Yet, the complex influences involved in real-world outcomes may also limit the validity that can be expected for any specific behavioral indicator of functional status. For example, some settings lack the opportunity or resources for independent living even though patients have the potential to function at such a level.³⁶ We found that a combination of cognitive and functional competence measures account for approximately one-third of independence-related outcome variance. The acceptable minimum of real-world validity for functional status measures remains to be determined and will assume increased urgency as schizophrenia research advances the frontiers of treatment discovery and evaluation.

Limitations bearing in our research include an inability to evaluate baseline cognitive and functional competence measures as predictors of change in community independence over time. All patients in the study were active participants in rehabilitation programs, but there was no significant increase (or decrease) in MSIF scores over the 10- to 11-month follow-up period. Hence, the data reflect a stable patient population in terms of community independence. In addition, it should be noted that the functional competence measure used in our study, the UPSA, has been modified since its introduction and exists in several versions with minor variations in content as well as difficulty level. The UPSA version used in the study was supplied by the instrument's developers but may differ slightly in item content from other versions in current use.

It would be worthwhile to determine if functional competence is largely redundant with cognitive performance across type of competence measure,^15,37 variations in cognitive tests, and different real-world indicators.^29,32 If our findings generalize, then clinicians and researchers should question the need to add these measures to existing methods of assessing and evaluating functional status in schizophrenia.

Funding

The Ontario Mental Health Foundation (R.W.H.); Community Schizophrenia Vocational Rehabilitation Schizophrenia Foundation (R.W.H.).