The Convergence between Self-reports and Observer Ratings of Financial Skills and Direct Assessment of Financial Capabilities in Patients with Schizophrenia: More Detail is Not Always Better

Abstract

Despite multiple lines of evidence suggesting that people with schizophrenia tend to overestimate their ability to perform everyday tasks such as money management, self-report methods are still widely used to assess functioning. In today’s technology driven financial world patients are faced with increasingly complex financial management tasks. To meet these challenges adequate financial skills are required. Thus, accurate assessments of these abilities are critical to decisions regarding a patient’s need for support such as a financial trustee. As part of the larger VALERO study, 195 patients with schizophrenia were asked to self-report their everyday financial skills (five common financial tasks) with the Independent Living Skills Survey (ILSS). They were also assessed with performance-based measures of neuro-cognition and functional capacity with a focus on financial skills. In addition, a friend, relative, or clinician informant was interviewed with the ILSS and a best estimate rating of functioning was generated. Scores on the performance-based measures of financial skills and neuropsychological tests were uncorrelated with self-reported financial activities. Interviewer and all informant judgments of financial abilities were also minimally correlated with performance on functional skills tests. Discrete financial skills appear to be challenging for clinicians to rate with accuracy without the use of direct assessments. Direct assessment of financial skills seems prudent when making determinations about the need for guardianship or other financial supervision.

Financial management is an essential skill for everyday functional independence. The ability to perform tasks, such as paying bills, manipulating currency, depositing and accessing money in bank accounts, and budgeting, are all necessary for individuals to successfully function within the community. Financial institutions are increasingly encouraging consumers to rely on the use of technology such as Automatic Teller Machines for common financial tasks such as accessing cash, making deposits, and general accounts management. Further, in the United States, retirement and disability compensation is typically delivered directly to bank accounts. Several studies have shown that financial management tasks, especially those that involve technology and no face-to-face customer service support, require several underlying cognitive abilities, such as working and long term memory, executive function, and reasoning (e.g., Czaja et al., 2006).

Individuals with severe mental illness, including schizophrenia, often have difficulty performing financial management tasks (Perivoliotis et al., 2004), as evidenced by the high proportion of these individuals who have a trustee to manage their funds (Marson et al;., 2006). In this regard, studies of everyday functioning among people with severe mental illness have been expanded to understand the skills-based determinants of these deficits (Harvey et al., 2009). Although it is well established that cognitive impairments are predictive of deficits in everyday functioning (Green et al., 2000), in an effort to target intervention strategies recent emphasis has been on mapping abilities with real-world functional deficits on specific tasks (Bowie et al., 2006; 2008). In this regard performance-based measures of functional capacity, the ability to perform everyday functional and social skills, have consistently found to be related to in social, residential, and vocational functioning in the “real world” in patients with schizophrenia and bipolar disorder. These studies have included outcomes such as the ability to achieve real-world milestones (e.g., the ability to pay rent or having money available; Harvey et al., 2012; Leung et al., 2008) as well as ratings of current and potential functioning generated by informants close to the patient (e.g. a close friend or relative; Sabbag et al., 2011). In this regard, several different performance-based assessments of financial management have been developed. For instance, the UCSD Performance Based Skills Assessment (UPSA; Patterson et al, 2001) has a financial management subtest, as does the Independent Living Scales (ILS; Loeb, 2006). Similar performance-based money management tests have been developed for other patient populations, including Alzheimer’s disease and HIV infection (e.g., the Advanced Finances subtest of the Everyday Functioning Battery; Heaton et al., 2004). Clinical rating scales that collect reports from patients or informants are also used to generate an impression of the patient’s money management skills, although they may be subject to reporter biases (Loewenstein et al., 2001).

In reality, many of the skills involved in money management are “micro-level” skills, including completing forms, paying bills correctly, and maintaining adequate reserves of money to anticipate needs until the “next check arrives.” Rating the level of competence in these skills on the part of patients may be problematic for many informants, such as case managers, because they may have never observed patients directly performing these types of functional activities. Similarly, self-reports from patients with schizophrenia in general are poorly associated with objective measures of outcomes. For example, the correlation between patients’ ratings cognitive performance and standard neuropsychological assessments are typically quite low (Keefe et al., 2006). Similarly, the convergence between self-reports of everyday functioning and reports from high-contact clinicians have been found to be essentially zero across several different data sets (Bowie et al., 2007; Sabbag et al., 2011), while ratings of functioning on the part of high-contact clinicians have been foind to be are correlated with patients’ performance on measures of functional capacity and neurocognition.

In this study, we examined the convergence between performance on specific tests of financial management ability on the part of patients with schizophrenia and detailed assessments of real-world functioning in these same domains. As part of a larger project, the Validation of Everyday Real World Outcomes (VALERO; Leifker et al., 2011; Harvey et al., 2011) initiative, we performed a detailed, multi-task, and performance-based examination of financial management skills and we collected self-reports and informant reports of financial management skills, yielding an interviewer judgment of functioning (described in Harvey et al., 2011). In this study we examined the convergence between performance on the targeted financial management tests, as well as more general functional capacity measures and neurocognition and 1). self-reported performance from patients, 2). Informant reports of functioning, and 3). interviewer judgments of real world financial performance.

Methods

Participants

These analyses are part of the larger VALERO study (Harvey et al., 2011), aimed at identification of the best methods for rating everyday functioning in people with schizophrenia. The study participants were male and female patients with schizophrenia (n=195) who were receiving treatment at one of three different outpatient service delivery systems, two in Atlanta and one in San Diego. Informants interviewed concerning the everyday functioning of each of the patients, were either a high-contact clinician (case manager, psychiatrist, therapist, or residential facility manager; 20% of cases) or a friend or relative (80% of cases). All research participants provided signed, informed consent, and the study was approved by local IRBs in Atlanta and San Diego. Patients self-identified their informants, who also signed informed consent forms to participate. In Atlanta, patients were either recruited at a psychiatric rehabilitation program (Skyland Trail) or from the general outpatient population of the Atlanta VA Medical Center. The San Diego patients were recruited from the UCSD Outpatient Psychiatric Services clinic, a large public mental health clinic, and other local community clinics and by word of mouth. Table 1 presents the demographic information.

Table 1.

Demographic Information, Clinical Symptoms, and Performance Based Scores for the Patient Sample

N	195
% Male	69
Racial Characteristics (%)
Caucasian	55
African-American	38
Other	7
Latino(%)	12
	M	SD
Age	44.03	11.73
Years of Education	12.97	2.52
WRAT-III Reading Score	47.72	5.58
Beck Depression Inventory-II	15.80	12.03
Total Score
PANSS Total Score	64.15	14.73
Performance Based Scores for the Patient Sample
	M	SD	Possible Range
UPSA-B Finances (scaled)	40.11	7.46	0–50
Advanced Finances (raw)	8.64	3.81	0–14
UPSA-B Total (scaled)	76.67	14.14	0–100
Modified MCCB (t)	37.90	6.94	20–80

All patients were administered a structured diagnostic interview, either the Structured Clinical Interview for the DSM (SCID; First et al., 1995 administered at the Atlanta sites) or the Mini International Neuropsychiatric Interview, 6^th Edition (MINI; Sheehan et al., 1998) administered at the San Diego site) by a trained interviewer. All diagnoses were subjected to a consensus procedure at each site. Patients were excluded for a history of traumatic brain injury with unconsciousness >10 minutes, brain disease that includes seizure disorder or neurodegenerative condition, or the presence of another DSM-IV diagnosis that would exclude the diagnosis of schizophrenia. None of the patients were experiencing their first psychiatric admission. Comorbid substance use disorders were not an exclusion criterion, in order to capture a broad array of patients, but patients who appeared intoxicated were rescheduled. Inpatients were not recruited. Patients resided in a wide array of unsupported, supported, or supervised residential facilities. Informants were not screened for psychopathology or substance abuse.

Procedure

All patients were examined with a performance-based assessment of neurocognitive abilities and functional capacity. Patients and informants also provided reports of social, residential, and vocational functioning by completion of a series of 6 questionnaires and interview-based procedures. The examiner who conducted the interviews with the patient and informant then generated ratings, based on his or her impression of the “true” status of the patient on all six functional rating scales Clinical ratings of symptoms were collected with the Positive and Negative Syndrome Scale and are presented in Table 1.

Performance-Based Assessment of Financial Management

We administered two different performance-based functional capacity measures that assessed financial management abilities. One was the brief version of the UCSD Performance-based Skills Assessment (UPSA-B; Mausbach et al., 2007). The UPSA-B is a measure of functional capacity in which patients are asked to perform everyday tasks related to communication and finances. For the Finances subtest, participants handle currency and count change, read a utility bill, and write and record a check for the bill. During the Communication subtest, participants role-play exercises using an unplugged telephone (e.g., emergency call; dialing a number from memory; calling to reschedule a doctor’s appointment). For this study, we examined the finances subtest as our primary variable of interest and used the total score for comparative analyses of more general functional capacity effects. Both of these subscales have scores ranging from 0–50, with higher scores reflecting better performance.

We also administered the Advanced Finances subscale of the Everyday Functioning Battery (EFB; Heaton et al., 2004), designed to examine financial management with a slightly more challenging procedure. The Advanced Finances test requires individuals to prepare bank deposits and checks to pay bills, maintain a checkbook balance, and organize payments such that a pre-specified amount of money is left available at the end of the task. Total scores on the Advances Finances subtest range from 0–13.

Neurocognition

We examined cognitive performance with a modified version of the MATRICS consensus cognitive battery (MCCB; Nuechterlein et al., 2008). For this study, we did not include the social cognition measure from the MCCB, the Mayer-Salovey-Caruso Emotional Intelligence Test Managing Emotions, because several recent meta-analyses (e.g., Fett et al., 2011; Ventura et al., 2013) found that social cognition measures are minimally correlated with neurocognitive test performance and that neurocognition and social have very different relationships with everyday outcomes. We calculated a composite score, an average of the 9 age-corrected T-scores based on the other tests in the MCCB, using the MCCB normative program, as our critical dependent variable.

Real World Financial Management

We used the Independent Living Skills Survey (ILSS; Wallace et al., 2000) as our index of real-world financial functioning because of its detailed examination of everyday functional performance in domain of financial management. The ILSS has separate patient and examiner forms and separate assessment procedures. On the patient form, the questions address whether the patient reports that s/he did or did not perform the critical functional skill in the past 30 days, although no request is made to rate the quality of their actual performance. The skills assessed include paying bills, making a deposit, budgeting, cashing a check, and managing money successfully. All of these items are parallel to performance-based items on the EFB and UPSA-B finances subscale. For the informant version, the questions are focused on real-world performance: the frequency of successfully performing the skilled acts in the last 30 days as required, ranging from “never” to “always” (scored on a 0–4 point range). In addition, the informant version rates the extent to which the individual performed the skilled acts without prompting. A total score is also calculated on the basis of the informant ratings across items, resulting in a range of scores from 0 to 20. Self-report items have no such total score, as all items are scored on a yes/no basis.

Data Analyses

In these analyses we examined performance on the UPSA-B Finances subscale and total score, the EFB, and the MCCB with self-reports, informant reports, and interviewer judgments and of the quality of patients’ performance of these skilled acts, separating the informants into friend or relative informants and high-contact clinicians. We used t-tests to compare scores on the performance-based tasks as a function of whether patients reported that they had performed each of these tests in the last 30 days. We computed Pearson product moment correlations between the four patient performance variables and the interviewer, informant, and patient reports of performance. As there were 20 t-tests and 20 correlations computed per analysis set, we adjusted the p values using the Bonferroni correction (p<.05/20=.001) to reduce the possibility of family-wise Type 1 error.

Results

Table 1 presents performance for the group of patients on the modified MCCB, the UPSA-B subscales, and the everyday functioning battery. Table 2 presents the scores on the performance-based assessments as a function of whether the patients reported performing the skills in question in the past 30 days. As can be seen in the table, the majority of patients reported that they had performed each of the financial management tasks in the past 30 days.

Table 2.

Performance on Financial Skills Tasks as a Function of Patients’ Self-Report of Performing of these Tasks

ILSS Item	No		Yes
Paid Bills	N=91		N=104
	M	SD	M	SD	t	p
UPSA-B Finances	40.30	6.52	40.81	6.58	−.39	.70
Advanced Finances	9.30	3.94	8.24	3.46	1.38	.18
UPSA-B Total	74.50	11.90	77.34	12.48	1.12	.23
MCCB Composite	37.80	7.30	37.80	6.70	−.04	.97
Bank Deposit/Withdrawal	N=99		N= 96
	M	SD	M	SD	t	p
UPSA-B Finances	41.48	6.99	39.83	7.71	1.24	.22
Advanced Finances	9.03	3.44	8.38	4.01	0.89	.38
UPSA-B Total	78.39	10.09	75.70	13.98	1.03	.31
MCCB Composite	38.30	8.23	36.47	6.33	0.67	.51
Budgeted	N=63		N=132
	M	SD	M	SD	t	p
UPSA-B Finances	39.28	8.00	40.39	7.34	−.48	.64
Advanced Finances	9.24	3.30	8.45	3.92	1.01	.31
UPSA-B Total	75.59	12.56	76.75	13.08	−.80	.43
MCCB Composite	39.45	7.06	37.42	6.75	1.29	.21
Cashed Check	N=125		N=70
	M	SD	M	SD	t	p
UPSA-B Finances	40.50	7.48	38.78	7.07	0.60	.62
Advanced Finances	8.62	8.70	8.32	4.06	0.35	.72
UPSA-B Total	76.48	10.80	76.63	13.52	0.26	.75
MCCB Composite	36.45	6.78	37.52	6.76	−.79	.43
Purchased Essentials	N=112		N=83
	M	SD	M	SD	t	p
UPSA-B Finances	40.95	6.47	40.01	7.35	0.63	.53
Advanced Finances	8.48	3.97	8.41	3.94	0.08	.94
UPSA-B Total	79.07	13.06	76.04	12.96	1.12	.26
MCCB Composite	37.44	7.29	37.92	6.85	−.33	.94

Next we used t-tests to examine the association between patients’ reports of whether they performed the financial tasks in question and their performance on the four performance-based ability measures. Scores on the 4 performance based measures are presented as a function of whether the research participants had performed each of the five functionally skilled acts in the last 30 days, leading to a total of 20 t-tests. None of these tests identified a statistically significant difference in performance (either at the Bonferroni adjusted .001 level or standard .05 level) between patients who reported that they recently performed the financial skills in question and those that said that they had not. Thus, there were no objective ability differences between patients who reported performing financially skilled acts in the last month and those who reported they did not perform the tasks, despite the fact that many patients reported performing these tasks recently.

There was also strikingly little relationship between reported functional performance across informants and scores on the modified MCCB, the UPSA-B Financial and Communication subscales, and the EFB. Table 3 presents the correlations between the patients’ financial management functioning as rated by the interviewer and the different informants, and performance on the two financial management tests, the MCCB, and the UPSA-B Finances subtest. As some of the friend or relative informants could not provide ratings that they considered valid, there are different sample sizes for each correlation and associated significance levels. For the interviewer ratings, only one of the 20 correlations exceeded the Bonferroni criteria for statistical significance. Specifically, interviewer ratings for the ability to make a bank deposit were correlated with the total scores on the EFB, which assesses this specific ability. Three other correlations were significant at conventional levels, but accounted for 4% or less shared variance. No friend or relative informant ratings of any real-world financial management exceeded the Bonferroni criteria for statistical significance with financial ability scores. The ability to make a bank deposit was correlated with the three performance based functional skills measures at conventional significance levels (and in a smaller sample size). The ability to purchase essentials was also correlated at conventional levels with performance on the UPSA-B financial skills subtest. Finally, high contact clinician ratings of the ability to make a bank deposit exceeded the Bonferroni criteria for statistical significance with UPSA-B total scores and correlated at conventional levels with UPSA-B financial skills.

Table 3.

Correlations Between Judgments of Patient Financial Skills and Patient Performance on Financial Skills Tests

	Patient Performance
	UPSA-B	Advanced	UPSA-B	MCCB
	Total	Finances	Finances	Total
Interviewer Rating
Of Ability
(n=195)
Paying Bills	.20**	.13*	.08	.07
Bank Deposit/Withdrawal	.03	.26***	.02	.07
Budgeting	.18*	.12*	.10	.04
Cashing Check	.01	.14*	.05	.04
Purchasing Essentials	.00	.06	.08	.05
Total Score	.19*	.09	.06	.04
Friend/Relative Rating
Paid Bills	.12	.01	.12	.07
(n=76)
Bank Deposit/Withdrawal	.31**	.26*	.31**	.07
(n=72)
Budgeting	.18	.14	.18	.06
(n=88)
Cashing Check	.17	.13	.15	.04
(n=69)
Purchasing Essentials	.14	.15*	.28**	.05
(n=89)
Total Score	.19*	.21*	.27*	.24*
Clinician Rating
(n=39)
Paying Bills	.13	.33	.00	.00
Bank Deposit/Withdrawal	.47***	.39*	.23	.12
Budgeted	.25	.23	.29*	.02
Cashing Check	.03	.02	.25	.28*
Purchasing Essentials	.29	.00	.10	.07
Total Score	.06	.23	.09	.21

^*p<.05

^**p<.01

^***p<.001

Finally, we examined the correlations between total scores (also in table 3) on the money management subscale and the four performance-based ability variables. The total score for money management rated by the interviewer achieved conventional criteria for statistical significance and accounted for about 4% variance in UPSA-B total scores, but not for any other variables. None of the correlations for clinician ratings achieved significance. Finally, all four of the correlations between total scores for money management and the four performance-based ability variables reached conventional levels of significance when rated by a friend or relative.

Discussion

This study examined the convergence among interviewer ratings, informant reports, self-reports and objective measures of financial management abilities among patients with schizophrenia. Overall, the findings indicated that interviewer ratings, informant reports, and self-reports of performance on highly specific everyday financial tasks were minimally associated with performance on measures that directly targeted component financial skills. A consistent exception was the ability to make a bank deposit. However, the relevance of this skill is itself one of the less important financial management tasks, as most deposits for disability compensation are now direct from the source to the bank account, bypassing the need to personally make a deposit.

These findings are quite important given that the global ratings across six different rating scales of real world functioning generated by these same interviewers based on interviews with these same informants were robustly correlated with performance on the three performance-based ability measures (Harvey et al., 2011) . Specifically, the findings suggest that while interviewers may be able to accurately generate relatively global ratings of everyday functioning, generating valid ratings of specific aspects of everyday functioning is more challenging. Thus, despite rating global aspects of everyday functioning with reasonable validity, breaking down “money management” into a series of highly detailed questions led to ratings that did not converge well with scores on tests measuring these same abilities. Friend or relative informants generated overall ratings of a limited set of the financial abilities that were correlated with all four of the performance-based ability variables, albeit with minimal shared variance. This is in contrast to our previous findings with global scores, where friend or relative informants provided ratings that did not converge with patient self-reports, patient test scores, or clinician ratings (Sabbag et al., 2011; 2012). These previous findings were based on much more global ratings of the ability to function in everyday settings and not based on detailed ratings of specific functional skills.

These findings suggest that clinicians who are attentive to the functioning of their patients may be able to form generally accurate “summary” impressions of functioning in different functional domains. Further, financial skills deficits may be identified when a crisis occurs, such as failure to pay bills such as rent. However, these ratings may be less accurate when high levels of detail are demanded. In contrast, friends or relatives may be able to accurately observe discrete behaviors, but are challenged when summary judgments, involving normative inferences, are required. Thus, different informants may be differentially informative across rating domains. This finding has potentially important implications for patient treatment and underscores the need for objective, robust and efficient measures of functional performance.

There are some limitations in this study. First, the ILSS ratings required on the part of patients simply address whether they have performed the act in question. There is no attempt in the scale to have them self-report their competence in performing those acts. It may be that some patients who actually did not perform the skilled acts, regardless of the reason, had the ability to do so or had a history of doing so. Previous studies asking patients how well they could perform financially skilled acts in general have found minimal correlations with ability variables. Second, we used a single informant per patient in this study, and as noted above, there may be variation in the extent to which different informants could observe the patient performing the skilled acts in question. There were also fewer clinician informants; in a previous paper we found that despite the smaller sample size there was greater convergence between clinician ratings of more global functional abilities and patients’ performance-based ability scores than between the ratings of other informants and patients’ performance (Sabbag et al. 2011).

These findings lead to two general conclusions. First, detailed assessments and highly specific questions do not guarantee increased validity of ratings of functioning, whether they are patient self-report, interviewer judgments, or the reports of various informants. The ILSS asks highly detailed questions which generated interviewer judgments and self-reports. These judgments and reports were minimally correlated with performance on tests of the abilities rated. Second, high-contact clinician informants who can generate more valid ratings of more general aspects of functioning (overall ratings of everyday functioning) do not necessarily generate similarly better ratings when asked to rate highly molecular skills. When the total scores were examined, a slightly better pattern of results was found, but there was no significant improvement on the part of clinicians. These data suggest that functional status rating scales targeted at clinicians may have to strike a balance between global impressions and coverage of component skills.

There are important implications of these findings for assessment of the capability and competence of patients to perform financial management tasks. Direct assessment of financial management skills should be considered as a standard practice for determining financial competence. Performing the UPSA-B Finance subtest and the EFB is no more time consuming than having the patient and an informant complete the ILSS Money Management subsection and then having a clinician compare their performance. Also, as noted above, many friend or relative informants stated that they could not provide valid ratings of the performance of the patients in financial domains. In contrast, all of the patients in the study had complete data on the performance-based measures of financial skills, suggesting that performance-based measures are easier to obtain than informant judgments. While it could be argued that the performance-based measures are actually the source of poor validity, it has been shown repeatedly that performance on these direct assessments of functional skills is correlated with real world outcomes in domains of independent living and vocational outcomes (Bowie et al., 2008). The high levels of validity and ease of administration of these tests suggest that they should be more widely adopted to increase the validity of critical competence decisions and areas for patient treatment. These assessments could also be relied on by case managers when making judgments about placement after discharge.