Factors Influencing Self-Assessment of Cognition and Functioning in Schizophrenia: Implications For Treatment Studies

Abstract

Awareness of illness is a major factor in schizophrenia and extends into unawareness of cognitive and functional deficits. This unawareness of functional limitations has been shown to be influenced by several different predictive factors, including greater impairment and less severe depression. As treatment efforts are aimed at reducing cognitive deficits, discovery of the most efficient assessment strategies for detection of cognitive and functional changes is critical. In this study, we collected systematic assessments from high contact clinicians focusing on their impressions of the cognitive deficits and everyday functioning in a sample of 169 community dwelling patients with schizophrenia. The patients provided self-report on those same rating scales, as well as self-reporting their depression and performing an assessment of cognitive performance and functional skills. There was essentially no correlation between patients' self reports of their cognitive performance and functional skills and either clinician ratings of these skills or the results of the performance-based assessments. In contrast, clinician reports of cognitive impairments and everyday functioning were correlated with objective performance data. Depression on the part of patients was associated with ratings of functioning that were both more impaired and more congruent with clinician impressions, while overall patients reported less impairment than clinicians. These results underscore the limitations of self reported cognitive functioning even with structured rating scales. Concurrently, clinicians provided ratings of cognitive performance that were related to scores on objective tests, even though they were unaware of the results of those assessments.

Cognitive impairment in schizophrenia has become an area of interest in research due to its impact on patients' everyday functioning and subsequent perpetuation of disability. Deficits in social (Wiersma et al., 2000), residential (Auslander et al., 2001) and occupational (Ho et al., 1997) domains are observed in patients with schizophrenia despite successful treatment of the active phase of the illness. Cognitive impairments are one of several factors known to predict impaired everyday outcomes. Current pharmacologic treatments have a positive effect on psychotic symptoms but a limited effect on cognition. Therefore, it is a major priority to develop treatment alternatives for cognitive deficits in schizophrenia.

The assessment of cognitive impairment has been accomplished with performance-based neuropsychological (NP) tests and functional capacity (FC) measures. Due to their potential to predict real-world functioning, both tools have been accepted for use in treatment studies and will be used for registration studies (Green et al., 2011). For the approval of a cognitive-enhancing drug for schizophrenia, the Food and Drug Administration (FDA) is promoting the development of a variety of methods for assessing treatment related outcomes in cognition. As a result, interview-based measures have been developed to measure cognitive functioning and cognitive changes with treatment. The Cognitive Assessment Interview (CAI) is a 10-item instrument that assess 6 of the 7 MATRICS cognitive domains, it can be completed by a patient, informant and rater, it has been recently validated and produces reliable ratings of cognitive functioning that were correlated with functional outcome (Ventura et al., 2013). The Schizophrenia Cognition Rating Scale (SCoRS; Keefe et al., 2006) is a similar 20-question interview with both the patient and an informant and has been successfully used to measure cognition outcomes after treatment with atypical antipsychotics (Harvey, Ogasa, et al., 2011). However, there are some intrinsic limitations with interview-based assessments, including both self-reports and caregiver assessments. First, patients can be poor informants because the role played by insight in cognition has not been well examined in interview-based assessments and is likely that interview-based methods require considerable patient insight (Medalia and Thyssen, 2010). Second, remotely delivered interventions may lead to patients not being seen in person, particularly in purely clinical scenarios. Third, very little is known about what influence caregiver characteristics have on their reports about the cognitive functioning of the patient (Sabbag et al., 2011), although there are clear variations in the validity of those reports. Last, it may be also be difficult to see non-clinician informants in person and therefore it will be difficult to evaluate their own cognitive abilities and emotional states. Some of these difficulties can be surmounted if high-contact clinicians generate the ratings, as there is considerable evidence that their ratings have considerably more validity than those of other informants (Bowie et al., 2007; Sabbag et al., 2011; 2012), including both the patient and non-caregiver friend or relative informants.

Lack of insight is a core feature of schizophrenia and studies have also shown that patients are unaware of their cognitive deficits (Medalia and Thysen., 2010). Previous studies demonstrated that self-reports of cognition are not well correlated with performance-based assessments (Keefe et al., 2006; Green et al., 2008; 2011). Similar problems may exist when patients are asked to report their own levels of everyday functioning (Bowie et al., 2007; Patterson et al, 1997; McKibbin et al, 2004). On the other hand, a recent study (Sabbag et al., 2011) has indicated that clinician ratings of the severity of real-world impairment were more strongly correlated with the results of performance-based assessments than impairment ratings generated by friends, relatives, other caregivers or patients themselves. Further, interview based assessments considering all possible informants, with a final judgment rendered by the interviewer, have been shown to be suitably correlated with the results of performance-based assessments (Keefe et al., 2006; Ventura et al., 2013) and to be sensitive to the effects of potentially cognitively enhancing agents (Harvey, Ogasa, et al., 2011).

It is still tempting to consider the use of self-reports in clinical trials, as there is some evidence that many people with schizophrenia may have trouble identifying potential informants (Patterson et al., 1996). In addition, it may be logistically challenging for the same interviewer to collect data from patients and informants. Since clinician informants seem to generate valid reports (Bowie et al., 2007; Sabbag et al., 2011), even without training, it is possible that collection of questionnaire data from these clinician informants would be suitable to use as outcome measures. As a result, it would be important to understand the convergence between clinician informant ratings of cognitive performance and performance-based assessments of patients, as well as understanding the relationship between real-world outcomes and these informant reports of cognitive functioning.

We present a systematic study of the validity of self-reported versus clinician reported cognitive performance. All clinicians provided mailed-in questionnaire responses, with no in-person assessments performed. These clinicians received no training in completion of the questionnaires. Clinicians were asked to rate patient's cognitive performance and everyday functioning using the Cognitive Assessment Interview (CAI, Ventura et al., 2013) and the Specific Levels of Function (SLOF; Schneider and Streuening, 1983) respectively. Patients were seen in person and reported their cognitive functioning with a structured interview focusing on the same forms completed by mail by clinicians (CAI) and completed the SLOF as a questionnaire. Patients were also assessed with performance based-assessments of functional capacity using the UCSD Performance-based Skills Assessment, brief version (UPSA-B; Mausbach et al., 2007) and neurocognition utilizing a modified version of the MATRICS Consensus Cognitive Battery (MCCB; Nuechterlein et al., 2008).

Methods

Participants

These analyses are the first publication for the VALERO (Validation of Everyday Real World Outcomes; Harvey, Raykov, et al., 2011) study (phase 2), aimed at identifying the determinants of self-assessment of cognition and everyday functioning in people with schizophrenia. The study participants were patients (n=207) with schizophrenia or schizoaffective disorder who were receiving treatment at one of three different outpatient service delivery systems, one in Miami, one in Atlanta and one in San Diego. In addition, high contact clinician informants (case manager, psychiatrist, therapist or residential facility manager) were interviewed concerning the functioning of each of the patients. All of these research participants (patients and informants) provided signed, informed consent, and this research study was approved by local IRBs in Miami, Atlanta and San Diego. In Atlanta, patients were either recruited at a psychiatric rehabilitation program (Skyland Trail); in Miami they were recruited from the general outpatient population of the University of Miami Medical Center. The San Diego patients were recruited from the UCSD Outpatient Psychiatric Services clinic, a large public mental health clinic, other local community clinics, and by word of mouth.

All patients were administered a structured diagnostic interview, the Mini International Neuropsychiatric Interview, 6^th Edition (MINI; Sheehan et al., 1998) by a trained interviewer. All diagnoses were subjected to a consensus procedure at each local site. Inclusion criteria were a diagnosis of schizophrenia, any subtype, or schizoaffective disorder. Patients were excluded for a history of traumatic brain injury with unconsciousness >10 minutes, brain disease such as seizure disorder or neurodegenerative condition, or the presence of another DSM-IV diagnosis that would exclude the diagnosis of schizophrenia. None of the patients were experiencing their first psychotic episode. Comorbid substance use disorders were not an exclusion criterion, in order to capture a broad array of patients, but patients who were unable to comply with assessments and appeared intoxicated were rescheduled. Inpatients were not recruited, but patients resided in a wide array of unsupported, supported, or supervised residential facilities. Informants were not screened for psychopathology or substance abuse. Patients characterized their racial status (Caucasian, African American, Other) and stated whether they were of Hispanic origin, according to the criteria of the US NIMH.

Procedure

All patients were examined with an assessment of NP and FC performance. They also provided self-reports of interpersonal functioning, everyday activities, and vocational skills by completion of the SLOF. Patients also rated their cognitive functioning using a structured procedure. High contact clinicians generated ratings of cognition and everyday functioning with the clinician versions of the cognition and functional status rating scales which were sent to them by mail or delivered in person. Ratings of symptoms as measured by the Positive and Negative Syndrome Scale (PANSS, Kay, 1991) were generated by a rater who was not aware of patient reports, patient performance, or clinician reports.

Performance-based assessment

Neurocognition

We examined NP performance with a modified version of the MATRICS consensus cognitive battery (MCCB). For this study, we did not include the social cognition measure from the MCCB, the Mayer–Salovey–Caruso Emotional Intelligence Test—Managing Emotions, because there are several reasons that social cognition measures may have a different relationship with everyday outcomes compared to neurocognitive measures. This minor modification of the MCCB makes the results similar to previous work, such as our own, that did not include social cognition measures (e.g., Bowie et al.,2008). We calculated a cognitive composite score, an average of 9 age-corrected T-scores based on the MCCB normative program.

Functional Capacity

We administered the brief version of the UCSD Performance-based Skills Assessment (UPSA-B; Mausbach et al., 2007) as our functional capacity measure. The UPSA-B is a measure of functional capacity in which patients are asked to perform everyday tasks related to communication and finances. During the Communication subtest, participants role-play exercises using an unplugged telephone (e.g., making an emergency call; dialing a number from memory; calling to reschedule a doctor's appointment). For the Finance subtest, participants count change, read a utility bill and write and record a check for the bill. The UPSA-B requires approximately 10 minutes, and raw scores are converted into a total score ranging from 0-100, with higher scores indicating better functional capacity.

Real-World Functional Outcomes

As we previously reported, the initial phase of the VALERO study indicated that everyday functioning rated with multiple rating scales was related to NP and FC performance (Harvey, Raykov, et al., 2011). The best rating scale of those examined, on the basis of its optimal individual correlation with the ability measures, was the Specific Levels of Functioning (SLOF; Schneider and Struening, 1983) scale. This scale is a 43 item, self or informant rated report of a patient's behavior and functioning on the following domains: Interpersonal Relationships (e.g., initiating, accepting and maintaining social contacts, effectively communicating), Participation in Community and Household Activities (shopping, using the telephone, paying bills, use of leisure time, use of public transportation), and Work Skills (e.g., employable skills, level of supervision required to complete tasks, ability to stay on task, completes tasks, punctuality). Note that the Work Skills domain comprises behaviors important for vocational performance, but is not a rating of behavior during employment. The SLOF's Physical Functioning, Self-Care and Socially Acceptable Behavior subscales were not used in the VALERO study on the basis of suggestions of the original RAND panel.

For 38 of the patients, their clinicians stated that they were unable to report on some elements of the SLOF Community Activities subscale. Rather than impute scores based on means or some other procedure, we conducted these analyses with the remaining 169 subjects.

Self-reported and Interviewer Rated Cognitive Functioning

We had patients and the high contact clinicians rate the Cognitive Assessment Inventory (CAI; Ventura, et al., 2013). This is a 10 item instrument that asks the person making the judgments about cognition to rate severity of impairments in a variety of cognitive domains. These domains are aimed at the dimensions of cognitive impairment typically studied in schizophrenia. Ratings are generated with on a 6-point (1-6) scale, with scores of 1 reflecting the least impairment. For the purposes of theses analyses, we calculated a total score for the clinician and self-reported scores on the CAI. The patient was asked the questions in a standard interview format. Clinicians were simply asked to complete the form on their own using the same instructions that the interviewer provided to the patients.

Psychopathology Measures

We assessed self-reported depressive symptoms with the Beck Depression Inventory-II (BDI-II; Beck et al., 1996), a 21-item questionnaire. Participants rated each of the 21 items on a scale from 0-4. A total depressive symptoms score was created by summing the 21 items (range 0 to 84).

Severity of psychotic and negative symptoms was assessed using the Positive and Negative Syndrome Scale (PANSS, Kay, 1991). This 30-item scale contains items measuring positive symptoms negative symptoms, and general aspects of psychopathology and was completed after a structured interview by a trained interviewer. We present means scores for the PANSS subscales, derived by factor analysis (Marder et al., 1997) for information purposes only.

Results

Table 1 presents demographic information on the patient participants and Table 2 presents the scores on the SLOF and CAI, for both self-reports and interviewer judgments and for all of the predictor variables, including the UPSA-B, modified MCCB, and the BDI-II. As can be seen in Table 1, the majority of patients were male and about half were Caucasian. One third were African American and one quarter had Hispanic ethnic background. The average age was mid 40's and the average years of education was slightly over high school. Clinicians came from three general categories. There were 159 case managers or social workers, 36 treating psychiatrists, and 12 residential facility staff members.

Table 1. Demographic and Clinical Variables in the VALERO II Patient Sample with Schizophrenia.

	n=214
Characteristic	n	%
Male	140	65.4
Race
Caucasian	117	54.7
African American	77	36
Other	20	9.3
Hispanic Ethnicity	50	23.4
	Mean	SD
Age (Years)	41.0	12.4
Education	12.3	2.2

Table 2. Functioning, Cognition, and Mood Symptoms.

Cognition and Everyday Outcomes as Reported by Patients and Clinicians
Variable	Clinician Ratings		Patient Reports		t	p
	M	SD	M	SD
SLOF Interpersonal Functions	22.45	5.91	24.69	6.62	3.51	.001
SLOF Activities Subscale	44.77	10.41	48.81	9.02	3.90	.001
SLOF Vocational Subscale	20.13	5.29	23.65	5.09	6.50	.001
Cognitive Assessment Inventory	30.38	11.24	21.78	8.66	8.15	.001
Cognition and Symptoms
Cognition Composite Score	37.41	8.71
UPSA-B Score	70.70	15.94
BDI-II	15.33	11.67
Negative Symptoms (PANSS)	16.61	6.64
Positive Symptoms (PANSS)	15.67	5.51

Between group comparisons on Ratings

For the first analysis, we computed paired t-tests comparing the self-reported and clinician rated scores on real-world functioning and cognition. Initial tests of the normality of distributions revealed that all of the variables used in the correlational analyses were normally distributed, with none of the variables having significant kurtosis or skew. For all four tests, the clinician ratings reflected more severe impairment than reported by the patients. Results of these t-tests are presented along with the mean scores in Table 2.

Correlational Analyses

Next, we computed Pearson correlations between patient reported and clinician rated SLOF scores and the CAI. These correlations are presented in Table 3. All four of these correlations were non-significant and the largest of these correlations accounts for approximately 1% of the variance. The next set of correlations was between the performance-based variables (Modified MCCB and UPSA-B), and depression reported by the patients and the clinician and patient reported scores on the SLOF and the CAI.

Table 3. Correlational Analyses.

Correlations Between Self-reported vs Clinician Rated Functioning Patient Reported
Clinician Rated
SLOF Interpersonal Functions	-.10
SLOF Activities Subscale	-.01
SLOF Vocational Subscale	-.02
Cognitive Assessment Inventory	.12
Correlations Between Self-reported vs Clinician Rated Functioning and Performance and Symptom Measures
	MCCB Total Score	UPSA-B	BDI Total
Clinician Rated
SLOF Interpersonal Functions	.16	.16	.00
SLOF Activities Subscale	.29*	.39*	.04
SLOF Vocational Subscale	.23*	.28*	.07
Cognitive Assessment Inventory	.41*	.31*	.01
Self-Reported
SLOF Interpersonal Functions	-.08	-.02	-.40*
SLOF Activities Subscale	.16	.00	-.26*
SLOF Vocational Subscale	.08	.12	-.34*
Cognitive Assessment Inventory	.12	.11	.44*

Clinician rated scores on the SLOF work and activities subscales were correlated with both MCCB scores and with performance on the UPSA-B. Further, clinician ratings on the CAI were also correlated with UPSA-B scores and MCCB scores. Finally, clinician ratings on the interpersonal functioning subscale on the SLOF were not correlated with the performance based measures or clinician ratings of cognition generated with the CAI. BDI scores were not correlated with any of the clinician rated SLOF subscales or clinician ratings on the CAI.

Patient reported functioning manifested a very different pattern of correlations with the predictor variables. There were no statistically significant correlations between patient performance on the MCCB or the UPSA-B and any of their self-reports of everyday functioning or cognition as evaluated by the CAI. In contrast, all four aspects of patients' self-reported functioning were significantly correlated with patient-reported BDI scores, with higher scores predicting lower self-reported functioning. Thus, patients' ratings of their functioning were consistently correlated with their subjective mood symptoms but not with their objective ability scores.

We computed correlations between BDI scores and the differences between patient reports and clinician ratings on the SLOF and CAI. For all four variables, higher BDI scores were correlated with less overestimation of functioning, all r>.26, all p<.001. Further, lower scores on the MCCB composite score were correlated greater overestimation of functioning for work and everyday activities, both r>.25, both p<.001, and lower scores on the MCCB were correlated with greater over-estimation of cognitive performance compared to clinician ratings, r=-.30, p<001.

Discussion

In our study we found that people with schizophrenia, on average, overestimate their cognitive and functional ability, when clinician ratings of their functioning are used as a reference point. There were no variables on which self-reported functioning was correlated with either clinician ratings of those same variables or with objective measures of ability. In contrast, self-reported depression was consistently correlated with all aspects of self-reported functioning. Clinician ratings of everyday functioning were correlated with performance on NP tests and the UPSA-B and unrelated to patient reported depression. Patient reported depression was also not associated with performance on either NP tests or functional capacity measures. The correlations between the performance-based assessments and clinician ratings of everyday functioning provide evidence of convergence between highly relevant sources of information, because the clinicians were not aware of these scores when they made their ratings. Clinician-rated CAI scores were also correlated with clinician ratings of everyday functioning, but were generated by the same individual.

There are some limitations of these data. Patients had to be able to complete a detailed assessment and not all patients with schizophrenia are likely able to do this. Patients without a high contact clinician were not enrolled. Finally, we used self reports of depression. A schizophrenia specific depression rating scale might have yielded different results.

These data raise serious questions about whether patients' reports provide information with any useful concordance to performance-based assessments. There is so little overlap between self-reported and objectively measured cognitive performance that we see that they cannot be seen to be measuring the same content. These results have substantial implications for future research on clinical interventions. Performance-based measures of cognitive functioning and functional capacity should remain the gold-standard of functional assessment in short term treatment studies. Clinician ratings seem to have high levels of validity and little bias, but any study using these outcomes would need to require that all patients have such an informant and that they have similar levels of familiarity with the patients.

Depression had a unique relationship with self-reported functioning in this study as it has had in previous research on depression and self assessment in schizophrenia (Bowie et al., 2007; Sabbag et al., 2012). Depression severity correlated with less overestimation of functioning by the patients, but seems to reflect a source of response bias that is markedly affecting the likelihood of accurate reporting of abilities. Clinicians did not rate more depressed patients as more cognitively or functionally impaired; this tendency was validated by the results of the performance-based assessments which were related to clinician ratings. Most likely the substantial signal for cognition and functioning originating from schizophrenia was more salient to clinicians than depressed mood, which they may not assess thoroughly or appreciate adequately.

Treatment of depression might have one of several impacts. Improving depression might reduce the response bias toward reporting everyday functioning as a function of mood state. It is not clear that accuracy of assessment would be influenced for all patients, as those patients whose depression was more severe were less likely to overestimate their functioning compared to the reference point of clinician ratings of functioning. As there is very little data available on treating depression in schizophrenia and essentially none on cognitive responses to treatment of depression, these questions will await later research. Further, the impact of depression on everyday functioning may be a function of the informant (Sabbag et al., 2011), as our previous work with consensus ratings found that depression was found to impact everyday functioning and our work relying entirely on clinician informants has found minimal impacts in zero-order correlations. Finally, direct treatment of metacognitive deficits might reduce self-assessment bias as well.