Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2019 Sep 1.
Published in final edited form as: Schizophr Res. 2018 Apr 17;199:75–82. doi: 10.1016/j.schres.2018.04.015

Self Assessment of Social Cognitive Ability in Schizophrenia: Association with Social Cognitive Test Performance, Informant Assessments of Social Cognitive Ability, and Everyday Outcomes

Juliet Silberstein 1, Amy E Pinkham 2,3, David L Penn 4,5, Philip D Harvey 1,6
PMCID: PMC6151277  NIHMSID: NIHMS961762  PMID: 29673732

Abstract

Background

Impairments in self-assessment are common in people with schizophrenia and impairments in self assessment of cognitive ability have been found to predict impaired functional outcome. In this study, we examined self-assessment of social cognitive ability and related them to assessments of social cognition provided by informants, to performance on tests of social cognition, and to everyday outcomes. The difference between self-reported social cognition and informant ratings was used to predict everyday functioning.

Methods

People with schizophrenia (n=135) performed 8 different tests of social cognition. They were asked to rate their social cognitive abilities on the Observable Social Cognition Rating Scale (OSCARs). High contact informants also rated social cognitive ability and everyday outcomes, while unaware of the patients’ social cognitive performance and self-assessments. Social competence was measured with a performance-based assessment and clinical ratings of negative symptoms were also performed.

Results

Patient reports of their social cognitive abilities were uncorrelated with performance on social cognitive tests and with three of the four domains of functional outcomes. Differences between self reported and informant rated social cognitive ability predicted impaired everyday functioning across all four functional domains. This difference score predicted disability even when the influences of social cognitive performance, social competence, and negative symptoms were considered.

Implications

Mis-estimation of social cognitive ability was an important predictor of social and nonsocial outcomes in schizophrenia compared to performance on social cognitive tests. These results suggest that consideration of self assessment is critical when attempting to evaluate the causes of disability and when trying to implement interventions targeting disability reduction.

Introduction

Everyday disability remains a reality for the majority of individuals with schizophrenia despite marked improvements in the treatment of positive symptoms with antipsychotic medications. Functional deficits are present across objective (e.g. living independently, maintaining employment, and building relationships) and subjective domains (e.g. quality of life, perceived illness burden; Brekke et al., 1993). Several factors have emerged as significant predictors of functional status in people with schizophrenia. Neurocognition, social cognition, the ability to perform everyday functional skills (i.e., functional capacity), and negative symptoms appear to be globally related to functioning and are more strongly correlated with outcomes than the severity of psychosis in most studies (Bowie et al., 2010; Bowie et al., 2008; Bowie et al., 2006; Fett et al., 2011; Green et al., 2000; Harvey et al., 2011; Tabares-Seisdedos et al., 2008). Yet, studies of the determinants of everyday functional deficits in schizophrenia have stalled at accounting for 50% or less of the variance in real world functioning (Bowie et al., 2010; Bowie et al., 2008; Bowie et al., 2006; Harvey et al., 2011). Improved life outcomes for individuals with schizophrenia, including work performance and independent living skills, hinges on furthering our understanding of the determinants of real world disability and identification of new treatment targets.

Meta-analyses suggest that social cognition more strongly predicts social outcomes than everyday activities (e.g. living independently), and neurocognition is often found to provide a minimal prediction of social outcomes (e.g. interpersonal relationships; Depp et al., 2012; Fett et al., 2011). Ultimately, understanding the association between different features of the illness and outcome domains will improve the ability of clinicians and clinical researchers to personalize treatment targets, such that the treatment of a person with social deficits may differ from treatments for a person with problems in vocational funcitoning.

A promising new lead in the search for transdiagnostic determinants of real-world functional outcome has emerged, introspective accuracy (IA). We define IA as the ability to accurately self-assess adequacy of performance on cognitive or social cognitive tasks (Fleming et al., 2010), to evaluate the general level of neurocognitive or social cognitive ability (Medalia and Thysen, 2010), and to evaluate the level of competence in the performance of functional skills (Hur et al., 2014; Metcalfe and Greene, 2007). In our view, IA overlaps with metacognition, defined as thinking about thinking, but is also distinct. IA can describe impairments that result from errors in the metacognitive process but can also be applied to many different domains in addition to cognitive performance, such as judgments of performance in social or other adaptively relevant situations (Harvey and Pinkham, 2015). IA is separable from metacognitive constructs such as clinical and cognitive insight in that both latter constructs focus on understanding clinical phenomena such as anomalous experiences and erroneous inferences (Beck et al., 2004; Riggs et al., 2012), whereas IA focuses on the self-awareness of levels of specific skills and abilities. IA is also distinct from the metacognitive domain of the Theory of Mind (ToM), the ability to infer the thoughts and emotions of others, in that Theory of Mind is other-focused (Dimaggio and Lysaker, 2010; Nelson et al., 1999). In contrast IA is self-focused (Koren et al., 2006). To provide an example, someone who inaccurately believes that they recognize emotions well might incorrectly perceive someone as angry (poor IA and poor ToM) and act accordingly. Conversely, an individual who acknowledges they have poor emotional recognition might first ask the person if they are angry before responding (good IA and poor ToM).

IA is not global by definition, meaning a patient might have insight in one domain and lack it in another (Yahav et al., 2011). Similar to other symptom domains, not every individual with schizophrenia presents with impaired self-assessment. Our initial work with individuals with schizophrenia has found IA impairments in about 50% of patients in the domains of neurocognitive performance, and functional abilities ( Bowie et al., 2007; Durand et al., 2015; Gould et eal., 2015; Harvey et al., 2012; Keefe et al., 2015; Riggs et al., 2012; Sabbag et al., 2011). Importantly, IA deficits across various domains have previously shown significant correlations with medication adherence, suicidality, everyday activities, vocational functioning, and social outcomes (Green et al., 2011; Holshausen et al., 2014; McKibbin et al., 2004; Patterson et al., 1997). Research from the VALERO II study suggested that impaired IA of neurocognitive ability was a more potent predictor of everyday functional deficits in social, vocational, and everyday activities domains than scores on performance-based measures of neurocognitive cognitive abilities and functional capacity (Gould et al., 2015). These findings are even more meaningful because of previous research suggesting that deficits in self-assessment can be targeted via psychotherapeutic interventions (Moritz et al., 2011; Moritz et al., 2014).

The current data comes from the final phase of the SCOPE study, which aimed to identify the best methods for assessing social cognition in individuals with schizophrenia (Pinkham et al., in press). This study expands the evaluation of IA and its functional significance to the domain of social cognition and everyday social functioning by examining the predictive association of discrepancies between informant ratings and self-reports of social cognitive abilities and four different domains of real-world everyday functioning: vocational functioning, everyday activities, interpersonal functioning, and socially acceptable behavior. We validated informant vs. self reports of social cognitive ability by correlating both of these reports and their differnce with performance on 8 different social cognitive tests. We also examined the convergence beween self assessment of social cognition and self assessment of everyday functioning. Our predictions were straightforward: we hypothesized that social cognitive IA, indexed by the difference between self-reported and informant rated social cognitive ability, would be a substantial predictor of impairments in interpersonal relationships and socially acceptable behavior. We expected that sel-reports of social cognitive ability and everyday functioning would be correlated with each other to a greater extent than these self reports were correlated with objective information obtained from social cognitive test performance and informant judgments of everyday functioning.

Method

Participants

Data collection occurred at three sites in this final phase of the SCOPE study (Pinkham et al., in press): The University of Texas at Dallas (UTD), The University of Miami Miller School of Medicine (UM), and The University of North Carolina at Chapel Hill (UNC). Participants were stable outpatients with diagnoses of schizophrenia or schizoaffective disorder (n=135). Only patients with a high contact informant were included in this study because previous evidence from earlier studies shows that such informants generate ratings with adequate validity (Sabbag et al., 2011). All informants whose data were used reported that they knew the patient “very well”. This lead to the exclusion of 73 patients who were reported on in the previous paper whose community informants indicated knowing the patient less than “very well”.

UTD patients were recruited from Metrocare Services, a non-profit mental health services provider organization in Dallas County, TX, and other area clinics. UM patient recruitment occurred at the Miami VA Medical Center and the Jackson Memorial Hospital-University of Miami Medical Center, and UNC patients were recruited from the Schizophrenia Treatment and Evaluation Program (STEP) in Carrboro, NC and the Clinical Research Unit (CRU) in Raleigh, NC. To be eligible, patients required a DSM-IV diagnosis of schizophrenia or schizoaffective disorder. Patients could not have any hospitalizations within the last two months and had to be on a stable medication regimen for a minimum of six weeks with no dose changes for a minimum of two weeks.

Clinical Symptom Ratings

The severity of positive and negative symptoms was rated with Positive and Negative Symptom Scale, a 30-item scale (Kay et al., 1987).

Social Cognition Measures

Observable Social Cognition Rating Scale (OSCARS)

Both patients and high contact informants completed the OSCARS. The OSCARS is an 8-item assessment of social cognition. Each OSCARS item is comprised of a question probing a social cognitive domain (theory of mind, emotional perception, cognitive rigidity, jumping to conclusions, and attributional style) followed by general example behaviors that reflect impairment in that domain. Participants ranked their abilities in each item on a 7-point scale with higher ratings indicating greater impairment. An additional question assessed the impression of global social cognitive impairment. The global ratings utilized a 10-point scale, again higher ratings indicated greater impairment (range 1–10). The patient was asked the questions in a standard interview format. Informants completed the form by themselves using the same instructions that the interviewer provided to the patients to rate the patient’s level of impairment. The OSCARS was administered at the baseline assessment only.

The OSCARs has previously been validated in a comprehensive study. In that study, the internal consistency of the OSCARS was .80 in patients, with test-retest reliability of the items ranging from .50 to .70. Informant ratings on the OSCARS were found to correlate with several different measures of social cognition and social functioning, including both performance-based and rating-scale measures. We examined the correlation of OSCARS total scores and the global ratings. Correlations between informant total scores and global ratings was r=.84, p<.001 and the correlation between the global rating and the total score for patients was r=.67, p<.001). The global score additionally showed significant correlations to all 8-items (p<.05). As such, this analysis used informant and self-reported global ratings as our outcome variable.

The performance-based social cognition tasks were previously reported on in terms of their psychometric properties, so they will simply be listed here.

Attributional Style/Bias

The Intentional Bias Task (IBT)

The IBT assesses the tendency to attribute intentionality to the actions of others (Rosset, 2008).

Social Perception

The Mini Profile of Nonverbal Sensitivity (MiniPONS)

The MiniPONS is a multi-channel test of accuracy in decoding interpersonal cues (face, body, and voice tone) (Bänziger et al., 2011).

The Social Attribution Task – Multiple Choice version (SAT-MC)

This task assesses social inference via perception of animations. Participants viewed a short animation of geometric shapes enacting a social drama and answered questions about the social themes present in the animation (Bell et al., 2010).

Emotion Processing

Penn Emotion Recognition Test (ER-40)

The ER-40 assesses facial emotion recognition abilities for 4 basic emotions (i.e. happiness, sadness, anger, or fear) and neutral expressions (Kohler et al., 2003).

Bell Lysaker Emotion Recognition Task (BLERT)

The BLERT measures recognition of seven emotional states: happiness, sadness, fear, disgust, surprise, anger, or no emotion. Participants identified the emotion shown in 21 videos of a male actor providing dynamic facial, vocal-tonal, and upper-body movement cues (Bryson et al., 1997).

Mental State Attribution

Reading the Mind in the Eyes Test (Eyes)

Eyes measures the capacity to understand mental states of others from expressions in the eye region of the face (Baron-Cohen et al., 2001).

The Awareness of Social Inferences Test, Part III (TASIT)

TASIT assesses detection of lies and sarcasm by asking participants to view short videos of social interactions and to answer questions about the intentions, beliefs, and meanings of the speakers (McDonald et al., 2003).

Hinting Task

Hinting examines the ability to infer the true intent of indirect speech (Corcoran et al., 1995).

Functional Measures

Social Skills Performance Assessment (SSPA)

Social competence was assessed with the SSPA, a role-play measure in which participants were asked to initiate and maintain a conversation in 2 social situations: meeting a new neighbor and negotiating with a landlord to fix a leak. Roleplays were audiotaped and coded by an expert rater blind to diagnosis on the following variables: interest, fluency, clarity, focus, overall abilities, and social appropriateness. The landlord role-play also coded for negotiation ability and persistence. The mean score across both role-plays was used as the dependent measure and could range from 1 to 5 (Patterson et al., 2001). We have previously reported that performance on the 8 social cognitive tests shared 34% variance with SSPA, so we do not repeat those analyses herein (Pinkham et al., in press).

Specific Level of Functioning (SLOF)

Real-world functional outcome was assessed via the 31-item version of the SLOF, a self- or informant-rated measure of functioning in Interpersonal Relationships (e.g., initiating, accepting and maintaining social contacts, effectively communicating), Social Acceptability (e.g. appropriateness of verbal and physical behavior), Participation in Community and Household Activities (e.g. shopping, using the telephone, paying bills, use of leisure time, use of public transportation), and Work Skills (e.g., employable skills, level of supervision required to complete tasks, ability to stay on task, completes tasks, punctuality). The SLOF’s Physical Functioning and Self-Care were not assessed. Patients were interviewed by a rater and informants completed the scale as a questionnaire, in line with our previous use of this scale in multiple studies (Bowie eet al., 2006; Harvey et al., 2011; Pinkham et al., 2016). Ratings for each item were made on a 1–5 point scale with higher scores indicating better functioning. An average item score across each subscale was used as the dependent variable (Schneider and Struening, 1983).

Procedures

Participants completed two study visits: baseline and a retest assessment conducted 2–4 weeks after the initial visit (mean interval=16.69 days). The OSCARS assessment was administered one time during visit 1 and was not repeated on visit 2. At visit 1, all participants provided informed consent and completed the performance- and interview-based social cognitive and functional outcome measures. For patients, visit 1 also included diagnostic assessment and an evaluation of symptom severity using the Positive and Negative Syndrome Scale (Kay et al., 1992). Diagnostic and symptom raters were trained to reliability using established procedures at each site. All informants received no training and had no information about any performance based, clinically rated, or self-reported data on the participants.

Data analyses

Impairments in IA were calculated by the difference between high-contact informants’ ratings of r global social cognition and patients’ self-rating of their global social cognition on the OSCARS and (maximum range: −9 to 9), wherein a negative discrepancy score indicated a rating of one’s social cognitive ability as being better than the informant ratings and a positive discrepancy score indicated. For the purposes of examining possible IA for everyday functioning, we calculated a difference score for all four SLOF subscales, subtracting patient scores from informant reports, such that negative scores again reflected the patients rating their functioning as better compared to informant ratings.

We aimed to determine the relative functional significance of IA. Pearson correlations were used to assess the relative strength of correlations between four SLOF functional outcomes domains (interpersonal, social acceptability, everyday activities, vocational) and a priori selected predictors: OSCARS ratings (informant, self-report, and IA), social cognitive test performance, a performance-based measure of social competence (SSPA), and negative symptoms. In a further test of IA, we also correlated the discrepancy scores between informant reported everyday functioning and self-reported everyday functioning on the 4 SLOF subscales with thse same predictor variables.

Next, we computed a set of preliminary regression analyses to identify whether self-rated or informant-rated OSCARS scores significantly predicted everyday functioning ratings of each of the four SLOF subscales. Subsequent sets of regression analysis were limited to the OSCARs ratings (self-reported or informant-reported) that showed significance. These analyses were computed with separate simultaneous entry regressions, predicting each of the four SLOF subscales. Our second set of regression analyses examined the contributions of informant-rated OSCARS scores and OSCARs difference scores to the prediction of the four SLOF subscales. In each analysis, we controlled for other possible predictors of everyday outcomes through blocked entry analysis.

Results

Table 1 presents the characteristics of the patients with informants. The patients were in theirearly 40’s on average, with slightly less education than their mothers. WRAT reading scores were close to the average range, consistent with their years of education. The sample was more than half Caucasian and over 10% of the sample was Hispanic in ethnicity.

Table 1.

Descriptive Information on the Sample of patients with high contact informants (n=135)

Variable Mean SD
Age 40.78 11.75
Years of education 13.36 2.57
Mother’s years of education 14.03 3.71
WRAT Standard Score 97.73 14.59
WASI Vocabulary t score 44.70 13.86
% male 63 %
Caucasian 53 %
Latino 11
Open in a new tab

Scores for clinician ratings of everyday functioning and social cognition, as well as self-reported social cognitive abilities, are presented in Table 2. Also presented are scores on the SSPA and the Marder negative symptom factor. Also shown are the discrepancy scores for everyday functioning ratings between self report and informant reports. The range of scores for the OSCARS was similar across information sources and both showed normal distribution, with a range of 1–9 for informants and 1–10 for patients. The Pearson correlation between informant rated and self-reported OSCARS scores was r=.33, p<.0001. A paired t-test found that self reported scores on the OSCARS did not differ significantly between the informants and the patients, t(128)=1.75, p=.13.

Table 2.

Scores on Ratings of Everyday Functioning and Predictor variables

Variable Mean SD Range
SLOF Interpersonal 3.57 0.90 1–5a,b
SLOF Social Acceptability 4.44 0.58 2.67–5a,b
SLOF Everyday Activities 4.45 0.78 2.58–5a,b
SLOF Vocational Functioning 3.75 1.00 1–5a,b
SLOF Interpersonal Difference −.40 1.02 −3–3.14c
SLOF Social Acceptability Difference −.24 0.60 −2.33–1.67 c
SLOF Everyday Functioning Difference −.24 1.09 −2.50–3.67 c
SLOF Vocational Difference −.30 0.97 −3.27–2.55 c
OSCARS Self-Reported 4.41 2.37 1–10d
OSCARS Informant Rated 3.99 2.35 1–9d
OSCARS Difference 0.42 2.72 −6–7 c
SSPA total Score 4.23 0.48 2.75–5b
Negative Symptoms 12.98 5.76 7–32d
Open in a new tab

Notes.

a

Mean item score

b

Higher scores reflect less impairment

c

Negative scores reflect patient ratings of less impairment

d

Higher scores reflect more impairment

Supplemental Table 1 presents the scores for the social cognitive variables in the study. In order to validate the association between social cognitive test performance and self-reported vs informant rated social cognitive abilities, as well as their difference we performed three regression analyses. In each, we regressed all 8 social cognition variables on global OSCARs score generated that information source. We chose this global strategy because of our previous findings that the social cognitive tests in the previous SCOPE study constituted a single factor, even when tested against multi-factorial models (Browne et al., 2016). For the informant global OSCARs score, the overall analysis was significant, F(7,127)=4.48, p<.001, R2=.15. For the patient self-report global OSCARs score, the overall analysis was not significant, F(7,127)=1.56, p=.19, p<.001, R2=.05. For the self-reported OSCARS global score, there were no social cognitive performance-based variables that manifested a statistically significant relationship with the self-report measure, all t<1.7, all p>.10. For the difference score, the overall analysis was not significant, F(7,127)=1.60, p=.18, p<.001, R2=.06. There were again no social cognitive performance-based variables that manifested a statistically significant relationship with the discrepancy score measure, all t<1.7, all p>.10. Thus, performance-based measures of social cognitive ability correlate with informant ratings of social cognitive ability but not with self reports or the difference between self reports and informant ratings of social cognitive ability.

The distribution of social cognitive discrepancy scores, displayed a normal distribution (range −7 to 7). Nearly a quarter (22.9%) of patients overestimated their global social cognition by two points or more (out of a 10-point scale) compared to high-contact informant ratings, and 19.4% underestimated their social cognition by two points or more Further, the discrepancy scores for SLOF subscales were also normal in their distributions, with the mean scores reflecting slightly higher reports of everyday functioning by patients compared to informant reports.

Correlations between everyday functioning and self-reported and informant rated social cognition, difference scores for social cognition, difference scores for everyday functioning, social competence, and negative symptoms are presented in Table 3. The difference between self-reported and informant-rated OSCARS scores was correlated with four out of four SLOF subscales, such that self reports of better social cognitive ability compared to informant impressions was associated with lower scores on informant ratings of everyday functioning. The difference between self-reported and informant-rated OSCARS scores was also significantly correlated with 3 of the SLOF difference scores as well. These correlations suggested that rating one’s social cognitive ability as better than how it was rated by the informant was related to rating one’s everyday functional ability as better as well. Informant-rated OSCARS scores also correlated significantly with four out of four SLOF subscales (all r −.63 to −.38, p<.001). However, self-reported OSCARS scores, social competence (SSPA), and negative symptoms all correlated with only one out of four SLOF subscales. Poorer performance on a test of social competence was associated with patients rating themselves as more capable than their informants saw them in domains of interpersonal and vocational functioning. Negative symptoms were not related to differences between patients and informants in reports of everyday functional abilities.

Table 3.

Correlations between Everyday Functional Outcomes and Multiple Potential Predictors: Self Reported and Informant Rated Social Cognition, Social Competence, and Negative Symptoms

Predictor Variable SLOF Subscale
Informant Ratings Discrepancy Scores
Interpersonal
Functioning		 Social
Acceptability		 Everyday
Activities		 Vocational
Functioning		 Interpersonal
Functioning		 Social
Acceptability		 Everyday
Activities		 Vocational
Functioning
Informant OSCARs −.53*** −.63*** −.38*** −.63*** .21* .13 .11 .08
Self report OSCARS −.15 −.18* −.11 −.10 .19* .13 .08 .23*
IA (OSCARS Difference) −.32*** −.39*** −.23** −.45*** .42*** .39*** .16 .47***
SSPA 0.14 −.01 −.01 .32*** −.18* −.03 −.13 −.23*
Negative Symptoms −.30*** 0.07 0.04 −.08 .00 −.16 .06 .09
Open in a new tab
*

p<.05

**

p<.01

***

p<.001

The first regression analyses examined which OSCARS global ratings (self-rated and informant-rated) significantly accounted for variance in each of the four functional outcome domains. As would be expected from the correlational analyses, informant-rated OSCARs significantly entered into all four models, all t<8.4, all p<.05, while self-reported OSCARS scores did not significantly enter any of the equations, all t<1.7, all p>.09. As such, self-reported OSCARS scores were not entered into subsequent regression analyses. See figure 1 for a scatter plot of the distributions of self reported and informant reported social cognitive ability and informant rated interpersonal functioning on the SLOF.

Figure 1.

Figure 1

Open in a new tab

Scatterplot of the correlation between OSCARS difference scores and Interpersonal Functioning ratings on the SLOF.

Table 4 presents the results of the block-entry regressions assessing the relative importance of performance-based social cognition, informant-rated social cognition, and social cognitive difference scores in predicting the four SLOF subscales of functional outcomes. For the variables as a group, all four regression analyses found that the set of performance-based social cognitive tasks did not contribute to the prediction of any of the four SLOF subscales (all F(7,120)<1.89, all p>.08). After controlling for social cognitive performance, informant-rated OSCARS scores predicted all four SLOF subscales, and the social cognitive difference scores predicted three out of four SLOF subscales. Informant-rated scores accounted for 36% of the variance in social acceptability and vocational functioning (both R2 incremental = .36, p=.001), 25% of the variance in interpersonal functioning (R2 incremental= .25, p=.001), and 14% of the variance in everyday activities (R2 incremental= .14, p=.001). As can be seen in the table, even after the entry of informant ratings of social cognition, social cognitive difference scores contributed approximately equivalent incremental variance across interpersonal functioning, social acceptability, and vocational functioning (all R2 incremental: .06–.08, all p<.001).

Table 4.

Regression Results Predicting SLOF Subscales Using Social Cognitive IA (OSCARS Difference scores) and Informant Rated OSCARS scores

Controlling for Social Cognitive Performance
Interpersonal Functioning
Predictor t p R2 incremental R2 Cumulative
Block 1 SC Performance 0.93 .53 .01 .01
Step 1 Informant OSCARS 6.72 .001 .25 .25
Step 2 SC Introspective Accuracy 3.48 .001 .06 .32
Social Acceptability
Predictor t p R2 incremental R2 Cumulative
Block 1 SC Performance 1.37 .08 .02 .02
Step 1 Informant OSCARS 8.29 .001 .36 .38
Step 2 SC Introspective Accuracy 4.72 .001 .07 .45
Everyday Activities
Predictor t p R2 incremental R2 Cumulative
Block 1 SC Performance 1.38 .07 .04 .04
Step 1 Informant OSCARS 3.68 .001 .14 .18
Vocational Functioning
Predictor t p R2 incremental R2 Cumulative
Block 1 SC Performance 1.40 .06 .05 .05
Step 1 Informant OSCARS 8.37 .001 .36 .41
Step 2 SC Introspective Accuracy 5.44 .001 .08 .49
Open in a new tab

Note. SC= Social Cognition

All four regressions were repeated with only social cognitive difference scores, and all four analyses were found to be significant (all F>7.28, all p<.008). Further, the variance accounted for by the OSCARS difference score was the same or higher as the variance accounted for in the previous model which included informant ratings of social cognition ability: Interpersonal: R2=.11; Social Acceptability: R2=.15, Everyday Activities: R2=.05; Work R2=.20, all p<.01. Figure 1 shows a scatter plot of the correlation between interpersonal functioning and the social cognitive difference score.

Table 5 describes our final set of regression models that examine the relative importance of negative symptoms and social competence on the prediction of functional outcomes. When vocational functioning was predicted with SSPA scores in the first block and the two OSCARS variables in the second, SSPA scores accounted for about 10% of the variance in work functioning. The remaining predicted variance was accounted for by informant OSCARS ratings and social cognitive difference scores. When we predicted interpersonal functioning with negative symptoms in the first block and the OSCARS variables in the second, negative symptoms accounted for 9% of the variance in interpersonal functioning and informant ratings and social cognitive difference scores accounted for an additional 26% of the variance.

Table 5.

Regression Results Predicting SLOF Subscales Controlling for Social Competence and Negative Symptoms

Controlling for Social Competence
Vocational Functioning
Predictor t p R2 incremental R2 Cumulative
Block 1 SSPA 3.72 .001 .10 .10
Step 1 Informant OSCARS 8.33 .001 .25 .35
Step 2 SC Introspective Accuracy 4.78 .001 .07 .42
Controlling for Negative Symptoms
Interpersonal Functioning
Predictor t p R2 incremental R2 Cumulative
Block 1 Negative Symptoms 3.55 .001 .09 .09
Step 1 Informant OSCARS 7.11 .001 .20 .29
Step 2 SC Introspective Accuracy 4.20 .001 .06 .35
Open in a new tab

Discussion

This study examined the association between self assessment of social cognitive abilities and everyday functioning. Individuals with schizophrenia self-evaluated their social cognitive ability on the OSCARs rating scale. Concurrently, high-contact informants rated these same abilities with an identical rating scale and rated the patients’ everyday functioning in social and nonsocial domains. The results demonstrated that patient judgments regarding their social cognitive ability had a minimal correlation to their functional outcomes as rated by informant as well as minimal correlation with their performance on social cognitive tests. However, the discrepancy between the patients’ and informants’ratings of patients’ social cognitive abilities predicted real world functioning. Patients who rated their social cognitive abilities as better compared to informant impressions showed poorer everyday outcomes in every functional domain. Regression analysis further suggested that estimation of social cognitive ability as better than seen by the informant, was associated with everyday outcomes in three out of four functional domains, adding incremental variance beyond that accounted for by informant ratings of social cognitive abilities and actual social cognitive performance.

We performed several additional analyses the validity of these conclusions. We examined the multiple correlation between 8 different social cognitive tests and OSCARS scores generated by the informants and the patients. While informant OSCARS scores correlated with social cognitive performance overall, self-reported OSCARS scores did not. Similarly, the difference scores were not correlated with social cognitive test performance. Thus, informant OSCARS have greater validity as an index of social cognitive than ability than self-reports and discrepancies between informant and self report. We also examined the correlations between the patient-informant difference scores for OSCARS and for SLOF everyday functioning ratings, finding a correlation suggesting that reporting that one was functioning better than the informant’s perspective was consistent across different domains of self-assessment.

In evaluating these results, it is important to keep in mind that reporting your social cognitive ability was better than seen by the informant was not ubiquitous. The distribution of over and underestimation was perfectly normal, with equal numbers of patients over and underestimating their ability compared to an absolute agreement with informants and 25% of each group mis-estimating by 2 or more points on a 10-point scale in either direction. This finding of an approximately equal proportion of cases over and underestimating their social cognitive performance compared to informants is essentially identical to our previous results examining over and underestimation of neurocognitive ability (Gould et al., 2015) and is consistent with the self reports of everyday functioning presented in this paper. The current results confirm that discrepancies between patient and informant assessments are bidirectional and highlight the need to understand the factors that contribute to what could be seen as underestimation in some patients but over-estimation in others. Given that these tendencies are likely to impact behavior directly, understanding the moderators of directionality is necessary (Gould et al., 2015).

The present study extends prior results that neurocognitive IA is a predictor of functional outcomes to social cognitive IA. Results from the VALERO II study on impaired neurocognitive introspective accuracy showed IA, which was also indexed by difference scores between clinician ratings and self-reports on a cognitive functioning rating scale, to be a more potent predictor of everyday functional deficits than scores on performance-based measures of neurocognitive abilities and functional skills (Gould et al., 2015). The present study suggests that accurate self-assessment of social cognition is also a stronger predictor of social outcomes than social cognitive ability. The study also suggests that informant ratings of patient social cognitive ability may offer an efficient way to get an accurate global snapshot of social cognitive ability.

Our results did not find social cognitive IA was the strongest predictor of everyday outcomes when informant ratings were also considered. However, this may be due in part to shared method variance between the informant ratings on the OSCARS and SLOF. Importantly, social cognitive IA showed a strong association to outcomes even in reference to negative symptoms and social competence. Social cognitive IA also added 6–8% variance to informant ratings in all models except everyday activities (all R2 incremental= .06–.08). In comparison, previous analyses of the SCOPE dataset (Pinkham et al., 2016; in press) found small correlations between social cognitive performance-based measures and functional outcomes. Half of the social cognitive tasks assessed showed significant, yet limited correlations with functional outcomes ad none shared more than 3.5% variance with everyday functioning. These social cognitive tasks were entered as a group for several of the current analyses, which reflects our previous findings on the factor structure of this set of social cognitive measures (Browne et al., 2016).

Several limitations require consideration. Functional outcomes where defined by informant reports. As noted above, this might conflate the correlation results between informant assessments of patients’ social cognitive ability and everyday disability. However, the fact that discrepancies between informant and self-reports added variance to the prediction of informant rated outcomes beyond informant opinions of social cognitive ability suggests that IA is still important as a predictor. Analyses excluding informant ratings of social cognition still found that indices of IA predicted everyday activities. An additional limitation remains the same from earlier SCOPE studies. Our sample is composed of predominately older, clinically stable individuals in chronic phases of schizophrenia, which may limit the generalizability of these findings. Note that a specific replication of the SCOPE findings in first episode patients found essentially similar characteristics of all of the social cognitive measures (Ludwig et al., 2017).

This study supports a growing body of work focused on understanding how discrete symptoms correlate to discrete outcomes (Depp et al., 2012; Fett et al., 2011). We found that both informant ratings of social cognition and social cognitive IA accounted for a larger percentage of social functional outcomes (i.e. interpersonal relationships) than non-social functional outcomes (i.e. everyday activities). These findings suggest that IA of social cognition could be an important research and treatment target aimed at improving social real-world outcomes such as interpersonal relationships, social acceptability, and vocational functioning in a quarter of individuals with schizophrenia. Overall, our findings suggest that IA of social cognitive ability might to be a more important predictor than ability itself, as assessed on performance-based tasks, when predicting social everyday disability. The significant correlations between informant ratings of social cognitive ability and social cognitive IA with functional outcomes indicate advancement in the understanding of discrete predictors of everyday disability.

Conclusions

  1. Informant-rated social cognitive competence is correlated with performance on tests of social cognitive ability.

  2. Patient-reported social cognitive competence is not correlated with performance on tests of social cognitive ability.

  3. Patients who report social cognitive competence that is better than reported by their informants have lower scores on everyday functioning.

  4. Patients who report social cognitive competence that is better than reported by their informants also report that their everyday functioning is better than that reported by heir informants.

  5. Differences between patient and informant reports of everyday are not correlated with patients’ social cognitive performance.

Supplementary Material

supplement

Footnotes

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

References

  1. Bänziger T, Scherer KR, Hall JA, Rosenthal R. Introducing the MiniPONS: A short multichannel version of the Profile of Nonverbal Sensitivity (PONS) J Nonverb Behav. 2011;35(3):189–204. [Google Scholar]
  2. Baron-Cohen S, Wheelwright S, Hill J, Raste Y, Plumb I. The ‘Reading the mind in the eyes’ Test revised version: A study with normal adults, and adults with Asperger syndrome or high-functioning autism. J Child Psychol Psychiatry. 2001;42(2):241–251. [PubMed] [Google Scholar]
  3. Beck AT, Baruch E, Balter JM, Steer RA, Warman DM. A new instrument for measuring insight: the Beck Cognitive Insight Scale. Schizophr Res. 2004;68(2–3):319–329. doi: 10.1016/S0920-9964(03)00189-0. [DOI] [PubMed] [Google Scholar]
  4. Bell MD, Fiszdon JM, Greig TC, Wexler BE. Social attribution test--multiple choice (SAT-MC) in schizophrenia: comparison with community sample and relationship to neurocognitive, social cognitive and symptom measures. Schizophr Res. 2010;122(1–3):164–171. doi: 10.1016/j.schres.2010.03.024. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Bowie CR, Depp C, McGrath JA, Wolyniec P, Mausbach BT, Thornquist MH, Luke J, Patterson TL, Harvey PD, Pulver AE. Prediction of real-world functional disability in chronic mental disorders: a comparison of schizophrenia and bipolar disorder. Am J Psychiatry. 2010;167(9):1116–1124. doi: 10.1176/appi.ajp.2010.09101406. [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Bowie CR, Leung WW, Reichenberg A, McClure MM, Patterson TL, Heaton RK, Harvey PD. Predicting schizophrenia patients’ real-world behavior with specific neuropsychological and functional capacity measures. Biol Psychiatry. 2008;63(5):505–511. doi: 10.1016/j.biopsych.2007.05.022. [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Bowie CR, Reichenberg A, Patterson TL, Heaton RK, Harvey PD. Determinants of real-world functional performance in schizophrenia subjects: correlations with cognition, functional capacity, and symptoms. Am J Psychiatry. 2006;163(3):418–425. doi: 10.1176/appi.ajp.163.3.418. [DOI] [PubMed] [Google Scholar]
  8. Bowie CR, Twamley EW, Anderson H, Halpern B, Patterson TL, Harvey PD. Self-assessment of functional status in schizophrenia. J Psychiatr Res. 2007;41(12):1012–1018. doi: 10.1016/j.jpsychires.2006.08.003. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Brekke JS, Levin S, Wolkon GH, Sobel E, Slade E. Psychosocial functioning and subjective experience in schizophrenia. Schizophr Bull. 1993;19(3):599–608. doi: 10.1093/schbul/19.3.599. [DOI] [PubMed] [Google Scholar]
  10. Browne J, Penn DL, Raykov T, Pinkham AE, Kelsven S, Buck BA, Harvey PD. Social cognition in schizophrenia: Factor structure of emotion processing and theory of mind. Psychiatry Res. 2016;242:150–156. doi: 10.1016/j.psychres.2016.05.034. [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Bryson G, Bell M, Lysaker P. Affect recognition in schizophrenia: a function of global impairment or a specific cognitive deficit. Psychiatry Res. 1997;71(2):105–113. doi: 10.1016/s0165-1781(97)00050-4. [DOI] [PubMed] [Google Scholar]
  12. Corcoran R, Mercer G, Frith CD. Schizophrenia, symptomatology and social inference: investigating “theory of mind” in people with schizophrenia. Schizophr Res. 1995;17(1):5–13. doi: 10.1016/0920-9964(95)00024-g. [DOI] [PubMed] [Google Scholar]
  13. Depp CA, Mausbach BT, Harmell AL, Savla GN, Bowie CR, Harvey PD, Patterson TL. Meta-analysis of the association between cognitive abilities and everyday functioning in bipolar disorder. Bipol Disord. 2012;14(3):217–226. doi: 10.1111/j.1399-5618.2012.01011.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. Dimaggio G, Lysaker P. Metacognition and Severe Adult Mental Disorders: From Research to Treatment. Routledge; London: 2010. [Google Scholar]
  15. Durand D, Strassnig M, Sabbag S, Gould F, Twamley EW, Patterson TL, Harvey PD. Factors influencing self-assessment of cognition and functioning in schizophrenia: implications for treatment studies. Eur Neuropsychopharmacol. 2015;25(2):185–191. doi: 10.1016/j.euroneuro.2014.07.008. [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Fett AK, Viechtbauer W, Dominguez MD, Penn DL, van Os J, Krabbendam L. The relationship between neurocognition and social cognition with functional outcomes in schizophrenia: a meta-analysis. Neurosci Biobehav Rev. 2011;35(3):573–588. doi: 10.1016/j.neubiorev.2010.07.001. [DOI] [PubMed] [Google Scholar]
  17. Fleming SM, Weil RS, Nagy Z, Dolan RJ, Rees G. Relating introspective accuracy to individual differences in brain structure. Science. 2010;329(5998):1541–1543. doi: 10.1126/science.1191883. [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Gold JM, Barch DM, Carter CS, Dakin S, Luck SJ, MacDonald AW, 3rd, Ragland JD, Ranganath C, Kovacs I, Silverstein SM, Strauss M. Clinical, functional, and intertask correlations of measures developed by the Cognitive Neuroscience Test Reliability and Clinical Applications for Schizophrenia Consortium. Schizophr Bull. 2012;38(1):144–152. doi: 10.1093/schbul/sbr142. [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Gould F, McGuire LS, Durand D, et al. Self-assessment in schizophrenia: Accuracy of evaluation of cognition and everyday functioning. Neuropsychology. 2015;29(5):675–682. doi: 10.1037/neu0000175. [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Green MF, Kern RS, Braff DL, Mintz J. Neurocognitive deficits and functional outcome in schizophrenia: are we measuring the “right stuff”? Schizophr Bull. 2000;26(1):119–136. doi: 10.1093/oxfordjournals.schbul.a033430. [DOI] [PubMed] [Google Scholar]
  21. Green MF, Schooler NR, Kern RS, Frese FJ, Granberry W, Harvey PD, Karson CN, Peters N, Stewart M, Seidman LJ, Sonnenberg J, Stone WS, Walling D, Stover E, Marder SR. Evaluation of functionally meaningful measures for clinical trials of cognition enhancement in schizophrenia. Am J Psychiatry. 2011;168(4):400–407. doi: 10.1176/appi.ajp.2010.10030414. [DOI] [PubMed] [Google Scholar]
  22. Harvey P, Pinkham A. Impaired self-assessment in schizophrenia: why patients misjudge their cognition and functioning. Current Psychiatry. 2015;14:6. [Google Scholar]
  23. Harvey PD, Raykov T, Twamley EW, Vella L, Heaton RK, Patterson TL. Validating the measurement of real-world functional outcomes: phase I results of the VALERO study. Am J Psychiatry. 2011;168(11):1195–1201. doi: 10.1176/appi.ajp.2011.10121723. [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Harvey PD, Sabbag S, Prestia D, Durand D, Twamley EW, Patterson TL. Functional milestones and clinician ratings of everyday functioning in people with schizophrenia: overlap between milestones and specificity of ratings. J Psychiatric Res. 2012;46(12):1546–1552. doi: 10.1016/j.jpsychires.2012.08.018. [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Healey KM, Combs DR, Gibson CM, Keefe RS, Roberts DL, Penn DL. Observable Social Cognition--A Rating Scale: an interview-based assessment for schizophrenia. Cogn Neuropsychiatry. 2015;20(3):198–221. doi: 10.1080/13546805.2014.999915. [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Holshausen K, Bowie CR, Mausbach BT, Patterson TL, Harvey PD. Neurocognition, functional capacity, and functional outcomes: the cost of inexperience. Schizophr Res. 2014;152(2–3):430–434. doi: 10.1016/j.schres.2013.08.004. [DOI] [PubMed] [Google Scholar]
  27. Hur JW, Kwon JS, Lee TY, Park S. The crisis of minimal self-awareness in schizophrenia: a meta-analytic review. Schizophr Res. 2014;152(1):58–64. doi: 10.1016/j.schres.2013.08.042. [DOI] [PubMed] [Google Scholar]
  28. Kay SR, Fiszbein A, Opler LA. The positive and negative syndrome scale (PANSS) for schizophrenia. Schizophr Bull. 1987;13(2):261–276. doi: 10.1093/schbul/13.2.261. [DOI] [PubMed] [Google Scholar]
  29. Kay SR, Opler LA, Fiszbein A. Positive and Negative Syndrome Scale: Manual. Multi-Health Systems, Inc; 1992. [Google Scholar]
  30. Keefe RS, Davis VG, Spagnola NB, Hilt D, Dgetluck N, Ruse S, Patterson TD, Narasimhan M, Harvey PD. Reliability, validity and treatment sensitivity of the Schizophrenia Cognition Rating Scale. Eur Neuropsychopharmacol. 2015;25(2):176–184. doi: 10.1016/j.euroneuro.2014.06.009. [DOI] [PMC free article] [PubMed] [Google Scholar]
  31. Kohler CG, Turner TH, Bilker WB, Brensinger CM, Siegel SJ, Kanes SJ, Gur RE, Gur RC. Facial emotion recognition in schizophrenia: intensity effects and error pattern. Am J Psychiatry. 2003;160(10):1768–1774. doi: 10.1176/appi.ajp.160.10.1768. [DOI] [PubMed] [Google Scholar]
  32. Koren D, Seidman LJ, Goldsmith M, Harvey PD. Real-world cognitive--and metacognitive--dysfunction in schizophrenia: a new approach for measuring (and remediating) more of the right stuff. Schizophr Bull. 2006;32(2):310–326. doi: 10.1093/schbul/sbj035. [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. Ludwig KE, Pinkham AE, Harvey PD, Kelsven S, Penn DL. Social cognition psychometric evaluation (SCOPE) in people with early psychosis: A preliminary study. Schizophr Res. 2017;190:136–143. doi: 10.1016/j.schres.2017.03.001. [DOI] [PMC free article] [PubMed] [Google Scholar]
  34. McDonald S, Flanagan S, Rollins J, Kinch J. TASIT: A new clinical tool for assessing social perception after traumatic brain injury. J Head Trauma Rehabil. 2003;18(3):219–238. doi: 10.1097/00001199-200305000-00001. [DOI] [PubMed] [Google Scholar]
  35. McKibbin C, Patterson TL, Jeste DV. Assessing disability in older patients with schizophrenia: results from the WHODAS-II. J Nerv Ment Dis. 2004;192(6):405–413. doi: 10.1097/01.nmd.0000130133.32276.83. [DOI] [PubMed] [Google Scholar]
  36. Medalia A, Thysen J. A comparison of insight into clinical symptoms versus insight into neuro-cognitive symptoms in schizophrenia. Schizophr Res. 2010;118(1–3):134–139. doi: 10.1016/j.schres.2009.09.027. [DOI] [PubMed] [Google Scholar]
  37. Metcalfe J, Greene MJ. Metacognition of agency. JEP Gen. 2007;136(2):184–199. doi: 10.1037/0096-3445.136.2.184. [DOI] [PubMed] [Google Scholar]
  38. Moritz S, Kerstan A, Veckenstedt R, Randjbar S, Vitzthum F, Schmidt C, Heise M, Woodward TS. Further evidence for the efficacy of a metacognitive group training in schizophrenia. Behav Res Ther. 2011;49(3):151–157. doi: 10.1016/j.brat.2010.11.010. [DOI] [PubMed] [Google Scholar]
  39. Moritz S, Veckenstedt R, Andreou C, Bohn F, Hottenrott B, Leighton L, Kother U, Woodward TS, Treszl A, Menon M, Schneider BC, Pfueller U, Roesch-Ely D. Sustained and “sleeper” effects of group metacognitive training for schizophrenia: a randomized clinical trial. JAMA Psychiatry. 2014;71(10):1103–1111. doi: 10.1001/jamapsychiatry.2014.1038. [DOI] [PubMed] [Google Scholar]
  40. Nelson TO, Stuart RB, Howard C, Crowley M. Metacognition and clinical psychology: A preliminary framework for research and practice. Clin Psychol Psychot. 1999;6(2):73–79. [Google Scholar]
  41. Patterson TL, Moscona S, McKibbin CL, Davidson K, Jeste DV. Social skills performance assessment among older patients with schizophrenia. Schizophr Res. 2001;48(2–3):351–360. doi: 10.1016/s0920-9964(00)00109-2. [DOI] [PubMed] [Google Scholar]
  42. Patterson TL, Semple SJ, Shaw WS, Halpain M, Moscona S, Grant I, Jeste DV. Self-reported social functioning among older patients with schizophrenia. Schizophr Res. 1997;27(2–3):199–210. doi: 10.1016/S0920-9964(97)00078-9. [DOI] [PubMed] [Google Scholar]
  43. Pinkham AE, Harvey PD, Penn DL. Social Cognition Psychometric Evaluation: Results of the Final Validation Study. Schizophr Bull. doi: 10.1093/schbul/sbx117. In press. [DOI] [PMC free article] [PubMed] [Google Scholar]
  44. Pinkham AE, Penn DL, Green MF, Harvey PD. Social Cognition Psychometric Evaluation: Results of the Initial Psychometric Study. Schizophr Bull. 2016;42(2):494–504. doi: 10.1093/schbul/sbv056. [DOI] [PMC free article] [PubMed] [Google Scholar]
  45. Riggs SE, Grant PM, Perivoliotis D, Beck AT. Assessment of cognitive insight: a qualitative review. Schizophr Bull. 2012;38(2):338–350. doi: 10.1093/schbul/sbq085. [DOI] [PMC free article] [PubMed] [Google Scholar]
  46. Rosset E. It’s no accident: Our bias for intentional explanations. Cognition. 2008;108(3):771–780. doi: 10.1016/j.cognition.2008.07.001. [DOI] [PubMed] [Google Scholar]
  47. Sabbag S, Twamley EM, Vella L, Heaton RK, Patterson TL, Harvey PD. Assessing everyday functioning in schizophrenia: not all informants seem equally informative. Schizophr Res. 2011;131(1–3):250–255. doi: 10.1016/j.schres.2011.05.003. [DOI] [PMC free article] [PubMed] [Google Scholar]
  48. Sabbag S, Twamley EW, Vella L, Heaton RK, Patterson TL, Harvey PD. Predictors of the accuracy of self assessment of everyday functioning in people with schizophrenia. Schizophr Res. 2012;137(1–3):190–195. doi: 10.1016/j.schres.2012.02.002. [DOI] [PMC free article] [PubMed] [Google Scholar]
  49. Schneider LC, Struening EL. SLOF: a behavioral rating scale for assessing the mentally ill. Soc Work Res Abstr. 1983;19(3):9–21. doi: 10.1093/swra/19.3.9. [DOI] [PubMed] [Google Scholar]
  50. Tabares-Seisdedos R, Balanza-Martinez V, Sanchez-Moreno J, Martinez-Aran A, Salazar-Fraile J, Selva-Vera G, Rubio C, Mata I, Gomez-Beneyto M, Vieta E. Neurocognitive and clinical predictors of functional outcome in patients with schizophrenia and bipolar I disorder at one-year follow-up. J Affect Disord. 2008;109(3):286–299. doi: 10.1016/j.jad.2007.12.234. [DOI] [PubMed] [Google Scholar]
  51. Wechsler D. Wechsler abbreviated scale of intelligence. Psychological Corporation; 1999. [Google Scholar]
  52. Weickert TW, Goldberg TE, Gold JM, Bigelow LB, Egan MF, Weinberger DR. Cognitive impairments in patients with schizophrenia displaying preserved and compromised intellect. Arch Gen Psychiatry. 2000;57(9):907–913. doi: 10.1001/archpsyc.57.9.907. [DOI] [PubMed] [Google Scholar]
  53. Yahav T, Maimon T, Grossman E, Dahan I, Medalia O. Cryo-electron tomography: gaining insight into cellular processes by structural approaches. Curr Opin Struct Biol. 2011;21(5):670–677. doi: 10.1016/j.sbi.2011.07.004. [DOI] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

supplement
NIHMS961762-supplement.docx (12.3KB, docx)

RESOURCES