Psychometric properties of the Observable Social Cognition Rating Scale (OSCARS): Self-report and informant-rated social cognitive abilities in schizophrenia

Abstract

Individuals with schizophrenia spectrum disorders (SSD) consistently show deficits in social cognition (SC) which is associated with real world outcomes. Psychosocial treatments have demonstrated reliable improvements in SC abilities, highlighting the need for accurate identification of SC deficits for efficient and individualized treatment planning. To this end, the Observable Social Cognition Rating Scale (OSCARS) is an 8-item scale with both self and informant versions. This study investigated psychometric properties of the OSCARS as both a self and informant-reported scale in a large sample of SSD (n =382) and individuals without a psychiatric diagnosis (n = 289). A two-factor structure (Social Cognitive Bias and Social Cognitive Ability) of the OSCARS demonstrated acceptable model fit with good internal consistency for both self- and informant-report. The OSCARS had adequate convergent, external, and predictive validity. Area Under the Curve (AUC) values suggest the OSCARS has some value in identifying individuals with impaired SC and social competence, although stronger AUC values were demonstrated when identifying individuals with impaired real-world functioning. Overall, psychometric properties indicate the OSCARS may be a useful first-step tool for clinicians to detect functioning deficits in SSD and efficiently identify individuals in need of additional assessment or psychosocial interventions.

1. Introduction

Social cognition (SC) encompasses cognitive domains related to how individuals think about themselves, others, social situations, and social interactions (Penn, Corrigan, Bentall, Racenstein, & Newman, 1997; Penn, Sanna, & Roberts, 2008). Individuals with schizophrenia spectrum disorders (SSD) consistently show deficits in National Institute of Mental Health - defined domains of SC: theory of mind, social perception, social knowledge, attributional bias, and emotional processing (Green et al., 2008; Savla, Vella, Armstrong, Penn, & Twamley, 2013). SC is strongly associated with functional outcomes, with evidence that SC explains more variance in social outcomes than neurocognition (NC; Fett et al., 2011; Halverson et al., 2019). To date, psychotropic medications do not produce robust effects on SC but psychosocial treatments demonstrate reliable improvements in SC abilities (Grant, Lawrence, Preti, Wykes, & Cella, 2017; Kurtz, Gagen, Rocha, Machado, & Penn, 2016; Sergi et al., 2007). While SC deficits are consistently reported in SSD, findings regarding the prevalence of intact SC in SSD are scant and less consistent, with estimates that 25% - 42% of individuals with SSD do not show impairment (Hajdúk, Harvey, Penn, & Pinkham, 2018; Rocca et al., 2016). Thus, there does appear to be a subset of individuals without impairment, and therefore, efficient and accurate identification of SC deficits in SSD is an important first step for effective and individualized treatment planning.

To this end, the Social Cognition and Psychometric Evaluation (SCOPE) Study investigated SC measures in SSD and recommended tasks with good psychometric properties (Pinkham et al., 2014). The SCOPE battery is a comprehensive assessment with rigorously tested psychometric properties (see Pinkham, Harvey, & Penn, 2018; Pinkham, Penn, Green, & Harvey, 2016) for SC assessment, however, an administration length of an hour and the need for trained raters may not be feasible for all referrals or treatment settings. An alternative approach to detect SC deficits is a brief rating scale. Observer and interview-based rating scales exist for NC deficits with demonstrated reliability and convergence with performance-based tasks (e.g., Schizophrenia Cognition Rating Scale; Keefe, Poe, Walker, Kang, & Harvey, 2006, Clinical Global Impression of Cognition in Schizophrenia; Ventura, Cienfuegos, Boxer, & Bilder, 2008). The development and validation of SC rating scales may similarly prove to be useful and efficient tools for clinicians and clinical research.

Healey et al. (2015) developed a rating scale of SC, the Observable Social Cognition Rating Scale (OSCARS). The OSCARS is an eight-item assessment available as a self- or informant-report measure with a total administration time of five minutes. OSCARS items were selected and reviewed by internationally recognized experts to probe areas of SC with demonstrated deficits in SSD while items considered to be indicators of social skill, selfawareness, and insight were removed. Items are rated on a Likert scale with higher scores reflecting greater impairment. The OSCARS includes a total summary score of all items and a separate single item “global score.” Whereas the OSCARS total score is the sum of individually rated areas of SC difficulties, the global score is a single-item overall assessment of social cognition (“overall impression of difficulty in these areas”) completed at the end of the OSCARS (analogous to the single-item General Assessment of Functioning (GAF) completed by clinicians after a clinical interview). In an initial validation study, the OSCARS was administered to informants of 62 individuals with SSD and 50 healthy controls (HC). The OSCARS demonstrated sufficient psychometric properties and significant relationships with functional outcomes (Healey et al., 2015). Differences in the factor structure of the OSCARS emerged based on diagnostic group. A two-factor (i.e., Social Cognitive Bias and Social Cognitive Ability) structure provided the best fit for the SSD group while a three-factor structure (i.e., Social Cognitive Bias, Social Cognitive Ability, and Social Cognitive Flexibility) emerged in the HC group. The Social Cognitive Bias factor comprised items indicating impulsivity, hostility, and rigidity; the Social Cognitive Ability factor comprised items involving reasoning and perceptual abilities; and the Social Cognitive Flexibility factor (found only in HC) comprised items involving subtle theory of mind ability and flexibility in social situations (Healey et al., 2015).

Increasingly, studies are utilizing the OSCARS single-item global score. Silberstein, Pinkham, Penn, & Harvey (2018) examined the OSCARS global score generated by individuals with SSD and high-contact informants. Informant reports were significantly correlated with both performance-based assessments of SC abilities and with real-world functional outcomes, while self-reports were not. Additionally, studies have used the difference between self- and informant-reported OSCARS global scores as an indicator of introspective accuracy, a component of metacognition defined as how well individuals can evaluate their own abilities and performance on tasks (Harvey et al., 2019; Jones et al., 2019; Silberstein & Harvey, 2019). Given the recent interest and dissemination of the OSCARS and its relationship with SC and functioning, a thorough psychometric investigation to complement the initial validation study is warranted. A more comprehensive understanding of factors that impact the OSCARS global score is also needed as this score has received recent attention as a potential measure of functioning. For example, informants may weigh certain OSCARS factors (e.g., Cognitive Bias versus Cognitive Ability) differently when making this summary rating compared with self-report.

The present study seeks to replicate findings from the OSCARS initial validation study and investigate the factor structure and psychometric properties in a larger sample of SSD and HC individuals. For the first time, psychometric properties of the OSCARS self-report will also be examined and compared with informant-report. The aims of the present study are 1) evaluate the structure and psychometric properties of the OSCARS self-report and informant-report in HC and SSD, 2) examine relationships between OSCARS factor scores and global score, and 3) examine relationships among the OSCARS, SC tasks, and functional outcomes. We hypothesize that 1) OSCARS informant-report will exhibit stronger relationships with functional outcomes compared with self-report and 2) informant-report will significantly predict impaired SC task performance, validating the OSCARS as a reliable proxy to assess SC compared with more resource-intensive performance tasks.

2. Methods

2.1. Participants

The OSCARS was administered as part of the SCOPE study; a multi-site, multi-phase, National Institute of Mental Health project to identify the best measures for assessing SC in schizophrenia (Pinkham et al., 2014). Participants (N = 671) included individuals with at least one source of OSCARS ratings (i.e., self or informant). SSD (n = 382) and HC (n = 289) participants were recruited from four different academic sites (i.e., Southern Methodist University – initial evaluation phase, University of Miami – all phases, University of North Carolina – second and final validation phases, and The University of Texas at Dallas – second and final validation phases) during either the initial (n = 213) second (n = 88), or final validation phase (n = 370) of the SCOPE Study (Pinkham et al., 2016; Pinkham et al., 2018).

SSD participants were stable outpatients (i.e., no hospitalizations within two months and a stable medication regimen for at least six weeks, no change in medication dosage for at least two weeks) recruited from community clinics, academic medical centers, and Veterans Affairs medical centers. Schizophrenia diagnoses were confirmed through a structured diagnostic interview, the Mini International Neuropsychiatric Interview (MINI; Sheehan et al., 1998), and the psychosis module of the Structured Clinical Interview for DSM Disorders (SCID; First, Spitzer, Gibbon, & Williams, 2002). HC participants were recruited through community advertisements and were administered the MINI to confirm no Axis I or Axis II diagnoses were present. Exclusion criteria for the entire sample included: IQ <70, presence of a medical condition with the potential to affect neurological functioning, vision or sensory limitations interfering with assessment, lack of proficiency in English, or presence of substance abuse (past month) or dependence (past six months). All study measures included in the present analysis were collected in one study visit while informant ratings were completed within two weeks of this visit (see Table 1 for demographics and clinical characteristics).

Table 1.

Sample Characteristics

	SSD (n = 382)	HC (n = 289)	p-valuea
Age, years	41.7 ± 12.0	41.1 ± 12.6	.51
Male, % (n)	66.0 (252)	57.4 (166)	.02
Education, years	13.0 ± 2.4	13.9 ± 1.9	<.01
WRAT-3 Standard Score	94.6 ± 14.9	98.8 ± 12.3	<.01
Race/Ethnicityb, % (n)
White	49.0 (187)	47.1 (136)	.63
Black	44.5 (170)	46.4 (134)	.63
Other	6.5 (25)	6.6 (19)	.99
Hispanic/Latinx	18.1 (69)	19.4 (56)	.67
Diagnosis, % (n)
Schizophrenia	49.2 (188)
Schizoaffective	49.7 (190)
Psychosis NOS	10 (4)
Medication Typec, % (n)
Typical	12.6 (48)
Atypical	73.0 (279)
Combination	4.7 (18)
No Antipsychotic	7.6 (29)
PANSS
Positive	16.0 ± 5.3
Negative	13.8 ± 5.2
General	31.5 ± 8.0
Total	61.3 ± 14.8

Note:

^aChi-squared for categorical variables (sex, race/ethnicity), t-test for continuous variables (age, education, WRAT standard score)

^bindividuals able to identify more than one race/ethnicity

^cMedication information unavailable for 7 participants.

WRAT - Wide Range Achievement Test – 3^rd Edition, PANSS = Positive and Negative Syndrome Scale, SSD = schizophrenia spectrum disorder, HC = healthy control; all values presented are M ± SD unless otherwise noted.

2.2. OSCARS Administration

HC and SSD individuals completed the OSCARS as a self-report measure. SSD participants were asked to identify an informant with whom they had the most regular contact. Only high-quality informants (i.e., professional role or more than two hours of contact per week; n = 174) were included in the present analyses (see Table 3). Informant report was not available for all SSD individuals. The eight OSCARS items include anchor points of severity and frequency for ratings of 1, 3, 5, and 7 to guide raters. The OSCARS global score is on a 10-point scale. See Appendices A and B for full versions of the OSCARS self-report and informant-report.

Table 3.

OSCARS Descriptive Statistics by Diagnostic Group

	SSD		HC
OSCARS	n	M ± SD	n	M ± SD
Self-Report Total Score	289	19.7 ± 9.1	289	12.1 ± 4.9**
Social Cognitive Ability		9.4 ± 4.6		5.8 ± 2.5**
Social Cognitive Bias		10.2 ± 5.4		6.3 ± 2.9**
Informant-Report Total Score	174	23.6 ± 10.4
Social Cognitive Ability		11.5 ± 5.3
Social Cognitive Bias		12.0 ± 5.9
Informant Role
Psychologist/Psychiatrist	62
Parent	28
Significant other	22
Social worker	18
Sibling	14
Friend	13
Child	9
Group home manager	4
Other	3
Aunt	1

Note:

^*Significant between-group differences

^**p<.01

SSD = schizophrenia spectrum disorder, HC = healthy control.

2.3. Social Cognition Tasks

Five task-based SC measures collected during all phases of the SCOPE Study with good psychometric properties were examined in relation to the OSCARS. The Bell Lysaker Emotion Recognition Task (BLERT; Bell, Bryson, & Lysaker, 1997; range 0–21) and the Penn Emotion Recognition Task (ER-40; Kohler et al., 2003; range 0–40), measured emotion perception. The Reading the Mind in the Eyes task (Eyes; Baron-Cohen et al., 2001; range 0–36), the Hinting Task (Hinting; Corcoran, Mercer, & Frith, 1995; range 0–20), and The Awareness of Social Inferences Test (TASIT; McDonald, Flanagan, Rollins, & Kinch, 2003; range 0–64) measured theory of mind and mental state attribution. Performance on all measures was indexed as the total number of correct responses.

Different measures of attributional bias were administered. The Ambiguous Intentions Hostility Questionnaire (AIHQ; Combs, Penn, Wicher, & Waldheter, 2007) was administered during the initial and second phases. The AIHQ generates three indices: an Aggression Index and Hostility Index based on rater-scored open-ended responses to vignettes and a Blame Index which is an average composite of Likert items completed by participants. The AIHQ was classified as “not recommended” in the SCOPE initial validation study due to limited relation with functional outcomes (Pinkham et al., 2016). However, recent research suggests the Blame Index exhibits improved psychometric properties and significant relationships with functional capacity and was therefore included in the present analyses (Buck et al., 2017). The Intentionality Bias Task (IBT; Rosset, 2008) was administered during the final validation phase to replace the AIHQ. The IBT generates an Intentionality Bias Index which is the percentage of actions from vignettes that participants endorse as intentional.

Detailed descriptions of all SC tasks are described in previous publications (Pinkham et al., 2016; Pinkham et al., 2018).

2.4. Neurocognition Measures

A subset of the MATRICS Consensus Cognitive Battery (Nuechterlein et al., 2008) was administered to assess NC. Category Fluency - Animal Naming, Symbol Coding, and Trail Making Test - Part A (Trails A) assessed processing speed. Letter-Number Span assessed working memory, and verbal learning was assessed with the Hopkins Verbal Learning Test – Revised (HVLT – R). Due to high correlations between tasks, a composite score was calculated and used for analyses. The Wide Range Achievement Test – 3 Reading subscale (WRAT-3) estimated premorbid IQ (Weickert et al., 2000).

2.5. Functional Measures

Three different measures assessed functioning in competence or performance contexts. The Social Skills Performance Assessment (SSPA; Patterson, Moscona, McKibbin, Davidson, & Jeste, 2001) is a role play task that assesses social competence and consists of two role plays yielding a 1–5 score for each role play and an average total score with higher scores reflecting better functioning. The SSPA was only collected during the initial and final validation phases. Another performance-based measure, the UCSD Performance-Based Skills Assessment, Brief (UPSA-B; Mausbach, Harvey, Goldman, Jeste, & Patterson, 2007) was administered to assess functional capacity. The Specific Levels of Functioning Scale (SLOF; Schneider & Struening, 1983) is a 31-item measure of domains of real-world functioning (i.e., interpersonal relationships, social acceptability, participation in activities, and work skills). The SLOF was administered as both a self-report and informant-report measure in the current study. The SLOF yields an average score across all domains (range 1–5) as well four specific domain scores (range 1–5) with higher scores reflecting better functioning.

2.6. Symptoms

The Positive and Negative Syndrome Scale (PANSS; Kay, Fiszbein, & Opler, 1987) assessed current symptomatology in the SSD group with higher scores reflecting more severe symptoms. PANSS Positive, Negative, and General scores as well as total score were calculated.

2.7. Statistical Approach

The structure of the OSCARS was assessed by fitting confirmatory factor analysis (CFA) models from the two and three-factor models presented by Healey et al., (2015) and comparing model fit statistics across nested models. First, unidimensional models were fit separately to each informant source (i.e., self and informant) within each diagnostic group. Next, one-factor then two-factor models were first fit, followed by three-factor models. Model indices were inspected and chi-square likelihood difference tests were applied to assess optimal model fit. Model fit was evaluated based on the following guidelines: non-significant chi-square (X²) statistic (Barrett, 2007), comparative fit index (CFI) and Tucker Lewis Index (TLI) values close to or greater than .95 (Hu & Bentler, 1999), standardized root square mean residual (SRMR) value less than .08 (Hu & Bentler, 1999), and a RMSEA value with a 90% confidence interval that does not include values greater than .10 (MacCallum, Browne, & Sugawara, 1996). Judgment of optimal model fit was informed by multiple model fit statistics and theoretical considerations rather than a single criterion (Bentler, 2007). Multi-group CFAs were fit with optimal models to assess measurement invariance (e.g., potential differences in OSCARS factor structure and loadings) across sites. Once an appropriate model fit was established for each informant source within each diagnostic group, internal consistency statistics were calculated.

Correlations between the OSCARS and task-based measures of SC, NC, and functional capacity and real-world outcomes were calculated to assess convergent, divergent, and external validity. Regressions predicting the OSCARS general score from factor scores were examined to assess relative influence of specific domains of observerable SC. Hierarchical regressions predicting functional outcomes and SC task performance examined predictive validity of the OSCARS. Finally, SC composite z-scores normed to average HC performance were calculated and Receiver Operating Characteristic (ROC) analyses assessed utility of the OSCARS to predict impaired SC (i.e., at least 1.5 standard deviations below HC performance). All analyses were done in R using the lavaan package (Rosseel, 2012) for factor analyses using maximum likelihood estimation.

3. Results

3.1. Sample Characteristics and Descriptives

HC and SSD groups did not significantly differ in age, race, or ethnicity, but the HC group completed more years of education, had a higher IQ, and included more females (see Table 1) which were included as covariates in subsequent analyses. Table 2 presents descriptives by diagnostic group for SC, NC, and functional outcomes and Table 3 presents OSCARS scores. As expected, the HC group demonstrated significantly better performance on all measures of SC, NC, and functional outcomes compared with the SSD group (p <.03 for all comparisons). Across sites, SSD groups showed differences on some demographic and clinical variables (see Supplementary Tables 1 and 2).

Table 2.

Social Cognition Tasks and Functional Outcome Measures by Diagnostic Group

	SSD	HC	p
Social Cognition Tasks
AIHQ – Blamea	8.8 ± 2.9	7.0± 2.4	<.01
BLERT	13.6 ± 4.1	15.9 ± 2.8	<.01
ER40	30.7 ± 4.9	33.0 ± 3.3	<.01
Eyes	21.0 ± 5.6	24.3 ± 4.4	<.01
Hinting	13.4 ± 3.7	16.1 ± 2.4	<.01
IBTb	44.0 ± 17.7	40.2 ± 14.7	.03
TASIT	44.8 ± 7.6	51.6 ± 6.2	<.01
Neurocognition Tasksc
Animal Naming	19.3 ± 5.7	22.7 ± 6.0	<.01
HVLT	20.9 ± 5.6	25.2 ± 4.6	<.01
Letter Number Sequence	11.8 ± 4.2	14.6 ± 3.8	<.01
Symbol Coding	42.8 ± 11.4	53.3 ± 12.4	<.01
Trails A	39.6 ± 17.4	31.3 ± 10.7	<.01
Functional Outcome Measures
SSPA Average	4.1 ± 0.5	4.6 ± 0.4	<.01
SSPA1	4.2 ± 0.6	4.7 ± 0.4	<.01
SSPA2	4.0 ± 0.6	4.5 ± 0.5	<.01
SLOF Self-Report Average	4.1 ± 0.6	4.6 ± 0.4	<.01
Interpersonal Relationships	3.6 ± 0.9	4.1 ± 0.7	<.01
Social Acceptability	4.4 ± 0.6	4.7 ± 0.4	<.01
Activities of Community Living	4.4 ± 0.9	4.8 ± 0.5	<.01
Work Skills	4.0 ± 0.8	4.7 ± 0.5	<.01
SLOF Informant Average	4.0 ± 0.6
Interpersonal Relationships	3.4 ± 0.9
Social Acceptability	4.4 ± 0.6
Activities of Community Living	4.5 ± 0.9
Work Skills	3.7 ± 0.9
UPSA Total	70.3 ± 14.4

Note:

^aAIHQ only collected during initial and second phases

^bIBT only collected during final phase

^cNeurocognition composite score used for analyses

AIHQ – Blame = Ambiguous Intentions and Hostility Questionnaire Blame Index, BLERT = Bell Lysaker Emotion Recognition Task, ER40 = Penn Emotion Recognition Task, Eyes = Reading the Mind in the Eyes Task, Hinting = Hinting Task, IBT = Intentionality Bias Task, TASIT = The Awareness of Social Inference Test, HVLT = Hopkins Verbal Learning Test, SSPA = Social Skills Performance Assessment, SLOF = Specific Levels of Functioning, UPSA = UCSD Performance-Based Skills Assessment, HC = healthy control, SSD = schizophrenia spectrum disorder; all values presented are M ± SD.

3.2. Construct Validity

Chi-square likelihood difference tests comparing one-, two-, and three-factor CFA models and model fit indices suggested the two-factor model proposed by Healey et al. (2015) was the optimal fit, yielding a Social Cognitive Bias Factor and a Social Cognitive Ability Factor across diagnostic group and informant source (see Table 4).

Table 4.

OSCARS Confirmatory Factor Analysis

	SSD Self-Report		SSD Informant-Report		HC Self-Report
OSCARS Item	SC Ability	SC Bias	SC Ability	SC Bias	SC Ability	SC Bias
1. Recognizing other people’s emotions, particularly negative emotions (sadness, fear and anger) based on facial expression, body language and/or vocal tone and rate?	.52		.75		.61
6. Understanding subtle jokes, sarcasm and insults in conversation?	.60		.74		.50
7. Seeing things from the perspective of others (i.e., putting themselves in other people’s shoes)?	.67		.88		.48
8. Understanding subtle social cues, hints and indirect requests? (an example of an indirect request is if your son/daughter wants a toy, but rather than say so directly, comments on how pretty it is)	.61		.84		.65
2. Interpreting social interactions in a malevolent, hostile manner?		.60		.79		.59
3. Making decisions quickly (i.e., jumps to conclusions) without examining other evidence?		.67		.75		.63
4. Being flexible in interpreting social situations?		.67		.70		.66
5. Can change or correct their interpretation of social interactions when wrong?		.78		.73		.73
	X2 (19) =37.2, p = <.01		X2 (19) =54.5, p = <.01		X2 (19) =69.3, p = <.01
	CFI = .97		CFI = .96		CFI = .91
Model Fit Statistics	TLI = .96		TLI = .94		TLI = .86
	SRMR = .03		SRMR = .04		SRMR = .06
	RMSEA = .06 [.03, .09]		RMSEA = .10 [.07, .14]		RMSEA = .10 [.07, .12]

Note: shaded regions reflect individual factors; SC = social cognition, SSD = schizophrenia spectrum disorder, HC = healthy control.

Multi-group two-factor CFAs were fit to test for configural invariance (i.e., same overall factor structure) and factor loadings or metric invariance (i.e., similar associations between individual items and latent factors) across sites. The OSCARS self-report configural invariance model demonstrated acceptable fit (RMSEA = .09, CFI = .90) and results of a X² difference test demonstrated no difference with the metric invariance model (X²(12) = 20.2, p = .06) supporting full metric invariance. Likewise, the OSCARS informant-report configural model demonstrated acceptable fit (RMSEA = 0.11, CFI = 0.95) with no difference observed with the metric invariance model (X²(12) = 20.5, p = .06). Overall, results demonstrate relative stability of the OSCARS factor structure and loadings across sites.

Factor loadings were acceptable to high (.48 – .88). Correlations between factors within self-report (r_SSD=.64 and r_HC= .60, ps <.01) and informant-report (r_SSD= .74, p <.01) were also high (see Supplementary Table 3). In the SSD group, medium correlations (rs .29 – 42, ps <.01) were observed between OSCARS self-report and informant-report (i.e., total, global, and factor scores), consistent with previous work (Silberstein et al., 2018). The SSD group, as expected, had significantly higher self-reported OSCARS scores (i.e., more impaired SC) compared with the HC group (see Table 3). Supplementary Table 4 presents regressions predicting OSCARS global scores from factor scores with results suggesting both SC Bias and SC Ability factor scores are significantly related to the OSCARS global scores. Slightly larger estimates for the SC Ability factor score were observed across informant source and diagnostic group.

3.3. Internal Consistency

Cronbach’s alphas were calculated as a measure of internal consistency for the OSCARS total and factor scores for both self-report and informant-reported outcomes (ωs .72 - .92). McDonald’s hierarchical omegas were also calculated (ωs .64 - .91) and presented (see Table 5) as a more conservative and robust measure and demonstrated good internal consistency (Dunn, Baguley, & Brunsden, 2014; Peters, 2014).

Table 5.

OSCARS, Social Cognition, Neurocognition, and Functional Outcome Correlations

		SSD Self-Report				SSD Informant-Report				HC Self-Report
	OSCARS Factors	Total	SC Ability	SC Bias	Global	Total	SC Ability	SC Bias	Global	Toaal	SC Ability	SC Bias	Global
	McDonald’s Omegas	ω =.83	ω =.69	ω =.78	-	ω =.91	ω =.83	ω = .88	-	ω =.80	ω =.64	ω =.75	-
Divergent Validity	NC Composite	−.13*	−.14*	−.09	−.18**	−.09	−.13	−.04	−.19**	−.12*	−.14*	−.08	−.16**
Convergent Validity	AIHQ – Blamea	-	-	-	-	.15	.12	.15	.11	.37**	.29**	.36**	.16
	BLERT	−.06	−.07	−.04	−.12*	−.20**	−.19*	−.18*	−.28**	−.20**	−.21**	−.16**	−.19**
	ER40	−.09	−.14*	−.04	−.10	−.02	−.02	−.01	−.10	−.17**	−.15*	−.15*	−.11
	Eyes	−.15**	−.19**	−.10	−.19**	−.02	−.06	.03	−.11	−.15*	−.13*	−.13*	−.19**
	Hinting	−.07	−.06	−.07	−.10	−.08	−.12	−.03	−.14	−.06	−.07	−.03	−.01
	IBTb	.14*	.14*	.12	.10	.07	.08	.05	.14	.07	.00	.11	−.01
	TASIT	−.07	−.11	−.03	−.05	−.04	−.07	.00	−.13	−.15*	−.21**	−.07	−.24**
External Validity	SSPA Average	−.15*	−.17*	−.11	−.06	−.09	−.17*	−.02	−.14	−.12	−.13*	−.09	−.14*
	SSPA1	−.17*	−.19**	−.12	−.08	−.12	−.15	−.07	−.14	−.17**	−.19**	−.12	−.17**
	SSPA2	−.09	−.10	−.06	−.03	−.05	−.14	.05	−.11	−.05	−.05	−.04	−.08
	SLOF Self-Report Total	−.40**	−.35**	−.36**	−.39**	−.40**	−.39**	−.37**	−.38**	−.47**	−.43**	−.41**	−.31**
	Interpersonal Relationships	−.38**	−.33**	−.36**	−.26**	−.24**	−.26**	−.19*	−.26**	−.49**	−.42**	−.46**	−.36**
	Social Acceptability	−.30**	−.19**	−.34**	−.24**	−.28**	−.21*	−.31**	−.26**	−.28**	−.17**	−.33**	−.25**
	Community Living	−.20**	−.20**	−.16*	−.30**	−.34**	−.33**	−.31**	−.29**	−.24**	−.27**	−.18**	−.11
	Work Skills	−.28**	−.26**	−.25**	−.26**	−.24**	−.24**	−.21*	−.27**	−.33**	−.34**	−.26**	−.19**
	SLOF Informant Total	−.19*	−.14	−.20**	−.21**	−.60**	−.54**	−.57**	−.67**
	Interpersonal Relationships	−.20**	−.14	−.21**	−.19*	−.53**	−.50**	−.49**	−.53**
	Social Acceptability	−.29**	−.21**	−.30**	−.18*	−.56**	−.42**	−.62**	−.59**
	Community Living	−.05	.01	−.08	−.13	−.30**	−.28**	−.29**	−.38**
	Work Skills	−.15*	−.17*	−.11	−.15*	−.48**	−.46**	−.44**	−.58**
	UPSA Total	−.14*	−.12*	−.13*	−.18**	−.13	−.20**	−.06	−.14

Note:

^*p <.05

^**p <.01.

^aAIHQ only collected during initial and second phases

^bIBT only collected during final phase

AIHQ – Blame = Ambiguous Intentions and Hostility Questionnaire Blame Index, BLERT = Bell Lysaker Emotion Recognition Task, ER40 = Penn Emotion Recognition Task, Eyes = Reading the Mind in the Eyes Task, Hinting = Hinting Task, IBT = Intentionality Bias Task, TASIT = The Awareness of Social Inference Test, HVLT = Hopkins Verbal Learning Test, SSPA = Social Skills Performance Assessment, SLOF = Specific Levels of Functioning, UPSA = UCSD Performance-Based Skills Assessment, SSD = schizophrenia spectrum disorder, SC = social cognition; ω = McDonald’s hierarchical omegas.

3.4. Convergent and Divergent Validity

Convergent and divergent validity were assessed through correlations between OSCARS scores and measures of SC (convergent validity), NC (divergent validity; see Table 5), and symptoms (divergent validity; see Supplementary Table 5). OSCARS total and factor scores demonstrated convergent validity with significant small to medium correlations (rs −.12 to −.38) with at least one measure of SC (with the exception of SSD self-reported SC Bias scores). The OSCARS did not show strong support for divergent validity with significant small to medium correlations (rs −.13 to −.38), essentially the same as those for SC, observed between NC composite scores and most OSCARS scores. OSCARS scores showed significant correlations with all PANSS scores except the negative symptoms subscale.

3.5. External Validity

Overall, the OSCARS demonstrated adequate external validity with measures assessing functional outcomes. Most OSCARS scores were significantly correlated with at least one performance-based measures of functioning (i.e., SSPA or UPSA) with small to medium correlations (rs −.12 to −.33, see Table 5), with the exception of SSD total informant-report, SSD informant-report SC Bias, and SSD informant-report global score. All OSCARS informant-reported scores also demonstrated external validity with self and informant-reported measures (i.e., SLOF) of functioning (rs −.67 to −.19, ps <.05). Most of the OSCARS self-report scores demonstrated external validity with self and informant-reported SLOF scores (rs −.39 to −.15), with the notable exception of non-significant relationships between all OSCARS scores and SLOF informant-reported Community Living Subscales, and some OSCARS scores with SLOF Interpersonal Relationships and Work Skills.

3.6. Predictive Validity

Predictive validity was examined through a series of hierarchical regressions predicting functional outcomes from OSCARS total scores after accounting for the variance explained by sex, years of education, and WRAT scores (see Table 6). For each analysis, predictor variables were entered into the model in the following order 1) sex, years of education, and WRAT scores 2) OSCARS total score 3) OSCARS global score. The OSCARS self-report and informant-based total scores were shown to significantly predict functioning on both the SLOF self-report (OSCARS self-report total, F(4, 182) = 10.5, p <.01, Δ R² = .10; OSCARS self-report global score, F(5, 181) = 10.0, p <.01, Δ R² = .02; OSCARS informant total F(4, 105) = 6.0, p <.01, Δ R² = .07), and SLOF informant-report total (OSCARS informant total F(4, 165)= 24.8, p <.01, Δ R² = .28; OSCARS global score F(5, 164) = 27.9, p <.01, Δ R² = .08). Self- and informant-reported OSCARS scores did not predict UPSA or SSPA performance.

Table 6.

Regressions Predicting Functional Outcomes from OSCARS in SSD

Outcome	Adjusted R2	Change in R2	Model Statistic	p
SSPA Average
OSCARS Self-Report
Total	.08	.01a	F (4, 222) = 5.8	<.01
Global	.08	<.01	F (5, 221) = 4.7	<.01
OSCARS Informant-Report
Total	.09	<.01a	F (4, 148) = 4.8	<.01
Global	.09	<.01	F (5, 147) = 4.0	<.01
SLOF Self-Report Total
OSCARS Self-Report
Total	.17	.10a*	F (4, 182) = 10.5	<.01
Global	.19	.02*	F (5, 181) = 10.0	<.01
OSCARS Informant-Report
Total	.15	.07a*	F (4, 105) = 6.0	<.01
Global	.15	<.01	F (5, 104) = 5.0	<.01
SLOF Informant Total
OSCARS Self-Report
Total	.07	.01a	F (4, 171) = 4.5	<.01
Global	.08	.01	F (5, 170) = 3.8	<.01
OSCARS Informant-Report
Total	.36	.28a*	F (4, 165) = 24.8	<.01
Global	.44	.08*	F (5, 164) = 27.9	<.01
UPSA Total
OSCARS Self-Report
Total	.24	<.01 a	F (4, 273) = 22.9	<.01
Global	.24	<.01	F (5, 272) = 18.7	<.01
OSCARS Informant-Report
Total	.20	<.01a	F (4, 161) = 11.5	<.01
Global	.20	<.01	F (5, 160) = 9.2	<.01

Note:

^achange in R² from covariate (sex, years of education, WRAT3 score) model

^*indicates significant model improvement, p <.05

SSD = schizophrenia spectrum disorders; Due to significant correlations between predictors, variance inflation factor (VIF) statistics were calculated, VIF values for all predictors across models < 5 indicating acceptable multicollinearity; SSPA = Social Skills Performance Assessment, SLOF = Specific Levels of Functioning, UPSA = UCSD Performance-Based Skills Assessment, SSD = schizophrenia spectrum disorder.

Predictive validity of the OSCARS was also examined through a series of hierarchical regressions predicting SC task performance from OSCARS total scores (see Table 7). Self- and informant-reported OSCARS scores did not predict SC task performance.

Table 7.

Regressions Predicting Social Cognition Task Performance from OSCARS in SSD

Outcome	Adjusted R2	Change in R2	Model Statistic	p
SC Composite
OSCARS Self-Report
Total	.29	<.01a	F(4, 275) = 29.0	<.01
Global	.29	<.01	F (5, 274) = 23.3	<.01
OSCARS Informant-Report
Total	.29	<.01a	F(4, 160) = 17.9	<.01
Global	.30	.01	F (5, 159) = 14.9	<.01

Note:

^achange in R² from covariate (sex, years of education, WRAT3 score) model

^*indicates significant model improvement, p <.05

SC = social cognition; Due to significant correlations between predictors, variance inflation factor (VIF) statistics were calculated, VIF values for all predictors across models < 5 indicating acceptable multicollinearity.

3.7. Predictive Validity Identifying Social Cognition Impairment

ROC analyses yielding AUC values examined the diagnostic utility of the OSCARS to identify individuals with SSD for comparison with Healey et al. (2015). Generally, AUCs of .70 −.79 reflect fair, .80 - .89 good, and .90 – 1.00 excellent values (Swets, 1988). AUC values for predicting SSD diagnoses were fair for OSCARS self-reported (AUC = .77 [95% CI .73, .81]) and informant-reported (AUC = .76 [95% CI .68, .85]) total scores.

However, AUC values may be inflated when distinguishing a clinical sample from a HC sample (Whiting, Rutjes, Reitsma, Bossuyt, & Kleijnen, 2003). Therefore, predictive utility of the OSCARS was also examined in a more conservative manner by distinguishing individuals with impaired SC task performance (i.e., SC composite score 1.5 standard deviations below average HC performance) from the rest of the sample. AUC values for the OSCARS self-reported total (AUC = .69 [95% CI .61, .77]) and global scores (AUC = .68 [95% CI .60, .76]) were fair. OSCARS informant-reported total (AUC = .58 [95% CI .46, .70]) and global scores (AUC = .64 [95% CI .51, .76]) were less impressive, albeit not significantly lower than self-report (Venkatraman’s test ps >.05, see Table 8). OSCARS total and global score cut points for identifying impaired SC based on optimal sensitivity and specificity values are presented in Table 8.

Table 8.

OSCARS Predicting Impaired Social Cognition Task Performance

OSCARS Scale	AUC [95% CI]	Score Range		Sensitivity	Specificity
		Low Risk	Elevated Risk
OSCARS Self-Report
Total*	.69 [.61, .77]	≤ 15	> 15	.59	.72
Global*	.68 [.60, .76]	≤ 3	> 3	.56	.70
OSCARS Informant-Report
Total	.58 [.46, .70]	-	-	-	-
Global*	.64 [.51, .76]	≤ 2	> 2	.21	.90

Note: AUC = Area under the curve; Impaired social cognition task performance indicated by performance 1.5 standard deviations or more below average performance of healthy control sample

^*p <.05.

3.8. Exploratory Predictive Validity Identifying Functional Impairment

A review of correlations between the OSCARS and other measures revealed roughly equivalent associations between the OSCARS, SC, and NC with strong correlations observed with functional outcomes (Table 5). Given this finding, as well as recent work demonstrating relationships between the OSCARS and functioning (e.g., Oliveri, Awerbuch, Jarskog, Penn, Pinkham, & Harvey, 2020; Silberstein et al., 2018), exploratory analyses examined the predictive validity of the OSCARS identifying impaired functioning. Impaired functioning was defined as 1.5 standard deviations or more below average HC performance (i.e., SSPA and SLOF). A published cut score for UPSA impairment (i.e., a score of 75; Mausbach et al., 2008) was used since this measure was only administered to the SSD group. All AUC values for the OSCARS predicting functional impairment were significant with the exception of impairment on the UPSA. Significant AUC values were fair to good (i.e., .60 - .82; see Table 9 for AUC values and suggested OSCARS cut scores to identify impairment).

Table 9.

OSCARS Predicting Impaired Functioning

Functional Measure	AUC [95% CI]	Score Range		Sensitivity	Specificity
		High Risk	Low Risk
SLOF Self-Report
OSCARS Self-Report
Total*	.82 [.77, .87]	≤ 16	> 16	.82	.74
Global*	.81 [.75, .86]	≤ 3	> 3	.75	.77
OSCARS Informant-Report
Total*	.75 [.66, .84]	≤ 14	> 14	.92	.45
Global*	.74 [.65, .83]	≤ 4	> 4	.70	.77
SLOF Informant-Report
OSCARS Self-Report
Total*	.62 [.54, .70]	≤ 17	> 17	.64	.56
Global*	.64 [.56, .72]	≤ 2	> 2	.80	.41
OSCARS Informant-Report
Total*	.85 [.79, .90]	≤ 22	> 22	.84	.76
Global*	.85 [.79, .91]	≤ 4	> 4	.80	.80
SSPA
OSCARS Self-Report
Total*	.65 [.59, .71]	≤ 18	> 18	.49	.75
Global*	.64 [.58, .71]	≤ 2	> 2	.59	.59
OSCARS Informant-Report
Total*	.60 [.52, .69]	≤ 19	> 19	.69	.54
Global*	.61 [.52, .70]	≤ 2	> 2	.76	.45
UPSAa
OSCARS Self-Report
Total*	.61 [.54, .67]	≤ 21	> 21	.47	.75
Global*	.59 [.52, .66]	≤ 3	> 3	.69	.53
OSCARS Informant-Report
Total	.59 [.50, .67]	-	-	-	-
Global	.58 [.50, .67]	-	-	-	-

Note: AUC = Area under the curve, SLOF = Specific Levels of Functioning Scale, SSPA = Social Skills Performance Assessment, UPSA = UCSD Performance-Based Skills Assessment; Impaired functioning indicated by performance 1.5 standard deviations or more below average performance of healthy control sample

^aNo healthy control comparison sample so impaired functioning indicated by published cut-off score of 75 (Mausbach et al., 2008)

^*p <.05.

4. Discussion

The present study is the first to demonstrate the psychometric properties of the OSCARS when administered as a self-report measure. It also replicated findings from Healey et al. (2015) demonstrating the psychometric reliability of the OSCARS when administered as an informant-based measure in an independent sample over five times the size of the original validation sample. Overall, results indicate the OSCARS is a reliable, easily administered, and timeefficient measure. Surprisingly, the predictive validity of the OSCARS predicting task-based SC performance and identifying SC impairments was only fair. Follow-up analyses revealed improved predictive validity identifying impairments in real-world functioning (i.e., SLOF), suggesting the OSCARS may be better conceptualized as a measure of functioning.

The OSCARS exhibited good construct validity (Aim 1) with factor analyses confirming a two-dimensional model with interpretable factors across informant-report and self-report administration. This two-factor structure (i.e., SC Ability and SC Bias) replicates previous findings for separate constructs of skill and bias in SSD (Buck, Healey, Gagen, Roberts, & Penn, 2016). Good construct validity was also evidenced by significantly better HC group scores compared with the SSD group on all OSCARS scores.

The OSCARS displayed good internal consistency and modest convergent validity with significant small to medium correlations with all SC tasks (Aim 3). The SCOPE study suggested SC tasks were not impressive in their prediction of everyday outcomes and instead exhibited stronger correlations with performance-based measures of NC (Pinkham et al., 2018). Consistent with this idea, small to medium correlations between the OSCARS and some functional outcomes demonstrated external validity. Divergent validity of the OSCARS was equivocal with significant correlations observed with measures of NC but no significant correlations with negative symptoms. Recent findings suggest NC and SC may be less distinct than previously thought, but roughly equivalent relationships between NC and SC with the OSCARS is surprising and suggests potential low divergent validity (Deckler, Hodgins, Pinkham, Penn, & Harvey, 2018). Alternately, previous work demonstrates negative symptoms and SC are distinct domains and non-significant relationships observed between the OSCARS and negative symptoms offers some support for divergent validity (Bell, Corbera, Johannesen, Fiszdon, & Wexler, 2013; Sergi et al., 2007).

Finally, the OSCARS also demonstrated equivocal predictive validity dependent on the type of outcome measure. Self-report and informant-report showed incremental validity predicting functional outcomes above and beyond demographic characteristics on self and informant-reported SLOF scores but not UPSA or SSPA performance. This null finding may reflect modest rather than strong correlations with functional outcomes, a replication of previous studies examining relationships between report-style cognition measures (e.g., Cognitive Assessment Interview) and functional outcomes (Ventura al., 2008). SC Ability and SC Bias each demonstrated significant relationships with the OSCARS global score (Aim 2) suggesting both factors influence self and informant-reported overall assessment of SC.

After establishing psychometric properties, three hypotheses investigated the performance and utility of the OSCARS. The first hypothesis received modest support. Informant-reported OSCARS scores significantly predicted informant and self-reported SLOF scores but not SSPA or UPSA performance. Results also modestly support the second hypothesis: OSCARS informant-report significantly predicted impaired SC task performance and ROC analyses predicting SSD diagnostic group and impaired SC demonstrated fair predictive validity. Interestingly, OSCARS self-report exhibited stronger AUC values for identifying impaired SC performance compared with informant-report, although this difference was not statistically significant. One explanation for this finding may be the inclusion of attribution bias in our SC composite score. Self-assessment has been shown more effective in evaluating effects of attribution style on outcomes (Vidarsdottir et al., 2019).

Exploratory analyses predicting functional impairment were conducted since the OSCARS demonstrated roughly equivalent correlations with SC and NC measures but noticeably stronger relationships with functional outcomes. The OSCARS demonstrated better utility predicting real-world functional impairments compared with SC impairments (i.e., the highest AUC values were observed when using the OSCARS to predict functional impairment measured by the SLOF). This finding provides preliminary support for the OSCARS as a measure of real-world functioning in SSD. However, the utility of the OSCARS as a screener for impairments in functioning was exploratory and future validation research is needed.

Good psychometric properties, modest correlations with performance-based SC tasks and NC tasks, and strong relationships with functional outcomes suggest the OSCARS may be best utilized as an efficient screening tool for clinicians to identify individuals with impaired functioning to receive psychosocial treatments or additional assessment. The OSCARS is a free instrument with an administration time of five minutes, is available as a paper questionnaire, and does not require a trained rater for administration or scoring. Additionally, results support the validity of both self- and informant-report versions. These qualities make the OSCARS an attractive and easily disseminated first-step measure to be used early in treatment initiation and planning. Early detection of impairment in functioning may facilitate enrollment in psychosocial interventions yielding improvements in treatment outcomes (e.g., Mueser, Deavers, Penn, & Cassisi, 2013).

Informants included in the present study were heterogenous with significant differences in informant roles observed between SSD and HC groups. Informant role likely influences the quality of relationship and subsequent informant-reported abilities. Future work should collect quality ratings of informant-individual relationships, in addition to quantity of time spent together, to investigate potential influences on OSCARS performance. Finally, some validity analyses (e.g., external and predictive validity) utilized the same informant source (e.g., informant-reported OSCARS and informant-reported SLOF) and significant findings may reflect shared measurement variance. However, method variance does not account for incremental validity in predicting functional outcomes and impairment from different informant sources (e.g., informant-reported OSCARS explains additional variance in self-reported SLOF outcomes). Another limitation is the use of an abbreviated MATRICS Battery to assess divergent validity. Significant relationships observed between the OSCARS and NC are congruent with recent models demonstrating significant and complex relationships between NC and SC, especially as they relate to functioning (e.g., Green, Horan, & Lee, 2019; Hasson-Ohayon, Goldzweig, LaviRotenberg, Luther, & Lysaker, 2018). However, the similar correlation coefficients observed across both NC and SC is surprising as stronger correlations were expected with SC. These findings suggest potential low divergent validity of the OSCARS and highlight the need to identify stronger measures of divergent validity (e.g., domains where no associations are expected) for assessing measures of SC.

Notwithstanding these limitations, the present study provides support for the OSCARS as an adequate brief measure of SC deficits in SSD with preliminary support that the OSCARS may be a good measure to identify impairment in functioning. Additionally, the OSCARS can be flexibly administered in either an informant or self-report format. Future work is needed to investigate discrepancies between self-reports and informant impressions as these may provide useful information about limitations in self-assessment of functioning and SC abilities. Both formats demonstrated acceptable psychometric properties and good validity with some support for use as a screening tool to detect impairment in SC and more promising preliminary support to detect impairment in real-world functioning. When making administration decisions, clinicians are encouraged to consider time-constraints and availability of informants given roughly equivalent psychometric properties of self and informant-report. Overall, findings best support that the OSCARS may be a useful first-step tool for clinicians to identify real-world functioning deficits and efficiently identify individuals in need of additional assessment or psychosocial interventions.

Highlights.

• Psychometric investigation of the brief Observable Social Cognition Rating Scale (OSCARS) in a large sample of individuals with schizophrenia spectrum disorder (n = 382) and individuals without a psychiatric diagnosis (n = 289).

• Novel presentation of self-report psychometric properties of the OSCARS and comparison with psychometric properties of informant-report.

• The OSCARS had good convergent validity and adequate external validity, predictive validity, and areas under the curve (AUC) values for predicting impaired social cognition and everyday functioning.

• The OSCARS may be a useful first-step tool for clinicians to detect social cognition and everyday functioning deficits in schizophrenia and efficiently identify individuals in need of additional assessment or psychosocial interventions.