The Social Cognition Psychometric Evaluation Study: Results of the Expert Survey and RAND Panel

Abstract

Background: In schizophrenia, social cognition is strongly linked to functional outcome and is increasingly seen as a viable treatment target. The goal of the Social Cognition Psychometric Evaluation (SCOPE) study is to identify and improve the best existing measures of social cognition so they can be suitably applied in large-scale treatment studies. Initial phases of this project sought to (1) develop consensus on critical domains of social cognition and (2) identify the best existing measures of social cognition for use in treatment studies. Methods: Experts in social cognition were invited to nominate key domains of social cognition and the best measures of those domains. Nominations for measures were reduced according to set criteria, and all available psychometric information about these measures was summarized and provided to RAND panelists. Panelists rated the quality of each measure on multiple criteria, and diverging ratings were discussed at the in-person meeting to obtain consensus. Results: Expert surveys identified 4 core domains of social cognition—emotion processing, social perception, theory of mind/mental state attribution, and attributional style/bias. Using RAND panel consensus ratings, the following measures were selected for further evaluation: Ambiguous Intentions Hostility Questionnaire, Bell Lysaker Emotion Recognition Task, Penn Emotion Recognition Test, Relationships Across Domains, Reading the Mind in the Eyes Test, The Awareness of Social Inferences Test, Hinting Task, and Trustworthiness Task. Discussion: While it was possible to establish consensus, only a limited amount of psychometric information is currently available for the candidate measures, which underscores the need for well-validated and standardized measures in this area.

Introduction

An NIMH Workshop on Social Cognition in Schizo phrenia defined social cognition as “the mental operations that underlie social interactions, including perceiving, interpreting, and generating responses to the intentions, dispositions, and behaviors of others.”^{1 (p1211)} This definition emphasizes a direct link between social cognition and social behavior and suggests that impaired social cognition in schizophrenia may be a primary mechanism through which patients show poor social functioning.

Social cognition has emerged as a major focus of study in schizophrenia, with the number of publications devoted to the topic increasing substantially since the turn of the century.² Fueling interest is a convergence of data indicating that individuals with schizophrenia are impaired in social cognitive abilities, including facial affect perception,³ recognition of emotional prosody,⁴ and theory of mind (ToM),⁵ as well as evidence indicating that individuals with persecutory ideation tend to show attribution biases such as blaming others, rather than situations, for negative events.^6–8 Growing evidence also indicates that impairments in social cognition may precede onset of the disorder^9,10 and are present early in the illness.^11–13

Interest in social cognition in schizophrenia has also increased because it contributes to a variety of real-world outcomes, such as social competence, community functioning, and quality of life.¹⁴ Several studies suggest that social cognition contributes to functional outcomes beyond the influence of neurocognition and may have a greater impact than cognition on social outcomes.^15–20 In addition, social cognition may mediate the relationship between neurocognition and social functions in both chronic^21–24 and first-episode patients.²⁵ Importantly, treating social cognitive deficits leads to improvements in real-world social outcomes,^26,27 including social adjustment,²⁸ social functioning,^29–31 social relationships,^32,33 aggressive incidents,³² and social skills.³⁴

These findings identify social cognition as an important target for pharmacological and psychosocial treatments. However, challenges remain in using social cognition as an endpoint. First, there is no consensus on exactly which abilities define the construct, and considerable conceptual and measurement-related overlap exists among domains.^1,35 The few factor analyses that have been completed suggest social cognition may be best parsed according to level of information processing (ie, perception vs inferential and regulatory processing) rather than domain of social information (eg, emotion vs mental state)^36,37; however, considerably more work is needed before the long-standing practice of parsing social cognition into domains can be discarded.

Perhaps a more critical issue is that there is no consensus about which measures best index a given social cognitive domain. In a recent commentary,³⁸ it was noted that “...there are few widely accepted standardized measures of social cognition that are available for use with schizophrenic populations” (p. 61). A recent meta-analysis of ToM⁵ concluded that “…heterogeneity of the methods used to assess ToM abilities contributes to the inconsistency of the reported findings” (p. 7).

Further, the majority of extant social cognition measures have poor psychometric properties. Yager and Ehmann³⁸ note, “…the psychometric properties of a large proportion of social cognitive measures have not been well-documented” (p. 61), and Bora et al⁵ conclude their meta-analysis with the recommendation that future research “use ToM tasks with better psychometric properties” (p. 8). In a recent review of measures of emotional prosody,⁴ 14 of 21 tasks were rated as “low quality,” with only 2 receiving excellent ratings. This problem is present across all social cognition domains.

The Social Cognition Psychometric Evaluation (SCOPE) study was designed to address these problems. The primary goals of SCOPE are to achieve a consensus on the crucial social cognitive domains in schizophrenia and to evaluate the psychometric properties of existing measures and their suitability for clinical trials. To meet these goals, SCOPE has 5 phases. Phase 1 was an extensive survey of experts in the field to identify the core domains of social cognition in schizophrenia and the best existing tasks assessing each domain. Phase 2 used methods similar to other NIMH measurement initiatives (eg, MATRICS, MATRICS-CT, and VALERO), in which a carefully selected group of expert panelists evaluate identified tasks using a consensus procedure, the RAND Appropriateness Method.³⁹ Based on these assessments, the most promising tasks within each domain were selected for further evaluation and development. In phase 3, these candidate measures will be administered to large samples of individuals with schizophrenia and healthy controls to assess the reliability and validity characteristics of each task. In phase 4, promising candidate measures that show inadequate characteristics will be modified and pilot tested to evaluate the results of the modifications. Finally, in phase 5, a large validation study will be conducted to determine the psychometric properties of the final measures. This portion of the project will also emphasize criterion validity by systematically examining the relationship between these refined social cognition measures and aspects of functional outcome. In this article, we report the methods and results of the first 2 phases.

Methods

Expert Surveys

Two different surveys were conducted. The first elicited nominations for important domains of social cognition and the best existing measures of those domains. The second was a follow-up survey to obtain additional information on the nominated domains and their definitions. For both surveys, the principal investigators (PIs; Drs P.D.H., D.L.P., and A.E.P.) compiled a list of experts in the fields of schizophrenia, social psychology, and autism based on literature searches and knowledge of these fields. Experts were selected if they performed psychological, neurobiological, psychophysiological, or neuroimaging research on social cognition, broadly defined. Experts from fields other than schizophrenia were included to incorporate important concepts from closely related areas.

From March to July 2012, 132 individuals were invited via e-mail. These invitations included a brief description of the SCOPE project and asked respondents to nominate the key domains of social cognition and tasks that they thought were the best measures of social cognition. The RAND panelists were also invited to submit nominations.

Upon completion of the survey, the PIs compiled a list of all domains that received more than 1 nomination. Definitions for each identified domain were then derived from the existing literature, and these domains and their definitions were presented back to the experts in a second, follow-up survey in July 2012. This survey asked experts to indicate (1) the degree to which they thought the construct represented a valid domain of social cognition, (2) the degree to which they believed this domain was important to their area of research, and (3) the degree to which they agreed with the definition provided. At the end of the survey, respondents were asked if they believed any domains should be added, combined, or split, and there was space for additional comments. All experts who had been invited to participate in the first survey were also invited to participate in the second.

The total number of nominated measures (n = 108) was too large to be evaluated by the RAND panel, so a subgroup of measures was selected for evaluation according to set criteria. These criteria were determined by the PIs with input from the primary study consultants (Drs Jean Addington and Michael Green). For measures that were easily categorized into 1 of the 4 consensus social cognitive domains from the surveys, a minimum of 2 measures were selected from each domain using the combination of the following criteria: (1) highest number of nominations, excluding those from the test developer, (2) highest average number of citations per year for the primary publication of the measure, (3) assessment of a subdomain or skill that was not already mostly or fully accounted for by a previously selected measure, and (4) evidence that the measure is sensitive to treatment effects. For measures that did not fit cleanly into 1 of the 4 domains, but still seemed promising, the following criteria were used: (1) some psychometric data were available, (2) if not used in schizophrenia, the measure had been used in 2 or more clinical populations, and (3) of those under consideration, the measure had among the highest average number of citations per year for the primary publication.

RAND Panel

The RAND/UCLA appropriateness method was originally developed as a way of combining the best available scientific evidence with the collective judgment of experts to reach consensus on the appropriateness of medical procedures. This method has since been used beneficially in efforts to develop consensus-based test batteries for use in clinical trials of schizophrenia (eg, MATRICS⁴⁰ and VALERO⁴¹), and it uniquely suits the goals of SCOPE. It begins by identifying a panel of experts who review all available scientific information about a given topic. Once reviewed, panelists individually provide initial ratings of the appropriateness of various assessments or procedures, or in our case, the extent to which social cognitive measures meet selection criteria. In a second round of ratings, panel members convene in a moderated meeting to discuss areas of disagreement and to rerate each criterion. These final ratings are then used to select the battery of tasks to be further developed.

Here, 11 experts agreed to serve as panelists (see list in the Appendix) and represented the areas of social psychology, social neuroscience, biostatistics, and schizophrenia (ie, social cognition and/or psychosocial and pharmacological treatment) (One panelist, Dr Park, was unable to attend the meeting and therefore did not complete the second round ratings. Additionally, no panelists reported financial conflicts of interest; however, several panelists [Drs Gur, Horan, Lysaker, and Payne] and Dr Green reported they were involved in the development of candidate measures. To limit any bias in the rating and subsequent selection of measures, involvement in measure development was declared at the beginning of the panel meeting, and final mean ratings were calculated both including and excluding members with potential conflicts. Ratings were essentially unchanged after the exclusions, and the final selection of tests was unaffected.). The panel was cochaired by Drs M.F.G. and A.E.P.

Evaluation Criteria

Characteristics upon which the RAND panel rated each measure were based on those used in MATRICS⁴⁰ and VALERO.⁴¹ These included (1) reliability—test-retest and interrater reliability as applicable, as well as internal consistency, (2) distributions—floor and/or ceiling effects and normality of distributions, (3) utility as a repeated measure—stability over time in the absence of intervention or sensitivity to intervention associated change, (4) convergent and discriminant validity—relationship to social cognitive measures relative to other abilities and constructs, (5) criterion validity—correlations with real-world social outcomes, (6) practicality for administration, and (7) tolerability for patients.

Data Preparation and RAND Process

For each candidate measure, the original publication and all articles citing that original publication were retrieved from 2 search engines: Web of Knowledge and Google Scholar. Two doctoral students in clinical psychology (Mr B.B. and Ms K.H.) examined all articles citing the measure for information pertaining to the 7 criteria and created a comprehensive database that included a brief description of each candidate measure, all data relevant to the 7 criteria, and basic information about each study from which data were drawn (ie, study authors, date, and sample). The final database was thoroughly reviewed by all 3 study PIs for accuracy and comprehensiveness and is available at the SCOPE Web site (http://psychiatry. med.miami.edu/division-of-psychology/research- collaborators). The database was then sent to panelists so they could make preliminary ratings of each measure on each criterion using a 9-point scale in which 1 was poor and 9 was superb. Results were compiled into a summary document displaying histograms of the ratings for each measure on each criterion as well as the mean, median, and SD of these ratings. A summary of this first round of ratings was provided to the panelists at the meeting.

The goal of the RAND panel meeting was to resolve discrepant ratings, which were defined as first round ratings having a SD greater than 1.5 and any individual ratings that were ±2 points from the median. Following discussion, the second round ratings were all more convergent and none met this criterion for discrepancy. These final ratings were then evaluated to identify the highest rated measures within each domain that would progress to the psychometric study.

Results

Expert Surveys

Fifty-nine experts in the fields of schizophrenia (n = 39), social psychology (n = 16), and autism (n = 4) responded to the first survey. A total of 168 nominations were received for possible domains of social cognition with 96 of these being unique terms. After combining terms referring to the same general process (eg, affect recognition and emotion identification), compilation of nominations revealed a consensus on 6 domains: emotion processing, social perception, ToM/mental state attribution, social metacognition, social reciprocity, and attributional style/bias. These 6 domains were the most frequently nominated and collectively accounted for 129 of the 168 nominations received.

Thirty-five experts (24 in schizophrenia, 5 in social psychology, and 6 in autism) responded to the second, follow-up survey. They supported the value and validity of 4 of the nominated domains: emotion processing, social perception, ToM, and attributional style/bias. These responses were also used to revise and augment the domain definitions, which are provided in table 1.

Table 1.

Nominated Domains of Social Cognition and Ratings from the Second Expert Survey

		Mean (SD)
Domain (Number of Nominations)	Definition	Valid	Important	Definition
Emotion processing (36)	This domain is broadly defined as perceiving and using emotions.1 It subsumes 3 subdomains that represent both lower level and higher level processes. At a lower perceptual level is the first subdomain emotion perception/recognition (identifying and recognizing emotional displays from facial expressions and/or nonface cues such as voice), and at a higher level are the 2 subdomains of understanding emotions and managing emotions	7.63 (1.70)	7.51 (1.87)	6.77 (2.16)
Social perception (25)	Social perception refers to decoding and interpreting social cues in others.42–44 It includes social context processing and social knowledge, which can be defined as knowing social rules, roles, and goals (RRGs), utilizing those RRGs, and understanding how such RRGs may influence others’ behaviors25,45	7.63 (1.61)	7.06 (1.51)	6.77 (1.70)
Theory of mind/mental state attribution (42)	This domain is defined as the ability to represent the mental states of others including the inference of intentions, dispositions, and/or beliefs.46,47 Theory of mind is also referred to as mentalizing, mental state attribution, or cognitive empathy48	7.63 (1.61)	7.20 (1.84)	6.46 (2.12)
Attributional style/bias (14)	Attributional style describes the way in which individuals explain the causes, or make sense, of social events or interactions1,46	6.85 (1.91)	5.74 (2.29)	6.40 (1.70)
Social metacognition (9)	Social metacognition refers to the ability to evaluate thinking, including both one’s own thoughts and those of others. This allows for the formation and modification of ideas about oneself in the present and about one’s identity and characteristics over time.49 Self-perception, including agency and self-knowledge is included as a subdomain	6.24 (1.58)	5.57 (2.13)	5.91 (1.85)
Social reciprocity (3)	This domain is defined as engaging in emotionally and socially appropriate turn-taking interactions with others (even if the interaction is only 1 turn each). Social reciprocity requires awareness of the interpersonal cues of others, appropriate responding to those cues, awareness of others’ reactions to themselves and their behaviors, and emotional engagement50	5.83 (1.90)	5.49 (2.17)	5.57 (1.61)

Note: Domains presented in bold font were supported in the second round survey and were identified as the current primary domains of social cognition. Ratings were provided on a scale from 1 to 9. For the first 2 questions about the validity and importance of the domain, 1 = not at all valid/important, 5 = moderately valid/important, and 9 = extremely valid/important. For the last question inquiring about the degree to which one agreed with the definition of each domain, 1 = disagree, 5 = neither agree nor disagree, and 9 = agree. Overall mean ratings for retained domains were significantly higher than those for the dropped domains [t(4) = 3.56, P = .02].

Experts from the first survey also nominated 108 different measures for assessing social cognitive ability. Using the selection criteria detailed above, 21 measures were selected for evaluation by the RAND panel. Table 2 provides a list of the selected measures along with citation information and number of nominations. Of note, the Mayer-Salovey-Caruso Emotional Intelligence Test (MSCEIT)⁵¹ received 8 nominations; however, because the basic psychometric properties of the MSCEIT have already been established,^52,53 it was not considered for inclusion in this portion of the project.

Table 2.

Candidate Measures Selected for Evaluation by the RAND Panel

Measure	SCOPE Domain of Social Cognition	Original Citation	Total Citations	Citations Per Year	Total Useable Articles	Number of Expert Nominations
Ambiguous Intentions and Hostility Questionnaire (AIHQ)	Attributional style/bias	Combs et al54	35	7	12	4
Internal, Personal, and Situational Attributions Questionnaire (IPSAQ)	Attributional style/bias	Kinderman and Bentall55	109	6.81	15	2
Bell Lysaker Emotion Recognition Task (BLERT)	Emotion processing	Bryson et al56	110	7.33	12	5
Diagnostic Analysis of Nonverbal Accuracy 2 (DANVA2)	Emotion processing	Nowicki and Duke57	331	18.56	10	2
Face Emotion Discrimination Test (FEDT)	Emotion processing	Kerr and Neale58	310	16.32	24	2
Penn Emotion Recognition Task (ER-40)	Emotion processing	Kohler et al59	276	30.6	15	10
Half Profile of Nonverbal Sensitivity (Half PONS)	Social perception	Ambady et al60	239	14.06	11	3
Interpersonal Perception Task (IPT-15)	Social perception	Costanzo and Archer61	178	7.74	16	3
Relationships Across Domains (RAD)	Social perception	Sergi et al62	4	1.33	3	1
Adult Faux Pas	ToM	Stone et al63	676	48.29	17	4
Brune Picture Sequencing Task	ToM	Brune64	110	12.22	13	3
Happe’s Stories	ToM	Happe65	820	43.94	8	5
Reading the Mind in the Eyes Test	ToM	Baron-Cohen et al66	1037	94.27	18	13
Silent Animations	ToM	Castelli et al67	708	70.8	14	5
The Awareness of Social Inference Test (TASIT)	ToM	McDonald et al68	94	10.44	11	11
The Hinting Task	ToM	Corcoran et al69	512	30.12	33	10
Visual Perspective Tasking Task	ToM	Langdon and Coltheart70	79	7.18	2	2
Affect Misattribution Procedure (AMP)	Novel	Payne et al71	297	42.43	4	1
Movie for the Assessment of Social Cognition (MASC)	Novel	Dziobek et al72	62	10.33	12	6
Need for Closure Scale	Novel	Webster and Kruglanski73	825	45.83	9	2
Trustworthiness Task	Novel	Adolphs et al74	904	64.57	10	2

Note: The citation provided for the ER-40 is the most widely used, and first, citation for the measure; however, the task utilized in this study was an expanded version that included 96 stimuli. The first time the ER-40 was published in its current form was Kohler et al.⁷⁵ Likewise, the citation provided for the Interpersonal Perception Task refers to the full task of 30 items. The original citation for the 15-item version is Costanzo and Archer.⁷⁶ SCOPE, Social Cognition Psychometric Evaluation.

RAND Panel

From the first round panel ratings, descriptive statistics were calculated for each candidate measure on each rating criterion (ratings available on request). These ratings were reviewed by the panel cochairs, and it was determined that 4 measures, the Interpersonal Perception Task-15, Visual Perspective Taking, Happe’s Stories, and the Affect Misattribution Procedure, received ratings that were substantially below the other measures and too low to justify further consideration (ie, mean of the median ratings for each criterion was less than 4). Thus, 17 measures were considered and rerated during the panel meeting.

Review by the cochairs identified 57 ratings that qualified as discrepant. Discrepancies were more prevalent in ratings of reliability and tolerability, and it became evident during the early portion of the meeting that some criterion definitions varied across panelists. Thus, the group refined the definitions of each criterion. Most notably, in considering reliability, all panelists agreed not to consider Cronbach’s alpha as a criterion because it was noted that a measure could have excellent internal consistency but low test-retest reliability, which is critical for clinical trials. Following clarifications, panelists discussed and rerated each item, at which point, no discrepant items remained. The final descriptive statistics of the second round, consensus ratings are provided in table 3.

Table 3.

Descriptive Statistics for Final (Round 2) RAND Panel Ratings

	Rating Criteria, Mean (SD), Median
	Reliability	Distributions	Utility as a Repeated Measure	Convergent and Discriminant Validity	Criterion Validity	Practicality	Tolerability	Scale Mean Score
AIHQ	6.27 (1.27), 6	4.64 (1.43), 5	3.10 (0.088), 3	2.40 (0.84), 2	2.73 (1.19), 3	5.73 (1.49), 6	6.18 (1.33), 6	4.50 (1.69)
IPSAQ	1.60 (0.70), 1.5	5.64 (1.21), 6	3.45 (1.44), 4	3.10 (0.74), 3	1.00 (0.00), 1	2.90 (0.88), 3	3.40 (1.08), 3	3.00 (1.48)
BLERT	5.27 (1.42), 5	6.18 (0.98), 6	6.09 (1.22), 6	6.09 (1.04), 6	4.91 (0.94), 5	8.00 (0.89), 8	8.18 (0.60), 8	6.43 (1.19)
DANVA2	1.80 (0.63), 2	4.70 (1.16), 5	1.30 (0.48), 1	4.18 (1.40), 5	4.60 (0.70), 4.5	6.09 (1.30), 6	7.00 (1.18), 7	4.16 (2.00)
ER-40	3.30 (0.82), 3.5	6.40 (0.070), 6.5	4.80 (1.03), 5	4.40 (0.84), 5	4.09 (1.38), 4	8.18 (0.98), 8	8.27 (0.65), 8	5.63 (1.96)
FEDT	4.27 (1.49), 4	2.70 (1.64), 2	4.50 (0.97), 5	5.36 (0.92), 5	4.36 (1.43), 4	6.82 (1.25), 7	7.18 (1.33), 7	5.00 (1.47)
Half PONS	2.00 (0.47), 2	7.10 (0.88), 7	2.60 (1.17), 2	5.00 (0.82), 5	5.64 (1.12), 6	4.82 (1.17), 5	4.10 (1.20), 4	4.51 (1.78)
RAD	5.80 (1.03), 6	6.80 (0.92), 7	2.90 (0.88), 3	5.20 (0.79), 5	5.18 (1.40), 5	4.82 (1.17), 5	3.80 (0.92), 4	4.97 (1.28)
Faux Pas	7.00 (2.16), 8	5.09 (1.45), 5	1.40 (1.27), 1	3.82 (0.98), 4	3.73 (1.42), 4	5.09 (0.94), 5	5.18 (1.33), 5	4.56 (1.73)
Brune PS	1.40 (0.84), 1	2.90 (0.88), 3	4.82 (1.08), 5	4.00 (1.10), 4	4.73 (1.01), 5	6.00 (1.34), 6	6.10 (0.88), 6	4.31 (1.70)
Eyes	4.18 (1.47), 4	5.64 (1.43), 6	3.91 (1.22), 4	5.36 (1.21), 6	4.10 (0.88), 4	6.55 (1.04), 6	4.80 (1.23), 5	4.91 (0.89)
SA	2.60 (0.97), 3	5.00 (1.55), 5	1.00 (0.00), 1	3.91 (1.14), 4	1.00 (0.00), 1	5.91 (1.30), 6	7.80 (0.79), 8	3.31 (2.53)
TASIT	5.50 (1.12), 6	5.27 (1.35), 5	4.73 (1.35), 4	4.10 (0.99), 4	3.80 (0.42), 4	4.27 (1.42), 4	4.70 (0.68), 5	4.63 (0.58)
Hinting	2.5 (0.53), 2.5	4.60 (0.70), 4.5	5.09 (0.94), 5	5.00 (1.18), 5	6.60 (0.84), 6	7.18 (1.08), 7	7.20 (0.92), 7.5	5.53 (1.68)
MASC	1.80 (0.63), 2	5.70 (0.68), 6	1.10 (0.32), 1	2.30 (0.82), 2.5	2.45 (1.44), 2	1.70 (0.82), 1.5	5.30 (0.95), 6	2.93 (1.82)
NCS	2.10 (0.57), 2	5.09 (0.83), 5	4.40 (0.97), 4.5	1.90 (0.74), 2	1.82 (1.08), 1	6.64 (1.29), 7	6.73 (1.27), 7	4.17 (2.24)
Trust	1.20 (0.63), 1	2.70 (0.68), 3	2.90 (0.74), 3	1.6 (0.70), 1.5	1.00 (0.00), 1	7.30 (0.68), 7	7.30 (0.68), 7	3.43 (2.74)
Mean criterion score	3.5 (1.92)	5.1 (1.33)	3.4 (1.58)	4.0 (1.33)	3.7 (1.72)	5.5 (1.95)	6.1 (1.53)
Qualitative rating	Fair	Good	Fair	Fair/good	Fair	Good	Good/very good

Note: AIHQ, Ambiguous Intentions Hostility Questionnaire; IPSAQ, Internal, Personal, and Situational Attributions Questionnaire; BLERT, Bell Lysaker Emotion Recognition Task; DANVA2, Diagnostic Analysis of Nonverbal Accuracy 2; ER-40, Penn Emotion Recognition Test; FEDT, Face Emotion Discrimination Test; Half PONS, Half Profile of Nonverbal Sensitivity; RAD, Relationships Across Domains; Faux Pas, Adult Faux Pas; Brune PS, Brune Picture Sequencing Task; Eyes, Reading the Mind in the Eyes Test; SA, Silent Animations; TASIT, The Awareness of Social Inference Test; Hinting, The Hinting Task; MASC, Movie for the Assessment of Social Cognition; NCS, Need for Closure Scale; Trust, Trustworthiness Task. Mean scores for selected tasks were significantly higher than mean scores for the nonselected tasks [t(15) = 2.55, P = .02].

Final Measures

Using the final RAND panelist ratings, the PIs selected measures that will be included in the psychometric study. Within domains, measures falling in the top half of the distribution of overall means were chosen. Due to the different number of tests in each domain evaluated by the RAND panel, this resulted in 1 task each for the domains of attributions and social perception, 2 tasks for emotion processing, and 3 tasks for ToM. Inclusion of multiple tasks for emotion processing and ToM ensures wider coverage of these multifaceted domains and is consistent with the greater number of nominated tasks within these domains. One task from the novel category was also selected (discussed below).

Selected measures are listed below, and a description of each is provided in table 4:

Table 4.

Description of Final Measures

Task	Format	Description
Ambiguous Intentions and Hostility Questionnaire, abbreviated version	Paper and pencil	This task is designed to evaluate hostile social cognitive biases. Participants read 5 hypothetical, negative situations with causes that are ambiguous (ie, they could be intentional or accidental), imagine the scenario happening to them, and record a reason why the scenario occurred. Independent raters later code this initial response to compute a hostility index (range = 1–5). Participants then use Likert scales to rate whether the other person/s performed the action on purpose (1 “definitely no” to 6 “definitely yes”), how angry it would make them feel (1 “not angry at all” to 5 “very angry”), and how much they would blame the other person/s (1 “not at all” to 5 “very much”). Finally, the participant is asked to write down how they would respond to the situation, which is later coded by 2 independent raters to compute an aggression index (range = 1–5)
Bell Lysaker Emotion Recognition Task	Stimuli presented via computer, responses recorded by experimenter	The BLERT measures the ability to correctly identify 7 emotional states: happiness, sadness, fear, disgust, surprise, anger, or no emotion. Participants view 21 ten-second video clips of a male actor, which provide dynamic facial, vocal-tonal, and upper body movement cues. After viewing each video, participants identify the expressed emotion. Performance is indexed as the total number of correctly identified emotions (ranging from 0 to 21)
Penn Emotion Recognition Test	Computer administered	The ER-40 assesses facial affect recognition ability. It includes 40 color photographs of static faces expressing 4 basic emotions (ie, happiness, sadness, anger, or fear) and neutral expressions. Stimuli are balanced for poser’s gender, age, and ethnicity, and for each emotion category, 4 high-intensity and 4 low-intensity expressions are included. Participants view 1 image at a time and choose the correct emotion label for each face. Accuracy scores, ranging from 0 to 40, serve as the primary dependent variable
Relationships Across Domains, abbreviated version	Paper and pencil	The RAD measures competence in relationship perception. The content and format are based on relational models theory, which proposes that individuals use their implicit knowledge of 4 relational models (ie, communal sharing, authority ranking, equality matching, and market pricing) to understand social relationships and predict the behavior of others. The abbreviated RAD is comprised of 15 vignettes involving different male-female dyads that represent one of the relational models. Participants read each vignette and answer 3 yes/no questions about whether a future behavior is likely to happen given the described relationship. Performance is indexed as the total number of correct responses (ranging from 0 to 45)
Reading the Mind in the Eyes Test	Stimuli presented via computer, responses recorded by experimenter	The Eyes task measures the capacity to discriminate the mental state of others from expressions in the eye region of the face. Participants view 36 photos of the eye region of different faces and choose the most accurate descriptor word for the thought/feeling that is portrayed. Four possible options are presented with each photo, and a glossary of mental state terms is provided for reference. Total score is the number of correct responses, and scores range from 0 to 36
The Awareness of Social Inferences Test, Part III	Stimuli presented via computer, responses recorded by experimenter	The TASIT is comprised of videotaped vignettes of everyday social interactions, and Part III, the Social Inference-Enriched test, assesses detection of lies and sarcasm. Participants watch each vignette and answer 4 standard questions per vignette that probe understanding of the intentions, beliefs, and meanings of the speakers and their exchanges. Scores range from 0 to 64
Hinting Task	Paper and pencil	The Hinting Task examines the ability of individuals to infer the true intent of indirect speech. The task includes 10 short passages presenting an interaction between 2 characters that are read aloud by the experimenter. Each passage ends with one of the characters dropping a hint, and participants are asked what the character truly meant. If the first response provided is inaccurate, a second hint is delivered, and participants may earn partial credit for that passage. Total scores range from 0 to 20
Trustworthiness Task, abbreviated version	Stimuli presented via computer, responses recorded by experimenter	Participants rate 42 faces for trustworthiness on a scale from −3 to 3. Faces are presented in gray scale and represent ethnically diverse males and females. The task assesses participants’ ability to make complex social judgments by comparing the participant ratings to normative data

Note: Abbreviations are explained in the first footnote to table 3.

1. Attributional style—Ambiguous Intentions and Hostility Questionnaire.

2. Emotion processing—Bell Lysaker Emotion Recog nition Task and Penn Emotion Recognition Test.

3. Social perception—Relationships Across Domains.

4. ToM—Reading the Mind in the Eyes, The Awareness of Social Inferences Test—Part III, and Hinting Task.

5. Novel—Trustworthiness Task.

Discussion

In this article, we present the results of a consensus process to identify the domains of social cognition that are most important within the field of schizophrenia research and the best existing measures of those domains. Two expert surveys identified and supported the value of 4 domains—emotion processing, social perception, ToM/mental state attribution, and attributional style/bias. Additionally, 8 measures were selected for psychometric evaluation via expert survey and RAND panel ratings.

The selected domains are fully consistent with the previous literature,^1,46,77 suggesting that the existing conceptual framework of domains appears to be useful to the field. The nomination of 2 additional domains, social metacognition and social reciprocity, however suggest avenues for expansion of social cognitive research. Factor analytic studies are also still needed to examine the independence of these domains and to determine if the underlying structure of social cognition truly maps meaningfully onto these identified domains. The domains presented here represent those that are prominent in schizophrenia research, but they have not been conclusively proven to be fundamentally important to the disorder. The data collected from the remaining phases of SCOPE will allow for such investigations.

Over 100 measures were nominated, and 21 were forwarded to the RAND panel for consideration. On the positive side, these numbers provide strong evidence for the health, breadth, and diversity of social cognitive research. From a more negative viewpoint, however, the use of so many different measures can result in the problems noted previously, namely the lack of standardized measures and potential for discrepant findings between studies that may be due simply to measurement or task idiosyncrasies rather than true differences. In this respect, a widely applicable, standardized social cognitive battery such as the one that will result from the SCOPE project is likely to advance our understanding of social cognitive impairment.

The RAND panel ratings underscore the dearth of well-validated and standardized measures. In first round ratings, no measure received an average rating in the very good to superior range, and the majority of measures fell within only the fair to good range. Likewise, after consensus, only 4 of the 21 measures received average ratings in the good range, with the remainder rated as only fair. The decrease in ratings can in part be explained by lower ratings for the reliability criterion. Because most studies have relied on Cronbach’s alpha to establish reliability and very few have reported test-retest reliability, this resulted in a lack of reliability information for most measures. The lack of psychometric information was also seen in the ratings for utility as a repeated measure and criterion validity because both criteria were among those with the lowest mean ratings. An important goal of SCOPE will be to provide such information about the final measures so that their appropriateness for clinical trials can be properly evaluated.

It is notable that in our first survey on domains of social cognition, empathy received 9 nominations, although some of these included empathy as a subcomponent of ToM. These nominations highlight the lack of clarity in the field about how empathy fits within the current conceptualization of social cognition and whether it is a process that subsumes, or is subsumed by, ToM. In the second survey, we included empathy as a subdomain of ToM, and several experts, citing recent conceptual work on this topic (eg, Shamay-Tsoory⁴⁸ and Decety and Svetlova⁷⁸), disagreed with this approach, noting that empathy should be included as its own domain. Based on this feedback, the SCOPE project now conceptualizes empathy as an overarching ability that includes both cognitive and affective components, and we have specified that only cognitive empathy is included in our definition of ToM. We have refrained, however, from including the overall construct of empathy as a separate domain because of a lack of nominations for tasks that capture the affective component of empathy and the need for additional work clarifying the relationships among empathy, ToM, and emotion perception in schizophrenia. Nevertheless, our surveys highlight empathy as an area of importance within the field that will likely be a focus of future work.

Further, although the definition of emotion processing is divided into lower level perception and higher level regulation, the selected final measures for this domain only pertain to the perception subdomain. From the expert survey, the MSCEIT⁵¹ was the only nominated measure of emotional regulation. As noted above, however, the basic psychometric properties of the MSCEIT have already been established, and it was therefore not considered for inclusion in the initial psychometric study. It will though be included in the final validation study to allow full representation of this domain.

Finally, within the novel category of tasks, the Trustworthiness Task was selected for further study despite receiving lower ratings than the Need for Closure Scale (NFC). Several factors contributed to this decision. First, in contrast to the self-report format of the NFC, the Trustworthiness Task has the advantage of being performance based. Second, the RAND panel noted that low ratings for this measure were due to a lack of data from patients with schizophrenia but commented favorably on its promise, citing its use in a variety of clinical populations including brain lesion patients and individuals with autism spectrum disorders. Third, high ratings for practicality and tolerability suggest a minimal tradeoff should the measure be found unsuitable.

Limitations of the current study and method also require consideration. The response rates for both expert surveys were relatively low, 45% and 35%, respectively, which may have influenced the results. Additionally, the final battery of tasks includes only 1 measure each from the domains of social perception and attributional style. This reflects the limited number of measures available for assessing these domains and suggests that these 2 areas may be particularly in need of further development. Finally, the expert survey and RAND panel methods also inadvertently favor more established, well-known tasks over those that are novel or innovative. We attempted to reduce this limitation by considering a “novel” category and including tasks that showed promise but that were not widely used in schizophrenia. Nevertheless, the majority of tasks selected for psychometric evaluation are those that are generally familiar to the field. While the goal of SCOPE is not to develop new measures of social cognition, state-of-the-art measures that capitalize on recent developments in the field are certainly needed and should be pursued in future work.

The first 2 phases of SCOPE have yielded expert consensus on the key domains of social cognition within schizophrenia research and identified the most promising measures of these domains at the present time. The lack of available information regarding the psychometric properties of these tasks has been demonstrated, and the need for studies designed to assess the psychometric properties of these measures is clear. The upcoming phases of SCOPE will address this need, and specifically, the next phase will directly examine the most problematic aspects of the current measures: reliability and utility as a repeated measure.

Funding

National Institute of Mental Health at the National Institutes of Health (R01 MH093432 to P.D.H., D.L.P., and A.E.P.).

Appendix: RAND Panel Members

Chair:

Michael Green, PhD (University of California, Los Angeles)

Cochair:

Amy Pinkham, PhD (Southern Methodist University)

Members:

Eric Granholm, PhD (University of California, San Diego and San Diego VA Medical Center)

Ruben Gur (University of Pennsylvania School of Medicine)

William Horan, PhD (University of California, Los Angeles)

Helena Kramer, PhD (Stanford University)

Ann Kring, PhD (University of California, Berkeley)

Matthew Lieberman, PhD (University of California, Los Angeles)

Antony Loebel, MD (Sunovion Pharma America)

Paul Lysaker, PhD (Roudebush VA Medical Center, Indianapolis, IN and Indiana University School of Medicine)

Sohee Park, PhD (Vanderbilt University)

Keith Payne, PhD (University of North Carolina at Chapel Hill)

Sophia Vinogradov, MD (University of California, San Francisco and San Francisco VA Medical Center)