The “Reading the Mind in the Eyes” test: Systematic review of psychometric properties and a validation study in Italy

Abstract

Introduction

The “Reading the Mind in the Eyes” test (henceforth, Eyes test) is a simple but advanced Theory of Mind test, and it is widely used across different cultures. This study assessed the reliability and construct (convergent and discriminant) validity of the Eyes test in Italy.

Methods

A sample of 18- to 32-year-old undergraduate students of both sexes (N = 200, males = 46%) were invited to fill in the Italian version of the Eyes test, the Empathy Quotient (EQ), the Toronto Alexithymia Scale (TAS), and the Marlowe-Crowne Social Desirability Scale (SDS).

Results

Internal consistency (Cronbach’s alpha) was .605. Confirmatory factor analysis provided evidence for a unidimensional model, with maximal weighted internal consistency reliability =.719. Test–retest reliability for the Eyes test, as measured by intraclass correlation coefficient, was .833 (95% confidence interval = .745 to .902). Females scored significantly higher than males on both the Eyes test and the EQ, replicating earlier work. Those participants who scored lower than 30 on the EQ (n = 10) also scored lower on the Eyes test than those who did not (p <.05). Eyes test scores were not related to social desirability.

Conclusions

This study confirms the validity of the Eyes test. Both internal consistency and test–retest stability were good for the Italian version of the Eyes test.

Introduction

According to the evolutionary psychology framework (Cosmides & Tooby, 1997), emotions are automatic/involuntary response coordination mechanisms, accompanied by distinct types of physiological arousal, selected by evolution to react to basic environmental challenges such as avoiding physical harm (fear), contaminants (disgust), discouraging intentional damage to basic resources (anger), orienting towards novel cues (surprise), signalling an actual or potential loss (sadness), and attracting companionship or potential mates (joy). In this framework, emotional states are seen as having been shaped by natural selection to enable animals to cope with the threats and opportunities of their physical and social environment (Barrett, Cosmides, & Tooby, 2010; Cosmides & Tooby, 1997).

Emotion recognition is a core component of social cognition, defined as the ability to interpret emotional expressions (in the face, voice, or posture) about others’ mental states, to predict their behaviour (Adolphs, 2009; Gallese, Keysers, & Rizzolatti, 2004). In both nonhuman and human primates, the ability to rapidly and correctly detect others’ emotional states is crucial for social interaction. In humans, effectiveness in deciphering others’ emotions confers an advantage socially and vocationally (Boyatzis & Satyaprasad, 1994; Elfenbein, Foo, White, Tan, & Aik, 2007; Leppänen & Hietanen, 2001). Conversely, emotion recognition deficits are implicated in psychopathology (Baron-Cohen, 2003, 2009; Decety & Moriguchi, 2007; Maggini & Raballo, 2004). In particular, the social-cognitive impairment that affects people with autism or schizophrenia involves selective difficulties in emotion recognition (Baron-Cohen, Jolliffe, Mortimore, & Robertson, 1997; Baron-Cohen, Wheelwright, Hill, Raste, & Plumb, 2001; Green et al., 2005; Larøi, Fonteneau, Mourad, & Raballo, 2010). Deficits in emotion recognition are reported in other conditions as well, such as anorexia nervosa (Harrison, Tchanturia, & Treasure, 2010), bipolar disorder (Derntl, Seidel, Kryspin-Exner, Hasmann, & Dobmeier, 2009; Ryu, An, Jo, & Cho, 2010), and social anxiety (Machado-de-Sousa et al., 2010). Thus, the investigation of emotion recognition and its contribution to social cognition has clinical relevance, and offers clues to the aetiology of these conditions.

We define emotion recognition as the ability to read subtle cues indicating the emotional state of another person. These cues can be both visual, principally in facial expression, and verbal, principally in prosody, both signalling an internal emotional state. Emotion recognition and the perception of behavioural social cues are integral to social cognition (Adolphs, 2009; Gallese et al., 2004; Gallese, Rochat, Cossu, & Sinigaglia, 2009).

Social cognition encompasses a broader set of abilities involved in the detection and processing of information on goals and intentions of conspecifics. Emotional and behavioural social cues enable us to derive information about others’ intentions, beliefs, goals, and desires. The processing of these social cues is sometimes referred to as employing a “theory of mind” (ToM), the ability to attribute mental states to the self and to others, to predict behaviour (Premack & Woodruff, 1978; Samson, 2009).

Different instruments have been developed to assess different components of social cognition. ToM tests generally consist of short stories and questions about them (Bird, Castelli, Malik, Frith, & Husain, 2004; Hallerbäck, Lugnegård, Hjärthag, & Gillberg, 2009; White, Hill, Happé, & Frith, 2009). These tests require intact verbal ability. Instruments testing emotion recognition typically consist of still facial photographs, as in the “Facial emotion identification task” (Ihnen, Penn, Corrigan, & Martin, 1998; Kee, Kern, & Green, 1998), the “Social cognition test” (Hall et al., 2004), or the NimStim Face Stimulus Set (Tottenham et al., 2009). Other tests use vocal stimuli, with the subject invited to identify emotional states from spoken phrases, as in the “Voice Emotion Identification Test” (Kerr & Neale, 1993) or the “Reading the Mind in the Voice” test (Rutherford, Baron-Cohen, & Wheelwright, 2002). A combination of verbal and dynamic visual stimuli is included in the “Cambridge Mind-reading Face Voice Battery” (Golan, Baron-Cohen, & Hill, 2006). Other tests use videotapes depicting social interactions, and the subject is asked about the nature and the quality of these interactions. Examples are the “Social Cue Recognition Test” (Ihnen et al., 1998) and the “Videotape Emotion Identification Test” (Kee et al., 1998). A finer but more time-consuming evaluation of emotion recognition is based on the assessment of computerised distorted facial pictures (morphing) (Scrimin, Moscardino, Capello, Altoè, & Axia, 2009). In all these tests the participant is asked to identify an emotion or to rate its intensity across a range of expressions.

The simplest emotion recognition test and one of the most widely used is the “Reading the Mind in the Eyes” test (henceforth, the Eyes test), which underwent a revision in the adult version, and which includes 36 still pictures of the eye regions illustrating emotionally charged or neutral mental states (Baron-Cohen et al., 1997; Baron-Cohen, Wheelwright, Hill, et al., 2001). The core of the Eyes test involves the matching of the semantic definition of a mental state (e.g., “worried”, “annoyed”) to the picture of the eye-region expression displayed in the screen. This is assumed to involve an unconscious, automatic and rapid matching of past memories concerning similar expressions with a lexicon of mental state terms (Baron-Cohen, Wheelwright, Hill, et al., 2001).

The Eyes test is considered an advanced ToM test, since participants have to put themselves into the mind of the person shown in the photograph, and attribute a relevant mental state to them. However, because the judgements can only be on the basis of facial expression, the test can also be considered an emotion recognition test. The Eyes test has been used in samples of people expected to show impairment of social cognition, such as people with autism, schizophrenia, and anorexia (Baron-Cohen, Wheelwright, Hill, et al., 2001; Harrison et al., 2010; Kettle, O’Brien-Simpson, & Allen, 2008). The Eyes test has also been used to assess the impact of pharmacological treatment on social cognition, in particular to assess the benefit of oxytocin administration in both typical individuals (Domes, Heinrichs, Michel, Berger, & Herpertz, 2007), and in people with autism (Guastella et al., 2010) or schizophrenia (Feifel et al., 2010). On the Eyes test, deficits are found in people with autism (Baron-Cohen, Wheelwright, Hill, et al., 2001), schizophrenia (Kettle et al., 2008; Köther et al., 2012; Schimansky, Rössler, & Haker, 2012), eating disorders, particularly those with anorexia nervosa (Harrison et al., 2010), and in depressed elderly people with a history of suicide attempt (Szanto et al., 2012). Patients with Parkinson’s disease (Tsuruya, Kobayakawa, & Kawamura, 2011) and Huntington’s disease (Allain et al., 2011) also show impairment on the Eyes test. Patients with borderline personality disorder may show enhanced performance compared to typical controls (Fertuck et al., 2009; Schilling et al., 2012), perhaps reflecting heightened attention to affective threat cues in the eye stimuli.

The ability to decode the feelings and thoughts of others from the eyes develops at a young age (Baron-Cohen, Wheelwright, Spong, Scahill, & Lawson, 2001; Gunther Moor et al., 2012). Neuroimaging studies suggest that the Eyes test causes activation of a network including the posterior superior temporal sulcus (pSTS) and the inferior frontal gyrus (Adams et al., 2009; Baron-Cohen et al., 1999, 2006; Gunther Moor et al., 2012). However, various higher order ToM tasks may be related to differing information-processing components, since different ToM tests may tap different domains. For example, Ahmed and Miller (2011) found lack of correlation between the Eyes test, a revised version of the Strange Stories test (Happé, 1994), and the Faux pas test (Gregory et al., 2002) in a sample including 123 typical people. Other studies have found the Eyes test to be highly correlated with the Strange Stories test (Baron-Cohen, Wheelwright, Hill, et al., 2001).

The Eyes test is based on facial pictures of Caucasian people, and has been used in predominantly Anglo-Saxon countries (Goding, Johnson, & Peterman, 2010; Guastella et al., 2010; Harrison et al., 2010; Irani et al., 2006; Kettle et al., 2008; Valla et al., 2010). There is evidence that facial pictures of American, Caucasian people expressing the six basic emotions can be readily identified and correctly labelled across cultures (Ekman, Sorenson, & Friesen, 1969; Izard, 1971). However, cultural differences have been reported in emotion recognition of complex stimuli based on facial pictures of Caucasian American people (Izard, 1971), and cultural differences have been reported even when using stimuli that are culturally compatible with the participant being tested (Elfenbein et al., 2002). Cultural differences may be even larger for complex mental states as those depicted in the Eyes test, but detailed studies on this topic are lacking. Studies based on the Turkish, Hungarian, Japanese, French, German, and Spanish versions of the Eyes test have been published so far (Bora et al., 2005; de Achávala et al., 2010; Havet-Thomassin, Allain, Etcharry-Bouyx, & Le Gall, 2006; Kelemen, Keri, Must, Benedek, & Janka, 2004; Kunihira, Senju, Dairoku, Wakabayashi, & Hasegawa, 2006; Voracek & Dressler, 2006). However, most of these studies did not detail information on the psychometric properties of the test in these cultures.

Information on reliability of the Eyes test are rarely reported. Two studies found low internal coherence, as measured by Cronbach’s alpha (Harkness, Jacobson, Duong, & Sabbagh, 2010; Voracek & Dressler, 2006), and two other studies found acceptable internal coherence as assessed with Cronbach’s alpha (Dehning et al., 2012) or Guttman’s split-half method (Serafin & Surian, 2004). Test–retest stability was found acceptable for both the adult version (Yildirim et al., 2011) and the child version (Hallerbäck et al., 2009) of the Eyes test. However, independent replication of these findings is lacking (Table 1).

Table 1. Psychometric characteristics of the Eyes test in past studies.

Study	Country	Sample	No. of items	Reliability	Mean score	Sex differences	Correlation with measures of empathy	Association with IQ
Ahmed & Miller (2011)	Athens, USA	123 healthy people 70 females, 53 males age: 19.0 ± 1.3	36	Unreported	27.3 ± 3.7	No differences by gender	Untested	Verbal IQ related to Eyes test scores
Alaerts et al. (2011)	Heverlee, Belgium	37 healthy people 22 females, 15 males age: 24.6 ± 5.1	36	Unreported	Unreported	Statistically significant advantage for females	Ad hoc emotion recognition task: r = .38	Untested
Ali et al. (2010)	London, UK	112 undergraduate students age: 20.7 ± 5.8	36	Unreported	Unreported	Unreported	EQ: r = .44	Untested
Baron-Cohen, Wheelwright, Hill, et al. (2001)	Cambridge, UK	103 undergraduate students 50 females, 53 males age: 20.8 ± 0.8	36	Unreported	Men: 27.3 (3.7) Women: 28.6 (3.2)	No differences by gender	Untested	No links with IQ
Carroll & Chiew (2006)	York, UK	48 university students 24 females, 24 males age: 21	36	Unreported	Men: 24.8 (4.6) Women: 27.8 (3.6)	Cohen’s d = 0.71 favouring females	EQ (n = 20): r = .36	No impact of IQ on the links between Eyes test and EQ
Chapman et al. (2006)	Cambridge, UK	76 children 37 girls, 39 boys age: 6 to 9 years old	28	Unreported	Boys: 15.2 (3.5) Girls: 16.3 (3.3)	No differences by gender	EQ-c: r = .56	No links with IQ
Cook & Saucier (2010)	Lethbridge, Canada	88 undergraduate students age: 22.2 ± 5.6	36	Unreported	Men: 26.1 (3.7) Women: 27.2 (4.1)	No differences by gender	EQ: r = .31	Untested
Dehning et al. (2012)	Jimma city, Ethiopia	237 medical students 30 females, 207 males age: 21.4 ± 2.7	36	Cronbach’s alpha: .70	First year: Men: 14.2 (4.7) Women: 16.3 (4.8) Final year: Men: 17.2 (4.3) Women: 21.2 (4.0)	Cohen’s d = 0.45 and 0.94, respectively in first and final year, favouring females	Untested	Untested
Golan & Baron-Cohen (2006)	Cambridge, UK	39 people with normal IQ (26 with autistic traits) 7 females, 32 males age: 25.1 ± 8.4	36	Unreported	Unreported	Unreported	Untested	Verbal IQ related to Eyes test scores
Hallerbäck et al. (2009)	Karlstad, Sweden	158 undergraduate students 83 females, 75 males age: 23.9 ± 3.3	28	Stable at retest according to the Bland-Altman plot	Men: 18.5 (1.9) Women: 19.2 (2.3)	Cohen’s d = 0.33 favouring females	Untested	Untested
Harkness et al. (2010)	Waterloo, Canada	93 college students 76 females, 17 males age: 18.9 ± 1.4	36/28	Cronbach’s alpha: .58 in the 36 items version, .63 in the 28 items version	Men: 17.6 (3.4) Women: 19.0 (3.6)	Nonsignificant advantage for the women	Untested	Untested
Kelemen et al. (2004)	Kecskemét, Hungary	40 controls 12 females, 28 males age: 41.2 ± 15.4	36	Unreported	27.8 (5.0)	Unreported	Untested	No links with IQ
Kenyon et al. (2012)	London, UK	57 controls age: 24.0 ± 2.6	36	Unreported	28.3 (3.2)	Unreported	Untested	IQ r = .28 in controls
Kettle et al. (2008)	Parkville, Australia	43 healthy controls 29 females, 14 males age: 19.2 ± 0.6	36	Unreported	Cannot be derived	No differences by gender	Untested	Untested
Kunihira et al. (2006)	Tokyo, Japan	40 undergraduate students age: cannot be derived	32	Unreported	Cannot be derived	No differences by gender	Untested	Untested
Lawrence, Shaw, Baker, Baron-Cohen & David (2004)	London, UK	53 adult volunteers 28 females, 25 males age: 32.5 ± 10.9	36	Unreported	Unreported	Unreported	EQ social skills factor: r .27	Untested
Muller et al. (2010)	Bordeaux, France	15 controls age: 37.0 ± 12.5	20	Unreported	13.0 (1.9)	Unreported	No links with Davis’ IRI	Untested
Serafin & Surian (2004)	North-east Italy	75 healthy people 41 females, 34 males age: 26.7 ± 5.4	36	Guttman’ split-half: .77	Men: 24.3 (3.7) Women: 25.5 (4.0)	No differences by gender	Untested	Untested
Smeets, Dziobek, & Wolf (2009)	???, Germany	64 undergraduate students 32 females, 32 males age: 25.9 ± 4.3	36	Unreported	Men: 28.4 (1.1) Women: 28.4 (1.1)	No differences by gender	Untested	Untested
Stanford, Messinger, Malaspina, & Corcoran (2011)	New York, USA	115 subjects 39 controls, 13 with schizophrenia, 63 with high risk of psychosis 38 females, 77 males	36	Unreported	Young people (n = 24): 25.7 (4.9)	No differences by gender	Untested	IQ related to Eyes test scores
Valla et al. (2010)	New York, USA	144 undergraduate students 79 females, 65 males age: 20.2 ± 1.8	36	Unreported	Men: 27.1 (3.5) Women: 27.3 (3.2)	No differences by gender	Untested	Untested
Voracek & Dressler (2006)	Vienna, Austria	423 healthy people age: 29.7 ± 11.4	36	Cronbach’s alpha: .63 in men, .60 in women	Men: 22.3 (4.5) Women: 23.3 (4.2)	Cohen’s d = 0.22 favouring females	EQ: r = .23	Untested
Yildirim et al. (2011)	Bakirkoy, Turkey	130 healthy people 72 females, 45 males age: 33.3 ± 8.9	36	Test-retest ICC: .65 (95% CI: .49 to .77)	Men: 23.2 (3.5) Women: 25.2 (3.1)	Cohen’s d = 0.61 favouring females	Untested	Untested

The Eyes test is usually used as a total score resulting from the sum of the correct guess on the 36 items, but the monodimensional structure of the test has never been formally investigated by factor analysis. The most replicated finding in past studies is the greater ability of female participants on the Eyes test. In six out of 17 studies, a statistically significant female advantage was observed, with Cohen’s d ranging from 0.22 to 0.94. Unfortunately, scores by gender are often lacking and only the statistical results are reported, which makes a meta-analysis of findings complicated.

Eyes test scores are often reported to correlate with self-reported empathy, but two studies failed to find a link with self-report measures of empathy. No link with the Interpersonal Reactivity Index (Davis, 1983) was found in the Muller et al. (2010) study, and only the social skills factor of the EQ was modestly related to the Eyes test in the Lawrence et al. (2004) study. Performance on the Eyes test requires intact verbal abilities, and a link with IQ, and verbal IQ in particular, was found in four out of eight studies (see Table 1).

One study examined culture differences on the Eyes test in a mixed American and Japanese sample, using both white American and Asian eyes stimuli. Authors found that both Japanese and white American participants performed better on same-culture (American participants vs. American eyes stimuli; Asian participants vs. Asian eyes stimuli) compared to other-culture stimuli, such as American participants versus Asian eyes stimuli or the reverse (Adams et al., 2009). Because of the problems with the identification of the Caucasian eyes stimuli, the reaction time and not the number of correct replies is considered in the Japanese version of the Eyes test (Kunihira et al., 2006). In a study carried out in Ethiopia, 237 medical students showed low scores on the Caucasian-based original Eyes test: on average below 22 as against 25–28 mean scores in typical Western student samples (see Table 1). Another study by Baron-Cohen et al. (1996) examined complex mental states as depicted in drawings of faces of a contemporary British artist (Hockney) and a seventeenth-century Spanish painter (Velazquez). In the Baron-Cohen et al. study, people from three different cultures (Britain, Spain, and Japan) showed good agreement in their judgement of most depicted mental states from facial information. However, cultural differences were found on three out of the 11 mental states: scheme, wariness, and guilt. Moreover, although there is evidence that members of different cultures reliably judge spontaneously expressed emotions, culturally influenced expressions are often displayed after initial, immediate, and universal emotional displays (Matsumoto, Olide, Schug, Willingham, & Callan, 2009; Matsumoto, Willingham, & Olide, 2009).

In Latin cultures, greater emphasis is placed on the expression of emotions than in Anglo-Saxon countries (Bhugra & McKenzie, 2003). This greater emphasis on emotional display might be related to differences in the ability of recognising emotions and complex affective states in others. The present study set out to assess reliability and construct (convergent and discriminant) validity of the Eyes test in Italy, a Southern European Mediterranean culture, to further extend its psychometric properties. We expected the Italian version of the Eyes test to display acceptable reliability, as far as internal coherence and test–retest stability are concerned, to show positive associations with measures of empathy and negative associations with measures of alexithymia, and to be unrelated to social desirability, differently from self-report measures of empathy.

Methods

Participants

Participants were 200 undergraduate students attending the courses of the University of Cagliari (Italy). Participants were enrolled from a wide array of courses: engineering (n = 45), psychology (n = 57), mathematics (n = 2), architecture (n = 6), political science (n= 15), medicine (n = 4), languages (n = 9), education (n = 13), classical studies (n = 19), pharmacy (n = 4), biology (n = 5), economics (n = 12), law (n = 7), geology (n = 1), physics (n = 1). This sample’s participation rate was 91.3%, from 219 subjects invited to take part in the study (seven refused to take part in the study; six refused to take the laboratory test after having completed the questionnaires; six did not complete the questionnaires). Participation was voluntary and no fee or other incentive was provided for taking part in the study. The institutional review board of the Department of Psychology of the University of Cagliari approved this study. The protocol of the research project conformed to the guidelines of the 1995 Declaration of Helsinki (as revised in Tokyo 2004). All participants provided informed consent.

Measures

Each participant was told the data would remain confidential, and received a booklet containing the questionnaires listed next, which they were asked to complete.

The Empathy Quotient (EQ)

The EQ is a 60-item questionnaire, with 40 questions tapping empathy and 20 filler items. Responses are given on a 4-point Likert scale. Scores can range from 0 to 80 (Baron-Cohen & Wheelwright, 2004; Lawrence et al., 2004). A cutoff score of fewer than 30 is useful to differentiate adults with autism from controls. We used the Italian version of the questionnaire (Preti et al., 2011), which shows acceptable internal consistency, concurrent and convergent validity, and good test–retest reliability.

The 20-item Toronto Alexithymia Scale (TAS-20)

The TAS-20 is a self-report scale that is comprised of 20 items, and it is the most frequently used self-assessment instrument to assess alexithymia because of its good reliability and construct validity (Bagby, Taylor, & Parker, 1994; Taylor, Bagby, & Parker, 2003; Taylor et al., 1988). We used the validated Italian version of the questionnaire (Bressi et al., 1996). Since the factorial structure of the TAS-20 has not been always supported (Müller, Buhner, & Ellgring, 2003), we used the total score only to assess alexithymia. Some studies report a high degree of alexithymia in individuals with autism (Berthoz & Hill, 2005; Bird et al., 2010; Lombardo et al., 2007).

The Marlowe-Crowne Social Desirability Scale (SDS)

This is a 33-item self-report questionnaire aimed at measuring social desirability (Crowne & Marlowe, 1960). Subjects rate the extent to which they agree (True) or disagree (False) with each item: the 15 items keyed False are likely but socially undesirable and are thought to measure denial and self-deception (e.g., “I am sometimes irritated by people who ask me favours”); the 18 items keyed True are improbable but socially desirable and are thought to measure a tendency to positive attribution, or to attribute to the self traits that are seen by society positively (e.g., “No matter who I’m talking to, I am always a good listener”) (Ramanaiah, Schill, & Leung, 1977). The Italian version of the SDS showed good psychometric functioning when tested in nonclinical populations and, as in past studies (Lane et al., 1990), showed a negative correlation with measures of psychopathology (Miotto, De Coppi, Frezza, Rossi, & Preti, 2002; Miotto & Preti, 2008).

General sociodemographic information from self-report data was collected for the following variables: age, sex, and socioeconomic status. As a measure of socioeconomic status we used the highest level of parental education (Galobardes et al., 2006), which was further classified into three categories: lower than high school diploma, high school diploma, college graduate or higher.

After completion of the booklet containing the questionnaires, each participant was invited to sit in a quiet room where s/he took the Reading the Mind in the Eyes test (Eyes test; Baron-Cohen, Wheelwright, Hill, et al., 2001). The adult, 36-item revised version of the Eyes Test was administered in the Italian version. We used the version of the Eyes test as reported in the Italian translation of The Essential Difference (Baron-Cohen, 2003, 2004); this version was based on translation and backtranslation of the descriptors, with a final check by the author.

Participants were randomly presented with a series of 36 photographs of the eye region of 19 actors and 17 actresses. Each photo was surrounded by four single-word mental state descriptors, e.g., bored, angry, one at each corner. One of these descriptors targeted the mental state depicted in the photo, and the others were foils. The Eyes test is based on a four-alternative forced-choice paradigm, with 25% correct guess rate. On each item, a response rate equal or above 50% indicates that participants are really detecting mental states and are not making random guess. Participants were instructed to choose which of the four descriptors best describes what the person in the photo is thinking or feeling. Participants could take as long as they wanted, and progressed to the next item when ready. They also could request an explanation of the meaning of the descriptors and could consult a prepared glossary if they were unsure of the definition of any descriptor word used. Score on the test is the number of descriptors correctly identified by the participants, i.e., the number of mental states correctly identified. The maximum score is 36.

Statistical analysis

All data were coded and analysed using the Statistical Package for Social Sciences (SPSS) for Windows (Chicago, IL 60606, USA), version 13. All tests were two-tailed, with α = .05. Categorical data were analysed in intergroup comparisons with χ², or Fisher’s exact test, when appropriate (n < 5 in any cell in binary comparison). Student t-test or the Mann-Whitney test, when appropriate, was used to compare the ordinal variables. Pearson’s r correlation coefficients were used to examine associations between two variables.

Scale reliability was measured by Cronbach’s coefficient alpha, a measure of internal consistency. For the Eyes test, reliability was also measured by maximal weighted internal consistency reliability from confirmatory factorial analysis (CFA) testing the unidimensional model (see later). To compare groups, reliability values of .70 are considered satisfactory (Bland & Altman, 1997).

Test–retest stability of the Eyes test was evaluated in a subgroup of 36 participants (one per item), who were invited to fill in the Eyes test again after 30 days from initial assessment. Follow-up completion rate was 80% for the test–retest reliability sample (36 out of 45 contacted: five refused to take part in the study because they did not have time; three accepted to take part in the study but did not show up on the day of the test; one took the test but did not return for the retest). Test–retest stability was assessed using intraclass correlation coefficient (ICC), with 95% confidence interval (CI). The ICC is dimensionless statistics that describes the reproducibility of repeated measurements in the same population: ICC values ≥.60 are considered acceptable for clinical use (Brennan & Silman, 1992). The Bland and Altman (1986) method was used, too, to assess agreement at retest. The Bland-Altman plot visualises the agreement between the scores of a test measured at two different assessment points by plotting the difference between test and retest scores against the mean value of test and retest scores for each participant. Confidence intervals for the mean difference are calculated to determine if the mean difference deviates significantly from zero, which should not occur. Graphically, the upper and lower limits of agreement are drawn, indicating the range within which 95% of the test scores of two assessments can be expected to vary.

The Eyes test is supposed to discriminate people with ToM deficits from typical people, and for this a summary score is often used. Therefore, the unidimensional model, assuming all items loaded on a single factor, was tested by CFA, which was carried out with EQS for Windows (Bentler, 2008), version 6.1. An alternative three-factor model, based on an items valence classification, was also tested, by distinguishing items with positive (eight items; i.e., playful, fantasising [Items 6 and 21], thoughtful, friendly, interested, flirtatious, confident), negative (12 items, i.e., upset, worried, regretful, accusing, doubtful, preoccupied [Item 9], defiant, hostile, cautious, distrustful, nervous, suspicious), and neutral (16 items, i.e., desire, insisting, uneasy, despondent, preoccupied [Item 22], cautious, sceptical, anticipating, contemplative, decisive, tentative, pensive, interested, reflective, serious, concerned) valence (Harkness, Sabbagh, Jacobson, Chowdrey, & Chen, 2005). Maximum likelihood under elliptical distribution was used to estimate fit. Chi-square is the traditional fit index for evaluating overall model fit, since it assesses the magnitude of discrepancy between the sample and fitted covariances matrices (Hu & Bentler, 1999, p. 2). However, the use of the chi-square likelihood ratio test to assess model fit is unsatisfactory for a number of reasons (Tanaka, 1993), including sensitivity to sample size. The ratio chi-square/degrees of freedom (df) was calculated in addition to the chi-square to evaluate model fitting, with ratios larger than 2 indicating poor fit (Byrne, 1989). In this study, we also used other parameters for fit estimate, namely: the comparative fit index (CFI), the root mean square error of approximation (RMSEA), and the standardised root mean square residual (SRMR). RMSEA ≤ .06, SRMR ≤ .09, and CFI ≥ .90 are considered acceptable (Browne & Cudeck, 1993; Hu & Bentler, 1999).

Convergent validity of the Eyes test was evaluated by testing the association of Eyes test scores with scores on the EQ and the TAS. The Eyes test was expected to covary with self-reported measures of empathy, such as the EQ, and to be negatively related to measures of emotion recognition deficits, such as the TAS. Discriminant validity of the Eyes test was evaluated by examining the links with the SDS. The Eyes test was expected to be unrelated to social desirability, as measured by the SDS.

Sample size

We carried out a preliminary power calculation to determine the minimum sample necessary to test our hypothesis. Based on a Pearson product-moment correlation test, a minimum sample of 82 participants would be needed to achieve 80% power to detect r =.30 (medium effect size) in the relationship between scores on the Eyes test and those of the measures of empathy or alexithymia at a two-sided significance level of .05. Using a nonparametric Spearman correlation index a sample of 104 participants would be needed, at the same levels of power and significance, since efficiency of the Spearman coefficient is about .91 of the corresponding Pearson coefficient (Lehmann, 1975). Calculation was performed using G*Power 3 (Faul, Erdfelder, Lang, & Buchner, 2007).

Results

The final sample included 200 participants (males = 46%). The mean age in the sample was 24.1 years ± 2.8 (range 19 to 32 years old), with no sex differences in age (males: 24.3 ± 2.7; females: 23.8 ± 2.9; t = 1.22, df = 198, p = .223). There was no sex difference in having a parent with a university degree (males n = 17; 18.5%; females, n = 12; 11.1%). Among participants, none was married (Table 2).

Table 2. General characteristics of the sample (n =200).

Sociodemographic group	N (%)	Eyes test Mean (SD)
Gender
Male	92 (46%)	24.1 (4.7)
Female	108 (54%)	25.5 (3.5)*
Age
18–24 (%)	115 (57%)	24.5 (4.6)
25–32 (%)	85 (43%)	25.3 (3.5)
Highest level of parental education
Lower than high school diploma	89 (44%)	24.6 (3.8)
High school diploma	82 (41%)	24.9 (4.4)
College graduate or higher	29 (15%)	25.1 (4.6)

^*t-test p <.05.

Reliability analysis

Internal consistency, as measured by Cronbach’s alpha, was .800 for the EQ, .743 for the TAS, .723 for the SDS, and .605 for the Eyes test.

Factorial analysis of the Eyes test

The unidimensional model proved acceptable on the basis of fit indexes, but for the CFI: Chi-square = 675.69, df = 594, p = .011, Chi-square/degrees of freedom = 1.1, CFI = .810, RMSEA = .026 (95% CI = .014 to .036), SRMR = .068. Maximal weighted internal consistency reliability for the unidimensional model was .719. The three-factor model did not converge with any adjustment in the distribution. A forced three-factor model by unweighted least square based on polychoric correlations was carried out with the FACTOR program (version 8.02; Lorenzo-Seva & Ferrando, 2006). The fit of the solution was poor: Chi-square=2017.98, df = 525, p < .0001, Chi-square/degrees of freedom = 3.8, CFI = .310, SRMR = .0716, RMSEA not provided. Item loading on the three factors did not follow the expected distribution. Items expected to belong to the “positive valence” or the “negative valence” factor loaded indifferently on each of the extracted three factors.

Distribution of response on the Eyes test

Females scored significantly higher than males on the Eyes test (t = −2.50, df = 198, p = .013, Hedges’ g = 0.34, 95% CI = 0.06 to 0.62); no other differences on the sociodemographic variables were found (Table 2). Since the scores were not normally distributed (Shapiro-Wilk test test: p <.001), differences on the Eyes test by gender were reanalysed with the nonparametric Mann-Whitney U-Test, which confirmed the difference, z = −2.09, p = .036. Overall, females were more able than males to correctly identify the target descriptor over the foils (Table 3). In two cases only the target descriptor was selected by less than 50% of the sample (Items 7 and 35). In Item 7, about 25% participants selected the foil word “friendly” rather than the correct word “uneasy”. In Item 35, participants were as likely to select the foil word “puzzled” (34%) as the correct word “nervous” (36.5%). These items can be considered difficult items, with lower margin for differentiation.

Table 3. Reading the Mind in the Eyes test–Distribution of responses in percentages^*.

									Corrected
Item	Answer A		Answer B		Answer C		Answer D		M	F	Factor loading
1	Playful	69.5	Comforting	24.5	Bored	3.5	Irritated	2.5	66.3	72.2	.168
2	Terrified	18.0	Upset	56.0	Annoyed	18.5	Arrogant	7.5	51.1	60.2	.242
3	Joking	0.0	Flustered	7.5	Convinced	27.5	Desire	65.0	60.9	68.5	.001
4	Joking	4.0	Insisting	65.5	Relaxed	27.5	Amused	3.0	65.2	65.7	−.031
5	Irritated	2.5	Sarcastic	13.5	Friendly	0.0	Worried	84.0	78.3	88.9	.148
6	Aghast	7.5	Fantasising	69.0	Alarmed	2.5	Impatient	21.0	60.9	75.9	.315
7	Apologetic	19.0	Friendly	25.5	Dispirited	10.0	Uneasy	42.5	39.1	45.4	.024
8	Despondent	67.0	Relieved	5.0	Excited	22.0	Shy	6.0	67.4	66.7	.164
9	Annoyed	5.0	Hostile	3.5	Preoccupied	90.5	Horrified	1.0	89.1	91.7	.254
10	Cautious	63.5	Insisting	26.5	Aghast	5.0	Bored	5.0	59.8	66.7	.050
11	Terrified	6.0	Amused	5.5	Flirtatious	17.5	Regretful	71.0	69.6	72.2	.076
12	Indifferent	19.0	Embarassed	8.5	Dispirited	1.0	Sceptical	71.5	73.9	69.4	−.058
13	Decisive	14.5	Anticipating	63.5	Shy	21.0	Threatening	1.0	54.3	71.3	.203
14	Irritated	10.5	Disappointed	6.5	Accusing	80.0	Depressed	3.0	78.3	81.5	.317
15	Contemplative	83.0	Flustered	2.5	Amused	2.5	Encouraging	12.0	78.3	87.0	.511
16	Irritated	2.0	Thoughtful	76.0	Sympathetic	19.5	Encouraging	2.5	67.4	83.3	.409
17	Doubtful	54.0	Affectionate	32.0	Aghast	9.5	Playful	4.5	58.7	50.0	.049
18	Decisive	92.0	Amused	2.5	Bored	2.5	Aghast	3.0	91.3	92.6	.231
19	Arrogant	4.0	Graeful	34.0	Tentative	52.5	Sarcastic	9.5	53.3	51.9	.104
20	Dominant	18.5	Friendly	73.5	Horrified	1.5	Guilty	6.5	68.5	77.8	.178
21	Embarassed	15.5	Fantasising	73.0	Panicked	2.5	Confused	9.0	67.4	77.8	.313
22	Preoccupied	90.5	Grateful	0.5	Imploring	4.5	Insisting	4.5	88.0	92.6	.317
23	Contended	5.5	Apologetic	9.5	Curious	22.5	Defiant	62.5	62.0	63.0	.152
24	Pensive	58.5	Irritated	9.0	Hostile	7.5	Excited	25.0	56.5	60.2	.096
25	Panicked	6.5	Incredulous	8.5	Interested	67.0	Despondent	18.0	65.2	68.5	.352
26	Alarmed	6.5	Shy	5.0	Anxious	12.0	Hostile	76.5	67.4	84.3	.333
27	Joking	2.5	Cautious	63.0	Reassuring	18.5	Arrogant	16.0	57.6	67.6	.361
28	Interested	70.0	Joking	3.5	Contended	22.0	Affectionate	4.5	69.6	70.4	.404
29	Impatient	9.5	Aghast	10.0	Reflective	66.5	Irritated	14.0	71.7	62.0	.140
30	Grateful	4.5	Flirtatious	91.0	Disappointed	2.0	Hostile	2.5	87.0	94.4	.264
31	Ashamed	7.5	Confident	66.5	Dispirited	18.5	Joking	7.5	63.0	69.4	.315
32	Serious	73.0	Ashamed	3.0	Alarmed	10.5	Bewildered	13.5	70.7	75.0	.394
33	Embarassed	9.5	Guilty	23.0	Concerned	54.0	Fantasising	13.5	55.4	52.8	.171
34	Aghast	2.0	Baffled	21.5	Terrified	5.5	Distrustful	71.0	71.7	70.4	.139
35	Puzzled	34.0	Nervous	36.5	Contemplative	13.5	Insisting	16.0	44.6	29.6	−.035
36	Ashamed	1.0	Nervous	7.0	Indecisive	15.5	Suspicious	76.5	77.2	75.9	.319

^*Correctly identified descriptors are marked in bold. Factor loading refers to the unidimensional model.

Figure 1 summarises the distribution of the Eyes test scores in the sample. As can be seen, a small fraction of the sample was particularly poor in the identification of the target descriptor: 15 subjects (7.5%) scored less than 20 on the Eyes test, corresponding to 1 standard deviation below the mean. This group can be said to have “low empathy” on the Eyes test. Males were more likely to have low empathy on the Eyes test: 13 (14.1%) versus 2 (1.9%), Fisher’s exact test p = .002; Cramér’s V = .232.

Figure 1. Distribution of test scores, 36 items (n = 200).

Measures of empathy and alexithymia

Females scored higher than males on the EQ (t = −5.34, df = 198, p <.0001, Hedges’ g = 0.76, 95% CI = 0.47 to 1.05). There were no differences by gender on the TAS or on the SDS (Table 4). In the sample 1 female (0.9%) but 9 male (9.8%) participants scored lower than 30 on the EQ, the cut-off score that best differentiates Autism Spectrum Conditions from controls (Fisher’s exact test p = .006). In the sample 3 female (2.8%) and 6 male (6.5%) participants scored higher than 61 on the TAS, the threshold that suggests the presence of alexithymia (Fisher’s exact test p = .30).

Table 4. Participants’ scores on the self-report scales (top) and relations between Eyes test scores and the other measures (bottom).

		Males (n = 92)			Females (n = 108)			Total sample (n = 200)
	Cronbach’s α	Minimum	Maximum	Mean (SD)	Minimum	Maximum	Mean (SD)	Mean (SD)	Median	IQR
Eyes test	.605	8	32	24.1 (4.7)	15	34	25.5 (3.5)	24.8 (4.2)	25	5
EQ	.800	25	64	41.8 (8.7)	27	67	48.3 (8.4)	45.3 (9.1)	45	13
TAS	.743	25	71	44.4 (9.1)	22	72	43.1 (9.0)	43.7 (9.1)	43	12
SDS	.723	3	29	16.6 (5.2)	5	28	16.3 (4.5)	16.5 (4.8)	16	7
		Pearson’s correlation coefficient
		Eyes test	EQ	TAS	Eyes test	EQ	TAS	Eyes test	EQ	TAS
EQ		−.039			−.070			.013
TAS		−.220*	−.358**		.019	−.455**		−.117	−.406**
SDS		.079	.302*	−.321*	.013	.410**	−.325*	.044	.321**	−.319*

SD =standard deviation; IQR = interquartile range. *p <.01, **p <.0001.

Eyes test, EQ, TAS, and SDS

In females, scores on the Eyes test did not correlate with scores on the EQ, the TAS or the two subscale of the SDS (Table 4, bottom). In males, scores on the Eyes test were negatively related to TAS scores, but not to EQ or SDS scores. EQ was negatively related to the TAS and positively related to the SDS, as expected; the TAS was negatively related to the SDS, again as expected. Those participants who scored lower than 30 on the EQ (n = 10) also scored lower on the Eyes test than the remaining participants (n = 190): 21.9 ± 5.1 vs. 25.0 ± 4.1, t = 2.32, df = 198, p = .021 (Hedges’ g = −0.74, 95% CI = −1.38 to 0.10). People with “alexithymia” scores on the TAS (n = 9) scored lower on the Eyes test than those in the nonalexithymia interval (n = 191), but the difference was not statistically significant: 22.6 ± 3.3 vs. 24.9 ± 4.2, t = 1.62, df = 198, p =.107 (Hedges’ g = −0.55, 95% CI = −1.22 to 0.12).

Test–retest stability at 1-month interval

In the subgroup involved in the test–retest assessment (n = 36), the mean score on the Eyes test was 23.8 (SD = 5.2) at the test and 25.1 (4.7) at the retest. Test–retest reliability for the Eyes test, as measured by ICC, was .833 (95%CI = .745 to .902). The mean difference between the first and the second assessment with the Eyes test in the 36 participants was 1.3 (SD = 4.4). The 95% CI for the mean difference was −0.18 to 2.79 (i.e., 0 is within the confidence interval, therefore the mean difference did not differ statistically from 0). By plotting the differences and the means of the two assessments in the Bland-Altman plot, two cases out of 36 (5.6%) were outside the upper and lower limits of agreement (Figure 2).

Figure 2. Bland-Altman plot of the Eyes Test test–retest assessment (n = 36).

Discussion

This study provides new evidence confirming the validity of the “Reading the Mind in the Eyes” test: Typical participants who score low on the Empathy Quotient (EQ) also scored lower on the Eyes test. Female participants scored higher than male participants on the EQ and the Eyes test, further providing evidence of validity of the Eyes test. This confirms the earlier reports of females scoring higher than males on tests of empathy (Baron-Cohen & Wheelwright, 2004; Lawrence et al., 2004; Von Horn, Bäckman, Davidsson, & Hansen, 2010; but see Eisenberg & Lennon, 1983). More importantly, Eyes test scores were not related to social desirability: that is, the Eyes test is free from the social desirability bias, as expected by its quasiecological nature.

As far as reliability is concerned, there is some agreement that Cronbach’s coefficient alpha is a poor index of unidimensionality, and that how much variance in a test is accounted for by one common factor might be a better index of the internal coherence of a test (Bentler, 2009; Sijtsma, 2009; Zinbarg, Yovel, Revelle, & McDonald, 2006). Reliability of the Eyes test was rarely reported in past studies. Voracek and Dressler (2006) found that Cronbach’s alpha was .63 in men and .60 in women. Harkness et al. (2010) found Cronbach’s alpha = .58 in a sample including 93 college students. Dehning et al. (2012) found Cronbach’s alpha = .70 in 237 Ethiopian medical students, despite a low mean score on the test (on average, below 22). By using a different Italian version of the adult Eyes test than the one used in this study, Serafin and Surian (2004) found a reliability value of .77, as measured by the Guttman split-half technique. In the present study, Cronbach’s alpha was .60, whereas maximal weighted internal consistency reliability for the unidimensional model provided a better estimate (.719), indicating acceptable internal consistency of the Eyes test in Italy.

Test–retest stability was reported to be acceptable in a past study from Turkey, with ICC = .65 (Yildirim et al., 2011). By using the child Swedish version of the Eyes test (28 items), Hallerbäck et al. (2009) found evidence of stability of the tool, according to the Bland and Altman (1986) method. We found good test–retest stability of the Italian version of the Eyes test, with ICC = .833, and good evidence of repeatability of the measurement according to the Bland and Altman method.

To our knowledge, this is the first study providing evidence that the Eyes test can be effectively scored as a single factor. Factor loading was far from being optimal: 21 items out of 36 did not reach the minimal acceptable threshold for factor loading (.25, i.e., around 5% of variance explained). Past studies used composite scores of the Eyes test, by subdividing items on the basis of their content. Three subscores were used in some studies, for “positive”, “negative”, and “neutral” items. We tested this three-factor structure of the Eyes test but the model was rejected on the basis of fit indexes and factor loading. This may be because the mental states in the test are sufficiently different to each other, so that they cannot easily be grouped into a small number of categories.

Past studies report consistent evidence for a correlation between EQ and Eyes test scores. One study also found a negative correlation (r = −.43) between TAS and Eyes test scores in a sample including 30 individuals with high-functioning autism or Asperger syndrome and 30 sex- and age-matched typical controls (Lombardo, Barnes, Wheelwright, & Baron-Cohen, 2007). In this study we did not find a correlation between EQ and Eyes test scores in both males and females. Moreover, a link with TAS scores was found in males only, and this may reflect that empathy functions differently between the two sexes (Valla et al., 2010). Males are more likely to display higher levels of autistic traits (Baron-Cohen, Wheelwright, Skinner, Martin, & Clubley, 2001), and a higher chance of alexithymia is reported in people with autism (Berthoz & Hill, 2005; Bird et al., 2010; Lombardo et al., 2007). Those who scored lower than 30 on the EQ, the cutoff score that best differentiates autism from controls, also had statistically lower scores on the Eyes test, providing some evidence for an association between Eyes test performance and autistic traits.

Analysis of single items also provided evidence for their discriminative properties: Some of the emotions depicted in the pictures were easily detected by the majority of participants, whereas other items were more difficult. Two items only were identified by less than 50% of participants. It has been proposed that the degree of difficulty of an item might vary from the original to the translated version because of cultural differences in the meaning attributed to the target definitions (Hallerbäck et al., 2009). This hypothesis can be tested by a comparative study only.

One limitation of the study is the inclusion of undergraduate students only. The lack of more varied strata of the population precludes the establishment of baseline for the general population. In particular, no evaluation of education level impact on scores can be calculated using undergraduate students only. Although previous studies have not found major effects of education or IQ on this test (Baron-Cohen, Wheelwright, Hill, et al., 2001), some studies have found an association between IQ and the Eyes test (Ahmed & Miller, 2011; Golan & Baron-Cohen, 2006; Kenyon et al., 2012; Stanford et al., 2011). However, in a sample of undergraduate students it is unlikely that IQ would show variations large enough to be detected statistically.

In conclusion, the “Reading the Mind in the Eyes” test is a freely available measure of facial affect recognition, easy to administer and easy to score. It has been translated into many languages, and this allows comparability. This study shows that the Eyes test, in its revised, adult version, is a valid and reliable measure, which can be used to assess differences in recognition of complex affective states, including nonbasic emotions.