Damage to Association Fiber Tracts Impairs Recognition of the Facial Expression of Emotion

Abstract

An array of cortical and subcortical structures have been implicated in the recognition of emotion from facial expressions. It remains unknown how these regions communicate as parts of a system to achieve recognition, but white matter tracts are likely critical to this process. We hypothesized that (1) damage to white matter tracts would be associated with recognition impairment and (2) the degree of disconnection of association fiber tracts [inferior longitudinal fasciculus (ILF) and/or inferior fronto-occipital fasciculus (IFOF)] connecting the visual cortex with emotion-related regions would negatively correlate with recognition performance. One hundred three patients with focal, stable brain lesions mapped onto a reference brain were tested on their recognition of six basic emotional facial expressions. Association fiber tracts from a probabilistic atlas were coregistered to the reference brain. Parameters estimating disconnection were entered in a general linear model to predict emotion recognition impairments, accounting for lesion size and cortical damage. Damage associated with the right IFOF significantly predicted an overall facial emotion recognition impairment and specific impairments for sadness, anger, and fear. One subject had a pure white matter lesion in the location of the right IFOF and ILF. He presented specific, unequivocal emotion recognition impairments. Additional analysis suggested that impairment in fear recognition can result from damage to the IFOF and not the amygdala. Our findings demonstrate the key role of white matter association tracts in the recognition of the facial expression of emotion and identify specific tracts that may be most critical.

Introduction

The ability to recognize facial expressions of emotion is a core component of social cognition (Darwin, 1872; Borod et al., 1986; Cole, 1998; Rolls, 1990; Fridlund, 1994; Russell and Fernández-Dols, 1997). Functional imaging (Borod et al., 1993; Davidson and Sutton, 1995; Phillips et al., 1997; Hariri et al., 2000; Iidaka et al., 2001; Kesler-West et al., 2001; Murphy et al., 2003) and lesion studies (Adolphs et al., 1994, 2000; Hornak et al., 1996; Shaw et al., 2005) have implicated a number of cortical and subcortical structures in this ability, suggesting that emotion recognition from facial expressions relies on a large-scale distributed network (Vuilleumier, 2005). This network includes multiple sectors of the occipito-temporal cortex (Bowers et al., 1985; Borod et al., 1998; Adolphs et al., 2000), the basal ganglia (Cancelliere and Kertesz, 1990), the frontal and parietal opercula (Damasio, 1994; Adolphs et al., 2000; Keane et al., 2002), the insula (Calder et al., 2000; Craig, 2002; Singer et al., 2004), the amygdala (LeDoux, 1992; Adolphs et al., 1994; Young et al., 1995), and the orbitofrontal cortex (Hornak et al., 1996; Rolls et al., 1999). White matter tracts are likely critical for these regions to communicate to achieve recognition. Indirect evidence from studies on the rapid interaction between visual and emotional processing (Rudrauf et al., 2008c) suggests involvement of long-range association fiber tracts that connect visual and emotion-related structures: the inferior-longitudinal fasciculus (ILF) and inferior fronto-occipital fasciculus (IFOF). The ILF connects the occipital cortex with the anterior temporal lobe and amygdala, whereas the IFOF begins in the occipital cortex, continues medially through the temporal cortex dorsal to the uncinate fasciculus, terminating in the orbitofrontal cortex (Catani et al., 2002, 2003).

These tracts could play a broad and critical role in mediating the recognition of the facial expression of emotion, beyond rapid processing. Although difficult to address using functional imaging, this hypothesis can be tested with the lesion method (Rudrauf et al., 2008a). Lesion studies often show involvement of white matter but rarely make white matter damage an explicit focus (Bowers and Heilman, 1984; Rapcsak et al., 1989, 1993; Ross et al., 1997). The current study is the first to take a whole-brain statistical approach, using a relatively large sample of subjects, to investigate white matter implication in emotion recognition.

We hypothesized that (H1) damage to white matter tracts could be associated with impairment in the recognition of the facial expression of emotion, and (H2) specifically, the degree of disconnection of the ILF and/or IFOF would be negatively correlated with performance in recognition of the facial expression of emotion.

We tested these hypotheses using (1) standard voxelwise lesion–deficit statistical mapping over the entire brain, (2) together with a novel method for analyzing lesion–deficit associations [generalized lesion-symptom mapping (GLSM)] (Rudrauf et al., 2008a) that incorporates probabilistic fiber tract information and (3) a case study of the patient with the most specific lesion located in the IFOF and, to a lesser extent, the ILF (with minimal gray matter involvement), testing the hypothesis that this lesion pattern would be sufficient to cause impaired recognition of facial emotions.

Materials and Methods

Participants.

One hundred four lesion subjects [48 left hemisphere, 42 right hemisphere, and 14 bilateral; 91 right handed (100); 3 left handed (−100), and 10 mixed handedness] initially participated in the study. One of the subjects (1652), with a left hemisphere lesion, was excluded after being identified as an outlier in the main disconnection analysis. All the statistical analyses were then performed with the 103 remaining subjects. (We note here that the results were virtually identical with and without the inclusion of the outlier subject.) Eighteen healthy age-matched comparison subjects (nine males and nine females; mean age, 56 ± 16 years) also took part in this study. The subjects were all neurologically and psychiatrically normal with normal visual discrimination ability, and mean ± SD intelligence quotient (IQ) was 108 ± 8. The subjects [lesion group (n = 103) and the healthy age-matched comparisons (n = 18)] and emotion recognition data were the same as those included in the study by Adolphs et al. (2000). The present study is a reanalysis of those data incorporating information on white matter tract disconnection.

All brain-damaged subjects were selected from the Cognitive Neuroscience Patient Registry of the Department of Neurology at the University of Iowa and met the inclusion criteria for the registry: they had stable (nonprogressive), circumscribed brain lesions. We included subjects with various lesion etiologies [hemorrhage (8 left, 6 right, 2 bilateral); infarct (30 left, 28 right, 8 bilateral), herpes simplex encephalitis (2 right, 1 bilateral), surgical resection of focal lesions (15 left, 10 right, 9 bilateral); temporal lobectomy mostly for epilepsy (8 left, 6 right); other (5 left, 1 right, 2 bilateral)]. (Some subjects had multiple etiologic mechanisms.) All subjects were characterized neuropsychologically and neuroanatomically in the chronic epoch (>3 months after onset of lesion), according to the standard protocols of the Benton Neuropsychology Laboratory (Tranel, 2007) and the Laboratory of Human Neuroanatomy and Neuroimaging (Damasio and Damasio, 1989; Damasio, 1995; Frank et al., 1997). All had IQ in the normal range, normal ability to discriminate faces (measured by the Benton facial recognition test), and normal or near-normal visual perception. Table 1 presents a summary of their neuropsychological profiles [note that subjects in the table are grouped based on average emotion recognition task performance (Impaired and Unimpaired)].

Table 1.

Components of the neuropsychological profile means and SDs are reported for all subjects (n =103)

	Unimpaired (n = 61)	Impaired (n = 42)	Case study (1981)
Age	51.8 (15.2)	55.8 (15.0)	66
Education	14.0 (2.7)	12.9 (2.7)	16
Full-scale IQ	104.4 (15.0)	95.2 (12.0)	108
Verbal IQ	103.4 (15.4)	96.5 (13.0)	106
Performance IQ	104.7 (17.4)	94.3 (14.0)	110
Wechsler Memory Scale-III (general memory)	103.0 (16.7)	91.3 (15.2)	115
Wechsler Memory Scale-III (immediate memory index)	101.0 (17.2)	89.8 (19.0)	118
Wide Range Achievement Test-Revised Reading	98.6 (15.1)	93.5 (16.3)	101
Wide Range Achievement Test-Revised Spelling	92.9 (18.0)	90.6 (17.3)	93
Wide Range Achievement Test-Revised Arithmetic	96.7 (11.7)	95.7 (14.7)	100
Benton facial recognition test	45.9 (14.2)	43.4 (4.9)	50
Warrington Faces (scaled)	7.8 (4.4)	5.5 (2.1)	7
Warrington Words (scaled)	11.0 (3.8)	9.5 (3.8)	11
Stroop (word)	42.4 (9.1)	40.0 (13.5)	42
Stroop (color word)	42.6 (6.5)	40.5 (15.2)	44
Stroop (color–word)	47.0 (10.6)	40.2 (9.8)	52
Rey–Osterrieth (copy)	32.2 (3.5)	30.5 (5.3)	34
Controlled oral word association	39.7 (12.7)	32.4 (13.5)	44
Hooper visual organization test	52.2 (8.6)	52.9 (8.0)	44
Depression composite (BDI and MMPI)	0.39 (0.69)	0.26 (0.5)	0

Scores are reported for patient groups separated based on the mean score for the emotion recognition task (Impaired, n = 42; Unimpaired, n = 61). Notably, scores on the various neuropsychological measures (i.e., face recognition, IQ, and verbal fluency) and demographic variables were not significantly different between groups (based on Wilcoxon's rank sum tests, two-tailed, Bonferroni's corrected for multiple comparisons, α = 0.00125).

All participants gave informed consent according to a protocol approved by the Institutional Review Board of the University of Iowa.

Emotion recognition task.

All subjects were tested with an established emotion recognition task, specifically from visually presented facial expressions (cf. Russell and Bullock, 1985; Adolphs et al., 1994, 1996, 1999; Hamann et al., 1996; Young et al., 1997) described fully by Adolphs et al. (2000). Emotion recognition involves the retrieval of conceptual knowledge about the emotion from perception of the facial expression. This ability amounts to much more than discrimination, classification, or naming. Our task, which asks subjects to rate the graded intensity of each of the six basic emotions, provides a rich measure of conceptual knowledge. In previous studies, it has been used to derive category membership (Young et al., 1997), emotion category structure (Russell and Bullock, 1985), and perceived similarity (Russell and Bullock, 1985; Adolphs et al., 1994; Hamann and Adolphs, 1999).

Procedurally, subjects were shown 36 black-and-white photographs of faces (Ekman, 1976), six each showing six basic emotions: happiness, sadness, anger, fear, disgust, or surprise. All the images were shown in randomized order in six separate blocks, with each block asking subjects to rate the intensity, on a scale from 0 to 5, of each of the six emotions, for each face. The rating profile for each face from each subject was then correlated with the corresponding average ratings produced by 18 age-matched healthy comparison subjects (nine males, nine females; mean age, 56 ± 16 years), normalized using a Fisher's Z transform, and averaged for each subject across the six faces for each emotion to quantify emotion recognition. The subjects were all neurologically and psychiatrically normal, with normal visual discrimination ability, and mean ± SD IQ was 108 ± 8.

The degree to which a subject's ratings correlate with control ratings provides a continuous measure of the degree to which the subject's conceptual knowledge of the facial expression of emotion conforms to normal knowledge. We estimated the mean and SD of correlation scores across the healthy comparison group by correlating each individual healthy comparison subject with the remaining ones (Fisher's Z transformed to normalize the scores).

For the standard voxelwise lesion–deficit analysis, we defined a dichotomous measure of impairment for the brain-damaged subjects using a cutoff of −1.65 SDs below the normal comparison mean, corresponding to a one-tailed α level of 0.05. This cutoff corresponded to a raw score of less than 0.79 (79% accuracy) for the average emotion recognition score. Refer to Figure 4 for histogram of Z-transformed performance distribution.

Figure 4.

Case study. A, Top row, Overlay of the tract and the lesion (red) of the subject on seven coronal slices (see legend). Bottom row, Overlay of the ILF and the lesion (red) of the subject on seven coronal slices (see legend). B, Chart representing the Fisher's Z-transformed correlation scores of the case study subject (red dots), the corresponding histograms of scores for the brain damage group (gray color scale) and the corresponding mean scores (green dots), the corresponding median scores for the brain damage group (yellow bars), and SDs of the healthy comparison subjects (green bars encompassing 1.65 SD), for all emotions and the average emotion scores. pos, Posterior; ant, anterior.

Other neuropsychological and psychiatric measures.

To address the specificity of the visual recognition impairment for emotional facial expressions in our subjects and to rule out the potential contribution of general object recognition and visual perception impairments, we performed the main fiber tract regression analyses with additional covariates including measures of basic visual and object recognition measures [Benton facial recognition task, judgment of line orientation, Rey–Osterrieth complex figure (Copy), Boston naming test, and Hooper visual organization test]. Because depression can occur as a consequence of stroke (in ∼30% of patients) and that depression is known to alter the ability to accurately perceive emotions (Rubinow and Post, 1992), we also included an additional covariate for depression [a composite measure based on scores from Beck Depression Inventory (BDI), Minnesota Multiphasic Personality Inventory (MMPI)] as in the study by Adolphs et al. (2000).

Lesion mapping.

We took as a starting point the lesion maps used by Adolphs et al. (2000), which were based on either T1-weighted magnetic resonance imaging (MRI) (the majority) or three-dimensional computed tomography (CT) scans. Lesion maps were generated using the MAP-3 method (Frank et al., 1997; Fiez et al., 2000; Damasio et al., 2004), a semiautomated method in which the boundaries of the lesions of a given subject are visually identified on MR or CT scans and manually transferred onto a normal reference brain (P.C. local standard space; resolution, 0.94 × 0.94 × 1.6 mm) based on the delineation of homologous anatomical landmarks. This procedure requires anatomical expertise but circumvents the problems of interindividual registration encountered with lesion data and the problems of combining subjects scanned with different imaging modalities. Lesion delineation and transfer were done using Brainvox (Frank et al., 1997). MAP-3 has been used to generate lesion overlap maps across a variety of cognitive domains (Tranel et al., 1997, 2001, 2003; Adolphs et al., 2000, 2002; Barrash et al., 2000; Damasio et al., 2004).

The general procedure is the following (for more details, see Damasio et al., 2004): (1) a normal template brain MRI scan is reconstructed in three dimensions from thin contiguous MR slices; (2) major sulci are identified and color coded in the template brain and the lesion brain; (3) the slices in the template brain are matched in orientation and thickness to those of the lesioned brain (MRI or CT) taking into consideration the intersection of the slices with the color-coded sulci; (4) the lesion contour on each slice is manually traced by an expert on the template brain (H.D.), taking into consideration the distance of the lesion contour to identifiable landmarks, such as sulci and subcortical structures, and respecting the gray and white matter components of the lesion; and (5) the collection of transferred traces defines a volume that can be saved as a binary mask of the lesion. One advantage of this time-consuming approach is that it preserves anatomical boundaries and tissue compartments in the mapping of the lesions onto the reference brain, enabling group-level analysis.

Standard lesion–deficit analysis.

We first approached hypothesis H1 with voxelwise lesion proportion difference maps (PM3) (Rudrauf et al., 2008b), which provide a standard description of lesion–deficit associations. This voxelwise lesion–deficit analysis also served to inform the new tractwise lesion–deficit analyses, described below, by (1) enabling us to construct covariates to account for the potential confound of cortical damage in the tractwise regression analyses (e.g., removal of gray matter confound to identify unique contribution of fiber tract damage to impairments recognition of the facial expression of emotion) and (2) allowing us to determine whether white matter tract lesion–deficit associations were corroborated by lesion–deficit associations in their cortical target regions or termini of the tracts.

PM3 expresses, for every voxel, the proportion of subjects whose lesion includes the voxel and who have a deficit (N_LD) relative to the total number of subjects with a deficit (N_D), minus the proportion of subjects with a lesion at the voxel and no deficit (N_LnD) relative to the total number of subjects with no deficit (N_nD). The formula can be expressed with the equation Prob (L | D) − Prob (L | nD), the conditional probability of a lesion (L) given a deficit (D) minus the conditional probability of a lesion given no deficit (nD). For example, at a given voxel, if all patients with a lesion have a deficit, the PM3 = 1, whereas PM3 = 0 when half the patients have a lesion and a deficit, and the remainder have a lesion and no deficit.

The PM3 maps were thresholded using exact statistics from permutation tests (Rudrauf et al., 2008b). We determined the statistical thresholds for the PM3 analysis based on preliminary power analyses, i.e., “effective coverage maps” (ECMs) (Rudrauf et al., 2008b). Effective coverage is defined as the voxel map in which effects can be detected at a given significance threshold, assuming there is the maximal lesion–deficit relationship permitted by the sample. A satisfactory effective coverage implies the ability to detect effects over a large percentage of the brain, especially within regions for which hypotheses have been formulated. We selected the one-tailed, uncorrected threshold of α = 0.01 as the main threshold for the PM3 analysis because higher thresholds would have led to an extensive loss in effective coverage. Relatively liberal thresholds are common in standard lesion studies because of their intrinsically low and heterogeneous statistical power. We show ECMs and effect maps obtained for threshold at 0.05, 0.01, and 0.001 (see Fig. 2). We primarily used the results of the PM3 analysis to identify gray matter regions to include in the fiber tract analyses as potential confounding effects.

Figure 2.

Voxelwise lesion–deficit analysis. Top row, Unthresholded PM3 results. The scale corresponds to the proportion of subjects with a deficit and a lesion at a given voxel among the subjects with a deficit minus the proportion of subjects with no deficit and a lesion at the voxel among the subject with no deficit. Middle row, Thresholded PM3 results (blue, <0.05; green, <0.01; orange, <0.001). Bottom row, Effective coverage map [blue, <0.05 (the threshold used in this study); green, <0.01; orange, <0.001] (see Results). Left, Lateral views of the left and right hemispheres. Right, Coronal slices (radiological display convention). occ, Occipital cortex; front, frontal cortex.

Fiber tract atlas registration and disconnection estimation.

To estimate the contribution of fiber tracts to the impairments in our subject sample, we first registered a probabilistic fiber tract atlas (Mori et al., 1999; Wakana et al., 2004; Hua et al., 2008) (http://lbam.med.jhmi.edu) to our lesion analysis reference brain (Fig. 1). The atlas [tract probabilistic maps in Montreal Neurological Institute (MNI) space], containing probabilistic maps of 10 white matter tracts, was originally developed from a diffusion tensor imaging (DTI) database of 28 normal subjects that were registered by affine transformation to the MNI–International Consortium for Brain Mapping template. These maps give the probability of the presence of a given tract at each voxel in the standard space.

Figure 1.

Examples of registered tracts. Probabilistic fiber tracts registered to our reference brain for lesion–deficit analyses. The tracts are seen through a semitransparent representation of the reference brain (tract probability is thresholded at 0.1). Four tracts are represented: the IFOF, the ILF, the SLF, and the UNCF. Top row, Lateral view of the right hemisphere. Bottom row, Ventral view of the right hemisphere.

We registered the MNI template to the MAP-3 reference brain (using a fifth order nonlinear registration of AIR 5.2.5) (Woods et al., 1998) and then applied the derived deformation fields to the fiber tract data to bring them into register with the standard template brain. Registration quality was evaluated by visual inspection in standard, ensuring correspondence of key landmarks (e.g., frontal pole, visual occipital pole, and anterior commissure). We note that orientation convention was conserved across registration of the different templates (the DTI atlas was flipped to radiologic convention for congruence with MAP-3 orientation convention). We focused on the six available association fiber tracts: the cingulum (cingulate gyrus process) (CgU), cingulum (hippocampal process) (CgL), the ILF, the IFOF, the superior longitudinal fasciculus (arcuate fasciculus) (SLF), and the uncinate fasciculus (UNCF). Tract probabilities were bottom thresholded at 0.1, and the few above-threshold voxels that fell into gray matter in the reference brain were masked out using a validated tissue segmentation algorithm (Grabowski et al., 2000). The intersection of the lesion of each subject with each association tract, computed as the number of voxels in the lesion volume that overlapped with the thresholded tract, was used to index the degree of disconnection. Because this index is correlated with sheer lesion volume, our regression analyses also covaried for lesion volume for each subject to achieve specificity.

Regression analysis of fiber tract involvement in impaired emotion recognition.

To test hypothesis H2, we adopted a newly developed approach to disconnection analysis, GLSM (Rudrauf et al., 2008a). The primary purpose of these analyses was to determine whether long-range association fiber tracts support the ability to recognize emotion and also to identify which specific tracts play critical roles. Building on the rationale that wherever the lesion occurs along the tract, the functional consequences should be similar, we used a multiple regression framework in which we attempted to predict emotion recognition score based on regressors estimating degree of fiber tract disconnection, after covarying for damage to other tracts and confounding damaged gray matter.

We used the general linear model to predict emotion recognition scores (Y, in the lower performing direction) based on regressors quantifying the estimated fiber tract disconnections. The generic model was the following:

Y corresponded to the negative Z-transformed correlations calculated from the emotion recognition task, such that large positive values would correspond to impairments. β₀ is the intercept. L_t1 is the estimated degree of disconnection for tract t1. β_t1 is the corresponding regression coefficient. Twelve fiber tract regressors were included, corresponding to each of the six association tracts of interest (CgU, CgL, ILF, IFOF, SLF, and UNCF), for each hemisphere. L_S is the total lesion size (gray and white matter), and β_S is the corresponding regression coefficient. L_g is a regressor indicating the presence of damage in cortical gray matter voxels associated with impairments in emotion recognition, derived from the PM3 analysis described above. β_g is the corresponding regression coefficient. The modeling of confounding gray matter was done using an omnibus binary gray matter regressor L_g (Rudrauf et al., 2008a) that indicates whether a subject has damage in gray matter regions significantly associated with impairment as determined by the PM3 analysis.

In these “tractwise” analyses, we used continuous dependent variables to describe impaired task performance. Our hypothesis is that damage to critical tracts impairs emotion recognition to the degree to which it hinders communication between disconnected cortical regions. We presumed that the degree of communication impairment would, in part, vary based on the number of disconnected fibers within the tract bundle. Because this could not be directly quantified, we used the degree of overlap between the probabilistic tract map and the lesion map of an individual subject as a proxy for the degree of disconnection of the tract. We further presumed that the degree of physiologic compromise of a critical tract would be correlated with the degree of behavioral impairment. Therefore, we also used continuous scores for the dependent variable. Thus, the tractwise analysis allowed us to formally correlate a continuous measure of tract damage with a continuous measure of neuropsychological performance (emotion recognition) while covarying for potential confounding factors.

To assess the validity and robustness of using a parametric approach with our data, we first compared, using the original sample of 104 subjects, the results obtained with the parametric analysis of the average emotion scores with corresponding results obtained with a nonparametric approach, based on a bootstrap analysis. We used a trial-with-replacement scheme to generate vectors of surrogate data based on the observed average emotion recognition scores. We generated 1000 such vectors simulating 1000 samples of 104 subjects. To each of these vectors of simulated recognition scores, we applied the same multiple linear regression analysis as that used on the correctly labeled data. We then looked at the distribution of t values generated by these 1000 analyses for the IFOF regressor (i.e., the regressor corresponding to the tract implicated by the parametric regression analysis). This simulated empirical distribution showed that 0.2% of the samples yielded t values of 2.96 or more for the right IFOF regressor, reproducing the parametric result. Second, we inspected histograms of residuals and plots of residuals against fitted values. This exercise identified the one subject whose data we excluded because it could not be fitted by the model and was clearly an outlier. Other than this outlier, there was no clear indication of departure from normality or unequal variance, given the sample size, thus justifying a parametric approach.

Seven main analyses were performed, one for each of the six emotions and one for the average correlation score across all six emotions. These analyses included all 12 tracts (6 tracts × 2 hemispheres) and resulted in a total of 12 × 7 = 84 tests. To control for multiple comparisons, we used the Dubey and Armitage-Parmar procedure, which takes into account the level of correlation among the dependent variables (Armitage and Parmar, 1986). The mean value of the dependent variable correlation matrix (diagonal excluded) was 0.47, resulting in a corrected threshold α* = 0.05, corresponding to an uncorrected threshold of α = 0.0049. All analyses were implemented in Matlab (MathWorks).

Analysis of tract specificity of lesions.

To provide additional support for the specificity of tract involvement based on the main regression analyses, we derived a measure to determine the degree of specificity of each subject's lesion for each tract (e.g., a subject may have a lesion that overlaps both the ILF and SLF, but because of the continuation of the lesion into more ventral and anterior brain regions, it may be more specific to the ILF). With this measure, we generated a list of subjects ranked by the specificity of tract involvement. For each subject [with a lesion on the right hemisphere (n = 42), because most effects were found on the right], we computed two parameters. First, we computed the proportion P_T of each tract T (all six association tracts on the right) that overlapped with the lesion of the subject and further divided each P_T by the sum of P_T across tracts to obtain a parameter with a sum equal to 1, behaving as a distribution of probability. Second, we computed, for each tract, the proportion P_L of each lesion that overlapped with the tract. P_L was then also normalized. For each subject and each parameter, we then computed the entropy H(S, P_i) (Shannon, 1948) of those distributions using the following formula:

the sum being calculated across all nT tracts. The entropy (conventionally expressed in units of “bits”) is maximum when each state of the probability space (i.e., disconnection of a given tract) is equiprobable. In this application, entropy will be maximal [max (H(S, P)) = log₂(6) = 2.58] when the white matter component of the lesion of the subject is not specific to any tract (i.e., when the white matter component of the lesion overlaps with multiple tracts). Conversely, the entropy will be zero when the “probability” of disconnection of one tract is unity and all others are zero, i.e., when the white matter component of the lesion of the subject is fully specific to one tract. We also computed the amount of damage in gray matter encompassed by the lesion of each subject to be able to identify subjects with minimal damage to gray matter.

To investigate tract specificity, we selected the first 25 subjects (with the lowest entropy values) from the ranked entropy list. The list of subjects so obtained was the same for both P_T and P_L. Across the sample of right hemisphere lesions, the minimum entropies were 0 bits for both parameters: the maximum, 1.9 and 2.3 bits for P_T and P_L, respectively, and mean ± SD, 1.2 ± 0.6 and 1.2 ± 0.6 bits. All 25 subjects with the lowest-ranked entropies had their lesions affecting either one or two tracts, with generally a larger peak for one of them.

Identification of the subject for the case study.

The subject in the case study (1981; see below) was identified based on anatomical criteria of both tract specificity and degree of white matter involvement. More specifically, subject 1981 was identified in the entropy analysis to have a lesion most specific to the hypothesized tracts of interest (right IFOF, ILF) and to have a lesion with minimal involvement of gray matter.

Auxiliary analyses: analyses investigating null results for main regression for ILF and SLF.

Neither ILF nor SLF was implicated in the main regression analysis (see Results). Because in each case there was a rationale to wonder whether the findings were false negatives, we performed supplementary analyses to investigate potential factors underlying the null results, e.g., shared variance (anatomical overlap/colinearity). We performed several supplemental regression analyses. First, each of the 12 tracts were independently entered into a regression model as single independent variables to determine, with a minimal model, whether each individual tract could contribute to emotion recognition impairment in our task. Second, because some effects were significant, we performed an additional regression analysis with restricted combinations of tract regressors (e.g., combination 1: right SLF, right IFOF; combination 2: right SLF, right ILF) to further assess the possible factors that could potentially account for the null effects in the full model with the 14 regressors.

Supplementary analysis: specificity of IFOF in fear recognition.

We performed a supplementary analysis to assess the specific role of the IFOF in the recognition of the facial expression of fear. In this analysis, we first identified subjects from the tract specificity analysis (list of 25 subjects with the lowest entropy values) with lesions most specific to the IFOF [with the lowest entropy value for IFOF (n = 6)]. Next, we calculated a weighted lesion–deficit value for each of these subjects (six with lesions most lowest entropy value for the IFOF) to generate a lesion overlap map weighted by the degree of fear recognition impairment. For example, a subject with a low score for fear recognition would be weighted more heavily in this lesion overlap analysis than a subject with a higher score. We calculated a weighted value by dividing the binary lesion map of each subject by the emotion recognition score for fear. Based on these weighted lesion–deficit measures, we computed a weighted lesion overlap map by adding the calculated weighted values (weighted lesion maps) across subjects for each voxel in the brain (for these results, refer to Fig. 5).

Figure 5.

Weighted lesion overlap map for subjects with specific lesions in the IFOF. Overlay of the outer boundary of the IFOF and weighted lesion overlap values overlaid on a series of coronal slices from posterior occipital lobe (top left) to anterior frontal lobe (bottom right) (see Results).

Results

Task performance

The performance of the subjects in the emotion recognition task (from facial expressions) is summarized in Table 2. (The scores are presented as both standard correlations and Fisher's Z-transformed correlations; for the distribution of scores in the Fisher's Z transform space, see also Figure 4B.) The correlation scores across all emotions had a mean Z-transformed value across subjects of Z = 1.17 (r = 0.81), and 42 of 103 subjects were considered impaired.

Table 2.

Impaired recognition of emotion in lesion subjects

	Average	Happy	Sad	Angry	Afraid	Disgusted	Surprised
HC	0.88	0.94	0.88	0.83	0.83	0.86	0.87
	(1.36, 0.15)	(1.77, 0.39)	(1.34, 0.18)	(1.19, 0.21)	(1.20, 0.19)	(1.28, 0.22)	(1.36, 0.21)
BD	0.81	0.92	0.74*	0.80	0.69*	0.77	0.73*
	(1.17, 0.25)	(1.74, 0.47)	(1.03, 0.31)	(1.16, 0.26)	(0.94, 0.37)	(1.08, 0.30)	(1.05, 0.39)
N	42	7	54	9	40	26	46
CS	0.69	0.88	0.67	0.71	0.40	0.52	0.76
	(0.84, 3.47)	(1.37, 1.04)	(0.82, 2.93)	(0.88, 1.48)	(0.42, 4.22)	(0.57, 3.20)	(1.00, 1.75)

The table gives the mean correlation of ratings given to each emotion, and its normally distributed transform is shown in parentheses (mean Fisher's Z transform, SD). HC, Correlation of ratings within the healthy comparison group; each healthy subject's ratings were correlated with those of the remaining group to derive these measures. BD, Correlation of ratings between the healthy comparison group and the brain-damaged group. n, Number of brain-damaged subjects (of a total of 103) whose correlation score was classified as impaired (defined as −1.65 SD below the mean of the healthy comparison group). CS, Case study (subject 1981).

*p < 0.05, significant difference in mean correlation rating between BD and HC.

Brain regions associated with emotion recognition impairment in the PM3 analysis

We performed a voxelwise lesion–deficit analysis to provide a standard description of the association between lesion location and impairment in emotion recognition from facial expressions (mean emotion recognition score). These analyses also served to inform the tractwise analyses, in the identification of potential cortical confounds and determination of white matter tract lesion–deficit associations.

We found lesion–deficit relationships in occipital, perisylvian, and frontal sectors of the right cerebral cortex and in the inferior frontal gyrus on the left (Fig. 2), consistent with previous findings by Adolphs et al. (2000). Specifically, the brain regions on the lateral surface in which lesions were found to be associated with deficits in emotion recognition in facial expressions included the following: bilateral frontal operculum (inferior frontal gyrus), right somatosensory cortices, right superior temporal gyrus, right inferior parietal lobule (including supramarginal gyrus), and right lateral occipital cortex. In the coronal sections, effects are apparent bilaterally in the anterior insula and portions of the caudate and putamen. The PM3 analysis also revealed extensive white matter involvement, particularly of the right hemisphere, extending from occipital cortices to the anterior frontal cortex. Statistical power was more limited for the left hemisphere, but it was sufficient to identify potential lesion–deficit associations on the left.

Main results of the fiber tract regression analyses

To investigate the role of fiber tract disconnection on the recognition of emotion from facial expressions, we first completed the main tractwise regression analyses including all 14 regressors (including for all tracts, gray matter, and lesion size).

In these analyses, only disconnection of the right IFOF significantly predicted impaired emotion recognition scores. This applied for the average emotion recognition score (t = 3.14; p < 0.0012; r = 0.32) (overall model: r² = 0.39; adjusted r² = 0.30; F_(14,88) = 4.09; p < 0.00002), and, individually, for sadness (t = 3.66; p < 0.00022; r = 0.36) (overall model: r² = 0.38; adjusted r² = 0.28; F_(14,88) = 3.79; p < 0.00005), anger (t = 3.02; p < 0.0016; r = 0.31) (overall model: r² = 0.43; adjusted r² = 0.34; F_(14,88) = 4.80; p < 0.000002), and fear (t = 3.03; p < 0.0016; r = 0.31) (overall model: r² = 0.40; adjusted r² = 0.31; F_(14,88) = 4.22; p < 0.00001). Descriptively, the IFOF tract in the right hemisphere appeared to follow clusters of local maxima in the PM3 map all along the tract (Fig. 3), also supporting the idea that long-range fibers are involved. Despite aforementioned limitations in statistical power for the left hemisphere, statistical power was equivalent for both hemispheres in the white matter compartments along the trajectory of the implicated tracts (Fig. 2). A post hoc test between left and right hemisphere of the estimated degree of disconnection of the IFOF in predicting performance was significant (t₍₈₈₎ = 2.68; p < 0.01), further supporting the finding. Thus, the laterality of our finding is likely not attributable to limitations in coverage and statistical power.

Figure 3.

Voxelwise lesion deficit analysis (PM3) alone and with overlay of IFOF, ILF, and SLF. Fiber tract perimeter (after thresholding the tract probability at 0.15) is depicted in white and overlaid with unthresholded PM3 results mapped on coronal slices of the right hemisphere, from the occipital lobe (left) to the frontal lobe (right). A, PM3 (no tract). B, IFOF. C, ILF. D, SLF.

To address possible confounds attributable to general visual (perception/recognition) or emotional (depression) impairments, we also performed the main regression analysis with additional covariates for basic visual perception, object recognition, and a composite measure for depression (BDI and MMPI). Critically, the results from this analysis did not differ from the original results, because damage to the IFOF on the right was still significantly associated with impaired emotion recognition (mean recognition score) from facial expressions (t = 2.76; p < 0.004). Thus, we have shown that basic visual object recognition does not appear to confound the finding of a significant association between IFOF disconnection and impairments in emotion recognition from facial expressions in our sample. These results are also consistent with those performed in the original study (Adolphs et al., 2000).

Case study

As mentioned in Materials and Methods, we identified a subject (1981) from the list of 12 subjects with lesions most specific to the right IFOF or ILF and with the smallest involvement of gray matter. This subject had damage caused by an infarct, involving only 0.08 cm³ of gray matter (smallest among the 103 subjects) and 4.6 cm³ of white matter (total sample mean = 9.3 and 11.7 cm³, respectively). This subject's lesion was primarily in the right IFOF (Fig. 4A) (68% of the lesion in the IFOF; 20% in the ILF) and followed quite precisely the posterior course of the right IFOF. We found clear impairments in the recognition of the facial expression of emotion in this subject (Fig. 4B), with emotion recognition scores several SDs below the mean of the normative group on several emotions (happiness, 1.04 SDs; sadness, 2.93 SDs; anger, 1.48 SDs; fear, 4.22 SDs; disgust, 3.20 SDs; surprise, 1.75 SDs; and average emotion score, 3.47 SDs). Table 2 documents the otherwise unremarkable neuropsychological profile for subject 1981, with normal perception, intelligence, attention, and memory. The subject's impairment in the recognition of the facial expression of emotion thus appeared to be quite specific.

Additional analyses addressing null results for ILF and SLF

Neither ILF nor SLF was implicated in the main regression analyses, but, in each case, there is a rationale to wonder whether the findings are false negatives. We hypothesized the ILF to be critical to emotion recognition, in addition to the IFOF, and the case study was consistent with a role for the ILF, given the partial overlap between the ILF and the case study subject's lesion. The SLF primarily overlapped with significant effects in the PM3 analysis: 69% (at p < 0.05) and 35% (at p < 0.01) of the voxels corresponding to the right SLF, excluding voxels overlapping with other fiber tracts.

We undertook supplementary analyses that addressed a potential role for colinearity (shared variance) in accounting for the absence of significant effects for these tracts. Shared variance arises from damage that tends to span anatomic structures attributable to either the characteristics of the lesions themselves and/or the use of a probabilistic atlas in which tracts can overlap.

Thus, anatomical proximity between structures (tracts, gray matter regions) and the relatively large size of the lesions on average lead to substantial correlations between the right SLF and IFOF regressors (0.72) and between the SLF and confounding perisylvian gray matter (0.89). Likewise, the right ILF and the right IFOF regressors showed substantial correlations (0.72), as did the ILF regressor and the cortical regressor (0.38).

We therefore performed supplemental regression analyses in which the 12 tracts were entered as the only independent variable, to assess the potential of the right SLF and ILF to predict average emotion score. In these analyses, only the right IFOF (t = 4.69; p < 0.000004; r = 0.42), the right ILF (t = 3.34; p < 0.0006; r = 0.32), and the right SLF (t = 4.28; p < 0.00002; r = 0.39) were associated with significant effects. Not surprisingly, the right IFOF presented the most significant effect.

We then performed additional analyses for the right SLF and right ILF with restricted combinations of regressors (see below) to further identify the factors potentially involved in the absence of significant effects for those two tracts in the main regression analyses with the full model. For the right SLF, the addition of the right IFOF regressor neutralized the significance of the SLF regressor (t = 1.16; p < 0.12). The addition of the right ILF or cortical regressor did not eliminate the significance of the SLF effects (respectively, t = 2.96, p < 0.002; t = 4.28; p < 0.00002). Similarly, when the right IFOF and right ILF were included together in the model, the ILF effect was no longer significant (t = −0.21; p < 0.58). When the right IFOF and the cortical regressor were included together, the right ILF effect was still significant (t = 3.34; p < 0.0006).

False-negative findings could also arise from lesion sampling effects because, in our subject sample, there were more lesions relatively specific to the IFOF. Twelve of the 25 subjects with lesions most specific to a given tract (based on the entropy analysis) had lesions primarily associated (highest peak in the probability distributions) with the IFOF; 23 of 25 had lesions associated with the IFOF when tracts with the second highest peak were also counted. For 8 of the 25 subjects, the highest peak in the probability distributions was for the SLF; 12 of 25 were counted when the second highest peak was also taken into account. Two of 25 subjects had highest peaks, and 6 of 25 had the first or second highest peaks, associated with the ILF. Thus, in these 25 subjects, lesions in the IFOF showed the highest specificity, in line with the ability of the regression to show significant effects for the IFOF, above and beyond the other tracts.

Supplementary analysis of the specificity of the IFOF in fear recognition impairments

We performed a supplementary analysis to further clarify the specificity of IFOF disconnection in fear recognition impairments. In particular, we were interested in assessing whether damage to the amygdala or disconnection of the amygdala could be contributing to the overall deficits in fear recognition observed in the group-level analysis and associated with disconnection of the IFOF, given the previous literature on the role of the amygdala in fear recognition (Adolphs et al., 1994, 1999; Young et al., 1995; Calder et al., 2001). This supplementary analysis was also motivated by the following reasons: (1) subject 1981, who showed a dramatic impairment in fear recognition, had a lesion that also encompassed the ILF (which connects visual cortices with the amygdala) (Catani et al., 2003); (2) in general, for the occipito-temporal components of the tract, damage affecting the IFOF is also likely to cause partial disconnection of the ILF, because of the proximity of the two tracts; and (3) the IFOF itself has a segment running near the amygdala, and thus it is likely that lesions to the amygdala will often involve the IFOF.

Accordingly, we reasoned that, if disconnection of, or lesions to, the amygdala are responsible for the fear recognition deficits, the lesion overlap of subjects with lesions tending to be maximally specific for the IFOF, weighted by the degree of fear recognition impairments (profound impairment associated with increased weight), should concentrate in occipito-temporal brain regions, which include the ILF and/or the amygdala. Conversely, if disconnections anywhere along the path of the IFOF can cause impairment in the recognition of fear, then substantial overlap should also be observed in the frontal sectors of this tract. In the list of 25 subjects with lesions maximally specific to a tract, we selected the six subjects with the lowest entropies associated with lesions in the IFOF (this corresponded to the first peak of the entropy distribution). We divided the binary lesion map of each subject by his/her emotion recognition score for fear, so that subjects with lower scores (in the direction of impairment) would have more weight. We then computed a weighted lesion overlap map by adding the values of weighted lesion maps across subjects for each voxel. Figure 5 shows the results of this analysis. The highest values in the weighted maps were found all along the tract, including in the frontal lobes, further supporting the causal role of lesions to the IFOF in fear recognition impairments.

Discussion

All analyses clearly implicated white matter in the visual recognition of the facial expression of emotion, which supports our general hypothesis that white matter damage can impair emotion recognition in this task. The tractwise analyses and case study supported the more specific hypothesis that association fiber tracts that connect visual and emotion-related cortical regions (IFOF and/or ILF) play an essential role in normal emotion recognition from facial expressions. The case study provides evidence that damage to one or both of these tracts is sufficient to impair emotion recognition in our task. The group-level tractwise analysis confirmed the right IFOF as a critical component of the large-scale network that subserves recognition of facial emotion. Damage to the right IFOF significantly predicted worse performance even when the following had been covaried out: (1) the presence of damage in any other association fiber tract; (2) overall lesion volume; and (3) the presence of damage in implicated gray matter sectors. We have demonstrated previously (Rudrauf et al., 2008a) the effectiveness of our approach in isolating one of several probabilistically overlapping tracts in a basic visual test bed.

The IFOF is positioned to mediate long-range interactions between the ventral visual stream, subserving object and face recognition, and emotion-related cortical regions (i.e., orbitofrontal cortex). It may facilitate the combination of affective responses and early visual information, allowing emotion recognition from facial expressions. Recent primate studies have found that neurons in the orbitofrontal, ventromedial prefrontal, and inferior prefrontal cortices respond to faces (Wilson et al., 1993; O'Scalaidhe et al., 1997), and analogous regions in humans respond to facial emotion (George et al., 1993; Hornak et al., 1996; Sprengelmeyer et al., 1998; Nakamura et al., 1999; Kawasaki et al., 2001). Furthermore, a recent study using magneto-encephalography (Rudrauf et al., 2008c) supported a role for long-range association fiber tracts (i.e., IFOF and ILF) and/or alternative subcortical pathways (Liddell et al., 2005) in rapid interactions between emotion and visual processing. The current study was not designed to adjudicate whether the IFOF mediates such rapid interactions or processing over longer timescale, because the present paradigm did not incorporate any processing speed conditions.

The ability to interpret the results has other limitations attributable notably to the specificity of our paradigm and lack of appropriate control tasks. It is possible that the IFOF would not appear to play a role in the processing of facial expression of emotion with other tasks relying less on the retrieval of conceptual knowledge. The IFOF might also play a more general role in the appreciation of visual emotional stimuli or in the apprehension and retrieval of emotional information.

Future research should investigate the specificity versus generality of the role of the same white matter tracts (e.g., IFOF) in the modulation of emotion recognition, using for instance different visual stimuli (natural scenes, body posture), cognitive strategies, and stimuli from other sensory modalities (e.g., auditory). We hypothesize that the same tracts (IFOF/ILF) will facilitate recognition of emotion from other visual stimuli (e.g., body posture) but not from other sensory modalities. Emotional expression in lip speech or prosody may involve more dorsal regions (i.e., superior temporal sulcus) and implicate other white matter tracts (i.e., SLF). Although our participants did not have basic visual discrimination or facial recognition impairments (e.g., Benton face recognition), a recent study suggests limitations in the sensitivity and specificity of these standard face recognition tasks (Duchaine and Weidenfeld, 2003). We controlled for this potential confound by including covariate terms for basic visual and object recognition performance in the participants, and this did not affect our results. Future investigations should include explicit control tasks.

Possible limitations attributable to the publicly available DTI fiber tract atlas that we used can be noted. The atlas was based on the FACT algorithm, which may overestimate tract volume, thus increasing the risk of colinearity between tracts attributable to the increased probability of shared voxels among tracts. It was also constructed with a younger subject population (25–40 years) than our patient sample (mean of 53.6 years). Although there is some evidence that the right IFOF volume decreases with age (Thomas et al., 2008), to our knowledge the effects of aging on tracts do not lead to a reorganization of their trajectory: it is therefore unlikely that false positives attributable to tract misattribution would result from aging. Age differences may be accounted for in future studies by increasing the number and age distribution of subjects in the probabilistic atlas. We refer the reader to the study by Rudrauf et al. (2008a) for a detailed validation of the method using the same atlas.

Our findings only significantly implicated the right IFOF. The post hoc comparison between left and right hemispheres corroborated the lateralization effect. However, statistical power was slightly broader on the right, although similar along sectors of the IFOF bilaterally. Lesion distribution among patients was diffuse and bilateral, but we cannot rule out left hemisphere contributions (and left IFOF) to the observed impaired processes. Assuming that the observed lateralization is not a false positive, we consider the following points in the broader context of emotional processing and valence. Right lateralization in our findings is compatible with the “right hemisphere hypothesis of emotion” (Levine and Levy, 1986; Mandal et al., 1996), although effects for positively valenced emotions (happiness) were limited. The lateralization effect for negative emotions partially supports the valence asymmetry model, because no effect was found on the left for positive emotions (Sackeim et al., 1982; Davidson, 1984; Davidson et al., 1987; Burton and Levy, 1989).

We cannot conclude that the right IFOF is the only association tract involved. The SLF and ILF showed significant effects when no other tract regressors were used. Shared variance between tract regressors and sparse sampling for the ILF makes it possible that the absence of significant effects for the SLF and ILF in the main analysis could be false negatives. Although descriptively the effects in the PM3 analysis (Fig. 3) overlap with the SLF, these effects appear to be less tract specific than the IFOF. The entropy analysis (proxy for specificity of lesion to a given tract) showed that few subjects had ILF-specific lesions. It is possible that isolated analysis of the tract exaggerates its contributions. Also, in the PM3 analysis, descriptively there was no evidence of lesion–deficit relationships in the sectors of white matter containing the ILF, whereas in the effective coverage maps, there were no apparent statistical power issues.

We expected the ILF to play an important role in facial expression recognition. This tract connects visual cortical areas with the amygdala (Catani et al., 2002, 2003; Schmahmann and Pandya, 2006). The amygdala is important for emotion processing and interactions between visual processing and emotion (LeDoux, 1998; Rolls, 2000; Vuilleumier, 2005) and has been associated with the recognition of fear from facial expressions (Adolphs et al., 1994, 1999; Young et al., 1995; Calder et al., 2001). Furthermore, disconnection of the ILF may contribute to deficits in the ability to recognize familiar faces (i.e., prosopagnosia) (Catani et al., 2003; Catani and Thiebaut de Schotten, 2008; Fox et al., 2008). Given the statistical limitations, we can neither conclude nor refute that the ILF is critical to emotion recognition in facial expressions.

We note that the basic PM3 analysis did not suggest any significant association between damage in the amygdala and fear recognition impairment. Our results emphasize the importance of attention to both gray and white matter in lesion studies. This may be especially important for the amygdala, a structure with a history of conclusions drawn from nonselective lesions (Kluver and Bucy, 1937), and more subtle impairments after selective lesions (Meunier et al., 1999). Future lesion studies of the role of the human amygdala in emotion recognition should quantify the extent of subjacent white matter involvement, given that there is a segment of the IFOF that runs close to the amygdala, which could be a potential confound.

Our findings strongly support the hypothesis that the right IFOF is a critical component of the neural system necessary for the recognition of the facial expression of emotion.