DIFFERENT ASPECTS OF FACIAL AFFECT RECOGNITION IMPAIRMENT FOLLOWING TRAUMATIC BRAIN INJURY: THE ROLE OF PERCEPTUAL AND INTERPRETATIVE ABILITIES

Abstract

It is well established that many individuals with traumatic brain injury (TBI) are impaired at facial affect recognition, yet little is known about the mechanisms underlying such deficits. In particular, little work has examined whether the breakdown of facial affect recognition abilities occurs at the perceptual level (e.g., recognizing a smile) or at the verbal categorization stage (e.g., assigning the label ‘happy’ to a smiling face). The aim of the current study was to investigate the integrity of these two distinct facial affect recognition sub-skills in a sample of 38 individuals with moderate-to-severe TBI and 24 demographically matched healthy individuals. Participants were administered an affect matching (perceptual skills) and an affect labeling (verbal categorization skills) task. Statistical analyses revealed that, while individuals with TBI showed significantly higher levels of impairment in the verbal categorization task than in the perceptual task, they underperformed healthy comparison participants on both tasks. These findings indicate that facial affect recognition impairment can occur at different cognitive stages following TBI, suggesting the necessity of careful screening to offer targeted treatment. Moreover, they provide further neuropsychological evidence supporting the notion that distinct types of sub-skills are necessary to achieve successful recognition of facial affects.

INTRODUCTION

Facial affect recognition deficits have been extensively reported in individuals with moderate-to-severe traumatic brain injury (TBI), and a recent meta-analysis estimated that up to 39% of individuals with TBI are likely to be impaired in this domain (Babbage et al., 2011). In the TBI literature, however, facial affect recognition has often been treated as a unitary cognitive skill, with relatively little attention to contributions of cognitive sub-skills that can impact successful recognition of facial affect (Jackson and Moffat, 1987; Knox and Douglas, 2009; Rigon, Turkstra, Mutlu, & Duff, 2016; Rosenberg, Dethier, Kessels, Westbrook, & McDonald, 2015).

Studies on healthy individuals have provided support for the notion that facial affect recognition requires the orchestration of distinct cognitive processes: (1) the perception of the facial features that carry information that is relevant for facial affect recognition (e.g., the perception of upturned corners of the mouth or of the configuration of the zygomaticus major, or a frown and the contraction of the corrugator supercilii), and (2) the ability to perform a verbal categorization of the perceptual features detected, and connect them with the corresponding affect label (e.g., form a connection between upturned lips, a smile, and the semantic label of ‘happiness’) (Adolphs, 2002; Hariri, Bookheimer, & Mazziotta, 2000; Palermo, O’Connor, Davis, Irons, & McKone, 2013). Although verbal categorization relies on the ability to correctly detect facial features, and thus requires successful perception of affect information, it also necessitates the ability to establish a connection between such features and the semantic knowledge of an affect; thus, perception of facial features is necessary to successfully achieve verbal categorization, but the opposite is not true (Barrett, Lindquist, & Gendron, 2007; Palermo, et al., 2013).

Affect perception and categorization have been assessed using different tasks. Perception has been tested with affect matching tasks, in which participants see three or more emotional faces and are asked to select which faces show the same (or different) emotions; while the combination of perception and categorization has been tested using tasks in which participants see a face and are instructed to choose the emotional label that best describes its affect (Hariri, et al., 2000; Neumann, Keiski, McDonald, & Wang, 2014). Scores on matching and labeling tasks are moderately to highly correlated in typical young adults, but are below the upper bound of the correlation, indicating that although matching and labeling tasks tap into highly related cognitive processes, they are not fully overlapping and represent theoretically separable cognitive processes (Palermo, et al., 2013).

Within the TBI literature, few studies have compared performance on matching and labeling tasks. Green and colleagues (2004) and Genova and colleagues (2015), and administered matching and labeling tasks to samples of individuals with moderate-to-severe TBI, and found they underperformed healthy comparison peers on both tasks; Ietswaart and colleagues (2008) found similar results in a sample that included individuals with TBI ranging from mild to severe (Genova et al., 2015; Green, Turner, & Thompson, 2004; Ietswaart, Milders, Crawford, Currie, & Scott, 2008). A study by McDonald and Crocker (2004) attempted to examine the relationship between matching and labeling performance in a sample of individuals with severe TBI; the authors reported that individuals with TBI were impaired at both tasks, but they did not find a correlation between performance on the two tasks (Croker and McDonald, 2005). However, it should be noted that the sample size the authors worked with was relatively small; moreover, for all four studies cited, the number of trials for each task was relatively small (Croker and McDonald, 2005).

In the TBI literature, matching and labeling tasks have often been used interchangeably as measure of facial affect recognition (Babbage, et al., 2011). The lack of exploration of the cognitive sub-skills involved in emotion recognition in individuals with TBI may be related to the nature of the tools used to study affect recognition in TBI (Palermo, et al., 2013). Most tasks employed were designed to detect differences between clinical and non-clinical populations (there are some exceptions, such as the Emotion Recognition Test), and to categorize participants as impaired or not (Rosenberg, et al., 2015; Rosenberg, McDonald, Dethier, Kessels, & Westbrook, 2014). Moreover, many of these tasks are administered as part of larger batteries that assess several different domains of cognition, and they tend to include few trials. As a consequence, they often yield ceiling effects and a left skewed distribution within healthy comparison samples, as well as within the 60 to 80% of individuals with TBI who are unimpaired at affect recognition (Babbage, et al., 2011). Although these batteries and tasks are ideal in clinical and bedside settings, as they allow rapid assessment of a client’s need for rehabilitation and monitoring in a specific cognitive area, they are not ideal in an experimental context because of the statistical and methodological challenges presented by data that are not normally-distributed or have ceiling or floor effects (Palermo, et al., 2013).

Determining whether individuals with TBI show specific impairment on perceptual or verbal categorization skills, or whether they are equally impaired at both affect recognition processes, is essential to inform intervention for psychosocial deficits following TBI. For instance, knowing that individuals with TBI are impaired at the perceptual stage of affect recognition could support a focus on detection of specific patterns of facial muscle movements and configurations. By contrast, a finding of impairment in verbal categorization of emotional faces would support an emphasis on associating correct emotional labels with facial configurations.

The aim of the current study was to determine whether affect recognition deficits following moderate-to-severe TBI are limited to matching or labeling tasks, or involve both perceptual and categorization skills. We employed an affect-matching task and an affect-labeling task recently developed by Palermo and colleagues (Palermo et al., 2017; Palermo, et al., 2013). The tasks have demonstrated validity and reliability for assessment of individual differences within healthy populations, and thus do not suffer from ceiling effects; and they were chosen due to their sensitivity to inter-individual variability, and thus their ability to detect even slight differences in affect recognition performance (Palermo, et al., 2013). Our theoretical approach, as well as the large sample and mixed-effect analysis method provide a significant methodological advance over previous work, and afford us sufficient power to examine group difference in specific sub-skills of facial affect processing.

MATERIAL AND METHODS

Participants

Thirty-eight individuals with moderate-to-severe TBI and 24 healthy comparison (HC) participants were tested for this study. Participants were recruited through the University of Iowa community and through the University of Iowa Brain Injury Registry (Rigon, Turkstra, et al., 2016; Rigon, Voss, Turkstra, Mutlu, & Duff, 2016, 2017).

The groups were not significantly different on age (t(37.19)=.951, p>.05), education(t(60)=−.93, p>.05) or sex (X²(1, N=62)=1.35, p>.05) (See Table 1).

Table 1.

Demographic information for the HC and TBI participants

	HC	TBI	p	Effect size
N	24	38	N/A	N/A
AGE (Mean±SD)	48.29±20.14	52.76±14.08	.35	d=.26
SEX (Females)	9	20	.24	φ=.144
EDUCATION (Mean±SD)	15.5±2.14	14.97±2.17	.36	d=.25
CHRONICITY(Months, Mean±SD)	N/A	122.29(±139.62)	N/A	N/A

Note: HC=Healthy comparison participants, TBI=Traumatic brain injury, p=p-value, SD=Standard Deviation, N/A=Not Applicable, d= Cohen’s d, φ=Phi. All effect sizes were < medium.

Inclusion criteria for individuals with TBI were (1) history of moderate-to-severe TBI, (2) chronic post-injury phase (all participants were > 12 months post injury), (3) aphasia quotient higher than 93.8 on the Western Aphasia Battery (WAB) (Shewan and Kertesz, 1980). Language deficits were ruled out to ensure that participants were able to correctly understand and follow instruction, and that poor performance on the tasks administered was not due to language deficits.

Participants had sustained their TBI a minimum of 17 months and a maximum of 523 months before testing (Mean=122.29, SD=139.62). One participant, included in the final sample, had sustained two separate TBIs. Causes of injury were falls (21), motor vehicle accidents (13), assaults (3), and non-motor vehicle accidents (2) (See Table 2 for further injury details).

Table 2.

Detailed injury information for all participants with TBI

ID	GCS	LOC	PTA	Etiology	CT/MRI findings	Chronicity
1	.	several hours	.	MVA	ICH	321
2	.	minutes	~2 weeks	MVA	.	393
3	3	2 weeks	22 days	Fall	SAH	67
4	8	.	.	Fall		44
5	.	.	.	Fall	CT, stability of SAH	55
6	.	.	.	Fall	CT	190
7	15	> 30 min	.	Fall	ICH	307
8	6	.	.	MVA	SAH and ICH	59
9	.	.	.	Fall	SAH	51
10	15	.	.	Fall	SAH, occipital fracture	64
11	.	none	<24 hours	Fall	SAH	39
12	.	<30 min	<24 hours	Assault	SAH	42
13	15	+	UKN	Assault	EDH, SDH	104
14*	.	<30 min, none	<24 hours, none	Fall, MVA	SDH	68
15	14	.	>24 hours	MVA	SAH	53
16	4	<30 min	>24 hours	N-MVA	ICH	24
17	7	+	>24 hours	Fall	EDH, SDH	19
18	3	+	>24 hours	Fall	SDH, parietal fracture	27
19	15	<30 min	<24 hours	Fall	ICH	24
20	.	+	.	Fall	SDH	20
21	.	+	>24 hours	MVA	SAH	32
22	10	<30 min	>24 hours	Fall	SAH, ICH	28
23	.	1 hour	1–2 days	Fall	SAH	26
24	6	+	.	MVA	ICH	81
25	.	.	.	N-MVA	SDH, SAH	17
26	3	10 days	.	MVA	EDH, SDH, ICH right orbital wall fracture	157
27	.	minutes	30 minutes	Fall	SAH	20
28	.	none	none	Fall	SDH	17
29	.	<1 minute	none	Fall	EDH	53
30	9	.	> 1 week	Fall	SAH, SDH	29
31	.	2 hours	.	Fall	skull fracture	523
32	.	.	months	MVA		187
33	3	.	.	MVA	SAH	195
34	3	1.5 months	.	MVA	internal decapitation	356
35	.	minutes	1 hour	MVA	SAH	156
36	.	induced coma, days	4 weeks	MVA	SAH	247
37	10	2 hours	1 week	Assault	.	509
38	.	2 minutes	24 hours	N-MVA	ICH	43

Note: MVA= Motored Vehicle Accident; NMVA= Non-motored Vehicle Accident; PTA= Post Traumatic Amnesia; LOC= Loss of Consciousness; SAH= Subarachnoid Hemorrhage; EDH= Epidural Hemorrhage; SDH= Subdural Hemorrhage; ICH=Intracranial hemorrhage; +=presence of LOC or PTA, but duration is unspecified. “.” indicates information not available.

^*Sustained two TBIs on two different occasions.

TBI severity was assessed using the Mayo Classification System (Malec et al., 2007). Participants were considered moderate-to-severe if at least one of the following criteria was met: (1) Glasgow Coma Scale (GCS) <13 (i.e., moderate or severe according to the GCS), (2) positive acute CT findings or lesions visible on a chronic MRI, (3) loss of consciousness (LOC)>30 minutes or post-traumatic amnesia (PTA) >24 hours, and (4) retrograde amnesia >24 hours. Injury-related information was collected using a combination of medical records and a semi-structured interview with participants. In the current sample, information on GCS was available for 18 participants, on LOC for 28 participants, information on retrograde or anterograde amnesia on 24 participants, and on CT or MRI findings for 33 participants.

Inclusion criteria for the HC group were: 1) no self-reported history of head or brain injury or loss of consciousness, 2) no history of neurological, psychiatric or learning disorders, and 3) aphasia quotient higher than 93.8 on the WAB.

Neuropsychological measures & psychological distress

To characterize the sample of participants and conduct post-hoc explanatory analyses to determine the association between affect recognitions skills and other cognitive domains we administered: (1) the California Verbal Learning Test (CVLT-Immediate), short-delay verbal recall (CVLT-Short Delay) and long-delay verbal recall (CVLT-Long Delay), to obtain a measure of verbal learning and memory (Delis, Freeland, Kramer, & Kaplan, 1988); (2) the Symbol Search and Coding subtests of the Wechsler Adult Intelligence Scale (WAIS), to obtain a composite index of Processing Speed (WAIS- PSI) (Holdnack, Xiaobin, Larrabee, Millis, & Salthouse, 2011); (3) the Trail Making Test (Trails A and B), to measure executive functioning and task-switching (Gordon, 1972); (4) the Brief Symptoms Inventory (BSI-18), to evaluate psychological distress and psychiatric disorders on dimensions of Somatization (e.g., pain, faintness, nausea), Depression, Anxiety, and Global Severity Index (GSI) (Derogatis and Melisaratos, 1983); (5) the Benton Facial Recognition Task (Benton), to exclude visuo-perceptual deficits in face recognition and compare facial recognition performance across the two groups (Levin, Hamsher, & Benton, 1975); and (6) the Positive and Negative Affect Schedule (PANAS), to assess state positive and negative affects across on the day of testing (administered at the beginning of the battery) (Watson, Clark, & Tellegen, 1988).

Several of these measures were collected as part of battery administered to the University of Iowa TBI registry patients (Rigon, Duff, McAuley, Kramer, & Voss, 2016; Rigon, Turkstra, et al., 2016; Rigon, Turkstra, Mutlu, & Duff, In press; Rigon, Voss, et al., 2016). Due to study attrition, results for WAIS, CVLT, Trails, and BSI were missing from one participant in the HC group.

Affect recognition tasks

Affect labeling and affect matching tasks were adapted from Palermo and colleagues (Palermo, et al., 2017; Palermo, et al., 2013) (See figure 1). The matching task (Matching) is an odd-expression-out task, in which participants saw three faces displaying an emotional expression. Two faces of different individuals showed the same expression, and a third showed a different expression, for a total of three different actors for each trial. Participants were asked to indicate the face displaying a different affect from the other two. The target affect and distractor affects were always one of the six basic emotions described by Ekman et al (anger, happiness, disgust, fear, sadness and surprise) (Ekman and Friesen, 1971). For further information on the development and characteristic of both the matching and the labeling task, see Palermo et al, 2013.

Figure 1. Facial Affect Recognition Tasks.

Facial affect recognition task, adapted from Palermo and colleagues (2013). A. Matching affect recognition task. B. Labeling recognition task.

The Matching task consisted of 100 trials, and each trial lasted a maximum of 10 seconds. First, participants saw all three faces simultaneously for 5 seconds. Trial duration was lengthened to adapt the task for a TBI population, but a limited duration was retained, as in out-of-the-lab settings facial affects are not displayed for indefinite amounts of time, and thus a time limit rendered the task more ecological; for further information on the original task (see Palermo and colleagues, 2013). After 5 seconds, the faces disappeared and participants were given 5 more seconds to make their choice by pressing the button (1, 2, or 3) corresponding to the face showing a different affect. Participants could respond whenever they made a decision, within the 10-second time for each stimulus. To prevent poor performance due to fatigue, every 20 trials participants were given the opportunity to take a break. The time available for each trial was selected based on the range of reaction times recorded on a sample of individuals with TBI on a previous emotion recognition study to ensure that each participant was able to produce a motor response in the allotted time, and that individuals with TBI were not penalized by the duration of the task (Rigon, Turkstra, et al., 2016).

The labeling task (Labeling) consisted of 144 trials, and each trial lasted a total of 10 seconds. On this task, a face was presented in the center of the screen for 2 seconds, followed by a blank screen for 8 seconds. To decrease the working memory demand, labels of the six basic affects remained visible at the bottom of the screen for the full 10 seconds, and participants were asked to select the one corresponding to the affect shown by the stimulus. Participants could select the correct answer any time in the 10-second interval. For both Labeling and Matching, as soon as participants selected their answer, the task moved to the next trial. Participants could not change their answer after making a decision.

The number of trials differed between Matching and Labeling because different strategies were required to answer correctly: in the Matching task, to successfully identify the different affect, participants were not required to form an association between the label of an affect and the corresponding facial features, and thus they were unlikely to notice a discrepancy in the number of trials for each affect. Conversely, in the Labeling task participants were specifically instructed to verbally categorize the specific affect, and a difference in the number of trials, for example, happy and sad affect might have confused participants and influenced their response. For further information about the development and features of the matching task, see Palermo et al., 2013.

The matching task was always administered before the labeling task. This was decided following the procedure adopted by the creators of the task (Palermo, et al., 2013).

Statistical analysis

For all neuropsychological measures, one-tailed independent sample t-tests were performed, based on the hypothesis that individuals with TBI would underperform HCs across all measures. As several neuropsychological and psychological distress measures were collected, and as the main purpose of collecting such data was not to test a research hypothesis but to further characterize the TBI sample, a liberal multiple comparison correction approach was adopted, and p<.01 was considered significant. The family-wise error rate (FWER) for group comparisons (for 14 tests and p<.01) was .13. For one participant with TBI, Part B of Trails had to be discontinued, and his score was re-coded as the lowest values from the dataset (Costa, 2014). Visual inspection revealed the presence of two outliers in the Trails B, one participant with TBI and one comparison participant, whose scores were over three standard deviations below the group average. For this reason, group comparisons were computed both with and without the outliers. Given the wide age and education range of participants, we opted for converting individual Trails B scores into z-scores over alternative approaches (e.g., subtracting Tails A score from Trails B score) (Tombaugh, 2004).

A Shapiro-Wilk task was used to assess normality of both Labeling and Matching scores for both tasks (all W<.951, all p>.1). Skewness for the data ranged from .1 to −0.8, indicating low to moderate skewness. Two-tailed Pearson correlations were used to examine the association between the two affect recognition tasks. Similarly, two-tailed Pearson correlations between the two affect recognition tasks and neuropsychological measures were computed to determine the within-group association between performance on both affect recognition tasks and skills in the cognitive domains assessed by the neuropsychological tests administered. For the Trails B, the scores of two outliers (one in the TBI group, one in the HC group) were changed to the lowest values in the dataset in order to reduce the risk of spurious or outlier-driven correlation. As stated above, p<.01 was considered significant. The family-wise error rate (FWER) for within-group correlations (for 13 tests and p<.01) was .12.

To examine group differences between the two affect recognition tasks, two different types of analysis were performed. The aim of the first analysis was to determine whether the degree of affect recognition deficit within the TBI group differed on the two tasks. To this end, the number of impaired individuals with TBI differed for Matching and Labeling was determined. In the first analysis, normative scores for each task were calculated by using the average of the HC group (normative data from the creators of the task were not utilized because of the slight modification made to the task for the current study, i.e. longer presentation times for the stimuli, and because of the markedly larger age difference in the current sample). Using the normative data, two different impairment thresholds were calculated: a more conservative threshold, in which individuals with TBI were marked as impaired in matching or labeling based on whether they performed more than 2 standard deviations below the average of the HC group; and a less conservative threshold, in which individuals with TBI were marked as impaired in matching or labeling based on whether they performed more than 1.5 standard deviations below the average of the HC group. Following this procedure, a McNemar’s test was employed to determine whether nominal dichotomous variable (impaired vs. non-impaired) differed between the two affect recognition tests. This allowed us to examine whether individuals with TBI were significantly more impaired on Matching than on Labeling (or vice versa).

The second analysis was performed to examine whether individuals with TBI were impaired on either task when compared with healthy individuals. We examined group differences on each task by controlling for the variance provided by factors such as subject and item characteristics, as is recommended (Baayen, Davidson, & Bates, 2008). For the second analysis, each answer was marked as either correct or incorrect, resulting in binary outcome variables. The data were modeled in R using a mixed-effect logistic regression approach (using the package lme4) (Bates, 2016). Therefore, instead of beta coefficients that express the slope of the relationship between predictors and the dependent variable, in logistic regression we compute the probability of the dependent outcome occurring given known predictors. Like a beta coefficient in multiple linear regression, in logistic regression a Z-statistic indicates whether the coefficient for each predictor is significantly greater than zero. To aid in interpretation of each coefficient, we report the odds ratio for significant predictors, which indicates the change in odds for the dependent variable associated with a 1-unit change in the predictor.

For both Labeling and Matching, raw emotion recognition scores (i.e., percent of correct response for each task) were entered as dependent variables. For Labeling, we tested the presence of a group-by-affect interaction as well as main effects of group and affect type. This was chosen based on previous reports that individuals with TBI are significantly more impaired when attempting to recognize specific emotion types, and negative emotions in particular (Rigon, Turkstra, et al., 2016; Rosenberg, et al., 2014). A likelihood-ratio test determined that the maximal random effect structure supported by the data (identified using a model simplification approach) included random intercepts for subject and item, as well as a random subject-by-affect type slope (X²(1)=779.66, p<.001).

For Matching, only group was entered as predictor. The decision not to examine the interaction between group and affect type was due to the consideration that selection of the correct answers was based both on the characteristics of the target stimulus and on the characteristics of the distractor; thus classifying a trial based uniquely on the affect of the target stimulus (or of the distractor) would not adequately characterize its nature. A likelihood-ratio test determined that the maximal random effect structure supported by the data included random intercepts for subject and item, as well as a random item-by-group slope (X²(2)=9.69, p<.01).

We entered age and sex as covariates in both Labeling and Matching models, as these have previously been found to be associated with affect-recognition abilities, with older adults and males performing worse than their counterparts (Kessels, Montagne, Hendriks, Perrett, & de Haan, 2014; Montagne, Kessels, De Haan, & Perrett, 2007; Rigon, Turkstra, et al., 2016).

RESULTS

Neuropsychological Tests

Participants with TBI performed significantly worse than HCs on the Trails A (t(52.54)=−4.52, p<.001), but not on Trails B (t(59)=.27, p>.05). When the two outliers were removed from the sample, individuals with TBI performed significantly worse than HCs on the Trails B as well (t(57)=−2.9, p<.01) (See Table 3 for estimates of effect sizes).

Table 3.

Group comparison on neuropsychological measures and psychological distress tasks

	HC (Mean±SD)	TBI (Mean±SD)	Group Difference (P-value, Cohen’s d)
Trails A (Z score)	.89±.58	−.32±1.48	p<.001, d=1.08
Trails B (Z score)	−.23±3.05	−1.07±2.68	p=.27, d=.29
Trails B (Excluding outliers)	.35±1.19	−.72±1.62	p=.003, d=.75
BFRT	46.62±3.49	46.24±3.81	p=.68, d=.1
GSI (BSI)	46.7±8.04	53.53±7.72	p=.001, d=.87
Somatization (BSI)	52.63±8.56	46.17±5.12	p=.001, d=.91
Depression (BSI)	52.52±9.73	48.65±8.86	p=.06, d=.42
Anxiety (BSI)	50.26±8.77	46.22±7.57	p=.04, d=.49
PANAS Positive	32.92±8.21	29.71±9.15	p=.16, d=.37
PANAS Negative	13.07±5	11.92±4.12	p=.35, d=.25
CVLT-Immediate	56.22±8.96	51.37±13.85	p=.05, d=.42
CVLT-Short	.46±.89	.08±1.15	p=.08, d=.37
CVLT-Long	.52±1.16	−.01±1.44	p=.06, d=.41
WAIS-PSI	111.30±15.64	97.79±13.95	p=.001, d=.91

Note: HC=Healthy comparison participants, TBI=Traumatic brain injury, p=p-value, SD=Standard Deviation, Trails A= Trail Making Test A, Trails B= Trail Making Test B, BFRT=Benton Facial Recognition Task.

All participants were able to correctly perceive faces. Participants with TBI did not score significantly lower than HCs on the Benton Facial Recognition Test (t(60)=−.4, p>.05). No participants scored in the impaired range (<38), although two participants in the TBI group and one participant in the HC group scored in the borderline range (all three participants had scores of 40).

Participants with TBI and HCs did not significantly differ on either the PANAS positive (t(60)=1.44, p>.05) or the PANAS negative (t(60)=.95, p>.05). On the BSI, there was a significant group difference on the GSI composite (t(59)=3.3, p<.01), mainly driven by individuals with TBI reporting more Somatization symptoms (t(59)=3.69, p<.001). There were no significant differences for Depression (t(59)=1.55, p>.05) or Anxiety (t(59)=1.83, p>.01), although participants with TBI reported more anxiety than HCs (p=.04).

There was no significant group difference on the CVLT-Immediate (t(58.67)=−1.66, p=.05), CVLT-Short Immediate (t(59)=−1.34, p>.05), or CVLT-Long (t(59)=−1.51, p>.05). As predicted, participants with TBI had lower WAIS-PSI scores than HCs (t(59)=−3.05, p<.001).

Performance validity

We used the z-scores for Trails A to assess performance validity (i.e., effort exerted by participants). Within the HC group, z-scores for Trails A ranged from −.22 to 2.04, suggesting that participants performed as expected based on their education level and age (Mean=.89, SD=.58). Within the TBI group, scores ranged from −5.25 to 1.62 (Mean=−.32, SD=1.47). Considering that Trails A is a measure of processing speed (Salthouse, 2011), and that individuals with TBI often show impairment in this domain (indeed, in this sample WAIS-PSI scores for the TBI group were lower than for the HC group) (Mathias and Wheaton, 2007) we did not consider this a cause for concern, or an indication of poor effort.

Matching vs. Labeling: Within-group comparisons

Individuals with TBI were slower than HCs both at Matching (t(59)=−2.60, p>.05; HC Mean= 3.98±2.84, TBI Mean= 6.12±3.99), and at Labeling (t(59)=−2.42, p>.05; HC Mean= 4.78±2.34, TBI Mean= 4.78±3.25). Averages times for both groups were less than 10 seconds, the maximum amount of time allowed for each trial.

Two-tailed Pearson’s correlations revealed that within both groups, performance on the Matching and Labeling tasks were highly and positively associated (TBI: r=.82, p<.001; NC: r=.61, p=.001) (See figure 2).

Figure 2. Correlation between matching and labeling performance within TBI and HC groups.

Pearson’s correlation between performance on the labeling task and performance on the matching task. For both groups correlations were highly significant.

For the Labeling task, on average HCs responded correctly to 78.41% of the items. The standard deviation was 7.84, so the more conservative threshold for impairment was 62.73 (2 standard deviations below the mean), and the less conservative threshold was 66.65 (1.5 standard deviations below the mean). Using the more conservative threshold, within the TBI group 17 (44.7%) individuals were impaired in the labeling task, while using the less conservative threshold 20 (52.6%) individuals were impaired.

For the Matching task, HCs responded correctly to 72.25% of the items. The standard deviation was 8.38, so the more conservative threshold for impairment was 55.48 (2 standard deviations below the mean), and the less conservative threshold was 59.67 (1.5 standard deviations below the mean). Using the more conservative threshold, within the TBI group 4 (10.5%) individuals were impaired in the matching task, while using the less conservative threshold 7 (18.4%) individuals were impaired.

For both thresholds, the more conservative threshold, a McNemar’s test revealed a significant difference between tests, with significantly more impairment for the Labeling task than for the Matching task (both ps<.001). No participant’s scores were in the impaired range on the Matching task but not the Labeling task.

The same analyses were repeated using percentiles as cut-off scores in order to exclude spurious findings as an effect of skewness, and they yielded the same results. The results are reported in the supplementary materials.

Logistic regression: Labeling task

There was a significant group effect of Labeling scores (Z=−3.26, p=.001, 95% CI [−.45, −.11]) with individuals with TBI being less likely to produce a correct answer to the average item. The log odds of selecting the correct label for the average stimulus for individuals with TBI was 1.29, while the log odds of selecting the correct label to the average stimulus for HCs were 1.84. In probability terms, the probability of a correct response to the average item was .78 for individuals with TBI, and .86 for HCs. The odds of a correct response were 1.74 times greater for HCs than for individuals with TBI (see Figure 3).

Figure 3. Probability of correct response – Group comparison (Labeling).

A. Percent of correct response. Error bars represent the standard deviation. B. Mean probability of giving the correct response to the average item in the labeling task for the TBI group and for the HC group. C. Each data point represents a participant’s probability to respond correctly.

In accordance with the literature, we also found a significant age effect, with the odds of selecting the correct label for the average item decreasing by 1.25 with a one standard deviation increase in age (Z=−2, p<.01, 95% CI [−.38, −.08]), and a significant sex effect, with the odds of selecting the correct label 1.19 times greater for women (Z=−1.97, p<.05, 95% CI [−.34, −.004]).

There was no significant interaction between group and affect type (all Zs<1.1, all Ps>.28, all 95% CI [−.32, .29]), revealing that HC’s superiority on the task did not significantly vary based on affect type.

Given the lack of an interaction, we examined the affect effect across groups by comparing each affect to performance on labeling of happy faces, as happiness has been widely reported as the easiest facial affect to recognize. The log odds of selecting the correct label to happy stimuli were 3.18, which means that participants had a probability of .96 of selecting the correct response when they were shown a happy face. The log odds dropped to .98 for anger (.72 probability of a correct response, the odds 9.98 times lower than for happiness, Z=−6.32, p<.001, 95% CI [−2.99, −1.61]), .94 for disgust (.72 probability of a correct response, 10.47 times lower than for happiness, Z=−6.04, p<.001, 95% CI [−3.07, −1.63]), −.02 for fear (.49 probability of a correct response, 27.14 times lower than for happiness, Z=−8.62, p<.001, 95% CI [−4, −2.60]), 1.74 for surprise (.85 probability of a correct response, 4.70 times lower than for happiness, Z=−4.24, p<.001, 95% CI [−2.24, −.86]), and 1.39 for sadness (.8 probability of a correct response, 6.62 times lower than for happiness, Z=−5.85, p<.01, 95% CI [−2.56, −1.22]). All affects were significantly harder to recognize than happiness.

For the HC group, the log odds of selecting the correct label to happy stimuli were 3.19, which indicates a probability of .96 of selecting the correct response when they were shown a happy face. The log odds dropped to 1.33 for anger (.79 probability of a correct response, the odds 6.42 times lower than for happiness, Z=−4.35, p<.001, 95% CI [−2.7, −1.02]), 1.21 for disgust (.77 probability of a correct response, 7.2 times lower than for happiness, Z=−4.35, p<.001, 95% CI [−2.9, −1.08]), .32 for fear (.58 probability of a correct response, 17.57 times lower than for happiness, Z=−6.81, p<.001, 95% CI [−3.69, −2.04]), 1.9 for surprise (.87 probability of a correct response, 1.28 times lower than for happiness, Z=−3.04, p<.01, 95% CI [−2.11, −.46]), and 1.86 for sadness (.86 probability of a correct response, 3.77 times lower than for happiness, Z=−3.01, p<.01, 95% CI [−2.19, −.46]). All affects were significantly harder to recognize than happiness.

For the TBI group, the log odds of selecting the correct label to happy stimuli were 3.17, which means that participants had a probability of .96 of selecting the correct response when they were shown a happy face. The log odds dropped to .7 for anger (.66 probability of a correct response, the odds 11.82 times lower than for happiness, Z=−6.32, p<.001, 95% CI [−3.23, −1.70]), .72 for disgust (.67 probability of a correct response, 11.72 times lower than for happiness, Z=−6.04, p<.001, 95% CI [−3.26, −1.66]), −.29 for fear (.42 probability of a correct response, 32.18 times lower than for happiness, Z=−8.62, p<.001, 95% CI [−4.26, −2.68]), 1.52 for surprise (.82 probability of a correct response, 5.19 times lower than for happiness, Z=−4.24, p<.001, 95% CI [−2.41, −.89]), and 1.08 for sadness (.74 probability of a correct response, 8.11 times lower than for happiness, Z=−5.84, p<.01, 95% CI [−2.79, −1.39]). All other affects were significantly harder to recognize than happiness (see Figure 4).

Figure 4. Probability of correct response – Breakdown by emotion and group (Labeling).

Mean probability of giving the correct response to the average item for each affect in the labeling task for the TBI group and for the HC group.

Logistic regression: Matching task

We found a significant group effect (Z=−3.07, p<.01, 95% CI [−.73, −.16]) with individuals with TBI being less likely to produce a correct answer. The log odds of selecting the correct label to the average stimulus for individuals with TBI were .91, while the log odds of selecting the correct label to the average stimulus for HCs were 1.36. In probability terms, the probability of a correct response to the average item was .71 for individuals with TBI, and .8 for HCs. The odds of a correct response were 1.56 times greater for HCs than for individuals with TBI (see Figure 5).

Figure 5. Probability of correct response – Group comparison (Matching).

A. Percent of correct response. Error bars represent the standard deviation. B. Mean probability of giving the correct response to the average item for each affect in the matching task for the TBI group and for the HC group. C. Each data point represents a participant’s probability to respond correctly.

As in the Labeling task, we also found a significant age effect, with the odds of selecting the correct answer for the average item decreasing by 1.32 with a one standard deviation increase in age (Z=−4.31, p<.001, 95% CI [−.41, −.15]). The sex effect was not significant, with the odds of selecting the correct label 1.24 times greater for women (Z=1.67, p<.1, 95% CI [−.04, .47]).

Correlations between affect recognition tasks and neuropsychological tasks

As reported in previous work (Montagne, et al., 2007; Rigon, Turkstra, et al., 2016), regarding demographic variables, performance on Labeling or Matching was not significantly associated with education within the TBI group (both rs<.28, both ps>.05) or within the HC group (both rs<.05, both ps>.05), nor with time post-injury in the TBI group (both rs<.08, both ps>.05). As previously reported in the literature (Kessels, et al., 2014), age was significantly and negatively correlated with both Labeling and Matching performance in both groups: in the TBI group the correlation between age and Matching score was r=−.49, p<.01, and the correlation with Labeling was r=−.47, p<.01; for the HC group, the correlation between age and Matching score was r=−.56, p<.01, and the correlation with Labeling was r=−.43, p<.05 (see Table S1).

We found no significant correlations between any of the BSI scales and Matching or Labeling within the TBI group (all rs<.31, all ps>.05), or within the HC group (all rs<.34, all ps>.05). Moreover, there was no significant correlation between the PANAS scales and Matching or Labeling within the TBI group (all rs<.25, all ps>.05), or within the HC group (all rs>−.33, all ps>.05).

Regarding the Benton Facial Recognition Test, there were no significant correlations with Labeling within the TBI group (r=.37, p<.05) or Matching (r=.25, p>.05). Within the HC group there were no significant correlations between Benton and both Labeling (r=.37, p=.08) and Matching (r=.4, p=.05) (Table S2).

For the neuropsychological tasks, within the TBI group there was a significant positive correlation between Trails A and Labeling (r=.43, p<.01), but the correlation with Matching was not significant (r=.31, p=.06). There was no significant correlation between Labeling or Matching with the Trails A (both rs<.32, both ps>.05). Within the TBI group, there were no significant correlations between Trails B and Matching (r=.38, p<.05) and Labeling (r=.32, p<.05). Within the HC group the Trails A did not significantly correlate with Matching (r=.43, p<.05), nor Labeling (r=.29, p>.05). None of the CVLT subtests were significantly correlated with Matching or Labeling in the TBI group (all rs<.27, all ps>.05) or the HC group (all rs<.48, all ps>.02). The WAIS-PSI was highly and positively associated with Matching (r=.5, p<.01) and Labeling (r=.47, p<.01) within the TBI group, and with Matching within the HC group (r=.69, p<.01). It was not significantly associated with Labeling in the HC group (r=.44, p=.08) (See Table S3).

DISCUSSION

The aim of the current study was to examine whether affect recognition impairment following TBI differed across two separate tasks measuring perceptual and verbal categorization affect recognition abilities. Although previous studies have examined affect recognition impairment following TBI, here we analyzed facial affect recognition skills as a non-unitary construct for the first time, adopting a theoretical framework that treats perceptual and interpretative abilities as partially overlapping but separate. To do so, we administered two tasks to individuals with moderate-to-severe TBI—an affect Matching task, aimed at assessing the perceptual cognitive processes involved in affect recognition, and an affect Labeling task, which tested the ability to form a connection between perceptual features and semantic labels.

We used two different types of analyses to compare performance on the two tasks, as well as to compare performance by the TBI and HC group. The first analysis (McNemar’s test on paired nominal data) revealed that 44% of individuals with TBI were impaired on the Labeling task, while only 10% on the Matching task (although the numbers raised to 52% and 18%, respectively, if a more relaxed threshold was used to determine impairment). For the Labeling task, the numbers align with findings from Babbage’s meta-analysis, which reported that up to 40% of individuals with moderate-to-severe TBI are likely to be impaired at affect recognition, although it should be noted that the tasks used here were created to examine inter-individual differences within healthy populations, and thus harder to successfully complete than the affect recognition tasks often used in the literature (Babbage, et al., 2011; Croker and McDonald, 2005; Palermo, et al., 2013). Regardless of how conservative the threshold, individuals with TBI who showed impairment on the Labeling task were significantly higher in number that those who were impaired at Matching, revealing for the first time that for many individuals, problems in affect recognition arise at the level of semantic categorization of facial affects, while detection of perceptual features appears to be intact. It should be noted that, using both a relaxed and conservative threshold to determine impairment, there were no individuals showing impairment on the Matching task but not on the Labeling task. This supports the theoretical framework adopted by the current study, which assumes that correct verbal categorization cannot occur in the presence of faulty detection of perceptual features carrying affect information, while the opposite (i.e., intact perceptual processing and disrupted semantic processing) can indeed occur (Palermo, et al., 2013).

Although only four (or seven, depending of the criterion used) individuals were classified as impaired using data from the HC sample as normative scores, it should be noted that a more sophisticated statistical analysis found that individuals with TBI underperformed HCs on both the Matching and Labeling task. Indeed, we observed a group effect on both tasks while correcting for age and sex, and while accounting for random by-subject and by-item variation (i.e., including a random effect structure) on both tasks. This, coupled with the other results we have reported here, suggests that although individuals with TBI are more likely to be impaired at semantic categorization, they also underperform healthy individuals at detection of perceptual features carrying salient emotional information.

Our findings have clinical implications. In clinical settings, care should be taken to determine exactly where the disruption lies, and instruments should be devised to assess both cognitive abilities (perception and interpretative) in a reliable way to tailor the treatment of facial affect recognition impairment (as well as its downstream consequences in the psychosocial domain) based on the individual profile of deficit. Moreover, it should be kept in mind that the failure and inconsistency in the response to treatment of facial affect recognition impairment might be due to the fact that the wrong aspect (perceptual vs. interpretative) is being targeted by a specific treatment. As different training strategies might target different facial affect recognition impairments (e.g., gradual learning of identification and discrimination of prototypical facial signs of the six basic affects might benefit people with perceptual impairment but not be necessary for individuals with interpretative impairments, who might benefit more from training of association between holistic processing of faces and emotional labels) it is important to know which efforts should be prioritized during the rehabilitation period. Thus, a theoretical framework that takes into account the different sub-skills necessary to achieve facial affect recognition, as well as they respective relationships, can benefit the field of TBI rehabilitation.

Interestingly, group comparisons revealed that individuals with TBI were not significantly worse than HCs at the Benton Facial Recognition Test, indicating that individuals with TBI do not have agnosia. It should also be noted that the four participants who showed impairment in the Matching task were not the worst performers on the Benton Facial Recognition Test in the TBI group, and all of them performed in the normal range (i.e., neither of the two participants whose score was classified as borderline was impaired on the Matching task, and the scores were respectively 41, 41, 50 and 46). This is in line with the results of the correlation analysis between performance on the Matching task and Benton Facial Recognition Test scores, which revealed a lack of significant correlation within both the TBI and HC group.

Another matter of interest in the investigation of the mechanisms underlying affect recognition impairment post-TBI is the presence of deficits pertaining to affective semantic knowledge, which is the knowledge of the situational determinants of an affect (Turkstra et al., 2017). This would include the contextual knowledge of when it is appropriate to experience a certain affect (e.g., when receiving a present, a person would usually be happy; when being chased by a rabid dog, they would be scared). The affective semantic system is often assessed using emotional inference tasks, in which participants are asked to assign a vignette or a short story (context) to the affect that it is likely to elicit (Croker and McDonald, 2005; Barbra Zupan, Babbage, Neumann, & Willer, 2016; B. Zupan, Neumann, Babbage, & Willer, 2015). Although one study found no impairment within a TBI group using a similar task, more recent studies, using more complex and nuanced vignettes reported significantly lower scores for individuals with TBI, indicating that TBI might in fact lead to impairment in the affective semantic system (Croker and McDonald, 2005; B. Zupan, et al., 2015). Although in the current study this system was not assessed, future work should attempt to establish the relationship between impairment in the labeling affect recognition tasks and affective semantic knowledge.

It should be noted that performance of the two affect recognition tasks was highly correlated, as previously reported for individuals with TBI or shorter tasks by Ietswaart and colleagues, and for healthy individuals by Palermo and colleagues, who used the tasks employed here (Ietswaart, et al., 2008; Palermo, et al., 2013). Although a previous study on TBI found a lack of correlation between performance on a matching and labeling task, this is likely to be due to the small sample size, as well as to the ceiling effects in each test reported by the authors (Croker and McDonald, 2005). Moreover, the only neuropsychological measure that showed high levels of correlation with the affect recognition tasks was processing speed: although this is likely to be a product of the fact that each task had a time limit for each trial (which was used to make the test more naturalistic, i.e., out-of-the-lab social interaction affects are never permanently on a face), it should also be noted that previous studies that gave participants an unlimited amount of time to respond have reported an association between processing speed and affect recognition, suggesting potential common underlying mechanisms (Rigon, Turkstra, et al., 2016; Yim, Babbage, Zupan, Neumann, & Willer, 2013).

A novel finding here was the lack of a group-by-affect interaction for the labeling task. This indicated that participants with TBI were significantly underperforming HCs in a way that did not significantly vary across affect. Although several other tasks, both by our group and others, have previously reported group-by-affect interactions (Rigon, Turkstra, et al., 2016; Rosenberg, et al., 2015; Rosenberg, et al., 2014), it is possible that the different finding here is due to 1) the fact that the tasks used here included a higher number of trials for all types affect, thus providing a more complete assessment and 2) the use of mixed-effect modeling, which allows to account to the variance provided both by items and participants. Especially considering the fact that TBI research often works with relatively small samples, future research on affect recognition might benefit from the use of more complex statistical models, as well as larger sample sizes. The results of the current study reveal that when subject and item intercepts, and well as affect by subject slopes are taken into consideration, individuals with TBI are not more impaired at recognizing one specific affect.

Certain limitations of the current study should be noted. Firstly, both the age range and the time since injury range were large in our study, which increased variability in the statistical analyses. Although we attempted to account for this by performing additional analyses and adding age as a covariate, future work might benefit from limiting age range and chronicity range within their TBI samples.

Secondly, it is important to note that the Labeling and Matching tasks, while differing on the cognitive abilities of interest for the current study (perceptual vs. interpretative), are also structured in different ways: participants have to choose between more options in the Labeling task than in the Matching task. It should be noted, however, that the Matching task, while requiring the participant to choose between only 3 options (vs. 6), was not significantly easier, overall, than the Labeling task: indeed, on average, healthy individuals made more errors in the former. Moreover, were the Matching task structured to perfectly mirror the Labeling task (i.e., one face and six options to choose from), it would create a much higher working memory load, as scanning six different faces to find a match would be significantly harder than reading six labels and choosing one. Thus, the current structure of the task is optimal considering the demands it poses on the participant.

Thirdly, although we estimated performance validity using the Trails A, we did not directly measure it, and thus cannot comment with certainly about degree of engagement in the tasks for the HC group. Moreover, as is common in TBI research, injury-related information were not present uniformly for all participants—the issue was obviated by using the Mayo Classification System.

Lastly, the Benton Facial Recognition task, while a normed and valid measure frequently used to assess acquired prosopagnosia, has also been reported to be less than effective when discriminating facial recognition impairment in certain populations (e.g., developmental prosopagnosia) (Duchaine and Nakayama, 2004). While our data suggests that individuals with TBI and HCs performed similarly on the test, future work should supplement this task with other facial recognition assessments to gain a more well-rounded perspective on the role played by early perceptual processes in emotion recognition following TBI.

Future work should also examine how impairment on different aspects of affect recognition maps onto real word behaviors, for instance by looking at correlations between performance on matching and labeling tasks and instruments that assess different psychosocial constructs, such as the LaTrobe Communication Questionnaire (cognitive and social communication) and the Vineland Adaptive Behavior Scales (overall psychosocial well-being) (Douglas, Bracy, & Snow, 2007; Sparrow and Cicchetti, 1985). It is possible that more perceptual deficits lead to problems in specific areas of social communication (e.g., the ability to detect feedback based on others’ reactions), while verbal categorization deficits map onto different types of psychosocial problems (e.g., social communication). Moreover, the question of why certain individuals with TBI have more pervasive deficits that include perceptual problems, while others seem to be only impaired at verbal categorization, remains. As there is evidence that these two cognitive abilities map onto different neural networks, it is possible that different profiles of impairment are originated by different patterns of disruption in functional and structural connectivity, and this knowledge would be critical in advancing our understanding of the neurobiology of social cognition, and of facial affect recognition in particular, and for the development of interventions that move beyond direct behavioral training (Barbey et al., 2015; Hariri, et al., 2000).

Conclusions

In the current study, we have shown that although individuals with moderate-to-severe TBI are more likely to show impairment in affect recognition tasks that require verbal categorization, they also underperform healthy individuals on more basic detection of facial features carrying salient emotional information. These findings provide further evidence supporting distinct roles of perceptual and interpretative skills in facial affect recognition, as well as the theoretical framework that provides the necessity of intact perceptual skills to achieve successful verbal categorization. Moreover, this suggests that clinicians targeting affect recognition deficits in individuals with TBI in treatment should assess and focus on both sets of cognitive skills, and take care to examine the specific profile of impairment for each individual.