Functional patterns of neural activation during vocal emotion recognition in youth with and without refractory epilepsy

Highlights

• •Neural activation to vocal emotion was assessed in youth with and without epilepsy.
• •Youth with epilepsy were less able to infer emotional intent in vocal expressions.
• •Patterns of activation in regions of the mentalizing network differentiated groups.

Abstract

Epilepsy has been associated with deficits in the social cognitive ability to decode others’ nonverbal cues to infer their emotional intent (emotion recognition). Studies have begun to identify potential neural correlates of these deficits, but have focused primarily on one type of nonverbal cue (facial expressions) to the detriment of other crucial social signals that inform the tenor of social interactions (e.g., tone of voice). Less is known about how individuals with epilepsy process these forms of social stimuli, with a particular gap in knowledge about representation of vocal cues in the developing brain. The current study compared vocal emotion recognition skills and functional patterns of neural activation to emotional voices in youth with and without refractory focal epilepsy. We made novel use of inter-subject pattern analysis to determine brain areas in which activation to emotional voices was predictive of epilepsy status. Results indicated that youth with epilepsy were comparatively less able to infer emotional intent in vocal expressions than their typically developing peers. Activation to vocal emotional expressions in regions of the mentalizing and/or default mode network (e.g., right temporo-parietal junction, right hippocampus, right medial prefrontal cortex, among others) differentiated youth with and without epilepsy. These results are consistent with emerging evidence that pediatric epilepsy is associated with altered function in neural networks subserving social cognitive abilities. Our results contribute to ongoing efforts to understand the neural markers of social cognitive deficits in pediatric epilepsy, in order to better tailor and funnel interventions to this group of youth at risk for poor social outcomes.

1. Introduction

Epilepsy has been strongly associated with social cognitive deficits (see reviews by Bora and Meletti, 2016, Edwards et al., 2017, Mirabel et al., 2020, Monti and Meletti, 2015, Stewart et al., 2016), which can manifest as difficulty with mentalizing (understanding others’ beliefs or intentions) or emotion recognition skills (ER; understanding others’ putative emotional states based on nonverbal expressions). Most of the research examining this clinical phenotype has focused on individuals’ ability to label emotions in facial expressions (e.g., Golouboff et al., 2008, Meletti et al., 2009, Sedda et al., 2013) and on underlying neural representations of these stimuli (e.g., Batut et al., 2006, Benuzzi et al., 2004, Labudda et al., 2014, Szaflarski et al., 2014, Vuilleumier et al., 2004). This literature has identified consistent deficits in emotion recognition and atypical patterns of neural activation and connectivity in response to emotional faces in adults and children with epilepsy (Broicher et al., 2012a, Meletti et al., 2009, Morningstar et al., 2021, Morningstar et al., 2020a). However, there is also evidence that individuals with epilepsy also process other types of nonverbal cues differently from healthy controls. Notably, a handful of studies have reported difficulties in the recognition of affective prosody (i.e., emotional tones of voice) in individuals with epilepsy (Bonora et al., 2011, Broicher et al., 2012b, Cohen et al., 1990, Hennion et al., 2015, Meletti and Bonora, 2013). However, there is comparatively little research on potential neural underpinnings of this social cognitive deficit—with a particular gap in knowledge about how vocal affect is processed by the developing brain. Expanding our understanding of the neural correlates of social cognition in youth with epilepsy would be beneficial for assessment (e.g., of patients’ risk for social difficulties) and treatment planning (e.g., enrollment in social skills training programs, counselling surrounding expected outcomes following surgical resection of relevant brain areas; Kirsch, 2006). To that end, the current study leveraged multivoxel pattern analysis to compare brain response to vocal emotional expressions in youth with and without refractory focal epilepsy.

1.1. Vocal emotion recognition in epilepsy

There is growing understanding that non-facial nonverbal cues, such as a speaker’s tone of voice (separate from the verbal content of their speech), provide important information about others’ social attitudes and emotional states (Johnstone and Scherer, 2000, Mitchell and Ross, 2013). However, compared to the interpretation of facial expressions, identifying emotional intent based on vocal prosody is more difficult (App et al., 2011, Scherer, 2003) and matures at a more protracted rate throughout childhood and adolescence (Morningstar et al., 2020b, Morningstar et al., 2018a). This social cognitive skill has been studied less extensively in the context of epilepsy than facial ER, with reviews of social cognition in epilepsy excluding vocal ER studies due to their small number (Bora and Meletti, 2016, Ives-Deliperi and Jokeit, 2019). Nonetheless, a handful of studies report that epilepsy is associated with deficits in vocal ER (see review by Monti and Meletti, 2015) across different emotion types (Bonora et al., 2011, Broicher et al., 2012b, Hennion et al., 2015, Meletti and Bonora, 2013). These findings suggest that epilepsy may be associated with broad forms of social cognitive deficits across multiple nonverbal modalities (Hixson and Kirsch, 2009).

Notably, adults with childhood-onset epilepsy often show the greatest impairment in social cognitive functions (see review by Besag and Vasey, 2019), suggesting that early insults to the brain (as a result of seizures or other associated sequelae of epilepsy) may be particularly damaging to abilities like mentalizing and ER. Although there is evidence that youth with epilepsy already show deficits in facial ER (Golouboff et al., 2008, Laurent et al., 2014, Morningstar et al., 2020c), only two studies to our knowledge have investigated vocal ER in children with epilepsy. One study found that 5- to 19-year-old children and adolescents with temporal lobe epilepsy (TLE) showed no impairments in the recognition of emotional prosody compared to typically developing youth (Laurent et al., 2014). The other study found that 6- to 11-year-old children with right-sided TLE scored lower on a vocal ER task than typically developing children (but did not differ from children with left-sided TLE; Cohen et al., 1990). As such, results are mixed. Moreover, neither of these studies investigated potential neural underpinnings of vocal emotion recognition in youth with or without epilepsy. Understanding the neural correlates of these tasks, and how they may be disrupted in epilepsy, can help identify potential mechanisms through which epilepsy (and associated syndromes) may impact social cognition. Moreover, given the importance of early intervention, understanding risk factors for social deficits in youth with epilepsy may greatly contribute to improving quality of life in this at-risk population (Besag and Vasey, 2019, Mirabel et al., 2020, Ronen et al., 2010, Steiger and Jokeit, 2017).

1.2. Neural correlates of vocal emotion recognition

Models of the neural representation of vocal emotional information (Schirmer and Kotz, 2006, Wildgruber et al., 2006) typically implicate regions of the temporal lobe (e.g., auditory cortex, temporal voice area, superior temporal gyrus; Belin et al., 2000, Ethofer et al., 2012), limbic and subcortical structures (e.g., amygdala, insula; Ethofer et al., 2009; Y. Zhang et al., 2019), and areas of the frontal cortex (e.g., dorsal medial prefrontal cortex, inferior frontal gyrus [IFG], orbitofrontal cortex; Adolphs et al., 2002, Alba-Ferrara et al., 2011, Ethofer et al., 2006, Wildgruber et al., 2002). Given the heavy involvement of the temporal lobe in the perception of vocal affect, it is perhaps not surprising that children and adults with mesial temporal lobe epilepsy (TLE) seem to fare worse on vocal ER tasks than individuals with extra-temporal foci or healthy controls (Broicher et al., 2012b, Cohen et al., 1990). Indeed, some studies have reported that lesions to the amygdala impair the recognition of fearful prosody (Brierley et al., 2004, Dellacherie et al., 2011, Scott et al., 1997), although the evidence is mixed (c.f., Adolphs and Tranel, 1999, Adolphs et al., 2001, Fowler et al., 2006).

In addition, neural networks supporting social cognition more broadly are likely to contribute to the recognition of vocal emotion in ER tasks. In typically developing youth, greater vocal ER ability was associated with heightened connectivity between frontal areas associated with language processing and the right TPJ (Morningstar et al., 2019)—a major node in social cognitive networks (Blakemore and Mills, 2014, Redcay, 2008). Indeed, networks such as the “mentalizing network” (including the temporo-parietal junction [TPJ], the posterior superior temporal sulcus [pSTS], the medial prefrontal cortex [mPFC], and the anterior temporal cortex; Mills et al., 2014) and the default mode network (DMN; including similar regions of the lateral parietal cortex [TPJ and pSTS] and mPFC, along with the posterior cingulate cortex [PCC] and the entorhinal cortex/hippocampus) are increasingly recognized as being central to the processing of socio-emotional stimuli (Adolphs, 2002, Adolphs, 2009, Amft et al., 2015, Dunbar, 1998, Kennedy and Adolphs, 2012, Li et al., 2014, Rushworth et al., 2013). Interestingly, the integrity of these networks can be impaired in individuals with epilepsy: although most studies have investigated the structure of these networks at rest (Ibrahim et al., 2014, Mankinen et al., 2012, Widjaja et al., 2013; Z. Zhang et al., 2010), there is evidence that connections between specific nodes of these networks may also be altered during social cognitive tasks (Fruhholz et al., 2015, Morningstar et al., 2021). For example, youth with epilepsy showed decreased connectivity between the pSTS and mPFC, but increased connectivity within the temporal lobe, than their typically developing peers when completing a facial ER task (Morningstar et al., 2021). Such findings suggest that epilepsy is associated with differential engagement of social cognitive networks when processing facial emotional stimuli. However, the literature on the neural underpinnings of prosody processing in individuals with epilepsy has primarily centered on lesion studies (review by Alba-Ferrara et al., 2018), with little understanding of functional patterns of neural activation during vocal ER tasks.

1.3. Goals and hypotheses

The current study investigated differences in task performance and in patterns of neural activation during a vocal ER task, in 8- to 21-year-old youth with refractory focal epilepsy and typically developing youth. Participants completed a vocal ER task while undergoing functional magnetic resonance imaging (fMRI). We examined group differences in youth’s vocal ER ability and applied multivoxel pattern analysis (MVPA) to examine areas of the brain in which patterns of neural activation differentiated youth with and without epilepsy. Extending beyond traditional univariate analyses of neural activation, MVPA can help identify regions of the brain in which neural signals are predictive of epilepsy status. Although MVPA is widely used to predict what stimulus was presented to participants, we applied an inter-subject pattern version of this technique (inter-subject pattern analysis; Wang et al., 2020) to predict group membership (epilepsy vs. no epilepsy) based on neural activation in response to a given stimulus. In other words, our analysis sought to determine, based on neural response to vocal emotional prosody, whether a given participant was diagnosed with epilepsy or not. This approach aligns itself with clinical goals of predicting potential deficits based on neuropsychological assessments. Moreover, compared to univariate activation-based analyses, MVPA is more sensitive to subtle, locally distributed effects (Haynes and Rees, 2006, Jimura and Poldrack, 2012, Kriegeskorte et al., 2006)—such as those associated with the detection of emotional information in the voice (Ethofer et al., 2009b, Kotz et al., 2013).

The paucity of literature on neural response to vocal emotions in individuals with epilepsy makes it difficult to articulate precise hypotheses. Nonetheless, based on functional studies with typically developing youth and lesion studies with patients with epilepsy (Adolphs et al., 2002, Alba-Ferrara et al., 2018, Dellacherie et al., 2011, Fruhholz et al., 2015, Milesi et al., 2014, Morningstar et al., 2019a, Sanz-Martín et al., 2006, Scott et al., 1997), we expected that activation patterns in regions involved in vocal emotion processing (e.g., superior temporal gyrus, mesial temporal structures like the amygdala, IFG) and/or mentalizing more broadly (e.g., right TPJ, mPFC, regions of the DMN) would discriminate youth with and without epilepsy.

2. Methods

2.1. Participants

Participants included 26 youth with refractory focal epilepsy (FE) and 42 typically developing (TD) youth. Youth with epilepsy were recruited from an epilepsy monitoring unit at a large urban children’s hospital (United States), while typically developing youth were recruited via digital flyers distributed to hospital staff. Written informed consent and/or assent was obtained prior to participation and all study procedures were approved by the local Institutional Review Board.

All participants were proficient in English, between the ages of 8–21, had typical or corrected vision and hearing, and were able to participate in the planned fMRI protocol. All participants in the epilepsy group had a primary clinical diagnosis of focal epilepsy. Of the original 68 participants, 4 were excluded for not completing the vocal ER task (2 TD, 2 FE youth) and 3 (FE youth) were excluded due to having previous surgical resections. No participant had a diagnosis of autism spectrum disorder. The final sample was composed of 21 youth with epilepsy (14 male, 7 female) and 40 typically developing youth (14 male, 26 female). Youth with and without epilepsy did not differ in age, t(59) = 0.12, p = .90. Groups differed in sex distribution, χ²(1, N = 61) = 5.56, p = .02, with a greater proportion of male participants in the FE than TD group.

IQ was assessed using subscales of the Wechsler Intelligence Scale for Children (WISC; Wechsler, 2014)/Wechsler Adult Intelligence Scale (WAIS; Wechsler, 2008). TD youth completed the Matrix Reasoning and Vocabulary subtests during their visit; the equivalent subscale scores and an estimate of full-scale IQ (FSIQ) were pulled from records of recent neurocognitive testing for FE youth. Youth with epilepsy did not differ from typically-developing youth in Matrix Reasoning scores, t(59) = 1.80, p = .08, but scored lower on the Vocabulary subscale, t(59) = 4.96, p < .001. No participant was diagnosed with intellectual disability.

In addition, the following information about participants with epilepsy was obtained from their medical charts by two independent coders (with any disagreements resolved by the first author): type of epilepsy (temporal, frontal, fronto-temporal, or other), the lateralization of seizure foci (left, right, bilateral, or unknown), the presence of mesial temporal sclerosis (MTS; determined by radiologist report), age of seizure onset, duration of illness (computed by subtracting age of onset from age at scan, in years), and the number of antiseizure medications (ASMs) prescribed at the time of the study (Table 1; see Supplemental Table 1 for more information about types of prescribed ASMs). Neither type nor lateralization of epilepsy was associated with age, sex, presence of MTS, age of seizure onset, duration, or number of ASMs (all ps > 0.05). Because of the similarities across participants, youth with epilepsy were considered as one group for all primary analyses.

Table 1.

Sample characteristics.

	Typically developing youth	Youth with epilepsy
Age (in years)	14.07 (3.48)	14.19 (3.60)
Sex	14 male (35%)	14 male (67%)
Type of epilepsy	–	4 frontal 13 temporal 2 fronto-temporal 2 other (occipital, unknown)
Lateralization of seizure foci	–	9 left 10 right 2 bilateral
Age of seizure onset (in years)	–	7.36 (4.52)
Duration of illness (in years)	–	7.27 (3.79)
Presence of MTS	–	6 yes 15 no
Number of ASMs prescribed at time of scan	–	2.81 (1.29)

Note. For age, age of seizure onset, duration of illness, and number of antiseizure medications (ASMs) prescribed, values represent the mean (standard deviation). Estimates do not include youth with resections who were excluded prior to analyses. MTS = mesial temporal sclerosis. Two participants with FE had cortical dysplasias outside of the temporal lobe (1 frontal-parietal; 1 parietal-occipital); no other participant with FE had noted dysplasias or brain injuries associated with epilepsy (e.g., no history of traumatic brain injury, hydrocephalus, or intracerebral hemorrhage).

2.2. Neuroimaging task

Youth participated in a forced-choice vocal ER task while undergoing functional MRI. Participants heard auditory recordings of other teenagers expressing one of five emotional tones of voice (anger, fear, happiness, sadness, and neutral) over pneumatic noise-cancelling earbuds. They were then asked to indicate which emotion the speaker was conveying via hand-held response boxes, choosing from the above five labels.

We opted to use stimuli that contained socially oriented linguistic content (rather than nonsense syllables, or rather than using non-linguistic vocalizations) to probe youth’s interpretation of socially relevant affective prosody (i.e., vocal emotional information embedded in speech content). The stimuli (total of 75 recordings) were produced by 3 teenage community-based actors (2 females) speaking the same 5 sentences (e.g., “I can’t believe you just did that”, “Why did you do that?”) in each of the above 5 tones of voice. These recordings were selected from a larger set of stimuli (Morningstar et al., 2017) based on judges’ ratings of recognizability and authenticity (Morningstar et al., 2018). On average, stimulus duration was 1.34 s (ranging from 0.89 to 2.03 s).

The task was split into three runs of 25 recordings each. Each run had a balanced, pseudo-randomized number of stimuli per emotion type. Stimuli were presented in an event-related design, with a jittered inter-trial interval of between 1 and 8 s (mean 4.5 s). Each event consisted of the stimulus presentation followed by a 5-second response period. Throughout the task, participants were looking through a mirror affixed to the head coil at a monitor at the head of the magnet bore. During stimulus presentation and during the inter-trial interval, participants viewed a fixation cross. When participants were making their response, a pictogram of the response boxes with labels above the associated buttons was shown.

2.3. Image acquisition and processing

Due to hardware updates during the study, MRI data were collected on two Siemens 3-Tesla scanners running identical software, using standard 32– and 64-channel head coil arrays. Both scanners had comparable acquisition protocols. The fMRI protocol included three-plane localizer scout images and anisotropic 3D T1-weighted anatomical scan covering the whole brain (MPRAGE), with a 1-millimeter isotropic voxel size. Imaging parameters for MPRAGE were: 176 sagittal slices, repetition time (TR) = 2200–2300 ms, echo time (TE) = 2.45–2.98 ms, and field of view (FOV) = 248–256 mm. Functional MRI data were acquired with echo planar imaging (EPI) acquisitions, with voxel dimensions of 2.5 × 2.5 × 3.5–4 mm, TR = 1500 ms, TE = 30–43 ms, FOV = 240 mm, and with the phase-encoding axis oriented in the anterior-posterior direction. EPI images were acquired using simultaneous multi-slice sequences.

The first six TRs (9 s) at the beginning of each run were discarded to allow saturation of MR signal. EPI images were preprocessed in AFNI, version 18.0.11 (Cox, 1996). Functional images were aligned to the first volume, realigned to the AC/PC plane, co-registered to the T1 image, and non-linearly normalized to the Talairach template. Within each functional run, voxel-wise signal was scaled to a mean value of 100 and signal values above 200 were winsorized to 200. TRs with motion outliers above 1 mm or signal outliers were regressed out of the final model.

2.4. Statistical analysis

2.4.1. Emotion recognition ability

Participants’ performance on the task was indexed using an estimate of sensitivity (P_r; e.g., Pollak et al., 2000), which combines participants’ hit rates (HR; correct responses) and false alarms (FA; incorrect responses; Pollak et al., 2000) for each emotion category. Similar to d’, P_r (i.e., HR – FA) is more appropriate when participants’ recognition is low (Snodgrass and Corwin, 1988), as is often the case in emotion recognition tasks with affective prosody embedded in speech (e.g., Morningstar et al., 2018b, Morningstar et al., 2019a). P_r values range from −1 to 1, where positive values represent more correct responses than incorrect responses (i.e., HR > FA) and negative values represent more incorrect responses than correct responses (i.e., FA > HR). Responses made within 150 ms of the start of the rating period were censored from analyses because of physiological implausibility. For each participant, a value of P_r was estimated for each emotion category. Behavioural responses were not recorded for two participants (1 FE, 1 TD) due to technical issues1.

A general linear model was computed to examine the effect of Group (between-subjects variable, 2 levels: typically developing youth, youth with epilepsy) and Emotion type (within-subjects variable, 5 levels: happiness, fear, anger, sadness, neutral) on P_r. Given known increases in vocal ER skills with age across childhood and adolescence (review in Morningstar et al., 2018), age in years (between-subjects variable, continuous) was also included in the model as a variable of interest. Sex was included as a covariate given group differences in sex distribution. Greenhouse-Geisser corrections were applied when indicated by Mauchly's test of sphericity.

2.4.2. Whole brain searchlight classification

Multi-voxel pattern analysis (MVPA) was used to discriminate between youth with and without epilepsy based on patterns of activation during the vocal ER task. Analyses excluded 4 (TD) participants due to an inconsistent amount of data between runs (i.e., shortened runs due to task protocol failures) and 2 (FE) participants due to excessive motion (i.e., over 40% of volumes censored during events—48% and 51%, respectively).

MVPA was conducted in several steps (Fig. 1). First, each participant’s time-series data were converted to a vector (one vector per run per participant), in which events were labelled by stimulus emotion. The hemodynamic response function was fit to that labelled time-series. Then, the first-order Legendre polynomial was removed from the BOLD signal in each voxel within each participant’s runs to remove baseline and linear trends. Each participant’s data were also Z-scored to assess relative variations in BOLD and dampen the influence of extreme signal values. Second, the data were divided into separate vectors for each emotion and stacked by participant. Each of the resulting stacked vectors was labelled with that individual’s epilepsy status (TD vs. FE). Using a leave-one participant-out (LOO) cross-validation approach, vectors were recursively split into training and test sets, where a test set consisted of all runs for a single participant. A linear support vector machine (SVM) classifier was implemented in Python 2.7 using PyMVPA’s default parameters (Hanke et al., 2009). The SVM was trained on all data in the training set to predict the epilepsy status of the left-out participant, using a roaming searchlight at each voxel of the Talairach template (radius = 6 mm, Kriegeskorte et al., 2006). To ensure results were independent of the characteristics of any one left-out participant, each participant was used as the test data exactly once. Classifier performance was operationalized at each voxel as the average accuracy in prediction across all test sets (i.e., across all participants).

Fig. 1.

Steps in MVPA analysis. Note. A) Participants’ time-series data were converted to a vector (one vector per run). Events were labelled by stimulus emotion (hypothetical events represented as coloured bars; orange = anger, yellow = fear, light green = happiness, light blue = sadness, dark grey = neutral). The hemodynamic response function was fit to each labelled time-series. The first-order Legendre polynomial was removed and data were Z-scored in this step. B) Data were divided into separate vectors for each emotion and stacked by participant. Participants’ stacked vectors were labelled with their epilepsy status (purple = typically developing [TD] youth; green = youth with focal epilepsy [FE]. C) Using a leave-one-out cross-validation approach, emotion-specific vectors were recursively split into training and test sets (where a test set consisted of all runs for a single participant). Using a roaming searchlight, a support vector machine was trained at each voxel. Classifier performance was calculated at each voxel as the accuracy with which it labelled each test set. This procedure resulted in an average accuracy brain map for each emotion. The process in C was repeated for each emotion category. D) Accuracy maps for each emotion were averaged to create a grand average map. This brain map represented the classifier’s performance across all emotions in the task. Please see 2.4.2 for additional details. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

The searchlight was performed for each emotion separately, resulting in an accuracy map for each emotion per participant. These maps were then averaged across participants to create an average accuracy map for each emotion. Finally, accuracy maps for each emotion were then averaged to generate a grand average map that represented the classifier’s performance across all emotions. Given imbalance in group sizes, the theoretical chance level for performance was 65.45% (rather than 50%). We applied this threshold to the grand average map and identified clusters of contiguous voxels (NN = 1, k = 20) in which classifier performance (i.e., accuracy in correctly predicting the epilepsy status of all test sets) exceeded chance.

2.4.2.1. Permutation testing of classification performance

While it is common practice to perform voxel-wise t-tests on classifiers’ accuracy score compared to chance level, this approach violates several assumptions of the t-statistic; instead, permutation tests are recommended to assess statistical significance (Stelzer et al., 2013). Permutation testing is a non-parametric bootstrapping method where the target labels are shuffled to create a null distribution. The empirical distribution is then tested against the null distribution to determine significance. We computed the average classifier performance by voxel within each of the clusters identified above. We then performed significance testing of this average performance against the theoretical chance level using permutation tests (Nichols and Holmes, 2002) for each emotion (1000 repetitions). To assess significance of the grand average map (i.e., significance across all emotions), the null distributions for each emotion were averaged for each cluster (see Fig. 3, second column).

Fig. 3.

Clusters of interest alongside permutation test results and emotion-specific patterns of activation Note. R = right. TPJ = temporo-parietal junction; PCC = posterior cingulate cortex; mPFC = medial prefrontal cortex. The first column depicts the 6 clusters in which patterns of activation to emotional voices predicted epilepsy status with > 65.45% classifier accuracy (i.e., above theoretical chance level) and a cluster correction threshold of k = 20 (NN = 1; see Table 3 for description of each cluster). Brain images are rendered in the Talairach-Tournoux template space. The second column depicts histograms of permutation test results for each cluster, with classifier accuracy on the x-axis and frequency on the y-axis. The orange vertical line represents the classifier’s accuracy averaged across all voxels in a respective cluster (R-TPJ = 0.6597, R mPFC = 0.6607, R hippocampus = 0.6583, PCC = 0.6587, R fusiform = 0.6597, Cerebellum = 0.6603); the blue curve represents the distribution of accuracy for voxels in a given cluster across 1000 permutation tests. In all clusters, the observed classifier accuracy is significantly higher than the distribution obtained from random permutations (all ps < 0.001). The third column contains radar plots illustrating emotion-specific activations in each cluster, for youth with (green) and without (purple) epilepsy. TD youth = typically developing youth; FE youth = youth with focal epilepsy. Values on each axis represent blood-oxygen-level-dependent signal (BOLD) during emotional stimuli (relative to an implicit baseline), averaged across voxels in the cluster. Refer to Supplemental Materials for more information on emotion-specific results. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

3. Results

3.1. Vocal emotion recognition ability

P_r values for each emotion type and group are listed in Table 2. A main effect of Group on P_r, F(1, 54) = 5.64, p = .02, η² = 0.10, indicated that youth with FE (M = 0.20, SD = 0.02) were less accurate on the vocal ER task than were TD youth (M = 0.25, SD = 0.02). There was also a significant effect of Age, F(1, 54) = 17.31, p < .001, η² = 0.24: parameter estimates indicate that age was positively associated with P_r. Lastly, there was an interaction of Group and Age, F(1, 54) = 4.15, p = .046, η² = 0.07, such that group differences in P_r were most pronounced in younger participants than in older participants (Fig. 2). No other effects were significant (ps > 0.07).2

Table 2.

Task performance (P_r) for each emotion by group.

	Typically developing youth	Youth with epilepsy
Anger	0.47 (0.19)	0.39 (0.27)
Fear	0.13 (0.11)	0.11 (0.16)
Happiness	0.13 (0.13)	0.15 (0.13)
Sadness	0.31 (0.18)	0.17 (0.20)
Neutral	0.23 (0.16)	0.17 (0.22)

Note. P_r = sensitivity (estimate of performance on the vocal emotion recognition task). Values represent means (standard deviations) for all participants.

Fig. 2.

Task performance (P_r) across age. Note. The figure depicts the Group × Age interaction on P_r (sensitivity index). TD youth = typically developing; FE youth = youth with focal epilepsy. Simple-slopes tests reveal that the slope of P_r across age is significantly different from 0 for both groups (TD youth t = 2.09, p = .04; FE youth t = 4.01, p < .001); * p < .05, *** p < .001.

Given group differences in Vocabulary scores on measures of cognitive ability (see 2.1) and the known influence of intelligence scores on ER task performance (e.g., Schlegel et al., 2020), we verified whether participants’ scores on the Vocabulary and Matrix Reasoning subscales of the WISC/WAIS were related to their accuracy on the vocal ER task. Correlation analyses revealed that both Matrix Reasoning (r = 0.31, p = .02) and Vocabulary (r = 0.44, p < .001) scores were associated with task performance, across both groups. We then performed a follow-up analysis to investigate whether group differences in ER accuracy were due to participants’ subscale scores. Using PROCESS 4.0 (Hayes, 2016) in SPSS, we tested whether these subscale scores were significant moderators of the relationship between Group (FE vs. TD youth) and P_r, controlling for age and sex. Results indicated that groups differed in ER performance, B = -0.26, p = .03, but that no other effects were significant (all ps > 0.05). Subscale scores neither predicted nor moderated group differences in ER accuracy—suggesting that FE youth’s relative deficits in vocal ER are likely independent from any difference in general cognitive abilities.

3.2. Brain regions predicting epilepsy status

The classifier’s accuracy in predicting participants’ epilepsy status was significantly above chance levels in six clusters (Table 3; Fig. 3). Clusters were located in a) the right TPJ, b) the right mPFC, c) the right hippocampus, d) the posterior cingulate cortex, at midline, e) the right fusiform, and f) the cerebellum (inferior semi-lunar lobule), at midline (Fig. 3, column 1). Permutation tests indicated that classifier performance in these clusters was significantly above theoretical chance levels (Fig. 3, column 2). The classifier’s accuracy across emotions in these clusters was 0.66 (with precision = 0.65, recall = 0.66, and F1 = 0.64). Chi-square tests indicated that the classifier accuracy did not differ for TD vs. FE youth, χ²(1, N = 55) = 0.11, p = .74, suggesting that results are not due to bias in the model (i.e., the classifier performs just as well with one group as with the other).

Table 3.

Brain regions in which activation to vocal stimuli was predictive of epilepsy status.

Region	k	x	y	z	Brodmann area
R TPJ	74	46	−59	29	39
R mPFC	23	26	54	−6	10
R hippocampus	26	36	−19	−11	N/A
PCC	43	−1	−46	24	23
R fusiform	61	49	−59	−19	40
Cerebellum	89	6	−61	−44	N/A

Note. Clusters listed represent areas in which activation to stimuli in the vocal emotion recognition task were predictive of epilepsy status in the multivoxel pattern analysis, with average classifier accuracy across voxels within each cluster > 65.45% (i.e., above theoretical chance level) and a cluster correction threshold of k = 20 (NN = 1). Regions are labelled based on AFNI’s TT_Daemon and DD_Desai_MPM atlases. R = right. TPJ = temporo-parietal junction; PCC = posterior cingulate cortex; mPFC = medial prefrontal cortex. k = cluster size in voxels. xyz coordinates represent the cluster’s peak (in accuracy), in Talairach-Tournoux space.

An inspection of the patterns of activation for each emotion type suggests that emotion-specific responses within each cluster varied by group (Fig. 3, column 3). For instance, FE youth showed reduced activation to fearful voices in all clusters, compared to TD youth; in contrast, activation to angry voices was markedly higher than that of TD youth across most identified clusters. Additional information about emotion-specific results can be found in Supplemental Materials.

3.3. Relationship between classifier performance and clinical variables

Within the group of participants with epilepsy, we investigated whether certain illness-related variables were associated with greater classifier accuracy within the clusters identified above. In other words, we examined whether the classifier performed better with some participants than others within the epilepsy group—and whether variables such as age of illness onset or seizure lateralization could explain this. To reduce the number of statistical tests required to answer this question, we computed classifier performance as a binary (0 or 1) between-subjects variable across clusters. We first obtained the average cross-validation accuracy for all voxels within a given cluster; these values were then averaged across all emotion types and all clusters for each participant, and rounded to the closest integer. Thus, a participant for whom cross-validation accuracy was on average 0.93 across voxels in all clusters and emotion types would be deemed to have been ‘accurately’ identified (rounded up to 1); a participant for whom average cross-validation accuracy was 0.10 would be deemed to be ‘inaccurately’ identified (rounded down to 0) by the classifier in these clusters of interest.

We then examined whether participants with epilepsy for whom the classifier was deemed accurate (n = 13) vs. inaccurate (n = 6) across all clusters differed in the type of epilepsy (temporal, frontal, fronto-temporal, or other), the lateralization of seizure focus (left, right, or bilateral), the presence of MTS (yes or no), the age at seizure onset (continuous variable, in years), or the number of antiepileptic drugs (AEDs; continuous variable) prescribed at the time of study. We also examined whether the two subsamples of participants with epilepsy differed in sex (female or male), age at scan (continuous variable, in years), and FSIQ (continuous variable). Chi-square tests were employed for categorical variables and independent samples t-tests were used for continuous variables.

Participants with epilepsy for whom the classifier was accurate vs. inaccurate did not differ in the type of epilepsy they were diagnosed with (p = .58), the lateralization of seizure focus (p = .78), the presence of MTS (p = .13), sex (p = .83), FSIQ (p = .07), or number of AEDs (p = .43). The two subsamples of participants differed in age at scan, t(17) = 2.25, p = .04, such that those for whom the classifier accurately predicted their epilepsy status were younger (M = 14.02, SD = 3.01) than those for whom the classifier was inaccurate (M = 17.46, SD = 3.27). In addition, there was a difference between subsamples of participants in age at seizure onset, t(17) = 2.33, p = .03, with those for whom the classifier was accurate having an earlier age of seizure onset (M = 6.18, SD = 4.01) than those for whom the classifier was inaccurate (M = 10.83, SD = 4.12). Thus, the classifier was more accurate at identifying epilepsy status for younger participants with epilepsy, and those whose seizures began at an earlier age.3

4. Discussion

The current study examined vocal ER ability and patterns of neural response to vocal emotional stimuli in youth with and without focal epilepsy. We found that youth with FE were less accurate on the vocal ER task than were TD youth. We also identified several areas of the brain—including the right TPJ, right mPFC, PCC, right hippocampus, right fusiform gyrus, and cerebellum—in which neural response to vocal emotion differentiated youth with FE from their TD peers. Taken together, our findings suggest that recognizing others’ emotional states from their tone of voice may be challenging for youth with epilepsy; moreover, their neural response to this type of socio-emotional stimuli may be different than TD adolescents’ in regions of brain networks subserving social cognitive functions. By beginning to delineate behavioural and neural manifestations of specific social cognitive deficits in youth with epilepsy, these results advance our understanding of a clinical phenotype that is commonly associated with epilepsy.

4.1. Vocal emotion recognition skills

Average P_r was of 0.24 (SD = 0.12) across emotion types. Although seemingly low, this level of performance reflects substantially more hit rates than false alarms; the vast majority of participants (all but 3 FE youth) performed above chance level. Moreover, this level of accuracy is typical of pediatric samples (who are consistently found to be less accurate in vocal ER than adults; e.g., Chronaki et al., 2015, Morningstar et al., 2018b), of affective prosody tasks (which are more difficult than tasks using nonlinguistic vocalizations as stimuli; e.g., Lausen and Hammerschmidt, 2020), and of tasks using youth-generated stimuli (which, although more socially relevant to youth, have been found to be less recognizable than those produced by adults; Morningstar et al., 2018). Emotion-specific patterns were consistent with a large body of previous literature, particularly in the TD group: for instance, anger and sadness were well-recognized, but happiness was not (see reviews by Johnstone and Scherer, 2000, Scherer, 2003).

Compared to TD youth, youth with FE were less accurate on the vocal ER task. This finding is in line with previous reports of vocal ER deficits in adults (Bonora et al., 2011, Broicher et al., 2012b, Hennion et al., 2015, Meletti and Bonora, 2013) and youth with TLE (Cohen et al 1990; c.f., Laurent et al., 2014). Moreover, although this small magnitude interaction effect should be interpreted with caution until it is replicated in a larger sample, we found that group differences in vocal ER were particularly pronounced in younger participants. Struggling to identify the emotional intent of peer-aged speakers has been linked with poorer psychosocial outcomes in typically developing youth (Maxim and Nowicki, 2003, McClure and Nowicki, 2001, Morningstar et al., 2019b, Nowicki and Carton, 1997, Nowicki and Duke, 1992, Rothman and Nowicki Jr., 2004); as such, this form of deficit may be contributing to social difficulties commonly experienced by children with epilepsy (Camfield and Camfield, 2007, Drewel and Caplan, 2007, Sillanpää and Helen Cross, 2009, Steiger and Jokeit, 2017). However, additional research will be needed to determine the functional consequences of vocal ER deficits in youth with epilepsy.

4.2. Pattern of activation to emotional voices

We leveraged multivoxel pattern analysis tools to examine whether we could detect areas of the brain that showed differential patterns of activation to emotional voices in youth with and without epilepsy. We found that activation patterns in six clusters were significantly predictive of epilepsy status in the current sample: the right TPJ, right mPFC, PCC, right hippocampus, right fusiform gyrus, and cerebellum.

Many of these regions are contained within neural networks implicated in social cognition, broadly construed. For instance, the mPFC and TPJ are considered important nodes in the “mentalizing network” (or “social brain”) implicated in evaluating and interpreting socio-affective cues (Kilford et al., 2016, Peelen et al., 2010, Redcay, 2008). These two clusters, along with the PCC and the hippocampus, are also thought to be part of the “default mode network” (DMN) in children and adults (e.g., Fair et al., 2008). Although the DMN has historically been conceptualized as task-negative, it has increasingly been linked to mentalizing, social cognition, and emotional processing functions (Li et al., 2014, Mars et al., 2012, Satpute and Lindquist, 2019, Schilbach et al., 2008, Spreng and Andrews-Hanna, 2015). Extending previous work noting aberrant connectivity between nodes of the mentalizing and DMN network in epilepsy (Cataldi et al., 2013, Liao et al., 2011, Morningstar et al., 2021; Z. Zhang et al., 2009), our findings suggest that youth with epilepsy may engage regions of these social cognitive networks differently than TD youth when perceiving and interpreting vocal cues of emotion. Differential activation in regions of social cognitive networks could denote atypical responses to emotional prosody, reduced integrity of neural networks involved in social cognition, or—given group differences in task performance—index alternative strategies for interpretation of such socio-emotional stimuli (Price and Friston, 1999).

Contrary to our hypotheses, areas involved in vocal emotion processing (e.g., superior temporal gyrus, mesial temporal structures like the amygdala, IFG) did not differentiate between youth with and without epilepsy. It is possible that group differences in basic auditory processing are less prominent than differences in the broader social cognitive functions that putatively underlie vocal ER tasks. In addition, the nature of the task—which required identification and labelling of stimuli—may have relied on frontal regions involved in higher-level analysis or second-order representation of others (e.g., the mPFC; Frith, 2007, Kilford et al., 2016) more so than regions involved in salience detection (e.g., the amygdala; Kennedy and Adolphs, 2012, Redcay and Warnell, 2018). As such, forward projections from the mesial temporal lobe to the frontal cortex may be more important for the interpretation of socio-emotional cues (Kirsch, 2006) in this type of labelling task than the amygdala itself. An interesting possibility for future work is to determine whether prediction based on the emotion participants perceived in emotional stimuli (regardless of its objective classification) would identify discriminative patterns of activation in the amygdala or IFG instead.

Rather, our findings highlighted other clusters of interest in the temporal lobe (fusiform gyrus, hippocampus) and the cerebellum. Differential activation in areas within the temporal lobe may be reflective of general pathology in this area in youth with epilepsy, given the high number of youth in the sample with seizure loci presumed to be in the temporal lobe. Moreover, given the presence of MTS in some of our participants with epilepsy, it is possible that damage to hippocampal and other mesial temporal structures may lead to reduced activation in this region, broadly speaking—which could have served as a predictive cue for the classifier we trained. However, given that clinical variables—including both diagnosis type and the presence or absence of MTS—did not differentiate subgroups of youth with epilepsy for whom the classifier was accurate or not, these results may be reflective of other task-specific forms of activation in these regions that differentiate individuals with and without epilepsy.

Interestingly, of the clinical variables investigated as potentially predictive of classifier success, no illness-related variables differentiated youth with epilepsy who were accurately vs. inaccurately identified by the patterns of activation in the above regions of interest. However, participants’ age and their age of seizure onset were predictive of classifier success. In other words, the classifier was most accurate at identifying epilepsy status for youth who were younger, or those for whom seizure onset occurred at a younger age. These results suggest that patterns of activation in these clusters may be particularly distinctive of epilepsy in younger patients, or in those with earlier childhood onsets, than for older patients. This pattern mirrors our behavioural results, in which group differences in vocal ER ability were also more pronounced in younger participants. We can only speculate about the reasons for this. One possibility may be that, as patients become older, their neural and behavioural responses to emotional stimuli regularize and are more similar to those of their TD peers. In contrast, younger children with FE—for whom ER abilities and neural organization are still immature—may show more atypical responses to emotional prosody as the underlying systems are maturing. Given that deficits in social functioning are often more pronounced when damage to the social brain occurs during early development (Kennedy and Adolphs, 2012), our findings highlight the potential need to allocate and prioritize intervention resources towards young children diagnosed with FE.

4.3. Strengths and limitations

The current study extended the growing literature on functional activation in task-based settings in youth with epilepsy, to try to better understand the neural mechanisms that potentially underlie social cognitive deficits in youth with epilepsy. Expanding on work with facial expressions (e.g., Ives-Deliperi and Jokeit, 2019), our study utilized novel methods to demonstrate differential activation in several regions of the DMN and mentalizing network, which have been previously implicated in vocal emotion recognition tasks (e.g., Morningstar et al., 2019) and social cognitive skills more broadly (Kilford et al., 2016, Schilbach et al., 2008, Spreng and Andrews-Hanna, 2015). Our findings add to the growing body of evidence that individuals with epilepsy process social and affective cues differently at a neural level. This is particularly notable in the context of a long history of investigating emotional processing in patients with epilepsy undergoing neurosurgery. Indeed, although invasive approaches to monitoring neural response intracranially have allowed for incredible advances in understanding the temporal dynamics of emotion processing across the brain (e.g., Guillory and Bujarski, 2014, Zheng et al., 2017), these methods nonetheless rely on the participation of individuals with epilepsy—who may process emotional stimuli differently than those without epilepsy.

Some limitations must be noted. First, our sample was of modest size and comprised a heterogenous sample of youth with epilepsy. Our analytical approach to neural response is robust to modest sample sizes and serves to increase power in situations where the clinical specificity of the sample of interest limits the ability to recruit large samples. However, our behavioural results would benefit from rigorous replication in a larger sample size.

Second, it is not always clear how differential patterns of neural activation to these social cues translate to interpretative deficits (Monti and Meletti, 2015), which is an outstanding question that the current investigation cannot answer. Moreover, it will be important to better establish the functional consequences of altered neural response and task performance on social outcomes for youth with epilepsy. Prospective longitudinal studies tracking neural and behavioural response to several emotion types conveyed via multiple nonverbal modalities (e.g., facial expressions, tone of voice, postures/gestures) and indices of social functioning are needed to better map how neural processing of social cues relates to both the ability to interpret them and the use of these skills in real-world settings.

Lastly, our sample of youth with epilepsy was heterogenous in diagnosis, age, medication regimen, and seizure localization. Although this likely introduced noise to our data, our results are likely relevant to a broad group of youth with refractory partial epilepsy and reflect the heterogeneity of the condition. Indeed, classifier accuracy was largely independent of most clinical variables related to the condition. However, other important variables that were unassessed in this study, such as seizure frequency and levels of anxiety or depression (which are commonly comorbid with pediatric epilepsy and have been associated with vocal emotion recognition deficits in youth; Emerson et al., 1999, Jones et al., 2007, Morningstar et al., 2019a, Morningstar et al., 2020b), may have played a role in our behavioural and/or neural findings. As such, it would be beneficial for larger studies to compare classifier results in subgroups of youth with epilepsy (e.g., late- vs. early-onset epilepsy, youth with and without comorbid conditions like autism, etc.) to add to our understanding of the neural correlates of social cognitive tasks in this at-risk population.

5. Conclusions

To contribute to our understanding of an important social cognitive skill in a population at risk for social functioning deficits, the current study compared task performance and neural response during a vocal emotion recognition task in youth with and without epilepsy. Compared to their typically developing peers, youth with epilepsy were less accurate in identifying speakers’ emotional intent in the task. A novel application of multivoxel pattern analysis revealed that they also showed distinctive patterns of neural response in regions located within social cognitive networks and the temporal lobe. The classifier was particularly able to distinguish younger children with epilepsy from their typically developing peers, suggesting that differences in how youth with and without epilepsy process social stimuli like emotional voices may be particularly pronounced early in development. Our findings encourage further investigations of the associations between neural processing of emotional stimuli, the capacity to interpret these social cues, and social functioning in pediatric epilepsy. Given the relationship between early damage to the social brain and poorer social function later in life (Kennedy & Adolphs, 2012), developing the capacity to identify children within this group who show divergent responses to emotional cues is necessary to adequately assess and funnel resources towards their socio-emotional development.

CRediT authorship contribution statement

M. Morningstar: Conceptualization, Methodology, Formal analysis, Writing – original draft, Visualization. C. Grannis: Conceptualization, Methodology, Software, Formal analysis, Writing – review & editing, Visualization. W.I. Mattson: Conceptualization, Methodology, Software, Writing – review & editing. E.E. Nelson: Conceptualization, Methodology, Resources, Writing – review & editing, Supervision, Funding acquisition.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Appendix A. Supplementary data

The following are the Supplementary data to this article: