Separate neural networks of implicit emotional processing between pictures and words: A coordinate-based meta-analysis of brain imaging studies

Abstract

Both pictures and words are frequently employed as experimental stimuli to investigate the neurocognitive mechanisms of emotional processing. However, it remains unclear whether emotional picture processing and emotional word processing share neural underpinnings. To address this issue, we focus on neuroimaging studies examining the implicit processing of affective words and pictures, which require participants to meet cognitive task demands under the implicit influence of emotional pictorial or verbal stimuli. A coordinate-based activation likelihood estimation meta-analysis was conducted on these studies, which revealed no common activation maximum between the picture and word conditions. Specifically, implicit negative picture processing (35 experiments, 393 foci, and 932 subjects) engages the bilateral amygdala, left hippocampus, fusiform gyri, and right insula, which are mainly located in the subcortical network and visual network associated with bottom-up emotional responses. In contrast, implicit negative word processing (34 experiments, 316 foci, and 799 subjects) engages the default mode network and fronto-parietal network including the ventrolateral prefrontal cortex, dorsolateral prefrontal cortex, and dorsomedial prefrontal cortex, indicating the involvement of top-down semantic processing and emotion regulation. Our findings indicate that affective pictures (that intrinsically have an affective valence) and affective words (that inherit the affective valence from their object) modulate implicit emotional processing in different ways, and therefore recruit distinct brain systems.

1. Introduction

Experimental stimuli that can reliably elicit specific human emotions are essential for laboratory research in multiple disciplines, including psychology, psychiatry, economics, and neuroscience. The most frequently used emotional stimuli in the visual modality are pictures and words (for a review, see Brosch et al., 2010). Emotional pictures (e.g., the International Affective Picture System: see Lang et al., 2008) depict faces, animals, non-moving objects, or scenes that can be emotionally arousing. Emotional words (e.g., the Affective Norms for English Words: see Bradley and Lang, 1999) mainly refer to norms and adjectives that are associated with specific emotional experiences. Both affective pictures and words have been widely employed to investigate emotion processing and emotion-cognition interactions (Izard, 2009; Song et al., 2017). In these studies, participants are often exposed to emotionally salient stimuli that are processed automatically, leading to the re-allocation of attentional resources required for ongoing cognitive tasks; accordingly, impaired behavioral performance in these tasks is considered a manifestation of implicit emotional influence (e.g., Buodo et al., 2002; Cromheeke and Mueller, 2014).

Although both pictorial stimuli and written words are frequently employed in emotion research, little is known about the comparability of emotional effects associated with these stimuli. This issue is not only methodologically, but also theoretically important. Many seminal proposals do not consider the potential systematic differences in empirical findings due to stimulus types, assuming that different types of stimuli engage similar neurocognitive processes (Cacioppo and Gardner, 1999; Dolan, 2002; Koole and Rothermund, 2011; Mauss and Robinson, 2009). This assumption has been supported by some studies showing similar results elicited by emotional pictures and words (e.g., Balconi and Cobelli, 2015; Dowd and Barch, 2010; Gianotti et al., 2008; J. T. Larsen et al., 2003; Wager et al., 2015). For example, emotional influence on cognitive tasks could be evoked by both pictures and words, and their effects show a similar pattern (Agusti et al., 2017; Beall and Herbert, 2008; Bruno et al., 2020). Carroll and Young (2005) found that using pictures as primes and words as targets (and vice versa) could successfully elicit the emotional priming effect, indicating that the activation of a specific emotion is compatible across stimulus categories (but see Meissner and Rothermund, 2015). However, a growing body of evidence indicates that there are fundamental processing differences between pictorial and verbal displays of emotion (Brosch et al., 2010). Essentially, some cognitive processes are selectively associated with either the picture condition (e.g., object categorization) or the word condition (e. g., semantic processing). Pictures may allow access to emotional information more automatically than words, as the latter have to go through phonological processing first (Brosch et al., 2010). Researchers also point out that emotional pictures may be more biologically relevant and therefore are more physiologically arousing than words (Keil, 2006; Vanderploeg et al., 1987), whereas words may lead to stronger top-down effects on emotional processing (Carretie et al., 2008). Moreover, pictures may have privileged access to the emotional systems compared to words, as indicated by the fact that the emotionality of a picture interferes with affective categorization of words, but the reverse is not true (Houwer and Hermans, 1994). Consequently, the classic “negativity bias” (i.e., greater sensitivity to negative than positive stimuli; see Cacioppo and Gardner, 1999; Taylor, 1991) is more prominent for picture stimuli than for word stimuli (Bayer and Schacht, 2014; Herbert et al., 2006; Sutton and Lutz, 2019). Overall, these findings indicate systematic differences in the nature and intensity of the two types of emotional materials (Carretie et al., 2008).

Neuroscientific studies have significant implications for this area of research, as they can uncover common and/or distinct brain systems engaged in different emotional contents. Although numerous studies have been devoted to exploring neural substrates of the processing of emotional pictures and words (Citron, 2012; Hinojosa et al., 2015; Olofsson et al., 2008), only a few of them have applied within-subjects experimental designs to directly compare stimulus domains (Bayer and Schacht, 2014). In an emotional rating task, Kensinger and Schacter (2006) found that both pictures and words increased the activation of widespread brain networks, including the amygdala, ventromedial prefrontal cortex, and dorsal medial prefrontal cortex (dmPFC). Also, Schlochtermeier et al. (2013) reported comparable emotion-related activity for pictures and words in the amygdala, anterior cingulate cortex, and visual areas. Using the event-related potential (ERP) technique, Luo and colleagues demonstrated that the encoding of both emotional facial expressions and emotional words (including norms and adjectives) involves three temporal stages, indicating a common model for emotional information processing (Luo et al., 2010; Yi et al., 2015; D. Zhang et al., 2014; D. Zhang et al., 2013a, 2013b). These findings are further supported by other ERP studies identifying similar neural responses between stimulus domains, leading to the conclusion that biological and symbolic emotional signals are decoded by the same brain systems (Schacht and Sommer, 2009; Tempel et al., 2013). In contrast, other studies have indicated that different types of stimuli exhibit different capacities to trigger emotion-related brain activity. Flaisch et al. (2015) detected strong activations of visual, parietal, temporal, frontal, and subcortical regions in the emotional picture categorization task, whereas only the extrastriate cortex was involved in the emotional word categorization task (see also Reisch et al., 2020). Leclerc and Kensinger (2011) found that emotional pictures and words were characterized by distinct activations in the amygdala and medial prefrontal cortex in both young and older populations. There is also ERP evidence for preferential processing of emotional pictures compared to words, manifesting as larger amplitudes, shorter latencies, and/or additional ERP components in the picture condition (Bayer and Schacht, 2014; Fruhholz et al., 2011; Hinojosa et al., 2009; Rellecke et al., 2011).

The possible reasons for such discrepancies remain unclear and understudied in the literature, but it is plausible that the difference in experimental paradigms has played an important role, as different paradigms may engage various cognitive processes that interact with emotional processing in diverse ways (Wentura, 2019). In addition, some previous results were based on limited-size samples, which have been demonstrated to be a source of heterogeneity (Bakker et al., 2012; Button et al., 2013). The latter issue could be addressed by using the meta-analysis technique, which is employed in the current study to examine the comparability of emotion effects elicited by different stimulus types. By quantitatively examining convergence across documented studies, meta-analysis is capable of providing a comprehensive overview of a research domain, assessing the consistency of previous findings, and overcoming the heterogeneity of experimental results based on underpowered samples (Gurevitch et al., 2018; Rosenthal and DiMatteo, 2001). Researchers have applied meta-analysis on behavioral and neuroimaging data to produce abundant knowledge about emotion-related concepts (e.g., Butzlaff and Hooley, 1998; Elfenbein and Ambady, 2002; Kober et al., 2008; Phan et al., 2002; Wager et al., 2003). Nevertheless, only a few meta-analyses on emotional processing have considered stimulus-specific effects. Some of these studies only included publications reporting an a priori region of interest (e.g., the amygdala) (Sergerie et al., 2008), some conducted follow-up modulation analyses on multiple stimulus types rather than independent meta-analysis for each stimulus type (Lee and Siegle, 2012), whereas others focused on the comparison across different modalities (e.g., visual, auditory, olfactory, and imagery) (Costafreda et al., 2008; Phan et al., 2002; Ruffman et al., 2008; Satpute et al., 2015). Moreover, Morawetz et al. (2017) found stronger dorsolateral prefrontal cortex (dlPFC) activation for pictures than for other stimulus materials (not limited to words) in emotion regulation tasks. In short, although these studies provide valuable knowledge on the emotional brain, they have not directly addressed our research interest.

In this study, we conducted a coordinate-based activation likelihood estimation (ALE) meta-analysis (Laird et al., 2005) to summarize brain imaging findings related to the emotional Stroop task (EST) and its variants (MacLeod, 1991; Williams et al., 1996). The EST represents one of the most influential paradigms for investigating (implicit) emotional processing in healthy and clinical populations (e.g., Cisler et al., 2011; Epp et al., 2012; Phaf and Kan, 2007; Williams et al., 1996). The EST differs from the classical Stroop Task in the nature of its interference effect (see R. J. Larsen et al., 2006 for a review): unlike the classical Stroop task in which the meaning of a color-word (e.g., “red”) and its ink color are incongruent (MacLeod, 1991; Stroop, 1935), the EST examines emotional effects elicited by pictorial or verbal stimuli on cognitive task (e.g., color naming) performance. In our opinion, comparing picture-based and word-based findings across different EST variations is suitable for addressing the current research topic. Focusing on the EST should also help to minimize confounding effects that arise from differences in experimental design and task demand across studies (Bayer and Schacht, 2014). Considering that a large number of studies using the EST only consist of negative and neutral (but not positive) conditions (e.g., Brennan et al., 2015; Dresler et al., 2012; George et al., 1994; Isenberg et al., 1999; Mitterschiffthaler et al., 2008; Whalen et al., 1998), we restricted our analyses to these two conditions to include as many available data as possible. Previously, we used meta-analysis to identify the brain mechanisms underlying the EST with emotional words as stimuli, revealing the involvement of the dmPFC and ventrolateral prefrontal cortex (vlPFC); however, the EST with emotional pictures was not included in that study (Feng et al., 2018). Here, we expected to observe that emotional pictures and words recruit both overlapping and distinct neural correlates.

Considering previous studies, we are particularly interested in the brain systems associated with emotional reaction (e.g., the amygdala) and emotion regulation (e.g., the lateral prefrontal regions) (Ji et al., 2019; Lee and Siegle, 2012; Pessoa, 2018; Xie et al., 2016). One hypothesis was that the brain systems associated with emotional reaction would be more active for emotional pictures than for words, seeing that pictures are perceived to be more biologically and evolutionarily relevant; the reverse would be true for the brain systems associated with emotion regulation, seeing that word processing is closely related to top-down regulation. Moreover, it was hypothesized that some regions were likely to be domain-specific; for instance, the fusiform gyrus would be preferentially involved in affective picture processing, whereas the temporal gyrus and inferior parietal areas would be preferentially engaged by emotional words, because their activation reflects the encoding of specific stimulus features (Friederici, 2011; Vuilleumier et al., 2001). Finally, regions identified in the meta-analysis were overlaid on a brain functional network atlas to reveal underlying large-scale network correlates (Chen et al., 2018; R. Zhang et al., 2017), and the psychological functions of these regions were examined using functional decoding analyses based on large-scale datasets from the Neurosynth database (Yarkoni et al., 2011). Together, these complementary analytical approaches aimed to provide data-driven quantitative inferences on the psychophysiological functions of the identified regions.

2. Materials and methods

2.1. Literature search and selection

Systematic and comprehensive searches of the PubMed, ISI Web of Science, and Google Scholar databases were performed in December 2020 according to the PRISMA guidelines (Shamseer et al., 2015). The keywords for the search used the combination of two categories of relevant terms as follows: (i) implicit emotional processing: “emotional distractor” OR “affective distractor” OR “emotional Stroop” OR “affective Stroop” OR “implicit emotional processing” OR “implicit affective processing;” and (ii) imaging modalities: “magnetic resonance imaging” OR “fMRI” OR “positron emission tomography” OR “PET” OR “neuroimaging.” In addition, we explored several other information sources, including: (1) the BrainMap database (http://brainmap.org); (2) the bibliography and citation indices of the pre-selected articles; (3) the reference list of relevant reviews (Carretié, 2014; Feng et al., 2018; Levin et al., 2007; Sussman et al., 2016); and (4) direct searches on the names of frequently occurring authors.

The studies were further assessed according to the following criteria. First, each publication reported an empirical study and was published in a peer-reviewed English language journal. Second, each study recruited healthy participants who performed an implicit emotional processing task, including the color-word emotional Stroop task, emotional counting Stroop task, and implicit affective-picture processing task, all of which are prevailing paradigms employed in cognitive and clinical psychology (Phaf and Kan, 2007; Williams et al., 1996). For instance, in the color-word emotional Stroop task, participants are required to perform a simple cognitive task (e.g., name the color of words) while ignoring the semantic meaning of those words (George et al., 1994; Williams et al., 1996); in the implicit affective-picture processing task, affective stimuli are presented as the background when participants are required to perform an irrelevant task, such as categorizing the gender of displayed affective faces (Haller et al., 2018). The affective contents of words or pictures used in these tasks do not directly conflict with the participants’ main tasks; instead, disruptions in task performance are suggested to indicate the emotional significance of affective words or pictures (Algom et al., 2004; Dalgleish, 2005). In other words, these tasks probe implicit processing of affective content. To distinguish the implicit processing of different kinds of affective content, we classified these studies into: (1) implicit negative word processing (i.e., negative words > neutral words), (2) implicit positive word processing (i.e., positive words > neutral words), (3) implicit negative picture processing (i.e., negative pictures > neutral pictures), and (4) implicit positive picture processing (i.e., positive pictures > neutral pictures). Fourth, we restricted the current meta-analysis to studies that employed functional magnetic resonance imaging (fMRI) or positron emission tomography (PET), and studies that reported whole-brain functional neuroimaging data (rather than region of interest [ROI] analyses). Fifth, the results were derived from a general linear model based on either binary contrast or parametric analyses. Finally, brain activation was presented in a standardized stereotaxic space (Talairach or Montreal Neurological Institute, MNI). Note that for the papers reporting Talairach coordinates, a conversion to MNI coordinates was employed using the icbm2tal algorithm (Lancaster et al., 2010) implemented in the GingerALE software (version 3.0.2, http://www.brainmap.org/). Filtering the search results according to the inclusion/exclusion criteria yielded a total of 66 published fMRI or PET studies for the main meta-analysis (Fig 1). The publications included in this meta-analysis are listed in Table S1.

Fig. 1. Flow chart of the study selection process for the meta-analysis.

fMRI, functional magnetic resonance imaging; PET, positron emission tomography.

2.2. Main ALE approach

A coordinate-based meta-analysis of the included studies was conducted using the revised ALE algorithm (in-house MATLAB scripts) (Eickhoff et al., 2009). ALE is a modeling technique used to determine the convergence of foci reported from different neuroimaging studies, with published foci in Talairach or MNI space (Turkeltaub et al., 2002). ALE interprets reported foci as spatial probability distributions, whose widths are based on empirical estimates of spatial uncertainty due to between-subject and between-template variability in neuroimaging data (Eickhoff et al., 2009). Within each of the included studies in this analysis, a modulated activation (MA) map or modeled anatomical map, was created by taking the maximum probability associated with any one focus (always the closest one) for each voxel (Turkeltaub et al., 2012). Notably, to prevent studies with multiple experiments based on the same subject sample from influencing ALE values more than others, different experiments from the same subject sample were combined into a single experiment rather than being treated as independent experiments (see also Turkeltaub et al., 2012).

The union of individual MA maps created from the maximum probability associated with the closest focus for each voxel (Turkeltaub et al., 2012) was then calculated to obtain an ALE map across the experiments. This ALE map was assessed against a null distribution of random spatial association between experiments using a non-linear histogram integration algorithm (Eickhoff et al., 2012). In addition, the average non-linear contribution of each experiment for each cluster was calculated from the fraction of the ALE values in the cluster with and without the respective experiment (Eickhoff et al., 2016). Based on the calculated contribution, we employed two additional criteria to select significant clusters: (1) the contributions for one cluster should be from at least two experiments so that the finding would not be derived by a single experiment; and (2) the average contribution of the most dominant experiments (MDE) should not exceed 50 % and the average contribution of the two most dominant experiments (2MDEs) should not exceed 80 % (Eickhoff et al., 2016). These additional criteria were set according to a recent simulation study to ensure that the identified clusters were derived from sufficient convergence across experiments rather than driven by only a few experiments (Eickhoff et al., 2016).

Applying the ALE algorithm, the reported coordinates of the brainareas associated with the emotional Stroop effect were converged across different experiments. Specifically, the neural signatures of the emotional Stroop effect were converged using the following meta-analytic strategies: (i) implicit negative word processing (i.e., negative words > neutral words; 34 experiments, 316 foci, and 799 subjects); and (ii) implicit negative picture processing (i.e., negative pictures > neutral pictures; 35 experiments, 393 foci, and 932 subjects). In addition, we identified four experiments (10 foci, 70 subjects) for implicit positive word processing and 15 experiments (126 foci, 412 subjects) for implicit positive picture processing. Considering that the number of experiments for positive word and picture processing was below the recommended number of experiments (i.e., 17 – 20) to obtain reliable results (for details on this issue, see Müller et al., 2018), the current study mainly focused on implicit negative word/picture processing. However, for completeness, we present the meta-analysis of implicit positive picture processing (the number of studies for positive word processing was too small to conduct a meaningful meta-analysis) as supplementary results (see also Fig. S5 and Table S4).

All maps were thresholded using a cluster-level family-wise error correction (P < 0.05) with a cluster-forming threshold of P < 0.001 (Eklund et al., 2016; Woo et al., 2014).

2.3. Modulation effects

We extracted per-voxel probabilities of activation in the meta-analysis for each of the identified brain regions to examine potential modulating effects of mean age, sex ratio, and task type across studies (see also Goodkind et al., 2015; Li et al., 2020; McTeague et al., 2017). The non-parametric Kruskal – Wallis H, Mann – Whitney U, and Spearman’s rank correlation tests were used as required.

2.4. Conjunction/contrast analysis

After obtaining consistent maxima separately for implicit negative word processing and implicit negative picture processing, a conjunction analysis was conducted to assess the correspondence. This was implemented by using the minimum statistic (Nichols et al., 2005), which is equivalent to identifying the intersection between two thresholded ALE results. In addition, differences between implicit negative word processing and implicit negative picture processing were tested by first performing a separate ALE analysis for each condition and then computing the voxel-wise difference between the ensuing ALE maps (Eickhoff et al., 2011).

2.5. Large-scale network analysis

To assess the underlying large-scale network correlates, we overlaid the identified clusters (see the Results section) onto seven canonical functional cortical networks: the fronto-parietal network (FPN), dorsal attention network (DAN), ventral attention network (VAN, also known as the salience network), somatomotor network (SMN), visual network (VN), cortical affective network (AFN, also known as the limbic network), and default mode network (DMN) (Choi et al., 2012; Liu et al., 2018; Yeo et al., 2011). These brain networks, which have been commonly employed in the literature across various domains, were derived from resting-state functional connectivity that represents the intrinsic organization of the functional connectome of the human brain rather than being specific to a particular domain (Yeo et al., 2011). In line with previous studies (Li et al., 2020; R. Zhang et al., 2017), the large-scale network analysis represents a follow-up functional characterization of the brain regions identified in the meta-analysis, in terms of the intrinsic network topology of the human brain. Many researchers have pointed out that the resting state networks show close correspondence with activation networks across task states, indicating a framework where the human brain is intrinsically organized into domain-general functional networks from the interactions of which affective and cognitive processes are constructed (Barrett and Satpute, 2013; Smith et al., 2009). Accordingly, many studies have successfully used Yeo et al. (2011)’s template as the functional network atlas to investigate various affective and cognitive functions (e.g., Zhang et al., 2017).

It is noteworthy that the template covers only the cerebral cortex; thus, a collection of subcortical areas (i.e., subcortical network [SCN]) was added to the template, resulting in eight brain networks in total (see also Liu et al., 2018). The relative distribution was calculated as the ratio of activated voxels of a given network to all activated voxels, whereas the absolute distribution was computed as the ratio of activated voxels of a given network to the voxels of the corresponding template network (see also Chen et al., 2018; R. Zhang et al., 2017).

2.6. Functional decoding

To explore which psychological topics were most relevant to the identified clusters (see the Results section), a meta-analysis was first performed based on version 0.6 of the Neurosynth database (Yarkoni et al., 2011). The database consists of 11,406 fMRI studies and over 410, 000 activity peaks that cover the published neuroimaging literature. The data collected from each study included the peak activities for contrasting variables reported in the study and the frequency of all words in the abstract. Notably, a set of 60 psychological topics were used (De La Vega et al., 2017), which was derived from the latent Dirichlet allocation topic modeling to remedy the redundancy and potential ambiguity in word terms (Blei et al., 2003).

Using all fMRI studies, a functional decoding analysis was then performed by training a naïve Bayes classifier, which is widely used in text classification (Lewis, 1998; Rennie et al., 2003). Two sets of studies were characterized and selected, one of which activated at least 5% of voxels and the other did not activate any voxel in the given ROI as the training sample (De La Vega et al., 2017). The area under the curve — receiver operating characteristic was used to measure the performance of the model with a 4-fold cross-validation, resulting in the conditional probability of the 60 psychological topics under each module. Please note that only those topics that yielded significant results (P < 0.01) in multiple comparisons using the false discovery rate by implementing a permutation test were reported. Finally, the log odds ratio between the probability of the top five topics activating the considered clusters and the probability of the topic not activating the considered clusters was extracted from the trained naïve Bayes model to generate functional decoding profiles.

3. Results

3.1. Main ALE meta-analysis

For implicit negative word processing, the ALE meta-analysis revealed significant convergence of activity in the left vlPFC, dlPFC, and dmPFC (Fig. 2a and Table 1). Thirteen out of 34 experiments contributed to the cluster in the left vlPFC (MDE = 13.06 %; 2MDE = 25.34 %, Table S1). Eight out of 34 experiments contributed to the cluster in the left dlPFC (MDE = 28.00 %; 2MDE = 45.64 %, Table S1). Nine out of 34 experiments contributed to the cluster in the left dmPFC (MDE = 17.41 %; 2MDE = 34.81 %, Table S1).

Fig. 2. Significant clusters from the main coordinate-based activation likelihood estimation (ALE) meta-analysis and contrast analysis.

(a) Consistent maximum for main ALE meta-analysis of implicit negative word processing. (b) Consistent maximum for main ALE meta-analysis of implicit negative picture processing. (c) Consistent maximum for the contrast of negative word > negative picture. (d) Consistent maximum for the contrast of negative picture > negative word. L, left; R, right; Amyg, amygdala; vlPFC, ventrolateral prefrontal cortex; dlPFC, dorsolateral prefrontal cortex; dmPFC, dorsomedial prefrontal cortex; FG, fusiform gyrus; AI, anterior insula; Hipp, hippocampus.

Table 1.

ALE meta-analysis results for implicit negative word and picture processing.

Laterality	Brain Regions	BA	MNI Coordinates			peak Z score	Cluster Size
			(mm)
			x	y	z		(mm3)
Negative words
L	ventrolateral prefrontal cortex	47	−46	30	−2	6.47	2104
L	dorsolateral prefrontal cortex	45	−46	18	20	4.23	720
L	dorsomedial prefrontal cortex	9	−6	52	38	4.32	1152
Negative pictures
L	fusiform gyrus	37	−40	−48	−16	5.34	1000
R	fusiform gyrus	37	40	−48	−20	7.00	2888
L	amygdala extending to hippocampus	\	−20	−6	−16	6.04	2712
R	amygdala extending to hippocampus	\	20	−4	−14	5.32	2208
R	anterior insula	13	38	22	6	4.95	1048
Conjunction between negative words and negative pictures
\			\	\	\	\	\
Negative words > negative pictures
L	ventrolateral prefrontal cortex	47	−52	32	−2	2.87	1152
L	dorsolateral prefrontal cortex	45	−48	16	18	2.36	472
L	dorsomedial prefrontal cortex	9	−8	52	40	3.28	944
Negative pictures > negative words
L	fusiform gyrus	37	−36	−50	−14	2.63	768
R	fusiform gyrus	37	38	−48	−20	6.91	2680
R	amygdala	\	20	−4	−14	3.09	1600
L	amygdala	\	−22	−2	−20	3.11	1440
L	hippocampus	\	−30	−20	−10	2.33	176
R	anterior insula	13	40	26	4	3.33	984

P_(FWE) < 0.05 at the cluster level with a cluster-forming threshold of P < 0.001 using 10,000 permutations. L, left; R, right; BA, Brodmann area.

These regions were primarily distributed in the DMN (relative: 80.30 %; absolute: 1.32 %) and FPN (relative: 15.58 %; absolute: 0.37 %) (Fig. 3a and 3b, left panel) and were predominantly linked to memory, language, communication, reading, and semantic processing (Fig 4a and S1a).

Fig. 3. Network distribution of clusters associated with implicit negative word and picture processing.

(a) Relative distribution of networks for brain regions identified in the main ALE meta-analysis of implicit negative word and picture processing in canonical brain networks. (b) Absolute distribution of networks for brain regions identified in the main ALE meta-analysis of implicit negative word and picture processing in canonical brain networks. (c) Relative distribution of networks for brain regions identified in the contrasts of negative words > negative pictures and negative pictures > negative words in canonical brain networks. (d) Absolute distribution of networks for brain regions identified in the contrasts of negative words > negative pictures and negative pictures > negative words in canonical brain networks. L, left; R, right; DMN, default mode network; VN, visual network; SCN, subcortical network; FPN, fronto-parietal network; AFN, cortical affective network; VAN, ventral attention network; DMN, dorsal attention network; SMN, somatomotor network.

Fig. 4. Functional decoding of brain regions involved in implicit negative word and picture processing.

(a) Functional decoding for brain regions identified in the main ALE meta-analysis of implicit negative word and picture processing. (b) Functional decoding for brain regions identified in the contrasts of negative words > negative pictures and negative pictures > negative words.

Notably, for all of the identified clusters, the modulation analysis revealed no effect of sex ratio (non-parametric correlations: Spearman ∣rho∣ < 0.10; P > .60; Mann–Whitney U tests via median-split: ∣U∣ < 0.67; P > .50), mean age (non-parametric correlations: Spearman ∣rho∣ < 0.20; P > .32; Mann–Whitney U tests via median-split: ∣U∣ < 1.88; P > .06), or task type (color-naming task vs. counting task; Mann–Whitney U tests: ∣U∣ < 1.96; P > .05).

For implicit negative picture processing, the ALE meta-analysis revealed significant convergence of activity in the bilateral fusiform gyrus, amygdala (extending to the hippocampus), and anterior insula (AI) (Fig. 2b and Table 1). Eight out of 35 experiments contributed to the cluster in the left fusiform gyrus (MDE = 22.32 %; 2MDE = 43.83 %, Table S2). Thirteen out of 35 experiments contributed to the cluster in the right fusiform gyrus (MDE = 13.68 %; 2MDE = 26.50 %, Table S2). Twelve out of 35 experiments contributed to the cluster in the left amygdala (MDE = 11.83 %; 2MDE = 21.07 %, Table S2). Thirteen out of 35 experiments contributed to the cluster in the right amygdala (MDE = 14.63 %; 2MDE = 27.71 %, Table S2). Twelve out of 35 experiments contributed to the cluster in the left AI (MDE = 26.96 %; 2MDE = 45.27 %, Table S2). Eleven out of 35 experiments contributed to the cluster in the right AI (MDE = 19.43 %; 2MDE = 36.33 %, Table S2).

These regions were primarily distributed in the SCN (relative: 46.47 %; absolute: 5.17 %) and VN (relative: 22.29 %; absolute: 1.01 %) (Fig. 3a and 3b, right panel) and were mainly related to face/emotion, emotion, fear, memory, and reward processing (Fig. 4a and S1b).

Notably, for all of the identified clusters, the modulation analysis revealed no effect of sex ratio (non-parametric correlations: Spearman ∣rho∣ < 0.19; P > .28; Mann–Whitney U tests via median-split: ∣U∣ < 1.14; P > .25), mean age (non-parametric correlations: Spearman ∣rho∣ < 0.32; P > .08; Mann–Whitney U tests via median-split: ∣U∣ < 1.05; P > .30), or task type (gender discrimination task vs. other tasks; Mann–Whitney U tests: ∣U∣ < 1.74; P > .08).

3.2. Conjunction

A conjunction analysis revealed that there was no common activation maximum for implicit negative words and pictures.

3.3. Contrast

The left vlPFC, dlPFC, and dmPFC were more activated during implicit negative word processing than during implicit negative picture processing (Fig 2c and Table 1), all of which were located in the regions identified in the main meta-analysis of implicit negative word processing (Fig. S2). These regions were primarily distributed in the DMN (relative: 83.79 %; absolute: 0.70 %) and the FPN (relative: 16.21 %; absolute: 0.21 %) (Fig. 3c and 3d, left panel) and were predominantly linked to memory, language, communication, reading, and semantic processing (Fig. 4b and S3a). These findings are in line with a recent study indicating that the subregions of the lateral frontal cortex identified in the current study were more located in the DMN than in the FPN (de la Vega et al., 2018).

In contrast, the contrast analysis showed that the bilateral fusiform gyrus, amygdala, left hippocampus, and right AI were more activated during implicit negative picture processing than during implicit negative word processing (Fig. 2d and Table 1), all of which were located in regions identified with the main meta-analysis of implicit negative picture processing (Fig. S4). These regions were primarily distributed in the SCN (relative: 43.20 %; absolute: 3.62 %) and VN (relative: 27.87 %; absolute: 0.95 %) (Fig. 3c and 3d, right panel) and were mainly related to face/emotion, emotion, fear, memory, and reward processing (Fig. 4b and S3b).

4. Discussion

Pictures and words have frequently been employed as emotion-evoking materials. Empirical findings based on these two types of stimuli are widely supposed to be comparable to some degree. However, the results of our coordinate-based ALE meta-analysis on brain-imaging studies using the EST and its variants showed largely separated neural networks of implicit emotional processing between pictures and words. On the one hand, implicit processing of negative pictures recruits the bilateral amygdala, left hippocampus, fusiform gyri, and right AI, which are mainly located in the SCN and VN. On the other hand, implicit processing of negative words recruits the left vlPFC, dlPFC, and dmPFC, which are important nodes of the DMN and FPN. Notably, the current study revealed no common brain regions involved in the implicit processing of negative pictures and words. Therefore, we suggest that implicit picture processing and word processing engage distinct brain systems.

4.1. Brain networks associated with emotional picture processing

Our functional decoding analysis indicated that the brain modules (i. e., the SCN and VN) associated with implicit negative picture processing were mainly related to the processing of fear, emotional faces, memory, and reward, consistent with previous reports (Bressler and Menon, 2010; Rosazza and Minati, 2011; Uddin et al., 2019; Xu et al., 2016). In line with the current findings, Kitada et al. (2010) found that facial expressions compared with pictures of shoes induced stronger activations in the SCN and VN, including the bilateral amygdala, right fusiform gyrus, insula, and hippocampus. In the history of affective neuroscience, the SCN has long been considered the evolutionarily preserved basis of basic emotions (for a review, see Dalgleish, 2004). More specifically, subcortical structures are shaped by evolution to generate autonomic, bottom-up emotional responses (e.g., fear, anger, and disgust) that are meaningful for survival and adaptation (Ohman et al., 2007). It is thus understandable that SCN activity increases as a function of the biological relevance of stimuli (Adolphs, 2008). Compared to words, pictures may be perceived as more biologically relevant and are therefore more capable of eliciting autonomic emotional reactions (Brosch et al., 2010; Feng et al., 2018; Hinojosa et al., 2009), which could help explain the current findings. In line with this idea, Phaf and Kan (2007) pointed out that pictures are evolutionarily prepared stimuli, but the same may not be true for words from the same emotional category, which do not have obvious perceptual characteristics.

As a key region in the SCN, the amygdala has been widely regarded as a core region in the brain circuitry of emotion (especially negative emotions such as fear and anxiety; see Bishop, 2007; LeDoux, 2000). The current meta-analysis revealed that only negative pictures, but not words, could reliably elicit bilateral amygdala activation in the EST (for the absence of amygdala engagement, see also Bremner et al., 2004; Dresler et al., 2012; Feng et al., 2018; Veroude et al., 2013). As mentioned above, pictures could be regarded as stimuli with a higher level of biological relevance than words. Thus, pictures may provide a more vivid (and accordingly more emotionally arousing) subjective experience than do words. Previous studies have found that amygdalar activity scales with the vividness of emotional events (Kensinger et al., 2011; Yonelinas and Ritchey, 2015). A key function of the amygdala is to produce emotion-related effects from episodic memories (Hamann et al., 1999; Phelps and Sharot, 2008). Compared to words, pictures may be more powerful in facilitating the retrieval of emotional memories, as they contain abundant visual details that could function as memory cues (Sharot et al., 2004).

Similar to the amygdala, the left hippocampus has been implicated in the processing of emotion-eliciting stimuli and emotional memory (Bellace et al., 2013; Garrett and Maddock, 2006). Many studies have suggested that the amygdala and hippocampus constitute a complex that sustains emotional memories (Brohawn et al., 2010; Madan et al., 2017; Richardson et al., 2004). According to Kensinger and Corkin (2004), the amygdalar-hippocampal network is responsible for emotional memory enhancement for arousing information, such that the amygdala modulates the encoding and storage of hippocampal-dependent memories (see also Phelps, 2004). In our opinion, the left hippocampus interacts with the amygdala during emotional picture presentation for the retrieval of emotional memories, because pictures more effectively evoke those memories than do words.

Within the VN, the fusiform gyrus is preferentially recruited by negative pictures. This finding could be attributed to the fact that faces have been widely applied in EST variants (particularly the implicit face-emotion processing task), and that neural activity of the fusiform gyrus has long been recognized as being face-selective (McCarthy et al., 1997). That said, the face-specificity hypothesis is not without controversy (for a review, see Duchaine and Yovel, 2015). For instance, some researchers have proposed that a subregion of the fusiform gyrus is responsive to visual word perception (Kanwisher and Yovel, 2006; McCandliss et al., 2003). Despite this debate, our results confirmed that the fusiform gyrus was activated in the picture condition, but not in the word condition, and that this region encodes the emotional valence of pictures.

In line with the current findings, it has been well-documented that there are extensive anatomical connections between the VN and key nodes (e.g., the amygdala) in the SCN (Tamietto et al., 2012; Tao et al., 2021). Moreover, a large body of evidence has shown that the functional connectivity between visual system and subcortical limbic areas are modulated according to various affective features of stimuli, including their salience, significance, ambiguity, and unpredictability (Pessoa and Adolphs, 2010; Uddin, 2015). For instance, the functional coupling of the amygdala and visual cortex increased in proportion to the arousal rating of pictures (Sabatinelli et al., 2005). Likewise, emotion regulation strategies, such as attentional deployment and reappraisal, attenuated the functional connectivity between the amygdala and visual cortex in response to negative pictures (Ferri et al., 2016; Sarkheil et al., 2019).

Finally, the right insula is also known to be sensitive to the difference between negative and neutral pictures, but not to that between negative and neutral words. Most researchers attribute the insula to the salience network, which is responsible for monitoring the salience of external inputs and internal events (Bressler and Menon, 2010). The insula is richly connected with key nodes of the SCN and VN (Gasquoine, 2014); for instance, this region and the amygdala are anatomically connected directly in a reciprocal way (Baur et al., 2013). Similar to the amygdala, the insula is an important region for the processing of basic emotions, such as anger, sadness, and disgust (Chang et al., 2012). In our opinion, the selective engagement of the insula in the emotional picture condition is also due to the biological relevance of pictures.

4.2. Brain networks associated with emotional word processing

The DMN and FPN are more involved in negative word processing than in negative picture processing. Our functional decoding analysis indicated that these brain systems were mainly related to higher-level functioning, including language, communication, reading, and semantic processing. Although the DMN is generally more activated during the resting state than most task states (Raichle et al., 2001), this network also participates in a wide range of cognitive functions (Buckner et al., 2008; Greicius et al., 2003). Regarding the FPN, researchers suggest that its main function is to provide rapid adaptive control of other brain systems (Dosenbach et al., 2008; Vincent et al., 2008). In line with the current findings, the importance of the DMN and FPN to semantic processing has been highlighted in previous research (e.g., Klepousniotou et al., 2014; Lanzoni et al., 2020). For instance, Wirth et al. (2011) compared semantic, phonological, and perceptual decision tasks and found that the entire DMN unit showed less deactivation for semantics than for the other two conditions. Here, the significance of the DMN echoes the idea of Binder and Desai (2011) that “resting” is a cognitively complex condition, including memory retrieval and semantic knowledge manipulation (see also Yeshurun et al., 2021). In a functional connectivity study, Xu et al. (2016) discovered that both the DMN and FPN were key parts of an intrinsic functional semantic processing network; specifically, the DMN may reflect a memory retrieval process during which episodic memories are elicited by semantic materials (see also Hassabis and Maguire, 2007; Spaniol et al., 2009), whereas the FPN is related to conceptual and language-related control (e.g., the cognitive control process of language production) (Geranmayeh et al., 2014). According to the above, it is not surprising that the DMN and FPN were selectively associated with emotional word stimuli in the current study.

Nevertheless, it is likely that the DMN and FPN are not restricted to general semantic processing, otherwise it would be difficult to explain why the activity of these networks is modulated by the emotional valence of word stimuli (negative > neutral). The key nodes within the DMN and FPN, including the dlPFC, dmPFC, and vlPFC, are engaged in various cognitive domains, ranging from selective attention and working memory to response selection (Duncan, 2013; Duncan and Owen, 2000). In general, the vlPFC is associated with the inhibitory control of prepotent responses (Aron et al., 2004; Swick et al., 2008); the dlPFC is associated with attention allocation based on current goals and task demands (Fan et al., 2002; Posner and Rothbart, 2007); and the dmPFC is associated with the monitoring of ongoing performance and the adaptive control of attention (Botvinick et al., 2001; Bush et al., 2000). Overall, these brain areas play critical roles in cognitive control and emotion regulation (Duncan and Owen, 2000; Ochsner and Gross, 2008). It is therefore not surprising that their activities have been consistently observed during the EST (Dresler et al., 2012; Herrington et al., 2005) and are negatively correlated with the emotional Stroop effect (Mincic, 2010; Price et al., 2011). As pointed out by Feng et al. (2018), the involvement of these regions indicates that a cognitive control network targeting emotion-cognition interactions is employed in the EST to maintain behavioral performance by suppressing interference from emotional distractors (see also Cromheeke and Mueller, 2014). Consistent with the current findings, Feng et al. (2018) discovered that the presentation of emotional words enhanced the regulatory engagement of the aforementioned cognitive control regions.

In short, the current findings on the implicit processing of negative words largely overlap with those regarding classical emotion regulation networks (Cisler et al., 2013; McRae et al., 2010; Ochsner et al., 2002). In our opinion, these findings indicate the importance of language processing in the cognitive regulation of emotion. According to a previous meta-analysis of neuroimaging studies on emotional regulation, the reappraisal (or reconceptualization) process of an emotional stimulus involves reformulating the mental representation of that stimulus in which language processing plays a key role (Kohn et al., 2014; see also Ochsner et al., 2004). This conjecture is in line with the emotional consciousness theory proposed by LeDoux and Brown (2017), which argues that for human beings, language is essential to conscious experiences of emotion. In particular, language allows symbolic representation of those experiences, organizes them into categories, and regulates them without being exposed to evolutionarily relevant stimuli (e.g., snakes and spiders).

Finally, our results associated with emotional words were generally left-lateralized (i.e., left vlPFC and dlPFC). This functional hemispheric asymmetry is consistent with previous findings in language research (Geranmayeh et al., 2014; Xu et al., 2016). For instance, many studies have highlighted the importance of the left hemisphere in semantic control (Geranmayeh et al., 2012; Whitney et al., 2011, 2012).

4.3. Summary

To sum up, this meta-analysis revealed distinct brain systems for the implicit processing of negative pictures and words: the SCN and VN are more involved in the emotional valence of picture stimuli than of word stimuli, possibly because this kind of stimuli is more biologically relevant than word stimuli; in contrast, the DMN and FPN are more involved in the emotional valence of word stimuli than of picture stimuli, indicating their functions in semantic processing and emotion regulation. According to these findings, we suggest that there is no generic valence processing systems for different types of affective stimuli (see also Yuan et al., 2019). Instead, our findings are in favor of multiple (in contrast to common) system models in which picture processing and word processing systems generally operate independently rather than converge on a common store (Federmeier and Kutas, 2001; Paivio, 1991). Therefore, it is understandable why conflicting results are often produced by studies using different methods for emotion induction. Accordingly, we suggest that emotion researchers should (1) be aware of the comparability of previous studies using different evocative stimuli, and (2) carefully choose a specific type of stimulus even though they are not interested in the perceptual characteristics of pictures or words. More uniform, standard stimuli and experimental paradigms are highly recommended (e.g., Izard, 2007). In particular, if researchers consider the amygdala as their region of interest, then using pictorial stimuli instead of using verbal stimuli would be more likely to identify the target region. The reverse may be true if researchers focus on the regulatory effect of cognitive control regions (e.g., the dlPFC, dmPFC, and vlPFC) targeting the influence of emotional material on ongoing task demands. Overall, we agree with the viewpoint of Phan et al. (2002) that individual imaging studies cannot fully unravel specific regional brain involvement in emotion due to limitations in experimental stimuli and task design; thus, more convergent research using alternative methodological approaches is needed. Further, these findings provide support to the constructionist hypotheses of the emotional brain; that is, emotion categories may not be unambiguously localized to distinct brain regions (Barrett, 2006). Instead, more general brain networks, which are commonly involved in psychological operations of both an emotional and non-emotional nature, are active during emotion experience (Lindquist et al., 2012).

The emotional Stroop effect based on the EST has been recognized as a reliable marker of the psychopathological symptom load in clinical populations, particularly those with emotional disorders (Cox et al., 2006; Epp et al., 2012; Pishyar et al., 2004). We suggest future studies to explore the clinical significance of the current findings. For example, in depression research, the majority of studies have suggested that a higher level of depressive symptoms is associated with hypoactivation in prefrontal areas (including a large part of the DMN and FPN) and hyper-activation in limbic areas (Berman et al., 2011; Drevets et al., 2008; Goodman et al., 2021). These activation patterns are often interpreted as reflecting rumination and excessive self-referential thoughts (Nolen--Hoeksema et al., 2008; Zhou et al., 2020). However, a large number of the above findings were task-induced; thus, the selection of different types of experimental materials (e.g., picture vs. word) may matter (Sheline et al., 2009; Suzuki et al., 2014). For instance, many studies have consistently found an association between depression severity and DMN activity in word-based memory tasks (Holt et al., 2016; Toki et al., 2014; Whalley et al., 2012); however, previous results were more heterogeneous when using pictures as stimuli (Dillon and Pizzagalli, 2013; Schweizer et al., 2018; Werner et al., 2009). In our opinion, it is possible that the effect of depression on executive functions and emotion regulation manifests as aberrant DMN activity when word processing is involved in the ongoing task. This idea could be examined by directly comparing depressive responses to words and pictures (see Atchley et al., 2012, for behavioral findings), which would help determine whether interventions aimed at enhancing depressive individuals’ executive functions and/or emotion regulation processes should rely on verbal material (Kovacs et al., 2006).

Finally, we would like to list several limitations and future directions for the benefit of future research. First, follow-up studies are necessary to investigate whether common or dissociable neural networks are involved in explicit emotional processing of pictures and words (e.g., during emotion evaluation tasks). Second, the current analyses focused on the comparison between negative and neutral conditions. As pointed out in the Materials and Methods section, the number of available EST studies using positive words was too small for a meaningful meta-analysis; therefore, we did not compare brain activation patterns between positive words and pictures in this study. Consequently, it remains unclear whether positive pictures and words share the same neural underpinnings. This possibility is worth examining with alternative paradigms, as the processing of positive and negative stimuli shows different patterns in many aspects (e.g., Leppänen and Hietanen, 2004; Ohman et al., 2001). Finally, the EST is an experimental task based on visual perception, which may limit the generalizability of our conclusions. It is imperative to investigate the reliability of our findings with stimuli in other modalities (e.g., auditory and olfactory) (Blood and Zatorre, 2001; Ferdenzi et al., 2011; Fritz et al., 2009).