Combination Across Domains: An MEG Investigation into the Relationship between Mathematical, Pictorial, and Linguistic Processing

Douglas K Bemis; Liina Pylkkänen

doi:10.3389/fpsyg.2012.00583

. 2013 Jan 3;3:583. doi: 10.3389/fpsyg.2012.00583

Combination Across Domains: An MEG Investigation into the Relationship between Mathematical, Pictorial, and Linguistic Processing

Douglas K Bemis ^1,^*, Liina Pylkkänen ^1,²

PMCID: PMC3535734 PMID: 23293621

Abstract

Debates surrounding the evolution of language often hinge upon its relationship to cognition more generally and many investigations have attempted to demark the boundary between the two. Though results from these studies suggest that language may recruit domain-general mechanisms during certain types of complex processing, the domain-generality of basic combinatorial mechanisms that lie at the core of linguistic processing is still unknown. Our previous work (Bemis and Pylkkänen, 2011, 2012) used magnetoencephalography to isolate neural activity associated with the simple composition of an adjective and a noun (“red boat”) and found increased activity during this processing localized to the left anterior temporal lobe (lATL), ventro-medial prefrontal cortex (vmPFC), and left angular gyrus (lAG). The present study explores the domain-generality of these effects and their associated combinatorial mechanisms through two parallel non-linguistic combinatorial tasks designed to be as minimal and natural as the linguistic paradigm. In the first task, we used pictures of colored shapes to elicit combinatorial conceptual processing similar to that evoked by the linguistic expressions and find increased activity again localized to the vmPFC during combinatorial processing. This result suggests that a domain-general semantic combinatorial mechanism operates during basic linguistic composition, and that activity generated by its processing localizes to the vmPFC. In the second task, we recorded neural activity as subjects performed simple addition between two small numerals. Consistent with a wide array of recent results, we find no effects related to basic addition that coincide with our linguistic effects and instead find increased activity localized to the intraparietal sulcus. This result suggests that the scope of the previously identified linguistic effects is restricted to compositional operations and does not extend generally to all tasks that are merely similar in form.

Keywords: language, cognitive neuroscience, temporal lobe, domain-generality, combinatorics

Determining how language relates to cognition more generally is vital both for understanding how language works and how it came into being. Answers to questions regarding the origin of language in large part depend on the delineation between language specific and domain-general mechanisms (Hauser et al., 2002; Pinker and Jackendoff, 2005). An accurate determination of this line can help shed light on the magnitude and form of the evolutionary jump necessary to bring about linguistic ability (Marcus, 2006). Further, determining the relationship between language and cognition can help to calibrate the confidence with which findings from other domains and even species can be extended to language (Wise, 2003; McElree, 2006). If basic linguistic processes are utilized across many domains, this places restrictions on the form of the underlying mechanisms (i.e., it cannot be narrowly linguistic in nature). In the present experiment, we use magnetoencephalography (MEG) to assess the degree to which neural signatures previously shown to reflect basic linguistic composition, such as between an adjective and a noun, are also elicited by similar processing in non-linguistic domains.

While an abundance of evidence suggests that neural resources utilized during linguistic combinatorial processing are also active within non-linguistic domains, these studies have primarily investigated the processing of relatively complex linguistic structures that quite intuitively might recruit domain-general mechanisms such as executive control, attention, conflict resolution, and memory processes. For example, center embedding expressions, as in the juice that the child enjoyed stained the rug, are well known to reliably produce increased activity in the left inferior frontal cortex (Broca’s area) when compared to simpler constructions (Stromswold et al., 1996; Caplan et al., 2000; Grodzinsky and Santi, 2008). However, beyond basic linguistic composition, sentences of this type also require listeners to select between potential antecedents, modulate attention between locations in the sentence, and retrieve past words or phrases from memory. Non-linguistic tasks that tap into each of the mechanisms involved in comprehending such expressions – selection and cognitive control (Badre and Wagner, 2007), attention (Nelson et al., 2009), and memory (Fiebach et al., 2007) – have also been observed to generate increased activity in Broca’s area, along with a multitude of other tasks spanning a wide range of domains and abilities (Duncan and Owen, 2000; Thompson-Schill et al., 2005). Further, a direct comparison between the processing of syntactically ambiguous sentences and the resolution of stroop conflict identified coincident activity within this region (January et al., 2009). Other types of complex sentences, such as those that require the listener to resolve syntactic ambiguity in an unfamiliar manner, activate wide neuronal networks shared by non-linguistic processes, such as motor coordination, error monitoring, and response selection (Stowe et al., 2004). Within electrophysiological paradigms, ERP effects canonically elicited by various types of linguistic violations have been evoked within non-linguistic domains as well, such as music (Patel et al., 1998), pictures (Cohn et al., 2012), and using abstract geometric patterns (Besson and Macar, 1987). Thus, there is a large body of evidence indicating that language comprehension interacts with and relies upon domain-general operations at some level (though, see Fedorenko et al., 2011), however, it is presently unclear how deeply this interrelationship permeates linguistic processing. Specifically, it remains unknown to what extent basic linguistic combinatorial mechanisms that lie at the heart of language comprehension operate within non-linguistic domains as well.

In the present work, we use MEG to investigate the domain-generality of such operations, and we examine the extent to which simple combinatorial processing brought about by pictures and numbers evokes effects similar to those observed during basic linguistic composition. In a previous language paradigm (Bemis and Pylkkänen, 2011, 2012 and, see Figure 9 for a summary) we assessed linguistic combinatorial processing by identifying increases in neural activity evoked during the composition of an object-denoting noun with a color-denoting adjective (“red boat”) and compared this activity to that evoked by the same noun in a non-combinatorial context, such as when it was preceded by a non-pronounceable consonant string (“xhl boat”), a burst of auditory noise, or, critically, a different object-denoting noun, presented during a non-combinatorial task (“cup, boat”). Crucially, in this paradigm no judgments were required on the critical stimuli – subjects were only asked to comprehend naturally and then judge whether a following picture matched the preceding verbal description. We then only analyzed the activity generated during the critical nouns, which were identical across all conditions. During the composition of an adjective with a noun we found increased activity localized to the left anterior temporal lobe (lATL) at ∼200–250 ms and the ventro-medial prefrontal cortex (vmPFC) at ∼400 ms (Bemis and Pylkkänen, 2011), with a potential additional contribution from the left angular gyrus (lAG) at a similar later time (Bemis and Pylkkänen, 2012). All three of these regions have been implicated in combinatorial linguistic processing by a substantial number of studies (Bottini et al., 1994; Humphries et al., 2001; Vandenberghe et al., 2002; Mar, 2004; Lau et al., 2008; Fedorenko et al., 2011; Pallier et al., 2011), and thus these results provide a basis against which to judge the similarities between non-linguistic and linguistic combinatorial processing.

**Combined linguistic ROI results**. Activity generated during all tasks is shown for the three ROIs implicated in basic linguistic composition. Shaded regions denote significant clusters of increased activity during combinatorial conditions, as identified by a permutation test (Maris and Oostenveld, 2007). We observed no increased activity during basic addition in any of the ROIs. Combinatorial processing in the non-linguistic pictorial task evoked increased activity in the vmPFC, but not in either the left ATL or left AG.

To assess the extent to which these effects generalize to non-linguistic domains, we replaced the word stimuli of these prior experiments with pictures and numbers. Pictures were chosen for their similarity to words in activating conceptual representations. Intuitively, both comprehending sentences and viewing visual scenes are capable of eliciting complex conceptual representations in the mind of the comprehender or perceiver, though little is known about the neural overlap in the two routes taken by these processes from perceptual input to conceptual realization. In the present study, we paralleled our linguistic task and investigated only minimal combination, as brought about through the viewing of a single colored shape. If the composition of a simple phrase such as “red boat” involves a semantic combinatory mechanism that is not dependent on linguistic input, then a picture of a red boat should engage the same mechanism, assuming a task that demands on the construction of a combined “shape + color” concept. In contrast, a picture of a red boat should not give rise to any specifically syntactic combinatory mechanisms, such as the construction of a noun phrase, as such mechanisms should be dependent on syntactic category information absent in pictures. Thus, any shared combinatory effects between words and pictures would support a semantic as opposed to a syntactic role for the implicated region.

As a second non-linguistic task, we chose a simple arithmetic task in which the “combinatorial” operation was the addition of two small numbers. While this task echoes aspects of the linguistic design, such as minimality and the functional transformation of two input elements to a single output, theoretical models of addition (Dehaene et al., 2003) do not generally include specific combinatorial mechanisms found in models of linguistic composition (Heim and Kratzer, 1998), such as the construction of syntactic phrases or the creation of new meanings. Thus, if this task elicited effects similar to those found for linguistic composition, it would suggest that the underlying mechanisms are of the most general combinatory nature, only loosely related to linguistic notions of syntactic and semantic composition. Thus, in sum our picture experiment aimed to assess whether combinatorial mechanisms involved in the comprehension of basic adjective-noun phrases are also involved in the construction of domain-general meaning, and the math experiment is designed to examine the extent to which these mechanisms may support very general combinatory mechanisms that convert multiple inputs into a single output regardless of their domain. This investigation diverges from prior studies into the domain-generality of linguistic composition in its simplicity: in each task we explicitly manipulate only a single step of composition.

Experiment 1: Pictorial Combination

The purpose of the pictorial manipulation was to explore the extent to which basic combinatorial linguistic mechanisms are utilized for non-linguistic stimuli. While language is by far the most common method of constructing complex ideas (or meaning) out of simpler pieces, the processing of visual scenes (such as when viewing the external world) can also evoke similar mental representations (i.e., those that encode complex relationships between abstract concepts), and this cognitive process requires combination between individual conceptual elements as well. While there has been extensive investigation into the neural relationship between picture processing and lexical access during language production (Indefrey and Levelt, 2004), to date there has been little work that attempts to measure the construction of complex conceptual representations from pictorial stimuli. There has, however, been a large amount of work investigating perceptual combination, or feature binding, during visual object perception (Treisman and Gelade, 1980; Treisman, 1999; Quinlan, 2003), however, such processes are not the intended object of investigation for the current study. In general, this work has attempted to characterize the mechanisms by which perceptual features of a visual object are bound together into a single representational unit. In other words, when we perceive a picture of a red boat, we extract a percept of the color associated with the object (a particular shade of red) and a percept of the shape associated with the object (a particular type of boat). These percepts are thought to be initially extracted separately (Treisman and Gelade, 1980), and thus must later be established as belonging to the same object in the world. In general, feature binding refers to this establishment procedure. By contrast, when constructing the meaning of the linguistic expression red boat, we extract the concept of redness (not necessarily tied to a particular shade of red) and the concept of boatness (not necessarily tied to a particular type of boat), and then combine these separate concepts into an abstract representation of a single conceptual object. There are, of course, recent claims of synonymy between these two processes (Wu and Barsalou, 2009; Lynott and Connell, 2010), however, this equivalence has yet to be conclusively demonstrated empirically. With this distinction in mind, we have designed the present paradigm to maximize conceptual processing during picture viewing, as we believe it is this process that is most likely to occur during both language processing and combination evoked by pictorial stimuli. It may still be the case, however, that combinatorial effects observed during this latter type of processing reflect perceptual combination to some degree. To evaluate this possibility, in addition to the linguistic ROIs we have included a posterior region in the analysis as well that has been heavily tied to perceptual processing (the left mid fusiform gyrus, lMFG – Tarkiainen et al., 1999; Grill-Spector, 2003).

To date, there have been very few investigations into the extent that linguistic conceptual combinatorial mechanisms are utilized when processing pictorial stimuli. Perhaps the most relevant studies are those that have replicated the paradigmatic N400 effect using incongruent picture stimuli instead of sentence with incongruent endings (Nigam et al., 1992; West and Holcomb, 2002; Cohn et al., 2012). Unfortunately, it is difficult to draw a direct connection between this result and our previous linguistic findings as this paradigm differs from our own in several important respects. First, activity is measured during the comprehension of relatively complex meanings (i.e., descriptions of action sequences unfolding over time) and second, effects are associated with the processing of unexpected stimuli. In contrast, our linguistic paradigm was specifically designed to isolate minimal combinatorial processing in a normal context. Thus, in order to determine the extent to which our previous results might reflect domain-general mechanisms, we designed a parallel non-linguistic paradigm that sought to maintain these crucial qualities using pictorial stimuli.

Specifically, we aimed to evoke semantic combination that was as alike as possible to the natural, simple intersective modification required by the prior linguistic manipulation but using non-linguistic pictorial stimuli. To this end (Figure 1), we replaced adjective-noun combinations with colored pictures depicting the color and shape denoted by the adjective and noun (e.g., “red boat” was replaced by a picture of a red boat) while single noun stimuli were replaced by simple outlines (e.g., “xtp boat” was replaced by an outline of a boat). Subjects were then asked to judge if a following test picture matched the preceding stimuli in both shape and color (for colored shape trials) or shape alone (for outline trials). Importantly, to avoid simple perceptual matching between the test and critical pictures, and so force subjects to perform conceptual combination, different tokens of each shape were used for the two sets of items. Thus items qualifying as matching were unambiguously of the same conceptually category (e.g., a boat), however they did not have the identical shape. Further, critical and test items were given different scalings and rotations to enhance the differences in form. Consequently, subjects could not simply perceptually compare the two pictures to each other. In order to assess the correspondence between the critical and task pictures in colored shape trials, subjects had to create a combined, conceptual representation of the color and shape percepts evoked by the critical picture beyond the lower-level binding of these percepts to a single object. It is this conceptual combinatorial processing that we expect to be functionally similar to that evoked by a linguistic adjective-noun combination. Though it should be noted that the identification of perceptual combinatorial effects within linguistic ROIs would be informative with respect to linguistic compositional mechanisms as well (cf. Wu and Barsalou, 2009). Conceptual combination was minimized during outline trials by requiring only the shape of the two pictures to match and by removing all color from the task pictures, thus eliminating the potential use of conceptual combination in the task. Thus, in sum, the present design attempted to use non-linguistic stimuli to evoke the creation of a complex conceptual object constructed by modifying an initial shape property with a color property; in other words, the same combinatorial semantic operation involved in the composition of an adjective and a noun. Additionally, we sought to evoke this combinatorial processing through as natural a context as possible while still maintaining the minimal simplicity of the previous design. We expect conceptual combinatorial processing to occur only during the processing of the colored shapes while outline stimuli should not evoke such processing. Consequently, if any of our previously identified combinatory effects reflect domain-general semantic composition, we would expect a similar effect to be elicited for the color-shape combinations in the present task as well.

**Pictorial experimental design**. The trial structure (shown at left) was the same across both conditions. Subjects first saw a critical shape (either a colored picture or an outline on a colored background) and were asked to judge if a following test shape matched the initial stimulus. On Outline trials **(A)**, only the shape was required to match. On Colored Shape trials **(B)**, both the color and shape were required to match. Only activity evoked during the viewing of the critical stimuli was subject to analysis. Subjects saw an equal number of matching and non-matching trials in both conditions.

There is both a theoretical and practical reason that combination was evoked from this subtle manipulation and not through the sequentially presentation of two pictures (i.e., a color spot followed by a shape outline). First, we wanted to maintain the naturalness of the linguistic experiment to the greatest degree possible, and, in general, people view complex visual objects all at once and not as two sequential presentations. Second, pilot testing determined that subjects were much more likely to use an explicit verbalization strategy when presented with two sequential stimuli compared to a single picture. As the present study is designed to investigate conceptual combination in the pictorial domain, this finding played a substantial role in deciding to present a single unified picture and not a sequentially broken version. Additionally, at no point were subjects told to expect linguistic stimuli or was any reference made to the specific names of any object during the instructions.

Materials and methods

Participants

Twenty-four non-colorblind, fluent English speakers participated in the study (16 women). All had normal or corrected-to-normal vision. Their average age was 23.1 years (18–42 range). The protocol was approved by the Institutional Review Board, and subjects provided written consent before beginning the experiment.

Stimuli

Subjects were presented with two trial types, Colored Shape and Outline, each of which contained an initial fixation, a critical shape, and a test shape (Figure 1). In Colored Shape trials, a subset of the test shapes from our previous linguistic experiment served as the critical stimuli (the bags, bells, boats, bows, cars, cups, houses, keys, lamps, leaves, locks, notes, planes, shoes, and trees). These pictures were manually created canonical depictions of the shapes denoted by each noun and each was filled with one of six colors (red, blue, pink, black, green, brown). Additionally, three versions of each picture were created by applying a random scaling factor between 105% and 115% and a random rotation of 0°–360° to the original Figure. Critical pictures in the Outline trials were created from these stimuli by transferring the color from the middle of the shape outline to an area around its border using Photoshop. Thus, the total number of colored pixels remained equal between the two sets of stimuli. In total, two different tokens of each shape type were used, one for the test shapes and one for the critical stimuli. No token was used in both sets and tokens were used in the same set in each experiment. In the Colored Shape trials, test images were filled with color, while in Outline trials tests were presented without color. At the beginning of each trial, a small cross appeared and was centered in the middle of the screen.

During the task, subjects viewed 240 trials, 120 of each trial type, and both trial types were mixed together. Both conditions contained an equal number of trials for which the test shape matched or did not match the preceding shape. For Colored Shape trials, the non-matching trials were divided equally among those that did not match the color and those that did not match the shape. Additionally, in these trials all test shapes matched at least one feature of the preceding shape. During each condition, each of the 15 possible description shapes was used eight times (each time randomly paired with one of the six possible colors), four times in matching trials and four times in non-matching trials. Trials and stimulus lists were randomized and constructed separately for each subject.

Procedure

Before the experiment, subjects practiced the task outside of the MEG room. Prior to recording, subjects’ head shapes were digitized using a Polhemus Fastrak 3D digitizer (Polhemus, VT, USA). The digitized head shape was then used to constrain source localization during analysis by coregistering five coils located around the face with respect to the MEG sensors. During the experiment, subjects lay in a dimly lit, magnetically shielded room. All stimuli were presented using PsyScope X and were projected onto a screen ∼50 cm from the subject’s eye. Shapes subtended between 6° and 10°. Each trial began with the presentation of a fixation cross for 300 ms, a blank screen for 300 ms, and either a colored shape or an outline on a colored background for 500 ms. This picture was then followed by a 300-ms blank screen and the test picture, which remained onscreen until the subject’s response. Inter-trial intervals followed a normal distribution with a mean of 400 ms and a standard deviation of 100 ms.

Neuromagnetic fields were recorded continuously with a whole-head, 157-channel axial gradiometer array (Kanazawa Institute of Technology, Nonoichi, Japan) at a sampling rate of 1000 Hz in a band between 0 and 200 Hz, with a notch filter at 60 Hz. The entire recording session lasted ∼25 min.

Data acquisition

Magnetoencephalography data evoked by the initial shape, from 100 ms before to 600 ms after its onset, were segmented out for each subject for each condition. These data were cleaned of potential artifacts by rejecting trials for which the subject answered either incorrectly or too slowly (defined as over 2.5 s after the appearance of the colored shape) or for which the maximum amplitude exceeded a threshold that varied between 2500 and 3500 fT depending on the amplitude range of each subject. Overall 9.1% (3.5% std.) of trials were excluded due to behavioral error or artifacts. Remaining data were then averaged for each subject for each condition and band-pass filtered between 1 and 40 Hz. For inclusion in further analysis, we required that subjects show a qualitatively canonical profile of evoked responses during the processing of the critical items. This profile was defined as the appearance of robust and prominent initial sensory responses. In the visual modality, we required the presence of either the M100 or M170 field pattern (Tarkiainen et al., 1999; Pylkkänen and Marantz, 2003) in the time window of 100–200 ms following the onset of the critical stimuli. In order to assess this criterion, preliminary grand average waveforms were constructed for each subject by averaging over both conditions. Five subjects failed to meet this requirement and were excluded from further analysis.

Data analysis

In order to maximally parallel our previous linguistic paradigm, the analysis procedure in the present experiment was identical to that of Bemis and Pylkkänen (2011), which contains a more detailed description of the procedure and statistical tests used to analyze the data.

As in that study, distributed minimum norm source estimates served as our primary dependent measure in the present analysis. These were created for each condition average for each subject using BESA 5.1 (MEGIS Software GmbH). Each estimated resulted in 713 non-directional sources spread across a smoothed cortex for which electrical activity estimates could be compared between conditions at each time point. Our main analysis then attempted to identify significant effects in activity that localized to the regions of interest (ROIs) previously associated with linguistic combination – the lATL, vmPFC, and lAG. While the use of different subjects across experiments precludes a direct comparison of localized activity, we attempted to minimize differences by maintaining the exact boundaries of the ROIs as used in Bemis and Pylkkänen (2011, 2012). These boundaries were initially based upon hemodynamic results demonstrating increased activity in the lATL and lAG during the comprehension of sentences compared to word lists (Mazoyer et al., 1993; Friederici et al., 2000; Vandenberghe et al., 2002) and MEG results associating increased activity in the vmPFC with the resolution of semantic mismatches (Pylkkänen and McElree, 2007; Brennan and Pylkkänen, 2008, 2010). Additionally, we investigated potential differences in perceptual processing that might arise from the different types of visual stimuli used in the Colored Shape and Outline trials by drawing a broad ROI centered roughly around the lMFG. This region has been identified as integral to the perceptual processing of many types of complex visual stimuli (Tarkiainen et al., 1999; Grill-Spector, 2003), and so effects due to perceptual processing are likely to manifest in this ROI. Following our earlier study, the boundaries for this ROI were based on previous functional results and encompassed a rather broad cortical region in order to capture as much relevant activity as possible. Researchers wishing to reproduce this region can contact the corresponding author for exact dimensions regarding this ROI and all other ROIs used throughout the study. We also performed a supplemental full-brain analysis to identify differences in activity beyond these predefined ROIs.

Differences in activity localized to each ROI were identified using a non-parametric, cluster-based permutation test (Maris and Oostenveld, 2007) with the same parameters as in Bemis and Pylkkänen (2011, 2012). The primary advantage of this test is that it can determine time windows of significantly different activity across the entire epoch while controlling for multiple comparisons. Thus, we did not need to specify precise temporal ROIs a priori. Briefly, this test works by first calculating a single test statistic for the observed data and then comparing this value to a distribution bootstrapped from randomly permuting the original data. In this analysis, the test statistic was calculated by first determining contiguous time points for which the difference between conditions at each time point reached a certain statistical threshold (p = 0.30 by a two-tailed, paired samples t-test) and then summing the t-statistics from each point-wise comparison within the cluster. Permutations of the data were then created by randomly assigning the condition labels within each subject. The same test statistic was then calculated for each of these permutations, and the p-value of the observed test statistic was set equal to the proportion of permuted data sets that produced a larger test statistic. All p-values reported in the results were derived from distributions of 10,000 permutations of the original data.

The primary purpose of the full-brain analysis was to supplement the ROI findings and confirm that effects observed within an ROI occurred primarily within the predefined boundary. Additionally, we sought to obtain a broad overview of any combinatory effects that occurred outside of these regions. Thus, we compared activity estimates at each source-time point in the epoch using a two-tailed, paired samples t-test and then applied a liberal set of statistical, spatial, temporal, and amplitude filters to the results to eliminate spurious activity differences. The cutoffs for these filters were set at p = 0.05, five contiguous sources, five contiguous time points, and at least a 1 nAm difference between the activity in each condition, averaged across all subjects. In the results and figures below, we discuss only effects attributable to an increase in activity during the combinatorial Colored Shapes condition compared to the Outline condition.

Results

Behavioral results

Reaction time and accuracy data (Figure 2) were submitted to paired samples t-tests. We found no significant difference in accuracy between responses in Colored Shape trials (M = 96.9%, SD = 2.5%) and Outline trials (M = 97.3%, SD = 2.7%), t(18) = 0.98, p = 0.34 (two-tailed). We did, however, observe significantly faster responses in Colored Shape trials (M = 649 ms, SD = 147 ms) compared to Outline trials (M = 666 ms, SD = 135 ms), t(18) = 2.55, p = 0.02 (two-tailed). Past studies have demonstrated faster responses when subjects judge pictures against composed, complex linguistic descriptions as opposed to against simple, one-word descriptions (Potter and Faulconer, 1979; Bemis and Pylkkänen, 2011, 2012). Thus, the present result would be expected if linguistic composition caused this prior facilitation, and subjects performed a similar combinatorial operation in the current Colored Shape trials, as intended. Alternatively, responses on the Outline trials may have been slower due to a mismatch between the presence and absence of color information on the critical and test shapes respectively. This possibility is explored in more depth within the discussion below.

ROI results

Figure 3 displays the time course of activity localized to the four ROIs during the processing of the critical stimuli. Only activity localized to the vmPFC ROI exhibited the profile expected for a combinatorial mechanism. In this ROI, we identified a significant cluster (376–454 ms) of increased activity in the Colored Shape condition (M = 3.75 nAm, SD = 1.41 nAm) compared to the Outline condition (M = 2.96 nAm, SD = 1.25 nAm), p = 0.0475 (10,000 permutations). No significant clusters of increased activity in the Outline trials were identified at any point in the epoch for activity localized to the vmPFC (all clusters, p > 0.10).

On the other hand, we did identify clusters of increased activity during the Outline trials compared to the Colored Shape trials localized to the other three ROIs. This effect was only significant in the lAG ROI, where we identified a significant cluster from 110 to 460 ms of increased activity in the Outline condition (M = 3.09 nAm, SD = 2.01 nAm) compared to the Colored Shape condition (M = 2.24 nAm, SD = 1.26 nAm), p = 0.0003 (10,000 permutations). We identified marginally significant clusters in the lATL (132–206 ms, p = 0.0957, 10,000 permutations) and lMFG (110–205 ms, p = 0.0969, 10,000 permutations) with increased activity in the Outline trials (lATL: M = 4.62 nAm, SD = 1.82 nAm; lMFG: M = 5.50 nAm, SD = 3.07 nAm) compared to the Colored Shape trials (lATL: M = 3.83 nAm, SD = 1.59 nAm; lMFG: M = 4.37 nAm, SD = 1.93 nAm). We identified no significant clusters of increased activity in the Colored Shape trials compared to the Outline trials for the lATL, lAG, or lMFG ROIs (all clusters, p > 0.10).

Full-brain results

Our full-brain analysis (Figure 4) confirms the combinatorial results from the ROI analysis, as a clear increase in activity during Colored Shape processing can be seen localized to the vmPFC from 300 to 400 ms. Further, this effect is located entirely within the predefined vmPFC ROI. Also, as in the ROI analysis, there are no observable activity increases in the full-brain analysis localized to the LATL, AG, or lMFG.

Outside of the predefined ROIs, the only visible increase in activity occurred relatively early (∼200 ms) and was localized in the left superior parietal cortex. This region has long been implicated in the control of spatial attention, especially during the processing of visual stimuli that require attention to multiple features (Corbetta et al., 1995). Thus, this activity may reflect an increase in attention during the processing of the colored shapes, potentially driven by the need to initially bind the color and shape percepts into a complex visual representation prior to conceptual combination.

Discussion

The present experiment was designed to investigate whether the neural mechanisms that support basic linguistic composition operate during non-linguistically driven combination as well. Thus, our pictorial paradigm paralleled our previous linguistic design (Bemis and Pylkkänen, 2011), and we recorded neural activity as subjects viewed a description of a colored shape prior to judging whether a following test shape matched the preceding description. In this case, however, the descriptions were pictorial rather than linguistic, and subjects had to determine whether the two stimuli were of the same conceptual class (i.e., the same shape type and color). Importantly, subjects could not simply perceptually match the two shapes to each other as we used different tokens of each type for the description and test pictures. Thus subjects were compelled to construct a more abstract complex representation, similar to that elicited by the composition of an adjective and a noun. Combinatorial activity was isolated by comparing the processing during these colored shapes to that evoked by a shape outline on a colored background. To minimize combinatorial processing in this condition, subjects judged only if a following outline was of the same shape type. We identified increased activity during the combinatorial condition localized to one of the regions also implicated in basic linguistic composition, the vmPFC ROI. This result suggests that the combinatorial mechanism subserved by this region during basic linguistic combinatorial processing operates outside the linguistic domain and is involved in conceptual combination more generally.

While our previous study is not alone in linking the vmPFC to combinatorial language processing (Mar, 2004), this region has canonically been associated with a variety of non-linguistic tasks. Increased activity in this region has been observed during theory of mind tasks (Sabbagh et al., 2004), emotional processing (Northoff et al., 2000), error monitoring (Bayless et al., 2006), and the calculation of reward and punishments (O’Doherty et al., 2001). Thus, our finding that a non-linguistic task drives activity in this region is broadly consistent with a large body of evidence implicating this region in non-linguistic processing. Our results do however more closely align at least one of the functions subserved by this region with basic linguistic processing. Specifically, the results of Bemis and Pylkkänen (2011) combined with the present results suggest that the vmPFC supports a combinatorial mechanism that operates during both basic linguistic composition and conceptual combination evoked by simple colored pictures. As the latter processing does not involve any syntactic composition, these results suggest that the vmPFC supports the composition of semantic representations during both linguistic and non-linguistic processing; a conclusion consistent with past work implicating this region in the resolution of grammatically correct but semantically mismatched expressions (Pylkkänen and McElree, 2007; Brennan and Pylkkänen, 2008).

It should be noted, of course, that these results do not imply the vmPFC houses the sole neural mechanism responsible for combining meaning. Even within language alone, the composition of two concepts requires a large and diverse set of cognitive operations (Murphy, 1988; Kamp and Partee, 1995), and the general ability to combine concepts surely involves the interactive effort of multiple neural regions. Therefore, it is unlikely that damage to any one single neural region will entirely incapacitate this ability. However, there remains a tension between the interpretation of the present results and the neurophysiological literate that primarily links vmPFC damage to deficits in impulse control (Mesulam, 2002), social interactions (Bechara et al., 1994), and cognitive control (Fellows and Farah, 2003). In combination with the neuroimaging results cited above, these findings suggest that the vmPFC plays an important role in complex, goal-oriented behaviors as well basic combinatorial operations. Taken as a whole, the vmPFC encompasses a rather large cortical region, which is both cytoarchitecturally heterogeneous and widely interconnected to a variety of both subcortical and cortical regions (Hof et al., 1995; Ongür and Price, 2000). Thus, it is quite likely that distinct neural subpopulations within this region participate in several different cognitive operations. Given the widely acknowledged propensity of the brain to reorganize following damage (Chen et al., 2002; Saur et al., 2006), the relative lack of deficits observed for basic combinatorial processing compared to goal-directed behavior following damage to the vmPFC may suggest a less centralized and more redundant neural complex supporting the former compared to the latter. Further work is clearly needed to disentangle the interaction of functional heterogeneity and neural reorganization following damage within this region. In particular, we believe it would be highly relevant to our interpretation to observe changes in the activity profile for both the present manipulation and our previous linguistic paradigm for patients with damage to the vmPFC, particularly across different time periods following trauma, in order to chart reorganization as a function of time (cf. Saur et al., 2006).

Despite this outstanding puzzle, our results suggest that non-linguistic conceptual combination recruits neural mechanisms that are active during basic linguistic composition as well. One consideration in drawing this conclusion is that combinatorial effects observed in the present contrast might reflect perceptual as opposed to conceptual combination. This possibility cannot be ruled out entirely from the present design, and we did, in fact, identify increased activity in the combinatorial condition localized to the superior parietal cortex, which has been linked to directing spatial attention and visual feature binding (Corbetta et al., 1995). However, this effect occurred earlier than that observed in the vmPFC, falling within the time frame canonically associated with visual processing (Tarkiainen et al., 1999). Contrastingly, neither the timing nor the location of the observed combinatorial effect – in the vmPFC at ∼400 ms – match the spatio-temporal profile of canonical visual effects, which occur earlier and in more posterior regions (Tarkiainen et al., 1999; Grill-Spector, 2003). Such an effect was instead observed in the Outline condition for activity localized to the lMFG from ∼100 to 200 ms. We also observed increased activity in the lAG and lATL ROIs during this condition, as well as slower reaction times at the test shape. One possibility is that this entire constellation of effects is related to the need to suppress color information, present within the critical shapes, during the decision at the test shapes, which do not contain color. Under this hypothesis, increased activity observed in the lMFG might reflect increased perceptual processing required to separate figure from ground (Schira et al., 2004), and activity localized to the lATL and lAG, both regions previously linked to single word lexical processing as well as combinatorial linguistic tasks (Horwitz et al., 1998; Mummery et al., 1999; Binder et al., 2000; Fujimaki et al., 2009), including covert object naming (Ellis et al., 2006), may reflect the enlistment of a verbal strategy in maintaining a separation between the color and shape information (cf. Sperling, 1963). While such extra processing during the Outline trials was not intended, it does suggest that Outlines, and not Colored Shapes, evoked more activity related to perceptual processing, and therefore the observed effect in the vmPFC is more likely to reflect conceptual as opposed to perceptual combinatorial processes.

There is the danger, of course, that our paradigm erred too far toward paralleling our previous linguistic experiment and simply evoked linguistic combination as before, through the covert use of verbalization to solve the task. Though informal interviews following the recording session suggested that subjects were not aware of such verbalization, introspective reports can hardly be taken as definitive evidence against this possibility (Dodge, 1912; Schwitzgebel, 2008). Likewise, while the absence of an lATL effect suggests a dissociation in processing compared to the previous experiment, this null result is at best suggestive and may reflect either a loss of power or more variability in the data. Thus, the present results do not entirely rule out the possibility that subjects relied upon a verbal strategy to complete the combinatorial pictorial task. It must be noted, however, that the question of whether or not subjects used a “verbal” or “non-verbal” strategy during the combinatorial task must be posed with care as it is, in fact, precisely the fusion of these two notions that we are interested in investigating, or, more exactly, the point at which they fuse together. Based on our results, we are suggesting that this point occurs during the construction of complex semantic representations. Specifically, we are suggesting that during non-linguistic conceptual combination, as brought about in the present task, a pictorial description activates two conceptual representations, such as those that encode red and boat, and that their subsequent combination makes use of a neural mechanism also operational during the linguistic composition of an adjective and a noun. This latter process, in highly simplified terms, involves the recognition of a linguistic item (such as a visually presented word), followed by the retrieval of information from the lexicon, followed by the activation of conceptual representations that are again bound together, in conjunction with syntactic structures, through the use of the same combinatorial mechanism. Thus, at a certain point, in this case conceptual combination, the “verbal” and “non-verbal” strategies become synonymous.

Two caveats to this suggested interpretation should be mentioned. First, though our results suggest a shared domain-general combinatorial process between linguistic and non-linguistic processing, the present data remains equivocal on the exact nature of the shared mechanism. While we are suggesting that the junction between linguistic and non-linguistic processing streams occurs at semantic composition, the point of contact may in fact occur earlier. For instance, the pictorial representations in the combinatorial task may activate lexical precursors to semantic composition, thus sharing the linguistic combinatorial route from this point forward. Further work is needed in order to more precisely isolate the moment of contact. Second, while the present results suggest the engagement of a linguistic combinatorial mechanism, they do not speak directly to whether or not this engagement is essential for completing the non-linguistic task. In other words, though our results indicate that a non-linguistic combinatorial task initiates the involvement of a linguistic combinatorial mechanism, the present paradigm cannot determine if this involvement is necessary for the completion of the task and thus whether the initiated process constitutes a truly domain-general mechanism, i.e., one that is indispensible for performing both the non-linguistic and linguistic task. One popular approach for assessing the necessity of linguistic processing is through the use of concurrent verbal shadowing (e.g., Hermer-Vazquez et al., 1999). Unfortunately, at the present time not only is it unclear how neural activity generated by shadowing can adequately be differentiated from that of the main task but, again, we are interested in identifying a point of contact between linguistic and non-linguistic processing. Thus, the introduction of extra, explicitly linguistic processing, even if intended to be entirely non-combinatorial, will only complicate this determination.

Aside from the central question of shared combinatorial processing, there are also questions, which plague many psychological experiments, about the relationship between the processing evoked during the experimental task and that which occurs “in the wild.” For example, it is unclear if conceptual combination, of the type we intended to study, occurs whenever a visual object is attended to or if it is task-dependent. Further, it remains to be seen if the effects observed in the present paradigm persist during the processing of more complex visual scenes in more natural circumstances. Efforts in this direction are currently ongoing in both the linguistic (Brennan et al., 2012) and pictorial domain (Hasson et al., 2010), though a direct comparison between the two has yet to be performed. Thus, the results of the present experiment suggest that a common combinatorial neural mechanism lies at the heart of both basic linguistic and non-linguistic combination. Much work, however, remains in elucidating the precise nature of this mechanism and the extent to which these two processing streams coincide.

Experiment 2: Mathematical Combination

In contrast to the picture experiment, which was designed to test a rather plausible hypothesis about shared semantic composition between language and pictures, our math experiment was designed to constrain the functional hypothesis space open to our previous findings, namely that they reflect highly general combinatory mechanisms engaged every time any computation needs to be performed on two symbols. Specifically, we sought to construct a paradigm similar in form to the previous linguistic task, but sufficiently different in function so as not to evoke the same combinatorial mechanisms. In other words, we desired to measure activity generated by an operation that converts two simple inputs into a single output, but that does not plausibly require the same compositional operations involved in basic linguistic combinatorics, such as syntactic or semantic combination. A negative result from such a task would support the hypothesis that the effects observed previously during linguistic composition reflect these more specific combinatorial mechanisms and are not always elicited by tasks that are merely similar in structure.

To this end, we chose a manipulation involving basic mathematical processing. Despite tantalizing similarities between the structure of language and mathematics, there is a growing body of evidence that suggests these two types of processing employ largely distinct mechanisms, especially in terms of combinatorial mechanisms. The superficial similarities between the two, however, are quite noticeable and have driven extensive investigation into the relationship between these two modalities. Like linguistic expressions, mathematical expressions exhibit hierarchical structural relations between basic elements, controlled by well-formedness rules (Friedrich and Friederici, 2009). Indeed, the most prominent mathematical processing model (Dehaene et al., 2003) posits a non-negligible overlap between linguistic and numerical processing. In this “triple-code” model, one of the three hypothesized codes underlying mathematical processing is an explicitly verbal representation of number. In support of this proposal, Dehaene et al. (2003) point out that certain types of mathematical deficits are often accompanied by aphasia (Dehaene and Cohen, 1997; Cohen et al., 2000). Additionally, the angular gyrus, often implicated in linguistic processing (Horwitz et al., 1998; Binder et al., 2000) including our own composition work (Bemis and Pylkkänen, 2012), shows increased activity during many types of mathematical processing (Dehaene et al., 1999; Fulbright et al., 2000). Finally, behavioral studies indicate that concurrent language tasks interfere with basic mathematical operations (Spelke and Tsivkin, 2001; Lee and Kang, 2002). Crucially however, the role of language in these mathematical tasks is not assumed to be combinatorial in nature. Instead, it is posited to reflect the retrieval of rote learned declarative facts through verbal associations (Dehaene et al., 2003). Linguistically-related effects have been observed primarily in mathematical tasks for which the answer has been learned and can be recalled without resorting to calculations involving magnitude (e.g., multiplication with small numbers). Such effects are absent or at least much reduced during similar types of operations for which the answer is not assumed to have been learned, such as simple subtraction problems (Chochon et al., 1999; Lee, 2000). Further, increased activity in traditionally linguistic cortical regions, such as the angular gyrus, has been shown to track the extent to which mathematical answers have been learned, showing more activity for exact compared to approximate problems (Dehaene et al., 1999) and smaller compared to larger sums (Stanescu-Cosson et al., 2000). Thus, despite the potentially integral role that linguistic representations play in certain types of mathematical processing, their hypothesized function is entirely distinct from combinatorial processing as employed during the comprehension of linguistic phrases.

This hypothesis has been put to the test on several recent occasions by attempting to produce canonical linguistic combinatorial effects using complex mathematical tasks, leveraging the potential hierarchical structural similarities between the two domains. To date, these attempts have failed to find effects during the processing of “combinatorial” mathematical expressions. Martín-Loeches et al. (2006) constructed “syntactic” violations parallel in structure to linguistic violation paradigms. Following Dehaene and Cohen (1995), they roughly equated mathematical operators (e.g., “+”) with function words and found that subsequent violations in which these operators appeared in syntactically incorrect locations failed to produce the anterior negativities that similar word-category violations do (Friederici et al., 1993), instead resulting in posterior parietal negativities. Similar results were obtained when examining ERP responses to mathematical parentheses, hypothesized to introduce “relative clause” like expectations (i.e., embedded combinatorial processing). An fMRI study of hierarchical mathematical expressions, compared to structurally flat ones, also failed to replicate effects found in parallel linguistic manipulations (Friedrich and Friederici, 2009). Finally, Varley et al. (2005) present patient data for which linguistic comprehension is severely impaired, while mathematical computation is preserved. These patients are able to correctly answer mathematical problems that depended on order (60–90 vs. 90–60) while not being able to distinguish parallel linguistic expressions (“John hit Mary” vs. “Mary hit John”) and can solve problems involving embedding [36/(3 × 2)] while not being able to comprehend embedded linguistic expressions (“The reporter that attacked the senator admitted the error”). Thus, to date, despite concerted efforts to show otherwise, there is no evidence that mathematical processing employs combinatorial mechanisms similar to those used in linguistic comprehension. It appears as though, despite superficial similarities in structural complexity, different processing mechanisms underlie the comprehension and parsing of complex mathematical and linguistic expressions.

Thus, in order to explore the bounds of the effects observed in our combinatorial linguistic manipulation, we designed a parallel mathematical task that required the simple addition of two small numbers (Figure 5). Though similar in structure to basic linguistic combination, this task should be functionally distinct, at least in terms of combinatorial processing. To maximize the structural parallelism, the mathematical design replicated the linguistic design in full, which included not only a comparison between one and two-word trials (red boat vs. xtp boat) in a task invoking composition, but also a control “list task” that served as a lexical-semantic control. In this control task subjects read lists of one or two nouns (xtp boat vs. cup, boat) and judged whether a subsequent picture matched any previously presented noun. Mirroring this structure, our math design was split into two complementary tasks, each of which contained two and one-element trials. In each task, subjects were shown either two sequential, small (1–5) Arabic numerals or a visually matched meaningless symbol followed by a single digit. In the critical addition task, subjects were asked to judge whether or not a following set of dots was equal to the sum of the preceding numbers. In the control list task, subjects were asked to judge if the set of dots was equal to any of the preceding numbers. As can been seen, these two tasks follow the general form of the two linguistic tasks, and in both cases the critical trials require subjects to perform an operation requiring two operands (either an adjective and a noun or two-numerals) that were presented one after the other. Further, both the linguistic and mathematical operations are simple and basic, with no judgment required at the presentation of the critical stimuli. Therefore, if the effects observed during the linguistic paradigm reflect general mechanisms that operate during any process that maps two input elements onto a single output, then we would expect to see similar effects in the present paradigm. For example, previous effects may reflect either the retrieval or encoding of a single object in memory with multiple elements or cues. During combination, various levels of representation of red and boat may be associated with the resulting complex representation of a red boat just as representations of 2 and 3 may be associated in memory with their resulting sum. No such single object is as easily associable with the separate elements in the list conditions. Further, in the addition condition the sum of the two-numerals may in part be retrieved from memory through the mutual use of both addends, 2 and 3, as cues (Dehaene et al., 2003) just as information related to a composed complex meaning may be retrieved from memory during language comprehension through the use of both linguistic elements. Again, no corresponding memory operation would be expected during processing in the list controls. On the other hand, if our previously observed linguistic effects reflect more specific combinatorial mechanisms, such as those required to construct complex syntactic or semantic representations, then we would not expect to observe the same effects in the present paradigm.

**Mathematical experimental design**. The trial structure (shown at top) was the same across all conditions. Subjects saw an initial stimulus (either a numeral, 1–5, or a non-sense symbol), a critical numeral (a numeral 1–5), and a test picture (a small set of dots). In the addition task **(A)**, they were asked to judge if a following set of dots was equal to the sum of all preceding numbers. In the list task **(B)**, they were asked to judge if the following set of dots matched any of the numbers. Only activity evoked during the viewing of the critical numerals was subject to analysis. Subjects saw an equal number of matching and non-matching trials in all conditions.

If this were to be our finding, then it would be desirable to obtain a positive result as well to demonstrate that the present paradigm does not lack the power to find effects related to basic addition. It is somewhat unclear, however, exactly what effects we should expect to see during simple addition. Within the triple-code model, basic addition as employed in our design is hypothesized to be solved either through the recall of memorized verbal facts or through calculation using the magnitude of the presented numbers (Dehaene et al., 2003). A wide range of data, however, indicates that mathematical calculation more generally produces increased activity in the intraparietal sulcus (IPS) with strong regularity (Chochon et al., 1999; Pesenti et al., 2000; Dehaene et al., 2004; Roitman et al., 2012). Therefore, as a sanity check for our task and to avoid a null result as the sole prediction from this manipulation, we included the IPS as a region of interest during the analysis of the mathematical data.

As a final note, it should be emphasized that while our design has the ability to assess whether or not previous effects observed in the lATL, vmPFC, and lAG during basic linguistic composition reflect the general form of our previous task (as opposed to combinatorial mechanisms) it does not, of course, have the ability to assess the domain-generality of all linguistic combinatorial mechanisms in relation to mathematics. While past evidence suggests that basic linguistic combinatorial mechanisms are not likely to be involved in basic addition, it may well be the case that shared mechanism exist at a more complex level of operation. Thus, a negative result with respect to the specific linguistic ROIs analyzed in the present experiment cannot be taken to imply a complete functional separation between linguistic and mathematical operations.