Mapping mental function to brain structure: How can cognitive neuroimaging succeed?

Abstract

The goal of cognitive neuroscience is to identify the mapping between brain function and mental processing. In this paper, I examine the strategies that have been used to identify such mappings, and argue that they may be fundamentally unable to identify selective structure-function mappings. I argue that in order to understand the functional anatomy of mental processes, it will be necessary to move from the brain mapping strategies that the field has employed towards a search for selective associations. This will require a greater focus on the structure of cognitive processes, which can be achieved through the development of formal ontologies that describe the structure of mental processes. I outline the Cognitive Atlas project, which is developing such ontologies, and show how this knowledge could be used in conjunction with data mining approaches to more directly relate mental processes and brain function.

1 Introduction

Imagine that fMRI had been invented in the 1860s rather than the 1990’s¹. Instead of being based on modern cognitive psychology, neuroimaging would instead be based on the faculty psychology of Thomas Reid and Dugald Steward, which provided the mental “faculties” that Gall and the phrenologists attempted to map onto the brain. Researchers would have presumably jumped from phrenology to fMRI and performed experiments manipulating the engagement of particular mental faculties, or examining individual differences in the strength of the faculties. They almost certainly would have found brain regions that were reliably engaged when a particular faculty was engaged, and potentially would also have found regions whose activity correlated with the strength of each faculty across subjects. In support of this assertion, Table 1 provides a demonstration of some modern neuroimaging data that the intrepid post-phrenologist might have appealed to in order to demonstrate the neural reality of his proposed faculties.

Table 1.

A mapping of Galls 27 faculties (from Whye, 2004) to potentially related neuroimaging results

Faculty	Modern equivalent for neuroimaging	Regions implicated	Reference
Impulse to propagation	Viewing of romantic lover versus other individuals	Basal ganglia	Aron et al. (2005)
Tenderness for the off-spring, or parental love	Mothers viewing own vs. other child	Amygdala, insula, anterior cingulate, superior temporal gyrus	Leibenluft et al. (2004)
friendly attachment or fidelity	Viewing friend vs. stranger	R temporoparietal cortex	Sugiura et al. (2005)
valour, self-defense	Punishment of defectors in economic games	Dorsal striatum	de Quervain et al. (2004)
murder, carnivorousness	Less active in murderers	Prefrontal cortex	Raine et al. (1994)
sense of cunning
larceny, sense of property	Activated in relation to hoarding behavior in OCD	left precentral gyrus and right orbitofrontal cortex	Mataix-Cols et al. (2004)
pride, arrogance, love of authority	related to arrogance scores	Prefrontal cortex	Yang et al. (2005)
ambition and vanity	Activation for judgment about self versus others	Medial PFC	Ochsner et al. (2005)
circumspection	Activation correlated with harm avoidance	Nucleus accumbens	Matthews et al. (2004)
aptness to receive an education, or the memoria realis	activation during reasoning tasks correlated with general intelligence	Parietal cortex	Lee et al. (2006)
sense of locality	Scenes vs. nonscenes	Parahippocampal cortex	Epstein & Kanwisher (1998)
recollection of persons	activated by judgments about face identity vs. occupation	Fusiform gyrus	Turk et al. (2005)
faculty for words, verbal memory	Use of memory strategies	Prefrontal cortex, extrastriate visual cortex	Kirchhoff & Buckner (2006)
faculty of language
disposition for colouring, and the delighting in colours	Greater activity in grapheme-color synesthetes	V4	Hubbard et al. (2005)
sense for sounds, musical talent	activation in MEG and gray matter volume correlated with musical aptitude	Auditory cortex	Schneider et al. (2002)
arithmetic, counting, time	activity correlated with arithmetic skill	Angular gyrus	Menon et al. (2000)
mechanical skill	greater activity for observing actions in skilled vs. unskilled groups	Left premotor, intraparietal, superior temporal	Calvo-Merino et al. (2005)
comparative perspicuity, sagacity
metaphysical perspicuity
wit, causality, sense of inference	more active for viewing causal vs. non-causal events	MT, STS, IPS	Blakemore et al. (2001)
poetic talent	generation of creative vs. uncreative narrative	Right medial frontal	Howard-Jones et al. (2005)
Good-nature, compassion, moral sense	Judging personal versus impersonal moral dilemmas	Medial prefrontal, posterior cingulate, angular gyrus	Greene et al. (2001)

Although few today would hold that nineteenth century faculty psychology is an accurate description of the structure of the mind, we can likely all agree that if the phrenologists had created task manipulations to isolate their proposed faculties using fMRI, something would have “lit up”. What would the patterns of activation associated with these faculties have looked like? If we believe, as I think most would agree, that each of the phrenologists’ putative faculties relies in actuality upon a combination of basic mental operations, then we would likely expect that the maps obtained for a given faculty would include a large set of activated regions that would tend to overlap across tasks meant to tap into different faculties. Regardless, one can be almost certain that Gall and his contemporaries would have taken these neuroimaging results as evidence for the biological reality of his proposed faculties.

The point of this example is not to appeal to a specious analogy between phrenology and neuroimaging, but rather to point towards a more fundamental issue. Neuroimaging studies rely upon a theory about the structure of the mind that specifies the component operations that comprise mental function, which I will refer to as a cognitive ontology (Bilder et al., 2009; Price & Friston, 2005). This ontology describes the “parts” of the mind, which in the end are the things that cognitive neuroimaging aims to map onto brain structure, just as biologists map cellular functions (e.g., translation) onto cellular structures (e.g., ribosomes). So long as the assumed ontology is at somewhat correlated with the true ontology, consistent structure-function mappings can be found, but these do not imply that the underlying ontology is correct. Instead, correctness of the ontology would be reflected in selective association between structures and functions. That is, if a specific structure or network is activated in association only one putative cognitive process, then one could argue that the reality of this process has been established.

A review of the neuroimaging literature suggests that selective association between mental processes and brain structures is currently impossible to find. Although popular accounts often imply unique structure-function mappings (e.g., the amygdala is the “fear area”, the anterior cingulate is the “conflict area”), closer examination of nearly every such claim uncovers counterexamples that are difficult to reconcile with a selective structure-function mapping. There are a number of possible reasons for this lack of selective mapping. First, the underlying ontology may be incorrect. For example, while we think that “working memory” is a unique function implemented in the brain, it may be the case that there is no such function implemented by the brain, and that what we call “working memory” is in reality a combination of some other functions. Second, it may be that the cognitive ontology is correct, but that the studies are not properly isolating the basic operations (i.e., the mapping from task manipulations to mental processes is incorrect). Third, it may be that the assumption that mental functions can be mapped to individual brain structures is incorrect, such that there is selective mapping but it occurs at the level of networks rather than individual structures.

The goal of this paper is to examine a set of questions that arise from a consideration of these possibilities. First, I will ask whether current research strategies may be problematic for the identification of selective associations even if they exist. Second, I will discuss the issue of cognitive ontologies, highlighting the need for a more formal approach to mapping of mental processes to brain structures. I will not directly address the question of localization in regions versus networks; it is a very important issue, but it has been addressed in detail by previous authors (e.g., McIntosh, 2000).

2 Neuroimaging research strategies

The most obvious strategy within cognitive neuroimaging is what one might call the where strategy:

1. Design a manipulation that is thought to modulate the engagement of some particular mental process.

2. Analyze neuroimaging data to identify regions whose activity is modulated by this manipulation.

3. Conclude that the active regions are involved in the manipulated process.

This was a common strategy in early stages of neuroimaging; for example, the early studies by (Petersen et al., 1988) determined that semantic processing relied upon the left inferior prefrontal cortex using subtraction of word repetition from verb generation. This approach is often disparaged as “blobology” or “neo-phrenology”, though it’s not actually clear what other approach one might use to bootstrap a new research enterprise.

As neuroimaging has matured, the where strategy has given way to what one might call the what strategy, which focuses more directly on characterizing the function of a specific brain region:

1. Design a task that independently manipulates two or more different mental processes, one of which is hypothesized to be performed by some particular region.

2. Examine the the imaging data to identify the relative response of the region in question to these manipulations.

3. Conclude that the region in question performs a particular one of the manipulated processes.

This approach reflects an incremental approach to reverse engineering of the function of individual brain regions. For example, a number of studies in the last ten years have examined the role of the left inferior prefrontal cortex in language processing. Early work suggested that it played a role in the retrieval of knowledge from semantic memory (Demb et al., 1995). However, subsequent work proposed that instead of performing retrieval, this region was involved in the selection of task-relevant information across both semantic and non-semantic domains (Thompson-Schill et al., 1997). This hypothesis was later disconfirmed by Wagner et al. (2001), who showed that the region was involved in semantic retrieval even when selection demands were help constant.

This approach has also in some cases led to what one might call the fractionation strategy:

1. Design a task that independently manipulates two or more different mental processes.

2. Identify the regions that are separately engaged by those different processes.

3. Conclude that the processes are performed by different regions.

For example, a number of early neuroimaging studies examined the distinction between processing of word meaning (semantic processing) and processing of word sounds (phonological processing). Based on an fMRI study that directly compared manipulations of these two types of processing, along with a meta-analysis of previous studies,Poldrack et al. (1999) proposed that semantic processing relied upon a more anterior portion of the inferior frontal gyrus, whereas phonological processing relied upon a more posterior portion of the gyrus. This distinction has been extensively replicated, and also extended. In particular, there is a region in the middle of the left inferior frontal gyrus, in between the regions engaged by semantic and phonological processing, that appears to play a different role from these other regions. One set of findings has highlighted the role of this mid-LIFG region in syntactic processing; a number of studies (reviewed by Bookheimer, 2002) have shown activation in this region for manipulations of the complexity of syntactic processing. Another result suggests that this mid-LIFG region may implement the selection operations proposed by Thompson-Schill et al. (1997). Badre et al. (2005) used a set of converging manipulations along with factor analysis to determine the task factors that modulated activation in the left inferior frontal gyrus. They found that whereas the anterior/inferior portion of the LIFG was sensitive primarily to semantic retrieval demands, the middle portion of the LIFG was sensitive to a factor that indexed the need for selection amongst competing alternatives.

These approaches have led to increasingly sophisticated functional characterizations of specific anatomical regions within relatively limited domains. However, in many cases the same region may be characterized in this way across multiple very different domains. For example, one set of studies has implicated the posterior portion of the left inferior frontal cortex in a more general process of temporal sequencing (Gelfand & Bookheimer, 2003). Yet another set of studies has suggested that this region forms part of a “mirror network” that is involved in the production and recognition of actions (Iacoboni et al., 1999; Nishitani et al., 2005). Each of these provides an explanation for some of the language phenomena that have been previously associated with left inferior frontal gyrus activity, but integrating all of these findings into a single theory is challenging.

It’s instructive to project forward and think about what the ultimate result would be from several decades of science using the current approach. It is tempting to conclude that this approach would help us learn “what each brain area does”, but the reality may be somewhat less informative. In particular, while this approach is likely to uncover a broad set of functions that rely upon each region, it is unlikely to identify a fundamental functional role in mental activity for a particular region (e.g., the basic computations that each region performs). As an analogy, imagine a group of people individually trying to understand the function of a knife blade. One person tests its ability to cut peaches. Upon finding that the blade cuts through peach flesh but not through the pit, they conclude that the knife is specialized for peach flesh removal. Another person might test its ability to screw various types of screws; finding that the knife blade works well to screw flathead and Phillips screws but not allen screws, they might conclude that it is specialized for a subset of screwing functions. While each of these is a valid description of the functions that the knife participates in, neither seems to be an accurate description of the fundamental function of a knife blade, which might be described as something like “either cutting or manipulating objects depending upon their hardness.”

3 Mining for functional characterization

How might we better zero in on the functional roles of individual brain regions? Answering this question requires that we determine which cognitive functions are associated with activity in each particular region. This in turn requires that we have a specified set of cognitive functions that can be mapped onto regions (which we refer to as a cognitive ontology), and that we know which task manipulations are associated with each of these functions. Once we have this ontology and its mappings to experimental manipulations, we can then examine imaging data to determine which functions each brain region or network is associated with. We have recent provided a proof of concept for this approach (Poldrack et al., 2009).

In this study, we performed meta-analysis on a data set that included fMRI data from subjects performing one of eight different tasks, which ranged very widely (e.g., from auditory working memory to reading words aloud to gambling judgments); for each subject, a single statistical map comparing the task versus rest/fixation was used in the analysis. We first developed a relatively coarse ontology of mental processes involved in these studies, which was based on the BrainMap behavioral domains framework with some additions. We then coded each task with regard to whether it engaged each of those particular mental processes (versus resting/fixation). With this mapping of tasks to processes, we could then map neuroimaging data from those tasks into a representation of how strongly each cognitive process was associated with engagement of a particular region or network. In Poldrack et al. (2009), we focused on mapping these concepts onto a set of six dimensions obtained using a neural network classifier for dimensionality reduction. These networks were associated with different sets of cognitive processes in ways that seemed sensible based on the existing literature. However, it is also possible to examine the mapping of activity from individual regions into the ontology space as well. Figure 1 shows tag cloud representations of the concepts that are associated with activity in several different anatomical regions. This shows expected patterns for a number of regions (e.g., primary auditory and visual cortices), but somewhat unexpected patterns for other regions (e.g., prefrontal regions). This most likely represents the very small size of the dataset, and the fact that individual cognitive processes are not well isolated across these eight tasks. Notwithstanding these issues, the analysis provides a proof of concept for the mapping of mental processes, rather than task manipulations, to specific regions.

Figure 1.

Loading of several individual regions onto mental concepts was computed by projecting the average activity for each of the 8 tasks in the Poldrack et al. (2009) dataset onto a matrix describing the relation between tasks and mental concepts. Strength of loading is depicted using a tag cloud, with the size of the term representing the strength of the loading and the color of the term representing positive [red] or negative [blue] loading. The upper panels depict reasonable loadings of basic processes onto cortical regions, given the known function of those regions. The bottom left panel shows predictable negative signals associated with decision making in the medial prefrontal region, which is known to exhibit deactivation across a wide range of cognitive tasks (Gusnard et al., 2001). The bottom right shows the pattern for right inferior frontal operculum, which shows an unexpected pattern with strongest loading on vision.

4 Towards a formal cognitive ontology of the mind: The Cognitive Atlas

In the analyses described above, we used an admittedly coarse and incomplete description of the mental processes that are associated with each task comparison. However, the successful use of this kind of analysis will require a comprehensive formally-specified ontology of mental processes. In other areas of bioscience the use of formal ontologies has grown rapidly, and these resources have provided the basis for a new generation of discovery tools (Bard & Rhee, 2004). Most prominent has been the Gene Ontology (Ashburner et al., 2000), which comprises a formal specification of gene and gene product function, including cellular components, biological processes, and molecular functions (accessible at http://www.geneontology.org). This ontology is used to annotate data, which involves specifying a relation between some particular gene product (e.g., CamKII) and a specific biological process (e.g., synaptic transmission) on the basis of some particular kind of evidence (e.g., from a mutant mouse phenotype). Fundamentally, an annotation amounts to a formal specification of the kinds of results that would usually be written into a paper. However, because the individual aspects of the annotation are specified in a formal knowledge base, they can be more readily used in meta-analysis.

Unfortunately, there is no existing formal ontology of mental processes. To remedy this, we have undertaken a project, called the Cognitive Atlas (http://www.cognitiveatlas.org), that aims to develop a comprehensive and current ontology for mental processes. The goal of the project is to provide a knowledge base that describes the “parts” and processes of the mind, just as the Gene Ontology describes the component parts and functions of a cell. Doing this in the context of psychology is substantially more difficult than in the context of biology; in particular, whereas most biologists agree in large part on the ontology of the cell, there are few psychological processes or entities whose existence is uncontroversial. The goal of the Cognitive Atlas is to allow the representation of these concepts in a way that captures the disagreement that is certain to occur among cognitive scientists as they discuss the structure of mental processes.

A schematic overview of the Cognitive Atlas is shown in Figure 2. A specific goal of the Cognitive Atlas is to allow the annotation of behavioral and/or neural data according to the mental processes that are thought to be engaged. Thus, the top level of representation in the knowledge base is that of mental concept, which refers to any concept that describes a mental function, structure, or process. The knowledge base allows users to define concepts, associate them with publications, and specify relations between concepts. For example, a user might enter a definition and a citation for the concept “working memory”, and then specify that working memory is a kind of memory. The next level of representation is that of task, which describes a manipulation that is performed in order to manipulate some mental process. For example, a user might describe the “Sternberg item recognition task” and provide a citation for the task. However, tasks are not directly associated with mental concepts; rather, mental concepts are associated with specific measures on a task, which we refer to as indicators. For example, on the Sternberg item recognition task, indicators might include recognition accuracy for a given set size, or the slope of accuracy as a function of set size. The system allows specification of relations between mental concepts and task/indicator combinations, such as a relation between the concept of “working memory” and recognition accuracy on the Sternberg item recognition task.

Figure 2.

Depiction of the structure of the Cognitive Atlas knowledge base. The left panel shows examples of mental concepts (such as “working memory”) and relations between them. The right panel shows an example of a particular mental task, the Sternberg Item Recognition Task, and two indicators for the task. It is these task/indicator combinations that link directly to mental concepts, as show by the “is measured by” relations in this figure.

The Cognitive Atlas is in its infancy, and is not yet sufficiently fleshed out to provide a solid basis for annotation of neuroimaging data. Like other social collaborative knowledge bases such as Wikipedia, its success will rely upon the engagement and involvement of a large number of interested researchers. If successful, it will provide a new means by which to more directly relate neuroimaging data to mental processes, and could afford a number of new analytic approaches, which I now turn to.

5 Selectivity and ontological reality

The foregoing analysis provides a means to identify which functions are mapped to particular regions, but does not speak to the selectivity of those regions for the functions in question. For example, the fact that the posterior STG region in Figure 1 is associated with the process of Audition does not imply that it is the only region that is associated with this process (and indeed, it is not). In order to ask this question, we need a different kind of analysis: Instead of asking which regions are associated with a particular process, which need to ask which regions have patterns of activity that are predictive of engagement of a particular process, as opposed to other processes (e.g., which regions have patterns of activity that reliably indicate that the process of Audition is engaged). To address this question, we can use the tools that have been developed within the field of machine learning, which are focused on determining the degree to which one can use data to make accurate predictions about new observations (Haynes & Rees, 2006; O’Toole et al., 2007). If we can predict which mental process is engaged on the basis of neuroimaging data, then this provides us with evidence for selective association.

In order to perform such predictive analysis, we need a large neuroimaging dataset that spans the cognitive ontology. Unfortunately, there are no such datasets available that include whole-brain neuroimaging data. However, the BrainMap database (Laird et al., 2005a) contains coordinate-based data from a large number of papers, and by using methods such as Activation Likelihood Estimation (ALE) (Laird et al., 2005b), it is possible to create simulated whole-brain activation maps from these coordinates. These data can then be used to determine the degree to which brain activity patterns can predict which cognitive processes are being engaged. An additional requirement is that the data be annotated using a cognitive ontology, in order to determine which data are associated with which mental processes. The BrainMap data are annotated using a relatively coarse ontology (which formed the basis for the one used in Poldrack et al. (2009), and thus cannot support fine-grained analysis of predictive ability, but it does at least provide the basis for a proof of concept.

We (Lenartowicz et al., 2010) performed such an analysis for a small subset of concepts within the domain of executive function (“cognitive control”, “response inhibition”, “task switching”, and “working memory”). We retrieved all papers matching those search terms, and manually confirmed (by agreement of three raters) each of these annotations. We also retrieved papers for the search term “bilingual language” as a control condition. The data for each study were projected into three-dimensional space using ALE, and then reduced by averaging within each of 117 anatomical regions of interest. Using this reduced representation of the neuroimaging data, we examined the ability to classify between the presence of two different mental processes (separately for each possible pair of concepts), using a k-nearest neighbor classifier.

A summary of the results is shown in Figure 3. All of the executive function concepts were distinguished from the bilingual language construct with relatively high accuracy (A’ > 0.8), which likely provides an upper bound on the accuracy of prediction from these data; given the very sparse nature of the coordinate data, this is quite impressive. Within the set of executive function concepts, there was more variability in the accuracy of classification. Some of these concepts (e.g., “working memory” and “task switching”) were readily discriminable, whereas others (e.g., “task switching” and “response inhibition”) were not as easily discriminated from one another.

Figure 3.

Selectivity analysis of several mental concepts using the BrainMap database Lenartowicz et al. (from 2010). Discriminability (A’) values were obtained using a k-nearest neighbor classifier (k=3). The bottom right triangle of the figure is a gray scale depiction of A’ values, with brighter tones denoting greater discriminability. The top left triangle is a reconstruction of which regions provided discriminability between each pair of concepts. WM: working memory, TS: task switching, RI: response inhibition, CC: cognitive control, BI: bilingual language.

This initial analysis has a number of shortcomings; in particular, we only performed pairwise comparisons, and thus we cannot determine the broad-scale selectivity of these concepts. In addition, the analysis does not provide direct evidence regarding which regions were most selectively associated with each concept. Nonetheless, this analysis provides encouraging initial evidence that it is possible to find relatively selective association between neuroimaging data and mental functions. At the same time, the analysis also demonstrates a new way to determine whether particular distinctions within the cognitive ontology may not be neurally plausible: Namely, if it is not possible to distinguish two concepts from one another (but is possible to distinguish both from a different process), this suggests that the ontological distinction between those two concepts should be reconsidered.

6 Conclusions

Cognitive neuroscientists have spent the last two decades attempting to map mental processes onto the brain, but have used a set of strategies that are fundamentally unable to identify selective structure-function associations. I have argued here that in order to understand the functional anatomy of the mental function, it is necessary to move from the brain mapping strategies that the field has employed towards a search for selective associations. This will require a greater focus on the structure of cognitive processes, which can be achieved through the development of formal ontologies that describe the structure of mental processes. Using such detailed ontologies along with large-scale data mining approaches, it may finally be possible to determine the joints at which the brain carves the mind.