Brainmap taxonomy of experimental design: Description and evaluation

Abstract

Coordinate‐based, voxel‐wise meta‐analysis is an exciting recent addition to the human functional brain mapping literature. In view of the critical importance of selection criteria for any valid meta‐analysis, a taxonomy of experimental design should be an important tool for aiding in the design of rigorous meta‐analyses. The coding scheme of experimental designs developed for and implemented within the BrainMap database provides a candidate taxonomy. In this study, the BrainMap experimental‐design taxonomy is described and evaluated by comparing taxonomy fields to data‐filtering choices made by subject‐matter experts carrying out meta‐analyses of the functional imaging literature. Fifteen publications reporting a total of 46 voxel‐wise meta‐analyses were included in this assessment. Collectively these 46 meta‐analyses pooled data from 351 publications, selected for experimental similarity within each meta‐analysis. Filter implementations within BrainMap were graded by ease‐of‐use (A–C) and by stage‐of‐use (1–3). Quality filters and content filters were tabulated separately. Quality filters required for data entry into BrainMap were classed as mandatory (five filters), being above the use grading system. All authors spontaneously adopted the five mandatory filters in constructing their meta‐analysis, indicating excellent agreement on data quality among authors and between authors and the BrainMap development team. Two non‐mandatory quality filters (group size and imaging modality) were applied by all authors; both were Stage 1, Grade A filters. Field‐of‐view filters were the least‐accessible quality filters (Stage 3, Grade C); two field‐of‐view filters were applied by six and four authors, respectively. Authors made a total of 115 content‐filter choices. Of these, 78 (68%) were Stage 1, Grade A filters; 16 (14%) were Stage 2, Grade A; and 21 (18%) were Stage 2, Grade C. No author‐applied filter was absent from the taxonomy. Hum Brain Mapp 25:185–198, 2005. © 2005 Wiley‐Liss, Inc.

INTRODUCTION

Human functional brain mapping (HFBM) is a large, steadily growing, highly inter‐disciplinary, clinically relevant research domain. The HFBM community has been remarkably successful in evolving sophisticated data‐analysis methods and in adopting them as de facto community standards. The core data‐analytic standards of the HFBM community are three: spatial normalization (coordinate‐based anatomy), voxel‐wise computation of statistical parametric images (SPI), and automated local‐maxima extraction. By these standards, task‐induced brain activations are reported as the x‐y‐z addresses of centers‐of‐mass extracted automatically and exhaustively from group‐averaged voxel‐wise SPIs. We estimate the HFBM literature conforming to these standards to be more than 2,500 articles, with approximately 500 new articles published each year [Laird et al.,2005a]. Such widespread use of highly quantitative analysis standards makes the HFBM literature fertile ground for quantitative meta‐analysis [Fox et al.,1998; Fox and Lancaster,2002; Fox et al.,2005]. Not surprisingly, the peer‐reviewed literature already contains more than 50 coordinate‐based meta‐analyses of the HFBM literature. The recent development of coordinate‐based, voxel‐wise meta‐analysis (CVM) methods [Chein et al.,2002; Laird et al.,2005b; Lancaster et al.,2005; Neumann et al.,2005; Turkeltaub et al.,2002; Wager et al.,2004] will surely accelerate this trend, as evidenced by the present special issue.

BrainMap [Fox and Lancaster,2002; Laird et al.,2005a; http://www.brainmap.org] is an online database of the HFBM literature that includes both the experimental results (x‐y‐z addresses of activated regions, P values, etc.) and a highly structured classification of meta‐data describing the experimental conditions. The purposes of the BrainMap database are to provide rapid, structured access to the HFBM literature and to enable a deeper understanding of this literature through quantitative meta‐analysis. At the time of writing, BrainMap contains more then 2,000 experiments from more than 500 HFBM publications; this represents approximately 20% of the HFBM literature. As described elsewhere in this special issue [Fox et al.,2005], all studies used in the meta‐analyses reported in this issue are being entered, which will increase the BrainMap data volume by approximately 50%. Hopefully, this precedent will continue, with published meta‐analyses contributing their coded data, thereby promoting replication and extension of their meta‐analyses.

In advance of the widespread application of these newly developed meta‐analysis methods to large, publicly shared data sets, which among other advantages will allow HFBM data to be explored and meta‐analyzed even by persons with no expertise in brain imaging, it is prudent to consider the experience of other fields with meta‐analysis and to develop some operational guidelines. In fields other than HFBM, meta‐analysis methods have been applied enthusiastically but injudiciously, resulting in harsh criticism and a skepticism that lingers about meta‐analysis as a method [for discussion and references, see Fox et al.,1998]. Perhaps the most useful lesson to be extracted from this experience is that “the meta‐analysis is itself a study requiring careful planning and execution” [Jones,1995]. Input‐data retrieval strategies should be carefully planned to identify all relevant studies, avoiding selection biases. Inclusion and exclusion criteria should be defined in advance. For CVM meta‐analysis, minimum inclusion criteria are the use of voxel‐wise SPI analysis and reporting coordinates in standardized space. Additional quality‐control criteria to consider are exhaustive reporting of observed activations (rather than the selective reporting of activations of interest practiced by some authors) and sample‐size minima [Lancaster et al.,2005]. These criteria are general, in that they don't bear on the hypotheses to be tested by specific meta‐analyses. Study‐specific criteria relate to the behavioral conditions employed by the individual studies and to commonalities of condition across study. These criteria can be chosen knowledgeably only by persons expert in the topic of the meta‐analysis. Obviously, study‐specific criteria are fundamental in the design of a cogent meta‐analysis. A rigorous meta‐analysis thus needs careful design and detailed specification in advance of the application of activation likelihood estimation [ALE; Turkeltaub et al.,2002] or other CVM methods.

In view of the critical importance of selection criteria for any valid meta‐analysis, an HFBM‐specific taxonomy of experimental design and execution could be an invaluable tool for designing rigorous meta‐analyses. The coding scheme for describing HFBM study design implemented within the BrainMap database provides a potentially suitable taxonomy [Fox and Lancaster, 1996; Laird et al.,2005a]. Because the BrainMap database also contains the results of thousands of HFBM experiments, it provides an environment within which to design and implement meta‐analyses. The purposes of the present study are: (1) to describe the BrainMap experimental‐design taxonomy; and (2) to evaluate the BrainMap taxonomy by comparison to choices made by subject‐matter experts carrying out HFBM meta‐analyses. As a longer‐term goal, this evaluation is intended to inform future refinements of the BrainMap taxonomy and the development of data‐filtering functions based on this taxonomy.

The BrainMap Taxonomy

The BrainMap meta‐data coding scheme is a concise description system composed chiefly of structured keywords that explain and categorize experimental methods, including the experimental question addressed, the imaging methods used, the behavioral conditions during which imaging was acquired, and the statistical contrasts carried out. (This is a partial listing.) The coding scheme has been iteratively refined and augmented to accommodate research trends. For example, the original coding scheme (1988) was specific for block‐design, two‐condition contrast (subtraction design) positron emission tomography (PET) studies. It was first modified to allow block‐design functional magnetic resonance imaging (fMRI) studies, and modified subsequently to allow event‐related and mixed block/event‐related designs. It has also been modified to allow multicondition contrasts and correlations with external variables, as exemplified by performance correlation analysis [Fox et al., 2000; Silbersweig et al.,1995]. The coding scheme is understood most readily by downloading the BrainMap Submit software, the manual, and an example article (http://www.brainmap.org). Key features are described briefly below.

Core Hierarchy

The BrainMap meta‐data coding scheme follows a hierarchy naturally occurring within the brain‐mapping literature. This hierarchy is the backbone of the new BrainMap meta‐data coding scheme (Fig. 1). Every paper reports experimental results drawn from one or more subject populations, the members of which have been functionally imaged during one or more scanning sessions, with each session being composed of one or more behavioral conditions. In BrainMap, an experiment is defined operationally by the production of a statistical parametric image (SPI). Each SPI is created by a statistical contrast of functional images selected based on criteria defining specific subsets of the populations, sessions, and conditions. Any experiment creates exactly one SPI. From each SPI one or more functional activations (locations) are extracted.

Figure 1.

The BrainMap coding scheme. The BrainMap taxonomy of experimental design is illustrated. Shaded boxes indicate the three levels within the taxonomy hierarchy: Paper, Experiment, and Location.

Paper‐Level Meta‐Data

Paper‐level meta‐data includes all items that refer to an entire publication and includes citation data: authors, journal, volume, issue, pages, year, and Medline number. A prose description provides a rapid grasp of the article. The prose description differs from the abstract in being entirely directed to the methods and highly stylized in format. Most paper‐level data apply comprehensively to subordinate levels and, therefore, do not require explicit relationships to be established to these lower levels. Population(s), condition(s), and session(s) include multiple entries that apply selectively to data at lower levels.

Conditions code the behavioral states used. Most HFBM studies use multiple behavioral states, which differ among one another in ways that are as specific and well characterized as possible. The behavioral differences among these conditions are the primary source of contrast in the functional images, as each condition recruits slightly different sets of functional areas. Conditions are described in considerable detail, using hierarchical keywords that categorize the type of sensory stimulus applied (e.g., visually presented words), the response made (e.g., words spoken aloud), and the instructions for each condition (e.g., say a verb corresponding to each presented noun).

Experiment‐Level Meta‐Data

An experiment is defined by the creation of a statistical parametric image. An SPI is formed by combining and contrasting image data from subsets of populations, sessions, and conditions, which are described fully at the article level. Experiment‐level coding, therefore, points above to the article level to specify its relations to these descriptors. These upward pointers allow for detailed understanding of the exact structure of the experiment reported. In the typical HFBM study, multiple experimental conditions are acquired; these are then combined and contrasted to create the SPIs of greatest possible statistical power or physiological or psychological significance. Although careful study of these experimental contrasts provides the fullest understanding of the meaning of the data, it is insufficient. Interpretation of the conditional contrasts is time consuming, requires a high degree of sophistication in the field of HFBM, and does not lend itself to rapid or systematic retrieval. To facilitate meta‐analysis, therefore, several types of hierarchically structured keywords have been developed that categorize research both in broad strokes and in critical details.

Context broadly categorizes the purpose of the work: normative mapping, age effects, developmental effects, disease effects, drug effects, and so forth. Behavioral domain categorizes the research in terms of neural/behavioral systems studied: cognition, action (motor), perception, emotion, autonomic, and so forth. Paradigm class categorizes the challenge presented, preferably in the jargon of the field: anti‐saccades task, n‐back task, Stroop task, Posner task, etc. These fields are all aids to meta‐analysis as they allow rapid, comprehensive retrieval of similar studies and filtering retrieved articles/experiments along well‐defined categories.

Additional fields at the experiment level aid detailed study of each article/experiment, to interpret the meaning of the results. For example, contrast identifies the most likely source(s) of statistical contrast effects in the SPI. Contrast effects can be critical to the experiment or not. Noncritical sources of conditional contrast include intentional and unintentional lack of experimental control. Use of low‐level control states (e.g., eyes‐closed rest) is a common practice that may seem overly casual but actually has considerable scientific benefit, not the least to the aspiring meta‐analyst. For example, studies of word production often use visually presented words as stimuli. If the control state doesn't include visually presented words (e.g., eyes‐closed rest or a fixation‐point control), there is contrast induced both by the stimuli and by the task. Use of a low‐level control avoids the assumption of “pure insertion,” often discussed as a problem with subtractive analyses. Use of a low‐level control also allows meta‐analyses to use noncritical as well as critical contrasts (e.g., a meta‐analysis of visual‐word form areas would benefit from the foregoing example).

Location‐Level Data

Location‐level data list and characterize the individual sites of activation extracted from an SPI. The most fundamental piece of data is the x‐y‐z coordinate, which reports the location of each activation. Generally speaking, the x‐y‐z coordinate is the center‐of‐mass of a volume including tens to hundreds of image voxels that exceed the significance threshold of the study. The x‐y‐z coordinate is the data type used in most HFBM meta‐analyses published to date. Supplemental data to allow for rule‐based subsampling of responses include the extent of the activation, statistical parameter, and anatomical names [provided by the Talairach Daemon; Lancaster et al.,2000]. Different forms of quantitative meta‐analysis will differ in which of these parameters they require as input.

Taxonomy Implementation

The experimental‐design taxonomy is implemented in the BrainMap environment. The BrainMap environment consists of: an SQL database management system, structured as described in the preceding paragraphs; a growing corpus of data, currently greater than 2,000 experiments; a Java client for data entry (Submit); and, a Java client for data retrieval, inspection, filter application, and meta‐analysis (Search&View). Because Search&View is the BrainMap application used for constructing meta‐analyses, it is the focus of the present evaluation. The same taxonomy is implemented in the Submit application. Search&View has four main windows, each of which controls a different stage of use (Fig. 2). The search window (Stage 1) constructs a query and launches a partial retrieval of meta‐data. The meta‐data retrieval is partial to allow the user to check the volume and nature of the data specified by the query before full retrieval. The results window (Stage 2), displays the results of initial search, listing articles that match the search criteria. At this stage, the user can access critical meta‐data and the prose description, to select articles for full download. The workspace window gives access to fully downloaded data (Stage 3). At this level, additional quality and context filtering options are provided to create and save a modified workspace. Workspace data can be output in ALE‐analysis format [Laird et al.,2005b; Turkeltaub et al.,2002], and in tab‐delimited spreadsheet format for network analysis using replicator dynamics network analysis [RDNA; Neumann et al.,2005], fractional similarity network analysis [FSNA; Lancaster et al.,2005], and related tools. ALE, RDNA, and FSNA functions have been implemented in Java and are being added as BrainMap workspace functions. The plot window is used to review the spatial distribution of retrieved data, with each point being an active link to all related meta‐data.

Figure 2.

BrainMap Search&View. The four windows that control the major stages of use are illustrated. The search window (upper left) constructs the initial query (Stage 1) and launches a first‐phase database search. The results window (upper right) displays the results of initial search, listing articles that match the search criteria and partial meta‐data (Stage 2). The filtering objective at this stage is to select articles for inclusion in the next stage of meta‐analysis. The workspace window (lower left) accesses fully downloaded data (Stage 3). At the workspace‐window level, additional quality and context filtering options are provided to create and save a modified workspace. The plot window (lower right) is used to review the spatial distribution of findings and to prepare output for quantitative meta‐analysis.

MATERIALS AND METHODS

The experimental design taxonomy of the BrainMap database was evaluated for its ability to emulate the data‐filtering choices made by subject‐matter experts carrying out HFBM meta‐analyses.

Evaluation Sample Retrieval and Filtering

The evaluation sample was limited to CVM studies. All contributions to the present special issue were included. In some instances, additional data regarding the filtering rules and the studies included had to be obtained by personal communication with the authors. Of three original descriptions of CVM, two [Chein et al.,2002; Wager et al.,2004] were excluded because they provided insufficient information about the data sample and filter rules; the third [Turkeltaub et al.,2002] was included. Tabular meta‐analyses and reviews were excluded, even if they reported summary coordinates, because they do not require explicit, rule‐based groupings.

Evaluation Strategy

The evaluation sample was reviewed to determine data‐retrieval and data‐filtering strategies. These are verbally synopsized for each member of the evaluation sample below. Author‐applied filters rules were subdivided into two categories: data‐quality filters and data‐content filters. The BrainMap taxonomy was then examined to determine whether the filter was present or absent. If present, filters were then characterized by the retrieval stage (Stages 1–3; Table I) at which it was applied, and by its ease of application (Grade A–C; Table II). Several of the author‐applied quality filters are absolute criteria for inclusion in the BrainMap database, in which case the stage‐of‐use and ease‐of‐use grading is not applicable. These quality standards were graded “M” for mandatory.

Table I.

Stage‐of‐use grading

Stage	Use
1	Before meta‐data retrieval
2	After meta‐data retrieval
3	After full retrieval

This grading system ranked the stage at which the user could apply the desired filter.

Table II.

Ease‐of‐use grading

Filter	Grade A	Grade B	Grade C
Automated filter	✓
Limited‐entry field	✓	✓
Prose field			✓

This grading system was used to assess the facility with which the BrainMap taxonomy implemented the filtering rules applied by the contributing authors.

RESULTS

Evaluation Sample

Fifteen CVM publications were analyzed. The 15 meta‐analyses reported 46 CVMs in total, with the number per article ranging from 1 to 7 (Table III). Collectively, these meta‐analyses pooled data from 351 publications, with an average of 23.4 publications per CVM. Meta‐analysis authors applied a variety of strategies to gather suitable publications. Many authors identified candidate studies using Medline or other online search engines. In some cases, this was achieved simply but effectively by searching for studies with a keyword designating the paradigm class (e.g., Stroop) under the MeSH subject heading of “Brain Mapping.” In other instances, authors searched iteratively, varying search criteria until no additional relevant articles could be identified. Additionally, some authors supplemented online searches with the bibliographies of retrieved articles. A few authors limited their input data to studies already entered into BrainMap, using it as their exclusive data source. In most instances, the size of the original sample retrieved from the literature and the amount by which filtering reduced this sample were not reported. From the instances in which it was reported, it seems that included studies represent a very small fraction of those retrieved. For example, in Laird et al. [2005c; Study 7], 54 articles were retrieved, but only 19 articles passed the filtering rules. Although these 19 articles reported 56 experiments, only 27 experiments passed the filtering rules.

Table III.

Evaluation sample

Sources	ALE Images	Papers	SPIs	Foci		Subject range
				Total	Range
Bolger et al.,2005	A English/Western	25	38	498	2–75	7–72
	B Chinese	9	18	317	3–39	6–20
	C Japanese Kana	5	8	73	1–24	9–17
	D Japanese Kanji	4	6	49	2–16	6–15
	E All Eastern	15	32	439	1–39	6–20
Brown et al., 2005	A Stutterers	8	9	154	4–41	5–20
	B Controls	7	7	74	2–30	10–18
Buchsbaum et al.,2005	A Wisconsin Card Sort	13	13	278	5–48	6–36
	B Go/no‐go	18	18	224	3–23	6–20
	C Task switching	18	18	231	4–75	5–48
Derrfuss et al.,2005	A Task switching	14	16	97	1–13	6–16
	B Stroop	11	11	64	1–14	7–16
Farrell et al.,2005	A Heat pain: right arm	11	11	249	6–39	6–16
	B Heat pain: left arm	15	16	170	4–53	6–27
Glahn et al.,2005	A Schizophrenics (SCZ)	7	7	62	5–15	10–30
	B Controls (CT)	7	7	60	5–12	10–27
	C SCZ > CT	6	6	38	1–12	11–30
	D CT > SCZ	4	4	40	3–14	6–30
Grosbras et al.,2005	A Saccades: all	30	33	385	3–42	4–20
	B Saccades: uncued	15	15	152	3–21	5–12
	C Saccades: cued	18	18	217	5–42	4–20
	D Attention shift: all	16	16	212	4–44	4–24
	E Attention shift: uncued	10	10	97	6–21	4–24
	F Attention shift: cued	6	6	78	4–44	4–19
	G Gaze perception	8	10	59	2–13	6–15
Laird et al.,2005	A Stroop: all	19	19	205	2–39	6–34
	B Stroop: verbal response	13	13	153	2–39	6–34
	C Stroop: manual response	6	6	52	2–10	12–18
Lancaster et al.,2005	A Stroop	19	19	205	2–39	6–34
Neumann et al.,2005	A Stroop	15	17	239	?	?
Owen et al.,2005	A N‐back: all	24	59	668	1–32	5–28
	B N‐back: verbal, identity	12	21	226	2–27	8–22
	C N‐back: nonverbal, identity	6	9	76	1–24	9–14
	D N‐back: nonverbal, location	5	11	150	2–28	5–8
Petacchi et al.,2005	A Auditory: all	15	27	233	3–34	4–18
	B Auditory: passive	10	19	138	3–22	4–18
Price et al.,2005	A Overt naming: covert controls	3	4	71	5–25	9–16
	B Covert naming: covert	6	6	78	6–24	8–19
	C Overt naming: overt controls	7	7	103	4–25	6–12
	D Covert naming: overt controls	2	2	36	16–20	8–10
	E Picture naming: all	16	19	288	4–25	6–19
	F A + B, covert controls	9	10	149	5–25	8–19
	G C + D, overt controls	8	9	139	4–25	6–12
Tan et al.,2005	A Phonology: alphabetic	13	16	109	1–11	6–24
	B Phonology: Chinese	6	7	109	4–38	6–12
Turkeltaub et al.,2002	A Single‐word reading	11	16	172	2–33	6–17

Source data used to assess the utility of the BrainMap meta‐data for filtering of input for meta‐analysis are listed. Sources are all peer‐reviewed, voxel‐based meta‐analyses, listed alphabetically by first author; full citations are provided in the bibliography. ALE images is a brief description of the ALE images reported by each source. Papers is the number of papers contributing to each ALE image. SPIs is the number of statistical parametric images contributing to each ALE image. Foci refers to the number of location triplets included in each ALE image. In subsequent discussion, each ALE image is referred to by the number of the paper (left‐most column) and the letter of the experiment (third column).

Quality Filters

In all 15 publications examined, authors applied quality filters, some explicitly and some implicitly. The quality filters applied and their implementation in the BrainMap database (Stage and Grade) are tabulated in Table IV. As all studies evaluated were CVM studies, all used only input data computed voxel‐wise (no region‐of‐interest studies), which reported activation foci as x‐y‐z addresses of the center‐of‐mass (or peak voxel) in standardized space; these are all mandatory BrainMap filters. All meta‐analyses limited input data to studies published in peer‐reviewed, English‐language journals, although only one author [Derrfuss et al., 2005c] explicitly stated this criterion; these are mandatory BrainMap filters. Only one study [Laird] explicitly restricted input data by the size of the group from which the SPIs were computed; thus, group size > 1 was an implicit filter (Table III, Subjects column). Twelve authors explicitly limited input data by imaging modality, accepting PET and fMRI studies (8 of 12); PET only (1 of 12); or fMRI only (3 of 12). No author included data from other imaging modalities, making this an implicit filter in the remainder. Group size and imaging modality are Stage 1, Grade A filters in BrainMap. Imaging field of view (FOV) was an explicit criterion in six studies, requiring that included studies image the entire brain [e.g., Farrell et al.,2005; Petacchi et al.,2005; Price et al.,2005] or, at least, span brain regions critical for the meta‐analysis [e.g., Grosbras et al.,2005; Owen et al.,2005]. Reporting FOV was an explicit criterion for four authors in that they excluded region‐of‐interest studies. All authors included articles reporting very limited lists of coordinates (e.g., <5; Table III, Foci column), which strongly suggests selective reporting. Imaging FOV and reporting FOV are not explicit filters in BrainMap. FOV can assessed using the plot screen (Stage 3), allowing studies to be assessed visually and selected/deselected for further use (Grade A). Alternatively, studies can be restricted by the number of reported foci, excluding studies with short lists of coordinates because of the implication of selective reporting (Stage 1, Grade A). One author limited input data by year of publication [Derrfuss et al.,2005].

Table IV.

Data quality filters

Filter	Stage	Grade	Explicit	Implicit	Total (of 15)
Voxel‐wise image analysis	M	M	15	0	15
Standardized space	M	M	15	0	15
Coordinates published	M	M	15	0	15
Peer‐reviewed publication	M	M	15	0	15
English‐language publication	M	M	2	13	15
Group size	1	A	1	14	15
Imaging modality	1	A	12	3	15
Imaging FOV	3	C	6	?	≥6
Reporting FOV	3	C	4	?	≥4
One experiment per study	2	A	2	2	4
Year of publication	1	A	1	0	1

All contributing authors applied a variety of data quality filters, described in the text. For the most part, authors explicitly stated which quality filters were applied. In some instances, the data used in the meta‐analysis met quality standards that were not explicitly stated by the authors. In these instances the quality filtering was scored as implicit. Grade refers to ease‐of‐use grades described in Table I; stage refers to the point at which the filter is applied as described in Table II. M indicates that this quality standard is mandatory for inclusion in BrainMap and, therefore, supersedes grade or stage.

FOV, field of view.

Content Filters

The content filters applied by each author are described in the following paragraphs, tabulated in Table V and tallied in Table VI. All authors limited data by subject population: 12 included only normal volunteers and 3 contrasted patient populations with normal controls. All meta‐analyses of speech and language functions included publications in which all subjects were right‐handed persons and whose native language was appropriate for the stimulus language, although no author made these explicit content filters.

Table V.

Data Content Filters

Author filters	BrainMap fields	BrainMap entries	Stage	Grade
Bolger et al.,2005
Written stimuli	Condition: stimulus: modality	Visual	1	A
Words	Condition: stimulus: type	Words	1	A
Overt reading	Expt: paradigm class	Reading (overt)	1	A
Covert reading	Expt: paradigm class	Reading (covert)	1	A
Passive control	Condition: instructions	Passive/rest	1	A
Native language	Paper: subjects: language	English, Chinese, Japanese	1	A
Normative study	Expt: context	Normal mapping	1	A
Normative study	Paper: subjects: diagnosis	Normal	1	A
Single‐words	Paper: prose description	—	2	C
Stimulus language	Paper: prose description	—	2	C
Brown et al., 2005
Stuttering study	Expt: context	Disease effects	1	A
Stuttering study	Paper: subjects: diagnosis	Developmental stuttering	1	A
Overt reading	Expt: paradigm class	Reading (overt)	1	A
Stuttering task	Paper: prose description	—	2	C
Buchsbaum et al.,2005
Wisconsin Card Sort	Expt: paradigm class	Wisconsin Card Sort	1	A
Go/no‐go	Expt: paradigm class	Go/no‐go	1	A
Task switching	Expt: paradigm class	Task switching	1	A
Normative study	Expt: context	Normal mapping	1	A
Normative study	Paper: subjects: diagnosis	Normal	1	A
Switch minus repeat	Expt: name	—	2	A
No‐go minus go	Expt: name	—	2	A
No‐go minus rest	Expt: name	—	2	A
Derrfuss et al.,2005
Task switching	Expt: paradigm class	Task switching	1	A
Stroop paradigms	Expt: paradigm class	Stroop	1	A
Frontal lobe	Locations	Lobe: frontal	1	A
Insula	Locations	Gyrus: Insula	1	A
Jan 2000–Jan 2004	Paper: citation: date	Jan 2000–Jan 2004	1	A
Normative study	Expt: context	Normal mapping	1	A
Normative study	Paper: subjects: diagnosis	Normal	1	A
Stroop; neutral control	Expt: name	—	2	A
Stroop: congruent control	Expt: name	—	2	A
Not language switching	Paper: prose description	—	2	C
Color–word Stroop only	Paper: prose description	—	2	C
Farrell et al.,2005
Tactile stimulus	Expt: behavioral domain	Perception: somesthesis: pain	1	A
Heat	Condition: stimulus: type	Heat	1	A
Normal controls	Expt: context	Normal mapping	1	A
Normal controls	Paper: subjects: diagnosis	Normal	1	A
Pain minus no‐pain	Expt: name	—	2	A
R or L arm	Paper: prose description	—	2	C
Glahn et al.,2005
Schizophrenia study	Expt: context	Disease effects	1	A
Schizophrenia study	Paper: subjects: diagnosis	Schizophrenia	1	A
Normal comparisons	Paper: subjects: diagnosis	Normal	1	A
N‐back	Expt: paradigm class	N‐back	1	A
N‐back–control	Expt: Name	—	2	A
SCZ > CT	Expt: name	—	2	A
CT > SCZ	Expt: name	—	2	A
Grosbras et al.,2005
Saccades	Expt: paradigm class	Saccades	1	A
Not anti‐saccades	Expt: paradigm class	Anti‐saccades	1	A
Not smooth pursuit	Expt: paradigm class	Visual pursuit	1	A
Attention shift tasks	Expt: paradigm class	Attention shift	1	A
Normative study	Expt: context	Normal mapping	1	A
Normative study	Paper: subjects: diagnosis	Normal	1	A
Voluntary	Paper: prose description	—	2	C
Visually cued	Paper: prose description	—	2	C
Gaze shift task	Paper: prose description	—	2	C
Laird et al.,2005
Stroop paradigms	Expt: paradigm class	Stroop	1	A
Verbal/manual response	Condition: response: modality	Oral/facial or hand	1	A
Normative study	Expt: context	Normal mapping	1	A
Normative study	Paper: subjects: diagnosis	Normal	1	A
Stroop: incongruent–congruent	Expt: name	—	2	A
Color–word Stroop only	Paper: prose description	—	2	C
Lancaster et al.,2005
Stroop paradigms	Expt: paradigm class	Stroop	1	A
Normative study	Context	Normal mapping	1	A
Normative study	Subjects: diagnosis	Normal	1	A
Stroop: congruent control	Expt: name	—	2	A
Color–word Stroop only	Paper: prose description	—	2	C
Neumann et al.,2005
Stroop paradigms	Expt: paradigm class	Stroop	1	A
Not: meta‐analysis	Expt: Context	Meta‐analysis	1	A
Normative study	Expt: context	Normal mapping	1	A
Normal controls	Paper: subjects: diagnosis	Normal	1	A
Incongruent–congruent	Experiment: name	—	2	A
Color–word Stroop only	Paper: prose description	—	2	C
Owen et al.,2005
N‐back	Expt: paradigm class	N‐back	1	A
Verbal stimuli	Conditions: stimulus: type	Words	1	A
Non‐verbal stimuli	Conditions: stimulus: type	Picture or words	1	A
Normative study	Expt: context	Normal mapping	1	A
Normative study	Paper: subjects: diagnosis	Normal	1	A
Not: 1‐back	Expt: name	—	2	A
Location monitor	Paper: prose description	—	2	C
Identity monitor	Paper: prose description	—	2	C
Not: reward tasks	Paper: prose description	—	2	C
Not: calculation tasks	Paper: prose description	—	2	C
Petacchi et al.,2005
Auditory paradigms	Condition: stimulus: modality	Auditory	1	A
Abstract stimuli	Condition: stimulus: modality	Tones, clicks, or noise	1	A
Not: words	Condition: stimulus: modality	Not: words	1	A
Not: music	Condition: stimulus: modality	Not: music	1	A
Not: environmental sounds	Condition: stimulus: modality	Not: sounds	1	A
Not: motor contrast	Expt: contrast: response	No	1	A
Normative study	Expt: context	Normal mapping	1	A
Normal controls	Paper: subjects: diagnosis	Normal	1	A
Low‐level baseline	Expt: name	—	2	A
Price et al.,2005
Picture naming	Expt: paradigm class	Picture naming	1	A
Overt response	Condition: response: modality	Oral facial	1	A
Covert response	Condition: response: modality	None	1	A
Silent baseline	Condition: response: modality	None	1	A
Speaking baseline	Condition: response: modality	Oral facial	1	A
Normative study	Expt: context	Normal mapping	1	A
Normal controls	Paper: subjects: diagnosis	Normal	1	A
Not: nonobject baseline	Paper: prose description	—	2	C
Not: action naming	Paper: prose description	—	2	C
Not: scene naming	Paper: prose description	—	2	C
Tan et al.,2005
Phonology tasks	Expt: behav domain	Cognition: language: phonology	1	A
Visual stimuli only	Condition: stimulus: modality	Visual	1	A
Alphabetic words	Condition: stimulus: type	Words	1	A
Chinese characters	Condition: stimulus: type	Chinese characters	1	A
Normative study	Expt: context	Normal mapping	1	A
Normative study	Paper: subjects: diagnosis	Normal	1	A
Non‐phonological decision control	Expt: name	—	2	A
Fixation control	Expt: name		2	A
Not: reading tasks	Expt: paradigm class	Reading (overt) or (covert)	1	A
Phonological decision task	Paper: prose description	—	2	C
Turkeltaub et al.,2002
Aloud reading	Expt: paradigm class	Reading (overt)	1	A
Normative study	Context	Normal mapping	1	A
Normative study	Subjects: diagnosis	Normal	1	A
Single words	Paper: prose description	—	2	C
Most basal control	Expt: name	—	2	A

All contributing authors applied various data content filters with which they isolated studies of interest. All applied content filters were stated explicitly and defined by the authors. All authors limited data by subject population. All meta‐analyses of speech and language functions included publications in which all subjects were right‐ handed persons, whose native language was appropriate for the stimulus language, although no author made these explicit content filters. Grade refers to ease‐of‐use grades described in Table I; stage refers to the point at which the filter is applied, as described in Table II.

Table VI.

Data content filter summary

Stage	Grade A	Grade B	Grade C	Σ
1	78	0	0	78
2	16	0	21	37
3	0	0	0	0
Σ	94	0	21	115

Authors applied a wide variety of filters for data content, which are listed in Table IV. For each stage and grade, the number of content filters applied is tabulated. For each author, a filter was counted only once, even if applied repeatedly (e.g., to compute several CVMs).

Bolger et al.,2005

Five voxel‐wise meta‐analyses were computed, each of which isolated brain regions recruited during reading (overt or covert) of written, single words. The five meta‐analyses differed in the writing system in which the reading tasks were carried out: English and other European languages pooled (1A); Chinese (1B); Japanese Kana (1C); Japanese Kanji (1D); and all three Asian systems pooled (1E). Input studies included a wide variety of tasks conditions (e.g., overt naming, rhyming, lexical decision, 1‐back, covert viewing, etc.). Control conditions were limited to a “resting baseline” (e.g., fixation, checkerboards, noise, etc.).

Brown et al., 2005

The neural correlates of stuttering were compared to those of fluent speech. Two voxel‐wise meta‐analyses were reported: stuttered speech in stutterers (2A), and fluent speech in controls (2B). Input data were limited to publications that: (1) reported persons who stutter and non‐stuttering controls; (2) used an overt speech task in both groups; and (3) included a stuttering condition in the stuttering group.

Buchsbaum et al.,2005

The brain basis of the Wisconsin Card‐Sorting Task (WCST) was addressed, targeting two hypothesized component cognitive processes: task switching and response suppression. Three voxel‐wise meta‐analyses are reported: WCST paradigms (3A), go/no‐go paradigms (3B), and task‐switching paradigms (3C). For the WCST meta‐analysis, input data were limited to experiments using “clearly recognizable” versions of the task and controls states for which the resulting contrast isolated “one or more of the basic cognitive processes involved.” For the task‐switching meta‐analysis, inclusion criteria were restrictive, being limited to contrasts of switch (task state) with repeat (control state) or comparable contrasts. In the go/no‐go meta‐analysis, studies were limited to: (1) no‐go (task) minus go (control); and (2) no‐go (task) minus rest (control). For each included study, input data were used only from a single contrast.

Derrfuss et al.,2005

Cognitive control functions of the inferior frontal junction were investigated using two voxel‐based meta‐analyses: one of task switching paradigms (4A) and one of Stroop paradigms (4B). The task switching meta‐analysis included task switching, set shifting and non‐probabilistic stimulus–response reversal, but excluded switching between languages, movements, or types of information held in working memory. Contrasts were selected to maximize the switching component; however, those using low‐level controls (e.g., rest or fixation) were excluded. For the Stroop meta‐analysis, only the traditional color–word Stroop was included. Contrasts were limited to incongruent (color–word, font–color mismatch) versus either neutral (non color–word) or congruent (color–word, font–color match). If both conditions were present, the neutral condition was preferred. Studies lacking activations in frontal lobe and anterior insula were excluded (two studies), and only activations from frontal lobe and insula were used as input.

Farrell et al.,2005

Brain systems underlying the perception of pain were isolated using two voxel‐wise meta‐analyses: right‐sided (5A) and left‐sided stimulation (5B). All included studies contrasted painful stimulation (task state) of the skin with nonpainful stimulation at the same site or with no stimulation (control states). Stimulus delivery system type was not used as a filter; these included: water bath, thermal contacts, and laser stimulation. Contrasts were restricted to those in which heat stimuli were applied to either upper limb singly, but not both simultaneously. For each included study, input data were used only from a single contrast per limb. For studies that reported multiple intensities of pain, only the most salient contrast was included.

Grosbras et al.,2005

Brain systems underlying overt and covert shifts of attention (i.e., with and without eye movements) were investigated using seven voxel‐wise meta‐analyses. Eye‐movement studies were limited to saccades, excluding anti‐saccades and smooth pursuit. Saccades studies were divided into two categories: voluntary and visually triggered. Voluntary saccades included spontaneous (uncued and unpaced), self‐paced and aurally cued saccades, but no visually cued saccades. Visually cued saccades included both unpredictable and predictable targets. ALE meta‐analyses were computed for all saccades (6A), voluntary saccades (6B), and visually triggered saccades (6C). Covert attention‐shift studies were divided into the same two categories and ALE images were computed for all (6D), voluntary (6E), and visually triggered (6F). Finally, studies in which shifts of attention were triggered involuntarily by observing a gaze shift in another person were grouped (6G). For the most part, control states were low level, including fixation and eyes‐closed rest, but this was not an explicit selection criterion.

Glahn et al.,2005

The n‐back task in schizophrenic and normal subjects was investigated through four voxel‐based meta‐analyses to highlight areas of hyper‐ and hypoactivation in schizophrenia. Meta‐analyses carried out on coordinates from articles applying the n‐back working‐memory paradigm to patients with schizophrenia and matched comparison subjects. Four ALE images were created: patients with schizophrenia alone (7A), healthy comparisons alone (7B), patients greater than controls (7C), and controls greater than patients (7D).

Laird et al.,2005

The Stroop paradigm was investigated using three voxel‐based meta‐analyses. Studies were restricted to those using the standard color–word Stroop, eliminating Stroop variants such as the emotional Stroop and the counting Stroop. Included experiments were limited to contrasts of incongruent condition (font–color, color–word name mismatch) with one of three control conditions: the congruent condition (font–color, color–word‐name match), a neutral condition (non color–word), or a nonlexical condition. Studies were categorized by response modality, as verbal (spoken response) or manual (button‐press response). The three ALE images created were: verbal plus manual (8A), verbal only (8B), and manual only (8C). For each included study, input data were used only from a single contrast (quality filter); in studies in which multiple experiments met the selection criteria, the experiment with the highest‐level control was selected (content filter).

Lancaster et al.,2005

The Stroop paradigm was investigated using a single ALE meta‐analysis (9A). All input data were obtained from the BrainMap database, using the selection criterion of the Laird “all Stroop” image (8A). The ALE image was then thresholded to create volumes of interest. Within these volumes of interest (VOI), co‐occurrence patterns were computed for the input data, probing both the presence/absence of activations per study and per VOI, and the presence/absence of other study variables obtained from the BrainMap database.

Neumann et al.,2005

The Stroop paradigm was investigated using a single ALE meta‐analysis (10A). All input data were obtained from the BrainMap database, using the selection criterion of the Laird et al. [2005c] “all Stroop” image (8A), but limited to the contrasts “incongruent versus congruent” and “incongruent verus neutral.” One BrainMap entry was eliminated, as it was a meta‐analysis [Laird et al.,2005; 8A–C]. As with the Lancaster et al. [2005] study (9), the resultant ALE image was thresholded to create VOIs. Within these volumes of interest, co‐occurrence patterns were computed for the input data, probing both the presence/absence of activations per study and per VOI.

Owen et al.,2005

The n‐back task was investigated through four voxel‐based meta‐analyses. Experiments using only n = 1 and those linking working memory with reward or calculation were excluded. The main meta‐analysis used all available data (11A). Studies were divided into three groups: identity tasks with verbal stimuli (11B), identity tasks with nonverbal stimuli (11C), and location tasks with nonverbal stimuli (11D).

Petacchi et al.,2005

Auditory functions of the cerebellum were investigated using studies in which abstract auditory stimuli, such as pure tones or band‐filtered noise, were contrasted to a low‐level baseline. Two voxel‐based meta‐analyses were carried out for all audition studies (12A) and for those studies that investigated passive auditory stimulation (12B). Experiments using words, music, or environmental sounds were excluded. Experiments with motor response were excluded, unless the control state included a well‐matched motor component. Multiple experiments from a single publication were included only if substantially different conditions were used. From studies that parametrically varied loudness, only one contrast was included. Multiple studies from a single laboratory were not included, using only the most recent publication. Studies limiting subject selection to trained musicians were excluded.

Price et al.,2005

The brain basis of picture naming and the effect of baseline condition on regional activation detection were investigated through seven voxel‐based meta‐analyses. Picture‐naming tasks were placed in four categories, based on whether the task and control were overt or covert (13A–D). Additionally, ALE images were computed for all studies (13E), and separately for covert controls (13F) and overt controls (13G). Studies not imaging the entire brain or using regions of interest to report data were excluded. Action naming and scene identification were excluded. Tasks without a nonobject baseline were excluded.

Tan et al.,2005

Neuroanatomical correlates of phonological processes associated with two different writing systems were studied using two voxel‐based meta‐analyses: alphabetic words (14A) and Chinese characters (14B). Only experiments that required an explicit, phonology‐related decision (e.g., rhyme judgment) were included, excluding reading tasks. For the alphabetic meta‐analysis, studies using English and German word stimuli were included. Control tasks were a nonphonological decision (e.g., font size) or fixation.

Turkeltaub et al.,2002

The neural processes associated with cognitive and sensorimotor function of reading was studied in the original ALE meta‐analysis of 11 PET studies (15A). Studies that acquired data on normal subjects who carried out overt reading of single, real words were pooled. If more than one control condition was contrasted to the reading condition, then the most basal control was selected. This innovative method of meta‐analysis was validated by comparison with new fMRI data on aloud word reading in normal adult subjects.

DISCUSSION

Fifteen publications reporting 46 voxel‐wise meta‐analyses in total were included in this assessment. Collectively, these authors applied a wide range of quality and content filters. Quality filters were fairly similar across authors, reflecting common views as to what constituted good quality in this literature. Most quality filters were stated explicitly. Some obvious quality filters (e.g., peer‐reviewed and English‐language publications) were made explicit by a minority of authors despite being followed by all or nearly all authors. Content filters were more variable, reflecting the varied content of the meta‐analyses; nevertheless, common strategies are evident. For example, virtually all authors retrieved/filtered articles by paradigm class (e.g., Stroop, n‐back, saccades, picture naming, etc.). The most likely reason for this is that paradigm classes are very effective keywords for searching online databases such as Ovid and PubMed. Another common strategy was to include/exclude by the contrast of two conditions, such as incongruent minus congruent (in the Stroop task), or saccades minus fixation. This filtering strategy is quite effective for isolating specific mental operations and therefore for making a coherent, powerful CVM.

BrainMap Taxonomy: Performance Summary

Overall, the BrainMap Taxonomy performed well, providing a wide range of quality and content filters that conformed closely to the filters authors chose to apply manually, i.e., by retrieving and studying the publications. No author‐applied filter was absent from the taxonomy. In all instances, authors applied the five quality filters required for inclusion in the BrainMap database (mandatory filters) either implicitly or explicitly. Two nonmandatory quality filters (group size and imaging modality) were applied by all authors; both are Stage 1, Grade A filters. In total, authors made 115 content‐filter choices. Of these, 78 (68%) were Stage 1, Grade A filters; 16 (14%) were Stage 2, Grade A; and 21 (18%) were Stage 2, Grade C.

The lowest level of performance (Stage 3, Grade C) of any filter was for the quality filter “field of view.” Several authors explicitly excluded studies that did not image the whole brain. At present, BrainMap does not have a yes/no field for whole brain FOV; this should be added. Ideally, for articles with limited FOV, the BrainMap taxonomy would include a description of the exact extent of the FOV imaged and from which coordinates were reported (i.e., in region‐of interest‐based studies). There are two practical limitations to this suggestion. First, many articles omit a description of their FOV. Second, for studies that did not align the imaging planes parallel to the anterior–posterior commissure X–Y plane, there is no concise (or published) description of FOV. Despite these limitations, the FOV over which coordinates are reported can be assessed rapidly by visualizing the coordinates reported using the BrainMap Search&View plot function (Fig. 2) to see their extent over the brain (Stage 3, Grade C). It can also be weakly inferred from the number of points reported for each study (Stage 2, Grade B), in that limited FOV studies will report fewer points. FOV was also used as a content filter, as several authors explicitly included only studies reporting activations in specific brain structures [e.g., the frontal lobes in Derrfuss et al.,2005]. This type of anatomical filtering is very well developed in BrainMap, allowing creation of volumes‐of‐interest either with a bounded box function or with anatomical boundaries (e.g., hemispheres, lobes, and gyri) provided through the Talairach Daemon [Lancaster et al.,2000].

The next lowest level of performance (Stage 2, Grade C) was observed for content filters that grouped or limited higher‐level filter functions. For example, Owen et al. [2005] grouped n‐back tasks into identity‐monitoring tasks and location‐monitoring tasks, but excluded 1‐back tasks. This level of detail may be present in the experiment name field (Stage 2, Grade A), but often is present only in the prose description field (Stage 2, Grade C). As stated above, the prose description is a terse, highly stylized description of the conditions imaged and statistical contrasts formed to generate the reported coordinates. It is without exception more focused and information dense than are either the abstract or the methods section of a publication. Even this Grade C filter is thus applied much more readily than an author's manual inspection of the original publications.

BrainMap Taxonomy: Scope Correspondence

Not only did the BrainMap taxonomy perform well when compared to author‐applied filters, the same can be said for the reverse comparison. Although author‐applied filters did not utilize the entire range of filter functions implemented through the BrainMap taxonomy, they did draw at least once on each of five major categories of Stage 1 filter. Filters accessed through the “experiments” meta‐data dimension (Fig. 2, panel 1) were used most heavily, as they included paradigm class (e.g., Stroop or saccades), context (normal mapping, disease effects, or meta‐analysis), and imaging modality. Paradigm class was particularly widely used, as it lended itself to keyword searches in citational databases, which was the starting point of most authors' retrieval process. Filters in the “conditions” meta‐data dimension were also used broadly, including stimulus modality (e.g., visual or tactile), stimulus type (e.g., words or heat), and instruction (e.g., discriminate or passive/rest). Filters in the “subjects” dimension were used by all authors to specify sample diagnosis (e.g., developmental stuttering, schizophrenia, normals, etc.) and to restrict data by study sample size (e.g., group‐averaged studies only). Information fields under the “citation” category were not used heavily as filter functions, with one author restricting studies to a time period [Derrfuss et al.,2005] and one author limiting the number of studies included per laboratory [Petacchi et al.,2005]. These fields are necessary for study tracking and proper attribution.

Content filter functions that are accessed at later stages of use (Stages 2 and 3) were minimally used as entry or exclusion criteria in this sample, although they have considerable potential as meta‐analysis filters. For example, behavioral domain and contrast allow grouping of cognitively similar studies that use different paradigms. As noted previously, this absence is explained most likely by the difficulty of applying this type of filter manually, although it is applied readily within BrainMap. Similarly, brain location was used as a filter only by Owen et al. [2005], applied jointly with the n‐back paradigm as a selection criterion. This level of filtering is applied readily within BrainMap, but cannot be applied through keyword searches of publication databases. The fact that BrainMap has high‐level filter functions that cannot be applied readily otherwise leads to the discussion of BrainMap as a data resource.

BrainMap as a Growing Resource

In view of the solid performance of the BrainMap taxonomy, we believe that our emphasis should be to grow this resource and to extend its applicability. One way to extend this resource is to expand its data volume. BrainMap currently contains more than 2,000 experiments, a data corpus that will grow substantially through the contributions associated with this Special Issue. We strongly encourage authors of meta‐analyses and of primary publications to use BrainMap as a vehicle for sharing their data and, thereby, simultaneously exhibiting scientific good citizenship and increasing their personal citation indices. Meta‐analyses can be shared not only by submitting the articles used in the meta‐analysis, but also in the form of a saved workspace, which can be used to replicate rapidly and extend the meta‐analyses. Another way to extend this resource is to incorporate tools for meta‐analysis (e.g., ALE, RDNA, and FSNA) within the BrainMap environment. These tools have been implemented in Java (the coding language of the BrainMap applications) and are being added to BrainMap at this time. Growing BrainMap into a “full‐service” meta‐analysis environment should also motivate ongoing data contributions. In our experience, the entire process of CVM, including literature retrieval, BrainMap coding, quality and content filtering, CVM computation and interpretation, is an excellent educational exercise, making BrainMap an important educational resource for graduate students and post‐doctoral fellows. Finally, the BrainMap taxonomy has enormous potential as a meta‐data schema for databases of functional images, either in‐house or public (e.g., the Dartmouth fMRI Data Center). The BrainMap development endorses and will happily support these and other extended applications of this community resource.