The Cognitive Paradigm Ontology: Design and Application

Abstract

We present the basic structure of the Cognitive Paradigm Ontology (CogPO) for human behavioral experiments. While the experimental psychology and cognitive neuroscience literature may refer to certain behavioral tasks by name (e.g., the Stroop paradigm or the Sternberg paradigm) or by function (a working memory task, a visual attention task), these paradigms can vary tremendously in the stimuli that are presented to the subject, the response expected from the subject, and the instructions given to the subject. Drawing from the taxonomy developed and used by the BrainMap project (www.brainmap.org) for almost two decades to describe key components of published functional imaging results, we have developed an ontology capable of representing certain characteristics of the cognitive paradigms used in the fMRI and PET literature. The Cognitive Paradigm Ontology is being developed to be compliant with the Basic Formal Ontology (BFO), and to harmonize where possible with larger ontologies such as RadLex, NeuroLex, or the Ontology of Biomedical Investigations (OBI). The key components of CogPO include the representation of experimental conditions focused on the stimuli presented, the instructions given, and the responses requested. The use of alternate and even competitive terminologies can often impede scientific discoveries. Categorization of paradigms according to stimulus, response, and instruction has been shown to allow advanced data retrieval techniques by searching for similarities and contrasts across multiple paradigm levels. The goal of CogPO is to develop, evaluate, and distribute a domain ontology of cognitive paradigms for application and use in the functional neuroimaging community.

Introduction

Data sharing efforts within the human neuroimaging community are rapidly growing, driven equally by NIH and NSF policies and a new generation of scientists who are committed to sharing to enable large-scale knowledge discovery in the brain (Amari et al. 2002; Derrfuss et al. 2009; Hamilton 2009; Laird et al. 2009; Nielsen 2009; Van Essen 2009). Initial progress in this domain was achieved by the fMRI Data Center (Van Horn et al. 2001), and has been advanced more recently by several other neuroimaging data repositories, such as the Biomedical Informatics Research Network (BIRN; (Keator et al. 2008; Fennema-Notestine 2009)), XNAT Central (Marcus et al. 2007), the OpenfMRI Project (http://openfmri.org/), the 1000 Functional Connectomes Project (http://fcon_1000.projects.nitrc.org/), and the Laboratory of Neuro Imaging (LONI, http://www.loni.ucla.edu/Research/Databases/). Data sharing projects also include databases designed to archive neuroimaging results in the form of reduced data (i.e., stereotactic coordinates of brain activation locations), such as the BrainMap database (http://brainmap.org; Fox and Lancaster 2002; Laird et al. 2005), SumsDB (Van Essen 2005), or the AMAT database (Hamilton 2009). The Cognitive Atlas (Poldrack RA et al. Manuscript under review) is developing a knowledge base covering terms and assertions regarding cognitive science, e.g., “Working memory is a kind of memory.” The ability to integrate data and information across these different neuroscience repositories is being spearheaded by the Neuroscience Information Framework (NIF; (Gardner et al. 2008)), across biomedical repositories more generally in BIRN, and internationally supported by the International Neuroinformatics Coordinating Facility working groups (INCF, www.incf.org).

These data sharing efforts have highlighted the need for standardized terminologies within neuroimaging data descriptions (Bug et al. 2008; Larson et al. 2009), and minimum information standards such as those being suggested by International Neuroinformatics Coordinating Facility (INCF 2010). Standardized terminologies can serve as a common vocabulary for data sharing; but once the effort is begun to define the terms clearly, the next step is defining how the concepts those terms represent are related—what is the relationship between Brodmann area 9 and the dorsolateral prefrontal cortex, and can we use that in integrating neuroimaging results, for example? The combination of standard terms and their relationships in a formalized language produces an ontology, i.e., a representation of a domain of knowledge. These ontologies, written in the Web Ontology Language (OWL; (W3C-OWL-Working-Group 2009)) for the most part, provide a framework for connecting meaning to data sets and repositories in a way that can be used by automated systems.

Within the development of ontologies for neurobiological data sharing, different domains have already been identified. RadLex, the ontology of radiological procedures for RSNA, describes radiological procedures including CAT scans, PET scans, and structural and functional magnetic resonance imaging (MRI) methods with some level of detail (Langlotz 2006). Neuroanatomical ontologies such as the Foundational Model of Anatomy (FMA) provide structured representations of body parts and neuroanatomical labels (Rosse et al. 2003; Cook et al. 2004; Golbreich et al. 2006). The Ontology for Biomedical Investigation (OBI) provides the basic structure for describing a generic experimental process, and teasing out the representation of the experimental sample from the analytical methods from the equipment used (Brinkman et al. 2010).

The ability to describe the cognitive paradigms used during the behavioral portions of a neuroimaging study is critical for sharing data and integrating information across experiments. Cognitive paradigms are not standardized; they are infinitely flexible, and can vary by choice of stimuli, timing, the instructions given to the subject, and the responses the subject is expected to make. There are some general classes of experiments that have been established when a researcher initially develops a paradigm and others continue to use it over many different experiments (the classic Stroop experiment (Stroop 1935) is an example); more common, however, is the experiment that is executed once, then modified for the following experiment or to emphasize something new in a different experiment (e.g., the emotional Stroop experiment (McKenna et al. 2004)). Moreover, many behavioral paradigms are used once and never repeated. To encompass the ever-increasing complexity of experiments being used by investigators, a standardized structure for describing cognitive paradigms in human neuroimaging, or cognitive psychology in general, needs to be flexible.

The BrainMap repository has dealt with this problem for several decades, beginning in the late 1980s as a database of published findings in the human neuroimaging literature, first in PET and then fMRI. The goal of BrainMap was to allow aggregation of studies and their findings to determine similarities and easily allow consistent patterns of findings across multiples studies to be identified. BrainMap now archives the results of 2,060 publications (as of December 2010), which includes 11,696 subjects reporting 76,872 activation locations across 9,712 experiments. The schema that BrainMap uses to describe experiments is generally applicable, but is not in itself a strict ontology; however, we use its approach and basic terminology as the backbone of the Cognitive Paradigm Ontology.

Ontologies within biomedical research have the option of being coordinated through the Open Biomedical Ontologies (OBO) Foundry, which is a community of ontology developers committed to interoperable, orthogonal biomedical ontologies (Smith et al. 2007). In keeping with the overarching philosophy of the OBO Foundry, we are building CogPO using the Basic Formal ontology (BFO; www.ifomis.org/bfo) as the upper-level ontology, and the Relationship Ontology (RO) as the foundational set of relationships (Smith et al. 2005; Arp et al. 2008). We add new relationships only as necessary. We used the Information Artifact Ontology (IAO) as the midtier ontology, which was the starting point for CogPO development.

The IAO (http://neurocommons.org/page/Information_Artifact_Ontology) is a spin-off of OBI that focuses on the more abstract concepts of information that are critical to scientific work, but not well-defined within the other ontologies. The IAO focuses on the relationship between the artifact (the document, the plan, the lab book, the computer program) and the information it contains about the experiments, the methods, the subjects, etc. Neuroimaging experimental paradigms represent plans for experimental procedures, generally; what is described in the literature is what was expected or planned, and in general what was done, but it does not necessarily describe what actually happened. A paper may report that subjects were told to push a button when the light flashed—most of the time that is what the subjects did, and the neuroimaging results are interpreted to represent that action, even though there will be some instances in which a given subject may not have responded correctly to the flash of light. For the purposes of CogPO, we develop the ontology to represent what was intended by the experimenters and what instructions were given, and not what the subject may have actually perceived about the stimuli or what they thought they were doing.

We present here the structure of CogPO, the key terms and relationships, and the application to several published neuroimaging paradigms.

Structure of CogPO

The core terms of CogPO are shown in Table 1, and their definitions and usage are given below. Throughout this paper we will denote terms and relations which are part of an ontology in italics to differentiate them from their use in common English (e.g., Stimulus as a term in CogPO rather than the stimulus in a given experiment). We chose the Basic Formal Ontology as our foundational ontology, in keeping with best practices for the OBO Foundry; we started with the Information Artifact Ontology, which also uses BFO as its foundation, and added CogPO within its structures. The CogPO wiki (www.wiki.cogpo.org) allows term-by-term browsing for a more complete view.

Table 1.

Basic concepts within CogPO

Concepts	Parent class	Definition	Restrictions
Behavioral Experimental Paradigm	OBI:’Planned process’, BFO: processual_entity	Within an Experimental Paradigm, the behavioral paradigm describes the behavioral aspects of the experiment: what stimuli are presented to the subject when, and under what conditions, and what the subject’s responses are supposed to be.	Must have_part at least one Behavioral Experimental Paradigm Condition
Behavioral Experimental Paradigm Condition	OBI:’Planned process’, BFO: processual_entity	A planned combination of stimuli and instructions regarding responses to the experimental subjects (who are only involved once the process is implemented).	(has_instructions some Instructions) and (has_stimulus some Stimulus) and (has_response_type some Response)
Stimulus Role	BFO:role	The role of a stimulus in a behavioral experiment is attributed to the object(s) which are presented to the subject in a controlled manner in the context of the experiment.
Response Role	BFO:role	The role of response is attributed to the overt or covert behavior which is elicited from the subject in an experimental condition.
Stimulus	BFO:ObjectAggregate	The object or set of objects, internal or external to the subject, which is intended to generate either an overt or covert response in the subject as part of an experimental condition.
Response	BFO:’process aggregate’	The overt or covert behavior which is elicited from the subject in an experimental condition.
Instructions	IAO:’action specification’, BFO: generically_ independent_continuant	Instructions are the information-bearing entity that sets up the rules for desired behavior from the subjects. An explicit direction that guides the behavior of the subject during the experimental conditions. Instruc- tions serve the function that they lay out what the response behaviors should be for any set of stimuli in the experiment.
Stimulus Modality	BFO:quality	The quality of the sensory perception of an explicit stimulus.	Inheres_in only Explicit Stimulus
Response Modality	BFO:FiatObjectPart	Class of body parts used to perform the actions which can play the role of an overt response
Relationships	Parent Class	Domain	Range
Has_stimulus	Has_participant	Behavioral Experimental Paradigm Condition	Stimulus
Has_response_type	Has_part	Behavioral Experimental Paradigm Condition	Response
Has_instructions	Has_participant	Behavioral Experimental Paradigm Condition	Instructions
Has_stimulus_modality	Bearer_of	Explicit Stimulus	Stimulus Modality
Has_response_modality	Has_participant	Overt Response	Response Modality

Every term has a label, meant to be the readable term understandable by the casual user. This is in contrast to the Uniform Resource Identifier (URI) for the term, which is meaningless to the casual user. The URI provides a unique identifier for a term that can be held constant while the definition is changed or the label is modified, such as to correct a spelling error or to change capitalization. When the URI is retired, the concept is no longer available within the ontology. Besides a label and a URI, we also have a definition, the definition editor(s), the source for the definition, and any comments or usage notes. We can use these to expound on the differences between these terms and the precise terms used in the BrainMap schema, where needed, or to capture and summarize the arguments about the term’s definition, so that the same points do not need to be revisited over time. The annotation fields for these are drawn from IAO.

Behavioral Experimental Paradigm

The first key term in CogPO is the Behavioral Experimental Paradigm. Even though CogPO is designated as a “cognitive” ontology, many of the included paradigms are emotional or interoceptive in nature and not rightly termed as involving cognition. We defined the behavioral experimental paradigm as a subtype of the IAO term planned process, as it refers to a planned process which occurred, describing what subjects were supposed to hear, see, or do during the experiment. The paradigm covers what is described in neuroimaging papers often under the heading of “Task”. It does not capture what may have happened in any particular subject’s neuroimaging session, when the subject fell asleep and failed to respond, or the video projector failed halfway through the last run and the subject couldn’t see the stimuli. These deviations from the protocol would be outside of CogPO’s scope; other researchers may need to represent such details explicitly, and could in theory do so using terms regarding the implementation of plans from within OBI.

The subtypes of Behavioral Experimental Paradigm are named paradigms that are recognized by BrainMap, which include the Stroop task, Oddball Discrimination, Delayed Match to Sample, etc. These can be defined by logical constraints when possible (e.g., a task is an auditory oddball task is and only if it is an oddball task with auditory stimuli). The list of subtypes can be expanded as the community recognizes that more paradigms are needed, e.g., if it is recognized that experiments with particular types of stimulus/response/instruction combinations always activate the anterior cingulate that could be logically defined as a new paradigm class. The Cognitive Atlas project is also defining classes of experimental paradigms in cognitive psychology, and a process for incorporating the new paradigm classes across both projects is in development.

Behavioral Experimental Paradigm Conditions

Within a recognized class of paradigms, there are variations in the stimuli used (e.g., is a given oddball task implemented with visual or auditory stimuli?), the requested responses (e.g., should subjects use their hand to respond, their foot, or an eye movement?), or the instructions given to the subject (e.g., upon viewing emotional faces, should the subject discriminate according to the emotion or gender of the face?). Thus the linchpin of CogPO is the Behavioral Experimental Paradigm Condition, another planned process which must include at least one stimulus, one instruction, and one response type. Following the standards set in neuroimaging experimental design, each experimental paradigm must have at least two conditions (i.e., an experiment with only one condition is not an experiment). For example, the Auditory Oddball Paradigm class contains paradigms that have at least two conditions, both of which have stimuli with an auditory modality, one of which requires a response (the target or oddball stimulus) and one of which does not (the standard stimulus).

In general, experimental studies do not repetitively use the same exact paradigms, but develop new ones, some that do not clearly fit into a recognized class. Paradigms that are used in a single study and never used again would simply be described as instances within the Behavioral Experimental Paradigm, with their conditions represented as instances of Behavioral Experimental Paradigm Condition. A new class of paradigm can be defined when enough individual experiments with similar conditions are described. In the experience of developing BrainMap over the years, once at least five instances exist that use the same combination of conditions, a new class can be named of which they are examples.

Stimulus, Response, and Instructions

The term “stimulus” is particularly challenging, since it is used in many different domains in generally similar but ill-defined ways. In effect, anything can be a stimulus, since anything can elicit a response from the subject, whether it is the lights being on in the experiment room (which may be irrelevant to the outcome of the experiment) or a dose of a drug (which is critical to the outcome). Thus in CogPO, we define the Stimulus Role, a role which is played by whatever is used deliberately as part of the experimental design to elicit a response from the experimental subject. The class Stimulus is an Object Aggregate within BFO, and is a class that defines all the things that can play the role of stimulus (e.g., acupuncture, tones, music, movie clips, faces, images, internal thoughts and images, etc). This is currently circumscribed by the finite list of stimulus types that the BrainMap schema recognizes, but can be expanded when necessary. There are two immediate subtypes of Stimulus: Explicit Stimulus, which is generated under the experimenter’s control and exists external to the subject at least at some point in time, and Implicit Stimulus, which is generated by the subject. Explicit stimuli such as visually presented images or tactile stimulation are the more commonly found in the neuroimaging literature; however, implicit stimuli are important in studies of resting state fluctuations, mental imagery, autobiographical memory, and meditation, to name a few.

Every Explicit Stimulus has at least one Stimulus Modality, which refers to the sensory system targeted by the stimulus; these can be restricted a priori in some cases. Tones have the stimulus modality of Auditory Modality, for example, while Food has the stimulus modality of Gustatory Modality, but Film Clips can have both the auditory and visual modality. A Stimulus Modality is a “quality” from BFO, as it is the quality of stimulating a particular sensory system, and is not the sensory system itself. The relationship between an Explicit Stimulus and its stimulus modality is represented in CogPO as has_stimulus_modality, which is a subclass of the OBO Relation bearer_of, that links objects and qualities, roles, or functions more generally. The relationship has_stimulus_ modality is restricted specifically to the domain of Stimulus and the range of Stimulus Modality.

The Response Role is played by the Response, the overt or covert behavior elicited from the subject in the experimental condition. A response can be an Overt Response, i.e., a response made with a body part that is externally observable (a button press or eye movement, for example); or a Covert Response performed internally, such as a silently rehearsing or naming objects. A response may be considered a combination of processes unfolding over time, and thus in CogPO we have made Response a subclass of the BFO term Process Aggregate. The Response Modality that each overt response must have is actually the body part used in the response. Thus the Button Press subclass of Response may have the Response Modality of Foot or Hand or whatever body part is used to push the button. The body part terms have been incorporated from the Foundational Model of Anatomy (FMA) using the MIREOT process (Xiang et al.), and have the parent class fiat object part from BFO. A fiat object part is a part of an object which is not demarcated by any physical discontinuities; the precise boundary where the wrist becomes the hand, the foot becomes the ankle, or the eye becomes the optical nerve is not physically determined.

The term Instructions are an Action Specification from IAO. They are currently one-word summaries from the BrainMap listing—Attend, Discriminate, Encode, Smile, etc. They each include a definition, and usage clarifications where needed. They do not have any other limitations or relationships at this point, though they could be restricted to particular response types or paradigm conditions if needed.

The relationships used within CogPO include the following from the Relationship Ontology (RO): has_part, has_participant, and the extension to RO to include bearer_of and inherent_in for the qualities and roles. The relationship between Behavioral Experimental Paradigm and some Behavioral Experimental Paradigm Condition is has_part. We propose new relationships that are specific to the domains and ranges of CogPO terms: a Behavioral Experimental Paradigm Condition has_stimulus of something that comes from the class Stimulus, and has_response of something that comes from the class of Response. Has_stimulus and has_response are both subtypes of has_part; has_stimulus_modality, which describes the relationship between a stimulus and its modality, is a subtype of bearer_of, while has_response_modality is a subtype of has_part (since response modalities are the body parts used to make the response).

Application to Published Cognitive Paradigms

To demonstrate CogPO’s application, we present the representation of two example cognitive paradigms from published fMRI papers. These two paradigms were selected since both are present in the BIRN Data Repository (http://www.birncommunity.org/resources/data/) as well as the BrainMap database, and therefore represent two potential examples for querying and automated reasoning across distributed data sources.

Auditory Oddball Paradigm

The first paradigm is a classic auditory oddball paradigm, as used in (Ford et al. 2009) and (Kim et al. 2009a), and which is a type of paradigm with 11 variations within the BrainMap repository. The task consists of a stream of tones being played at a constant interval. In the representative example of this task (Ford et al 2009), the tones are space 500 ms apart and each one lasts 100 ms. The standard tone is pitched at 1000 Hz and occurs 95% of the time, while the remaining 5% of the time the tone is pitched at 1200 Hz. Subjects are instructed to push a button with the finger of their right hand when the less frequently presented “oddball” tone is heard, and nothing when the standard tone is heard.

The Ford et al. 2009 paradigm was used by the Functional Imaging Biomedical Informatics Research Network (FBIRN) in one of its multi-site neuroimaging studies of schizophrenia, the “Phase II” study. It can be stored in the CogPO ontology as the idiosyncratic string “FBIRN Phase II AudOdd”, which is an instance of the class Auditory Oddball Paradigm. It has two experimental conditions. The first condition can be stored as the “FBIRN Phase II AudOdd Standard Tones” condition, which has_stimulus Tones, has_response some Covert Response, and has_instructions Attend. The second condition is the “FBIRN Phase II AudOdd Target Tones” condition, which has_stimulus Tones, has_response Button Press, and has_ instructions Discriminate. Within the larger CogPO framework, an auditory oddball paradigm is defined as being a paradigm that has at least two conditions, which both have a stimulus type with the auditory modality. One of the two conditions has no response while the other condition requires a response. Some published auditory oddball experiments have three or more conditions (e.g., standard tones, target tones, and novel tones ((Kiehl et al. 2005)), or do not use Tones as their stimuli (e.g. the sounds of barking dogs and sheep were used in a variation of an oddball task by (Altmann et al. 2007)). Thus when defining the auditory oddball paradigm, we could not restrict the number of conditions to only two, or the stimulus to only Tones.

Within the CogPO ontology the stimulus class of Tones is constrained to have the stimulus modality of Auditory Modality, since tones cannot be presented via any of the other stimulus modalities (e.g., visual or tactile). Thus when the repository is searched for all paradigms with an auditory stimulus, the “FBIRN Phase II AudOdd” paradigm is identified, among others. More detail regarding the stimuli could be included, if necessary. For example, it may be important for researchers comparing auditory oddball papers to know what the specific pitch and timing of the tones were, or the digital onset and offset details of the sounds. CogPO could be extended to cover this additional information by adding properties to the stimulus class of Tones to represent the frequencies or durations of the presented tones. However, these extensions are best driven by specific use cases, and have not been needed within the scope of the BrainMap repository; thus CogPO 1.0 does not include that level of detail at this point.

Sternberg Paradigm

The second example paradigm we discuss is the version of the Sternberg (1966) task used in several previous neuroimaging studies (Brown et al. 2009; Kim et al. 2009b; Potkin et al. 2009), and is a type of paradigm with 23 variations in the BrainMap database. The representative example of this task is the FBIRN Serial Item Recognition Paradigm (SIRP), whose design is shown in Fig. 1 below. This is a more complex paradigm than the auditory oddball task. The FBIRN SIRP includes multiple encoding and recall conditions in which the memory load is parametrically modulated, and a fixation condition in which subjects passively watch scrambled images, which serves as a baseline for the experiment. In the encoding conditions, subjects are briefly shown one, three, or five digits, which they are instructed to remember with no overt response. In the immediately following set of probe conditions, they are shown a series of digits and are instructed to indicate with a button press whether the presented digit was one of the memory set or not. This creates a total of seven conditions: one fixation condition, with the stimulus type of Fixation Point, the instructions Fixate and the response type Covert Response; three encode conditions, with the stimulus type of Digits, the instructions Encode and the response type Covert Response; and three probe conditions, with the stimulus type of Digits, the instructions Recall and the response type Button Press. As with the auditory oddball paradigm, more details could be included if needed. For example, the delay between the end of an encode condition and the subsequent probe condition can be manipulated in such experiments, and may need to be represented in future versions of CogPO. This information could potentially be included in an additional property that has a temporal duration. The number of items in the memory set could also be made explicit as another property.

Fig. 1.

The experimental conditions of the Sternberg paradigm from Brown et al. 2009 discussed in the text. The horizontal arrangement shows an example of the timecourse of the experimental conditions. The number of items to be remembered could be 1, 3, or 5 digits; the different levels of memory load were presented multiple times with new datasets each time in pseudorandom, balanced orders, inter-digitated with fixation conditions

CogPO Release to the Community

CogPO Version 1.0 was released in November 2010, and is open for community feedback through several forums. It is presented as part of a CogPO specific wiki at www.wiki.cogpo.org. Any member of the research community can register on this wiki and comment on or create new concepts, properties, or relationships. CogPOver1 .owl is available for download from www.cogpo.org/ontologies/. CogPO 1.0 has also been submitted to the Neuroscience Information Framework Standard ontology (NIFSTD) and can be queried through the Neurolex wiki and examined within the context of a larger ontology of neuroanatomical and experimental terms. From within NIFSTD it can also be used to annotate datasets for specific querying and comparisons. Lastly, CogPO 1.0 is also available for visualization or download from the National Center for Biomedical Ontologies (NCBO)’s Bioportal at www.biportal.ncbo.org.

Discussion

As data sharing initiatives continue to expand within the functional neuroimaging community, the need for comprehensive ontologies also increases. As early as 2002, it was realized that without a standardized representation of behavioral experiments, “image data are incomparable and cannot be databased to their fullest potential”(Toga 2002). To address this need, we have developed an OWL representation of experimental conditions for cognitive or behavioral paradigms that are commonly used in human neuroimaging or psychological experiments. Our first version of the Cognitive Paradigm Ontology (CogPO) includes an initial description of stimulus types presented to the subject, the instructions given to the subject, and the response type requested from the subject.

The CogPO .owl files are available on the CogPO website (www.cogpo.org), for download and use by the research community; the terms are also available for browsing and comparison with other biomedical ontologies through the NCBO BioPortal (http://bioportal.bioontology.org/). Feedback on the terms, their definitions and relationships can be provided via the CogPO wiki (www.wiki.cogpo.org), where community-generated modifications and suggestions can be discussed and curated prior to inclusion in upcoming releases of the ontology. This is in keeping with the Neuroscience Information Framework (NIF) NeuroLex practices, of providing a wiki page for each term that lays out its definition, relationship within the class hierarchy, logical constraints, and editors or source of each definition. Each CogPO class is also linked both on the Wiki and in the OWL files to a PubMed ID for a neuroimaging paper that exemplifies that term in its experimental paradigm. The Metaneva project (www.metaneva.org) is a related project within the non-human neurophysiology domain to standardize vocabularies and meta-data descriptions for single-unit recording experiments. They also borrowed from the BrainMap terminology, and as a result have already suggested terms that will be included in the next release of CogPO.

We envision the immediate uses of CogPO being as annotations for databases of cognitive neuroscience or psychology experiments, or for textual markup in the literature, which will quickly identify where it needs expansion to deal with novel applications. Experiment scripts and programs from Eprime, Presentation, CIGAL and other software programs could be annotated with the appropriate CogPO terms, so that users of those particular experimental scripts and programs would be easily able to annotate their data and link it into broader data-sharing efforts. Semantic Web applications that use RDF and OWL as interchange formats could in principle include a rdf: tag which links to a URI for a CogPO term, which would be helpful in search and retrieval applications. The XML-based Clinical and Experimental Data Exchange (XCEDE) format for meta-data sharing (Keator et al. 2006; Gadde et al. 2011) provides a structure that can house CogPO URIs as descriptors of the cognitive paradigms used in an experiment, as well.

We acknowledge that Stimulus and Response are temporary concepts, which we expect to become an inferred class as CogPO interacts with other ontologies. For example, tones are not really a stimulus type, but instead they are a kind of sound. Similarly, a button press can be thought of in many ways other than as an experimental response. However, the development of a full ontology of sounds and their characteristics, or hand movements and their uses, was outside the scope of CogPO. When such an ontology exists, we will modify CogPO to refer to that ontology’s concept of sounds or actions, with the caveat that that concept in question can play the Stimulus Role or Response Role as relevant. An example of this interplay between ontologies is already shown with Response Modality, whose subclasses have been linked from the Foundational Model of Anatomy. We anticipate that we will eventually have links to ontologies of sounds, shapes, food, actions etc., and Stimulus and Response will become inferred classes of things that play the relevant roles in an experimental condition.

CogPO 1.0 allows for a description of the experiment as might be drawn from the published study, and reflects an intended experimental paradigm, from which any given dataset might deviate to some extent. CogPO 1.0 also does not include all possible information regarding the cognitive paradigm. Within a published description of an experiment, information is often provided regarding characteristics of the stimuli that are not included here, such as the specific color of a target square, or the visual angle of a checkerboard stimulus, or the frequency and duration of a tone. These characteristics can be very domain-specific, and potentially infinite in their variations. Thus we have chosen to develop a high-level description of these concepts as an initial version of standardizing descriptions of experimental paradigms. However, if in the future, the precise angle of a corner in some visual stimulus is found to be a critical detail whose inclusion in CogPO will result in more powerful data mining applications, then these additional descriptors of the stimulus, response, and instructions can be added to a later version of the ontology. Annotating the data to indicate that in a particular dataset the subject failed to press the button, for example, or pressed the button more than was instructed, requires an extension that is not available in this current formulation. This kind of deviation in particular is a challenging issue: representing logically that a button press was expected but not received is similar to attributes required to describe cancerous tissue samples as having “too many” cells of a particular type to be healthy, or describing a mouse as being “smaller than normal”—conceptually it is quite simple, but within a logically rigorous framework it is more challenging. Discussions regarding these issues are ongoing within OBI, NIF, PATO and other large-scale ontological development projects (Mabee et al. 2007; Washington et al. 2009; Mungall et al. 2010). We fully expect that CogPO will grow and evolve as the need to represent more of these details arises.

The BrainMap database, from which the basic classes and definitions of CogPO were drawn, includes additional metadata beyond what is needed to characterize the behavioral paradigm. BrainMap focuses on the reduced data of coordinates that are published in functional neuroimaging studies, which are derived from comparing neuroimaging data from at least two experimental conditions, or the same condition across at least two subject groups. Thus, BrainMap includes terminology for describing subject groups (e.g., males, females, both; healthy, or with a particular diagnosis; children, adult, elderly adult, etc.), the context of the study (e.g., the investigative purpose, such as to identify the effects of disease, gender, drugs, or normal human brain mapping), and the behavioral domain under study (e.g. the cognitive process isolated by the experimental contrast, such as working memory, speech, audition, etc.). Future versions of CogPO will be expanded to include some of this information; the descriptions of subject types and characteristics can be pulled from other ontologies, but the context of the study (e.g., pre or post-treatment) is particularly important both in human and animal behavioral studies, and its inclusion in CogPO may result in improved data retrieval and discovery.

CogPO was designed specifically to address descriptions of tasks employed in neuroimaging studies, but does not currently include links to the cognitive processes that are elicited by those tasks. While these concepts are included in the BrainMap database as “behavioral domains”, they are also being standardized and described in the Cognitive Atlas (Poldrack 2006; Poldrack RA et al. Manuscript under review). In BrainMap, the behavioral domain terms are assigned on the basis of what the study authors include in the published paper; the experimental paradigms are generally part of the operationalization of the behavioral domain or domains. However, these terms may change with time based on the results of ongoing research. For example, previous debates over short term, intermediate term, and long-term memory are no longer phrased in those terms. What is classified as attention versus what is executive function versus what is impulse control within psychological research may change with time. If these links are explicit within the ontology then the ontology must be reconstructed or refactored every time the research community consensus about cognitive processes changes. This contributes to the particular problem of “speciation” in cognitive paradigms (Bilder et al. 2009) that stems from the ongoing changes to paradigms due to shifts in conceptual understandings. The CogPO approach is less susceptible to the whims of conceptual shifting within the domain, in that while one’s research interests may progress from “working memory” to “executive function”, one may continue to be interested in paradigms in which (1) a target set of letters were presented, followed by a probe letter, (2) after which subjects indicated if the probe letter matched any target, (3) using a button press response. These conditions and the comparisons between them have the potential be considered as operationalizations of measuring multiple cognitive processes, including other processes than those that were intended by the original experimental designers.

The link between the CogPO project for ontology development and the Cognitive Atlas project for building a more explicit knowledge base of cognitive processes is ongoing; to date, the paradigm class conditions that are included in both projects have the same definitions, but the potential for extended interoperability goes much further than that. At present, we are developing methods of harmonization that will allow CogPO to link its paradigm classes to cognitive processes identified in the Cognitive Atlas, but we expect to create these links only when the associated definitions are suitably stable and have been vetted by the community.

Comparing results across different experiments via large-scale meta-analyses of image data in the BIRN Data Repository or coordinate data in BrainMap may, in fact, help identify meaningful distinctions in how cognitive processes are classified in a project such as the Cognitive Atlas. For example, data collected across multiple behavioral paradigms may be observed to support a single neuroanatomical circuit, leading to the conclusion that the individual processes can be considered as a unitary cognitive construct. Alternatively, a single cognitive process can be teased apart into multiple constituent components based on varying experimental paradigms that produce different brain activation patterns. The paradigm classes and behavioral domains should eventually be defined from the bottom up, based on patterns in the annotated experimental results, rather than from the top down, based on researcher intent and the then-current view of cognitive processes. We expect that as CogPO becomes richer we will include similarity measures between paradigms to infer classes automatically rather than defining paradigm classes a priori.

A caveat in the current formulation of CogPO is its definition of an experimental paradigm having at least two discrete conditions. This stems from the original model of representing neuroimaging results from a statistical contrast. While effective, this leads to difficulty in representing continuously varying stimuli which technically have an infinity of conditions (as in classical retinotopic mapping studies such as (DeYoe et al. 1996)), and in single-condition studies. Current neuroimaging studies will sometimes have only a single condition—e.g., resting or watching a movie—followed by complex multivariate analyses of the data to identify intrinsic, covarying neuroanatomical patterns over time without specifically referencing a particular stimulus (e.g., (Spiers et al. 2007)). While single-condition studies can be represented in CogPO, they do not have a named paradigm class. This limitation will be considered in future releases; given the broader meta-data regarding context and analyses which should be included in representations of cognitive experiments and their results, this may be an unnecessary restriction.

CogPO represents a single building block in the description of experiments in a structured framework, which should ultimately facilitate the representation of the actual experimental process leading to the published results, and the assertions that the published results claim to support. Projects such as the Cognitive Atlas, Knowledge Engineering from Experimental Design (Burns et al. 2009), NEMO (Frishkoff et al. 2007; Frishkoff et al. 2009), RadLex and OBI are also developing representational structures for the details of the experimental methods, the variables and interpretations of the results, in cognitive neuroscience and related domains. The large-scale, automated derivation of novel findings regarding brain function and dysfunction will require integration across these frameworks and their broader use in annotating data repositories, in conjunction with automated reasoning engines.

Information Sharing Statement

The Cognitive Paradigm Ontology wiki page can be accessed at www.wiki.cogpo.org; the .owl file is downloadable from www.cogpo.org/ontologies/CogPOver1.owl. CogPO 1.0 is also searchable through the Neuroscience Information Framework (www.neuinfo.org), and the National Center for Biomedical Ontologies BioPortal (www.bioportal.bioontology.org). The BrainMap database is available at www.brainmap.org.