Representation, Pattern Information, and Brain Signatures: From Neurons to Neuroimaging

Summary

Human neuroimaging research has transitioned from mapping local effects to developing predictive models of mental events that integrate information distributed across multiple brain systems. Here we review work demonstrating how multivariate predictive models have been utilized to provide quantitative, falsifiable predictions; establish mappings between brain and mind with larger effects than traditional approaches; and help explain how the brain represents mental constructs and processes. Although there is increasing progress towards the first two of these goals, models are only beginning to address the latter objective. By explicitly identifying gaps in knowledge, research programs can move deliberately and programmatically toward the goal of identifying brain representations underlying mental states and processes.

Introduction

In recent years, human neuroimaging research has undergone a paradigm shift from brain mapping to developing integrated, multivariate brain models of mental events. Traditional brain mapping approaches analyze brain-mind associations within isolated brain regions or voxels, or a series of them tested one at a time. A local brain response is treated as the outcome to be explained by statistical models, and effects in local regions are aggregated into maps. Brain models reverse this equation: sensory experiences, mental events, and behavior are the outcomes to be explained. A model specifies how to combine brain measurements to yield a prediction about the identity or intensity of a mental process (Figure 1). For example, a model might predict (or decode) the category of the objects a person is viewing (Haxby et al., 2001; Issa et al., 2013), which action a person is about to perform (Haynes et al., 2007; Soon et al., 2008), or the intensity of pain experience (Marquand et al., 2010; Wager et al., 2013). Thus, brain maps and models have fundamentally different goals: Whereas maps describe local encoding of information, models attempt to specify the parts of a neural system and how their joint activity predicts mind and behavior.

Figure 1.

Brain maps versus brain models. (A) The objective of conventional brain mapping is to identify which brain regions are reliably more active as a function of different kinds of stimulation or manipulations of mental state (in addition to error, ɛ). The classical outcome of brain mapping study is a parametric map indicating the extent to which every brain measure (voxel) is associated with a given mental state. The objective of developing a multivariate brain model is to account for, and thus predict, an individual person’s mental state or behavior (outcomes) based on their brain activity. (B) Brain maps are displayed for comparisons of brain responses between emotional faces and shapes, reward and punishment (Barch et al., 2013), and painful pressure applied to the thumb and rest (study 5 from Kragel et al., 2018). (C) Brain models can vary in complexity, ranging from the average activity of individual brain regions (e.g., a bilateral amygdala mask [left] (Swartz et al., 2015)) to more complex patterns of brain activity optimized through statistical learning procedures (e.g., the top 1% of voxels that predict crowdfunding choices [center] (Genevsky et al., 2017), or the Neurological Pain Signature [right] (Wager et al., 2013)).

Some models are simple, associating activity in a single brain region with an outcome. But increasingly, brain models are multivariate: they explain outcomes as patterns of brain activity and/or structure across large numbers of brain features, often distributed across anatomical regions and systems, or even types of measurements (fMRI activity, connectivity, structure, and/or neurochemistry). Multivariate models have been developed for diverse types of mental events and states—including object recognition (Haxby et al., 2011), speech content (Formisano et al., 2008), wakefulness (Tagliazucchi and Laufs, 2014), autonomic responses (Eisenbarth et al., 2016), memory (Harrison and Tong, 2009; Polyn et al., 2005), decision making (Hampton and O’Doherty J, 2007; Kahnt et al., 2011), semantic concepts (Huth et al., 2016; Mitchell et al., 2008), cognitive tasks (Poldrack et al., 2009), attention (Esterman et al., 2009; Rosenberg et al., 2016), pain (Marquand et al., 2010; Wager et al., 2013), prosody (Ethofer et al., 2009), emotion (Kragel and LaBar, 2015; Saarimaki et al., 2016; Wager et al., 2015), empathy (Krishnan et al., 2016), and the content of dreams (Horikawa et al., 2013). They have also been applied to diverse neurological and mental disorders (for reviews, see Arbabshirani et al., 2017; Woo et al., 2017a).

In this review, we discuss the theoretical underpinnings that make multivariate brain models an informative and powerful approach, and provide a brief history of the expanding set of modeling tools and approaches in the field. We also explore the promise and challenges of a specific type of model, brain ‘signatures’ or neuromarkers, which identify brain patterns that predict mental and behavioral outcomes across individuals (Gabrieli et al., 2015). In the application areas listed above and beyond, the reversal of predictors and outcomes affords several advantages: 1) a better match to how mental and behavioral information is encoded in neurons; 2) larger effect sizes in brain-outcome associations than standard local region-based approaches; 3) quantitative predictions about outcomes that can be empirically falsified; 4) models with defined measurement properties that can, under certain use cases, be tested and validated across studies and labs; and 5) a path towards validating mental constructs and understanding how the brain carves the mind at its joints – i.e., which psychological distinctions are paralleled by strong neurological ones (Lenartowicz et al., 2010).

Finally, we discuss the difficult issues surrounding mental constructs and their validation, and how predictive brain models can help redefine the way we categorize and understand the mind. The progression from brain maps to models of mental states provides a strong foundation for empirical and theoretical development. But it also raises some fundamental issues about how researchers define and evaluate mental constructs, and what it means to identify a ‘brain representation’ that underlies them (see Appendix for key definitions). As the science of multivariate brain models develops, the field must grapple with these questions. Scientists are already engaged in the demanding work of iteratively identifying potential mental constructs, developing neural measurement models for them, and validating, refining, and redefining those constructs based on empirical data. An explicit formalization of this process can identify gaps in current research and accelerate progress towards a fundamental goal of cognitive neuroscience and related fields: establishing mappings between mind and brain.

Theoretical Assumptions about Neural Representation

Brain mapping in neuroimaging emerged from a convention of thinking of mental processes as being modular and implemented in isolated, local brain regions. This view is grounded in longstanding assumptions in philosophy of mind (Lindquist & Barrett, 2012) and studies showing that lesions of distinct cortical areas produce deficits in speech production, language comprehension, perception and action (reviewed in Banich, 2004; Brett et al., 2002). This work supported the notion that the brain can be thought of as a collection of functional modules—independent, separable processing units that access one another’s inputs and outputs, but not intermediate processes (Fodor, 1985; Marr, 1977). Though challenged on theoretical grounds (e.g., see Barrett, 2009a; Jonas and Kording, 2017; Sarter et al., 1996), this basic assumption was adopted in early neuroimaging studies, and it became popular to analyze brain-mind associations by analyzing each brain voxel independently (Brett et al., 2002; Logothetis, 2008).

In contrast, multivariate predictive models emerged from theories grounded in neural population coding and distributed representation. Neurophysiological studies have established that information about mind and behavior is encoded in the activity of intermixed populations of neurons. Many studies identify information encoded in single neurons—but often, activity in even the most stimulus- or task-predictive individual neurons contains too little information to accurately predict behavior. A literature on population coding demonstrates that behavior can often be more accurately predicted by joint activity across a population of cells (Averbeck et al., 2006; Pouget et al., 2000), including motor control (Georgopoulos et al., 1986), face perception and identification (Chang and Tsao, 2017; Young and Yamane, 1992), object recognition (Hung et al., 2005; Kiani et al., 2007), control of eye movements (Lee et al., 1988), odor perception (Iurilli and Datta, 2017), numerosity (Tudusciuc and Nieder, 2007), and more.

Population-coding studies have shown that most cells are not strongly selective for a single stimulus or action category, such as object type (Kiani et al., 2007) or saccade direction (Sparks et al., 1990), but rather respond to complex combinations of categories (Rigotti et al., 2013). Firing rates for non-preferred categories are stable and reproducible, and including them in predictive (i.e., ‘decoding’) models results in stronger classification performance than models containing only strongly category-selective neurons (Kiani et al., 2007). Further, they provide strong classification performance after removing strongly responsive neurons from predictive models (Rigotti et al., 2013; Tudusciuc and Nieder, 2007). Deactivation of neurons strongly responsive to one category (e.g., one saccade direction) does not eliminate that category of responses, but rather results in predictable shifts in behavior consistent with population coding (Lee et al., 1988). And in addition to variation in the mean activity in populations of cells, co-variation between neurons is also important (Ni et al., 2018). These findings indicate that when it comes to information encoding, the whole is often greater than the sum of its parts.

Population codes have several adaptive benefits that may have driven their evolution, including robustness, noise filtering, and the ability to encode high-dimensional, nonlinear representations that can be used flexibly (Pouget et al., 2000), Distributed representation permits combinatorial coding (Osborne et al., 2008), providing the capacity to represent a great deal of information with limited neural ‘real estate’. Neurons are elements that can be combined into a nearly infinite number of system states, exponentially increasing the network’s coding capacity (Rolls, 2007). Such generative systems are pervasive. For example, 26 Latin letters are the basis for all the words in the English language. By contrast, the Chinese Zhonghua Zihai character dictionary includes over 85,000 logograms, each representing a single word or concept (Russell and Cohn, 2012).

These advantages have inspired artificial neural networks that capitalize on these principles (O’Reilly et al., 2012; Rumelhart et al., 1986). Neurons in these models encode features of input objects (e.g., images, text, etc.) in a highly distributed, ‘many-to-many’ fashion. Each neuron represents many object features, and a representation of an object feature is distributed across many neurons, providing a rich way of representing similarities and associations across objects. Neurons in deep network layers encode complex combinations of features, which has proven critical for the improved predictive accuracy of deep learning models relative to other models (Krizhevsky et al., 2012; LeCun and Bengio, 1995; LeCun et al., 2015). Such models can also be used decode, and create, never-before-seen objects (Gregor et al., 2015; Miyawaki et al., 2008; Nguyen et al., 2016).

Multivariate modeling of how activity spanning many brain voxels jointly encodes behavioral outcomes in human neuroimaging is an extension of population coding concepts in cellular neuroscience. Because human neuroimaging provides an indirect measure of neural activity that is more consistent with local field potentials and bulk calcium imaging than the activity of single neurons (Logothetis et al., 2001; Nir et al., 2007; Schulz et al., 2012), the activity in any individual voxel is not viewed as indicative of any specific computation or process, but as part of a distributed representation that can be dynamically transformed during cognition to accomplish different functional tasks. Rather than attempting to localize independent functional modules, as is done in conventional univariate approaches, multivariate methods characterize relationships between distributed patterns of activity and categories of mental events and behaviors (Haynes, 2015; Norman et al., 2006; Poldrack and Farah, 2015).

A Brief History of Multivariate Brain Models

Advances in Multi-Voxel Pattern Analysis

Multivariate brain models are a diverse family of models, encompassing multiple goals and analytic approaches. One goal is to accurately predict outcomes (i.e., maximize variance explained by the model), which is useful for future prediction (prognosis). But there are other, complementary goals. Models can be designed to (a) generalize to new groups of people, mental states or behaviors, or testing contexts; (b) discriminate one category of mental events or behaviors from another; and/or (c) be more or less easily interpreted in the context of other neuroscientific data and validated against other findings. Models that are accurate, generalizable, and interpretable provide more than predictions; They provide explanations for the neural bases or mental events. Models also vary in the assumptions they make about how mental events are represented in the brain, with different goals and assumptions indicating different study designs and analytic methods.

Over the past two decades, the variation in the types of models employed has grown dramatically, as assumptions about how the brain represents mental events have changed and some goals originally thought to be unreachable now seem possible. Figure 2a shows a timeline of some of the most important developments, and corresponding choices about model goals and structure. We group these advances into multiple stages, each adding a set of techniques to the neuroscientist’s toolbox.

Figure 2.

Advances in multivariate brain modeling. (A) Timeline of methodological developments in predictive brain modeling. Advances in predicting behavioral and mental outcomes are influenced by ideas about local coding, distributed coding, and generalizability. These ideas fostered complementary tools for analyzing brain data. (B) Decisions involved in developing multivariate models of brain activity. Three classes of decisions involve the intended generalizability of a model (should it work for a single individual, or a whole population?), the spatial scale of modeling (should activity within a local searchlight, a single brain system, or the whole brain be modeled?), and the complexity of relationships linking brain and mind (e.g., should a linear or quadratic function be used to map brain activity to model outcomes?). The right most column depicts multivariate brain models that are the result of different methodological decisions.

Local Information Coding Within Individuals

Early studies were grounded in the assumption that information is primarily encoded in local brain regions, in the activity of functional neuronal columns and ensembles with a fine spatial scale (i.e., ~1 mm or less, depending on the system; Duong et al., 2001; Fukuda et al., 2006) and whose precise topography varies across individuals (Issa et al., 2013). Modeling efforts thus focused on predicting mental states within individuals in spatially localized regions. The goal was not to develop a useful overall model of perception or behavior, but—as in traditional brain mapping—to understand local brain representation.

Using this approach, several seminal papers showed that brain activity in early visual cortex could be used to predict the orientation of line gratings a person was viewing (Kamitani and Tong, 2005), other work demonstrated that activity patterns identified in this way could be used as probes of working memory. For example, models developed to identify the perceived orientation of line gratings could be used to infer the contents of working memory in the absence of visual stimulation (Harrison and Tong, 2009). These studies and seminal work in other domains (e.g., Hampton and O’Doherty J, 2007; Kay et al., 2008; Kuhl et al., 2011) have helped establish predictive analyses within local regions of interest as a way of understanding local representation of mental events. An extension, searchlight mapping (Kriegeskorte et al., 2006), involves multivariate prediction within local spherical ‘searchlights’ across the brain to construct brain maps of where information about a mental/behavioral outcome is encoded, which has become a popular technique for mapping local brain information content (Haynes et al., 2007; Peelen et al., 2010; Rissman et al., 2010).

Although these studies illustrated a groundbreaking new approach, they are limited in some important ways. First, showing that a local fMRI model predicts an outcome above chance does not, in itself, license the use of the model as a marker, or proxy for a brain representation. Using a brain model to infer the presence or strength of a mental event requires assuming—or, ideally, showing—that (a) the putative brain marker is causally related to the mental event, rather than confounding processes; (b) the brain marker is sufficient to capture the brain representation of the mental event and detect it with high sensitivity; and (c) the brain marker is specific to the mental event of interest (Davis et al., 2017; Woo et al., 2017a). The brain marker has high positive predictive value for the mental event – an indicator that activation of the brain marker implies that the mental event occurred – if and only if the latter two criteria are met. These criteria are particularly difficult to meet in single-subject, local decoding models.

Another limitation is that because single-subject decoding identifies a different model (e.g., different pattern of parameter estimates based on the observed fMRI activation) for each individual participant, it allows a great deal of flexibility in capturing artifacts and confounding processes not related to the mental process being studied (Davis and Poldrack, 2013; Gilron et al., 2017; Todd et al., 2013). Potential confounds (e.g., time-varying effects such as learning, habituation, fatigue) are not typically modeled in local decoding studies, and avoiding systematic biases requires specialized experimental designs (e.g., within-person counterbalancing and stratification on confounds) that may be impractical in many cases. Individualized models also cannot be tested for accuracy, generalizability, or susceptibility to confounds in new studies, without bringing the same individuals back for re-testing. That is, it is possible to replicate the finding that premotor activity predicts future choices (Haynes et al., 2007; Soon et al., 2008), but not possible to test whether the precise models used for each individual represented intended actions or, rather, another correlated process unrelated to willed action.

Finally, local prediction relies heavily on the assumption that information is contained predominantly in fine-grained, local patterns. If this is not the case, then predictive models will perform poorly if restricted to focal brain regions. And the lower the decoding accuracy (or related measures of effect size; see Box 1), the more likely it is that the brain measure is too noisy to serve as a proxy measure, the brain area targeted may play only a minor role in representing the mental process, or the association between brain and mental/behavioral outcome is artefactual. Unfortunately, it is not easy to know how accurate local searchlight models are in much of the published literature, because post-hoc effect sizes in searchlight maps are optimistically biased if significant regions are selected from among many regions tested (Reddan et al., 2017).

Box 1. Effect Size as a Function of Spatial Scale.

Measures of effect size (e.g., Cohen’s d, Pearson’s r, or Glass’ Δ) indicate the strength of an observed relationship independent of sample size. The goal of predictive modeling is to develop models of brain activity that can detect specific mental states with large effect sizes.

Although many neuroimaging studies report effect sizes, most are not appropriately designed to estimate them in an unbiased manner (Reddan et al., 2017). Most studies report effects for a small subset of the most significant voxels from mass-univariate tests, which are not representative of future performance due to a voxel selection bias that makes them over-optimistic (Varma and Simon, 2006). This bias can be avoided by nested cross-validation and prospective, out-of-sample testing.

Multivariate brain models that accumulate information from many different neural sources increase sensitivity and effect sizes when the brain information of interest is distributed across regions. Recent studies examining brain representations of negative emotion (Chang et al., 2015) and vicarious pain (Krishnan et al., 2016) have shown that predictive models utilizing information spanning the entire brain perform better than models based on local patterns of activity within spherical “searchlights” (Kriegeskorte et al., 2006). These results suggest that complex emotional states are best characterized in terms of global brain states, as opposed to activity within localized neural substrates (Barrett, 2017).

Figure I.

The effect of spatial scale on model performance. The plot shows the average cross-validated performance (Pearson’s r, averaged over 500 iterations) of models designed to predict pain reports following thermal stimulation using a 2-fold subject-independent cross-validation (data from study 2 of Wager et al., 2013; n = 33). The x-axis denotes the number of voxels used in each model, which were sampled randomly from a uniform distribution spanning the entire brain (black) or individual resting-state networks (colored lines, inset render shows each network from a medial view). Solid curves display parametric fits of the form A – Be^−v/C, where v is the number of voxels, A is the performance of the whole-brain model, B is the performance of a single voxel, and C determines the rate of increase. Sampling voxels from the whole brain produces the most predictive models, compared to sampling within a single resting-state network or searchlight, although only ~1,000 randomly sampled voxels are needed to achieve this performance.

Within the extremes of local searchlights and whole-brain models, predictive models can decode information at multiple different spatial scales. Figure I depicts how information is encoded across multiple brain systems, by plotting the performance of pain-predictive models developed using brain activity from multiple resting-state networks (i.e., uniform random sampling from the entire brain) versus models that sample from a single resting-state network and the most predictive searchlight from the entire brain. Models using features that span multiple networks ($\stackrel{-}{r}$ = .544 for the whole-brain models) have larger effect sizes than those using features from a single network ($\stackrel{-}{r}$ = .487 for the visual network models), indicating that information about pain experience is contained in patterns of activity that span multiple brain systems.

Thus, the development of brain measures as indicators for mental constructs like perception, working memory, pain, etc. is an important goal, but it requires inferences that are difficult to establish in single studies, let alone single participants. Several developments in the field address different aspects of these limitations, as we describe below.

From Local to Brain-Wide Decoding

Rather than focusing on individual regions, other studies started from the assumption that information is encoded in distributed brain systems, and that characterizing complex behaviors may require models that capture patterns of activity across these systems. This assumption leads to models that make predictions based on joint patterns of activity and/or connectivity across many voxels (currently up to hundreds of thousands) spanning the brain.

This approach was slower to develop, in part due to the potential for overfitting when there are many more model parameters (e.g., voxels) than observations, producing models that do not generalize well (for more explanation in the neuroimaging context, see Pereira et al., 2009). However, machine learning techniques that regularize or reduce the complexity of predictive models with large numbers of features help to overcome this challenge and make whole-brain models viable (Gramfort et al., 2013; Grosenick et al., 2013; Michel et al., 2011). Studies emerged using brain-wide patterns to decode the contents of memory (Polyn et al., 2005) and semantic information (Mitchell et al., 2008), and differentiate among cognitive task types (Poldrack et al., 2009). More recent studies have shown that signals containing information about reward and punishment (Vickery et al., 2011), working memory (van Ast et al., 2016), semantics (Huth et al., 2016; Huth et al., 2012), pain (Brodersen et al., 2012; Marquand et al., 2010), sustained attention (Rosenberg et al., 2016), and other functions are not confined to single brain regions or systems, but are widely distributed across brain regions. A new direction is direct comparisons of models that operate at different spatial scales, permitting inferences about where and how broadly mental/behavioral information is encoded (Swisher et al., 2010). Early model comparison studies suggest that information about at least some classes of mental events is indeed distributed across regions and systems (e.g., Chang et al., 2015; Krishnan et al., 2016; see Box 1).

Broadening the spatial scale of a model does not, in itself, address limitations inherent in single-subject models, including (a) susceptibility to confounds (especially diffuse neuromodulatory effects that are not specific to the target mental state), (b) poor interpretability of the model parameters (i.e., patterns within and across voxels), and (c) inability to test the generalizability and specificity of already-trained models across participants, contexts, and types of mental events. In addition, despite regularization and related modeling techniques, estimates of these parameters are generally noisier than standard univariate maps and their interpretation is more complex. For example, brain features important in the model may capture and control for sources of noise in the data, rather than being directly related to mental events (Haufe et al., 2014).

The interpretability of model parameters is particularly complicated when nonlinear mappings, such as those implemented by commonly used Radial Basis Function Support Vector Machines and increasingly popular deep neural nets, are used to predict mental states from brain activity. These approaches create mappings between parameters (generally, brain features) and outcomes that are complex and nonmonotonic (Kamitani and Tong, 2005; Norman et al., 2006). A classic example of this problem involves decoding object identity from retinal activity. A complex non-linear model can use retinal activity to predict the semantic category of object one is viewing, even though individual neurons in the retina do not respond based on semantics. The representation of categories is encoded in the model, but not directly in the activity of retinal cells. A linear model will fail to predict semantic categories—an advantage in this case—because it relies only on information encoded in the system in a linear fashion. Despite this challenge facing nonlinear models, seminal studies using deep nets to model processing in the ventral visual pathway have shown a striking convergence with biological data (DiCarlo et al., 2012; Horikawa and Kamitani, 2017). More generally, issues related to interpretability can be partially addressed by assessing their relationship with univariate encoding weights (Haufe et al., 2014; Jimura and Poldrack, 2012; Woo et al., 2017b) and evaluating the reproducibility of model weights across individuals (Strother et al., 2002; Varoquaux et al., 2017).

From Modeling Individuals to Populations

Given the limitations raised above, researchers have increasingly focused on identifying models that generalize across individuals. Models that predict outcomes in a group of subjects are constrained to have identical model parameters and estimates across individuals, reducing idiosyncratic artifacts (Todd et al., 2013) and increasing interpretability. Additionally, model performance can be tested on out-of-sample individuals, yielding estimates of person-level performance as is the case for diagnostic tests used in medicine.

This approach assumes that there is useful information contained in patterns of brain activity that are consistent across individuals—meso-scale and systems-level activity. In spite of initial assumptions to the contrary, cross-subject decoding proved effective in a number of different areas, including identification of attentional states (Mourao-Miranda et al., 2005), detecting the semantic category of perceived objects (Shinkareva et al., 2008), and diagnosis of dementia (Davatzikos et al., 2009), depression (Drysdale et al., 2017), chronic pain (Baliki et al., 2012; Mansour et al., 2016; Tetreault et al., 2016), and other clinical outcomes (for reviews see Gabrieli et al., 2015; Woo et al., 2017a). These studies shaped the landscape of predictive models by establishing population-level models that generalize across participants.

One potential disadvantage is that population-level models are not always as predictive as individualized models (e.g., see direct comparisons in Clithero et al., 2011; Haxby et al., 2011; Lindquist et al., 2017; Shinkareva et al., 2008). One important limitation is inter-subject variability in structural and functional anatomy that reduces generalizability across subjects (Cox and Savoy, 2003). Statistical theory shows that the relative costs of between-person relative to within-person prediction depends on the ratio of between-person variance (individual differences) to within-person variance (individual measurement error; Lindquist et al., 2017). Larger individual differences and the ability to collect large amounts of data per person shift the advantage towards within-person models. However, as the amount of data that can be collected on one individual is often limited, as in standard brief ‘localizer’ tasks, in some cases between-person models perform nearly as well as within-person models (Lindquist et al., 2017) or are actually more accurate (Chang et al., 2015). Several new models, including hyperalignment (Haxby et al., 2011) and other methods that align functional (rather than anatomical) regions across participants, can dramatically reduce inter-subject variability in functional anatomy, increasing the accuracy and specificity of population-level models.

Several other important limitations remain, including limitations in interpretability (Haufe et al., 2014), the potential for confounds, and questions about whether a given model generalize to different contexts (i.e., predict related outcomes in superficially different settings). However, these limitations can be partially overcome with considered choices of training and testing data and model structure, as we describe below.

Generalization Across Contexts

A major challenge with all types of predictive models is ensuring that they reflect a particular target mental process (e.g., pain, attention, etc.). In some cases, the model may track correlated superficial variables, and in others, it may track the mental process in only one context. For example, is a brain classifier that predicts when individuals are viewing angry vs. fearful faces picking up on the concepts of ‘anger’ and ‘fear’ in general, or rather some particular aspects of the faces and eye-movement patterns during viewing? An important direction in multivariate modeling is to explicitly train models that are robust to variations of the experimental context—e.g., angry vs. neutral pictures, sounds, memories, etc. Systematically generalizing over experimental contexts makes models more likely to reflect a targeted mental category, rather than correlated sensorimotor and cognitive processes.

Several recent studies develop models that generalize across superficially different exemplars of a mental process. Work on modeling emotion categories (fear, anger, etc.) has trained population-level models to generalize across music and film clips (Kragel and LaBar, 2015), short movies and mental imagery (Saarimaki et al., 2016), and diverse emotion-induction methods (Wager et al., 2015). Other studies have predicted emotion and affective valence in a way that generalizes across visual and gustatory stimuli (Chikazoe et al., 2014); face, voice, and body cues (Peelen et al., 2010); and across direct perceptions and causal inferences about context (Skerry and Saxe, 2014).

As with other approaches, there are limitations here as well. Different manipulations may prove more or less effective at inducing targeted mental states; for example, video clips are frequently more robust elicitors of emotions than mental imagery or autobiographical recall (Westermann et al., 1996), and certain types of experience may be more difficult to manipulate with certain types of stimuli (Wager et al., 2015). This introduces a confound between category and intensity, which can be accounted for by (a) approximately matching stimuli from different categories on intensity, and (b) controlling for intensity during model training. The same principle applies to other potential confounds, and while prior work has accounted for them in some cases, future modeling work should consider them carefully. In addition, it is often infeasible to manipulate more than a few variables in a single study. Combining contextual variation with population-level modeling can help by integrating data across multiple studies with more contextual heterogeneity in the combined dataset (e.g., Kragel et al., 2018). Finally, some mental constructs are hypothesized to vary as a function of context (e.g., Barrett, 2017; Barrett and Satpute, 2017; Barsalou, 2008; Wilson-Mendenhall et al., 2011). For example, ‘anger’ and ‘fear’ are intrinsically linked to different action tendencies, and it may be impossible to separate emotions from these tendencies.

Brain Signatures: Towards Strong Inference in Relating Mind to Brain

Though multivariate brain models come in many varieties, a primary goal is to predict mental events and thereby understand the brain representations underlying them. This can include (a) detecting whether a mental process has been engaged; (b) inferring the strength or intensity of engagement; (c) inferences about which mental categories are similar or distinct in their brain representation; (d) inferring how changes in context or treatments affect the engagement of a mental process and its brain representation; and more.

We argue that a particular class of models—brain ‘signatures’—is especially useful for this purpose. This class of models uses distributed information within and across brain systems to make population-level, between-subject predictions about the strength of engagement of a mental process, ideally across contexts, and to distinguish it from other categories of mental events (see Box 2 for guidelines on developing brain signatures). Such signatures are effectively brain biomarkers, or ‘neuromarkers’ (Gabrieli et al., 2015). In our usage, the terms ‘signature’ and ‘biomarker’ are essentially interchangeable. A brain signature is not assumed to be unique to a particular mental process in the way that a handwritten signature is unique to a person. Its sensitivity, specificity, generalizability, and other measurement properties are empirical matters. Likewise, biomarkers in medicine may be more or less accurate, more or less specific to a particular disease, etc. Signatures with desirable properties should be carried forward and tested more extensively, and those with poor measurement properties discarded or redefined (Woo et al., 2017a). By extension, just because model development targets one type of mental event, we should not assume that the targeted event class is the best description of what the signature measures. For example, ‘pain signatures’ trained to track pain may measure the engagement of attention or negative affect. Testing alternative psychological descriptions is also an empirical process, the heart of what we refer to as construct identification below.

Box 2: Guidelines for the Development of Multivariate Brain Markers

Although the exact approach depends on the goal of a specific project, some general guidelines—outlined below—apply to many multivariate model-development efforts.

Choice of outcome:

Outcomes can be either categorical or continuous, and can vary within-person, between-person, or both. One common approach is binary classification with two categories (e.g., two task conditions, or patients vs. controls). These models can be useful, but they may often yield brain measures that are not specific. ‘Greedy’ classifiers tend to reflect a range of processes that differ between the two conditions. For example, classifying pain vs. no-pain conditions can capture activity related to arousal, and attention as well as pain. Regression models trained on parametric variations across multiple conditions (e.g., pain intensity ratings) offers a partial solution, and allow for the estimation of dose-response relationships between brain responses and outcomes.

Choice of experimental conditions and level of analysis:

Models that capture variation of interest across multiple outcome levels (e.g., intensity of affect or cognitive performance) within-person can increase specificity and permit the evaluation of sensitivity across a dynamic range. This also reduces the potential for floor and ceiling effects. And importantly, training models on within-person outcomes reduces sources of noise that can make prediction of between-person differences intractable, including nuisance variation in both outcome and brain measures: for example, (a) scale rating usage (for self-reported assessments) and (b) vascularization, hematocrit, receptor density, arousal level, and other factors that produce inter-personal variation in brain responses.

Designing for specificity:

In addition to target conditions of interest, it is also desirable to include a range of ‘foil’ conditions that are similar to the outcome of interest (i.e., engage some overlapping processes). For example, when predicting taste aversion, foils could include other aversive conditions (pain, touch, pictures, sounds) and other attention-demanding conditions. Including such foils during model training can increase specificity.

Designing for generalizability:

Including in training datasets multiple instances of the outcome of interest that vary superficially – e.g., different subjects, studies, populations, and task variants—can increase the generalizability of the model to a broader range of cases.

Avoiding bias in evaluating performance:

Cross-validation can be a valid and reasonable way to estimate the performance of a classifier in new samples. During cross validation, one part of the data forms the training set (e.g., some participants, in models designed to generalize across participants) and the remaining data forms the test set for performance evaluation. The procedure is repeated for several repetitions (‘folds’), until all samples have served as part of the test set. It is useful to stratify holdout sets on the outcome, keeping training and test sets balanced on outcome values, and balance outcomes on potential confounds if possible.

Cross-validation is useful, but it can provide over-optimistic estimates when (a) there is dependence among observations in training and test sets; (b) the intent is to generalize to populations that are not identical to the original one (e.g., different demographics and sample characteristics); and (c) fitting multiple models on the same dataset, e.g., to optimize parameters or feature selection. The latter is a form of researcher-induced bias that is often unreported in papers, making it difficult to assess whether reported results are optimistic or not.

We offer the following recommendations to mitigate dependence-related issues: Prediction of time series data should control the influence of auto-correlation, for instance by using rolling-hvg block cross-validation (Racine, 2000) or leaving out entire runs. Ideally, prediction should be performed across participants if possible, depending on study goals (e.g., 5-fold cross-validation, leaving out all images collected from subjects in the test set). Dependence across participants can be related to cohort effects (time of day, experimenter, time of year), family/twin structure, and other factors. If dependencies exist between participants (e.g., dyads, twin samples), those should be part of the same holdout sets. Dependence across participants can also be caused by mean-centering, z-scoring, or removing covariates estimated on the full sample; caution should be exercised when applying these types of transformations.

Permutation testing:

Permuting the order of outcome labels can be used to test for bias and statistical significance of the overall model. In the absence of bias, the permuted distribution should be centered on chance. For such tests, assessing root mean squared error is desirable, as correlations between predictions and outcomes can be negatively biased and accuracy is a coarse (imprecise) metric with small samples.

Independent hold out sets:

To guard against over-optimism, performance should further be evaluated on a completely independent hold-out test set, which should be tested only once to assess the final performance. All training, feature selection, and tweaking of machine-learning parameters should be done in one part of the data, and then tested only once in a completely novel, unseen dataset.

Prospective testing:

given the early stages of brain patterns as biomarkers or diagnostic patterns, it will be crucial to keep testing and externally validating existing markers on new and independent datasets (in the sense of both convergent and discriminant validity) and to test the effects of different types of experimental manipulations and other factors. Prospective application requires creating a method (e.g., software) for applying the model to a new individual case and obtaining a model prediction, without reference to any normative sample or outcome information. We encourage the creation and sharing of such methods to encourage prospective testing.

In addition, a signature may not be a complete description of a mental process. It may be useful as an indicator without capturing relevant brain processes, just as a disease biomarker need not capture all aspects of disease physiology. Thus, there is ample room for multiple signatures for the same class of mental event.

The specific modeling choices involved in constructing brain signatures—distributed information and population-level modeling—allow these various types of empirical development and validation to occur across studies and laboratories, dramatically enhancing the ability to (a) falsify models by making strong predictions, (b) develop models with desirable measurement properties, (c) establish reproducibility across studies, (d) use pre-defined models as targets for interventions, and (e) identify the psychological constructs that are measured by brain signatures and develop new psychological ontologies.

Falsifying Models

Thinking of brain signatures as measures highlights one of their main advantages: they provide specific, quantitative predictions that can be tested and falsified. This enables a cycle of model formulation and rigorous testing that has been crucial for scientific advancement across fields (Platt, 1964). In physics, it has led to strong, theory-driven predictions that were empirically tested only many years later, like Einstein’s prediction based on the general theory of relativity (Einstein, 1916) that starlight should bend around the sun as it reaches earth. This prediction was tested years later by Eddington (Dyson et al., 1920), validating the theory’s predictive utility.

For example, a ‘pain signature’ should respond whenever pain is strongly believed to be present, but not otherwise. If it does not respond to pain, and methodological errors can be ruled out, then the hypothesis that the signature reflects all types of pain can be ruled out, paving the way for new refinements. Alternatively, the signature might reflect only some types of pain or pain from some sources, leading to new hypotheses that the brain includes multiple distinct processes that can be labeled as painful. If the signature responds to events that are clearly not painful, like aversive images, bitter taste, or breathlessness, then the signature can be falsified as being unique to pain, and the understanding of what it measures can be refined.

Focus on Measurement Properties

Because much of the focus in both brain mapping and multivariate searchlight approaches has been on explaining local brain representations, relatively little attention has been paid to the measurement properties of brain signals as defined by signal detection theory (McNicol, 2005) –e.g., their sensitivity, specificity, positive predictive value for behavior, and generalizability. A second advantage of brain signatures is the fact that they have definable measurement properties that allow models to be empirically tested in subsequent research.

In addition, a closer match between multivariate patterns and underlying neural representation naturally leads to better measurement properties. This point is supported by studies of hyperacuity, the observation that multivariate models are sensitive to information coded at spatial resolutions finer than the resolution of neuroimaging data acquisition (Carlson, 2014; Kamitani and Tong, 2005; Wardle et al., 2017). It is also supported by observations of multiscale sensitivity, where distributed information within and across regions provides better prediction than single regions. These advantages have resulted in larger effect sizes for brain-wide multivariate compared to local multivariate and univariate models in direct comparisons in several studies (e.g., Chang et al., 2015; Krishnan et al., 2016)(see Box 1). Because the models that best match underlying neural representations are likely to fit best, model comparisons—univariate vs. multivariate, local region vs. distributed network—provide a way to use neuroimaging to probe the nature of underlying brain representations.

Reproducibility

The ability to reproduce results across studies and laboratories is a key part of scientific progress. Findings that cannot be replicated may simply be false positives, or the effects may be too context-sensitive for cumulative scientific progress and utility in practical applications. Questions about the reproducibility of scientific findings have become a major issue across fields (Munafò et al., 2017). But establishing reproducible findings has been particularly problematic in fields that involve massive numbers of tests, as in neuroimaging and genetics. It is also problematic when a large number of context variables may change how a process works, as in translational neuroscience (Begley and Ioannidis, 2015) and some areas of psychology (Doyen et al., 2012; Yong, 2012). Neuroimaging-based mapping of mind to brain lies at the intersection of these danger zones.

Reproducibility is limited in standard voxel-wise maps by a combination of noisy voxel-level measurements and the massive numbers of tests involved. The more stringent the multiple comparison threshold applied, the less likely that studies with the same true underlying neural activity will yield the same results—in effect, low statistical power can ensure that each study identifies a tiny, and often different, part of the true underlying pattern (e.g., Yarkoni, 2009). By contrast, signature-based approaches integrate brain information into a single optimized prediction, and test predictions on new, independent individuals. This avoids the need for multiple comparisons and provides unbiased estimates of effect size (Reddan et al., 2017) when testing how experimental interventions affect pattern expression.

Pooling information over multiple brain regions can yield measures with much larger effect sizes. Whereas local effect sizes are limited (usually “moderate” effects around Cohen’s d = 0.5 (Poldrack et al., 2017)), brain signatures often demonstrate large effect sizes when evaluated in independent studies. For example, a pain-predictive model called the Neurologic Pain Signature (NPS; Wager et al., 2013) yields effect sizes ranging from d = 1.2 to 3.50 for high vs. low pain (Krishnan et al., 2016; Wager et al., 2013). In a recent analysis of N = 603 participants across 20 studies from different sites worldwide (Zunhammer et al., In press), the NPS response is greater for pain vs. rest in 95.4% of participants, with an effect size of Hedges’ g = 2.30, 95% CI [1.92, 2.69] across studies. The Picture Induced Negative Emotion Signature (PINES; Chang et al., 2015) differentiated emotionally negative from neutral images with an effect size of d = 4.69. The Vicarious Pain Signature (VPS; Krishnan et al., 2016) yielded effect sizes ranging from d = 1.63 to 1.75 for high vs. low observed pain (Krishnan et al., 2016; Lopez-Sola et al., 2017a).

These signatures have been evaluated in multiple studies, which have tested their properties in different ways. The NPS’s responsiveness to pain has been replicated in 14 independent published study cohorts and one large-scale analysis (Zunhammer et al., In press), which have begun to characterize its profile of sensitivity, specificity, and responses to drug and psychological interventions (e.g., Lindquist et al. 2017; Woo et al. 2017a). It responds to some interventions, including the opiate remifentanil (Wager et al., 2013), serotonin reuptake inhibitor citalopram (Ma et al., 2016b), and some conditioning paradigms that influence pain expectancies (Woo et al., 2017b). However, it is insensitive to others—including cognitive reappraisal (Woo et al., 2015), perceived control (Brascher et al., 2016; Woo et al., 2017b), reward (Becker et al., 2017), and placebo (Zunhammer et al., In press)—indicating that it tracks some of the neurophysiological processes that contribute to pain self-reports but not others. The PINES response has been doubly dissociated from the NPS, indicating that it measures a distinct set of brain processes (Chang et al. 2015), and has been used as an outcome for emotion regulation (Gilead et al., 2016). In the latter study, taking the perspective of a “tough” individual reduces PINES responses to negative images. The double-dissociation of NPS responses to somatic pain and VPS to vicarious pain has been replicated in two additional independent studies in Krishnan et al. (2016) and Lopez-Sola et. al (2017a).

Novel Targets (and Measures) for Interventions

Given the better match to underlying processes and the improved measurement properties, multivariate brain models are promising targets for causal interventions that directly or indirectly influence neural activity (e.g., neurostimulation or neurofeedback, respectively). Population-level brain ‘signatures’ are particularly useful in evaluating these properties, because they enable cross-study testing and cumulative science. Brain stimulation targeting multivariate brain models has proven effective, particularly in shaping memory performance (Ezzyat et al., 2017; Ezzyat et al., 2018; Rose et al., 2016). Often these techniques are aimed at changing activity in a single brain region, but their effects might be more widespread and instead alter representations that are distributed across multiple brain systems (Antal et al., 2008; Bestmann et al., 2004; Martin et al., 2013). In this case, multivariate markers can serve as outcome measures by measuring target processes altered by neurostimulation, and identifying brain mediators of effects on behavioral or clinical outcomes.

Construct Validation: Understanding Mental Events with Biologically Grounded Models

At the heart of the enterprise of mapping brain to mind is the definition of categories of mental events that should map onto brain processes. Painful heat, cold, and chemical stimuli all involve different peripheral receptors and populations of neurons; are the experiences they evoke all examples of a single type of ‘pain’ that is represented similarly in the brain? Or is the category ‘pain’ more like the category ‘furniture’, a convenience of human thinking and language? Conversely, we use the term ‘pain’ to describe both sensations related to bodily injury and, sometimes, romantic rejection. Are their brain representations distinct, or should the category ‘pain’ be extended to non-somatic events? The answers to these questions and others determine how we conceptualize the organization of mind and brain, and often have practical implications as well. Causing someone else somatic pain is grounds for legal prosecution; should causing emotional pain be considered equally harmful under the law?

All mental categories are ultimately psychological constructs, conceptual categories organized into taxonomies of mental events (or alternatively as ontologies; see Poldrack, 2006; Poldrack and Yarkoni, 2016). Mental constructs have traditionally been ‘folk categories’ (Barrett, 2017) defined based on similarities in phenomenology and linguistic usage rather than biological systems. Likewise, disease categories have historically been based on observable symptoms (stomach pain, shivering, etc.) rather than biological causes (bacterial, viral, etc.). Re-categorizing diseases based on their underlying biological pathology was the critical conceptual shift that separates modern allopathic medicine from previous systems for diagnosing and treating illness. Recent initiatives like the Research Domain Criterion (RDoC) framework attempt to accomplish a similar shift in how we think about mental illness (Insel et al., 2010; Insel, 2014).

Whether the outcomes are disease categories or other mental constructs, comparing brain models and their patterns of sensitivity and specificity can be used to validate existing mental constructs and even infer new ones, and thus use the brain to redefine how we think about the mind (Coltheart, 2013; Box 3). At present, studies aiming to develop multivariate brain models implicitly attempt to validate constructs, but do not systematically take advantage of construct validation theory. A paradigm shift towards explicitly evaluating brain models with a formal construct development approach may lead to a better understanding of both the brain and mind.

Box 3: Why Mental Constructs Should Be Grounded in Biology

There is a long history in science of classification, induction (Mill, 1884), and of searching for natural kinds (Venn, 2006). There are several tenets held by proponents of natural kind views that, although not necessarily valid (see e.g., Searle, 1995 for criticism), shed light on the importance of biology in understanding mental constructs (adapted from Hacking, 1991):

Independence: it is a fact about nature that there are kinds of things; the differences among things are the work of nature, whereas the recognition of those things is the act of man

Definability: there are multiple ways to characterize natural kinds (e.g., they share a causal mechanism, or they have similar properties)

Utility: depending on the purpose, some classifications of objects are more useful than others

Uniqueness: there is a unique best taxonomy in terms of natural kinds that represents nature as it is, although it is not known to us

Thus, classification can be viewed as a way of organizing objects into useful groupings. Regardless of whether there is truly an optimal classification scheme, taxonomies can be compared to one another and evaluated based on their utility. Categories are often formulated in ways that prioritize human communication, rather than advancing understanding or utility in preventing disease or promoting health. This has historically been the case in medicine and psychology. In medicine, the classification of many cancers has traditionally been based on the tissue of origin and the degree of differentiation. While this classification has been useful, evolving research has shown that classification based on the presence or absence of specific genetic or molecular abnormalities may be more useful. The development of therapies tailored to these different subtypes has revolutionized cancer treatment.

Our understanding of the mind has not yet made such a transition. Constructs like ‘pain’ and ‘emotion’ have been defined based on symptoms reported by individuals – self-reports are clustered into categories such as ‘chest pain’, ‘heartache’, ‘depression’, or ‘anxiety’. ‘Memory’ is one mental construct that has been heavily influenced by neuroscience research; types of ‘memory’ are increasingly defined based on the biological systems that underlie them (Squire, 2004). However, we do not yet have biologically grounded ontologies for many other constructs.

Construct Validation

Principled approaches to defining and validating constructs can be found in measurement theory (Cronbach & Meehl, 1955), but the strategies they use have rarely been applied to neuroscience (for a discussion, see Barrett, 2009b; Barrett, 2011). A central tenet is the acknowledgement that constructs are not directly observable; rather, they are inferred from performance on multiple measures, called indicators (see, e.g. Strauss and Smith, 2009). For example, psychometric studies assume that constructs such as ‘general intelligence’ or ‘math ability’ cannot be directly observed, but that tests of math and reading can be used as indicators that reflect latent, underlying abilities. If different types of math tests with different material and presentation formats correlate together (convergent validity), it might be inferred that they all measure (load on) the same construct, and one can develop composite measures that track the latent construct better than any single indicator. If math tests are relatively uncorrelated with another coherent set of tests (discriminant validity)—say, of language performance—it might be inferred that the tests measure ‘math ability’ instead of ‘general intelligence’, willingness to follow instructions, etc. This approach uses the similarity structure across indicators to infer the nature of otherwise unobservable constructs.

Construct validation theory provides principled ways of evaluating multivariate brain models, and a path towards using brain models to infer which constructs have coherent neurophysiological mechanisms. Brain models provide putative measures (i.e. potential indicators) of latent constructs. Just as individual test items can be pooled together to measure a construct (e.g., subscales of clinical inventories; Henry and Crawford, 2005), brain activity across voxels and systems can be combined to create measures related to latent constructs.

For instance, if the same brain measure is activated by multiple distinct types of pain, but not by manipulations of other emotional or cognitive processes, it displays both convergent and discriminant validity as a measure of the construct ‘pain’ (Kragel et al., 2018). Similarly, if a single brain measure is correlated with performance on multiple tasks requiring motor response inhibition (Wager et al., 2005), it shows some convergent validity as a measure of the construct ‘inhibition’. However, it is not clear what the boundary conditions on the construct are. The brain pattern could be related narrowly to motor inhibition; to a broader construct of ‘inhibition’ that encompasses actions, thoughts, percepts, and memories; or very broadly to ‘cognitive control’. The pattern of convergent and discriminant evidence—manipulations the model is sensitive to and specific against, respectively—identifies the construct. In the inhibition example, such an analysis suggests new tasks that would be informative; for example, inhibition of memories and other cognitive control tasks that are not obviously related to motor inhibition. Recognizing that constructs are inferred makes it clear that we need explicit strategies for inferring what brain models actually measure, and conversely refining psychological taxonomies by identifying constructs with convergent and discriminant validity at the brain level.

Seen in this light, recent studies of generalizability across contexts and stimulus modalities begin to establish convergent validity for affective valence (Chikazoe et al., 2014; Kahnt et al., 2014) and other constructs. Some studies include manipulations of multiple contexts and multiple putative constructs, establishing convergent and discriminant validity in the same study (Kragel and LaBar, 2015; Saarimaki et al., 2016). ‘Mega-analysis’ of person-level image data across studies can extend this process, allowing systematic sampling of multiple constructs each with multiple, distinct manipulations in a way that would be difficult in individual studies. For example, Kragel et al. (2018) analyzed participant-level contrast images from manipulations of pain, negative emotion, and cognitive control. Eighteen studies (with 270 participants) were selected to include three distinct methods for engaging each putative construct (e.g., noxious thermal, mechanical, and visceral stimulation for the construct ‘pain’), with two representative studies in each method. Modeling the similarity structure across constructs, methods, and studies provided convergent validity for a common representation of ‘pain’ in the anterior midcingulate cortex and ‘negative emotion’ in the ventromedial prefrontal cortex. The study also provided evidence that ‘cognitive control’ may not map as clearly onto one underlying brain representation, but should rather be subdivided into more fine-grained subtypes.

Another way of validating constructs is external validity, which involves using a brain model to predict real-world outcomes (Knutson and Genevsky, 2018). For example, brain responses in the ventral striatum while viewing items predicts later purchasing decisions (Genevsky et al., 2017); ventromedial prefrontal responses predict long-term behaviors such as attempts to quit smoking (Chua et al., 2011); and amygdala activity predicts future anxiety (Swartz et al., 2015). More complex multivariate patterns predict the progression to chronic pain (Vachon-Presseau et al., 2016), and whether prefrontal brain stimulation will be an effective treatment for depression (Drysdale et al., 2017). In another series of studies, brain signatures for six different emotion categories developed in one study (Kragel and LaBar, 2015) were applied to resting-state data in an independent study and shown to correlate with individual differences in mood and personality traits (Kragel et al., 2016). Individuals with higher self-reports of depressive symptoms had greater expression of a ‘sadness’ signature, whereas anxious individuals showed greater expression of a ‘fear’ signature.

Other studies have validated constructs using interventions. Rose et al. (2016) used searchlight mapping identify regions in which fMRI pattern activity related to an item in working memory (i.e., faces, words, or the direction of motion). They found that persistent activity drops to baseline over time, but transcranial magnetic stimulation of these regions preferentially reactivated memoranda-related patterns and enhanced subsequent memory. Interventions including brain stimulation, neurofeedback, and pharmacology can help validate brain measures by showing that they mediate intervention effects on behavior. Directly manipulating the brain also provides stronger inferences about causal effects of the brain system(s) measured.

Population-level models can bring these various kinds of validation together by allowing them to be tested across studies. The NPS has been tested on data from many laboratories around the world, allowing a provisional (and ongoing) identification of the construct it measures (see Fig. 3). It tracks pain evoked by noxious peripheral stimulation of multiple types, including thermal (Brascher et al., 2016; Wager et al., 2013), mechanical (Krishnan et al., 2016), electrical (Krishnan et al., 2016; Ma et al., 2016a), capsaicin-potentiated heat, laser, and visceral stimuli, demonstrating convergent validity for evoked pain (see Zunhammer et al., In press for a meta-analytic assessment). It does not respond to non-noxious warm stimuli (Wager et al., 2013), threat cues (Krishnan et al., 2016; Ma et al., 2016a; Wager et al., 2013), social rejection-related stimuli (Wager et al., 2013), observed pain (Krishnan et al., 2016; Lopez-Sola et al., 2017a), or aversive images (Chang et al., 2015), demonstrating discriminant validity against some kinds of related, non-somatic processes. It has external validity in predicting hypersensitivity in fibromyalgia patients (Lopez-Sola et al., 2017b), though its generalizability to other forms of clinical pain remains unknown, and in showing responses to interventions thought to modulate pain, such as the opiate remifentanil (Wager et al., 2013; Zunhammer et al., In press).

Figure 3.

Examining predictive models at multiple spatial scales. Brain-wide multivariate models can be understood by examining how pattern expression (i.e., model output for test data) varies across established brain networks and regions. (A) Brain-wide expression of the Neurologic Pain Signature (NPS), a signature developed to predict physical pain intensity (Wager et al., 2013), and the Vicarious Pain Signature (VPS), a signature developed to predict observed pain intensity, in comparisons of high versus low levels of heat pain (red) and observed-pain (purple)(data from study 1 of Krishnan et al., 2016). These two signatures are independently affected by these two manipulations. (B) Decomposing a distributed pattern into subsystems: Expression of the NPS and VPS within 7 resting-state networks (Yeo et al., 2011). Wedge plots of the same dataset depict normalized local pattern expression (using the signature weights in the local region), with red indicating positive values and blue negative values. The darker shaded area indicates the standard error of the mean across individuals. The NPS primarily has positive expression during pain in the ‘ventral attention’ and ‘somatomotor’ networks during heat-pain, and negative expression in the ‘dorsal attention’ and ‘limbic’ networks. In contrast, the VPS has more evenly distributed expression across cortical networks, with a peak in the ‘visual network’ during observed-pain. (C). Meso-scale organization: Heat pain and observed pain also have distinct profiles of local pattern responses in the diencephalon, based on an anatomical delineation of thalamic nuclei and hypothalamus into 17 distinct regions (Krauth et al., 2010; Niemann et al., 2000). NPS expression during pain is positive in many thalamic nuclei and negative in the habenula, whereas VPS is expressed most reliably during observed pain in the hypothalamus. Error bars reflect standard error of the mean. (A) adapted from (Krishnan et al., 2016). dAttention = dorsal attention, vAttention = ventral attention, Pulv = pulvinar, LGN = lateral geniculate nucleus, MGN = medial geniculate nucleus, VPL = ventral posterolateral nucleus, VPM = ventral posterior medial nucleus, Intralam = intralaminar nuclei, Midline = midline thalamic nuclei, LD = lateral dorsal nucleus, VL = ventral lateral nucleus, LP = lateral posterior nucleus, VA = ventral anterior nucleus, VM = ventral medial nucleus, MD = mediodorsal nucleus, AM anteromedial nucleus, AV = anteroventral nucleus, Hb = habenular nucleus, Hythal = hypothalamus.

Challenges and Caveats

In addition to challenges facing the interpretation of model parameters, researchers are often tempted to go beyond the data and jump to broad conclusions about the biological significance of models. For instance, if a model predicts behavior, differentiates emotion categories, etc., it might be assumed to reflect brain systems that are i) prewired or biologically determined (i.e., occur independently of learning or experience), ii) stable or invariant to context (new sample, individuals, test conditions, metabolic state of the body, etc.), and iii) superior to alternative explanations (that this classification scheme is the ‘right’ or ‘best’ classification). None of these conclusions are logical sequelae of the models or examples we discuss here. However, some are testable: new samples can be evaluated, at least in models that generalize across individuals, and context can be systematically varied. The innateness of a brain model can be indirectly inferred by evaluating it across development, or in populations with markedly different cultures and experiences. Such variations are at the core of construct development.

Towards Biologically Driven Construct Development

By explicitly identifying gaps in knowledge, research programs can move more deliberately and programmatically toward the goal of identifying brain representations for mental states and processes. This process is likely to be an iterative one: developing brain models that predict and explain mental constructs will require frequent revisions to both brain models and construct definitions. One goal is to maximize simple structure: to iteratively refine both psychological constructs and brain measures to approximate, as closely as possible, a 1:1 correspondence between them. Revisions to models will teach us about how the brain encodes mental states as we currently define them, and revisions to constructs will help us develop new, neuroscience-informed ideas about how the mind works.

Neuroimaging has contributed to a tension between psychology and neuroscience, as researchers have taken various positions on what studying the brain can tell us about the mind. One expression of this tension is a series of articles challenging whether neuroimaging has taught us anything about how the mind works (Berman et al., 2006; Coltheart, 2013; Henson, 2005; Mather et al., 2013; Poldrack, 2008; Poldrack and Wagner, 2004). While there are various empirical answers to this challenge, progress is often hard to track because it does not come in the form of proving or disproving a critical theory about the mind, but by shifts in the assumptions about how the mind works, which are often implicit, metaphorical, and embedded in our current understanding of physics, computation, and biology.

As an example, computational models of mind are grounded in the traditional concept of the five senses—a basic, implicit concept that few cognitive scientists question. But how many senses are there? Neuroscience has taught us that there is not simply one sense of ‘touch’, but multiple types of somatic sensation that are mediated by distinct pathways and mechanisms (e.g., separate pathways exist for sensation of light touch, deep pressure, painful pressure, inflammation, and other somatic events). For many purposes, including specifying the computational processes involved and predicting outcomes, these must not be lumped together – and constructs like ‘touch’ have hampered progress towards understanding and clinical applications. In addition, some neuroscience findings challenge the assumption that our five exteroceptive senses are independent from one another; for example, auditory information is encoded in patterns of activity in primary visual cortex, and vice versa (Amedi et al., 2007; Luo et al., 2010).

The same can be said of ‘cognition’, ‘emotion’, ‘memory’, ‘language’ and other mental constructs – there is meaningful variation within each mental construct and meaningful similarity amongst constructs (Barrett, 2009a; Barrett and Satpute, 2013). Biological understanding allows us to discover meaningful categories, as well as new ideas about how assumed categories (like ‘working memory’) can be affected by processes previously thought to be unrelated (like ‘inflammation’). Neuroscience can teach us much about the mind if we are open to using its insights to make novel inferences.

Conclusions

A new wave of multivariate predictive models is relating mind to brain in new, more powerful ways. In many domains of cognition, sensation, and affect, such models have highly reproducible relationships with mental phenomena and behavior. They can have very large effect sizes, and in some cases are sufficient to make accurate inferences about individual persons. They can generalize across individuals and testing contexts, providing quantitative models that can be falsified, and can have explanatory power for understanding the brain bases of mental events. This literature represents early steps in a process of construct validation, wherein mental constructs are validated against brain measures. In the future, neuroimaging research programs explicitly designed for construct validation will yield yet more generalizable and useful measures. Such models can serve as targets for interventions—psychological, pharmacological, and neurological. Ultimately, such models can also be used to refine psychological concepts to bring them closer in line with their biological underpinnings, and thus reinvent how we think about the mind.

Appendix: Definition of Key Terms

Brain representation: a latent variable that is inferred based on shared variance between measures of brain activity and outcomes of interest (e.g., observed behavior, self-report, or physiological responses)

Brain signature: a multivariate brain model that includes features that span multiple brain systems and is designed to make predictions on data from a population of individuals

Behavioral domain: a set of conceptually related observable behaviors that accomplish a common goal

Generalizability: the ability of a model to perform well when tested in different conditions, e.g., in different scanning sessions, experimental manipulations, individuals in the same study, or different studies

Mental construct: categories of mental phenomena that are inferred by the observation of multiple measurements and are not directly reducible to any single measure or indicator

Mental process: a sequence of operations (or events) thought to produce observable outcomes (e.g., behavior or brain activity)

Mental state: the status of ongoing mental processes at a given point in time

Multivariate brain model: an explicit mapping that transforms multivariate observations of brain activity (or connectivity) into an outcome of interest (either discrete or continuous).

Positive predictive value: the proportion of positive predictions (cases that a model assigns as being the positive class) that are true positives (cases that have positive ground truth labels); PPV = TP/(TP+FP), where PPV = positive predictive value, TP = true positive, and FP = false positive

Reliability: the ability of a model to produce consistent results in the same conditions

Sensitivity: the proportion of true positives that are predicted to be positive, also known as the true positive rate; TPR = TP/(TP+FN), where TPR = true positive rate, TP = true positive, and FN = false negative

Specificity: the proportion of true negatives that are predicted to be negative also known as the true negative rate; TNR = TN/(TN+FP), where TNR = true negative rate, TN = true negative and FP = false positive