The importance of task design and behavioral control for understanding the neural basis of cognitive functions

Abstract

The success of systems neuroscience depends on the ability to forge quantitative links between neural activity and behavior. Traditionally, this process has benefited from the rigorous development and testing of hypotheses using tools derived from classical psychophysics and computational motor control. As our capacity for measuring neural activity improves, accompanied by powerful new analysis strategies, it seems prudent to remember what these traditional approaches have to offer. Here I present a perspective on the merits of principled task design and tight behavioral control, along with some words of caution about interpretation in unguided, large-scale neural recording studies. I argue that a judicious combination of new and old approaches is the best way to advance our understanding of higher brain function in health and disease.

Introduction

It is an exciting time for systems and cognitive neuroscience. Methods for collecting and analyzing data are improving at a remarkable pace [1,2], and questions once limited to human and nonhuman primate studies are now being addressed in smaller animal models for which large-scale data collection and powerful circuit dissection techniques are more tractable [3^••,4]. However, the pursuit of advanced technology and ‘big data’ should not come at the expense of well-defined hypotheses and rigorous behavioral control. For although we wish to understand the inner workings of the mind, we only have access to a coarse distillation thereof, namely behavior. The degree of insight attainable in an experiment is therefore limited not by how many neurons we record, but by the quality of the mapping we can create between internal states and behavioral reports.

The first part of this review describes some principles of task design that originate in the quantitative study of perception or movement, yet should prove useful for probing processes that lie squarely in between. The second part consists of a brief commentary on recent approaches in the literature that might benefit from such ‘old fashioned’ behavioral tools. My goal is not to suppress enthusiasm for new methods or to criticize any particular approach, but to encourage renewed emphasis on smart task design and careful quantification of behavior to help make the most of these advances.

What is meant by behavioral control, and why is it important?

If we want to understand how neural activity gives rise to our sensations, thoughts, memories, and decisions, we must come to grips with the fact that everything we know about such internal processes can only be inferred by observing external behavior. This is obviously true for animal models, but also applies in human subjects for whom introspection can be misleading and whose verbal reports are just another form of behavioral assay. Making things worse, this inference problem is ill posed: many internal states could lead to the same behavioral outcome.

So how do we make progress? The key is to constrain the space of possibilities as much as possible. Psychophysics does this by (a) carefully controlling sensory input, (b) measuring behavioral responses in a principled, quantitative fashion, and (c) accounting for response variability with an underlying statistical process, which then informs the search for neural mechanism. Similarly, research in computational motor control relies on monitoring the output of the system (i.e. eye or limb movements) with high resolution, and constructing normative models that explain the variability of behavior in terms of what is being optimized, and how [5–7]. In both cases, this kind of groundwork has been highly successful in advancing our understanding of sensory and motor processes. My contention is that the same degree of rigor will be needed to support inferences about the neural basis of cognitive functions.

Lessons from psychophysics

When vision scientists gained access to the physiology of visual neurons in the mid-20th century, there was great initial excitement but also caution [8,9]. Brindley [10] and Teller [11] developed the concept of a ‘linking hypothesis’ as way to formalize what one can and cannot conclude about how perception works based on properties of sensory neurons. This idea, together with the insights of Barlow [12], Marr [9], and others, began to codify the brain–mind relationship in a way that emphasized testable predictions and an understanding of the computations the system must perform. The resulting synthesis included a recognition that strong interpretive statements about neurophysiology should be predicated on a robust characterization of behavior (e.g. psychophysics) and a mathematical framework that connects this behavior to a postulated internal state.

Several authors have recently articulated the basic principles of psychophysics as they pertain to neuroscience [3^••,13^••,14,15]. Here are a few considerations to keep in mind when designing a behavioral task, and especially when training animal subjects to perform it.

Account for errors

One of the most critical benchmarks is to ensure that subjects are working ‘at threshold’ or nearly so — that is, trained to asymptotic performance and fully engaged in the task. To illustrate this point, imagine we want to understand the stochastic process that leads to errors in a perceptual judgment. To do this we have to convince ourselves that the observed errors are not generated by a separate process, distinct from the one we wish to study. This is why, whenever possible, it is important to include multiple levels of difficulty that span the range from (near) perfect to chance performance, and to vary them randomly from trial to trial. If your subjects can perform near 100% correct on the easiest trials, even when they cannot predict trial difficulty in advance, you can rule out a large class of so-called ‘lapses’ — also termed guesses or random choices — for explaining behavior on more difficult trials.

It might seem like a maximum of 85–90% correct is sufficient, but consider that a lapse rate of 10–15% on a binary choice task — not uncommon in animal studies, particularly with rodents — implies that the subject is guessing on 20–30% of the trials (because half the guesses will be correct). Crucially, these trials are random from the experimenter’s point of view and cannot simply be detected and removed. No scientist would tolerate a software glitch that covertly replaced a substantial fraction of their data with random numbers, yet this is essentially what a high lapse rate permits, at least in principle. Although sophisticated behavioral modeling [16] can isolate the contribution of lapses from other sources of variability, this only applies to probabilities over ensembles of trials. It does not prevent lapses from corrupting attempts to relate single-trial neural activity to behavior. It is worth noting here that rodents are capable of achieving very low lapse rates, at least in certain tasks [17^•]. However, if high performance cannot be demonstrated, this may be a sign that the chosen species is not a good model for the task or process being studied.

In summary, experimenters should make every attempt to design tasks and training procedures such that subjects are capable of near-perfect performance on the easiest conditions, even though most analyses ought to focus on the difficult ones.¹ Along with selecting an appropriate task structure (e.g. 2-alternative forced choice) [3^••,18] and normative modeling framework, this strategy will facilitate an accounting of how different types of errors arise. This issue seems particularly important for interpreting large-scale, exploratory studies, which may be more likely to zero in on features of neural activity that end up being red herrings due to uncontrolled behavioral variables.

Care about time

Time is fundamental to cognition, not only in the explicit way in which it supports abilities like learning and prediction, but more generally for structuring and regulating internal processes. The real world is not partitioned into discrete trials [13^••], so the brain has evolved to implement rules for terminating or switching between processes in the absence of external cues. In many cases, simply measuring the time it takes subjects to respond can be a powerful constraint for models of the internal process governing behavior [19–21]. Most importantly for the present topic, response-time (RT) tasks [22,23] are often the best way to identify the relevant time window(s) for analysis of neural data.

For example, many types of decisions involve the accumulation of evidence up to a threshold, or bound [24]. To study this process and its neural correlates it is essential to demonstrate behaviorally that the subject is indeed accumulating evidence. With well-trained subjects and an appropriate balance between reward and punishment (e.g. to discourage fast guessing), a choice-RT design allows the experimenter to analyze neural data restricted to the epoch of decision formation, barring sensory and motor delays.

In cases where RT measurements are not attainable or inappropriate for the task, other temporal constraints can be quite useful. One approach is to ‘compel’ subjects to initiate the motor response at a specific time [25], even — remarkably — before the sensory information for the decision becomes available [26]. These designs help to isolate the decision process from motor preparation and other variables such as the speed-accuracy trade-off. Alternatively, one can embrace this trade-off and simply vary the stimulus duration, using accuracy as a function of time to infer the presence of a bound and to estimate the time window of accumulation [27,28]. To promote engagement and discourage procrastination, it is often advisable to draw durations from a distribution with an early peak and a long tail, such as an exponential. In contrast, durations that are fixed [29,30] — or uniformly distributed with a relatively long minimum [31^••] — may permit subjects to delay accumulation, or worse, vary its timing unpredictably across trials, thus complicating the interpretation of neural data.

These issues would be esoteric were it not for the ubiquity of putative integration or diffusion-like mechanisms in the literature. What is less widely recognized is that searching for their neural implementation must be accompanied by behavioral evidence that such a process is actually taking place, and in what time frame.

Lessons from computational motor control

So far I have argued that the rigorous measurement and modeling of behavior in the psychophysical tradition can benefit the study of complex cognitive functions as well. A similar payoff can accrue to those who borrow experimental methods and computational approaches from the field of motor control [5–7]. Furthermore, it turns out the motor system may offer a more direct glimpse of cognitive processing than is widely appreciated.

In a series of studies [32,33,34^•], Cos, Cisek and colleagues analyzed arm kinematics and electromyographic (EMG) activity while human subjects decided between two reach targets in a free-choice task. The choice options differed in their biomechanical costs, and the authors also varied factors such as requirements for precision and stability. The results [32,33] suggested that the decision process takes these factors into account well before movement onset, and does so very rapidly after visual specification of the choice options [34^•].

The latter study exploited the known relationship between evoked EMG potentials and corticospinal excitability, using transcranial magnetic stimulation (TMS) of motor cortex to probe the state of motor preparation at different times within a trial. Similarly, Selen et al. [35] built upon what was known about intermediate-latency and long-latency stretch reflexes [36] to discover that the motor system receives a continuous flow of information related to a perceptual task. Subjects discriminated the direction of motion in a random-dot display and indicated their choice with a reaching movement while grasping a manipulandum. Selen et al. [35] then perturbed the manipulandum at random times to initiate a reflexive EMG response whose gain reflects the current preparatory state. They found that these reflex gains tracked the evolving decision variable (i.e. accumulated sensory evidence), as though a provisional plan of action was being continuously fed to motor-related structures in the brain during decision formation — an idea supported by neurophysiology in oculomotor tasks as well [24,37–39].

These and other findings [40] are consistent with the view that decisions, at least when the outcome is communicated with a motor act, arise as a competition between provisional behaviors or intentions [41,42], rather than being completed in a space of goods [43] or abstract categories [44] prior to action selection. But no matter where the reader stands on this particular issue, the key point is that insights into ostensibly covert cognitive processing can be made through careful measurement and manipulation of the motor system, guided by normative theory (e.g. optimal feedback control) [5,36]. Just as an accounting of the source of errors is an advantage of threshold psychophysics, so can a detailed understanding of movement and its variability help limit the range of explanations applied to neuronal datasets. It seems possible that better measurement and control of motor processes could help explain some of the ‘perplexing’ features of neuronal responses in higher brain areas, features that have recently motivated a reliance on complex dynamical structures and decoding schemes.

Pitfalls and limitations of large-scale population analyses

It is tempting to think that our understanding of the brain would immediately take a giant leap forward if only we could measure more of it. To be sure, major discoveries throughout history have often followed from technological advances that allowed greater access to the components of a system. We should not, however, allow the promise of big data to excuse obfuscation or sloppy thinking [1]. Neither should the sophistication of statistical methods dictate what sort of result constitutes an established finding about neural mechanism, as opposed to a description of the data or exploratory analysis for hypothesis generation [45,46].

The reverse inference problem

Naturally, larger and more complex datasets create new challenges for statistical inference. This has been a topic of great interest in the neuroimaging community, with several high-profile critiques highlighting the perils of improper correction for multiple comparisons [47,48] and the inflation of effect size from non-independence or circularity errors [49,50]. A more general problem is the inherent weakness of so-called reverse inference [51]: the attribution of specific mental processes based on brain activity patterns. With fMRI, the validity of reverse inference depends on how frequently a pattern of activity occurs across a variety of experimental contexts — the ‘base rate’ or prior probability of the observation. Certain regions or networks tend to be active in many studies across multiple cognitive domains, thus it can be problematic to make strong claims about particular mental states based on such activation.

Although this problem is exacerbated by the coarse spatial and temporal resolution of fMRI, even cellular-resolution imaging and multielectrode physiology are not immune. Hypothetically, suppose that some interesting and novel feature of population activity is found to be associated with an experimental manipulation or task. This is a perfectly reasonable basis for forward inference (how the brain responds to the manipulation), but it does not imply that the feature underlies the cognitive process being investigated,² nor even that its presence is particularly diagnostic. Again it depends, among other things, on the base rate of the observation, which typically cannot be estimated in a single study using these techniques.

An example from the motor system may again be illustrative. Churchland et al. [52^•] reported that the evolution of population activity in motor and premotor cortex exhibits a striking rotational structure during reaching movements, one that seems well suited for generating the measured EMG responses. Several data-shuffling controls indicated that rotational dynamics are not trivially observed in any dataset of similar size and complexity, nor were they observed when the animal was stationary. In a sense, this rules out a high base rate for such dynamics, but only within the actual recorded population. One might wonder how often rotations could be extracted from populations in other brain areas, including those unrelated to movement preparation. Indeed, partly in response to this concern, Churchland and colleagues (Churchland, Lara, and Cunningham, Soc. Neurosci. Abs. 2015; Russo, Perkins, London, and Churchland, Soc. Neurosci. Abs. 2015) recently reported an absence of strong dynamics or rotational structure, both upstream (supplementary motor area) and downstream (the muscles) of primary motor cortex, even when the movement itself was periodic. These findings strengthen the conclusions of their earlier work [52^•], and provide a template for how to deal with the reverse inference issue as it applies to large-scale neurophysiology.

What we can and cannot learn from high-dimensional structures

Like Churchland et al. [52^•], many recent studies (e.g. [30,53–56]) describe the activity of a large neural population as a coordinate in a high-dimensional space, where the number of dimensions is equal to the number of neurons. Typically some form of dimensionality-reduction is performed, and the positions or trajectories within the reduced state space (and/or performance of a statistical classifier) are used to infer something about the neural representation. This strategy is clearly worthwhile —indeed often necessary — for visualizing the data and furnishing a conceptual framework in which to generate new hypotheses [45,46,57]. It is generally unwarranted, however, to jump to the conclusion that state-space dynamics or advanced decoding algorithms constitute a mechanism for biological neural computation. Take for instance the observation that a linear decoder can be trained to read out task-related variables from a population whose constituent neurons respond with complex or ‘mixed’ selectivity [53,54]. This provides a useful description of the signals available in the data, but says very little about whether the brain takes advantage of these signals in a way remotely like the experimenter does.

In an analysis of prefrontal cortex activity during two different working memory tasks, Rigotti and colleagues [53] elegantly demonstrated the potential role of nonlinear mixed selectivity, including its relationship to dimensionality and how such representations confer an advantage for linear classifiers. Intriguingly, the dimensionality of the neural representation collapsed on error trials, suggesting a behavioral role for high dimensionality. However, this interpretation suffers from an incomplete understanding of the source of errors, that is, whether they indicate failures of working memory, behavioral response selection, or were simply lapses. The fact that stimulus identity could still be decoded with high accuracy on error trials was taken as support for the second possibility [53], but this presupposes the particular read-out strategy being evaluated.

Raposo et al. [54] analyzed posterior parietal cortex activity while rats performed a multisensory event-rate discrimination task. They found that, although individual neurons did not fall into clear functional categories, the population as a whole could be read out to predict the animal’s choice and the stimulus modality (visual or auditory). However, the stronger claim [54] that activity is decoded in a particular way to support performance —or that exploration of different state space dimensions in different epochs plays a causal role — seems premature without tighter links to behavior and a biologically plausible model for combining sensory inputs and forming a decision. This brings up the related point that decoding analyses should take the perspective of the subject rather than the experimenter. The task of the subject (and thus the neural population) is to determine, from the evidence, which choice is the correct one — not which choice it is about to make, and not a task-irrelevant property of the trial such as stimulus modality [54].

Returning to fMRI, similar caveats — and other, more nuanced ones [58,59] — apply to the class of methods known as multivariate (or multi-voxel) pattern analysis (MVPA), which also uses machine-learning algorithms to extract information from a distributed pattern of activity. This type of analysis can reveal the representational content of the BOLD signal across a set of voxels, but not whether or how that information is used by the rest of the brain. Again the point here is deeper than just ‘correlation is not causation’. It is that the way a classifier solves a problem — ignoring connectivity and limitations on synaptic plasticity — likely bears no resemblance to what the brain is actually doing, and may not say much about where it is doing it [58].³

Lastly, at the risk of being accused of Luddism, I would submit an appeal to the simplicity and directness of an earlier time. Imagine if Hubel and Wiesel had recorded many neurons simultaneously while animals viewed natural movies. Complex dynamics would surely have been observed, and a support vector machine might have been put forth as a candidate mechanism for vision. But one wonders what would have become of the simple and powerful insights arising from the selectivity of single cells for orientation, motion, color, and binocular disparity [60] — not to mention faces, toilet brushes and so on. The trajectory of systems neuroscience may have been much different.

Of course this is somewhat tongue-in-cheek, as cognition is not vision (although it seems unlikely that evolution would produce radically different neural mechanisms for the two functions [41]). Arguably, however, the framing of hypotheses in terms of computations reflected in the activity of single neurons [12,61–64] was not from a lack of imagination or statistical prowess on the part of the founders of systems neuroscience. It was based on careful observations and reasoning about the structure of the nervous system and the specific computations required of a given task.

It has been said that computations are the bridge between neural circuits and behavior [65], but not all computations are equally likely to serve this purpose. As the field continues to enumerate a set of fundamental or ‘canonical’ neural operations [13^••,65–67], it is worth being mindful and transparent about the distinction between a tool employed by scientists and a mechanism employed by the brain. It may turn out to be prophetic that the statistics of population activity in higher brain areas are well suited for the application of machine learning classifiers. And perhaps it is true that neural networks wire themselves up in just the right way to permit computation with state space trajectories governed by recurrent dynamics. There are reasons to believe the answers will often be more straightforward than that (e.g., constraints of anatomy and biological noise [68], and elements of Barlow’s ‘neuron doctrine’ [12]), if only we ask the right questions. Either way, we cannot hope to find clarity unless neural data collection is coupled with well-motivated and controlled behavioral measurements.

Concluding remarks

The preceding commentary is offered in a spirit of goodwill, and with several caveats, first and foremost that the author is manifestly not an expert in many of the methods and topics discussed. I do not wish to imply that all experimental variables or computations should be observable at the level of single neurons, or that behavioral methods must be refined until they are. I am not saying that machine learning and dynamical systems approaches cannot have explanatory value on their own. And I would not suggest that all cognitive functions can be construed as a psychophysical task, or fully understood within a framework inspired by motor control. What I do hope to convey is that the philosophy and toolkit of traditional systems neuroscience can still contribute much in the way of rigor and parsimony, even (or especially) in the era of large-scale recording and sophisticated data analysis.

The BRAIN initiative and related ventures have focused much-needed attention on the development of tools for measuring and perturbing activity across large neural circuits. This effort will surely revolutionize our understanding of normal and diseased brain function. But if we wish to decipher the neural basis of cognition, we must strive for a better understanding of the cognitive states we wish to explain, states that can only be revealed through the lens of behavior.