A null model for cortical representations with grandmothers galore

Abstract

There has been extensive discussion in the literature about the extent to which cortical representations can be described as localist or distributed. Here we discuss a simple null model that encompasses a family of related architectures describing the transformation of signals throughout the parts of the visual system involved in object recognition. This family of models constitutes a rigorous first approximation to explain the neurophysiological properties of ventral visual cortex. This null model contains both distributed and local representations throughout the entire hierarchy of computations and the responses of individual units are meaningful and interpretable when encoding is adequately defined for each computational stage.

How does the brain represent information? A cartoonish answer was entertainingly and endearingly depicted in a recent Pixar film, Inside Out, where we get to peek inside the mind of a young girl, and find five characters, each representing basic emotions: Joy, Sadness, Fear, Disgust and Anger. The notion of a little character inside the brain to represent each emotion could easily be extended by adding more and more characters to represent other sensations, thoughts, actions, decisions, even grandmothers. There could be one character that gets activated whenever we see a chair, another one for grandma, another one whenever we listen to Beethoven, another one whenever we are hungry and another one when we raise our right arm. Despite the seemingly naïve nature of this scheme, after replacing “characters” by “neurons”, this idea seems to be at the heart of how lay people and many Cognitive Scientists and Neuroscientists think about brain representations.

I argue here that this cartoon version of cortical representations is neither completely wrong nor completely right. We need to dig deeper into what we mean by representations and translate those definitions into the biophysical language of neurons. What emerges is an even simpler, and yet far more wonderful and elegant biologically plausible description of how brains store information, a null model for cortical representations. I will focus the discussion here on visual information, specifically on how we see and recognize shapes, objects and faces. I restrict the discussion to visual objects because we know more about the visual system and the transformation of inputs along the visual hierarchy than about any other modality or any other aspect of cognition. This should not be interpreted to imply that we fully understand visual object processing; there remain fundamental unanswered questions the field. Despite these lacunae, we are beginning to develop biologically inspired computational models that provide a reasonable approximation and first sketch of visual processing.

I will argue that the term “grandmother cell” and several derivatives and synonyms have not been well defined (Gross, 2002). In the Cognitive Science literature, the question about information representation has elicited significant discussion between so called “localist” representations (McClelland & Rumelhart, 1981) and “distributed” representations (McClelland, Rumelhart, & Group, 1986); see (Bowers, 2009) for a clear and detailed review and discussion of the literature. It seems clear to most investigators that there is no one-to-one map between a single neuron and perception (i.e. it cannot be the case that there is only one neuron in the entire brain that responds to a given photograph of grandma and it cannot be the case that a given neuron responds only to one particular photograph of grandma and to no other possible photograph of grandma or other stimuli). Once we abandon the idea of a one-to-one map, one formulation of the question about how information is represented in the brain asks whether single neurons store “meaningful things” or not. For example, it could be argued that the word grandmother constitutes a meaningful thing and we can ask whether there are neurons that respond selectively to grandmother and not to other stimuli. Toy computational models have been built where there are units that represent letters or words. Yet, when discussing real brains, this apparently simple question is not sufficiently well defined. Do we mean to imply that there is a single neuron in the entire brain that responds to grandmother? What do we mean by “neuronal response”? How selective does the neuron need to be in this definition? Do we mean to imply that the neuron(s) would respond to grandmother above baseline and to no other possible stimulus in the world? Would the neuron(s) respond to the word “grandma” and the misspelled word “grandmotter”? Would the neuron respond to a picture of grandma, to any picture of grandma, to any grandma, to grandma's clothes, to her gait, her voice, her accent, etc.? The universe of related questions is infinite and so is the list of discussions that can be generated around such poor definitions. Instead, I will provide a brief, and admittedly incomplete, description of the cascade of processes that lead to representing visual information, arguing that this involves both implicit and explicit representations and that ultimately defining these operations is the crux of the problem to understand cortical representations.

Basic tenets of representation

The basic requirements for representing information in the brain have been lucidly articulated by several authors before (e.g. (Barlow, 1972; Crick & Koch, 2003; Parker & Newsome, 1998) among others). To constrain the discussion here, we consider an experiment where a visual stimulus is presented (e.g. a picture of grandma), the subject reports his/her perception (e.g. indicating yes/no whether he/she recognized grandma), and we scrutinize the responses of multiple neurons. I will re-state the main postulates by copying, expanding, rephrasing and adding examples to the ideas in Parker and Newsome (Parker & Newsome, 1998), to define the basic tenets of representation of visual information in cortex:

1. Reflection of perception. The neuronal responses elicited by presentation of the stimulus should be directly comparable to perception and behavior as reported by the subject in terms of magnitude, timing, duration and specificity. Even though we obviously cannot see grandma without a retina, this tenet implies that the retinal ganglion cells are not part of the explicit visual representation of grandma.

2. Timing. The neuronal responses should occur before or at the moment of perception (and not afterwards). This is a basic and fundamental definition of causality and has implications for how neuronal responses are defined, analyzed and interpreted. Visual recognition of an object flashed on the screen takes place within approximately 150 ms of stimulus onset (Kirchner & Thorpe, 2006; Potter & Levy, 1969; Thorpe, Fize, & Marlot, 1996) and therefore the neuronal responses should occur within this time frame.

3. Selectivity. The neurons in question should signal relevant and selective information when and only when the organism perceives the stimulus (in the scenario described here, a two-alternative forced choice discrimination of whether grandma is present). There are abundant studies documenting selective responses to stimuli throughout visual cortex, e.g. in primary visual cortex (Hubel & Wiesel, 1962), area MT (Baker, Petersen, Newsome, & Allman, 1981) and inferior temporal cortex (Desimone, Albright, Gross, & Bruce, 1984). Throughout the brain, neurons fire spontaneously even in the absence of their preferred stimuli. Here we focus on elevated responses beyond this baseline firing.

4. Stability. The neuronal representation should be stable over prolonged periods of time. The representation of grandma should be the same today and a month from now. This is not to say that the brain cannot acquire or forget information. Plasticity is a fundamental property of cortical circuits but there should exist a stable representation of information in cortex. Recent studies have shown that the representation of visually selective information can be very stable, e.g. (Bansal et al., 2012; Bondar, Leopold, Richmond, Victor, & Logothesis, 2009; McMahon, Jones, Bondar, & Leopold, 2014; Tolias et al., 2007).

5. Single trial interpretability. Differences in the firing patterns of the candidate neurons (or a subset thereof) to different external stimuli should be sufficiently reliable in a statistical sense to account for, and be reconciled with, the precision of the organism's responses in single trials. In other words, we should be able to reliably read out information about the stimulus from the neuronal responses, in single trials, and with a linear read-out mechanism. The imposition of linearity here is important; the retina contains all the information about the picture of grandma but it cannot be linearly read out. Machine learning techniques have been extensively used to extract selective visual information in single trials, e.g. (Hung, Kreiman, Poggio, & DiCarlo, 2005). The imposition of single trials interpretability is critical: the brain cannot average across trials to make a decision (the question of noise in neuronal responses is further elaborated upon in the Discussion).

6. Predictive of perceptual fluctuations. Fluctuations in the firing of some set of the candidate neurons to the repeated presentation of identical external stimuli should be predictive of the observer's judgment on individual stimulus presentations including errors and other behavioral changes, e.g. (Newsome, Britten, & Movshon, 1989). For example, in the context of bistable perception, an observer's subjective report for the presence or absence of the stimulus should correlate with the neuronal responses, as observed in inferior temporal cortex (Sheinberg & Logothetis, 1997) but not in primary visual cortex (Leopold & Logothetis, 1996).

7. Invariance. The representation should be robust to certain transformations of the input. For example, in the context of recognizing an object, recognition is essentially unaffected by a wide range of changes in the object's size, illumination, rotation and other transformations. Robustness is not complete: it is possible to disrupt recognition if the image transformation is sufficiently large. Yet, recognition is remarkably robust to image changes and the neuronal representation should reflect this degree of invariance (e.g. (Deco & Rolls, 2004; DiCarlo, Zoccolan, & Rust, 2012; Ito, Tamura, Fujita, & Tanaka, 1995; Riesenhuber & Poggio, 1999; Sary, Vogels, & Orban, 1993)).

8. Task independence. The firing patterns of the neurons in question should not be affected by how behavior is reported. Grandma is grandma, regardless of whether we ask subjects to indicate yes/no by pressing buttons, to verbally report her name or to passively view the picture. There are of course plenty of neurons in cortex that are strongly modulated by the task demands, but there should exist a stable representation that is task-independent. For example, to a first approximation, neurons in inferior temporal cortex carry information about object shapes that is conveyed to pre-frontal cortex where investigators have described strong modulation depending on the task demands (Freedman, Riesenhuber, Poggio, & Miller, 2001; Meyers, Freedman, Kreiman, Miller, & Poggio, 2008). Task demands can modulate the representation (e.g. visual attention can significantly enhance neuronal activity throughout visual cortex (Reynolds & Chelazzi, 2004)) but it should still be possible to decode the information about grandma in a task-invariant manner.

9. Susceptibility to stimulation. Direct interference with the firing patterns of some set of the candidate neurons (e.g. by electrical, chemical or optogenetic stimulation) should lead to measurable changes at the perceptual level. These perceptual changes should be contingent on the spatial and temporal specificity of the external manipulation. Current injection has been shown to bias behavioral responses in monkeys in a content-specific fashion in different parts of visual cortex, e.g. (Afraz, Kiani, & Esteky, 2006; Salzman, Britten, & Newsome, 1990).

10. Susceptibility to lesions. Susceptible to Temporary or permanent removal of all or part of the candidate set of neurons should lead to a measurable perceptual deficit, e.g. (Afraz, Boyden, & DiCarlo, 2015; Dean, 1976; Holmes, 1918).

Starting from the very beginning

When photons impinge on the eyes, retinal ganglion cells (RGC) respond vigorously when there is a change in luminosity within their receptive fields. To get right to the point: is this a grandmother-like representation or a distributed one? The question is never even discussed among retinal physiologists and hardly makes sense. We have a rather elaborate understanding of how photons are transformed into electrical signals and we are beginning to elucidate the circuitry that leads to RGC firing at a very fine level of detail (e.g. (Field et al., 2010)). The picture of grandma is implicitly represented by the activity of a large number of RGCs; one could rephrase this statement to argue that the information about a complex visual stimulus such as grandma is distributed over multiple RGCs. At the same time, the representation of light changes in a specific location is explicitly represented by RGCs. The RGCs therefore behave like grandmother cells in terms of explicitly representing changes in luminosity within their receptive field. Coarsely speaking, we can represent the firing rate r of a given RGC as $r=g\left(\sum _{i=1}^N{w}_i{x}_i\right)$, where g is a non-linearity (e.g. a threshold), w_i denote the synaptic weights of the N units that project onto the RGC, and x_i denote the activities of the N inputs. Although admittedly oversimplified, this formulation forms the foundation of the basic intuition of how neurons integrate information and remains at the heart of models of network activity (Dayan & Abbott, 2001).

RGCs project to neurons in the lateral geniculate nucleus (LGN), and these cells in turn project to primary visual cortex (V1). Neurons in primary visual cortex respond vigorously and selectively to oriented bars presented within their receptive field. Neurons in primary visual cortex do not respond uniquely to a single orientation. A neuron that shows maximal response to a bar of, say, 45-degree orientation will also respond to bars of 44-degree and 46-degree orientation. There is a tuning curve with a certain width that describes how selective the neuron is for different orientations. Investigators have described simple and complex cells in V1, the latter pooling information from multiple simple cells to achieve robustness: whereas the responses of a simple cell are sensitive to the exact location of the oriented bar within its receptive field, a complex cell will yield approximately the same response when the bar is translated to different positions within the receptive field. Again, in the study of primary visual cortex the discussion about grandmother cell-like representations is notably absent. There are multiple computational models that provide a reasonable account of how orientation tuning emerges in V1, e.g. (Carandini et al., 2005; Hubel & Wiesel, 1962; Olshausen & Field, 1996; Priebe & Ferster, 2012). The fundamental questions to completely understand the responses of neurons in layer 4 of primary visual cortex involve fully elucidating the activity of the relevant inputs (x_i in the formulation above), the strength of the relevant connections (w_i in the formulation above) and the specific nature of the computations including thresholds, time constants, synaptic adaptation, synaptic depression and dendritic non-linearities. Information about grandma remains distributed over a large number of V1 cells. We cannot linearly decode the presence of grandma from an ensemble of V1 neurons. However, the presence of an oriented bar at a specific location can be linearly and robustly decoded in single trials from the activity of a small number of V1 neurons.

In sum, in the pathway from the retina to V1, information is represented both in a distributed way and in a grandmother-like way. If we define neuronal response as an enhanced firing rate beyond baseline firing and we define a “meaningful thing” as a stimulus with the right contrast, color, orientation and spatial frequency localized within the receptive field of a given V1 neuron, then this neuron behaves like a grandmother cell. The representation is redundant because there are multiple similar neurons and is achieved by virtue of integrating the activity over a distributed representation of the same stimulus at the level of the LGN.

The ventral visual cortex

Ascending through the visual hierarchy, we have a decent idea of the structures and anatomical pathways that transform visual information along the so-called what pathway involved in object recognition in macaque monkeys (Connor, Brincat, & Pasupathy, 2007; Felleman & Van Essen, 1991; Markov et al., 2012). Yet, we understand much less about the details of the computations that transform information from one visual area to the next.

The ventral visual stream can be described by an approximate hierarchy: V1 neurons project to area V2, V2 neurons in turn project to area V4, V4 neurons project to inferior temporal cortex (ITC). In addition to these projections there are abundant horizontal connections within each area and top-down connections (e.g. V2 projects back to V1). Moving from V1 to ITC, there is a progressive increase in the size of the receptive fields, in the degree of complexity of the type of features represented at each stage and also in the degree of tolerance to image transformations. For example, some neurons in V4 are particularly sensitive to the extent and type of curvature of the visual stimulus (Pasupathy & Connor, 2002). Although the specific circuits that give rise to selectivity along the ventral stream are not clearly understood, it is possible to approximate the type of selective responses to different shapes using the same type of ideas articulated above by non-linearly pooling inputs from simpler responses (Cadieu et al., 2007).

Inferior temporal cortex constitutes a vast expanse of the ventral stream. Lesions in inferior temporal cortex typically lead to specific deficits in visual object recognition (e.g. (Afraz, Boyden, & DiCarlo, 2015; Benton & Wav Allen, 1972; Dean, 1976)). Neurons in inferior temporal cortex respond to complex shapes including abstract ones like folded paperclips or fractal patterns and real world objects including faces (Desimone, Albright, Gross, & Bruce, 1984; Logothetis, Pauls, & Poggio, 1995; Logothetis & Sheinberg, 1996; Naya, Sakai, & Miyashita, 1996; Tanaka, 1996; Tsao, Freiwald, Tootell, & Livingstone, 2006). Their responses also show a significant degree of robustness to a large number of stimulus transformations including changes in preferred stimulus position within the receptive field, scale, illumination, rotation, and many others (Ito, Tamura, Fujita, & Tanaka, 1995; Logothetis, Pauls, & Poggio, 1995; Sary, Vogels, & Orban, 1993; Tovee, Rolls, & Azzopardi, 1994). It is possible to linearly decode in single trials the identity of objects from the activity of small ensembles of ITC neurons (Hung, Kreiman, Poggio, & DiCarlo, 2005).

Consider an ITC neuron that responds selectively to a complex shape, say a face. To be clear, what we mean in practice is that the investigator records the activity of this neuron and counts the number of spikes in a given time window in response to presentation of a battery of multiple stimuli, say on the order of several hundred different shapes. The stimuli that contain faces elicit not only a response above baseline but also a response that is statistically stronger than the responses to the other stimuli in the set. If we were able to trace the input connections of this ITC neuron, and one day in the not too distant future we may be able to do this, we would likely observe that the inputs are distributed over multiple V4 cells. As a null hypothesis, we imagine the transformation of V4-like responses into ITC-like responses to obey the same principles that govern the transformation of multiple LGN inputs converging to give rise to orientation selectivity in V1. In other words, at each stage of processing, there is a transformation from a distributed representation to a grandmother-like representation. Each stage of processing thus connotes a distributed representation and also a localist one, depending on what particular aspects of the stimulus we are considering. Many of the discussions about localist versus distributed representations in the brain have centered on high-level “meaningful things” such as grandmothers, words, or the concept of love or fear. Unfortunately, for most of these high-level things, we still do not really understand the degree of selectivity and invariance of the relevant neurons, let alone the inputs that can give rise to such representations. In the context of visual information, the simplest null model is that we can build a representation of progressively more complex shapes by adequate pooling and filtering signals from the previous processing stage. This leads to an elegant null model for cortical representations, with grandmothers galore as long as we define what is represented at each stage and also with clearly distributed representations.

Computational models of visual recognition

The brief qualitative description of the transformation of visual inputs in the previous section has been formalized in the form of a theoretical framework and quantitatively instantiated into algorithms for visual recognition (DiCarlo, Zoccolan, & Rust, 2012; Fukushima, 1980; Mel, 1997; Olshausen, Anderson, & Van Essen, 1993; Perret, Oram, Hietanen, & Benson, 1994; Riesenhuber & Poggio, 1999; Rolls, 1991; Serre et al., 2007). These biologically inspired algorithms are characterized by a bottom-up hierarchy of linear and non-linear computations that start at the pixel level and progressively build a complex representation of the visual inputs. Ascending through the visual hierarchy, units display larger receptive fields, selectivity to more complex visual features and increased tolerance to changes in those features. The specific implementation details vary across different models but the general principles are the ones articulated in the previous section: units pooling information from previous layers, weighted linear sums followed by thresholding, non-linearities that give rise to tolerance. As a reasonable first approximation, these models assume that “cortex is cortex” and hence that the same mathematical operations are performed at each level (except that each level operates on different inputs and therefore conveys different outputs).

There have been multiple implementations of this type of network architecture (e.g. (Miconi, Groomes, & Kreiman, 2015; Riesenhuber & Poggio, 1999; Serre et al., 2007; Wyatte, Curran, & O'Reilly, 2012; Yamins et al., 2014) among many others). These architectures also constitute the basic foundation of deep convolutional networks, which have been successful in many engineering applications, most notably in computer vision (e.g. (Krizhevsky, Sutskever, & Hinton, 2012)).

This family of models constitutes only a first order approximation to the complexities of the cortical circuitry; for example, most of these models involve exclusively feed-forward projections and it is well known that there are abundant horizontal and feedback connections throughout neocortex (Douglas & Martin, 2004). Despite this and many other simplifications, these models have been shown to capture basic properties of neuronal responses throughout the ventral visual stream (Riesenhuber & Poggio, 1999; Serre et al., 2007).

An attractive feature of computational models is that we can inspect the activity of every unit and we can explain with perfect accuracy all the inputs and outputs to any possible stimulus. The question of localist versus distributed representations has been discussed before in the context of certain computational models (e.g. (Bowers, 2009; McClelland & Rumelhart, 1981; McClelland, Rumelhart, & Group, 1986)) but those computational models have not been directly linked to neurophysiological measurements. Is information in deep hierarchical architectures for visual recognition represented in a localist manner or a distributed one? Each unit in these models is selective and represents a meaningful thing. In the typical computational implementation of these models, every unit shows selectivity. The biological implementation could differ substantially: there is a large number of different types of neurons in every patch of cortex and we still lack a fundamental understanding of the functional properties of these neuronal types and whether all of them show the same type and degree of selectivity. Each unit is part of a small group of units (compared to the total number of units in each area) that codes for that meaningful thing, we can precisely pinpoint what that meaningful thing is for each unit, injecting current into those units leads to an enhanced representation of the features they encode and removing those units from the model leads to a visual scotoma for those features. In other words, every unit in the model is a grandmother unit, as long as we define grandmother in terms of the unit's specific preferences. At the same time, each unit is part of a large ensemble that projects onto the next stage to build a more complex feature. There is no real dichotomy between localist and distributed representations in these models. All the computations are clearly defined in terms of a few well-defined rules, a simple, yet beautiful null model for cortical representations.

Discussion

It does not make sense to discuss distributed versus localist representations in the brain without defining clearly the identity of the neurons in question and the specific type of information that is represented. Accepting the family of models succinctly described above as a plausible first-order description for the functions of ventral visual cortex, one could argue that we have only described the representation of visual features that range from luminance changes to oriented bars to more complex shapes but we have not specifically described how to represent chairs, dogs, grandmothers and other objects. However, these models provide a very explicit description of how to represent those shapes. Indeed, those models are routinely tested in object recognition tasks that involve dozens to millions such objects (e.g. (Krizhevsky, Sutskever, & Hinton, 2012; Serre et al., 2007)). In a typical scenario, a set of training images is presented to the models to simulate the activity of every unit in response to every image. Next, a machine learning approach is used to train the weights of a classifier to map the activity of units in one or more layers of the model to the image labels (e.g. label “grandmother” for the corresponding set of pictures that include grandma). Finally, the model is tested with a different set of images to evaluate how well it can label those new images. In many cases, a linear kernel is used for the classifier. In this case, a given linear classifier unit takes the distributed input from several units in an earlier layer, multiplies the activity of those units by suitable weights and thresholds the results, finally deciding in a binary way whether the image should be labeled “grandmother” or not. These classifier units are localist par excellence; the inputs to those units contain a clearly distributed representation of visual information. This process is repeated once and again throughout the hierarchical model.

While in some instantiations of these models there may be a single classifier unit for each label, in the brain there is a significant degree of redundancy (Barlow, 1972). The notion of a localist representation does not depend on having a one-to-one map between units and meaningful things. This is a typical misconception of the definition of grandmother cells that has been cleared in multiple earlier discussions. In the original parable by Lettvin, there were 18,000 such cells that responded uniquely to “mother” (Gross, 2002). At no stage of processing do we have only one neuron representing information. There are multiple RGCs that represent luminance changes in a given point in space, multiple V1 neurons that respond to a certain orientation in a given location, and multiple ITC neurons that represent complex visual features at a given location.

There are many important nuances to the representation of information in the brain that deserve further discussion. Once we allow for the notion established in the previous paragraph that the map between neurons and meaningful things is not one-to-one, we may ask how many neurons are involved in representing each feature. Neurophysiological recordings typically show that representations tend to be sparse. Sparseness can be loosely defined as using only a small number of neurons out of the entire set of possible neurons to represent a particular feature (for a more formal definition, see equation 4 in (Olshausen & Field, 1996)). Sparseness can also be loosely defined by the related but distinct notion that a neuron responds to only a small fraction of all possible stimuli. In computational models, introducing sparseness as a constraint can alter the resulting representation and lead to the development of biologically plausible feature tuning properties. For example, there has been considerable discussion about the idea that information is sparsely represented in primary visual cortex (e.g. (Olshausen & Field, 1996)). Sparseness is an attractive property because it leads to a more efficient representation that is easier to decode in later stages by virtue of the independence of the inputs, and it may also have energetic efficiency implications (Laughlin, van Steveninck, & Anderson, 1998). Some investigators draw a sharp distinction between localist representations, sparse distributed representations and dense distributed representations (e.g. (Bowers, 2009)). In the null model outlined here, the distinction is merely a quantitative one. Each visual feature at each location is represented by a relatively small number of units (but more than one unit).

Two essential causal manipulations provide evidence that we are in the right track towards understanding the representation of visual information: disturbing the representation through small lesions or by injecting currents into local circuits. Lesions in V1 lead to localized scotomas in the visual field and lesions in higher visual areas lead to specific visual deficits such as the inability to detect color, motion or complex shapes (e.g. (Dean, 1976; Zeki, 1990, 1991)). Eliminating a single neuron is unlikely to lead to clear behavioral manifestations, presumably due to the robustness and redundancy in the representation. We currently do not have the technical ability to selectively lesion or inactivate all the neurons that respond to a specific feature. Yet, a recent study demonstrated that optogenetic silencing of a relatively small group of neurons in a patch of cortex containing neurons selective for face gender can lead to a small but significant impairment in gender discrimination (Afraz, Boyden, & DiCarlo, 2015). Advocates of distributed representations would be right to point out that a large number of neurons is removed in these lesion studies; future technological developments will allow a tighter titration of the behavioral consequences of eliminating specific types and different numbers of neurons from cortical circuits. The specificity of the ensuing effects in lesion studies supports the notion that there is a localized, interpretable representation of visual information in cortical circuits. This specificity in the consequences of local lesions is also consistent with the type of effects one would expect from silencing local groups of units in the computational models outlined in the previous section.

Another important causal manipulation that can shed light on the nature of the neural representations involves examining the effects of electrical stimulation. Injecting current through high-impedance microelectrodes placed extracellularly probably alters the activity of hundreds to several thousand neurons in the vicinity of the electrode, depending on the type of electrode, the current intensity, current injection pattern, brain area and tissue excitability (Logothetis et al., 2010; Tehovnik, 1996), in addition to indirect effects on other brain areas. Several studies have demonstrated that perceptual and behavioral effects can be elicited via current injection in this fashion. Stimulating V1 leads to perception of localized phosphenes (e.g. (Brindley & Donaldson, 1972; Dobelle & Mladejovsky, 1974; Tehovnik, Slocum, Smirnakis, & Tolias, 2009)). Similarly, stimulating other parts of visual cortex can lead to specific visually-triggered behaviors including motion detection enhancement upon stimulating area MT (Salzman, Britten, & Newsome, 1990) or face detection enhancement upon stimulating inferior temporal cortex (Afraz, Kiani, & Esteky, 2006). Recently, a series of elegant studies have advanced the possibility of stimulating individual neurons, albeit not in the visual system (e.g. (Brecht, Schneider, Sakmann, & Margrie, 2004), reviewed in (Doron & Brecht, 2015)). Rather surprisingly, these studies have demonstrated that stimulating single neurons can elicit both sensory percepts as well as motor outputs. For example, adding approximately 15 action potentials to baseline activity in the rat barrel cortex, which is involved in the representation of whisker tactile stimulation, led to significant and specific behavioral reports. The results of this type of experiment are typically weak and show a significant degree of variability. Furthermore, the mechanisms of cortical propagation in these so-called nanostimulation experiments are poorly understood; it has been suggested that the behavioral manifestations arise as a consequence of amplification via recurrent connections. Despite these considerations, it is difficult to argue that the activity of individual neurons is uninterpretable given the results of these micro and nanostimulation studies. Rather, the stimulation studies lend further support to the notion of local circuits with interpretable responses that can be activated via stimulation to elicit specific perceptual sensations or motor outputs. The results of these stimulation experiments are also consistent with the consequences of activating local groups of units in the type of computational models outlined in the previous section.

Information from the top layers of the ventral visual stream is conveyed to multiple brain areas, most notably the medial temporal lobe and frontal cortex. Neurophysiological recordings in the human medial temporal lobe have described a plethora of interesting selective responses to places (Ekstrom et al., 2003), object categories (Kreiman, Koch, & Fried, 2000b; Mormann et al., 2011) and specific individuals or landmarks (Quian Quiroga, Reddy, Kreiman, Koch, & Fried, 2005). These responses could be interpreted as additional examples of a localist representation for high-level information. However, as fascinating as those signals are, the neuronal responses in the medial temporal lobe structures do not fulfill all the requirements outlined above required for a representation of visual information, most notably the fact that removing those neurons does not lead to visual impairments (Squire, Stark, & Clark, 2004). Rather, the medial temporal lobe structures play a fundamental role in interpreting the degree of familiarity and novelty of current stimuli, associating multi-modal information in the context of emotional valence and prior knowledge, to form and retrieve memories (Brown & Aggleton, 2001; Cameron, Yashar, Wilson, & Fried, 2001; Eichenbaum, 2004; Gelbard-Sagiv, Mukamel, Harel, Malach, & Fried, 2008; Kreiman, 2007; Kreiman, Koch, & Fried, 2000a; Mormann et al., 2014; Rutishauser, Mamelak, & Schuman, 2006; Rutishauser, Schuman, & Mamelak, 2008; Squire, Stark, & Clark, 2004).

Information from the ventral visual cortex is not only conveyed to the medial temporal lobe but also to frontal cortex structures that play a critical role in interpreting the visual inputs in a task-dependent manner to make decisions leading to behavioral outputs (Miller & Cohen, 2001). Neurons in pre-frontal cortex show dynamic tuning to task-dependent variables. Recently, it has been argued that each neuron in pre-frontal cortex is tuned to multiple different aspects of an object sequence memory task rather than a single one (Rigotti et al., 2013). This so-called mixed-selectivity does not imply that the responses of individual units cannot be interpreted. This question goes back to the definition of “meaningful things”. If a neuron shows an increased firing rate during the presentation of a specific stimulus and also before a given motor response, it may be challenging to label the neuron as “purely visual” or “purely motor”. However, the inadequacy of these labels pre-defined by the investigators does not preclude from decoding the properties of neuronal responses. The neuron in question still shows a reproducible response that can be correlated to the sensory events, decisions and motor outputs during the task. The interpretational challenges arise only because of our attempt to force specific anthropomorphic descriptions based on language to the changes in firing rate. These language-based descriptions of neuronal preferences have been relatively successful in early sensory areas but it seems likely, and far more powerful, that we will require mathematically defined operations to explain the responses in higher parts of cognition. Even within the complex and fascinating realm of high-level cognitive information represented in frontal cortex, there is still hope for interpreting the activity of individual neurons.

The apparent “mixed” selectivity alluded to in the previous paragraph is not the only conceptual barrier to interpreting the activity of cortical neurons. Another important problem arises when we consider the response dynamics and the dependence on behavioral tasks. Neuronal responses throughout the visual system show complex temporal dynamics that evolve on scales of tens to hundreds of milliseconds (e.g. (Hung, Kreiman, Poggio, & DiCarlo, 2005; Meyers, Freedman, Kreiman, Miller, & Poggio, 2008; Richmond, Optican, & Spitzer, 1990; Ringach, Hawken, & Shapley, 1997; Smith, Majaj, & Movshon, 2005; Woloszyn & Sheinberg, 2009)). Additionally, the responses throughout visual cortex can be significantly modulated by task demands (e.g. (Bansal et al., 2014; Baylis & Rolls, 1987; Gilbert, Li, & Piech, 2009; Ramalingam, McManus, Li, & Gilbert, 2013; Rigotti et al., 2013; Vogels, Sary, & Orban, 1995; Woloszyn & Sheinberg, 2009)). The family of models described in the previous section does not really account for neural dynamics (other than the serial passing of signals from one layer to the next) nor does it clearly describe how the same neural representation can be used for multiple different tasks. Furthermore, if the interpretation of spike trains from a given neuron evolves over time due to internal dynamics and/or due to task demands, it is unclear how post-synaptic neurons can decode such time-varying inputs. Changes over long temporal scales (e.g. due to learning) may be easier to decipher since they may be accompanied by concomitant modifications in the synaptic weights, yet we need to be able to also understand changes in the meaning of spike trains that occur within tens to hundreds of milliseconds. In the extreme version, the same firing rate of a given neuron could convey distinct types of information at two different time points. Understanding these dynamics and task-dependent effects opens the doors to interesting questions both for localist and distributed representations and will require significant extensions to the simplified family of models described here.

A prominent example of dynamics in neural network models is illustrated in attractor networks. Attractor networks constitute particularly interesting computational models that are rather distinct form the family of bottom-up architectures described in the previous section (e.g. (Hopfield, 1982)). Briefly, units are interconnected in an all-to-all fashion and the state of the network is defined by the activity of all units. Each unit participates in representing every possible input and every input is represented by all units. In some sense, each unit in this type of attractor network is not representing any meaningful thing in and of itself. This seems to be a major departure from the hierarchical feed-forward models described here as a null model. Even though there has not been clear compelling biological evidence for or against the possibility of an attractor network type of neocortical circuits, given the dense interconnectivity of cortical circuits, it is not inconceivable that these ideas may help us better understand the dynamics of computations in cortex. Still, these networks typically do not explicitly specify how and when information is read out from these models. Brains need a way of converting the representation of visual information into decisions and actions (e.g. to indicate that a particular image does or does not contain a picture of grandma). We need an additional step to peek inside the attractor network, examine its state and determine whether that state corresponds to one of the stored patterns. The simplest way to think about how this read-out mechanism could be implemented with biological circuits is to wire all of the units in the network to another set of units that are capable of discriminating different states. These read-out units would behave as localist units deciphering the distributed the representation in the attractor network.

Another important issue that should be discussed is the apparent unreliability of neuronal responses. Upon repeated presentation of an identical stimulus, neurons seem to respond capriciously, and the variance in the spike counts can be as high as the mean spike count. The high degree of trial-to-trial variability in the spike trains elicited in response to a given stimulus has puzzled neuroscientists trying to understand how neurons encode information for decades (Perkel, Gerstein, & Moore, 1967). This degree of variability seems to pose a challenge to a localist representation where the activity of each neuron in each trial matters and should be interpretable. One possibility is that the activity of each neuron can be described as a localist representation only on average, but this seems like an artificial and convoluted argument; the brain must function in individual trials. A more attractive possibility is that the apparently high noise levels are a consequence of heterogeneity in the experimental conditions. Even when investigators think that the conditions are identical from one trial to the next, there exist multiple internal and small external variables that can be different. In support of this argument, cortical responses can show smaller degrees of trial-to-trial variability when there is better control of experimental conditions (e.g. (Bair & Koch, 1996; Churchland et al., 2010)). Furthermore, several studies examining responses in primary or early sensory neurons where there may be less fluctuation in internal conditions have demonstrated that variability can be very low, in some cases, as low as the theoretical limits imposed by the discrete nature of spikes (e.g. (Berry & Meister, 1998; Kreiman, Krahe, Metzner, Koch, & Gabbiani, 2000; van Steveninck, Lewen, Strong, Koberle, & Bialek, 1997)). Additionally, variability can also be significantly reduced in slice recordings (Holt, Softky, Koch, & Douglas, 1996; Mainen & Sejnowski, 1995; Stevens & Zador, 1998). In sum, under conditions where it is possible to exert better control over the experimental conditions, trial-to-trial variability is significantly reduced and the responses of individual neurons become more interpretable in single trials.

Even if we accept the rudimentary sketch proposed as a null model for the cortical representation of visual information, we still have very little to say about a myriad of high-level and abstract ideas such as how the brain encodes “love”, or “I would like to talk to grandma today”, or “I miss grandma's delicious food”. In a completely speculative and admittedly naïve tone, we can imagine that evolution works by duplicating, copying, improving and refining existing circuits. Hence, the basic computational principles uncovered when scrutinizing the representation of visual information could well extrapolate to other aspects of cognition. Needless to say, there is a vast and fascinating uncharted territory awaiting for neuroscientists to probe the representation of high-level cognitive information.