Probing Computation in the Primate Visual System at Single-Cone Resolution

Abstract

Daylight vision begins when light activates cone photoreceptors in the retina, creating spatial patterns of neural activity. These cone signals are then combined and processed in downstream neural circuits, ultimately producing visual perception. Recent technical advances have made it possible to deliver visual stimuli to the retina that probe this processing by the visual system at its elementary resolution of individual cones. Physiological recordings from nonhuman primate retinas reveal the spatial organization of cone signals in retinal ganglion cells, including how signals from cones of different types are combined to support both spatial and color vision. Psychophysical experiments with human subjects characterize the visual sensations evoked by stimulating a single cone, including the perception of color. Future combined physiological and psychophysical experiments focusing on probing the elementary visual inputs are likely to clarify how neural processing generates our perception of the visual world.

INTRODUCTION

Transduction and processing of physical signals encoded by populations of sensory receptors connect us to the outside world. Sensory transduction sets fundamental limits on the information available about the physical world, and the processing of this information by downstream neural circuits controls how we perceive and interact with the environment. Therefore, studying how sensory systems work requires manipulating signals in the receptor population as precisely as possible, while measuring downstream neural signals and behavior.

The most studied sensory modality—vision—begins with the encoding of visual scenes by the activation of opsin in the mosaic of photoreceptors. Individual photoreceptor signals are, in a sense, the elementary units of vision. Therefore, understanding the neural computations that mediate visual perception and behavior amounts to understanding how the patterns of signals in the photoreceptor mosaic are processed by the retina and the brain.

The utility of understanding the neural processing of signals from the array of sensory receptors is illustrated by the history of studying rod-mediated (low light) vision. Psychophysical experiments in the early twentieth century illustrated that individual rod photoreceptors could be activated by single photons, and that human (and animal) observers are sensitive to the activation of one or a few rods (Hecht et al. 1942, Barlow 1956, Sakitt 1972, Aho et al. 1988, Tinsley et al. 2016; reviewed in Field & Sampath 2017). Physiologists, exploiting these insights and utilizing experiments that activated individual rods, used these measurements to decipher the mechanisms of phototransduction, synaptic transmission, and circuit architecture that support vision when photons are scarce (Baylor et al. 1984, van Rossum & Smith 1998, Schneeweis & Schnapf 1999, Field & Rieke 2002, Berntson et al. 2004). These advances led to important insights about the biological sources of noise that limit vision on a starlit night and the downstream computations that extract the relevant signal (Field et al. 2005, Dunn & Rieke 2006, Ala-Laurila & Rieke 2014).

Daylight vision, in which photons are abundant, is mediated by the cone photoreceptors. In the last decade, technical advances have made it possible to characterize the cone mosaic of the living human and nonhuman primate retina (Figures 1 and 2) and to control the pattern of light absorption at the resolution of the cone mosaic. These advances have enabled an emerging understanding of the effects of single-cone stimulation, both behaviorally and physiologically. This unprecedented ability allows for an appraisal of fundamental assumptions made by previous studies, the establishment of direct links between cone activations and visual performance, and an exploration of new aspects of visual system function inaccessible by traditional methods. This review briefly summarizes the technical advances that have made it possible to study vision at the upper limit of the spatial and spectral resolution set by the cone mosaic, both in vivo and ex vivo, and outlines the recent progress made using these methods.

Figure 1.

(a) Image of the cone mosaic in an individual human subject acquired using an adaptive optics-scanning laser ophthalmoscope. Each spot in the image is one cone. Image is from a patch of retina at approximately 1.5° eccentricity. Image provided by Dr. R. Sabesan. (b) Arrangement of L (red), M (green), and S (blue) cones in the mosaic shown in panel a, obtained using adaptive optics microdensitometry. The L:M ratio of ~2.5:1 and sparse S cone submosaic are typical for humans, as is the near-random packing arrangement of the L and M cones. Abbreviations: L, long wavelength–sensitive; M, middle wavelength–sensitive; S, short wavelength–sensitive. Panel b reproduced from figure 2 in Sabesan et al. (2015), published under the Creative Commons Attribution 4.0 International Public License.

Figure 2.

RF maps at single-cone resolution. (a) RFs of four major cell types obtained with high-resolution white noise in a single recording. Reverse correlation analysis reveals separate puncta of sensitivity in each RF, corresponding to individual cone inputs. L and M cone types have different spectral properties revealed by the color of the puncta. Outlines of RF centers are shown (black lines). (b) Cone mosaic reconstructed from multiple RGCs recorded simultaneously (same experiment as in panel a). Circles represent locations of L (red), M (green), and S (blue) cones. Cones obtained from RFs of the four cells from panel a are highlighted (black lines). (c) Cone input strength in an OFF midget cell RF. (Top) Cone locations denoted by filled circles; color scheme corresponds to relative cone input strength: weaker input (blue) and stronger input (yellow). Input strengths deviate from a Gaussian model. (Bottom) Image of the same RF showing cone types. Thickness of each converging white line is proportional to the relative cone input strength to the RGC. (d) Example mosaic of OFF midget cells. Cells of the same type tile the retina, sampling from most available cones. Outlines of individual cells are coordinated with minimal overlap. Note the mixed L and M input in most OFF midgets. Abbreviations: L, long wavelength–sensitive; M, middle wavelength–sensitive; RF, receptive field; RGC, retinal ganglion cell; S, short wavelength–sensitive. Figure adapted from Field et al. (2010).

VISUAL STIMULATION AT SINGLE-CONE RESOLUTION

Ex vivo recordings from isolated primate retina have now made it possible to study how signals from individual cones propagate through neural circuitry to retinal ganglion cells (RGCs), which transmit the signals to the brain. Large-scale, high-density microelectrode arrays enable recording from hundreds of RGCs simultaneously, while focusing a display directly on the retina, with the physiological optics of the eye removed. The main challenge of maintaining sensitive and stable light responses for many hours is achieved by keeping the pigment epithelium attached. High-resolution spatiotemporal noise stimuli coupled with reverse correlation analysis (Chichilnisky 2001) reveal the locations and input strength of all cones in the receptive fields (RFs) of many simultaneously recorded RGCs (Field et al. 2010; for earlier work focused on individual S cone inputs, see Chichilnisky & Baylor 1999) (Figure 2). The targeted stimulation of cones with tiny spots permits exploration of interactions between cone signals (Li et al. 2014, Freeman 2015). This method makes it possible to measure cone locations and spectral types over patches of the cone mosaic that span several degrees of visual angle and to construct RF maps at single-cone resolution across complete populations of several RGC types (Field et al. 2010, Doi et al. 2012). However, these methods are currently applicable only to midperipheral and peripheral retina because, near the fovea, the high density of RGCs makes it difficult to isolate their spikes.

In vivo methods have been mainly used in psychophysical experiments, where an observer reports on the presence or appearance of a stimulus, but the advances have also been adopted for in vivo physiological studies. A key challenge here is that the spread of light caused by blur from the physiological optics generally makes the retinal image of even the smallest external stimulus exceed the size of a foveal cone. Two main methods have been used to overcome optical blur: laser interferometry and adaptive optics.

Interferometry stimulates the photoreceptor mosaic with high spatial-frequency gratings (over 100 cycles/degree). With this approach, cone density and packing geometry were measured psychophysically in the living human eye (Williams 1985, 1988; Coletta & Williams 1987). Interferometry was also used while recording from primate lateral geniculate nucleus (LGN) neurons, permitting inferences about the number of cones providing input to foveal parvocellular neurons (McMahon et al. 2000). Interferometric methods, however, do not allow for the selective activation of individual cones.

In adaptive optics, the eye’s optical aberrations are measured with a wave front sensor (Liang et al. 1994, Liang & Williams 1997, Hofer et al. 2001, Thibos et al. 2002), and a deformable mirror is adjusted under computer control to cancel the aberrations. Imaging systems equipped with adaptive optics now routinely image individual cones at or near the center of the human fovea (Liang et al. 1997, Carroll et al. 2005, Williams 2011, Sincich et al. 2016) (Figure 1). It is also possible to identify the spectral type of individual cones with in vivo microdensitometry, which compares the amount of photopigment bleaching by lights of different wavelengths (Roorda & Williams 1999, Hofer et al. 2005a, Sabesan et al. 2015). When combined with tracking of fixational eye movements in real time (Arathorn et al. 2007) and characterization of and compensation for transverse chromatic aberration (Harmening et al. 2012, Privitera et al. 2016, Winter et al. 2016), adaptive optics systems allow visual stimuli to be targeted to individual cones. This technique has been developed for both psychophysical (Harmening et al. 2014; see also Tuten et al. 2012) and physiological (Sincich et al. 2009) measurements.

FUNDAMENTALS OF RETINAL PROCESSING REVEALED THROUGH SINGLE-CONE STIMULATION

The central unifying concept in visual neuroscience is the RF of a neuron, which summarizes how its activity is driven by the visual image (Kuffler 1953). Decades of work have focused on the structure of RFs at different stages in the visual hierarchy, and the concepts have been used to probe other sensory systems as well. However, most neurons have no direct access to the visual image. Instead, visual neurons are driven by inputs from preceding neurons in the visual hierarchy, forming a chain of computations that leads back to the elementary inputs—photoreceptors. For daylight vision, three types of cone photoreceptors [sensitive to short (S), middle (M), and long (L) wavelengths] synapse with bipolar cells, which in turn synapse with multiple RGC types. In addition, horizontal and amacrine cells mediate lateral interactions (for reviews, see Dacey 2004, Baccus 2007, Field & Chichilnisky 2007, Thoreson & Mangel 2012, Diamond 2017). Recently, the technical advances that permit single-cone stimulation (see above) have made it possible to understand how RFs of RGCs are composed from the inputs of photoreceptors via retinal interneurons and how these RFs are collectively organized over space. These studies have resulted in the first large-scale functional map of connectivity in a neural circuit at cellular resolution.

Univariance

Single-cone stimulation permitted the first test of a fundamental assumption regarding RF structure—that the effective inputs from different cones differ only in magnitude, not in time course or other properties. We refer to this property as univariance, following a concept from color vision (see Brindley 1970). An implication of univariance is that, for a given stimulus, downstream neural circuits cannot identify the cone in which the signal originates based on the RGC response, because it is confounded with stimulus strength. A test of univariance was enabled by stimulating individually targeted cones within the RF of peripheral RGCs (Li et al. 2014). In general, it was possible to adjust the stimulus contrast for different cones to evoke identical responses in a RGC, confirming the key experimental prediction of univariance. Furthermore, the cone strengths obtained by the contrast adjustment approach were consistent with cone strengths derived from high-resolution white noise stimulation and reverse correlation, an efficient tool for measuring RFs at single-cone resolution in a large population of RGCs simultaneously (Chichilnisky 2001, Field et al. 2010).

Spatial Structure of the Retinal Ganglion Cell Receptive Field

The RFs of most cell types in the early visual system consist of a strong, compact center and a weaker, larger antagonistic surround (Kuffler 1953, Dawis et al. 1984, Field & Chichilnisky 2007, Crook et al. 2009, Gauthier et al. 2009b). In the most common model of this RF structure, the univariant strength of photoreceptor inputs creates a Gaussian spatial profile for the RF center and a broader Gaussian profile for the opposing surround (Rodieck & Stone 1965). This model successfully reproduces RGC responses to spots, gratings, and coarse checkerboards (Rodieck & Stone 1965, Enroth-Cugell & Robson 1984, Crook et al. 2008, Dacey et al. 2014, Cooper et al. 2016, Wool et al. 2018). However, many experiments have also indicated substantial variability in RF shape and spatial structure at a coarse scale (Brown et al. 2000; Gauthier et al. 2009a; Lee et al. 2012, 2017; McMahon et al. 2000; Passaglia et al. 2002). At single-cone resolution, local inhomogeneities were observed in several cell types in the tiger salamander retina (Soo et al. 2011). Similar irregularities are visible in published single-cone RF maps of peripheral primate RGCs (Field et al. 2010, Li et al. 2014), although this property was not analyzed in detail. Direct examination of the spatial distribution of cone strengths in midget and parasol cells (Figure 2), the most numerous RGC types in the primate retina, reveals substantial deviations from the Gaussian model. These deviations could strongly influence the responses of RGCs to naturalistic stimuli, limiting the applicability of the Gaussian model. To extend these findings to the central visual field, the most promising method may be a combination of in vivo recordings in the retina or LGN and visual stimulation with adaptive optics (see Sincich et al. 2009, Yin et al. 2014).

Collective Organization of Receptive Fields

The RFs of each RGC type are organized into a semiregular tiling pattern, or mosaic, uniformly covering the surface of the retina and thus visual space (Devries & Baylor 1997). Furthermore, local spatial irregularities in the RFs of neighboring cells of the same type complement one another like puzzle pieces (Gauthier et al. 2009a, Liu et al. 2009), suggesting a finely tuned developmental program governing visual field coverage. With stimulation at single-cone resolution, this coordination between neighboring RGCs of the same type was revealed in complementary patterns of cone input at the boundaries between neighboring RFs (see Field et al. 2010) (Figure 2). Thus, within a mosaic, the sampling of cone inputs by RGCs appears to be precisely orchestrated in development. Indications of cone sampling coordination between different cell types are also emerging, both at a coarse scale in the relative positioning of dendritic fields (Jang & Paik 2017) and at single-cone resolution in the anticorrelation of cone inputs to overlapping RGCs of different types in the salamander (Soo et al. 2011). Functionally, such coordination appears to make information processing more efficient, allowing for more uniform sampling of the visual world and decorrelating responses of RGCs with highly overlapping RFs (Gauthier et al. 2009a, Soo et al. 2011, Doi et al. 2012). Developmentally, this kind of coordinated cone sampling could be produced by competition between bipolar cells for synaptic input from cone pedicles (Johnson & Kerschensteiner 2014; see also Thompson et al. 2017). Similar competition could occur in other parts of the nervous system as well. Thus, tackling this problem in the retina may help reveal general rules of synaptic organization across cell types during development.

Nonlinear Receptive Field Structure Revealed Through Single-Cone Stimulation

Many models of RGC function rely on the assumption that visual inputs are combined linearly over space and time. However, nonlinearities in retinal circuits profoundly shape RGC responses (Gollisch & Meister 2010, Gollisch 2013), enhancing properties such as sensitivity to fine textures that are common in natural scenes (Schwartz et al. 2012, Turner & Rieke 2016), responses to looming (Münch et al. 2009) and objects moving relative to backgrounds (Olveczky et al. 2003), and direction selectivity (Barlow & Levick 1965, Amthor & Grzywacz 1991, Demb 2007, Zhou & Lee 2008, Kuo et al. 2016). These nonlinearities contribute to the failure of RGC models to generalize to natural stimuli. A major nonlinearity in retinal circuits is the rectified synaptic transfer between bipolar cells and RGCs, which transmits bipolar cell depolarization but not hyperpolarization. This nonlinearity produces subunits within the RF, thought to represent the RFs of contributing bipolar cells (Hochstein & Shapley 1976, Demb et al. 2001). However, until recently, these subunits had never been directly visualized (Shah et al. 2016, Liu et al. 2017).

Given that most bipolar cells receive input from a few cones, the spatial properties of the bipolar RGC nonlinearity can be understood more precisely with single-cone stimulation. This approach has been used to estimate the location and cone convergences of nonlinear subunits within the RFs of midget RGCs (Freeman et al. 2015) by independently stimulating cones with fine-grained white noise while measuring the spiking responses of RGCs. A subunit model was then fitted to the data with parameters that maximized the likelihood of the measured RGC responses given the spatial pattern of cone stimulation. This approach revealed local groups of cones that interacted linearly with one another, but nonlinearly with other cones outside the group. The validity of the model was tested by presenting an increment of light to one cone and a decrement to another: Cone signals within a subunit interacted linearly and cancelled, while cone signals in different subunits failed to cancel, producing a nonzero response in the RGC (Figure 3). Thus, RGC activity driven by independent cone stimulation can be used to map the spatial locations and sizes of subunits, capturing nonlinear computations across space at the scale of the RGC mosaic (Figure 3). Furthermore, it likely reveals the functional connectivity and RFs of the bipolar cells connecting stimulated photoreceptors to recorded RGCs. Recently developed methods (Liu et al. 2017, Maheswaranathan et al. 2018) may help to uncover the subunit structure in the RFs of other cell types as well.

Figure 3.

Nonlinear spatial summation in the RF. (a,b) Examples of OFF midget responses to stimulation of two individual cones in isolation (left and right raster subpanels) and concurrently with opposite contrast (middle raster subpanels). In panel a, during concurrent stimulation, the response to the light increment in this OFF cell is rectified before it reaches the cell and has no effect on firing (nonlinear summation). In panel b, RGC produces almost no response to concurrent stimulation, presumably due to linear summation and the cancellation of opposing cone signals in a common bipolar cell prior to rectification. These two cones form a linear subunit within the RF. (c) Schematic illustration of how the subunit model works: (left) schematic circuit connections forming the RF and (right) computational operations within this circuit. (d) Estimated subunits within a mosaic of OFF midget RFs. Circles denote cone locations and converging lines represent cone inputs to individual OFF midget RFs (see also Figure 2). Cone pairs and triplets with linear summation (i.e., subunits) are colored and share a gray outline. Abbreviations: RF, receptive field; RGC, retinal ganglion cell. Figure adapted from figures 1, 2, and 6 in Freeman et al. (2015), published under the Creative Commons Attribution 4.0 International Public License.

Single-Cone Receptive Field Maps Help Establish the Circuits for Color Vision

The perceptual opposition of red and green color sensations, and blue and yellow color sensations, has long been central in theories of color vision [Hering 1964 (1878)]. More recently, the discovery of opponent signals in primate RGCs (Gouras 1968) suggested that perceptual opponency may originate in the retina (De Monasterio et al. 1975; but see De Valois & De Valois 1993) and indicated that opposing signals from different cone types play a key role in shaping the color information conveyed to the brain. Blue-yellow opponent signals are carried by small bistratified RGCs (Dacey & Lee 1994), which receive excitatory input from S cones and suppressive input from L and M cones, via specific and distinct bipolar cell circuits. Red-green opponency has long been identified with midget RGCs and parvocellular LGN neurons that, in some cases, receive opposing inputs from L and M cones (Derrington et al. 1984, Martin et al. 2001, Field et al. 2010, Wool et al. 2018). The most likely mechanism of this opponency is a different combination of L and M cone inputs in the RF center and surround. For example, a pure L cone center and pure M cone surround would produce strong L-M opponency for a stimulus covering the entire RF. However, most peripheral midget cells receive mixed L and M inputs in both the RF center and the surround (Figure 2) with no anatomical evidence for cone selectivity (Martin et al. 2001, Field et al. 2010, Wool et al. 2018), raising the possibility that any opponency in midget cells is due to random sampling of the cone mosaic. Therefore, to understand whether red-green opponency is produced by specific circuit mechanisms or random sampling requires probing how individual inputs of different cone types over space are sampled by midget RGCs.

The sampling of cone inputs by many simultaneously recorded midget cells was measured to test this random connectivity hypothesis (Field et al. 2010). For each midget cell, a measure of cone input purity (dominance by either L or M cones) was compared statistically to the same purity measure obtained with the identities of the L and M cones artificially permuted. Permutation significantly reduced the degree to which L or M cone input dominated the RF center, narrowing the distribution of cone purity in the population of midget cells by 15–20%, but did not change purity in the RF surround (Field et al. 2010). This result strongly favors a mixed but somewhat biased sampling model for the RF center, and random sampling for the surround, of midget RGCs. Importantly, unlike other studies (Wool et al. 2018), this approach does not make any assumptions about the shape of the RF profiles of midget cells or about the L and M cone distributions and spatial locations in a particular cone mosaic. Instead, it uses the measured values of these key factors that shape cone opponency in RGCs.

THE SENSATION AND PERCEPTION OF SINGLE-CONE STIMULATION

Single-Cone Activation Produces a Reliable Sensation

When a stimulus activates a single cone, psychophysical measurements of detection threshold tell us how faithfully that activation is preserved for sensation. A fundamental question, then, is whether activation of a single cone is detectable. Experiments using adaptive optics to present very small monochromatic spots against an otherwise dark background have shown that the answer is yes (Hofer et al. 2005b). A light-capture model indicated that most of the stimulus energy was absorbed by a single cone (see also Makous et al. 2006). In these experiments, however, fixational eye movements precluded knowledge of which cone was stimulated on each presentation of the stimulus. Thus, it was not possible to assess how sensitivity varies from cone to cone. Such measurements would provide clear behavioral correlates to the substantial variation (and potential coordination) in cone inputs to RGCs described in the physiological measurements above.

Single-Cone Detection Thresholds Depend on Eccentricity and May Vary Between Nearby Cones

Recent technical advances (see above) now enable repeated targeting of an individual cone for visual stimulation in behaving humans and thus measurement of the minimum detectable light intensities of cone-sized stimuli (Harmening et al. 2014). Stimulus increments directed at single cones were visible near the fovea, with the threshold for visibility increasing as a function of eccentricity. Beyond 5° eccentricity, subjects were not able to detect single-cone increments against the background studied or over the range of intensities that could be delivered. A possible explanation for the increase in thresholds with eccentricity is that the number of cones pooled at the site of perceptual decision may increase with eccentricity, so that single-cone activation must be detected against a larger amount of noise coming from unstimulated cones (see also Field & Rieke 2002). Accuracy of targeting was confirmed by lower detection thresholds in locations containing a cone than in locations between the cones (about a one-and-a-half-fold difference). This level of increase was consistent with an optical model in which cone-targeted stimuli primarily activated a single cone. However, the possibility that multiple cones contributed to detection threshold when individual cones were targeted could not be completely ruled out.

Heterogeneity of Single-Cone Sensitivity

Preliminary results (Bruce et al. 2014, Sincich et al. 2016) suggest that there may be systematic variation in sensitivity between adjacent cones. Any such variation does not appear to be related to gross differences in the reflectivity of cones, as assessed by structural images acquired using adaptive optics (Bruce et al. 2015), and could potentially be related to variation in cone optical waveguide coupling or inhomogeneous sampling by downstream neurons (see section titled Discussion).

Single-Cone Thresholds Depend on Context, Revealing Postreceptor Processing

Threshold measurements can also be leveraged to study how signals from multiple cones interact and to relate these interactions to anatomical and physiological findings (Tuten et al. 2017). Thresholds for individually targeted L and M cones were differentially elevated by adapting backgrounds of different wavelengths in a manner consistent with the type of the targeted cone. In addition, the threshold elevations produced by background lights varied systematically with the number of L and M cones in the local neighborhood of each targeted cone (Figure 4a). Modeling and comparison with the anatomy and physiology of horizontal cells (Wässle et al. 1989; Dacey et al. 1996, 2000) provided estimates of H1 and H2 horizontal cell RF size in the living eye. This work highlights the power of psychophysical experiments that target individual cones to advance our functional understanding of postreceptoral processing, particularly when the results are treated in the context of retinal anatomy and physiology.

Figure 4.

(a) Relative thresholds of individually activated L cones (red circles), with stimuli presented against a background that activated M and S cones more than L cones. Thresholds increase systematically with the number of M and S cones in the local neighborhood of the targeted L cones. A corresponding effect (not shown here) was observed when M cones were targeted against a background that activated L cones more than M cones. Panel a adapted from figure 3 in Tuten et al. (2017), published under the Creative Commons Attribution 4.0 International Public License. (b) Measured size of the neural summation area for one subject (green circle) superimposed on an image of the foveal cone mosaic of the same subject. Panel b adapted from figure 3 in Tuten et al. (2018), published under the Creative Commons Attribution 4.0 International Public License. (c) The fraction of times different color terms were used to describe brief (550 nm) monochromatic flashes whose retinal size was commensurate with a single cone. The possible color names were red, orange, yellow, yellow-green, green, blue-green, blue, purple, and white. Each bar shows data for a single subject, with subjects ordered from left to right according to increasing L:M cone ratio. Panel c adapted with permission from figure 4 in Hofer et al. (2005b). (d) Single-cone naming data for targeted cones, showing a patch of mosaic where the type of individual cones was determined using microdensitometry. The color naming results for individual targeted cones are shown by a white annulus around each such cone. The fraction of times the terms white, red, and green were used is indicated by the fraction of the annulus rendered in the corresponding color. The cone type is indicated by the color—L cone (red) and M cone (green)—in the center of the annulus. Panel d adapted from figure 3 in Sabesan et al. (2016), published under the Creative Commons Attribution NonCommercial 4.0 International Public License. Abbreviations: L, long wavelength–sensitive; M, middle wavelength–sensitive; S, short wavelength–sensitive.

Summation of Cone Signals to Increments Is Complete Over Local Regions of Visual Space

Ultimately, we would like to know how signals from multiple cones are combined to mediate visual perception. A classic psychophysical approach to this broad question is to measure how the detection threshold varies with stimulus size. Measurements of this sort reveal an area of complete summation, within which incident photons contribute equally to detection (Riccò 1877). Before the advent of adaptive optics, measurements of the summation area were limited by optical blur (Davila & Geisler 1991). Two groups have now measured the foveal area of summation using adaptive optics (Dalimier & Dainty 2010, Tuten et al. 2018), with the more recent measurements also stabilizing the stimuli against motion blur from fixational eye movements. Comparison of the resulting summation areas and images of foveal cones in the same subjects (Figure 4b) revealed that neural mechanisms of detection sum over approximately 24 cones (the range across subjects was 17–37) (Tuten et al. 2018). Although the precise mechanisms underlying this summation remain to be identified, the area is much larger than the RF of a midget RGC in the fovea, indicating that the summation mechanism arises in a different RGC type (e.g., parasol cells) or perhaps at a later stage of visual processing (e.g., visual cortex).

A Wide Range of Color Percepts Can Be Induced by Single-Cone Stimulation

At the scale of individual cones, trichromacy breaks down because there is only a single cone at each retinal location. This breakdown, however, is well hidden in the color perception of spatially extended stimuli; the visual system spatially integrates signals from different cone classes in a manner that generally results in high-acuity spatial vision combined with excellent color acuity (Williams et al. 1991). The measurements described above address the combination of signals from cones of different types by RGC RFs. The ability to stimulate individual cones in the living human eye has now enabled the investigation of parallel questions posed in terms of perception.

Briefly flashed spots, presented near the fovea against a dark background and with adaptive optics to minimize blur, evoked a wide range of color percepts (Hofer et al. 2005b, 2012; Koenig et al. 2014). Indeed, subjects required the set of names red, green, and blue and also white, yellow, purple, and orange to describe the colors they saw in response to the spots (Figure 4c). The five subjects varied considerably in the relative numbers of L and M cones in their cone mosaics, as measured using adaptive optics microdensitometry. This observation seems inconsistent with an elemental sensations hypothesis (Helmholtz 1896, Krauskopf & Srebro 1965, Hofer et al. 2005b), in which there is a single elemental color percept associated with the activation of each cone type (e.g., L cones signal red). Under this hypothesis, accounting for the wide range of observed colors would require assuming that more than one cone was inadvertently stimulated. This possibility, however, is inconsistent with the observation that the largest fractions of white responses were given by subjects with the most skewed L:M cone ratios. This pattern is the opposite of what perceptual mixing of the three cone type–specific sensations predicts, because the number of trials on which L, M, and S cones were stimulated would be the smallest for the subjects with the most skewed cone ratios (Hofer et al. 2005b).

A Wide Range of Small-Spot Color Names Is Consistent with a Bayesian Model

The wide range of color names elicited by small spots presented against a dark background can be accounted for by a Bayesian model of optimal stimulus reconstruction from the pattern of mosaic cone activations (Brainard et al. 2008). The key insight underlying the model is that, because there are spatial and spectral correlations in natural images, information about the visual input at a retinal location is available not only from the cone at that location but also from nearby cones. This insight plays out through the Bayesian model to predict how the activation of a single cone is interpreted perceptually. In particular, when the activation is against an otherwise dark field, perception should depend on the fine structure (i.e., the cone type makeup) of the surrounding cone mosaic. If a single cone is activated while other nearby cones of the same type are not, the information provided about the external stimulus is different than in the case where a single cone is activated while other nearby cones of different types are not. The Bayesian model says that the visual system is sensitive to this type of difference and works out the implications in detail (see Brainard et al. 2008, figures 7 and 8).

Stimulation of Individually Targeted Cones Elicits a Reliable Color Percept

The inferences that could be made from the data described above are limited by fixational eye movements, which preclude knowledge of which cone was activated on any given trial. This limitation was overcome using real-time eye tracking in conjunction with adaptive optics (Sabesan et al. 2016; see also Schmidt et al. 2018). Subjects provided largely reliable color names in response to the repeated activation of individually targeted cones (Figure 4d). Different color names, however, were provided in response to the activation of different individual L cones as well as to the activation of different individual M cones, confirming variation in the sensation elicited by different cones of the same type. Of note, though, is that subjects in this study required only the color terms red, green, and white, in contrast to the wide range of color names used by the subjects in the Hofer et al. (2005b) study. The reasons for this interesting interstudy difference are not known. Several potentially important stimulus parameters (e.g., duration, background, intensity, retinal stabilization) were not matched across the studies. Systematic variation of these parameters should clarify our understanding of the neural computations linking single-cone activation to color appearance.

DISCUSSION

Above we have reviewed recent work focused on neural circuits and perception at the elementary resolution of the visual system. This work has begun to clarify how signals from individual cones are represented by retinal circuits and how these signals shape what we see. In addition, the ability to stimulate cones in combination has provided new insights about how information is processed downstream of the cones. Below we outline opportunities and challenges for future work.

Decades of research have revealed the degree to which a single photon absorption in a handful of rods can produce a signal that propagates faithfully from the retina to the brain, producing reliable perception despite the many potential sources of corrupting biological noise (Field & Sampath 2017). Although cones do not reliably signal the absorption of individual photons (Donner 1992, Koenig & Hofer 2011), similarly faithful transmission of cone signals may be crucial for high-resolution daylight vision. Robust responses to the stimulation of a single cone have been recorded in peripheral midget and parasol RGCs, and the trial-to-trial variability in these responses can be characterized (Li et al. 2014) (see also Figure 3). These measurements reveal the fidelity with which single-cone signals are transmitted to the brain. Matching stimulus properties in single-cone psychophysical experiments would extend this understanding through to perception. Ideally, the comparisons would be made at corresponding retinal locations: Thus, an important technical goal is to enable single-cone physiology at locations closer to the fovea (Sincich et al. 2009, Yin et al. 2014). A second challenge is to further ensure that possible diluting effects of light scatter and optical blur on the purity of single-cone activation are well matched in the physiology and psychophysics (Hofer et al. 2005b, Field et al. 2010, Harmening et al. 2014).

Signals originating from single cones flow to diverse classes of RGCs (Field et al. 2010). The sampling of cone signals by these populations and the corresponding perceptual visual experience of humans have the potential to be connected experimentally using the approaches described in this review. For example, a full view of how robustly single-cone signals are transmitted by diverse RGC populations to the brain, and the variability of these signals over space, may help to explain the mechanisms underlying the psychophysically measured neural summation area as well as the color perceptions associated with single-cone stimulation. As a concrete example, a striking observation in physiological data is that single-cone stimulation with decrements of light produces much stronger and more reliable responses in OFF cells than equivalent increments of light produce in ON cells (Li et al. 2014) (see also Figure 3). The mechanism of this asymmetry is not understood, although it may arise from asymmetries in cone photoresponses themselves (see Endeman & Kamermans 2010; F. Rieke, personal communication). A straightforward prediction is that single-cone decrement signals may be more detectable by humans than increment signals. Such asymmetries have been observed for larger stimuli (Cohn 1974). Confirmation of this prediction at the single-cone level would suggest that mechanisms in the first cells in the visual hierarchy control the detectability of the most elementary signal. Deviations from the prediction would suggest that the behavioral detectability of the elementary signal depends heavily on the population of RGCs that transmit it to the brain. A related possibility is that the psychophysical characterization of the spatial spread of threshold elevation produced by single-cone stimulation could reveal the spatial structure of the postreceptoral mosaic of the cell type that mediates detection (Boehm et al. 2017) and provide a direct signature of the RF properties of the underlying mechanisms of detection.

Ultimately, we want to understand how neural computations so effectively integrate the discrete signals from individual cones of different spectral types to produce percepts that seamlessly represent the visual world. Earlier studies of the limits of human visual acuity (Westheimer & McKee 1977, Geisler 1984, Williams 1985, Smallman et al. 1996, Westheimer 2012) indicate that this integration preserves spatial information at the resolution of the underlying cone mosaic, while at the same time combining information across cones of different spectral types to enable trichromatic color vision (Williams et al. 1991, Brainard et al. 2008). Experiments that probe vision and neurophysiology at this resolution hold significant promise for generating a coherent understanding of how the activity of individual neurons contributes to perception.