Are we underestimating the richness of visual experience?

Abstract

It has been argued that the bandwidth of perceptual experience is low—that the richness of experience is illusory and that the amount of visual information observers can perceive and remember is extremely limited. However, the evidence suggests that this postulated poverty of experiential content is illusory and that visual phenomenology is immensely rich. To properly estimate perceptual content, experimentalists must move beyond the limitations of binary alternative-forced choice procedures and analyze reports of experience more broadly. This will open our eyes to the true richness of experience and to its neuronal substrates.

The world of visual experience is often characterized as being immensely “rich,” meaning that it seems to have a very densely detailed structure. Yet, when subjects in controlled experiments are prompted with queries about their experience, little of this richness comes across, and so we are left to question the quality of such introspective characterization. As summarized in a recent review (Cohen et al., 2016) the evidence points to a “severely limited amount of information that subjects can report at any given moment.” Cohen et al. suggest that our impression of richness is related to perceptual experience of “ensemble statistics,” features of visual experience that are about detailed structure in the world. However, we shall argue that this inference is problematic for two reasons: first, despite many claims to the contrary, introspective reports of a richly detailed visual experience—e.g. of a colorful visual periphery—are often in fact “supported” by solid psychophysical evidence. The reliable correspondence between phenomenological report and “first order” psychophysics is worth deeper consideration than it often receives. Second, limitations in what subjects can report are partially imposed by the strictures of traditional psychological research, in particular the requirement that subjects generate only pre-defined, high-level categorical responses, ranging from binary choices (e.g. face or nonface, male or female, same or different, horizontal or vertical, leftward or rightward), to one of 10 numbers or 25 letters. If such strictures could be removed, the lower bound of “experiential bandwidth” (to use Cohen and colleagues’ term) could increase significantly. As we discuss below, introspective reports of rich experience should not be discounted, and new paradigms are needed that can take experience as something to be explained, rather than as something to be explained away.

First, a psychophysical defense of introspection: a popular view in cognitive neuroscience is that visual experience is surprisingly sparse, consisting of coarse descriptions of a scene (its “gist”) and the few objects that are grasped by selective attention. Cohen et al. (2016) suggest that if we also take into account “ensemble statistics”—coarse but local descriptions of complex regions of a scene that are outside the attentional focus—the “surprising” part of sparseness can be explained. To dissolve this conflict, the sparseness argument relies to some extent on the claim that introspection is a poor tool for investigating experience. If introspection is simply a poor tool, introspective reports of rich experience can be rejected out of hand (Cohen et al., 2016). Color vision in the peripheral visual field is often invoked in the discussion to make this point (Lau and Rosenthal, 2011; Cohen et al., 2016). Owing to the low cone density of the peripheral retina, it is (wrongly) inferred that color vision should be restricted to the central visual field. Accordingly, various explanations are offered as to why subjects “claim” to experience a colorful periphery: memory or expectation may be “filling in” reports (Cohen et al., 2016) or poor attention may be inflating judgments about peripheral phenomenology (Lau and Rosenthal, 2011). However, such claims have been repeatedly refuted by psychophysical studies that scale stimuli according to cortical magnification factor, i.e. by the amount of cortex devoted to each degree of retinal area (Anstis, 1998; Block, 2007; Tyler, 2015). A recent mini-review details how peripheral color experience is not essentially different from foveal color experience (Tyler, 2015)—the only difference is in the objective resolution of the visual field (Gordon and Abramov, 1977; Hansen et al., 2009; Webster et al., 2010; Tyler, 2015). Thus, peripheral color experience has no more to do with memory, expectation or selective attention than does foveal color experience. A related claim that peripheral vision is blurry and that introspective claims of peripheral clarity should, therefore, be doubted is similarly ill-founded (Anstis, 1998). More to the point of the “ensemble statistic” idea, when the emphasis is on experimental rigor it does not appear that any known summary statistic is capable of capturing the appearance of peripheral vision (Wallis et al., 2016). Indeed, we do see much local spatial detail even in peripheral vision [with the caveat that we cannot “recognize” what we see there as well as we do foveally without a more drastic rescaling of the stimulus (Pelli and Tillman, 2008)]. Translated into everyday experience, then, there is no prima facie reason to doubt an observer’s introspective reports of colorful, clear experience extending across her visual field as when she views a clear blue sky: such a report is psychophysically plausible. These considerations undermine a popular criticism of introspective reports as illusory or even delusional, especially in the visual periphery. If we take normal psychophysics as a guide, it seems that we do experience what we think we do.

Of course, naïve reports about subjective experience should not be taken at face value. Indeed, there is a strong tradition in perception science of taking phenomenology seriously as a critical component of psychophysical experiments (Spillmann, 2009). Consciousness science has honed this tradition with the goal of separating sensitivity, which can rely entirely on nonconscious processing, from introspective accuracy, using metacognition methods that assess the availability of a perceptual judgment to introspection (Seth et al., 2008; Fleming and Lau, 2014). For example, using metacognition to quantify the accuracy of introspection, recent studies showed that attention-bound working memory and pre-attentive sensory (iconic) memory are equally accessible to introspection, debunking the claim that the latter is unconscious or illusory (Vandenbroucke et al., 2014). Another recent study showed that memory for incidental visual experience of specific, nontarget faces in crowds is far better than a sparse experience account might support, well in line with our intuitions about incidental visual experience (Kaunitz et al., 2016). In these cases, metacognitive accuracy objectively demonstrates that we do know what we think we know and forget what we think we forget. Introspective reports of rich visual experience have a solid psychological basis.

If our intuitions about our rich visual experience have more validity than the sparse account supposes, how can we better estimate this richness? To begin, we should reconsider how the existing wealth of psychophysical data can ground a much more realistic view of the contents of visual experience than the focus on objects and ensembles gives us. But the future may require loosening the straightjacket of simple discrimination and identification tasks that have long been the bread and butter of psychophysical studies. While forced choice reports are easy to obtain and analyze, they fail to adequately deal with the complexity of experience, and create the illusion that phenomenology is made up entirely of high-level categories that are experimentally convenient to construct and analyze, such as letters or digits, nameable objects, and statistical summaries (Cohen et al., 2016).

A better reflection of phenomenal content may be found in the analysis of natural speech or drawings. Consider (Fei-Fei et al., 2007), in which subjects had to verbally describe briefly flashed (and masked) real-world photographs. Even for very brief exposures, spoken utterances are far richer in content than binary discrimination or reaction tasks have revealed. While in the past these data were unsuitable for rigorous analysis, one might now collect subjective reports from workers on Amazon’s Mechanical Turk and analyze these using natural language processing algorithms; such methods may allow us to assess the richness of a complex free “report,” and even may give us access to subjects’ confidence in their free reports on a phrase-to-phrase basis [e.g. via methods used in marketing research to analyze “sentiment content” (Pang and Lee, 2008; Liu, 2012)]. Potentially, even richer reports could be obtained by asking artistically talented subjects to draw what they see, and mining their sketches using appropriately trained deep learning machine vision algorithms. Developing “richness assessment” analyses would be the easy part, however. The deeper challenge would be assessing the degree to which a report adheres to an actual, introspective experience (as opposed to e.g. a confabulation based on unconsciously processed information or prior information); but exactly this same challenge is faced by traditional, and even state-of-the-art, approaches to consciousness science. Helping to calibrate such methods may be theories of the neural substrate of consciousness: if a theory predicts that a certain experience should occur under such-and-such circumstances, and if a subject’s analyzed reports suggest such an experience, then we can have greater confidence in such techniques.

As an example of this approach, take Sperling’s classic “partial report” experiment (Sperling, 1960) (Fig. 2), often cited to emphasize the limited capacity of phenomenal vision, since subjects can report at most 3–4 of 12 briefly flashed letters. But is that really all they see, perhaps augmented by some summary statistics? A moment’s reflection indicates that, if only they were asked, subjects could report much more—one certainly perceives that there are many black marks, that they are arranged in rows and columns, in a rectangular array, without depth nor motion, within a rectangular display, against a bright homogeneous background that is spatially extended, being composed of a multitude of distinguishable locations, each with its specific neighbors, and so on. Indeed, the immense number of phenomenally distinct regions and their topographic relations, which configure the experience of space itself, is typically taken for granted rather than included in the catalog of conscious contents (Tononi et al., 2016). Furthermore, all these marks are “bound” together within the same experience in a complex pattern of relations: e.g. the letter reported as “A” is located at a particular spatial location in the rightmost column in the upper row of the array, an upper-case character composed of two straight oblique edges and a horizontal edge each with internal spatial structure, and on and on (Tononi et al., 2016). While subjects may not be able to recognize specific identities of many features such as the specific colors or identities of individual Sperling figures (as in e.g. Ward et al., 2016), they can effortlessly report that what they saw were letter-like figures, distinct from other possible textures that could occupy the same location (De Gardelle et al., 2009). Similarly, subjects can report, with high confidence, what they did not see—such as the burning Twin Towers, a Bernese Mountain dog, the President, and so on. While subjects do not normally report any of these aspects—positive or negative—of phenomenal vision, it is not because they do not experience them, but simply because they are not asked.

Figure 2.

Sperling’s (1960) original display. Subjects had to report as many letters of a briefly flashed display as possible, ostensibly revealing the limits of phenomenal vision. Previous experiments focused on the accuracy and the number of the remembered letters. However, conscious phenomenology is far richer than a focus on letter identity would suggest.

We argue that the richness of visual experience has been neglected by psychologists who traditionally emphasize button pressing and categorization, and who have focused too much on high-level, categorical properties of visual experience. Our premise is that scholars of consciousness should not treat introspective reports of phenomenology with distrust and reject them—in fact, when we look at the psychophysics we may be surprised at how much support there is for “naïve phenomenology.” If we can accept that visual phenomenology is actually very much like what it immediately seems to be, then the greatest difficulty will not be in establishing the richness of visual phenomenology—we should not treat phenomenology as a doubtful hypothesis, but as a thing to be “explained”—but in characterizing it properly and linking it to neural and behavioral evidence. Relying exclusively on traditional button-press methodology has led many scientists to identify the neural correlates of consciousness with the neural correlates of reports of simple categories (Tsuchiya et al. 2015; Koch et al., 2016). There is more to experience than meets the button.