Lack of awareness despite complex visual processing: Evidence from event-related potentials in a case of selective metamorphopsia

Significance

How does the neural activity evoked by visual stimuli support visual awareness? In this paper we report on an individual with a rare type of neural degeneration as a window into the neural responses underlying visual awareness. When presented with stimuli containing faces and target words—regardless of whether the patient was aware of their presence—the neurophysiological responses were indistinguishable. These data support the possibility that extensive visual processing, up to and including activation of identity, can occur without resulting in visual awareness of the stimuli.

Abstract

Visual awareness is thought to result from integration of low- and high-level processing; instances of integration failure provide a crucial window into the cognitive and neural bases of awareness. We present neurophysiological evidence of complex cognitive processing in the absence of awareness, raising questions about the conditions necessary for visual awareness. We describe an individual with a neurodegenerative disease who exhibits impaired visual awareness for the digits 2 to 9, and stimuli presented in close proximity to these digits, due to perceptual distortion. We identified robust event-related potential responses indicating 1) face detection with the N170 component and 2) task-dependent target-word detection with the P3b component, despite no awareness of the presence of faces or target words. These data force us to reconsider the relationship between neural processing and visual awareness; even stimuli processed by a workspace-like cognitive system can remain inaccessible to awareness. We discuss how this finding challenges and constrains theories of visual awareness.

A large body of research in cognitive science is aimed at describing the markers of visual awareness. Theorists generally agree that awareness requires integration across multiple formats and levels of representation, but the nature of that integration in cognitive and neural terms remains under debate (1–4). One approach to these issues is to delineate the levels of processing or types of representation that always, sometimes, or never reach awareness. In neurologically intact populations, masking techniques—such as continuous flash suppression, binocular rivalry, inattentional blindness, and backward masking—are used to control the stimuli that participants are aware of and investigate concomitant processing. However, these paradigms are complex, with difficulties arising in ensuring that stimuli are fully masked from awareness, and in matching conditions with awareness to those without awareness.

Deficits of visual awareness resulting from neuropathology offer alternate approaches to the study of awareness. Research on blindsight, hemispatial neglect, and other conditions has offered insights into the cognitive and neural correlates of awareness (5–9), and more broadly, studies of individuals with cognitive deficits have shed light on the processes underlying vision, language, memory, and higher-order thought (for recent reviews, see refs. 10 and 11).

We report the case of an individual (initials: R.F.S.) with a neurodegenerative disease who shows neurophysiological evidence of complex cognitive processing in the absence of awareness. The phenomena we report raise questions about the conditions necessary for awareness of visual stimuli. R.F.S. presented with a category-specific metamorphopsia (distorted perception) for Arabic digits (e.g., the number 8). When presented visually with a digit in the range 2 to 9, R.F.S. was completely unable to recognize the digit or even perceive its shape. Instead, he reported seeing a tangle of black lines (Movie S1). Critically, the distorted percept also affected overlapping, embedded, or immediately-adjacent visual stimuli, disrupting his awareness of these stimuli. We report a series of experiments probing the cognitive and neural fate of the distorted stimuli as an avenue to understanding the processes underlying normal visual awareness. First, through a series of behavioral experiments, we confirmed the observation that R.F.S. was unable to report the identity of visual stimuli embedded in the digits 2 to 9. Furthermore, we showed that he exhibited chance-level accuracy on two-choice discrimination tasks for these distorted stimuli, suggesting that his deficit did not spare the sorts of implicit knowledge sometimes found in other deficits of visual awareness (e.g., blindsight) (12–14).

Next, we used event-related potentials (ERP) to probe the neural processing of the stimuli that R.F.S. could not report. We tested whether the presence of the distorted digits disrupted neural processing of embedded stimuli, examining 1) visual processing of face stimuli, using the N170 component and 2) processing of visual words in the context of a target-detection task, using the P3b component. R.F.S. was not aware of the embedded stimuli, much less of the critical distinctions among them (face vs. nonface, target vs. nontarget word); yet the ERP results clearly demonstrated preserved neural sensitivity to these distinctions. Furthermore, the ERP responses occurring in the absence of awareness did not differ from those observed when R.F.S. was aware of the relevant distinctions among stimuli.

These data provide a clear demonstration of sophisticated cognitive and neural processing dissociated from reportable experience, forcing us to reconsider the conditions giving rise to visual awareness. Our findings suggest that the relationship between the cognitive processes necessary for registering a stimulus as a known identity and conscious awareness is more complex than previously understood, requiring a reexamination of theories of visual awareness.

Results

A Case of Category-Specific Metamorphopsia.

R.F.S. is a left-handed engineering geologist who was 60 y old at the start of this study. He suffered an acute neurological event in October 2010 involving headache, transient aphasia and amnesia, temporary loss of vision, and abnormal bitemporal sharp waves recorded by electroencephalography (EEG). In early 2011, he began to experience a range of motor and somatosensory deficits, including upper limb tremor, episodes of phantom touch, involuntary muscle spasms, dystonia, and difficulty walking, which progressed in severity over the course of the next several years. Neuropsychological testing revealed impairment in long-term memory [e.g., Wechesler Memory Scale (WMS)-III: Logical Memory I scaled score of 2, Logical Memory II scaled score of 6 (15); Hopkins Verbal Learning Test-Revised first percentile (16)] as well as difficulty accurately tracing and perceiving the orientation of some written digits. A diagnosis of corticobasal syndrome (probable corticobasal degeneration) was made in May 2011 on the basis of symptomology and parietal atrophy visible on MRI (Fig. 1) (17, 18). Subsequent MRIs have shown progressing cerebral, midbrain, and cerebellar volume loss consistent with corticobasal syndrome.

Fig. 1.

Coronal slice of R.F.S.’s brain. MRI scan, August 2011, presented in neurological convention (left side of image is right hemisphere). Slice taken at y = −49 Talairach. Radiologists noted abnormal atrophy in the parietal lobes.

By August 2011, R.F.S. was completely unable to recognize, name, copy, or comprehend the digits 2 to 9, presented either singly or in multicharacter strings (e.g., “8,” “476,” “A7”). He described the appearance of these digits as a tangle of black lines with no discernible relationship to any recognizable shape (Movie S2). R.F.S. further stated that if he looked at the same digit on multiple occasions, the tangle of lines looked different each time, and so he could not learn to identify the digits from the distorted percepts. When a colored digit was shown, R.F.S. described the color as spread across the general area of the digit as a background color (Fig. 2 and Movie S1). The perceptual distortions were not diminished by any stimulus parameters we manipulated, including size, exposure duration, visual field location, contrast, and flicker. Even under conditions requiring two-choice discrimination, R.F.S. demonstrated no ability to discriminate the shapes of the digits 2 to 9. When pairs of these digits were presented for same-different judgments, he performed at chance (28 of 52).

Fig. 2.

Category-specific metamorphopsia: R.F.S. was unable to perceive, describe, or copy the form of Arabic digits 2 to 9. This figure depicts a direct copying task with the number 8: Stimulus (Left) and R.F.S.’s copy (Right). R.F.S. was given various pens and markers to choose from to complete the task. He began by drawing the black lines, and then added the orange background. See Movie S1.

In contrast to his gross impairment for the digits 2 to 9, R.F.S.’s visual-perceptual skills were unimpaired for other visual stimuli, including stimuli similar to digits. Most striking was that he had no difficulty naming, copying, comprehending, or otherwise processing the digits 0 and 1. Selective sparing or impairment of 0 and 1 has been reported in multiple previous cases of cognitive impairment (19–24). These digits have been argued to have a special cognitive status, potentially stemming from their role in syntactic operations with numerals, semantic value as the absence of quantity and unity, or unique realization in verbal numeral names. Also potentially relevant to the sparing of 0 and 1 in R.F.S.’s case is that these digits have particularly simple shapes and closely resemble letters (for 0, the letter O, and for 1 the uppercase letter I and the lowercase letter l).

R.F.S.’s letter recognition was largely intact, although he exhibited mild perceptual distortion for a subset of letters (M, N, P, R, S, Z). This distortion, which usually did not prevent accurate recognition of the letter, may be a milder form of the impairment affecting the digits 2 to 9. For other types of visual stimuli, R.F.S.’s perception appeared normal. He made no errors in visual picture naming, copying simple shapes, and naming typographical symbols (e.g., #, $, +).

R.F.S. also had no impairment of numerical abilities other than Arabic digit identification. On the Arithmetic subtest of the Wechsler Adult Intelligence Scale-Revised (WAIS-R) (25), which is administered verbally, he achieved a scaled score of 14 (91st percentile). Furthermore, he was intact in his ability to read and comprehend number words (e.g., “five”) and Roman numerals (e.g., “XIV”); he initially used these alternative forms to write numbers and perform calculations after the onset of his difficulties with digits. R.F.S. was also able to learn novel “surrogate” digits we developed to replace the Arabic digits 2 through 9 (SI Appendix, Fig. S1). Because he could do numerical work with the surrogate digits, R.F.S. was able to remain employed as an engineering geologist for several years after the onset of his disorder (until his retirement in October 2014). R.F.S. continues to use the surrogate digits in daily life (e.g., for jotting down phone numbers or working on his computer, which we modified to display numbers in surrogate-digit form).

Impaired Digits Affect Nearby Stimuli.

Not only does R.F.S. experience gross perceptual distortion for the digits 2 to 9, but these digits also drastically impair his perception of other stimuli presented in close spatial proximity. R.F.S. was presented with displays consisting of a large letter or digit with a line drawing of an object (e.g., a pear) embedded within the strokes of the character (Fig. 3). In the first block, R.F.S. was asked to name the object and the letter or digit; in blocks 2 and 3 he was asked to name the object only. R.F.S. was 100% correct in naming the letters (10 of 10) and the digits 0 and 1 (2 of 2) in block 1, but he was unable to name any of the digits 2 to 9 (0 of 8), responding with “don’t know.” His performance in naming the line drawings showed the same pattern in all three blocks of trials: He was perfect when the drawings were embedded in either letters (30 of 30) or the digits 0 and 1 (6 of 6). However, he was unable to name any drawings embedded in the digits 2 to 9 (0 of 24) and stated that he could not even tell a drawing was present. These results demonstrate that R.F.S.’s deficit in perceiving the digits 2 to 9 also affects other visual stimuli in the same spatial location.

Fig. 3.

Impaired digits affect embedded stimuli. R.F.S. was unable to name the object embedded in this 6, but was accurate for the G. Across 60 trials, R.F.S. was 0% correct when objects were embedded in digits 2 to 9 and 100% correct when embedded in a letter or digit 0 or 1.

No Evidence of Implicit Knowledge Found with Discrimination Tasks.

R.F.S. reports a lack of awareness for the identity of digits 2 to 9 and stimuli embedded within them. However, despite the lack of conscious awareness, R.F.S. might have implicit knowledge that, while not accessible to awareness, might be revealed with appropriate testing methods. Consider the phenomenon of blindsight, in which some individuals with visual-system damage report no awareness for a visual stimulus yet perform at above-chance levels in discrimination tasks probing knowledge of stimulus properties, such as shape or location. For example, blindsight patient D.B. was tested on perceptual discrimination between Xs and Os presented in blind regions of his visual field (14, 26). D.B. reported having no conscious visual experience for the stimuli and stated that he was just guessing on the task, yet he performed at above-chance levels on the X/O discrimination. Results of this nature have been taken as evidence for implicit knowledge in the absence of awareness, not only in the study of blindsight but also in research on other deficits of visual awareness, such as hemispatial neglect (27).*

We conducted a series of two-choice perceptual discrimination experiments to determine whether any evidence of implicit identification could be found in R.F.S. for stimuli embedded in impaired digits. Stimuli in these experiments were large characters with embedded stimuli, and on each trial R.F.S. made a forced-choice decision by button-press about which of two stimulus types was embedded. Across five tasks, he was required to discriminate between simple visual shapes (×/+, ◇/□, ↑/↓), line drawings (instrument/vehicle drawing), and words (animal/object words). In all tasks, each stimulus was embedded in a large 8 or H (Fig. 4). Prior to each task, R.F.S. was informed about the stimuli and the response alternatives. R.F.S. insisted that on the digit trials he could not perceive anything on which to base his decision, but we encouraged him to guess. In each task, each of the two response alternatives was correct on half of the trials.

Fig. 4.

No implicit awareness of identity of embedded stimuli. R.F.S.’s task was to make a decision about the figure presented inside the character, as in these examples. Row 1: “square or diamond?”, Row 2: “up or down arrow?”, Row 3: ”animal or thing word?” His performance was 49% (at chance) when the figures were inside an 8, and 99% when the figures were inside an H.

When asked to distinguish between embedded “×” and “+” shapes, R.F.S. was 98% correct (46 of 47 correct) on the letter trials and 50% correct (23 to 46 correct) on the digit trials. (Despite the instruction, R.F.S. did not respond on all trials, and we excluded no-response trials from analyses.†) In distinguishing between an embedded diamond and square, R.F.S. was 100% correct (96 of 96) on the letter trials and 41% correct (37 of 90 correct) on the trials with digits 2 to 9. When the embedded stimuli were up and down arrows, his accuracy was 100% on letter trials and 53% (25 of 47 correct) on digit trials. In distinguishing between animal words and object words, R.F.S. was 98% correct (46 of 47 correct) on letter trials and 50% correct (22 of 44 correct) on digit trials. Finally, in distinguishing between drawings of vehicles and musical instruments, R.F.S. responded with 98% accuracy (47 of 48) on the letter trials and 56% accuracy (24 of 43 correct) on the digit trials.

Across these five tasks, R.F.S. performed with 99% accuracy (283 of 286 correct) on stimuli embedded in the letter H, and 49% accuracy (131 of 270 correct) on stimuli embedded in the digit 8, which did not differ significantly from chance, z = 0.043, P = 0.71. Chance-level performance on discrimination tasks could result from extreme response biases in which one of the response alternatives is chosen all or nearly all of the time (e.g., virtually always responding “diamond” in the diamond/square task). However, R.F.S. did not exhibit such biases with digit-embedded stimuli: Across the five tasks, the more frequently chosen response alternative was selected, on average, only 56% of the time, with values for individual tasks ranging from 52 to 68%. Accordingly, all tasks provided ample opportunity to detect departures from chance performance.

R.F.S. evidenced no ability to discriminate visual stimuli embedded in a distorted digit. Perceptual discrimination and forced-choice paradigms are often employed to reveal implicit knowledge (2, 4, 14, 30); our tasks revealed no such knowledge for R.F.S. We conclude that in addition to a lack of explicit awareness, R.F.S. also had no implicit knowledge of the shapes and identities of the embedded stimuli.

The Nature of R.F.S.’s Deficit.

R.F.S.’s deficit of awareness for the digits 2 to 9 and embedded stimuli is both profound and remarkably stable, over the nearly 8 y in which we tested him, and across a wide range of testing conditions. The stability of the deficit makes it particularly well-suited to studying the neural correlates of (the lack of) visual awareness. However, given the rare form of R.F.S.’s metamorphopsia, how can we be sure that his deficit is genuine? With any unusual deficit, there is the possibility that the underlying dysfunction is psychiatric, psychogenic, or “functional,” rather than an impairment of basic perceptual/cognitive processes. We believe this unlikely in the present case for multiple reasons. Primarily, R.F.S. has an organic neurological condition: Corticobasal syndrome was diagnosed in the same period as the metamorphopsia and has caused striking, severe, and progressive impairments. Second, at the time of our study R.F.S. was seeing a psychiatrist for help in adjusting to his condition, and the psychiatrist had no suspicion that any of his perceptual, cognitive, or physical symptoms reflected a functional disorder. In addition, R.F.S.’s performance in two-choice discrimination was not below chance, as is often found in cases of malingered deficits (31–34); nor was evidence for malingering found in a neurological evaluation (Test Of Memory Malingering, administered June 2011). Furthermore, R.F.S. did not use his metamorphopsia (or other deficits) to gain disability payments or other benefits and continued to work as long as he was physically able, making use of the surrogate digits we created for him. Finally, R.F.S.’s selective digit metamorphopsia is strikingly analogous to reported cases of selective face metamorphopsia: Category-specific perceptual distortion of faces, such as warping, blurring, or “melting” of features (35–44).

On the basis of these arguments and the results presented in previous sections, we draw the following conclusions about R.F.S.’s deficit. First, the impaired performance observed for digits and embedded stimuli does not reflect a psychiatric disorder or malingering, as indicated by the absence of below-chance discrimination and other evidence discussed above. Second, when R.F.S. is presented with a digit in the 2 to 9 range, he is consciously aware only of a tangle of lines (“spaghetti”) and is unaware of the digit’s identity. Third, when visual stimuli (e.g., pictures, symbols, words) are embedded with digits, R.F.S. is not consciously aware of the identity of the embedded stimuli or even of their presence. These conclusions follow from R.F.S.’s consistent verbal reports about his perceptual experience and knowledge. Fourth, testing with two-choice discrimination procedures fails to show implicit knowledge of the sorts sometimes revealed by these procedures in conditions such as blindsight. Accordingly, we are confident in using R.F.S.’s deficit to investigate neural processing in the presence of disturbed perceptual awareness.

R.F.S.’s deficit provides an experiment of nature that allows us to directly contrast conditions in which he is and is not aware of the distinctions between visual stimuli, a key comparison for evaluating theories of awareness (4, 45–47). To this end, we examined two ERP components that have been argued to index visual awareness of particular stimulus properties: The N170 and the P3b. We compared their presence across conditions in which R.F.S. was, and was not, able to perceptually distinguish the relevant stimulus properties.

Neurophysiological Evidence of Face Detection Despite Lack of Awareness: N170 Component.

We first examined the N170 component, which has been found to reliably distinguish face from nonface stimuli (48–52). The N170 is a temporo-occipital negativity observed between 160- and 200-ms after stimulus onset, with an amplitude that is reliably larger for faces relative to other visual stimuli (48, 51). We presented faces embedded in a digit (“8”) or a letter (“H”) and compared these to scrambled faces embedded in the same characters (Fig. 5 A and B). We verified that R.F.S. was unable to distinguish between digit stimuli containing faces and those containing scrambled faces, consistent with previous testing with embedded figures. Thus, this experiment allowed us to test whether R.F.S.’s neural processing reflects a distinction that he is unaware of behaviorally: An N170 response for faces relative to nonfaces embedded in digits would suggest neural face detection despite a lack of awareness of the presence of a face.

Fig. 5.

Neurophysiological evidence of face detection despite lack of awareness: N170 component. We found robust differences of the amplitude of the N170 component when faces were compared to nonfaces, embedded in both letters and digits. Asterisks reflect significance of a permutation test comparing the two conditions (**P < .01, ***P < .005). (A) Sample stimuli and averaged response at P9 and P10 electrodes evoked by faces (red) vs. scrambled faces (blue) embedded in the letter H. (B) Same as A, embedded in the digit 8. (C) Difference waves at P9 and P10 electrodes evoked for faces minus scrambled faces embedded in a digit (green) vs. a letter (pink). Gray highlighting indicates time window of interest (170 to 200 ms). (D) Scalp topography indexing main effect of face presence/absence in the time window of interest.

When the embedding character was a letter, R.F.S. was aware of the embedded stimulus and could readily distinguish faces from scrambled faces. Under these conditions, the standard N170 effect (face – scrambled face) was significant (amplitude difference = −0.58 μV; P = 0.009) across the P9 and P10 electrodes in a time window from 170 to 200 ms after stimulus onset (Fig. 5A).

When the embedding character was a digit in the 2 to 9 range, R.F.S. was unaware of the distinction between stimuli with faces and those with scrambled faces. When questioned after presentation of practice stimuli, between test blocks, and at the end of the experiment, R.F.S. consistently described the stimuli in the digit conditions as “spaghetti,” and stated that he could not see any embedded stimuli, much less determine whether a face or scrambled face was present. Nevertheless, we observed a robust N170 effect in this condition (face – scrambled face amplitude difference = −0.68 μV; P = 0.002) (Fig. 5B). In fact, the N170 effect was equivalent in magnitude under circumstances in which R.F.S. was not aware whether a face was present (digit embedding) and circumstances in which he could clearly perceive its presence [letter embedding; Letter(face – scrambled face) – Digit(face – scrambled face) = 0.094 μV; P = 0.84] (Fig. 5C). In addition, when we compared the neural response to faces across the letter and digit conditions, we failed to find any difference (faces in letters – faces in digits = −0.01 μV; P = 0.98). These results indicate the face and scrambled-face stimuli were processed to a point that differentiated faces from scrambled faces and that this difference was indistinguishable in the digit and the letter conditions.

Neurophysiological Evidence of Task-Dependent Target Detection Despite Lack of Awareness: P3b Component.

The previous results indicate neurophysiological activity tracking a visual distinction that R.F.S. cannot report behaviorally. To probe higher-level processing, we examined the P3b ERP component with a task that involved distinguishing target from nontarget words. The P3b is found in response to infrequent, oddball, or target stimuli, relative to frequent or distractor stimuli. We aimed to elicit this component by embedding target and nontarget words in large digit and letter characters, thereby manipulating R.F.S.’s awareness of the distinction between trials with target words and trials with nontarget words. The P3b component is thought to reflect the integration of incoming information into working memory, as required for task-dependent target detection (53, 54). Importantly for the present study, the P3b has been shown to be highly sensitive to subjective reports of stimulus visibility (55), and is widely believed to index conscious awareness (56–58; compare with ref. 59) or postperceptual processing related to task-relevance (47, 60).

For the P3b target-based response, we presented R.F.S. with large letter and digit characters with embedded visual words (Fig. 6). Two of the words were designated as targets and the rest were nontarget distractors. When the large character was a letter, R.F.S. could perceive the embedded words normally, and therefore could easily distinguish target from nontarget words. Accordingly, we expected to observe a P3b effect in this condition. When the words were embedded in digits, R.F.S. was unable to consciously distinguish targets from nontargets. He reported on repeated questioning that he could not see any sign of embedded stimuli, much less identify these stimuli as particular words. Based on previous literature, we would not expect a P3b effect in the absence of awareness for the target/nontarget status of a stimulus word.

Fig. 6.

Neurophysiological evidence of task-dependent target detection despite lack of awareness: P3b component. We found robust P3b differences on the P3b component when target words were compared to nontargets, embedded in both letters and digits. Asterisks reflect significance of a permutation test comparing the two conditions (****P < .001). (A) Sample stimuli and response at Cz evoked by targets (red) vs. nontargets (blue) embedded in a letter. (B) Same as A, embedded in a digit. (C) Difference waves at Cz for targets – nontargets embedded in a digit (green) vs. a letter (magenta). Gray highlighting indicates time window of interest (550 to 600 ms). (D) Scalp topography indexing main effect of target presence/absence in the time window of interest.

In the letter condition, we found a robust P3b effect at Cz from 550 to 600 ms after word onset (target – nontarget amplitude = 1.30 μV; P = 0.0001), establishing the modulation of this component under circumstances in which R.F.S. was able to recognize the embedded target words. Next, we examined the same site and time window in the digit condition, where R.F.S. was unaware of the target/nontarget status (or even the presence) of the embedded words. Here again we found a robust P3b effect (target – nontarget amplitude = 1.21 μV; P = 0.0002). A direct comparison of P3b magnitude in the letter and digit conditions did not reveal a difference [letter (target – nontarget amplitude) – digit(target – nontarget amplitude) = −0.081 μV, P = 0.99]. Finally, comparing the response to the target word in the digit and letter conditions revealed no difference (target: letter – digit amplitude = 0.03 μV; P = 0.99).‡

Discussion

We report the case of R.F.S., an individual with selective metamorphopsia that grossly disrupts perception of the digits 2 to 9 and stimuli presented in close proximity to these digits. Assessing the neural processing occurring in this profound deficit of visual awareness, we found responses demonstrating high-level neural processing despite the lack of awareness: The N170 and P3b ERP components elicited by stimulus contrasts R.F.S. was unaware of were robust and indistinguishable from those elicited by the same contrasts under full awareness. How can this surprising pattern of results be interpreted? We first discuss our N170 and P3b results in light of current understanding of these ERP components. We then offer a hypothesis about the nature of the dysfunction in R.F.S.’s digit processing, and how this dysfunction leads to impaired awareness for digits and nearby stimuli. Finally, we consider the implications of our interpretation for theories of visual awareness.

A Normal N170 Is Typically Found Only When Participants Are Aware of the Face/Nonface Contrast.

In a recent review, Rossion (61) argued that the N170 component specifically reflects interpretation of a stimulus as a face: Across the literature, regardless of the method of rendering face stimuli invisible (e.g., sandwich masking, continuous flash suppression, object substitution masking, inattentional blindness; all limited to nonemotional faces) the N170 component is either eliminated, attenuated, or even reversed when participants are not aware of the face stimuli (62–68). One possible exception to this pattern is a study by Vuilleumier et al. (69) that examined the N170 component in an individual with visuospatial neglect who was typically only able to report faces when they were presented in his nonneglected hemifield. Vuilleumier et al. found that reported and unreported faces elicited standard and indistinguishable N170 components relative to nonface shapes. Nevertheless, two differences were found between reported and neglected stimuli: A diminished P1 and a diminished P170 (N170 companion component along the midline) for neglected faces. R.F.S., however, showed typical and indistinguishable responses to faces and nonfaces in digits across the N1, P1, and N170/P170 components. Combined with the result of the study by Vuilleumier et al., we propose that under certain situations, the N170 does not entail awareness of the presence of a face.

The P3b as a Marker of Conscious Awareness.

The P300 component is widely thought to index processing of contextual information and engagement with task demands, and two subcomponents can be distinguished: The P3a and the P3b (54). The P3a is evoked in oddball paradigms when an infrequent or unexpected stimulus appears, even in a passive task without responding required. The P3b, a later and more sustained component, reflects processing the presence of a task-relevant target. The P3b is thought to be a marker of either conscious awareness (3, 56–58, 70) or short-term memory buffering (47, 69, 71). Despite this debate, both sides agree that awareness of the target is necessary for the P3b to be elicited (72). One notable exception comes from Silverstein et al. (59), who demonstrated a P3b to subliminally presented words in a relatively passive oddball paradigm. However, there was no comparison with a P3b response to visible words, leaving open the possibility that the response was attenuated relative to a standard P3b (56, 73).

Critically, in our experiment the P3b effect was present in the digit condition despite R.F.S.’s inability to consciously detect or distinguish the embedded target and nontarget stimuli, and the effect did not differ in magnitude from the P3b elicited under conditions of full awareness. These results provide a counterexample to the necessity of awareness of a stimulus for that stimulus to elicit a P3b response. We suggest that the complex cognitive operations underlying target-word detection and integration of incoming information into representations of task context (and hence the P3b neural signature) can occur in the absence of visual awareness.

One might argue that we have not correctly isolated the P3b component; we believe this to be unlikely for several reasons. First, our target-matching task requires the cognitive processes underlying the P3b as opposed to the P3a: Specifically, the matching of incoming information to task-set representations maintained in working memory (74). Second, in tasks that are designed to elicit both the P3a and P3b, the P3a occurs earlier in time and is relatively shorter in duration while the P3b is later and sustained. Despite the name, the P3b component is not typically elicited at precisely 300 ms after stimulus onset. For R.F.S., the target – nontarget ERP was late and sustained, beginning around 550 ms and extending well beyond 700 ms. The relatively late timing of R.F.S.’s P3b is consistent with delayed P3b responses in cortical-basal degeneration (75). Furthermore, this timecourse is consistent with the interpretation of the P3b as an initial ignition and then sustained reverberation of information within a global neuronal workspace (70, 72).

Reconciling the Paradox: Disruption of Awareness yet Normal Neural Responses.

To our knowledge, our study is unique in reporting a complete lack of awareness for the identity of stimuli that nevertheless elicit a pattern of neural processing typically found with awareness. This finding provides a strong constraint on our understanding of the neural correlates of visual awareness by suggesting that some situations exist in which typical neural processing—and the underlying cognitive processes necessary to generate such neural signals—occurs but awareness and reportability do not ensue.

The ERP components we examined appear by all measures normal, even though R.F.S. was unable to distinguish the embedded face/nonface and target/nontarget word stimuli. How can R.F.S.’s behavioral and neural results be reconciled? In answering this question, we also address a more fundamental paradox: How can R.F.S. have a category-specific deficit (elicited only by digits) with low-level visual consequences (affecting conscious perception of the digit and nearby visual stimuli)? This discussion requires considering what types of processing are sufficient for visual awareness of a stimulus.

What Are We Typically Aware of?

Many theories of visual awareness posit integrative processes. This assumption agrees with intuitions about awareness: We see the world as a collection of interpreted objects (e.g., “8” as “a black eight”) rather than a collection of raw sensory features (e.g., curved edges and black). The integrative aspect of awareness could arise in different ways: Via feedback/recurrency (4, 76–78) or via feedforward signals to a second-order representation (2, 79). Regardless, the effect is the same: We are aware of bound units, not of unbound pieces of information (a sense of the color black, the identity eight), and these units are typically tied to specific spatial locations. By way of cashing out this process, we posit that integrated percepts are normally a combination of visual information (e.g., form, color, texture) and identity information, grouped together at a given position in space. [This proposal bears some resemblance to feature integration theory and the idea of “object files” (80); see discussion of how object files might relate to consciousness in ref. 81.] Under typical circumstances, all aspects are tied together: The identity of a symbol or word is linked to its location and its physical properties like color and size. It is by virtue of this integration of visual, identity, and position information that one becomes aware of visual stimuli.

What Is R.F.S. Aware of?

This proposal about the integrated contents of visual awareness suggests an interpretation for R.F.S.’s deficit. We hypothesize that for R.F.S., the integration process produces an uninterpretable percept. We assume that his processing of visual shape information is sufficiently intact to distinguish between digits and other similar visual forms, and further that stored digit representations are correctly accessed from the computed shape information. In other words, we assume that the (nonconscious) process of digit identification occurs normally; otherwise, we have no way of explaining how the deficit could be selective to digits. We propose that R.F.S.’s deficit affects the signals sent from digit-identification processes for integration with visual information (via feedback connections to lower levels or feedforward connections to higher levels). Specifically, we assume that the to-be-integrated signals are abnormal in some way, and consequently cannot be properly integrated with the visual information about the digit or other stimuli at the same spatial location. Consequently, R.F.S. becomes aware of an unrecognizable (distorted) percept at the location of the digit. On this account, the stored representation of the correct digit is activated but the outcome of this process is not available to awareness.§ (We note that this is consistent with R.F.S.’s verbal descriptions of a digit as unrecognizable, as can be heard in Movie S2 when holding a large foam eight: “too strange for words … [what I’m seeing] doesn’t have a shape.”) In the case of an embedded stimulus, the N170 and P3b results suggest that the embedded visual information also undergoes identification processes normally (e.g., faces and words are nonconsciously identified), but the abnormal signals from the digit identification process disrupts the integration of visual and identity information at the spatial location of the stimuli. In the P3b experiment, processes beyond unconscious word recognition must also occur without awareness: The embedded visual information is processed, the word is identified (“tuba”) and linked to information about the task goals (“tuba” is a target). However, this information cannot be integrated with the disrupted output of digit identification and the resulting awareness contains no usable information from this location, disrupting any knowledge about the identity of the digit, the word, or target presence.

We have no specific claims to offer about the normal content of the signals from identification processes that enter into integration processes, how these signals are disrupted, or why the disrupted signals lead to the specific form of disturbed awareness experienced by R.F.S. (“spaghetti”). Nevertheless, we suggest that our assumptions offer a plausible interpretation for R.F.S.’s impaired performance and have significant implications for understanding visual awareness.

Integrative Theories of Visual Awareness.

Three theories of visual awareness are particularly relevant to our findings: Recurrency/feedback theories, higher-order theories, and the global neuronal workspace hypothesis. Recurrency theories posit that awareness of a visual stimulus depends upon recurrent processing between lower- and higher-level visual areas (4, 76–78). By hypothesis, visual awareness of a particular stimulus arises only when there is consistency between the content of higher- and lower-level representations, integrating features and categories into a coherent percept. A failure of coherence is assumed to result in a disruption of awareness. Within this framework, we could interpret R.F.S.’s performance by assuming that feedforward signals required for (nonconscious) identification of digits are normal, but corrupted feedback signals are sent from category-specific digit identification processes to earlier levels of processing. On this account, the corrupted feedback results in a failure to integrate the higher- and lower-level information and distorted perceptual awareness of both the digit itself and stimuli presented roughly at the same location (on or near the digit). One implication of this interpretation is that N170 and P3b components arise from largely feedforward processing. Multiple authors have suggested that face categorization and the N170 can arise purely on a feedforward processing sweep (77, 82). In contrast, the P3b has been argued to involve recurrent or integrated processing (56–58). If feedback processing is in fact required for the P3b to arise, one would have to assume that feedback is occurring in the absence of awareness. Given this assumption, however, our data would be difficult to reconcile with recurrency theories of visual awareness.

Higher-order theories of consciousness stipulate that awareness arises from second-order representations of oneself in a particular perceptual (or cognitive) state (2, 79). For a stimulus to be perceived normally, second-order representations must integrate low-level visual properties with a higher-level interpretation of a stimulus. As with recurrent theories, a failure of integration will disrupt awareness. Within this context, we could interpret R.F.S.’s performance by again positing corrupted signals emerging from digit-identification processes. Processes aimed at generating second-order representations could not successfully integrate the abnormal signals from the identification level with the (normal) signals from lower perceptual levels, leading (as in the feedback interpretation) to impaired awareness of digits and nearby stimuli. On this account, the stages of cognitive processing indexed by the N170 and P3b must be assumed not to depend on second-order representations, allowing normal neural signatures of the embedded stimuli to be observed despite a disruption of visual awareness.

Finally, we consider whether global neuronal workspace theories could account for our findings. These theories espouse a strong parallel between consciousness and cognitive processing; the assumption of such a close relationship is not present in other theories of visual awareness (79, 83). But like the other two theories, global neuronal workspace theories also rely on integration: Certain types of complex cognitive processes—those that are explicitly rehearsed, goal-driven, or context-sensitive—are believed to occur in an integrated domain-general global neural workspace, and by virtue of this fact are conscious (3, 84, 85). Global workspace theories posit that information from any sensory system, given the proper task demands, can integrate with other sensory and memory information and responses can be generated using multiple modalities. Under some circumstances, the P3b effect can be a marker of “ignition” of the global neuronal workspace, a neural marker of awareness (3, 58, 70). For R.F.S., one might imagine that the digits and embedded figures are not processed by the global neuronal workspace, thus not reaching his visual awareness. The N170 effect might reflect a low-level face detection signal outside the workspace, consistent with this theory. However, our P3b results provide compelling evidence that R.F.S. is computing complex, task-sensitive representations in the absence of awareness. We found that the P3b ERP signature did not change as a function of awareness. To square these results with global neuronal workspace theory, one would have to argue that the cognitive processes of task-guided target matching (with word stimuli) can occur independently of global workspace ignition (as measured by the P3b).

Theorizing about consciousness is notoriously difficult, and the foregoing theories are largely general approaches to the study of visual awareness. As with any result, our findings do not eliminate any of these broad theories from the realm of possibility, but they do place important constraints on how recognition and associated neural processing relate to visual awareness. We hope that future revision of these theories will include our striking findings with patient R.F.S. as one key data point toward understanding visual awareness.

Conclusion.

We have reported a case of an individual with a deficit of visual awareness—digit metamorphopsia—that also prevents recognition of visual stimuli in the same location. We used this phenomenon to address the neural correlates of visual awareness and found robust neural responsiveness to faces (N170) and target words (P3b), even when they could not be consciously distinguished from nonface and nontarget stimuli. These ERP components can thus be neither necessary for consciousness nor sufficient to indicate consciousness. Unconscious identification of complex forms (faces, words) can occur successfully without the output of the identification process becoming available to visual awareness. Our results are consistent with awareness depending on integration across levels of representation, but require careful reconsideration of how neural responses relate to stimulus reportability.

Methods

Participants.

Patient R.F.S. was 60 y old at the start of data collection in 2011. He suffered an acute neurological event in October 2010, which resulted in headache, transient aphasia and amnesia, and temporary loss of vision. MRI conducted at this time was normal but EEG showed bitemporal sharp waves and a cerebral angiogram revealed diffuse enlargement of intracranial internal carotid arteries (left cavernous segment diameter 5 to 6.5 mm). In May 2011, a diagnosis was made of progressive neurological disease (corticobasal degeneration) in the parietal lobes and subcortical structures. R.F.S. was referred to our laboratory in June 2011 after a neuropsychological evaluation indicated difficulty processing digits.

The procedures reported in this paper were approved by the Johns Hopkins Homewood Institutional Review Board and the Johns Hopkins Medicine Institutional Review Board. R.F.S. provided written informed consent prior to his participation.

Experimental Procedures.

Exp. 1: Embedded stimuli verbal report experiment.

The large characters were presented in 600-pt Calibri font, in a light gray color (RGB: 192 192 192) with a black outline on a white background, for unlimited duration (until response). The 30 embedded line drawings were selected from the Snodgrass and Vanderwort set, and included 7 animals, 7 foods, 5 body parts, and 11 inanimate objects (86). Each line drawing was presented twice, once inside a letter and once inside a digit, across 60 trials. The trials were split into three blocks of 20 trials (10 digit, 10 letter) and order was randomized. R.F.S.’s task in block 1 was to name the object and the letter or digit; in blocks 2 and 3 he named the object only.

Exp. 2: Embedded stimuli two-choice perceptual discrimination experiment.

The stimuli in this experiment were large characters (8 or H) with embedded stimuli; R.F.S. made a decision between two possible responses. Across five tasks he was required to discriminate different stimuli: diamond/square (◇/□), times/plus (×/+), up/down arrow (↑/↓), instrument/vehicle line drawing, or word category (animal/object). For the tasks with embedded shapes, the task on each trial was to indicate which of two shapes had appeared. For the line drawing and words tasks, stimuli were exemplars from two categories, and the task was to indicate which category was presented. Responses were recorded by button press. Trials with no response were excluded from the analysis.

Exp. 2a (times/plus).

There were 48 digit trials and 48 letter trials; of these half contained an “×” and half a “+.” The two symbols were the same shape differing by rotation of 90°, and referred to as “times sign” and “plus sign.”

Exp. 2b (diamond/square).

There were 96 digit trials and 96 letter trials; half of these contained the outline of a diamond and half a square, of the same dimensions (differing by rotation of 90°).

Exp. 2c (up/down arrow).

There were 48 digit trials and 48 letter trials; half contained an up-facing arrow and half contained a down-facing arrow.

Exp. 2d (animal/thing word).

There were 48 digit trials and 48 letter trials; half of each contained a word referring to an animal and the other half referring to an inanimate object. The same words were used in the 8 and H conditions and no words contained letters that R.F.S. has difficulty recognizing.

Exp. 2e (vehicle/instrument drawing).

There were 48 digit trials and 48 letter trials. Half of each contained a Snodgrass line drawing of a vehicle and the other half contained a line drawing of a musical instrument.

Exp. 3: Eliciting the N170 component.

Stimuli in this experiment were images of faces (one male, one female) and scrambled faces superimposed within the central stroke of a large letter or digit character. The faces were scrambled by randomizing the position of pixels within each face image, ensuring that the average pixel intensity of the scrambled stimuli was matched to the face stimuli. The large characters were 0, 1, 5, 6, 7, 8, G, H, (all in Calibri font with the exception of the zero in Consolas). The face/scrambled face image was aligned within the central stroke of the character (Fig. 5). R.F.S. was seated ∼65 cm from the screen. The large characters subtended a visual angle of ∼10.5 × 5.6 to 8.0° and the embedded face/scrambled face ∼2.5 × 2.0°. To maintain R.F.S.’s attention to the stimuli, infrequent “go” trials consisted of a “#” or “?” symbol of the same size as the other characters, and he was instructed to press a button when one of these symbols appeared. These trials were discarded from analysis.

The trial sequence was as follows: 1,500-ms fixation cross, 1,000-ms character presentation, face/scrambled face image superimposed on character for 300 ms, 1,500-ms blank screen. Each block contained four repetitions of the following trials in a random order: Two trials of each character with embedded face (2 × 8), two trials of each character with embedded scrambled face (2 × 8), and two response trials, for a total of 136 trials per block. Eight blocks were presented for a total of 1,088 trials. We used E-Prime 2.0 software to control stimulus presentation and response collection (87).

We examined responses at electrode sites P9/P10 in the time window from 170 to 200 ms after onset of the face/scrambled face embedded in the large character, selected based on a review of the literature (88).

Exp. 4: Eliciting the P3b component.

Visual stimuli were words superimposed on a large letter or digit character (Fig. 6). All words were four letters in length and did not contain any letters which R.F.S. has difficulty perceiving (M, N, R, S, Z). The large characters were 3, 5, E, and P, in Calibri font, as well as the characters representing 5 and 7 in the surrogate digit system developed for R.F.S. (SI Appendix, Fig. S1). As in Exp. 3, R.F.S. was seated ∼65 cm from the screen. The large characters subtended ∼11.0 × 8.3° of visual angle and the embedded words 1.6 × 4.1°.

R.F.S. was told that the large characters would include some letters, some of his surrogate digits (which he could perceive normally), and some Arabic digits. He was instructed to press a button when he saw a word referring to a musical instrument (“TUBA” or “FIFE”) overlaid on a digit, and to withhold responses on all other trials. As a consequence of this instruction, R.F.S. responded to target words in the surrogate-digit condition but not in the Arabic digit condition (where he was unaware of the stimulus words) or in the letter condition (where he withheld responding because the large character was not a digit). The conditions we compared—the Arabic digit condition and the letter condition—were therefore matched in having no responses.

The trial sequence was as follows: First a 1,500-ms fixation period, then the character briefly presented alone (10 ms), followed by the character with the embedded visual word (600 ms), followed by auditory word presentation (900 ms). Each block contained 110 trials with each type of large character (letter, surrogate digit, Arabic digit), for a total of 330 trials per block. For each large-character type, the embedded word was a target on 22 of the 110 trials, and a nontarget on the remaining 88 trials. Six blocks were presented, for a total of 1,980 trials in the experiment. We used E-Prime 2.0 software-controlled stimulus presentation and response collection (87).

For the P3b, we examined responses at Cz in the time window from 550 to 600 ms after onset of the visual word (ending prior to onset of the auditory word at 600 ms).

EEG data collection and preprocessing.

For both the N170 and P3b experiments, EEG was recorded at 250 Hz using a Geodesics 256-channel sensor net and Net Station v4.3. Impedances were kept under 50 kΩ where possible. The data were filtered using a 0.1- to 30-Hz bandpass filter, referenced online to Cz electrode and then rereferenced to the average reference. Prior to offline averaging, all single-trial waveforms were automatically screened for amplifier blocking, muscle artifacts, horizontal eye movements, and blinks over epochs of 1,000 ms, starting 200 ms before the onset of the critical stimuli. For each condition, no more than 6 trials (of 416) were removed during this screening procedure. Average ERPs were computed over artifact-free trials for critical stimuli in each condition. The mean amplitude of the ERPs over selected electrode sites in the relevant time windows (relative to a 100-ms prestimulus baseline) was submitted to statistical analysis.

Statistical analysis.

Preprocessed ERP data consisted of cleaned and baseline-corrected EEG segments corresponding to each trial. Each comparison was assessed for statistical significance using a permutation analysis (89). For each timepoint in the time window, a one-way ANOVA was used to assess the difference between signals (across trials) from the different conditions. The set of F-values from each time point within the time window was averaged to form a single mean F-test statistic. When the comparison involved two difference waves, the F-value corresponding to the interaction term from a two-way ANOVA was used to generate the test and pseudo mean F-values. A permutation test was used to compute P values that corrected for analytical flexibility in selecting the time window. Specifically, a set of 10,000 pseudo F-values were computed by: 1) Randomly permuting the condition labels (without replacement) to each trial, 2) computing a set of mean F-values by sliding the time window within a range of feasible time points (N170: 150 to 200 ms; P300: 250 ms to 600 ms), and 3) selecting the largest mean F-value within the set of mean F-values from the moving window and placing it in the pseudo F-distribution. Statistical significance was quantified as the rank of the actual mean F-test statistic within the pseudo F-distribution with the P value consisting as the number of pseudo F-values greater than that F-value, divided by the 10,000 values in the distribution. This analysis was run in a custom MATLAB script (v9.6.0.1135713), making use of tools from EEGLab (90). The preprocessed EEG data and custom MATLAB analysis scripts generated during this study are available through the Open Science Framework at https://osf.io/72JHA (91).