Challenges for theories of consciousness: seeing or knowing, the missing ingredient and how to deal with panpsychism

Abstract

Significant progress has been made in the study of consciousness. Promising theories have been developed and a wealth of experimental data has been generated, both guiding us towards a better understanding of this complex phenomenon. However, new challenges have surfaced. Is visual consciousness about the seeing or the knowing that you see? Controversy about whether the conscious experience is better explained by theories that focus on phenomenal (P-consciousness) or cognitive aspects (A-consciousness) remains, and the debate seems to reach a stalemate. Can we ever resolve this? A further challenge is that many theories of consciousness seem to endorse high degrees of panpsychism—the notion that all beings or even lifeless objects have conscious experience. Should we accept this, or does it imply that these theories require further ingredients that would put a lower bound on beings or devices that have conscious experience? If so, what could these ‘missing ingredients’ be? These challenges are discussed, and potential solutions are offered.

This article is part of the theme issue ‘Perceptual consciousness and cognitive access’.

1. Introduction

Since the 1990s, behavioural science, neuroscience, biology, computational science and philosophy have joined forces to unravel one of the greatest mysteries, that of consciousness. How is it possible that with 1.5 kg of mainly fat and protein enclosed in a bony shell we have conscious experiences? Considerable progress has been made since Koch et al. [1], but we are clearly not there yet. Several promising theories and ideas have emerged, which have been backed up with lots of experimental data. This has led to consensus on some aspects of how, when and where brains do or do not produce consciousness. But strong controversy remains, in particular on what aspects of consciousness are the explanandum. Does the real mystery of consciousness lie in the fact that we experience the world that surrounds us, or in the ability to reflect on it and cognitively manipulate what we perceive; is consciousness about seeing or about knowing what we see? In the first part of this paper I will review the current state of that debate, starting with a brief overview of what we know about visual processing, and then delving into the question about where, when and how the transition from unconscious to conscious visual processing occurs.

In the second part, I will address a more recent issue about how theories of consciousness deal with the problem of panpsychism: the notion that all animals, or even all living beings and possibly even non-living items possess some form of consciousness. Major theories of consciousness can be argued to endorse high degrees of panpsychism. Does that falsify these theories? Should we accept panpsychism? Or is there a third way out? I will provide arguments for the latter, by showing that human and fly consciousness both exist, while at the same time are so different that swatting a fly should not worry you too much. It does appear, however, that many theories of consciousness are too ‘simple’. They seem to be missing a key ingredient, and more neurobiological avenues may point us in the right direction.

Note that the paper will focus almost entirely on visual consciousness. That is not because other sensory modalities of more executive faculties have no relevance. It is just a consequence of my ignorance on these other matters.

2. The intelligent reflex arc

There is little controversy on how visual processing proceeds. With every saccade—and we make about three per second—the eye lands on a new scene and the image is processed by the retina. In about 50 ms, this information has reached primary visual cortex, and from there on is distributed along a large number of other visual areas. Within 100–200 ms (depending on species and brain size) the whole brain is updated about the new image in front of us. During this rapid feedforward sweep, many features have been extracted from the image [2]. Neurons have detected shapes, colours, motion or depth. Higher level cells have signalled the presence of faces or animals [3] or other complex constellations that we are strongly familiar with (e.g. letters and words in humans [4]). We even have a rough idea about what the image is about: scene gist is extracted within that time-frame as well [5]. You might say that almost everything there is to know about the image has been signalled by the powerful machinery of visual cortex and its many low- and high-level feature selective cells (figure 1).

Figure 1.

Three successive stages of visual processing after stimulus presentation. The feedforward sweep proceeds within 150–200 ms, during which low- and high-level features are extracted and translated into a (potential) motor output. This stage is unconscious. Recurrent processing starts within 100 ms after stimulus presentation, at first between low-level visual areas, and then more widespread between visual cortex. Neural correlates such as V1 contextual modulation, PPI and VAN (see text) are shown. These recurrent interactions enable phenomenal (P-) consciousness of the visual stimulus: you SEE. Eventually, recurrent interactions spread through the whole brain, causing ‘global ignition’. At the neural level this is expressed in P300 responses, and the involvement of fronto-parietal areas. The result is access (A-) consciousness, the ability to cognitively manipulate the stimulus, your reaction to it, its consequences, etc.: you KNOW. (Online version in colour.)

It does not end there. The feedforward sweep continues to proceed, now feeding into motor and frontal cortices. Here, visual information is transformed into execution—what can you do with what you see? If a ball is looming towards you, you instinctively dodge or catch it. The feats of such feedforward sensorimotor transfer are often quite remarkable. The eyes can saccade towards a visual stimulus within 120 ms [6], manual reaction times can be as short as 180 ms. Tennis players can return a ball serviced at 130 mph (world record is 163 mph), implying a reaction time of 400 ms, which includes deciding between forehand or backhand, doing a backswing and hitting the right way. You may have noticed yourself miraculously catching a glass of red wine that unexpectedly fell from a table, directly on its way to cause a horrible stain on the white carpet beneath. Before you knew what happened, the glass was in your hands, saving the day.

The feedforward sweep is the brain at its best. It shows how it can function as a reflex arc, not very different from the simple sensorimotor transfer that occurs in the knee-jerk reflex. Infinitely more complex and intelligent, of course, but a reflex arc nonetheless, where sensory input is as swiftly as possible translated into motor output. Note that this even involves some forms of decision-making: which of the many objects in a natural scene do we react to? How do we respond? In such reflex-like responses this strongly depends on the saliency or innate value of the external stimuli [6], but also internal variables (e.g. danger [7]) are taken into account. The feedforward sweep, therefore, is not a rigid process. What pathways are followed, and how deep each of these is penetrated depends on temporal and focused attention, task sets, expectation, brain state and many other factors [8–11].

3. Where does the magic happen?

From a behavioural or neuroscience point of view, this is all fairly straightforward. Of course some mysteries remain, among which the most prominent are how the brain detects objects for which there are no dedicated neurons, how it is possible to extract scene gist as fast as we can (the ‘seeing the forest before the trees' problem) and how we select among the many motor plans that are activated by the multitude of stimuli in our surroundings. But when we take into consideration how we experience our brains doing this, an entirely different matter arises, that of consciousness. We are not mere automatons executing reflexes ‘in the dark’. We see the ball that we hit during a tennis match, we feel the urgency when we grasp that glass. Somewhere along the way, a conscious experience of what we see arises. Where, when and how does that ‘magic’ happen? And why?

The short answer to the when and how is: after the fact. Feedforward processing itself is unconscious. The arguments for that have been laid out in many past reviews [12–15], showing considerable consensus on the matter. In short, when visual processing is artificially restricted to the feedforward sweep, it appears that features are extracted and that potential motor responses are activated yet that people remain completely unconscious of these events or of the visual stimulus that has set them in motion. The most widely applied method for doing so has been backward masking—i.e. showing a visual stimulus very briefly and then have it followed by another stimulus, the mask. This typically renders the visual stimulus completely invisible [16]. However, information about it travels pretty much throughout the brain, activating visual [17], motor [18] and even prefrontal areas [19,20]. The feedforward sweep, as intelligent as it may be, apparently does not suffice to give you a conscious sensation of the image that ignited it. So what is generating the conscious sensation that we automatically have when something hits our eyes?

What typically follows feedforward processing is recurrent or re-entrant processing. Via horizontal and feedback connections, neurons that initially responded to very different parts of the scene, or that had extracted different types of lower- or higher-level features, start to influence each other's activity patterns (figure 1) [2,21]. Neurons in the primary visual cortex (V1) typically have small receptive fields that are selective for particular features—say the orientation of line segments within a scene. Because of their small receptive fields, the neurons will respond identically, regardless of these line segments belonging to either a background or a figure. That is, up to about 100 ms. After that, the neurons' responses start to diverge for figure and background, as if they suddenly start to care about the larger perceptual context of the line segments they are responding to [22]. Quite strikingly for neurons in as ‘low’ a visual area as V1, this ‘contextual modulation’ of neural responses typically follows the perceptual interpretation that subjects have of the scene [23–25]. That is possible because it is mediated by horizontal connections and feedback connections from higher level areas [26], as lesions to these areas abolish the modulation [21,27]. Modelling has shown how neurons first do their independent job of detecting features and objects in a hierarchical cascade of feedforward processing, and then start to influence each other's activity via intricate inhibitory and excitatory interactions mediated by horizontal and feedback connections [28]. What we observe at the single unit level as contextual modulation can be observed in humans using scalp-recorded event-related potential (ERP) potentials in response to similar stimuli [29,30]. Latency is typically larger than in monkeys, more in the order of 200 ms, but the characteristics are very much the same.

Recurrent processing does not stop in visual cortex. The more activity advances towards frontal and motor regions, the larger the network of recurrent interactions becomes. The inclusion of motor-related activity (e.g. task relevance) may cause the visual activity to modulate accordingly [31]. Fronto-parietal activity may induce attention-related modulation of visual responses [32], which can reach back all the way to V1 [33]. Evaluation by medial prefrontal cortex will cause emotional valence to influence sensory activity [34], and so on. Thus, the network of recurrent interactions grows in size and complexity, making responses of neurons in this network more and more interdependent (figure 1). With that, the neural signals also come at longer latencies than those of purely visual recurrent signals. Attentional modulation of V1 neurons may start at 150 ms or more in monkeys [33]. In humans, using electroencephalography (EEG), we are dealing with latencies of up to 300–400 ms [15], for example, as expressed in P300 responses [35].

It is important to not let these interactions get out of hand; if everything is functionally connected to everything, all neurons will eventually do the same—which is known in pathology as epilepsy. Several mechanisms are in place for that. Recurrent modulation is selective: only when neurons are activated by the feedforward sweep can their activity be modulated by feedback [36]. This may be mediated by feedback connections primarily targeting NMDA receptors [37], whose ion channels only open when the membrane potential is already depolarized. In addition, feedback connections may specifically target inhibitory interneurons, given that they release excitatory neurotransmitters (glutamate) [38] yet primarily exert inhibitory effects on neural activity in distant regions [39].

Ample evidence emphasizes the importance of these recurrent interactions in conscious experience. They are abolished in anaesthesia, while feedforward processing is not [40,41]. Masking likewise selectively disrupts recurrent processing [17,42,43]. Dichoptic masking can render visual stimuli invisible for prolonged periods of time, which does not affect selective processing of faces in the fusiform cortex (FFA) yet disrupts recurrent interactions between the FFA and early visual cortex, as shown with psychophysiological interaction (PPI) analysis of fMRI [44]. Transcranial magnetic stimulation (TMS) can target visual areas with high temporal precision, which has shown that conscious percepts typically depend on early activity in high-level areas and on later activity in lower level ones, indicating that information has to be sent back along the visual hierarchy to become conscious [45]. Various manipulations of conscious perception consistently show to influence EEG activity in the 200 ms latency range—known as the visual awareness negativity (VAN)—which is the typical latency of recurrent visual processing [46]. In monkeys trained to report the conscious perception of texture defined figures, contextual modulation of V1 neurons was selectively absent, while feedforward signals were unaffected [47]. In sum, recurrent interactions between visual neurons expressed as contextual modulation at the single neuron level or as 200 ms latency EEG signals (VAN) or PPI in humans, is crucial for the emergence of conscious experience.

Also, the disruption of longer latency recurrent interactions causes a failure to report conscious percepts. For example, when people fail to attend to stimuli, they will report not having seen them, as is the case in conditions like change blindness, inattentional blindness or the attentional blink. These manipulations typically leave early recurrent signals relatively intact, while selectively disrupting later recurrent signals (e.g. P300 or N400 EEG responses [48] [49], but see [50]).

While the importance of recurrent processing for visual consciousness is widely accepted, the main controversy lies in the question of what extent of recurrent processing is sufficient for conscious experience to arise. The two main positions and their respective arguments (briefly) are

• 1. Only when recurrent processing includes the fronto-parietal network do we experience the visual input. At the basis of this line of reasoning is that for sensory information to become conscious it must become available to what is called the ‘global (neuronal) workspace’ (global ignition in figure 1) [15,51,52]. This global availability enables the cognitive manipulation of that information (e.g. storage in working memory), conscious access to the information, and eventually also the ability to report about it [53]. Hence, when subjects report not to have seen visual (or other modality) stimuli—as happens in cases like change blindness, attentional blink, neglect, and so on—this is taken at face value: it is then considered unconscious [54]. The neural correlates of such global availability are the involvement of fronto-parietal activation, and the later onset (P300) recurrent interactions [15]. Closely linked are theories that consider consciousness a higher-order thought (HOT) [55], implying that a higher-order representation of the visual experience must be present before it becomes conscious. A specific version of this idea is that consciousness requires (or is somehow linked to) metacognition, which can be briefly summarized as the ‘knowing that you see’ or ‘the knowing that you know’ [56,57].

• 2. Recurrent processing between visual areas suffices for conscious visual experience to arise. The hallmark of conscious vision, it is argued, is the integration of visual features into a coherent single scene. Once this is achieved, all the necessary and sufficient requirements for conscious vision have been fulfilled [14]. Unconscious vision, on the other hand, is characterized by unbound visual features, detected by neurons in distributed areas [58]. The boundary between unconscious and conscious vision is, therefore, put at the transition between feedforward and recurrent processing (figure 1b) [59]. Adding more widespread recurrent interactions—including fronto-parietal activation—may give you the ability to cognitively manipulate and report the conscious visual percept, but this is not explaining the unconscious/conscious transition itself [60]. Conditions like change blindness, neglect, etc. are considered failures of attention and report, not of conscious vision [14,61]. Experimental support comes primarily from findings showing that report, cognitive access or attention do not change the quality of (neural) visual representations [62]; they just add report, access and attention [63].

Further arguments in favour of the one or the other have been laid out extensively elsewhere [14,15,53,60]. The discussion is reminiscent of some older philosophical discussions, for example about the distinction between A (access) and P (phenomenal) consciousness [64–66], or that of easy versus hard problems of consciousness [67]. The controversy between the two ideas seems to be in a stalemate, with support for one or the other swinging back and forth [54,60,61,68]. There is sufficient data for either point of view, it mainly boils down to the two theories having clearly different explananda: are you interested in explaining the seeing or the knowing that we see? Crucial is whether we should consider consciousness as closely intertwined with attention and cognition (as in the first view) or as a function or phenomenon with its own ontological status (as in the second view) [14,58,69,70].

Not that both types of theories would not argue to explain the ‘seeing’. HOT, for example, argues that the higher-order representation of visual experience renders the experience conscious, hence truly ‘seen’, even when that higher-order representation has nothing to do with knowing or other higher levels of cognition. Similarly, in global workspace theory, it is argued that local recurrence leads to stimuli being ‘pre-conscious’ and that only after global ignition the ‘seeing’ arises.¹ Either way, however, according to these theories something beyond (recurrent) visual processing is necessary to explain the seeing. Similarly, neural correlates of these types of theories (when given) typically go beyond visual cortex and require the involvement of fronto-parietal or other non-visual regions. The second type of theories, however, confine all the necessary ingredients for seeing to the visual processes (or visual pathways) themselves. It may be fruitful to evaluate the two types of theories with respect to their explanatory power [13,14]. For example, does frontal involvement mechanistically explain something fundamental about the difference between conscious and unconscious vision, or rather about the difference between the ability to report or not [63]?

A more recent idea states that consciousness is all about the integration of information (information integration theory, IIT [72–74]). In this theory, it is taken as axiomatic that conscious experience is information that is integrated into a structured, united and intrinsic whole, at the exclusion of infinite other possible wholes. The properties of networks are linked to these axiomatic properties of consciousness in that representational systems, such as the brain, can be assigned a value (PHI) based on their connectivity, and to what extent it supports the integration of information. The higher this value, the more conscious the system is. Some aspects of the theory do quite well at explaining why highly coherent states (such as epilepsy) may yield unconsciousness instead of consciousness, or why some connectivity types (such as those of the cerebellum) do not support consciousness, while others (such as in the cortex) do. IIT would predict low PHI for strictly feedforward systems, and higher PHI for recurrent ones. Strictly visual recurrency would have lower PHI than global recurrency, but the theory is somewhat ambiguous about the extent of information integration (and hence recurrency) that is necessary for the conscious experience to arise [75].

4. Escaping panpsychism, or what it is like to be a fly

Current models of consciousness all suffer from the same problem: at their core, they are fairly simple, too simple maybe. The distinction between feedforward and recurrent processing already exists between two reciprocally connected neurons. Add a third and we can distinguish between ‘local’ and ‘global’ recurrent processing. From a functional perspective, processes like integration, feature binding, global access, attention, report, working memory, metacognition and many others can be modelled with a limited set of mechanisms (or lines of Matlab code). More importantly, it is getting increasingly clear that versions of these functions exist throughout the animal kingdom, and maybe even in plants.

Effects of anaesthesia are a good example of the problem we run into. At comparable doses of the volatile anaesthetics isoflurane or halothane, animals as different as worms, flies, goldfish, ducks, rats, horses, monkeys and man stop exhibiting responses to nociceptive or otherwise threatening stimuli [76]. Moreover, in all animals ‘higher’-level cognitive functions are more susceptible to anaesthesia than more primitive, reflex-like reactions. In the nematode, a creature with 302 neurons, the order of functions that ‘go’ under increasing levels of isoflurane anaesthesia are male mating, coordinated movement, chemotaxis and pharyngeal pumping until the worm stops moving at all [76]. Flies have larger brains, and neural activity shows similar spectra of local field potentials as can be found in mammals, with lower frequencies having more power than higher frequencies (roughly 1/f spectrum). In man and other mammals, it has been consistently shown that the higher frequencies (γ band) are indicative of feedforward or local processing, while lower frequencies (α and β bands) are linked to feedback or recurrent processing and top-down influences. Using Granger causality analysis of local field potentials, a similar distinction was found in the fly. Moreover, feedback and recurrent processing are more susceptible to anaesthesia than feedforward activation in mammals like man, monkey, rodents or ferrets. The same was recently found in the fly [77]. So in sum, both from functional and neural perspectives anaesthesia has the same effects in species as different as humans or flies. It may not even stop there. Plants such as the Venus flytrap stop responding to stimulation of their trigger hairs (normally excited by prey insects) when exposed to ether [78], a volatile anaesthetic also working in man.² The natural counterpart of anaesthesia—sleep—is equally ubiquitous in the animal and plant kingdoms. Animals that do not sleep have yet to be found [79], and many plants show clearly different metabolic activity patterns (and appearances) during day and night. We experience the difference between awake and asleep as one of the most fundamental transitions of consciousness, and the fact that conscious sensations are lost during sleep is possibly the number one reason why we believe there is consciousness. If this transition between awake and asleep is present in all living beings, does that imply that to these other beings the transition ‘feels’ equally dramatic? And does it then follow that these beings should, therefore, be granted consciousness—when awake? We run into similar problems when we assign specific functions as central to consciousness. Cognitive function like attention, access or metacognition, often linked to consciousness in ‘type 1’ theories, seem similarly widespread throughout the animal kingdom. Attentional processes such as increased reaction times in the presence of distractors, or the suppression of behavioural responses to non-attended stimuli are present in bees and flies [80]. Ants can navigate home in the absence of a sensory stimulus (such as the scent of the nest) guiding their way. They seem to do so via an internally represented ‘home vector’, that is based on an integration of its own past movements relative to the polarization plane of the sky. This way, the ant, after having found food, can run straight back to the nest, regardless of the often haphazard way it left it. Importantly, however, the ant also can invert its home vector to then run straight back from the nest to the location where the food was found. Similarly, bees have the well-known capacity to convert similar vectors into dances that are signalling other bees where food can be found. Crabs combine several home vectors for other purposes than just homing, for example, to defend their burrow [81]. At the core, such capacities fulfil all requirements for what we call ‘cognitive manipulation’ or ‘access’ in higher animals like man, given that access is generally defined as the making flexibly available of sensed or memorized information for different types of behaviour. In line with this, bees can abstract at least two different concepts from a set of complex pictures and combine them in a rule for subsequent choices.

Metacognition is generally studied using paradigms where animals can choose to opt-out if they consider the task too difficult. The idea is that in doing so the animal expresses knowledge of its own performance, or of the probability that his stimulus–response coupling is the right one [82]. Also, opting out has been used to distinguish blindsight from the conscious vision in monkeys [83]. Honeybees were presented with visual targets that either indicated a positive (sucrose) or negative (quinine) reinforcement. The more difficult the visual discrimination became, the more often the bees opted to not respond at all, thereby increasing their overall performance. Moreover, they easily transferred this strategy to another task [80]. Whether it would qualify for metacognition is debatable, but even plants (peas) can adopt different root growing strategies depending on the variability of nutrients in their soils, showing risk appetite and aversion not unlike animals do [84]. What would follow from such findings? If we take the functional perspective, and state that attention, access or metacognitive faculties are the hallmarks of consciousness, we would have to conclude that these simple animals have conscious sensations of their surroundings, home vectors and stimulus response-couplings. If not willing to accept that, one would have to resort to meta- or higher-order variants of these functions and deny consciousness to animals that just execute the ‘simple versions’. But what higher order variants would that be: ‘super-attention’, ‘super-access’ or ‘super-metacognition’ of some sort? We cannot argue that attention, access or metacognition are central to consciousness in humans, and then suddenly retreat if animals that we do not ‘believe’ to have consciousness execute those same functions. It is either that these functions are not constituting consciousness, or that the lower animals have consciousness too.

Functional and neural definitions of consciousness as currently formulated thus direct us towards a rather panpsychistic view,³ where all animals and possibly even plants would be conscious, or at least express the unconscious/conscious dichotomy. One may even start to worry over cultures of neurons in a Petri dish, or slices of cortex or hippocampus as often studied in electrophysiological experiments. Note that IIT, in particular, would probably assign higher PHI (and hence consciousness) to such preparations than to flies [74]. Another obvious extension would be consciousness in artificial intelligence (AI) systems. It has been argued that current AI generally lacks mechanisms for global access and availability of information or for metacognition (required for consciousness according to view 1 above), so that fears of conscious machines are premature [85]. Others, however, have opposed this view, arguing that many AI systems, in fact, do have these properties (either in rudimentary or more advanced forms). Moreover, if conscious experience is better explained by recurrent interactions or information integration (view 2 above), we may already have created conscious machines [86].

Do such considerations falsify current theories of consciousness? It seems entirely plausible that other animals than us have conscious sensations, fitting with the propensity of science to remove man from its pedestal, acknowledging that we are just animals too. On the other hand, it feels unsatisfactory to equate consciousness as we experience it to whatever happens in the mind of a fly. IIT as well as recurrent processing theories of consciousness offer some potential solutions to this problem. Both theories would argue that while the conscious/unconscious dichotomy should be present in many animals (or even devices), the ‘what it is like's’ may differ vastly between them. Of the senses, vision and olfaction are most important in the fly, and it has been shown that there is multisensory integration between the two. The visual system of the fly is dominated by neurons and pathways that detect motion (even second order and illusory motion), not surprising given its behaviour. Second are pathways related to colour vision. Much less prominent are systems that would enable the fly to discriminate between all sorts of objects and shapes [87]. So what ‘is it like’ to see like a fly? Obviously, conscious content can only be about information that the neural machinery detects. Recurrent processing theory (RPT) would argue that a fly sees an amalgam of motion and colour, quite ‘objectless’ in fact. Objectless vision is not unlike what patients with damage to visual object areas see, like the famous D.F., studied extensively by Goodale & Milner, who cannot identify objects and shapes, yet is seeing features like colour and motion [88]. Apperceptive agnosia patients have similar sensations of unbound basic features [89,90]. In blindsight, humans report to even have no conscious visual sensations at all (in the affected hemifield), apart from a vague sensation of objectless motion when things move fast enough [91,92]. So what the fly sees may be similar to the limited experiences of blindsight and other patients, who would qualify their vision as severely impaired or even ‘unconscious’. Add to that the obvious other differences between fly and human visual systems (resolution, numbers of neurons and areas, etc.), and it will be quite clear that we would probably qualify what the fly sees as not (or hardly) seeing at all.

IIT adds the prediction that conscious content is also determined by what is excluded by the current state of the system. For example, colour neurons in our brains contribute to our phenomenal experience even when silent, because they exclude the presence of colour, while patients with damage to colour regions do not even understand or ‘miss’ the notion of colour [72,75], much like we do not ‘miss’ perceiving the polarization of light. Because our brains support so much more concepts and sensory modalities than brains of flies, our visual experience would be richer than that of the fly even if the fly brain had the same visual machinery as we have.

This hints towards an important conclusion. What we consider highly impaired vision or even blindness, is what the fly may see consciously. Both species have an unconscious/conscious dichotomy. When a fly escapes from a swat it executes a very intelligent reflex arc, where it positions its middle legs to jump in a direction orthogonal to the approaching danger [93], not unlike what we do when a large object is looming. After the fact, when recurrent processing (or information integration) has kicked in, the fly may experience the fear associated with it (flies show fear conditioning [94]). But the experience will never transcend beyond ‘something nasty that moved’. In our minds, however, we will come to recognize that the looming object was a pick-up truck, running off the road, steered by a bearded man. We heard the engine roaring, felt the gush of wind when the car almost hit us. And in doing so we exclude millions of other possible percepts. So although our reflex may be similar to that of the fly, the sensory experience that follows is infinitely richer. In fact, what the fly experiences may be very similar to what we experience in a masking experiment,⁴ or when we choose ‘not seen’ on a perceptual awareness scale. What we call unconscious may be the richest experience a fly or worm ever experiences.

Should we then conclude that flies do not have consciousness after all? That is missing the point. The notion of conscious versus unconscious is still useful—or even necessary—to grasp what goes on in the mind of a fly. The unconscious to conscious transition occurs in flies just like it does in humans, in the sense that it is a radical—and probably nonlinear—phase transition, supported by the transition between feedforward to recurrent processing. Accordingly, the amount of integrated information (PHI) suddenly increases strongly, which is accompanied by a sudden transition from processing ‘in the dark’ to conscious phenomenology. Therefore, to the fly, being awake will ‘feel’ clearly different from being anesthetized. Hence, there should be a clear and rather dichotomous distinction between the two states from the perspective of the fly, just as there is this distinction in our phenomenology when we go from awake to asleep. But conscious experience in the fly is just not comparable in richness, quality and extent to ours.

Such considerations call for another distinction, that between unconscious and non-conscious, or death. No matter how deep the sleep or anaesthesia, there remain clear differences between brains that live and those that do not, both in terms of remaining functionality and neural processing. Anesthetized brains process information in a feedforward fashion, and potentially also exhibit limited recurrent information exchange; PHI will be low but not zero. Clinically, we know there are differences between brain death, coma, vegetative state or minimally conscious state. In sleep, we recognize different depths. In other words, there are gradations of unconscious. These gradations are all accompanied by differences in the extent of neural processing, for example, expressed in the spread of neural signals evoked by TMS of localized regions of cortex [96,97]. But when dead, all that stops; clearly another nonlinear transition in the level of consciousness. Does that imply that being asleep or anesthetized ‘feels’ different from being dead? Nobody knows, of course, but it is an interesting speculation. It is getting increasingly clear that sleep is not always as unconscious as we may think it is. Sensory processing continues, allowing a child sobbing in the distance to wake up a worried mother, or outside events to get integrated into dreams. People with neurophysiological signs of sleep may feel awake [98]. This idea of ‘multiple gradations’ of unconsciousness is also suggested by studies in awake subjects, where masked and unattended items may feel equally ‘unconscious’ (as in ‘not accessible by the self’), yet for entirely different reasons, and supporting taxonomically different types of unconsciousness (in that case unconscious and pre-conscious) [99].

Together, we arrive at a view where we can objectively define strong and nonlinear transitions in states of consciousness that most likely occur in many animals. Because of differences in cognitive machinery that these animals are endowed with (different sensory modalities and capabilities, emotion, social cognition, language, self, etc.), these different states of consciousness support highly different phenomenal experiences. These will feel so entirely different that what we ‘experience’ during deep sleep is similar in richness to what a fly experiences during wakefulness (figure 2). It would be human-chauvinist to deny the fly its state transitions, however, and claim that it would not feel different to the fly to be awake, anesthetized or dead. More importantly, however, it would deny consciousness its own ontological status, separate from the cognitive machinery it supports [14].

Figure 2.

Phenomenal content is conscious in the fly, yet comparable to unconscious content in humans. The figure depicts the richness or quality of phenomenal content (calculated as PHI in IIT) on the vertical axis. In both fly and man there is a nonlinear transition in this richness, marking the transition between unconscious to conscious processing, corresponding to the transition from feedforward to recurrent processing at the neural level. At the lower end, there is a second nonlinear transition, that between dead or alive. In both species, it therefore makes sense to talk about ‘conscious’ versus ‘unconscious’. However, the content of conscious processing in the fly is comparable to unconscious processing (e.g. deep sleep) in humans. (Online version in colour.)

5. The missing ingredient

Another, and highly related problem is equally present in theories of consciousness—that of the missing ingredient. Recurrent processing will not grind to a complete halt in anaesthesia. Although monkey experiments have shown the abolishment of interareal (e.g. between V1 and extrastriate areas) or maybe even local (horizontal) interactions [40], it is conceivable that at more local levels (between nearby neurons) recurrent interactions remain. In brain death, recurrent processing is probably completely absent, as even feedforward signals such as the N20 SSEP are absent. In coma, however, some recurrent signals may be present, expressed in burst suppression EEG, where the complete absence of coherent activity alternates with bursts of synchronous cortical activity. In vegetative state,⁵ connectivity is similar to that of deep sleep or anaesthesia, all showing coherent EEG signals [96]. If some levels of recurrent processing are compatible with unconsciousness, ‘something else’ beyond mere recurrency apparently is needed for consciousness.

IIT offers a solution by arguing that not all recurrency equally supports the integration of information. With too little connectivity, the information stays too ‘localized’, but when units are connected too strongly neurons lose their ability to convey independent information, also lowering PHI.⁶ Indeed, the high amplitude and widespread slow waves seen in sleep, anaesthesia or vegetative state EEG are signals of too strong coherency [97]. This may also be the reason why in monkey experiments anaesthesia completely disrupts neural signals that are signalling perceptual organization [40] (i.e. meaningful information integration), while at the same time more ‘meaningless’ interactions remain [100].

Another additional ingredient may be that some types of recurrency are more likely to induce changes to the connectivity of the neural network, and do so at different time scales, and using different molecular mechanisms. A longstanding idea is that recurrent interactions between neurons, particularly if they involve burst firing of neurons, are likely to induce synaptic plasticity, involving NMDA receptor activation and subsequent CA²⁺ influx, which induces a cascade of molecular and genetic events, changing the efficacy of the synapses involved [101]. Indeed, NMDA receptor activation has been tightly linked to cortico-cortical feedback [37], and NMDA receptor blockers impair feature integration [102]. Some have proposed that the final common pathway of many anaesthetics is NMDA receptor inactivation [103]. Synaptic plasticity typically occurs at time scales too slow for a direct involvement in generating conscious experience, yet NMDA receptor activation may be one of the required preconditions for effective large-scale recurrent interactions to occur [104]. Also, more rapid changes in synaptic efficacy occur, such as paired pulse facilitation or suppression, each having various potential underlying mechanisms [105]. Another important role may be played by γ-aminobutyric acid, as it is actively involved in the dynamics of the neuronal competition underlying binocular rivalry [106]. In sum, it may be that recurrent interactions are just the first step in a more complex cascade of molecular events that induce changes to the network at various time scales. These changes to the network—and not the recurrency per se—may then be the ingredient that generates the conscious experience, and these changes may only happen when recurrency fulfils certain temporal or spatial conditions (figure 3).

Figure 3.

Recurrent processing-induced network plasticity as a potential ‘missing ingredient’. A pre-stimulus network state is depicted, with varying strengths of connections. A stimulus evokes a feedforward sweep, followed by recurrent processing. Recurrent processing may then either induce changes to the strengths of connections (via long or short term, or pre- or postsynaptic plasticity; right) or not (left). Only in the case of network changes will conscious experience arise. (Online version in colour.)

IIT has formulated some axioms about conscious experience, for which it seeks a physical substrate (PSC), for example, in the brain. This would pose a problem, however, when these axioms are not unique to conscious experience yet also hold for unconscious processing. That seems to be the case for two of these axioms, those of information and of integration. The axiom of information states that ‘experience is specific, being composed of a particular set of phenomenal distinctions (qualia), which make it what it is and different from other experiences. … the content of my current experience might be composed of seeing a book (rather than seeing no book), which is blue (rather than not blue), and so on…’ [75, p. 451]. This is hardly specific to conscious processing, as all unconscious processing equally involves specific information, for example, in that neurons encode all sorts of features and high-level objects (at the exclusion of others) even when anesthetized, under conditions of masking, etc. [58]. The axiom of integration states that ‘experience is unitary, meaning that it is composed of a set of phenomenal distinctions, bound together in various ways, that is irreducible to non-interdependent subsets' [75, p. 452]. This smacks of what others call feature binding, Gestalt grouping, perceptual organization, etc. While previously thought to be exclusively linked to conscious perception, feature binding does occur to quite some extent also in unconscious visual processing [107]. This holds in particular for the binding of features that are detected simultaneously by a single neuron, such as orientation and movement or colour (so-called base-groupings), as these are already extracted during the feedforward sweep [108]. More problematic is the integration of distant features that constitute many perceptual illusions or perceptual inferences, which often also occur unconsciously [58]. Only some levels of perceptual organization or Gestalt grouping seem to require consciousness. In that sense, IIT suffers from the same ‘missing ingredient’ as the other theories mentioned, in that consciousness is more than the mere integration of information.

Note that theories that propose higher cognitive functioning as the key ingredient for consciousness also suffer from the ‘missing ingredient’ problem, as many high-level cognitive functions occur unconsciously. For example, masked no-go stimuli activate prefrontal inhibition networks in such a way that they induce a slowing of reactions [20], indicating that cognitive control can be executed unconsciously. Also, task switching, attention shifting, conflict monitoring, error detection and evidence accumulation can occur unconsciously [109,110]. Do such results suggest the existence of unconscious access or metacognition? Using unconscious cues to adapt or control behaviour, to change or make decisions or to switch strategies certainly qualifies for unconscious access, as these are often ways in which conscious access is operationalized. The case for unconscious metacognition is more difficult.⁷ However, post error slowing is very similar to opting out, in that the system seems to know something about its own performance and adapts its behaviour accordingly. Post error slowing does occur for unconscious errors [111], yet is typically weaker than after conscious errors. When comparing conscious to unconscious cognition it is often found that the former is more flexibly deployed and has stronger effects on behaviour as well as stronger associated brain signals. This does not solve the ‘missing ingredient’ issue, however, unless that ingredient is ‘more’ or ‘stronger’. Some have, therefore, proposed that cognitive functioning per se is not the key ingredient of consciousness but learning to continuously redescribe the cause–effect relations of these functions, whether simple or complex, and internal or external. Once these relations are well established, they leave the conscious domain (as in well-learned skills) [112]. Naccache [113] proposes that a multitude of functions (or ingredients) may be simultaneously required to produce consciousness. In his dialectic approach, the starting point would be the set of cognitive functions that in a healthy awake adult support self-reportability (in its widest sense, so also non-verbal), as this is the aspect of consciousness that is central to the phenomenon, at least according to longstanding philosophical and introspective intuitions. The next step would be to find the neural mechanisms that support such self-reportability, which in the healthy human seem to be recurrent interactions between highly educated and sophisticated cortical regions [52]. Only when all the necessary ingredients coincide, that is when both sufficiently educated cortical areas are active and engage in recurrent interactions, does consciousness as experienced by a normal adult emerge. It should be possible to infer the psychological states going along with recurrent interactions between less sophisticated cognitive modules (say in animals, babies, etc.), such that at some point one would have to conclude that ‘consciousness as we experience it’ is no longer present. The same logic could be applied to difficult to categorize patients suffering from disorders of consciousness [114].

6. Concluding remarks: let us talk

I have tried to highlight some pregnant issues in our current knowledge about consciousness and how it may relate to what happens in brains. The focus was on three theories, in particular, being recurrent processing theory (RPT), global neuronal workspace theory (GNWT) and integrated information theory (IIT). I noticed the stalemate between RPT and GNWT, primarily caused by them having different explananda. I discussed how the three theories endorse panpsychism and how this could potentially be solved, and that each of them still miss ‘key ingredients’ to satisfactorily distinguish unconscious from conscious processing. Many other theories exist, of course. Readers are encouraged to comment on how some of these other theories may solve the issues raised. Because the science of consciousness will clearly benefit from an open exchange of ideas, based on the wealth of experimental data that exists and is produced every day, I hope I have inspired such an exchange.

Data accessibility

This article has no additional data.

Competing interests

I declare I have no competing interests.

Funding

I received no funding for this study.