A predictive processing theory of sensorimotor contingencies: Explaining the puzzle of perceptual presence and its absence in synesthesia

Abstract

Normal perception involves experiencing objects within perceptual scenes as real, as existing in the world. This property of “perceptual presence” has motivated “sensorimotor theories” which understand perception to involve the mastery of sensorimotor contingencies. However, the mechanistic basis of sensorimotor contingencies and their mastery has remained unclear. Sensorimotor theory also struggles to explain instances of perception, such as synesthesia, that appear to lack perceptual presence and for which relevant sensorimotor contingencies are difficult to identify. On alternative “predictive processing” theories, perceptual content emerges from probabilistic inference on the external causes of sensory signals, however, this view has addressed neither the problem of perceptual presence nor synesthesia. Here, I describe a theory of predictive perception of sensorimotor contingencies which (1) accounts for perceptual presence in normal perception, as well as its absence in synesthesia, and (2) operationalizes the notion of sensorimotor contingencies and their mastery. The core idea is that generative models underlying perception incorporate explicitly counterfactual elements related to how sensory inputs would change on the basis of a broad repertoire of possible actions, even if those actions are not performed. These “counterfactually-rich” generative models encode sensorimotor contingencies related to repertoires of sensorimotor dependencies, with counterfactual richness determining the degree of perceptual presence associated with a stimulus. While the generative models underlying normal perception are typically counterfactually rich (reflecting a large repertoire of possible sensorimotor dependencies), those underlying synesthetic concurrents are hypothesized to be counterfactually poor. In addition to accounting for the phenomenology of synesthesia, the theory naturally accommodates phenomenological differences between a range of experiential states including dreaming, hallucination, and the like. It may also lead to a new view of the (in)determinacy of normal perception.

How can it be true, as I think it is, that we are perceptually aware, when we look at a tomato, of the parts of the tomato which, strictly speaking, we do not perceive. This is the puzzle of perceptual presence.

(Noë, 2006, p. 414)

In this quote Alva Noë identifies a central challenge for theories of perception. In normal circumstances perceptual content is characterized by subjective veridicality; that is, the objects of perception are experienced as real, as belonging to the world. When we perceive the tomato we perceive it as an externally existing object with a back and sides, not simply as a specific view—a “perspectival take”—on an external scene (Noë, 2004). Subjective veridicality —or “perceptual presence”—may seem sufficiently obvious to not require explanation, but this assumption is readily challenged by well-established examples of perception in which raw sensory experience (“qualia”) remains but perceptual presence is lacking. These include afterimages, some forms of hallucination, and—as we will see— synesthesia.

According to Noë, perceptual presence is explained by a “sensorimotor theory” on which perception depends on a practical mastery of sensorimotor dependencies or “sensorimotor contingencies” (SMCs) (O'Regan & Noë, 2001). The theory inherits from Gibsonian notions of “affordances” (Gibson, 1979) and from enactive cognitive science (Thompson & Varela, 2001) which stress the importance of brain-body–world interactions in cognitive processes. On sensorimotor theory, the perception of a tomato as a (perceptually present, real, subjectively veridical) tomato is given by practical mastery of the SMCs governing how the sensory responses elicited by the tomato will behave in a variety of situations. A strong point of this theory is that it suggests why there are differences in qualitative character between modalities, the reason being that different modalities instantiate different SMCs (O'Regan & Noë, 2001). However, sensorimotor theory faces two major challenges. The first is to specify at the level of neural mechanism what is meant by a SMC and by their mastery. The second is to account for instances of perception which apparently do not involve SMCs. As we will see, synesthesia exemplifies this second challenge.

An alternative and increasingly influential theory of perception derives from the idea of the brain as a “prediction machine” (Clark, 2013; Hohwy, 2013; Seth, 2013). Applied to perception, this idea views perceptual content as a form of inference about the causes of sensory signals. In the currently popular formulation of “predictive processing” (PP)¹ or the “Bayesian brain” this holds that perceptual content is determined by hierarchically organized generative (i.e., predictive) models (HGMs) of the external causes of sensory signals, induced by a process approximating Bayesian inference (Friston, 2009). Perception of a tomato, on this view, involves the brain deploying a high-level generative model predicting the sensory responses elicited by the tomato. In contrast to sensorimotor theory, PP emphasizes neural mechanisms as both necessary and sufficient for perceptual experience (at least at any particular instant). While accumulating evidence is providing strong (though indirect) support for PP (Bubic, von Cramon, & Schubotz, 2010; Clark, 2013; Hohwy, 2013; Koster-Hale & Saxe, 2013; Yuille & Kersten, 2006), the theory has until now not addressed the key challenge of perceptual presence as identified within sensorimotor theory. Neither has it yet been applied to synesthesia.

Here, I address these challenges by integrating insights from predictive processing and from sensorimotor theory to derive a Predictive Perception account of SensoriMotor Contingencies (PPSMC). Adapting a recent formulation by Friston and colleagues (Friston, Adams, Perrinet, & Breakspear, 2012), I propose that normal (veridical) perception is underpinned by counterfactually-rich generative models, which means that these models encode not only the likely causes of current sensory inputs, but also the likely causes of those sensory inputs predicted to occur given a large repertoire of possible (but not necessarily executed) actions- hence the term “counterfactual.” These counterfactually-rich generative models add mechanistic focus to the notion of “mastery of sensorimotor contingencies” central to sensorimotor theory (Noë, 2004; O'Regan & Noë, 2001). My specific claim is that the subjective veridicality (or perceptual presence) of normal perception depends precisely on the counterfactual richness of the corresponding generative models. If true, this implies that perceptual presence will be lacking when the corresponding generative models are counterfactually poor.

Synesthesia provides an ideal test case for the application of PPSMC. Synesthetes enjoy a remarkably rich perceptual world in which stimuli in one modality (inducers) reliably induce additional perceptual experiences (concurrents) either in the same modality (e.g., grapheme-color synesthesia in vision) or in a different modality (e.g., sound-vision synesthesia) (Grossenbacher & Lovelace, 2001; Sagiv & Frith, 2013; Ward, 2013). Synesthesia is by definition highly consistent across time (Baron-Cohen, Wyke, & Binnie, 1987; Eagleman, Kagan, Nelson, Sagaram, & Sarma, 2007) and automatic (i.e., synesthetes have little control over the onset and appearance of a concurrent; Mattingley, 2009). Importantly, inducers are not substituted by concurrents; for example, music-taste synesthetes continue to hear music as well as tasting it (Beeli, Esslen, & Jancke, 2005). This highlights a common but as yet unexplained feature of synesthetic phenomenology, namely that concurrents while often perceptually vivid (Ward, 2013; Ward, Jonas, Dienes, & Seth, 2010) lack perceptual presence. In other words synesthetes have intact reality-checking with respect to their concurrents so that, unlike the perception of an inducer (or a tomato), characteristic synesthetic experiences are subjectively non– veridical.

Existing theories of synesthesia do not account for this key distinction between perceptual reality (which synesthetic experiences often have) and subjectively veridicality or presence (which they typically lack). Such theories are usually based on the simple premise that synesthesia involves additional functional co– activation of brain regions implicated in processing concurrents and inducers (Bargary & Mitchell, 2008; Rouw, Scholte, & Colizoli, 2011), whether by increased cross-activation at low levels of stimulus processing (Ramachandran & Hubbard, 2001a) or disinhibition or direct feedback from higher levels (Grossenbacher & Lovelace, 2001; Hubbard, Brang, & Ramachandran, 2011; Smilek, Dixon, Cudahy, & Merikle, 2001). In either case the subjective non– veridicality of synesthetic concurrents remains unexplained. By contrast, PPSMC can account both for the perceptual reality of synesthetic concurrents and for the fact that, for synesthetes, these concurrents are pre-reflectively not part of the “real” world (i.e., they lack perceptual presence). The claim is that synesthetic concurrents depend on counterfactually– poor generative models of their external causes, as compared to those related to inducers, because there is no corresponding rich world-related statistical structure for such models to learn. In other words, counterfactually-poor generative models encode a smaller repertoire of likely causes of sensory inputs conditioned on possible actions, as compared to counterfactually-rich models. I will show that this hypothesis accounts as well for phenomenological distinctions differentiating synesthesia from other perceptual modes including imagery, hallucinations, and dreams.

The remainder of this paper is organized as follows. I first outline the basic conceptual structure of PP, avoiding the detailed mathematics. I then describe the main tenets of sensorimotor theory, and show how counterfactual PP can operationalize the notion of mastery of SMCs and solve the problem of perceptual presence. This constitutes the core contribution of PPSMC. After providing some examples from normal perception, I turn to the specific case of synesthesia. The subsequent sections summarize the relevant features of this condition, describe why it poses a serious and unresolved challenge for sensorimotor theory, and elaborate on a new account based on PPSMC. I then discuss how the theory accounts for individual differences in synesthetic phenomenology and for other unusual perceptual experiences. The final sections broach broader issues including the distinction between cortical “dominance” and “deference” (Hurley & Noë, 2003b) and the possible indeterminacy of normal perception. For ease of reference, Table 1 provides a glossary summarizing the more technical terminology used throughout.

TABLE 1.

A glossary of some of the technical terminology and abbreviations used in this paper. Order of presentation is alphabetic

Active inference	An extension of PP (and part of the free energy principle), which says that agents can suppress prediction errors by performing actions to bring about sensory states in line with predictions.
Bayesian inference	A principle for estimating the probable causes of observed data (the posterior) given prior “beliefs” about these causes, and a generative model of the likelihood of observing some data given specific priors.
Counterfactual predictive processing	An extension of PP which says that generative models encode not only the likely causes of sensory signals, but also the likely causes and values (and precisions) of sensory signals that would occur given a repertoire of possible (but unexecuted) actions.
Doxastic veridicality	The property that perceptual content is understood cognitively to reflect a property of the real world. Perceptual content can have subjective veridicality in the absence of doxastic veridicality (e.g., in Charles Bonnet hallucinations).
Free energy principle	A generalization of PP according to which organisms minimize an upper bound on the entropy of sensory signals (the free energy). Under specific assumptions, free energy translates to prediction error.
Grapheme-color synesthesia	A common form of synesthesia in which graphemic (e.g., letter) inducing stimuli give rise to additional color experiences (concurrents).
Hidden causes and hidden controls	Hidden causes (controls) are causal factors responsible for sensory signals (motor actions) that are not directly available to perception, so that their existence and behavior must be inferred.
Hierarchical generative model (HGM)	A Bayesian implementation of PP in which posteriors at one level form the priors at one level lower, an arrangement which allows priors to be induced from the data stream itself (“empirical” Bayes).
Objective veridicality	The property that perceptual content reflects (at least partly) features of the real world.
Perceptual presence	The phenomenological property that perceptual content is experienced as part of—as continuous with—the real world; equivalent here to subjective veridicality.
Precision (weighting)	The precision of a probability distribution is the inverse of its variance and is a measure of uncertainty. Dynamic precision weighting (associated with attention) can modulate the balance between top-down and bottom-up signal flow: For example, low prediction-error precision corresponds to high confidence in top-down prior beliefs, so that prediction errors are less able to update these beliefs.
Predictive processing (PP)	A Bayesian scheme, dating at least to Helmholtz, which conceives of perception as a process of probabilistic inference on the likely causes of sensory signals. The scheme can be generalized to cognition and action (see active inference).
Sensorimotor contingencies (SMCs)	SMCs describe ways in which sensory signals change given actions in specific contexts; they are “rules” describing sensorimotor dependencies.
Sensorimotor theory	A cognitive theory according to which perception is constituted by the exercise of a practical mastery of sensorimotor skills or contingencies: On this theory, perception is an activity.
Subjective veridicality	The phenomenological property that the perceptual content is experienced as being part of the real world. As used here it is equivalent to perceptual presence (see above).

PREDICTIVE PERCEPTION AND SENSORIMOTOR CONTINGENCIES

Predictive processing, perception, and action

Predictive processing (PP) has a long history, originating with the insights of Hermann von Helmholtz and reaching recent prominence in the “Bayesian Brain” hypothesis (see Clark, 2013; Hohwy, 2013; Pouget, Beck, Ma, & Latham, 2013). The basic idea is that, in order to support adaptive responses, the brain must discover information about the likely external causes of sensory signals, without any direct access to these causes, using only information in the flux of the sensory signals themselves (Friston, 2009). According to PP, perception solves this problem via probabilistic, knowledge-driven inference on the causes of sensory signals. Applied to cortical networks, the concept of PP overturns classical notions of perception as a largely “bottom-up” process of evidence accumulation or feature detection. Instead, PP proposes that perceptual content is specified by top-down predictive signals emerging from multi-level hierarchically-organized generative models (HGMs) of the causes of sensory signals, which are continually modified by bottom-up prediction error signals communicating mismatches between predicted and actual signals across hierarchical levels (Friston, 2009; Lee & Mumford, 2003; see Figure 1). In this view, even low-level fine-grained perceptual content depends on a cascade of predictions flowing from very general abstract expectations which constrain successively more detailed predictions.

Figure 1.

A schematic of hierarchical PP across three cortical regions; the “lowest” on the left (R1) and the “highest” on the right (R3). Bottom-up (red) projections originate from “error units” (orange) in superficial cortical layers and terminate on “state units” (light blue) in the deep (infragranular) layers of their targets, while top-down (dark blue) projections conveying predictions originate in deep layers and project to superficial layers of their targets. Both prediction error signals and predictions are characterized by precisions (inverse variances) which determine the relative influence of top-down and bottom-up signal flow (see also Figure 2). Top-down precision weighting (dashed lines) is equivalent to modulating the post-synaptic gain of prediction-error projection neurons, possibly involving dopaminergic and cholinergic neuromodulation. Triangles represent pyramidal cells; circles represent inhibitory interneurons. Figure adapted from Friston (2009).

Although unequivocal neural evidence is still lacking (Clark, 2013; Pouget et al., 2013), PP is supported by an encouraging convergence of ideas and evidence from cognitive neuroscience, machine learning, and statistical physics. From cognitive neuroscience comes accumulating evidence that perceptual content—and features of the underlying neural activity—can be shaped or determined by pre– stimulus expectations (Bubic et al., 2010; Egner, Monti, & Summerfield, 2010; Kok, Brouwer, van Gerven, & de Lange, 2013; Kok, Rahnev, Jehee, Lau, & de Lange, 2012; Melloni, Schwiedrzik, Muller, Rodriguez, & Singer, 2011). For example, Egner and colleagues showed that repetition suppression (decreased cortical responses to familiar stimuli) is abolished when repetitions are unexpected, an effect compatible with PP but not predicted by standard accounts based on adaptation or sharpening of representations (Egner, Summerfield, Trittschuh, Monti, & Mesulam, 2008). From machine learning and statistical physics comes the mathematical machinery showing how hierarchical inference can be implemented by cortical networks via approximations to Bayesian inference (Dayan, Hinton, Neal, & Zemel, 1995; Friston, Kilner, & Harrison, 2006; Hinton & Dayan, 1996; Lee & Mumford, 2003).

Putting the detailed mathematics aside, an appreciation of the basic concepts is helpful. These rest on principles of Bayesian inference, which provide a computational mechanism for estimating the probable causes of data (the posterior) given the observed conditional probabilities of the data (likelihoods, i.e., the probability of observing some data given particular causes) and prior “beliefs” about the probable causes. In other words, Bayes’ theorem relates a conditional probability (which can be observed) to its inverse (which cannot be observed, but knowledge of which is desired). While exact Bayesian inference is computationally challenging and often intractable, a variety of approximate methods exist some of which have plausible neurobiological implementations. Prominent among these is Friston's “free energy” framework (Friston, 2005, 2009; Friston, Daunizeau, Kilner, & Kiebel, 2010; Friston et al., 2006), which, following earlier work by Hinton and colleagues (Dayan et al., 1995; Hinton & Dayan, 1996), shows how hierarchically– organized generative models (HGMs) can be induced from data by assuming that the brain minimizes an upper bound on the evidence for this data (this is the “free energy,” a quantity derived from statistical physics). The generalization of Bayes’ to a hierarchical scenario implies that posteriors at one level form the priors at one level lower, thus enabling a form of bootstrapping with respect to the priors—this is the idea of “empirical Bayes” that enables priors to be induced from the data stream itself.

Several aspects of PP applied to brain function are particularly relevant to sensorimotor theory and to synesthesia. First, an important implication of PP is the existence of a strong continuity between perception and imagination or imagery (Albright, 2012; Clark, 2012) (see also later). As emphasized by Hinton, to be able to perceive an object requires a generative model capable of autonomously creating, in a top-down fashion, fictive (i.e., surrogate) sensory signals that could originate from that object (Hinton, 2007). More generally, the key role of top-down predictive or generative models in perception points to a strong continuity not only with imagery but also with associative recall, dreaming, and other self– generated perceptual or quasi-perceptual states. As we will see later, the formation of strong associations via correlated sensory input (or via neonatally undifferentiated generative models) may induce HGMs that predict intra– or inter-modal fictive sensations associated with inducing stimuli which could in turn account for the existence of synesthetic concurrents.

Second, key to PP is the minimization of prediction error (or free energy) as a means of determining the most likely causes of sensory signals. Crucially, this can be accomplished in two ways: HGMs can be changed to accommodate unexpected sensory signals (perceptual inference and learning) or actions can be performed to confirm sensory predictions (active inference) (Friston, 2009; Friston et al., 2010). In most interpretations of PP, and especially in the free energy framework, these processes are understood to happen simultaneously and continuously. Actions themselves can be considered as arising from the fulfilment of proprioceptive predictions (minimization of proprioceptive prediction error) via motor reflexes (Friston et al., 2010). The close coupling of perception and action in this framework implies that PP involves the induction of HGMs that predict the dynamics of sensorimotor interactions, providing a link to the notion of “sensorimotor contingencies” (SMCs; Clark, 2012; K. Friston, 2012; O'Regan & Noë, 2001). We will see later that the concept of active inference is critical in understanding the role of counterfactual probability densities in PPSMC.

Third, generative models underlying perception and action in PP are hierarchical, with higher levels encoding more abstract, contextual, and multi– or amodal aspects of sensorimotor interactions. Importantly, the influence of prediction errors shaping posteriors and updating priors can differ across hierarchical levels leading to distinct consequences for perception and action (Adams, Stephan, Brown, Frith, & Friston, 2013; Edwards, Adams, Brown, Parees, & Friston, 2012). For example, if low-level sensory prediction errors have weak influence as compared to high-level prediction errors (e.g., by precision weighting, see below), posterior distributions specifying perceptual content will be strongly influenced by low-level priors. This is because these priors will be able to reshape high-level predictions on the basis of highly weighted prediction errors reaching high levels, while the low-level priors themselves will be relatively resistant to reshaping given the weak influence of low-level prediction errors.

Fourth, and relatedly, predictions and prediction errors within PP are associated with precisions (inverse variances, i.e., a measure of uncertainty). Controlling precisions provides a mechanism by which the influence of sensory (or motor) prediction errors on perceptual (or proprioceptive) predictions can be modulated rapidly and flexibly at specific hierarchical levels. Higher precision of prediction errors implies increasing their gain or weighting, and so enhances their impact on posterior distributions (see Figure 2). Note that precision is different from accuracy: A prediction can be precise but inaccurate, imprecise but accurate, or indeed any other combination. In hierarchical settings agents will have expectations about precisions which can translate into changes in precision weighting (Feldman & Friston, 2010; Hohwy, 2012), the optimization of which has been associated with attention (Feldman & Friston, 2010). Abnormal precision weighting has been suggested to underlie a variety of unusual perceptual states including psychotic hallucinations (Adams et al., 2013; Fletcher & Frith, 2009) and functional and motor symptoms in hysteria (Edwards et al., 2012).

Figure 2.

The influence of precisions on Bayesian inference and predictive processing. A. High precision-weighting of sensory signals (red) enhances its influence on the posterior (green) and expectation (black dashed line) as compared to the prior (blue). B. Low precision-weighting of sensory signals as compared to priors has the opposite effect on posteriors and expectations.

Predictive processing, sensorimotor contingencies, and counterfactual HGMs

An alternative view on the nature of perception is provided by “sensorimotor theory” (Noë, 2004; O'Regan & Noë, 2001). On the sensorimotor view, perceptual content is determined by a practical “know how” or “mastery” of sensorimotor dependencies or “sensorimotor contingencies” (SMCs). As O'Regan and Noë put it:

[S]eeing is a skillful activity whereby one explores the world, drawing on one's mastery of the relevant laws of sensorimotor contingency.

(O'Regan & Noë, 2001, p. 966)

For example, on sensorimotor theory the conscious visual experience of redness is given by practical² mastery of the SMCs governing how red things behave under a variety of situations. Note that sensorimotor theory associates conscious perception specifically with the mastery of SMCs, not just with their online operation (Noë, 2004). Even more so than the PP view just outlined, sensorimotor theory emphasizes the close coupling of perception and action. However, unlike the PP view, and despite the attention the theory has received over many years, possible neural or mechanistic implementations remain unspecified (except in highly reduced computational simulations, see e.g., Buhrmann, Di Paolo, & Barandarian, 2013, or in abstract color spaces, Philipona & O'Regan, 2006). In addition, the existence of synesthesia has long presented a challenge to sensorimotor theory in its purest form (Gray, 2003; Hurley & Noë, 2003a; Noë & Hurley, 2003). This is because the phenomenal character of a concurrent (e.g., the color of a grapheme) seems to have little to do with the SMCs underwriting the perception of the inducer. Gray highlights the “alien colour effect” as particularly troublesome for sensorimotor theory (Gray, 2003). This occurs when “colored-hearing” synesthetes experience an incongruent (visual) color concurrent on hearing an (auditory) color-name inducer (e.g., the heard word “red” might elicit a concurrent visual experience of the color green; see also Gray et al., 2006). According to Gray, this type of synesthetic association would not just be irrelevant to, but would actively interfere with, ongoing sensorimotor dependencies—consider the confusion that would be occasioned by hearing the phrase “a red bus is approaching”—so that such associations should be eliminated as SMCs are “mastered.” Interestingly, while conceding that sensorimotor theory is indeed challenged by synesthesia, Hurley and Noë do suggest that synesthetic color experiences may be phenomenally distinct from normal color experiences, a line of argument developed here as well. One item of empirical evidence consistent with this is that, in tests of visual search, synesthetic concurrents do not “pop-out” in the same way as happens in normal perception (Edquist, Rich, Brinkman, & Mattingley, 2006; Laeng, Svartdal, & Oelmann, 2004; Sagiv, Heer, & Robertson, 2006; Ward et al., 2010).

On the other hand, sensorimotor theory offers a valuable and distinctive perspective on subjective veridicality, or perceptual presence, which as mentioned marks a key difference between normal and synesthetic perceptual experience (even staunch critics of sensorimotor theory might agree with this, see for example, Block, 2005). On this theory, veridical perceptual scenes are world-revealing: They comprise a world of objects rather than “perspectival takes” on objects, as exemplified by the quote from Noë with which this paper began. The solution offered by sensorimotor theory is that the experience of the tomato as an object is given by an implicit knowledge of the ways in which our perspectival takes would alter subsequent visual inputs. In other words, we know, at some level, how moving our eyes and our bodies would reveal additional sensory information about the tomato, and it is this knowledge—what Noë and O'Regan call “mastery” of SMCs—that endows perceptual presence to our experience of the tomato as a tomato-object, with its immediately given subjective veridicality. Importantly, this knowledge is typically “sub-personal” in the sense that it need not be in the form of explicitly held or declarable beliefs (Roberts, 2009). This view aligns strongly with the general tradition of enactive perception according to which experience is intrinsically tied up with motor dispositions, such that perceptual content is constitutively dependent on brain-body–world interactions (Noë, 2004).

What could it mean, in terms of neurocognitive processes, to have knowledge or mastery of the SMCs underlying perceptual presence? Recall that according to PP, perception depends on the operation of HGMs of hidden (external) causes that best explain changing patterns of sensory inputs, driven by the minimization of prediction error between counter-flowing top-down predictions and bottom-up sensory evidence. Prediction error can be reduced either by changing the content of the HGM, or (simultaneously and continuously) by performing actions to bring about sensory input in-line with current predictions. These models can therefore be understood to instantiate sub– personal knowledge about perception-action couplings relevant to SMCs (Clark, 2012; Friston, 2012).³

However, the notion of mastery of SMCs relevant to perceptual presence seems to suggest the involvement of more than just a generative model predicting ongoing sensorimotor flow. The incorporation of an explicitly conditional (or even meta-conditional) aspect seems essential: Were this (rather than that) action to be performed, then this (and not that) would be the most likely causes of the sensory signals that would be likely to occur. One way to accommodate this essential conditionality within a PP framework is to consider that HGMs could encode counterfactual probabilities. That is, HGMs not only represent the most likely hidden causes of current sensory input, they also encode how sensory inputs would change on the basis of a repertoire of possible actions, even if those actions are not performed. In other words, HGMs encoding conditional aspects of SMCs would incorporate explicitly counterfactual probabilistic models of the behavior of hidden causes of fictive sensory signals (and their precisions) given particular actions. This extended interpretation of PP can be called Predictive Perception of SensoriMotor Contingencies (PPSMC). It is strikingly compatible with Noë's view of perceptual experience:

Qualities are available in experience as possibilities, as potentialities, but not as complete givens. Experience is a dynamic process of navigating the pathways of these possibilities.

(Noë, 2006, p. 428).⁴

Helpfully, the notion of counterfactual probabilistic models has already been proposed within Friston's free-energy framework, where they are used to model saccadic eye-movements as tests of perceptual hypotheses (Friston, 2012; Friston et al., 2012). In this view, saccades are guided on the basis of prior beliefs about which movements would maximize precision or confidence (minimize uncertainty) in perceptual predictions. This entails a generative model of how the world is actively sampled which explicitly incorporates counterfactual probability distributions about the sensory consequences of possible actions, where these models encode conditional predictions about sensory values and their associated precisions.

Consider an example, adapted from Friston et al. (2012). Imagine that you are sitting in a garden and notice some fluttering movements in your visual periphery. Your brain forms the hypothesis (mediated by a HGM) that the fluttering is caused by a bird. This minimizes the associated prediction errors and induces a perceptual state including a peripherally-located bird (perhaps with some phenomenal indeterminacy, see later). This, so far, is standard PP applied to perception. The next step is that the brain selects prior beliefs about gaze direction that will minimize the uncertainty about the current HGM related to the peripheral fluttering. This selection depends on conditional expectations of the precision of counterfactual aspects of the HGM (i.e., predictions about how precisions would change given a repertoire of possible actions). As mentioned, this entails that the HGM explicitly encodes counterfactual predictions about sensory values and associated precisions (and their likely hidden causes), where the term counterfactual refers to the specific sense of possible but non-actualized situations.

For Friston, the next step is that the selected priors will produce proprioceptive predictions related to the oculomotor system and (counterfactual) sensory predictions about the visual consequences of executing the saccade. Finally, action will fulfil the proprioceptive predictions (via oculomotor reflexes that minimize proprioceptive prediction errors) leading to foveation of the bird and—if indeed a bird is foveated—the uncertainty in the bird-related HGM will be reduced (minimized).

It is important to say exactly what is distinctive about PPSMC as compared to standard predictive coding (e.g., Rao & Ballard, 1999) and active inference (Friston, Mattout, & Kilner, 2011). Standard predictive coding involves the predictive modeling of sensory responses and does not care about the impact of actions on these responses, so counterfactual probability densities are not implicated. Active inference says that sensory prediction errors can be suppressed by performing actions to confirm perceptual predictions. In this sense, counterfactual probability densities are implicit in dynamics of priors predicting the sensory consequences of actions. In contrast to both (and leveraging the framework of Friston, 2012; Friston et al., 2012), PPSMC requires that counterfactual predictions be explicitly incorporated as part of the priors in a HGM. That is, a counterfactually-rich HGM will model predicted future states (sensory signals, their external causes, and associated precisions) under a broad repertoire of different “controls” (those signals, not directly accessible to an agent, that cause movements). The explicit encoding of precisions and precision expectations within counterfactual components of the model can be used to guide action selection via criteria such as maximization, as described above (Friston, 2012; Friston et al., 2012). However, they can also be considered as underpinning fundamental aspects of perceptual phenomenology, as I argue next. To put it more simply, active inference is roughly equivalent to the concept of a SMC, while PPSMC operationalizes the notion of mastery or knowledge of SMCs within sensorimotor theory.

In this light, and as already suggested, PPSMC leads to a view of perception that directly addresses Noë's puzzle of perceptual presence, the puzzle that we can be perceptually aware of parts of an object that strictly speaking we do not directly perceive. As Friston puts it:

Being able to predict what is currently seen also enables us to predict fictive sensations that we could experience from another viewpoint

(Friston et al., 2012, p. 17)

Putting all this together, I suggest that perceptual presence is underpinned by the engagement of counterfactually-rich HGMs within the framework of PP. We experience normal perception as world– revealing precisely because the generative models underlying perceptual content specify a rich repertoire of explicit counterfactual probability densities encoding (mastery of) SMCs. Importantly, this departs from standard versions of sensorimotor theory which, in contrast to PP, are explicitly non– inferential (Noë, 2004). It also bears clarifying that this application of counterfactual HGMs differs from that offered by Friston and colleagues, who—while recognizing similarities between active inference and SMCs (Friston, 2012) do not equate counterfactual PP with the notion of mastery of SMCs (as done here), and are concerned specifically (at least so far) with understanding active perceptual sampling, rather than with the phenomenology of perceptual presence.

Presence, precision expectations, and action

It is worth taking a few lines to clarify the roles of precision expectations and action in the above account of perceptual presence. To rehearse, the contention is that the subjective veridicality (perceptual presence) associated with perceptual content is underpinned by the counterfactual richness of the corresponding HGMs, where richness refers to the range of conditional sensorimotor relations that are counterfactually encoded. This means that particular content (e.g., a bird, a tomato) will be experienced as perceptually present to the extent that the corresponding HGM encodes a rich repertoire of predicted sensory signals (and their associated precisions and likely external causes) conditioned on possible actions.

If counterfactual predictions have low expected precision, one might wonder how they could modify current perceptual content, since low precision prediction errors generally do not lead to updating of higher-level priors. However, it is important to distinguish between perceptual content (which is dependent on non-counterfactual aspects of the HGM that can be updated or modulated by high precision prediction errors) and perceptual presence, which (on this account) depends on the richness of counterfactually-encoded sensorimotor dependencies. Thus, low-precision counterfactual predictions can still modulate the perceptual presence associated with a specific content according to their richness and structure. On the other hand, counterfactual expectations of high precision can drive action, as described above in the example of saccade generation. However, on the present view, action is not constitutively necessary for perceptual presence; that is, it is possible to have a rich repertoire of counterfactual predictions endowing a high level of perceptual presence to some content, without any of these predictions simultaneously driving action. This means that one can experience an object as perceptually present without being driven to walk around it.

Examples of PPSMC

As a further illustration of the impact of counterfactual richness on perception, consider how we experience images.⁵

In Magritte's classic painting (see Figure 3), we are invited to reflect on the image of a pipe as precisely that—an image, and not a real pipe in the proximate world. What accounts for the image-like experience, in contrast to the subjectively veridical experience of pipe-ness that would be occasioned by visual exposure to a real pipe? According to PPSMC, the SMCs governing our interaction with an image of a pipe are very different (and substantially less rich) than those that would govern our interactions with an actual pipe. Images certainly change as a function of selective sampling, but they change in very different ways than do the referent. However flexibly we might inspect Figure 3 we would not be able to directly examine the rear of the depicted pipe. Thus, the best HGMs to account for our interaction with the image are those that entail selection of a high-level prior of image-hood, which would specify a distinct and impoverished repertoire of intermediate-level counterfactual probability densities (with respect to the sensory and precision consequences of possible actions regarding the pipe referent) as compared to a high-level prior of object– hood.

Figure 3.

Rene Magritte's The Treachery of Images (1928–1929). © ADAGP, Paris and DACS, London 2014.

A second example, adapted from Noë (2004) involves the perception of ellipses and circles. Inspection of Figure 4A (while avoiding looking at 4B) should elicit the perceptual content of (an image of) an ellipse, for that is what is there. Inspection of Figure 4B, however, involves both the visual impression of an ellipse and of a circular form, thanks to the perspectival context given by the rest of the picture. On the face of it this is puzzling: In some sense, we see both an ellipse and a circle given the same sensory impressions. Sensorimotor theory would explain this by saying that in Figure 4B we recognize a richer repertoire of SMCs than in Figure 4A. Roughly the same explanation applies for PPSMC. The additional context available in Figure 4B engages a high-level prior of object-hood (albeit within an image, see above) which activates a broader repertoire of intermediate-level counterfactual probability densities that in turn underpins the experience of a perceptually present circular image. In contrast, for Figure 4A we have only the SMCs activated by the superficial elliptical shape.

Figure 4.

A. A context-free ellipse, underlying the visual impression of an image of an ellipse. B. A context-laden ellipse, underlying the visual impression of images of both an ellipse and of a circular form, as a result of enriched counterfactual SMCs. Example adapted from Noë (2004).

Interim summary

The hierarchical generative models (HGMs) proposed within predictive processing provide a natural way to operationalize the idea of sensorimotor contingencies described within sensorimotor theory. This is because, within predictive processing, generative models are induced by minimization of prediction errors as a result of both perception and active inference, thus bringing together perception and action. Extending this idea, counterfactually-rich generative models explicitly encode the conditional nature emphasized by the mastery of sensorimotor contingencies relevant to the puzzle of perceptual presence. That is, a counterfactually-rich hierarchical generative model explicitly encodes probabilistic representations of the external causes and expected values and precisions of fictive sensations conditioned on a repertoire of possible actions, thus capturing the key notion within sensorimotor theory of somehow perceiving parts of an object not directly available within the ongoing sensorimotor flux. This theory of predictive processing of sensorimotor contingencies (PPSMC) unifies two distinct traditions of thinking about perception and in doing so is well placed to account for some otherwise puzzling phenomena. Prominent among these is synesthesia, which I now turn to as an illuminating test case for PPSMC.

PPSMC AND SYNESTHESIA

The existence of synesthetic concurrents

As mentioned, sensorimotor theory has struggled to account for synesthesia because the perceptually real nature of concurrents (at least for projector subtypes, see later) is apparently inconsistent with the SMCs underwriting perception of the inducing stimuli (Gray, 2003; Gray et al., 2006; Hurley & Noë, 2003a; Noë & Hurley, 2003). PPSMC, on the other hand, provides a way to understand both the existence of synesthetic concurrents and their usually overlooked subjective non-veridicality (lack of perceptual presence).

On any PP account, perception depends on probabilistic inference of the likely external (hidden) causes of sensory signals. This should hold true for synesthetic concurrents as well. This means that HGMs that encode (predict) inducers as causes of the corresponding (veridical) sensory inputs must also encode (predict) the inexistent sensory inputs (and their likely causes, and precisions) corresponding to the concurrent. Following general associationist principles (Albright, 2012; James, 1890), it is plausible that these additional components to a predictive model emerge via associative processes during development and learning, possibly facilitated by genetic predispositions leading to structural cross-wiring (Bargary & Mitchell, 2008). On this view, reliable (inter– and intra-modal) correlations in sensory signals will induce generative models that predict their co-occurrence via a single set of external causes. An alternative but also associationist view is that humans are born with strong cross-modal associations which are pruned away or inhibited during development. This “neonatal synesthesia” hypothesis implies that different sensory modalities have to be “learned” during development in order to be separated (Maurer & Mondloch, 2006). On this view, HGMs underlying perception are initially highly multimodal, encoding strong priors on cross– modal correlations which are selectively weakened over time, but less so for synesthetes than neurotypics. A third alternative is that there may be a short developmental window in which synesthetic correspondences could be “burned in” by even single salient events involving novel associations coupled to reward or emotional salience. It is not necessary here to arbitrate among these positions (see Deroy & Spence, 2013): What matters is that there is good evidence for associationist accounts of cross– and intra-modal perceptual associative learning, relevant highlights of which are summarized next.

At the behavioral level, intra-modal and cross– modal associations, both innate and learned, have well-established effects on perceptual experience. For example, in the famous McGurk effect (McGurk & MacDonald, 1976), the visual appearance of moving lips can strongly influence the auditory experience of a clearly spoken phoneme. There is compelling evidence of cross-modal perceptual correspondences even in non-human animals: Chimpanzees are better at classifying bright stimuli when they are presented along with high-pitched sounds as compared to low-pitched sounds, suggesting an association between brightness and pitch similar to that observed in humans (Ludwig, Adachi, & Matsuzawa, 2011). In a related study of neurotypic humans, smoothness, softness, and roundness of stimuli positively correlated with luminance of a preferred color; smoothness and softness also positively correlated with the color itself (Ludwig & Simner, 2013). Moreover, tactile sensations were associated with specific colors (e.g., softness with pink). This again suggests the existence of cross-modal associations which could underwrite synesthetic concurrents via their encoding in multimodal HGMs. Interestingly, Deroy and Spence suggest in a recent review that cross-modal correspondences underlying synesthesia may be grounded in sensorimotor associations. For example, angular and round visual shapes are usually associated with non-words “takete” and “maluma” respectively, supporting the idea that the sharp vocal transitions engaged when vocalizing “takete” map onto visually sharp angular transitions (and viceversa for “maluma”) (Deroy & Spence, 2013). This idea, which recalls some earlier suggestions (e.g., Ramachandran & Hubbard, 2001b), fits nicely with that of cross-modal correspondences being encoded in predictive models of SMCs.

More direct behavioral evidence has been sought by recent studies which have employed associative training regimes to induce forms of synesthesia in neurotypical subjects (Colizoli, Murre, & Rouw, 2012; Kusnir & Thut, 2012; Meier & Rothen, 2009; Rothen, Wantz, & Meier, 2011). While the results of these studies are mixed, at least some behavioral criteria for synesthesia are met following training. For example, trained grapheme-color associations have been shown to elicit Stroop effects similar to those observed in natural synesthetes, although performance was unchanged on more perceptually oriented tasks such as perceptual crowding (Colizoli et al., 2012; Meier & Rothen, 2009). In one very recent study employing a highly demanding training regime, a majority of subjects reported the appearance of subjective synesthetic concurrents even outside the laboratory, although not consistently (Bor and colleagues, unpublished data).

At the neural level, learned associations have clear effects on neural responses. Importantly, these effects generalize beyond so-called “association” (infero– temporal, IT) cortex. In a classic study, Schlack and Albright trained macaques to associate static images of arrows with moving dot patterns and found that, after training, neurons in cortical area MT which normally respond only to movement also responded to the static images, and furthermore, that these neurons showed an orientation tuning to arrow direction corresponding to their “natural” tuning curves for motion (Schlack & Albright, 2007). They interpreted this activity as a neural correlate of visual imagery of motion evoked by top-down recall, an interpretation consistent with a role for top-down predictive signaling in eliciting concurrent-related perceptual content. A wide range of similar neural evidence is surveyed in (Albright, 2012). In humans, the neural underpinnings of cross-modal correspondences are perhaps less well characterized. One recent study identified the involvement of right intraparietal cortex in a similar though not identical location to that implicated in synesthesia (Bien, ten Oever, Goebel, & Sack, 2012). Other lines of evidence show that stimuli presented in one modality can modulate (de Haas, Schwarzkopf, Urner, & Rees, 2013; Noesselt et al., 2010) or elicit (Liang, Mouraux, Hu, & Iannetti, 2013; Luo, Liu, & Poeppel, 2010) specific responses in other modalities.

It is now possible to say how predictive perception could lead to the emergence and persistence of synesthetic concurrents. The idea is that on encountering an inducer, a synesthete's brain deploys a suite of generative models which predict the external (hidden) causes of the sensory inputs associated to the inducer, as well as those indirectly associated to the concurrent via the synesthetic relationship. This relationship, encoded in the HGM, could emerge from any combination of the associationist mechanisms outlined above. The combined content of this HGM across hierarchical levels generates an integrated perceptual content incorporating both the inducer and the concurrent.

Crucially, these HGMs must be resistant to reshaping by sensory prediction errors that would arise when an inducer is encountered in the absence of stimuli directly representing the concurrent. One might suspect that concurrents should be eliminated over time by reshaping of prior beliefs to explain away persistent prediction errors (equivalent to the elimination of a SMC within sensorimotor theory). This, of course, is the worry motivating the “alien colour effect” challenge mentioned earlier (Gray, 2003). One way to understand how this resistance might happen is by considering precisions and the hierarchical nature of generative models (Adams et al., 2013; Edwards et al., 2012). The suggestion is that synesthetic associations are encoded within intermediate-level generative models corresponding to associative or secondary sensory cortical processing. These intermediate-level models are endowed with unusually high prior precision weighting so that these priors overwhelm concurrent-related sensory prediction errors flowing from lower levels. These aberrant precisions may in turn emerge from differences in associative mechanisms differentiating synesthetes from neurotypics, perhaps implicating differences in attentional processing. At the same time, these intermediate-level models pass on precise prediction errors to higher-level models that are trying to explain concurrent-related percepts that they did not predict. These higher-level models are reshaped over time to predict the synesthetic percepts in a context-sensitive fashion, which is consistent with the context dependency of many synesthetic phenomena (Ward, 2013). This account is just one among a variety of possible mechanisms⁶ which could assure resistance of synesthetic HGMs to reshaping by prediction error signals by calling on aberrant encoding of precisions and precision-weighting.

The subjective non-veridicality of synesthetic concurrents

The key phenomenological property of the subjective non-veridicality of synesthetic concurrents can now be explained by appealing to differences between the counterfactual richness of the HGMs associated with inducers and concurrents. For inducers (and in normal perception), perceptual content depends on counterfactually-rich HGMs of the behavior of hidden causes of fictive sensory signals in response to hidden controls relating to possible actions. In contrast, for concurrents, the corresponding HGMs are hypothesized to be counterfactually poor because the hidden causes giving rise to concurrent– related sensory signals do not embed a rich and deep statistical structure for the brain to learn (i.e., to encode into counterfactual representations). In particular, there is very little sense in which concurrent percepts depend on active sampling of their hidden causes. (For example, synesthetic visual responses to music are unaffected by shutting or moving the eyes; Ward, 2013). According to PPSMC, this comparative counterfactual poverty explains why synesthetic concurrents are subjectively non-veridical.

The hierarchical nature of generative models in perception again comes into play here. Just as highlevel models learn to predict the concurrent-related perceptual hypotheses of intermediate-level models (see above), high-level models may also learn to predict the counterfactual poverty of these intermediate-level models. That is, the subjective non-veridicality of synesthetic concurrents gets built right into the structure of the generative model as a result of high-level models trying to predict the behavior of hidden causes at (counterfactually-poor) intermediate levels.

In short, for concurrents, there is only a very weak sense in which generative models hierarchically encode the counterfactual probability distributions reflecting fictive sensations (and their likely external causes and precisions) conditioned on possible actions, where these distributions are exactly those which correspond to the notion of perceptual presence (subjective veridicality) described within sensorimotor theory.⁷

Let's consider a concrete example, contrasting a sensorimotor account of the (normal) perception of a tomato with grapheme-color synesthesia. On sensorimotor theory, to perceive the tomato is to have mastery of corresponding SMCs. That is, we have a sense of the perceptual presence of the whole tomato because we (sub-personally) know how to move our eyes and bodies in order to obtain more information about it. This account, however, struggles to explain grapheme-color synesthesia because there seem to be few or no relevant SMCs related to concurrents (Gray, 2003; Hurley & Noë, 2003a). On a PP account, we perceive the tomato as a tomato because HGMs induce higher-order invariants relevant to object-hood (Clark, 2012). In this view SMCs reflect deeply layered hierarchical models of hidden causes of sensory signals. The key new move in PPSMC is to propose that HGMs underlying perception explicitly encode mastery of SMCs as counterfactual probability densities: i.e., predictions of fictive sensations (and their expected precisions and likely external causes) conditioned on specific inferential sampling actions, and that this underlies the phenomenology of presence (or its absence). Our perception of the tomato (or inducer) is counterfactually rich, involving deeply layered probabilistic models of how possible actions could lead to changes in values and precisions of sensory signals. This richness leads to the tomato being perceived as perceptually present, available, and externally real. By contrast, the synesthetic perception of a concurrent is counterfactually poor, lacking any deep hierarchical structure linking possible actions to fictive sensory signals and precisions, which leads to it being perceived as unreal and as lacking in external object-hood.

Projector and associator synesthesia subtypes

There is considerable individual variability among synesthetes in the phenomenology of concurrents and in inducer-concurrent relationships. Within grapheme-color synesthesia, one relatively common (though still controversial) distinction is between projector subtypes—who experience concurrents as occupying specific extra-personal spatial locations, usually but not always co-located with their inducers —and associator subtypes, who experience concurrents with an ill-defined or absent spatiality (Dixon, Smilek, & Merikle, 2004; Rouw & Scholte, 2010). Typically, projectors report their concurrents as being more perceptually vivid than do associators. At the limit, associators may be equivocal about whether there is any specific phenomenology at all, beyond an involuntary and automatic conceptual association (Ward, Li, Salih, & Sagiv, 2007).

The distinction or continuum between projectors and associators in grapheme-color synesthesia is naturally accommodated by the present theory. For associators, the lack of spatial specificity negates an entire set of SMCs that would otherwise be brought to bear, whether for inducers or (projected) concurrents. This further constrains the counterfactual richness of the underlying HGMs, leading to a further diminution of perceptual presence and, toward the extremes, attenuation of the perceptual reality of the concurrent (shading ever further towards imagery). It could even be argued that the issue of perceptual presence only really emerges in the case of projectors, for whom there is a salient phenomenological contrast in this dimension between the appearance of inducers and concurrents.

Other forms of synesthesia

A great variety of synesthetic conditions have been documented besides the canonical case of grapheme– color synesthesia. A prominent example is mirror– touch synesthesia in which observed touch on another elicits the sensation of being touched (Banissy, Cohen Kadosh, Maus, Walsh, & Ward, 2009). This form of synesthesia unsurprisingly, has been linked to the operation of the mirror neuron system (Banissy et al., 2011; Mroczko-Wasowicz & Werning, 2012), which itself yields nicely to an interpretation in terms of PP since mirror-system processing is effectively about predicting the sensory consequences of observed actions (albeit with low precision on proprioceptive prediction errors in order to prevent explicit action mirroring or echopraxia). Another interesting example is sound-vision synesthesia (Ward, Huckstep, & Tsakanikos, 2006), in which auditory inducers evoke visual concurrents. This form of synesthesia is highly “productive” in the sense that it is hard (perhaps impossible) to find sounds that do not elicit visual concurrents. Again it seems that a PP account could apply given innate or learned cross-modal HGMs linking sound and vision (Ludwig et al., 2011). The productivity of soundvision synesthesia as compared to (for example) grapheme-color synesthesia might be rooted in the fact that the latter depends on learned inducers whereas sound inducers presumably do not need to be learned. The relative selectivity (non– productiveness) of grapheme-color synesthesia may (speculatively) arise because only certain graphemes are extensively learned in association with colors.

” Spatial form” or “sequence space” synesthesia is a relatively common variant in which numbers are experienced as having spatial locations, usually lying along a “number line” with a specific arrangement in extra-personal space (Eagleman, 2009; Sagiv, Simner, Collins, Butterworth, & Ward, 2006). Like grapheme-color synesthesia, this type of synesthesia is subjectively non-veridical in that the spatial location and embedding of numerical inducers is not confused with the spatial organization and appearance of the external world. We can again appeal to PPSMC to explain this non– veridicality by assuming that the synesthetic spatial embedding is comparatively counterfactually impoverished, and that the emergence of the relevant cross-modal correspondences may again originate in associations implicit in both synesthetes and non– synesthetes alike (Price & Mattingley, 2013). For example, reaction time measures indicate a left-to– right association with increasing magnitude (Dehaene, Bossini, & Giraux, 1993).

Imagery in synesthetes and neurotypics

There is some evidence that synesthetes (at least in grapheme-color synesthesia) have more vivid visual imagery than neurotypic controls (Barnett & Newell, 2008). The overlap between perception and imagery implied by PPSMC is consistent with this finding inasmuch as both imagery and synesthetic concurrents involve having higher confidence in top– down priors in the face of potentially conflicting sensory error signals, perhaps as a result of low– precision weighting of these sensory signals.

There are, however, important phenomenological differences between imagery and synesthesia. Imagery typically lacks perceptual reality and is associated with experiences of volition, while synesthesia (at least for projectors, see above) has the opposite characteristics (Ward, 2012). Why might this be? One possibility is that imagery involves transient enhancement of prior precision at high hierarchical levels, while synesthesia involves a more permanent enhancement of prior precision at intermediate levels (in both cases relative to prediction error precisions). The high intermediate– level precision for synesthesia explains the perceptual reality of concurrents, while the reshaping of high-level models predicting the concurrent explains their involuntary nature (i.e., they become expected). By contrast, transient modulation of highlevel priors is not sufficient to reshape intermediate– level models to drive perceptual reality, and neither is there the opportunity for yet-higher–level models to learn the properties of the levels immediately below, so that the related experiences are associated with the explicit engagement of these models and hence as voluntary. (See Edwards et al., 2012 for a similar account of the involuntary nature of many hysterical symptoms.)

There also appears to be differences between the nature of imagery in synesthetes and in non– synesthetes who have strong imagery capabilities. Notably, the latter tend to be poor at remembering colors, while synesthetes with strong imagery tend to be good at this task (Yaro & Ward, 2007). This suggests that synesthetes may be more “accurate” than non– synesthetes in both their imagery and perception, supporting enhanced memory. This is compatible with evidence that grapheme-color synesthetes perform better on tests of color perception as compared to non– synesthetes (Banissy, Walsh, & Ward, 2009; Yaro& Ward, 2007), and with the remarkable consistency of synesthetic concurrents, which is indeed used as an operational criterion (Rothen, Seth, Witzel, & Ward, 2013). Possibly, these differences in color memory and perception may emerge from compensatory increases in prior precision for synesthetes as compared to non– synesthetes.

Relation to other forms of perceptual content

If the PPSMC account of perception is on the right lines, it should be compatible with other forms of perceptual content—besides synesthesia and imagery —that also vary in their perceptual reality and veridicality. In order to accommodate these forms it is useful to differentiate veridicality into three sub-types: (1) subjective (i.e., whether the perceptual content appears, phenomenologically, as part of the external world—recall that this is distinct from perceptual reality which denotes the existence of a vivid perceptual phenomenology); (2) doxastic (i.e., whether the perceptual content is understood cognitively to reflect part of the external world); and (3) objective (i.e., whether perceptual content, at least to some extent, does indeed reflect properties of the external world). The distinction between subjective and doxastic veridicality is subtle but significant. For example, lucid dream states and certain non– delusional hallucinations (e.g., Charles Bonnet syndrome) involve perceptual content that appears, perceptually or subjectively, to be continuous or confusable with the external world (i.e., there is perceptual presence or subjective veridicality), while at the same time the individual retains intact reality checking, at the (doxastic) level of beliefs, that the content is not in fact part of external reality. On this view, synesthetic concurrents and imagery lack all three forms of veridicality, while the perception of inducers has all three.

Table 2 summarizes a selection of perceptual forms along the above dimensions. Organized this way, it is clear that many different combinations are possible. The claim is that, across the board, the absence of subjective veridicality in the presence of perceptual reality is grounded in a comparative counterfactual poverty of the underlying HGMs. Normal perception, as a benchmark, involves perceptual reality and all forms of veridicality (excluding visual illusions which may fail objective veridicality). Dreams and many hallucinations have both perceptual reality and subjective veridicality. On the PPSMC theory this implies that there are counterfactually-rich HGMs underlying the perceptual content, which is consistent with this content generally recapitulating content experienced in different contexts during normal perception. Interestingly, doxastic veridicality can be either present (for non-lucid dreams or hallucinations accompanied with delusions) or absent (for lucid dreams and non-delusional hallucinations) for these perceptual forms. The transition to doxastic veridicality poses an interesting challenge. Following Fletcher and Frith (2009) this may have to do with the reshaping of higher-level abstract priors in order to “explain away” persistent lower– level or intermediate-level perceptual prediction errors. As argued throughout this paper, synesthetic concurrents (for both projectors and associators) are characterized by a lack of subjective veridicality grounded in counterfactually-poor HGMs. In this regard they may be experientially similar to afterimages which also combine perceptual reality with subjective non-veridicality, again plausibly because of counterfactually impoverished generative models. Finally, imagery and associative recall lack perceptual reality as well as all forms of veridicality (though perceptual reality may be maintained to a limited extent for “strong” imagers).

TABLE 2.

Varieties of perceptual content differentiated by their perceptual reality, subjective veridicality, doxastic veridicality, and objective veridicality (see main text). Hallucinations without delusions include Charles Bonnet syndrome, Lewy Body dementia, and Parkinson's disease dementia (Santhouse, Howard, & ffytche, 2000). Hallucinations with delusions include canonical schizophrenic psychotic episodes as well as certain drug-induced hallucinations, including, for example, those elicited by psylocibin

	Perceptual reality	Subjective veridicality	Doxastic veridicality	Objective veridicality
Normal perception	✓	✓	✓	✓
Dreaming (non-lucid)	✓	✓	✓	✗
Hallucinations (with delusions)	✓	✓	✓	✗
Dreaming (lucid)	✓	✓	✗	✗
Hallucinations (without delusions)	✓	✓/✗	✗	✗
Synesthesia (projector)	✓	✗	✗	✗
Afterimages (e.g., retinal)	✓	✗	✗	✗
Synesthesia (associator)	✓/✗	✗	✗	✗
Imagery/associative recall	✗	✗	✗	✗

Interim summary

Synesthesia involves consciously perceived cross– or intra-modal correspondences between inducing stimuli and concurrents. Based on PPSMC I hypothesize that the existence of synesthetic concurrents is rooted in multimodal hierarchical generative models entailed by innate or learned associations interacting with unusually high prior precisions at intermediate hierarchical levels. These associations likely reflect widespread neurotypic cross– and intra-modal correspondences, and— crucially—their subjective non-veridicality arises because concurrent-related aspects of these generative models lack the counterfactual richness of those related to inducers. It remains as an outstanding challenge to address this framework to the large variety of synesthetic conditions, as well as—more generally—to other perceptual states which differ along the dimensions of perceptual realism and veridicality.

DISCUSSION

In this section I encounter some broader issues and open questions raised by the PPSMC and its application to synesthesia.

Cortical dominance and cortical deference

The relationship between sensorimotor theory and neural activity has been used to support a distinction between cortical dominance, in which perceptual content is shaped by the local neuronal properties of an activated region, and cortical deference, in which content depends on how an area participates in sensorimotor dependencies involving brain, body, and environment (Hurley & Noë, 2003a). Sensorimotor theory, as originally conceived, aligns well with cortical deference in emphasizing that perceptual content is a way of interacting with an environment. Experiments using “sensory substitution” are often highlighted in support of cortical deference. For example, use of the Bach-y– Rita's famous “tactile-visual–sensory-substitution” device (Bach-y-Rita, 1972) has been argued to lead to “quasi-visual” perceptions despite no direct activation of visual cortex (Hurley & Noë, 2003a). Cortical dominance is exemplified by phantom limb syndrome where, for example, stroking of the face may be felt as stroking of a phantom arm, reflecting activation of a region of somatosensory cortex previously dedicated to processing arm-related input (Ramachandran & Rogers-Ramachandran, 1996). Sensorimotor theory accounts for this instance of cortical dominance by noting that, because an amputated arm cannot be moved, there are no new SMCs to be learned which would modify the corresponding perceptual content. Accordingly, when new phantom-related SMCs are facilitated, as happens during “mirror-box” therapy (Ramachandran & Rogers-Ramachandran, 1996), dominance becomes deference and sensations are referred to the phantom limb.

Synesthesia represents a potentially more problematic case of cortical dominance for sensorimotor theory, because here (in contrast to phantom limbs) dynamical SMCs should eliminate interfering concurrent experiences, especially in cases like the alien colour effect (see earlier and Gray, 2003). As mentioned, Hurley and Noë admit the challenge posed by synesthesia. They scout a solution by noting that synesthetic phenomenology is distinct from normal phenomenology (though they do not focus on veridicality), and they suggest that concurrents may depend on a kind of “cortical dangling” occasioned by the involvement of the relevant cortical substrates (e.g., V4 in GCS) in other SMCs (Hurley & Noë, 2003a; Noë & Hurley, 2003).

PPSMC suggests a more nuanced approach to the distinction between deference, dominance, and the relation to synesthesia. On this view, perceptual content of all kinds is shaped by multiple interacting brain regions encoding HGMs which predict the dynamics of multimodal sensorimotor dependencies. Therefore, it is only in extreme or unusual cases in which content maps on to local neuronal substrates independent of behavioral context (dominance) or is fully determined by this context (deference). An emphasis on cortical networks rather than regions dilutes the deference/dominance distinction and is consistent with a range of experimental evidence mentioned earlier regarding modulation and elicitation of multimodal cortical activity following unimodal input (de Haas et al., 2013; Liang et al., 2013; Luo et al., 2010; Noesselt et al., 2010), as well as with observations of occipital cortical activation following training on a tactile substitution device in blind subjects (Ortiz et al., 2011). More generally, the balance between deference and dominance in shaping perceptual content will depend on the precision weighting of sensory and perceptual error signals in reshaping multimodal HGMs. The notion of “cortical dangling” in synesthesia can therefore be given focus as the operation of HGMs encoding inducer– concurrent associations and their corresponding sensorimotor dependencies, with the counterfactual poverty of concurrent-related sensorimotor dependencies underlying the phenomenological differences (i.e., subjective non-veridicality) between concurrents and inducers.⁸

PPSMC and phenomenological indeterminacy in vision

There is long-standing debate regarding the determinacy of visual experience. On one side is the notion that conscious scenes are fully determinate, in the sense that out of a vast repertoire of possibilities, just a single determinate perceptual scene is experienced at any one time, corresponding—to some degree—to a specific interpretation of an external state-of–affairs. It is not easy to find explicit positive statements of this view since it seems implicitly assumed by many contemporary theories of consciousness. On the other side is the idea that visual experience is intrinsically indeterminate, at least in the periphery, and that the apparent determinacy of perceptual content may be illusory (Cohen & Dennett, 2011; Kouider, de Gardelle, Sackur, & Dupoux, 2010). This is compatible with Dennett's argument that language encourages us to depict our mental states (including perceptual contents) as perhaps more determinate than they actually are (Dennett, 1991). It is also illuminated by the interesting example of “perceptual metamers” (Freeman & Simoncelli, 2011; see Figure 5), in which two images appear subjectively indistinguishable (when viewed under the right conditions) despite one including severe distortions. One way to understand this phenomenon is to suppose that visual content, at least in the periphery, is phenomenologically indeterminate. In line with this view, nearly 40 years ago Jerome Lettvin described peripherally crowded letters as having “lost form without losing crispness” and as “only seem[ing] to have a ‘statistical’ existence” (Lettvin, 1976).

Figure 5.

Perceptual “metamers”. A. Undistorted image. B. ‘Metamerized’ image. When viewed with central fixation (and at the appropriate distance) the images are subjectively indistinguishable, despite the metamerized image incorporating large distortions. Figure provided by J. Freeman and E. Simoncelli (see Freeman & Simoncelli, 2011).

Considering counterfactual HGMs in the context of PPSMC introduces a new angle into this debate. Madary has suggested that the “indeterminate implicit anticipations” encoded by counterfactually-rich generative models may plausibly be associated with a phenomenal indeterminacy in the periphery (Madary, 2012). This argument can be taken further by considering that perceptual presence (subjective veridicality) seems to be deeply associated with subjective determinateness. A tomato perceived at fixation is experienced as perceptually real, subjectively veridical, and also as highly determinate —it is not partly “there” and partly “not-there.” However, if perceptual presence depends on counterfactually-rich HGMs, then—perhaps counter– intuitively—subjectively determinate presence may depend constitutively on the deployment of a suite of indeterminate (i.e., probabilistic) counterfactual anticipations encoding SMCs. In other words, even (or especially) when we have an apparently phenomenologically determinate experience of a tomato, this determinacy may actually depend constitutively on a highly probabilistic counterfactually-loaded HGM replete with indeterminate implicit anticipations of the effects of possible actions on sensory signals and their precisions. This provides an unexpected response to Madary's Wittgensteinian question (and the title of his paper): “How would the world look if it looked as if it were encoded as an intertwined set of probability distributions?” Surprisingly, such a world might not “look” indeterminate at all. Rather, the implicit operation of counterfactually-rich HGMs might furnish exactly those properties which endow normal perceptual phenomenology with its apparent determinism, presence, and world-revealing nature.⁹

Open questions

PPSMC and its application to synesthesia exposes a rich seam of open questions. Some of these are identified below.

• A defining but poorly understood criterion for synesthesia is the consistency of the inducer-con– current relationship (Baron-Cohen et al., 1987; Eagleman et al., 2007; Rothen et al., 2013). Consistency is difficult to fake and for this reason is often used as an objective test for synesthesia (especially for grapheme-color synesthesia). Could consistency be explained via PPSMC by synesthetes deploying highly precise perceptual predictions?

• How can empirical tests target specifically the putative counterfactual elements of HGMs within PPSMC? What tests could interrogate the differences in counterfactual richness hypothesized to underlie the differences in perceptual presence elicited by inducers and concurrents? One possible approach may lie in “synesthesia training” experiments where associations are reinforced with systematic variation of conditional sensorimotor dependencies.

• Do all instances of synesthesia involve a lack of perceptual presence? Does the degree of perceptual presence associated with a concurrent change during the development of a synesthetic association? There is a need for increased breadth and depth in the phenomenological study of synesthesia to better inform and constrain neurocognitive models, including the present one.

• Can PPSMC be extended to fully account for the phenomenological differences between synesthesia (in its many guises), normal perception, and the range of other perceptual states discussed previously (hallucinations, dreaming, imagery, and the like)? A promising approach is to elaborate in more detail roles for precision weighting of prediction errors (Adams et al., 2013), with a focus on the precision of counterfactual predictions.

• Synesthesia can emerge either developmentally or following sensory deafferentation or brain injury (Ward, 2013). Does PPSMC—which has dealt implicitly with developmental synesthesia —also account for acquired cases? It could be that, following deafferentation, the precision of corresponding sensory prediction errors is drastically reduced, leading to bottom-up signals in this modality becoming susceptible to reshaping via cross-modal or amodal top-down predictions shaped during prior experience.

• Precision weighting in cortical PP is associated with neuromodulatory molecules such as dopa– mine and acetylcholine (and maybe oxytocin) which affect post-synaptic gain (Friston, 2009). What role do these molecules play in the development of natural synesthesia and might they also be able to induce transient synesthesia in neurotypics (like, for example, lysergic acid diethylamide; see Luke & Terhune, 2013)? Others have associated synesthesia with serotonin (Brang & Ramachandran, 2008) raising related issues of how this neurotransmitter influences perceptual inference.

• Can PPSMC fully account for the phenomenology of sensory substitution (Bach-y-Rita, 1972), for example the quasi-visual experience elicited by tactile-visual sensory substitution? From the perspective of cortical deference and dominance, tactile-vision and synesthesia are opposites. Others view sensory substitution as an acquired form of synesthesia (Ward & Wright, 2012). According to PPSMC, both engage predictive models of sensory inputs with distinct combinations of sensory prediction errors, top-down predictions, precision weightings, and counterfactual richness.

• How does PPSMC relate to the “global” sense of conscious presence that is selectively lost in conditions like depersonalization and derealization disorders (Sierra & David, 2011)? I have previously suggested that (global) presence has to do with the precision of interoceptive predictions (i.e., predictions about the causes of interoceptive signals) within a framework of “interoceptive predictive coding” or “interoceptive inference” underpinning emotion and affect (Seth, 2013; Seth, Suzuki, & Critchley, 2011). One approach may be to consider that, in virtue of subjective veridicality, HGMs may recruit interoceptive responses regarding the affective properties of the perceived object, thereby contributing to a global sense of presence.

CONCLUSIONS

This paper has outlined a predictive processing theory of sensorimotor contingencies. PPSMC proposes that the generative models underlying perceptual content incorporate counterfactual probabilistic representations of hidden causes linking fictive sensory signals (and their expected precisions) to possible actions. This gives mechanistic focus to the notion of mastery of sensorimotor contingencies as described within sensorimotor theory, and explains subjective veridicality or “perceptual presence” in terms of the counterfactual richness of the underlying generative models. Applied to synesthesia, the theory accounts for the subjective non-veridicality of concurrents by counterfactual poverty: Concurrents are experienced as lacking in perceptual presence because of counterfactually impoverished generative models of how hidden causes would modify sensory signals (and associated precisions) in response to actions. This is because, as compared to inducers (or the world generally), there is no corresponding rich world-related statistical structure for any such model to learn.

The theory is compatible with associative accounts of developmental synesthesia (whether neonatal or learning-based), and it accommodates the relations between synesthesia, normal perception, and other perceptual and quasi-perceptual states including dreaming, imagery, hallucinations, and the like. The theory also resolves the challenges posed by synesthesia to enactive or sensorimotor accounts of perception by proposing that the local substrates of perception (which remain wholly in the brain) implement generative models that learn both actual and counterfactual sensorimotor contingencies. Finally, the theory suggests a response to the interesting question posed by predictive perception of “what would the world look like if it looked as if it were encoded as a set of intertwined probability distributions” (Madary, 2012). If these distributions are largely counterfactual, the answer may be that the world would look as if composed of externally existing objects with determinate features. Much as it actually seems to.