A touchy subject: advancing the modulated visual pathways account of altered vision near the hand

Abstract

A growing body of evidence demonstrates that human vision operates differently in the space near and on the hands; for example, early findings in this literature reported that rapid onsets are detected faster near the hands, and that objects are searched more thoroughly. These and many other effects were attributed to enhanced attention via the recruitment of bimodal visual-tactile neurons representing the hand and near-hand space. However, recent research supports an alternative account: stimuli near the hands are preferentially processed by the action-oriented magnocellular visual pathway at the expense of processing in the parvocellular pathway. This Modulated Visual Pathways (MVP) account of altered vision near the hands describes a hand position-dependent trade-off between the two main retinal-cortical visual pathways between the eye and brain. The MVP account explains past findings and makes new predictions regarding near-hand vision supported by new research.

One of the important discoveries in cognitive neuroscience is that our capacity for spatial representation does not consist of a single mechanism. Despite our awareness of a unitary spatial layout, there exist multiple neural maps of space, each anchored to a different frame of reference [1]. This means that distinct populations of cells represent an object location in eye-centered, head-centered, body-centered, and hand-centered frames of reference [2,3]. The hand-centered representation of space has been of particular interest given the relevance of this space for immediate action. Indeed, over the past decade, accumulating evidence suggests that the near-hand space has a unique status in sensory and perceptual processing [4,5]. The present review is concerned with the modulations in visual processing in the near-hand space; specifically, we first establish the context of this new literature, and then advance a new theory to account for altered vision in near-hand space.

The space within reach as a historical precursor to near-hand space

The foundational ideas for the current research on visual processing in near-hand space originally emerged from neuroscientific and neuropsychological studies on nearspace (peripersonal space) that showed how perception operates differently for stimuli within reaching distance [6]. This perceptual demarcation was convincingly illustrated by research on monkeys showing that periarcuate neurons respond selectively to items within or beyond reach [7]. Another population of neurons in the macaque respond to both tactile stimuli on the hands and visual stimuli near the hands [3,8]. The receptive fields of these neurons are yoked to the hands, providing the neural basis for later claims that these neurons are involved in enhanced attention near the hands [9]. Importantly, this neural tuning to action-relevant spaces is flexible, such that when monkeys are given a tool to increase the length of their reach, the receptive fields of their reach-selective neurons rapidly adjust in size to accommodate the new reachable space [10].

The functional specificity of spatial representation at the neural level was integral to the supposition that the space near the hands might also be perceptually unique. The neurophysiological data from monkeys were corroborated by human neuropsychological cases where patients experienced hemineglect that was selectively confined to either the nearspace [11] or the space beyond reach [12], thus demonstrating a dissociation between the near and far maps of space. When the hemineglect is bound to nearspace, the neglected area can be expanded by using a tool to interact with the distant environment [13]. In contrast, another case study has shown that hemineglect for the space beyond reach can be reduced by reaching out with a tool [14]. These studies are important because they show how the area of attentional neglect can be functionally determined. Perhaps the most influential demonstration of the importance of hand posture on attentional processing in the neuropsychological literature comes from the case study of patient WM, who experienced left neglect after a stroke ([15], although see [16]). Remarkably, his left hemifield visual impairment was attenuated when reaching through that space; when stimuli were out of reach, there was no effect.

When examining this literature, it is important not to conflate nearspace with near-hand space. As we will see, researchers built on the nearspace literature describing perception for stimuli within reach to describe a new visual demarcation in the space near the hands. The idea that vision is altered near the hands is predicated on the existence of different spatial representations corresponding to different affordances (e.g. separate representations for reachable, graspable, or walkable space).

Early evidence for altered attention near the hands

The neuroscientific and neuropsychological studies on nearspace established the presence of multiple spatial representations corresponding to different actions. This literature provided cognitive and perceptual researchers the motivation to investigate vision near the hands in normal humans (for review see [17] and [18]). The first major study in this literature demonstrated that detection of peripheral targets in a spatial cueing paradigm [19] is faster when they appear on the side of the display closest to an outstretched hand [9]. This effect only occurred on the palm side of the hand, and attenuated with distance [20], consistent with the theory that near-hand space is prioritized for attentional processing via the recruitment of bimodal visual-tactile neurons responding to stimuli near the hand. This same mechanism was adopted to account for the finding that attention disengages slower from stimuli near the hands [21]. In that study, visual search slopes were steeper, inhibition of return was greater, and the attentional blink effect was exacerbated near the hands – all consistent with delayed disengagement. To complicate matters, singleton distractors near the hand produced a larger cost without any evidence for faster target detection near the hand [22].

Subsequent research showed performance on a diverse array of attention-demanding tasks is altered in the space near the hands; for example, stimuli near the hands seemed to enjoy a more robust representation in visual working memory [23], flanking distractors outside the hands produce smaller interference with vision at fixation [24], and shifting attention between global and local stimulus features is markedly slower near the hands [25]. Hand proximity affects a truly diverse array of abilities, from the low-level perceptual task of figure-ground segregation [26] to inducing deficits in reading [27].

Problems with early accounts of altered vision near the hands

Although no existing reviews specifically claim attention can account for all of these findings, altered visuospatial attention (e.g., rapid selection or delayed disengagement) caused by hand proximity is implicated in every finding in the preceding section. Against this background, our lab recently considered an alternative account of altered vision near the hands [28]. In arguing for the need for an alternative account, Gozli et al. identified three major inconsistencies with an account that is solely based on attention. First, in attentional orienting tasks, the effect of hand-proximity and the effect of a peripheral cue are additive ([9,20]; see also [29]), which suggests that the two effects are driven by distinct mechanisms [30]. Second, even when the location of the impending target is known with certainty, targets appearing on the hand are detected faster than targets on other control objects [31]. Given that attentional orienting is unnecessary in case of target location certainty [32], something other than attention must facilitate detection near the hand. Third, the finding that hand proximity modulates figure-ground segregation [26] is difficult to account for in terms of attention, given that figure-ground segregation occurs pre-attentively [33,34]. These observations, paired with the myriad effects of altered vision near the hands, make it difficult for any single attentional theory to account for all of these results.

The modulated visual pathways account of altered vision near the hands

The unifying account of altered vision near the hands supported in this review argues that differences in visual cognition between near-and far-hand postures can be explained by the balance of processing between the two principal retinocortical pathways. The magnocellular and parvocellular pathways function in parallel, and diverge early, starting at the retinal ganglion cells [35,36]. They project to separate areas of the thalamus and occipital cortex. Critically, the action-oriented dorsal pathway receives input primarily from the M pathway [37,38]. This anatomical connection was the basis for the prediction that action-relevant stimuli, such as stimuli appearing near the hands, would tip the balance of processing toward the M pathway. The other critical premise is that the M and P pathways are mutually inhibitive, and that biasing one pathway should impair the other [39–41]. Stimuli near the hands, then, should boost activity in the M pathway relative to the P pathway. This Modulated Visual Pathways (MVP) account is appealing because it can parsimoniously account for altered vision near the hands while making new predictions (see Figure 1).

Figure 1.

Summary of the MVP account of altered vision near the hands. (A) Shows the typical near-versus far-hand manipulation employed by researchers to examine vision near the hands. (B) Illustrates the predicted effect these hand postures would have on the primarily magnocellular dorsal pathway (top) and the primarily parvocellular ventral pathway (bottom). A green arrow indicates preferential processing. (C) Lists some of the features and functioned performed by the magnocellular and parvocellular pathways and their projections. These are the visual faculties expected to be modulated by hand posture according to the MVP account.

The first prediction to be tested was derived from the observation that the M and P pathways carry different information. The M cells carry a transient signal, and are much faster than their P counterparts [42]. Conversely, P cells have much narrower receptive fields, which gives them a much higher spatial acuity [43]. If hand proximity biases the M pathway at the expense of the P pathway, then stimuli near the hands should be perceived with higher temporal sensitivity, and stimuli far from the hands should be perceived with higher spatial sensitivity. This exact trade-off was observed in temporal and spatial gap detection tasks near and far from the hands [28].

Having found initial evidence for the MVP account, we tested additional predictions derived from the expectation that hand proximity would bias the M pathway; for example, the M pathway has a demonstrated preference for low spatial frequency (LSF) images [44], and is involved in rapid gist perception [45]. Accordingly, Chan et al. [46] reasoned that LSF images appearing near the hands should be identified especially quickly. Across a series of experiments, they had observers view objects displayed in LSF or high spatial frequency (HSF) with their hands near or far (Experiment 1) or with the palms facing the stimuli or facing out (Experiment 2), and made a categorical judgment on object size (bigger or smaller than a shoebox). Results showed a response time (RT) advantage for LSF images with the hands near but not far from the stimuli, and only with palms facing the stimuli. This result is important because an attentional account of altered vision near the hands would not predict an advantage for LSF compared to HSF information near the hands.

Although the segregation between M and P pathways occurs early in the visual processing stream, increased contribution of one pathway over the other has consequences beyond early vision. Notably, the M and P pathways play an important role in our ability to bind and track objects over space and time. Below, we describe four studies from our lab and colleagues showing that the MVP account predicts specific effects on object perception near the hands.

First, increased magnocellular contribution should affect the temporal aspect of object representations. Specifically, if stimuli near the hands enjoy enhanced temporal sensitivity due to a magnocellular enhancement, then hand proximity should reduce errors in temporal segmentation. Due to insufficient temporal resolution, two distinct objects appearing close in time might be temporally fused into a single whole. Increasing temporal sensitivity through hand proximity should reduce those errors. Object substitution masking (OSM) is a failure to perceive objects presented within a temporally trailing four-dot mask [47]. This masking is caused by a failure of object segmentation due to the rapidly updated percept of the empty, trailing four-dot pattern [48]. When observers held their hands around a display during the OSM paradigm, masking was reduced, consistent with the expectation that hand proximity should increase temporal segmentation [49]. However, it is worth noting that since OSM is attenuated by focused attention to the masked stimulus [47], this study cannot disambiguate the MVP and attentional accounts of altered vision near the hands. In a later study, Goodhew and colleagues presented OSM stimuli with or without pulsed luminance pedestals [50]. This presentation method is known to saturate the M pathway, so they could compare OSM with P processing only, or with the M and P pathways activated [51]. They found that OSM was enhanced under pulsed pedestal presentation, confirming a critical role for the M pathway on temporal object segmentation [52]. This corroborates the earlier finding that hand proximity increases temporal segmentation by biasing the M pathway [49].

To fully establish the role of the M pathway in object perception near the hands, Goodhew et al. [52] employed the object correspondence paradigm, which tests how object perception persists when a moving object is temporarily occluded from vision. This object correspondence effect prevents us from losing track of occluded objects, and it depends on the object’s spatiotemporal trajectory and surface features before and after occlusion [53]. Because the magnocellular pathway is insensitive to colour, object correspondence using colour surface features should be impaired near the hands. Indeed, observers were slower to use colour information to detect changes to objects after brief occlusion near the hands. This result firmly demonstrates the modulation of the M pathway for object perception near the hands. Importantly, the MVP account makes a clear prediction about the P pathway and its role in object perception as well: due to the M and P pathways’ mutually inhibitive nature, hand proximity should impair the P pathway, leading to selective impairments in object perception.

In addition to the impact on the temporal aspect of object representation, an increased magnocellular contribution will impact the manner in which multi-featured objects are represented. Because the P pathway projects primarily to the ventral stream, and the ventral stream plays an important role in forming representations of complex objects [54,55], having the hands near the display should impair the binding of features into objects. When an object is first perceived, the disconnected features are bound into a unified object file [56]. One way to measure this feature binding is with the object-specific preview task [56]. In this task, observers are primed with two placeholders labeled with different symbols. The symbols then offset, the placeholders move, and then one of the symbols reappears for participants to identify as quickly as possible. Critically, the symbol can appear in the original placeholder (repeat condition), it can appear in the other placeholder (repeat – different frame condition), or it can be entirely new. Symbols are identified faster when they reappear in the original placeholder compared to when it appears in the other placeholder, presumably because they have already been bound together into one object file [56]. When the symbols reappear in a new placeholder, responses are slower because a new object file must be created. Gozli et al. [57] utilized this paradigm with the expectation that an impaired P pathway would erase or attenuate the advantage of viewing a symbol in its original placeholder, because the object file would not be fully formed. When subjects perform this task far from the hands, they exhibit the same-object file advantage. However, consistent with the MVP prediction, performing the task near the hands erased this advantage completely, suggesting that symbols were not bound to their placeholders in the near-hand space [57].

Finally, hand proximity modulates attentional capture by gestalt grouping (Huffman, Gozli, Welsh and Pratt, “Hand position influences perceptual grouping”, in preparation). Sudden onset objects formed by gestalt grouping principles such as good continuation are known to capture attention [58]. Because perceptual grouping is primarily a ventral stream process [59] receiving input from the P pathway, this object-based attentional capture should be greatest when the hands are far from the display. Indeed, observers exhibit slower responses to invalidly cued targets appearing inside a gestalt object (attentional capture) when the hands were far from the display, but not near the display. This effect is predicted by the MVP account but not the attentional account of near-hand effects on vision.

Although the MVP account is deeply rooted in well-established visual neuroscience, all of the reviewed evidence has thus far been behavioural. However, a recent study has shown that visually evoked potentials (VEPs) during selective attention near and far from the hands conform to predictions made by the MVP account [60]. In this study, participants attended to small red and blue dots, while ignoring surrounding checkerboard stimuli that alternated in a time-locked pattern with the VEPs. The scanning data therefore reflected the processing of the irrelevant distractor checkerboards. The P200 component of the VEP was significantly curbed for stimuli near the hands, which is reflective of impairment to feature-based processing [61]. This decrement to feature-based processing is consistent with difficulties in forming object files near the hands compared to far from the hands [57].

Other researchers outside our lab have also begun to report experiments derived from the MVP account. For example, observers are highly sensitive to LSF Gabor patches near the hands and HSF patches far from hands, consistent with the spatial frequency preferences of the M and P pathways ([62] Experiment 1). The effect of higher sensitivity to LSF stimuli near the hands is completely erased by diffuse red light ([62] Experiment 2), which is known to upset magnocellular processing [44,63,64]. This effect is totally unpredictable by any account other than MVP. Finally, these researchers showed that delayed attentional disengagement is erased under diffuse red light, but not under neutral green light ([62] Experiment 3). This last experiment strongly implicates the magnocellular pathway in an effect previously attributed to altered attention near the hands [18,21].

The success of MVP in accounting for altered vision near the hands does not rule out the possibility that attentional enhancement via the recruitment of bimodal visual-tactile representations is involved in the effects described in this literature. Recently Adam et al. [65] supported this view by demonstrating that letter recognition accuracy was dynamically affected by hand proximity. Letters were better recognized when they appeared closest to a hand moving through space. Furthermore, performance attenuated with distance when the hand was static. That the effect was yoked to the hand and attenuated with distance is consistent with both an attentional account and the MVP account of altered vision near the hands.

It should also be noted that the MVP account does not explain all hand-related effects on vision. We recently reported effects on altered vision for stimuli on the hands, as opposed to near the hands [66]. In a spatial cueing paradigm, like the one used to investigate attentional prioritization near the hand [9,20], observers consistently exhibited interactions between hand position (whether stimuli appeared on, near, or far from the hands) and the validity of cues; observers were very slow to orient attention on, to, or from the hands or tools in use [66]. This effect clearly implicates an interaction with attention for the perception of stimuli on the hands, which cannot be easily accounted for by MVP. This new effect likely reflects the different functional status stimuli in contact with the body have over stimuli near the body, and it is not inconsistent with the view that the recruitment of visual-tactile representations enhances attention. Interestingly, we failed to observe a typical object-based attention effect on the hands [67], which is consistent with MVP’s prediction that object-based attentional capture, a P pathway faculty, should not occur near the hands (Huffman et al., in preparation). This study therefore serves as a reminder that multiple mechanisms can account for some effects.

Furthermore, as Bush and Vecera [68] recently showed, a distinction should be made between two types of hand-proximity manipulation. The first type occurs in situations when either the hand-proximal or the hand-distal (but not both) are potential target locations [9]. The second type occurs when the hand-proximal and hand-distal are both potential target locations [21,28]. Bush and Vecera argued that the MVP account might only apply to the former case, whereas the latter manipulation might induce an attentional imbalance in favour of the hand-proximal space. It is possible that with a single-hand manipulation, the hand simply serves as a landmark for attention, increasing the visual salience in one hemifield. In this sense, it is difficult to distinguish the effect of a hand from the effect of any other salient object. Accordingly, a genuine effect of hand-proximity should survive even when the observer’s hands are hidden from view [69]. Nevertheless, as pointed out by Bush and Vecera [68], the distinction between the two types of hand proximity deserves further study.

Conclusion

In summary, the literature on altered attention near the hands is vastly improved and clarified by the advent of the MVP. Where the early accounts had difficulty reconciling a diverse array of near-hand effects on vision, the MVP succeeded. The MVP account is parsimonious in that it calls on distinctions between parallel, low-level visual pathways to explain complex visual behaviour. It makes clear, falsifiable predictions that are at odds with the attentional account of near-hand vision. There may still be a role for attention in some or all of these effects, but the MVP has proven to be a mainstay in this new and thriving literature.