Numerous studies have show that extra-retinal signals can disambiguate motion information created by movements of the eye or head [1]. We show a new form of cross-modal sensory integration in which the kinesthetic information generated by active hand movements essentially captures ambiguous visual motion information. Several previous studies have shown that active movement can bias observers' percepts of bistable stimuli [2-4]; however, these effects seem best explained by attentional mechanisms [5]. We show that kinesthetic information can change an otherwise stable perception of motion, providing evidence of genuine fusion between visual and kinesthetic information. The experiments take advantage of the aperture problem, in which the motion of a one-dimensional grating pattern behind an aperture, while geometrically ambiguous, appears to move stably in the grating normal direction. When actively moving the pattern, however, the observer sees the motion to be in the hand movement direction.
Subjects viewed grating patterns displayed on a monitor through a mirror. Behind the mirror, subjects moved a cube on a tabletop that was co-aligned optically with the visual display (Fig. 1A). We measured the motion of the cube in real time and the grating was rendered to move with subjects' hands as if mounted on the top face of the cube. Informal initial demonstrations showed that when subjects moved the cube in a curvilinear path, they perceived the grating to move along a similar curvilinear path; observers passively viewing the grating perceived it to move along a line perpendicular to the grating, albeit with changing speeds over time (Fig. 1B). We performed two experiments to quantify the influence of kinesthesis on motion perception. Subjects, without seeing their hand or the cube, moved the grating along approximately straight-line paths in self-chosen random directions and reported the perceived grating motion direction. Each block in Experiment 1 contained randomly intermixed, equal numbers of trials with circular or square apertures. The orientations of the gratings were randomly chosen from (0°, 40°, 80°, 120°, 160°) away from the horizontal. Square apertures were always oriented 45° relative to the grating normal direction.
When the aperture was circular, subjects reported the pattern moving largely in the hand movement direction (Fig. 1C), suggesting that visual motion signals were modulated by kinesthetic signals from the hand. When the aperture was square, subjects' judgments showed more variability, with subjects appearing to adopt different strategies in different trials. While for most subjects, kinesthetic modulation happened on some trials, on others it did not, with judgments sometimes matching the motion of either set of terminators where the grating intersected the aperture boundary (Fig. 1D). The multiple percepts were consistent with the findings in [6], where the percepts of similar stimuli were multi-stable, changing from the grating normal direction to those of the terminators. On the one hand, subjects' judgments for the square aperture stimuli show that kinesthetic information was not strong enough to resolve the multi-stability. On the other hand, it indicates that subjects were reporting their visual motion percepts as instructed rather than simply the perceived hand motion.
To further test that subjects' perceptual reports in Experiment 1 reflected true sensory fusion, we ran a control experiment in which we added a 200ms delay between the visual motion and subjects' hand motion. Eight new subjects were given the same instructions as in Experiment 1 and were not told about the delay. Only circular apertures were used. Seven subjects' perceived motion directions clustered around the grating normal directions, independent of hand movement direction (Fig. 1E).
Previous research on the effects of active movement on motion perception has shown that hand movement direction biases subjects' percepts of bi-stable motion stimuli (e.g. counter-phase flickering gratings) to one or another of the bi-stable percepts [2]. These studies did not enforce consistency between the visual motions and the observer-generated movements and further experiments suggest that the effects were due to attentional modulation [5]. Though we cannot exclude any role of attention in our study, several aspects of the stimuli and the results suggest that the percepts reflect true fusion of visual and kinesthetic information. First, motion percepts for passively viewed gratings through a circular aperture are not bi-stable or ambiguous – they appear reliably to move in the grating normal direction. While attention can modulate the perceived motion of a grating behind a square aperture by focusing on the terminators, when attention is focused on the grating itself, one cannot willfully change the direction of perceived motion. Thus, the kinesthetic information in Experiment 1 dramatically changes what is otherwise a reliable percept of motion in the perpendicular direction. Second, the kinesthetic modulation shown in Experiment 1 is subject to a strong temporal congruency constraint, a factor known to be crucial in other multi-sensory integration phenomena [7].
Some subjects experienced changes in their percepts between the grating normal and hand movement directions within one trial. We instructed subjects who reported this bi-stability to report the more dominant percept. The malleability of the effect depends on individuals and on specific stimulus parameters. For example, larger apertures lead to more consistent kinesthetic capture, perhaps because of a decreased influence of terminator motion. When the perceived motion of the grating switches to the grating normal direction the hand no longer appears to be physically moving the grating; thus, we speculate that differences in the strength / stability of kinesthetic capture are related to sensory evidence and implicit priors for subjects' causal inferences about the relationships between hand movement and the visual stimulus.
The results are consistent with Bayesian theories of perceptual inference that explain the perpendicular motion percept as a result of integrating visual motion information with an internalized prior for slow speeds [8]. When kinesthetic information about the motion of the stimulus is available, it can override such prior biases. While we can only speculate on the physiology involved in kinesthetic and visual motion integration, Blake, et al. did find that haptic motion information from touch activated the MT complex [3] and a recent study by Lunghi et al. [9] showed that haptic signals could resolve binocular rivalry. Both results suggest that the integration may happen as early as the first stage of visual processing known to be causally linked to visual motion percepts [10].
Supplementary Material
Acknowledgments
This work was supported by NIH grant R01 EY017939 to the second author. The authors would like to thank Leslie Richardson for her help running subjects.
References
- 1.Wexler M, van Boxtel JJ. Depth perception by the active observer. Trends Cogn Sci. 2005;9:431–438. doi: 10.1016/j.tics.2005.06.018. [DOI] [PubMed] [Google Scholar]
- 2.Yabe Y, Taga G. Treadmill locomotion captures visual perception of apparent motion. Exp Brain Res. 2008;191:487–494. doi: 10.1007/s00221-008-1541-3. [DOI] [PubMed] [Google Scholar]
- 3.Blake R, Sobel KV, James TW. Neural synergy between kinetic vision and touch. Psychol Sci. 2004;15:397–402. doi: 10.1111/j.0956-7976.2004.00691.x. [DOI] [PubMed] [Google Scholar]
- 4.Ishimura G, Shimojo S. Voluntary Action Captures Visual Motion. Investigative Ophthalmology & Visual Science. 1994;35 [Google Scholar]
- 5.Wohlschläger A. Visual motion priming by invisible actions. Vision Research. 2000;40:925–930. doi: 10.1016/s0042-6989(99)00239-4. [DOI] [PubMed] [Google Scholar]
- 6.Castet E, Charton V, Dufour A. The extrinsic/intrinsic classification of two-dimensional motion signals with barber-pole stimuli. Vision Research. 1999;39:915–932. doi: 10.1016/s0042-6989(98)00146-1. [DOI] [PubMed] [Google Scholar]
- 7.Soto-Faraco S, Kingstone A, Spence C. Multisensory contributions to the perception of motion. Neuropsychologia. 2003;41:1847–1862. doi: 10.1016/s0028-3932(03)00185-4. [DOI] [PubMed] [Google Scholar]
- 8.Weiss Y, Simoncelli EP, Adelson EH. Motion illusions as optimal percepts. Nat Neurosci. 2002;5:598–604. doi: 10.1038/nn0602-858. [DOI] [PubMed] [Google Scholar]
- 9.Lunghi Claudia, B P, Concetta Morrone M. Touch disambiguates rivalrous perception at early stages of visual analysis. Curr Biol. 2010;20:R143–R144. doi: 10.1016/j.cub.2009.12.015. [DOI] [PubMed] [Google Scholar]
- 10.Britten KH, Shadlen MN, Newsome WT, Movshon JA. Responses of neurons in macaque MT to stochastic motion signals. Vis Neurosci. 1993;10:1157–1169. doi: 10.1017/s0952523800010269. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.