Skip to main content
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2011 May 25.
Published in final edited form as: Curr Biol. 2010 May 25;20(10):R436–R437. doi: 10.1016/j.cub.2010.03.053

Kinesthetic information disambiguates visual motion signals

Bo Hu 1, David C Knill 1
PMCID: PMC2937283  NIHMSID: NIHMS227211  PMID: 20504749

Numerous studies have show that extra-retinal signals can disambiguate motion information created by movements of the eye or head [1]. We show a new form of cross-modal sensory integration in which the kinesthetic information generated by active hand movements essentially captures ambiguous visual motion information. Several previous studies have shown that active movement can bias observers' percepts of bistable stimuli [2-4]; however, these effects seem best explained by attentional mechanisms [5]. We show that kinesthetic information can change an otherwise stable perception of motion, providing evidence of genuine fusion between visual and kinesthetic information. The experiments take advantage of the aperture problem, in which the motion of a one-dimensional grating pattern behind an aperture, while geometrically ambiguous, appears to move stably in the grating normal direction. When actively moving the pattern, however, the observer sees the motion to be in the hand movement direction.

Subjects viewed grating patterns displayed on a monitor through a mirror. Behind the mirror, subjects moved a cube on a tabletop that was co-aligned optically with the visual display (Fig. 1A). We measured the motion of the cube in real time and the grating was rendered to move with subjects' hands as if mounted on the top face of the cube. Informal initial demonstrations showed that when subjects moved the cube in a curvilinear path, they perceived the grating to move along a similar curvilinear path; observers passively viewing the grating perceived it to move along a line perpendicular to the grating, albeit with changing speeds over time (Fig. 1B). We performed two experiments to quantify the influence of kinesthesis on motion perception. Subjects, without seeing their hand or the cube, moved the grating along approximately straight-line paths in self-chosen random directions and reported the perceived grating motion direction. Each block in Experiment 1 contained randomly intermixed, equal numbers of trials with circular or square apertures. The orientations of the gratings were randomly chosen from (0°, 40°, 80°, 120°, 160°) away from the horizontal. Square apertures were always oriented 45° relative to the grating normal direction.

Figure 1.

Figure 1

Experimental set-up and results. (A) Subjects viewed a one-dimensional grating pattern appearing on the display that was co-aligned optically with the top of a cube held by subjects' right hand against a fronto-parallel table. A 12-degree aperture surrounding the grating was rendered binocularly on a plane 10mm above the visual pattern. The motion of the cube was measured by an Optotrak 3020 system (NDI, ON, Canada) and the grating was rendered at the same velocity as the hand movement. After viewing the pattern for 3500 ms, subjects oriented a line probe to indicate the perceived direction of grating motion. (B) When subjects moved a grating along the direction indicated by the green arrow, they saw the motion of the grating in that direction, whereas a second observer saw it moving unambiguously along the grating normal direction, indicated by the red arrow, because of the aperture problem. (C—E) Subject's perceived motion directions as a function of hand movement direction. The top row illustrates the stimuli used in corresponding plots. Each point was from one trial. The angles were relative to the grating normal direction. (C) One typical subject's data in trials with circular apertures in Experiment 1. The points lay along the line of unity slope, suggesting that the visual motion signals were modulated by kinesthetic signals from the hand movement. Seven subjects showed similar results (see Supplemental Data). (D) The same subject's data in trials with square apertures in Experiment 1. The points were along either the unity line or the horizontal line at 45°, one of the two directions of terminator motion. The results varied across subjects (see Fig. S1). (E) One subject's data in Experiment 2. The points clustered around the abscissa, indicating little kinesthetic modulation when there was a delay between the visual motion and hand motion. Seven of the eight subjects showed the same results. One subject reported perceived motion in the direction of hand movement (see Supplemental Data). (F) Mean and SEM of the influence of the kinesthetic signal from the hand movement. The measure of influence that we used combines the estimated proportion of trials in which subjects' motion percepts were modulated by kinesthetic information and the weight given to kinesthetic information on those trials (see Supplemental Data). A value of 1 means that subjects' judgments matched the hand movement direction on all trials. The influence was close to 1 when the aperture was circular, showing that the perceived motion was largely determined by the kinesthetic signals. The influence was smaller and the variability greater when the aperture was square, because of subjects' mixed strategies in motion judgment. The influence was close to 0, indicating vision dominance, in the delay condition – the apparent small non-zero weight being due almost entirely to one subject who reported hand movement direction on all trials (see Fig. S2).

When the aperture was circular, subjects reported the pattern moving largely in the hand movement direction (Fig. 1C), suggesting that visual motion signals were modulated by kinesthetic signals from the hand. When the aperture was square, subjects' judgments showed more variability, with subjects appearing to adopt different strategies in different trials. While for most subjects, kinesthetic modulation happened on some trials, on others it did not, with judgments sometimes matching the motion of either set of terminators where the grating intersected the aperture boundary (Fig. 1D). The multiple percepts were consistent with the findings in [6], where the percepts of similar stimuli were multi-stable, changing from the grating normal direction to those of the terminators. On the one hand, subjects' judgments for the square aperture stimuli show that kinesthetic information was not strong enough to resolve the multi-stability. On the other hand, it indicates that subjects were reporting their visual motion percepts as instructed rather than simply the perceived hand motion.

To further test that subjects' perceptual reports in Experiment 1 reflected true sensory fusion, we ran a control experiment in which we added a 200ms delay between the visual motion and subjects' hand motion. Eight new subjects were given the same instructions as in Experiment 1 and were not told about the delay. Only circular apertures were used. Seven subjects' perceived motion directions clustered around the grating normal directions, independent of hand movement direction (Fig. 1E).

Previous research on the effects of active movement on motion perception has shown that hand movement direction biases subjects' percepts of bi-stable motion stimuli (e.g. counter-phase flickering gratings) to one or another of the bi-stable percepts [2]. These studies did not enforce consistency between the visual motions and the observer-generated movements and further experiments suggest that the effects were due to attentional modulation [5]. Though we cannot exclude any role of attention in our study, several aspects of the stimuli and the results suggest that the percepts reflect true fusion of visual and kinesthetic information. First, motion percepts for passively viewed gratings through a circular aperture are not bi-stable or ambiguous – they appear reliably to move in the grating normal direction. While attention can modulate the perceived motion of a grating behind a square aperture by focusing on the terminators, when attention is focused on the grating itself, one cannot willfully change the direction of perceived motion. Thus, the kinesthetic information in Experiment 1 dramatically changes what is otherwise a reliable percept of motion in the perpendicular direction. Second, the kinesthetic modulation shown in Experiment 1 is subject to a strong temporal congruency constraint, a factor known to be crucial in other multi-sensory integration phenomena [7].

Some subjects experienced changes in their percepts between the grating normal and hand movement directions within one trial. We instructed subjects who reported this bi-stability to report the more dominant percept. The malleability of the effect depends on individuals and on specific stimulus parameters. For example, larger apertures lead to more consistent kinesthetic capture, perhaps because of a decreased influence of terminator motion. When the perceived motion of the grating switches to the grating normal direction the hand no longer appears to be physically moving the grating; thus, we speculate that differences in the strength / stability of kinesthetic capture are related to sensory evidence and implicit priors for subjects' causal inferences about the relationships between hand movement and the visual stimulus.

The results are consistent with Bayesian theories of perceptual inference that explain the perpendicular motion percept as a result of integrating visual motion information with an internalized prior for slow speeds [8]. When kinesthetic information about the motion of the stimulus is available, it can override such prior biases. While we can only speculate on the physiology involved in kinesthetic and visual motion integration, Blake, et al. did find that haptic motion information from touch activated the MT complex [3] and a recent study by Lunghi et al. [9] showed that haptic signals could resolve binocular rivalry. Both results suggest that the integration may happen as early as the first stage of visual processing known to be causally linked to visual motion percepts [10].

Supplementary Material

Figure S1. Results of individual subjects in Experiment 1. (A-B) Subject's perceived motion directions as a function of hand movement direction in the trials with circular apertures (A) and square apertures (B). The angles are normalized with regard to the grating normal directions—zero degrees would correspond to a perceived motion direction perpendicular to the gratings. The icon on the top-left corner shows the selected model of each subject. (C) Kinesthetic influence of the trials with circular and square apertures, respectively, in Experiment 1. Error-bars are 1-SE. (D) Histograms of subjects' hand movement directions in Experiment 1. Each radial line represents the direction in one trial.

Figure S2. Results of individual subjects in Experiment 2. (A) Subject's perceived motion directions as a function of hand movement direction. (B) Kinesthetic influence in Experiment 2. Error-bars are 1-SE. (C) Histograms of subjects' hand movement directions in Experiment 2.

Acknowledgments

This work was supported by NIH grant R01 EY017939 to the second author. The authors would like to thank Leslie Richardson for her help running subjects.

References

  • 1.Wexler M, van Boxtel JJ. Depth perception by the active observer. Trends Cogn Sci. 2005;9:431–438. doi: 10.1016/j.tics.2005.06.018. [DOI] [PubMed] [Google Scholar]
  • 2.Yabe Y, Taga G. Treadmill locomotion captures visual perception of apparent motion. Exp Brain Res. 2008;191:487–494. doi: 10.1007/s00221-008-1541-3. [DOI] [PubMed] [Google Scholar]
  • 3.Blake R, Sobel KV, James TW. Neural synergy between kinetic vision and touch. Psychol Sci. 2004;15:397–402. doi: 10.1111/j.0956-7976.2004.00691.x. [DOI] [PubMed] [Google Scholar]
  • 4.Ishimura G, Shimojo S. Voluntary Action Captures Visual Motion. Investigative Ophthalmology & Visual Science. 1994;35 [Google Scholar]
  • 5.Wohlschläger A. Visual motion priming by invisible actions. Vision Research. 2000;40:925–930. doi: 10.1016/s0042-6989(99)00239-4. [DOI] [PubMed] [Google Scholar]
  • 6.Castet E, Charton V, Dufour A. The extrinsic/intrinsic classification of two-dimensional motion signals with barber-pole stimuli. Vision Research. 1999;39:915–932. doi: 10.1016/s0042-6989(98)00146-1. [DOI] [PubMed] [Google Scholar]
  • 7.Soto-Faraco S, Kingstone A, Spence C. Multisensory contributions to the perception of motion. Neuropsychologia. 2003;41:1847–1862. doi: 10.1016/s0028-3932(03)00185-4. [DOI] [PubMed] [Google Scholar]
  • 8.Weiss Y, Simoncelli EP, Adelson EH. Motion illusions as optimal percepts. Nat Neurosci. 2002;5:598–604. doi: 10.1038/nn0602-858. [DOI] [PubMed] [Google Scholar]
  • 9.Lunghi Claudia, B P, Concetta Morrone M. Touch disambiguates rivalrous perception at early stages of visual analysis. Curr Biol. 2010;20:R143–R144. doi: 10.1016/j.cub.2009.12.015. [DOI] [PubMed] [Google Scholar]
  • 10.Britten KH, Shadlen MN, Newsome WT, Movshon JA. Responses of neurons in macaque MT to stochastic motion signals. Vis Neurosci. 1993;10:1157–1169. doi: 10.1017/s0952523800010269. [DOI] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Figure S1. Results of individual subjects in Experiment 1. (A-B) Subject's perceived motion directions as a function of hand movement direction in the trials with circular apertures (A) and square apertures (B). The angles are normalized with regard to the grating normal directions—zero degrees would correspond to a perceived motion direction perpendicular to the gratings. The icon on the top-left corner shows the selected model of each subject. (C) Kinesthetic influence of the trials with circular and square apertures, respectively, in Experiment 1. Error-bars are 1-SE. (D) Histograms of subjects' hand movement directions in Experiment 1. Each radial line represents the direction in one trial.

Figure S2. Results of individual subjects in Experiment 2. (A) Subject's perceived motion directions as a function of hand movement direction. (B) Kinesthetic influence in Experiment 2. Error-bars are 1-SE. (C) Histograms of subjects' hand movement directions in Experiment 2.

RESOURCES