Summary
Judgments about the perceptual appearance of visual objects require the combination of multiple parameters, like location, direction, color, speed, and depth. Our understanding of perceptual judgments has been greatly informed by studies of ambiguous figures, which take on different appearances depending upon the brain state of the observer. Here we probe the neural mechanisms hypothesized as responsible for judging the apparent direction of rotation of ambiguous structure from motion (SFM) stimuli. Resolving the rotation direction of SFM cylinders requires the conjoint decoding of direction of motion and binocular depth signals [1, 2]. Within cortical visual area V5/MT of two macaque monkeys, we applied electrical stimulation at sites with consistent multiunit tuning to combinations of binocular depth and direction of motion, while the monkey made perceptual decisions about the rotation of SFM stimuli. For both ambiguous and unambiguous SFM figures, rotation judgments shifted as if we had added a specific conjunction of disparity and motion signals to the stimulus elements. This is the first causal demonstration that the activity of neurons in V5/MT contributes directly to the perception of SFM stimuli and by implication to decoding the specific conjunction of disparity and motion, the two different visual cues whose combination drives the perceptual judgment.
Graphical Abstract
Highlights
-
•
Targeted search for brain sites encoding conjunctions of stereo and motion
-
•
Electrical microstimulation alters perceptual choice and resolves visual ambiguity
-
•
Causal evidence for direct role of V5/MT in cue combination
-
•
Stimulation at cortical sites with conjoint encoding is particularly effective
Results
Neurons in V5/MT are selective for motion direction and binocular disparity [3–5]. When these visual parameters are examined separately, recording and electrical microstimulation experiments show that V5/MT neurons selective for these parameters make causal contributions to perceptual signals for motion [6–8] and binocular depth [9–11]. Rotating structure from motion (SFM) cylinders comprise two transparent surfaces of random dots, which move in opposite directions; assignment of dots with opposite motion directions to different visual depth surfaces defines rotation direction (Figures 1A and 1B). While V5/MT neurons can be tuned to both a motion direction and a binocular depth [9], it is only at the level of area MST that motion preference reverses for near and far disparities, which would allow a more complete representation of the stimulus [12]. Using electrical microstimulation, we tested whether V5/MT neurons causally contribute to judgments about this stimulus, which is resolved by the specific conjunction of motion and depth information. If neurons in V5/MT are not read out in this task specifically with reference to their conjoint coding, this microstimulation is expected to have no systematic effect on perceived rotation.
Two monkeys were trained to discriminate the direction of rotation of a SFM cylinder (defined by the added binocular disparity) with an eye movement response (Figure 1C). The monkeys worked at close to the psychophysical threshold for binocular disparity. After the electrode entered V5/MT, we recorded multiunit activity (MUA) every 100 μm and searched for sites that showed consistent tuning to both the same direction of motion and the same binocular disparity over a 300 μm stretch of cortex (see also Figure S1 available online). We retracted the electrode to the middle of an identified stretch and quantitatively assessed tuning for direction of motion and for binocular disparity in the cylinder stimulus (Figures 2A, 2B, 2D, and 2E). Cylinder stimuli were matched in position, size, direction of motion, and speed to the preferences of neurons at the chosen site. While the animals judged the direction of rotation on each trial, we applied electrical microstimulation on 50% of those trials, pseudorandomly interleaved with nonstimulated trials for comparison. Animals received fluid reward for correct responses to the visual stimulus, regardless of microstimulation. Data were analyzed from 48 cortical sites with significant MUA tuning to cylinder disparity (ANOVA, p < 0.05) in two monkeys (see the Supplemental Experimental Procedures).
Effect of Electrical Stimulation on Cylinder Discrimination
We hypothesized that stimulation at a cortical site where MUA is tuned to rightward motion and near disparity should boost the rightward motion signal at the front surface of the cylinder: the animal should consequently choose counterclockwise (CCW) rotation more often (Figures 1A and 1B). Figures 2A–2C show such a site tuned to CCW rotation: across all cylinder disparities, stimulation of this cortical site increased the proportion of CCW choices. The electrical signal seemed to be integrated with the neural signals evoked by the visual stimulation across the range of tested disparities. This is not simply due to boosting choices favoring the rightward motion direction. In the same monkey, we stimulated another site with MUA preference for rightward motion but in conjunction with far disparity; this time, we significantly increased the choices of clockwise (CW) rotation (Figures 2D–2F). Neither disparity selectivity alone nor direction selectivity alone can explain this pattern of results (see also Figure S2).
We calculated the effect of electrical microstimulation as if it were an additional component of the visual stimulation (see the Supplemental Experimental Procedures). Across all sites in both animals, we fitted the psychometric functions with pairs of cumulative Gaussians that were allowed to differ only in their mean (see Figures 2C and 2F) and measured the horizontal shift between the positions of the 50% points between the two curves. Stimulation sites were assigned positive shifts if the monkey’s choices were shifted toward the neural preference, as determined by the MUA tuning for cylinder stimuli; negative shifts were in the opposite direction. In one monkey (ICA), the mean shift was equivalent to addition of 0.010° (SD ± 0.038) of binocular disparity to the stimulus across 28 sites; in the other monkey (FLE), the mean shift was 0.020° (SD ± 0.029) across 20 sites. We assessed the significance of each measured shift by fitting a single cumulative Gaussian to the combined data at a single site and comparing log likelihoods (χ2, p < 0.05). For ICA, 16 out of 28 sites showed a significant shift with a mean of 0.017°; for FLE, 15 out of 20 sites were individually significant with a mean of 0.026°. For both monkeys, most shifts were positive and therefore in the direction predicted by the cylinder preference at the microstimulation site (Figures 3A and 3B).
To compare stimulation effects across stimulation sites and monkeys regardless of stimulus eccentricity and psychometric threshold, we normalized the magnitude of the stimulation effect by dividing the horizontal shift by the psychometric threshold at each site. This “normalized shift” is conceptually equivalent to the use of “normalized equivalent deviations” [13, 14]. Again, positive shifts represent sites where the effect of electrical microstimulation increases perceptual reports in favor of the neurons’ cylinder rotation preference. Across all 48 cortical sites, the median normalized shift is 0.38 (significantly different from 0, Wilcoxon sign rank, p < 0.0001; Figure 3C) and did not differ significantly between monkeys (Wilcoxon rank sum, p = 0.19; see also Figure S3).
Both near and far disparity sites have significant shifts (Figure 3D), with median normalized shifts not significantly different (far = 0.32; near = 0.39; Wilcoxon rank sum, p = 0.40). Consistent with previous reports [4, 5, 15, 16], in V5/MT we identified fewer sites (9/48) at which neurons were tuned to far disparities. Nonetheless, for sites with a particular preference for motion direction, microstimulation had opposite effects on SFM judgments depending on the disparity selectivity of the site. Our results cannot be explained by an effect of microstimulation only on direction judgments, even if combined with some strategy such as only reporting the motion of the near surface. When we plotted normalized shifts, broken down by preferred response direction, we identified significant shifts for both CW (target left) and CCW choices (target right) (Figure 3E). The median, normalized shifts for CW rotation (0.38, n = 28) and for CCW rotation (0.37, n = 20) did not significantly differ (Wilcoxon rank sum, p = 0.70). The distributions of normalized shifts were in both cases not significantly different from each other (χ2 test, p > 0.05). The measured shifts with microstimulation could not be explained by changes in vergence or the use of paired rather than independent Gaussian fits (see the Supplemental Experimental Procedures; see Figures 3A and 3B for a comparison of different Gaussian fits).
Comparison of Microstimulation in the Cylinder and a Planar Disparity Task
DeAngelis et al. [11] showed that electrical microstimulation of disparity-tuned sites in V5/MT influenced perceptual decisions in a depth-discrimination task, based on altering the proportion of dots that carried a binocular disparity signal. From their study, means and slopes for individually fitted sigmoid curves using logistic regression were available. We refitted our data with the same logistic regression used by those authors for direct comparison between the two studies. All results continue to be expressed as normalized shifts as defined above, but change of the fitted model, particularly the use of independent sigmoid curves for stimulated and nonstimulated cases, alters the detailed numerical values in comparison with the previous section.
For disparity discrimination, normalized shift due to microstimulation is correlated with the disparity tuning index (DTI) obtained from MUA at the stimulated site [11]. We also calculated the DTI for our MUA data. The mean DTIs found in the two studies were not significantly different (DeAngelis et al., 0.62, n = 65; this study, 0.66, n = 48; t test, p = 0.33). However, in our data set, only one monkey showed a significant correlation between DTI and normalized shift (combined data, Spearman’s r = 0.14, p = 0.33; FLE, r = 0.45, p < 0.05, n = 20; ICA, r = 0.07, p = 0.74, n = 28; Figure 4). On visual inspection, our data set had fewer sites with very low DTI and nonsignificant shifts. One reason could be that we applied more-stringent selection criteria for inclusion of a cortical site, which constrained the range of DTIs in our data set. When we examined the relationship between DTI and the logistic regression shifts by animal using an analysis of covariance in our data set (n = 48), the slopes for the two monkeys did not differ significantly (p = 0.14), but there was a main effect of animal (p = 0.03). This was not the case for sites with significant shifts induced by microstimulation (p = 0.25, n = 32).
Across the range of DTI, we find systematically larger effects of microstimulation in our task—a median normalized logistic shift of 0.42 (SD ± 0.63, n = 48) in our data compared with 0.22 (SD ± 0.24, n = 65) for DeAngelis et al.’s (Wilcoxon rank sum, p < 0.01; Figure 4). The same holds if we restrict the comparison to sites with significant shifts (0.84 ± 0.64, n = 32, versus 0.33 ± 0.22, n = 43; Wilcoxon rank sum, p < 0.0001). Electrical microstimulation of V5/MT seems to have a stronger effect in our task that depended upon the conjunction of motion and disparity cues than a task based on disparity alone. This reinforces the view that the microstimulation effects we report result from a specialized functional organization of V5/MT for the relationship between disparity and motion, rather than as a byproduct of separate compartments in V5/MT, each compartment contributing separately to disparity and motion perception.
Discussion
Electrical microstimulation of V5/MT biased choices in the direction expected from local neuronal selectivity for a specific combination of two different visual cues. We demonstrate that electrical signals in V5/MT can causally influence the perceived conjunction of disparity and motion, and thus alter the perception of SFM figures.
Previous experiments established that electrical microstimulation in area V5/MT can influence the perceived direction of motion or the reported binocular depth separately [7, 11]. However, the effect on SFM cylinders cannot result from simply boosting neuronal signals for either left or right motion direction alone without reference to binocular disparity: this would boost signals in the near and far surfaces without any consistent effect upon cylinder rotation. Our data confirm this specificity: the effect of stimulation reverses sign with disparity preference. We therefore reasoned that stimulation at sites with no disparity selectivity would be less informative. We did not stimulate at such locations, and so we cannot exclude the possibility of a small effect at such sites (e.g., if these signals were attributed to near surfaces by default). Any such effect, however, cannot explain the reversal with disparity preference.
The requirement for decoding conjunctions may explain why the effect of microstimulation here is larger than for disparity discrimination alone [11]. V5/MT contains some neurons that are not selective for the conjunction of motion and binocular disparity and other neurons that are. Therefore, the number of neurons potentially available for our task is smaller and more specific than those available for a simple disparity task. Provided that the electrode is located carefully, this means that the signal injected by microstimulation drives a greater fraction of the available neurons, so the psychophysical effect is larger. Although great care was taken to ensure that the stimulation parameters and the procedure for choosing stimulation sites were closely similar between the two studies, it is always possible that other subtle differences contribute. Regardless, the difference in size of microstimulation effect between the two studies warrants further study.
This argument also presupposes that neurons selective to the conjunction are clustered together in V5/MT, as suggested by our MUA recordings. The idea that that clustering is related to the strength of the electrical microstimulation effect is supported by a recent study in extrastriate visual area MSTd on multisensory heading perception [17]. Early combined microstimulation and recording experiments in motor cortex estimated that the effective spread of a 20μA current would be about 100–150μm [18]. These dimensions would be consistent with stimulating neurons within a direction column and a patch of similar disparity preference [11, 19, 20]. Horizontal connectivity within V5/MT is clustered at approximately 2 mm intervals [21], consistent with connecting neurons of similar direction and disparity preference. If the spread of microstimulation is largely determined by the architecture of local circuitry, our results suggest that these local connections reflect the conjunction of disparity and motion signals.
DeAngelis and Newsome [22] studied a motion discrimination task, in which the visual stimulus was matched carefully to both the disparity and motion selectivity of the V5/MT receptive field: in two out of three monkeys, microstimulation effects for motion were stronger at sites that were less well tuned to disparity. However, when there was a microstimulation effect at a disparity selective site, it was generally stronger when the motion stimulus was presented at the disparity preferred at that site. Also in V5/MT, Sasaki and Uka [23] showed that the correlation of neuronal activity with perceptual choice was similar for discrimination of either binocular disparity or direction of motion. These various results are not inconsistent with conjoint coding for motion and disparity, but in both tasks, the other stimulus component could be (and was at least partially) discounted. In contrast, the perceptual decisions about SFM stimuli in our task required the decoding of specific conjunctions of motion and depth signals, rather than about the visual motion or depth parameter in isolation.
Consistent with the results presented here, larger correlations between single neuron firing and perceptual choice (choice probability) have been found in V5/MT for discriminating the direction of rotation of the SFM cylinder stimulus, as used in this study, compared to motion discrimination or depth discrimination tasks alone [6, 9, 10]. Larger choice probabilities have been proposed to indicate a closer link with perception [24], but the link between choice probability and contribution to the perceptual task is not straightforward [25]. One way to reconcile these studies is to suggest that when an animal’s task depends on only a single perceptual attribute, V5/MT neurons with conjoint encoding properties may be involved in a perceptual decision, but these neurons are “read out” in a way that is capable of ignoring the coding for conjunction.
A readout from neurons conjointly coding disparity and motion, as found here for V5/MT, is required as a substrate for a number of psychophysical interactions between motion and stereo, including the depth-dependent motion aftereffect [26]. Neurophysiological recordings in V5/MT previously correlated neuronal responses to resolving perceptual ambiguity for bistable percepts like SFM and binocular rivalry [9, 27], but the level of direct contribution of such signals to perceptual decisions has been debated [28, 29]. For example, greater numbers of temporal and prefrontal neurons appear to follow perceptual dominance in other visual tasks [30, 31]. Our results suggest that these responses related to perceptual dominance might be causally driven by perceptual signals located in visual areas at earlier stages of processing.
In conclusion, our results show a strong causal link between activity in extrastriate visual area V5/MT and perception of SFM stimuli. V5/MT appears to have a specific and robust contribution to perceptual decisions about complex visual objects defined by conjunctions of motion and binocular depth. The coding of perceptual conjunctions within neural circuits may be a general encoding strategy within extrastriate cortex for cues that are commonly encountered in combination in the visual environment.
Acknowledgments
K.K. is a Royal Society University Research Fellow; N.C. held an Usher Cunningham Studentship (DPAG and Exeter College, Oxford). We would like to thank Greg DeAngelis for allowing us to use his data and Rachel Denison for help with the analysis. This work was supported by the Wellcome Trust and the Intramural Research Program of the US National Institutes of Health.
Published: July 18, 2013
Footnotes
This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Supplemental Information includes Supplemental Experimental Procedures, three figures, and one movie and can be found with this article online at http://dx.doi.org/10.1016/j.cub.2013.06.023.
Supplemental Information
References
- 1.Wallach H., O’Connell D.N. The kinetic depth effect. J. Exp. Psychol. 1953;45:205–217. doi: 10.1037/h0056880. [DOI] [PubMed] [Google Scholar]
- 2.Treue S., Husain M., Andersen R.A. Human perception of structure from motion. Vision Res. 1991;31:59–75. doi: 10.1016/0042-6989(91)90074-f. [DOI] [PubMed] [Google Scholar]
- 3.Dubner R., Zeki S.M. Response properties and receptive fields of cells in an anatomically defined region of the superior temporal sulcus in the monkey. Brain Res. 1971;35:528–532. doi: 10.1016/0006-8993(71)90494-x. [DOI] [PubMed] [Google Scholar]
- 4.Maunsell J.H.R., Van Essen D.C. Functional properties of neurons in middle temporal visual area of the macaque monkey. II. Binocular interactions and sensitivity to binocular disparity. J. Neurophysiol. 1983;49:1148–1167. doi: 10.1152/jn.1983.49.5.1148. [DOI] [PubMed] [Google Scholar]
- 5.Krug K., Parker A.J. Neurons in dorsal visual area V5/MT signal relative disparity. J. Neurosci. 2011;31:17892–17904. doi: 10.1523/JNEUROSCI.2658-11.2011. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Britten K.H., Newsome W.T., Shadlen M.N., Celebrini S., Movshon J.A. A relationship between behavioral choice and the visual responses of neurons in macaque MT. Vis. Neurosci. 1996;13:87–100. doi: 10.1017/s095252380000715x. [DOI] [PubMed] [Google Scholar]
- 7.Salzman C.D., Britten K.H., Newsome W.T. Cortical microstimulation influences perceptual judgements of motion direction. Nature. 1990;346:174–177. doi: 10.1038/346174a0. [DOI] [PubMed] [Google Scholar]
- 8.Salzman C.D., Murasugi C.M., Britten K.H., Newsome W.T. Microstimulation in visual area MT: effects on direction discrimination performance. J. Neurosci. 1992;12:2331–2355. doi: 10.1523/JNEUROSCI.12-06-02331.1992. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Dodd J.V., Krug K., Cumming B.G., Parker A.J. Perceptually bistable three-dimensional figures evoke high choice probabilities in cortical area MT. J. Neurosci. 2001;21:4809–4821. doi: 10.1523/JNEUROSCI.21-13-04809.2001. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Uka T., DeAngelis G.C. Contribution of area MT to stereoscopic depth perception: choice-related response modulations reflect task strategy. Neuron. 2004;42:297–310. doi: 10.1016/s0896-6273(04)00186-2. [DOI] [PubMed] [Google Scholar]
- 11.DeAngelis G.C., Cumming B.G., Newsome W.T. Cortical area MT and the perception of stereoscopic depth. Nature. 1998;394:677–680. doi: 10.1038/29299. [DOI] [PubMed] [Google Scholar]
- 12.Roy J.P., Komatsu H., Wurtz R.H. Disparity sensitivity of neurons in monkey extrastriate area MST. J. Neurosci. 1992;12:2478–2492. doi: 10.1523/JNEUROSCI.12-07-02478.1992. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Gaddum J.H. H.M. Stationary Office; London: 1933. Reports on Biological Standards. III. Methods of Biological Assay Depending on a Quantal Response. [Google Scholar]
- 14.Backus B.T. The Mixture of Bernoulli Experts: a theory to quantify reliance on cues in dichotomous perceptual decisions. J. Vision. 2009;9 doi: 10.1167/9.1.6. 6.1–6.19. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Bradley D.C., Andersen R.A. Center-surround antagonism based on disparity in primate area MT. J. Neurosci. 1998;18:7552–7565. doi: 10.1523/JNEUROSCI.18-18-07552.1998. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.DeAngelis G.C., Uka T. Coding of horizontal disparity and velocity by MT neurons in the alert macaque. J. Neurophysiol. 2003;89:1094–1111. doi: 10.1152/jn.00717.2002. [DOI] [PubMed] [Google Scholar]
- 17.Gu Y., Deangelis G.C., Angelaki D.E. Causal links between dorsal medial superior temporal area neurons and multisensory heading perception. J. Neurosci. 2012;32:2299–2313. doi: 10.1523/JNEUROSCI.5154-11.2012. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Stoney S.D., Jr., Thompson W.D., Asanuma H. Excitation of pyramidal tract cells by intracortical microstimulation: effective extent of stimulating current. J. Neurophysiol. 1968;31:659–669. doi: 10.1152/jn.1968.31.5.659. [DOI] [PubMed] [Google Scholar]
- 19.Albright T.D., Desimone R., Gross C.G. Columnar organization of directionally selective cells in visual area MT of the macaque. J. Neurophysiol. 1984;51:16–31. doi: 10.1152/jn.1984.51.1.16. [DOI] [PubMed] [Google Scholar]
- 20.DeAngelis G.C., Newsome W.T. Organization of disparity-selective neurons in macaque area MT. J. Neurosci. 1999;19:1398–1415. doi: 10.1523/JNEUROSCI.19-04-01398.1999. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Ahmed B., Cordery P.M., McLelland D., Bair W., Krug K. Long-range clustered connections within extrastriate visual area V5/MT of the rhesus macaque. Cereb. Cortex. 2012;22:60–73. doi: 10.1093/cercor/bhr072. [DOI] [PubMed] [Google Scholar]
- 22.DeAngelis G.C., Newsome W.T. Perceptual “read-out” of conjoined direction and disparity maps in extrastriate area MT. PLoS Biol. 2004;2:E77. doi: 10.1371/journal.pbio.0020077. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Sasaki R., Uka T. Dynamic readout of behaviorally relevant signals from area MT during task switching. Neuron. 2009;62:147–157. doi: 10.1016/j.neuron.2009.02.019. [DOI] [PubMed] [Google Scholar]
- 24.Parker A.J., Newsome W.T. Sense and the single neuron: probing the physiology of perception. Annu. Rev. Neurosci. 1998;21:227–277. doi: 10.1146/annurev.neuro.21.1.227. [DOI] [PubMed] [Google Scholar]
- 25.Nienborg H., Cumming B.G. Decision-related activity in sensory neurons reflects more than a neuron’s causal effect. Nature. 2009;459:89–92. doi: 10.1038/nature07821. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Anstis S.M. Movement aftereffects contingent on binocular disparity. Perception. 1974;3:153–168. doi: 10.1068/p030153. [DOI] [PubMed] [Google Scholar]
- 27.Logothetis N.K., Schall J.D. Neuronal correlates of subjective visual perception. Science. 1989;245:761–763. doi: 10.1126/science.2772635. [DOI] [PubMed] [Google Scholar]
- 28.Gold J.I., Shadlen M.N. The neural basis of decision making. Annu. Rev. Neurosci. 2007;30:535–574. doi: 10.1146/annurev.neuro.29.051605.113038. [DOI] [PubMed] [Google Scholar]
- 29.Nienborg H., Cohen M.R., Cumming B.G. Decision-related activity in sensory neurons: correlations among neurons and with behavior. Annu. Rev. Neurosci. 2012;35:463–483. doi: 10.1146/annurev-neuro-062111-150403. [DOI] [PubMed] [Google Scholar]
- 30.Sheinberg D.L., Logothetis N.K. The role of temporal cortical areas in perceptual organization. Proc. Natl. Acad. Sci. USA. 1997;94:3408–3413. doi: 10.1073/pnas.94.7.3408. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Panagiotaropoulos T.I., Deco G., Kapoor V., Logothetis N.K. Neuronal discharges and gamma oscillations explicitly reflect visual consciousness in the lateral prefrontal cortex. Neuron. 2012;74:924–935. doi: 10.1016/j.neuron.2012.04.013. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.