Temporal and spatial limits of pattern motion sensitivity in macaque MT neurons

Abstract

Many neurons in visual cortical area MT signal the direction of motion of complex visual patterns, such as plaids composed of two superimposed drifting gratings. To compute the direction of pattern motion, MT neurons combine component motion signals over time and space. To determine the spatial and temporal limits of signal integration, we measured the responses of single MT neurons to a novel set of “pseudoplaid” stimuli in which the component gratings were alternated in time or space. As the temporal or spatial separation of the component gratings increased, neuronal selectivity for the direction of pattern motion decreased. Using descriptive models of signal integration, we inferred the temporal and spatial structure of the mechanisms that compute pattern direction selectivity. The median time constant for integration was roughly 10 ms, a timescale characteristic of integration by single cortical pyramidal neurons. The median spatial integration field was roughly one-third of the MT receptive field diameter, suggesting that the spatial limits are set by stages of processing in earlier areas of visual cortex where receptive fields are smaller than in MT. Interestingly, pattern direction-selective neurons had shorter temporal integration times than component direction-selective neurons but similar spatial integration windows. We conclude that pattern motion can only be signaled by MT neurons when the component motion signals co-occur within relatively narrow spatial and temporal limits. We interpret these results in the framework of recent hierarchical models of MT.

to determine the motion of complex visual patterns, the visual system processes moving stimuli in two stages. In early visual cortical areas, neurons extract signals related to the simple, oriented motion components of a moving pattern. Further downstream, these component signals are selectively integrated to compute the direction of pattern motion, that is, the overall direction of a moving pattern, not that of its individual components. Some neurons in primary visual cortex (V1) encode the attributes of simple motion, direction, and speed (Hubel and Wiesel 1968). When presented with complex patterned motion stimuli, such as plaids composed of two superimposed drifting gratings, these neurons respond selectively to the motion of the underlying component gratings rather than the overall pattern motion (Movshon et al. 1985); these neurons are designated “component direction selective” (CDS), and essentially all V1 neurons are CDS. V1 sends direction-selective information to extrastriate cortical area MT/V5 (Maunsell and Van Essen 1987; Movshon and Newsome 1996; Zeki 1974, 1978), where pattern motion is explicitly represented in neuronal responses (Born and Bradley 2005; Movshon et al. 1985). Although many MT neurons are CDS, a notable subset of them (∼33%) signals the direction of pattern motion; these neurons are designated “pattern direction selective” (PDS).

Because of their key role in the representation and perception of visual motion, PDS neurons are the cornerstone of many models of pattern motion integration (Adelson and Movshon 1982; Grzywacz and Yuille 1990; Movshon et al. 1985; Rust et al. 2006; Simoncelli and Heeger 1998; Wilson et al. 1992). A common motif in these “hierarchical” models is an integration stage in which the weighted combination of directional V1-like signals gives rise to pattern direction selectivity in MT. To date, these models have only considered the time-averaged, steady-state responses of MT neurons, in large part due to our limited understanding of how pattern motion is dynamically computed for continuously changing stimuli. Recent studies have suggested that the integration properties of MT neurons, both temporal and spatial, may carry useful information about the mechanisms that compute pattern motion (Clark and Bradley 2008; Majaj et al. 2007; Pack and Born 2001; Smith et al. 2005; Solomon et al. 2011). In the present study, we measured the temporal and spatial limits on the pattern motion computation using a novel set of stimuli, plaids with component gratings that alternate in time or space, herein referred to as “pseudoplaids.” We found that the limits of both spatial and temporal integration for pattern motion signaling by MT neurons were quite stringent, suggesting that the computation of pattern motion within the large receptive fields of MT neurons depends critically on mechanisms that are tightly localized in both time and space.

MATERIALS AND METHODS

Surgical procedures.

Fifteen adult male macaque monkeys (Macaca nemestrina and M. fascicularis) were prepared for single-unit recording, as described in detail elsewhere (Cavanaugh et al. 2002). The animals were maintained under anesthesia and paralysis with continuous intravenous infusion of sufentanil citrate (6 μg·kg⁻¹·h⁻¹, initial dose) and vecuronium bromide (Norcuron, 100 μg·kg⁻¹·h⁻¹) in 5% dextrose/Normosol-R solution. Vital signs were monitored (heart rate, blood pressure, lung pressure, EEG, ECG, body temperature, urine flow and specific gravity, and end-tidal Pco₂) and maintained within appropriate physiological limits. The pupils were dilated with topical atropine sulfate, and the corneas were protected with oxygen-permeable hard +2D contact lenses. Supplementary lenses, chosen via direct ophthalmoscopy, were used to make the retinas conjugate with the experimental display. Experiments typically lasted 4–5 days. All experimental procedures were conducted in compliance with the NIH “Guide for the Care and Use of Laboratory Animals” [DHEW Publication No. (NIH) 85-23, Revised 1996, Office of Science and Health Reports, DRR/NIH, Bethesda, MD 20205] and were reviewed and approved by the New York University Animal Welfare Committee.

Neurophysiology.

A craniotomy and durotomy were centered roughly 6 mm posterior to the lunate sulcus and 16.5 mm lateral to the midline. Electrode penetrations were confined to a parasagittal plane and directed downward at an angle of 20° from horizontal, passing through the lunate sulcus and entering the posterior/ventral bank of the superior temporal sulcus. We recorded well-isolated, single-unit activity extracellularly using quartz-platinum-tungsten microelectrodes (Thomas Recording, Giessen, Germany). We defined single units qualitatively, using a dual window discriminator; their voltage traces were distinct from background multiunit activity in both shape and amplitude. We identified area MT physiologically from the onset of brisk direction-selective responses in single- and multiunit activity, with receptive field diameters roughly equal to eccentricity (typically 5° to 15°). Signals were amplified, bandpass filtered (300 Hz to 10 kHz), fed into a time-amplitude discriminator, and time-stamped with 100-μs resolution. Electrolytic lesions were made at the end of each recording track for histological confirmation of MT recording sites.

Visual stimuli.

We presented visual stimuli on the screen of a gamma-corrected CRT monitor (Eizo T966) located 80 cm from the eyes. The monitor was refreshed at 120 Hz and had a resolution of 1,280 × 960 pixels and a mean luminance of 33 cd/m², which remained constant throughout all stimulus presentations. Stimuli were generated using Expo software on an Apple Mac Pro computer (http://corevision.cns.nyu.edu/expo). For each neuron recorded, we initially mapped the receptive fields through each eye; subsequent tests were run monocularly through the dominant eye. We determined the optimal direction, spatial frequency, drift rate, and position of the neuron's receptive field using full-contrast, sinusoidal gratings. The optimal receptive field size was estimated on the basis of a spatial summation tuning curve. For each neuron, we presented gratings at nine sizes, covering four octaves centered around a qualitative estimate of the receptive field diameter. The mean firing rates to each size were fit with a well-established descriptive model, the integral-difference-of-Gaussians (Cavanaugh et al. 2002), and the peak of the fit was used as a measure of the optimal receptive field size. Stimuli were vignetted with a soft-edged, circular aperture and presented against a background at the mean luminance.

To test the temporal and spatial scales that limit the pattern motion computation, we presented sinusoidal gratings and plaids and two types of pseudoplaids: temporal and spatial. Individual gratings were presented at 50% contrast and at the optimal spatial frequency and drift rate. Plaids were generated by superimposing two drifting gratings separated in direction by 120°. Pseudoplaids had the same component gratings as true plaids, but they were separated in time or space (see below). We also presented appropriate “half-pseudoplaid” controls: single component gratings temporally or spatially modulated with the same structure as the corresponding pseudoplaid. Responses to half-pseudoplaids were used to construct the pattern and component response predictions for pseudoplaid stimuli. Specifically, this allowed us to generate predictions that incorporated our experimental manipulations of the temporal and spatial structure of component motion signals. Each pattern was presented in 12 motion directions (30° steps). Stimuli were presented randomly interleaved in a continuous stream (Smith et al. 2005), with at least 20 repeats per condition; this also included the presentation of full field mean luminance to measure baseline firing rate. Each stimulus was presented for 267 ms in the temporal pseudoplaid experiment and for 500 ms in the spatial pseudoplaid experiment. For every stimulus condition we report the number of impulses per stimulus duration.

Temporal pseudoplaids were created by presenting each of the component gratings of a plaid in a temporally alternating fashion, every 1, 2, 3, 4, 8, or 16 video frames (corresponding to full-cycle alternation periods from 17 to 267 ms). Figure 1, A–C, shows a schematic of the presentation sequence of component gratings for temporal pseudoplaids of three different periods (0, 33, and 267 ms). If both components are presented on each video frame (Fig. 1A), the result is a true plaid, which we treat as a temporal pseudoplaid with an alternation period of 0 ms. Note that for all temporal pseudoplaids, both component gratings had a fixed total presentation time (50% of the frames in each stimulus epoch); what varied was the time between the sets of frames containing each component grating. The phase of the component gratings during each alternation period was consistent with continuous motion (i.e., only the opacity of each component grating was alternated).

Fig. 1.

Stimulus construction. A–C: temporal pseudoplaids were composed of 2 superimposed drifting gratings separated by 120° and temporally interleaved. Stimulus schematics show the onset and duration of each component grating (light and dark gray) for temporal pseudoplaids with different alternation periods: 0 ms (equivalent to a true plaid), 33 ms, and 267 ms. D–F: spatial pseudoplaids were composed of 2 drifting gratings presented within patches that tessellated the receptive field (dashed line); patches containing each of the 2 components were spatially interleaved. Spatial pseudoplaids with different patch numbers are shown: an infinite number of patches (equivalent to a true plaid), 16 patches, and 4 patches. White arrows indicate the drift direction of component gratings.

Spatial pseudoplaids were created by presenting each of the component gratings of a plaid within separate circular patches that tiled the receptive field; patches containing component 1 were spatially alternated with those containing component 2 in a regular lattice. We used circular patches to minimize motion biases due to terminator cues and the regular lattice to ensure that equal proportions of each component grating were presented. To reduce edge artifacts, each component patch was vignetted with a Gaussian profile contrast envelope that faded gradually to zero over the outermost 25% of the patch radius. Figure 1, D–F, shows examples of spatial pseudoplaids with different numbers of patches. As the number of patches was decreased, the patch size was increased to maintain the same total area covered by each component grating; what varied was the spatial distribution of component-specific stimulus energy, not the total amount. For example, a 4-patch pseudoplaid would have patch diameters equal to one-half the receptive field size, whereas a 16-patch pseudoplaid would have patch diameters equal to one-quarter of the receptive field size. We treat a true plaid whose components are physically added as a spatial pseudoplaid with an infinite number of patches. To prevent asymmetries or inequalities in the locations or numbers of component grating patches, only 4n² numbers of patches were used (for n = 1, 2, 3, 4, …). Typically 4, 16, 36, and 64 patches were tested. For each neuron, the maximum number of patches presented was limited by its optimal spatial period; we only used stimuli that contained at least one grating cycle in each patch.

Separating the component gratings of plaid stimuli in time or space introduces spectral energy at frequencies not present in the true plaid, but we do not expect this additional spectral content to influence the response of MT neurons. First, the spectral spread was confined to either the temporal or spatial frequency dimension, never both. This ensured that we did not introduce cues consistent with any particular velocity of motion. Second, in the case of spatial pseudoplaids, we restricted the extent of additional spatial spectral energies by selecting patch sizes containing at least one cycle of the component grating at the optimal spatial frequency. This prevented the introduction of spectral power in the pattern motion direction. Last, our half-pseudoplaid controls empirically measured the influence of this additional spectral content on the direction selectivity of MT neurons; we found no evidence for such modulation in our data set.

Pattern index.

We computed an index of pattern direction selectivity from the responses to drifting plaids and gratings, after correcting for response latency as described in Smith et al. (2005). First, we generated two predicted-direction tuning curves for the complex visual patterns (plaids/pseudoplaids): a pattern prediction (a tuning profile that matches the tuning for gratings/half-pseudoplaids) and a component prediction (a tuning profile that matches the linear combination of responses to each component grating/half-pseudoplaid). We then computed the partial correlations between the observed responses and each prediction, and transformed these correlations into normal deviates using Fisher's r-to-Z transform. Our measure of pattern direction selectivity, the pattern index (PI), is the difference between the Z-transformed pattern correlation (Z_P) and the component correlation (Z_C). Using the pattern index, we classified neurons as pattern or component direction selective (PDS and CDS, respectively). To be classified as PDS, PI must be positive; to be classified as CDS, PI must be negative. Furthermore, the absolute value of this index had to exceed a criterion value of 1.28 standard deviations (equivalent to P < 0.1); otherwise, neurons were considered intermediate.

Bootstrap statistics.

For every neuron, we resampled the original trial-by-trial data with replacement, keeping the total number of trials unchanged. From these resampled data, we computed the pattern indexes for plaid and pseudoplaid conditions, as well as relevant model fit parameters (described in the text). This procedure was repeated 10,000 times, and we took the limits of the inner 95% of the distributions of bootstrap estimates as confidence bounds.

RESULTS

To determine the temporal and spatial limits over which MT neurons integrate their input signals to compute visual pattern motion, we studied neuronal responses to a novel family of patterned stimuli in which the two component gratings were separated in time or in space: pseudoplaids. These stimuli allowed us to manipulate the temporal and spatial structure of the component motion signals and to infer the way MT neurons integrate these signals. We report results from 139 neurons in total, recorded in 15 macaque monkeys.

Temporal limits on pattern direction selectivity.

Previous work investigating the temporal dynamics of MT responses suggests that the computation of pattern motion is not derived instantaneously from afferent inputs but evolves over time (Pack and Born 2001; Smith et al. 2005; Solomon et al. 2011). These studies, however, only explored the case where both component motion signals were simultaneously presented. In this study, we took a related but different approach and determined the limits of pattern motion integration for component motion signals that were temporally interleaved. We presented temporal pseudoplaids to 112 neurons in MT. We used temporal pseudoplaids in which the component gratings alternated at different rates (as in Fig. 1, A–C; see materials and methods). As controls, we presented standard sinusoidal gratings, as well as half-pseudoplaids in which a single component grating was alternated with frames of mean luminance. Figure 2A shows polar direction tuning curves (black) for a typical MT neuron tested with single gratings (left) and plaids (right); the pattern (red) and component (blue) predictions are overlaid for comparison. This neuron showed strong direction selectivity for motion down and to the left. It was strongly pattern direction selective, as indicated by the similarity of the measured plaid tuning curve and the pattern prediction (right). Figure 2, B and C, show the neuron's responses to half-pseudoplaids of different alternation periods (left) and to temporal pseudoplaids with different alternation periods (right); the pattern (red) and component (blue) predictions are again overlaid for comparison. When the alternation period was brief (33 ms), responses to the half-pseudoplaid (left) were similar to grating responses, even though half of the video frames in the half-pseudoplaid control were of mean luminance; when the alternation period was long (267 ms), responses were appreciably weaker. At an alternation period of 33 ms, the tuning curve for plaids (right) was intermediate between the pattern and component predictions. At a period of 267 ms, the tuning curve matched the component prediction, and the response magnitude was commensurate with the weaker half-pseudoplaid response.

Fig. 2.

Responses of an example MT neuron to temporal pseudoplaids and controls. A: direction tuning curves to gratings (left) and plaids (right) for an example pattern direction-selective (PDS) neuron. Overlaid on plaid responses are the pattern (red) and component (blue) predictions; baseline firing is also plotted [green; ∼0 impulses/s (imp/s)]. B and C: direction tuning curves to half-pseudoplaids (left), composed of 1 component grating alternating in time with mean luminance, and temporal pseudoplaids (right) are shown at 2 alternation periods: 33 ms (B) and 267 ms (C). Direction selectivity was similar across all control stimuli (gratings and half-pseudoplaids). Selectivity for pattern motion gradually decreased as the temporal separation between component motion signals increased. At an alternation period of 0 ms, the response profile matched the pattern prediction; at 33 ms it resembled a mixture of the pattern and component predictions; at 267 ms it matched the component prediction, indicating a complete breakdown of selectivity to the direction of pattern motion.

We wondered how the observed shift in the time-averaged direction tuning curve was represented on a finer timescale. When component gratings are presented synchronously, as in a true plaid, we expect an elevated but unmodulated response throughout the stimulus duration for different directions of motion. In contrast, when component gratings are alternated slowly, as in a pseudoplaid of 267-ms alteration period, we expect a strongly modulated response with a characteristic “high/low” profile: responses would be higher in epochs containing the preferred component grating and lower in epochs containing the nonpreferred component grating. Since the component gratings were separated by 120°, the response magnitude in each epoch should depend on the neuron's direction tuning bandwidth to gratings (see Fig. 3A, top), and it is unlikely that both components would strongly drive responses. Moreover, the shortest alternation period at which responses become modulated may indicate the limit of temporal integration of the two component motion signals, beyond which the neuron responds to each component grating separately. Figure 3 shows peristimulus time histograms (PSTHs) for the example neuron from Fig. 2 to gratings, plaids, and temporal pseudoplaids. For gratings and plaids (Fig. 3A), we observed an elevated but unmodulated response for all directions that drove the neuron. For temporal pseudoplaids (Fig. 3B), we show cycle PSTHs where responses were wrapped based on the alternation period; each half of the PSTH shows the response to one of the two component gratings, as indicated schematically by dark and light gray bars. At the shortest alternation period (17 ms), pseudoplaid responses were qualitatively similar to true plaid responses and there was no discernible modulation synchronized with the component alternation. PSTHs corresponding to the true direction of pattern motion (red band) showed the strongest responses, indicating that the neuron continued to signal pattern motion. As the alternation period increased, responses became more modulated, indicating that the neuron responded during one of the component epochs, but not the other. At long alternation periods (>67 ms), responses to the true direction of pattern motion were weaker than those to the two directions of component motion (blue bands), suggesting a selective weakening of the pattern motion response.

Fig. 3.

Response peristimulus time histograms (PSTHs) to temporal pseudoplaids. Responses of the same example neuron (as in Fig. 2) are shown. A: PSTHs of the responses to gratings (top) and true plaids (bottom) drifting in different directions. Each PSTH lasts 267 ms and contains 32 bins. B: cyclical PSTHs of the responses to temporal pseudoplaids drifting in different directions (columns) and at different alternation periods (rows). Each cyclical PSTH is wrapped at the alternation period of the stimulus. The directions of each component grating (dark and light gray bars) are indicated separately on the abscissa. PSTHs corresponding to the optimal pattern direction (from A) are highlighted by a red band. PSTHs corresponding to plaids containing a component grating drifting in the optimal direction are highlighted by a blue band. Selectivity for the direction of pattern motion decreased as the alternation period increased. At the fastest alternation period (17 ms), the neuron showed strong, unmodulated responses at the true direction of pattern motion (red). At longer alternation periods (>67 ms), responses at that same direction were weak. Instead, strong, modulated responses appeared at plaid directions in which the component gratings were drifting in the optimal direction (blue), suggesting the neuron was independently signaling each of the component gratings.

The pattern of modulation changes shown in Fig. 3 was consistent across the population of neurons recorded. When neurons integrated the component gratings to compute pattern motion, cycle histograms were weakly modulated; when responses became dominated by component signals, cycle histograms showed more modulation. We quantified this modulation with an F₁/F₀ index, where F₁ is the amplitude of the cyclical PSTH at the alternation frequency, and F₀ is the average spike rate. This index ranges from 0 to 2, with higher values indicating stronger response modulation. We compared the modulation indexes computed from the responses to pseudoplaids containing a component grating drifting in the preferred direction (blue band) at short and long alternation periods (17 and 267 ms, respectively). This allowed us to examine changes in response modulation for all MT neurons regardless of pattern index. The median modulation index increased from 0.16 at short periods to 1.11 at long periods (paired Wilcoxon signed rank sum test, P < 0.001).

To quantify the change in direction tuning curves across the population of neurons recorded, for each alternation period, we correlated each observed pseudoplaid response with its associated pattern and component predictions (see materials and methods). Figure 4. A–C, shows scatter plots of the partial correlations to the pattern and component predictions for alternation periods of 0, 33, and 267 ms. Dashed lines separate neurons into three categories (see materials and methods): PDS (red), CDS (blue), and intermediate (gray). Data are color-coded according to their responses to true plaids (0-ms alternation period); results for the example neuron shown in Figs. 2 and 3 are shown in black. In this condition, we observed a continuous distribution of PDS, CDS, and intermediate neurons (Movshon et al. 1985; Smith et al. 2005), showing approximately equal proportions of each type. Because we were particularly interested in determining the spatiotemporal limits of pattern motion integration, we sampled more PDS neurons than other classes. As the alternation period increased, all neurons showed a gradual reduction in their pattern correlation and a paralleled increase in their component correlation, reflecting systematic shifts in their direction tuning curves. This is evidenced by the movement of data points down and to the right for all types of neurons. Thus even neurons that were intermediate or CDS showed some signature of a pattern motion response, and this signature was weakened for increasingly longer alternation periods in the same manner as it was for PDS neurons. This is consistent with evidence that MT neurons lie along a continuum from pure PDS to pure CDS, with most showing signs of both kinds of behavior (Rust et al. 2006).

Fig. 4.

Selectivity for pattern motion across temporal alternation period. A–C: scatter plots of the Z-transformed correlations between the observed responses to temporal pseudoplaids and each of the component (abscissa) and pattern predictions (ordinate). Each panel shows data at 1 of 3 alternation periods (0, 33, and 267 ms) and for all neurons tested (n = 112). Significance bounds (dotted lines) classify neurons in this correlation space into 3 types: PDS (red, n = 46) in the top left zone, component direction selective (CDS; blue, n = 27) in the bottom right zone, and intermediate neurons (gray, n = 39) in the middle zone. Data for the example neuron (from Fig. 2) are highlighted (black). Data in B and C are color-coded based on the classification of responses to true plaids (as in A). As the alternation period increased, all neurons showed a gradual reduction in their pattern correlation (Z_P) and an increase in their component correlation (Z_C), reflecting changes in their direction tuning curves.

Although the observed changes in pattern direction selectivity could have resulted from changes in the responses to control stimuli (i.e., gratings/half-pseudoplaids) that were used to derive predictions, we found this not to be the case. Any distortions in the tuning curves for control stimuli (due to higher response variability, additional peaks, etc.) would result in a reduction in both the pattern and component correlations. In contrast, as shown in Fig. 4, we found a reduction in pattern correlation concomitant with an increase in component correlation.

The difference between the pattern and component partial correlations, known as the pattern index, quantifies the strength of pattern direction selectivity (see materials and methods). Figure 5 shows how the pattern index varied with alternation period for 10 representative example neurons. The example neuron (in Fig. 2) is highlighted (black); solid lines are fits of a descriptive model described below. When component gratings were alternated at high rates (e.g., alternation period of 17 ms, or every other video frame), MT neurons responded as if a true plaid was presented. As the alternation period increased, there was a smooth and rapid decrease in the pattern index. The decay in pattern direction selectivity reached a plateau at alternation periods greater than 133 ms. At the longest alternation period (267 ms), all neurons, regardless of their classification in response to true plaids, responded to the individual component gratings, providing an upper limit measure of maximum component direction selectivity. This breakdown in pattern direction selectivity suggests that the computation of pattern motion depends on mechanisms with a very limited temporal integration window.

Fig. 5.

Pattern index across temporal alternation period. To capture changes in neuronal selectivity for the direction of pattern motion, we computed the pattern index (see materials and methods) at each temporal alternation period tested. This is shown for 10 example MT neurons (gray) that show a range of profiles; data for the example neuron (from Fig. 2) is highlighted (black). Solid lines are model fits to the data. As the alternation period increased, we observed a rapid decrease in the pattern index. The breakdown in pattern selectivity suggests that pattern motion is computed over a relatively small temporal scale.

Modeling temporal limits of integration.

We modeled the changes in pattern index we observed by assuming that component motion signals are temporally low-pass filtered before being combined. Intuitively, our model (Fig. 6) proposes that the pattern index is proportional to the degree to which the two filtered component signals overlap in time and are thus simultaneously available for the pattern motion computation; in addition, each neuron has a characteristic maximum pattern index. This model is mechanistically agnostic and serves only to provide a simplified functional description of the temporal characteristics of the pattern motion computation. Figure 6A shows a schematic of the presentation times of the two component gratings (shown as pulse trains) for a rapidly alternating pseudoplaid; at any given time, only one grating is present. In the model, each pulse train is blurred in time; operationally, it is convolved with an exponential decay function (Fig. 6C), as follows:

$$b\left(t\right)=s\left(t\right)\otimes e^{-t/\tau },$$ (1)

where s(t) is the component stimulus pulse train, τ is a time constant, and b(t) is the resulting blurred stimulus. After blurring, the resulting signals (Fig. 6D, light and dark gray) have some overlap (red). During this temporal overlap, both component signals are available and can therefore be used to compute pattern motion. We make the simple assumption that the pattern index is proportional to this overlap, as follows:

$$\text{PI}\approx \frac{b_1\left(t\right)\cdot b_2\left(t\right)}{\Vert b_1\left(t\right)\Vert \hspace{0.17em}\Vert b_2\left(t\right)\Vert },$$ (2)

where b₁(t) and b₂(t) are the two temporally blurred stimuli and PI is the pattern index; this computation is performed at each alternation period. When pulse trains with slower alternation periods (Fig. 6B) are convolved with the same exponential decay function, the resulting window of overlap is smaller (Fig. 6E). In general, for any given τ, as the alternation period increases, there is a smooth and rapid decay in the overlapped signals. The extent of temporal blur, as parameterized by the value of τ, also affects the reduction in the overlap window. When the same pulse trains (Fig. 6, F and G) are convolved with a slower exponential (Fig. 6H), the resulting window of overlap is larger (Fig. 6, I and J), predicting a slower decay in the pattern index.

Fig. 6.

A model of the temporal integration of component motion signals. A and B: schematics of the 2 component gratings (light and dark gray), shown for temporal pseudoplaids in which these signals were rapidly and slowly alternating, respectively. C: a temporal (exponential decay) filter is applied to the pulse trains in A and B; this filter has a short time constant (τ). D and E: the resulting temporally blurred pulse trains have periods in which the 2 component gratings overlap in time (red) and are simultaneously available for computing pattern motion. If a filter with a longer time constant (H) is applied to the same component gratings (F and G), the temporal overlap will increase (compare E and J).

To quantify the temporal integration behavior for each neuron, we found the value of τ that best accounted for the decay in pattern index (fits shown in Fig. 5), minimizing the χ² error between the observed pattern indexes and those predicted by the model. The model successfully captured the decay in pattern index across increasing alternation periods (median R² = 0.84, median P < 0.004). Figure 7 shows the derived time constants plotted against the pattern indexes computed from responses to true plaids, across the population (n = 112). Gray bands are the bootstrapped 95% confidence limits for each estimate of pattern index and τ. The distribution of time constants was unimodal with a median of 10.3 ms; the time constant for the example neuron was similar (black; 11.2 ms). The time constants were significantly correlated with the pattern index (Spearman's r: −0.47, P < 0.001): neurons with higher pattern indexes had smaller time constants. For a few neurons (5/112), we observed no systematic changes in pattern index at different alternation periods. For these data (2 PDS, 2 CDS, and 1 intermediate neuron), the model could not fit the data and we therefore report their time constant as “undefined” (right); these neurons typically had low, inconsistent firing rates during the stimulus presentation.

Fig. 7.

Population time constants. A scatter plot of the time constant derived from model fitting and the pattern index for all neurons tested. Each point represents data from 1 neuron; data for the example neuron (in Fig. 2) is highlighted (black). Time constants for neurons that could not be fit by the model are marked as “undefined.” Significance bounds for classifying PDS and CDS neurons (see materials and methods) are shown by dotted lines. The median time constant was 10.3 ms (inverted triangle). PDS neurons had shorter time constants on average compared with CDS neurons, suggesting that they were less tolerant to the temporal separation of component motion signals.

When separating the components of a plaid in time (or in space, see below), we expect them to act independently to elicit responses and therefore expect the pattern index to drop toward a low value. For true plaids, the pattern index for component cells is already low, meaning that any manipulation of the plaid that reduces the pattern index has less range over which to operate and the resulting estimate might therefore be more vulnerable to noise contamination. If this had a systematic effect on the estimate of time constant, CDS neurons would have artifactually longer time constants. To check this possible source of bias, we simulated the relationship between neuronal variability and the time constants of the pattern computation. First, we computed the lower bounds for the pattern index based on the observed neuronal variability of the recorded responses to gratings and half-pseudoplaids. We compared the differences between the observed pattern indexes derived from true plaid responses and the simulated lower bounds (i.e., the maximal distance to the lower bound) and found no indication that these distances were systematically related (data not shown; Spearman's r = −0.2, P = 0.09). Second, we simulated the effect of introducing additional noise on the time constants by parametrically adjusting the variability of our neuronal responses (scaled from 1/5 to 5-fold, bootstrapped, n = 2,000 iterations). We found an increase in the variability of the estimated time constants as more noise was introduced but no reliable increase in median value. These simulations show that the relationship between our measured time constants and the pattern selectivity of MT neurons was not only a consequence of CDS neurons having weaker pattern direction selectivity.

Spatial limits on pattern direction selectivity.

We also measured the spatial limits on the pattern motion computation using analogous methods. We presented spatial pseudoplaids to 87 neurons in MT, where a spatial pseudoplaid consists of component gratings presented in spatially alternating patches that tessellate the neuron's receptive field (as in Fig. 1, D–F; see materials and methods). As controls, we also presented full receptive field sinusoidal gratings and half-pseudoplaids, in which patches of a single component grating were spatially alternated with patches of mean luminance. Figure 8A shows polar direction tuning curves (black) for a typical MT neuron tested with single gratings (left) and plaids (right); the pattern (red) and component (blue) predictions are overlaid for comparison. The neuron showed strong direction selectivity for motion up and to the left. It was strongly pattern direction selective, as indicated by the similarity of the measured plaid tuning curve and the pattern prediction (right). Figure 8, B and C, shows the responses of this neuron to half-pseudoplaids with different numbers of patches (left) and to the corresponding spatial pseudoplaids (right); the pattern (red) and component (blue) predictions are again overlaid for comparison. Responses to the half-pseudoplaids (left) were similar to grating responses in shape, but weaker, presumably because the stimulus area was halved. When there were 16 patches (Fig. 8B), the tuning curve for plaids (right) was intermediate in shape and reduced in amplitude. When the number of patches was reduced to four (Fig. 8C), the tuning curve matched the component prediction, and the response magnitude was commensurate with the weaker half-pseudoplaid response. Note that spatial pseudoplaid stimuli elicit weaker responses than true plaids due to the sparser coverage of the receptive field; the spatial configuration and shape of the patches were chosen to minimize biases in visual drive.

Fig. 8.

Responses of an example MT neuron to spatial pseudoplaids and controls. A: direction tuning curves to gratings (left) and plaids (right) for an example PDS neuron. B and C: direction tuning curves for half-pseudoplaids (left), composed of 1 component grating alternating in space with mean luminance, and spatial pseudoplaids (right) are shown at 2 scales of spatial separation: 16 and 4 patches. Direction selectivity was similar across all control stimuli. Selectivity for pattern motion gradually decreased as the spatial separation between the component motion signals increased, and fewer patches were presented. For an infinite number of patches (true plaid), the response profile matched the pattern prediction; for an intermediate number of patches (16), it resembled a mixture of the pattern and component predictions; for even fewer patches (4), it matched the component prediction, indicating a complete breakdown of selectivity to the direction of pattern motion. Plotting conventions are as described in Fig. 2.

To quantify the change in direction tuning curves across the population, we again correlated the pseudoplaid responses with each of the pattern and component predictions. Figure 9, A–C, shows scatter plots of the partial correlations to the pattern and component predictions for spatial pseudoplaids with an infinite number of patches (true plaid) and for 16, and 4 patches. As with temporal pseudoplaids (Fig. 4), neurons were classified and color-coded based on the pattern index computed from their responses to true plaids and gratings; the example neuron is highlighted (black). As the number of patches decreased, all neurons showed a strong reduction in pattern correlation and a moderate increase in component correlation, reflecting systematic shifts in their direction tuning curves. As with temporal pseudoplaids, this same trend was evident for PDS, CDS, and intermediate neurons.

Fig. 9.

Selectivity for pattern motion across spatial separation. A–C: scatter plots of the Z-transformed correlations between the observed responses to spatial pseudoplaids and each of the component (abscissa) and pattern predictions (ordinate). Each panel shows data for spatial pseudoplaids with a different number of patches (infinite, 16, and 4 patches) and for all neurons tested (n = 87). Data for the example neuron (as in Fig. 8) are highlighted (black). As the spatial separation between component gratings increased, and fewer patches were presented, all neurons showed a gradual reduction in their pattern correlation, reflecting changes in their direction tuning curves. Plotting conventions are as described in Fig. 4.

We computed the pattern index for each neuron and for each number of patches tested. Figure 10 shows the pattern index for different numbers of patches for 10 representative neurons. The example neuron (Fig. 8) is highlighted (black); solid lines are the fits of a descriptive model described below. The pattern index fell smoothly and completely as the number of patches decreased. This breakdown in pattern selectivity suggests that pattern motion is computed by mechanisms that integrate over a small area.

Fig. 10.

Pattern index across spatial separation. To capture changes in neuronal selectivity for the direction of pattern motion, we computed the pattern index at each spatial separation tested. This is shown for 10 example MT neurons (gray) that show a range of profiles; data for the example neuron (from Fig. 8) is highlighted (black). Solid lines are model fits to the data. As the spatial separation of component gratings increased, and fewer patches were presented, we observed a gradual decrease in the pattern index. The breakdown in pattern selectivity suggests that pattern motion is computed over a relatively small spatial scale.

Modeling spatial limits of integration.

We modeled the decrease in pattern index with decreasing patch number by assuming that the component grating signals were spatially blurred before being combined in MT (Fig. 11). As with the temporal integration model, we assume the pattern index to be proportional to the overlap in the spatially filtered component motion signals, up to a maximum value characteristic of each neuron. The component gratings were not physically superimposed, and we assume that the pattern computation is based on spatially blurred component signals; we quantified the blur needed to explain the observed decrease in pattern index. We first generated spatial maps of the two component signals; Fig. 11, A and B, indicates the locations of each of the two component gratings (schematically represented as light and dark gray) for pseudoplaids of 16 and 4 patches. These maps were vignetted (see materials and methods) and convolved with a two-dimensional Gaussian profile, as follows:

$$b\left(x,y\right)=s\left(x,y\right)\otimes e^{-\left(x^2+y^2\right)/2{\sigma }^2},$$ (3)

where s(x,y) is the vignetted spatial map for each grating location, σ is the space constant of the Gaussian, and b(x,y) is the resulting blurred map. These blurred maps contain regions where signals from the component gratings overlap. We assume that in these regions, the component signals could be used to compute pattern motion (see Eq. 2). We formalize this as follows:

$$\text{PI}\approx \frac{b_1\left(x,y\right)\cdot b_2\left(x,y\right)}{\Vert b_1\left(x,y\right)\Vert \hspace{0.17em}\Vert b_2\left(x,y\right)\Vert },$$ (4)

where b₁(x,y) and b₂(x,y) are the blurred locations for each grating across space and PI is the pattern index. Figure 11, D and E, shows the location and relative strength of the overlap areas for pseudoplaids of 16 and 4 patches, respectively; for a given space constant, the area of overlap decreases as the number of patches decreases (compare Fig. 11, D and E). If the same spatial pseudoplaids (Fig. 11, F and G) are convolved with a Gaussian with a wider space constant (Fig. 11H), the resulting area of overlap is more extensive (Fig. 11, I and J), yielding a higher pattern index.

Fig. 11.

A model of the spatial integration of component motion signals. A and B: schematics of the location of the 2 component gratings (light and dark gray), shown for spatial pseudoplaid in which these signals were presented in many/small patches or few/large patches (16 and 4 patches, respectively). C: a spatial (2-dimensional Gaussian) filter is applied to each of the stimuli in A and B; this filter has a small space constant (σ). D and E: the resulting spatially blurred stimuli have regions in which the 2 component gratings overlap in space (red) and are simultaneously available for computing pattern motion. Darker areas represent greater spatial overlap; each contour line represents 4% overlap. If a filter with a wider space constant (H) is applied to the same stimuli (F and G), the area of spatial overlap will increase (compare E and J).

To quantify the spatial limit on pattern integration, we computed the value of σ that best accounted for the observed decay in pattern index for each neuron; fits were well matched to the data (median R² = 0.997, P < 0.05). We constrained patches to contain at least one cycle of grating at the optimal spatial frequency, so most neurons (66/87) could not be tested with spatial pseudoplaids containing more than 16 patches. As a result, if for a given neuron the pattern index for pseudoplaids of 16 and 4 patches were similar (16/87), the model became degenerate and fit the data with an asymptotically small space constant (close to 0). These neurons effectively exhibited an “all-or-none” phenomenon for spatial integration (Majaj et al. 2007); any separation of the component gratings resulted in a complete breakdown of pattern direction selectivity.

Figure 12 shows the space constants plotted against the pattern indexes, computed from responses to true plaids, across the population. The median space constant was 9.4% of the receptive field (data from all-or-none neurons were excluded). Gray bands indicate the 95% confidence limits for the pattern index and σ, showing that our fits were not biased and that we were not overfitting the data. Most space constants fell between 3% and 20% of the receptive field diameter, with no particular relationship to the pattern index. The fitted space constants were somewhat larger than the classical receptive fields of V1 neurons at the eccentricities tested but comparable to the size of their suppressive surrounds (Angelucci et al. 2002; Cavanaugh et al. 2002).

Fig. 12.

Population space constants. A scatter plot of the space constant derived from model fitting and the pattern index for all neurons tested (n = 87). Each point represents data from 1 neuron; data for the example neuron (in Fig. 8) is highlighted (black). Space constants for neurons whose selectivity for pattern motion broke down even for the smallest spatial separation are marked separately as “all-or-none.” Bounds for classifying PDS and CDS neurons are shown by dotted lines. The median space constant was 9.4% of the MT receptive field (inverted triangle). There was no clear relationship between the space constants and the pattern index. RF, receptive field.

We wondered if the time and space constants of individual MT neurons were correlated. Figure 13 shows a scatter plot of these derived metrics for 61 neurons that were tested with both temporal and spatial pseudoplaids; of these, 2 had undefined time constants and 10 had near-zero space constants (i.e., exhibiting an all-or-none behavior). We found no correlation between these measures and conclude that the limits of temporal and spatial integration are unrelated (Spearman's r: −0.14, P = 0.35).

Fig. 13.

Relationship between temporal and spatial integration. A scatter plot of the time constants and space constants for all neurons whose temporal and spatial integration were tested with temporal and spatial pseudoplaids (n = 61). Each circle represents data from 1 neuron; data were derived from the model fits. There was no correlation between the time and space constants across neurons, indicating that the mechanisms underlying the temporal and spatial integration of component motion signals were unrelated.

DISCUSSION

Neurons in extrastriate area MT signal the direction of moving visual patterns, and their responses have been directly linked to the perception of visual motion (Britten et al. 1992; Movshon et al. 1985; Newsome and Pare 1988; Salzman et al. 1990). For pattern motion to be computed accurately, MT neurons must integrate component motion signals across time and space. We examined the temporal and spatial integration limits that govern this pattern motion computation with pseudoplaids in which the component gratings alternate in time or space (Fig. 1). These stimuli allowed us to vary the temporal and spatial structure of the component motion signals systematically. Neuronal selectivity for pattern motion broke down rapidly when the component gratings were temporally or spatially separated. This loss of pattern direction selectivity was captured by simple descriptive models (Figs. 6 and 11) in which the component motion signals were first low-pass filtered (blurred) before being combined. The inferred characteristics of these blurring operations are of interest because they suggest the temporal and spatial structure of the mechanisms that compute pattern motion.

Temporal and spatial limits to motion integration.

For temporal pseudoplaids, all MT neurons, regardless of their sensitivity to pattern motion in true plaids, showed reduced sensitivity to pattern motion as the alternation period of the two component gratings increased (Fig. 4). The loss of pattern direction selectivity at the longest alternation periods was expected because the stimulus had long epochs in which the component gratings were presented alone. Of interest, however, is the rate at which pattern direction selectivity fell with alternation period. In our model, this is well described by a single time constant, capturing the temporal integration properties of the entire network involved in the pattern motion computation. Our observation that this is about 10 ms (Fig. 7) suggests that the integration of motion signals takes place with a relatively short timescale, comparable to the timescale with which individual pyramidal cells operate (Cardin et al. 2007; Connors et al. 1982; McCormick et al. 1985; Nowak et al. 2003) and similar to the timescale with which MT neurons signal simple, one-dimensional motion (Bair and Movshon 2004). This would appear to rule out a computation with a recurrent architecture, which, because of the extra synaptic delays in a multineuronal circuit, is necessarily slower in its temporal resolution than the neurons of which it is composed.

Similarly, for spatial pseudoplaids, as the spatial separation of the component gratings increased and fewer patches were presented, all MT neurons showed a reduction in pattern direction selectivity (Fig. 9). The decay in the pattern index was gradual (Fig. 10) and was well captured by a model with a single stage of Gaussian spatial low-pass filtering (blur). The scale of this blur is small compared with an MT receptive field (about one-third of the receptive field size). This agrees with previous work showing that pattern motion seems to be computed locally in MT (Majaj et al. 2007). Our finding that the spatial integration of component signals is on the order of one-third of the MT receptive field size suggests that component signals presented outside this spatial window have little effect on a neuron's selectivity for the direction of pattern motion.

Our data show that MT neurons have different temporal integration limits depending on their pattern direction selectivity (Fig. 7). PDS neurons have relatively shorter time constants and are therefore less tolerant to temporal separations of component motion signals than CDS neurons. Consistent with this, PDS neurons may pool over afferents with faster normalization dynamics than CDS neurons. In contrast to our temporal integration finding, our data show that PDS and CDS neurons have similar spatial integration limits (Fig. 12). The most parsimonious explanation would be that these limits are set before the convergence of component signals in MT and reflect the properties of afferent mechanisms responsible for shaping the direction tuning of MT neurons.

Implications for models of pattern motion selectivity.

Many models seek to explain how MT neurons compute the motion of complex moving patterns (Bowns 2002; Grzywacz and Yuille 1990; Movshon et al. 1985; Nowlan and Sejnowski 1995; Simoncelli and Heeger 1998). The question is how to interpret our measured stringent spatial and temporal integration requirements for pattern motion in the context of these models.

The temporal requirements of the pattern motion computation seem to be consistent with a feedforward architecture in which afferent signals subserving the pattern computation are summed in a single neuronal layer. If the interaction between the component motion signals is elaborate, recurrent, or spread over multiple stages, it would be difficult to imagine how it could preserve the temporal fidelity of a single neuron, characteristic of the ∼10-ms time constants we inferred. Moreover, other studies have suggested that MT is driven by feedforward networks with high temporal precision (Bair and Movshon 2004; Buracas et al. 1998; Perge et al. 2005). Although our data do not rule out the role of recurrent networks in the pattern motion computation, they can be parsimoniously explained using a feedforward mechanism.

It is not entirely obvious how to best interpret our measurements of the spatial requirements for pattern motion integration. Recent attention has been focused on models of MT motion integration that are largely feedforward in their architecture (Nishimoto and Gallant 2011; Rust et al. 2006), but these do not deal directly with the spatial structure of MT inputs, and the classical view of MT receptive fields is that they integrate signals across their entire receptive field, although many of these results simply show that signals combined more or less additively across the MT receptive field. In addition, the results of Majaj et al. (2007) and Hedges et al. (2011) imply that signals arising from different locations in the MT receptive field are largely independently processed, which is consistent with our findings.

The model by Rust et al. (2006) provides a framework with which we can understand our results. In this model, the activity of MT neurons is the result of a cascade of signal transformations occurring both in area MT and in its afferents (from V1 and perhaps other cortical areas). The activity of the afferent neurons is modified by two forms of gain control. The “untuned” form corresponds to the classical contrast gain control in V1 (Carandini et al. 1997; Geisler and Albrecht 1992; Heeger 1992), whereas the “tuned” form is related instead to the documented properties of surround suppression in V1, where responses are suppressed by stimuli near but outside the receptive field that are similar in orientation and direction to those driving the classical receptive field center (Angelucci et al. 2002; Cavanaugh et al. 2002; Sceniak et al. 2001). This model assigns a particularly important and specific role to this tuned gain control mechanism in the computation of pattern motion.

The spatial limits on pattern motion integration are too large in extent to correspond in a simple way to the classical receptive fields of individual V1 neurons, typically 10–20% the diameter of receptive field sizes of MT neurons at the same eccentricity (Cavanaugh et al. 2002; Desimone and Ungerleider 1986; Freeman and Simoncelli 2011; Gattass et al. 1981). Moreover, neither V1 nor V2 neurons show significant sensitivity to pattern motion (Gegenfurtner et al. 1996; Movshon et al. 1985). However, the extent of the limits on pattern motion integration correspond quite well to a larger area corresponding to the extent of the surround of the receptive field of V1 neurons, typically three times larger than the receptive field center (Angelucci et al. 2002; Cavanaugh et al. 2002; Tsui et al. 2010). This means that the falloff in pattern motion sensitivity for pseudoplaid stimuli corresponds to the spatial extent of the surround; when pseudoplaids place gratings of different orientations in the surround, tuned normalization is weakened or abolished, and in the model of Rust et al. (2006), this would in turn weaken pattern motion sensitivity.

Relationship to perception.

In this study we have determined the temporal and spatial limits on pattern motion selectivity under opiate anesthesia, and from these we have inferred something of the bottom-up structure of the neural computations that set these limits. When motion is presented in the natural world, it is often complex, consisting of multiple component motions; components that are not always necessarily present at all times and in all regions of space, and not always associated with single objects. Previous psychophysical studies have explored how the spatial integration of nonoverlapping component motions influence the perception of pattern motion (Alais et al. 1998; Amano et al. 2009, 2012; Dobkins et al. 2004; Lorenceau 1998; Lorenceau and Zago 1999; Mingolla et al. 1992; Rubin and Hochstein 1993). However, it is difficult to relate these results to our findings quantitatively because these studies often used component signals with sizes and spatial frequencies incommensurate with eccentricity. Qualitatively, our results are in agreement: the higher the density of component signals, the stronger the pattern motion sensitivity. Furthermore, psychophysical studies have shown that the integration of motion signals over space is strongly influenced by perceptual context and scene organization (McDermott et al. 2001; see Nishida 2011), factors that played no role in our experiments. Further investigations of the influence of these top-down signals can now be framed by the basic computational structures that we have uncovered.

GRANTS

This work was supported by National Eye Institute Grants EY002017 and EY004440 (to J. A. Movshon) and by a Robert Leet and Clara Guthrie Patterson Trust Postdoctoral Fellowship (to R. D. Kumbhani).

DISCLOSURES

No conflicts of interest, financial or otherwise, are declared by the authors.

AUTHOR CONTRIBUTIONS

R.D.K. and J.A.M. conception and design of research; R.D.K. and Y.E.-S. performed experiments; R.D.K. analyzed data; R.D.K., Y.E.-S., and J.A.M. interpreted results of experiments; R.D.K. prepared figures; R.D.K. drafted manuscript; R.D.K., Y.E.-S., and J.A.M. edited and revised manuscript; R.D.K., Y.E.-S., and J.A.M. approved final version of manuscript.