Interaction of cortical networks mediating object motion detection by moving observers

Abstract

The task of parceling perceived visual motion into self- and object motion components is critical to safe and accurate visually guided navigation. In this paper, we used functional magnetic resonance imaging to determine the cortical areas functionally active in this task and the pattern connectivity among them to investigate the cortical regions of interest and networks that allow subjects to detect object motion separately from induced self-motion. Subjects were presented with nine textured objects during simulated forward self-motion and were asked to identify the target object, which had an additional, independent motion component toward or away from the observer. Cortical activation was distributed among occipital, intra-parietal and fronto-parietal areas. We performed a network analysis of connectivity data derived from partial correlation and multivariate Granger causality analyses among functionally active areas. This revealed four coarsely separated network clusters: bilateral V1 and V2; visually responsive occipito-temporal areas, including bilateral LO, V3A, KO (V3B) and hMT; bilateral VIP, DIPSM and right precuneus; and a cluster of higher, primarily left hemispheric regions, including the central sulcus, post-, pre- and sub-central sulci, pre-central gyrus, and FEF. We suggest that the visually responsive networks are involved in forming the representation of the visual stimulus, while the higher, left hemisphere cluster is involved in mediating the interpretation of the stimulus for action. Our main focus was on the relationships of activations during our task among the visually responsive areas. To determine the properties of the mechanism corresponding to the visual processing networks, we compared subjects' psychophysical performance to a model of object motion detection based solely on relative motion among objects and found that it was inconsistent with observer performance. Our results support the use of scene context (e.g., eccentricity, depth) in the detection of object motion. We suggest that the cortical activation and visually responsive networks provide a potential substrate for this computation.

Introduction

Much is known about how the brain processes visual motion. Psychophysical sensitivity has been characterized and the neural substrate of a vast number of specific motion tasks has been identified. However, most of this knowledge is specific to how a stationary observer perceives different aspects of motion or how a moving observer uses the pattern of visual motion (optic flow) generated by static objects to judge the direction of self-motion (heading). Comparatively, little is known about how the human brain identifies the movement of objects while the observer is also moving, although ecologically, this is one of the most ubiquitous motion processing problems.

During self-motion, the entire visual field moves, and in order to determine whether (and how) objects are moving within it, the visual system has to separate information about the movement of the object from the optic flow produced by the observer's self-motion. There is robust evidence that the visual system can effectively use extra-retinal cues (proprioceptive and vestibular information, efference copies of the motor command, concurrent auditory cues, etc.) to contribute to the separation of the sources of perceived motion (Wallach 1987; Gogel 1990; Vaina et al. 2010a; Calabro et al. 2011).

However, recent studies suggested that visual information alone is sufficient to estimate an object's motion during self-motion (Rushton and Warren 2005). In order to accurately detect moving objects in this situation, it is not enough to simply detect retinal motion, since the entire scene is moving. Several possibilities exist for how observers may solve this problem. For example, it has been shown that observers are sensitive to deviations of a moving object from a radial flow field, both based on the object's direction (Royden and Connors 2010) and speed (Royden and Moore 2012). However, relying on an object's direction, deviation alone would fail for objects moving on trajectories parallel to the observer (e.g., moving cars approaching in a lane adjacent to the observer). While deviations from the radial speed gradient could identify such objects, such deviations also occur for static objects at different depths due to motion parallax. Distinguishing moving objects from parallax-induced motion requires knowledge of the scene's 3D configuration and the object's location within the scene. Because of this, relying on relative motion cues alone may not be sufficient to accurately and reliably detect moving objects during self-motion.

One mechanism that has been proposed to solve this problem suggests that observers may use their sensitivity to optic flow to estimate and subtract self-motion from the observed flow field to isolate object motion, termed flow parsing (Rushton and Warren 2005; Rushton and Duke 2007; Warren and Rushton 2007, 2009). By subtracting the induced self-motion from the visible flow field, the motion that remains reflects scene-relative object motion, or parts of the scene where the motion cannot be explained solely by the observer's movement. If performed using the 2D flow field alone, this approach would suffer the same difficulty in distinguishing parallax-induced motion from world-centric object motion as the relative motion strategy discussed above, but if based on 3D motion vectors, or if using a 3D scene reconstructions, this computation would accurately indicate moving objects. This approach is consistent with results showing that the presence of a self-motion optic flow field induces a world-centric frame of reference when observers perceive 3D object motion (Matsumiya and Ando 2009).

A similar approach has been suggested by Pauwels et al.'s (2010) biologically inspired parallel processing model for the extraction of object motion by a moving observer. In a six-stage hierarchical model based on the computational properties of the dorsal visual processing stream, the authors demonstrate the effectiveness of a distributed, parallel processing hierarchal architecture for the separation of self- and object motion. This may suggest that the neural implementation of object motion detection during self-motion is likely to draw upon a distributed network of cortical areas in the dorsal stream. To determine the neural underpinnings of object motion detection in humans, it is therefore important both to establish the areas involved in this task as well as how those areas communicate and organize into networks.

In this paper, we were interested in determining whether subjects use a simple (though inaccurate) relative motion computation to detect moving objects during self-motion or whether they incorporate scene context when detecting object motion. Further, we aimed to determine the brain areas and networks that mediate object motion detection in the presence of self-motion. We addressed these questions by combining psychophysics, functional magnetic resonance imaging (fMRI) and functional connectivity analysis of the fMRI data using partial correlation and multivariate Granger causality analyses to identify the functional areas and the connected networks involved in the detection of a moving object during self-motion. We suggest that object motion extraction and detection is mediated by distinct cortical networks as revealed by a clustering analysis of the connectivity data. The results show two clusters of visually responsive areas that are likely involved in the detection of object motion and scene context, and a cluster of fronto-parietal areas involved in higher level functions such as the interpretation of the stimulus for action.

Methods

Subjects

Seven subjects (ages 19–26, mean 21.5; 4 female) participated in the fMRI scans. Subjects were enrolled in the study after they had given their written informed consent; the study protocol was approved by the Massachusetts General Hospital institutional review board and followed the guidelines of the Declaration of Helsinki. All subjects had normal or corrected to normal vision, were right handed, and none had any history of neurological or psychiatric illness. Subjects participated in at least 1 h of “pre-fMRI” practice on the task to ensure that performance had stabilized.

Apparatus

Stimuli were generated on and presented by a MacBook Pro running Matlab using the Psychophysical Toolbox (Brainard 1997; Pelli 1997) and OpenGL libraries and projected by a Notevision6 LCD projector onto a translucent acrylic screen. Subjects saw the screen through a Buhl Optical collimating lens placed on the head coil. To ensure accurate synchronization, a USB trigger code was sent at the start of each fMRI acquisition to the stimulus presentation computer, which was used to begin the visual stimuli. Subject responses were recorded with a 4 button, MRI compatible button box.

Object motion stimulus and task

Stimuli consisted of 9 spherical objects distributed within a 25 × 25 × 60 cm simulated OpenGL environment. Objects were high contrast (28.3 cd/m² on a 0.3 cd/m² background) textured spheres with a mean initial diameter of 1.5°. The display area was divided into 9 equally sized wedges, each containing one object at a random eccentricity up to 9° (using a square-root distribution to create a uniform density), to prevent occlusion. Subjects were instructed to fixate a red cross (20 × 20 arcmin) at the center of the display throughout the testing period. The display was viewed binocularly, but no stereo cues were presented, so depth was inferred only through changes in object size (Fig. 1).

Fig. 1. Stimulus illustration. All objects expanded during simulated self-motion. Observers had to detect the object that had an additional, independent motion vector.

Each trial began with the display of a static scene containing 9 objects, with contrast at 0 % and gradually increasing so that the objects became visible (2 s), then remaining visible and stationary (1 s). Objects were then moved and scaled consistent with forward observer translation of 3 cm/s (relative to a 30 cm simulated distance to the objects, such that the radial velocity was up to 1.66°/s for the most eccentric objects, or 0.84°/s for objects of mean eccentricity), lasting 1 s in duration. One object (the “target” object) had an independent forward or backward motion vector of 2,4, 6 or 8 cm/s within the scene in addition to the induced self-motion. After the motion interval, the screen was cleared for 250 ms. Then, the target and three other randomly selected objects re-appeared and were labeled with numerals, 1–4. Subjects performed a 4AFC task to identify which object was the target (although subjects had to monitor all 9 objects since the labels did not appear until after the stimulus motion finished).

fMRI scanning sessions consisted of four 240 s acquisitions. Within each acquisition (details below), target speeds and directions were mixed, repeating each condition 5 times. This resulted in 160 total trials per subject (40 trials per target speed). Performance on the 4AFC task was converted to d′ (Green and Bridsall in Green and Swets 1966), such that 0 indicated chance performance on the task.

A control fMRI task was used to determine which activations were due to the object motion component of the task. In the control fMRI task, subjects were presented with the same textured objects, but their motion resulted from the self-motion only, thus creating a radial motion pattern (with no target object moving independently). Subjects were asked to report the direction of self-motion (forward or backward). This task was used to determine activation due to the presence of moving objects from simulated self-motion (but without the segmentation and detection of an independently moving object), or to the right-handed button press response. This task was matched to the stimulus parameters (self-motion speed, object sizes and luminance), but not to task difficulty.

MR scanning and data analysis

Data acquisition

Data were collected at the Athinoula A. Martinos Center for Biomedical Imaging, using a 3T Siemens TrioTim 60 cm (RF coil ID) whole-body MRI. Two high resolution, 3D T1-weighted structural MRI scans were first obtained for registering the functional data using a 3D magnetization-prepared rapid acquisition gradient echo (MPRAGE) sequence (TR 2,530 ms, TE 3.39 ms, inversion time 1,100 ms, flip angle 7°) with 128 slices of 1.33-mm thickness and 256 × 256 in-plane sampling (1 × 1 mm resolution). Functional volumes were acquired using an interleaved, gradient echo EPI sequence every 2 s for each 6 min acquisition (TR 2,000 ms, 180 TRs; TE 30 ms, flip angle 90°, distortion factor = 20 %, phase = 100). Four acquisitions were obtained for each subject. We acquired 33 slices of 3-mm thickness spanning the entire cerebral cortex, with in-plane sampling of 64 × 64 (resolution of 3.125 × 3.125 mm). Slice positions were based on an AutoAlign sequence for consistency in slice positioning across subjects (van der Kouwe et al. 2005).

Functional analysis

Data processing and analysis was performed using Freesurfer v4.5 to obtain a 3D anatomical reconstruction of the cortical surface and registration to Talairach coordinates (Dale et al. 1999; Fischl et al. 1999). Functional data were automatically registered to the structural reconstruction, then manually checked and adjusted to ensure accurate alignment. Functional data were motion-corrected using the AFNI motion correction tool and spatially smoothed with a Gaussian smoothing kernel with a full-width, half-maximum of 5 mm. Optimized event-related sequences (Burock et al. 1998; Burock and Dale 2000) were used to specify the timing of visual stimuli. Events were coded post hoc based on target speed, target direction and subject response (correct/incorrect). Activation contrasts were computed between the motion and static intervals to isolate activity associated only with the motion of the stimuli, as well as for per-trial activation as a function of direction, speed and response, using a group weighted random effects model. To identify voxels with task-specific responses for both the object motion and control tasks, a first-level statistical analysis was performed using a gamma HRF model including motion correction regressors. Individual activations were combined in a group analyses by first mapping each subject's activation to the MNI305 brain and then using a surface-based weighted random effects linear model on the concatenated time courses across subjects. Regions of interest (ROIs) were chosen as surface clusters of at least 80 mm with group activation of p < 0.01. To avoid biasing the results toward or against areas with specific correlations to behavior, all speed conditions were combined when defining ROIs. ROIs were automatically mapped to each subject's native space using their spherical registrations. When available, functional labels were assigned on the basis of the literature-defined functional areas with similar Talairach and Tournoux (1988) coordinates (http://collaborate.bu.edu/bravi/FAQ/BrainAreasSummary). Areas not belonging to known functional visual areas were assigned anatomical labels as determined during the anatomical reconstruction. In order to confirm which of our functionally active corresponded to hMT, each subject was given an hMT localizer using astandard method (Tootell et al. 1995), which we have previously employed (Michels et al. 2005).

Connectivity and network analysis

To determine the functional associations and connectivity among ROIs, we first computed partial correlations based on the time courses of all functionally defined ROIs. Connectivity analyses were based on the entire original (non-spatially smoothed) 180 TR time courses, which were motion corrected and slice-time corrected (Smith et al. 2004), and averaged across all functionally active voxels (p < 0.01) within each ROI and across all four test runs. Associations were computed as the linear partial correlation between each pair of ROIs conditioned on the time courses of all other ROIs using the partialcorr function in the Matlab statistics toolbox. For each association, a correlation coefficient and significance (based on the t statistic) were calculated and used to establish which areas had functional associations. p values were combined across subjects using Fisher's method to obtain a χ² statistic and group p value. Group p values were corrected for multiple comparisons using a false discovery rate correction with FDR <=0.01.

Community structure was investigated using two approaches. First, a node-reordering algorithm (Rubinov and Sporns 2010) that clusters highly connected nodes was applied to the partial correlation map. This resulted in a sequence in which nodes with similar patterns of connectivity were located in proximity. Second, a weighted, undirected modularity algorithm (Newman 2006; Leicht and Newman 2008) was applied to the partial correlation data. The algorithm finds non-overlapping clusters of nodes by maximizing the number of within-cluster connections while minimizing the across-cluster connections, resulting in clusters of highly connected areas.

To hypothesize about the directionality of connections, we coupled the partial correlation approach with a first-order, multivariate Granger causality (mGC) (Granger 1969; Kaminski et al. 2001; Roebroeck et al. 2005) analysis. mGC was computed for pairs of ROIs that had a significant (FDR < 0.01) associations as determined by the partial correlation analysis. Masking the Granger connections by the significant partial correlations was done to reduce the likelihood of false positives (see Supplement). Connections among pairs of ROIs are conditioned by the time courses of all other ROIs and are evaluated by determining whether knowing the time points in ROI-2 improves the prediction for ROI-1 after all other ROIs have been included. For each subject, mGC was computed by using the Granger Causality Connectivity Analysis toolbox for Matlab (Seth 2010), which tests whether the Granger coefficients are significantly different than zero using an F statistic and their corresponding p values. Group data were computed using Fisher's method, as in the partial correlation analysis.

Results

Neural substrate for object motion detection by a moving observer

Group activation was computed using a weighted random effects model across all subjects for activation during the object motion task relative to a baseline condition in which all nine objects were visible, but stationary (Fig. 2). Based on group activity, we defined nine bilateral, five left hemisphere only and one right hemisphere only ROIs (see Table 1).

Fig. 2.

Weighted random effects model of group activation (n = 7) on the object motion task compared to an interval with all objects present, but static registered to the MNI305 standardized brain. Color indicates significance of activation [−log(p)]. Only significant voxels falling within surface clusters of at least 80 mm² are shown. Blue text indicates functionally active ROIs, as listed in Table 1. Anatomical labels (white text) include the transoccipital sulcus (TOS), inferior temporal sulcus (ITS), superior temporal sulcus (STS), central sulcus (CS), post-central sulcus (PoCS), intra-parietal sulcus (IPS) and parieto-occipital sulcus (POS)

Table 1. Mean values across subjects for defined ROIs.

ROI	Talairach coordinates			−log(p)	% Signal change	# Vtx	# Subj	BA
	x	y	z
LH V1	−12.76	−76.24	5.52	3.96 (1.31)	0.82 (0.27)	128.1 (42.49)	7	18
RH V1	10.20	−78.02	10.38	4.02 (1.01)	0.85 (0.23)	123.7 (47.30)	7	17
LH V2	−11.20	−73.92	16.29	4.25 (1.31)	0.82 (0.25)	123.4 (47.33)	7	18
RH V2	14.26	−69.64	−0.57	3.81 (0.90)	0.89 (0.21)	92.3 (31.42)	7	18
LH LO	−22.73	−72.93	−5.00	3.55 (0.59)	0.91 (0.10)	115.6 (36.98)	7	18
RH LO	27.82	−77.54	−5.37	3.75 (0.62)	1.14 (0.25)	119.1 (36.49)	7	18
LH V3A	−24.13	−90.07	10.40	2.76 (0.69)	0.65 (0.13)	18.9 (12.19)	7	18
RH V3a	29.83	−80.95	16.27	3.07 (0.41)	0.79 (0.06)	67.4 (17.04)	7	19
LH KO	−34.48	−79.08	10.97	3.41 (1.12)	0.79 (0.18)	30.7 (15.97)	7	19
RH KO	33.89	−82.39	5.72	3.12 (0.63)	0.85 (0.12)	28.9 (9.91)	7	19
LH hMT	−39.64	−70.34	−7.05	4.37 (1.27)	1.26 (0.22)	91.9 (27.50)	7	19
RH hMT	44.82	−63.36	−4.06	3.42 (0.77)	1.15 (0.28)	30.9 (18.58)	7	37
LH VIP	−31.39	−72.16	24.27	4.21 (1.91)	0.99 (0.32)	32.1 (8.43)	7	19
RH VIP	30.96	−66.43	30.51	3.54 (0.54)	0.99 (0.10)	66.4 (16.77)	7	39
LH DIPSM	−19.91	−62.13	38.02	3.60 (0.82)	1.11 (0.15)	17.1 (5.34)	7	7
RH DIPSM	38.74	−45.20	37.24	2.81 (0.49)	0.78 (0.11)	30.7 (19.35)	7	40
LH PoCS	−44.28	−24.83	40.87	5.99 (1.26)	1.35 (0.30)	63.7 (29.18)	7	2
LH PoCGd	−44.29	−20.36	49.80	4.49 (0.85)	1.13 (0.27)	45.6 (10.97)	7	3
LH PoCGv	−59.12	−10.40	27.00	3.59 (1.65)	0.82 (0.27)	15.1 (6.54)	7	4
LH CS	−47.40	−6.49	28.79	2.64 (0.91)	0.67 (0.12)	16.0 (9.13)	7	6
RH CS	57.69	−9.44	22.02	2.73 (0.61)	0.67 (0.20)	14.3 (6.68)	7	4
LH SubCS	−51.41	−14.76	16.36	3.95 (0.67)	0.85 (0.17)	25.7 (7.02)	7	43
LH FEF	−34.79	−16.18	57.69	4.94 (1.95)	1.49 (0.59)	16.4 (4.93)	7	4
RH Precun	9.31	−59.49	48.16	3.34 (1.08)	1.06 (0.27)	23.0 (8.39)	7	7

Anatomical area is based on the group results mapped to the MNI305 brain. Activation values are shown as significance [ −log(p)] and mean % BOLD signal change. Values in parentheses are standard deviation across subjects. Talairach coordinates are the mean of all active voxels in the ROI. “# Subj” is the number of subject for whom the ROI was defined (all subjects for each ROI), and BA indicates the Brodmann Area of the mean Talairach coordinate. ROIs include LO lateral occipital, KO kinetic occipital, hMT human motion complex, VIP ventral intraparietal, DIPSM dorsal intraparietal sulcus medial, PoCS post-central sulcus, PoCGd/v post-central gyrus dorsal/ventral, CS central sulcus, SubCS sub-central sulcus, FEF frontal eye fields, Precun precuneus

Cortical activation was distributed among occipital, occipito-temporal, parietal and parieto-frontal regions (Fig. 2). There was increased BOLD signal in the regions corresponding to the location of V1 and V2 (Fischl et al. 2008; Hinds et al. 2008), as well as several other cortical areas that have previously been shown to be strongly responsive to visual motion. Based on the Talairach coordinates of the center of mass of each area and on anatomical location, we identified these regions as corresponding to V3A (Tootell et al. 1997; Mendola et al. 1999; Sunaert et al. 1999; Vaina and Soloviev 2004; Vaina et al. 2010b), the kinetic occipital area (KO, or V3B, Smith et al. 1998; Tyler et al. 2006), the human motion complex (hMT, Tootell et al. 1993; Vaina et al. 1998, 2001; Sunaert et al. 1999; Ffytche et al. 2000; Orban et al. 2003) and the ventral intraparietal sulcus (VIP, Bremmer et al. 2001; Orban et al. 2003), as well as a large, ventral area in the lateral occipital area cortex (LO, Malach et al. 1995; Kourtzi and Kanwisher 2001; Amedi et al. 2002). Bilateral activation was found in the medial region of the dorsal intraparietal sulcus medial (DIPSM, Orban et al. 2003; Durand et al. 2009) and along the central sulcus. All these cortical areas had a significant bilateral response to the motion stimulus, and the activation and topographic location were consistent across subjects (see Table 1). In addition to the motion responsive ROIs, several fronto-parietal cortical regions showed significant and consistent activation across subjects. Increased activation was present in the left hemisphere including two regions of the post-central gyrus (ventral and dorsal, PoCG-v and PoCG-d, respectively), the post-central sulcus, sub-central sulcus and frontal eye fields (FEF, Orban et al. 1999). Right hemisphere-specific activation was found only in the precuneus.

Activations of areas V3A, KO, hMT, VIP and DIPSM were significantly greater in the object motion task than in the control task that displayed only the expanding pattern motion of the objects. However, in both the object motion and control tasks, there was a similar, overlapping activation in V1, V2 and LO (see Supplemental Figures 1 and 2). Among the fronto-parietal left hemisphere areas, activation was observed only in the central sulcus, though shifted slightly compared to the object motion task. Since subjects gave responses using the same right-handed button press in both tasks, these results suggest that the left CS activation may be attributed to the motor response associated with pressing a button, but the other left fronto-parietal activation seen in the object motion task cannot be attributed to the button press.

We used subjects' responses recorded during the scans to measure task condition-dependent performance. Figure 3 summarizes subject performance as a function of target speed and direction. Performance in the scanner was significantly worse than the performance of these subjects during pre-fMRI training (3-way ANOVA for testing location: F_1,106 = 27.44, p < 0.001, speed: F_3,106 = 54.29, p < 0.001, and direction: F_1,106 = 28.97, p < 0.001), likely due to better testing conditions in the lab, but in both cases, the trends (performance proportional to speed; asymmetry between approaching/receding objects) were maintained from our previously reported psychophysical performance on this task (Calabro et al. 2011). Event-related, trial-by-trial performance data used in an exploratory voxel-wise analyses did not show any significant group activation clusters associated with the direction of the target object, nor between trials with correct and incorrect responses.

Fig. 3.

Performance of subjects during fMRI scans (d′) as a function of the speed of the target object both during the fMRI scans, and in pre-scan testing in the laboratory. Trials in which an invalid button press was recorded were excluded. Negative speeds indicate receding objects; positive speeds indicate objects approaching the observer. Data points are offset for clarity, but used the same speeds (± 2, 4, 6, 8)

However, there were significant correlations between BOLD signal and performance when comparing across different speeds. For every subject, we normalized the BOLD signal and task performance in the scanner for each speed (grouped across directions). Normalization was relative to the subject's own overall BOLD signal and performance, respectively, combined across all conditions (based on Gilaie-Dotan et al. 2001). We computed correlation between the normalized BOLD signal and performance with a generalized linear model using the data from all subjects (Fig. 4). The activation of several areas was significantly correlated to behavior after correcting for multiple comparisons (FDR < 0.05): bilateral V1 (lh: t_1,26 = −3.38, p = 0.0023; rh: t_1,26 = −2.83, p = 0.008), rh V2 (t_1,26 = −4.64, p < 0.001), rh V3A (t_1,26 = −2.26, p = 0.03), lh KO (t_1,26 = −2.48, p = 0.02), lh LO (t_1,26 = −3.16, p = 0.004), lh hMT (t_1,26 = −2.80, p = 0.001) and lh VIP (t_1,26 = −2.53, p = 0.017). Interestingly, only cortical regions known as visually responsive areas had significant correlations to behavior, and all correlations were negative (there was less BOLD signal as performance improved). The correlation between BOLD signal and performance is in agreement with previous studies showing increased BOLD response with more difficult stimuli (Heekeren et al. 2004; Kayser et al. 2010), showing that activation in these areas depended on stimulus parameters and suggesting that these areas are strongly involved in forming a perceptual representation of the task.

Fig. 4.

Behavioral correlations between normalized performance during the scan and BOLD signal. Each data point corresponds to a single target object speed (grouped across directions) for a single subject. Each subject is indicated by a different marker symbol

Cortical connections mediating object motion detection

Partial correlations were computed among all ROIs and are shown in Fig. 5. A modularity algorithm (Rubinov and Sporns 2010) applied to the partial correlation adjacency matrix revealed 4 clusters of ROIs. One cluster (black boundary in Fig. 5) consisted of bilateral V1 and V2 with 5 of a possible 6 associations statistically significant after correcting for multiple comparisons (ρ > 0.1, p < 0.001). The intra-hemispheric V1 to V2 associations were strongest, with ρ = 0.35 and 0.47 for the RH and LH, respectively. A second cluster (Fig. 5, green boundary) consisted of visually responsive areas including bilateral LO, V3A, KO and hMT had both within- and cross-hemispheric associations. Within this cluster, 21 of 28 associations were significant, with strongest correlations between intra-hemispheric hMT and LO (ρ = 0.27 and 0.20 for LH and RH, p < 0.0001) and between KO and V3A (ρ = 0.23 and 0.18 for LH and RH, p < 0.0001). A third network (Fig. 5, yellow boundary) involved visually responsive areas in the parietal lobe (bilateral VIP and DIPSM), and the right precuneus, with 8 of 10 associations significant. Unlike the previous two clusters that were mostly within one hemisphere, the strongest association in this cluster was cross-hemispheric, between LH and RH VIP (ρ = 0.35, p < 0.0001). Finally, a fourth cluster involving higher level, primarily left hemisphere areas (Fig. 5, blue boundary) included PoCS, PoCG (dorsal and ventral), FEF, SubCS as well as bilateral CS. The node-reordering algorithm (the sequence of ROIs listed in Fig. 5) resulted in a sequence in which all ROIs were placed next to the rest of the ROIs in their cluster with one exception (left V3A), reinforcing the clusters found by the modularity algorithm.

Fig. 5.

Connectivity map among the functionally defined ROIs showing connections that were significant among subjects (FDR < 0.01) as measured by a partial correlations and b multivariate Granger causality. Color indicates the uncorrected significance of the group connection [ −log(p)]. In (a), connections are undirected, and for directional connections in (b), origin area (“connections from”) is indicated on the y-axis and destination area (“connections to”) on the x-axis

We compared the partial correlation results to those obtained in the control task (self-motion only) (see Supplemental Figure 3). While the control task elicited the same connectivity in the V1/V2 cluster, the connectivity in the LO/KO/hMT/V3A and left fronto-parietal clusters was sparser. Furthermore, the connectivity was almost absent in the VIP/DIPSM cluster. This suggests that the connectivity among the cortical network of intra-parietal sulcus areas is specific to the object motion task and may be involved in the separation of object and self-motion components from the perceived motion field.

Among the connections resulting from the partial correlation analysis, we were interested to see whether we could classify the directionality of the connection in order to speculate on the processing architecture supporting the object motion task. To this end, we measured mGC for all pairs of ROIs that had significant (FDR < 0.01) associations in the partial correlation analysis. This was done to mitigate potential false positives. Figure 5b shows the significance of the Granger coefficients for connections that were significant for both their mGC and partial correlation relationships. mGC measures the proportional change in the residual error of time course estimation to provide an estimate of which associations can be considered causal, directional connections. Interestingly, none of the associations within the V1/V2 cluster had significant causal connections, suggesting that these correlations were driven by a similar or common source, but did not directly affect each other. In the other clusters of visually responsive areas, two ROIs—right KO and right VIP—were the largest recipients of connections. Right KO had strong incoming connections from bilateral hMT and VIP, while right VIP was driven primarily by bilateral hMT and DIPSM. This suggests a network architecture in which visual information represented in hMT feeds into VIP, and both hMT and VIP feed into KO.

Object motion detection: scene context or relative motion?

To understand the functional roles of the activations and networks (described above), we investigated the psychophysical mechanism being used by subjects in the task. Specifically, we were interested to determine whether the stimulus motion alone (the optic flow field) could account for subjects' performance, or whether observers were incorporating scene-context information, such as the eccentricity and/or depth of the objects, when identifying the target object. Previous psychophysical studies have shown that subjects are sensitive to deviations in direction (Royden and Connors 2010) and speed (Royden and Moore 2012) from the optic flow field. We suggest two possible strategies that subjects could have employed to solve our task without incorporating scene context. First, subject responses could have been based solely on retinal speed of the objects. Although this strategy would be ineffective for most task conditions (since the target object was not generally the fastest or slowest object in the scene), it would produce more correct responses at high target speeds, similar to the behavioral trend shown by our subjects (Fig. 3). However, we have previously shown that performance did not change when the speed of observer motion changed (which shifted the distribution of retinal speeds), indicating that subject performance was not linked to absolute retinal speed (Calabro et al. 2011).

Second, subjects could have based their responses on the relative motion among objects, independent of their locations in the scene. By this strategy, subjects could have considered all the motion vectors (speed and direction) and choose the object with the maximum magnitude and/or unique direction (e.g., moving inward if all other objects are moving outward). To address whether observer performance could be explained by this relative motion strategy, we analyzed performance of all subjects while participating in the fMRI experiment for object speeds of 2, 4, 6 and 8 cm/s (both approaching and receding) during forward observer translation of 3 cm/s (Fig. 6, shaded region). The results showed that observers' results on the task was consistent with our previous reports of their performance outside the scanner and depended on both the speed and direction of the object motion, with faster and approaching objects easier to detect.

Fig. 6.

Observer performance (filled circles) on the object motion detection task compared to simulations using relative speed (open circles) or speed and direction (open squares) among objects to select the target. Shaded region indicates ±1 SD across subjects; *'s indicate data points where no correct responses were obtained from the simulations (d′ = −Inf)

We compared observers' performance to a model that selected a response based only on the relative motion among objects. The model had the motion vectors for all objects and selected its response as the object with the maximal (for expanding object motion conditions) or minimal (for receding conditions) speed or direction (inward/outward). Since object motion was always parallel to the observer's path (and line of sight), the object had no directional deviation from the radial flow field produced by the world-static objects, so that directional deviations did not provide a means to identify the target object. Thus, the directional information was limited to whether the object was moving inward or outward. The model therefore measured the proportion of trials in which the target could be uniquely identified on the basis of relative speed and/or direction. Since objects with a larger eccentricity had a faster self-motion-induced speed, and since each object had a randomly chosen eccentricity, the target object did not necessarily have the highest speed, which was the primary limitation on the model's performance. We performed simulations to determine whether a relative motion strategy could quantitatively explain subject performance by computing the percent of trials (of 10,000 simulated trials per object speed condition) in which the target object moved faster than any other object. Results are plotted in comparison with subject performance in Fig. 6. For approaching (positive) velocities, the relative speed strategy could account for performance of approaching object trajectories, but importantly, for receding objects (negative object velocities), the model did not provide above-chance correct responses, since the speed of the target object nearly always fell within the range of the speeds of the objects whose motion was due only to the motion of the observer.

Although receding objects could not be uniquely identified by their speed, they could be identified by their direction. When the target was receding faster than the observer's forward motion (object speeds of −4, −6, −8 cm/sec), it had a net motion away from the observer, and the target was the only object moving radially inward. Thus, we conjecture that a strategy combining speed and direction should result in task performance without any errors for receding objects, as verified by our simulations (Fig. 6, open squares). In contrast with the results of these simulations, human observers had significant difficulty in detecting receding objects (speeds < 0) and performed well below the prediction of the relative speed and direction strategy. A similar problem is encountered if we consider the use of motion-in-depth cues (expansion/contraction) of the objects, rather than direction. For receding objects, only one object would contract over time. Thus, the same problem occurs: if subjects were to use the “direction” of looming, we would expect their performance to be significantly better than we have observed. The comparatively poor performance for receding object conditions suggests that observer performance was not consistent with target selection strategies based on relative speed and/or direction among the objects.

Instead, the data suggest that subjects used more than just the motion vectors to determine whether an object moved relative to the scene during simulated egomotion (e.g., contextual information such as the object's location and depth within the scene). This is consistent with mechanisms of object motion detection such as the flow-parsing hypothesis put forward by Rushton and Warren (2005), Rushton and Duke (2007), Warren and Rushton (2007, 2009) and the parallel processing model of Pauwels et al. (2010), both of which consider the object relative to the entire scene to determine its world-centric object motion. However, whether either of these proposals or another explanation best represents the mechanism at work in our task of object motion detection remains an issue for future study.

Discussion

In this study, we examined the cortical areas and networks involved in the detection of a moving object by a moving observer. We used an ROI-based activation analysis, in conjunction with partial correlation and multivariate Granger causality analyses of connectivity, to show that the perception and representation of the stimulus is mediated by highly active and interconnected visually responsive areas, including LO, V3A, KO, hMT, VIP and DIPSM. In addition to the visual processing areas, activation was also found in a network of higher level, primarily left hemisphere fronto-parietal regions, including the CS, PoCS, PoCG, SubCS and FEF.

The results of the psychophysical experiment suggests that observers do not use a relative motion strategy for detecting moving objects during self-motion. Such an approach would be prone to errors by failing to discriminate moving objects from parallax-induced motion for objects at different depths. Instead, our psychophysical data and mathematical models suggest that to detect moving objects during egomotion, observers take into account the scene context of objects (e.g., position and/or depth within the scene) This is compatible with, but does not specifically indicate the use of, the flow-parsing models proposed by Rushton and Warren (2005) and Pauwels et al. (2010).

Network organization

The modularity of the connectivity pattern revealed that the distinct regions of activation in the object motion task could be divided into four clusters. Each cluster was distinguished by its functional properties, suggesting specific computational roles. The V1/V2 cluster was both activated and connected in the self-motion only control task, as well as the object motion task, suggesting that while these areas had increased activation in response to object motion, the difference was only in magnitude, and the involvement of these areas was not specific to the object motion task. The cluster including hMT/LO/V3A/KO was strongly activated and connected in the object motion task, but not the control task (with the exception of LO activation), suggesting that these areas had an increased role in the object motion task and were tightly linked to stimulus parameters (as revealed by the correlation to task difficulty). ROIs in the third cluster, containing VIP/DIPSM, were not activated nor connected to each other in the control task, but did contain regions correlated to behavioral performance, suggesting that they were involved in processing stimulus features not present in the control task. The ROIs in the cluster of primarily left hemisphere fronto-parietal areas were not activated by the control task, nor were they correlated to behavior in the object motion task. This suggests that the activation of ROIs in this cluster was specific to the object motion stimulus, but were not dependent on the stimulus properties per se.

In this study, we were interested primarily in two network clusters consisting of visually responsive areas, one consisting of bilateral LO, V3A, KO, hMT, and one of VIP and DIPSM. Activations in these areas had significant behavioral correlations (between BOLD % signal change and task difficulty, as Gilaie-Dotan et al. 2001; Vaina and Soloviev 2004), suggesting involvement in the perception and representation of the object motion stimulus. Furthermore, with the exception of LO, these areas had increased activation in response to the object motion stimulus compared to the control stimulus which contained only pattern motion consistent with self-motion. Previous neuroimaging studies have implicated these areas in a variety of aspects of visual motion processing. Area hMT has been significantly implicated in human motion processing, including tasks of navigation based on optic flow (for a review, see Vaina and Soloviev 2004). Recent studies have shown that activation in hMT is accompanied by parietal activity during detection and maintenance of path information during locomotion (Billington et al. 2010). Area VIP in the Macaque (Colby et al. 1993; Bremmer et al. 1997; Duhamel et al. 1998), and its human homolog identified by fMRI (Bremmer et al. 2001), responds to motion in depth and is particularly relevant since the object trajectories in our stimulus were in depth, toward or away from the observer. The kinetic occipital area (KO/V3B) has been associated with the detection of kinetic edges (Dupont et al. 1997; Van Oostende et al. 1997; Tyler et al. 2006). Area KO/V3B received connections from both hMT and VIP, suggesting that it pools information from these visually responsive areas. DIPSM has been linked to the perception and processing of 3D structure (Durand et al. 2009), which may be critical to determining the expected optic flow field based on self-motion alone.

This visually responsive network was complemented by a network of higher level, primarily left hemisphere, fronto-parietal regions (especially along the post-central sulcus and gyrus, and central sulcus). Of these regions, only the central sulcus was activated during the control fMRI experiment, suggesting that it could be attributed to the motor activation associated with the button press. Activation in the post-central gyrus has been associated with attention to the upper visual field, and in the pre-central gyrus with attention along the horizontal meridian (Mao et al. 2007), as well as pre-motor functions (Field and Wann 2005). We suggest that the role of these areas is to link perception to action planning. In particular, this network of cortical areas may be interpreting visual information represented by the networks of the visual areas, for planning actions that help a moving observer avoid or intercept a moving object.

Neuronal computations for object motion detection during self-motion

Our psychophysical data suggest that subjects do not perform the object motion task by simply comparing the motion of nearby objects, but rather incorporate scene context in their judgments. This allows for a more robust detection of moving objects since a strategy based on relative motion alone would be prone to errors in distinguishing object motion from induced motion parallax of static objects at different depths. Instead, we suggest that when estimating scene-relative object motion, observers use a strategy that incorporates scene context such as depth and position within the flow field.

We were particularly interested in the activation of hMT, since the properties of its homolog in the Macaque provide a plausible model for object motion detection during self-motion. Neurons in the Macaque dorsal medial superior temporal area (MSTd) have been shown to be highly selective to radial motion patterns, and self-motion (Tanaka and Saito 1989; Duffy and Wurtz 1991a, b), while the lateral MST (MSTl), responds well to motion properties associated with object motion (Tanaka et al. 1993; Eifuku and Wurtz 1998, 1999). In principle, these two areas might be thought to be sufficient for the task described in this study. However, our fMRI study showed that activation in many cortical areas involved in processing visual motion, including occipito-temporal and parietal areas, was correlated to behavioral performance. This suggests that, rather than hMT alone providing the neuronal substrate for object motion detection during self-motion, there is a network of occipital and parietal areas interacting to mediate object motion detection in our task.

Computationally, several models for object motion detection are consistent with our psychophysical data supporting the use of scene context and are interesting to consider given the cortical activation and connectivity exhibited during our task. One possibility is that observers perform a 3D vector subtraction, in which the difference between a self-motion translation vector and an ego-centric object motion vector would produce the scene-relative object motion. The drawback of this approach is that the 3D motion vector must be reconstructed (and vector subtraction must be computed) for every object in the scene, making it inefficient in search situations. Alternatively, the visual system could use a 3D representation of the scene and perform a mental translation to produce an expected flow field. A 2D vector subtraction between the expected and actual flow fields would leave motion only at those locations in the scene where scene-relative object motion exists. Although this would be more efficient for search, it requires an accurate model of the absolute depth and position of objects in the scene. The latter proposal mirrors the model proposed by Pauwels et al. (2010), which isolated moving objects when observed by a moving (binocular) camera using known cortical visual processing computations (e.g., edge detection, stereo extraction, optic flow), including the subtraction of ego-flow from optic flow.

Both approaches involve 3D scene and object manipulations, as well as a difference operation, either between the self- and object motion vectors or between the expected and observed flow fields. The association of intra-parietal areas to 3D structure-from-motion processing provides the possibility that the activation we have seen in VIP and/or DIPSM is related to this 3D manipulation. Furthermore, previous studies have linked KO/V3B to the processing of kinetic boundaries across stimulus types (Van Oostende et al. 1997). The connections from hMT and VIP to KO/V3B therefore suggest an intriguing possibility that KO/V3B provides the substrate for the difference operation required by models of object motion detection, by computing the difference between the visual representations provided by hMT and VIP, respectively.