Topological change disturbs object continuity in attentive tracking

Abstract

The question of what is a perceptual object is one of the most central and also controversial issues in cognitive science. According to the topological approach to perceptual organization, the core intuitive notion of an object—the holistic identity preserved over shape-changing transformations—may be characterized precisely as topological invariance. Here we show that, across a series of multiple-object tracking tasks, performance was not disrupted when the moving items underwent massive featural changes. However, performance was significantly impaired when the items changed their topological properties of holes, demonstrating that topological invariance constrains what counts as an object in the first place. Consistent with previous findings, fMRI studies indicated that the anterior temporal lobe may be involved in the formation of object representation defined by topological constraints.

What is a perceptual object? This question has become one of the most central and also controversial issues in many areas of the cognitive sciences (1–13). After decades of research, we still do not yet know exactly what counts as an object in the first place. In natural language and in everyday life, objects can be almost anything (e.g., from a drop of water, to a bird, to a car). It is a major challenge to define precisely, beyond commonsense labeling, what constitutes an object, or, in other words, how we abstract from every variety of objects the basic constraint on objecthood applied to all objects by definition.

We propose that the core intuitive notion of an object is its holistic identity preserved over shape-changing transformations (14). This identity can be characterized precisely as topological invariance^†, and the extraction of topological properties serves as the starting point for the formation of an object representation (14). Consider a flying bird. The actual shape of the bird always is subject to change because of its motion (which may be nonrigid) or because of changes of illumination. Nonetheless, the phenomenal impression will be that, regardless of the changes in its featural properties (e.g., location, orientation, size, and shape), the bird retains its identity as a single connected entity (an object) over such deformations. Here the invariant of connectivity is precisely one of the topological properties. (Topology sometimes is referred to as “rubber-sheet geometry.” The topological transformation can be imagined as rubber-sheet deformations such as bending, twisting, stretching, and shrinking but disallowing tearing apart or gluing together parts. The main topics of interest in topology are the properties that remain unchanged by such continuous deformations. In this kind of rubber-sheet distortion, the number of holes remains invariant and hence is a topological property.)

A key but counterintuitive prediction of this topological account of perceptual objects is that the topological change of a figure (e.g., the sudden appearance of a hole in a solid figure) should disturb its object continuity. A topological change in holes should lead to the stimulus being perceived as the emergence of a new object. In contrast to the perception of a new object in response to topological change, the object identity may survive various nontopological changes, such as changes of shape and color.

We investigated this prediction, namely that whether representations of an attended object survive topological disruptions as objects move, in a modified multiple-object tracking (MOT) paradigm (Fig. 1). Most studies of MOT have used moving items with constant shapes, with the exception of a few studies (10, 11) that addressed the roles of shapes of items in tracking^‡. Here we specifically manipulated topological changes (introducing or removing holes within moving items) and compared the effects of these topological changes on tracking performance against changes in various form properties, luminous flux, and color, in 16 behavioral experiments and four sets of fMRI experiments.

Fig. 1.

Illustration of the modified MOT paradigm manipulating the topological transition of holes in comparison with nontopological transitions in moving items. The gray shading in an initial screenshot indicates that an item blinked, designating a target, at the beginning of each trial. The gray arrow at the end screenshot indicates an item identified by subjects as a target. (A) Samples of the presentation sequence of screenshots, including initial and the end screenshots, in the S–ring topological-transition experiment (B) Samples of screenshots in the square–rectangle transition experiment in which the extreme shape change in elongated and slender rectangles (Fig. 2L) is particularly illustrated.

Results

All the behavioral and neuroimaging results consistently supported the topological definition of perceptual objects: Although object continuity can survive a broad spectrum of nontopological changes, all the topological changes in holes (Fig. 2 B–H) consistently disturbed object continuity (Fig. 3), showing the emergence of new objects. The topological definition of objects also received support from the fMRI results: The topological transitions in attentive tracking mainly activated the anterior temporal lobe (ATL) (Fig. 4), consistent with previous fMRI findings that the ATL is involved in topological perception (e.g., refs. 16 and 17).

Fig. 2.

(A–O) Schematic depiction of the stimulus pairs for the 16 behavioral experiments. Because of space limitations, the extremely elongated and slender rectangle (15° x 0.75°) in pair L is not shown to scale here but is illustrated in Fig. 1B. The dark and light gray shading indicates alternate transitions of item pairs in tracking.

Fig. 3.

Mean attentive-tracking performances of the 16 behavioral experiments. The results are shown in three grouped categories of baseline, topological changes (including the subcategory of holes differing in size in experiment 9), and nontopological changes. The category of nontopological change is shown in three further subcategories: shape change, color change, and the subjective emergence of new objects. Error bars indicate SEM.

Fig. 4.

(A and B) fMRI activation patterns from the first set of fMRI experiments. (A) Grouped topological transitions compared with the grouped local shape transitions [n = 30, P < 0.025 false-discovery rate (FDR)-corrected, size >10 voxels]. (B) Grouped topological transitions compared with the grouped nontopological (including shape and color) transitions (n = 46, P < 0.05 family-wise error-corrected, size >15 voxels). (C) fMRI activation patterns of the fourth set of fMRI experiments using high-resolution scans for the grouped topological transitions compared with the grouped local shape transitions (n = 36, P < 0.002 FDR-corrected, size >40 voxels).

Experiment 1 measured the tracking accuracy in a baseline condition (Fig. 2A). As in a canonical MOT task (3), eight white disks on a black background moved independently and unpredictably across the screen, keeping their shapes unchanged throughout their animation. At the beginning of each trial, four randomly selected items blinked to indicate their status as targets. Subjects then tracked the four targets and at the end of the trial indicated which four items were the original targets. Consistent with previous studies (11), subjects were able to perform the MOT task with a high accuracy of about 97%.

In experiment 2, one of the conditions of topological changes, a disk initially turned into a ring while in motion (Fig. 2B). Because the ring has a hole, and the disk does not, the transition between the disk and the ring represents a change in topology. In this disk–ring transition, each of the eight items suddenly changed (ring into disk or disk into ring) every 1–3 s, with the restriction that no simultaneous changes were allowed. This transition occurred 8–10 times for each disk in a single trial. Otherwise the motion of the disks followed the same trajectories at the same speed as in the baseline condition. We found that the change of topology in holes indeed impaired tracking performance: The tracking accuracy was significantly lower than in the baseline condition [t(18) = −3.23, P < 0.005] (Fig. 3).

A potential confound in the disk–ring transition may be that the creation and deletion of a hole could affect other features, such as luminous flux and spatial frequency components commonly considered in the study of vision. To address this issue, experiment 3 tested a new condition of topological transition in which the disks were replaced by S-shaped figures (hereafter referred to as S) (Fig. 2C). The S and the ring are topologically different in that the ring has one hole and the S has none. However, they were made to have equal luminous flux, nearly the same spatial frequency components, perimeter length, equal averaged edge crossings, and other local features; the shape of S also was made irregular to eliminate the possible effects of subjective contours (14). In the S–ring condition, the items moved and changed in the same manner as in the disk–ring condition, except that the stimuli now alternated between the S and the ring. It was found that the S–ring change significantly impaired tracking accuracy [t(18) = −6.13, P < 0.001, in comparison with the baseline] (Fig. 3). There was no significant difference in tracking performance between the S–ring and the disk–ring conditions [t(18) = −0.35, P > 0.7] (Fig. 3); in both cases, a topological transition was present.

Considering the symmetric nature of topological changes (that is, both the change of a solid figure to a hollow one and its reverse, the change of a hollow figure to a solid one represent a topological change in holes), experiment 4 was run with the initial ring–S switch instead of the initial S–ring switch, as shown in Fig. 2D. Similar results were obtained. In experiment 4, as in experiment 3, the initial ring–S transition disrupted object continuity [t(18) = −5.08, P < 0.001, in comparison with the baseline], and no significant difference in tracking performance was found between experiments 3 and 4 [t(18) = −1.09, P > 0.2] (Fig. 3).

Although the S–ring change was designed to control for spatial frequency components, it could still be argued that, because the S carries a horizontally oriented bar in the middle, there is more horizontal-edge energy and higher horizontal spatial frequencies in the S than in the ring; a neuron merely preferring horizontal edges or horizontal higher spatial frequencies could distinguish the S from the ring without the need to pay explicit attention to topology. To address this issue, experiment 5 designed a 9-shaped figure (hereafter referred to as ) and a 5-shaped figure (hereafter referred to as ) instead of the ring and the S, respectively (Fig. 2E). The and the share exactly the same horizontal line segments but differ in topology: The has a hole, but the has none. This control for the horizontal bar rendered the use of the horizontal-orientation energy very unlikely. Nevertheless, the result still demonstrated that the topological change significantly disrupted the tracking accuracy [t(23) = −3.77, P < 0.002, in comparison with the baseline], and no significant difference in tracking performance was found between the and the S–ring transitions [t(23) = 0.60, P > 0.5] (Fig. 3), both of which represent the topological change of no-hole to one-hole.

It still may be argued that, even though the stimulus pair of the excluded the possible use of the local features such as the edge energy and the oriented spatial frequency components in forming new objects, the emergence of a new object involved in the rings of the disk–ring and the S–ring may be based on the fact that the ring carries a white region at the middle, which stimulated an on-center cell. Experiment 6 addressed this argument based on the on-center cell: a θ-like form (hereafter referred to as θ) was used instead of the ring (Fig. 2F). The θ was designed by adding a central bar to a ring-like form so that no white part was contained at its middle. The θ and the S also were made to have the same area and were oriented such that their central bars were parallel. However, the θ and the S differ topologically in the presence of holes: The θ contains two holes, and the S contains none. Hence experiment 6, in addition to addressing the counterexplanations, tested the generality of the topological definition of objects from no-hole to one-hole to no-hole to two-holes, another kind of topological change. The S–θ transition, which represented the topological change of no-hole into two-holes and also controlled for the on-center cell detector, the edge energy, the horizontally oriented spatial frequency components, and the luminous flux, still disrupted the attentive tracking [t(18) = −4.00, P < 0.002, in comparison with the baseline] (Fig. 3).

So far, the topological definition of perceptual objects has been tested against the transitions between solid and hollow forms, including no-hole to one-hole and no-hole to two-holes. Experiments 7 and 8 further generalized the topological definition to the topological changes of one-hole to two-holes (Fig. 2G) and one-hole to four-holes (Fig. 2H). These experiments tested whether the transitions of a ring into a two-hole disk and a ring into a four-hole disk also damaged tracking. To control for changes in local features, such as luminous flux, spatial frequency components, and perimeters, the sums of the areas of the two small holes contained in the two-hole disk and of the areas of the four small holes contained in the four-hole disk were made to be equal to the areas of the big holes contained in the rings. Such manipulation also controlled for differences in the orientation-specific energy of spatial frequencies between the transitions of the ring to two-hole disk and the ring to four-hole disk, as shown by power-spectrum analysis [e.g., ref. 18]. The results confirmed that the topological change of one-hole into two-holes and one-hole into four-holes also damaged the tracking performance in comparison with the baseline [t(18) = −4.16, P < 0.002, and t(18) = −5.52, P < 0.001].

Experiment 9 further generalized the topological account to the holes of different sizes. We measured tracking performance in the transitions between a disk and a ring in which holes varied in size from small to large (0.5°, 0.6°, 0.75°, and 1.0°) (Figs. 2B and 3). When visibility was given due consideration^§, all the changes in holes interrupted tracking accuracy in comparison with the baseline [t(18) = −3.96, P < 0.002; t(18) = −4.68, P < 0.001; t(18) = −3.23, P < 0.005; and t(18) = −4.30, P < 0.001, respectively]. Particularly, as the data show (Fig. 3), there was no correlation between the size of the holes and tracking accuracy (P > 0.7), indicating that the holes were perceived as abstract (mathematical) topological entities in the formation of new objects, independent of nontopological geometric or physical properties, including their concrete size.

A further challenge to the topological definition remains: Is the effect of this topological change unique compared with highly salient but nontopological changes? A broad range of nontopological changes were contrasted with the topological change. Experiment 10 contrasted the disk–square transition in shape (Fig. 2I), in which eight items alternated between disks and squares but otherwise moved and changed as in experiments 2–9. The disk and the square differ in shape and other local features but are topologically equivalent. Unlike the topological transition of disk–ring, the shape transition of disk–square did not interfere with tracking performance [t(18) = −0.84, P > 0.4, in comparison with the baseline] (Fig. 3).

Even though the S–ring pair controlled well for various local featural properties, the topological account still may be challenged by one further counterexplanation: This disturbance of tracking performance might be caused not by the topological change in holes per se but by the fact that the shape transformations of S–ring and disk–ring are more extensive than the shape transformation of disk–square. Experiment 11 tested tracking accuracy in the S–disk transition (Fig. 2J). The S and the disk differ extensively in shape but are topologically equivalent. The result showed that the S–disk transition did not damage tracking performance [t(18) = −1.25, P > 0.2, in comparison with the baseline]. A similar result was also obtained in experiment 12, in which an initial disk–S switch (Fig. 2K) was used instead of the initial S–disk switch [t(18) = −1.17, P > 0.2, in comparison with the baseline] (Fig. 3). In both cases, the object representation survived such extensive but nontopological transformation.

Experiment 13 pushed the idea of massive shape transformation to an extreme by assessing the effects of a transition from a small square into an elongated and slender rectangle (Fig. 1B and Fig. 2L). In this dramatic transition of square–rectangle, the hollow elongated and slender rectangles and the hollow small squares remain topologically equivalent, as illustrated in Fig. 1B. We found again that such extensive shape transformations did not damage the tracking performance [t(18) = −0.94, P > 0.3, in comparison with the baseline]. The result is counterintuitive but strongly supports the validity of the topological definition of objects: Without the change in topology, even very extensive shape transformations do not disrupt object identity.

It still may be argued that, even if extensive shape transformations are not sufficient to lead to the perception of a new object, the subjective emergence of new objects may do so [e.g., when a disk changes its shape into two smaller disks linked by a bar (Fig. 2M)]. Despite the subjective impression that the two smaller disks may tend to appear as two new objects, the topological definition of objects holds that this is not the case, as long as the two smaller disks are connected, even by a thin bar (Fig. 2M). The topological interpretation therefore is that this new shape will be perceived as the same object as the disk, without disturbing object continuity. Consistent with this interpretation are the findings in experiment 14 that the transition of disk–dumbbell did not impair tracking performance [t(18) = −0.97, P > 0.3, in comparison with the baseline].

There could be one more concern that the impaired performance might be caused simply by difficulty in tracking hollow figures (i.e., rings) rather than by the topological change in holes per se. Experiment 15 ran the baseline condition again but used eight rings instead of eight disks (Fig. 2N). The result confirmed that it was indeed the topological change in holes per se that disturbed the object continuity [t(18) = −0.45, P > 0.6, in comparison with the baseline].

So far, the studies have focused on the shape or geometrical dimension. Experiment 16 introduced a further contrast, using the color dimension, a highly salient color transition of white–red (Fig. 2O), in which the eight disks kept the same shape but alternated between white and red, otherwise moving and changing with the same dynamics as in the topological transitions. Unlike the topological change, the highly salient color change did not disturb attentive tracking [t(18) = −0.64, P > 0.5, in comparison with the baseline] (Fig. 3).

In the present study, it was stipulated as a critical control in any condition that no overlap or occlusion of any items was allowed during their motion. The reason for choosing this control is that from a 2D topological analysis^†, a variation in overlap or occlusion represents a change in a topological property, namely connectivity. This absence of overlap and occlusion did, in fact, allow tracking performance to reach an accuracy of about 97% in the baseline condition, replicating the accuracy (about 96%) reported in the previous studies that also disallowed overlapping and occlusion of tracked items (11). When overlapping and occlusions are allowed, the visual system could perceive the transition of two disconnected items into the two overlapping/occluded items as the emergence of a new object, and this perception damaged object continuity even when organizational factors, such as common fate, made the two overlapping items two distinguishable items in depth. Such results support the topological account that proximity (2D connectivity) is superior to organizational factors such as common fate^¶.

This consistent and large set of behavioral data, summarized in Fig. 3, led us to investigate the neural substrate for the representation of perceptual objects defined by topology. All of the tasks and stimuli were similar to those used in the behavioral experiments.

In one condition of the first set of fMRI experiments, the activation in response to the disk–ring transition was contrasted with the activation in response to the disk–square transition. The result revealed a major activation in the ATL (Talairach coordinates: −54, −8, −10; 52, −2, −24) (Fig. S1A and Table S1)], consistent with previous findings that the ATL is involved in topological perception (16, 17).

Two more conditions (S–ring vs. S–disk and S–ring vs. color change) were conducted to test the generality of activation in the ATL caused by the topological change. The areas of common activation again were found in the ATL in each individual condition [Talairach coordinates: −56, −6, −14; −56, 10, −14 (Fig. S1B) and Talairach coordinates: −54, −18, −10, respectively (Fig. S1C)]. The common and more robust activation in the ATL was shown further when topological changes were grouped as one category in contrast with the grouped baselines of local shape changes [Talairach coordinates: −56, −6, −12 (Fig. 4A)] and with the grouped baselines of all nontopological (including color and shape) changes [Talairach coordinates: −52, −12, −12; 54, −12, 8 (Fig. 4B)].

Two control fMRI experiments further tested whether this activation in the ATL was specific to the topological transition. One experiment compared activation to the topologically equivalent shape transitions of disk–S vs. disk–square. Another compared attentive tracking without shape changes in moving items vs. passive viewing (no tracking). The result of attentive tracking vs. passive viewing mostly replicated the activation pattern reported before in ref. 22, as shown in Fig. S1D. Nevertheless, no significant fMRI activation in the ATL was found in these two experiments (Fig. S1 D and E, and Tables S2 and S3).

Further high-resolution scans focusing on the temporal lobe, where the common activation described above was found, confirmed the results: Activation in the ATL was found for both the conditions of disk–ring vs. disk–square and S–ring vs. S–disk [Talairach coordinates −44, 12, −24 and Talairach coordinates −46, −2, −26, respectively (Fig. S1 F and G, and Table S4)]. Again, contrasting the grouped topological category with the grouped baselines of local shape changes showed common and robust activation in the ATL [Talairach coordinates: −48, 4, −22 (Fig. 4C)].

Discussion

All the behavioral and neuroimaging results consistently supported the topological constraint on perceptual objects in the following aspects. The topological account of perceptual objects was generalized to different kinds of topological transitions, including the transition from no-hole to one-hole (Fig. 2 B, C, and E), no-hole to two-holes (Fig. 2F), one-hole to no-hole (Fig. 2D), one-hole to two-holes (Fig. 2G), and one-hole to four-holes (Fig. 2H), all of which consistently disturbed object continuity (Fig. 3). The holes differ in shape [e.g., circular (Fig. 2 B–D, G, and H), half-circular (Fig. 2F), square (Fig. 2L), rectangular (Fig. 2 E and L), and extremely elongated and slim holes (Fig. 2L)] and size. Together, all these results indicate the general and abstract nature of holes in the formation of new objects, independent of detailed geometric or physical properties. In contrast, however, object continuity survived a broad spectrum of nontopological changes (Fig. 3), including the massive shape deformations involved in various local feature changes (Fig. 2 I–L), salient color change (Fig. 2O), and the subjective emergence of new objects (Fig. 2M). The topological definition of objects also received support from the fMRI results. That topological transitions mainly activate the ATL (Fig. 4) is consistent with previous fMRI findings in studies of apparent motion (16) and hemispheric asymmetry (17) that topological perception mainly activates the ATL.

Most current models of vision follow a proposed processing hierarchy in which local features are detected first, before an attention-demanding integration process takes place to build objects. In contrast to this ‘local-first’ proposal, the topological approach emphasizes a “global-first” approach to perceptual organization (e.g., refs 14. and 23), claiming that global topological invariants are extracted at the very beginning of visual processing to form basic constraints on object coding. Together, all the present behavioral and fMRI experiments consistently support the view that the topological definition of objects provides a coherent account and is able to make specific predictions about object identity during shape-changing motion. The extraction of topological properties then serves as the starting point for the formation of an object representation.

Materials and Methods

Behavioral Experiments.

Subjects.

In total, 185 subjects (age 18–27 y) participated in the 16 behavioral experiments. In each experiment, 10 subjects participated, except for experiment 5, which had 15 subjects, and experiment 9, which had 30. All the subjects had normal or corrected-to-normal vision and were right-handed.

Stimuli and procedures.

All the stimulus pairs in Fig. 2, except pair O were white-on-black drawn forms [International Commission on Illumination (CIE) chromaticity coordinates and luminance value: x = 0.320, y = 0.320; 10.2 cd/m²] on a 19-in computer screen (subtending 36° × 27° at the viewing distance of 57 cm). In pair O (the color transition of white–red), the white disks changed color to red (CIE chromaticity coordinates and luminance value: x = 0.624, y = 0.340; 6.05 cd/m²).

At the start of each trial, all four randomly selected items blinked for 2 s to indicate their status as targets. In the remaining 20 s, all eight items moved linearly in an independent and random manner with the restriction that no overlap or occlusion was allowed for the items during their motion. Each item changed suddenly every 1–3 s with the constraint that no two items could change simultaneously. Such transitions occurred 8–10 times for each item in a single trial. The velocities of the moving items ranged from ≈4–7°/s. Subjects were instructed to track the four targets attentively, starting at the beginning of each trial, and pointed out these targets at the end of each trial. No feedback was given. For each condition, each subject completed 20 testing trials after three practice trials.

No special instructions were given concerning fixation, because different fixation conditions have been found not to affect performance on MOT tasks (24).

All statistical tests for the behavioral study used two-tailed independent t test.

fMRI Experiments.

The stimuli in fMRI were similar to those used in the behavioral study, except that a white cross (0.4° × 0.4°) at the center of each display was used as a fixation point. Subjects were instructed to maintain their fixation during scan to eliminate possible confounding influence of eye movement on fMRI activation pattern. Structural and functional images were acquired with a 3-T scanner (TRIO; Siemens). A standard single-channel head coil or a 12-element Head Matrix coil and a gradient echo planar imaging sequence were used to acquire functional images. More details are given in SI Text.