On a common circle: Natural scenes and Gestalt rules

Abstract

To understand how the human visual system analyzes images, it is essential to know the structure of the visual environment. In particular, natural images display consistent statistical properties that distinguish them from random luminance distributions. We have studied the geometric regularities of oriented elements (edges or line segments) present in an ensemble of visual scenes, asking how much information the presence of a segment in a particular location of the visual scene carries about the presence of a second segment at different relative positions and orientations. We observed strong long-range correlations in the distribution of oriented segments that extend over the whole visual field. We further show that a very simple geometric rule, cocircularity, predicts the arrangement of segments in natural scenes, and that different geometrical arrangements show relevant differences in their scaling properties. Our results show similarities to geometric features of previous physiological and psychophysical studies. We discuss the implications of these findings for theories of early vision.

One of the most difficult problems that the visual system has to solve is to group different elements of a scene into individual objects. Despite its computational complexity, this process is normally effortless, spontaneous, and unambiguous (1). The phenomenology of grouping was described by the Gestalt psychologists in a series of rules summarized in the idea of good continuation (2, 3). More quantitative psychophysical measurements have shown the existence of association fields (4) or rules that determine the interaction between neighboring oriented elements in the visual scene (5, 6). Based on these rules and on the Gestalt ideas, pairs of oriented elements that are placed in space in such a way that they extend on a smooth contour joining them will normally be grouped together.

These psychophysical ideas have been steadily gaining solid neurophysiological support. Neurons in primary visual cortex (V1) respond when a bar is presented at a particular location and at a specific orientation (7). In addition, the responses of V1 neurons are modulated by contextual interactions (6, 8–15), such as the joint presence of contour elements within the receptive field and in its surround. This modulation depends on the precise geometrical arrangement of linear elements (6, 16) in a manner corresponding to the specificity of linkage of cortical columns by long-range horizontal connections (17, 18). Thus, neurons in V1 interact with one another in geometrically meaningful ways, and through these interactions, neuronal responses become selective for combinations of stimulus features that can extend far from the receptive field core.

The rules of good continuation, the association field, and the connections in primary visual cortex provide evidence of interaction of pairs of oriented elements at the psychophysical, physiological, and anatomical level. The nature of the interaction is determined by the geometry of the arrangement, including spatial arrangement and the orientation of segments within the visual scene. An important question is whether this geometry is related to natural geometric regularities present in the environment. It is well known that natural images differ from random luminance distributions (19, 20), but the structural studies of natural scenes have not yet addressed the existence of geometrical regularities. We address this issue here by studying whether particular pairs of oriented elements are likely to cooccur in natural scenes as a function of their orientation and relative location in space. Our results are focused on two different aspects of the organization of oriented elements in natural scenes: scaling and geometric relationships. We will show that these two are interdependent.

Scaling measurements involve studying how the probability of finding a cooccurring pair changes as a function of the relative distance. A classic result in the analysis of natural scenes is that the luminance of pairs of pixels is correlated and that this correlation is scale-invariant (19, 20). This indicates that statistical dependencies between pairs of pixels do not depend on whether the observer zooms in on a small window or zooms out to a broad vista. The scale invariance results from stable physical properties such as a common source of illumination and the existence of objects of different sizes and similar reflectance properties (21). We show here that for particular geometries, the probability of finding a pair of segments follows a power law relation and thus is scale-invariant. We show further that a very simple geometric rule, consistent with the idea of good continuation, predicts the arrangement of segments in natural scenes.

Materials and Methods

Images were obtained from a publicly available database (http://hlab.phys.rug.nl/imlib/index.html; ref. 22) of about 4,000 uncompressed black and white pictures, 1,536 × 1,024 pixels in size and 12 bits in depth, with an angular resolution of ≈1 min of arc per pixel. This particular database was chosen because of the high quality of its pictures, especially in their lack of motion and compression artifacts, which would otherwise overwhelm our statistics. To obtain a measure of local orientation, we used the steerable filters of the H₂ and G₂ basis (23). By using steerable filters, the energy value at any orientation can be calculated by extrapolating the responses of a set of basis filters. A G₂ filter is a second derivative of a Gaussian and the H₂ filter is its Hilbert transform. H₂ and G₂ filters have the same amplitude spectra, but they are 90° out of phase; that makes them quadrature pair basis filters. The size of the filters used was 7 × 7 pixels. A measure of oriented energy was obtained by combining both sets of filters E(ϕ) = G₂²(ϕ) + H₂²(ϕ) (23). This measure is repeated at every pixel of the image to obtain the energy function for each image (n) of the ensemble {E_n(x, y, ϕ)}. To study the joint statistics of E(x, y, ϕ), we discretized the different orientations at 16 different values, 0 = (−π/32, π/32), 1 = (π/32, 3π/32), . . . , 15 = (29π/32, 31π/32), as shown in the color representation of orientations of Fig. 1. With this information one can obtain a measure of the statistics of pairs of segments by calculating the correlation (weighting the cooccurrences of segments by their energy).

where N is the total number of images and the integral is over each of the images of the ensemble. We were interested in measuring long-range correlations so we studied values of Δx, Δy = {−256, 256}. The correlation matrix has dimensions 512 × 512 × 16 × 16 and each point results from averaging 4,000 integrals over a 1,536 × 1,024 domain. To simplify the computations, for the general case, we decided to store at each pixel, for every image, the maximum energy value E(ϕ_max) and its corresponding orientation ϕ_max. An energy threshold E_T was arbitrarily set to match the visual perception of edges in a few images. Pixels in an image were considered “oriented” if E(ϕ_max) ≥ E_T, and “nonoriented” otherwise. This unique threshold value was applied to all images in the ensemble. Thus, for each image, we extracted a binary field E_n^bin(x, y) = {0, 1} and an orientation field Ang_n(x, y) = {1, . . . , 16}. From this binary field we can construct a histogram of cooccurrences: how many times an element at position (x, y) was considered oriented with orientation ϕ and at position (x + Δx, y + Δy) a segment was considered oriented with orientation ψ. Thus, formally, the histogram is obtained as C, taking as the Energy function E_n(x, y, ϕ) = 1 if ϕ = Ang_n(x, y) and E_n^bin(x, y) = 1; E_n(x, y) = 0 in any other case. The computation is reduced to counting the cooccurrences in the histogram H(Δx, Δy, ϕ, ψ) with Δx = {−256, 256}, Δy = {−256, 256}, ϕ, ψ = (0, π/16, 2π/16, . . . , π). From the histogram we obtained a measure of statistical dependence. Although choosing the threshold followed computational reasons, cortical neurons perform a thresholding operation and, thus, the measure of linear correlation (weighting cooccurrences by their energy) is not necessarily a more accurate measure of statistical dependence. The histogram was used for all of the data shown in Figs. 2 A–C, 3, 4, and 5. For Fig. 2D, for the particular case of collinear interactions, we computed the full linear cross-correlation. This computation is considerably easier because it is done for fixed values of orientation and direction in space. The two measures shown (Laplacian correlation and collinear correlation) were obtained according to the formulas:

for Laplacian filtering, and

for collinear oriented filtering.

Figure 1.

An example of the filtering process we applied to an image. (a) The original image. (b) The image after processing with local-oriented filters (66). The maximal orientation was calculated at each point. The image was converted to binary by considering char “oriented” only the pixels that, after being filtered at their maximal orientation, exceeded a given threshold. In the figure, the maximal orientation is shown by using a color code.

Figure 2.

Scaling behaviors for different geometrical configurations. (A) The number of cooccurrences between two segments in the relative positions within the line that the orientation of the first segment spans is shown for different orientations of the second segment. This measure was averaged over all possible orientations of the first segment. The collinear configuration is the most typical case and displays a scale invariant behavior as indicated by the linear relationship in the log–log plot. (B) The strength of the correlation and the degree to which it can be approximated to a power law are more pronounced for the particular case in which the reference line segment is vertical. (C) The same measure when the two segments are at a line 90° apart from the orientation of the first segment. In all three cases, black corresponds to iso-orientation, red to 22.5° with respect to the first segment, green to 45°, blue to 67.5°, and yellow to 90°. (D) Full crosscorrelation as a function of distance for Laplacian filtering (red circles), oriented filters in the collinear vertical direction (black circles), and for both cases after shuffling the images. The Laplacian filtered image is decorrelated, as can be seen from the fact that it shows the same structure as its shuffled version (cyan circles). Collinear configuration shows long-range correlations, which follow a power law of exponent 0.6 (blue line, y = x^−0.6) and are not present when the image is shuffled (green circles).

A quantitative signature of scale invariance is given by a function of the form C = r^−a (power law) where C is the correlation, r the distance, and a constant. If the scale is changed r → λr = r′ the function changes as C(r) = λ^−ar^−a = kC(r′) where k is a constant. A power law is easily identified as a linear plot in the log–log graph, which is clear from the relation log(C) = −alog(r).

The axis of maximal correlation (Fig. 5b) was calculated as follows. For each pair of orientations (ϕ, ψ), a measure of cooccurrence was calculated integrating across 16 different lines of angles of values (0, π/128, 2π/128, . . . , π) over distances of [−40, 40] of the center of the histogram. Thus, for an angle θ and orientations (ϕ, ψ) the measure of cooccurrence is: P_{ϕ, ψ}(θ) = Σ_{i = −40}⁴⁰ H(cos(θ ) * i, sin(θ) * i, ϕ, ψ). We then calculated the direction of maximal correlation θ_max(ϕ, ψ) and grouped all angles with common relative orientation ϕ − ψ = ξ. We had 16 different values for each ξ and from these 16 different values we calculated the mean P(θ, ɛ) = < θ_max(ψ, ψ + ɛ) > _ψ and the standard error. To calculate the mean energy as a function of relative orientation (Fig. 3) we integrated the histogram in spatial coordinates for each pair of orientations in space, and, as before, the different pairs where grouped according to their relative difference in orientation to calculate a mean value and a standard deviation, E_{ϕ, ψ} = ∫_{x = −100}¹⁰⁰ ∫_{y = −100}¹⁰⁰ H(x, y, ϕ, ψ)dxdy and E(ϕ) = 〈E_{ɛ, ɛ + ϕ}〉_ɛ. The code was parallelized by using MPI libraries and run over a small Beowulf cluster of Linux workstations.

Figure 5.

Quantitative analysis of the spatial maps. Orientation of the axis where cooccurring pairs of oriented elements of relative orientation (ξ = ψ − ϕ) are maximized. The axis of maximal probability was calculated relative to the orientation of the segment in the center (ψ). This was done for the 16 possible orientations of (and the corresponding values of ϕ = (ψ + ξ)(modulo16), and we computed for each the mean and standard error. The solid line corresponds to the solution predicted by the cocircular rule.

Figure 3.

The number of cooccurring pairs of segments as a function of their relative difference in orientation (ξ = ψ − ϕ). These values were obtained after integrating the histogram of cooccurrences in space for different angular configurations. Each point in the graph (ξ) corresponds to the average and the standard deviation of the 16 different configurations obtained by choosing one of the 16 possible values for the first orientation (ϕ) and then setting ψ = (ϕ + ξ)(modulo16).

In general, horizontal and vertical directions had better statistics because there are more horizontal or vertical segments than oblique in the images; these special orientations are also the most prone to artifacts from aliasing, staircasing, and the ensemble choice. Because we are interested in this study in the correlations as a function of relative distance and orientations, all of the quantitative measurements were performed by averaging overall orientations. However, the results shown still held true for each individual orientation.

Results

All 4,000 images used in this study were black and white, 1,536 × 1,024 pixels in size, and 12 bits in depth. We used a set of filters to obtain a measure of orientation at each pixel of every image of the database (23). The filters were 7 × 7 pixels in size and thus provided a local measure of orientation. The output of the filter was high at pixels where contrast changed abruptly in a particular direction, typically by the presence of line segments or edges, but also corners, junctions, or other singularities (Fig. 1). If the output of the filters were statistically independent, then we would expect a flat correlation as a function of (Δx, Δy, ϕ, ψ). In polar coordinates (r, θ, ϕ, ψ), the two problems that we address are naturally separated: the scaling properties result from studying how the histogram depends on r (distance), whereas the geometry does it from the dependence of the histogram on θ, ϕ, and ψ.

We studied the number of cooccurring pairs of segments as a function of their relative distance for different geometries (Fig. 2 A–C). The different geometric configurations correspond to the different orientations of the segments and their relative position within an image. We first studied the number of cooccurrences as a function of distance in the line spanned by the orientation of the reference segment, averaged across all possible orientations of the reference line (Fig. 2A). When both segments have the same orientation, we observe a scale invariant behavior, indicated by a linear relationship in the log–log plot (see Materials and Methods). Also it can be seen from this plot that collinear cooccurrences are more frequent than any other configuration. Fig. 2B shows the probability of cooccurrences is higher for the vertical orientation, and that scale invariance extends over a broader range.

The scaling properties are qualitatively different for segments positioned side-by-side, along a line orthogonal to the orientation of the first segment (Fig. 2C). Iso-oriented pairs were again the most frequent, but their cooccurrence in the orthogonal direction to the orientation of the first segment (Fig. 2C, black line) does not appear to be scale invariant. This is reflected by the presence of a kink as opposed to a straight line (power law) in the log–log plot, indicating well-defined scales with different behavior.

It is worth comparing the scale of interactions one observes by using different kinds of filters. Before filtering images, the luminance shows correlations, which follows a power law behavior (19, 20). After applying a Laplacian filter (equivalent to a center-surround operator, which measures nonoriented local contrast), the image is mostly decorrelated (Fig. 2D, red circles) (24, 25). This is seen in the exponential decay of the correlations, and in the fact that the correlations show similar behavior after a pixel-by-pixel shuffling of the image (Fig. 2D, cyan circles). The strength and scaling of the correlations across the collinear line changes radically when an oriented filter is used. In this example, to make a direct comparison between the various filters, we weighted each pair of segments by their energy value (linear crosscorrelation, instead of applying a threshold as in the earlier calculations). This calculation was done for the vertical reference line orientation, which showed long-range correlations (Fig. 2B, black circles), over much longer distances than observed with the Laplacian filter. Moreover, these correlations were not present when measured in the shuffled images (Fig. 2, green circles). It is clear from the above analysis that, when oriented filters are used, strong correlations that extend over large distances are revealed. The next question is how these correlations depend on the relative orientation of the line elements, and whether these dependencies have any underlying geometry. We first calculated the total number of cooccurrences as a function of the relative difference in orientation. Cooccurrences decreased as the relative orientation between the pair of segments increased, being maximal when they were iso-oriented and minimal when they were perpendicular (Fig. 3).

The next observation concerns spatial structure. The probability of finding cooccurring pairs of segments was not uniform, but rather displayed a consistent geometric structure. If the two segments were iso-oriented, their most probable spatial arrangement was as part of a common line, the collinear configuration (Fig. 4a). As the relative difference in orientation between the two segments increased, two effects were observed. The main lobe of the histogram (which in the iso-oriented case extends in the collinear direction) rotated and shortened, and a second lobe (where cooccurrences were also maximized) appeared at 90° from the first (Fig. 4 a–e). This effect progressed smoothly until the relative orientation of the two segments was 90°, where the two lobes were arranged in a symmetrical configuration, lying at 45° relative to the reference orientation. Thus, pairs of oriented segments have significant statistical correlations in natural scenes, and both the average probability and spatial layout depend strongly on their relative orientation. Remarkably, the structure of the correlations followed a very simple geometric rule. A natural extension of collinearity to the plane is cocircularity. Whereas two segments of different orientations cannot belong to the same straight line, they may still be tangent to the same circle if they are tilted at identical, but opposite, angles to the line joining them. Given a pair of segments tilted at angles ψ and ϕ, respectively, they should lie along two possible lines, at angles (ϕ + ψ)/2 or (ϕ + ψ + π)/2, in order to be cocircular (Fig. 4f). This is the arrangement we observed in natural scenes. The measured correlations, given any relative orientation of edges, were maximal when arranged along a common circle. To quantify this we calculated the orientation of the axis where cooccurrences were maximal. We did that for different relative orientations and compared it to the value predicted by the cocircularity rule (Fig. 5). This is particularly remarkable in that the comparison is not a fit, because the cocircularity rule has no free parameters.

Figure 4.

Plot of the spatial dependence of the histogram of cooccurring pairs for different geometrical configurations. (a) The probability of finding a pair of iso-oriented segments as a function of their relative position; a pair of segments at relative orientation of 22.5° (b), 45° (c), 67.5° (d), or 90° (e). (f) Cocircularity solution for a particular example of two segments. The solutions to the problem of cocircularity are two orthogonal lines, whose main have values (ψ + ϕ)/2 or (ψ + ϕ + π)/2. For the example given, ϕ (red segment) = 20°, ψ (blue segment) = 40°, and the two solutions (green lines) are 30° and 120° (all angles from the vertical axis).

Discussion

We have shown that there are strong, long-range correlations between local-oriented segments in natural scenes, that their scaling properties change for different geometries, and that their arrangement obeys the cocircularity rule. The filters we used for edge detection in our images were an oriented version of Laplacian-like filters in that they were local but had elongated, rather than circularly symmetric, center-surround structures. This change is analogous to the difference between filters in the lateral geniculate nucleus (LGN) and simple cells in the primary visual cortex. Thus, given that Laplacian filtering decorrelates natural scenes (24), it was surprising to find the long-range correlations and scale-invariant behavior of the collinear configuration. It is important to remark that our measure of correlation does not differ only in the type of filters used (elongated vs. circular symmetric), but also in the fact that we measured the correlations along a line containing the pair of segments. Long contours are part of the output of the Laplacian filters and thus the image should show correlations that might be hidden when integrating them across an area—essentially because a curve has zero area and thus the correlations along a curve are not significant when integrated over the two-dimensional field of view. The findings of long-range correlations of oriented elements extends the notion that the output of linear local-oriented filtering of natural scenes cannot be statistically independent¶ and shows that those correlations might be very significant through global portions of the visual field for particular geometries.

The cocircular rule has been used heuristically to establish a pattern of interactions between filters in computer vision (1, 27–29), and psychophysical studies suggest that the human visual system utilizes a local grouping process (“association field”) with a similar geometric pattern (4). Our finding provides an underlying statistical principle for the establishment of form and for the Gestalt idea of good continuation, which states that there are preferred linkages endowing some contours with the property of perceptual saliency (2). An important portion of the classical Euclidean geometry has been constructed by using the two simplest planar curves, the line and the circle (30); we show here that those are, in the same order, the most significant structures in natural scenes.

We have reported the emergence of robust geometric and scaling properties of natural scenes. This raises a question as to the underlying physical processes that generate these regularities. Although our work was solely based on statistical analysis, we can speculate on the possible constraints imposed by the physical world. In a simplifying view, we can think of a natural image as composed by object boundaries or contours, and textures. Collineal pairs of segments are likely to belong to a common contour; thus, our finding of scale invariance for collineal correlations is in agreement with the idea that scale-invariance in natural images is a consequence of the distribution of apparent sizes of objects (21). Parallel segments, on the contrary, may be part of a common contour as well as a common texture, which would explain the two scaling regimes we observed. Cocircularity in natural scenes probably arises because of the continuity and smoothness of object boundaries; when averaged over objects of vastly different sizes present in any natural scene, the most probable arrangement for two edge segments is to lie on the smoothest curve joining them, a circular arc. These ideas, however, require an investigation that is beyond the scope of this paper.

The geometry of the pattern of interactions in primary visual cortex parallels the interactions of oriented segments in natural scenes. Long-range interactions tend to connect iso-oriented segments (17, 18), and interactions between orthogonal segments, which span a short range in natural scenes, may be mediated by short-range connections spanning singularities in the orientation and topographic maps in the primary visual cortex (31). The finding of a correspondence between the interaction characteristics of neurons in visual cortex and the regularities of natural scenes suggest a possible role for cortical plasticity early in life, in order for the cortex to assimilate and represent these regularities. This plasticity might be mediated by Hebbian-like processes, reinforcing connections on neurons whose activity coincides (i.e., their corresponding stimuli are correlated under natural visual stimulation). Such plasticity could extend to adulthood to accommodate perceptual learning of novel and particular forms (32).

Although we find coincidences between the pattern of interactions in V1 and the distribution of segments in natural scenes, the sign of the interactions plays a crucial role. Reinforcement or facilitation of cooccurring stimuli (positive interaction) results in Hebbian-like coincidence detectors, whereas inhibiting the response results in Barlow-like detectors of “suspicious coincidences” that ignore frequent co-occurrences (33). Interestingly, the Hebbian idea and the decorrelation hypothesis represent two sides of the same coin. From our measurements of the regularities in natural scenes, and previous studies on the higher order receptive field properties in primary visual cortex, it appears that both types of operations exist. The response of a cell in V1 is typically inhibited when a second flanking segment is placed outside of its receptive field along an axis orthogonal to the receptive field orientation. This interaction is referred to as side-inhibition, which is strongest when the flanking segment has the same orientation as the segment inside the receptive field (13, 15, 34). In the present study, we found that iso-orientation is the most probable arrangement for side-by-side segments in natural scenes, which therefore constitutes an example, in the domain of orientation, of decorrelation through inhibition. This inhibition may mediate the process of texture discrimination (13, 16, 35). The property of end-inhibition has also been interpreted as a mechanism to remove redundancies and achieve statistical independence (36). The finding that responses of V1 neurons are sparse when presented with natural stimuli (37) and models of normalization of neuronal responses in V1 tuned to the statistics of natural scenes^¶ also supports the idea that the interactions in V1 play an important role in decorrelating the output from V1. This is consistent with the general idea that one of the important functions of early visual processing is to remove redundant information (38–40), and suggests that interactions in V1 may continue with the process of decorrelation that is achieved by Laplacian (24) and local-oriented filtering (41, 42).

But the visual cortex also can act in the opposite way, reinforcing the response to the most probable configurations. This is seen in the collinear configuration, which is the one that elicits most facilitation, and therefore illustrates how V1 can enhance the regularities in natural scenes. The fact that those correlations are significant over the entire visual field and are highly structured suggests that this is not a residual, or second-order, process. The opposing processes of enhancement of correlations and decorrelation may be mediated by different receptive field properties that can exist within the same cell. The same flank can inhibit or facilitate depending on the contrast (26, 43), suggesting that V1 may be solving different computational problems at different contrast ranges or a different noise-to-signal relationship. The dialectic behavior of visual cortex shows that the interplay between decorrelation (extraction of suspicious coincidences) and enhancement of a particular set of regularities (identification of form) may be mediated by the same population of neurons. Although the decorrelating process may be required to operate in the orientation domain to solve the problem of texture segmentation, particular sets of coincidences, which are repeated in the statistics, such as the conjunction of segments that form contours, need to be enhanced in the process of identification of form.

Abbreviation

V1

primary visual cortex