Crowding: A cortical constraint on object recognition

Abstract

The external world is mapped retinotopically onto the primary visual cortex (V1). We show here that objects in the world, unless they are very dissimilar, can be recognized only if they are sufficiently separated in visual cortex: specifically, in V1, at least 6 mm apart in the radial direction (increasing eccentricity) or 1 mm apart in the circumferential direction (equal eccentricity). Objects closer together than this critical spacing are perceived as an unidentifiable jumble. This is called “crowding”. It severely limits visual processing, including speed of reading and searching. The conclusion about visual cortex rests on three findings. First, psychophysically, the necessary “critical” spacing, in the visual field, is proportional to (roughly half) the eccentricity of the objects. Second, the critical spacing is independent of the size and kind of object. Third, anatomically, the representation of the visual field on the cortical surface is such that position in V1 (and several other areas) is the logarithm of eccentricity in the visual field. Furthermore, we show that much of this can be accounted for by supposing that each “combining field”, defined by the critical spacing measurements, is implemented by a fixed number of cortical neurons.

Introduction

Although the conclusion of this review is about the brain physiology of object recognition, most of the research that led there was psychophysical study of acuity and reading. The link is new and surprising.

In the peripheral visual field, objects that would each be identifiable on their own become impossible to recognize when presented close together (Fig. 1). For each place (and direction) in the visual field there is a critical spacing (center to center) that must be exceeded for unimpaired recognition. Objects closer together than this critical spacing are perceived as an unidentifiable jumble. This is crowding. It severely limits visual processing, including speed of reading and searching.

Figure 1.

Crowding. While fixating the (red) minus, it is easy to identify the isolated letter on the left, but impossible to identify the middle letter on the right. The flankers, “a” and “e,” spoil recognition of the target. Fixating the (green) plus, it is easy to identify the r on the right. Reducing the eccentricity of the target reduces your critical spacing, which relieves crowding. Reprinted from [19].

Crowding, today, can be appreciated as a key fact about how people visually recognize objects. Object recognition has been studied intensely for a long time as a computational problem to which human visual perception provides a solution [4,5]. However, from its discovery until recently, crowding has been studied entirely outside of the research effort explicitly directed at object recognition. Today’s books on object recognition are including crowding for the first time[6]. For most of its history, crowding has been studied as a phenomenon of acuity and reading.

While it is obvious that vision is a complex function with many important parameters of performance, the variation of vision among people has always been, and is still, characterized predominantly by one number: acuity. Acuity is the threshold size for recognizing an object, typically a letter.

The familiar acuity chartused today, hardly changed from Snellen’s original [7], is designed to measure single-letter acuity. With that purpose, the letters are widely spaced, with a center-to-center spacing of about twice the letter size. This works well in normal observers viewing directly. That is, the acuity of normal observers is unaffected by the presence or absence of the letters surrounding the directly-viewed target letter. However, that two-letter spacing is not wide enough for strabismic amblyopes. Their measured acuity improves substantially if the surrounding letters are removed (by using a piece of paper with a hole in it to isolate the target letter). It turns out that the same is true for normal observers viewing peripherally, which is relevant to measuring acuity of patients with central field loss (e.g. macular degeneration) [8,9].

For Plato [10] and essentially everyone since, until recently, it seemed obvious that single-letter acuity accounted for the sharp decline in reading speed that occurs when a book is held too far away. However, increasing viewing distance, by moving the book farther away, confounds the effects of size and spacing. Repeating the experiment with text specially printed with extra-wide letter spacing reveals that under normal reading conditions the maximum distance of the book for fast reading is determined by letter spacing, not size [9]. If the spacing is wider, we can read quickly from farther.

Around 1900, it was discovered that reading consists of a series of eye fixations (four per second) rather than a continuous glide along the line of text [11]. It was natural to suppose that what the observer could take in from each fixation would be limited by his or her visual span. Visual span is the number of characters in a line of text that an observer can identify without moving his or her eyes. (This is not trivial because the fixations of reading are brief, 250 ms, but the visual span is usually measured with unlimited viewing time.) In fact, this simple model (reading speed proportional to visual span) accounts well for the effect of most visual variables on reading speed [12].

Visual span has long been a popular explanation for speed of reading and searching, but, until recently, only Woodworth [13] and Bouma [1] insisted that the visual span is limited by crowding (letter spacing), not acuity (letter size) [14].

This historical review of the origins of the idea of critical spacing has set the stage for our logical argument, which uses the critical spacing idea but is independent of its history. Three findings together yield a strong conclusion about the architecture of cortical processing in visual object recognition.

1. Critical spacing: Bouma’s proportionality constant

Wishing to understand how “lateral interaction” would limit how many letters a person reading can pick up in each fixation, Bouma measured the critical spacing for letter identification (a target letter between two flanking letters) as a function of eccentricity. He discovered that the critical spacing Δϕ is proportional to eccentricity ϕ,

$$\mathrm{\Delta }\varphi =b\varphi ,$$ (1)

where b is now called Bouma’s constant [1,14]. Bouma originally estimated that b is “roughly” 0.5 [1], and later refined this to about 0.4. He reported this result only for letters and only at one letter size. (More extensive measurements appear in Fig. 2.) However, Eq. 1, which looks so innocent, is a breakthrough. It represents his finding that the critical spacing varies enormously with position in the visual field, rather than being determined by object size.

Figure 2.

Critical spacing. The axes indicate position in the visual field, relative to the fixation point (grey “+” in upper left). In the upper right, also gray, we show a target letter between two symmetrically arranged flankers. The colored contour lines trace out the center-to-center target-to-flanker spacing the observer required to achieve 80% correct identification of the target letter. At each eccentricity, the black, red, and green curves represent different letter sizes. The results show that the critical spacing is proportional to radial eccentricity and independent of letter size. The symmetry of the plots results from the symmetry of the stimulus, which always had symmetrically opposed flankers. Each measured critical spacing is plotted as two opposing points. This is similar to the earlier measurements of Toet and Levi [3], but including several letter sizes. The anisotropy of critical spacing (radial over circumferential) is about 1.5 for this observer; Toet and Levi report an average of 2 for six observers, with large variations among observers. Letter sizes are 0.3, 0.43, and 0.55 deg at 5° and 6° ecc.; 0.43, 0.55, and 0.83 deg at 7.8° ecc.; 0.53, 0.83, and 1.2 deg at 12° ecc.; and 0.83, 1.2, and 1.5 at 13° ecc. Reprinted from [14].

Much of our daily visual experience is very nearly scale-invariant. We readily recognize letters and friends over a nearly thousand-fold range of size (in deg of visual angle). This led practically everyone to suppose that visual object recognition is scale invariant, though, in fact, the processing is scale-dependent [15,16]. Similarly, it has long been known that acuity is worse at more peripheral locations in the visual field, but it has been supposed that position in the visual field has little or no effect on the identifiability of objects that are big enough to exceed the acuity limit.

Bouma’s proportionality (Eq. 1) turns this conventional wisdom on its head. It identifies spacing, not size, as the key parameter, and it is determined by position in the visual field, not size of the objects.

2. Generality: the Bouma law

Interest in crowding has grown and grown to reach its present height [17–19]. It was important to learn that critical spacing (center to center) is independent of letter size [20–23]. Critical spacing has been measured for a wide range of tasks and objects. Eq. 1 holds up well. Extra terms have been added to the equation to fit data at very small and large eccentricities [3,14]. It is not surprising that the value of Bouma’s proportionality constant varies among studies that use different tasks and criteria for critical spacing [19].

Flanking objects that are less similar to the target produce less impairment of identification. However, studies that have disentangled the amplitude of crowding from spatial extent consistently find that similarity of the flankers to the target only affects the amplitude, not the spatial extent, of crowding [19,21,24,25].

Thus it seems that critical spacing (more precisely, the spatial extent of crowding) depends only on the location in the visual field (and direction from target to flanker), independent of object and size. This is demonstrated in Fig. 3.

Figure 3.

Critical spacing is independent of object and size. Fixating on the (red) minus, you will be unable to identify the middle object in each row, unless you isolate it by hiding the flanking objects with your fingers (or two pencils). Grating patches, like those in the top two rows, are often taken to be one-feature objects. In the first row, is the middle grating vertical or tilted? The ± is our estimate of the fixation point where you can just barely identify the target (Eq. 1 with b = 0.4). You can assess the accuracy of this threshold estimate by noting that the task is easy when you fixate to the right of the ±, and hard at the left. Critical spacing depends solely on position (and direction), in the visual field, which does not vary among rows in this demonstration. Note that halving object size has no effect on critical spacing. Critical spacing is independent of spatial frequency [21,61]. Reprinted, with modifications, from [19]. [Original sources: The gratings were created in MATLAB. The letters are in the Courier font. The animal silhouettes are in our Animals font, which is available for research purposes. The men, women, and telephone signs are from aiga.com [62]. The ladder is licensed from and copyright Stockbyte. The rocking chair is copyright 2008 Jupiter Images Corporation. In the following credits, we use the convention (Photographer/Name of collection/Source). The stool (C Squared Studios/Photodisc/Gettylmages), pretzel (Steve Wisbauer/Photodisc/Getty Images), hamburger (Ryan McVay/Photodisc/Getty Images), and pizza (Raimund Koch/Riser/Getty Images) are from Getty Images.]

3. Brain mapping

Independent of the psychophysical work on crowding, a succession of neuroscientists has used various techniques to produce ever-better maps describing the retinotopic correspondence between object position in the visual field and location of the response on the cortical surface in monkey [26–30] and man [2,31–39]. To a good approximation, in V1, and several other areas of human visual cortex, position d on the cortical surface of the response is the logarithm of the object’s eccentricity ϕ in the visual field,

$$d=\frac{log\varphi }{\beta }-\alpha ,$$ (2)

where α and β are empirical constants, unique to each visual area [2]. β is 0.06/mm for V1, 0.05/mm for V2, 0.05/mm for V3, 0.04/mm for LO1, 0.06/mm for LO2, and 0.05/mm for hV4 [2]. (Virtues and limitations of this logarithmic approximation are discussed in [29,40]. It requires modification to fit the smallest and largest eccentricies.)

Conclusion

Because of the logarithmic mapping (Eq. 2), a spacing Δϕ at the visual field that is proportional to eccentricity (Eq. 1) results in a fixed spacing Δd at the cortex,

$$\mathrm{\Delta }d=d_{\varphi +b\varphi }-d_{\varphi }=log(1+b)/\beta .$$ (3)

Taking b = 0.4, as in Fig. 3, and the value of β for V1, we find that Δd is 6 mm. Thus the logarithmic transformation of the proportional critical spacing at the visual field results in a fixed critical spacing at the cortex — 6 mm at V1 —independent of eccentricity [19,41,42]. (This is consistent with the estimated 6 mm range of dichoptic interactions in the region of V1 corresponding to the blindspot [43].) Up to this point, we have supposed that the objects are spaced radially (increasing eccentricity).

Anisotropy

A similar conclusion can be drawn for circumferential spacing of objects (equal eccentricity), but the critical spacing is much smaller — 1 mm at V1 — independent of eccentricity. Psychophysically, circumferential critical spacing is roughly half the radial critical spacing, with much variation within and across individuals [3]. Brain imaging finds anisotropy in the cortical magnification factor. In V1, Larsson and Heeger find itto be 1/3 as large in the circumferential as in the radial direction [2]. (Others report less anisotropy of cortical magnification, especially near the horizontal meridian, in monkey [44,45] and man [39].) At V1, these two factors make the circumferential critical spacing (1/2)x(1/3) = 1/6 of that in the radial direction, that is, 1 mm.

Other areas

The same conclusion applies to each of the logarithmically mapped visual areas, with differentvalues radially and circumferentially. The anisotropy is even more pronounced in higher-tier visual areas [2]. To recognize an object, the brain must combine its features. Perhaps one of these cortical areas combines features over the critical spacing [2,42,46–55].

The great anisotropy of critical spacing at the cortex surprised us, and we have no explanation for it, but it is not paradoxical, as there is no strong reason to expect isotropy. The strongest argument that we can make for invariant critical spacing at the cortex applies to the areal cortical magnification factor (mm² per deg²), the product of the radial and circumferential cortical magnification factors. The areal magnification is less variable than the linear magnification [30]. Let us call the area around the target circumscribed by the critical spacing in every direction (i.e. the ellipses in Fig. 2) the “combining field,” which corresponds to the older terms “association field,” “perceptual hypercolumn,” and “isolation field” [23,41,56]. Suppose that a certain fixed number of cortical neurons is required to do the computation (combining of features) that is associated with each combining field. Further suppose a fixed overlap factor of combining fields (the average number of fields that include any given point). The density of cortical neurons (per mm²) in visual cortex seems to be fixed, stable across individuals and species [57,58], so a fixed number of neurons will occupy a fixed area a. The large variations in brain size (and size of each visual area), within and across species, imply that smaller brains have fewer neurons and provide fewer combining fields, which would have to be larger (larger critical spacing) to provide the same coverage (see Appendix). Thus it would be interesting to measure critical spacing across species that have different brain size. Similarly, within a species, especially man, it would be interesting to look for correlation, across individuals, of critical spacing (measured psychophysically) and areal cortical magnification (measured, e.g., by fMRI).

We have concluded, firstly, that, for object recognition, there is a fixed critical spacing, in mm, for each of the logarithmically mapped visual areas (with different values radially and circumferentially), independent of the object’s kind, size and location. Secondly, we have no explanation for the radial-circumferential anisotropy, but if we embrace it as an empirical fact, then assuming that a fixed number of neurons implement each combining field provides a plausible model for the critical spacing results, and predicts a testable inverse relation between brain size and critical spacing at the visual field.

Symbols

b

Bouma’s proportionality constant for Eq. 1. “Roughly 0.5” [1]

ϕ

eccentricity (in deg) in the visual field [1]

Δϕ

critical spacing (in deg) at the visual field

d

position (in mm) on the cortical surface

Δd

critical spacing (in mm) on the cortical surface

αand β

here correspond to the Larsson & Heeger parameters a and b in Fig. 5 of [2]

η

anisotropy of crowding, the ratio of radial over circumferential critical spacing. About 2 [3]

a

area (mm²) of cortex occupied by the set of neurons that implement one combining field

A

area (in mm²) of cortex available

N

number of combining fields that can be implemented by an area A of cortex

o

overlap factor. The number of combining fields that include any given point in the visual field

Appendix: Brain size and critical spacing

Here we show that supposing that a fixed area of cortex is devoted to each combining field predicts that critical spacing will track changes in brain size. Specifically, changes in area of the cortex will produce inversely proportional changes in b log(1+b), where b is Bouma’s constant, which determines the critical spacing Δϕ = bϕ.

For eccentricity, we take a section of a cortical map (e.g. V1) that is accurately logarithmic, using only eccentricities in the range ϕ_min < ϕ < ϕ_max, with bounds, e.g., ϕ_min = 1° and ϕ_max = 5°. For polar angle, we take the whole semicircular range of one hemisphere’s map, 0 < ψ < 180°. The critical spacing radially, outwards, from target to flanker is Δϕ = bϕ. Circumferentially the critical spacing is b′ϕ, where we define b′ = b/η, and η is the anisotropy (η ≈ 2) of the critical spacing (ratio of radial to circumferential). The ratio of eccentricities of a critically spaced flanker (outward, radially) and target is (1+b)ϕ/ϕ = 1+b. We suppose that that the radial critical spacing inward is in the same proportion, so that the ratio of greatest to smallest eccentricity of the combining field is (1+b)². Circumferentially the critical spacing, expressed as polar angle, is Δψ = (180/π)b′ϕ/ϕ = (180/π)b/η. The polar extent of the field is 2Δψ. If we “tile” the visual field (within the specified bounds) with abutting fields then at any given polar angle the number n of abutting fields radially from ϕ_min to ϕ_max will be

$$n=log({\varphi }_{max}/{\varphi }_{min})/log[{(1+b)}^2]$$ (A1)

At any given eccentricity, number n′ of fields abutting circumferentially will be

$$n^{\prime }=180/(2\mathrm{\Delta }\psi )$$ (A2)

$$=\pi /(2b/\eta ).$$ (A3)

The total number of fields tiling the visual field will be their product, n′n. Allowing for an overlap factor of o, the total number N of fields needed to service the specified visual area is

$$N={on}^{\prime }n.$$ (A4)

We have supposed that the neurons required to implement a combining field occupy an area a (in mm²). Let the section of cortex have area A (in mm²). This provides enough neurons to make N combining fields,

$$N=A/a.$$ (A5)

Putting together our two equations for N, we have

$${on}^{\prime }n=A/a,$$ (A6)

which expands to

$$o[\pi /(2b/\eta )]log({\varphi }_{max}/{\varphi }_{min})/[2log(1+b)]=A/a,$$ (A7)

which we can rearrange to

$$blog(1+b)=A^{-1}ao\pi \eta log({\varphi }_{max}/{\varphi }_{min})/4.$$ (A8)

On the left, we have an expression in b, Bouma’s constant. Note that for small b, log(1+b) is approximately b. Thus the left-hand expression is roughly b². On the right, the cortical area A is the most variable quantity, known to vary two-fold within a species. The area a is our invention, the area occupied by the number of neurons required to implement a combining field. We suppose that this is constant for a given region of the visual cortex within a species, but would depend on the number of dimensions (features) that are being combined [59,60], which differs among regions of the cortex and may differ between homologous regions in different species. The remaining terms—overlap factor o and anisotropy η—seem unlikely to change much. Thus, supposing that a fixed number of cortical neurons implements each combining field predicts that changes in area of the cortex will produce inversely proportional changes in b log(1+b), where b is Bouma’s constant, determining the critical spacing Δϕ = bϕ. QED.