Crowding - an essential bottleneck for object recognition: a mini-review

Abstract

Crowding, generally defined as the deleterious influence of nearby contours on visual discrimination, is ubiquitous in spatial vision. Crowding impairs the ability to recognize objects in clutter. It has been extensively studied over the last 80 years or so, and much of the renewed interest is the hope that studying crowding may lead to a better understanding of the processes involved in object recognition. Crowding also has important clinical implications for patients with macular degeneration, amblyopia and dyslexia.

There is no shortage of theories for crowding–from low-level receptive field models to high-level attention. The current picture is that crowding represents an essential bottleneck for object perception, impairing object perception in peripheral, amblyopic and possibly developing vision. Crowding is neither masking nor surround suppression. We can localize crowding to the cortex, perhaps as early as V1; however, there is a growing consensus for a two-stage model of crowding in which the first stage involves the detection of simple features (perhaps in V1), and a second stage is required for the integration or interpretation of the features as an object beyond V1. There is evidence for top-down effects in crowding, but the role of attention in this process remains unclear. The strong effect of learning in shrinking the spatial extent of crowding places strong constraints on possible models for crowding and for object recognition.

The goal of this review is to try to provide a broad, balanced and succinct review that organizes and summarizes the diverse and scattered studies of crowding, and also helps to explain it to the non-specialist. A full understanding of crowding may allow us to understand this bottleneck to object recognition and the rules that govern the integration of features into objects.

INTRODUCTION

“It is as if there is a pressure on both sides of the word that tends to compress it. Then the stronger, i.e. the more salient or dominant letters, are preserved and they ‘squash’ the weaker, i.e. the less salient letters, between them.” This is Korte’s original, and often referred to, description of crowding (Korte, 1923), and the reader can experience this phenomenon by viewing Fig. 1. Note that the term "crowding" ("Gedränge") has no counterpart in German perception research (including reading) and was first used by the Scandinavian Ophthalmologist, Ehlers in 1936 (Strasburger, personal communication).

Fig. 1. Crowding.

The reader can experience crowding by fixating the dot, and trying to identify one letter: in isolation (a), surrounded by 4 random flanking letters (b), surrounded by 2 horizontally placed random flanking letters (c), surrounded by 2 vertically placed random flanking letters (d).

Crowding, generally defined as the deleterious influence of nearby contours on visual discrimination, is a form of inhibitory interaction which is ubiquitous in spatial vision. It has been extensively studied in several waves since Korte’s original observations. In the 1960’s Flom, Heath & Takahashi (1963) discovered that crowding occurs when target and flankers were presented to different eyes, suggesting a cortical locus. In the 1970’s Bouma (1970; 1973) discovered that the extent of crowding is a more or less constant fraction of the target eccentricity and that crowding occurs in a variety of tasks (Andriessen & Bouma, 1976). In the 1990’s He, Cavanagh & Intrilligator (1996), demonstrated that orientation-specific adaptation is not affected by crowding, implying that the influence of crowding on spatial resolution may take place beyond the primary visual cortex, and there has been a recent explosion of interest in crowding, as evidenced by the publication of more than two dozen articles in a special issue on crowding in the Journal of Vision in 2007 (http://journalofvision.org/7/2).

Crowding impairs the ability to recognize objects in clutter, so much of the recent interest in crowding is in the hope that studying crowding may lead to a better understanding of the processes involved in object recognition. Crowding also has important clinical implications for patients with macular degeneration, amblyopia and dyslexia.

There is no shortage of ideas about the explanation for crowding–from low-level receptive field models to high-level attention. There are few comprehensive reviews of crowding (e.g., Flom, 1991), and none recent¹. The goal of this review is to try to provide a broad, balanced and succinct review that organizes and summarizes the diverse and scattered studies of crowding, but also helps to explain it to the non-specialist. The outline of the review is as follows:

1. What is (and is not) crowding?

2. Some measurement considerations and definitions

3. The “true facts” about crowding

4. Distinguishing crowding from other phenomena.

5. Crowding–an essential bottleneck for object recognition.

6. Theories of crowding

7. Where is the locus of crowding?

8. Consequences of crowding - crowding may limit reading.

9. Development of crowding

10. Can perceptual learning alleviate crowding?

11. Coda

i) What is (and is not) crowding?

In peripheral vision, a letter that is easily recognized on its own becomes unrecognizable if surrounded by other letters (Fig 1). In foveal vision crowding typically only occurs over very small distances (4–6 arc min., e.g. Flom et al, 1963; Toet & Levi, 1992; Liu & Arditi, 2000) or is reported not to occur at all (Strasburger, Harvey & Rentschler, 1991). In contrast, crowding in peripheral vision occurs over very large distances (up to 0.5 times the eccentricity of the target - Bouma, 1970; Toet & Levi, 1992; Kooi, Toet, Tripathy & Levi, 1994) where the retinal point spread functions of the target and flanks are clearly separate. Extensive crowding also occurs in the central visual field of strabismic amblyopes.

• Crowding is nearly ubiquitous in spatial vision

Crowding effects have been reported to occur in a wide variety of tasks including: letter recognition (Bouma, 1970; Flom, Weymouth & Kahnemann, 1963; Toet & Levi, 1992); Vernier acuity (Westheimer & Hauske, 1975; Levi, Klein & Aitsebaomo, 1985); orientation discrimination (Andriessen & Bouma, 1976; Westheimer, Shimamura & McKee, 1976); stereoacuity (Butler & Westheimer, 1978) and face recognition (Martelli, Majaj & Pelli, 2005; Louie, Bressler & Whitney, 2007). Crowding occurs for chromatic stimuli with equiluminant backgrounds, with similar extents to crowding in the luminance domain (Tripathy & Cavanagh, 2002). Thus, crowding is not simply a property of “luminance channels”. Additionally, crowding also occurs for moving stimuli (Bex, Dakin & Simmers, 2003; Bex & Dakin, 2005). However, there are notable exceptions. For example, there is little or no effect of crowding on simple detection of a target (Andriessen & Bouma, 1976; Levi, Klein & Hariharan, 2002a; Pelli et al., 2004; Livne and Sagi, 2007). Moreover, there is a question about whether genuine crowding actually occurs in the fovea (Strasburger et al., 1991; Levi et al., 2002a; addressed in the “True Facts” section); thus I shall use quotes when discussing foveal “crowding”.

Pelli et al. (2004) argue that the effects of flankers on the Landolt C and Vernier acuity tasks may not represent genuine crowding, and may rather reflect masking due to stimulation of the same feature detector by the target and flankers; however both tasks fulfill their criteria for crowding (see Sections iii and iv).

Crowding: What it’s not

Inspection of Fig. 1 makes it obvious that crowding does not result in reduced apparent contrast - rather, crowded letters are high contrast but indistinct or jumbled together. Tyler & Likova (2007) also note their strong subjective impression of a “gray, or inchoate, smudge between the two outer letters, including the inner parts of those letters.” (see their Fig. 2). Although introspection may give one a sense of what crowding is, individuals differ in their reports, and from here on we stick to consideration of quantitative measurements.

Fig 2.

The 2-dimensional shape of “crowding” in foveal (the small dot in the center) and peripheral vision (at 2.5. 5 and 10 degrees). From Toet & Levi, 1992.

ii) Some measurement considerations and definitions

In general, the goal of crowding experiments is to quantify the “critical spacing” and to compare it under different conditions/tasks/eyes/etc. The critical spacing is the distance at which flanks degrade performance (to be discussed further below). Bouma (1970) found that “for complete visual isolation of a letter presented at an eccentricity of ϕ ^o, it follows that no other letters should be present (roughly) within 0.5 ϕ ^o distance”.

A number of different strategies have been used to quantify crowding. Many studies quantify the impact of flankers on task performance (e.g., letter recognition) by using a fixed size target, which, in isolation is correctly identified say 90% of the time. In this paradigm, systematically varying the flanker distance and plotting “percent correct” versus flanker distance allows the experimenter to quantify both the strength and extent of crowding (Flom et al., 1963; Bouma, 1970, Chung, 2007 - and many others).

A second approach, used to assess the influence of the flanks on target perception in a number of recent studies, is to measure a “threshold” (e.g., contrast) for identifying the target (Strasburger, 1991; Chung, Levi & Legge, 2001; Levi et al., 2002a, Pelli et al., 2004). This method enables the experimenter to independently vary target and flank size, distance and other parameters. This method is time consuming, but informative for disentangling the effects of various stimulus and flanker parameters and for disentangling variations in the extent (critical spacing) and the strength (e.g. threshold elevation produced by flanks at a given distance) of crowding.

A third approach is to make two parametric measurements of letter identification (e.g. Levi, Song & Pelli, 2007). One measure is flanked acuity (Fig 1b) where the flank spacing is a multiple of the target size (e.g., spacing equals 1.1 × size). By itself, this measurement would confound size and spacing; however, also making a comparable measurement of isolated letter acuity (Fig 1a), i.e. with infinite spacing, allows one to disentangle the two if the flanked acuity is much larger than and thus not limited by unflanked acuity (as it is in peripheral vision and in the central field of strabismic amblyopes). This method has several practical advantages. It saves time by reducing the two-dimensional space of size vs. spacing to just one dimension, spacing = 1.1 × size. Moreover, it allows one to use the same procedure for isolated and flanked letters. It enables measurement of small critical spacings without overlapping the letters. It allows testing with normal text spacing (i.e., the spacing used in normal print is ≈ 1.1 times the letter size) over the whole range, reinforcing its relevance to reading. The cost of co-variation is that it confounds the variables. However, a number of studies (discussed below) find that peripheral and amblyopic letter identification is limited by the center-to-center spacing, not the letter size (Strasburger et al. 1991; Levi et al., 2002a & b; Tripathy & Cavanagh, 2002; Hariharan et al. 2005; Pelli et al., 2004; 2007). As noted above, measuring the size threshold with two different spacings enables one to distinguish the effects of letter size and spacing.

A key goal of many crowding experiments is to quantify the critical spacing. The critical spacing has been estimated in a variety of ways. For example, for the first approach described above, by fitting a cumulative Gaussian function and estimating the target-to-flank distance for a criterion performance (e.g., 75% correct - Toet & Levi, 1992 and many others). For the second approach, the critical spacing is typically estimated by fitting the threshold vs. flank distance data with a curve (e.g., a Gaussian function) in order to quantify the amplitude of the crowding and its spatial extent. When comparing critical spacing across studies one should bear in mind that its value depends on the threshold criterion (see Pelli et al., 2007, Fig. 7). Moreover while some investigators use the center-to-center separation of target and flanks, others specify the distance (separation) between the closest edges. This can often lead to confusion. For example, Levi et al (2002b) reported that the critical spacing was ≈ 0.1 times the target eccentricity (specified as the separation between the closest edges), while Bouma (1970), Toet & Levi (1992) and many others report that it is ≈ 0.5 times the target eccentricity (specified as the center-to-center separation of letter target and letter flanks). However, the apparent difference may be an illusion since the letters are typically constructed to be 5 times the stroke width. Thus the center-to-center separation of letter targets from letter flanks may be separated by as much as 5 times the separation between the closest edges (e.g., in the case where targets and flanks are abutting). In this review I shall refer to the critical spacing as the center-to-center separation of targets and flankers.

Finally, several new methods have provided important insights into crowding: amongst these, the use of band-limited stimuli shows that peripheral crowding is not simply due to a shift in the spatial scale of analysis in early visual cortex (Levi et al., 2002b; Chung et al., 2001); spatial error analysis has shown that target and flankers are frequently confused (Strasburger, 2005) and novel second-order classification image techniques reveal that that crowding reduces the use of valid features, whilst increasing the number of invalid features used by the visual system (Nandy & Tjan, 2007).

iii) The “true facts” about crowding

This section summarizes a number of features or hallmarks of crowding that are widely considered to be “true”. Some, but not all of these have been often replicated, and they represent the “facts” that have to be explained by any viable theory of crowding. The theoretical implications will be discussed in a later section.

• In peripheral vision, crowding is proportional to eccentricity, it is size invariant, and is quite distinct from ordinary masking

In peripheral vision, the spatial extent of crowding depends on eccentricity, and can be as large as ≈ 0.5 times the target eccentricity. Several recent studies have varied both target size and eccentricity, and show that the extent of peripheral crowding is more or less invariant to target size (Levi et al., 2002b; Tripathy & Cavanagh, 2002; Pelli et al., 2004, 2007), i.e., the critical spacing of crowding scales with eccentricity, so if the entire stimulus (including fixation) is scaled, performance is unchanged. Moreover, as discussed in section iv, the threshold versus contrast function for crowding is quite unlike that for ordinary masking (Pelli et al., 2004).

The strength and extent of peripheral crowding are much greater than the strength and extent of masking (Andriessen & Bouma, 1976; Levi et al., 2002b), so that in peripheral vision, the suppressive spatial interactions due to nearby flanks are not likely to be a consequence of simple contrast masking.

Note that except near the limit of resolution (Danilova & Bondarko, 2007) in the normal fovea the extent of “crowding” is proportional to stimulus size and cannot easily be distinguished from ordinary masking (Levi et al., 2002a). In the fovea, the extent of “crowding” for Gaussian or Gabor targets, is proportional to stimulus size (Levi et al., 2002a; Hariharan et al., 2005) over a more than fifty-fold range of target sizes. Over this large range, foveal “crowding” is scale-invariant. This means that the critical spacing is proportional to the signal size, keeping the signal at the same eccentricity (zero), and both the strength and extent of foveal “crowding” can be predicted directly by the strength and extent of masking (Levi et al., 2002a). Threshold elevation for crowding in an orientation discrimination task is closely similar to the threshold elevation for the detection of a Gabor patch among Gabor flankers, which has been ascribed to lateral masking (Polat & Sagi, 1993), so in foveal vision, the suppressive spatial interactions due to nearby flanks are similar in the two tasks and both may simply be a consequence of contrast masking. At the limit, foveal crowding extends over a tiny distance just 4–5 minutes of arc (Flom et al., 1963; Bouma, 1970; Toet & Levi, 1994; Danilova & Bondarko, 2007).

Is the fovea special? This is not yet clear. The fovea may appear to be special because it is subject to optical blur that is well matched to the neural limits of resolution. For example, with letters, crowding and masking may get confused in the fovea because of the effects of blur, so that what looks like crowding is actually partly masking. In the fovea, it has proven almost impossible to make the stimuli (i.e., target and flankers) smaller than the internal blur and still be able to measure performance using letters. The near perfect scaling with target size (Levi et al., 2002a) found in the fovea is not seen in the periphery until the stimulus blur exceeds the (very substantial) internal blur at that eccentricity. It is interesting to note that with Vernier targets (which are small and impervious to blur), crowding is qualitatively similar in foveal and peripheral vision (Levi, Klein & Aitsebaomo, 1985). Perhaps the same would be true with letters if we were able to adapt our methods appropriately to the foveal landscape, for example using adaptive optics and stabilized viewing.

• In peripheral vision, crowding is inhomogeneous

Crowding in peripheral vision is not isotropic. In the lower visual field vertically arranged flanks (Fig. 1d) are more damaging than horizontally arranged flanks (Fig. 1b), while in the left visual field, the opposite is true. Fig. 2 (from Toet & Levi, 1992) illustrates both the radial arrangement of crowding fields, which have been replicated by others (Petrov & Popple, 2007; Pelli et al., 2007), and the dramatic increase in the extent of crowding fields with eccentricity. Toet & Levi found that, on average, crowding extends from about 0.1 times the target eccentricity in the tangential direction to ≈ 0.5 times the target eccentricity in the radial direction. There is also a horizontal vertical asymmetry in crowding. A recent study (Feng, Jiang & He, 2007) shows that in all four quadrants of visual space, crowding is significantly stronger when the target and distractors are horizontally rather than vertically arranged.

At various points in this review, the critical spacing is described as being either 0.5 times the target eccentricity or ≈ 0.1 times the target eccentricity. As noted earlier, in part this reflects the varied criteria used by different investigators (e.g. center-to-center spacing vs edge-to-edge). It may also reflect the anisotropies described above, so it’s important to specify the spatial arrangement of the flankers in the visual field.

The extent of crowding is also reported to be field dependent. He, Cavanagh and Intrilligator (1996) report that flankers have a stronger effect on orientation discrimination (i.e., reduce percent correct responses more), and the “resolution of attention” (the minimum spacing at which observers can select individual items–Intriligator & Cavanagh, 2001) is coarser in the upper visual field than in the lower field. These results are important in relation to the nature and locus of crowding (discussed later).

• In peripheral vision, crowding is asymmetric

Peripheral crowding is asymmetric. In his original crowding paper, Bouma (1970) noted that two flankers (one on each side of the target letter) were much more potent than one, and that crowding was stronger with a single flanker at an eccentric locus greater than the target compared to a single flanker at an eccentric locus nearer to the fovea (at the same angular separation from the target). In a follow-up study (Bouma, 1973) compared performance (percent correct) for recognition of “inner” letters (i.e., letters closer to the fovea) and “outer” letters (i.e., more peripheral letters) as a function of string length. Remarkably, he found that outer (more peripheral) letters have higher recognition scores than inner (less peripheral) letters, whether in the left or right visual field. For outer letters there was little influence of string length, whereas scores for inner letters decreased with string length. This result is counterintuitive, since one might expect that an inner letter would be more visible than an outer one (which is at a larger eccentricity) and thus be a more potent mask. Bouma proposed a form of visual interference “of a masking type, characteristic for eccentric vision and predominantly acting towards the fovea”. This inner/outer asymmetry has been replicated a number of times for recognition of letters (Banks, Bacharach & Larson, 1977; Banks, Larsen & Prinzmetal, 1979: Bex, Dakin, & Simmers, 2003; Chastain, 1982; Krumhansl, 1977) and for the identification of Gabor patch orientation (Petrov, Popple & McKee, 2007). Indeed, Petrov et al., (2007) suggest that the inward–outward anisotropy can be used as a litmus test for crowding. Motter and Simoni (2007) provide a very simple explanation for this asymmetry in terms of cortical geometry: “although the angular separations for near and far flankers are the same in visual space, the far flanker is actually closer to the target than the near flanker after mapping to cortical space”.

• Crowding depends strongly on target/flanker similarity

While many of the classical studies of crowding (e.g. Bouma, 1970) used targets and flankers that were similar in their properties (polarity, contrast, size), for understanding the mechanisms of crowding, it is important to ask to what extent crowding is tuned. A number of studies have addressed the question “to what extent do target and flankers need to be similar for crowding to occur?” This issue has been widely studied given it’s importance to understanding the mechanisms involved in crowding and the relationship between crowding and other well studied phenomena, such as pop-out and masking.

Crowding is selective along a number of dimensions, with crowding being stronger and more extensive when target and flankers are similar. These dimensions include shape and size (Nazir, 1992; Kooi et al., 1994), orientation (Andriesen & Bouma, 1976; Levi et al., 2002b; Hariharan et al., 2005) spatial frequency (Chung et al., 2001), depth (Kooi et al. 1994) color (Bouma, 1969; Kooi et al., 1994 - in most but not all observers) and motion (i.e., motioned defined targets do not interact with luminance defined flanks in a Vernier task - Banton & Levi, 1993). Tuning along many of these dimensions would be expected based on low-level considerations and also on the basis of grouping (see section vi). But the story is made a bit more complex by the details. For example, for discriminating the orientation of a near threshold T (Kooi et al., 1994) or Landolt C (Hess, Dakin & Kapoor, 2000; Hess, Dakin, Kapoor & Tewfik, 2000) crowding is both stronger and more extensive, when target and flankers are the same contrast polarity (e.g. both black or both white) than when they are opposite polarity (e.g. target black, flankers white). This strong polarity tuning has been an important piece of evidence used to support both low-level and high-level explanations for crowding (see Section vi). However, when the stimuli are larger than the resolution limit the polarity tuning evaporates, and crowding is similar in strength and magnitude when target and flankers are the same polarity or the opposite polarity (Hess, Williams & Chaudry, 2001; Ehrt, Hess, Williams & Sher, 2003; Hariharan et al., 2005).

The polarity selectivity also depends on the temporal properties of the stimulus. Polarity tuning is strong at low temporal frequencies but is absent beyond 6-8 Hz (Chakravarthy & Cavanagh, 2007). Interestingly, Chakravarthy & Cavanagh found no polarity effect at any temporal frequency for lateral masking of target detection.

Contrast tuning (Kooi et al. 1994; Chung et al., 2001; Pelli et al., 2004; Livne & Sagi, 2007) is interesting because it represents a departure from the simple grouping by contrast hypothesis, which would predict that identical target and flank contrasts would result in the strongest crowding, and because the effect of contrast is quite distinct from that for ordinary masking. The strength of crowding depends monotonically on the target:flank contrast ratio (Chung et al., 2001). Importantly, at any target-to-flank spacing, the threshold and saturation contrasts of the flanks to affect the signal are the same (Pelli et al., 2004). This threshold versus contrast (TvC) function is entirely different from the usual TvC function for masking. Based on the spacing independence of the threshold and saturation contrasts, Pelli et al. conclude, that “the fixed threshold and saturation contrasts of the mask are determined not at the variously distant sensor that detects the signal, but, instead, at a sensor local to the mask. In other words, the effects of signal and mask are mediated by separate feature detections.”

Equally interesting are the cases where there is an absence of selectivity. One case where tuning fails relates to the properties or order that define the target and flanks. In almost all of the studies described so far, the stimuli (both targets and flankers) are defined by changes in luminance. That is, they are luminance defined (LD) or first-order stimuli. However, there are many other ways to define a visual stimulus: via contrast, texture, color or motion. These are second-order stimuli. As noted above, while there is no interaction between motion defined targets and luminance defined flankers for a Vernier task (Banton & Levi, 1993), there is substantial cross-over crowding between static targets and flanks that are of different order types. Specifically, Chung, Li and Levi (2007) reported substantial interaction between first- and second- order targets and flanks suggesting that the processing of these stimuli is not independent at the stage of processing at which crowding occurs.

• Crowding occurs under dichoptic conditions, even at the blindspot

Another case where tuning fails is eye-of-origin. A number of studies, using very different stimuli and tasks have shown convincingly that crowding is indifferent to whether the target and flanks are presented to the same eye or to different eyes (target to one eye, flanks to the other–Flom, Heath & Takahashi, 1963; Westheimer & Hauske, 1975; Levi et al., 1985; Kooi et al., 1994). The fact that crowding occurs when target and flanks are presented to separate eyes immediately places the site of the interaction at or beyond the site of combination of information of the two eyes. Remarkably, this dichoptic interaction even occurs when the flankers are presented around the blindspot of one eye, and the target in the “monocular” region corresponding to the blindspot of the other eye (Tripathy & Levi, 1994). This is both surprising and interesting because there is a complete absence of direct retinal afferents from one eye to this region of cortex.

• Crowding in the amblyopic fovea extends over long distances, and in strabismic amblyopia, is distinct from ordinary masking

Extensive crowding occurs in the central field of strabismic amblyopes. As first reported by Irvine (1945), “single letters or direction of … an E could be identified by the amblyopic eye, if viewed one letter at a time, but when placed in conjunction with other letters in a line, confusion affected the interpretation.” While the extent of crowding in the central field of amblyopic observers was originally thought to be proportional to the amblyope’s uncrowded acuity, and similar to normal observers when the resolution deficit was taken into account (Flom et al, 1963; see also Flom, 1991), the story seems to be a bit more complicated. Using Landolt C’s, similar to Flom, Hess & Jacobs (1979) reported that some strabismic amblyopes demonstrated much more extensive crowding than predicted by their acuity loss (Hess, Dakin, Tewfik & Brown, 2001; Levi et al., 2002c; Hariharan, 2005). In the central field of strabismic amblyopes, the strength and extent of crowding are much greater than the strength and extent of masking (Levi et al., 2002c), so we concluded that in the central field of strabismic amblyopic vision, the suppressive spatial interactions due to nearby flanks are not a consequence of simple contrast masking.

A recent large-scale study suggests that crowding is qualitatively different in strabismic and anisometropic amblyopes (Bonneh et al., 2004; also Song, Pelli & Levi, In preparation). In anisometropic amblyopes crowded acuity is proportional to the amblyope’s uncrowded acuity, whereas in strabismic amblyopes it is worse, often very much so.

iv) Distinguishing crowding from other phenomena

There are a number of distinct but seemingly related phenomena that occur when a target is presented in the presence of other features: these include masking, lateral interaction and surround suppression. While all share the characteristic of making a target more difficult to see, each of these phenomena are distinct and most likely reflect different neural processes. The difficulty is how to decide whether a particular set of observations reflects genuine crowding or one of the other phenomena such as masking. Pelli et al. (2004) suggest a diagnostic test for crowding, based on the criterion that “the critical spacing for crowding scales with eccentricity, independent of signal size, whereas for ordinary masking it scales with signal size, independent of eccentricity.”

Below we briefly discuss overlap masking, lateral masking and surround suppression and compare and contrast them to crowding.

Overlap or Ordinary Masking

Detection of a feature or target (e.g. a patch of grating) is rendered more difficult (i.e., threshold is elevated) when the target is overlaid on a mask (e.g., a suprathreshold grating). This is overlap or ordinary masking (Pelli et al., 2004). Overlap masking is tuned to spatial frequency and orientation and is strongest when the target and mask are similar, e.g. in spatial frequency and orientation (Graham, 1989). Overlap masking elevates thresholds for detection and identification similarly (Thomas, 1985) by directly stimulating the channel or feature detector that is most sensitive to detecting the signal. In overlap masking, thresholds increase as a power function of the mask contrast, with an exponent of ≈ 0.4–0.7 (Legge & Foley, 1980; Stromeyer & Klein, 1974).

Lateral Masking

Target and flanker need not overlap in order to interact. For example, flanking (Gabor) patches can impair detection of a (Gabor) target when they are very close (overlapping or abutting), and facilitate detection at wider spacing (Polat & Sagi, 1993). Polat & Sagi explain these effects on the basis that channels are excited by other channels within a distance of two wavelengths from their location and are inhibited by channels from larger distances. In this view, the excitation by nearby channels elevates thresholds and the inhibition from remote channels leads to facilitation. Both claims are controversial, and others have ascribed the facilitation to uncertainty effects (Levi et al., 2002a; Petrov, Verghese & McKee, 2006) and/or pedestal effects that occur in ordinary masking (Solomon & Morgan, 2000).

Surround Suppression

Physiological experiments in cortex reveal two distinct forms of neural suppression–overlay suppression (DeAngelis, Robson, Ohzawa & Freeman, 1992; Carandini, Heeger & Movshon, 1997) similar to the psychophysical overlay masking described above, and surround suppression, in which the mask is presented outside the classical receptive field of the “target” neuron (DeAngelis, Freeman & Ohzawa, 1994; Cavanaugh, Bair & Movshon, 2002). This suppression is divisive (DeAngelis et al., 1994) and probably reflects divisive contrast normalization (Heeger, 1992; Foley, 1994). Recent work suggests that in human observers, contrast thresholds may also be affected by two distinct forms of suppression. Using a double-masking paradigm, Petrov, Carandini & McKee (2005) showed that in humans, overlay suppression precedes surround suppression in the processing sequence, and unlike overlay suppression, surround suppression is only strong in the periphery ( >1° eccentricity). Although surround suppression shares a number of important attributes with crowding, a close comparison suggests that they are distinctly different (Petrov, Popple & McKee, 2007). For example, unlike crowding, surround suppression is not tuned to interocular disparity, and does not show the inward–outward anisotropy evident in crowding. Moreover, while crowding shows a strong dependence on flanker polarity (Kooi et al., 1994) surround suppression does not depend on the phase of the surround grating.

Can we now identify studies/facts that are unambiguously crowding NOT masking/surround suppression etc? i.e., Is there a “litmus test” that unambiguously dissociates between crowding and masking/surround suppression? Two such tests have been suggested: eccentricity dependence coupled with size independence, which is quite unlike ordinary masking (Levi et al., 2002b; Pelli et al., 2004); and the inward–outward anisotropy since it does not occur in surround suppression (Petrov et al., 2007). The very marked radial/tangential anisotropy (Toet & Levi, 1992) may be a third.

v) Crowding–an essential bottleneck for object recognition

Perhaps the great resurgence of interest in crowding comes from the recognition that crowding represents an essential bottleneck for recognizing objects in peripheral vision. As such, crowding seems to offer the promise of a window into the mechanisms underlying object recognition. Section vi describes a number of distinct theories of crowding, but the emerging consensus appears to be a two-stage model in which the first stage involves the detection of simple features (perhaps in V1), and the second stage is the integration (or interpretation) of the features as an object downstream from V1. In crowding, the target and flank features are detected independently and, when both fall within the “integration field”, they are merged into a percept that is often described as jumbled or indistinct. In this view, the bottleneck is a consequence of limited resources. As noted by Pelli et al (2004), the visual system probably has integration fields that span a large range of sizes (at any eccentricity); however, in peripheral vision, we lack small integration fields, so we use fields which may be inappropriately large. “The large ones are cheap, because it takes only a few to tile the visual field” and we can always fixate directly with the fovea (which has small fields) to scrutinize an object of interest. A similar explanation can be applied to an attentional account.

The concept of an “integration field”, as discussed above is equivalent to a second-stage receptive field that combines the features detected by the earlier stage into an object. However, the integration field need not be a fixed and static receptive field. May and Hess (2007) have argued that the integration field for crowding may actually be the “association field” proposed to mediate contour integration (Field, Hayes & Hess, 1993). The association field integrates information across neighboring first-stage filters tuned to similar orientations and is thought to be involved in texture segmentation.

The specific computations and combining rules that result in crowding are not yet well understood. However, it seems likely that they include segmentation (Wilkinson, Wilson & Ellemberg, 1997), contour integration (Field, Hayes & Hess, 1993; May & Hess, 2007), compulsory pooling (Parkes et al., 2001), feature binding (Treisman & Schmidt, 1982; Neri & Levi, 2006) and selective attention (He et al., 1996). Some of the specific ideas about the computations and mechanisms of segmentation, integration and feature binding, as they relate to crowding, are discussed below.

It remains to be tested whether a two-stage model such as this will be able to quantitatively account for all of the mysteries surrounding crowding, and provide insights into how to relieve the “pressure on both sides of the word that tends to compress it” in peripheral and amblyopic vision. The hope is that studying crowding may allow us to understand the bottleneck to object recognition and the rules that govern the integration of features into objects.

vi) Theories of crowding

There are a large number of extant ideas/theories for crowding that range from optical to low-level receptive fields to high-level attention. Several of these ideas are outlined below.

Optical Proposals

• The “Physics” of the stimulus

At the “front end”, is the suggestion that crowding is a consequence of the “physics” of the stimulus (Liu & Arditi, 2000; Hess, Dakin & Kapoor, 2000). For example, the foveal “crowding” effect has, at least in part, been ascribed to the effect of the eye’s point spread function when the letters are small and closely spaced (Liu & Arditi, 2000; Liu, 2001). It has also been argued that in foveal vision nearby flanks displace the “critical spatial frequency band” used to detect the orientation of the gap (horizontal vs. vertical) in a Landolt C, to higher spatial frequencies, thereby reducing the visibility of the cue (Hess et al., 2000).

The optical explanation would predict that crowding only occurs for small targets near the limit of visual acuity and does not occur for large blurred stimuli. In a recent study, Chung & Tjan (2007) measured spatial frequency tuning for filtered letters over a range of sizes and flank separations. Consistent with Hess et al., they found a shift in peak spatial frequency for all letter sizes, but only at the smallest letter separation. However, the shift was tiny - the peak tuning frequency for flanked letters is only 0.17 octaves higher than for unflanked letters at the fovea, and 0.19 octaves at 5 degrees eccentricity. This finding suggests that the human visual system shifts its sensitivity toward a higher (object) spatial-frequency channel when identifying letters in the presence of nearby letters. However, the size of the shift is insufficient to account for the large effect of crowding in the periphery (see also Danilova & Bondarko, 2007).

Neuronal Proposals

• Large Receptive Fields (spatial scale-shift)

One class of neuronal explanations is that crowding occurs when the target and flank overlap within the same neural unit (e.g. both fall within a single receptive field). This proposal predicts that crowding would occur over a range of target sizes, rather than just at the acuity limit, and that the flanking distance would be proportional to the target size. In their classical study, Flom, Weymouth & Kahneman (1963) estimated the extent of spatial interactions by having observers judge the orientation of a near acuity threshold Landolt C, and varying the distance of surrounding flanks from the C. They suggested that the distance over which spatial interaction occurs is related to the size of the receptive fields that are most sensitive to the target. Since peripheral vision is characterized by reduced visual acuity, larger receptive fields will be engaged (since small receptive fields have low sensitivity), and this “scale shift” will result in proportionally larger crowding distances. This “scale shift” hypothesis makes several predictions. First, it predicts that in peripheral vision, the spatial extent of crowding (with broadband stimuli) will scale with (be proportional to) the uncrowded acuity. Indeed, for Vernier acuity (Levi & Klein, 1985; Levi, Klein & Aitsebaomo, 1985) the spatial extent of crowding appears to scale with the unflanked Vernier acuity in both amblyopic and peripheral vision (note that Flom et al., did not report data for the normal periphery). Secondly, the scale shift hypothesis also predicts that the spatial extent of crowding will depend on the size or spatial frequency of the target.

In the normal fovea, the extent of “crowding” does indeed depend on target size over a 50-fold range of target sizes (Levi, Klein & Hariharan 2002a). Thus, the large extent of peripheral crowding obtained with near acuity threshold targets might simply be a consequence of the large target sizes used in testing peripheral crowding (due to poor peripheral acuity). According to the scale shift hypothesis, when tested with the same physical size (and spatial frequency) targets, foveal and peripheral crowding should be similar. As noted below, this hypothesis is incorrect.

Do large peripheral receptive fields account for crowding?

While the optical quality of the eye changes little with eccentricity within the central 10 degrees or so (e.g., Jennings & Charman, 1978; Losada, Navarro & Santamaria, 1993), there are marked neural changes in both the retina and the visual cortex. In particular, the size and spacing of both retinal and cortical receptive fields increase with eccentricity. We (Levi, Klein & Hariharan, 2002b) argued that both crowding and masking must occur after information from the two eyes converges, since both are orientation specific and because crowding occurs when the target is presented to one eye and flanks to the other (Flom, Heath & Takahashi, 1963b; Kooi et al., 1994). Thus, it is reasonable to ask whether the large extent of peripheral crowding reflects pooling of target and flanks by the large peripheral receptive fields of cortical area V1. Specifically, does the extended peripheral crowding reflect a spatial scale shift to larger receptive fields (lower spatial frequencies) in the periphery? This is effectively the ‘scale-shift’ hypothesis of Flom et. al. (1963a). Most studies of peripheral crowding use broad-band stimuli (e.g. letters) that are close to the acuity limit. Thus, crowding in peripheral vision is typically measured with larger (broad-band) stimuli than in the fovea, and the resulting data may indeed reflect a shift in spatial scale. However even with stimuli composed of narrow-band features, crowding is largely independent of stimulus size in the periphery, depending only on eccentricity, and, peripheral crowding extends over a greater distance even when tested with the same size (and spatial frequency) stimuli as the fovea (Levi et al., 2002b). Thus it is unlikely that the extended peripheral crowding reflects a simple shift in spatial scale of first stage filters toward larger (low spatial frequency) filters at an early stage of processing in V1 (see also Livne & Sagi, 2007).

• Perceptive Hypercolumns

This notion flows directly from the eccentricity dependence (ϕ) of crowding. Specifically, as noted above, in peripheral vision, the extent of crowding for letters is ≈ 0.5ϕ (in the radial direction). For Vernier acuity, crowding extends approximately 0.1ϕ at all eccentricities, about the size of a hypercolumn in primate visual cortex, leading to the suggestion that the extent of crowding corresponds to a fixed spacing on the cortex, and that crowding occurs when competing stimuli fall within the same (or an adjacent) “perceptive hypercolumn” as the target (Levi, Klein & Aitsebaomo, 1985). The perceptive hypercolumn might be considered to be the integration field described above.

This notion is appealing, since it predicts both the eccentricity dependence and the relative target size independence (but see Tripathy & Cavanagh, 2002). Moreover, it does not treat the fovea as a “special case”. A foveal perceptive hypercolumn is ≈ 4 arc minutes, about the size of just recognizable letter in the fovea and the distance over which flanks interfere with foveal Vernier acuity (Westheimer & Hauske, 1975; Levi et al., 1985). However, it leaves open several important questions, perhaps chief amongst these, why does the extent of crowding depend on target-flanker similarity? One possibility, suggested by Pelli & Tillman (2008), is that dissimilar flanks may actually operate over the same extent, but have a substantially weaker effect. This intriguing notion requires further investigation.

• Long-range horizontal connections

Another appealing explanation for crowding is long-range horizontal connections between neurons with like orientations in cortical area V1. These horizontal connections may be both excitatory and inhibitory (e.g., Fitzpatrick, 2000; Gilbert, 1998), and extend up to about 1–2mm in primate area V1 (Rockland & Lund, 1983; Blasdel et al., 1985; Fitzpatrick et al., 1985; Lund et al., 1993; Amir et al., 1993). Based on recent estimates of human cortical magnification, these cortical distances translate to approximately 0.1–0.2ϕ in peripheral vision (Beard, Levi & Klein, 1997). Thus, long-range inhibitory connections have approximately the requisite length to account for the extent of ≈ 0.1–0.2ϕ of peripheral crowding (at least for Vernier acuity). However, the fixed cortical distance of long-range connections predicts interactions over a fixed retinal distance, rather than interactions that are related to target size in the fovea.

• Contrast Masking

The term “crowding” is ordinarily used to describe the fact that adjacent objects (letters or flanks) reduce the discriminability of a target. Several aspects of crowding make it mysterious–for example, in peripheral vision crowding extends over very long distances (e.g. Bouma, 1970; Toet & Levi, 1992; Levi, Klein & Hariharan, 2002a), where the target and flanks do not overlap. The term “masking” is often used to describe the fact that a pattern (referred to as a mask) can reduce the detectability or discriminability of a target. Masking generally occurs for targets and masks that overlap, and the phenomenon and mechanisms of masking are reasonably well understood (e.g., Legge & Foley, 1980; Foley, 1994). Crowding and masking both impair visual discrimination, thus, it is reasonable to ask whether they are two sides of the same coin (i.e., to test the hypothesis that foveal crowding is simply masking by remote flanks (e.g., Chung et al., 2001). Although several recent studies (Pelli et al., 2004; Chung, Levi & Legge, 2001) have addressed this question, it has been difficult to compare the two, since they are typically measured with very different stimuli and tasks. To test this hypothesis Levi et al., 2002a measured and compared crowding in a direction-identification experiment with masking by remote flanks in a detection experiment (similar to the experiments of Polat & Sagi, 1993, 1994) using targets and flanks comprised of identical Gabor features. As noted above, in the fovea, both the strength and extent of foveal “crowding” can be predicted directly by the strength and extent of masking (Levi et al., 2002a). Thus, in foveal vision, except at the resolution limit (Danilova & Bondarko, 2007), the suppressive spatial interactions due to nearby flanks are similar in the two tasks and are simply a consequence of contrast masking. In contrast, the strength and extent of peripheral crowding are much greater than the strength and extent of masking (Levi et al., 2002b), so we concluded that in peripheral vision, the suppressive spatial interactions due to nearby flanks are not simply a consequence of simple contrast masking. The notion that crowding does not suppress feature detection gets strong support from the classification image study of Nandy & Tjan (2007).

Computational Proposals

• Abnormal Integration at a stage beyond feature detection

A recent proposal is that crowding reflects limited resolution at a stage beyond the initial filtering stage (He at al., 1996; Chung et al., 2001; Levi et al., 2002b; Pelli et al., 2004). Most crowding tasks require that the observer not only detect the features, but to also isolate and localize them. Note that in this context, a feature is "a component of the image that is detected independently of the others" (Pelli, Farell & Moore, 2003; Pelli, Burns, Farell & Moore-Page, 2006).

As noted above, in peripheral vision observers can easily detect the features under conditions where crowding is strong. Thus, our speculation is that peripheral crowding occurs because the target and flanks are combined or pooled at a second stage, following the stage of feature extraction. In peripheral vision, this pooling takes place over a long distance. Another way of saying this is that the second stage filter or template for identifying a target in peripheral vision is not well matched to the target (Levi, McGraw & Klein, 2000).

Nandy & Tjan (2007) used a novel classification image technique to study crowding. The noise fields in the classification images contain sufficient information to reveal the second-order correlation structures of subtemplate features, enabling them to infer the shape of the putative features used by the human observer and to compare them to features used by an ideal-observer model. This comparison provides a metric for feature validity. They found a decrease in feature validity (relative to the ideal observer) during crowding, consistent with the prediction of inappropriate feature integration. However, as they note, they also found a decrease in the number of valid features, which is not predicted by spurious feature integration per se. To account for these results, Nandy & Tjan suggest that the process of inappropriate feature integration must also somehow suppress the detection of valid features, as might occur if the process of feature integration is a competitive one like the association field model of Field, Hayes, & Hess (1993). Indeed, May & Hess (2007) propose precisely such a model to explain crowding.

A variation of this proposal is that crowding is “feature conjunction” gone awry. We perceive individual objects as possessing multiple attributes or features. A critical task of the visual system is to bind those features into a single percept; however, feature binding can fail resulting in the experience of illusory conjunctions of physically disjunct features (Treisman & Schmidt, 1982; Wolfe & Cave, 1999). The failure of feature binding is evident in patients with both subcortical (Ward et al., 2002) and cortical (Cohan & Rafal, 1991) damage, including visual neglect (Esterman et al., 2000) and Balint’s syndrome (Friedman-Hill et al., 1995; Robertson et al., 1997). Is crowding a failure of feature binding? In their extensive review, Pelli et al. (2004) suggest that illusory conjunction and crowding may both be symptoms of excessive feature integration. However, crowding and feature binding have never been studied with similar stimuli and methods, so they are difficult to compare. Moreover, there are large set-size effects in feature conjunction experiments, whereas it has been reported that there is no set size effect in crowding (Wilkinson et al., 1997; Pelli et al., 2004). For example, increasing the number of flankers from two (in the radial direction) to four (2 radial and 2 tangential) had have little or no effect (Pelli et al., 2004).

The absence of a set size effect in crowding appears to pose a serious problem for the notion of an integration field, since one might think that increasing the number of flanks within the integration field would increasingly confuse target recognition. However, it’s hardly surprising that adding 2 flankers in the tangential direction (where the critical spacing is small) had little effect beyond that of the 2 flankers in the radial direction (where crowding is strong and the critical spacing is large). Pelli et al. did not test the effect of adding additional flankers (beyond 2) in the radial direction. In contrast, Põder & Wagemans (2007) found a substantial set-size effect in crowding. They had observers identify a briefly presented ‘target’ (one of eight Gabor patches that varied in color, spatial frequency or orientation) flanked by 3, 5 or 7 surrounding patches that were selected from the remaining 7 patches. The authors found a strong effect of the number of flankers, and their error analysis revealed that observers made both feature errors and target mis-localizations. They propose a simple feature integration model that predicts many of the irregularities.

What limits feature-integration? Neri & Levi (2006) measured spatial resolution for discriminating targets that differed from nearby distractors in one feature either color or orientation (feature resolution), or their conjunction (conjunction resolution). An example of our results is shown in Fig 3. We found that in the normal fovea (open circle) when the individual features can be resolved, so can their conjunction (the circle falls on the solid 1:1 line). In the periphery (at an eccentricity of 7 deg–solid circle) resolution of conjunctions is much coarser (≈ 3 times) than the individual feature resolution. Thus when the individual features are easily resolved, their conjunction may be invisible. These results can be quantitatively explained by a model of primary visual cortex (V1) in which orientation and color maps are imperfectly co-registered topographically (thick dotted line). The + in Fig. 3 shows the size of a “perceptive hypercolumn”, corresponding to 0.1 times the target eccentricity or about 1mm of cortical distance (Levi et al., 1985). In periphery, the resolution threshold for individual features is much smaller than 0.1 times the target eccentricity (≈ 0.03 times) whereas the resolution threshold for feature conjunction is slightly larger than 0.1 times the target eccentricity. A similar impairment in resolving conjunctions is found in the fovea of amblyopic eyes (square in Fig. 3).

Fig 3.

The threshold resolution for feature conjunctions vs feature identification for normal fovea (open circle) periphery (solid circle) and the central field of a strabismic/anisometropic amblyope (square). The dotted line is the prediction of a model constrained by properties of V1. The + shows the size of a "perceptive hypercolumn". (Data and model replotted from Neri & Levi, 2006).

Feature-binding is widely believed to involve extrastriate cortex (Robertson 2003). The Neri & Levi results do not argue against the view that the read-out is performed by circuitry in ventral cortex, where neurons display properties that are suited to the task (Mazer and Gallant 2003; Bichot et al. 2005) and the foveal representation is particularly emphasized (Brewer et al. 2005); however, they do show that feature-binding is subject to constraints imposed by early processes in V1. It is not clear to what extent a model of this sort could account for all of the “facts” of crowding; however the constraints that V1 imposes on feature conjunction are likely to be present under conditions of crowding.

• Loss of position information

Peripheral vision is characterized by high degrees of spatial uncertainty (Pelli, 1985). Could increased spatial uncertainty explain the greater strength and extent of peripheral crowding? A number of studies implicate a loss of position information in the explanation for crowding (Krumhansl & Thomas, 1977; Wolford, 1975). For example, Strasburger (2005) found that observers frequently confuse the target with the flankers. Thus, “in a large part of the region where crowding occurs, the recognition of a character, irrespective of where in a string it is, is nearly as good as that of a singular character.” Popple & Levi (2006) found that five or six widely spaced stimuli were seen in the correct order, but only four crowded stimuli. Similar to Strasburger, we found that the proportion of confusions between neighboring positions was higher than predicted by chance performance, and proposed that errors might be attributed directly to a noisy stimulus representation, with both object and positional uncertainty under crowded conditions. Both the peripheral and amblyopic visual systems are known to have a high degree of positional uncertainty.

Nandy & Tjan (2007) observed that crowding did not have any systematic effect on the (already high) intrinsic spatial uncertainty of the periphery; however, their classification image study provides strong support for the notion that “source confusion” is a major source of crowding. Specifically they found that in peripheral vision there was a strong correlation between observers’ errors and “parts of the flanks that resemble target features”. Interestingly no such correlation was observed in the normal fovea.

• Crowding as “texture perception”

In several respects this notion is similar to the proposition that patterns that generate considerable neural activity within a constrained region of the periphery are processed as textures, so that the details of individual patterns are not available for discrimination (e.g., Wilkinson, Wilson & Ellemberg, 1997; Orbach and Wilson, 1999). In their model, the texture processing reflects a second-stage analysis by complex cells.

A closely related notion was suggested by Parkes et al. (2001). They found that in peripheral (but not central) vision, crowding in an orientation discrimination task was distinct from masking. Importantly, they were able to show that in a cluttered display, the orientation signals were pooled rather than being lost through masking. They concluded that crowding reflects compulsory averaging of signals, and is the term we use to define texture perception “when we do not wish it to occur”. Several lines of evidence argue against this compulsory averaging. For example, in the study by Livne & Sagi (2007) the mean orientation of the flankers was unchanged across conditions, yet configuration had a substantial effect on crowding.

A more likely feature combination rule is the “maximum rule” model of signal detection theory. In a cluttered display, each feature (target and flanks) is analyzed independently by local feature detectors that are perturbed by independent noise, but only the detector shouting the loudest gets passed along to the next level. This “maximum” combination rule has been shown to account nicely for the high confidence errors that observers make in judging target tilt in a crowded display (Baldassi, Megna & Burr, 2006).

• Configural grouping

A closely related notion is that the visual system, and particularly the periphery has a predilection to form “single perceptual groups by combining elements and features according to what are essentially Gestalt principles” (Banks et al., 1979; Estes, Allmeyer & Reder, 1976). Thus, Banks et al explain the inner/outer asymmetry on the basis that the outer flank is more potent because the configuration as a whole (target plus flanks considered as a group) is more eccentric when the flank is outside than when it is inside. As evidence, they show that the asymmetry is eliminated with a “grouped” mask, i.e., multi-element flankers that group together separately from the target, reducing the cause of the crowding and thus the asymmetry. While this is potentially a very important clue to understanding crowding, it is difficult to gauge the significance of this finding, since the effect is small. They found that observers’ correct identification responses only increased from 51 percent correct with a single peripheral mask to 60 percent with a grouped peripheral mask (and increased by only about 1% when the masks were central). Similar grouping effects have been reported by others (e.g. Banks & White, 1984; Põder, 2006). A particularly dramatic example can be seen in Fig. 3 of Põder (2006). Põder’s experiments involve detecting a blue horizontal bar amongst red horizontal and vertical bars, so the blue feature “pops out” and this pop out increases as the number of red distractors increase. However, this task only requires feature detection, which is not susceptible to crowding². In contrast, a number of studies have shown that in peripheral vision, crowding is not relieved by adding additional flankers (Wilkinson, Wilson & Ellemberg, 1997; Strasburger et al., 1991; Parkes et al., 2001; Felisberti et al., 2005; Pelli et al., 2004). It is unclear whether the success or failure of grouping in relieving crowding relates to the strength of grouping, or to other factors.

Using a different approach, Livne & Sagi (2007) provide strong evidence for configural effects in peripheral crowding. Specifically they asked observers to judge the orientation of a Gabor patch surrounded by eight flanking Gabor patches arranged in a circle, with the flank orientations in different configurations. They found that crowding was weakest (or absent) when the flank orientations varied smoothly around the circle, creating the impression of a closed contour, and strongest when the smooth variation in flank orientation was interrupted by changing the orientations of four of the eight flanks.

Recent work shows that in peripheral vision there are configural effects in crowding of high-level representations of objects. Specifically, recognition of an upright target face is more strongly impaired when surrounded by a crowd of nearby upright faces, than by a crowd of inverted faces. This high-level configuration effect only occurred in peripheral vision, and was not found for non-face objects (Louie et al., 2007). Both this work, and the study of Livne & Sagi suggest that crowding may operate at a level where configuration has already been extracted.

Interestingly, in foveal vision, adding more flankers in a Vernier task partially relieves “crowding”, so that thresholds are about halved when the number of flankers is increased from 4 to 8 (Malania, Herzog & Westheimer, 2007). When the flankers are part of a coherent texture, they lose their effectiveness in foveal “crowding”.

While grouping the flankers into a coherent texture may or may not relieve crowding in the periphery, when the target and flankers are “ungrouped” from each other by making them dissimilar in a fundamental property such as color, polarity or depth so that the target “pops out”, crowding is greatly reduced (Nazir, 1992; Kooi et al, 1994; Põder, 2007). So it appears that grouping by similarity is crucially important to crowding.

Attentional Proposals

A number of studies have used cueing to manipulate and study the effects of attention on crowding. Almost all have shown either no effect or a small effect of cueing on crowding (Nazir, 1992; Wilkinson et al., 1997; Scolari, Kohnen, Barton & Awh, 2007). Crowding occurs when the observer knows precisely when and where the target will be, so it seems unlikely that crowding can be explained on the basis of a lack of attentional deployment. Below we consider other attentional accounts.

One exception is that cueing appears to relieve the effects of crowding in a change detection task. However, rather than a high-level interpretation, Freeman & Pelli (2007) suggest that change detection is so easy that observers can tolerate feature degradation and letter distortion, giving them immunity from crowding.

• Reduced resolution of attention

When several features are presented together, perception of the spatial details of an individual feature depends on: a) the ability of the visual system to resolve each feature (visual resolution), and b) the ability of mechanisms at a subsequent stage, to isolate each feature. Intriligator & Cavanagh (2001) refer to this as "attentional resolution”, and they showed that in peripheral vision, the limits imposed by visual resolution and attentional resolution are quite different. He, Cavanagh & Intriligator (1996) have argued that peripheral crowding results from limitations set by attentional resolution.

The obvious question maybe a semantic one. Is “attentional resolution” simply another word for “crowded acuity”? Simply labeling it attention doesn’t necessarily make it attention. However, He et al. provided the first direct evidence that crowding occurs beyond V1. Specifically, they showed that orientation specific adaptation occurs under conditions where crowding reduces the observers’ ability to report the adapting grating’s orientation to chance performance. Although this is not direct evidence for an attentional locus, it did seem to provide strong evidence that crowding occurs beyond V1 (where adaptation effects are thought to occur). However, this interpretation has been called into question by the recent work of Blake, Tadin, Sobel, Raissian and Chong (2006). Their results showed that the threshold-elevation aftereffect was reduced substantially during crowding, and that the strong aftereffect reported by He and Cavanagh could be explained by the response saturation which occurred at their very high adapting contrast level. Blake et al suggest that their findings indicate that the neural events that underlie crowding are inaugurated at least in part at an early stage of visual processing, “because the threshold-elevation aftereffect arises, at least in part, from adaptation at the earliest stages of cortical processing.”

There are other lines of evidence that have been used to argue for an attentional explanation for crowding. Some are circumstantial rather than direct. These include the report that crowding, like other attentional effects, is stronger in the upper than in the lower visual field (He et al., 1996) and the fact that the polarity effect in crowding, has a temporal resolution similar to that for attention (Chakravarthy & Cavanagh, 2007).

One potentially very important piece of evidence in support of an attentional explanation is the very recent report that crowding is specific to the attentional selection region and does not occur outside it (Cavanagh & Holcombe, 2007; Cavanagh, personal communication). Their experiment tested whether crowding is specific to the locus of attentional selection (He et al, 1996; Intriligator & Cavanagh, 2001) or occurs preattentively throughout the visual field (Pelli et al, 2004). Using a moving attention technique, they created a display where the spatial and temporal properties of attention to the crowding array were different when attention moves compared to when it remains at a single location. Cavanagh & Holcombe argue that since the array itself is not changed, preattentive crowding would not be affected. Specifically, attention to a single location revealed a flickering test letter flanked and crowded by flickering distractors. However, when attention followed a moving guide, the test letter had no distractors along the target’s radial arm. Cavanagh & Holcombe report that crowding is evident when attention is directed to a fixed location but is greatly diminished when attention moves with the guide. They suggest that crowding is specific to the arrangement of distractors within a moving attentional focus–and not set by the arrangement of distractors in retinotopic coordinates.

As noted earlier, crowding is reduced when target and flanker are dissimilar (e.g. Nazir, 1992; Kooi et al., 1994). In Cavanagh & Holcombe’s experiment, in the fixed location condition, both target and flankers appear to flicker in place. However in the moving attention condition, the target appears to move, while the flankers don’t. A possible alternative explanation is that perhaps the strength of crowding depends on whether the target and flankers are both seen in one fixed coordinate system or in two coordinate systems: that of the moving target and the fixed flanker. Nonetheless, since the retinal stimuli are identical in both cases, this result strongly suggests a top-down role in crowding. At this writing, this work has not yet been published except as an abstract; however, because of its outstanding importance for understanding crowding mechanisms, it deserves further testing.

• Exogenous attention

Crowding is reduced when target and flankers differ in color (Nazir, 1992; Kooi et al., 1994) or when the target and flankers are the same color, but the target appears on a different colored background blob (Põder, 2007). Interestingly, the effect of color persisted even when the target and flank colors varied unpredictably, removing the top-down effect of knowing the target (and flank) colors, but the effect was increased in the course of 3 repeated trials with the same target/flank colors (Põder, 2007). Põder argues that since crowding is similarly reduced both by different colored flankers and by the target appearing on a different colored blob both results can be explained on the basis that “exogenously controlled attention is attracted to the location of a salient color singleton (either a target itself or a colored blob), and facilitates visual processing in that location”. He argues further that the colored blob experiment rules out a non-spatial color based selection (because the target and flanks are the same color); however, it is seems likely that there would be leakage of the background color into the target.

While it is clear from almost all of these studies that crowding represents a failure of feature selection, without a clear definition, it is not yet clear to what extent it involves “attention”.

vii) Where is the locus of crowding?

We can localize crowding to the cortex, perhaps as early as V1; but we can’t be much more specific than that. However several facts help to inform our speculations. First, as noted by Blake et al. (2006) the threshold-elevation aftereffect was reduced substantially during crowding, and this aftereffect arises, at least in part, from adaptation “at the earliest stages of cortical processing.” Second, because crowding occurs when the target is presented to one eye and the flanks to other, crowding must occur at or beyond the site of binocular combination (Flom et al., 1963). Third, a recent fMRI study (Arman, Chung & Tjan, 2006) suggests that the effects of crowding are not evident in the BOLD response in V1, but are evident as early as V2. Fourth, since crowding is similar in extent for chromatic and achromatic stimuli, crowding is likely to take place at a “common” locus where feature binding occurs (Tripathy & Cavanagh, 2002). Fifth, since there is cross-crowding between first-and second-order targets and flankers, crowding is likely to occur at a stage where first- and second-order information are combined (Chung, Li & Levi, 2007).

The first two points above are consistent with crowding occurring in the visual cortex, perhaps as early as V1, as is the computational model of feature conjunction described by Neri & Levi (2006). Thus, while we can be pretty certain that crowding occurs in the cortex, the precise locus is unknown, and what follows is speculative. The last three points suggest the possibility of a locus downstream from V1. This notion gains support when one considers the large extent of crowding, ≈ 0.5ϕ, and the substantial anisotropy (Bouma, 1970; Toet & Levi, 1992). On average, receptive fields in primate V1 are about 0.1ϕ, in V2 approximately 0.2E and in V4, approximately 0.5ϕ (Motter, 2002).

V4 is an appealing locus for several reasons: V4 receptive field size and anisotropy of ≈ 2:1 (Pinon, Gatass & Sousa, 1998) are similar in size and orientation to the marked radial/tangential anisotropy of crowding (Toet & Levi, 1992). Moreover, physiological studies point to cortical area V4 as a plausible site for the combination of signals from different stimulus types (Ferrera, Nealey & Maunsell, 1992; 1994; Logothetis & Charles, 1990), consistent with both the similar extent of crowding for luminance and chromatic stimuli, and with the finding that the crowding occurs at a stage where first- and second-order information are combined (Chung, Li & Levi, 2007). Indeed, estimates of human V4 receptive field size (Smith, Singh, Williams & Greenlee, 2001) are in good agreement with the full extent³ of crowding in peripheral vision (Chung, Li & Levi, 2007). Moreover, neurons in V4 are modulated by attention (e.g. Lee, Williford & Maunsell, 2007). While none of the arguments above by themselves are compelling, together they make a reasonable case for a role for V4 in crowding.

It is important to state that without a real model for crowding, V4 is only one of several higher-level possible cortical sites that have been proposed. For example, Michael & Desmedt (2004) have suggested that the pulvinar may be a plausible site for crowding in humans. They found that patients who suffered stroke in the pulvinar nucleus of the posterior thalamus were immune to the effects of interference in a reaction-time task; however, it’s not clear that their task (pressing a button to indicate whether the target disk was red or blue in the presence of a nearby Kanisza square) is a genuine crowding task, given that the “inteference index” (reaction-time with Kanisza square minus reaction-time without the square) was maximally 5%.

Nandy and Tjan (2007) suggest that crowding may be closely related to the process of contour completion, which is a competitive feature-integration process. Specifically, they suggest that under ambiguous conditions (at the local detector level), a process of bias competition, where disjointed patterns in the receptive field of a single neuron will ‘‘compete’’ for the control of the neuron’s firing rate, may serve as a neural substrate for the competitive feature-integration process. This process is found in cortical areas V4 and higher (Chelazzi, Miller, Duncan, & Desimone, 2001; Desimone & Duncan, 1995; Luck, Chelazzi, Hillyard, & Desimone, 1997; Moran & Desimone, 1985; Reynolds, Chelazzi, & Desimone, 1999). Interestingly, Chelazzi, & Desimone suggest that bias competition may operate via divisive normalization (e.g., Carandini, Heeger, & Movshon, 1997; Legge & Foley, 1980). The idea that crowding might operate via divisive normalization at a second stage (i.e., beyond the stage of feature detection) has been suggested previously (Chung, Levi & Legge, 2001; Tripathy & Cavanagh, 2002).

Paradoxically, in a recent fMRI study Beck & Kastner (2007) report that there is less suppressive interaction in area V4 for four identical stimuli than for four stimuli that differ in color and orientation, suggesting that competition is more likely to occur between than within groups. It is not clear how their configuration relates to crowding, which is much stronger when targets and flankers are identical than when they are dissimilar.

viii) Consequences of crowding - crowding sets a limit on reading speed

What are the consequences of crowding for everyday life? We live in cluttered visual environments, so crowding makes most of the objects in our peripheral visual field unrecognizable. Understanding crowding may help us model object recognition in normal and abnormal vision (Wilson, 1991; Pelli et al., 2004; Neri & Levi, 2006). One crucial life activity is reading. In normal vision, reading with peripheral vision is slow, and simply scaling the letter size and spacing does not undo the reduction of rate with eccentricity (e.g. Chung et al., 1998; Chung, 2002; but see Latham & Whitaker, 1996). Anecdotes and the few extant studies report that reading through the amblyopic eye is impaired (e.g. Bach et al., 1990; Stifter et al., 2005a & b).

Does crowding limit reading? Recent work (Pelli et al., 2007; Levi, Song & Pelli, 2007) suggests that the critical spacing for reading is equal to the critical spacing for crowding (see Pelli & Tillman, 2008 for a review). When text is more closely spaced than critical, reading is slowed. In their model (Pelli et al., 2007), reading rate is proportional to the uncrowded span, i.e., the number of character positions in a line of text that are spaced apart more than the critical spacing. This important idea (that crowding limits reading) is surprising and perhaps controversial, because reading rate has generally been thought to be linked to letter size rather than spacing. To directly test the effect of size vs. spacing on amblyopic reading, Levi et al., (2007) measured reading performance versus letter size with normal and doubled letter spacing (i.e., 1.1 and 2.2 times the letter size) (Fig. 4). In the size graph (top), doubling the spacing displaces the data a factor of 2 to the left, showing that spacing matters. In the spacing graph (bottom), doubling the size has no effect (both data sets lie on the same curve), showing that size does not matter. Note that for this amblyopic observer (and others), when text is more widely spaced than critical, reading is fast. Thus crowding appears to represent a limit for reading in normal and amblyopic central vision (Levi, Song & Pelli, 2007). This is consistent with the conclusions of Legge and his co-workers that the “visual span” (i.e., the number of letters in a line of text that can be recognized reliably without moving the eyes) represents a sensory bottleneck for reading (Legge, Chung & Mansfield, 2001; Yu, Cheung, Legge & Chung, 2007; Legge, Cheung, Yu, Chung, Lee & Owen, 2007), except that Legge’s "visual span" is not specifically and solely attributed to crowding.

Fig. 4.

Reading rate versus size (top) and spacing (bottom) for each eye of an amblyope (strabismic & anisometropic). The small symbols are for normal spacing (1.1 × size, these data also appear in Fig. 4) and the large symbols are for double spacing (2.2 × size). The thick lines are the best fit of the uncrowded-span model (Pelli et al., 2007) to the normal-spacing data. The thin lines in the top graph are copies, shifted left (arrows) by a factor of 2, to predict the double-spacing data if spacing limits reading.

While the critical spacing for crowding and reading are also identical in peripheral vision (suggesting that crowding represents one limit to fast reading in the periphery), reading slows with increasing eccentricity even when crowding is controlled for (by scaling with eccentricity), for reasons that remain mysterious. But that’s another story!

Crowding and dyslexia

Developmental dyslexia (or Specific Reading Disability) is a common, but multi-factorial, complex disorder that affects reading, and several studies have reported abnormal crowding in dyslexia (Atkinson, 1991). For example there are several interesting reports that dyslexics have a less crowding in the periphery (i.e., a wider area in which correct identification occurs in the peripheral field), but more “crowding” in the fovea than normal readers (Geiger & Lettvin, 1987; Lorusso, L., Facoetti, Pesenti, Cattaneo, Molteni & Geiger, G. (2004). For dyslexics, vocal reaction time for target words embedded in words arrays is faster when the inter-letter spacing is increased (Spinelli, DeLuca, Judica & Zoccolotti, 2002). Moreover, letter or number identification is impaired when a target letter is embedded in an array (Enns, Bryson & Roes, 1995; Hawelka & Wimmer, 2005); however, the abnormality may be related to memory rather than crowding, since the deficits occur only when the probe is presented after, but not before the array is presented. In their review, Pelli & Tillman (2008) note that dyslexic children have much smaller uncrowded spans than age-matched controls, but that they read much more slowly than predicted by the uncrowded span, suggesting that something other than crowding must be involved in the dyslexic impairment.

In testing dyslexics with letters and words, there is a chicken and egg problem in that it’s not always clear whether it is the reading disability that results in poor performance or vice versa. A recent study (Shovman & Ahissar, 2006) addressed the question using stimuli that were similar to single words in their visual characteristics but had none of the phonological, morphological and semantic characteristics. Their results show convincingly that crowding (by adding flankers) was essentially identical in dyslexic and controls.

ix) Development of crowding

The amblyopic fovea is immature in many ways (Levi & Carkeet, 1993). The large critical spacing for crowding in the amblyopic fovea may reflect a failure of the amblyopic visual system to follow the normal developmental reduction. Unfortunately, little is known about the developmental time course of crowding; however, the limited data suggest that while resolution acuity is nearly adult-like at age 3 years, “crowding effects may still impair performance on practical acuity tasks more than for the adult” at age 5 (Atkinson & Braddick, 1983; Atkinson, 1991; but see Manny, Fern & Loshin, 1987 for a contrary view).

There is firm data on the development of the normal visual span (Kwon, Legge & Dubbels, 2007), and, at least in adults, the visual span is the uncrowded span, which is inversely related to the critical spacing (Pelli et al., 2007). Thus, as far as reading is concerned, the amblyopic deficit could be just a failure of the foveal critical spacing for crowding to shrink from childhood to normal adult size; however we need to know much more about the development of crowding than we do at present.

x) Can perceptual learning alleviate crowding?

An interesting and important practical question is whether crowding can be alleviated through practice. To date only three studies have tackled this question. Wolford, Marchak & Hughes (1988), showed that 8 days of practicing the identification of letters flanked by H’s had almost no effect on crowding (termed lateral masking by them). More recently Huckauf & Nazir (2007) found that training improved recognition of 3-letter strings, with learning specific to the trained strings and also to the trained eccentricity. Transfer to other conditions was only found after more extensive training and transfer occurred earlier when words were used as the stimuli. Huckauf & Nazir argue that, in part, crowding “results from the absence of higher level representations”, and that these representations can be acquired through learning. While potentially very important, the use of percent correct and the attendant difficulties of ceiling effects (their isolated letters were correctly identified 88% of the time), make it difficult to assess whether the changes were really related to learning to “uncrowd”, or whether they simply relate to learning to recognize letters better in the periphery.

The clearest evidence for learning to uncrowd comes from the recent study by Chung (2007). She trained eight observers to identify crowded letters through repeated measurements (6000 trials–many more trials than Huckauf & Nazir’s “long training”) of identifying the middle letters of trigrams at a letter (center-to-center) separation equivalent to 0.8× the x-height. Importantly, the letters and flankers were randomly drawn from the entire alphabet. Following training letter identification performance improved from 34 to 64%-correct at the trained letter separation. Importantly, improvements also occurred at other untrained separations such that for all observers, the critical spacing (the letter separation that yielded 50% correct identification of the target letters) shrank substantially (on average, by 38%-Fig. 5). For one observer the critical spacing shrank from ≈ 2.8 deg to 1.2 deg, a change of almost 60%. This remarkable reduction in the extent of crowding places a strong constraint on any model for crowding. For example, it is not clear how a simple “fixed wire length” model (e.g. based on fixed long-range horizontal connections) can accommodate this kind of shrinkage. Similarly, it is not clear that a V1 receptive field model could be used to explain these results without additional assumptions.

Fig. 5. Learning to “uncrowd”.

This figure re-plots the critical spacing (extent of crowding in degrees) for each observer before (abscissa) and after (ordinate) extended practice (6000 trials of identifying flanked letters). Re-plotted from Chung, 2007.

xi) Coda

Crowding is an enigma wrapped in a paradox and shrouded in a conundrum. Despite a great deal of new (and old) work, we do not yet have a full understanding of crowding. The current picture is that crowding represents an essential bottleneck for object perception, impairing object perception in peripheral, amblyopic and possibly developing vision.

Crowding is nearly ubiquitous in spatial vision and neither masking nor surround suppression provide an adequate explanation for crowding. We can localize crowding to the cortex, perhaps as early as V1; but we can’t be much more specific than that. It is clear that V1 contains small receptive fields that can analyze individual features; however, one constraint on using these small V1 receptive fields for analyzing objects that involve conjunctions of features may lie in imperfect co-registration of the feature maps (Neri and Levi, 2006). This type of model can quantitatively explain the failure of feature conjunction in peripheral and amblyopic vision. Whether an explanation of this sort can also quantitatively account for the “compression” (Korte, 1923), source confusion (Tjan & Nandy, 2007; Strasburger, 2005), asymmetries (Bouma, 1973; Petov et al., 2007), grouping effects (Banks et al., 1979; Banks & White, 1984) and 2-dimensional shape (Toet & Levi, 1992) of crowding remains to be tested. However, there seems to be a growing consensus for a two-stage model in which the first stage involves the detection of simple features (perhaps in V1), and a second stage is required for the integration or interpretation of the features as an object downstream from V1. Segmentation, selection, feature-binding and contour integration all appear to be implicated in this process. While there is clear evidence for top-down influences, the role of attention in this process remains unclear. The strong effect of learning in shrinking the critical spacing also places strong constraints on models for crowding. Both the details and the physiology of crowding need to be sorted out for a full understanding.

A full understanding of crowding may allow us to understand the bottleneck to object recognition and the rules that govern the integration of features into objects.