Multistability, cross-modal binding and the additivity of conjoined grouping principles

Abstract

We present a sceptical view of multimodal multistability—drawing most of our examples from the relation between audition and vision. We begin by summarizing some of the principal ways in which audio-visual binding takes place. We review the evidence that unambiguous stimulation in one modality may affect the perception of a multistable stimulus in another modality. Cross-modal influences of one multistable stimulus on the multistability of another are different: they have occurred only in speech perception. We then argue that the strongest relation between perceptual organization in vision and perceptual organization in audition is likely to be by way of analogous Gestalt laws. We conclude with some general observations about multimodality.

1. Introduction

This paper is an attempt to refocus the question of the relation between multistable stimuli in vision and audition. The question before us is whether multistability in one affects multistability in the other. We believe that such effects are uncommon; hence, we focus on parallels between the behaviour of multistable stimuli in vision and multistable stimuli in audition. We will show that in both domains the interaction of sources of multistability (grouping principles) is subject to similar laws of additivity.

The question of cross-modal influences in the perception of multistable stimuli requires prior answers to two other questions:

• — Let us assume that stimuli in the two modalities undergo binding if and only if they seem to come from a single source or event in the environment. If the assumption holds, is the binding of the stimuli in the two modalities a prerequisite for such a cross-modal influence? We believe that the answer to this question is Yes.

• — Can one design a visual stimulus that is multistable to the eye and an auditory stimulus that is multistable to the ear such that they can undergo binding? We believe that the answer to this question is (with the exception of speech) No.

Because of these answers, we are sceptical regarding cross-modal influences on the perception of multistable stimuli. Nevertheless, as we will show later, there are fascinating and important cross-modal parallels between multistable stimuli.

We begin our paper by listing four conditions for the binding of auditory and visual stimuli. Then we compare and contrast emergence and binding. Then we argue that only emergent properties can be multistable. We continue by asking under which conditions two multistable stimuli can affect each other. We then answer in the negative the question of whether there are stimuli that are multistable in each modality and lend themselves to binding according to these determinants. Next, we describe parallel results we have obtained in vision and in audition regarding the additivity of organizational cues when we apply them conjointly to multistable stimuli within each modality. We conclude that the exploration of the scope and limits of cross-modal analogies in the perception of multistable stimuli is a fruitful direction for future research.

2. Binding

For cross-modal influence to occur, one or more of four conditions must hold: synchrony, common embodiment, common source or causality.

(a). Binding by synchrony

Two notable examples of binding by synchrony are the McGurk effect [1] and the ventriloquism effect [2,3]. Both involve speech. In the McGurk effect, a voice repeats the same syllable, such as /b/, whereas visible lip movements correspond to other syllables, such as /g/. The nature of the lip movements has a strong effect on our perception of the speech stimulus. In the ventriloquism effect, a sound comes from one location in space, whereas the lip movement comes from another location. The sound seems to emanate from the location of the mouth. Both of these effects diminish (within limits—see van Wassenhove et al. [4] regarding the McGurk effect) with asynchrony between sound and lip movements [5–11].

(b). Binding by common embodiment

A beautiful illustration of cross-modal influences is the Köhler ‘takete/maluma’ demonstration [12,13] (in the earlier version it was ‘takete/baluba’). He would show a person the two drawings in figure 1 and ask, Which is takete and which is maluma? People are near unanimous that the rounded form goes with maluma, the angular one with takete. The effect holds across different linguistic groups [14] and extends to preliterate toddlers [15].

Figure 1.

Köhler's demonstration of binding by common embodiment. (a) The maluma figure. (b) The takete figure.

The phenomenon is probably due to analogies between the gestures required to generate the drawing and produce the sound. The angular drawing requires abrupt reversals of direction; analogously, the sound requires the tongue to sharply strike the palate. In contrast, the rounded drawing requires a single flowing gesture; analogously, the corresponding sound requires a gentle touching of the lips.

Experimental evidence of this phenomenon is now available. Parise & Spence [16] varied the stimulus onset asynchrony (SOA) between the sound and the image, and asked the subject to say whether the sound preceded the image or not. The resulting psychometric functions showed that when the pairs were consistent (a rounded shape paired with a more smooth-sounding pure tone or an angular shape paired with a rough-sounding square wave), participants were more tolerant of SOA, i.e. they were less likely to say that inconsistent pairs were synchronous. Thus, they were able to show greater binding of consistent auditory and visual stimuli than inconsistent ones.

(c). Binding by common source

The relation between vision and touch offers an example of binding by common source. There is a correlation between the likelihood that visual and haptic signals come from different objects and the spatial separation between them. Gepshtein et al. [17] asked how this separation affects the binding. Their stimuli were two variable-slant virtual planes presented multimodally, such that the two planes were parallel within and between modalities, but the apparent distance between the planes differed between modalities. In the visual modality, they generated the planes by a random-element stereogram so that they appeared to be transparent; in the haptic modality, they generated the planes by a force-feedback device connected to thimbles into which the participants inserted their index finger and thumb. The observers' task was to view these surfaces with both eyes and/or ‘grasp’ them with the index finger (from above) and thumb (from below) and estimate the distance between them. They first determined for each observer the conditions under which the increase in discrimination precision with two modalities relative to performance with one modality is maximal. They manipulated the distance between the signals, and found that discrimination precision was optimal when the signals came from the same location.

(d). Binding by causality

Suppose that a marimba player strikes a key and follows it with a long or a short gesture. The perceived duration of the sound is longer if the gesture is long than if is short [18,19]. Schutz & Kubovy [18] demonstrated that this effect was due to cross-modal causality, by first showing that the effect depended on the percussive element of the sound. For example, when the sound associated with the long and short gestures (marimba or piano), the effect was present. But when they substituted a non-percussive sound—clarinet, violin or voice—for the percussive sound, the effect disappeared. Thus, they established the first condition of causality: appropriateness. A second condition is temporal order: a cause must precede its effect. They showed that even if they played the sound 400 or 700 ms after the visible impact, the effect was present. But if the sound preceded the impact by 400 or 700 ms, then the effect vanished.

In another experiment, Armontrout et al. [20] replaced the video of the percussionist with an animation in which only a white disk on a black background was visible: the disk followed the path of the mallet head, and no impact surface was visible. Despite this extreme simplification, the results held. Thus, the effect does not depend on learned context, such as a person causing the motion.

However, when they reversed the direction of the trajectory of the disk after impact—so that it seemed to go below the impact surface—the effect vanished, even though only the direction changed while velocities and accelerations remained identical. This suggests that this causality effect is hard-wired.

3. Multistability

(a). Related and unrelated ambiguities

On our way to determining whether multistable stimuli in different modalities can affect each other, we inquire about the conditions under which multistable stimuli affect each other within a modality. To that end, we distinguish between coherent and unrelated ambiguities. Coherent ambiguities share an emergent property.

Consider the possibility that the reversals of one Necker cube might affect the reversals of another. Adams & Haire [21] studied the relative reversal rates of a small Necker cube nested in a larger one under two conditions: in one, the orientations of the two cubes were the same (figure 2a); in the other, they were opposite (figure 2b).

Figure 2.

Examples of coherent and unrelated ambiguities: the Adams & Haire [21] nested cubes. (a) Same orientations. (b) Opposite orientations.

In both conditions, none of the participants reported a higher reversal rate for the larger cube. When their orientations were the same, their reversal rates were similar: the nested cube reversed faster for only 5 per cent of the participants. But when the orientations of the small cube and the large cube were opposite, there was no relation between their reversal rates: the nested cube reversed faster for 87.5 per cent of the participants.

The drawing of the cubes in the same orientation has coherent ambiguities. At a given moment, each produces one of two emergent properties—cube seen from right and above or cube seen from left and below. When both fit into the same spatial framework, they share this emergent property. This allows them to be seen as a single object: a solid cube with a cubic chamber inside. There is no question of independent multistability of these two cubes because they have no separate perceptual standing. In contrast, when the two cubes are in opposite orientations, one cannot make them coherent—at a given moment the emergent property produced by one is different from the emergent property produced by the other. We call these ambiguities unrelated.

A more striking example of unrelated ambiguities (figure 3a) comes from Bradley & Petry [22]. The figure is multistable in two ways.

Figure 3.

An example of unrelated ambiguities: Ambiguity 1: is the Necker cube interpreted as a cube seen from a vantage point above to its right or from a vantage point below to its left? Ambiguity 2: is the paper cutout of a Necker cube seen floating in front of eight black disks painted on a white background (illustrated in figure 3b) or is it seen through eight holes in a white surface, against a black backdrop? (illustrated in figure 3d and rendered in figure 3c) (a) The Bradley & Petry [22] cube. (b) Default interpretation: modal completion. (c) The modified Bradley & Petry [22] cube. (d) Alternative interpretation: amodal completion).

First, it shows the usual multistability of the Necker cube—cube seen from right and above or cube seen from left and below.

Second, it is multistable with respect to depth, or layering. In both interpretations, we see the Necker cube as a white paper cutout. But in one, we see it floating above eight black disks painted on white paper (figure 3b), whereas in the other, we see it through eight holes in a sheet of paper against a black backdrop seen through the holes behind the cutout (figure 3c,d). In the first case, we see the edges of the cube uninterrupted; this is the phenomenon of modal completion discovered by Schumann [23] and made important by Kanizsa [24,25]. In the second case, the completion is amodal, we see that the page occludes parts of the cutout, but we see the cutout as being complete.

Because unrelated ambiguities are logically and perceptually independent, we conjecture that unrelated ambiguities never affect each other.

These notions of coherence and independence of ambiguous patterns are still intuitive, but they offer a principled way to think about the influence of one multistable pattern on another.

(b). Cross-modal effects of one multistable stimulus on the multistability of another

We know of only one example of cross-modal coherent ambiguities. The phenomenon in question—the verbal transformation effect [26]—occurs both in audible speech (AVTE) and in visible speech (VVTE). In the AVTE, as we listen to a repeatedly played syllable, such as a /pse/, after a number of repetitions we hear it as a /sep/ and vice versa. In the VVTE, discovered by Sato et al. [27, experiment 1], as we look at images of moving lips that we see as speaking a /pse/, we eventually see it as saying /sep/. Sato et al. [27] found that reversals in the VVTE occurred in synchrony with reversals in the AVTE.

Why are these ambiguous stimuli coherent? The reason is that they both can produce the same emergent phonological entities: a /pse/ and a /sep/.

We are doubtful that other cases exist because of fundamental differences between perceptual organization in the two modalities. To illustrate, we compare two patterns: one visual, the other auditory (figure 4).

Figure 4.

Two examples of grouping. (a) Visual. (b) Auditory.

The visual pattern (figure 4a) forms a tight grouping of three items, and a loose grouping of two. We cyclically play the auditory pattern, represented in figure 4b, called an auditory necklace (to which we will return later). If we extended figure 4a indefinitely to the right and left, and played figure 4b repeatedly, either diagram would represent either stimulus.

In perceiving both patterns, groupings of three elements emerge. Grouping by proximity (spatial or temporal, as the case may be) emerges in both modalities; in that respect, they are analogous. But the visual pattern produces a further set of emergent properties based on mirror symmetry (a topic explored by Strother & Kubovy [28]), which are not present when we listen to the auditory stimulus because that requires audition to reverse time. In contrast, the auditory pattern produces its own emergent property, which has no analogue in the perceptual organization of the visual stimulus: an accent on the initial note of the pattern (the first note of the run of three), its ‘clasp’.

What do the stimuli in the two modalities have in common that could allow them to influence each other? What would make these ambiguous patterns coherent, and not independent? Either space or time.

(i). Space

Space cannot be the common medium for multistability in the two modalities, thanks to the theory of indispensable attributes [29–32]. This theory specifies the aspects of stimulation required for perceptual numerosity in each modality. This is critical for understanding perceptual organization. Without multiple perceptible entities, there is nothing to organize, no Gestalts, no multistability.

Imagine the following thought-experiment: we ask an observer who is looking at two coloured spots of light (figure 5a), How many entities are visible? We assume that the observer answers, Two. (If the observer gives any other answer, then the experiment is invalid.) If we collapse the display with respect to wavelength (figure 5b), then the observer will still say, Two. Hence, wavelength is not an indispensable attribute for visual numerosity. But if we collapse over space (figure 5c), then the observer will respond (because of colour metamerism), One. Hence, spatial location is an indispensable attribute for visual numerosity.

Figure 5.

Theory of indispensable attributes: the visual thought-experiment. (a) Start. (b) Collapse over wavelength: not indispensable. (c) Collapse over space: indispensable.

The auditory thought-experiment runs as follows: we ask an observer who hears two sounds played over two loudspeakers (figure 6a), How many entities are audible? We assume that the observer answers, Two. If we collapse the display with respect to space (figure 6b), then the observer will still say, Two. Hence, spatial location is not an indispensable attribute for visual numerosity. But if we collapse over frequency (figure 6c), the observer will respond (because of auditory localization mechanisms), One. Hence, frequency is an indispensable attribute for auditory numerosity.

Figure 6.

Theory of indispensable attributes: the auditory thought-experiment. (a) Start. (b) Collapse over space: not indispensable. (c) Collapse over frequency: indispensable.

Because space cannot serve as a common medium for the mutual influence of an auditorially multistable and a visually multistable stimulus, we turn to time.

(ii). Time

One can construct a sequence of visible events that form a multistable stimulus accompanied by a sequence of audible events that also form a multistable stimulus, and ask whether the perceptual organization of one affects the perceptual organization of the other. Cook & Van Valkenburg [33] have provided the best evidence to date that they do not. They used a four-tone bistable sequence (figure 7). In such sequences, one hears the middle tone group either with the high or with the low tone, leaving the other tone perceptually isolated. Because the frequency of the middle tone determines the perceptual organization of this stimulus, they used an adaptive method to determine the frequency at which the sequence is most ambiguous, where the probabilities that the middle tones group with the high tone and the low tone are equal.

Figure 7.

Four-tone bistable tone sequence used by Cook & Van Valkenburg [33]. (a) The middle tones are heard grouped with the high tone, leaving the low tone perceptually isolated. (b) The middle tones are heard grouped with the low tone, leaving the high tone perceptually isolated.

Concurrently with the sounds, they flashed two horizontally separated light-emitting diodes three times on the left and once on the right (or vice versa) to form two temporal groupings at two spatial locations. The main variable in the experiment had two levels: the single flash coincided with the high tone or it coincided with the low tone.

The participants reported whether they heard the pattern of figure 7a or 7b. The results showed that the visual perceptual organization had no effect on the auditory perceptual organization. In a further experiment, they obtained a temporal ventriloquism effect (perceived simultaneity of the two isolated events) even when the tone and the light were 120 ms out of sync. Together, these findings suggest that unimodal grouping precedes interaction between auditory and visual stimuli.

Similarly, Pressnitzer & Hupé [34] have shown that although ambiguous auditory streaming [35,36] and moving visual plaids [37] follow similar temporal courses, they alternate independently.

(iii). Binocular rivalry

Of course, unambiguous stimulation in one modality can have an effect on multistability in another. These examples are not central to our concerns for two reasons. First, they are not about one multistable stimulus affecting another. Second, they are generally about binocular rivalry, which is probably the product of different mechanisms than the classic multistable phenomena that we have been discussing [38]. For the sake of completeness, we briefly review them.

Regarding the effect of haptics, Binda et al. [39] and Lunghi et al. [40] found that orientation-matched haptic stimuli can extend the dominance and reduce the suppression of visual stimuli. Maruya et al. [41] found that the congruence between voluntary action and the motion of one of the rival visual stimuli prolongs the dominance times of that stimuli.

For audition, Conrad et al. [42] found that when the perceived unambiguous motion of a sound is directionally congruent with one of two binocularly rivalrous visual motion stimuli, it prolongs the dominance periods of the directionally congruent visual motion percept. Kang & Blake [43] found that an unambiguous amplitude-modulated sound can influence the rate of alternation in a synchronously flickering binocularly rivalrous grating.

4. Cross-modal analogies in the perception of multistable stimuli

We now can state our view of the relationship between the perceptual organization of multistable stimuli in vision and audition: they proceed independently but show notable parallels. In particular, we find that in both vision and audition, when one applies more than one grouping principle to the same stimulus (an operation we call conjoining), they act additively.

(a). Conjoined grouping principles

As soon as Wertheimer [44] formulated the laws of perceptual organization in vision (anticipated, as Ash [45], points out, by Schumann [46] and Müller [47]), he pitted grouping by proximity against grouping by similarity, as shown in figure 8 [48, pp. 74–75]. In so doing, he showed how to conjoin two grouping principles, i.e. apply both principles to the same collection of elements.

Figure 8.

Wertheimer's [44] rectangular dimotif dot lattices apply grouping by proximity and grouping by similarity concurrently to the same stimuli. (a) Wertheimer's figure (xii): both proximity and similarity favour columns (in terms defined in figure 9, |b|/|a| =1.083). (b) Wertheimer's figure (xiii): proximity favours columns (in terms defined in figure 9, |b|/|a| = 1.104) but similarity favours rows.

Naturally enough, this raised the question of the joint effect of conjoined grouping principles: are they additive, i.e. act independently, or are they non-additive, i.e. create a new emergent property? This is a task for mathematical models.

(b). Quantifying grouping principles in vision

Our goal in this section is to answer the following question: When grouping by proximity and grouping by similarity are concurrently applied to the same visual pattern, what rule governs their joint application? Later, we will ask the same question about grouping in auditory perception.

To study grouping, researchers (starting with Wertheimer [44], our figure 8) often used dot lattices. To describe the earlier research, we use the taxonomy of dot lattices (Kubovy [49], who extended the work of Bravais [50]). Figure 9 summarizes the taxonomy.

Figure 9.

Dot lattices. (a) Defining features. (b) Two-dimensional space and nomenclature.

A dot lattice is a collection of dots in the plane that is invariant under two translations, a (with length |a|) and b (whose length is |b| ≥ |a|). These two lengths, and the angle between the vectors, γ (constrained for purely geometric reasons by 60° ≤ γ ≤ 90°), define the basic parallelogram of the lattice, and thus the lattice itself. The diagonals of the basic parallelogram (shown in figure 9a, but not represented in figure 9b) are c and d (where |c| ≥ |d|). In its canonical orientation, a is horizontal; the angle θ (measured counterclockwise) is the measure of the orientation of a dot lattice (θ = 15° in figure 9a); we call |a| the scale of the lattice.

If we are not interested in the scale of a lattice, then we can locate dot lattices in a two-dimensional space with dimensions |b|/|a| and γ (figure 9b). In this space, we can locate six different types of lattices, which differ in their symmetry properties.

Before we can explore how the effects of two grouping principles combine when we apply them conjointly, we must measure the strength of grouping by proximity on its own.

Oyama [51] had shown how to measure the strength of grouping by proximity without recourse to grouping by similarity. Using rectangular dot lattices at fixed orientation, he recorded the amount of time subjects reported seeing the competing vertical and horizontal groupings. He found that the ratio of the time they saw the vertical and the horizontal organizations is a power function of the ratio of the vertical and horizontal distances.

Using dot lattices at near-equilibrium, Kubovy & Wagemans [52] and Kubovy et al. [53] demonstrated that we can understand grouping by proximity as the outcome of a probabilistic competition among potential perceptual organizations. Figure 10 shows their results schematically. Consider two dot lattices (in which we assume that |a| = 1): in the first, |b| = 1.1 and γ = 76.48°; in the second |b| = 1.2 and γ = 77.68°. The corresponding lengths of c are |c| = 1.3 and 1.39, and the lengths of d are |d| = 1.65 and 1.72.

Figure 10.

The attraction function of grouping by proximity.

It is remarkable that all the values of log[p(v)/p(a)] (where response v ∈ {b,c,d}) fall on the same line, which we call the attraction function. Its slope, ξ, is a person-dependent measure of sensitivity to proximity:

where vector v ∈ {b, c, d}. We define ϕ(v) = p(v)/p(a), and call it strength of grouping along v. Thus

(i). The additivity of grouping principles in vision

Once we have determined how grouping varies as a function of relative distance, we can determine the effect of conjoined grouping principles. This requires us to plot a family of attraction functions and determine whether we can make these functions parallel, suggesting additivity (and hence independence). Kubovy & van den Berg [54] acquired their data using dimotif dot lattices as shown in figure 11. Using these stimuli, they obtained additive results shown schematically in figure 12.

Figure 13.

Multiplicative model of conjoined grouping that implies additivity of grouping by proximity and grouping by similarity when choice probabilities are represented as log-odds, as in figure 12.

Figure 11.

Two dimotif dot lattices. In both, grouping by proximity favours a, but more weakly in the dot lattice on left, where |a| = 1.25|b|, than in the dot lattice on the right, where |a| = 1.5|b|. In the dot lattice on the left, grouping by similarity favours b3 (δ > 0, where δ is a measure of dissimilarity between two kinds of dots), whereas in the dot lattice on the right it favours a (δ < 0), but because the differences between dot-colours are smaller on the left than on the right, the strength of the grouping by similarity (δ = 2) on the left is smaller than the strength of grouping by similarity on the right (δ =− 3).

Figure 12.

A schematic of the results obtained by Kubovy & van den Berg [54] using the dimotif lattices described in figure 11, showing that the conjoined effects of proximity and similarity are additive when choice probabilities are represented as log-odds. The line for δ = 0 (light grey) corresponds to the attraction function in figure 10, i.e. all the dots have the same colour, and therefore, grouping by similarity cannot affect the results. The results are equivalent to the multiplicative model of figure 13.

When we conjoin two principles of grouping, what determines the perceived outcome? At least with respect to proximity and similarity, they affect the outcome independently. The conjoint effect of these two grouping principles is equal to the sum of their independent effects. Just as with the attraction function, the additivity of laws is as far as one can imagine from the spirit of Gestalt, even though grouping is an emergent property, and thus a Gestalt.

(c). Quantifying grouping principles in audition

The segmentation of auditory rhythmic patterns is an important function of the auditory perceptual system, particularly in the processing of speech and music [55–57]. For example, if represents a note and represents a rest, then

is a rhythmic pattern with eight repeating elements. Such patterns can be multistable. You can hear the preceding pattern as either a cyclic version of

or of

We call such stimuli auditory necklaces (ANs) because they are cyclical sound patterns of equal-duration beats (notes or rests) that are best visualized as arranged in uniform spacing around a circle (figure 14). Assume that when we play an AN, we play it fast at first, eventually decelerating into a moderate tempo, in order not to bias the listener—which note will the listener hear as the initial note (its ‘clasp’)?

Figure 14.

An auditory necklace 11100110 of length n = 8; i.e. it is eight beats long. It has five notes and three rests, grouped into two sequences of notes called runs, and two sequences of rests called gaps.

To describe an AN, we need the notions of run, which is a sequence of consecutive notes, and gap, which a sequence of consecutive rests. We describe an AN by

• — its length, n, and

• — the number of runs it contains (which is the same as the number of gaps it contains).

Figure 14 shows an example: , which we code 11100110 (where 1s are notes and 0s are rests). It has two runs (111 and 11) and two gaps (00 and 0).

Garner and co-workers [58–61] proposed two principles to account for the segmentation of ANs—the run and the gap principles. According to the run principle, the first note of the longest run will be the clasp, whereas according to the gap principle, the first note following the longest gap will be the clasp. Most listeners hear the AN in figure 14 either as 11100110 (run principle) or as 11011100 (gap principle).

(i). The additivity of organizational principles in audition

Previous researchers have used two tasks to study the segmentation of ANs:

• — Reproduction. One can have participants report each pattern by tapping it out or by writing it down, from which one can infer the clasp note [58–61]. This made data collection slow and as a result the datasets were too small to be adequately modelled.

• — Tap the clasp. One can ask participants to strike a key to coincide with the moment of the clasp [62]. Although the amount of data collected by this method can be copious, it is noisy because participants have trouble synchronizing their taps with the onset of the clasp beat. Another drawback of this task is that the motor control required might affect the perception of the patterns.

We improved upon these procedures by devising a procedure that did not require participants to synchronize their responses with events in real-time. A circular array of n icons (where n is the length of the AN) appeared on the computer screen at the beginning of each trial (figure 15). Each icon was associated with one beat (a note or a rest) of the AN. The position of the icon corresponding to a potential clasp can be in any of the n locations around the circle with equal probability. As each beat of an AN occurred, a square highlighted the icon associated with that note and then moved clockwise to coincide with the next beat.

Figure 15.

The response screen. At the moment depicted, the cross is highlighted.

The participants' task was to click on the icon associated with the beat they heard as the clasp. They could do this at any moment, without regard for the currently highlighted icon; i.e. they did not have to synchronize their response with a sound or an event shown on the screen.

In the research we are briefly reporting here (as yet unpublished), we used a sample of ANs with two runs (a sequence of one or more consecutive notes) and two gaps (a sequence of one or more consecutive rests). We called the runs A and B (denoted r_A and r_B ). We denoted the gap preceding r_A as g_A, and the gap preceding r_B as g_B. We manipulated the lengths of the runs and the gaps.

The response variable was binomial—choosing r_A or r_B.¹ To quantify the run and gap principles, we calculated the run-length difference: Δr = (r_A) − (r_B) and the gap-length difference: Δg = (g_A) − (g_B).

The best model² of the data is additive with Δr and Δg as predictors. Figure 16 shows the effect of Δr and Δg on the probability of choosing r_A as the clasp (in log-odds scale). The parallel lines demonstrate the additivity of the two principles, and for reasons as yet unknown the effect of gap length difference is much larger than the effect of run length difference.

Figure 16.

p(r_A ) as a function Δ r and Δ g. Error bars span ± 1 s.e.

5. Final comments and future directions

We have previously laid out our views of the intricate relationships between visual perceptual organization and auditory perceptual organization [31]. There we argued for three kinds of relationships between the two modalities:

• — a duality, based on the theory of indispensable attributes, and the differing functions of these modalities. The visual system's primary function is to identify reflective surfaces, and its secondary function is to take sources of light into account. The auditory system's primary function is to identify sources of sound, and its secondary function is to take reverberations from surfaces into account;

• — a dependence, according to which auditory localization is primarily in the service of visual orientation; and

• — audio-visual object formation, which is the result of binding (as described in the first part of this paper).

Here we have shown that when we conjoin Gestalt principles within a modality, they operate additively, i.e. do not form a new emergent property, a new Gestalt. In the terminology we developed earlier, they are independent, not coherent. More generally, we claim that binding and emergence are different phenomena: when an auditory and a visual input undergo binding, a new property does not emerge. The one exception is speech: when a person's lips move and we hear a concurrent sound, we experience a common emergent property—meaning. In contrast, even though we experience the sight of a hammer hitting a nail and the sound of the impact as one, the percept retains two fully separable modal aspects; it is not a trans-modal experience.

To make this idea clear, we turn to Gibson, whose position was tantamount to the claim that fire was an emergent property of four kinds of stimulation:

A fire is a terrestrial event with flames and fuel. It is a source of four kinds of stimulation. One can hear it, smell it, feel it, and see it, or get any combination of these detections, and thereby perceive a fire. For this event, the four perceptual systems are equivalent. If the perception of fire were a compound of separate sensations of sound, smell, warmth and colour, they would have had to be associated in past experience to explain how one of them could evoke memories of the others. But if perception is simply the pickup of information [it] will be the same whatever system is activated.

… the problem of perception is not how the sound, the odor, the warmth, or the light that specifies fire gets discriminated from all the other sounds, odors, warmths, and lights that do not specify fire. (Gibson [63, pp. 54–55])

A byproduct of our analysis may be a new understanding of why Gibson's approach to multimodal integration may not be fruitful.

These ideas may also clarify the issue of multimodal multistability. For multimodal multistability to occur, there must exist trans-modal Gestalts that can undergo multiple forms of perceptual organization. Because we have no evidence that trans-modal Gestalts exist outside of speech, we are sceptical about multimodal multistability as a general phenomenon.