Binocular vision in insects: How mantids solve the correspondence problem

Abstract

Praying mantids use binocular cues to judge whether their prey is in striking distance. When there are several moving targets within their binocular visual field, mantids need to solve the correspondence problem. They must select between the possible pairings of retinal images in the two eyes so that they can strike at a single real target. In this study, mantids were presented with two targets in various configurations, and the resulting fixating saccades that precede the strike were analyzed. The distributions of saccades show that mantids consistently prefer one out of several possible matches. Selection is in part guided by the position and the spatiotemporal features of the target image in each eye. Selection also depends upon the binocular disparity of the images, suggesting that insects can perform local binocular computations. The pairing rules ensure that mantids tend to aim at real targets and not at “ghost” targets arising from false matches.

Animals that come equipped with binocular vision are confronted with the matching or correspondence problem. Somehow they must determine which retinal images in the two eyes correspond to the same feature in visual space. The task can be very complicated in primates. They recognize fine details of visual patterns in different depth planes and various interpretation rules are needed to guide the binocular matching of corresponding images (for review, see ref. 1). Insects on the other hand seem to use binocular vision mainly to localize distinct moving objects such as prey rather than to reconstruct the three-dimensional structure of the whole scene (for review, see ref. 2). Thus, their task is to pick out one object either against a stationary background or, more problematically, when there are several moving objects within their visual field (3, 4). How successfully do insects do this and how is selection accomplished?

An attractive insect in which to study this question is the praying mantis. Mantids are ambush predators that spot their prey binocularly. The visual fields of the two compound eyes overlap widely in front and retinal disparity has turned out to be essential for the estimation of prey distance: strikes at prey are only elicited when disparities indicate values corresponding to the normal catching range (5, 6). Binocular interactions are also involved in controlling the fixating head saccades that typically precede the strike. These saccades are preprogrammed and their size is specified by the weighted mean of the error angles of the target images on the two retinae relative to their frontally located foveas (7, 8). Thus binocular target localization in mantids is expressed by two distinct behavioral responses, the head saccades and the strike. Both responses are performed by a stationary insect, which makes them easily accessible to observation and experiment.

The aim of this paper is to examine how mantids select one out of two targets, using their head saccades as a behavioral indicator. We shall see that mantids use quite simple matching rules and strategies to solve the correspondence problem.

MATERIALS AND METHODS

Experiments were performed on Sphodromantis viridis, a large African mantid with a body size of 80 mm and an interocular separation of 8 mm. The binocular field of view is at least 80°, and the interommatidial angles in the frontal foveas are as small as 0.6° compared with more than 2° in the peripheral eye. The optimal catching range begins at a distance of about 20 mm from the mantid’s head and ends at 60 mm, but prey is detected and fixated far beyond this distance. Typical prey has a size of about 15 mm, corresponding to a visual angle of 20° in the middle of the catching range (9).

A mantid selected for an experiment was fixed up-side down to a holder, so that its head, legs, and abdomen were free to move (Fig. 1a). The mantid viewed one or two computer-generated targets projected onto a pair of screens with a video projector. Each target consisted of a stack of 10 black elements. Up and down movements were generated by adding new elements at one end of the stack and by removing simultaneously elements at the other end. In one type of target (denoted here as “simple” target), the elements were aligned to form a black bar. In another type, each newly added element was displaced randomly by a small amount either to the left or to the right relative to the central axis of the stack. This target more closely mimicked the erratic movements of a real prey and is denoted as “complex” target. The targets moved side by side in a vertical direction over a range of ±36° relative to the mantid’s horizon. The angular paramters of the stimuli on the insect’s eyes slightly changed according to their position, and all of the following values refer to the mean. Accordingly, the target velocity was 80°/sec, and the sizes of the simple and complex targets were, unless stated otherwise, 4° × 20° and 6° × 20°, respectively. Important in this study were experiments in which one or the other eye’s views of the targets were obscured. Appropriate occlusion situations were created by opaque Plexiglas bars, positioned by means of micromanipulators (Fig. 1).

Figure 1.

(a) Experimental set-up with a praying mantis fixed up-side down to a holder. S, display screens with a simple and a complex target moving downwards; H, horizon of the mantid; M, mirror to monitor head saccades from below (the video camera is behind the insect); O, occluder to obsure targets for one or the other eye. (b) Illustration of the mantid head with projection of paired targets with a center-to-center separation of 30°. An occluder is adjusted to obsure the right target in the left eye view. Solid arrow, horizontal head angle relative to a position midway between targets; broken arrows, viewing directions of foveas; broken line, inner border of the catching zone (20 mm from the head; outer border is at 60 mm).

In each trial the mantids were first required to fixate a single target moving upward (giving their head a defined direction in azimuth) and then to track the test stimuli moving downward. Visual tracking was video-taped and the horizontal head angle after completion of tracking saccades was measured to an accuracy of 1°. Finally, the head angles were sampled in 3° intervals and accumulated in histograms normalized so that the total frequency equalled 1.

RESULTS

In a first set of experiments, mantids were confronted by two targets in different constellations, but with a fixed center-to-center separation of 30°. As Fig. 1b shows, the retinal images from such paired targets could be matched in four different ways. Two of these are correct and correspond to the matching of a lateral image in the left eye with a medial image in the right eye and vice versa. The two other matches are false and arise from pairwise matchings of the two lateral or the two medial images. In the latter cases, one would expect the mantids to fixate positions midmay between the targets, but as Fig. 2 shows, this did not normally happen. The mantids were able to fixate one or other target, preferring a target that fell closer to the fovea to a more peripheral one (Fig. 2b), a target within to a target outside the catching range (Fig. 2c), and a complex target to a simple target (Fig. 2d). However, there was uncertainty when two identical objects appeared at equal angles from the visual midline (Fig. 2a). In this case an initial saccade brought the target images of one or the other object closer to the foveas, and fixation was then completed with a second saccade. To a lesser extent, the same holds true when the two targets differed in their viewing distance (Fig. 2c), showing that over the catching and fixation range, distance itself is not very important in the control of head saccades. However, unlike the experiment of Fig. 2c, the angular size of real prey is reduced when it moves outside the catching range. Indeed, when the experiment was modified such that both targets had the same absolute size of typical prey (12 mm × 15 mm), the mantids always fixated the closer target on its initial saccade. Thus, these results suggest that the mantid’s saccade generating system can resolve potential matching ambiguities and can select one of several objects.

Figure 2.

Binocular fixation behavior in the presence of paired targets. Plots of distributions of fixation positions after the first (upper histograms) and the second saccade (lower histograms) of visual tracking. Targets are separated by 30° (dotted lines). With one exception the viewing distance of the display at the horizon is 40 mm. (a) Two identical (simple) targets with the head pointing midway prior to tracking (n = 524 saccades from 9 mantids). (b) Same stimulus conditions as in a, but the head points 6° ± 1.5° to the left prior to tracking (n = 232 saccades from 5 mantids). (c) Left target is within and right target is outside the catching range (40 mm and 140 mm, respectively), but both are equal in angular size and angular elevation when moving downwards (n = 476 saccades from 10 mantids). (d) Left target is complex and right target is simple (n = 298 saccades from 6 mantids). Note that head saccades in all four situations undershoot targets. This also holds true, when only the left or the right target is presented. Arrows at bottom of histograms show mean head angles of such trials.

For more detailed investigation of the matching rules, mantids were confronted with paired stimuli with occluders positioned to obscure one or the other eye’s views of the targets. First, occluders were placed so that the retinal information was restricted either to the medial or the lateral pair of images. The question is whether the mantids under these artificial conditions fixate the “ghost” targets that they had largely ignored in the experiments of Fig. 2. The results show that fixating saccades were evoked that brought the head midway between the two targets (Fig. 3 a and b), implying that the saccadic system does in fact interpret the paired stimuli as resulting from a single object. However, the rate of saccades was much reduced, perhaps because the disparities of the two images lay well outside those corresponding to the normal strike and fixation range (compare with Fig. 1b). Indeed the image matchings evident in Fig. 3 a and b were not expressed when an occluder was placed such that one target was seen by both eyes and the other by only one eye. There were now two possible matches and the mantid selected consistently the one corresponding to a target within the striking range (Fig. 3 c and d). This result suggests that horizontal disparity is one important determinant of the process of pairing and selection.

Figure 3.

Occlusion experiments to demonstrate the role of disparity in target selection. Same target constellation and viewing conditions as in Fig. 2a, but only the first saccade of visual tracking is considered. The viewing distance of the display is 40 mm (middle of the catching zone). Targets are presented to nine mantids and occluders are placed to mask the lateral eye regions in a (n = 105 saccades), the medial regions in b (n = 136 saccades), the lateral region of the right eye in c (n = 145 saccades), and the medial region of the left eye in d (n = 391 saccades). Arrows at bottom of histograms show mean head angles when only the left or the right target is presented.

Target selection is also influenced by the monocular features of the retinal images. When mantids with one eye totally occluded were presented with two identical (simple) targets, they made saccades toward the lateral target lying nearer to the fovea (Fig. 4a). In contrast, the mantids’ visual attention was divided between the paired stimuli, when the medial and more peripheral target was complex (Fig. 4b). A similar effect was observed when the medial target was simple but considerably larger than the lateral one. These results show that retinal images in each eye are weighted according to their position and their spatiotemporal features.

Figure 4.

Fixation behavior of monocular mantids. Head points midway between the targets prior to tracking and the seeing eye of seven mantids views either a pair of simple targets (a) (n = 115 saccades) or a simple and a complex target (b) (n = 156 saccades). Arrows at bottom of histograms show mean head angles when only the left or the right target is presented.

What is the effect of this weighting on binocular coordination? To study this question occluders were first arranged so that each eye saw only one of two targets, and the rate of saccades was determined for three different situations: either both targets were simple or complex or one target was simple and the other was complex. The results in Fig. 5a show that saccade frequencies are lowest for a pair of simple targets, highest for a pair of complex targets, and in between for the combination of both stimuli. These findings indicate that the weights of the monocular images are averaged during binocular coordination. The resulting signal is weighted in turn by the disparity between the images. This is suggested by the observation, that the ghost targets in Fig. 5a evoked fewer saccades than the monocularly viewed single real targets in Fig. 5b, obviously by virtue of the unfavorable disparity between the images of the ghosts. Furthermore, when one eye viewed a complex and the other both a complex and a simple target, the mantids behaved according to the matching rule of Fig. 3 and fixated the complex target (Fig. 5c). However, the complex target still exerted an effect when the situation was reversed, i.e., when the simple target was seen by both eyes and the complex target was seen by only one eye. In this case, the mantids showed a tendency to fixate positions in between the targets as though the paired stimuli at a preferred disparity value would no longer fully suppress the less favored match arising from the two lateral images (Fig. 5d). Thus, these results suggest that the strength of binocular matchings is determined by the combined effect of the monocular weights of the images and the disparity between them. In normal vision, then, the most effective combination of monocular and binocular retinal cues, and the one that will guide the fixation response, arises from a typical prey object that moves in front within the catching range of the mantid. It is this object that the visual system preferably selects from among others as suggested by Fig. 2. Finally, it should be noted that the weighting of the monocular images also affects the magnitude of the binocular saccadic command signals. The combination of a complex target to one eye and a simple target to the other results in fixation responses that are biased toward the complex target (Fig. 5 e and f). Thus, binocular positional information in the saccadic system is not invariant to the features of the retinal images (see also ref. 7).

Figure 5.

Occlusion experiments to demonstrate the interplay between monocular and binocular retinal cues. In a, occluders are placed to mask the lateral eye regions. Three different constellations of simple and complex targets are presented. Columns represent the frequency of the occurrence of one or several saccades when targets move downwards (n = 445 trials from five mantids). In b, occluders are arranged as in a, but only one target (simple or complex) is presented (n = 225 trials from the same mantids as in a). In c (n = 144 saccades from four mantids) and in d (n = 314 saccades from seven mantids), the situation is as in Fig. 3d except that either the left or the right target is complex. In e (n = 96 saccades from four mantids) and in f (n = 212 saccades from five mantids), the situation is as in Fig. 3 a and b, except that the left target is complex. Arrows at bottoms of histograms in c–f show mean head angles when only the left or the right target is presented.

In the discussed experiments so far, the paired targets were widely separated (30°) and in this condition individual targets are fixated. This no longer holds true when the target separation is reduced to about 20° or less. In situations like those of Fig. 2, the mantids then still show some preference for a complex target, but in all other constellations, they tend to fixate positions in between the targets throughout their tracking runs as though they were aiming at the center of gravitiy of the pair of targets. Occlusion experiments suggest that this behavior results from the influence of false matches and their fusion with the correct ones. The obvious consequence of such a fusion is that the paired stimuli are treated as a single image, an assumption that is supported by the experiment in Fig. 6. This experiment is based on the observation that mantids normally do not fixate and track objects which considerably exceed the size of typical prey (compare Fig. 6 a and b). Thus when mantids are presented with several targets occupying together an angular range of an abnormally large object, the occurrence or absence of saccades indicates whether they can select one target out of several alternatives or whether they treat the set of stimuli as a single object. The results in Fig. 6 c–e show that the rate of saccades drastically drops from a high value to zero, when the separation of nearby targets is reduced from 35° to 18° and finally to 9°. It seems, therefore, that the mantid’s capacity to separate different objects is restricted to a rather coarse angular scale. The resolving power of the peripheral eyes would allow, of course, a much better performance. In the frontal eye regions of Sphodromantis, the interommatidial angles are less than 1° and this means that the limitation on target separation is not the resolution of the retina but rather the resolution of the central brain.

Figure 6.

Spatial limits of target separation. Mantids are presented with a single target in a and b and with three, five, and nine targets in c–e, respectively. Target constellations in c–e subtend the same 70° horizontal visual angle as the single target in b. Columns represent the frequency of the occurrence of one or several saccades when targets move downwards through the mantid’s binocular field of view (n = 367 trials from seven mantids). No saccades occur in b (because the size of the target exceeds those of typical prey) and in e where neighboring targets are 9° apart.

CONCLUSIONS

The findings of this paper suggest that both monocular and binocular cues from the images of a set of objects interact to deliver appropriate matching solution, so that the mantid will tend to fixate the most attractive and the most easily catchable target. Complications arise when targets are close together. However, this limitation does not impair the mantid’s fixation behavior because normal prey at the catching distance subtend a visual angle of 20° or even more (9). Indeed interpreting narrowly spaced targets as a single image greatly simplifies binocular depth vision when it comes to the strike. If the resolution were finer than 20°, two targets (or pattern elements of a single target) could generate ghosts within the catching zone (20–60 mm from the mantid’s head, see Fig. 1b) and thus elicit an erroneous strike. However, the coarse resolution means that such configurations are treated as a single fused image and as a consequence the mantid’s assessments of range normally refer to a single real target (S.R., J. Herberholz, and M. Stein, unpublished observations). Thus it seems that the mantid’s strategy is to keep potential ghosts out or at least at the border of the range that is most relevant in its visual behavior. Perhaps this is the simplest method a visual system can adopt to cope with the correspondence problem.

It remains to be seen what kind of neuronal mechanisms are involved in binocular target selection of mantids. A simple scheme that explains the above results astonishingly well is based on binocular neurons arranged in arrays such that their optical axes provide an appropriate scaling of visual space. The receptive fields of nearby neurons overlap to a allow a greater positional accuracy by interpolation. They have inhibitory surrounds and the size of their excitatory centers is tuned to tolerate the range of disparities that is relevant in visual behavior. Activities in locally distinct sets of nearby neurons then reflect the combined effect of monocular and binocular retinal cues and specify both of the components necessary for guiding target selection and fixation–the attractivity value of targets and their binocular direction. The binocular control of the strike is presumably more demanding in terms of neuronal processing. It involves precise distance information and the disparity selectivity must be finer than that required for the selection of targets by the saccadic system. There is some morphological evidence for binocular retinotopic projections in the mantid brain (10), but their role in binocular vision has yet to be demonstrated.