Skip to main content
Proceedings of the National Academy of Sciences of the United States of America logoLink to Proceedings of the National Academy of Sciences of the United States of America
. 2016 Jun 16;113(27):7638–7643. doi: 10.1073/pnas.1603120113

Cross-modal object recognition and dynamic weighting of sensory inputs in a fish

Sarah Schumacher a,1, Theresa Burt de Perera b, Johanna Thenert a, Gerhard von der Emde a
PMCID: PMC4941484  PMID: 27313211

Significance

Being able to spontaneously recognize objects across different senses increases the flexibility of a multisensory system, because the reliability of different sensory inputs depends on the environment and range. However, this ability has only been found in mammals, suggesting such a high-level function might be associated with complex mammalian brain structures. We show that the weakly electric fish Gnathonemus petersii, is also capable of spontaneous cross-modal object recognition, discriminating objects learned visually with the active electric sense and vice versa. Furthermore, the modality-specific inputs are weighted dynamically according to their reliability at different ranges. This finding suggests that these complex cognitive abilities are present in fish in addition to mammals, and therefore do not require a cerebral cortex.

Keywords: multisensory integration, perception, sensory transfer, sensory reliability, sensory conflict

Abstract

Most animals use multiple sensory modalities to obtain information about objects in their environment. There is a clear adaptive advantage to being able to recognize objects cross-modally and spontaneously (without prior training with the sense being tested) as this increases the flexibility of a multisensory system, allowing an animal to perceive its world more accurately and react to environmental changes more rapidly. So far, spontaneous cross-modal object recognition has only been shown in a few mammalian species, raising the question as to whether such a high-level function may be associated with complex mammalian brain structures, and therefore absent in animals lacking a cerebral cortex. Here we use an object-discrimination paradigm based on operant conditioning to show, for the first time to our knowledge, that a nonmammalian vertebrate, the weakly electric fish Gnathonemus petersii, is capable of performing spontaneous cross-modal object recognition and that the sensory inputs are weighted dynamically during this task. We found that fish trained to discriminate between two objects with either vision or the active electric sense, were subsequently able to accomplish the task using only the untrained sense. Furthermore we show that cross-modal object recognition is influenced by a dynamic weighting of the sensory inputs. The fish weight object-related sensory inputs according to their reliability, to minimize uncertainty and to enable an optimal integration of the senses. Our results show that spontaneous cross-modal object recognition and dynamic weighting of sensory inputs are present in a nonmammalian vertebrate.


To behave adaptively, an animal must be able to perceive and react appropriately to environmental stimuli. Sensory information can often be obtained through multiple sensory channels and can interact in a number of ways before a behavioral output is produced. To increase the flexibility of a multisensory system, information about objects in the environment can be transferred between different senses; this enables some animals to use spatial information acquired with one particular sensory system to recognize objects with another one (cross-modal object recognition). In contrast to simple forms of cross-modal information transfer, which are based on the formation of direct associations between two specific stimuli (14), cross-modal object recognition requires additional and more complex conditions to be met. These conditions are as follows: (i) The information provided by the two senses has to match in content (i.e., both senses have to provide information about the same characteristic object property; e.g., shape, surface structure). (ii) The sensory inputs have to be encoded in a way that allows temporally disjointed information from two senses to be identified as identical, despite these senses relying on different physical stimuli. (iii) Characteristic object features have to be stored in a neuronal representation that is accessible by multiple senses. So far, spontaneous cross-modal object recognition has only been described in humans (5), apes (6), monkeys (7), dolphins (8), and rats (9), and little is known about the neuronal structures that are involved in this process.

A reliable percept is fundamental for cross-modal object recognition. Although the interaction of multiple sensory channels offers many advantages, the integration of conflicting information from different senses could also lead to a decrease of perceptual reliability. Therefore, to obtain a reliable percept, not all available senses contribute equally, and the observable behavioral output tends to be dominated by certain senses. Which sense dominates and the degree to which each sensory input contributes to the overall percept depends on the conditions and the task, and might be determined by the reliability of the different sensory inputs under the given conditions (1013) and prior experience (14, 15). In humans, for example, vision is dominant during spatial tasks (16, 17), whereas the acoustic and the haptic senses dominate over vision during tasks that require temporal assessments (18, 19). Because conditions may change rapidly, this “weighting” of sensory inputs has to be dynamically adjustable. Dynamic weighting of sensory inputs enables animals to integrate multisensory information optimally to obtain a reliable percept of the environment (11), but like cross-modal object recognition has so far only been described in mammals (11, 20, 21).

Weakly electric fish Gnathonemus petersii possess multiple senses, which potentially could be used for fine-scale spatial interrogation of their surroundings. These fish can discriminate between nearby objects using active electrolocation (22), a process during which object-evoked distortions in a self-generated electric field (electrical images) (23, 24) are perceived with special electroreceptor organs in the skin (25). Active electrolocation is a near-field sense, which works only at short distances from the fish (22). In addition to this active electric sense, G. petersii possess a visual system with highly specialized eyes (26). The visual system is adapted to the crepuscular and nocturnal activity of the fish and their habitat (turbid black water streams in Central and West Africa). In the “grouped retina,” the photoreceptors are packed into bundles within a tapetum lucidum, which improves vision under dim light and in turbid waters (26). Previous studies have shown that the visual and electrosensory inputs can be integrated when sensing the surroundings (27), and both senses provide spatial information (matching content) about objects.

Here we used G. petersii to test for spontaneous cross-modal object recognition and dynamic weighing of sensory inputs. We applied a two-alternative forced-choice procedure, during which the fish were trained to discriminate between two objects using only vision or only their active electric sense and we tested them subsequently with the untrained sense. Crucially, we varied access to object information using the two senses by altering features of the objects themselves without having to surgically manipulate the fish. This was an important approach as it reduced the uncontrolled effects of modifying animals’ senses.

Results

To test for spontaneous cross-modal object recognition, 10 G. petersii with no experimental experience were trained to discriminate between two objects of different shapes placed at a distance of 1 cm from the viewing gate. By using objects made from different materials, either the visual sense only (red colored electrically transparent agarose objects; group 1, n = 5) or the active electric sense only (metal objects covered by hoods; group 2, n = 5) provided information, which could be used for object discrimination (Fig. S1). After the fish reached a preassigned learning criterion (75% correct choices on 3 consecutive training days), they were subjected to transfer tests, during which they could use only the untrained sense for the discrimination task: in the visually trained group, the active electric sense (metal objects encased in a cube of electrically transparent agarose presented in the dark), and the visual sense in the electrically trained group (red-colored electrically transparent agarose objects). Control tests were performed to ensure that the fish could not use any other cues (e.g., electrical or lateral line input) to discriminate between the red-colored, electrically transparent agarose objects, and that the hoods did not influence the electrical performance during training or the tests (Control Tests and Figs. S2S6).

Fig. S1.

Fig. S1.

Training objects of the visual trained group (A and B) and the electrically trained group (CE). To prevent electrical discrimination, the objects of the visually trained group were made of red-colored electrically transparent agarose. For training with the electric sense, metal objects covered with hoods made of black cotton fabric were used. In all fish, a sphere was used as the positive object and either a cross (A and C) or a cuboid (B and D) served as negative object.

Fig. S2.

Fig. S2.

Discrimination performance during the dark control. These control tests were conducted to ensure that no additional cues were used to discriminate between the electrically transparent, red-colored agarose objects. Tests were conducted at 1-cm (green) and 3-cm (light green) object distances before and after electric silencing (dark green) with the visually trained fish (fish 1–5) and four of the electrically trained fish (fish 6 and 8–10). A χ2 test showed that none of the performances was significantly different from 50% chance level (P > 0.05), indicating that the fish were not able to use additional cues such as electrical or lateral line input to discriminate between the objects. This result shows that during the tests with light, only vision was used for discrimination. For a further description, see Fig. 1.

Fig. S6.

Fig. S6.

Discrimination performance of the visually trained fish (fish 1–5) in complete darkness with (red) and without (dark red) agarose cubes encasing the aluminum objects. These control tests were conducted to exclude influences of the agarose cubes on the discrimination performance during the electrical tests. The exact Fisher test showed no significant differences between both conditions (n.s.: P > 0.05). For further description, see Fig. 1.

After training, all fish of both training groups were able to discriminate between the objects with similar performances of 85–92% correct choices. However, when the objects were placed at a distance of 1 cm from the viewing gate, the results of the two groups during the transfer tests differed (Fig. 1). The discrimination performance of all fish in the visually trained group remained constant during transfer tests, during which only the electric sense was available for discrimination (Fig. 1A), indicating that the fish were able to spontaneously discriminate between the objects electrically without previously being trained with this modality. In other words, the fish were capable of cross-modal object recognition. In contrast, only two of five fish trained with only the active electric sense available reached a performance significantly different from chance level when tested only with visual information available for object recognition. These two fish transferred information from the active electric sense to vision (Fig. 1B).

Fig. 1.

Fig. 1.

Visual (blue) and electrical (red) discrimination performance, under training conditions and during the transfer tests of (A) visually trained fish (fish 1–5) and (B) electrically trained fish (fish 6–10). All trials were conducted with two objects that only differed in shape placed at a distance of 1 cm behind the respective gates. The number of trials conducted with each condition is indicated within the bars. Training results include all training trials after reaching the learning criterion. The dashed line indicates the 50% chance level. A χ2 test was conducted to test whether the performances were significantly different from chance level (*P ≤ 0.05; **P ≤ 0.01; ***P ≤ 0.001).

We were unable to ascertain whether, in the three unsuccessful fish (fish nos. 7, 8, and 10), transfer from the electric sense to vision failed to occur at the processing stage or whether transfer did occur, but was masked by a dominance of the electric sense. During the tests when objects could only be perceived visually, the electrically trained fish were presented with conflicting information from the electric sense (providing the information that no object is present) and from vision (providing the information that an object is present). Therefore, the inability to discriminate between the objects in the visual transfer tests might have been driven by a reliability-based dominance of the electric sense at short distances. To investigate this hypothesis, four fish [fish nos. 6 and 8–10 (fish 7 died)] from the electrically trained group were subsequently tested at different distances from the objects in three experimental situations: first, with only the active electric sense providing information about the objects (to provide a measure of the reliability of the electric sense at different distances); second, with only vision available (to test the transfer from the electric sense to vision); and third, with both senses available for object discrimination (a control).

At longer distances, the discrimination performance of the fish in the electrical tests decreased in line with decreasing reliability of the electric sense (Fig. 2, red curves). However, the performance in the transfer tests, in which vision alone was available for object discrimination, increased with distance, eventually reaching a similar level as in the electrical training (Fig. 2, blue curves). At even longer distances, the performances decreased again. These tests reveal that transfer must have occurred from the active electric sense to vision (there was no failure of transfer at the processing stage), but at short distances may have been masked in all but two fish at the point of the behavioral output.

Fig. 2.

Fig. 2.

Discrimination performance of the electrically trained fish [fish 6 (A), 8 (B), 9 (C), and 10 (D); same fish as in Fig. 1B] tested at different distances with only the electric sense (red circles), only vision (blue squares), and both senses (black triangles) available for object discrimination. At least 30 trials were conducted for each distance. The electrical (red line) and visual (blue line) results were fitted with a Gaussian fitting curve and a sigmoidal fit was used for the results of the tests with both senses available (black line). The R2 value is given in the corresponding color for each curve in the figure. Results above the dotted line are significantly different from chance level (P ≤ 0.05, χ2 test).

When tested with both senses (vision and the electric sense) available for object discrimination, the information from both modalities corresponded, leading to a high discrimination performance at both short and long distances (Fig. 2, black curves). Even at intermediate distances, where in both uni-modal cases the performances were near threshold-level (at a distance of about 2 cm), the fish now discriminated effectively between the objects (Fig. 2). Control tests ensured that this effect was not due to differences in the experimental conditions (Fig. S3).

Fig. S3.

Fig. S3.

Discrimination performance of four of the electrically trained fish (fish 6 and 8–10) during control tests, which ensured that there was no influence of the hoods on the discrimination performance. Electrical tests were conducted with the hood-covered metal objects (black) and with conductive, red-colored agarose objects in the dark (gray) at 2-cm respective to 3-cm object distances. The exact Fisher test was used to test for significant differences between the performances (n.s.: P > 0.05). There were no significant differences between both conditions, showing that there was no influence of the hoods on the discrimination performance. Thus, performance differences between electrical tests and tests with the conductive red-colored agarose objects in light (tests with both senses) (Fig. 2, red and black curve) were not due to differences in the experimental conditions but originated from the additional visual input. For a further description, see Fig. 1.

To pursue this idea further and to test whether the inability to discriminate between the objects visually at short distances might be due to the visual sense not functioning, the electric organ of each of the four electrically trained fish (fish 6 and 8–10) was surgically silenced. To do this, the spinal cord was sectioned anterior to the electric organ, which is located in the caudal peduncle (28). Following this procedure, the performances in the visual transfer tests at short distance (1 cm) increased to a level similar to that during electrical training (Fig. 3). This revealed that at short distances the eyes of the animals do provide information about object shape, and that in the absence of any electrical input the behavioral output is driven by the visual input. Control experiments after the tests at varying distances, before the fish were electrically silenced ensured that the changes in the discrimination performance were not due to experience gathered during the experiments with both senses (Fig. S5).

Fig. 3.

Fig. 3.

Discrimination performance at 1-cm object distances during the visual transfer tests of the electrically trained fish (fish 6 and 8–10; same fish as in Figs. 1B and 2) before (blue) (same data as in Fig. 1B) and after (dark blue) electric silencing (surgical deactivation of the electric organ). For further description, see Fig. 1.

Fig. S5.

Fig. S5.

Discrimination performance of the electrically trained fish during control tests, which were conducted to exclude influences of the principal surgical procedure for electrical silencing on the discrimination performance. Electrical (red) and visual (blue) tests were conducted before the sham operation (intact, same data as in Fig. 1), after the sham operation and after electrically silencing (only for the electrical tests). The exact Fisher test was used to test for significant differences between the performances of the intact fish compared with after the sham operation and after electric silencing (***P ≤ 0.001; n.s.: P > 0.05). For a further description, see Fig. 1.

Control Tests

During training in the visual trained group and during the transfer tests in the electrically trained group, red-colored agarose objects that had approximately the same conductivity as the surrounding tank water were used. Even though the resistance of the objects was measured and matched that of the tank water, the fish might have been able to discriminate between these objects electrically using differences between the resistance of the objects compared with the tank water that were not technically measurable. Furthermore during the visual experiments an influence of the lateral line on the discrimination performance could not be excluded through the object design. Therefore, to ensure that the fish used no other sensory cues than visual ones to discriminate between the red-colored, electrically transparent agarose objects, control tests at 1-cm and 3-cm distances were conducted before and after the fish were electrically silenced. The objects were presented in the dark (light intensity <0.01 lx) excluding visual discrimination. To observe the outcome, the tank was illuminated with infrared light of 850 nm (IR Illuminator, S8030-3D-l-IR; ITAKKA), which is invisible for G. petersii (54), and was observed through an infrared sensitive camera (DCR-HC40E; Sony). All fish of both training groups were unable to discriminate the objects under these conditions (Fig. S2), showing that neither the lateral line system nor the electrical input was sufficient for the discrimination task, and that the fish used vision to discriminate between the objects during the training respective to the transfer tests with light.

A second control was conducted to test whether the black cotton hoods that were used during the training of the electrically trained group influenced the discrimination performance. During the tests with both senses available, the performance of the electrically trained fish increased compared with the tests with only the electric sense available for object discrimination especially at longer distances (Fig. 2, black curve). This could be explained by the additional visual input or it could have been an effect of the changes in the experimental condition. During tests with only the electric sense available, the objects were covered with hoods to prevent visual discrimination. These hoods were removed during the tests with both senses available, which could have led to an increase in performance. Furthermore during the tests with both senses available, red-colored agarose objects were used instead of the metal objects that were used during electrical tests. These were made of a high conductive saline solution (see description of range tests), so the different object material might have also influenced the performance. To exclude these factors, control tests with the conductive red-colored agarose objects were conducted in the dark at 2-cm respective to 3-cm distances. The results of these control tests were almost identical to the results of the tests with the hood-covered metal objects (Fig. S3), revealing that neither the hoods nor the different object material influenced the discrimination behavior and leaving the additional visual input as an explanation for the performance increase in the test with both senses available. The lack of influence of the cotton hoods furthermore supports the assumption that the cotton hoods were electrically transparent.

A double blind control under the training conditions was conducted to make sure that there was no influence of the experimenter on the decision of the fish. During this control, a person that had previous experience with the experimental procedure but had never worked with the tested fish before and did not know which object was positive, conducted the experiments. The performance of the fish under these conditions was no different compared with the results with the original experimenter (Fig. S4).

Fig. S4.

Fig. S4.

Discrimination performance of one fish of each training group (fish 1 trained with vision and fish 6 trained with the active electric sense) with the usual experimenter (dark purple) and with another experimenter during the double blind control (light purple). The tests aimed at excluding influences of the experimenter on the discrimination performance and were conducted under training conditions with both fish. The exact Fisher test showed no significant differences between both conditions (n.s.: P > 0.05). For a further description, see Fig. 1.

After the sham operation, a control was conducted to ensure that the principal surgical procedure had no influence on the performance. The fish were tested with only the electric sense and with only vision available. In all fish, there was no significant change in performance during these tests compared with before the sham operation (Fig. S5, intact vs. sham). This result shows also that the performance changes after the real operation were not influenced by previous experiments with both senses available.

After the surgery, control tests with the hood-covered aluminum objects were conducted to test whether the electrically silenced fish were unable to discriminate between the objects electrically. Furthermore this control ensured that no additional electrical cues arising from the metal objects influenced the discrimination performance by stimulating the passive (ampullary) electrosensory system of the fish. None of the electrically silenced fish was able to discriminate between the objects without visual cues and the performance of the fish decreased highly significantly compared with the performance of the intact fish (Fig. S5, silenced).

In the visually trained group, aluminum objects encased in a cube of electrically transparent agarose were used in complete darkness during the electrical test to prevent influence of the lateral line system. To ensure that the agarose cubes did not influence the discrimination performance, control tests were conducted, during which the discrimination performance of the visually trained fish with and without the agarose cube were compared in the dark. These tests showed that the agarose cubes did not influence the discrimination behavior significantly (Fig. S6).

Discussion

Our results show that, similar to mammals, the weakly electric fish G. petersii is capable of spontaneous cross-modal object recognition. Cross-modal object recognition increases the flexibility of a multisensory system as it allows animals to recognize objects under varying conditions (e.g., day and night) and to exploit the advantages of their long-range (vision) and short-range (active electrolocation) sensory systems optimally. Because the senses have to provide information with matching content for spontaneous cross-modal object recognition, in most mammals, this ability is restricted to vision and the haptic/tactile sense (57, 9). Similarly, dolphins are capable of performing cross-modal object recognition between vision and active echolocation (8). Here we show, for the first time to our knowledge, that cross-modal object recognition is also possible between vision and the active electric sense. Although the mechanosensitive lateral line system or the passive electric sense also could have provided object information and might influence object recognition in a natural environment, control experiments showed no involvement of these sensory systems in our experiments (Figs. S2 and S5).

During our study, G. petersii used their active electric sense and vision to acquire information about the shape of the two objects to be discriminated. Thus, the animals had to recognize that the information provided by the two sensory systems about the same spatial object feature were identical, even though they were relying on different physical stimuli and the sensory information was arriving sequentially. This means that the sensory information had to be encoded in the brain in a way that allowed the two sensory channels to exchange information and compare and match object related inputs. This highly cognitive ability could be achieved in two different ways: First, information about certain object features (in our experiment pertaining to shape) is encoded in a generic form, regardless of the input channel (i.e., both channels use a matching format of encoding). This would enable the fish to recognize objects cross-modally without any previous experience, in other words cross-modal object recognition would be an innate ability. Alternatively, information originating from multiple senses might not match in format. Instead, the fish could have learned to associate visual and electric inputs of basic features also common in other environmental objects when exposed to the features in the past. So, for example, a fish might have learned to associate a visual and electric image of a curved edge or a corner. Subsequently, these associations would have been generalized to new objects and new situations. In this case, cross-modal object recognition would be dependent on sensory experience and would not be innate. At present, we do not know which scenario is correct for G. petersii. Studies with humans have shown, however, that newborn infants are cable of cross-modal recognition of object shape and texture using touch and vision, suggesting that information is encoded generically (2931); however, this has remained untested in any other animal.

To recognize objects cross-modally, information about characteristic object features has to be stored in some kind of neuronal representation, which is subsequently accessible by the other sense. This could be achieved either through a comparison of modality-specific representations or through a single multimodal representation stored in a multisensory brain area (9, 32, 33). In mammals, cross-modal object recognition is associated with cortical structures such as the prefrontal cortex (6), the perirhinal, or the posterior parietal cortex (5, 7). Because fish lack an isocortex, the ability of cross-modal object recognition cannot depend on the existence of these mammalian brain structures per se. However, a recent study has shown a cryptic laminar and columnar organization of the dorsolateral pallium (DL) of a gymnotiform weakly electric fish which, together with other organizational structures, supports the hypothesis that there is a homology between the teleost DL and the mammalian cortex (34). Furthermore the pallium of G. petersii is known to receive inputs from the auditory, the visual, the electrosensory, and the lateral line systems (35), and lesion experiments in goldfish have shown that the teleost telencephalon is involved in spatial learning tasks (3638), making it a prime candidate for the location of cross-modal object recognition in G. petersii. Other brain areas such as the tectum opticum, the torus semicirularis, and the valvula cerebelli also receive multiple sensory inputs in G. petersii and therefore could also be involved in cross-modal transfers.

In G. petersii, cross-modal object recognition is influenced by dynamic weighting of the sensory inputs. The fish weight object-related sensory inputs according to their reliability, to minimize uncertainty and to enable optimal multisensory integration. At short distances, the active electric sense dominates the behavioral output during object discrimination. When the object was close by, the conflict between vision and the active electric sense (one sense providing the information that an object is present, the other sense providing the information that no object is present) was resolved in favor of the active electric sense, leading to a reduction in ability to discriminate between the objects visually in the intact electrically trained fish (Figs. 1 and 2). However, after being electrically silenced, the same fish were able to discriminate between the objects visually at short range at the same level as during electrical training (Fig. 3). This finding suggests that in the absence of any electrical input, the visual information was no longer overwritten and the behavioral output was driven by the visual input, supporting the hypothesis that when intact, the ability to discriminate between the objects visually was masked by the dominance of the active electric sense. At short range, the reliability of the electric sense exceeds that of the visual sense, which has a relatively low spatial resolution (minimal visual angle of about 3°) (26, 39, 40). In contrast, active electrolocation provides the fish with fine-scale 3D spatial information and additionally informs the fish about the electrical properties of an object (41, 42). The electric sense is very reliable at short distances because environmental factors like light level, turbidity, or small suspended particles do not interfere with active electrolocation (in contrast to vision, which suffers, for example, from reflection, refraction, scattering, and attenuation) (43). In addition, the observed dominance of the electric sense at short range might be based on the prior experience that in nature there are no nearby objects that cannot be perceived electrically. Because of the huge conflict between the visual and the electrical inputs during our experiments, integration might have even broken down (segregation), so that the visual information may have been discounted leading to the fish ignoring the objects even though they could have been perceived by the visual sense (15).

The results obtained after visual training of the fish show that the weighting of the sensory inputs can be adjusted through learning (Fig. 1A). The repetition and rewarding of the visual stimulus without electrical object input during visual training may have remapped the system so that the visual training was eventually successful. In experiments with humans and monkeys (44, 45) it is possible to adapt to discrepancies in sensory inputs, if they are consistent and occur over many repetitions (15).

At longer object distances, the reliability of the electric sense decreases rapidly due to its small working range (22, 46). Consequently, the dominance of the electric sense over vision decreases, and the conflict between vision and the electric sense is solved in favor of the visual information (Fig. 2). These results correspond well with findings in humans (10), monkeys (21), and rats (11) and suggest that dynamic weighting of sensory inputs is a fundamental process necessary for multisensory integration, and is conserved across vertebrates. However, it is not known yet how the inputs of multiple senses are weighted and integrated in the brain. Multisensory information could, for example, be processed in a centralized or decentralized manner. In a centralized system information received through all sensory systems would be fed into a single integration center, where the multiple inputs would be integrated (47, 48). In contrast, in a decentralized system the integration would be achieved through the interconnection of many multisensory areas (49).

Here we show, for the first time to our knowledge, that spontaneous cross-modal object recognition as well as dynamic weighting of sensory inputs exists not only in mammals but also in fish. This has important implications for our understanding of the mechanisms and the neuronal requirements underlying these functions, by revealing that the teleost brain, which is usually considered to be simple in relation to those of birds and mammals, is nonetheless capable of performing these complex cognitive tasks.

Methods

Ten naive G. petersii were individually housed in tanks, which also served as the experimental arenas (Fig. 4). Fish were trained in a two-alternative forced-choice procedure to swim through a gate with an object (Fig. S1) that was positively associated with a food reward (a chironomid insect larva) and to avoid a second gate with a negative object, which was associated with a mild punishment (fish being chased back to start position). The position of the positive object was changed behind the gates pseudorandomly after Gellermann (50). During training the objects were placed 1 cm behind the gates. To ensure that during training and testing the fish kept a certain minimal distance during object inspection, a distance grid was placed directly behind each gate (Fig. 4). Distance grids were made from thin cotton threads, with a mesh size of 15 mm (diagonal), which ensured unimpeded electrolocation through this mesh. The fish could pass the grids by pushing them aside. Although only the minimal distance of the fish to the objects was restricted and the fish could choose its distance to the gate freely, observations of the behavior during object inspection showed that the fish always inspected the objects from immediately behind the gates. This inspection behavior did not change throughout the tests.

Fig. 4.

Fig. 4.

Experimental set-up. G. petersii were individually housed in 75 cm × 40 cm × 40 cm tanks, which also served as the experimental arenas. (A) Schematic side view. (B) Schematic top view. These tanks were divided into two compartments (40 cm × 40 cm and 35 cm × 40 cm) by a partition with two closable gates behind which were positioned two objects (indicated in red) 1 cm from the gate. Distance grids, which were placed directly behind the gates and which could be passed by pushing them aside, made sure that the fish kept the correct minimal distance to the objects. The larger compartment (experimental area) was again divided into two compartments, one gate leading to each compartment.

The experiments were carried out in accordance with the guidelines of German law, with the animal welfare regulations of the University of Bonn, and with the “Guidelines for the treatment of animals in behavioural research and teaching,” Association for the Study of Animal Behaviour (51). All experiments, except the dark controls, were conducted at an ambient light level of 3–6 lx (measured just above the water surface), which lies in the optimal intensity range for visual object discrimination in G. petersii (52). The conductivity (95–110 µS/cm) and the temperature (25–27 °C) of the water were kept constant.

Fish were divided into two training groups, which were either able to use vision or the active electric sense for object discrimination. Access via these senses was controlled by manipulating the objects or the ambient surroundings. Objects used in the visually trained group were constructed of electrically transparent, red-colored agarose (Fig. S1 A and B). The conductivity of those objects was adjusted to the conductivity of the tank water (approximately 100 µS/cm); therefore they were “electrically invisible” to the fish (53) (Control Tests and Fig. S2). Red food color (Lebensmittelpaste Rot) was added to deionized water (conductivity <10 µS) until a conductivity of 40 µS/cm was reached. By adding agarose powder (Agarose BP 160–100; Fisher Scientific) (2 g per 100 mL) the conductivity was increased to approximately 100 µS/cm. This mixture was boiled and cast in molds. After cooling down, the agarose became stiff and the objects could be used in the discrimination experiments. Because it was not possible to measure the conductivity of the stiffened agarose directly, the resistance of 250 mL stiff agarose within a beaker was compared with the resistance of 250 mL tank water using a multimeter (M-3650B; Voltcraft) to test whether their electrical properties were identical. For both measurements the measuring electrodes were positioned 5 cm apart. There was no measureable difference between the agarose and the tank water. Control tests ensured that the fish could not use electrical cues to discriminate between the objects (Control Tests and Fig. S2). Red color was used because the cones of G. petersii are most sensitive to red light (absorption maximum: 615 nm) (26).

During training in the electrical group (group 2) aluminum objects were covered with hoods made of opaque, black cotton fabric to prevent influence of vision and the lateral line system on the discrimination performance (Fig. S1 CE). Because the cotton fabric was soaked with the tank water, the hoods themselves were electrically transparent. To ensure that the hoods had no influence on the discrimination performance, control tests were conducted (Fig. S3).

For all fish a sphere (Ø 3 cm) was used as the positive object (S+). In both groups, three fish were trained with a cross (width, 4 cm; height, 4 cm; depth, 1.7 cm) as the negative object and two fish with a cuboid (4 cm × 2.2 cm × 1.7 cm) as the negative object (S−). Because the volume and the material of S+ and all S− were the same, the fish could use only the shape to discriminate the objects.

After the preassigned learning criterion of at least 75% correct choices on 3 consecutive days was reached, test trials (see below), which were neither rewarded nor punished, were introduced every third trial. After 3–5 days, the training-to-test trial ratio was increased to 2:2. With each test condition, 30 trials were conducted with each fish. The number of training trials per fish ranged between 182 and 934.

Transfer Tests.

During the transfer tests the fish could use only the previously untrained sense for the discrimination task. The fish of the visually trained group were tested in the dark at 1-cm distance with aluminum objects that were encased in cubes of electrically transparent agarose so that the positive and the negative object had the same outer shape and were only electrically distinguishable excluding vision and the lateral line system. The electrically trained group was tested visually with the red colored, electrically transparent agarose objects placed 1 cm behind the gates.

Range Tests.

Four of the electrically trained fish were tested at different object distances (0.2 cm, 1–7 cm, and 9 cm) with only the active electric sense available (aluminum objects covered with black cotton hood), with the visual sense available (red-colored, electrically transparent agarose objects) and with both senses available. For the latter, red colored conductive agarose objects were used. These objects were produced in the same way as the electrically transparent agarose objects but instead of using deionized water a high conductive saline solution (>10 S/m) was used. The tests with both senses available were conducted last to ensure that they would not influence the results of the visual tests. Besides that, the tested sense and object distances were chosen pseudorandomly for each day.

Tests with “Electrically Silenced” Fish.

Four fish (fish 6 and 8–10) of the electrically trained group were electrically silenced by cutting the spinal cord just anterior to the electric organ located in the caudal peduncle. Before the actual operation a sham operation was conducted. The fish were narcotized in a 100 mg/L solution of MS 222 (Acros Organics), the operation site was locally anesthetized with Xylocain Gel (AstraZeneca; 22876), and the skin was penetrated with a dissecting needle. Afterward, the fish were tested electrically and visually with a 1-cm object distance as a control (Control Tests). For the real operation, fish were treated as in the sham operation described above but the dissecting needle was inserted dorsally into the vertebral canal to transect the spinal cord. After the fish were electrically silenced, visual tests were conducted with the red-colored electric neutral agarose objects at 1-cm distance. Because the electric input was missing for training trials, every third trial was rewarded independently of the choice made by the fish to maintain motivation.

Acknowledgments

We thank Horst Bleckmann, Marian Dawkins, Adrian Thomas, Vera Schlüssel, Helmut Schmitz, and Graham Taylor for discussion and helpful comments on the manuscript. This study was supported by the German Research Foundation (DFG Em43/17-1).

Footnotes

The authors declare no conflict of interest.

This article is a PNAS Direct Submission.

This article contains supporting information online at www.pnas.org/lookup/suppl/doi:10.1073/pnas.1603120113/-/DCSupplemental.

References

  • 1.Yehle AL, Ward JP. Cross-modal transfer of a specific discrimination in the rabbit. Psychon Sci. 1969;16(5):269–270. [Google Scholar]
  • 2.Seraganian P, Popova YI. Cross modal transfer of a conditional flexion response in dogs. Pavlov J Biol Sci. 1976;11(3):162–174. doi: 10.1007/BF03000293. [DOI] [PubMed] [Google Scholar]
  • 3.Proops L, McComb K, Reby D. Cross-modal individual recognition in domestic horses (Equus caballus) Proc Natl Acad Sci USA. 2009;106(3):947–951. doi: 10.1073/pnas.0809127105. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4.Guo J, Guo A. Crossmodal interactions between olfactory and visual learning in Drosophila. Science. 2005;309(5732):307–310. doi: 10.1126/science.1111280. [DOI] [PubMed] [Google Scholar]
  • 5.Gaydos HF. Intersensory transfer in the discrimination of form. Am J Psychol. 1956;69(1):107–110. [PubMed] [Google Scholar]
  • 6.Davenport RK, Rogers CM. Intermodal equivalence of stimuli in apes. Science. 1970;168(3928):279–280. doi: 10.1126/science.168.3928.279. [DOI] [PubMed] [Google Scholar]
  • 7.Cowey A, Weiskrantz L. Demonstration of cross-modal matching in rhesus monkeys, Macaca mulatta. Neuropsychologia. 1975;13(1):117–120. doi: 10.1016/0028-3932(75)90057-3. [DOI] [PubMed] [Google Scholar]
  • 8.Herman LM, Pack AA, Hoffmann-Kuhnt M. Seeing through sound: Dolphins (Tursiops truncatus) perceive the spatial structure of objects through echolocation. J Comp Psychol. 1998;112(3):292–305. doi: 10.1037/0735-7036.112.3.292. [DOI] [PubMed] [Google Scholar]
  • 9.Winters BD, Reid JM. A distributed cortical representation underlies crossmodal object recognition in rats. J Neurosci. 2010;30(18):6253–6261. doi: 10.1523/JNEUROSCI.6073-09.2010. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10.Ernst MO, Banks MS. Humans integrate visual and haptic information in a statistically optimal fashion. Nature. 2002;415(6870):429–433. doi: 10.1038/415429a. [DOI] [PubMed] [Google Scholar]
  • 11.Sheppard JP, Raposo D, Churchland AK. Dynamic weighting of multisensory stimuli shapes decision-making in rats and humans. J Vis. 2013;13(6):4. doi: 10.1167/13.6.4. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12.Young MJ, Landy MS, Maloney LT. A perturbation analysis of depth perception from combinations of texture and motion cues. Vision Res. 1993;33(18):2685–2696. doi: 10.1016/0042-6989(93)90228-o. [DOI] [PubMed] [Google Scholar]
  • 13.Alais D, Burr D. The ventriloquist effect results from near-optimal bimodal integration. Curr Biol. 2004;14(3):257–262. doi: 10.1016/j.cub.2004.01.029. [DOI] [PubMed] [Google Scholar]
  • 14.Ernst MO, Bülthoff HH. Merging the senses into a robust percept. Trends Cogn Sci. 2004;8(4):162–169. doi: 10.1016/j.tics.2004.02.002. [DOI] [PubMed] [Google Scholar]
  • 15.Ernst MO, Di Luca M. Multisensory perception: From integration to remapping. In: Tommershauser KPKJ, Landy MS, editors. Sensory Cue Integration. Oxford Univ Press; New York: 2011. pp. 224–250. [Google Scholar]
  • 16.Howard IP, Templeton WB. Human Spatial Orientation. Wiley; New York: 1966. [Google Scholar]
  • 17.Rock I, Victor J. Vision and touch: An experimentally created conflict between the two senses. Science. 1964;143(3606):594–596. doi: 10.1126/science.143.3606.594. [DOI] [PubMed] [Google Scholar]
  • 18.Shams L, Kamitani Y, Shimojo S. Illusions. What you see is what you hear. Nature. 2000;408(6814):788. doi: 10.1038/35048669. [DOI] [PubMed] [Google Scholar]
  • 19.Shams L, Kamitani Y, Shimojo S. Visual illusion induced by sound. Brain Res Cogn Brain Res. 2002;14(1):147–152. doi: 10.1016/s0926-6410(02)00069-1. [DOI] [PubMed] [Google Scholar]
  • 20.Alais D, Newell FN, Mamassian P. Multisensory processing in review: From physiology to behaviour. Seeing Perceiving. 2010;23(1):3–38. doi: 10.1163/187847510X488603. [DOI] [PubMed] [Google Scholar]
  • 21.Fetsch CR, Turner AH, DeAngelis GC, Angelaki DE. Dynamic reweighting of visual and vestibular cues during self-motion perception. J Neurosci. 2009;29(49):15601–15612. doi: 10.1523/JNEUROSCI.2574-09.2009. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22.von der Emde G, et al. 3-Dimensional scene perception during active electrolocation in a weakly electric pulse fish. Front Behav Neurosci. 2010;4:26. doi: 10.3389/fnbeh.2010.00026. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23.Rasnow B. The effects of simple objects on the electric field of Apteronotus. J Comp Physiol A Neuroethol Sens Neural Behav Physiol. 1996;178(3):397–411. [Google Scholar]
  • 24.Caputi AA, Budelli R, Grant K, Bell CC. The electric image in weakly electric fish: Physical images of resistive objects in Gnathonemus petersii. J Exp Biol. 1998;201(Pt 14):2115–2128. doi: 10.1242/jeb.201.14.2115. [DOI] [PubMed] [Google Scholar]
  • 25.Jorgensen JM. Morphology of electroreceptive sensory organs. In: Bullock TH, Hopkins CD, Popper AN, Faye RR, editors. Electroreception. Springer; New York: 2005. pp. 47–67. [Google Scholar]
  • 26.Kreysing M, et al. Photonic crystal light collectors in fish retina improve vision in turbid water. Science. 2012;336(6089):1700–1703. doi: 10.1126/science.1218072. [DOI] [PubMed] [Google Scholar]
  • 27.Moller P. Multimodal sensory integration in weakly electric fish: A behavioral account. J Physiol Paris. 2002;96(5-6):547–556. doi: 10.1016/S0928-4257(03)00010-X. [DOI] [PubMed] [Google Scholar]
  • 28.Rojas R, Moller P. Multisensory contributions to the shelter-seeking behavior of a mormyrid fish, Gnathonemus petersii Günther (Mormyridae, Teleostei): The role of vision, and the passive and active electrosenses. Brain Behav Evol. 2002;59(4):211–221. doi: 10.1159/000064908. [DOI] [PubMed] [Google Scholar]
  • 29.Streri A. Cross-modal recognition of shape from hand to eyes in human newborns. Somatosens Mot Res. 2003;20(1):13–18. doi: 10.1080/0899022031000083799. [DOI] [PubMed] [Google Scholar]
  • 30.Streri A, Gentaz E. Cross-modal recognition of shape from hand to eyes and handedness in human newborns. Neuropsychologia. 2004;42(10):1365–1369. doi: 10.1016/j.neuropsychologia.2004.02.012. [DOI] [PubMed] [Google Scholar]
  • 31.Sann C, Streri A. Perception of object shape and texture in human newborns: Evidence from cross-modal transfer tasks. Dev Sci. 2007;10(3):399–410. doi: 10.1111/j.1467-7687.2007.00593.x. [DOI] [PubMed] [Google Scholar]
  • 32.Ettlinger G, Wilson WA. Cross-modal performance: Behavioural processes, phylogenetic considerations and neural mechanisms. Behav Brain Res. 1990;40(3):169–192. doi: 10.1016/0166-4328(90)90075-p. [DOI] [PubMed] [Google Scholar]
  • 33.Lacey S, Campbell C, Sathian K. Vision and touch: Multiple or multisensory representations of objects? Perception. 2007;36(10):1513–1521. doi: 10.1068/p5850. [DOI] [PubMed] [Google Scholar]
  • 34.Trinh AT, Harvey-Girard E, Teixeira F, Maler L. Cryptic laminar and columnar organization in the dorsolateral pallium of a weakly electric fish. J Comp Neurol. 2016;524(2):408–428. doi: 10.1002/cne.23874. [DOI] [PubMed] [Google Scholar]
  • 35.Prechtl JC, et al. Sensory processing in the pallium of a mormyrid fish. J Neurosci. 1998;18(18):7381–7393. doi: 10.1523/JNEUROSCI.18-18-07381.1998. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 36.Portavella M, Vargas JP, Torres B, Salas C. The effects of telencephalic pallial lesions on spatial, temporal, and emotional learning in goldfish. Brain Res Bull. 2002;57(3-4):397–399. doi: 10.1016/s0361-9230(01)00699-2. [DOI] [PubMed] [Google Scholar]
  • 37.Rodríguez F, et al. Spatial memory and hippocampal pallium through vertebrate evolution: Insights from reptiles and teleost fish. Brain Res Bull. 2002;57(3-4):499–503. doi: 10.1016/s0361-9230(01)00682-7. [DOI] [PubMed] [Google Scholar]
  • 38.Broglio C, et al. Hallmarks of a common forebrain vertebrate plan: Specialized pallial areas for spatial, temporal and emotional memory in actinopterygian fish. Brain Res Bull. 2005;66(4-6):277–281. doi: 10.1016/j.brainresbull.2005.03.021. [DOI] [PubMed] [Google Scholar]
  • 39.Pusch R, Wagner HJ, von der Emde G, Engelmann J. Spatial resolution of an eye containing a grouped retina: Ganglion cell morphology and tectal physiology in the weakly electric fish Gnathonemus petersii. J Comp Neurol. 2013;521(17):4075–4093. doi: 10.1002/cne.23397. [DOI] [PubMed] [Google Scholar]
  • 40.Francke M, et al. Grouped retinae and tapetal cups in some Teleostian fish: Occurrence, structure, and function. Prog Retin Eye Res. 2014;38:43–69. doi: 10.1016/j.preteyeres.2013.10.001. [DOI] [PubMed] [Google Scholar]
  • 41.von der Emde G. Non-visual environmental imaging and object detection through active electrolocation in weakly electric fish. J Comp Physiol A Neuroethol Sens Neural Behav Physiol. 2006;192(6):601–612. doi: 10.1007/s00359-006-0096-7. [DOI] [PubMed] [Google Scholar]
  • 42.von der Emde G, Ronacher B. Perception of electric properties of objects in electrolocating weakly electric fish: Two-dimensional similarity scaling reveals a City-Block metric. J Comp Physiol A Neuroethol Sens Neural Behav Physiol. 1994;175(6):801–812. [Google Scholar]
  • 43.Hopkins C. Electrical perception and communication. In: Squire L, editor. Encyclopedia of Neuroscience. Vol. 3. Academic Press; Oxford: 2009. pp. 813–831. [Google Scholar]
  • 44.Adams WJ, Banks MS, van Ee R. Adaptation to three-dimensional distortions in human vision. Nat Neurosci. 2001;4(11):1063–1064. doi: 10.1038/nn729. [DOI] [PubMed] [Google Scholar]
  • 45.Zaidel A, Turner AH, Angelaki DE. Multisensory calibration is independent of cue reliability. J Neurosci. 2011;31(39):13949–13962. doi: 10.1523/JNEUROSCI.2732-11.2011. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 46.Pedraja F, Aguilera P, Caputi AA, Budelli R. Electric imaging through evolution, a modeling study of commonalities and differences. Plos Comput Biol. 2014;10(7):e1003722. doi: 10.1371/journal.pcbi.1003722. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 47.Magosso E, Cuppini C, Serino A, Di Pellegrino G, Ursino M. A theoretical study of multisensory integration in the superior colliculus by a neural network model. Neural Netw. 2008;21(6):817–829. doi: 10.1016/j.neunet.2008.06.003. [DOI] [PubMed] [Google Scholar]
  • 48.Ursino M, Cuppini C, Magosso E, Serino A, di Pellegrino G. Multisensory integration in the superior colliculus: A neural network model. J Comput Neurosci. 2009;26(1):55–73. doi: 10.1007/s10827-008-0096-4. [DOI] [PubMed] [Google Scholar]
  • 49.Zhang WH, Chen A, Rasch MJ, Wu S. Decentralized multisensory information integration in neural systems. J Neurosci. 2016;36(2):532–547. doi: 10.1523/JNEUROSCI.0578-15.2016. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 50.Gellermann LW. Chance orders of alternating stimuli in visual discrimination experiments. Pedagog Semin J Genet Psychol. 1933;42(1):206–208. [Google Scholar]
  • 51. Association for the Study of Animal Behaviour (2006) Guidelines for the treatment of animals in behavioural research and teaching. Anim Behav 71:245–253. [DOI] [PubMed]
  • 52.Schuster S, Amtsfeld S. Template-matching describes visual pattern-recognition tasks in the weakly electric fish Gnathonemus petersii. J Exp Biol. 2002;205(Pt 4):549–557. doi: 10.1242/jeb.205.4.549. [DOI] [PubMed] [Google Scholar]
  • 53.Heiligenberg W. Electrolocation of objects in the electric fish Eigenmannia (Rhamphichthyidae, Gymnotoidei) J Comp Physiol. 1973;87(2):137–164. [Google Scholar]
  • 54.Ciali S, Gordon J, Moller P. Spectral sensitivity of the weakly discharging electric fish Gnathonemus petersi using its electric organ discharges as the response measure. J Fish Biol. 1997;50(5):1074–1087. [Google Scholar]

Articles from Proceedings of the National Academy of Sciences of the United States of America are provided here courtesy of National Academy of Sciences

RESOURCES