Cognitive ability is heritable and predicts the success of an alternative mating tactic

Abstract

The ability to attract mates, acquire resources for reproduction, and successfully outcompete rivals for fertilizations may make demands on cognitive traits—the mechanisms by which an animal acquires, processes, stores and acts upon information from its environment. Consequently, cognitive traits potentially undergo sexual selection in some mating systems. We investigated the role of cognitive traits on the reproductive performance of male rose bitterling (Rhodeus ocellatus), a freshwater fish with a complex mating system and alternative mating tactics. We quantified the learning accuracy of males and females in a spatial learning task and scored them for learning accuracy. Males were subsequently allowed to play the roles of a guarder and a sneaker in competitive mating trials, with reproductive success measured using paternity analysis. We detected a significant interaction between male mating role and learning accuracy on reproductive success, with the best-performing males in maze trials showing greater reproductive success in a sneaker role than as a guarder. Using a cross-classified breeding design, learning accuracy was demonstrated to be heritable, with significant additive maternal and paternal effects. Our results imply that male cognitive traits may undergo intra-sexual selection.

1. Introduction

Cognitive traits, the neuronal processes concerned with acquiring, processing, retaining and using information [1–3], equip an individual with the capacity to solve novel tasks. Cognitive traits may be favoured under natural selection if they influence individual fitness, for example through enhanced foraging ability or food caching [1]. Cognitive ability may also potentially undergo sexual selection [4,5]. Sexual selection is selection that acts on differences in fitness among individuals due to the number and identity of their mates, and is a powerful form of selection that shapes mating system evolution [6]. Hitherto, sexual selection has not been considered a major driving force in the evolution of cognition [1]. However, cognitive traits have the potential to influence mate finding, mate choice or success in fertilization, depending on the selective forces generated by a particular mating system. Variance in male reproductive success is typically greater than in females, and often results in the evolution of elaborate phenotypic traits in males. In the case that cognitive traits undergo sexual selection, a prediction is that selection for cognitive ability will be stronger in males than females, in the same way that other traits, such as nuptial coloration, weapons or alternative mating behaviours, are typically more strongly expressed in males than in females. For selection to act on cognitive traits, there must also be heritable variation with consequences for fitness [3,4,7].

Some recent studies have implicated a role for male cognitive abilities in the capacity to attract mates, successfully father offspring and efficiently perform parental care, particularly in birds [8–12]. In Drosophila melanogaster, it was shown that male cognitive traits were eroded under relaxed sexual selection, demonstrating a potentially significant role of cognition in the mating system of this species [13]. The results of this study imply a heritable basis to cognitive traits, and that their expression can carry a fitness cost in some circumstances. However, little emphasis has so far been placed on several key aspects of cognition, namely understanding how cognitive traits evolve, how they are associated with reproductive success and what trade-offs constrain their evolution.

Here, we investigated the role of cognitive ability in the mating system of rose bitterling (Rhodeus ocellatus), freshwater fish that spawn their eggs in the gills of living mussels. Dominant males aggressively defend territories to monopolize mussels and perform elaborate courtship towards females [14,15]. Females place their eggs into the gills of a mussel through its exhalant siphon. Males fertilize the eggs by ejaculating over the inhalant siphon of the mussel, with water filtered by the mussel carrying the sperm to the eggs. Pre-oviposition ejaculations, whereby males release sperm over a mussel before a female spawns, are common. Bitterling spermatozoa remain viable within a mussel gill for an unusually prolonged period and are capable of fertilizing eggs at least 14 min after ejaculation [16,17]. The risk of sperm competition in bitterling is high [18]. Males that control access to mussels enjoy high reproductive success, and male dominance is determined by body size [19,20], with smaller, subordinate males adopting alternative mating tactics. The dominant ‘guarder’ and subordinate ‘sneaker’ mating tactics are not fixed, and male mating behaviour is opportunistic, with males capable of playing either role depending on environmental context [21,22], which is the typical situation in teleost fishes [15]. The sneaker tactic commonly involves pre-oviposition ejaculations [14,15,17], with male reproductive success determined by how males distribute their ejaculates among mussels in relation to the distribution of spawning females, the spatial distribution of mussels and the distribution of rival ejaculates [17], while minimizing the risk of sperm depletion [18].

The bitterling mating system therefore would appear to favour males that can match their mating role to the local environment, and that have the spatial cognitive ability to distribute their ejaculates optimally among mussels to maximize their reproductive success [17]. In particular, sexual selection for spatial cognition would be predicted to operate more strongly on males in a sneaker role, as guarder males typically enjoy higher fertilization success than sneakers, with sneaker success linked more directly to an ability to anticipate spawning events with pre-oviposition ejaculations. Here, we examine the potential role of cognitive traits in mating system evolution by testing the predictions that: (1) performance in a spatial learning task predicts male reproductive success; (2) spatial cognitive ability favours males performing alternative mating tactics; and (3) performance in a spatial learning task is heritable.

2. Material and methods

(a). Study system

Experimental R. ocellatus used in trials were the second generation descended from 200 wild-caught fish from the River Yangtze basin, China. During the experiment, they were 18 months old. Prior to experiments, fish were housed in stock aquaria measuring 600 (length) × 300 (width) × 400 (depth) mm. For learning trials, groups of 16 randomly selected males and females were taken from stock aquaria. The same 16 males were subsequently used in competitive mating trials to assess the reproductive success of males of known performance in spatial learning ability. To quantify the heritability of spatial cognition, males and females from the same stock as the learning and mating trials were used, though not the same individuals. Stock and holding aquaria, as well as maze and mating trial aquaria, were all housed in an environmentally controlled room with a 16 L : 8 D cycle at 23°C. Fish were fed a mixture of commercial dried fish flake food and bloodworm (Chironomus spp.) twice daily.

(b). Learning accuracy

Learning accuracy was quantified using maze apparatus for fish [19,20], comprising a square plastic box with opaque walls measuring 500 (length) × 500 (width) × 300 (depth) mm. A central chamber (300 × 300 mm) with opaque walls was connected to four outer chambers by separate 50 mm wide openings (electronic supplementary material, figure S1a). The walls of each outer compartment were a different colour—red, blue, green or black—to act as clear landmarks for navigation. The choice of experimental colours was based on cyprinid colour vision, which is tetrachromatic [21]. Each outer chamber had a 40 mm diameter Petri dish placed at its farthest corner. A webcam, connected to a laptop computer, was suspended directly above the maze so that a fish in the maze could be observed remotely without disturbance.

Test fish were individually housed in holding aquaria measuring 300 (length) × 200 (width) × 220 (depth) mm. Each fish was randomly assigned to the red, blue, green or black chamber as a reward chamber. To measure learning accuracy, each fish was given a single familiarity trial before testing. A food reward of five to eight live whiteworms (Enchytraeus spp.) was placed in the Petri dish in the test chamber to which the test fish was assigned. To control for the effect of olfactory cues, water in which whiteworms were stored, and infused with their odour, was pipetted into all test chambers immediately prior to each test. The test fish were gently transferred to a clear plastic release cylinder in the central compartment of the maze and allowed at least 2 min to settle. The cylinder was then raised remotely, freeing the fish and enabling it to explore the maze. All fish located and ate the food reward within 2 h of release.

On the day following their familiarity trial, the fish was returned to the release cylinder for the start of the trial proper. A record was kept of the frequency with which the fish made an error and entered a chamber without a food reward, scored as occasions when the fish passed at least halfway through the chamber entrance. After 10 min, if the fish had not located the food reward, it was gently guided into the rewarded chamber with a hand net and allowed to feed. After feeding, fish were transferred back to their holding aquaria. Fish were not fed prior to testing and so were motivated to locate the food reward. Every fish was tested once each day for 7 days, with the total number of errors over this period summed as a learning accuracy score. After completion of seven trials, fish were measured for standard length (SL; tip of the snout to the base of the tail fin).

(c). Emergence trials

To control for the potentially confounding effects of variation in ‘shyness–boldness’ sensu [23], a behavioural assay was performed. The assay measured time to emerge from a refuge (for simplicity, hereafter referred to as ‘emergence time’), estimated as the time taken for an individual to emerge from shelter in a novel environment [23].

Tests followed an established protocol [24] and were conducted in glass aquaria measuring 300 (length) × 200 (width) × 200 (depth) mm. Test aquaria had a gravel substrate and water to a depth of 150 mm. The aquarium was bisected with a sliding opaque partition placed 100 mm from one end. Artificial plants that reached the water surface were placed behind the screen to provide a refuge area of dense submerged vegetation. The remaining 200 mm section of the aquarium was bare of cover (electronic supplementary material, figure S1b). Fish were gently released into the vegetated end of the test aquarium. After a 5 min acclimation period, the partition was raised, allowing the fish to explore the whole aquarium. An observer recorded the time taken for the fish to emerge a full body length from the vegetated end into the open part of the aquarium. A fish that failed to emerge within 10 min was assigned a score of 600 s. After completion of a trial, the test fish was returned to its holding aquarium. On the following day, the fish was retested using the same protocol. Repeatability was high (r = 0.61). The mean of the two emergence scores was used as an index of shyness–boldness for that individual. After completion of trials, males were retained singly in their holding aquaria for testing in competitive mating trials. Females were returned to stock aquaria and were not used further.

(d). Competitive mating trials

The reproductive success of the 16 males used in learning and emergence trials was measured in mating trials by permitting them to compete with rivals for fertilizations, acting in the role of both a guarder and a sneaker. Rivals were from the same stock of fish, but played no other role in the study. Trials were performed in an aquarium measuring 1250 (width) × 300 (length) × 300 (depth) mm. Two size-matched Unio pictorum mussels were placed in sand-filled cups and situated at each end of the aquarium. The aquarium had a sand substrate and was furnished with 20 artificial plants, distributed haphazardly, to add environmental heterogeneity.

The focal male was gently released into the aquarium and randomly assigned either a guarder or a sneaker role. In the case where the male was to play the role of a guarder, a rival male was released into the test aquarium that was 20% (by SL) smaller than the focal male. If the male was to play the role of a sneaker, the rival was 20% larger. Male mating role in R. ocellatus is determined by relative body size [25]. The two males were left for 24 h to establish dominance roles. In every case, the larger male played the role of guarder and the smaller individual acted as a sneaker during matings. After 24 h, a female in reproductive condition (with an extended ovipositor) was selected from a stock tank and gently released into the aquarium. After 1 h (which is sufficient time for repeated spawning acts), the female was captured and measured, and a small portion of the tail fin was removed and fixed in 95% ethanol. A second female was then released and the process repeated.

After completion of a trial, a finclip was taken from the focal and rival male, and fixed in ethanol for paternity analysis. The focal male was returned to his holding aquarium. The total length of the mussels was measured and their gills checked for eggs. If eggs were present, the mussel was dissected and the eggs allowed to develop in a water-filled 70 mm diameter Petri dish in an incubator at 23°C for 5 days. After 5 days, the embryos were fixed in ethanol for parentage analysis. Rival males and females were released in stock tanks and were not used again in trials.

Focal males were again tested a minimum of 2 days after the first trial (mean ± s.d. of 2.8 ± 1.0 days), but in the opposite role to the one they played in their first trial. Thus, focal males that had played a guarder role subsequently played the role of a sneaker, and vice versa, with the order in which they played these roles randomized, and using a new rival male and pair of females. Finclips were collected from the rival male and both females, and fertilized eggs were incubated and subsequently fixed in ethanol. It was not necessary to finclip focal males again. At completion of trials, all 16 focal males had engaged in competitive matings with a rival in both a guarder and sneaker role. The study generated a total of 439 fertilized embryos. Of these, a total of 416 embryos (95%) survived 5 days to fixation in ethanol, with a mean (±s.e.) of 13.0 (±1.2) embryos per trial.

(e). Parentage analysis

For parentage analysis, DNA was extracted from ethanol-preserved tissue using established methods [26]. A set of eight microsatellite loci [26,27] was chosen on the basis of their variability and informative value, and combined in two multiplex PCR reactions, with a mean of 13 (range: 6–23) alleles per locus. The length of the DNA fragments was analysed using GeneMapper software. DNA was successfully extracted and analysed for a total of 408 embryos. Of these, paternity was assigned with 95% confidence for 364 embryos in Cervus v. 3.0 (error rate set to 0.01) [28]. In one replicate, only three eggs were recovered and data for this replicate were excluded from the subsequent analysis. All other embryos were included in analyses, with a mean of 12 (range: 5–25) embryos per replicate. For one male, a finclip from a rival male was not properly fixed when the focal male played a guarder role. In this case, paternity could only be estimated with 95% certainty for the male in a sneaker role but not as a guarder. Because of the paired nature of our subsequent analysis, this replicate was excluded from the dataset.

(f). Heritability: in vitro fertilizations

The heritability of learning accuracy was measured using a North Carolina Type II breeding design using in vitro fertilizations to generate a series of replicated half-sib families [29]. Eight blocks, each with a set of 2 × 2, male × female, factorial crosses were conducted using fish from the same stock. Within each block, both males were crossed with both females, with a replicate of each cross. This design generated two replicates of four families of maternal and paternal half-siblings, in each of eight blocks, with a total of 64 replicated families in the final combined design. A comparable design was used successfully in previous heritability studies with R. ocellatus [30], and permits the relative contribution of additive and non-additive genetic effects for a trait of interest to be measured, and to identify maternal and paternal contributions to additive genetic variance.

To generate crosses, experimental females were isolated until they ovulated a batch of eggs (obvious from the female's extended ovipositor). The eggs were gently stripped from the female and divided into approximately two equal groups in separate 70 mm diameter Petri dishes containing freshwater (mean = 8.0 ± 3.1 s.d. eggs per group). Sperm was stripped from the two experimental males by gently pressing their abdomens and mixed in 9 ml of teleost saline [31]. A 1 ml subsample of this sperm solution was diluted with a further 9 ml of saline. Sperm suspensions were pipetted over the eggs and the covered Petri dishes were left on the laboratory bench for 30 min. The fertilized eggs were washed in freshwater and the long axis of every egg measured under a binocular microscope (Nikon Eclipse E200) with an eyepiece micrometer to the nearest 0.1 mm. They were subsequently incubated at 23°C until the yolk sac was absorbed and the fish began exogenous feeding, a period of approximately 30 days. A daily record was kept of embryo survival. After the onset of exogenous feeding, fish were transferred in family groups to aquaria measuring 300 (length) × 200 (width) × 210 (depth) mm and fed twice daily on formulated zebrafish granules, supplemented with live Artemia.

(g). Heritability: offspring learning accuracy

Offspring learning accuracy was assayed after approximately 12 weeks (mean = 86.3 ± 2.9 s.d. days) with a simplified version of the learning trials used for adults. A simplified design was used to facilitate screening of a large number of fish. Mean (±s.e.) offspring SL at this age was 21.8 (±1.9) mm. A single fish was tested from each family generated from factorial crosses, with 64 fish tested in total. Fish were sexually immature and were selected randomly from each family.

Learning accuracy was measured in a series of dichotomous choice chambers. These comprised a glass aquarium measuring 300 (length) × 200 (width) × 220 (depth) mm containing 7 l of fresh water. Halfway along the aquarium, there was a sliding partition that retained the test fish in the rear portion of the aquarium (electronic supplementary material, figure S1c). The front of the aquarium was partitioned into two 70 × 100 mm choice chambers, with 40 mm openings at each side. A single 30 mm diameter Petri dish was situated immediately inside each choice chamber, such that they could not be seen from outside the chamber. A red or blue plastic marker, measuring 5 × 5 mm, was attached to the front of the aquarium, so that it was visible to the test fish from the rear of the aquarium as a landmark for navigation. The side of the aquarium on which the blue or red markers were attached was randomized. Test fish were randomly assigned to either the red or blue chamber as a reward chamber.

To minimize isolation stress while confined at the rear of the aquarium, fish were able to see neighbours in adjacent aquaria. However, the front half of aquaria were screened from neighbours with an opaque barrier. Thus, when making the decision to enter the test chambers at the front of the test aquarium, the fish were visually isolated from their neighbours. This conformation ensured fish were not visually isolated between trials, but could not be influenced by the behaviour of neighbours during trials.

To measure learning accuracy, fish were initially given a familiarity trial. Test fish were introduced to the test chamber on the day prior to the start of trials, but not fed. The following day, a food reward of five to eight live whiteworms was placed in the Petri dish in the test chamber to which the test fish was assigned (red or blue). To control for the effect of potential olfactory cues, water in which whiteworms were stored, and infused with odour, was pipetted into both chambers. The central partition was raised, and the fish were allowed to explore both chambers and feed on the whiteworms. All fish located and ate the food reward within 2 h of release.

On the day following the familiarity trial, the fish were confined behind the central partition. Whiteworms were replaced in the Petri dish and the partition was removed. A record was kept of the frequency with which the fish made an error and entered a chamber without a food reward, scored as occasions when the fish passed at least halfway through the chamber entrance. If after 10 min the fish had not located the food reward, it was gently guided into the rewarded chamber with a hand net and allowed to feed. After feeding, fish were confined at the back of the aquarium, behind the partition. Fish were not fed prior to testing and so were motivated to locate the food reward. Every fish was tested once each day for 6 days, with the total number of errors over this period summed as a learning accuracy score.

(h). Statistical analysis

Before applying statistical models, a data exploration was undertaken following the protocol described in [32]. The data were examined for outliers in the response and explanatory variables, homogeneity and zero inflation in the response variable, collinearity between explanatory variables, and the nature of relationships between the response and explanatory variables. Data analyses were performed using R [33].

Sex difference in learning accuracy was modelled using a generalized linear model (GLM) with log-link function to preclude negative fitted values. Assuming estimates of learning accuracy (accuracy_i) were Poisson distributed with mean μ_i, the model contained a linear effect for fish length (sl), emergence time (emg) and sex (fSex) as main terms, and took the form:

Male reproductive success from competitive mating trials was modelled using a binomial generalized linear mixed model (GLMM) with an observation-level random intercept. Male length (SL) was collinear with emergence time, and rival SL was collinear with male mating role. Consequently, emergence time and rival SL were dropped from the analysis [32]. The model took the form:

where success_ij is the reproductive success in the jth competitive mating trial for male i out of the N_ij trials conducted. π_ij is the probability of successful fertilization of eggs by male i in the jth competitive mating trial. The model contained a linear effect for male SL (sl) and mussel length (mussel) as main terms, and an interaction between male mating role (fRole) and learning accuracy (accuracy). The random intercept m_i was included to introduce a correlation structure between observations for the same male. ɛ_ij is an observation-level random effect to accommodate overdispersion in the data [32,34]. Continuous covariates were standardized to enhance numerical optimization of the model [32]. In the model, the interaction between male role and learning accuracy measured the contribution of learning accuracy to reproductive success in the roles of guarder and sneaker. As an additional measure of this effect, the difference in reproductive success of males playing a sneaker compared with a guarder role was correlated with learning accuracy.

The colour of the test chamber in learning trials, and the order in which males played either a sneaker or guarder role in competitive mating trials, made no significant contribution to models and were dropped from analyses.

In the analysis of the heritability of learning accuracy, two-way ANCOVA was used for each 2 × 2 factorial block to compare effects of sire, dam and their interaction on learning accuracy at 12 weeks. Under this design, the kth offspring phenotype from cross i × j takes the form

where μ is the mean population phenotype, and s_i and d_j are the additive effects on phenotype from the ith male (sire effect) and jth female (dam effect), respectively. I_ij is the non-additive sire × dam interaction, and e_ijk is the deviation of observed phenotype of the kth offspring of male i and female j from model predictions, and comprises phenotypic variance resulting from segregation, dominance and environment [29]. The model assumes within-family variance is uncorrelated with among-family variance, with total phenotypic variance the sum of sire, dam, interaction and error variance:

Sums of squares were combined to calculate mean squares and degrees of freedom for all families combined [29]. Because the amount of egg yolk can significantly affect offspring fitness [15,35], egg size was included as a covariate in the analysis as a maternal effect. Narrow-sense heritability (h²) was estimated as V_A/V_P, where V_A is additive genetic variance and V_P total phenotypic variance [29].

3. Results

(a). Sex differences in learning accuracy

Males showed significantly better learning accuracy in maze trials than females (table 1). Mean (±s.e.) learning accuracy score for males was 8.6 (±0.74) errors and for females 11.0 (±0.84). There was no significant effect of fish SL or emergence time (table 1) on learning accuracy, and no sex difference in body size (t₃₀ = 1.47, p = 0.153) or emergence time (t₃₀ = 1.30, p = 0.202).

Table 1.

Summary of the GLM for Poisson-distributed data to examine sex differences in learning accuracy in rose bitterling (R. ocellatus). n_obs = 32.

model parameter	estimate	s.e.	z	p
intercept	1.510	0.591	2.55	0.011
sl	0.014	0.010	1.36	0.171
emg	−0.001	0.001	−0.32	0.750
sex(female)	0.289	0.127	2.27	0.023

(b). Male reproductive success and learning accuracy

There was a significant interaction between male mating role and learning accuracy in predicting reproductive success in competitive mating trials (table 2). When males played a sneaker role, learning accuracy predicted mating success, but not as a guarder (figure 1). There was a significant correlation between the difference in reproductive success in the sneaker role compared with territorial (sneaker − territorial) and learning accuracy (t₁₃ = 2.68, p = 0.019). Overall, male reproductive success was significantly higher in a dominant guarder role than as a subordinate sneaker (table 2), though some males performed better in the role of sneaker than as a guarder (figure 2). Mean mussel size weakly predicted male reproductive success (table 2). There was no significant contribution of male SL to reproductive success (table 2).

Table 2.

Summary of the GLMM for binomial distributed data to examine mating role differences in the reproductive success of male rose bitterling (R. ocellatus). Individual males were fitted as random intercepts, with standard deviation of 0.75. An observation-level random intercept was included in the model with standard deviation of 0.43. n_obs = 30.

model parameter	estimate	s.e.	z	p
intercept	−0.223	0.297	−0.77	0.441
role(guarder)	1.054	0.327	3.23	0.001
accuracy	−0.739	0.372	−1.97	0.047
sl	0.560	0.307	1.82	0.068
mussel	0.434	0.200	2.17	0.030
role(guarder) × accuracy	0.754	0.335	2.25	0.024

Figure 1.

Fitted values for male reproductive success against standardized learning accuracy scores for males playing a guarder and sneaker role in competitive mating trials modelled using a binomial GLMM. Grey bands indicate 95% confidence intervals around the fitted line. Black circles are observed values for male reproductive success. Note that a low standardized learning score indicated completion of the maze task with few errors.

Figure 2.

Observed reproductive success of males playing a guarder and sneaker role in competitive mating trials. Linked black circles are the same individual, open circles represent mating role means (±s.e.).

(c). Heritability of learning accuracy

Learning accuracy showed significant additive male and female effects (table 3). There was no male × female interaction on learning accuracy. Narrow-sense heritability of learning accuracy was estimated as h² = 0.27.

Table 3.

ANCOVA for rose bitterling (R. ocellatus) offspring learning accuracy for in vitro fertilizations.

source	d.f.	SS	MS	F	p	variance	%
egg size	8	5417	677.1	1.88	0.118	39.6	8
female (F)	8	6741	842.6	3.74	0.040	30.2	6
male (M)	8	10 964	1370.4	6.09	0.010	63.1	13
F × M	8	1801	225.2	0.63	0.747	0	0
error	21	7561	360.1			360.1	73

4. Discussion

Darwin was the first to recognize that cognitive traits potentially undergo sexual selection [6]. Our results demonstrated a link between male performance in a spatial task and reproductive success, which depended on mating context. Accuracy of learning predicted the reproductive success of males adopting sneaky mating tactics, but not the success of males playing a dominant, guarder mating role. We also measured significant paternally and maternally inherited additive genetic variance for learning accuracy, raising the possibility that spatial cognition may undergo sexual selection in rose bitterling.

Our findings implicate a possible causal link between male performance in a spatial learning task in the capacity to fertilize the eggs of females in a competitive environment, and especially in the role of a sneaker. The mechanism by which spatial cognition might contribute to male reproductive success was not directly measured in this study. However, male reproductive success in bitterlings is closely linked to the way the male distributes ejaculates in space and time, particularly for sneakers [16,17,26,36,37]. In nature and in the laboratory, male bitterlings systematically patrol mussels in their own territory, as well as those of their neighbours, examine the exhalant siphons of mussels and frequently ejaculate over them (termed pre-oviposition ejaculation) [14,15]. Non-territorial males also engage in the same behaviour, which takes place even in the absence of females, though the presence of a female who is ready to mate significantly increases the rate of male inspection and ejaculation [17,37]. Males appear to obtain information about the presence of their own (and possibly rival) spermatozoa by examining mussel siphons, which may provide them with cues about how to distribute their sperm among mussels [17,37,38]. To maximize their reproductive success, males must anticipate female oviposition decisions, as well as the ejaculatory behaviour of rivals, often among numerous mussels distributed over a wide area, and place their sperm into particular mussels at appropriate time intervals to minimize their risk of sperm depletion [17,18,38,39]. Males also modulate ejaculation size [39–41] in response to the intensity of sperm competition. Thus, in the bitterling mating system, optimizing the size, distribution and timing of ejaculates may impose cognitive demands on males, particularly those playing sneaker mating tactics, which selects for enhanced spatial cognitive ability.

For selection to operate on cognitive traits, a requirement is that they must show heritable variance. The heritability of cognitive ability has rarely been estimated, though where it has, it appears to be significant [4,5]. In humans, more than half of individual differences in intelligence are attributed to additive genetic variation [42,43]. For other taxa, systematic analyses of cognition are lacking [3–5,7]. Our estimates of heritability of performance in a spatial learning task in rose bitterling indicated that approximately one-quarter of variance in learning accuracy was heritable. The heritability of learning accuracy was wholly additive, and both maternally and paternally inherited. Significant additive variance for learning accuracy implies that the trait would respond positively to directional sexual selection. It also implies that selection on spatial learning in the study population has not been consistently strong, as variance in a trait is typically depleted under strong positive selection [29]. However, while spatial cognitive ability may enhance fitness in some circumstances, the evolution of cognitive traits faces constraints and need not always experience positive selection. Thus, there are potential trade-offs between the fitness benefits of enhanced cognitive performance, and costs associated with cognitive traits [44]. In a wild population of great tits (Parus major), parents that were able to solve a cognitive task produced larger clutches than those that failed to solve the task [9]. Task solvers spent less time foraging and foraged over a smaller area than non-solvers, implying that they were more efficient foragers than non-solvers. However, solvers were more sensitive to disturbance and were more likely to desert offspring. The result was that, on average, solvers and non-solvers fledged a similar number of offspring. In a laboratory population of the guppy (Poecilia reticulata), artificial selection on brain size enhanced cognitive ability in females, but not males, despite brain size responding to artificial selection in both sexes. Larger-brained individuals paid a fitness cost in terms of producing fewer offspring, potentially as a trade-off between energetically expensive brain tissue and investment in other organs [45]. In this study, cognitive traits were favoured when males played a sneaker role, but not in a guarder role, implying there may be a trade-off in the traits that make a successful sneaker and guarder. The guarder role in bitterling typically generates higher reproductive success than the sneaker role, though this varies with fish density [16,26].

A prediction from our results is that those males suited to a guarder role will have greater reproductive success at low densities, where the reproductive success of guarder is known to be greatest [16,26]. By contrast, males with superior cognitive ability would be predicted to perform better at high male densities, where sperm competition and male ability to optimally distribute their ejaculates plays a more critical role in male reproductive success [16,26,36,46,47]. A predicted outcome is that selection on male cognitive traits will vary among populations, and within populations among breeding seasons, thereby maintaining variance for cognitive traits. Bitterling populations occur at highly variable densities [48], and males exhibit wide variation in behavioural and morphological traits [36], offering exceptional material for examining selection on cognitive traits in nature. Further investigation of the role of cognitive traits in species that express alternative mating tactics will demonstrate the generality of our conclusions for other mating systems. It would also be informative to examine domain specificity in rose bitterling and establish whether learning accuracy can predict enhanced fitness in other contexts.

In conclusion, this study demonstrates a potential role for spatial cognitive traits in the mating system of a fish. Male performance in a spatial learning task showed additive genetic variance and may undergo intra-sexual selection, particularly under environmental conditions that favour the expression of alternative mating tactics. This is the first non-human study to show genetic variance for spatial cognitive ability with a direct link to reproductive success.

Ethics

All work was approved by the ethical committees of the IVB (no. 163-12) and complies with the legal regulations of the Czech Republic.

Data accessibility

The data associated with this paper are available on Dryad (http://dx.doi.org/10.5061/dryad.hs31q).

Authors' contributions

M.R. and C.S. conceived and designed the experiments, conducted data analyses and wrote the paper. A.P. and C.S. performed the experiments. M.R. conducted genetic analyses. All authors reviewed and approved the paper.

Competing interests

We declare we have no competing interests.

Funding

The research was funded through the Czech Science Foundation (P505/12/G112).