Transitions in sex determination and sex chromosomes across vertebrate species

Abstract

Despite the prevalence of sexual reproduction across eukaryotes, there is a remarkable diversity of sex determination mechanisms. The underlying causes of this diversity remain unclear, and it is unknown if there are convergent trends in the directionality of turnover in sex determination mechanisms. We used the recently assembled Tree of Sex database to assess patterns in the evolution of sex determination systems in the remarkably diverse vertebrate clades of teleost fish, squamate reptiles, and amphibians. Contrary to theoretical predictions, we find no evidence that the evolution of separate sexes is irreversible, as transitions from separate sexes to hermaphroditism occur at higher rates than the reverse in fish. We also find that transitions from environmental sex determination to genetic sex determination occur at higher rates than the reverse in both squamates and fish, suggesting that genetic sex determination is more stable. However, our data are not consistent with the hypothesis that heteromorphic sex chromosomes are an “evolutionary trap”. Rather, we find similar transition rates between homomorphic and heteromorphic sex chromosomes in both fish and amphibians, and to environmental sex determination from heteromorphic versus homomorphic sex chromosome systems in fish. Finally, we find that transitions between male and female heterogamety occur at similar rates in amphibians and squamates, while transitions to male heterogamety occur at higher rates in fish. Together, these results provide the most comprehensive view to date of the evolution of vertebrate sex determination in a phylogenetic context, providing new insight into longstanding questions about the evolution of sexual reproduction.

Introduction

The vast majority of eukaryotes reproduce sexually, and male and female reproductive phenotypes are broadly conserved across a vast array of taxa. Despite this conservation, there is an enormous diversity in the mechanisms used to determine sex across eukaryotes (Bull, 1983; Bachtrog et al., 2014; Beukeboom & Perrin, 2014). This diversity of sex-determination mechanisms is encapsulated in vertebrates, a clade in which nearly all known mechanisms of sex determination are present (Bachtrog et al., 2014; The Tree of Sex Consortium, 2014). This extensive diversity begs the question of both how and why transitions among sex determination mechanisms occur. Although there is an extensive body of theoretical work predicting when and why we might expect transitions in sex determination systems, we have lacked sufficient empirical data to critically test these hypotheses (Bull, 1983; Bachtrog et al., 2014; Beukeboom & Perrin, 2014). Furthermore, little is known about the mechanisms that underlie transitions among sex determination mechanisms. As a first step, identifying whether particular transitions occur at a higher rate than others can provide into whether there are evolutionary or mechanistic constraints on transitions among sex determination systems.

One of the most fundamental transitions in sex determination mechanisms is between the presence of both sexes within the same individual (hermaphroditism) and the presence of two sexes in different individuals (called gonochorism in animals and dioecy in plants). Hermaphroditism is only found in 5% of animal species (Jarne & Auld, 2006; Eppley & Jesson, 2008) but is quite common in flowering plants, with some form of hermaphroditism observed in 94% of species (Renner & Ricklefs, 1995; Renner, 2014). Although the transition to separate sexes was once considered to be irreversible (Bull & Charnov, 1985), recent work has suggested that transitions from dioecy to hermaphroditism might commonly occur in plants (Barrett, 2013; Käfer et al., 2014, 2017; Renner, 2014). Indeed, a recent study in flowering plants revealed no consistent trends in the rates of transition between hermaphroditism and dioecy (Goldberg et al., 2017). However, the rates of transition between hermaphroditism and gonochorism have not yet been investigated in any group of animals. The prevalence of both hermaphroditism and gonochorism in fish makes it an excellent clade to investigate these transition rates.

Even across gonochoristic vertebrate species, there is still a large diversity of sex determination mechanisms (The Tree of Sex Consortium, 2014). Many fish and non-avian reptiles have environmental sex determination (ESD), in which environmental cues, such as temperature during development, are used to determine sex. Many other species of fish, reptiles and amphibians, as well as all known birds and mammals, have genetic sex determination (GSD). The evolution of ESD is thought to be favored when environmental variation has a differential effect on the fitness of males and females (Charnov & Bull, 1977; Warner & Shine, 2008). By contrast, the evolution of GSD is favored in unpredictable environments or in environments with low variability (Bull, 1983). In addition, because single locus GSD results in roughly even sex ratios, it has been assumed to be more stable than ESD, which can result in highly skewed sex ratios (Conover & Heins, 1987; Pen et al., 2010), particularly during periods of environmental instability (Jensen et al., 2018). Sex determining systems with unbalanced sex ratios are prone to invasions by mechanisms that restore balanced sex ratios (Fisher, 1930). Therefore, the balanced sex ratio in GSD might be more resilient to invasion by new sex determining mechanisms, while ESD might be prone to invasion and replacement by systems that restore equal sex ratios. As a result, we might expect a bias toward GSD systems. A recent analysis found mixed support for this hypothesis, with no differences in transition rates between GSD and ESD in turtles, but a higher rate of transition from ESD to GSD in squamate reptiles (Pokorna & Kratochvíl, 2009; Gamble et al., 2015; Sabath, Itescu et al., 2016). However, the rates of transitions between GSD and ESD have not been tested in other suitable groups, such as fish.

Although GSD can be polygenic (Moore & Roberts, 2013), it is more often under single locus control, resulting in systems where either the male or female is heterozygous at the sex-determination locus. In single-locus GSD systems, the genetic difference between males and females can be as small as a single nucleotide variant (Kamiya et al., 2012), however this difference has often progressed to heteromorphic sex chromosomes (Charlesworth et al., 2005; Bachtrog et al., 2013; Wright et al., 2016). In these cases, male heterozygosity progresses to distinct X and Y sex chromosomes, and female heterozygosity to Z and W sex chromosomes. Heteromorphic sex chromosomes are most notably found in the therian mammals (Bellott et al., 2014; Cortez et al., 2014), birds (Zhou et al., 2014; Bellott et al., 2017), and snakes (Matsubara et al., 2006; Vicoso et al., 2013). These clades exhibit a surprising degree of conservation (but see Gamble et al., 2017), given the rapid sex chromosome turnover exhibited by other vertebrate clades. This has led to the suggestion that highly heteromorphic sex chromosomes, once established, are an evolutionary trap, and that transitions from heteromorphic sex chromosomes will be rare due to the accumulation of recessive mutations and loss of essential genes on either the Y or W (Bull, 1983; Bull & Charnov, 1977; Pokorná & Kratochvíl, 2009). This is somewhat at odds with theoretical suggestions that sex chromosomes might cycle quickly due to a combination of sexually antagonistic selection (van Doorn & Kirkpatrick, 2007, 2010) and the accumulation of deleterious mutations on the non-recombining Y or W chromosome (Blaser et al., 2013, 2014). The latter will be more acute as sex chromosome divergence progresses, leading to the counterintuitive idea that older, more degenerate systems would be more prone to transitions. Limited empirical support for the evolutionary trap hypothesis was previously found in squamates (Pokorná & Kratochvíl, 2009; Gamble et al., 2015), but this hypothesis requires further investigation.

In species with GSD, there is further variation in whether the male or the female is the heterogametic sex (The Tree of Sex Consortium, 2014). Here, we will refer to species with male heterogamety as XY and female heterogamety as ZW, regardless of whether there are heteromorphic sex chromosomes present in that species. Transitions between XY and ZW systems have been proposed to result from a variety of evolutionary forces, including drift, selection on pleiotropic effects of sex-determination genes, selection on sex ratio, and sexually antagonistic selection (Bull & Charnov, 1977; Werren & Beukeboom, 1998; Jaenike, 2001; van Doorn & Kirkpatrick, 2010; Bachtrog et al., 2011; Veller et al., 2017). Theory suggests that if transitions between XY and ZW systems are driven by sexually antagonistic selection or even by drift, then the new sex-determination system should be epistatically dominant to the ancestral system (van Doorn & Kirkpatrick, 2010; Veller et al., 2017). Because new W chromosomes have been shown to be dominant to the ancestral Y chromosome in species of fish (Kallman, 1984; Ser et al., 2010) and amphibians (Ogata et al., 2008), it was suggested that there could be a bias in transitions from XY to ZW systems in these groups (van Doorn & Kirkpatrick, 2010). However, empirical tests of this hypothesis have been limited and the few studies that have been conducted have produced mixed results. While a previous study in amphibians found that transitions from ZW to XY systems are more common than the reverse (Hillis & Green, 1990), another found no support for a difference in transition rates between XY and ZW systems across amphibians (Evans et al., 2012). Additional work is clearly needed to determine whether there are any biases in the direction of transitions between male and female heterogamety.

Here, we take advantage of the recently assembled Tree of Sex database (The Tree of Sex Consortium, 2014) and the diversity of sex-determination systems found in three major vertebrate groups (fish, amphibians, squamate reptiles; Figure 1) to examine transitions among sex determination systems. The complete dataset includes information on 705 species of fish, 173 amphibians and 487 squamate reptiles, and as such constitutes the broadest and most comprehensive analysis of vertebrate sex determination evolution conducted to date in a phylogenetic context. We used these data to compare the rates of transition between: (1) gonochorism and hermaphroditism in fish; (2) ESD and GSD in fish and squamates; (3) homomorphic and heteromorphic sex chromosomes in fish and amphibians; (4) homomorphic vs. heteromorphic sex chromosomes and ESD in fish; and (5) XY and ZW sex determination systems in fish, squamates and amphibians (Table 1).

Figure 1. The distribution of sex determination across three vertebrate clades.

Species are coded as being either XY heteromorphic (dark blue), XY homomorphic (light blue), ZW heteromorphic (dark red), ZW homomorphic (light red), unknown homomorphic (?, gray), having environmental sex determination (ESD, green), or being hermaphrodites (yellow). Species that had some degree of ESD were classified as such, regardless of their chromosomes. Note that in the actual analyses we estimated parameters across 10 different datasets, with slightly different taxonomic coverage; for the purposes of this figure, we selected one of these at random.

Table 1.

Summary of analysis

Clade	State 1: # species in dataset (avg # tree-matched species)	State 2: # species in dataset (avg # tree-matched species)	State 3: # species in dataset (avg # tree-matched species)	Prob. of higher transition rate
1. Transitions between gonochorism and hermaphroditism
Fish	Gonochorism: 371 (178.8)	Hermaphroditism: 309 (165)	N/A	0.949 gonochorism to hermaphroditism
2. Transitions between ESD and GSD
Fish	ESD: 61 (22)	GSD: 310 (156.8)	N/A	0.984 ESD to GSD
Squamates	ESD: 49 (22)	GSD: 389 (279)	N/A	1.0 ESD to GSD
3. Transitions between homomorphic and heteromorphic sex chromosomes
Fish	Homomorphic: 145 (82.2)	Heteromorphic: 126 (64.1)	N/A	0.699 heteromorphic to homomorphic
Amphibians	Homomorphic: 94 (60.9)	Heteromorphic: 45 (31.1)	N/A	0.697 heteromorphic to homomorphic
4. Transitions between homomorphic vs. heteromorphic sex chromosomes and ESD
Fish	Homomorphic: 137 (76.7)	Heteromorphic: 125 (64.8)	ESD: 52 (16.4)	0.437 heteromorphic to ESD
5. Transitions between XY and ZW sex determination systems
Fish	XY: 204 (110.5)	ZW: 92 (47)	N/A	0.999 ZW to XY
Squamates	XY: 116 (88)	ZW: 231 (160)	N/A	0.779 ZW to XY
Amphibians	XY: 67 (46.2)	ZW: 32 (28)	N/A	0.560 ZW to XY

Methods

Data

We matched data from the Tree of Sex database (The Tree of Sex Consortium, 2014) to recently published large-scale phylogenies of ray-finned fishes (Rabosky et al., 2013, with 11 erroneously placed species removed from the original; M. Alfaro, pers. comm), squamate reptiles (Pyron et al., 2013, Pyron & Burbrink, 2014), and amphibians (Pyron & Wiens, 2011; Eastman et al., 2013). All trees were ultrametric with branch lengths in units of millions of years. The Tree of Sex dataset we used is the same as that in the original publication with the addition of recently discovered XY systems in two species (Boa imperator, Python bivittatus) of snakes (Gamble et al., 2017). However, other recent studies in squamates are not included in this dataset (Koubová et al. 2014; Pokorná, Rens et al., 2014; Pokorná, Rovatsos et al., 2014; Gamble et al. 2015; Rovatsos et al., 2014, 2015, 2016). In many cases, we had congeneric matches between species in the tree and species in the dataset. In these cases, we used a recently developed algorithm (Pennell et al., 2016), implemented in the R package ‘phyndr’ (https://github.com/traitecoevo/phyndr), to swap species in the phylogeny which were not included in our dataset with ‘phylogenetically equivalent’ species that were (see Pennell et al., 2016 for full details on the algorithm). Unlike data imputation approaches (see Rabosky, 2015), the algorithm is conservative such that it is guaranteed not to introduce any biases into analyses of character evolution, so long as the taxonomy is phylogenetically informative. Since there may be many combinations of phylogenetically equivalent swaps, we ran each analysis across ten different tree and trait combinations to ensure that our analyses were robust to sampling artifacts. Furthermore, for species in which there were multiple records in the dataset, we randomly selected one of these to include in each of the ten analyses. The alternative datasets gave essentially identical results.

Overview of analyses

We addressed five different questions in our analyses. For each analysis, we coded the characters as discrete states and fit a Markov model of trait evolution (Pagel, 1994) using a Markov chain Monte Carlo (MCMC) procedure implemented in the R package ‘diversitree’ (FitzJohn, 2012). For all rates, we set a broad exponential prior (mean of 0.1). We ran all chains for 50,000 generations and removed the first 10,000 samples as burn-in. As stated above, each analysis was run across ten related trait/tree datasets in order to mitigate sampling error, and the results from all individual analyses were summarized together. To examine the support (or lack thereof) for differences in transition rates, we computed differences in rates across the posterior and examined the extent to which the distribution of differences overlapped with zero. In the absence of a reliable procedure for estimating Bayes Factors for MK models in R, we think this is the most accurate way to present our results. All analyses were conducted in R v3.3.1. Code to reproduce all analyses and results is available at https://github.com/mwpennell/vert_trans.

Transitions between gonochorism and hermaphroditism

The prevalence of hermaphroditism in fish (Devlin & Nagahama, 2002; Mank et al., 2006) makes it possible to test for differences in rates of transition between gonochorism and hermaphroditism in this clade. In order to evaluate the relative transition rates between gonochorism and hermaphroditism, we fit a two state Markov model, collapsing all species with separate sexes into a single category of gonochorism. In our fish dataset, there were 371 records of gonochorism (178.8 of which matched to the tree on average across the ten runs) and 309 records of hermaphroditism (165 matched to the tree on average).

Transitions between ESD and GSD

For this analysis, we focused on fish and squamates, as both ESD and GSD have evolved repeatedly in both of these clades. In order to investigate transitions between GSD and ESD, we fit a two-state Markov model similar to the gonochorism/hermaphroditism analysis above. Species were coded as having one or the other form of sex determination. For any species that had some degree of both genetic and environmental sex determination, we coded these species as having environmental sex determination. We also repeated the analysis with these ambiguously coded species excluded, and this did not qualitatively affect our results (Supplemental Figure 1). For fish, there were 310 GSD records in the database (156.8 matched to the tree on average) and 61 ESD records (22 matched to the tree on average). For squamates, there were 389 GSD records (279 matched to the tree on average) and 49 ESD records (22 matched to the tree on average).

Transitions between homomorphic and heteromorphic sex chromosomes

For this analysis, we considered both fish and amphibians. We did not analyze the squamate data due to the rarity of conclusive evidence for homomorphic sex chromosomes in this group (Gamble et al., 2015; Gamble, 2016), precluding the estimation of meaningful character correlations (Maddison & FitzJohn, 2015). Restricting the analysis to gonochoristic species, we coded species for two variables: 1) whether they had cytogenetically visible (i.e. heteromorphic) sex chromosomes or not; and 2) whether they were male or female heterogametic. We removed the few species from the datasets (fish: 15 records; amphibians: 1 record) where the Y (or W) sex chromosome has been completely lost (i.e., XO and ZO systems) as we expect the evolutionary dynamics to be different from those of true XY or ZW systems (Bull, 1983; Maddison & Leduc-Robert, 2013; Blackmon & Demuth, 2014). Three fish species with polygenic sex determination were also excluded. Across fish, this coding scheme resulted in 83 XY homomorphic, 97 XY heteromorphic, 51 ZW homomorphic, 29 ZW heteromorphic, and 11 unknown homomorphic records (average tree-matched counts: 53.2, 47, 23.4, 17.1, and 5.6). For amphibians, these numbers are 37 XY homomorphic, 29 XY heteromorphic, 18 ZW homomorphic, 16 ZW heteromorphic, and 39 unknown homomorphic records (average tree-matched counts: 25.9, 18.3, 14, 12.8, and 21).

Transitions between homomorphic vs. heteromorphic sex chromosomes and ESD

We were only able to perform this analysis on fish, as we were not able to meaningfully estimate parameters in the other clades. For this analysis, we coded gonochoristic species as being ESD, homomorphic GSD, or heteromorphic GSD. For species with GSD, we did not distinguish between whether they were male or female heterogametic, in contrast to the previous analysis. Species that had both GSD and some level of ESD were not included. This resulted in 137 species with homomorphic GSD, 125 species with heteromorphic GSD, and 52 species with ESD. After matching to the tree, we ended up with an average of 76.7 homomorphic species, 64.8 heteromorphic species, and 16.4 ESD species per analysis. We then fit a simple 3-state Markov model and estimated the six (i.e., forward and reverse) transition rates between the three states. For simplicity, we only report the comparison of the transition rate from homomorphic GSD to ESD versus the transition rate from heteromorphic GSD to ESD.

Transitions between XY and ZW sex determination systems

For this analysis, we used fish, squamate and amphibian datasets. We again considered only gonochoristic species with single-locus GSD. We coded all species as being either male (XY) or female heterogametic (ZW) and assumed for simplicity that the probability of an invasion by a novel sex chromosome did not depend on whether a species had homomorphic or heteromorphic sex chromosomes. This assumption is consistent with our finding that the transition rates between homomorphic and heteromorphic sex chromosomes are similar in both XY and ZW systems (see below). There were 204 XY and 92 ZW systems (average tree-matched counts: 110.5 XY, 47 ZW) in the fish dataset, 116 XY and 231 ZW systems (average tree-matched counts: 88 XY, 160 ZW) in the squamate dataset, and 67 XY and 32 ZW systems (average tree-matched counts: 46.2 XY, 28 ZW) in the amphibian dataset. We fit a simple, two state Markov model and estimated net transition rates between XY and ZW systems.

Results

Transition rates from gonochorism to hermaphroditism are higher than the reverse

In our fish dataset, there were roughly equal numbers of gonochoristic (n = 371) and hermaphroditic (n = 309) species (across the ten datasets, there were on average approximately 179 gonochoristic and 165 hermaphroditic species matched to the tree). However, 94.9% of the posterior distribution supports that transitions to hermaphroditism occur at a higher rate than transitions to gonochorism (Figure 2; Table 1). The median rate of transition from gonochorism to hermaphroditism is 2.4 times higher than the reverse.

Figure 2. Transitions from gonochorism to hermaphroditism occur at a higher rate than the reverse in fish.

Posterior distribution of the difference in transition rates between gonochorism and hermaphroditism in fish. Across the 10 datasets, 94.9% of the posterior distribution supports the conclusion that transitions from gonochorism to hermaphroditism occur at a higher rate than the reverse.

Transition rates from ESD to GSD are higher than the reverse

There is strong support for the conclusion that transitions from ESD to GSD occur at a higher rate than the opposite transition in both fish and squamates (Figure 3; Table 1; fish: 98.4% of the posterior distribution; squamates: 100% of the posterior distribution). The transition rate from ESD to GSD is six times higher than the reverse in fish, and 17.3 times higher than the reverse in squamates.

Figure 3. Transitions from environmental sex determination to genetic sex determination occur at a higher rate than the reverse in fish and squamates.

Posterior distribution of the difference in transition rates between genetic sex determination (GSD) and environmental sex determination (ESD) for fish (left panel) and squamates (right panel). Across the 10 datasets, 98.4% of the posterior distribution in fish and 100% of the posterior distribution in squamates supports the conclusion that transitions from ESD to GSD occur at a higher rate than the reverse. Indeed, across the posterior distribution, the rate of transition from ESD to GSD is, on average, around 6 (fish) and 17.3 (squamates) times higher than the reverse.

Similar rates of transition between homomorphic and heteromorphic sex chromosomes

We did not detect a significant difference in the rates of transitions between homomorphic and heteromorphic sex chromosomes; in both the fish and amphibian datasets approximately 70% of the posterior distribution supports the transition from heteromorphic to homomorphic as having occurred at higher rates and 30% suggests the reverse (Figure 4; Table 1). In fish and amphibians, both XY and ZW systems show similar rates of transitions from homomorphic to heteromorphic sex chromosomes (data not shown).

Figure 4. No differences in transition rates between homomorphic and heteromorphic sex chromosomes in either fish or amphibians.

Posterior distribution of the difference in transition rates between homomorphic and heteromorphic sex chromosomes in fish (left panel) and amphibians (right panel). Across the 10 datasets, 69.9% of the posterior distribution in fish and 69.7% of the posterior distribution in amphibians support a higher rate of transitions from heteromorphic to homomorphic sex chromosomes than the reverse, but these results are not significant.

Similar rates of transition to ESD from homomorphic vs. heteromorphic sex chromosomes

The transition rates from either homomorphic or heteromorphic sex chromosomes to ESD are significantly greater than zero (Supplemental Figures 2, 3). However, there is no significant difference between the transition rate from homomorphic sex chromosomes to ESD versus the transition rate from heteromorphic sex chromosomes to ESD (Figure 5; Table 1). These data suggest that the presence of heteromorphic sex chromosomes does not preclude transitions to ESD, at least in fish.

Figure 5. No difference in transition rate from homomorphic sex chromosomes to ESD versus heteromorphic sex chromosomes to ESD in fish.

Posterior distribution of the difference in transition rates between homomorphic sex chromosomes and ESD, and heteromorphic sex chromosomes and ESD. Across the 10 datasets and the entire analyses, 43.7% of the posterior distribution supports a higher rate of transitions from heteromorphic sex chromosomes to ESD, but this is not significant. The posterior distributions of the transition rates between heteromorphic sex chromosomes and ESD and between homomorphic sex chromosomes and ESD are significantly greater than zero (Supplemental Figures 2 and 3).

Rates of transition between XY and ZW systems differ among clades

In the fish dataset, 99.9% of the posterior distribution supports the conclusion that there is a higher rate of transition from ZW to XY systems than the reverse (Figure 6; Table 1). In squamates, 77.9% of the posterior distribution supports a higher rate of transition from ZW to XY systems. However, the bimodal distribution of posterior probabilities likely reflects that there are two different configuration of rates that produce the same distribution at the tips and suggests that our model may not adequately describe this data (Figure 6; Table 1). In amphibians, there is no significant difference in the rate of transition between ZW and XY systems (Figure 6; Table 1).

Figure 6. Transitions from ZW to XY sex determination systems occur at a higher rate than the reverse in fish, but not in squamates or amphibians.

Posterior distribution of the difference in transition rates between XY and ZW systems for fish (left panel), squamates (middle panel), and amphibians (right panel). Across the 10 datasets, 99.9% of the posterior distribution supports the conclusion that there is a higher rate of transitions from ZW to XY sex determination systems in fish. However, there is not significant support for this conclusion in other clades: only 77.9% of the posterior distribution in squamates and 56.0% of the posterior distribution in amphibians suggest a higher rate of transitions from ZW to XY sex determination systems than the reverse.

Discussion

Using the Tree of Sex database (The Tree of Sex Consortium, 2014), we compared the rates of transitions among different mechanisms of sex determination in three vertebrate clades with the most extensive variation in these fundamental traits. Here, we discuss how these results have provided new insight into the theoretical predictions about transitions in sex determination mechanisms, the caveats of our analyses, and the implications of the evolution of sex determination mechanisms, particularly sex chromosomes, for speciation.

The evolution of separate sexes is not always irreversible

In contrast to the hypothesis that the evolution of separate sexes is irreversible (Bull & Charnov, 1985), we find that transitions from gonochorism to hermaphroditism occur at higher rates than the reverse in fish. Thus, we conclude that the evolution of separate sexes is not always an evolutionary “one-way street”, consistent with recent studies in flowering plants (Barrett, 2013; Käfer et al., 2014, 2017; Renner, 2014; Goldberg et al., 2017). Nonetheless, our result is perhaps counter-intuitive, as it would suggest that there should be more hermaphroditic lineages in fish. However, there are actually fewer hermaphroditic species than gonochoristic species in our database (Figure 1; Table 1). Although it is possible that speciation rates are higher for gonochoristic lineages or extinction rates are higher for hermaphroditic lineages, we are unable to formally test these possibilities with the current dataset.

Hermaphroditism is particularly common in reef-dwelling fish (Ghislelin, 1969; Smith, 1975), and it is worth noting that the bright colors of many reef-fishes, which make them popular in the aquarium trade, may also have led to a relative over-sampling of these lineages in our dataset (see Caveats of our analyses). Nonetheless, there also may be extrinsic or intrinsic factors that select against gonochorism in reef environments. Given the relative rarity of dioecy in plants (Renner & Ricklefs, 1995; Renner, 2014), these factors might be shared between plants and reef fish. Alternatively, if gonochorism is the ancestral state in fish (still to be formally tested), then the genetic and developmental mechanisms that underlie the evolution of hermaphroditism from a gonochoristic ancestor might preclude a reversion to separate sexes. Such a constraint does not appear to be present in flowering plants, as the evolution of separate sexes from a hermaphroditic ancestor has occurred many times, and there are no differences in the rates of transition between hermaphroditism and dioecy (Barrett, 2013; Käfer et al., 2014, 2017; Renner, 2014; Goldberg et al., 2017). Future work is needed to identify the evolutionary and genetic mechanisms underlying transitions between hermaphroditism and separate sexes in plants and fish, as well as the potential ecological or life history conditions that might predispose these lineages to hermaphroditism.

ESD is less stable than GSD

Consistent with the hypothesis that species with ESD might have unequal sex ratios and therefore be prone to invasions by GSD to restore balanced sex ratios (Fisher, 1930), we find that transitions from ESD to GSD occur at higher rates than the reverse in both fish and squamates. Our results are also consistent with previous studies in squamates, which have also found that transitions from ESD to GSD are more common than the reverse (Pokorna & Kratochvíl, 2009; Gamble et al., 2015; Sabath, Itescu et al., 2016). However, there is no difference in transition rates between ESD and GSD in turtles, possibly due to the longer lifespan of turtles (Sabath, Itescu et al., 2016). Such longer-lived species are less affected by seasonal variation in the environment that could lead to biased sex ratios and extinction in short-lived species (Bull & Bulmer, 1989; Valenzuela & Lance, 2004). Indeed, turtles with ESD have longer average lifespans than turtles with GSD, and other lineages of reptiles including crocodylians and tuatara, which only exhibit ESD, are also long-lived (Sabath, Itescu et al., 2016). However, broader climatic shifts, such as human-induced climate change, have been shown to lead to major skews in sex-bias (Jensen et al., 2018), suggesting that ESD may be prone to invasion by GSD even in long-lived species. These results highlight that there are many sources of selection on sex determination mechanisms and that additional comparative studies across many systems are needed to shed further light on mechanisms driving these transitions.

Heteromorphic sex chromosomes are not always an evolutionary trap

The observation that some groups, including birds and mammals, have evolutionarily stable heteromorphic sex chromosomes has led to the suggestion that GSD, particularly heteromorphic sex chromosomes, acts as an evolutionary trap that prevents transitions to other mechanisms of sex determination due to the degeneration of either the Y or W (Bull, 1983; Bull & Charnov, 1977; Pokorná & Kratochvíl, 2009). Although previous analyses in squamates are consistent with this hypothesis, these analyses did not distinguish between GSD with homomorphic sex chromosomes and GSD with heteromorphic sex chromosomes (Pokorná & Kratochvíl, 2009; Gamble et al., 2015). This is in part due to the difficulty of reliably identifying systems with homomorphic sex chromosomes in squamates (Gamble et al. 2015; Gamble, 2016). Thus, we performed two complementary analyses to test the evolutionary trap hypothesis. In the first analysis, we found no differences in transition rate between homomorphic and heteromorphic sex chromosomes in either fish or amphibians. In the second analysis, we found that rates of transitions to ESD from homomorphic sex chromosomes were not significantly different than rates of transitions to ESD from heteromorphic sex chromosomes in fish. Our data are consistent with a recent study in Drosophila demonstrating that ancient sex chromosomes have reverted to autosomes (Vicoso & Bachtrog, 2013). Taken together, these data suggest that heteromorphic sex chromosomes might not always be an evolutionary trap that precludes transitions to other systems.

However, these results should be considered as preliminary. It is important to emphasize that we performed these analyses on a dataset in which the classification of heteromorphic sex chromosomes is based mostly on the presence of cytogenetically distinct sex chromosomes (The Tree of Sex Consortium, 2014). For the vast majority of species, this is the only data available. However, cytogenetic methods have low resolution and will greatly underestimate both the number of homomorphic and heteromorphic systems and therefore do not necessarily reveal the extent of degeneration found on a sex chromosome (Ross & Peichel, 2008; Gamble et al., 2015; Gamble, 2016). Testing whether there are differences in transition rates between sex chromosomes with high versus low levels of degeneration, as posited by the evolutionary trap hypothesis, will need to await detailed molecular analyses of sex chromosomes across many systems. Excellent efforts towards this goal have recently been made in some groups of squamates (e.g. Rovatsos et al., 2014, 2016; Gamble et al., 2015), but much more data are needed, which will be facilitated by new methods that rely on next-generation sequencing approaches to detect sex chromosomes (Vicoso & Bachtrog, 2013; Gamble, 2016; Muyle et al., 2016). Such analyses will also enable tests of whether high loads of deleterious mutations might actually promote turnover of degenerate sex chromosomes (Blaser et al., 2013, 2014).

Transition rates between XY and ZW systems differ among clades

We found clade-specific patterns in transition rates between female heterogametic (ZW) and male heterogametic (XY) systems. In fish, transitions from ZW to XY systems occur at higher rates than the reverse. By contrast, we found no differences in transition rates between XY and ZW systems in squamates or amphibians. Our results in amphibians are consistent with those of Evans et al. (2012), although an earlier study in amphibians found evidence for a bias from ZW to XY systems (Hillis & Green, 1990). However, both our study and Evans et al. (2012) used a larger dataset, updated phylogeny, and different methodologies.

Importantly, none of these results support the prediction that transitions from XY to ZW systems should occur at higher rates in these groups (van Doorn & Kirkpatrick, 2010). This prediction was based on both theoretical findings that transitions occur most readily when the new sex chromosome is dominant to the ancestral sex chromosome (van Doorn & Kirkpatrick, 2010; Veller et al., 2017), and empirical findings that new W chromosomes are dominant to ancestral Y chromosomes in multi-factorial sex determination systems found in fish and amphibians (Kallman, 1984; Ogata et al., 2008; Ser et al., 2010). To our knowledge, there are no systems in which a new Y chromosome is dominant to an ancestral W chromosome, but this could be because there are very few systems in which both types of sex chromosomes segregate. Furthermore, both W-linked and Y-linked sex determination loci are often dominant (Bachtrog et al., 2014), suggesting that there should not necessarily be a bias in the dominance relationships between W and Y-chromosomes. One alternative hypothesis to explain the prevalence of ZW to XY transitions in fish is that stronger sexual selection in males might promote transitions to XY systems if the sex-determination locus is linked to loci that are beneficial in males (Rice, 1986; Bachtrog et al., 2011). However, it is not known whether sexual selection is generally stronger in fish than in amphibians or squamates. A second alternative hypothesis is that transitions to XY systems might be favored because dominant masculinizing mutations on a Y chromosome can protect against female sex-ratio biases caused by cytoplasmic sex ratio distorters (Beukeboom & Perrin, 2014). The presence of numerous transitions between XY and ZW systems in vertebrates provide an excellent opportunity to further explore these hypotheses.

Caveats of our analyses

Because we were making inferences about the evolutionary dynamics of sex determination systems across large phylogenetic scales, there are a number of caveats to our analyses. First, it was necessary to assume homogeneous transitions rates within each clade that we studied (i.e., rates were assumed to be the same within fish but could differ between fish and squamates). Although transition rates between traits are often highly variable (e.g., Beaulieu et al., 2013) and several methods exist to estimate the phylogenetic position at which they change (e.g., Drummond & Suchard, 2010; Beaulieu et al., 2013; King & Lee, 2015), such methods have not been extended to the multi-state case we consider in some of our analyses, particular to the linear model formulation (FitzJohn, 2012). Therefore, for the sake of consistency and coherence, we maintained this rate-homogeneity assumption across all the analyses. As a result, ours is likely a conservative approach, as this assumption is will tend to obscure true differences rather than induce spurious ones.

Second, we coded all characters as having discrete states, even though this is artificial in some cases. In particular, in our analysis of transitions between homomorphic and heteromorphic sex chromosomes, we assumed that all heteromorphic chromosomes were alike, even though there in in fact a continuum of sex chromosome divergence, ranging from systems in which the two sex chromosomes are nearly identical in size and gene content to those in which the sex-limited Y or W chromosome is severely diminished. However this simplification is necessary in the absence of more refined data on the relative size of the sex chromosomes or a phylogenetic model that can adequately describe the process of chromosomal differentiation. Discrete models of evolution such as this one may make our analysis prone to phylogenetic pseudoreplication, wherein apparently strong evolutionary associations between characters resulted from only a few evolutionary events (Read & Nee, 1995; Maddison & FitzJohn, 2015; Uyeda et al., 2017). There is no clear solution to this problem, nor is there any reliable diagnostic test for phylogenetic pseudoreplication. However, there is a general increase in the robustness of estimates as the number of independent evolutionary transitions increases in a dataset. As apparent from Figure 1, each of the transitions we considered has occurred multiple times in different parts of the phylogeny such that we can be reasonably confident the associations we have found are unlikely to be spurious.

Third, our dataset is sparse. For example, in our largest dataset (gonochorism vs. hermaphroditism) we have 680 records for bony fish from a clade of more than 27,000 species. The squamate and amphibian datasets similarly comprise less than 5% of extant species. However, if our dataset represents a random sample (at least with regard to their sexual system) of the existing diversity, our estimates of transition rates will be unbiased (Pagel, 1994). However, if some taxonomic groups or states are disproportionately represented in our dataset, this could lead to erroneous estimates. For example, it is quite plausible that species with homomorphic sex chromosomes are underrepresented in our database; as discussed above, these are more challenging to recognize than heteromorphic sex chromosomes with mostly cytogenetic data. Likewise, some taxonomic groups are far more likely to be included in the database for a variety of reasons. While some methods have been developed to assess and mitigate the effect of such sampling biases on evolutionary inference from discrete characters (FitzJohn et al., 2014), the sampling in our database is still too low to apply these methods here. As such, we encourage readers to keep this important caveat in mind.

Fourth, our analyses make the assumption that the system of sex determination does not influence macroevolutionary rates of speciation or extinction. But as Maddison (2006) eloquently pointed out, failing to consider an association between a character and diversification rates when such an association actually exists can lead to estimates of transition rates being biased. A number of methods, most notably the *SSE (State-dependent speciation and extinction) family of models (e.g., Maddison et al., 2007; FitzJohn, 2012) have been developed to simultaneously model the evolution of traits and the diversification process, thus mitigating the potential for bias. Indeed, there are theoretical predictions that differences in sexual systems may lead to differences in the rates at which new species form or go extinct (see below). However, we chose to ignore speciation and extinction for two reasons. First, as we state above, the number of species included in our dataset is small relative to the number of species in these clades; as such, we would have essentially no power to detect any differences in diversification rates (FitzJohn et al., 2009). Second, our data spans large taxonomic groups, such that are likely to be many differences in speciation rates unrelated to the traits of interest; recent work has shown that BiSSE (and related methods) are susceptible to being misled by such background variation in rates (Rabosky & Goldberg, 2015). No general solution to this problem has yet been developed. When more data become available and/or novel approaches to studying state-dependent diversification are developed, it would be very interesting to revisit this work to understand the role of speciation and extinction in shaping the phylogenetic distribution of sexual systems.

Does the evolution of sex determination mechanisms influence speciation?

The evolution of sex determination, particularly sex chromosomes, also has important implications for speciation. In particular, two empirical patterns, Haldane’s Rule and the large-X effect, provide evidence that heteromorphic sex chromosomes play an important role in the evolution of post-zygotic hybrid sterility and inviability in animals (Haldane, 1922; Coyne and Orr, 1989; Presgraves, 2008). These patterns have led to the hypothesis that the presence of heteromorphic sex chromosomes might facilitate speciation, particularly the evolution of post-zygotic incompatibilities (Rieseberg, 2001; Philips & Edmands, 2012). Limited support for this hypothesis has been found. For example, higher levels of intrinsic post-zygotic isolation are found in species with sex chromosomes than in those without sex chromosomes, and in species with more heteromorphic or larger sex chromosomes (Turelli & Begun, 1997; Lima, 2014). Philips & Edmands (2012) tested this hypothesis using net diversification intervals as a proxy for speciation rate in reptiles; they found that speciation occurs more rapidly in squamates, in which heteromorphic sex chromosomes are common, than in turtles and crocodylians, in which heteromorphic sex chromosomes are rare or absent. However, these results were equivocal as net diversification intervals in birds, which have heteromorphic sex chromosomes, are similar to those in turtles and crocodylians (Philips & Edmands, 2012). Indeed, other analyses have found no association between diversification rates and the presence of ESD or GSD in turtles, squamates or birds (Organ & Janes, 2008; Sabath, Itescu et al. 2016). Thus, there is not strong evidence that the presence of heteromorphic sex chromosomes influences speciation rates in vertebrates.

It is also possible that speciation rates might instead reflect the rates of turnover in sex chromosomes (Demuth, 2014). Indeed, in stickleback fish, a fusion between an existing Y chromosome and an autosome created a neo-sex chromosome system that harbors loci involved in both behavioral isolation and hybrid male sterility between species, suggesting that sex chromosome turnover has facilitated speciation in this case (Kitano et al., 2009). However, testing the role of sex chromosome turnover or transitions in sex determination mechanisms in speciation using methods such as BiSSE requires more complete datasets than we have assembled here for vertebrates (FitzJohn et al., 2009). A few studies have been conducted in plants; however, no consistent association was found between sexual system and diversification rates (Leslie et al., 2013; McDaniel et al., 2013; Villareal & Renner, 2013; Sabath, Goldberg et al., 2016). Thus, the relative importance of sex chromosome turnover or transitions in sex determination mechanisms to the process of speciation is mostly unclear. Additional studies to characterize the diversity of sex determination mechanisms that we have highlighted here in vertebrates, as well as the diversity present in plants and invertebrates (The Tree of Sex Consortium, 2014), will provide a rich resource for future research to address this question.