Reconciling Species Concepts: An Ecological Perspective

ABSTRACT

Species concepts remain contentious, both in paleoanthropology and in modern taxonomy. The lineage‐based concept emphasizing evolutionary independence is most fundamental, but in practice is generally represented by proxy evidence of morphological or genetic divergence. This has resulted in a troubling proliferation of species names in the hominin fossil record. Pseudo‐extinctions where lineages persisted under a new species name need to be distinguished from cases where lineages ended terminally—the implications for ecological adaptability are diametrically opposed. Furthermore, the ecological criterion for species coexistence is widely overlooked. The competitive exclusion principle holds that species sharing closely similar niches cannot continue to coexist in the same place at the same time. Notably, the largely vegetarian Paranthropus lineage remained distinct from the diverging, more versatile Homo lineage until fading from the fossil record during the later Pleistocene. Claims that additional hominin species existed are ecologically suspect unless supported by evidence of adequate niche separation. Modern examples where there has been equivocation in lineage recognition are illustrated for bovids, giraffids, baboons, and elephants. Furthermore, the mechanisms that resulted in the displacement of Neanderthals by modern humans are reappraised from an ecological perspective. Representations of evolutionary divergence as a bushy tree need to be superseded by the emerging paradigm of reticulate lineages diverging and coalescing through time and space.

1. Introduction

Twenty or more species of hominin have been identified in this clade since it emerged in the fossil record around 7 Ma (Wood and Boyle 2016). Only one has survived to the present day— Homo sapiens . Nineteen or more named species fell by the wayside, evidently extinct. Why were hominins so prone to extinction?

Recognition of the origins and extinctions of species rests on concepts of what makes a species distinct. I draw on the article by Martin et al. (2024), advocating a unifying lineage‐based concept of species. It in turn builds on an earlier appraisal by de Queiroz (2007) distinguishing fundamental species concepts from practical species delineation. Functionally, species represent independently evolving segments of lineages that have diverged from other such lineages. Independence is commonly achieved geographically, enabling genomes and the morphological features they generate to drift apart. Eventually, barriers to mating or lack of viability of the offspring produced reinforce the spatial isolation. This is the generally conceived model of speciation, achieved progressively through time or facilitated by splitting into subsequently diverging lineages. However, over evolutionary time scales the evidence is restricted to morphological distinctions among fossilized remains, frequently assessed by distinguishing ancestral from derived traits. Judgment is needed as to whether these would have been sufficient to form barriers to genetic merging. Given the judgmental uncertainty, the tendency has been for species labels to proliferate, notably among hominins (Martin et al. 2024). Some of the changes in names do not actually represent extinctions: the lineage persisted in an altered form.

However, a fundamental ecological requirement must be met in order for species populations to coexist. The competitive exclusion principle holds that no two species can occupy the same niche in the same place at the same time; at least not for very long. Hence, prospective species must differ sufficiently in some feature of physiology, resource use, or predator evasion, defining their exclusive ecological niche, if they are to persist in sympatry. Some time ago, I modeled in some detail how morphological features contribute to niche separation among large mammalian herbivores, the group of organisms with which I am most familiar (Owen‐Smith 1985). Dietary distinctions appear related to differences in dentition, oral morphology, digestive anatomy, and body size. Several of these features are preserved in the fossil record. An expanded assessment considering also the effects of the risk of predation on habitat occupation is presented in Owen‐Smith (2021). An extant large herbivore fauna of almost 100 species resulted from the end‐Miocene radiation and subsequent niche divergence, while 20 or more species may coexist within locally protected areas like Serengeti National Park.

Among hominins, niche separation emerged following the Pliocene‐Pleistocene transition with the appearance of the dietary more versatile Homo clade alongside the large‐jawed primarily vegetarian product of the Australopithecus lineage (i.e., Paranthropus spp), in both southern and eastern Africa. Named species of Homo proliferated thereafter, some remotely in Eurasia. But propositions that multiple species of Homo coexisted, in time and space, in parts of Africa during the Pleistocene (e.g., Bobe and Carvalho 2019) seem dubious ecologically. How might they have differed sufficiently in the resources that they depended on, or how they extracted these food types, or where they found them, or in some other feature conferring niche separation? Earlier during the Pliocene, a generic distinction between Ardipithecus and Australopithecus is widely recognized even though these forms evidently connect an enduring lineage. There is an elevation in adaptive grade associated with the transition to bipedal locomotion that took place and its consequences for resource exploitation, but where it was manifested in time remains unclear. How Homo naledi, a somewhat smaller form with surprisingly archaic features for its existence around 280 ka, is connected remains puzzling. Perhaps it is a locally adapted offshoot of the Paranthropus lineage, which accordingly did not become extinct around 1 Ma.

Complications do arise in defining the “same” place and time. A braided channel concept of reticulate evolution has recently come to the fore (Scerri et al. 2018). This recognizes how morphological divergence in disparate local populations could be transient when distributions shift over the course of time. As a result, genes shaping local adaptations can become shared within a broader metapopulation (de Queiroz 2007; Foley and Lahr 2024). The outcome may be a rather complex pattern of spatial and temporal variation in morphological features (Anton and Middleton 2023; Hautavoine et al. 2024). This complicates inferences concerning how particular features contribute to reproductive isolation and niche separation, especially within a widely distributed genus like Homo.

2. Species Assignments Among Extant Large Mammals

There has also been equivocation in species assignments among extant large mammal taxa. I will discuss some examples in the context of species delineation.

2.1. Ungulates

Like Homo, hartebeests (Alcelaphus spp) are widely distributed across Africa from the Cape formerly to as far north as Egypt. An authoritative treatment of their biology assigns regional variation to eight subspecies, based mainly on variation in horn shape and coat color (Gosling 2013). Lichtenstein's hartebeest ( A. buselaphus lichtensteini), which inhabits the Miombo woodland region of central Africa, was proposed as a distinct species, being most divergent in horn shape, but this was not supported genetically. The allied hirola (or Hunter's hartebeest; Beatragus hunteri ) inhabiting the northeastern arid region of Africa is seemingly a relic of early cladogenesis and remains specifically differentiated. Similar patterns of subspecific distinction prevail for other widely distributed bovids.

In contrast, the nine recognized subspecies of giraffe ( Giraffa camelopardalis ) have recently been assigned to four species (Fennessy et al. 2016). This is based on distinctions in genetics (Winter et al. 2018; Bertola et al. 2024), supported by morphological differences in the shape of bony protuberances on the skull along with coat markings (Kargopoulos et al. 2024). Evidence of the implications for resource partitioning or mate recognition was not provided. This proposed taxonomic revision is inconsistent with the lineage concept as outlined above. Notably, prominent “ossicones” on the skull are exclusively a male feature, related to how giraffe males fight using blows with their heads. This implies that females differing in their rudimentary ossicones would qualify morphologically as a different species. Distinguishing sex differences from species differences is, of course, a pervasive problem in paleontology.

2.2. Baboons

The baboons (Pappio sp) are closest ecological counterparts of the hominins, being ground‐dwelling, savanna‐living omnivores. They are distributed across Africa, from the Cape to Ethiopia and into Arabia, in morphologically distinct forms occupying spatially disparate ranges. These regional populations are regarded as representing five or six distinct species by some observers and all as subspecies by others (Newman et al. 2004; Rogers et al. 2019). However, the morphological features used to differentiate the geographic units are mainly the color tinge, thickness, and distribution over the body of the fur coat (Jolly 2014), which would not be discernible in fossils. Ecological differences in resource use underlying the distinct distribution ranges are not apparent (Fischer 2019) and interbreeding takes place where ranges overlap. Genetic analyses reveal multiple episodes of admixture and introgression, showing evolutionary divergence, hybridization, and reticulation since the origin of the genus Pappio in southern Africa ~2 Ma (Rogers et al. 2019). Ecologically, there is no justification for regarding the named baboons as distinct species. Their closest ecological counterparts are the geladas (Theropithecus spp), similar to baboons in appearance and terrestrial habit but more narrowly grazers in their diet. The gelada clade went extinct late in the Pleistocene, apart from the sole relic ( Theropithecus gelada ) occupying the Ethiopian highlands, possibly due to competition from the baboons spreading northward. Although the geographic conditions enabling species divergence exist, baboons have not diversified ecologically, perhaps because there is insufficient niche space for an ecologically distinct form of baboon to evolve.

2.3. Elephants

The African forest elephant ( Loxodonta cyclotis ) and savanna elephant ( Loxodonta africana ) have recently become separated at the species level, despite quite minor morphological differences. Underlying this is a complex pattern of genetic divergence and introgression (Rohland et al. 2010; Palkopoulou et al. 2018). The ancestral form diversifying in Africa during the late Miocene, earlier named Elephas recki, has recently been revised to Palaeoloxodon recki and linked genetically to the European straight‐tusked elephant P. antiquus . The African savanna and forest elephants became separated genetically between 2.6 and 5.6 Ma, but the forest lineage continued to share genes with P. antiquus in southern Europe afterward. Hybridization continues to occur between forest and savanna elephants where their geographic distributions adjoin, especially in the Albertine rift region of Uganda and the adjoining Congo, accompanied by ecological mixing in the habitats occupied (Mondol et al. 2015; Kuhner et al. 2025). Despite this complexity in genetic exchanges, only two elephant lineages persist in Africa, incompletely isolated spatially despite their distinct habitat affiliations. Other elephant‐like proboscideans faded out from the fossil record during the course of the Pleistocene.

3. Neanderthal Displacement

Shifting attention back to hominins, there is a well‐founded example of splitting and divergence in hominin lineages that turned out to be unviable (Nowell 2023). Homo neanderthalensis confined to Eurasia shared a common ancestor with H. sapiens in Africa 600 ka or earlier (Stringer and Crete 2022). Numerous morphological and genetic distinctions between the two lineages justified distinguishing them at the species level (Stringer and Crete 2022; Meneganzin and Bernadi 2023). Their geographic isolation ended when modern humans spread through Eurasia after ~60 ka. There is evidence of earlier invasions, in a cave in Greece dated around 200 ka, and in the Levant dated ~100 ka (Li et al. 2024). However, these appear to have been temporary incursions during mild interglacial conditions (Stringer and Crete 2022). The incursion by modern humans had reached western Europe by 47 ka. No fossil remains or cultural artifacts of the Neanderthals endured after 40 ka (Higham et al. 2014; Iasi et al. 2024). The H. neanderthalensis lineage had become globally extinct, apart from the small package of genes they had contributed to H. sapiens via the interbreeding that did occur.

Despite their shared dietary dependence on hunting large mammals, especially during glacial winters, the Neanderthals and modern humans differed in their hunting methods: the Neanderthals relied on short spears thrust at close quarters, while modern humans brought with them projectile weapons, that is, throwing spears and arrows launched from bows (Banks 2008; Shea and Sisk 2010; O'Driscoll and Thompson 2018). Following from this distinction, the Neanderthals would have focused their kills on the more vulnerable segments of their prey populations, that is, the young and the old. In contrast, anatomically modern humans were enabled by their weapons to kill even healthy adults. Resource competition would have been accentuated by the effects of hunting by early modern humans encompassing the adult segment on the demography of the shared prey populations. As any demographically structured population model demonstrates, mortality incurred during the prime adult stage reduces the proportion of the population surviving into the old age range (Owen‐Smith 2015). The resultant demographic skew would have restricted the prey base for the Neanderthals. This phenomenon helps explain why, in Africa, cursorial carnivores like cheetahs ( Acinonyx jubatus ) and African wild dogs ( Lycaon pictus ) are generally so uncommon despite their high hunting success. Demographic skew of their shared prey base by lion kills spread across all life‐history stages forces these smaller carnivores to roam widely in seeking vulnerable prey (Creel et al. 2023). Notably, the extinction of the Neanderthals was followed shortly by the last recorded dates for several of their herbivorous prey species, along with the demise of the large carnivores dependent on this same food resource (Stuart 1991; Barnosky et al. 2004; Sandom et al. 2014). Modern humans were the superior hunters technologically, with devastating consequences for the persistence of the Neanderthals and eventually for the survival of all of the megaherbivores weighing more than 1000 kg across the northern hemisphere, together with their apex predators.

Numerous hypotheses have been proposed to explain the demise of the Neanderthals. As in conservation biology, a distinction needs to be made between processes that reduce a population and those that accelerate the trend towards extirpation once populations become small (Caughley 1994). From an ecological perspective, the global displacement of the Neanderthals by modern humans was primarily an outcome of trophic overlap, albeit exacerbated by the climatic stress associated with the impending glacial extreme (Banks 2008; El Zaatari et al. 2016). Through their hunting pressure, modern human invaders depleted the basic food resource that the Neanderthals had depended on. Other circumstances could have contributed. The Neanderthals might have experienced direct competition for the shelter provided by caves, especially during the increasingly cold winters experienced as the Last Glacial Maximum approached, with the larger groups formed by the modern humans dominant (Marean 2016). Other factors proposed as causal would have come into play once the population size of the Neanderthals had shrunk sufficiently, such as genetic inbreeding, accentuating the extinction vortex (Slimak et al. 2024). In these circumstances, the overlapping niche with modern humans occupied by the Neanderthals in their dietary dependence proved no longer viable ecologically, despite their substantial morphological and genetic divergence from modern humans. This illustrates how lineage independence is not consolidated until adequate ecological separation is attained.

4. Implications for Extinctions, Biodiversity and Rewilding

Why does it all matter? “Pseudo‐extinctions” where diverging lineages continued to propagate their gene pools, albeit under a new name, need to be distinguished from instances where lineages terminate, leaving no genetic descendants. In the first situation, the organisms adapted successfully, while in the second outcome the organisms failed to cope with changing circumstances. Whether adaptive divergence within enduring lineages warrants a new species label depends on the implications for niche distinctions. The proliferation of species labels associated with the hominin clade does not mean that it was as precariously viable as it might seem. Some pruning is needed, as was advocated by Martin et al. (2024).

Comparative assessments of biological diversity need to apply a common criterion for species recognition. Paleo‐communities could appear more speciose simply because a more widely encompassing concept of species was used. Baboons and giraffes are no more diverse in species generated than similarly widespread antelope. Species need to be distinguished consistently from subspecies.

Actions undertaken to restore past species assemblages (“rewilding” initiatives) need to be guided by whether the extinct forms represented distinct species or merely geographically deviant representatives of extant species. If the former, animals should be drawn from a specific source population in order to re‐establish the missing species niche. If the latter, animals from any subspecies would restore the functional contribution of the extirpated animals (although it may still be desirable to retain subspecies distinctions, if possible).

5. Conclusions

It is obfuscating to have five alternative concepts invoked for distinguishing species: evolutionary, reproductive, morphological, ecological, and genetic. The independent lineage concept, as consolidated by Martin et al. (2024), is functionally fundamental. The biological species concept is concerned with the mechanisms that isolate lineages genetically, through mate recognition or hybrid infertility. Morphological features provide the evidence establishing whether organisms have become isolated through mate recognition, physiological disparities, or ecological incompatibility. Ecological isolation operates over longer time periods and may become effective only after populations that had diverged in allopatry overlap in their geographic ranges. Prior to that eventuality, ecological equivalents may occur in discrete localities. Resource distinctions should encompass not only what was eaten but also potential differences in how these food types were acquired, for example, by scavenging versus active hunting, as reflected in dietary composition. Genetic distinctions may indicate the time that has passed since lineages split but not whether speciation occurred within this period. When the fossil evidence is fragmented and distorted, as it is for hominins, parsimony in species recognition is the more defendable standpoint (Martin et al. 2024; see also Villmoare and Kimbell 2024).

The classical metaphor of evolution as a bushy tree needs to be superseded by a more dynamic representation taking the form of a reticulate flux, anastomosing and rejoining to constitute a metapopulation (Scerri et al. 2018; Van Holstein and Foley 2022). Concordantly, ecological niches are best depicted as trajectories through time and space as organisms adapt, or fail to adapt, to changing circumstances (Owen‐Smith 2021). Resolving alternative species concepts from the hominin fossil record is hampered by the extremely fragmentary nature of the evidence. Fossilized remnants of associated large mammals are vastly more substantial, but remain largely unanalyzed within the newly emerging paradigm. This is where more analytic attention is needed, while searches for missing hominin fossils continue.

Remarkably, the lineage that led to H. sapiens endured unbroken through the environmental vicissitudes of the Pliocene and Pleistocene, despite the names applied in different places and stages. In the process, its ecological niche became transformed so radically from forest‐dwelling frugivore to the most feared predator in the African savannas and beyond (Owen‐Smith 2021; Zannette et al. 2023).

Author Contributions

Norman Owen‐Smith: conceptualization (equal), formal analysis (equal), methodology (equal), writing – original draft (equal), writing – review and editing (equal).

Data Availability Statement

The author has nothing to report.