No known hominin species matches the expected dental morphology of the last common ancestor of Neanderthals and modern humans

Significance

The identity of the last common ancestor of Neanderthals and modern humans is a controversial issue. This debate has been often addressed by means of descriptive analyses that are difficult to test. Our primary aim is to put questions about human evolution into a testable quantitative framework and to offer an objective means to sort out apparently unsolvable debates about hominin phylogeny. Our paper shows that no known hominin species matches the expected morphology of this common ancestor. Furthermore, we found that European representatives of potential ancestral species have had affinities with Neanderthals for almost 1 My, thus supporting a model of early divergence between Neanderthals and modern humans.

Abstract

A central problem in paleoanthropology is the identity of the last common ancestor of Neanderthals and modern humans ([N-MH]LCA). Recently developed analytical techniques now allow this problem to be addressed using a probabilistic morphological framework. This study provides a quantitative reconstruction of the expected dental morphology of the [N-MH]LCA and an assessment of whether known fossil species are compatible with this ancestral position. We show that no known fossil species is a suitable candidate for being the [N-MH]LCA and that all late Early and Middle Pleistocene taxa from Europe have Neanderthal dental affinities, pointing to the existence of a European clade originated around 1 Ma. These results are incongruent with younger molecular divergence estimates and suggest at least one of the following must be true: (i) European fossils and the [N-MH]LCA selectively retained primitive dental traits; (ii) molecular estimates of the divergence between Neanderthals and modern humans are underestimated; or (iii) phenotypic divergence and speciation between both species were decoupled such that phenotypic differentiation, at least in dental morphology, predated speciation.

Uncertainty about ancestry is typical for many nodes of the hominin phylogeny. Species are usually suggested as common ancestors based on their geography, chronology, and morphological affinities (1–3), but these criteria can be unsatisfactory for two fundamental reasons: first, the inherent incompleteness of the hominin fossil record precludes a clear understanding of the spatial and temporal range of variation of hominin species; and second, comparative morphological analyses are frequently descriptive in nature, making them difficult to evaluate objectively.

Fossil dental remains are central to our understanding of hominin evolution due to their commonness in archaeological and paleontological sites, their normally good state of preservation, and the minor influence that environmental factors—apart from wear—have on their morphology during an individual's life. Teeth provide the most consistently complete representation of the hominin fossil record and are thus fundamentally important for assessing the plausibility of competing evolutionary scenarios and avoiding inadvertently biased interpretations (Tables S1 and S2). The combination of quantitative evolutionary theory, geometric morphometrics, and dental anthropology provides a powerful framework for reconstructing statistical expectations of the morphology of unknown ancestors (ref. 4 but see also ref. 5) and for making posterior comparisons with actual fossil remains.

The main aim of our study is to evaluate whether several known candidate species have morphologies that are statistically consistent with them being the last common ancestor of Neanderthals and modern humans ([N-MH]LCA). To be a credible ancestor, a fossil must have lived at the appropriate time and be statistically close to the ancestral morphology, taking into account the uncertainty of the ancestral reconstruction that arises from lability of the evolutionary process and alternative hypotheses of the phylogeny on which it is based. To implement this, we used geometric morphometric data to reconstruct ancestral dental shapes and their confidence ellipsoids using the phylogenetic generalized linear model (GLM) approach (ref. 6; see Materials and Methods and SI Text for detailed explanations). Ancestral reconstructions are based on an independent phylogeny, ideally one built from nondental characters to avoid circularity (Fig. 1 and Fig. S1). Note, however, that the phylogenetic relationships of hominin species are vigorously debated and that they are based mainly on craniodental traits, although not on the quantitative dental shape traits we use (Figs. S2 and S3). The ancestral dental reconstructions were based only on species whose phylogenetic position is relatively uncontroversial; species whose phylogenetic position is contested were evaluated as potential candidate ancestors for nodes of relevant age. We considered 12 phylogenetic topologies that are collectively representative of the range of credible alternative hypotheses about hominin relationships (7). The ages of fossils and nodes were calibrated using comprehensive reviews of the chronology, geographical location, morphology and ecology of hominin species (8). In addition to the [N-MH]LCA, we evaluated ancestor candidates at two other important nodes of hominin phylogeny, the last common ancestor (LCA) of the genus Homo (excluding transitional hominins such as Homo habilis and Homo rudolfensis) and the LCA of Paranthropus and Homo, as controls for whether our statistical framework is too strict. Our method is not intended to be a phylogenetic reconstruction technique, but a means to evaluate compatibility of potential ancestral species within the framework of established phylogenies.

Fig. 1.

Phylogeny used for ancestral state reconstruction and candidate species. Numbers in parentheses represent the averaged chronology of the corresponding species (although see ref. 56); italic numbers represent branch lengths in millions of years; bold gray numbers represent ages of nodes. Branch lengths corresponding to a 1-Ma divergence between Neanderthals and modern humans are represented to the left of the slashes and values corresponding to a 0.45-Ma divergence to the right. H. erectus has been dated to 1 Ma (values before the slash in the H. erectus branch) or to 0.5 Ma (values after the slash). Candidate species are represented below the phylogenetic tree with bars showing their approximate temporal spans (black, candidates for the [P-H]LCA; dark gray, candidates for the [H]LCA; light gray, candidates for the [N-MH]LCA). H. erectus and the SH group have been included in the framework phylogeny in four (H. erectus) and six (SH) phylogenetic scenarios (Fig. S1) and evaluated as candidate species in the remaining scenarios. H. ergaster has been evaluated as a candidate species for the [H]LCA and the [N-MH]LCA. H. habilis has been evaluated as a candidate for the [P-H]LCA and the [H]LCA. Vertical guides indicate the age of the evaluated nodes. Fig. S1 shows individualized representations of the 12 phylogenetic topologies.

Phylogenetic scenarios differed from one another in the number and age of tip species and in the age of the nodes (branch lengths; Fig. 1 and Fig. S1). European Middle Pleistocene fossils were grouped into one of two operational taxonomic units based on previously recognized morphological differences (9): the Homo heidelbergensis group (represented in our study by fossils from Arago Cave) and the Sima de los Huesos (SH) group. SH hominins, unlike other European Middle Pleistocene fossils (such as those from Arago, Mauer, and Petralona), are widely considered to be an early chronospecies of classic Neanderthals (3, 9–11). However, the markedly derived dentitions of SH hominins (12) appear to contradict their placement as an early member of the Neanderthal lineage and their morphology substantially influences the reconstruction of the [N-MH]LCA; therefore, we evaluated SH both as a taxon on the phylogenetic tree and as a potential candidate for the [N-MH]LCA. For the former, SH was considered to be a node in the Neanderthal lineage at 350 ka following ref. 13, an age that is less controversial than a subsequent, older assessment of circa 600 ka (14). We considered two alternative estimates for the divergence time between Neanderthals and modern humans (1, 11). One scenario dates this divergence to 450 ka based on molecular clock estimates (11, 15); the other dates this divergence to 1 Ma based on morphological affinities observed in the fossil record (see Results and Discussion). We also considered two dates for Homo erectus, 1 Ma or 500 ka, to account for the different age estimates of key Javanese (16) and Chinese fossils (17). We included H. erectus as a potential candidate species for the [N-MH]LCA in some phylogenetic scenarios. For each candidate, we evaluated its compatibility with the statistical reconstruction of the morphology of the LCA under each of the 12 phylogenetic scenarios. Species were considered implausible as ancestors if one or more of their teeth was incompatible with the expected ancestral morphology under all 12 of the phylogenetic scenarios.

Results

The statistical reconstruction of the dental morphology of the [N-MH]LCA is characterized by symmetric and lingually reduced upper premolars (P³ and P⁴) and by slightly asymmetric lower premolars (P₃ and P₄). The upper first molar (M¹) has a slightly skewed morphology and the lower first molar (M₁), a Dryopithecus pattern (also called “Y5”) with five well-developed main cusps. The Dryopithecus pattern is the shared ancestral condition for all hominin lower molars, and consists of a five-cusped molar in which the mesiolingual cusp (metaconid) and the distobuccal cusp (hypoconid) are in contact and create a Y-shape fissure pattern. The upper second molar (M²) of the reconstructed common ancestor has a mildly reduced hypocone, whereas the upper third molar (M³) has strong reduction such that a high proportion of individuals are expected to lack the distolingual cusp. Lower second molars (M₂) of the reconstructed [N-MH]LCA are expected to have marked reduction of the fifth cusp and frequent occurrence of non-Y5 morphologies (mainly cruciform patterns). The lower third molar (M₃) of the expected ancestral phenotype has five cusps that are not arranged in a Y pattern, and a reduction of the lingual half of the molar relative to its buccal half (Fig. 2). This reconstruction of the dental morphology of the [N-MH]LCA is generally consistent with all 12 phylogenies, one of which is shown in Fig. 2 (the one that includes the SH group as a pre-Neanderthal population, that uses the earlier date of 1 Ma for the [N-MH]LCA, and which dates H. erectus to 1 Ma; Fig. 3 and Figs. S4 and S5 are based on the same phylogeny).

Fig. 2.

Dental morphology of the [N-MH]LCA. Estimated ancestral morphologies are represented as TPS-grids showing the deformation from the base of the phylogeny into the estimated ancestral shape. Photographs (with landmarks represented as blue points) show examples of typical morphologies found in the Neanderthal lineage (NEA) and in the modern human (H. sapiens) lineage (SAP). Mesial margins are represented to the left, distal margins to the right, buccal margins to the top, and lingual margins to the bottom of the figure. Not to scale.

Fig. 3.

Projection of the simplified hominin phylogenetic tree (represented by lines connecting terminal species and nodes) with terminal species (large dark-gray filled circles), ancestor shapes (small light-gray filled circles) and candidate species (open circles) into the upper and lower first molar morphospaces. Confidence ellipses (95%) around the three nodes discussed in the text are represented with the three ancestors located in their centers (solid line, confidence ellipse of the [N-MH]LCA; dashed line, confidence ellipse of the [H]LCA; dotted line, confidence ellipse of the [P-H]LCA). (A) Upper first molar morphospace. (B) Lower first molar morphospace. Plots corresponding to premolars and second and third molars are provided in Fig. S4. Note that confidence ellipses in these graphs correspond only to the first and second principal components (PCs), whereas P values in Table 1 are based on all of the PCs. Phylogeny: BOI, P. boisei; ERE, H. erectus; NEA, H. neanderthalensis; ROB, Paranthropus robustus; SAP, H. sapiens; SH, SH group. Candidates: AFA, A. afarensis; AFR, A. africanus; ANT, H. antecessor; ERG, H. ergaster; GEO, Homo georgicus; HAB, H. habilis; HEI, H. heidelbergensis.

Our results reveal that no known species of the relevant age is statistically consistent with the expected morphology of the [N-MH]LCA, even given the considerable uncertainty of the reconstructed ancestral morphologies. Graphical reconstructions of the nodes in dental morphospace are shown in Fig. 3 and Fig. S4, along with confidence ellipsoids that show the associated uncertainties that arise from the stochastic nature of the evolutionary process. The size of each confidence ellipsoid is directly related to the lengths of the branches connected to the node. At least two postcanine teeth fall outside the 95% confidence intervals of the [N-MH]LCA reconstruction for all candidate species under all 12 phylogenetic scenarios (Table 1). Furthermore, taxa such as H. heidelbergensis have several postcanine teeth that are morphologically consistent with the [N-MH]LCA only in one or two phylogenetic scenarios (Table S3). The addition of other European fossils to the H. heidelbergensis sample, such as those from Mauer and Petralona, could alter this result, but this is unlikely because they are similar (not only in dental, but also in cranial and mandibular morphology) to the Arago specimens relative to other taxa in our analysis (2, 3, 9, 18). Our results demonstrate that morphological evidence for any of these taxa being the [N-MH]LCA is, at best, weak. For these species to be credible ancestors, the age of the Neanderthal–modern human split would have to be radically different from the range we considered or selection on the dentition would have to have caused the ancestor to radically deviate from the expectations of Brownian motion evolution. In addition to ruling out these candidate species, our results indicate that all three European candidate groups are statistically closer to the Neanderthal lineage than to the modern human lineage (Table 2 and Table S4), which suggests that European hominins have had significant dental affinities with Neanderthals for almost 1 My (the oldest species, Homo antecessor, has been dated to ca. 950 ka; ref. 19). African Homo ergaster and Asian H. erectus do not share this European affinity, but they have nearly equal dental affinities with Neanderthals and with modern humans (Table 2 and Table S4). H. ergaster was evaluated as a [N-MH]LCA candidate because it is the youngest African species in our sample, despite it being considerably older than this ancestor is expected to be. Interestingly, H. ergaster fits the expected ancestral phenotype better than some of the younger candidate species (Table 1). Asian H. erectus was also evaluated as an ancestral candidate because the fossil specimens included in our study lived during the right time interval (0.5–1 Ma), but it has generally a worse fit with the [N-MH]LCA than H. ergaster.

Table 1.

Procrustes distances between expected ancestral morphologies and candidate species

	Procrustes distance* (P value)†
	P3	P4	M1	M2	M3	P3	P4	M1	M2	M3
[N-MH]LCA
SH group	0.045 (<0.001)	0.027 (0.072)	0.024 (0.373)	0.034 (0.710)	0.049 (0.392)	0.051 (0.042)	0.042 (0.103)	0.023 (0.432)	0.031 (0.169)	0.044 (0.074)
H. heidelbergensis	0.046 (0.785)	0.061 (0.147)	0.052 (0.006)	0.072 (0.001)	0.059 (0.255)	0.057 (0.310)	0.036 (0.487)	0.025 (0.991)	0.034 (0.194)	0.042 (0.980)
H. antecessor	0.038 (0.234)	0.037 (0.280)	0.051 (0.001)	0.058 (0.001)	N/A	0.081 (0.036)	0.046 (0.259)	0.030 (0.934)	0.048 (0.016)	0.041 (0.360)
H. erectus	0.031 (0.999)	0.023 (0.805)	0.028 (0.867)	0.053 (0.015)	0.067 (0.030)	0.084 (0.039)	0.043 (0.622)	0.029 (0.733)	0.038 (0.628)	0.038 (0.678)
H. ergaster	0.051 (0.206)	0.044 (0.920)	0.053 (0.013)	0.077 (0.001)	N/A	0.061 (0.250)	0.032 (0.608)	0.038 (0.913)	0.034 (0.670)	0.042 (0.811)
[H]LCA
H. georgicus	0.061 (0.456)	0.028 (0.580)	0.050 (0.918)	0.049 (0.307)	0.091 (<0.001)	0.059 (0.009)	0.030 (0.988)	0.033 (0.983)	0.036 (0.955)	0.050 (0.583)
H. ergaster	0.044 (0.394)	0.040 (0.995)	0.050 (0.060)	0.060 (0.170)	N/A	0.057 (0.256)	0.035 (0.694)	0.035 (0.894)	0.034 (0.765)	0.037 (0.805)
H. habilis	0.041 (0.627)	0.037 (0.596)	0.026 (0.868)	0.032 (0.300)	0.039 (0.602)	0.036 (0.813)	0.046 (0.497)	0.021 (0.965)	0.030 (0.919)	0.034 (0.971)
[P-H]LCA
H. habilis	0.028 (0.847)	0.027 (0.944)	0.024 (0.948)	0.029 (0.751)	0.023 (0.953)	0.046 (0.569)	0.040 (0.724)	0.022 (0.990)	0.031 (0.965)	0.037 (0.947)
A. africanus	0.031 (0.970)	0.029 (0.647)	0.029 (0.732)	0.031 (0.934)	0.056 (0.410)	0.043 (0.932)	0.026 (0.992)	0.025 (0.925)	0.035 (0.745)	0.043 (0.573)
A. afarensis	0.041 (0.962)	0.033 (0.868)	0.030 (0.688)	0.055 (0.282)	0.049 (0.509)	0.055 (0.951)	0.031 (0.960)	0.037 (0.664)	0.035 (0.729)	0.037 (0.642)

Bold represents P < 0.05 in all phylogenetic scenarios (candidate species is incompatible with ancestry in the specified tooth). N/A, not applicable (no M³ of H. antecessor and H. ergaster are included in the sample).

^*Mean Procrustes distance obtained in the 12 evaluated phylogenetic scenarios.

^†Highest P value observed in the 12 different phylogenetic scenarios. P values represent the probability that a fossil species falls within the expected distribution of an ancestral node under the assumption of a Brownian motion model of evolution.

Table 2.

Morphological affinities of candidate species with Neanderthals and with modern humans

Candidate	P3	P4	M1	M2	M3	P3	P4	M1	M2	M3
SH group	NEA	NEA	NEA	—	SAP	NEA	SAP	NEA	NEA	NEA
H. heidelbergensis	SAP	NEA	—	NEA	NEA	SAP	NEA	NEA	NEA	NEA
H. antecessor	NEA	NEA	NEA	NEA	N/A	SAP	NEA	SAP	NEA	NEA
H. erectus	SAP	NEA	SAP	NEA	NEA	SAP	NEA	SAP	SAP	SAP
H. ergaster	SAP	NEA	SAP	NEA	N/A	SAP	SAP	SAP	NEA	NEA

Affinities based on Procrustes distances (Table S4). N/A, not applicable; NEA, higher affinity with Neanderthals; SAP, higher affinity with modern humans; —, higher but not significant affinity with Neanderthals.

Our method identified compatible candidates for other nodes in the phylogeny (Fig. S5). Our data show that H. ergaster and, especially, H. habilis fit the expected morphology of the LCA of the genus Homo ([H]LCA) at all tooth positions in one or more of the 12 phylogenies (Table 1 and Table S3). Similarly, both Australopithecus afarensis and Australopithecus africanus fit the expected morphology of the LCA of Paranthropus and Homo ([P-H]LCA) at all tooth positions in most or all of the phylogenetic scenarios (Table 1 and Table S3). Despite being age-inappropriate, we also evaluated H. habilis as a candidate for the [P-H]LCA because of its suggested affinities with Australopithecus species (20). Interestingly, it has even stronger compatibility with the expected morphology of this ancestor than do A. afarensis and A. africanus (Table 1 and Table S3). The fact that several candidates are compatible with deeper nodes is not surprising because the long branch lengths leading to older nodes result in broader confidence ellipses than at shallower nodes with shorter branches (21). Regardless of uncertainty, Procrustes distances show that the fit between candidates and their corresponding nodes is substantially better at the older nodes than at the [N-MH]LCA (Table 1). The main exception is the SH group, which has some teeth that are comparatively close to the [N-MH]LCA in terms of Procrustes distances, but which is incompatible as an ancestor because some of its teeth are strongly derived and have a very low probability of conforming to the expected ancestral morphology.

Discussion

Our results demonstrate that, even though there are plausible candidate species for deep nodes of the hominin phylogeny (A. afarensis, A. africanus, and H. habilis for the [P-H]LCA and H. habilis and H. ergaster for the [H]LCA), we still do not have a good fossil candidate for the phenotypic LCA of Neanderthals and modern humans. This result is striking because confidence intervals of ancestral reconstructions are very broad, which means that existing candidates are very incompatible. Species falling outside the confidence intervals for a given node can be excluded from ancestry with high confidence presuming the model of Brownian motion evolution is applicable. The use of several different phylogenetic topologies in our study makes the rejection of these candidates even stronger.

Even though none of the candidate species we evaluated were compatible with the [N-MH]LCA, note that our study did not include African populations of H. heidelbergensis or African specimens dated to ca. 1 Ma. These groups deserve scrutiny as potential ancestors, especially because they lived at the right time and their cranial morphological affinities are ambiguous (2, 18, 22–24). Middle Pleistocene fossils from Africa (sometimes referred to as Homo rhodesiensis) have been considered by some as part of the lineage leading to H. sapiens (22) and by others as belonging to the ancestral species of Neanderthals and modern humans (3, 18, 23, 24). African fossils dated to ca. 0.5–1 Ma thus merit continued study and are currently the most promising source of candidates for the [N-MH]LCA.

The lack of a plausible [N-MH]LCA candidate among European groups and their dental affinity with Neanderthals back to 1 Ma suggests three possible scenarios (covered in the next three sections), which are not necessarily mutually exclusive.

Neanderthals and the [N-MH]LCA Could Be Primitive in Their Dental Morphology.

One possible explanation of our results is that the constellation of characters shared by European Pleistocene fossils represents the primitive condition for the clade Homo neanderthalensis–H. sapiens and that the true ancestral phenotype was closer to the Neanderthal condition than expected under a Brownian motion model. This scenario would require relative stasis in the evolution of dental morphology in the Neanderthal lineage and either neutral or directional evolution in the H. sapiens lineage. If this were the case, our ancestral reconstruction method would fail to recover the correct morphology because it assumes similar, nondirectional evolutionary changes in the lineages stemming from a node and subtending it. We have tested and rejected this possibility elsewhere (25), and so consider this scenario unlikely. Whereas some authors have suggested that Neanderthals are morphologically more primitive than modern humans (26), quantitative analysis of evolutionary patterns in these lineages are consistent with a Brownian process of nonadaptive, neutral evolution in both dental (25) and cranial features (27). Moreover, detailed, comprehensive evaluations of postcanine dental morphology have revealed that several dental traits shared by European late Early and Middle Pleistocene populations and classic Neanderthals are, in fact, derived (28–31).

The suite of dental traits shared by European hominins suggests that a separate European clade with Neanderthal affinities existed since at least 1 Ma. This does not necessarily imply a continuity between all European Pleistocene populations during the last 1 My (32), but it points to all these taxa being part of a Neanderthal phenotypic clade separate from the lineage leading to modern humans. It has been suggested that this European group could have been composed of successive migrant populations who shared their common phenotypes via gene flow (ref. 33, see also refs. 15 and 34), resulting in a lack of linearity in the pattern of acquisition of Neanderthal traits (33). It is worth noting that extensive hybridization would hamper ancestral reconstruction because the expected morphological differences would be attenuated by gene flow between diverging populations.

Molecular Divergence Times May Be Underestimated.

The paleontological estimate of the divergence time between Neanderthals and modern humans at ca. 1 Ma is substantially older than most molecular estimates, which set the divergence at less than 500 ka (11, 15). The divergence between Neanderthals and modern humans has also been dated to 300–400 ka using neutrally evolving cranial measurements (35), an estimate that is highly dependent on assumptions and parameters that are not known without error, but which is broadly consistent with the youngest molecular clock estimates. In light of the young molecular and cranial dates, several analyses have dismissed the early divergence model of Neanderthal origins (1, 3, 11). However, a late divergence model has been recently challenged on the basis of slower mutation rates that result in earlier split times in ape and human evolution (refs. 36–38 but see also refs. 39 and 40). Attempts to reconcile paleontological and molecular dates have to face the fact that alternative molecular estimates for the timing of the [N-MH]LCA vary considerably, with confidence intervals ranging from 300 to 850 ka (11, 37). Our results are however incongruous with even the lowest boundary of molecular confidence intervals, and thus lend support to the early divergence model and suggest that molecular estimates of divergence time can be underestimated.

Phenotypic Divergence and Speciation Could Be Decoupled.

A third possible alternative that can conciliate molecular and paleontological observations is that phenotypic divergence in dental traits between Neanderthals and modern humans significantly predated the completion of speciation. This scenario could explain why some European Pleistocene populations appear to be derived in dental traits but still show primitive cranial and postcranial features. Such mosaicism is particularly evident in H. antecessor’s combination of some derived Neanderthal dental features (29–31) and generally plesiomorphic nondental traits (at least for the clade H. neanderthalensis-H. sapiens; refs. 41–43), and in SH’s cranial morphology, which is transitional to the Neanderthal form (10), combined with its strongly derived dental phenotype (12).

It is most likely that the large discrepancy between molecular dates and phenotypic differentiation results from the wide confidence intervals of the molecular estimates and from the uncertainty associated with dating fossil materials. Regardless, our results indicate that dental morphology developed derived features as early as almost 1 Ma in the Neanderthalization process (12), raising interesting questions about which evolutionary forces may have caused this early dental differentiation with respect to other traits and over what time interval occurred. The evolutionary mechanisms behind this process are likely to consist of a mosaic of random factors (such as genetic drift) and nonrandom factors (such as lineage-specific patterns of environmental selection and rates of evolution of different traits; ref. 44). New paleontological, molecular, and quantitative phenotypic studies are needed to elucidate the exact nature of these factors.

Conclusions

Our study presents a statistical framework for addressing questions about ancestry and the relationships of particular fossils that can serve as a model for making paleoanthropological hypotheses more testable. The quantitative approach we propose offers an objective means to test hypotheses that capitalize on the comparatively rich dental fossil record and address apparently unresolvable debates about hominin phylogeny. The same approach can be applied to other skeletal parts via 2D and 3D quantifications of morphology, and it can be extended to include models of evolution other than Brownian motion processes should there be evidence that they are appropriate. Our approach thus provides a promising and extensible statistical context for evaluating fossil remains of uncertain phylogenetic position.

Materials and Methods

Materials.

Our study analyzed the morphology of 1,200 hominin teeth representing their 10 postcanine tooth positions (upper and lower premolars and molars) from 13 different species/morphotypes (Tables S1 and S2). We chose to split controversial named taxa for which all teeth are represented in our samples to evaluate their ancestral status independently on their own merits, but we lumped taxa when their dentitions were incompletely represented so as to provide more statistical power (e.g., Paranthropus aethiopicus was included in Paranthropus boisei and H. rudolfensis in H. habilis). The only exceptions were H. ergaster and H. antecessor, which were retained as separate candidate species despite their lack of M³ due to their strong candidacy as [N-MH]LCA. Although other taxonomic arrangements are plausible (e.g., merging H. erectus, H. ergaster, and Dmanisi fossils under H. erectus sensu lato), the purpose of our study is not to address taxonomic questions, but to evaluate the plausibility that previously recognized taxa occupy ancestral positions. In this regard, the most informative approach is a detailed evaluation of morphologically, geographically, and chronologically cohesive groups.

Geometric Morphometrics.

Dental morphology was quantified using 2D landmarks and sliding semilandmarks (45) digitized from photographs of the occlusal surface of premolars and molars. Depending on the tooth, 4–8 landmarks and 30–40 semilandmarks were used (Fig. S2 and S3). Coordinates were digitized using tpsDig2 software (46). Orientation of dental specimens was standardized such that the occlusal surface was parallel to the lens of the camera. Unworn and moderately worn teeth were included in our evaluation, whereas severely worn teeth were excluded. A generalized Procrustes analysis (47) and a principal components analysis (48) were carried out using the mean configurations of the species (Dataset S1) to provide shape variables. Ancestral state calculations were carried out on these variables in the principal components space (Fig. 3 and Fig. S4) using all principal components (5).

Ancestral State Reconstruction.

The GLM method for estimating ancestral traits (6, 49, 50) was used, implemented in Mathematica by one of us (51). As is standard practice (52), our implementation of ancestral state reconstruction assumes that the traits are evolving under a Brownian motion model of the sort that results either from genetic drift or from selection whose direction and magnitude varies randomly through time and across space. Our previous study indicated that hominin dental evolution is consistent with a Brownian motion model, although considerable stabilizing selection has been identified in first molars and slight directional trends in lower third premolars and upper third molars (25). Estimates of ancestral states are not badly affected by stabilizing processes, only by long-term unidirectional selection that occurs in parallel in different lineages (21, 53).

Ancestral node reconstruction identifies the single most likely ancestral condition and a multivariate confidence ellipsoid (95%) containing other plausible but less probable ancestral phenotypes (6, 50, 54). The likelihood distribution of the ellipsoids was used to calculate the probability that fossils of appropriate age could feasibly occupy those nodes (SI Text).

The reconstructed ancestral shape variables were converted back into landmark configurations (55), after which the Procrustes distances of candidate species to the expected ancestor were calculated. Procrustes distance is a measure of shape difference between superimposed conformations of landmarks calculated as the square root of the sum of the squared differences between the positions of homologous landmarks. The species that falls within the confidence ellipsoid and has the smallest Procrustes distance to the hypothetical ancestor can be considered the best candidate to occupy the ancestral position. Procrustes distance was also used to calculate the affinity of candidate species with Neanderthals and modern humans taking into account the phylogenetic topology (SI Text).