Morphometric analysis of molars in a Middle Pleistocene population shows a mosaic of ‘modern’ and Neanderthal features

Abstract

Previous studies of upper first molar (M¹) crown shape have shown significant differences between Homo sapiens and Homo neanderthalensis that were already present in the European Middle Pleistocene populations, including the large dental sample from Atapuerca-Sima de los Huesos (SH). Analysis of other M¹ features such as the total crown base area, cusp proportions, cusp angles and occlusal polygon have confirmed the differences between both lineages, becoming a useful tool for the taxonomic assignment of isolated teeth from Late Pleistocene sites. However, until now the pattern of expression of these variables has not been known for the SH sample. This fossil sample, the largest collection from the European Middle Pleistocene, is generally interpreted as being from the direct ancestors of Neanderthals, and thus is a reference sample for assessing the origin of the Neanderthal morphologies. Surprisingly, our study reveals that SH M¹s present a unique mosaic of H. neanderthalensis and H. sapiens features. Regarding the cusp angles and the relative occlusal polygon area, SH matches the H. neanderthalensis pattern. However, regarding the total crown base area and relative cusps size, SH M¹s are similar to H. sapiens, with a small crown area, a strong hypocone reduction and a protocone enlargement, although the protocone expansion in SH is significantly larger than in any other group studied. The SH dental sample calls into question the uniqueness of some so-called modern traits. Our study also sounds a note of caution on the use of M¹ occlusal morphology for the alpha taxonomy of isolated M¹s.

Introduction

Previous analyses on the shape and metric features of M¹s in the genus Homo have revealed significant differences between Homo sapiens and Homo neanderthalensis lineages (Bailey, 2002a, 2004; Bermúdez de Castro et al. 2007; Gómez-Robles et al. 2007, 2011a; Quam et al. 2009). Since M¹ is considered to be the least variable and most diagnostic of the upper molars (Dahlberg, 1945; Bailey, 2002b; Hlusko et al. 2004), its anatomical details are expected to reflect the underlying genetic signal more strongly and thus it potentially provides evolutionary information. This circumstance is of particular utility for the taxonomic identification of isolated teeth (e.g. Smith, 1976; Morris, 1986; Wood et al. 1988; Klein, 1999; Tattersall, 1999; Bailey, 2002a, 2004; Harvati et al. 2003). Typical Neanderthal M¹s have been described as strongly skewed rhomboids with bulging and protruding hypocones (Bailey, 2004). This morphotype has been also identified in the European Early and Middle Pleistocene populations (Bailey, 2004; Quam et al. 2009), including the large European Middle Pleistocene sample from Atapuerca-Sima de los Huesos (SH) (Gómez-Robles et al. 2007, 2011a), confirming the early appearance of some so-called Neanderthal features in Europe (Gómez-Robles et al. 2007, 2011a; Martinón-Torres et al. 2007, 2012). In addition, the pattern of expression of other typical Neanderthal M¹ features such as the relative cusp proportions (e.g. a small metacone and large hypocone) or the small size of the relative occlusal polygon has been also identified in the small Homo heidelbergensis sample analyzed by Bailey (2004) and Quam et al. (2009). However, to date, the pattern of expression of these features was unknown for the SH collection. The SH sample is currently the largest hominin collection worldwide for the Middle Pleistocene and provides an exceptional opportunity to study intra-population variability (e.g. Arsuaga et al. 1997a; Lorenzo et al. 1998; Bermúdez de Castro et al. 2004; Gómez-Olivencia et al. 2007; Gómez-Robles et al. 2008, 2009, 2011b,c; Prado-Simón et al. 2012). This hominin population has been assigned to H. heidelbergensis, assuming the latter to be a chronospecies of the European Neanderthal lineage (Arsuaga et al. 1991, 1993, 1997b). However, the precise relationship of the SH hominins with the Neanderthal lineage is still under debate (e.g. Stringer, 1993; Tattersall & Schwartz, 2006, 2009; Hublin, 2009). On dental grounds, the SH hominins present all of the morphological traits that are usually considered typical of H. neanderthalensis (Bermúdez de Castro, 1988, 1993; Martinón-Torres et al. 2012), although in some features they appear even more derived than classic Neanderthals (Bermúdez de Castro & Nicolás, 1995, 1996; Gómez-Robles et al. 2007, 2008, 2011a,b, 2012; Prado-Simón et al. 2012; Martinón-Torres et al. 2013). In this context, it could be expected that SH M¹s would also coincide with H. neanderthalensis in their pattern of cusp proportions, cusp angles, occlusal polygon and crown size. In this study, we aim to contribute to the understanding of the evolution of M¹ morphology by studying these features in the SH M¹ sample.

Materials and methods

Teeth with a wear degree higher than category 3 (Molnar, 1971) were excluded from the SH M¹ sample (n = 29; Martinón-Torres et al. 2012), resulting in a sample of 16 M¹s. When antimeres were present, the data from the right side were used, resulting in a minimum number of 12 individuals (AT-20, AT-26, AT-139, AT-196, AT-587, AT-812, AT-959, AT-2076, AT-3177, AT-3178, AT-4317, AT-5899). The SH results were compared with data from other Homo fossil and modern human samples obtained from the published literature. Tables 1 and 2 list the specimens and source of the M¹ comparative sample. The H. heidelbergensis sample does not include the SH specimens, which are analyzed and compared separately. Measurements were taken by one of us (P.S.). For that purpose we employed standardized images of the occlusal surface of the molars. Photographs were taken with a Nikon D1H digital camera fitted with an AF Micro-Nikkor 105 mm, f/2.8D attached to a Kaiser Copy Stand kit RS-1 with grid baseboard, column, and adjustable camera arm. For maximum depth of field, we used an aperture of f/32, and the magnification ratio was adjusted to 1 : 1. A scale was placed parallel to and at the same distance from the lens as the occlusal plane. Images were measured with fiji image analysis software (Schindlein et al. 2012). Each picture was calibrated three times and, following Bailey (2004), we used the average of these calibrations as the final calibration, and set a scale bar in the picture.

Table 1.

Comparative sample for the absolute and relative TCBA and cusp areas

Sample	Label	n	Source
Early Homo	EaHOMO	18	Wood & Engleman (1988); Wood (1991); Quam et al. (2009)
Homo erectus	ERE	5	Wood & Engleman (1988); Quam et al. (2009)
Homo antecessor	ANT	5	Gómez-Robles (2010)
Homo heidelbergensis	HEI	4	Bailey (2004); Quam et al. (2009)
Sima de los Huesos	SH	12	This study
Homo neanderthalensis	NEA	21	Bailey (2004); Arsuaga et al. (2007); Quam et al. (2009)
Middle Paleolithic H. sapiens	MP HSAP	7	Quam et al. (2009)
European Upper Paleolithic H. sapiens	UP HSAP	15	Bailey (2004); Arsuaga et al. (2007); Quam et al. (2009)
Contemporary Homo sapiens	HSAP	59	Bailey (2004); Quam et al. (2009)
Pooled fossil later Homo1	FLHOMO	55	Quam et al. (2009)
Pooled later Homo2	LHOMO	114	Quam et al. (2009)

¹Includes all late Homo fossils except for Homo sapiens.

²Includes all late Homo fossils.

Table 2.

Comparative sample for the cusp angles and occlusal polygon areas

Sample	Label	Source
Homo antecessor	ANT	Gómez-Robles (2010)
Homo erectus	ERE	Bailey (2004)
Homo neanderthalensis	NEA	Bailey et al. (2008)
Middle Paleolithic Homo sapiens	MP HSAP	Bailey et al. (2008)
Upper Paleolithic Homo sapiens	UP HSAP	Bailey et al. (2008)
Contemporary modern human sample	HSAP	Bailey (2004)

We measured the crown base area, individual cusp base areas, cusp angles and occlusal polygon area. We followed the methodology already employed by other authors in similar studies (Wood & Engleman, 1988; Bailey, 2004; Moggi-Cecchi & Boccone, 2007; Quam et al. 2009; Gómez-Robles et al. 2011a). We first traced the tooth perimeter. Individual cusp base areas were measured by following the primary fissures separating the cusps (Fig. 1). If the fissure was partially obscured, its course was estimated following the direction of the visible part of fissure (Bailey, 2004) and with the help of other dental features as anatomical reference (Biggerstaff, 1969). When necessary, the correction of interproximal wear was made according to Wood & Engleman (1988). When the metaconule or C5 was present, its area was measured and divided evenly between the adjacent main cusps (Quam et al. 2009; Gómez-Robles et al. 2011a). When a crista obliqua was present we followed the recommendations of previous authors and continued the course of the central groove between the protocone and the metacone (Gómez-Robles et al. 2011a). Cusps were considered equal in size if their relative areas did not differ by more than 1.0%. Total crown base area (TCBA) was calculated as the sum of individual cusp areas. Relative areas of each cusp were calculated by dividing the base area of each cusp by the total crown base area.

Fig. 1.

Upper first molar from Sima de los Huesos site (left) showing the separation of the different cusp areas (middle), the occlusal polygon and the cusp angles (right) at the protocone (angle A), paracone (angle B), metacone (angle C) and hypocone (angle D). Mesial: top of the figure; distal: bottom; buccal: right; lingual: left. White bar: 5 mm.

Cusp angles were measured by connecting the cusp apices of major cusps (Bailey, 2004). The position of the cusp apices on the images was estimated by checking their position in the original fossil tooth (Fig. 1). The occlusal polygon area was measured as the area defined by the lines connecting the cusp apices (Bailey, 2004). Relative occlusal polygon area, as an expression of the internal compression of tooth cusps, was calculated by dividing the occlusal polygon area by the total crown base area (Bailey, 2004).

Not all the variables could be scored in all the teeth (e.g. if a cusp is partially broken we cannot calculate the crown base area but we may be still able to measure angles or the occlusal polygon). Each tooth was measured twice and the average was used in the analyses. The differences between the two relative cusp measures were employed to calculate the intraobserver error, which was lower than 1.2% in all cases. For the interobserver error, we recorded the differences between the relative cusp areas of 7 M¹s measured by P.S. and J.M.B d C., which was less than 2.2% in all cases. Previous analysis of the M¹s morphology demonstrated that the error tied to the location of the cusp tips for the occlusal polygon is very low, with interobserver correlations higher than 0.97 (Gómez-Robles et al. 2007, 2011a). Statistical analyses were computed with past 2.14 (Hammer et al. 2001).

Linear correlations have been calculated by the Pearson coefficient among absolute cusp area values, among relative cusp area values, and between crown base area and each cusp areas. The relationship between crown base area and cusp areas was also analyzed by log-log reduced major axis regression, to consider the allometric patterns. A principal component analysis (PCA) based on correlation coefficients was also computed using the four cusp areas to verify the general contribution of the variables to the whole variation.

To examine the differences among SH and the comparative samples for the cusps and occlusal polygon areas and angles we performed a F-test (to test the equality of the variance) followed by a one-sample t-test from parameters (mean, n and variance) since the individual values of the comparative sample were not available. A two-sample t-test was also employed to determine whether the size difference between relative cusp sizes in SH was significant. Differences were considered significant when P ≤ 0.05.

Results

Intragroup variation

Table 3 provides the mean and the standard deviation (SD) for the TCBA and absolute cusp areas for the M¹ samples. The absolute values of the cusp areas are all correlated, with correlation coefficients ranging from 0.61 (protocone vs. hypocone) to 0.90 (metacone vs. paracone; Table 4). On the other hand, correlations between relative values are not significant. Total crown base area is correlated with the absolute cusp areas (with coefficients ranging from 0.80 for hypocone to 0.95 for paracone and metacone), but not with the relative cusp areas (Table 5).

Table 3.

Mean and standard deviation for the absolute and relative total crown base area and cusp areas of M¹s. The range of variation is included for the SH sample

	Absolute (mm2)					Relative (%)
Sample	TCBA (mm2)	Protocone	Paracone	Metacone	Hypocone	Protocone	Paracone	Metacone	Hypocone
SH (12)	107.7 ± 11.5 (84.1–129.8)	37.1 ± 4.3 (30.9–46.0)	26.9 ± 3.1 (20.8–32.0)	22.4 ± 3.2 (15.8–28.9)	21.1 ± 2.0 (16.7–24.9)	34.5 ± 1.6 (32.0–36.7)	25.0 ± 1.0 (23.6–26.5)	20.8 ± 1.1 (18.8–22.2)	19.7 ± 1.2 (17.7–21.3)
NEA (21)	112.3 ± 16.6	33.7 ± 6.1	28.8 ± 4.2	22.9 ± 3.9	26.8 ± 5.1	29.9 ± 2.4	25.8 ± 2.1	20.6 ± 1.8	23.7 ± 2.1
HEI (4)	115.5 ± 17	34.8 ± 6.8	28.3 ± 4.8	24.2 ± 4.8	28.2 ± 3	31.1 ± 3.7	24.8 ± 0.9	20.1 ± 2.6	24 ± 1.6
ANT (5)	123.8 ± 9	37.5 ± 2.6	29.4 ± 1.4	26.3 ± 2.4	30.6 ± 3.1	30.3 ± 0.7	23.8 ± 0.7	21.2 ± 0.7	24.7 ± 0.8
ERE (5)	115.5 ± 6.8	34.5 ± 3	28.7 ± 2.7	26.5 ± 3.2	25.8 ± 1.6	29.9 ± 2.2	24.9 ± 2.3	22.9 ± 1.7	22.3 ± 0.4
MP HSAP (7)	111.3 ± 12.7	34.7 ± 2	27.5 ± 2.4	23.7 ± 3.7	25.8 ± 7.8	31.3 ± 2.3	24.8 ± 1.6	21.3 ± 2.5	22.8 ± 5
UP HSAP (15)	99.6 ± 10.2	31.7 ± 3.6	25.2 ± 2.4	22.6 ± 3.3	20.3 ± 3.7	31.8 ± 1.5	25.7 ± 2.3	22.4 ± 1.7	20.1 ± 3
HSAP (59)	96.6 ± 14.0	29.9 ± 4.8	24.8 ± 3.7	22.1 ± 3.9	19.7 ± 3.8	31 ± 2	25.8 ± 2.1	22.9 ± 1.8	20.4 ± 2.5
EaHOMO (18)	134.9 ± 15.3	39.5 ± 5.3	31.1 ± 3.5	33.5 ± 4.6	30.8 ± 5.4	29.3 ± 2.1	23.2 ± 2.2	24.8 ± 2	22.8 ± 2.4
FLHOMO (55)	109 ± 14.5	33.3 ± 4.7	27.5 ± 3.6	23.5 ± 3.8	24.8 ± 5.5	30.6 ± 2.2	25.5 ± 2.1	21.5 ± 2	22.5 ± 3.1
LHOMO (114)	102.2 ± 15.4	31.4 ± 5	26 ± 3.8	22.7 ± 3.9	22 ± 5.3	30.8 ± 2.1	25.6 ± 2.1	22.2 ± 2	21.4 ± 3

Abbreviations are defined in Tables 1 and 2.

Table 4.

Correlation matrix between cusp values

	Proto	Para	Meta	Hypo
Absolute values
Proto		0.002	0.001	0.037
Para	0.8		0	0.006
Meta	0.82	0.9		0.01
Hypo	0.61	0.74	0.71
Relative values
Proto		0.103	0.088	0.161
Para	−0.49		0.813	0.541
Meta	−0.51	0.08		0.305
Hypo	−0.43	−0.2	−0.32

Above diagonal: p values; below diagonal: R.

Table 5.

Correlation between TCBA and cusp values

			RMA log-log
	R	P <	Slope	95%CI
Absolute
Proto	0.92	0	1.04	0.58–1.30
Para	0.95	0	1.1	0.76–1.26
Meta	0.95	0	1.34	1.10–1.68
Hypo	0.8	0.002	0.92	0.49–1.28
Relative
Proto	−0.09	0.786
Para	0.11	0.734
Meta	0.5	0.096
Hypo	−0.43	0.159

The allometric analysis of the four absolute cusp areas on the total crown base area cannot reject isometry for the protocone, paracone, and hypocone (Table 5). In these cases the slope of the log-log regression is between 0.92 and 1.10, with a wide confidence interval because of the small sample size. In contrast, the metacone displays a slope of 1.34, with a confidence interval barely excluding isometry (1.10–1.68) and suggesting positive allometry (Table 5).

Principal component analysis (Fig. 2) shows a strongly dominant first component (82% of the total variance) associated with size. A second component (10%) separates two subgroups of six specimens each by the inverse contribution of hypocone and protocone (i.e. large hypocone and small protocone vs. small hypocone and large protocone). However, because of the small sample size this component is not stable according to the Jolliffe cutoff (Jolliffe, 1986) and broken stick (Jackson, 1993) criteria, and must be interpreted with caution. A log-log regression between the absolute values of protocone and hypocone shows a slope with a value of 0.88, but with a confidence interval that cannot discard isometry (0.35–1.39).

Fig. 2.

Principal component analysis of the individual absolute cusp areas, showing the first two principal components, the variance explained by each, and the vectors of the variable loadings. The first component (PC1) is associated with increase of all the variables (general size). The second component is mostly associated with the inverse proportions of the hypocone and the protocone.

Total crown base area (TCBA) and absolute and relative cusp size in SH M¹s

The TCBA in the SH sample is small and only larger than the contemporary and the Upper Paleolithic H. sapiens samples (Table 3). It is significantly different from the early Homo, Homo antecessor, and contemporary H. sapiens samples (Table 6).

Table 6.

Statistical comparison of absolute and relative cusp areas

M1	NEA	HEI	ANT	ERE	MP HSAP	UP HSAP	HSAP	EaHOMO	FLHOMO	LHOMO
SH TCBA	NS	NS	**	NS	NS	NS	**	**	NS	NS
SH Protocone	NS	NS	NS	NS	NS	**	**	NS	**	**
SH Paracone	NS	NS	NS	NS	NS	NS	NS	**	NS	NS
SH Metacone	NS	NS	*	*	NS	NS	NS	**	NS	NS
SH Hypocone	**	*	**	**	NS	NS	NS	**	**	NS
SH Protocone rel	**	**	**	**	**	**	**	**	**	**
SH Paracone rel	NS	NS	**	NS	NS	NS	*	**	NS	NS
SH Metacone rel	NS	NS	NS	**	NS	**	**	**	*	**
SH Hypocone rel	**	**	**	**	NS	NS	NS	**	**	**

*Significant at P < 0.05.**Significant at P ≤ 0.01. NS, nonsignificant. Abbreviations are defined in Tables 1 and 2.

The absolute protocone area in SH is large, close to H. antecessor and early Homo values (Table 3). It is significantly different from the Upper Paleolithic and contemporary H. sapiens samples, the pooled later Homo and the pooled fossil later Homo samples. The relative protocone area is significantly larger than that of any of the comparative groups (Table 6).

The mean of the paracone size of SH M¹s is, in absolute terms, smaller than that of any other group except for the Upper Paleolithic and the contemporary H. sapiens groups and the pooled later Homo sample (Table 3). The differences are only significant when compared with the early Homo group (Table 6). The SH relative paracone area is significantly larger than that of H. antecessor and the early Homo groups, and significantly smaller than that of the contemporary H. sapiens group.

The SH M¹s absolute metacone area is small and only larger than that of the contemporary H. sapiens group (Table 3). The differences are significant when compared with the early Homo group, H. antecessor and H. erectus (Table 6). The relative metacone area is only larger than that of H. heidelbergensis and H. neanderthalensis and it shows significant differences with all groups except for H. antecessor, H. heidelbergensis, H. neanderthalensis and the Middle Paleolithic H. sapiens samples.

Regarding the SH M¹s hypocone, its absolute size is smaller than that of any other group except for the Upper Paleolithic and the contemporary H. sapiens samples (Table 3). It has significant differences with all the studied groups except for the pooled later Homo sample and the three H. sapiens samples (Table 6). The relative hypocone area of SH M¹s is the smallest of all the studied groups, and it differs significantly from all of them except the three H. sapiens groups.

The relative size sequence of SH M¹s is protocone > paracone > metacone > hypocone. This is the same pattern we find in the Upper Paleolithic, contemporary and late Homo samples, and it is different from the pattern found in H. antecessor, H. erectus, H. heidelbergensis (excluding SH), H. neanderthalensis and the Middle Paleolithic H. sapiens sample, in which the hypocone is equal or larger than the metacone (Table 3). Only in the early Homo group is the metacone larger than the paracone.

Cusp angles

The SH protocone angle (A) is the largest in all the groups, followed by Middle Paleolithic H. sapiens and the H. antecessor specimen, and it is significantly different from contemporary H. sapiens, which is the group with the smallest angle (Tables 7 and 8). The mean of the SH paracone angle (B) is only larger than that of H. neanderthalensis (which is the group presenting the lowest value) and the Middle Paleolithic H. sapiens sample, but the differences are not significant with any of the studied groups. Compared with the other groups, the SH metacone angle (C) is large and only smaller than that of H. neanderthalensis, although the differences are not significant. However, the differences with the Middle Paleolithic and the Upper Paleolithic H. sapiens samples are significant (Table 8). Finally, the SH hypocone angle (D) is the smallest of all the groups presented here, followed by Neanderthals, and is significantly different from all groups except for H. neanderthalensis (Tables 7 and 8).

Table 7.

Cusp angles

	A (protocone)	B (paracone)	C (metacone)	D (hypocone)
SH (10)	109.4 ± 8.2	73.0 ± 8.9	111.5 ± 6.2	66.2 ± 5.9
NEA (17)	106.1 ± 5.2	66.7 ± 6.7	118.0 ± 10.0	69.0 ± 6.1
HSAP (24)	101.4 ± 10.1	74.3 ± 4	106.2 ± 5.5	78.6 ± 7.7
MP HSAP (4)	109 ± 4.5	72.5 ± 2.5	102.0 ± 1.9	79.6 ± 6.1
UP HSAP (5)	106.3 ± 4.4	71.1 ± 2.7	110.3 ± 4.9	73.3 ± 4.8
ANT (1)	107.8	74.5	106	71.3
ERE (2)	105.3	75.2	96.8	82

Abbreviations are defined in Tables 1 and 2.

Table 8.

Statistical comparison of angles

	NEA	HSAP	MP HSAP
SH A	NS	*	NS
SH B	NS	NS	NS
SH C	NS	**	**
SH D	NS	**	**

^*Statistically significant (P ≤ 0.05). NS, nonsignificant.Abbreviations are defined in Tables 1, 2 and 6.

Occlusal polygon

The means of the SH M¹s and H. antecessor relative occlusal polygons are the same, and the smallest of all the studied groups (Table 9). SH shows significant differences with all the groups that allow statistical comparison except for H. neanderthalensis (Table 9). Thus, SH M¹s share a small relative occlusal polygon with H. neanderthalensis and H. antecessor, whereas H. sapiens and H. erectus share expanded occlusal polygons.

Table 9.

M¹ relative occlusal polygon area (OPA)

	OPA relative (%)
SH (12)	25.7 ± 4.2
NEA (17)	26.7 ± 1.8
MP HSAP (4)	33.3 ± 2.7**
UP HSAP (5)	32.7 ± 1.9**
HSAP (24)	37.5 ± 5.4**
ANT (1)	25.7
ERE (2)	32.9

^**Statistically significant (P ≤ 0.01). Abbreviations are defined in Tables 1 and 2.

Discussion

Previous studies have identified species-specific patterns of morphometric variation in upper first molars (M¹) across the genus Homo (e.g. Wood & Engleman, 1988; Bailey, 2004; Gómez-Robles et al. 2007, 2011a; Martinón-Torres et al. 2007; Moggi-Cecchi & Boccone, 2007; Quam et al. 2009). These differences were particularly significant between H. sapiens and H. neanderthalensis, suggesting they could be a useful tool for the taxonomic identification of isolated dental remains from Late Pleistocene sites (e.g. Smith, 1976; Morris, 1986; Bailey, 2002b, 2004; Harvati et al. 2003). According to Bailey (2004) and Quam et al. (2009) the typical M¹ Neanderthal features were generally absent in the H. sapiens sample and present already in their small European Middle Pleistocene samples, suggesting an early appearance of these features in the Neanderthal evolutionary lineage. A geometric morphometric study (Gómez-Robles et al. 2007, 2011a) confirmed that the large Middle Pleistocene M¹ sample from Atapuerca-SH also displays the typical skewed rhomboidal shape with bulging hypocone that characterizes the crown outline in Neanderthals, and that this morphotype was also present in the Early Pleistocene sample from Gran Dolina-TD6, attributed to H. antecessor (Bermudez de Castro et al. 1997). However, to date, there have been no data about cusp proportions and occlusal polygon from SH for comparison. Given the strong degree of Neanderthal affinities seen throughout the skeleton in the Sima de los Huesos hypodigm (e.g. Arsuaga et al. 1997b; Carretero et al. 1997; Martínez & Arsuaga, 1997; Rosas, 2001; Gómez-Olivencia et al. 2007; Bermúdez de Castro et al. 2009; Martinón-Torres et al. 2012), it was expected that SH M¹s would also display the typical Neanderthal features for these variables. Surprisingly, this study reveals that SH M¹s present a mosaic of typical H. neanderthalensis and typical H. sapiens features. SH M¹s displays the same strongly skewed (measured by the angles) and relatively small occlusal polygon as Neanderthals. However, the relative contribution of SH individual cusps to the total crown base area differs from Neanderthals and resembles that of modern humans.

The TCBA of the SH M¹s is small and falls within the range of variation of modern humans. The small postcanine size of the SH population was already ascertained in previous studies (Bermúdez de Castro, 1986; Bermúdez de Castro & Nicolás, 1995, 1996; Gómez-Robles et al. 2012). Although the identification of a general dental reduction process that affects late Homo groups is not surprising (Wolpoff, 1971; Brace et al. 1987), the marked degree and pattern of reduction seen in the SH sample, accompanied by the loss of main cusps in lower molars and M² and M³, is exceptional in a Middle Pleistocene population and even in Neanderthals (Bermúdez de Castro & Nicolás, 1995; Gómez-Robles et al. 2012; Martinón-Torres et al. 2012). SH hominins have mandibular posterior teeth that are smaller in absolute terms than those of other Pleistocene groups (Bermúdez de Castro, 1986) and are comparable to modern humans not only in size but in other traits such as similar intermolar size ratios and frequent absence of a hypoconulid (Bermúdez de Castro & Nicolás, 1995; Gómez-Robles, 2010; Gómez-Robles et al. 2011c). Similarly, our study reveals that SH presents the reduced M¹ tooth crown size that, according to Quam et al. 2009, was only attained in Upper Paleolithic times.

Regarding M¹ cusp size, the SH absolute cusp areas (except for the protocone) are small within the genus Homo, and close to/within H. sapiens range of variation. In contrast, the relative protocone area is the largest of the entire analyzed sample and significantly different from all the studied groups. The SH does not follow the pattern of relative hypocone enlargement that has been considered typical of H. heidelbergensis/H. neanderthalensis and H. antecessor (Bailey, 2004; Gómez-Robles et al. 2007, 2011a). Instead, the SH population shows a protocone expansion and hypocone reduction that characterizes H. sapiens and, thus, it refutes the hypothesis that modern human M¹ cusp proportions emerged late in the human lineage (Quam et al. 2009). However, the protocone expansion is significantly larger in SH than in any of the studied samples, including H. sapiens, and could be distinct from all other groups. Although the SH hypocone has been described as relatively large because of its prominent and bulging aspect, the similarities between SH and H. neanderthalensis M¹s rely on the relative position of this cusp with regard to the other cusps and to the tooth contour (Gómez-Robles et al. 2007, 2011a,b) rather than on, actually, a large size.

The small size of the relative metacone area in SH has been considered, together with an expanded hypocone, a derived feature of H. neanderthalensis (Bailey, 2004; Quam et al. 2009) and H. antecessor (Gómez-Robles et al. 2007, 2011a). The absolute metacone area was the only cusp to show some positive allometry in the SH sample (i.e. SH teeth with smaller TCBA would display a proportionally smaller metacone). However, with the available sample, the relative metacone area of SH is not significantly different from the Middle Paleolithic H. sapiens group, so we cannot confirm that this is a trait exclusively derived in the H. neanderthalensis direction. However, the non-significant results could be related to the limited size sample and thus should be interpreted with caution.

Regarding the relative occlusal polygon area, SH M¹s display the same pattern of internally compressed apices as H. heidelbergensis/neanderthalensis, a feature that is also present in H. antecessor but absent in H. sapiens.

Finally, this study suggests that the SH sample shares the pattern of relatively small angles at the hypocone (D), together with a relatively large angle at the metacone (C) with H. neanderthalensis, and it is likely to be responsible for the typical buccolingual skew of the rhomboid in the latter (see also Gómez-Robles et al. 2007, 2011a). Homo antecessor would differ from H. erectus in showing a distal movement of the hypocone, as suggested by the smaller angle D (Table 5 Fig. 3). The mesial movement of the metacone in the SH sample would have the effect of decreasing angle D and increasing angle C with respect to the pattern ascertained in H. antecessor (Fig. 3). Finally, the H. neanderthalensis pattern is the result of a mainly lingual shift of the metacone that enlarges the angle of the metacone (C) and reduces the angle of the paracone (B) relative to SH. This evolutionary sequence would match the changes suggested by Bailey (2004) for the Neanderthal lineage with regard to the primitive condition. Although SH could represent an early stage of the derived Neanderthal pattern, it is important to remember that with our sample the differences between the groups are not significant. Homo antecessor, SH hominins and H. neanderthalensis all have large metacone angles and small hypocone and paracone angles compared with the primitive form (H. erectus). Regarding the cusp angles, H. sapiens also presents the primitive pattern.

Fig. 3.

Schematic representation of evolution of the cusp angles. Arrows represent the main cusp shift in each of the represented groups. ERE, Homo erectus; SAP, Homo sapiens; ANT, Homo antecessor; SH, Sima de los Huesos; NEA, Homo neanderthalensis.

Summarizing, SH M¹s present a mosaic of H. neanderthalensis and H. sapiens features (Table 10). In 2004, Bailey pointed out that the differences in relative cusp base area were not necessarily responsible for the differences in tooth shape. Indeed, her sample of Middle Paleolithic H. sapiens fell within the Neanderthal variation for the relative cusp size, but was close to contemporary modern humans for cusp angles. Conversely, SH falls close to modern humans with regard to the TCBA and cusp sizes but is close to H. neanderthalensis in the occlusal polygon compression (this study) and general shape (Gómez-Robles et al. 2007, 2011a). Thus, we agree with Bailey (2004) that the Neanderthal tooth shape cannot be attributed solely to the differences in relative size of the main cusps.

Table 10.

Comparative summary of the SH M1 features

	ANT	NEA	HSAP
SH M1 features
Strongly skewed crown outline with a rhomboid shape and a bulging and protruding hypocone	x	x
Small relative occlusal polygon	x	x
Cusp angles: small D and large C	x	x
Small TCBA			x
Relatively large protoconea			x
Relatively small hypocone			x
Relatively small metaconeb	x	x

ANT, Homo antecessor; NEA, Homo neanderthalensis; HSAP, Homo sapiens.

^aThe relative expansion of the protocone is significantly larger than in any other studied group and could be distinct for SH (see text).

^bThe differences between the Middle Paleolithic H. sapiens and SH were not significant.

The apparently independent evolution of the metric and the morphological variables in the M¹s raises different evolutionary interpretations. One possibility is that the pattern of SH cusp proportions is related to the small size of the M¹ in this population, explaining in this way the resemblances between SH and H. sapiens in both features. Previous studies on upper molars have shown that differences in molar size affect the relative cusp size (Macho & Moggi-Cecchi, 1992; Moggi-Cecchi & Boccone, 2007; Williams & Corruccini, 2007). Following a patterning cascade model (Jernvall, 2000; Jernvall & Jung, 2000; Jernvall et al. 2000), the delayed onset of dental formation and/or the early termination of cusp morphogenesis in late Homo species would result in a paedomorphic tooth that does not complete its potential cusp pattern and, thus, distal elements are reduced and/or lost. However, this mechanism is particularly evident in M²s and M³s rather than in M¹s, and would affect distal rather than mesial cusps (Corruccini, 1979; Kondo & Yamada, 2003; Kondo & Townsend, 2006; Takahashi et al. 2007). This would explain the hypocone reduction in SH and H. sapiens, but not the increase in protocone size, which is particularly pronounced in the SH sample. Thus, another mechanism should be invoked to relate this pattern of cusp proportion to the dental reduction process ascertained in both groups.

Interestingly, the proportional increase of the protocone along the upper molar series was also noted by Macho & Wood (1995) in their H. sapiens sample. In addition, other researchers (Kondo et al. 2005; Takahashi et al. 2007) reported some type of interaction between the upper molar lingual cusps (in M²s and M³s), such that when the hypocone of a tooth tended to be reduced in size, the protocone of that tooth would tend to be larger. Finally, recent studies on molar variation in baboons have provided exciting venues for interpretation in this field (Hlusko et al. 2007; Koh et al. 2010). These authors have found strong genetic correlations between cusps, in particular in maxillary molars between the paracone-metacone and the protocone-hypocone (Koh et al. 2010), which do not conform to expectations from the sequence of cusp formation and/or calcification. According to Koh et al. (2010), the closer anatomical relationship between the mesiodistal length of maxillary molars with the rest of the cranium may serve as a constraint to the mesiodistal axis variation and a key factor to maintain a less variable molar row length.

Our analysis of the SH intragroup variation suggests that there are not obvious patterns of covariation associated with the relative cusp sizes. Although it is not confirmed by the bivariate analysis, the principal component analysis suggests an inverse relationship between the size of the protocone and hypocone within the SH samples, i.e. larger protocones tend to be found with small hypocones and vice versa. The results may be spurious correlations associated with the small sample size. However, at the same time, non-significant results could also be associated with the limited statistical power, and should be interpreted with caution. However, and as was mentioned above, it has been statistically demonstrated for other hominid samples that the protocone may increase at the expense of the hypocone (e.g. Takahashi et al. 2007; Koh et al. 2010). We could hypothesize that the hypocone reduction ascertained in SH and H. sapiens could be explained by an early termination of a small crown, which would affect the development of the last developing cusp. As a consequence, selective pressures would have favoured a protocone increase to compensate for mesiodistal length loss (see Koh et al. 2010). The need for the relative stability in these two cusps would be enhanced because of their role as ‘functional’ or ‘occluding’ cusps (Hiiemae & Kay, 1972; Molnar & Gantt, 1977; Molnar & Ward, 1977) and particularly the protocone, to guarantee optimal mastication (Macho & Wood, 1995). Cusp position and size depends on the combined effects of the secondary enamel knot spacing, the rates of cell proliferation and the onset and speed of the mineralization process halting the proliferation phase (Harris & Dinh, 2006). We suggest that, together with changes in the proliferation rates and/or the onset and speed of mineralization, the delayed onset of the hypocone formation would have reduced the effects of the hypocone inhibitory field, favouring a protocone enlargement.

Summarizing, we propose that as a response to the hypocone reduction caused by the strong overall dental reduction in SH and H. sapiens, selection favoured a mechanism to ensure the stability of the protocone and/or the protocone-hypocone axis, as they are the ones bearing the higher functional demands in mastication (see Macho & Wood, 1995; Koh et al. 2010).

How do we explain the expression of H. sapiens features in the SH sample? Assuming that H. neanderthalensis and H. sapiens are sister clades, this cusp proportion could be indeed a synapomorphy inherited from a hypothetical last common ancestor, and those traits would be eventually lost in H. neanderthalensis, whereas they would become typical of H. sapiens. This would explain why pre-Neanderthal and pre-sapiens groups, close to the node of divergence, present variable degrees of ‘sapiens’ and ‘Neanderthal’ features, respectively (Bermúdez de Castro et al. 2012; Martinón-Torres et al. 2012). For example, the departure of the Qafzeh sample from the typical H. sapiens pattern of cusp proportions was already noticed by Bailey (2004). This departure would be in line with other dental features such as the pronounced labial convexity (Martinón-Torres et al. 2012) or the perikymata distribution pattern (Guatelli-Steinberg & Reid, 2010) on upper incisors, which are closer to the typical patterns of Neanderthal than of H. sapiens. However, it is important to note that the ‘sapiens’ cusp pattern expressed by the SH M¹s has not been identified in any earlier hominin group, including the small H. antecessor sample, which was first interpreted as the last common ancestor of Neanderthal and modern lineages (Bermudez de Castro et al. 1997; but see also Martinón-Torres et al. 2010; Bermúdez de Castro & Martinón-Torres, 2012). The small sample size of this species might preclude a real assessment of H. antecessor intragroup variability and the possibility of a polymorphic expression of this pattern in this group.

However, if the small TCBA and ‘modern’ pattern of SH cusp proportions is not a synapomorphy, then these features would have evolved separately in SH and H. sapiens and it would represent a case of parallelism. The small size and relative cusp proportions of SH M¹s could be a particularity of the SH group and thus could not be extrapolated to the rest of the European Middle Pleistocene groups. Indeed, the relative protocone expansion ascertained in SH is significantly larger than that of H. sapiens, and could be distinct for the Atapuerca sample. Previous dental analyses have already pointed out the highly derived state of the Sima de los Huesos hominins, particularly in the small size of the postcanine dentition and the qualitative traits derived from this metric reduction, even more marked than in Neanderthals and some modern humans (Bermúdez de Castro & Nicolás, 1995, 1996; Gómez-Robles et al. 2012; Martinón-Torres et al. 2012). The derived traits in SH (even more derived than Neanderthals themselves; see also Prado-Simón et al. 2012) are combined with other dental and skeletal features that do not display such derived character states, such as cranial, facial and postcranial features, which show intermediate or primitive conditions with respect to classic Neanderthals (e.g. Arsuaga et al. 1997b; Carretero et al. 1997; Martínez & Arsuaga, 1997; Gómez-Olivencia et al. 2007). However, the mosaic of primitive and derived features displayed by the M¹ would enlarge the list of dental features (Bermúdez de Castro & Nicolás, 1995; Gómez-Robles et al. 2007, 2008, 2011a,b, 2012; Martinón-Torres et al. 2012; Prado-Simón et al. 2012) that do not follow the gradual and anagenetic model suggested by other researchers to explain the origins of the Neanderthal lineage, where chronologically earlier populations are expected to be less Neanderthal than later groups (Dean et al. 1998; Hublin, 1998; Hublin & Roebroeks, 2009). However, to better understand the biological processes that conclude with the appearance of H. neanderthalensis it is important to take into account as many skeletal features as possible. The M¹ is only an isolated piece which by itself can hardly provide a general picture. Indeed, this study sounds a note of caution in the use of isolated remains and/or features for taxonomical assignments, and strongly urges consideration of the variability of expression and the polarity of the traits under study.

Concluding remarks

The aim of this study was to contribute to the understanding of the M¹ evolution in H. sapiens and H. neanderthalensis by providing the TCBA, cusp angles, cusp proportions and relative occlusal polygon area of the largest European Middle Pleistocene sample to date, the Atapuerca-Sima de los Huesos population. The combination of Neanderthal and ‘modern’ features of SH highlights the importance of understanding the variability and the polarity of the traits under study before making taxonomical assignments. Interestingly, the Middle Paleolithic H. sapiens sample from Qafzeh also presents contradicting taxonomic signals, being similar in shape to H. sapiens but similar to Neanderthals in the pattern of cusp proportions (Bailey, 2004). The analysis of the Early and Middle Pleistocene dental samples from Atapuerca have obliged to reconsider the uniqueness of traits that have been classically interpreted as typical of Neanderthals or typical of H. sapiens. Finally, our results ratify the highly derived state of the Atapuerca-SH population, and questions whether the SH sample is representative of all the European Middle Pleistocene populations or presents some distinct or exclusive features. Future findings and studies may contribute to this question by providing more information on the intra and inter-population variability of the Middle Pleistocene hominins.