New insights into differences in brain organization between Neanderthals and anatomically modern humans

Abstract

Previous research has identified morphological differences between the brains of Neanderthals and anatomically modern humans (AMHs). However, studies using endocasts or the cranium itself are limited to investigating external surface features and the overall size and shape of the brain. A complementary approach uses comparative primate data to estimate the size of internal brain areas. Previous attempts to do this have generally assumed that identical total brain volumes imply identical internal organization. Here, we argue that, in the case of Neanderthals and AMHs, differences in the size of the body and visual system imply differences in organization between the same-sized brains of these two taxa. We show that Neanderthals had significantly larger visual systems than contemporary AMHs (indexed by orbital volume) and that when this, along with their greater body mass, is taken into account, Neanderthals have significantly smaller adjusted endocranial capacities than contemporary AMHs. We discuss possible implications of differing brain organization in terms of social cognition, and consider these in the context of differing abilities to cope with fluctuating resources and cultural maintenance.

1. Introduction

Comparisons of the morphology of Neanderthal and anatomically modern human (AMH) brains have previously identified a number of similarities, for example, in the degree of asymmetry and gyrification [1], as well as non-allometric widening of the frontal lobes [2]. However, differences in brain morphology have also been noted. For instance, in addition to their uniquely globular brain shape [3], it has recently been reported that the temporal pole is relatively larger and more forward-projecting, the orbitofrontal cortex being relatively wider and the olfactory bulbs being larger in AMHs compared with other hominins, including Neanderthals [4]. In addition, Neanderthals show lateral widening but overall flattening of their parietal lobes, whereas AMHs have uniform parietal surface enlargement [5]. These differences have led to the suggestion that Neanderthals and AMHs reached similarly enlarged brains along divergent developmental [3] and evolutionary [6] pathways.

Most of the work on Neanderthal and fossil AMH brains has relied on endocasts and the internal morphology of the cranium. However, this approach is limited to investigating external surface features and the overall shape and size of the brain, and provides no information about internal brain organization. For instance, identification of the lunate sulcus on endocast surfaces is highly ambiguous, making the size of the primary visual area (V1) difficult to measure from endocasts. An alternative approach has been to use known relationships between overall brain volume and the volume of specific brain areas in extant primates to estimate the respective brain region volumes in fossil crania [7]. While this approach is broadly reliable, it does assume that identical overall brain volumes imply identical brain organization. Although there are conserved patterns of scaling in brain organization across mammals [7,8], this is not always the case: the size of the visual system, for example, varies independently of the size of other sensory systems [9,10]. More importantly, there are well-known examples among primates where mosaic brain evolution has resulted in brain regions that are significantly smaller or larger than would be predicted by overall brain size alone. Among the great apes, for instance, gorillas and orang-utans have unusually large cerebella and relatively small neocortices [11]. Assuming that similar endocranial volumes equate with identical organization within the brain may be seriously misleading.

We hypothesize that the similarly sized brains of Neanderthals and AMHs were organized differently for at least two reasons. First, Neanderthals had larger bodies than AMHs and, hence, they would have required proportionately more neural matter for somatic maintenance and control [1,12]. Second, Neanderthals lived at high latitudes, where they would have experienced lower light levels than tropical hominins. Since even recent humans living at high latitudes require larger eyeballs to attain the same level of visual acuity and/or sensitivity as individuals living at lower latitudes [13], Neanderthals would probably have had larger eyes than contemporary AMHs, who had only recently emerged from low-latitude Africa. Components of the visual system scale with each other, from orbits and eyes [14,15] (contra [16]) through to the cortical primary (V1) and downstream visual areas (V2, V3 and V5/MT) in the brain [9,15,17–22]. This means that if Neanderthals had larger eyes than AMHs, they would also have had larger visual cortices. For a meaningful comparison of brain volume, fossil brains need to be adjusted for at least these two effects.

Here, we first show, using orbit size as a proxy [13,15], that Neanderthals had larger visual systems than contemporary AMHs. We then examine the implications of this and the difference in body size for the brains of these taxa.

2. Material and methods

We used endocast volumes for 21 Neanderthals and 38 AMHs dated 27–200 ka [23] (see the electronic supplementary material, table S1). We excluded AMHs younger than 27 ka so that the AMH specimens were as close in time to the Neanderthal specimens as possible. We excluded from the analyses specimens dated earlier than 200 ka as being taxonomically arguable, but include them for reference in figure 1. Using alternative cranial volume databases [24,25] produces essentially the same results, indicating that our findings are robust to discrepancies in volume determinations. However, we present only the analyses using the endocast dataset here.

Figure 1.

(a) Absolute and (b) standardized endocranial volumes for different hominin taxa, split into date groups (given in thousands of years ago: ka). The three boxes represent (i) Homo heidelbergensis (Hh) and possible Denisovans (?D), (ii) the Neanderthal lineage, from archaics (AHn) to Neanderthals (Hn) dated 76–200 ka and 27–75 ka, and (iii) the Homo sapiens lineage, from archaics (AHs) to AMHs (Hs) dated 76–200 ka and 27–75 ka. Circles indicate the value for each individual fossil specimen. The horizontal bars show group means ± the s.e.m. (cumulative for b). The light grey shading illustrates that Neanderthals dated 27–75 ka have the same sized brains as Homo sapiens in terms of absolute endocranial volumes, but that once body and visual system sizes are taken into account, the Neanderthal means lie outside the standard errors of the AMH means. The dashed lines in (b) illustrate the AMH means for both date groups, to ease comparison with the Neanderthal means.

In order to standardize the AMH and Neanderthal endocranial volumes in terms of body mass, we calculated the ratio between living human and fossil mean body masses for each of the fossil date groups in a study by Ruff et al. [26] and use this ratio to scale for body size. We multiplied the absolute brain volume of each fossil in our endocast dataset [23] by this date-group-specific body mass correction factor to give the equivalent endocranial volume expected for a living human. Since the relationship between brain and body size is not isometric, we used body mass raised to the 0.646 power in this calculation, which Isler et al. [27] report as the common slope for ln-transformed endocranial volume regressed on body mass across primates once grades are taken into account.

Orbital areas were from Kappelman [28], orbital height (OBH) and breadth (OBB) measurements were supplied by C.S., and orbital volumes were measured by E.P. (see [13] for more details of the volume method). Volumes did not include brow-ridges, since the orbits were filled with beads up to a line continuous with the lateral and medial rims, following Schultz [14]. In any case, brow-ridge size is not related to orbital volume across primates [29].

We calculated visual cortex corrections using a number of computational steps, which we summarize before giving details below. Insufficient volumetric data are available for primate visual areas beyond V1 to produce an equation relating orbital size directly to total visual cortex volume. Instead, we (i) estimated fossil visual cortical surface areas from OBH (the measurement for which we had most fossil data) using a primate-derived equation, and (ii) converted these into volumes by multiplying by cortex depth. We calculated this depth by (iii) assuming a 2 mm thickness for grey matter (i.e. multiplying the surface area in mm² by 2 mm to give volume in mm³; details below) and (iv) using a primate equation to estimate white matter volume from grey matter volume. We then (v) totalled grey and white matter volumes to give ‘total’ visual cortex volume.

In order to estimate total visual surface area from orbit diameter in the fossils, we conducted a cross-primate comparison by fitting a phylogenetically controlled generalized linear model (PGLM) using the pgls function of the caper package [30] in R to log₁₀ visual cortex surface area (V1 and V2 from [31]) plotted against log₁₀ orbital diameter [32,33] for five diurnal anthropoid primate species for which overlapping data were available: Homo sapiens, Macaca mulatta, Saguinus fuscicollis, Saimiri sciureus and Callithrix jacchus. We used the consensus chronogram tree from the 10k Trees Project website [34]. The PGLM analysis yielded a significant positive linear relationship between log₁₀ orbital diameter (mm) and log₁₀ ‘total’ visual cortex (represented by V1 and V2) surface area (mm²) independently of phylogeny: t₃ = 10.137, p = 0.002, adjusted R² = 0.962, log₁₀ total visual SA =−1.89 (95% CIs: −3.506 to −0.28) + 4.25 (95% CIs: −2.914 to 5.58) × log₁₀ orbit diameter.

Since total visual cortical surface area data were only available for five genera, we also present the results of a PGLM fitted to log₁₀ primary (as opposed to total) visual cortex (V1) volume [35] plotted against log₁₀ orbit diameter [32,33] for 15 diurnal anthropoids, to confirm that orbital dimensions provide a good proxy for visual cortical size in larger samples as well. The PGLM fitted to log₁₀ V1 volume plotted against log₁₀ orbital diameter revealed a significant relationship independently of phylogeny: t₁₃ = 8.412, p = 1.286e⁻⁰⁵, adjusted R² = 0.833, log₁₀ V1 = 0.54 (95% CIs: −0.25 to 1.34) + 2.38 (95% CIs: 1.77 to 2.99) × log₁₀ orbit diameter.

We used OBH to estimate the total visual surface area for each individual fossil in the endocast volume dataset for which C.S. provided data, and converted these surfaces into total combined grey and white matter volumes. First, we calculated total grey matter volume by multiplying the surface area estimates by 2 mm, which is a reasonable estimate of cortical grey matter thickness in both humans [36–38] and macaques [38]. We then estimated white matter volume (cm³) from grey matter volume (cm³) using a reduced major axis (RMA) regression equation, using anthropoid primate neocortical data from [39]: t₁₉ = 56.28, p < 0.001, r² = 0.994 log₁₀ white =−0.81 (95% CI: −0.72 to −0.93) + 1.32 (95% CI: 1.24 to 1.43) × log₁₀ grey. These parameters do not fall outside the 95% confidence intervals of those associated with a PGLM regression model using a tree from the 10k Trees Project. However, we chose to use the RMA model because this is more appropriate when there is measurement error on both axes and the relationship between variables is symmetrical [40]. Finally, we summed the estimated white matter volumes with the grey matter volume estimates to get total visual cortex volume.

Once we had total visual cortex volume estimates for each fossil, we calculated the residual to the AMH mean for each date group for each Neanderthal fossil. We also calculated the difference between the species’ means in each date group (27–75 ka, 118 cm³; 76–200 ka, 59 cm³).

After correcting for body mass differences as outlined above, we standardized the Neanderthal endocranial volumes by subtracting the difference in visual cortex volume between AMHs and Neanderthals (individual residuals or differences between date-specific means where OBH was unavailable) from the respective Neanderthal values. In other words, all Neanderthal endocranial volumes were recalibrated as if they were organized in the same way as the average AMH brain (i.e. without enlarged visual cortices), while also taking individual differences within the Neanderthal species into account as far as possible.

The computation of these standardized brain volumes requires a series of steps, which each introduce error into the subsequent estimates. In this study, we are interested in comparisons between taxon means: rather than being interested in the error accrued for the standardized brain estimate for each individual specimen, what is crucial is the compounded error attached to the calculated mean per taxon group. In other words, the central issue is the parameter (taxon variance) error rather than the population error. We therefore calculated the cumulative standard error of the mean standardized endocranial volume for each taxon/date group by taking, at each computational step, not only the mean estimate but also the mean estimate plus or minus the standard error (s.e.; for means) or standard error of the estimate (s.e.e. for regressions, following Ruff et al. [41]). We then took these three values—(i) mean estimate, (ii) mean estimate plus s.e./s.e.e. (iii) mean estimate minus s.e./s.e.e.—into the next step of the estimation process. Once we had calculated the standardized endocranial volumes, we then calculated the sample mean for the mean, upper (cumulative ‘mean plus s.e./s.e.e.’ estimates) and lower (cumulative ‘mean minus s.e./s.e.e.’ estimates) estimates for each of the taxon and date group subgroups of specimens. The means of the upper and lower estimates for the sample were taken as values representing the mean ± the cumulative s.e. We plot these cumulative standard error bars in figure 1b.

One-sample Kolmogorov–Smirnov tests with Lilliefors corrections found no deviations from normality for any variables. Effect sizes are reported as R².

3. Results

Independent-sample t-tests applied to three different datasets confirmed that Neanderthals had significantly larger orbits than contemporary AMHs (table 1). This implies that Neanderthals also had significantly larger eyeballs and visual cortices.

Table 1.

Orbital dimensions compared between anatomically modern humans (AMHs) and Neanderthals from all date groups.

	Neanderthals			AMHs			independent t-test (two-tailed)
	mean	s.d.	n	mean	s.d.	n	t	p-value	R2
orbital area (mm2)	1403.98	94.24	6	1223.39	130.00	10	2.952	0.011	0.38
orbital height (mm)	36.41	1.11	12	30.63	2.59	15	7.777a	<0.0001	0.71
orbital breadth (mm)	44.07	2.19	11	41.80	2.78	15	2.245	0.034	0.17
orbital volume (cm3)	34.15	3.39	5	29.51	2.07	4	2.387	0.048	0.45

^aCorrected for unequal variances.

Independent-sample t-tests found no significant difference in raw/absolute endocranial volume between AMHs and Neanderthals dated 27–75 ka (figure 1a and table 2). However, older AMHs dated 76–200 ka had significantly larger endocranial volumes than Neanderthals of a similar date (table 2). Neanderthals dated to 27–75 ka had significantly larger endocranial volumes than those dated 76–200 ka (two-tailed independent-sample t-test, corrected for unequal variances: t_17.687 = 3.513, p = 0.003, R² = 0.411).

Table 2.

Independent-sample t-test statistics for comparisons between AMHs and Neanderthals for raw and standardized endocranial volume, split by date group.

date groups (ka)	endocranial volume variables (cm3)		taxon		t-test (two-tailed)
			AMHs	Neanderthals	t	p-value	R2
27–75	absolute endocranial volume	n	32	13	0.008	0.994	n.a.
		mean	1473.84	1473.46
		s.d.	135.96	180.61
	corrected endocranial volume	n	32	13	4.439	<0.0001	0.31
		mean	1332.41	1133.98
		s.d.	126.45	157.76
76–200	absolute endocranial volume	n	6	8	7.169	<0.0001	0.81
		mean	1535.50	1272.00
		s.d.	48.87	78.96
	corrected endocranial volume	n	6	8	8.451	<0.0001	0.86
		mean	1405.11	1097.47
		s.d.	48.39	78.207

Comparison of ‘corrected’ or ‘standardized’ endocranial volumes shows that adjusting for differences in body and visual system size results in the disparity between the two Neanderthal date groups disappearing (figure 1b). In effect, the younger Neanderthals (27–75 ka) show no increase in non-somatic/non-visual brain size compared with the older Neanderthals (76–200 ka) and archaic humans. These younger ‘standardized’ Neanderthal endocranial volumes are significantly smaller than those of contemporary AMHs within the 27–75 ka date group (table 2). This suggests that later Neanderthal brains comprised a significantly larger proportion of neural tissue associated with somatic and visual function compared with the brains of contemporary AMHs.

In the 27–75 ka date group, the mean standardized Neanderthal endocranial volume (mean = 1134, 95% CIs = 1037–1231, n = 13) lies outside the 95% CIs of the mean endocranial volume of recently living humans (mean = 1373, 95% CIs = 1335–1410, n = 55 living in the last approx. 200 years; data from [13]). By contrast, the 95% CI of the fossil AMH mean (mean = 1332, 95% CIs = 1287–1377, n = 32) overlaps that of recently living humans.

4. Discussion

We have demonstrated that Neanderthals had significantly larger orbits than contemporary AMHs, which, owing to scaling between the components of the visual system, suggests that Neanderthal brains contained significantly larger visual cortices. This is corroborated by recent endocast work, which found that Neanderthal occipital lobes are relatively larger than those of AMHs [42]. In addition, previous suggestions that large Neanderthal brains were associated with their high lean body mass [1,43,44] imply that Neanderthal also invested more neural tissue in somatic areas involved in body maintenance and control compared with those of contemporary AMHs.

In recent humans (dated to the last approx. 200 years), larger visual systems translate into larger brains [13]. We might therefore expect that larger Neanderthal visual cortices (and somatic areas) would similarly drive overall brain enlargement in this taxon compared with AMHs. However, we have shown that this is not the case for specimens dated 27–75 ka; Neanderthals in this date group do not show significantly larger brains than contemporary AMHs. This suggests that (i) Neanderthal and AMH brains were organized differently, and, (ii) by implication, because a greater proportion of the overall brain tissue in Neanderthals was invested in visual and somatic systems, proportionally less neural tissue was left over for other brain areas in Neanderthals compared with AMHs. Note that our analysis considered only the principal visual areas in the occipital lobe: given that the visual system projects through the parietal and temporal to the frontal lobes, our case could only be strengthened if comparative data on these projection areas were available and could be included in the analysis.

Overall, our findings tie in with the suggestion that the Neanderthal and AMH lineages underwent separate evolutionary trajectories [6]. Starting from the brain size of their common ancestor Homo heidelbergensis, we suggest that Neanderthals enlarged their visual and somatic regions, whereas AMHs achieved similarly large brains by increasing other brain areas (including, for example, their parietal lobes) [5]. Furthermore, it seems that the Neanderthal route followed a more strictly allometric trajectory [6]. Human primary visual areas are smaller than expected for a primate of our brain size [45]; larger Neanderthal V1s may thus be more in line with the expectations for a generic large-brained primate, adding support to this argument.

Macroscopic measures such as regional volume index neural network characteristics such as the number of neurons and synapses [17,46,47]. Consequently, the differences in the partitioning of brain tissue might have substantial implications for cognitive processing in Neanderthals compared with contemporary AMHs. For instance, there is a well-established relationship between brain and bonded group size across anthropoid primates [48–57], as well as between specific areas of the frontal lobe and active social network (total number of personal contacts) size at the within-species individual level in both macaques [58] and, more importantly, humans [59–61]. In addition, neuroimaging studies have shown that this relationship between key brain region volumes and group size is mediated by mentalizing (theory of mind) competences [42]. In humans, the total network/group size comprises a number of nested layers [62] and the greatest number of minds an individual can keep track of at any one time constrains the size of their most intimate support clique [63], which in turn sets a limit on the overall group size that they can maintain [62,64–66]. The mean size of the active network for living humans predicted by cross-primate neocortex ratio comparisons has been corroborated across not only historical and modern traditional subsistence societies, but also in online social environments [54,67,68]. This suggests that throughout human evolution, brain structure and cognitive function have placed a constraint on bonded group size and social complexity.

While we cannot partition fossil brains down to the refinement of specific frontal regions, there is at least sufficient evidence from comparative studies of primates [69–71] to justify using whole brain volumes to estimate cognitive capacities as a first step. To do so, we followed Dunbar [54] in using an ape PGLM equation to predict group size from the standardized endocranial volumes. Note that this equation predicts a group size of approximately 144 for living humans rather than the 150 predicted by neocortex ratio, because using cranial volumes results in additional estimation error by including brain regions, such as the visual system, that are not part of the social or mentalizing network [11], and this results in a shallower slope. However, we use cranial volume here because it reduces the number of interpolation steps, and so reduces the cumulative error variance on the fossil estimates.

Neanderthals dated 27–75 ka were predicted to have had smaller cognitive group sizes (M = 115, s.d. = 19, n = 13) than contemporary fossil AMHs, whereas fossil AMHs (M = 139, s.d. = 15, n = 32) seem to have had group sizes more in line with those demonstrated for the mean personal network sizes of living humans (figure 2). What little archaeological evidence there is offers support for this: compared with Neanderthals, contemporary Eurasian AMHs had larger [72], more geographically extensive social networks [73,74]. Group size is a convenient index of the cognitive ability to deal with increasing social complexity and may thus evidence more general differences in sociocognitive abilities between these taxa.

Figure 2.

Group sizes estimated from standardized endocranial volumes for Neanderthals and AMHs in the 27–75 ka date group. The dashed line indicates the mean group size expected for living humans based on the size of their neocortex (150 individuals).

Such differences may have had profound implications for Neanderthals. First, assuming similar densities, the area covered by the Neanderthals' extended communities would have been smaller than those of AMHs. Consequently, the Neanderthals' ability to trade for exotic resources and artefacts would have been reduced [75], as would their capacity to gain access to foraging areas sufficiently distant to be unaffected by local scarcity [76]. Furthermore, their ability to acquire and conserve innovations may have been limited as a result [77], and they may have been more vulnerable to demographic fluctuations, causing local population extinctions.

Whereas AMHs appear to have concentrated neural investment in social adaptations to solve ecological problems, Neanderthals seem to have adopted an alternative strategy that involved enhanced vision coupled with retention of the physical robusticity of H. heidelbergensis, but not superior social cognition. For instance, only in Neanderthals, not AMHs, does body mass [26], and hence brain volume [78], increase over time. While the physical response to high latitude conditions adopted by Neanderthals may have been very effective at first, the social response developed by AMHs seems to have eventually won out in the face of the climatic instability that characterized high-latitude Eurasia at this time.