Genetic contributions to dental dimensions in brown-mantled tamarins (Saguinus fuscicollis) and rhesus macaques (Macaca mulatta)

Abstract

Objectives:

The use of dental metrics in phylogenetic reconstructions of fossil primates assumes variation in tooth size is highly heritable. Quantitative genetic studies in humans and baboons have estimated high heritabilities for dental traits, providing a preliminary view of the variability of dental trait heritability in non-human primate species. To expand upon this view, the heritabilities and evolvabilities of linear dental dimensions are estimated in brown-mantled tamarins (Saguinus fuscicollis) and rhesus macaques (Macaca mulatta).

Materials and Methods:

Quantitative genetic analyses were performed on linear dental dimensions collected from 302 brown-mantled tamarins and 364 rhesus macaques. Heritabilities were estimated in SOLAR using pedigrees from each population, and evolvabilities were calculated manually.

Results:

Tamarin heritability estimates range from 0.19 to 0.99, and 25 of 26 tamarin estimates are significantly different from zero. Macaque heritability estimates range from 0.08 to 1.00, and 25 out of 28 estimates were significantly different from zero.

Discussion:

Dental dimensions are highly heritable in captive brown-mantled tamarins and free-ranging rhesus macaques. The range of heritability estimates in these populations are broadly similar to those of baboons and humans. Evolvability tends to increase with heritability, although evolvability is high relative to heritability in some dimensions. Estimating evolvability helps to contextualize differences in heritability, and the observed relationship between evolvability and heritability in dental dimensions requires further investigation.

Introduction

Variation in dental morphology provides a powerful toolset that anthropologists can use to clarify the often-obscured picture of primate evolution. Teeth preserve well in the fossil record and are recovered in large numbers relative to other skeletal elements at paleontological sites. The use of tooth size and morphology in the reconstruction of fossil primate relationships is widespread (e.g. Gómez-Robles, Bermúdez de Castro, Martinón-Torres, Prado-Simón, & Arsuaga, 2012; Gómez-Robles, de Castro, Martinón-Torres, & Prado-Simón, 2011; Hunt & Vitzthum, 1986; Quam, Bailey, & Wood, 2009; Ross, Williams, & Kay, 1998; White, Suwa, & Asfaw, 1994; Wood & Abbott, 1983). These reconstructions rely on several assumptions regarding variation and change in dental morphology, including the assumption that variation in dental morphology has been primarily influenced by heritable differences. While dental metrics are already useful for both phylogenetic and dietary reconstruction, greater knowledge of the heritability of primate tooth crown dimensions will allow us to generate increasingly precise hypotheses to explain patterns of dental evolution in primates.

The rate at which a trait responds to natural selection depends on the selection intensity and the degree to which the trait is heritable in the population under selection (Lush, 1937; Lynch & Walsh, 1998). Adaptive hypotheses, therefore, assume traits of interest were reproductively advantageous and heritable. The assumption of heritability can be assessed in the teeth of present-day primate populations through the estimation of quantitative genetic parameters. Estimates of heritability from diverse extant primate populations may demonstrate patterns that were shared by fossil primates, thereby providing valuable information about the genetic architecture of traits used to understand primate evolution and adaptation.

Traditional quantitative genetic methods can be used to estimate heritability using mathematical models in which continuous phenotypic variance (σ²_P) is produced by environmental and genetic variance within the population. The environmental variance (σ²_E) represents the population-level variation in the phenotype produced by non-genetic factors, including differences in diet, disease, behavior, and measurement error. The genetic variance of a population (σ²_G) can be broken into constituent parts including the additive genetic variance (σ²_A) and the impacts of dominance and epistasis. Non-additive contributions to σ²_G, such as dominance and epistasis, are not consistent from generation to generation, so the ratio of additive genetic variance to phenotypic variance (σ²_A/σ²_P), called the narrow-sense heritability (h²), is most frequently used in evolutionary quantitative genetics research (Falconer & Mackay, 1996). Because primates tend to reproduce and develop slowly, and require considerable investment to house and breed, the use of quantitative genetic methods in anthropological research have been limited. Despite these challenges, heritabilities of cranial and dental variables have been estimated previously in humans and non-human primates (Cheverud, 1996; Cheverud & Buikstra, 1981; Joganic et al., 2017, 2012; Šešelj, Duren, & Sherwood, 2015; Sherwood, Duren, Demerath, et al., 2008; Sherwood, Duren, Havill, et al., 2008).

At present, much of our understanding of the heritability of dental morphology has emerged from studies of human twins (Biggerstaff, 1973, 2005; Boraas, Messer, & Till, 1988; Corruccini, Sharma, & Potter, 1986; Dempsey & Townsend, 2001, 2001; Liu, Deng, Cao, & Ono, 1998; Sharma, Corruccini, & Henderson, 1985; G. Townsend, Harris, Lesot, Clauss, & Brook, 2009; G.C. Townsend & Martin, 1992; Grant C. Townsend & Brown, 1978). The quantitative genetic parameters of human and non-human populations with complex pedigrees can also be analyzed using maximum likelihood (ML) estimation (Shaw, 1987), as performed previously on human and non-human primate dental features (Hlusko, Lease, & Mahaney, 2006; Hlusko, Sage, & Mahaney, 2011; Hlusko, Suwa, Kono, & Mahaney, 2004; Hlusko, Weiss, & Mahaney, 2002; Hlusko & Mahaney, 2009; Koh et al., 2010; Stojanowski, Paul, Seidel, Duncan, & Guatelli‐Steinberg, 2018; Stojanowski, Paul, Seidel, Guatelli-Steinberg, & Duncan, 2017). Thus far, dental heritabilities have been estimated from a single non-human primate population, the Southwest National Primate Research Center (SNPRC) baboons (Papio spp.). These studies show that molar size (Hlusko et al., 2002), molar cusp size (Koh et al., 2010), molar crown features (Hlusko & Mahaney, 2003), and tooth dimensions (Hlusko & Mahaney, 2009; Hlusko et al., 2011) are significantly heritable in this captive baboon population. The variation in tooth size, morphology and development observed in primates gives reason to suspect that the genetic variance of tooth size differs across living primate populations. Heritability estimates may also vary due to environmental effects related to living conditions (Charmantier & Garant, 2005; Kruuk, Slate, & Wilson, 2008; Pemberton, 2010; Weigensberg & Roff, 1996); the inclusion of free-ranging or wild populations in quantitative genetic studies will help to assess the impact of living conditions on the heritability of tooth size. Hence, quantitative genetic analyses of multiple species and populations living in different settings will allow for better assessment of the genetic and environmental contributions to variation in tooth size, and will bring greater attention to the complexity of interpreting quantitative genetic parameters across populations.

The additive genetic coefficient of variation, often called the evolvability (I_A), describes genetic variance in a trait relative to its size. I_A is useful for comparisons across populations because it not impacted by differences in σ²_E (Hansen, Pélabon, & Houle, 2011; Houle, 1992). I_A values often demonstrate different patterns between traits and populations than h² values (Hansen et al., 2011). The estimation of both h² and I_A in this study allows for the first published comparison of these parameters in primate dental traits.

Estimating the heritabilities and evolvabilities of dental traits in multiple primate populations is crucial to understanding the impact and maintenance of genetic variance in primate dental traits. This study uses h² and I_A estimates from brown-mantled tamarin (Saguinus fuscicollis) and rhesus macaque (Macaca mulatta) linear dental measurements to address two primary questions. First, how do dental trait heritabilities vary across a diverse set of primate species? Second, how do heritability estimates from dental measurements compare to evolvability estimates for the same traits? Identifying similarities and differences in dental trait heritabilities across non-human primate species will allow us to assess whether patterns observed are characteristic of a population, a family, or of primates. Comparison of h² and I_A will provide a useful test of these two parameters, specifically whether they reveal the same patterns in dental traits or whether, in combination, they provide new information about the genetic architecture of dental dimensions.

Materials and Methods

Populations and pedigrees

The Oak Ridge brown-mantled tamarin population was bred in captivity for biomedical research over several decades (Cheverud, 1996; Cheverud, 1995; Clapp & Tardif, 1985). The associated skeletal collection, now housed at the Osteometric Variation Analysis Laboratory (OVAL) at the University of Tennessee, includes hundreds of brown-mantled tamarin specimens that are part of an extended pedigree. A pedigree of 386 individuals, spanning four generations, was used in this study. Dams and sires are known for 190 individuals; the other 196 individuals are founders.

The Cayo Santiago rhesus macaques were introduced to the island near Puerto Rico in 1938 as a free-ranging population maintained for biomedical and behavioral research (Dunbar, 2012). Records of maternal parentage have been collected since the early 1950s and skeletal remains have been collected and maintained since the 1970s (Rawlins & Kessler, 1986). The skeletal collection, housed at the Caribbean Primate Research Center Laboratory of Primate Morphology and Genetics at the University of Puerto Rico, now contains hundreds of specimens from the Cayo Santiago population.

Although many paternity identities in the Cayo Santiago macaque population have been determined through genetic testing (Widdig et al., 2016), paternities are not known for most individuals in the skeletal collection. To maximize the use of the known maternities from this population, individuals with known mothers, based on behavioral observation, were assigned a “dummy sire”. In previous studies, these dummy sires were related to only one offspring in the pedigree (Joganic et al., 2012; Konigsberg & Cheverud, 1992), so that all individuals with the same dam were half siblings. The use of this half-sibling dummy sire model is likely to produce coefficients of relatedness that are smaller than the degree to which individuals are truly related across the population; this method may therefore inflate heritability estimates. To assess the impact of dummy sires on the estimation of heritabilities in the Cayo Santiago macaque population, all heritabilities were estimated twice in this population: once for all traits with a half-sibling dummy sire model, and once with a set of dummy sires assigned so that all individuals with the same dam were assigned the same dummy sire (Adams, 2011; Myers, Janzen, Adams, & Tucker, 2006). Estimates of σ²_A in this population are likely overestimated for the half-sib model and under-estimated in the full-sib model, so that the actual h² falls between these estimates. A pedigree containing 400 individuals was used for the macaque population. There are 66 founders with no known dam, and dams are known for the remaining 334 individuals. For the half-sibling model, 334 dummy sires were added to the pedigree and, for the full-sibling model, 152 dummy sires were added to the pedigree. The relationship between half-sibling and full-sibling h² estimates was assessed using a Pearson’s correlation test in SAS/STAT 14.1.

Measurements

Linear dental measurements were collected from 302 brown-mantled tamarin skeletons and 364 rhesus macaque skeletons. All measurements were taken using Mitutoyo nib-style digital calipers with a digital input tool to minimize human error during data entry.

Mesiodistal (MD) lengths and buccolingual (BL) breadths were measured from all teeth on half of the toothrow, excluding any teeth with wear or damage that could impact the dimensions of the tooth crown. Mesiodistal length for incisors, premolars, and molars was measured as the maximum length parallel to the lingual margin of the tooth crown and buccolingual breadth was measured as the maximum breadth perpendicular to the lingual edge of the tooth crown. For canines, mesiodistal length was measured as the maximum mesiodistal length, and the buccolingual breadth was the maximum breadth perpendicular to the mesiodistal length measurement. The left and right sides of the toothrow were considered interchangeable based on the evidence of complete pleiotropy between antimeres shown by previous studies (Hlusko et al., 2011; Stojanowski et al., 2017), so the half with the least damage and fewest missing teeth was measured for each individual. Measurements on teeth are denoted using standard terminology, wherein I = incisor, C = canine, P = premolar, and M = molar, with subscripts and superscripts indicating mandibular and maxillary teeth respectively.

Intra-observer measurement reliability was assessed by measuring ten individuals from each population three times. Calculation of measurement reliabilities was performed in Excel, where:

Reliability = 1 – (repeated measure variance / population variance)

Additional analyses are performed on all traits with reliability greater than 60%, and measurements with reliability below 80% are marked and discussed separately throughout this paper. Previous quantitative genetic studies of tooth dimensions have not reported measurement reliabilities for linear dental measurements (Hlusko et al., 2002; Stojanowski et al., 2017), although standard error of measurement estimates are provided for the baboon data elsewhere (Hlusko, 2000). Measurements of the tamarin teeth are especially prone to poor reliability since they are very small, and so it was deemed important to account for reliability in this study. Standard errors of measurements were also estimated as percentages and were less than 4% for measurements analyzed here. Incisor labio-lingual breadth was not measured due to the noticeable impact of wear on this trait in both samples.

Analyses

Following traditional quantitative genetic theory, the total phenotypic variance in a trait, σ²_P, can be decomposed into genetic and environmental variance, σ²_G and σ²_E respectively, so that:

$${{\text{σ}}^2}_{\text{P}}={{\text{σ}}^2}_{\text{G}}+{{\text{σ}}^2}_{\text{E}}$$ (1)

Variance related to dominance could not be estimated in the study populations because many full sibling relationships would be necessary for the estimation of σ²_D and are rare in the tamarin and macaque pedigrees. For this reason, the additive genetic variance (σ²_A) was estimated in place of σ²_G and the resulting heritability estimates reflect the narrow-sense heritability (h² = σ²_A/σ²_P), rather than the broad-sense heritability (h² = σ²_A/σ²_P). The phenotype of interest was modeled as

$$y=\mu 1_n+\left(X–1_{n}s’\right)\beta +a+e+\text{E}$$ (2)

where y is the n x 1 vector of phenotypic measurements, μ is the mean phenotype of the population, X is the n x k matrix of k covariates, 1_n is a vector of n ones, s is the vector of baseline covariates (equal to 0 for discontinuous covariates such as sex and birthplace, and equal to the mean value for continuous covariates such as age). β is the k x 1 vector of regression coefficients, a is the vector of additive genetic variances and e is the vector of random environmental effects (following Wang et al., 1997). The error term E has been added. The phenotypic covariance between individuals 1 and 2 for trait y is used to calculate σ²_A and σ²_E as

$$\text{Cov}\left(y_1,y_2\right)=2\Phi {{\text{σ}}^2}_{\text{A}}+I_n{{\text{σ}}^2}_{\text{E}}$$ (3)

where Φ is the n x n matrix of kinship coefficients and I_n is an n x n identity matrix. A general model, in which h² is estimated, is compared to restricted models, in which h² is constrained to zero, using likelihood-ratio tests. The likelihood-ratio test statistic (Λ) is calculated as

$$\Lambda =-2({\text{log-likelihood}}_{\text{general}}-{\text{log-likelihood}}_{\text{restricted}})$$

The likelihood-ratio test statistic follows a chi-squared distribution, providing the probability that the restricted model, in which h² is equal to zero, fits the data as well as the general model, in which h² is estimated without restriction. Heritability estimates are considered significantly different from zero when p<0.05.

Univariate quantitative genetic analyses were performed for each measurement in both populations using maximum likelihood-based variance decomposition performed in the open-source software package SOLAR 8.1.1 (Almasy & Blangero, 1998). The effects of covariates were estimated simultaneously using the screening function, which uses likelihood-ratio tests to compare models in which covariates are included to those without covariates. When the model with the covariate was significantly more likely than the model without the covariate (p<0.1), the covariate was included in the final model. For the macaque population, the effects of sex, age, and age-by-sex were estimated, whereas the effects of sex and birthplace (wild or captive birth, hereafter WC) were screened in the tamarin population. Age and age-by-sex covariates were screened to account for effects of tooth wear in the macaques, but age estimates were not available for any wild-born tamarins. For this reason, age was not included as a covariate for tamarin analyses. Traits were also screened for kurtosis, and an inverse normal distribution was applied to traits with kurtosis exceeding 0.8 (following Hlusko et al., 2002).

The heritabilities estimated in SOLAR are residual h² (h²_r), meaning phenotypic variance associated with significant covariates is removed prior to h² estimation. Heritability estimates described in the results and discussion are h²_r values. To assess the utility of I_A for comparing the genetic variance of primate dental traits, I_A was calculated manually based on total h². Total h² was calculated from h²_r by correcting for variance removed through the inclusion of covariates. The estimate of σ²_A was equal to the product of the total h² and σ²_P. I_A is estimated as σ²_A divided by the squared trait mean without sex correction, all multiplied by 100 to be expressed as a percentage (Hansen & Houle, 2008). To determine how inverse normal transformation would impact I_A estimation, I_A was estimated for traits with high kurtosis using transformed and untransformed trait values. The difference between transformed and untransformed I_A estimates was greater than 0.01 for ten traits, so I_A estimates are not presented for those traits.

Results

Measurement reliability

Measurement reliabilities are provided in Table 1. Sixteen of twenty-eight measurements from the brown-saddled tamarin population and twenty-six of twenty-eight measurements from the rhesus macaque population are reliable. Low reliability of three traits in the tamarin sample (I¹ length, I² length, I₁ length) merit their exclusion from additional analyses.

Table 1.

Measurement reliability for dental dimensions, grey-shaded cells indicate measurements with reliability below 80%, darker grey cells indicate measurements excluded from further analyses. MD: mesiodistal crown length, BL: buccolingual crown breadth, I: incisor, C: canine, P: premolar, M: molar.

	Saguinus fuscicollis				Macaca mulatta
	Maxillary		Mandibular		Maxillary		Mandibular
	MD	BL	MD	BL	MD	BL	MD	BL
I1	0.55		0.32		0.96		0.69
I2	0.17		0.65		0.99		0.99
C	0.81	0.86	0.69	0.77	0.97	0.99	0.97	0.99
P2	0.91	0.76	0.88	0.75
P3	0.81	0.88	0.87	0.77	0.83	0.97	0.93	0.95
P4	0.80	0.91	0.82	0.88	0.85	0.95	0.69	0.91
M1	0.90	0.72	0.93	0.78	0.93	0.88	0.95	0.93
M2	0.91	0.98	0.88	0.61	0.95	0.91	0.96	0.91
M3					0.97	0.97	0.98	0.94

Tamarins

Results of univariate analyses of dental dimensions in the Oak Ridge tamarins are provided in Table 2. Inverse normal transformation was applied to four traits (P⁴ breadth, C₁ breadth, P₂ breadth, M₁ breadth) to account for high measures of kurtosis. Covariate effects are incorporated into the final models for all but eight of the twenty-five dental measurements that were analyzed. Captive birth (WC) has a significant negative effect relative to wild birth on fifteen measurements. Sex is a significant covariate for four traits (P² breadth, P³ breadth, M² length, P₂ length). Covariates account for up to 7.8% of the total variance in a trait.

Table 2.

Heritability estimates from tamarin dental traits. Sample size (N), residual heritability (h²_r), standard error for h²_r (SE), significant covariates (C), and variance accounted for by significant covariates (σ²_C) from analyses in SOLAR are provided. Bold h²_r values are significantly different from zero (p<0.05). Trait mean, phenotypic variance (σ²_P), total heritability (h²) and evolvability (I_A) were calculated separately. MD: mesiodistal crown length, BL: buccolingual crown breadth. Traits with measurement reliability below 0.80 are shaded in gray.

Tooth	Trait	N	Mean	σ2P	h2r	SE	C	σ2C	h2	IA
C1	MD	273	2.34	0.05	0.70	0.11			0.70	0.66
	BL	271	1.95	0.03	0.65	0.13	WC	0.9	0.64	0.51
P2	MD	271	1.80	0.03	0.45	0.13	WC	7.6	0.42	0.34
	BL	274	2.14	0.03	0.56	0.11	Sex	0.7	0.56	0.32
P3	MD	256	1.57	0.02	0.29	0.13			0.29	0.19
	BL	280	2.44	0.04	0.61	0.12	Sex, WC	2.5	0.59	0.37
P4	MD	255	1.59	0.02	0.32	0.16			0.31	0.20
	BLK	276	2.64	0.04	0.68	0.11	WC	5.8	0.64
M1	MD	282	2.19	0.03	0.88	0.09			0.88	0.51
	BL	282	2.74	0.03	0.75	0.09	WC	2.6	0.73	0.28
M2	MD	250	1.43	0.03	0.31	0.13	Sex	1.4	0.30	0.40
	BL	270	2.27	0.05	0.88	0.07			0.88	0.87
I2	MD	266	1.34	0.02	0.48	0.14	WC	7.8	0.45	0.37
C1	MD	270	2.16	0.04	0.57	0.11	WC	3.5	0.55	0.48
	BLK	270	2.46	0.05	0.82	0.10	WC	1.0	0.82
P2	MD	274	2.11	0.04	0.39	0.13	Sex	2.0	0.38	0.36
	BLK	279	1.95	0.03	0.42	0.11	WC	0.9	0.42
P3	MD	245	1.71	0.02	0.29	0.14			0.29	0.19
	BL	272	1.78	0.02	0.82	0.10	WC	1.0	0.81	0.54
P4	MD	222	1.74	0.02	0.19	0.16	WC	4.1	0.18	0.12
	BL	242	1.84	0.03	0.49	0.13	WC	6.2	0.46	0.38
M1	MD	235	2.11	0.03	0.47	0.15	WC	6.4	0.44	0.26
	BLK	241	1.91	0.02	0.99	0.10	WC	2.6	0.96
M2	MD	218	1.97	0.02	0.45	0.18			0.45	0.28
	BL	241	1.63	0.01	0.92	0.10			0.92	0.45

^Kindicates inverse normalization was used to correct for skew

Quantitative genetic analyses of tooth size in the Oak Ridge tamarin population yield significant, non-zero h² estimates for twenty-five out of the twenty-six analyzed measurements. Only the h² of P₄ length is not significantly different from zero. Heritability estimates range from 0.185 (P₄ length) to 0.985 (M₁ breadth) in this tamarin population. Accounting for standard error, h² estimates overlap for most traits. Buccolingual breadth measurements produce larger h² estimates than mesiodistal length measurements. Eight of the ten largest h² estimates belong to buccolingual breadth dimensions and nine of the ten smallest h² estimates belong to mesiodistal length dimensions. Within each tooth, the buccolingual dimension produces a greater h² than the mesiodistal dimension for all teeth except C¹ and M¹.

Macaques

Results of univariate analyses of half-sibling and full-sibling pedigree models for the Cayo Santiago rhesus macaques are provided in Table 3. Inverse normal transformation was applied to twelve half-sibling traits and eleven full-sibling traits to account for high measures of kurtosis. Covariate effects are incorporated into the analyses for all 28 macaque dental measurements. Sex is a significant covariate for all traits across both pedigree models. Age has a significant effect on 13 traits across both pedigree models. Sex-by-age interaction has a significant effect on 6 half-sibling pedigree traits and 9 full-sibling pedigree traits. Covariates account for between 5.9% and 86.2% of the total phenotypic variance in a trait.

Table 3.

Heritability estimates from full- and half-sibling analyses of macaque dental traits. Sample size (N), residual heritability (h²_r), standard error for h²_r (SE), significant covariates (C), and variance accounted for by significant covariates (σ²_C) estimated in SOLAR are provided. Bold h²_r values are significantly different from zero (p<0.05). Trait mean, phenotypic variance (σ²_P), total heritability (h²) and evolvability (I_A) were calculated separately. MD: mesiodistal crown length, BL: buccolingual crown breadth. Traits with measurement reliability below 0.80 are shaded in gray.

Tooth	Trait	N	Mean	σ2P	Half-sib h2r	Full-sib h2r	Half-sib SE	Full-sib SE	C	Half-sib σ2C	Full-sib σ2C	Half-sib h2	Full-sib h2	Half-sib IA	Full-sib IA
I1	MD	258	6.24	0.12	0.89	0.41	0.24	0.15	sex	14.3	16.6	0.76	0.34	0.24	0.11
I2	MD	266	4.88	0.12	0.40	0.26	0.22	0.15	sex	37.8	38	0.25	0.16	0.13	0.08
C1	MDK	246	7.27	2.81	0.74	0.62	0.18	0.20	sex	62.4	61.6	0.28	0.24	-	-
	BL	251	6.19	1.46	0.89	0.43	0.19	0.18	sex, age, sex*age	85.9	86.2	0.13	0.06	0.48	0.23
P3	MDK	332	5.23	0.11	0.52	0.42	0.15	0.13	sex, sex*age	26.6	26.5	0.39	0.31	0.16	0.13
	BL	337	6.40	0.12	0.71	0.47	0.15	0.15	sex	15.9	16.6	0.60	0.40	0.17	0.11
P4	MDK	337	5.31	0.09	0.35	0.30	0.14	0.12	sex	7.6	7.6	0.32	0.28	-	-
	BL	332	6.90	0.13	0.57	0.45	0.14	0.14	sex, age, sex*ageFS	20.5	20.6	0.45	0.35	0.12	0.09
M1	MD	335	7.63	0.13	0.70	0.45	0.19	0.17	sex, age	16.4	17.9	0.58	0.37	0.13	0.09
	BLK	263	7.21	0.12	0.26	0.21	0.20	0.16	sex	19.4	19.4	0.21	0.17	0.05	0.04
M2	MD	342	8.80	0.18	0.46	0.46	0.15	0.14	sex	18.4	18.3	0.38	0.38	0.09	0.09
	BL	306	8.46	0.20	0.72	0.48	0.16	0.15	sex, age	31.2	32.1	0.49	0.32	0.14	0.09
M3	MD	259	8.86	0.21	0.41	0.35	0.17	0.16	sex, age, sex*age	42.4	42.3	0.23	0.20	0.06	0.06
	BLK	252	8.40	0.31	0.64	0.52	0.16	0.15	sex, age, sex*age	38.8	38.4	0.39	0.32	-	-
I1	MD	254	4.19	0.06	0.54	0.34	0.20	0.16	sex	10.0	10.1	0.48	0.31	0.16	0.10
I2	MD	241	4.01	0.29	0.45	0.36	0.17	0.14	sex	10.7	10.6	0.40	0.32	0.73	0.59
C1	MDK	235	4.61	0.88	0.52	0.33	0.26	0.22	sex, age	65.2	65.3	0.18	0.11	-	-
	BLK	213	7.37	3.42	0.42	0.31	0.23	0.18	sex, age	66.3	66.3	0.14	0.11	-	-
P3	MD	310	8.79	4.40	0.21	0.12	0.18	0.11	sex, age, sex*age	81.4	81.4	0.04	0.02	0.23	0.13
	BL	303	4.53	0.19	0.54	0.31	0.23	0.16	sex	49.9	50.5	0.27	0.15	0.25	0.14
P4	MD	322	5.83	0.12	0.64	0.55	0.14	0.13	sex, age, sex*age	11.4	11.3	0.57	0.49	0.20	0.17
	BLK	318	5.12	0.09	0.33	0.28	0.15	0.13	sex	8.9	8.9	0.30	0.26	-	-
M1	MDK	299	7.46	0.10	0.55	0.44	0.24	0.18	sex, age, sex*ageFS	19.8	20.9	0.44	0.35	0.08	0.07
	BL	236	5.87	0.07	0.50	0.08	0.43	0.18	sex	18.6	17.8	0.41	0.07	0.09	0.01
M2	MDK	332	8.58	0.16	0.51	0.36	0.18	0.16	sex	14.0	14.2	0.44	0.31	0.09	0.07
	BLK	305	7.16	0.14	0.44	0.31	0.17	0.14	sex, age	21.7	21.6	0.35	0.24	0.09	0.07
M3	MD	256	10.72	0.51	1.00	0.68	-	0.18	sex	9.8	5.9	0.90	0.64	0.40	0.28
	BL	253	7.49	0.16	0.59	0.64	0.15	0.16	sex, age, sex*ageFS	28.1	27.7	0.43	0.46	0.12	0.13

^Kindicates inverse normalization was used to correct for skew

^FSindicates that a covariate was statistically significant only in the full-sibling pedigree analysis

Quantitative genetic analyses of tooth size in the Cayo Santiago macaque population yield significant non-zero heritabilities for the same 25 traits (p<0.05) in the half- and full-sibling pedigree models. For three traits (M¹ breadth, P₃ length, M₁ breadth), h² estimates are not significantly different from zero. Estimates of h² from half- and full-sibling models are closely correlated (r=0.718, p<0.001). Heritability estimates are consistently smaller for full-sibling models than for half-sibling models of the same trait, the only exception being M₃ breadth (half-sibling h² = 0.591, full-sibling h² = 0.636). Half-sibling h² estimates range from 0.214 (P₃ length) to 1.0 (M₃ length), whereas full-sibling h² estimates range from 0.080 (M₁ breadth) to 0.675 (M₃ length). As in the tamarin sample, the margins of error for most traits in the macaque population overlap, although the confidence intervals of h² estimates at the extremes do not overlap. Buccolingual breadth measurements do not consistently have greater h² estimates than mesiodistal length measurements in the macaque population.

Evolvability

I_A estimates in the tamarin population range from 0.117 to 0.870 (Table 2). Evolvability estimates in the macaque population range from 0.047 to 0.732 for half-sibling pedigree models, and from 0.014 to 0.587 for full-sibling models (Table 3). I_A estimates are generally greater for tamarin dental traits than for half-sib and full-sib models of macaque dental traits. C¹ length, M² breadth and P₃ breadth have the greatest I_A estimates in the tamarin population, and C¹ breadth, I₂ length, and M₃ length have the greatest I_A estimates in the macaque population.

Discussion

Comparing heritabilities in macaques and tamarins to baboons and humans

Heritability estimates from the dental dimensions of these tamarin and macaque populations are moderate to high, with average h² values of 0.57 in tamarins and 0.55 and 0.39 in half- and full-sibling macaque analyses respectively. The range of h² estimates is broad in both tamarins and macaques, although tamarins and macaques exhibit slight differences in the distribution of h² estimates. Tamarins tend to have greater h² in buccolingual breadths than mesiodistal lengths whereas there is no consistent difference between buccolingual breadth and mesiodistal length h² estimates in macaques.

Maximum-likelihood h² estimates of linear dental dimensions have been published for the SNPRC baboon population (Papio spp.) (Hlusko & Mahaney, 2009), and a contemporary human population from James Island, South Carolina (Stojanowski et al., 2017). Combined with h² estimates from teeth of brown-mantled tamarins and rhesus macaques, h² estimates from baboons and humans indicate that dental dimensions are highly heritable in primates. Patterns of heritability observed in previous studies can now be assessed in a more phylogenetically diverse primate sample by comparing these four species.

In the South Carolina Gullah population, maxillary tooth lengths yield consistently greater h² estimates than the lengths of homologous mandibular teeth (Stojanowski et al., 2017). The authors hypothesize that maxillary h² estimates are greater than mandibular h² estimates due to greater constraints on the development of maxillary teeth, yet recognize that the pattern is weak when standard error for h² estimates is taken into account. In the Oak Ridge tamarins, Cayo Santiago macaques, and SNPRC baboons (data from Hlusko et al., 2011), h² values are not consistently greater in maxillary dental dimensions. The pattern observed by Stojanowski et al. (2017) may therefore be characteristic of humans or of the South Carolina Gullah people specifically but does not appear to be shared among non-human primates.

Heritability estimates in the South Carolina Gullah population align with dental morphogenetic field theory (Butler, 1939; Dahlberg, 1945): the h² of the key or pole tooth (where a key or pole tooth is the mesial-most tooth of its type in the maxilla or mandible) tends to be greater than the h² of more distal teeth of the same type (Stojanowski et al., 2017). This pattern is also observed in the Oak Ridge brown-mantled tamarin premolar and molar lengths, but not premolar or molar breadths. The pattern is less consistent in the Cayo Santiago macaque tooth length and breadth h² estimates. A similar trend is not observed in SNPRC baboon tooth dimension h² estimates (Hlusko et al., 2011).

As has been noted previously (Stojanowski et al., 2017), the morphogenetic field pattern observed in the South Carolina Gullah population may not be biologically significant. The differences between the h² estimates being compared are often smaller than the margins of error associated with each h² estimate. Furthermore, morphogenetic field theory (Butler, 1939) and more recent models of odontogenesis (Mitsiadis & Smith, 2006) describe how tooth morphology is genetically regulated, and not the degree to which tooth dimensions are heritable. The morphogenetic field pattern observed in human and tamarin tooth lengths may reflect environmental and common environmental effects during odontogenesis. Common environmental effects can inflate h² estimates in features that form early in development (Asadi Fozi, Van Der Werf, & Swan, 2005; Koivula, Strandén, & Mäntysaari, 2009; Kruuk & Hadfield, 2007), such as pole tooth crown dimensions. Estimation of genetic correlations between dental dimensions in these non-human primate populations will bear more directly on questions of dental patterning in primates.

Dental dimensions are consistently highly heritable in non-human primates based on comparisons of h² estimates from the SNPRC baboons, Oak Ridge tamarins, and Cayo Santiago macaques. The estimation of significant heritabilities in papionins and tamarins, despite differences in dental formula, body size, and evolutionary history, is evidence of the phylogenetic utility of primate dental dimensions. Based on these high h² estimates from dental dimensions, we may also expect groups of closely related individuals to have similar dental dimensions. Such groups will not accurately represent population- or species-level variation in dental traits. Anatomical and archaeological museum collections containing groups of closely related individuals may therefore underestimate species-level dental variation (Šešelj et al., 2015). Knowledge of the history of skeletal collections and how they were acquired can help to avoid bias in comparative studies.

Comparing heritability and evolvability estimates

Low h² estimates indicate reduced additive genetic variance or increased phenotypic variance associated with a given trait in a population (Falconer & Mackay, 1996; Fisher, 1930). Life history traits and traits closely associated with fitness, such as litter size and number of offspring, tend to have lower h² than traits more distantly associated with fitness, such as morphological traits (Mousseau & Roff, 1987). Houle (1992) attributes the observed difference in h² between life history and morphological traits to environmental variance, and states that estimates of I_A may be better suited to between-population comparisons of σ²_A (Hansen et al., 2011; Houle, 1992).

Heritability estimates for one tamarin dental dimension (P₄ length) and three macaque dental dimensions (M¹ breadth, P₃ length, M₁ breadth) are not significantly different from zero. Most of these traits (tamarin P₄ length, macaque M¹ breadth and P₃ length) have low h² estimates with standard error values similar to those of other traits. This could indicate recent selection reducing σ²_A or greater environmental effects increasing σ²_P in these traits. The h² estimates for macaque M₁ breadth differ considerably between the full- and half-sibling pedigree analyses and should be considered inconclusive.

Low h² of macaque P₃ length is of interest since the P₃ is morphologically derived, being mesiodistally expanded in males of most anthropoid primate taxa to form part of the canine-premolar honing complex. The functional relationship between the honing premolar and the canine teeth may impact the h² of P₃ length in this macaque population if the honing complex is integrated genetically as it is phenotypically (Delezene, 2015; Greenfield, 1996); genetic correlation estimates are needed to test this hypothesis. The confounding influence of extreme sexual dimorphism in macaque P₃ length may also impact h² estimation for this trait, yet h² estimates for canine dimensions are significant in this macaque population despite their extreme sexual dimorphism. Hlusko et al. (2011) also found that the h² of P₃ length is not significantly different from zero in the SNPRC baboons. The low h² of macaque and baboon P₃ length could result from recent selection on the honing complex in cercopithecoid primates. I_A estimates can be examined alongside h² estimates to separate traits with low σ²_A from those with high σ²_P.

Estimates of h² and I_A tend to covary across dental dimensions; those traits with the highest h² estimates have the highest I_A estimates (Figure 1). This contradicts the idea that h² and I_A are independent (Hansen et al., 2011), perhaps because all of the variables analyzed are measured on the same scale. Two tamarin traits (C¹ length, M² breadth) and two macaque traits (I₂ length, P₃ length) have large I_A relative to h², indicating that both σ²_A and σ²_P are high relative to the trait mean. With the exception of tamarin M² breadth, traits with high I_A relative to h² fall in or near the canine-premolar honing complex. Genetic and phenotypic correlations in the honing complex may contribute to increased σ²_A and σ²_P in this region. The high I_A of tamarin M² breadth may reflect an impact of third molar agenesis in the callitrichine lineage on genetic and developmental regulation of molar size.

Figure 1.

Residual heritability (h²_r) and evolvability (I_A) estimates in brown-mantled tamarins (white triangles) and half- (grey circles) and full-sibling (black circles) analyses of rhesus macaques. Traits with especially high I_A relative to h²_r are labeled.

Estimation of I_A provides a view of the genetic variance in macaque and tamarin dental traits that shows more clearly whether low σ²_A or high σ²_P is responsible for differences in h². Where comparisons of h² estimates alone show few clear patterns, comparisons of I_A and h² estimates together show high σ²_A and high σ²_P in and around the canine-premolar honing complex. This could result from a wide range of evolutionary and environmental scenarios, including diversifying selection acting to increase or maintain sexual dimorphism in these traits or from genetic correlations increasing σ²_A in some traits more than others. Although h² and I_A are closely related in most of the dental traits analyzed here, instances of disassociation can be informative and justify the estimation of both h² and I_A in quantitative genetic analyses.

Effects of captive vs. free-ranging habitat

This study provides an unprecedented opportunity to compare h² and I_A estimates of dental traits between captive and free-ranging non-human primate populations. The free-ranging Cayo Santiago macaque population are expected to have smaller h² estimates because, as a free-ranging population, they are expected to encounter less favorable environmental conditions than captive tamarins or baboons (Charmantier & Garant, 2005). The overall similarities in h² estimates between free-ranging macaques and captive tamarins shown here indicate, however, that environmental differences have a negligible impact on estimates of tooth dimension h² in these populations. The environmental conditions in which the Cayo Santiago macaques live may not be less favorable than those of the captive tamarins, perhaps because they are provisioned. Alternatively, dental h² may not be impacted by the differences in environmental conditions that exist between captive and free-ranging settings. Differences in mean dental traits have been observed between captive and wild primate populations. Captive-birth was a significant covariate for fifteen of twenty-six traits in the Oak Ridge brown-mantled tamarins, and the dental dimensions of the SNPRC baboons are significantly different from those of a wild baboon population (Hlusko & Mahaney, 2007). Despite the effect of captivity on trait means, these results and previous analyses find no evidence that captivity impacts the σ²_P or h² of primate dental dimensions (Hlusko & Mahaney, 2007). The effect of captivity on dental dimension h² estimates should also be assessed in a wild primate population without provisioning.

Challenges

The sampling variance that accompanies the estimation of h² of dental measurements in primate populations is large and limits the strength of conclusions that can be drawn from these analyses. As the Cayo Santiago skeletal collection grows to include more individuals with known paternity, it will be possible to estimate quantitative genetic parameters in this population with less error. The continued collection of skeletal materials and dental impressions at long-term primate research sites will greatly contribute to the study of skeletal and dental trait heritability. Estimates of I_A were not presented for traits characterized by extreme sexual dimorphism, because the distribution of these data had to be transformed prior to h² estimation. With larger and more powerful samples, the effect of sexual dimorphism on h² could be examined by estimating h² in males and females separately (Wolak, Roff, & Fairbairn, 2015). Although the available pedigree and phenotype data for the Cayo Santiago rhesus macaques are not yet adequate for the estimation of quantitative genetic parameters separately in each sex, this may be highly informative in future studies.

Conclusion

This study provides the first published estimates of tooth size heritability in a platyrrhine population and in a free-ranging cercopithecoid population, and the first tooth size evolvability estimates from a primate. These quantitative genetic parameters demonstrate the high heritability of dental dimensions in a phylogenetically diverse extant primate sample and a positive relationship between heritability and evolvability in primate dental traits. Although developmental patterns are weakly observed in the dental heritability estimates of tamarins and macaques, estimation of the genetic correlations will yield greater insight. Comparisons of h² and I_A estimates indicate that dimensions in and near the canine-premolar honing complex may exhibit greater additive genetic and phenotypic variance than other dental traits, perhaps due to sexual dimorphism and genetic integration with the honing complex. Continued preservation of skeletal and dental material and pedigree data from captive and wild primate populations at long-term study sites will be invaluable to future research on the heritability and evolvability of skeletal and dental morphology in primates.