Skip to main content
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2020 Apr 1.
Published in final edited form as: Environ Int. 2019 Feb 12;125:470–477. doi: 10.1016/j.envint.2019.01.070

Blood Lead Levels and Timing of Male Sexual Maturity: A Longitudinal Study of Russian Boys

Paige L Williams a,b, Andrea Bellavia c, Susan A Korrick c,d, Jane S Burns c, Mary M Lee e, Oleg Sergeyev f,g, Russ Hauser b,c, Russian Children’s Study Team
PMCID: PMC6388626  NIHMSID: NIHMS1521510  PMID: 30769180

Abstract

Background:

Higher blood leads levels (BLLs) have been linked to neurologic deficits and impaired growth, but few studies have evaluated their association with timing of sexual maturity or pubertal progression in boys.

Methods:

In a longitudinal cohort of Russian boys enrolled at age 8-9 and followed to adulthood, BLLs were measured at study entry, and pubertal staging (genitalia and pubic hair) and testicular volume (TV) measurements were obtained annually. We used interval-censored regression models to estimate differences between boys with higher BLL (≥ 5 μg/dL) and lower BLL in mean ages at sexual maturity (genitalia stage 5, pubic hair stage 5, or TV≥20 mL) and duration of pubertal progression (onset to maturity), adjusting for potential confounders. Mediation analyses were conducted to quantify the percent of lead’s effect attributable to its association with reduced somatic growth.

Results:

Among 481 evaluable boys, 28% had BLL ≥ 5 μg/dL. Adjusted mean ages at sexual maturity were 14.7 years for genitalia, 16.1 for pubic hair, and 13.9 for TV. In adjusted models, boys with BLLs ≥ 5 μg/dL had later maturity than those with lower levels by 4-5 months depending on pubertal indicator. In mediation analyses, height and body mass index at age 11 accounted for 40-71% of the shift in age at maturity for boys with higher compared to lower BLLs. Higher BLLs were not associated with pace of pubertal progression.

Conclusions:

Higher lead levels were associated with later attainment of sexual maturity in males, but not with the duration of pubertal progression. A high proportion of the delay in sexual maturity for boys with higher as compared to lower BLL was shown to be attributable to mediating effects of BLL on reduced growth.

Keywords: puberty, growth, sexual maturity, blood lead, endocrine disruptors, interval-censoring, mediation analysis, pubertal progression

1. Introduction

There is increasing evidence that environmental exposures play a role in childhood growth and development.1 Heavy metals such as lead, mercury, and arsenic are known endocrine disruptors2, and evidence suggests that lead in particular may interfere with pubertal development and growth.311 Epidemiological studies have identified an association between blood lead and diminished early childhood growth, particularly in boys.37 Lead exposure decreases concentrations of growth hormone, insulin-like growth factor 1 (IGF-1), testosterone and other hormones responsible for growth and pubertal development in animal studies, suggesting an endocrine mechanism for such effects.811

The Centers for Disease Control and Prevention recently lowered the reference value for blood lead levels (BLLs) from 10 to 5 ug/dL,12 acknowledging that even BLLs well below 5 μg/dL are associated with neurologic deficits.1314 The most common routes of lead exposure are ingestion via diet from lead contamination in packaging or soils, ingestion of water contaminated by lead-containing pipes or fixtures, and childhood “hand-to-mouth” behaviors, including ingestion of particles or dust from deteriorating lead-based paint. Lead exposure may also result from combustion products of past leaded gasoline use or other industrial emissions.4 BLLs in some areas of Russia and other countries remain particularly high.1518 While levels in the US have declined over time, rates in some subgroups remain high and both geographic and racial disparities persist.14, 19

Using data from a longitudinal cohort of 516 Russian boys, we previously reported an association of higher peripubertal BLLs with both timing of pubertal onset and reduced growth based on three years of follow-up, and reduced IGF-1.2021 We recently extended these findings to demonstrate associations between higher peripubertal BLLs and shorter height through age 18 years, with the most pronounced decrements between ages 12-15 years.2223 These findings suggest a persistent effect of lead on linear growth particularly during later pubertal development, which may affect subsequent timing of sexual maturity. In this analysis, we consider the association of peripubertal BLLs with timing of sexual maturity and duration of pubertal progression, and examine the proportion of any identified delays attributable to associations of BLLs with reduced somatic growth.

2. Patients and Methods

2.1. Study population

The Russian Children’s Study is a prospective cohort study of 516 boys in Chapaevsk, Russia, enrolled from 2003-2005 at ages 8-9 years, as previously described.20,22,23 The study was approved by the Institutional Review Boards of the Chapaevsk Medical Association, Harvard T.H. Chan School of Public Health, and Brigham and Women’s Hospital. The parent or guardian signed an informed consent and the boy signed an assent form; at age 18 years boys signed an informed consent. We excluded 17 orphans (missing maternal and birth characteristics) and 10 boys with chronic health conditions which could impact childhood development, leaving 489 eligible boys; 481 (98%) had available BLLs and were included in this analysis. At study entry, each boy’s mother or guardian completed a health and lifestyle questionnaire that included birth and medical history, demographic and socio-economic status indicators. A validated Russian Institute of Nutrition semi-quantitative food frequency questionnaire was used to ascertain the child’s dietary intake.24

2.2. Growth measurements and pubertal assessments

At study entry and at annual follow-up visits, a standardized anthropometric examination by a trained research nurse was performed per written protocol and without knowledge of the boys’ lead levels. Body mass index (BMI; kg/m2) was calculated from weight (measured to nearest 100 grams) and height (to nearest 0.1 cm) measurements. Age-adjusted z-scores were calculated for height (HTZ) and BMI (BMIZ) using World Health Organization (WHO) standards (http://www.who.int/childgrowth/).25 Pubertal status was staged from 1 to 5 via examination by a single clinician (O.S.) according to internationally accepted criteria.2628 Pubarche (pubic hair stage, P) was determined by extent of terminal hair growth. Genital staging (G) was assessed by genital size and maturity. Testicular volume (TV) was measured using an orchidometer. Three different measures of sexual maturity were considered as separate indicators: TV ≥ 20 ml of either testis, genitalia stage 5 (G5), and pubic hair stage 5 (P5). Duration of pubertal progression was defined as time from pubertal onset (TV>3 mL, genitalia stage ≥2 (G2), pubic hair stage ≥2 (P2), respectively) to sexual maturity, separately for each pubertal indicator.

2.3. Blood Lead Analysis

A 3.0 ml venous blood sample was collected in a trace metal–free vacutainer tube (Becton-Dickinson, Franklin Lakes, NJ) at study entry, when boys were aged 8 to 9 years old. Whole blood samples were diluted with a matrix modifier solution and analyzed by Zeeman background-corrected flameless graphite furnace atomic absorption (ESA Laboratories, Chelmsford, MA). The limit of detection (LOD) was 1.0 μg/dl. For 14 of 481 boys (2.9%) with levels below the LOD, BLLs were set to 0.5μg/dl.

2.4. Statistical analysis

We considered longitudinal data based on initial entry visits and up to 12 annual follow-up visits conducted through June 2017. Our primary analyses compared pubertal outcomes between boys with ‘higher’ (≥5 μg/dL) versus ‘lower’ (<5 μg/dL) peripubertal BLLs; sensitivity analyses were also conducted using log-transformed BLLs as a continuous predictor, and by treating BLL in three categories: low (≤2 μg/dL), moderate (3-4 μg/dL), or high ≥5 μg/dL). Interval-censored approaches for survival outcomes were used to account for the fact that sexual maturity was only known to occur between study visits rather than on a specific date; this approach also allowed for right-censored outcomes when sexual maturity had not yet occurred as of a boy’s last study visit. Interval-censored models were fit assuming a normal distribution for age at sexual maturity using accelerated failure time models.2933 We also compared our findings with those based on a Weibull model and results were very similar, but the normal model provided a better fit based on Akaike’s Information Criterion. We estimated the mean age at sexual maturity (with 95% confidence intervals (CIs)) separately for each of the three pubertal indicators, overall and stratified by BLL categories. We also present the estimated shift in mean age for those with higher vs lower BLLs with and without adjustment for confounders.

For comparison with other cohorts and to allow additional context for pubertal progression measures, we also fit similar interval-censored models for the outcome of pubertal onset, separately for each pubertal indicator (G2, P2, and TV>3 mL). For pubertal onset, the interval-censored modeling approach also accounted for left-censored outcomes for boys with pubertal onset prior to enrollment (note: no boys were sexually mature at entry so none were left-censored for that outcome).

Duration of pubertal progression could only be estimated among those boys with observed pubertal onset (either before study entry or during study follow up). Due to the lack of standard statistical software for evaluating doubly-censored outcomes, this duration was approximated by setting the age at pubertal onset as the midpoint between the two visits during which onset occurred, and treating the age at maturity as interval-censored (between the two visits at which it was noted to occur) or right-censored. Boys with pubertal onset prior to study entry were estimated to have had onset 6 months prior to entry. We again used accelerated failure time models for interval-censored data assuming a normal distribution for progression times, and present the estimated shift in mean duration of pubertal progression for those with higher vs lower BLLs, with and without adjustment for confounders.

We evaluated potential confounders identified from previous literature and our own research, including the boy’s birth weight, prenatal exposure to maternal alcohol and tobacco, maternal age at son’s birth, household characteristics including income level, parental education, and whether the biological father lived in the same household, the boy’s physical activity, and his nutritional status determined by caloric intake and percent of fat and protein intake. Potential confounders with p<0.20 in unadjusted models were included in multivariable models, which were then reduced to include only those whose inclusion resulted in a change in estimated lead effect greater than 10%. An additional sensitivity analysis was conducted further adjusting for maternal age at menarche (which was unavailable on 39 women). We did not adjust for growth measures at the time of BLL measurement because growth measures could be considered to be on the causal pathway between lead exposure and pubertal maturation. Instead, we applied a mediation approach to more carefully address the complex relationship between BLL, growth, and timing of maturity.

Mediation analysis was conducted to partition the total effect of higher vs lower BLLs on the age at sexual maturity into a direct effect of the lead exposure and an indirect effect of lead acting through its effects on height and BMI Z-scores (mediators) at age 11 years.31 This time point was chosen because no boys had reached sexual maturity prior to age 11 in this cohort, thus ensuring that the mediators were measured prior to the outcome of interest. In addition, this time point reflected growth several years after the measurement of BLLs, thus reducing the possibility of confounding by reverse causation due to dilution effects related to body size at the time of BLL measurement. The age at maturity was modeled using interval-censored models as a function of BLL without and with inclusion of the mediators (and other covariates) to obtain the total effect and direct effect of higher BLL, respectively.33 HTZ and BMIZ were jointly evaluated as potential mediators, and their indirect effects were calculated using the “product method” as described by VanderWeele et al.34,35 We present results in terms of proportion mediated, calculated as the ratio between the indirect and the total effects. Possible exposure-mediator and mediator-mediator interactions were also explored.36,37 However, only isolated evidence of interaction was detected, and the proportion mediated remained similar with and without inclusion of interactions. Thus, all results presented were obtained from models without interaction terms. Details of mediation methods are provided in online Supplemental Methods. All analyses were conducted with SAS Version 9.4, and statistical significance was set at alpha=0.05.

3. Results

3.1. Study population and blood lead concentrations

Among 489 eligible boys enrolled at age 8 or 9 years, 481 (98%) had BLLs measured. Birth, maternal, and household characteristics are presented in Table 1. The boys had a mean BMI of 15.9 at study entry, and other baseline growth measures consistent with WHO Child Growth Standards.25 The median BLL was 3 μg/dL, with 28% ≥ 5 μg/dL (range: 0.5-31 μg/dL). Boys with higher BLL at entry tended to be shorter and weigh less, and more often had lower socioeconomic status as reflected by parental education and household income than those with lower BLL. Retention rates were relatively high, with ~4% loss to follow-up per year and 64% retained to at least age 18 years. As of June 2017, the median follow-up (interquartile range, IQR) was 10.5 (8.0, 11.7) years, and age at latest visit was 19.0 (16.1, 20.1) years.

Table 1:

Demographic, maternal, and body size characteristics among 481 Russian Boys enrolled between 2003 and 2005 and followed until 2017

Characteristic Overall N=481) High Blood Lead (≥ 5μg/dL) (N=134) Low Blood Lead (<5 μg/dL) (N=347)
Growth Measurements at Study Entry
 Age (years) 8.4 (0.5) 8.5 (0.5) 8.4 (0.5)
 Height (cm) 130.2 (6.3) 128.5 (6.9) 130.8 (5.9)
 Weight (kg) 27.2 (5.5) 25.9 (5.2) 27.6 (5.6)
 Body mass index 15.9 (2.3) 15.6 (2.1) 16.1 (2.3)
 Total caloric intake (tertiles)
  High 155 (32%) 61 (46%) 94 (27%)
  Moderate 163 (34%) 38 (28%) 125 (36%)
  Low 160 (33%) 34 (25%) 126 (36%)
 Percent calories from fat 34.0 (5.9) 33.4 (6.1) 34.3 (5.8)
 Percent calories from protein 11.6 (1.6) 11.5 (1.6) 11.6 (1.6)
Growth Measurements at Age 11 years (n=426)
 N 426 122 304
 Height (cm) 143.7 (6.8) 141.5 (6.9) 144.5 (6.6)
 Height Z-score 0.06 (1.01) −0.27 (1.01) 0.20 (0.98)
 Body mass index 17.3 (3.1) 16.7 (2.8) 17.6 (3.2)
 BMI Z-score −0.14 (1.36) −0.40 (1.27) −0.03 (1.38)
Birth and Neonatal History
 Birth weight (kg) 3.34 (0.53) 3.25 (0.55) 3.38 (0.51)
 Gestational age (weeks) 39.0 (1.8) 38.9 (2.0) 39.1 (1.6)
 Preterm (gestational age<37 weeks) 37 (8%) 14 (10%) 23 (7%)
 Breastfeeding duration (weeks) 27.7 (34.5) 35.4 (45.1) 24.6 (28.9)
Household Characteristicsa
 Low household income 293 (61%) 99 (74%) 194 (56%)
 Low parental education 38 (8%) 18 (13%) 20 (6%)
 No biological father residing at home 166 (35%) 56 (42%) 110 (32%)
Maternal Characteristics and Exposures During Pregnancy
 Mother’s age at son’s birth 23.9 (5.1) 24.3 (5.2) 23.7 (5.0)
 Mother’s age at menarche 13.3 (1.4) 13.4 (1.3) 13.3 (1.4)
 Maternal alcohol consumption 59 (13%) 24 (18%) 35 (10%)
 Maternal tobacco smoking 36 (8%) 15 (12%) 21 (6%)
 Any smoking in household 228 (48%) 73 (55%) 155 (46%)

Values reported are mean (SD) or N (%). Summary statistics calculated among those with non-missing data; unavailable data included dietary information (n=3), birth weight (n=3), gestational age (n=4), breastfeeding duration (n=11), family income level (n=1), parental education level (n=1), mother’s age at son’s birth (n=5), maternal alcohol consumption (n=16), maternal tobacco smoking (n=13), maternal age at menarche (n=39), and household smoking during pregnancy (n=9).

a

Low household income defined as < US $250/month at study entry. Low parental education classified as secondary school or less.

3.2. Mean Ages at Pubertal Onset and Sexual Maturity, and Association with Blood Lead Levels

Mean ages (95% CI) at pubertal onset and sexual maturity for the study population as a whole are summarized in Figure 1, and ranged from 9.5-11.8 years for pubertal onset, and from 13.9-16.1 for sexual maturity (depending on pubertal indicator). The mean duration of pubertal progression was estimated as 5.1 years for genitalia, 4.5 for pubic hair, and 3.5 years for maturation of TV (Supplemental Figure 1). Between 64% and 89% of pubertal outcomes were determined to occur during study follow-up, depending on the outcome and pubertal indicator (see Supplemental Table 1 for a summary of censoring and timing of events relative to study entry and latest study visit).

Figure 1. Adjusted mean ages (years) at pubertal onset and sexual maturity, overall and by blood lead level.

Figure 1.

Pubertal onset defined by stage 2 for genitalia (G2), stage 2 pubic hair (P2), or testicular volume (TV) >3mL. Sexual maturity defined by stage 5 for genitalia (G5) and pubic hair (P5) or TV ≥ 20mL. “HighLead” defined as blood lead level (BLL) ≥ 5μg/dL and “LowLead” as BLL < 5μg/dL. Adjusted mean ages were estimated using the same accelerated failure time models summarized in Table 2, adjusted for specific covariates for each outcome as indicated, with birth weight set to its mean level (3.34 kg) and categorical covariates weighted according to their percent in the study population.

Boys with higher BLL had pubertal onset 7.7-8.4 months later on average than those with lower BLL, after adjustment for birthweight, nutritional intake, and socioeconomic covariates [Figure 1 and Table 2; note that Figure 1 presents mean ages in years while Table 2 summarizes differences in terms of months, but the differences themselves are equivalent and based on the same underlying models]. Boys with higher BLL also had significantly later adjusted mean age at sexual maturity, with 4.2-5.1 month later attainment compared to boys with lower BLLs in adjusted models (see Table 2). There was no difference in mean time in progression from pubertal onset to sexual maturity by BLLs based on pubic hair or TV, but a mean decrease of about 3.8 months (95% CI: −7.9, 0.4) in progression from G2 to G5 for boys with higher compared to lower BLLs (Table 2, Supplemental Figure 1). Estimated differences in median ages at pubertal outcomes were generally very similar when based on a Weibull model rather than under a normal distribution (Supplemental Table 2), with the exception that the Weibull model yielded a much larger difference in age at sexual maturity by TV than the normal model (12.5 vs 7.0 months).

Table 2.

Mean shifts in timing of pubertal outcomes (pubertal onset, sexual maturity, and time to progression) for boys with higher (≥5 μg/dL) compared to lower blood lead levels

Unadjusted Models Adjusted Modelsa

Pubertal Indicator N Shift in mean age in months (95% CI) P-value N Shift in mean age in months (95% CI) P-value
Age at Pubertal Onset

Genitalia (G2) 481 9.01 (4.32,13.79) <0.001 472 8.40 (3.70,13.10) <0.001
Pubic Hair (P2) 481 7.76 (3.14,12.37) 0.001 478 8.12 (3.46,12.78) <0.001
Testicular Volume (TV>3mL) 481 8.05 (3.84,12.26) <0.001 475 7.68 (3.46,11.90) <0.001

Age at Sexual Maturity

Genitalia (G5) 481 3.60 (−0.02,7.23) 0.051 473 4.20 (0.56,7.84) 0.024
Pubic Hair (P5) 481 5.64 (1.10,10.19) 0.015 480 4.23 (−0.31,8.77) 0.068
Testicular Volume (TV≥20mL) 481 7.00 (3.53,10.47) <0.001 475 5.14 (1.70,8.58) 0.003

Duration of Pubertal Progression

Genitalia (G2 to G5) 447 −3.62 (−7.85,0.61) 0.093 443 −3.76 (−7.93,0.42) 0.078
Pubic Hair (P2 to P5) 415 −0.37 (−5.35,4.60) 0.88 411 −1.82 (−6.91,3.28) 0.48
Testicular Volume (TV >3 mL to ≥20 mL) 432 −1.10 (−4.84,2.64) 0.56 428 −1.19 (−4.92,2.54) 0.53
a

Adjusted models include the following additional covariates:

For pubertal onset: birthweight, high caloric intake, low maternal education, and biological father not in home (for G2); high caloric intake and no biological father in home (for P2); birthweight, high caloric intake, and no biological father in hone (for TV2).

For sexual maturity: high caloric intake and mother<20 yrs at birth of son (for G5); low income level and no biological father residing in household (P5); birthweight, low maternal education level, and no biological father residing in household (TV5).

For pubertal progression, restricted to those with observed pubertal onset during follow-un: birthweight and mother<20 yrs at birth of son (for G2 to G5); low income level and low maternal education (for P2 to P5); birthweight (for TV > 3mL to ≥20mL).

In sensitivity analyses, associations remained similar after further adjustment for maternal age at menarche (data not shown). Later adjusted mean ages at onset and maturity were confirmed in models evaluating log-transformed BLLs (Supplemental Table 3). These models indicated a 3.7 to 5.0 month delay in pubertal onset, and a 2.8 to 4.9 month delay in sexual maturity per 1 unit increase in log lead levels (which corresponds approximately to a shift from the 25th percentile to the 75th percentile of observed BLLs). Boys with peripubertal BLL in the moderate range (3-4 μg/dl) did not have significantly later ages of onset or maturity than those with low BLL (0-2 μg/dl), but a significantly increasing trend in mean ages over the three lead ranges was apparent (Supplemental Table 4).

While boys with higher BLL had later attainment of sexual maturity, few had not attained maturity by age 18 according to genitalia stage or TV, among those followed to at least age 18. Only 1 did not attain mature genitalia until after age 18 (0.3% of 377 evaluable), and only 3 did not attain maturity by TV until after age 18 (0.8% of 389 evaluable). A higher percentage did not attain mature pubic hair until after age 18 (8.1% of 335 evaluable), and this finding was more common among boys with higher than lower BLLs (13.3% vs 6.1%, Fisher’s exact p=0.041).

3.3. Association of Child and Maternal Characteristics with Timing of Pubertal Outcomes

We observed significant associations of household characteristics, nutritional status, birth characteristics, and maternal factors with age at sexual maturity. Higher caloric intake was associated with earlier age at sexual maturity according to genitalia staging. Lower household income, lower parental education, biological father not residing in household, and younger maternal age at delivery were associated with later ages at sexual maturity. Higher birthweight was associated with earlier age at sexual maturity based only on TV. Older maternal age at menarche was associated with later age at sexual maturity for all three pubertal indicators. Similar associations were observed for these covariates with boys’ age at pubertal onset. Birthweight and socioeconomic covariates showed associations with time to progression from pubertal onset to sexual maturity, but no association with maternal age at menarche was observed. Covariates included in final adjusted models for each pubertal outcome are noted in Table 2.

3.4. Analysis of Growth Measures as Mediators of BLL effect on Sexual Maturity

In mediation analyses, models further included growth measurements at age 11 to better understand what portion of the shift in mean age at sexual maturity was attributable to the effect of BLL on growth. Boys with higher BLLs demonstrated significantly lower HTZ and BMIZ at age 11.22 The direct and indirect effects of BLL on age at sexual maturity are summarized in Table 3. The association of peripubertal BLL with HTZ at age 11 accounted for 34%-53% of BLL’s total effect on age at maturity, while BMIZ at age 11 accounted for only 6%-23%. The indirect effect of BLL acting through its combined effect on both HTZ and BMIZ accounted for 40% of the effect on age at maturity based on TV and over two-thirds of its effect on maturity based on genitalia and pubic hair.

Table 3.

Mediation analysis for direct and indirect effects of blood lead levels on age at sexual maturity, accounting for growth measures (height and BMI Z-scores) at age 11

Pubertal Indicator Sample Size Effect Type Adjusted mean shift in agea (months), (s.e.) %of Total P-value
Genitalia (G5) 422 Direct 1.46 (1.77) 32.7 0.41
Total Indirect 3.01 (0.71) 67.3 <0.001
---Indirect HTZ 2.37 (0.66) 53.0 <0.001
---Indirect BMIZ 0.64 (0.34) 14.3 0.062
Total 4.42 (1.86) 100.0 0.018
Pubic hair (P5) 425 Direct 1.44 (2.12) 29.1 0.50
Total Indirect 3.52 (0.91) 70.9 <0.001
---Indirect HTZ 2.36 (0.73) 47.5 0.001
---Indirect BMIZ 1.16 (0.59) 23.4 0.048
Total 4.71 (2.32) 100.0 0.042
Testicular Volume (≥20 mL) 422 Direct 3.12 (1.68) 59.8 0.063
Total Indirect 2.10 (0.62) 40.2 <0.001
---Indirect HTZ 1.78 (0.58) 34.2 0.002
---Indirect BMIZ 0.32 (0.26) 6.1 0.22
Total 5.25 (1.76) 100.0 0.003
a

Mediation analyses were adjusted for the same covariates indicated in Table 2, with or without adjustment for height z-score (HTZ) and BMI z-score (BMIZ) at age 11. Total effects differ slightly from those shown in Table 2 because models were restricted to a subset of 422-425 boys with available growth measures at age 11. Also note that direct and indirect effects do not necessarily sum to the total effect, given the use of the preferred product approach for calculating indirect effects.

4. Discussion

The effects of lead on pubertal onset and sexual maturity have primarily been studied in girls, with the exception of our own prior evaluations focusing on pubertal onset in this cohort and the Birth to Twenty cohort in South Africa.3841 Thus, evaluating associations of lead exposure with timing of pubertal milestones among boys is a critical research gap. Based on our longitudinal study of boys followed from pre-puberty to young adulthood, we demonstrated later attainment of both pubertal onset and sexual maturity for boys with higher peripubertal blood lead concentrations. The shift of 4-5 months in mean age at maturity for boys with higher compared to lower BLLs was less than that observed for pubertal onset (7-8 months), suggesting that effects of peripubertal BLLs are attenuated over time.

Despite the later attainment of sexual maturity for Russian boys in our cohort with higher blood lead levels, the overall mean ages at genital maturity were generally similar to those published for other US and European cohorts, although most of these studies were based on cross-sectional designs and only followed boys to age 16-17 years, which may have influenced model estimates.4249 The mean ages at genital maturity ranged between 14.3-14.9 years for longitudinal studies4244, compared to 14.7 years in our cohort, and from 14.5-15.9 years for cross-sectional studies.4549 The mean age at pubic hair maturity in our cohort (16.1 years) was also within 6 months of those from most other studies.4447,49

We also evaluated the duration of pubertal progression, accounting for the interval-censored nature of sexual maturity outcomes. While the pubertal progression duration varied according to the specific pubertal indicator (genitalia, pubic hair, or TV), we observed no consistent associations of progression duration with BLLs. However, while not significant, the shorter estimated time from G2 to G5 for boys with higher BLL provides support for the possibility of some attenuation of lead effects as boys reach maturity. This potential attenuation of lead effects is consistent with our previous evaluations of differences in linear growth in boys with higher BLL as compared to boys with lower BLL, which appeared to decrease as they approached age 18.22

The true duration of progression is the time from pubertal onset to sexual maturity, but because neither of these events is observed exactly, the outcome is technically “doubly-censored”. Because few standard statistical methods (and no standard software packages) have been developed to address these types of outcomes, we approximated the duration of progression by only considering the right-hand time point (maturity) to be interval-censored and essentially assumed that the left-hand time point (pubertal onset) was observed. Since the evaluation of this outcome is restricted to those with pubertal onset, approximation of the age at onset as the midpoint between the two visits at which it was observed seemed reasonable, although it does require an assumption that measurement error in this estimation does not depend on the BLL. There are few evaluations of pubertal progression from longitudinal cohorts which follow children from pre-puberty through to sexual maturity, so there are few other metrics for progression with which to compare our approach; statistical approaches for evaluating and comparing different metrics for duration of progression deserve further attention.

We had previously reported associations of BLLs with timing of pubertal onset,20 and have updated and expanded our analysis here to account for several additional years of follow-up, and now present these results as adjusted mean shifts in ages at onset. These estimates allow comparison of the magnitude of shifts for onset as compared to maturity. Few previous studies have addressed the association of BLLs with age at male sexual maturity or duration of pubertal progression. One recent study observed no association between blood lead levels at age 13 and pubertal development trajectories, but reported slower progression of pubertal hair in boys with higher cord blood levels.41 However, this study did not provide estimates of the average age at sexual maturity or duration of pubertal progression.

In addition to demonstrating an association of high BLLs with later maturity, we also used novel mediation methods to show that these shifts were substantially explained (mediated) by lower growth among boys with higher peripubertal BLLs. Similar to other studies evaluating male maturity, we found that diminished height at age 11 played a stronger role in mediating the effects of lead than did lower BMI at this age, but the combined effects of BLL on height and BMI explained up to 71% of the shifts in age at maturity related to higher BLLs.31 While these growth measures explained a high proportion of the overall effects of lead on delay in sexual maturity, it is notable that the effects of lead were not fully mediated by growth measures. This finding suggests that there are other direct effects of lead on timing of maturity, either via other pathways (eg. hormonal or metabolic) or via growth measures or trajectories at other ages. Mediation analysis approaches for survival outcomes have only been recently applied to interval-censored outcomes, but the framework of accelerated failure time models allows straight-forward identification of indirect and direct effects of the primary exposure of interest.31,32,35

Clinical effects of a shift of 4-5 months in the age at sexual maturity are anticipated to be subtle and may not reflect clinical delay for most youth, but may nevertheless translate on a population level to a higher than expected percent with clinically delayed maturity. Few boys did not attain maturity until after age 18 by genital staging or TV; however pubic hair maturity after age 18 was significantly more common among boys with higher BLL than lower BLL. Clinical implications of delayed maturity include decreased final adult height, which has been previously reported in this cohort, as well as increased risk of specific cancers. 22,50 In addition, public health implications of alterations in timing of sexual maturity include reduced self-esteem and decreased bone mineral density.51,52

Our findings are consistent with those of animal studies, which have demonstrated delayed sexual maturity in mice with elevated BLLs.811,53 In addition, lead exposure has been shown to affect reproductive hormone levels in girls, including decreases in levels of IGF-1, luteinizing hormone, and inhibin B, which may reflect hypothalamic–pituitary-gonadal (HPG) function.54 As a known endocrine disruptor, lead is believed to produce its adverse effects on reproductive function and pubertal development through the HPG axis, although multiple endocrine pathways may be involved.9,55

Strengths of our study include enrollment of all boys within a small age range, longitudinal follow-up over more than a decade with a high retention rate, annual evaluations of pubertal staging by the same clinical investigator, and collection of detailed data on growth and important covariates, including dietary intake and socioeconomic indicators. One limitation of our pubic hair assessments is the practice of pubic hair shaving among boys in this region, which may result in some misclassification of pubic hair staging. However, we would not expect boys with higher BLLs to be more likely to shave than those with lower BLLs; non-differential measurement error in our pubic hair assessments may have decreased the precision of our effect estimates but should not introduce bias.

An additional limitation was that we had only a single BLL measurement for each boy, and this single measurement may not reflect exposure in critical windows, such as prenatal or early childhood periods.56,57 BLLs are recognized to reflect recent exposure over the past weeks or months and may not accurately characterize longer-term cumulative exposures.57,58 At the same time, previous studies have reported relatively high reproducibility of BLLs in children, possibly reflecting consistent sources of exposure over time.59 In addition, the use of BLLs measured 2-3 years prior to the growth measurements included in our mediation analysis reduces the chance of reverse causation, which could result from lead concentrations being diluted by body size at the time of lead measurement.7 Still, given within-boy correlation in growth measurements over time, the possibility of such reverse causality cannot be completely ruled out. Finally, we recognize that the LOD of 1 μg/dL for our lead measurement was higher than that of more recent studies,7 but since <3% of BLLs fell below the LOD this would be expected to have little impact on our findings.

We recognize that the effect of BLLs on timing of sexual maturity via its effect on growth may be partially attributable to the previously demonstrated association of BLLs with later pubertal onset, which in turn affects subsequent growth. Our approach evaluated the overall association between BLLs and growth through multiple pathways, including alterations in the timing of pubertal onset. Future studies may be warranted to explore how other windows of potential susceptibility to lead exposure (in addition to the peripubertal exposure time period studied here) may impact trajectories of growth and pubertal development.

5. Conclusions

In conclusion, Russian boys with higher BLLs experienced delays in pubertal onset and sexual maturation that may have adverse long-term consequences. Despite decreasing trends in blood lead levels internationally, particularly in resource-rich settings, lead exposure continues to be a significant public health problem in low and middle income countries. The decrements in growth observed for boys with higher BLLs were found to be important contributors to the later attainment of sexual maturity, which suggests potential targets for interventions that could ideally be initiated prior to the adolescent growth spurt.

Supplementary Material

1

HIGHLIGHTS:

  • Higher blood lead has been linked to impaired growth and delayed pubertal onset.

  • Few studies have addressed the effects of blood lead on age at male sexual maturity.

  • We found significant delays in maturity among Russian boys with higher lead levels.

  • This delay was partly due to reduced growth among boys with higher lead levels.

  • Higher blood lead did not appear to slow pubertal progression.

Acknowledgments:

This study was supported by grants from U.S. EPA (grant R82943701) and the National Institute of Environmental Health Sciences (grants R01 ES014370 and P30 ES000002). Paige Williams had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. We thank the study participants, the Chapaevsk government (Vitaly Ashepkov, Dmitry Blynsky and Nikolay Malakhov), and Chapaevsk Medical Association, Chapaevsk Central Hospital and Chapaevsk Children’s Polyclinic staff as well as chiefs (Vladimir Zeilert, Nadezhda Saraeva, Svetlana Nikolaeva, Anatoly Kochkaryov, Andrey Pismenniy). We thank our colleague Boris Revich (Institute for Forecasting, Russian Academy of Sciences), and also Anna Safronova and Mihail Starovoytov (Russian Institute of Nutrition).

Funding sources: Funded by U.S. EPA (grant R82943701) and National Institute of Environmental Health Sciences, NIH (grants R01 ES014370 and P30 ES000002).

Abbreviations:

BLL

blood lead level

BMI

body mass index

G2

genitalia stage 2 (onset of puberty according to genitalia)

G5

genitalia stage 5 (sexual maturity according to genitalia)

HPG

hypothalamic-pituitary-gonadal

IGF-1

insulin-like growth factor 1

IQR

interquartile range

LOD

limit of detection

P2

pubic hair stage 2 (onset of puberty according to pubic hair)

P5

pubic hair stage 5 (sexual maturity according to pubic hair)

TV

testicular volume

WHO

World Health Organization

Footnotes

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

Declarations of Interest: The authors have no personal or financial conflicts of interest relevant to this article to disclose.

References

  • 1.Thayer KA, Heindel JJ, Bucher JR, Gallo MA. Role of environmental chemicals in diabetes and obesity: a National Toxicology Program workshop review. Environ Health Perspect. 2012; 120(6):779–89. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 2.Casals-Casas C, Desvergne B. Endocrine disruptors: from endocrine to metabolic disruption. Annu Rev Physiol. 2011; 73:135–62. [DOI] [PubMed] [Google Scholar]
  • 3.Schwartz J, Angle C, Pitcher H. Relationship between childhood blood lead levels and stature. Pediatrics 1986; 77(3):281–288. [PubMed] [Google Scholar]
  • 4.Shukla R, Dietrich KN, Bornschein RL, Berger O, Hammond PB. Lead exposure and growth in the early preschool child: A follow-up report from the Cincinnati lead study. Pediatrics 1991; 88(5):886–89. [PubMed] [Google Scholar]
  • 5.Kim R, Hu H, Rotnitzky A, Bellinger D, Needleman H. A longitudinal study of chronic lead exposure and physical growth in Boston children. Environ Health Perspect 1995; 103(10):952–7. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6.Ballew C, Khan LK, Kaufmann R, Mokdad A, Miller DT, Gunter EW. Blood lead concentration and children’s anthropometric dimensions in the Third National Health and Nutrition Examination Survey (NHANES III), 1988-1994. J Pediatr 1999; 134:623–630. [DOI] [PubMed] [Google Scholar]
  • 7.Deierlein AL, Teitelbaum SL, Windham GC et al. Lead exposure during childhood and subsequent anthropometry through adolescence in girls. Environ Int 2018; 10.1016/j.envint.2018.11.031. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8.Sokol RZ, Wang S, Wan YJ, Stanczyk FZ, Gentzschein E, Chapin RE. Long-term, low-dose lead exposure alters the gonadotropin-releasing hormone system in the male rat. Environ Health Perspect 2002; 110(9):871–4 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9.Ronis MJ, Badger TM, Shema SJ, Robertson PK, Shaikh F. Reproductive toxicity and growth effects in rats exposed to lead at different periods during development. Toxicol Appl Pharmacol 1996; 136:361–371. [DOI] [PubMed] [Google Scholar]
  • 10.Ronis MJ, Gandy J, Badger T. Endocrine mechanism underlying reproductive toxicity in the developing rat chronically exposed to dietary lead. J Toxicol Environ Health 1998; 54:77–99. [DOI] [PubMed] [Google Scholar]
  • 11.Ronis MJ, Badger TM, Shema SJ, et al. Endocrine mechanism underlying the growth effects of developmental lead exposure in the rat. J Toxicol Environ Health 1998; 54(2):101–120. [DOI] [PubMed] [Google Scholar]
  • 12.Centers for Disease Control (US). Preventing lead poisoning in young children. Atlanta (GA); Department of Health and Human Services (US); 1991. [Google Scholar]
  • 13.Jusko TA, Henderson CR Jr, Lanphear BP, Cory-Slechta DA, Parsons PJ, Canfield RL. Blood lead concentrations <10 μg/dL and child intelligence at 6 years of age. Environ Health Perspect 2008; 116(2): 243–248. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14.Centers for Disease Control and Prevention (CDC). Blood lead levels in children aged 1-5 years - United States, 1999-2010. Morb Mortal Wkly Rep. 2013. April 5;62(13):245–8. [PMC free article] [PubMed] [Google Scholar]
  • 15.Rubin CH, Esteban E, Reissman DB, et al. Lead poisoning among young children in Russia: Concurrent evaluation of childhood lead exposure in Ektaterinburg, Krasnouralsk, and Volgograd. Environ Health Perspect 2002; 110:559–562. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16.Snakin VV and Prisyazhnaya AA. Lead contamination of the environment in Russia. Sci Total Environ 2000; 256: 95–101. [DOI] [PubMed] [Google Scholar]
  • 17.Tong S, von Schirnding YE, Prapamontol T. Environmental lead exposures: a public health problem of global dimensions. Bull World Health Organization 2000; 78(9); 1068–1077. [PMC free article] [PubMed] [Google Scholar]
  • 18.Mathee A, von Schirnding Y, Montgomery M, Rollin H. Lead poisoning in South African children: the hazard is at home. Rev Environ Health 2004; 19(3-4):347–59. [PubMed] [Google Scholar]
  • 19.Jones RL, Homa DM, Meyer PA, et al. Trends in blood lead levels and blood lead testing among US children aged 1 to 5 years, 1988-2004. Pediatrics 2009; 123; e376–85. [DOI] [PubMed] [Google Scholar]
  • 20.Williams PL, Sergeyev O, Lee MM, et al. Blood lead levels and delayed onset of puberty in a longitudinal study of Russian boys. Pediatrics 2010; 125(5): e1088–96. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21.Fleisch AF, Burns JS, Williams PL, et al. Blood lead levels and insulin-like growth factor 1 concentrations in peripubertal boys. Environ Health Perspect.2013; 121(7):854–8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22.Burns JS, Williams PL, Lee MM, et al. Peripubertal blood lead levels and growth among Russian boys. Environ Int. 2017; 106: 53–59. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23.Sergeyev O, Burns JS, Williams PL, et al. The association of peripubertal serum concentrations of organochlorine chemicals and blood lead with growth and pubertal development in a longitudinal cohort of boys: a review of published results from the Russian Children’s Study. Rev Environ Health. 2017; 32(1-2): 83–92. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24.Martinchik AN, Baturin AK, Baeva VS, et al. Development of a method of studying actual nutrition according to analysis of the frequency of consumption of food products: creation of a questionnaire and general evaluation of the reliability of the method. (in Russian) Vopr Pitan 1998; 3:8–13. [PubMed] [Google Scholar]
  • 25.World Health Organization. The WHO Child Growth Standards. http://www.who.int/childgrowth/en/. Published 2011 Accessed 29 May 2018.
  • 26.Tanner JM and Whitehouse RH. Clinical longitudinal standards for height, weight, height velocity, weight velocity, and stages of puberty. Arch Dis Child 1976; 51:170–179. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 27.Marshall WA, Tanner JM. Variations in the pattern of pubertal changes in boys. Arch Dis Child 1970; 45:13–23 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 28.Karpati AM, Rubin, Kieszak S, Marcus M, Troiano RP. Stature and pubertal stage assessment in American Boys: The 1988-1994 Third National Health and Nutrition Examination Survey. J Adolesc Health 2002; 30:205–212. [DOI] [PubMed] [Google Scholar]
  • 29.Euling SY, Herman-Giddens ME, Lee PA, et al. Examination of US puberty-timing data from 1940 to 1994 for secular trends: panel findings. Pediatrics 2008; 121 (Suppl 3):S172–S191. [DOI] [PubMed] [Google Scholar]
  • 30.Williams PL, Abzug MJ, Jacobson DL, et al. Pubertal onset in children with perinatal HIV infection in the era of combination antiretroviral treatment. AIDS 2013; 27:1959–1970. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 31.Bellavia A, Williams PL, DiMeglio LA, et al. Delay in sexual maturation in perinatally HIV-infected youths is mediated by poor growth. AIDS 2017; 31:1333–1341. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 32.Fulcher IR, Tchetgen Tchetgen EJ, Williams PL. Mediation analysis for censored survival data under an accelerated failure time model. Epidemiology. 2017; 28(5):660–666. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 33.Gelfand LA, MacKinnon DP, DeRubeis RJ, Baraldi AN. Mediation analysis with survival outcomes: accelerated failure time vs. proportional hazards models. Front Psychol 2016; 7:423. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 34.MacKinnon DP. Introduction to statistical mediation analysis. New York, NY: Taylor & Francis Group/Lawrence Erlbaum Associates; 2008. [Google Scholar]
  • 35.VanderWeele TJ. Causal mediation analysis with survival data. Epidemiology 2011; 22:582–585. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 36.VanderWeele T, Vansteelandt S. Mediation analysis with multiple mediators. Epidemiol Method 2013; 2:95–135. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 37.Valeri L, Vanderweele TJ. Mediation analysis allowing for exposure-mediator interactions and causal interpretation: theoretical assumptions and implementation with SAS and SPSS macros. Psychol Methods.2013;18(2):137–50 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 38.Selevan SG, Rice DC, Hogan KA, Euling SY, Pfahles-Hutchens A, Bethel J. Blood lead concentration and delayed puberty in girls. N Engl J Med 2003; 348:1527–1536. [DOI] [PubMed] [Google Scholar]
  • 39.Wu T, Buck GM, Mendola P. Blood lead levels and sexual maturation in U.S. girls: The Third National Health and Nutrition Examination Survey, 1988-1994. Environ Health Perspect 2003; 111:737–741. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 40.Naicker N, Noris SA, Mathee A, Becker P, Richter L. Lead exposure is associated with a delay in the onset of puberty in South African adolescent females: Findings from the Birth to Twenty cohort. Sci Total Environ 2010; 408:4949–4954. [DOI] [PubMed] [Google Scholar]
  • 41.Nkomo P, Richter LM, Kagura J et al. Environmental lead exposure and pubertal trajectory classes in South African adolescent males and females. Sci Total Environ 2018; 628-629: 1437–1445. [DOI] [PubMed] [Google Scholar]
  • 42.Roche AF, Wellens R, Attie KM, Siervogel RM. The timing of sexual maturation in a group of US white youths. J Pediatr Endocrinol Metab 1995; 8:11–18. [DOI] [PubMed] [Google Scholar]
  • 43.Susman EJ, Houts RM, Steinberg L, et al. Longitudinal development of secondary sexual characteristics in girls and boys between ages 9 ½ and 15 ½ years. Arch Pediatr Adolesc Med 2010; 164:166–173. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 44.Lam T, Williams PL, Lee MM, et al. Prepubertal serum concentrations of organochlorine pesticides and age at sexual maturity in Russian boys. Environ Health Perspect. 2015; 123(11):1216–21. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 45.Willers B, Engelhardt L, Pelz L. Sexual maturation in East German boys. Acta Paediatr 1996; 85:758–788. [DOI] [PubMed] [Google Scholar]
  • 46.Herman-Giddens ME, Wang L, Koch G. Secondary sexual characteristics in boys: estimates from the national health and nutrition examination survey III, 1988-1994. Arch Pediatr Adolesc Med 2001; 155:1022–1028. [DOI] [PubMed] [Google Scholar]
  • 47.Juul A, Teilmann G, Scheike T, et al. Pubertal development in Danish children: comparison of recent European and US data. Int J Androl 2006; 29:247–255. [DOI] [PubMed] [Google Scholar]
  • 48.Papadimitriou A, Douros K, Kleanthous K, Papadimitriou DT, Attilakos A, Fretzayas A. Pubertal maturation of contemporary Greek boys: no evidence of a secular trend. J Adol Health 2011; 49:434–436. [DOI] [PubMed] [Google Scholar]
  • 49.Herman-Giddens ME, Steffes J, Harris D, et al. Secondary sexual characteristics in boys: data from the Pediatric Research in Office Settings Network. Pediatrics 2012;130:e1058–1068. [DOI] [PubMed] [Google Scholar]
  • 50.Lope V, Garcia-Esquina E, Ruiz-Dominguez JM, et al. Perinatal and childhood factors and risk of prostate cancer in adulthood: MCC-Spain case-control study. Cancer Epidemiol 2016; 43:49–55. [DOI] [PubMed] [Google Scholar]
  • 51.Golub MS, Collman GW, Foster PM, et al. Public health implications of altered puberty timing. Pediatrics 2008; 121 (Suppl 3):S218–S230. [DOI] [PubMed] [Google Scholar]
  • 52.Michaud PA, Suris JC, Deppen A. Gender-related psychological and behavioral correlates of pubertal timing in a national sample of Swiss adolescents. Mol Cell Endocrinol 2006; 254-255:172–178. [DOI] [PubMed] [Google Scholar]
  • 53.Iavicoli I, Carelli G, Stanek EJ III, Castellini N, Calabrese EJ. Effects of low doses of dietary lead on puberty onset in female mice. Reprod Toxicol 2004; 19:35–41. [DOI] [PubMed] [Google Scholar]
  • 54.Gollenberg AL, Hediger ML, Lee PA, Himes JH, Buck Louis GM. Association between lead and cadmium and reproductive hormones in peripubertal U.S. girls. Environ Health Perspect 2010; 118(12):1782–1787. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 55.Buck Louis GM, Gray LE Jr, Marcus M, et al. Environmental factors and puberty timing: expert panel research needs. Pediatrics 2008; 121(Suppl 3): S182–S207. [DOI] [PubMed] [Google Scholar]
  • 56.Selevan SG, Kimmel CA, Mendola P. Identifying critical windows of exposure for children’s health. Environ Health Perspect 2000; 108(Suppl 3): 451–455. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 57.Braun JM, Hoffman E, Schwartz J, et al. Assessing windows of susceptibility to lead-induced cognitive deficits in Mexican children. Neurotoxicology 2012; 33(5): 1040–1047. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 58.Hu H, Shih R, Rothenberg S, Schwartz BS. The epidemiology of lead toxicity in adults: measuring dose and consideration of other methodologic issues. Environ Health Perspect 2007; 115: 455–462. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 59.Sexton K, Adgate JL, Fredrickson AL, Ryan AD, Needham LL, Ashley DL. Using biological markers in blood to assess exposure to multiple environmental chemicals for inner-city children 3-6 years of age. Environ Health Perspect 2006; 114: 453–459. [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

1

RESOURCES