Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2023 Mar 1.
Published in final edited form as: Neurotoxicol Teratol. 2022 Jan 31;90:107075. doi: 10.1016/j.ntt.2022.107075

Sexually dimorphic associations between prenatal blood lead exposure and performance on a behavioral testing battery in children

Francheska M Merced-Nieves a, John Chelonis b, Ivan Pantic c, Lourdes Schnass c, Martha M Téllez-Rojo d, Joseph M Braun e, Merle G Paule b, Rosalind J Wright a,f, Robert O Wright a,f, Paul Curtin a
PMCID: PMC8957713  NIHMSID: NIHMS1779377  PMID: 35108597

Abstract

Background

Associations between lead (Pb) and neurodevelopment have been studied widely in the context of global measures of cognitive function, such as IQ. Operant test batteries consist of behavioral tasks that can be used to target discrete cognitive and behavioral mechanisms, which contribute to global cognitive faculties.

Objectives

The goals of this study were to identify Pb-associated deficits in cognitive development and determine the underlying mechanisms involved, utilizing an operant test battery. We evaluated effect modification by child sex.

Methods

This study utilized data from a prospective cohort in Mexico City. We included 549 participants aged 6-to-7 years with complete data on prenatal blood Pb measurements, Operant Test Battery (OTB) tasks, and demographic covariates. General linear models were used to examine the association of Pb levels at each prenatal timepoint and OTB performance. Effect modification by child sex was evaluated using 2-way interaction terms.

Results

In three of the operant tasks, we observed that higher late-pregnancy blood Pb concentrations were associated with greater response latencies. In the temporal processing task, we observed that higher late-pregnancy Pb exposure was associated with worse overall task performance. Further, in two operant tasks, the effects of Pb were dependent on the sex of the child, such that the effects of Pb were more pronounced in females in the condition position responding task, but stronger in males in the temporal processing task.

Conclusions

Our results suggest that prenatal Pb concentrations yield broad dysregulation of executive functions, which can be attributed to dysregulation of temporal processing. In addition, we observed sex differences in two operant tasks suggesting that some Pb effects on neurocognitive function may be sexually dimorphic.

1. Introduction

Lead is a ubiquitous heavy metal with a wide range of past and persistent uses in multiple contexts, from plumbing and water infrastructure to lead-based paint and pigments. Although now banned, lead was once widely used as a fungicide in household paint and as a gasoline additive, which led to widespread contamination. There is an extensive body of literature showing the adverse effects of prenatal and postnatal blood lead (Pb) exposure on neurodevelopment[1–4]; and although there has been a global decrease in exposure, recent evidence suggests that even “low-level” exposure has detrimental effects and should be of concern[5, 6].

Numerous studies have shown the detrimental effects of Pb exposure on global measures of cognition and standardized tasks like the Bayley Scale of Infant Development (BSID)[7–9], the McCarthy Scales of Children’s Ability (MSCA)[10–12], and the Weschler Intelligence Scale for Children – Revised (WISC-R)[13, 14]. Other studies have focused on the effects of Pb on specific neurocognitive and behavioral domains, showing that Pb exposure has detrimental effects on multiple domains such as executive function, memory, attention, inhibition, internalizing behaviors, learning, and motor function [15–26]. While many human studies have reported deficits in specific neurocognitive functions as a result of lead exposure, to our knowledge, none of these studies have utilized operant tasks that could be utilized to provide common measures of shared behavioral functionality across species. Operant tasks, provide reliable and specific measures of motivation, working memory, attention, and time perception among other functions, which contribute to global cognitive faculties, and are able to suggest corresponding anatomical domains [27–31].However, studies which examine the effects of neurotoxic exposures in these testing batteries are rare in the human literature. Animal literature utilizing operant tasks have shown that low-level Pb exposure has a detrimental effect on operant measures that test learning and memory[32]. Further, the animal literature has been able to elucidate how Pb exposure targets specific mechanisms and brain areas that are involved in these tasks. For example, Cory-Slechta and colleagues[33] demonstrated that dopamine activity in the nucleus accumbens is altered by Pb exposure during weaning. Their results showed that higher exposure was associated with higher overall response rates and decreases in stimulus control.

Surprisingly, only a small number of studies have looked at the role of sex as an effect modifier of Pb exposure. In nonhuman studies, a number of studies have highlighted the importance of sex as a potential modifier of cognitive and behavioral processes, including sensorimotor processing, exploratory behavior, or task-specific cognitive tasks such as maze performance[34–37]. While human studies have considered the role of sex[38], they have typically only adjusted for sex without addressing its role as an effect modifier. Although limited, there is evidence to suggest that there are sex-specific associations between Pb exposure and neurodevelopment[38, 39]. Specifically, prenatal Pb exposure has been associated with a decrease in intelligence quotient (IQ) in males, but not females[13, 40, 41]. Ris and colleagues[42] also found inverse associations between prenatal Pb exposure and measures of attention, with males performing worse than females. Understanding these sex differences could aid in the understanding of susceptibility to Pb exposure and targeting populations for interventions.

In the present study, an operant test battery was used to evaluate general and sex-specific effects of Pb exposure in 6- to 7-year-old children from a longitudinal, prospective birth cohort based in Mexico City where exposure to Pb remains common. This exposure remains common, mainly due to the continued use of Pb glazed traditional pottery which is used for cooking and food storage across the country, cosmetics, and treatments for intestinal cramps (e.g. greta, azarcon, and rueda); creating occupational and non-occupational exposure routes[43, 44]. The goals of this study were to identify Pb-associated deficits in cognitive development as evaluated by operant tasks that can be used across species, determine the underlying mechanisms involved in this dysregulation, and evaluate the dependency of these effects on the sex of the child.

2. Methods

2.1. Study participants

The children in this study are participants of a prospective birth cohort study called: Programming Research in Obesity, GRowth, Environment and Social Stressors (PROGRESS) based in Mexico City. Women were considered eligible for enrollment in the study if they were 18 years or older, less than 20 weeks gestation, free of heart or kidney disease, did not use steroids or anti-epilepsy drugs, did not consume alcohol on a daily basis or any psychoactive drugs, had access to a telephone, and planned to reside in Mexico City for the following 3 years[45]. Briefly, we enrolled 948 pregnant women between 2007 and 2011 who delivered a live birth and followed their offspring longitudinally thereafter. Of the original 948 children, 650 are actively followed in 2-year intervals with blood Pb measured at each visit along with neurodevelopmental tests. The study protocols were approved by the institutional review boards (IRBs) and informed consent was obtained during the first visit. The following analyses included 549 participants with complete data on blood Pb measurements, Operant Test Battery results at 6-to-7 years of age, and covariates.

2.2. Blood Pb measurements

Maternal blood was collected during the second trimester (2T), third trimester (3T), and on the day of delivery. Blood was also collected from the infant’s umbilical cord at delivery. Samples were collected in royal blue trace metal Vacutainer (Becton-Dickinson and Company, Franklin Lakes, NJ, USA) tubes containing EDTA. Digested samples were analyzed using external calibration with seven calibration points using an Agilent 8800 ICP Triple Quad (ICP-QQQ) (Agilent technologies, Inc., Wilmington, DE, USA) in MS/MS mode with Lutetium as the internal standard at the Lautenberg Environmental Health Science Laboratory at Mount Sinai, NY which participates biannually in the New York State, Department of Health as well as the Center de Toxicologie-INSPQ, Quebec, Canada external performance testing programs.

2.3. Operant Test Battery

2.3.1. Apparatus and procedure

Children were assessed using a version of the National Center for Toxicological Research (NCTR) Operant Test Battery (OTB) when they were between 6 and 7 years of age (from 2014 to 2018). A detailed description of the apparatus has been previously published[28, 46]. Briefly, the participant sat across from the testing panel that was mounted in a large wooden cabinet (Figure 1). The panel consisted of two types of response manipulanda – circular press-plates and levers and two different types of stimulus lights – correct and incorrect response indicator lights and serial position indicator lights. At the bottom of the apparatus there was a container where nickels, after a correct response was made, were delivered by a dispenser mounted inside the wooden cabinet. Children exchanged the nickels for a toy of their choosing at the end of the visit.

Figure 1.

Figure 1.

Open in a new tab

Diagram of participant positioning and the operant test panel. Illustration by Jill Gregory, used with permission of ©Mount Sinai Health System. Starting at the top of the panel, there is a speaker and below that there are three circular press-plates. Below the circular press-plates, there are two different types of stimulus lights – correct and incorrect response indicator lights (smiley face) and serial position indicator lights (colored rectangles). At the bottom of the apparatus there was a container where nickels were delivered by a dispenser mounted inside the wooden cabinet.

All tasks were administered by a trained psychologist. Video instructions in Spanish were given before each task. Once the child acknowledged they understood the instructions, the psychologist left the examination room and continuously monitored the child’s behavior through a one-way mirror. The current manuscript examined data from four behavioral tasks: Conditioned Position Responding (CPR), Temporal-Response Differentiation (TRD), Delayed Matching-to-Sample (DMTS), and Incremental Repeated Acquisition (IRA).

2.3.2. Conditioned Position Responding (CPR)

In the CPR task[28, 47], only the three circular press-plates were used. At the start of each trial, the center press-plate was illuminated with one of four colors – red, yellow, blue, or green. If the center plate color was red or yellow, then a press on the left-side plate was correct. If the center plate color was blue or green, a press on the right-side plate was correct. The variables for this task included: percent task completed (number of nickels earned/total possible); accuracy (number of correct responses/number of trials); observing response latency (average time the child took to make a response on the center press-plate once it was illuminated); choice response latency (average time the child took to make a response on either of the side press-plates after they were illuminated); and ineffective response rate (number of responses per minute the child made on any press-plate that was not illuminated). The main neurobehavioral function this task measures is color and position discrimination [28]. We utilized response latencies to provide a measure of general response time, and accuracy to specifically measure task performance.

2.3.3. Temporal Response Differentiation (TRD)

In the TRD task[48], the child had to hold the far-left lever down for 10 to 14 seconds. Lever hold durations were divided into timing holds (holds greater than or equal to 2 s in duration) and bursts (holds less than 2 s in duration)[49]. The variables for this task included: percent task completed (number of nickels earned/total possible); number of timing holds; average duration of timing holds (average time between the press and release of lever); standard deviation of the duration of timing holds (variability of the length of time between the press and release of the lever); average latency (average time between the release of the lever and the next lever press); and number of burst responses. The main neurobehavioral function this task measures temporal perception [28]. We measured average latency to provide a measure of general response time, and average duration of timing holds to specifically measure task performance.

2.3.4. Delayed Matching-to-Sample (DMTS)

In this task, only the press-plates were used. At the start of a trial, the center press-plate was illuminated with one of seven possible geometrical shapes – square, triangle, plus sign, vertical bar, horizontal bar, circle, or an X[46]. After there was a 1 to 32 seconds delay; followed by all three plates being illuminated with different shapes, one of which matched the original shape. The child was required to press the plate that “matched” the geometrical shape they saw previously. The variables for this task included: percent task completed (number of nickels earned/total possible); observing response latency (average time the child took to make a response on the center press-plate once it was illuminated); standard deviation of observing response latency (variability of the length of time the child took to make a response on the center press-plate once it was illuminated); overall choice latency (time it took the child to make a response when all press-plates were illuminated with a shape); standard deviation of overall choice latency (variability of the length of time it took the child to make a choice); and overall accuracy (number of correct responses/number of trials, regardless of time delay). The main neurobehavioral function this task measures is short-term memory and attention [28]. We measured response latencies to provide a measure of general response time, and overall accuracy to specifically measure task performance.

2.3.5. Incremental Repeated Acquisition (IRA)

In the IRA task, the levers, correct and incorrect response indicator, and the serial position indicator lights were used[50]. Briefly, the child was required to learn to press the levers in a specific sequence dictated by the illumination of the serial position indicator lights. This sequence consisted of a maximum of six lever presses, in other words, a six-link response chain. The serial position indicator lights signified which link of the chain the child was currently on. At the start of the task, the far-right serial position indicator would illuminate (red). The child’s initial task was to determine which of the four levers to press when the red serial position indicator was illuminated. An incorrect response was followed by the illumination of the incorrect response indicator for 2 seconds. A correct response was followed by the illumination of the correct response indicator for 1 second and the delivery of a nickel. At the start of the two-link chain, the second to right serial position indicator would illuminate (green) and the child had to determine which lever corresponded to this light cue. After the child pressed the correct lever, the green serial position indicator light was extinguished and the red serial position indicator illuminated. The child then had to press the lever that previously corresponded to this red light to receive a nickel and start the next trial for the two-lever sequence. The process continued for each response chain – three-link (orange), four-link (blue), five-link (yellow), and six-link (pink). The measures analyzed included: percent task completed (number of correct sequences/total possible sequences); accuracy of chains completed (number of errorless response chains completed/total number of chains completed); effective response rate (responses that resulted in the illumination of the correct/incorrect light indicators in responses per second); and ineffective response rate (responses during delays or illumination of correct/incorrect response indicator lights in responses per second). Two additional accuracies were also measured: memory accuracy and search accuracy. Memory accuracy was calculated using responses during the illumination of a stimulus light after the subject had learned which lever was correct for that stimulus light. Search accuracy used responses during the illumination of a stimulus light before the subject had learned which lever was correct for that stimulus light. The main neurobehavioral function this task measures is learning behavior [28]. We measured response rates to provide a measure of general response time, and accuracy to specifically measure task performance.

2.4. Statistical Analyses

All statistical analyses were performed using R version 3.6.2. Descriptive statistics were calculated for covariate and outcome measures and are reported in Table 1. For each outcome, all observations that were ± 3 standard deviations from the mean were considered outliers and eliminated from subsequent analyses. The number of observations excluded from each of the 23 outcomes studied varied from 2 to 16 observations, reflecting 0.4% to 3% of the total sample analyzed. General linear models were used to examine the association of blood Pb levels at each timepoint and each OTB variable. For each exposure-outcome model, effect modification by child sex was evaluated using 2-way interaction terms. Based on biological plausibility and a priori knowledge, models were adjusted for the child’s age at testing, maternal education (< high school, high school, > high school), and socioeconomic status (low, medium, high). Sensitivity analyses with postnatal blood lead exposure were performed to assess the robustness of any observed association. Models were adjusted for child’s postnatal blood lead levels (age range at the time of sample collection: 4.01, 6.46 years).

Table 1.

Demographics of mother-child pairs i ncluded in OTB an alyses

CPR task (n=549) TRD task (n=547) DMTS task (n=548) IRA task (n=544)
N (%) Mean (SD) N (%) Mean (SD) N (%) Mean (SD) N (%) Mean (SD)
Child age (years) 6.7 (0.5) 6.7 (0.5) 6.7 (0.5) 6.7 (0.5)
Child sex
 Female 271 (49.4) 270 (49.4) 270 (49.3) 268 (49.3)
 Male 278 (50.6) 277 (50.6) 278 (50.7) 276 (50.7)
Maternal education
 < high school 225 (41) 223 (40.8) 224 (41) 233 (41)
 high school 197 (35.9) 197 (36) 197 (36) 195 (35.8)
 > high school 127 (23.1) 127 (23.2) 127 (23) 126 (23.2)
Social Economic Status (SES)
 Low 296 (54) 295 (53.9) 295 (53.8) 294 (54)
 Medium 199 (36) 198 (36.2) 199 (36.3) 196 (36)
 High 54 (10) 54 (9.9) 54 (9.9) 54 (10)
2T blood Pb levels (μg/dL) 1 3.7(2.7) 3.7(2.7) 3.7(2.7) 3.7(2.7)
3T blood Pb levels (μg/dL) 2 3.9 (2.8) 3.9 (2.8) 3.9 (2.8) 3.9 (2.8)
Maternal blood Pb levels day of delivery (μg/dL) 3 4.3 (3.2) 4.3 (3.1) 4.3 (3.1) 4.3 (3.1)
Cord blood Pb levels (μg/dL) 4 3.4 (2.6) 3.4 (2.5) 3.4 (2.5) 3.4 (2.5)
Postnatal Pb levels (μg/dL) 5 2.4 (2.6) 2.4 (2.6) 2.4 (2.6) 2.4 (2.6)
Open in a new tab
1

Due to missing values: CPR (n=519); TRD, DMTS, IRA (n=544)

2

Due to missing values: CPR(n=463); TRD, DMTS, IRA (n=483)

3

Due to missing values: CPR (n=441); TRD, DMTS, IRA (n=457)

4

Due to missing values: CPR (n=301); TRD, DMTS, IRA (n=312)

5

Due to missing values: n=235

Abbreviations: OTB, Operant Test Battery; CPR, Conditioned Position Responding; TRD, Temporal Response Differentiation; DMTS, Delayed Matching-to-Sample; IRA, Incremental Repeated Acquisition; 2T, 2nd trimester; 3T, 3rd trimester

3. Results

3.1. Descriptive Statistics

Table 1 presents demographic data for the study population included in the analyses, divided by task. The average age of the participants was 6.7 years (SD: 0.5 years). The proportion of males (50.6%) and females (49.4%) was similar for each task. At the time of enrollment, about 59% of the mothers had at least a high school degree. Based on our indicator of Socioeconomic Status (SES), about 54% of the mothers were considered low SES. Prenatal Pb levels had a mean and SD of 3.7±2.7 μg/dL (2T; gestational age range: 16 to 22 weeks), 3.9±2.8 μg/dL (3T; gestational age range: 29 to 35 weeks), 4.3±3.1 μg/dL (delivery), 3.4±2.5 μg/dL (cord blood), and 2.4±2.6 μg/dL (postnatal; age range: 4.01 to 6.46 years). Table 2 presents the average performance on each of the OTB tasks.

Table 2.

Descriptive statistics for each Operant Test Battery t asks (Mean (SD))

Overall Males Females
Condition Position Responding (CPR)
Percent of task completed (%) 96.9 (9.6) 96.6 (10.2) 97.2 (8.9)
Accuracy (correct/total trials) 84.7 (17.2) 84 (17.8) 85.4 (16.6)
Observing response latency (seconds) 2.9 (1.5) 2.8 (1.6) 2.9 (1.4)
Choice response latency (seconds) 1.3 (2.2) 1.2 (0.5) 1.4 (0.9)
Ineffective total rate 1.8 (2.8) 2 (3.1) 1.6 (2.4)
Temporal Response Differentiation (TRD)
Percent of task completed (%) 21.6 (20.9) 23.9 (22.2) 19.3 (19.3)
Timing holds 33.3 (16.5) 33.4 (16.3) 31.8 (15.5)
Timing holds (average time) 15.2 (11.5) 13.8 (8.2) 14.4 (8.4)
Timing holds (SD) 9.2 (7.6) 8.9 (7.5) 9.5 (7.7)
Average latency (average) 3 (2.5) 2.9 (2.6) 3.1 (2.3)
Average latency (SD) 5.7 (6) 5.8 (6.4) 5.6 (5.5)
Number of bursts 31.2 (66.5) 30.3 (60.5) 32.3 (72.2)
Delayed Matching-to-Sample (DMTS)
Percent of task completed (%) 66.9 (16.9) 65.3 (17.1) 68.6 (16.5)
Overall choice latency (average) 3.5 (1.4) 3.6 (1.3) 3.5 (1.5)
Overall choice latency (SD) 3.1 (2.1) 3 (1.9) 3.1 (2.3)
Observing response latency (average) 3.4 (2.1) 3.6 (2.2) 3.2 (1.9)
Observing response latency (SD) 3.1 (3.6) 3.2 (3.6) 2.9 (3.5)
Overall Accuracy 86.3 (8.5) 86 (8.2) 86.7 (8.8)
Incremental Repeated Acquisition (IRA)
Percent of task completed (%) 82.3 (25.2) 83.9 (24.8) 80.5 (25.6)
Chains completed 27.7 (7.9) 27.8 (7.3) 27.5 (7.0)
Complete chains accuracy 39.5 (22.5) 41.1 (22.2) 37.9 (22.8)
Memory accuracy 64.9 (20.9) 66.6 (20.2) 63.1 (21.5)
Search accuracy 36.9 (10.1) 37.6 (10.2) 36.2 (10)
Effective response rate 0.59 (0.2) 0.62 (0.2) 0.57 (0.2)
Ineffective response rate 0.27 (0.3) 0.27 (0.3) 0.27 (0.3)
Open in a new tab

3.2. Conditioned Position Responding

In models for choice response latency, there was a significant interaction between blood Pb measured at the day of delivery and child sex with females taking 0.04 s longer than males with each 1-unit increase of Pb exposure at delivery, and 0.07 s longer with each 1-unit increase of cord blood Pb exposure (Table S1; Figure 2). As shown on Table S2, no other statistically significant results were observed for exposures measured at other time points (2T and 3T) and choice response latency. We likewise found no significant differences between exposure measured, sex, or sex-based interactions with exposure in the other parameters tested.

Figure 2.

Figure 2.

Open in a new tab

Covariate-adjusted models associating blood Pb levels and choice response latency in the Condition Position Responding task in both maternal blood at delivery and cord blood.

3.3. Temporal Response Differentiation

In models for the number of timing holds the children made, there was a significant interaction between cord blood Pb levels and child sex with males having a 1-hold decrease with each 1-unit increase of Pb exposure (Table S2; Figure 3–1a). In average duration of timing holds models, there were significant interactions between 3T, maternal blood day of delivery, and cord blood Pb levels and child sex with males having 0.72 s and 0.47 s longer timing holds with each 1-unit increase of Pb exposure during 3T and maternal levels the day of delivery, respectively (Table S2; Figure 3–2a and 2b). Interestingly, in the interaction for cord blood males had a non-significant increase of 0.38 s with each 1-unit increase of exposure, whereas, females had a significant decrease of 0.52 s with each 1-unit increase of exposure (Figure 3–2c). In models for the SD of timing holds, there were significant interactions between 3T, maternal blood day of delivery, and cord blood Pb levels and child sex with males having greater variability in their timing holds with higher exposure than females (Table S2; Figure 3–3a to c). In models for average latency, there was a significant interaction between 3T blood Pb levels and child sex with males having 0.14 s longer average latency with each 1-unit increase of Pb exposure (Table S2; Figure 3–4a). In models for bursts, there was a significant interaction between cord blood Pb levels and child sex (Table S2) with males having 2.61 fewer bursts with each 1-unit increase of Pb exposure. As shown in Table S2, no significant effects were observed.

Figure 3.

Figure 3.

Open in a new tab

(1a) Covariate-adjusted association between cord blood Pb levels and timing holds (interaction p-value = 0.03). (2a) Covariate-adjusted associations between 3T lead levels and average timing holds (interaction p-value = 0.02). (2b) Covariate-adjusted associations between maternal delivery lead levels and average timing holds (interaction p-value = 0.02). (2c) Covariate-adjusted associations between cord blood lead levels and average timing holds (interaction p-value = 0.02). (3a) Covariate-adjusted associations between 3T lead levels and the SD of timing holds (interaction p-value = 0.006). (3b) Covariate-adjusted associations between maternal delivery lead levels and the SD of timing holds (interaction p-value = 0.001). (3c) Covariate-adjusted associations between cord blood lead levels and the SD of timing holds(interaction p-value = 0.002). (4a) Covariate-adjusted association between 3T Pb levels and average latency (interaction p-value = 0.01). (5a) Covariate-adjusted association between cord blood Pb levels and number of burst(interaction p-value = 0.04).

Figure 4.

Figure 4.

Open in a new tab

(a) Covariate-adjusted association between cord blood lead levels and percent of task completed in the Delayed Matching-to-Sample. (b) Covariate-adjusted association between cord blood lead levels and observing response latency.

3.4. Delayed Matching-to-Sample

We found a negative association between increasing levels of cord blood Pb and percent task completed, such that children with higher cord blood exposure completed less of the DMTS task (Table S3; Figure 4a). There was a positive association between cord blood Pb levels and observing response latency. Thus, children with higher exposure to Pb at the end of pregnancy had longer observing response latencies (Table S3; Figure 4b). As shown in Table S3, no significant effects were observed for exposures measured at other time points; we likewise found no significant differences between sexes or sex-based interactions with exposure.

3.5. Incremental Repeated Acquisition

We found a negative association between increasing levels of blood Pb levels during 3T and effective response rate (Table S4). Hence, children with higher exposure had a decrease in effective response rate (Figure 5). As shown on Table S5, no other statistically significant results were observed.

Figure 5.

Figure 5.

Open in a new tab

Covariate-adjusted association between third trimester blood lead levels and effective response rate in the Incremental Repeated Acquisition task.

3.6. Sensitivity analyses

In TRD models, findings were essentially unchanged when we adjusted for postnatal exposure in our models (Table S5). Results were no longer statistically significant in the models for choice response latency (CPR), percent of task completed and observing response latency (DMTS), and effective response rate (IRA). There was no significant effect of postnatal blood lead levels in any of these outcomes.

4. Discussion

4.1. Summary of Results

We evaluated the relationship of prenatal lead exposure with performance on an Operant Test Battery among 6- to 7-year-old children in Mexico City. Our findings indicate two general patterns of effects associated with Pb exposure. First, in three of the four operant tasks measured, we observed that higher late-pregnancy blood Pb concentrations were associated with greater response latencies (choice response in the Conditioned Position Responding (CPR) task, observing response when the sample stimulus was displayed in the Delayed Matching-to-Sample (DMTS) task, and average latency to initiate a response in the Temporal Response Differentiation (TRD) task). Second, in the TRD task, we observed that higher late-pregnancy Pb exposure was associated with worse task performance, as evidenced by deficits in the number, duration, and variability of timing holds. Further, in both the CPR and the TRD tasks, the association of blood Pb concentrations were modified by the sex of the child, such that the associations of Pb in the CPR task were most evident in females, while Pb associations in the TRD task were most evident in males. Taken in sum, these results suggest that prenatal blood Pb concentrations yield broad dysregulation of executive functions which can be attributed in part to a discrete dysregulation of temporal processing.

Prior studies have shown that Pb exposure has a detrimental effect on cognitive faculties such as IQ, working memory, executive function, and motor function in young children[1, 2, 6, 9, 51]; these more global faculties each depend on the integration of multiple contributing mechanisms, which were more specifically assessed in the present study through varying operant tasks. The consistency of Pb-related delays in response timing across three OTB tasks suggests a dysregulation of executive function domains involved in response timing, action selection, and/or motor coordination. This discrete effect suggests that Pb-associated deficits in temporal processing may underlie dysregulation of global cognitive function.

Previous research[48, 49] has interpreted TRD task performance in the context of scalar timing theory. The scalar timing theory identifies two processes by which an organism tracks time: an internal clock speed and reference memory, with the former determining the length of timing productions (shorter versus longer holds) while the latter interfaces with working memory to preserve the target hold duration. The Pb-related deficits in TRD performance observed in this study were sex-dependent, such that males had longer hold durations with higher exposure, while females had shorter hold durations. These findings suggest that Pb differentially affects internal clock speed. In addition, we observed fewer bursts (holds less than 2 seconds) in males suggesting they were not disengaging from lever holds and further suggesting that there may be a motoric aspect to the effects of Pb. In addition, the sex-specificity of these deficits likely relates to working memory processes as well, given that males exhibited greater variability in their timing holds than females, consistent with dysregulated reference memory. Together, these findings implicate multiple aspects of temporal perception as a key locus of Pb neurotoxicity and suggest that sex is a critical biological moderator of these effects.

4.2. Brain Structures and Potential Underlying Mechanisms

As mentioned previously, there is literature suggesting specific brain areas involved in the performance of these operant tasks [47]. The prefrontal cortex seems to play an important role in performance on the CPR task [47, 52]. The hippocampus has been shown to play an important role in short-term memory and learning tasks (i.e. DMTS, CPR, and IRA tasks) [53, 54]. Another important brain structure is the nucleus accumbens, which seems to be involved with motivational/reward systems and these systems are needed for the completion of the battery [55].

There are five brain structures suspected of playing a role in temporal processing and perception – the frontal cortex, basal ganglia, parietal cortex, hippocampus, and cerebellum [56]. Each brain area and its integration facilitates: memory activation to predict and monitor the accuracy of the estimation (frontal cortex); the planning of movements based on stimuli (parietal cortex); the organization and recruitment of episodic memory (hippocampus); and the execution of motor control (basal ganglia and cerebellum)[56]. While Pb exposure does not have specific “signature effects” it has been demonstrated to affect all of these regions, particularly the frontal cortex and hippocampus[57, 58].

The subcortical structures of the basal ganglia, composed of the dorsal striatum and ventral striatum, are critical to timing of interval-length processes involved in motor coordination, and may represent a critical locale of Pb-associated neurotoxicity, particularly given prior findings which indicate susceptibility of this area to Pb exposures. Dopamine is essential for the maintenance and regulation of this system[59], and dopaminergic and glutamatergic dysregulation in this system have been identified as a common underlying pathologies in neurodegenerative diseases, particularly Parkinson’s Disease and Huntington’s Disease, that suffer both temporal and motor deficits[60–63]. Further, animal studies show that exposure to Pb was associated with an increase in dopamine activity in the nucleus accumbens[33, 64], which is the main component of the ventral striatum. Antagonism of the N-methyl-D-aspartate receptor (NMDAr) is another potential underlying mechanism by which Pb might disrupt temporal processing and perception[64, 65]. The NMDAr is a glutamate receptor found in the central nervous system and has a role in brain development; particularly in the domains of learning and memory[64, 66].

4.3. Limitations and Strengths

A major strength of this study is the use of an Operant Test Battery, which allowed for the objective assessment of both global cognitive deficits in the aggregation of task performance and for the assessment of specific aspects of neurodevelopment like time estimation and executive function. The observation of deficits in response latencies across multiple tasks provides consistent evidence of global dysregulation of executive function, which would be difficult to otherwise interpret from a single task. Likewise, the specificity of performance deficits only in the TRD task allows for the attribution of deficits in global cognitive function to a specific, discrete mechanism relating to temporal processing. These findings are consistent with previous research by Paule[27] showing that these behavioral tests show differential sensitivity to a variety of psychoactive chemicals. Hence, exposures of sufficient magnitude may result in poor global performance (i.e., a decrease in response latencies across multiple behavioral tasks), which can be subsequently dissected and attributed to discrete cognitive processes through task-specific performance. In addition, variations of these tasks have been used in animal models which allow for more direct translation between animal and human studies. Another strength is the focus on prenatal exposure to Pb, as there is limited literature on this critical window of exposure[51]. Evaluating the role of sex as a potential effect modifier was another strength, as there is limited literature on its role[38]. There are also some limitations that should be taken into account when considering the findings and implications of this study. Given the homogenous sample, based on race and SES, generalization may be limited to populations experiencing similar developmental and environmental contexts to this cohort.

Another potential limitation implicit to the analysis of early-life exposures is the potential impact of uncontrolled variables, including unmeasured exposures that may occur outside the developmental window considered here. There are factors implicit in our study design were employed to minimize these potential confounders. First, although we could not measure and control for exposures throughout the full course of development, we did measure postnatal blood level levels, and controlled for these in models evaluating the effects of prenatal exposures. Second, the availability of postnatal measures allowed us to confirm that postnatal blood lead levels were not significantly correlated with blood levels in the prenatal period; whether the effects observed are attributable to prenatal or postnatal lead exposure requires further study.

4.4. Conclusions

Our results provide additional evidence that Pb exposure impacts neurodevelopment and, importantly, expands the literature to include operant behavioral metrics which have been scant in humans. Here we demonstrate the use of operant testing procedures to attribute Pb-associated deficits to discrete cognitive mechanisms. Because operant testing is common in experimental animal research, this approach provides a translational bridge between human and animal toxicological studies and provides specific targets for future research along these lines. In particular, the current results provide a translational validation of prior toxicological studies in animal models which identified Pb-associated dysregulation of striatal function, which plays a critical role in the coordination of interval timing. The adoption of these methods in future exposure-related studies can likewise empower researchers with tools to bridge human and animal models, and aid in the dissection of global and discrete exposure-related neurodevelopmental deficits.

Supplementary Material

1

Highlights.

  • Operant testing procedures were used to attribute lead-associated deficits to discrete cognitive mechanisms

  • Higher prenatal lead (Pb) concentrations yield broad dysregulation of executive functions

  • Prenatal Pb concentrations dysregulations can be, in part, attributed to dysregulation of temporal processing

  • Sex differences were observed in two operant tasks suggesting that some Pb effects on neurocognitive function may be sexually dimorphic

Acknowledgements

This research was funded in part by NIH grants: T32HD049311, R01ES014930, R01ES013744, R24ES028522, P30ES023515, R01ES026033, R01MH122447, and R01ES029511.

This article reflects the views of the authors and should not be construed to represent FDA’s views or policies. This work was reviewed by the FDA-IRB and was determined to be exempt because the FDA investigator had no access to any subject PHI. The research that was conducted at NCTR related to this manuscript was supported by protocol S00786.

Footnotes

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

Conflict of Interest

The authors declare no conflict of interest.

Declaration of interests

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

References

  • 1.Bellinger DC, The Protean Toxicities of Lead: New Chapters in a Familiar Story. International Journal of Environmental Research and Public Health, 2011. 8(7): p. 2593–2628. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 2.Braun JM, et al. , Assessing windows of susceptibility to lead-induced cognitive deficits in Mexican children. NeuroToxicology, 2012. 33(5): p. 1040–1047. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3.Rice DC, Behavioral effects of lead: commonalities between experimental and epidemiologic data. Environ Health Perspect, 1996. 104 Suppl 2(Suppl 2): p. 337–51. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4.Santa Maria MP, Hill BD, and Kline J, Lead (Pb) neurotoxicology and cognition. Applied Neuropsychology: Child, 2019. 8(3): p. 272–293. [DOI] [PubMed] [Google Scholar]
  • 5.Bellinger DC, Chen A, and Lanphear BP, Establishing and Achieving National Goals for Preventing Lead Toxicity and Exposure in Children. JAMA Pediatrics, 2017. 171(7): p. 616–618. [DOI] [PubMed] [Google Scholar]
  • 6.Dórea JG, Environmental exposure to low-level lead (Pb) co-occurring with other neurotoxicants in early life and neurodevelopment of children. Environmental Research, 2019. 177: p. 108641. [DOI] [PubMed] [Google Scholar]
  • 7.Dietrich KN, et al. , Lead exposure and neurobehavioral development in later infancy. Environ Health Perspect, 1990. 89: p. 13–9. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8.Kim S, et al. , Association between maternal exposure to major phthalates, heavy metals, and persistent organic pollutants, and the neurodevelopmental performances of their children at 1 to 2years of age- CHECK cohort study. Science of The Total Environment, 2018. 624: p. 377–384. [DOI] [PubMed] [Google Scholar]
  • 9.Téllez-Rojo MM, et al. , Longitudinal Associations Between Blood Lead Concentrations Lower Than 10 μg/dL and Neurobehavioral Development in Environmentally Exposed Children in Mexico City. Pediatrics, 2006. 118(2): p. e323–e330. [DOI] [PubMed] [Google Scholar]
  • 10.Forns J, et al. , Exposure to metals during pregnancy and neuropsychological development at the age of 4 years. NeuroToxicology, 2014. 40: p. 16–22. [DOI] [PubMed] [Google Scholar]
  • 11.Lewis M, et al. , Prenatal exposure to heavy metals: effect on childhood cognitive skills and health status. Pediatrics, 1992. 89(6 Pt 1): p. 1010–5. [PubMed] [Google Scholar]
  • 12.Bellinger D, et al. , Low-level lead exposure and children’s cognitive function in the preschool years. Pediatrics, 1991. 87(2): p. 219–27. [PubMed] [Google Scholar]
  • 13.Taylor CM, et al. , Effects of low-level prenatal lead exposure on child IQ at 4 and 8 years in a UK birth cohort study. NeuroToxicology, 2017. 62: p. 162–169. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14.Wasserman GA, et al. , Lead exposure and intelligence in 7-year-old children: the Yugoslavia Prospective Study. Environ Health Perspect, 1997. 105(9): p. 956–62. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15.Fruh V, et al. , Prenatal lead exposure and childhood executive function and behavioral difficulties in project viva. Neurotoxicology, 2019. 75: p. 105–115. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16.Canfield RL, et al. , Low-level lead exposure, executive functioning, and learning in early childhood. Child Neuropsychol, 2003. 9(1): p. 35–53. [DOI] [PubMed] [Google Scholar]
  • 17.Kordas K, et al. , Deficits in cognitive function and achievement in Mexican first-graders with low blood lead concentrations. Environ Res, 2006. 100(3): p. 371–86. [DOI] [PubMed] [Google Scholar]
  • 18.Canfield RL, Gendle MH, and Cory-Slechta DA, Impaired neuropsychological functioning in lead-exposed children. Dev Neuropsychol, 2004. 26(1): p. 513–40. [DOI] [PubMed] [Google Scholar]
  • 19.Roy A, et al. , Lead exposure and behavior among young children in Chennai, India. Environ Health Perspect, 2009. 117(10): p. 1607–11. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20.Heidari S, et al. , Correlation between lead exposure and cognitive function in 12-year-old children: a systematic review and meta-analysis. Environ Sci Pollut Res Int, 2021. 28(32): p. 43064–43073. [DOI] [PubMed] [Google Scholar]
  • 21.Lucchini RG, et al. , Neurocognitive impact of metal exposure and social stressors among schoolchildren in Taranto, Italy. Environ Health, 2019. 18(1): p. 67. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22.Kim Y, et al. , Association between blood lead levels (<5 μg/dL) and inattention-hyperactivity and neurocognitive profiles in school-aged Korean children. Sci Total Environ, 2010. 408(23): p. 5737–43. [DOI] [PubMed] [Google Scholar]
  • 23.Lanphear BP, et al. , Low-level environmental lead exposure and children’s intellectual function: an international pooled analysis. Environ Health Perspect, 2005. 113(7): p. 894–9. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24.Bellinger DC, et al. , A pilot study of blood lead levels and neurobehavioral function in children living in Chennai, India. Int J Occup Environ Health, 2005. 11(2): p. 138–43. [DOI] [PubMed] [Google Scholar]
  • 25.Palaniappan K, et al. , Lead exposure and visual-motor abilities in children from Chennai, India. Neurotoxicology, 2011. 32(4): p. 465–70. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 26.Chiodo LM, Jacobson SW, and Jacobson JL, Neurodevelopmental effects of postnatal lead exposure at very low levels. Neurotoxicol Teratol, 2004. 26(3): p. 359–71. [DOI] [PubMed] [Google Scholar]
  • 27.Paule MG, Use of the NCTR Operant Test Battery in nonhuman primates. Neurotoxicol Teratol, 1990. 12(5): p. 413–8. [DOI] [PubMed] [Google Scholar]
  • 28.Paule MG, et al. , Quantitation of complex brain function in children: preliminary evaluation using a nonhuman primate behavioral test battery. Neurotoxicology, 1988. 9(3): p. 367–78. [PubMed] [Google Scholar]
  • 29.Paule MG, et al. , Ketamine anesthesia during the first week of life can cause long-lasting cognitive deficits in rhesus monkeys. Neurotoxicol Teratol, 2011. 33(2): p. 220–30. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 30.Schulze GE, et al. , Acute effects of marijuana smoke on complex operant behavior in rhesus monkeys. Life Sci, 1989. 45(6): p. 465–75. [DOI] [PubMed] [Google Scholar]
  • 31.Mazur J, Learning and Behavior. 4th ed. 1998, Upper Saddle River, NJ: Prentice-Hall. [Google Scholar]
  • 32.Rocha A and Trujillo KA, Neurotoxicity of low-level lead exposure: History, mechanisms of action, and behavioral effects in humans and preclinical models. Neurotoxicology, 2019. 73: p. 58–80. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 33.Cory-Slechta DA, O’Mara DJ, and Brockel BJ, Nucleus accumbens dopaminergic medication of fixed interval schedule-controlled behavior and its modulation by low-level lead exposure. J Pharmacol Exp Ther, 1998. 286(2): p. 794–805. [PubMed] [Google Scholar]
  • 34.Anderson DW, Mettil W, and Schneider JS, Effects of low level lead exposure on associative learning and memory in the rat: Influences of sex and developmental timing of exposure. Toxicol Lett, 2016. 246: p. 57–64. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 35.Anderson DW, Pothakos K, and Schneider JS, Sex and rearing condition modify the effects of perinatal lead exposure on learning and memory. Neurotoxicology, 2012. 33(5): p. 985–95. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 36.Betharia S and Maher TJ, Neurobehavioral effects of lead and manganese individually and in combination in developmentally exposed rats. Neurotoxicology, 2012. 33(5): p. 1117–27. [DOI] [PubMed] [Google Scholar]
  • 37.Tartaglione AM, et al. , Sex-Dependent Effects of Developmental Lead Exposure in Wistar Rats: Evidence from Behavioral and Molecular Correlates. Int J Mol Sci, 2020. 21(8). [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 38.Singh G, et al. , Sex-Dependent Effects of Developmental Lead Exposure on the Brain. Front Genet, 2018. 9: p. 89. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 39.Llop S, et al. , Gender differences in the neurotoxicity of metals in children. Toxicology, 2013. 311(1–2): p. 3–12. [DOI] [PubMed] [Google Scholar]
  • 40.Desrochers-Couture M, et al. , Prenatal, concurrent, and sex-specific associations between blood lead concentrations and IQ in preschool Canadian children. Environ Int, 2018. 121(Pt 2): p. 1235–1242. [DOI] [PubMed] [Google Scholar]
  • 41.Jedrychowski W, et al. , Very low prenatal exposure to lead and mental development of children in infancy and early childhood: Krakow prospective cohort study. Neuroepidemiology, 2009. 32(4): p. 270–8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 42.Ris MD, et al. , Early exposure to lead and neuropsychological outcome in adolescence. J Int Neuropsychol Soc, 2004. 10(2): p. 261–70. [DOI] [PubMed] [Google Scholar]
  • 43.Caravanos J, et al. , Blood lead levels in Mexico and pediatric burden of disease implications. Ann Glob Health, 2014. 80(4): p. 269–77. [DOI] [PubMed] [Google Scholar]
  • 44.Diaz-Ruiz A, et al. , Glazed clay pottery and lead exposure in Mexico: Current experimental evidence. Nutritional Neuroscience, 2017. 20(9): p. 513–518. [DOI] [PubMed] [Google Scholar]
  • 45.Braun JM, et al. , Relationships between lead biomarkers and diurnal salivary cortisol indices in pregnant women from Mexico City: a cross-sectional study. Environ Health, 2014. 13(1): p. 50. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 46.Chelonis JJ, et al. , Comparison of delayed matching-to-sample performance in monkeys and children. Behavioural Processes, 2014. 103: p. 261–268. [DOI] [PubMed] [Google Scholar]
  • 47.Paule MG, et al. , Operant test battery performance in children: correlation with IQ. Neurotoxicol Teratol, 1999. 21(3): p. 223–30. [DOI] [PubMed] [Google Scholar]
  • 48.Baldwin RL, et al. , Effect of Methylphenidate on Time Perception in Children With Attention-Deficit/Hyperactivity Disorder. Experimental and Clinical Psychopharmacology, 2004. 12(1): p. 57–64. [DOI] [PubMed] [Google Scholar]
  • 49.Chelonis JJ, et al. , Developmental aspects of timing behavior in children. Neurotoxicology and Teratology, 2004. 26(3): p. 461–476. [DOI] [PubMed] [Google Scholar]
  • 50.Baldwin RL, et al. , The use of an incremental repeated acquisition task to assess learning in children. Behavioural Processes, 2012. 91(1): p. 103–114. [DOI] [PubMed] [Google Scholar]
  • 51.Hu H, et al. , Fetal lead exposure at each stage of pregnancy as a predictor of infant mental development. Environ Health Perspect, 2006. 114(11): p. 1730–5. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 52.Friedman HR and Goldman-Rakic PS, Activation of the hippocampus and dentate gyrus by working-memory: a 2-deoxyglucose study of behaving rhesus monkeys. J Neurosci, 1988. 8(12): p. 4693–706. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 53.Goldman PS, Rosvold HE, and Mishkin M, Selective sparing of function following prefrontal lobectomy in infant monkeys. Exp Neurol, 1970. 29(2): p. 221–6. [DOI] [PubMed] [Google Scholar]
  • 54.Kojima S, Kojima M, and Goldman-Rakic PS, Operant behavioral analysis of memory loss in monkeys with prefrontal lesions. Brain Res, 1982. 248(1): p. 51–9. [DOI] [PubMed] [Google Scholar]
  • 55.Rada PV, Mark GP, and Hoebel BG, In vivo modulation of acetylcholine in the nucleus accumbens of freely moving rats: I. Inhibition by serotonin. Brain Res, 1993. 619(1–2): p. 98–104. [DOI] [PubMed] [Google Scholar]
  • 56.Fontes R, et al. , Time Perception Mechanisms at Central Nervous System. Neurol Int, 2016. 8(1): p. 5939. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 57.Verina T, Rohde CA, and Guilarte TR, Environmental lead exposure during early life alters granule cell neurogenesis and morphology in the hippocampus of young adult rats. Neuroscience, 2007. 145(3): p. 1037–47. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 58.Cory-Slechta DA, Garcia-Osuna M, and Greenamyre JT, Lead-induced changes in NMDA receptor complex binding: correlations with learning accuracy and with sensitivity to learning impairments caused by MK-801 and NMDA administration. Behav Brain Res, 1997. 85(2): p. 161–74. [DOI] [PubMed] [Google Scholar]
  • 59.Lanciego JL, Luquin N, and Obeso JA, Functional neuroanatomy of the basal ganglia. Cold Spring Harb Perspect Med, 2012. 2(12): p. a009621. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 60.Birtwistle J and Baldwin D, Role of dopamine in schizophrenia and Parkinson’s disease. Br J Nurs, 1998. 7(14): p. 832–4, 836, 838–41. [DOI] [PubMed] [Google Scholar]
  • 61.Chase TN and Oh JD, Striatal dopamine- and glutamate-mediated dysregulation in experimental parkinsonism. Trends Neurosci, 2000. 23(10 Suppl): p. S86–91. [DOI] [PubMed] [Google Scholar]
  • 62.Brunner D, et al. , Temporal control deficits in murine models of Huntington’s disease. International Journal of Comparative Psychology, 2015. 28. [Google Scholar]
  • 63.Curtin PC, et al. , Cognitive Training at a Young Age Attenuates Deficits in the zQ175 Mouse Model of HD. Front Behav Neurosci, 2015. 9: p. 361. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 64.Bauter MR, et al. , Glutamate and dopamine in nucleus accumbens core and shell: sequence learning versus performance. Neurotoxicology, 2003. 24(2): p. 227–43. [DOI] [PubMed] [Google Scholar]
  • 65.Ramírez Ortega D, et al. , Cognitive Impairment Induced by Lead Exposure during Lifespan: Mechanisms of Lead Neurotoxicity. Toxics, 2021. 9(2). [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 66.Toscano CD and Guilarte TR, Lead neurotoxicity: from exposure to molecular effects. Brain Res Brain Res Rev, 2005. 49(3): p. 529–54. [DOI] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

1
NIHMS1779377-supplement-1.docx (91.8KB, docx)

RESOURCES