Skip to main content
International Journal of Environmental Research and Public Health logoLink to International Journal of Environmental Research and Public Health
. 2023 Mar 28;20(7):5288. doi: 10.3390/ijerph20075288

Pyrethroid and Chlorpyrifos Pesticide Exposure, General Intellectual Abilities, and Executive Functions of School Children from Montevideo, Uruguay

Danelly Rodríguez 1, Gabriel Barg 2, Elena I Queirolo 2, James R Olson 1,3, Nelly Mañay 4, Katarzyna Kordas 1,*
Editors: David R S Lean, Fernando P Carvalho
PMCID: PMC10093823  PMID: 37047904

Abstract

Children’s developing brains are susceptible to pesticides. Less is known about the effect of exposure to chlorpyrifos and pyrethroids on executive functions (EF). We measured urinary 3,5,6-trichloro-2-pyridinol (TCPy), a metabolite of chlorpyrifos, and urinary 3-phenoxybenzoic acid (3-PBA), a general, nonspecific metabolite of pyrethroids in first-grade children from Montevideo, Uruguay (n = 241, age 80.6 ± 6.4 months, 58.1% boys). EFs were assessed with the Intra-dimensional/Extra-dimensional shift (IED), Spatial Span (SSP), and Stockings of Cambridge (SOC) tests from the Cambridge Neuropsychological Test Automated (CANTAB) Battery. General intellectual ability (GIA) was assessed using the Woodcock–Muñoz Cognitive battery. Median (range) urinary TCPy and 3-PBA levels were 16.7 (1.9, 356.9) ng/mg of creatinine and 3.3 (0.3, 110.6) ng/mg of creatinine, respectively. In multivariable generalized linear models, urinary TCPy was inversely associated with postdimensional errors on the IED task β [95% CI]: −0.11 [−0.17, −0.06]. Urinary 3-PBA was inversely associated with the total number of trials −0.07 [−0.10, −0.04], and the total number of errors −0.12 [−0.18, −0.07] on the IED task. When TCPy and 3-PBA were modeled together, the associations did not differ from single-metabolite models. We found no evidence of effect modification by blood lead level (BLL). Pesticide exposure may affect EF performance in urban children.

Keywords: pyrethroids, chlorpyrifos, pesticide exposure, cognition, child health, executive function

1. Introduction

Exposure to pesticides is a health concern globally, especially in countries in Latin America and the Caribbean [1]. For more than a decade, Uruguay has had one of the highest pesticide uses per hectare in Latin America, with peaks in 2010–2014 [2]. Children are especially sensitive to the neurotoxic consequences of environmental contaminants, including pesticides [3]. Numerous cohort studies in the rural and urban regions of the United States (U.S.) have described associations of prenatal exposures to pyrethroids and organophosphates (OPs), including chlorpyrifos, with poor IQ in 3–11-year-old children [4,5,6,7,8,9,10,11,12]. Similarly, studies in rural regions of Latin America have reported associations of prenatal exposures to organochlorine pesticides with developmental motor delays in Mexican and Ecuadorian 2-year-old children [13,14,15,16]. A followup study suggested that exposures to organochlorines in utero were associated with deficits in IQ, spatial orientation, and quantitative and verbal skills [17,18,19].

Although extensive evidence suggests that prenatal exposure to pesticides impacts IQ across childhood, there is an evidence gap in understanding the links between childhood pesticide exposure and executive functions (EF). Cross-sectional studies conducted in Latin America found relationships between exposure to OPs and EFs such as working memory, visual perception, selective attention, and language in school-age children [20,21,22,23]. However, these studies were based on measures of household proximity to agricultural crops, rather than biomarkers (i.e., metabolites) of exposure.

Studies that examined urinary biomarkers of childhood OP exposure, found inverse associations between urinary dialkyl phosphates (DAP) and EF domains, including working memory [24] and cognitive flexibility [25] in 6–7-year-old children. However, few studies have investigated exposure to chlorpyrifos and pyrethroids, which continue to be widely used, and aspects of children’s cognition. Even fewer of these studies measured specific metabolites of chlorpyrifos or pyrethroid exposures in children or assessed the effects of multiple pesticides together on cognition. Because of its cost-effectiveness, chlorpyrifos use is ubiquitous. The prevalence of chlorpyrifos use on crops worldwide is in the 5–39% range [26]. During the last decade, pyrethroids use has also increased to replace other more toxic pesticides and represents 38% of the world insecticide market [27]. In Uruguay, 38,000 tons of pesticides (pyrethroids and chlorpyrifos among others) were produced in 2012 [2]; many of these present high levels of toxicity, and more than 85% of the active ingredients used are imported [2].

One study in Costa Rica discovered that urinary 3,5,6-trichloro-2-pyridinol (TCPy), a specific metabolite of chlorpyrifos, and 3-phenoxybenzoic acid (3-PBA), a general biomarker of exposure to pyrethroids, was associated with deficits in processing speed and memory, as measured by the Wechsler Intelligence Scale (WISC-IV), in 6–9-year-old children [28].

In addition to gaps in exposure assessment, studies have paid less attention to specific neurodevelopmental domains such as EFs, and very few studies thus far have included children from urban areas. To address these gaps, we examined the cross-sectional association of urinary TCPy and urinary 3-PBA with general intellectual ability (GIA) and EFs of urban school children from Montevideo, Uruguay. We hypothesized inverse associations between urinary TCPy and 3-PBA and the cognitive performance measures.

Additionally, previous studies have indicated that lead exposure, even at lower levels, is linked to EF deficits in children [29,30]. Because lead, like pesticides, is a ubiquitous neurotoxicant and remains a public health problem in Uruguay [23,24,25], we investigated the extent to which children’s blood lead levels (BLLs) modify the association between pesticide exposure and tests of cognitive function. We hypothesized that the effect of pesticide exposure on cognitive performance will be higher in children with BLL ≥ 3.5 μg/dL compared to those with levels below 3.5 μg/dL.

2. Materials and Methods

2.1. Setting and Recruitment

The present study is a post hoc analysis of data from the Salud Ambiental Montevideo (SAM) study, originally designed to examine metal exposure, including arsenic, cadmium, and manganese among school-age children in Montevideo, Uruguay [31,32,33,34,35,36]. A total of 357 first-grade children out of 673 eligible children were recruited from elementary schools in locations with suspected or confirmed metal contamination, as previously described [36,37]. Children and parents completed questionnaires, neuropsychological tests, and biological sampling of urine and blood. Research ethics boards approved the study protocols at the Pennsylvania State University, the Catholic University of Uruguay, the Faculty of Chemistry at the University of the Republic of Uruguay, and the University at Buffalo. Of the enrolled 357 children, 116 did not have data on urinary biomarkers or full cognitive evaluation, which resulted in a study sample of 241 children.

2.2. Sociodemographic Measures

The caregivers, usually mothers, reported demographic information about themselves, their children, and their household environment. Characteristics for children included sex, age, and medical history. A social worker assessed the developmental environment of the household during a home visit using a Spanish version of the Home Observation for Measurement of the Environment (HOME) Inventory [38]. The HOME inventory score sums across 59 items related to parental accountability, motivation, emotional environment, learning materials and opportunities, active stimulation, household participation, parental involvement, and physical surroundings.

2.3. Analysis of Pesticide Metabolites in Urine

First void urine samples were collected by children with the assistance of caregivers and brought to school in collection cups provided for this purpose. The samples were kept at −20 °C until shipment to the U.S. Pyrethroid and chlorpyrifos metabolite analysis was completed at the Analytical Toxicology Laboratory, University at Buffalo. One metabolite of chlorpyrifos and 4 metabolites of pyrethroid pesticides were analyzed. The chlorpyrifos-specific metabolite, TCPy, was analyzed by gas chromatography–mass spectrometry (GC/MS) using chemical ionization in negative ion mode, with 13C-15N-3,5,6-TCPy as an internal standard, as described previously [39]. The general pyrethroid metabolite, 3-PBA, along with the alpha-cypermethrin (αCM)-specific metabolite, cis-3-(2,2-dichlorovinyl)-2,2-dimethyl-(1-cyclopropane) carboxylic acid (cis-DCCA), and the lambda-cyhalothrin (λCH)-specific metabolite, lambda cyhalothric acid, (λ-CA), were quantified in urine by negative ion chemical ionization by GC/MS, as described previously. In addition to cis-DCCA, trans-DCCA (trans 3-(2,2-dichlorovinyl)-2,2-dimethylcyclopropane carboxylic acid) was also measured since both are metabolites of cypermethrin [40]. Limits of detection (LOD) were as follows: TCPy (0.22 ng/mL), 3-PBA (0.45 ng/mL), cis-DCCA (0.09 ng/mL), trans-DCCA (0.18 ng/mL), and λCA (0.11 ng/mL).

Creatinine concentration in urine was measured at the Analytical Toxicology Lab, University at Buffalo, using the Jaffe reaction [41]. The urinary concentrations of pesticide metabolites were expressed as ng/mL urine (wet weight) and ng/mg creatinine (creatinine adjusted), according to the following formula:

Creatinine adjusted pesticide level (ng/mg creatinine) = [pesticide metabolite
(ng/mL urine)] [creatinine (mg creatinine/mL urine)]

2.4. Lead and Hemoglobin in Blood

A 25-gauge safety butterfly blood collection set (Vacutainer, Becton Dickinson, Franklin Lakes, NJ, USA) was utilized to collect blood samples. Fasting venous blood was collected in a serum tube with a clot activator and separator gel (Becton Dickinson, Franklin Lakes, NJ, USA). A drop of venous blood was extracted from the serum tube instantly and used to measure hemoglobin with a portable hemoglobinometer (HemoCue Inc., Lake Forest, CA, USA). Quality control was conducted daily with standard controls (low, medium, high). A second blood sample was collected into a heparin-coated tube (Vacutainer, Becton Dickinson, Franklin Lakes, NJ, USA). Blood samples were then stored on ice and transported to the Toxicology Laboratory “CEQUIMTOX” (Specialized Center for Chemical Toxicology) of the Faculty of Chemistry, University of the Republic of Uruguay. CEQUIMTOX participates in the Center for Disease Control’s (CDC) Lead and Multi-Element Proficiency Program (LAMP) and the Interlaboratory Program of Quality Control for Lead in Blood, in Spain (PICC Pb-S).Whole blood samples were analyzed by Atomic Absorption Spectrometry (AAS, VARIAN SpectrAA-55B) using flame (FAAS) or graphite furnace (GFAAS) ionization methods, depending on the volume of whole blood available. The graphite furnace was used for blood samples with volumes below 2 mL. In 2009–2010, the LOD was 2.5 µg/dL for FAAS and 2.0 µg/dL, then 0.8 µg/dL for GFAAS. Subsequently, in 2011–2013, the FAAS LOD was 1.8 µg/dL, and GFAAS was 0.8–1.0 µg/dL.

2.5. Woodcock–Muñoz Tests of General Cognitive Abilities

The Woodcock–Muñoz test battery is a Spanish version of the Woodcock–Johnson battery [42,43] that is typically utilized to assess cognitive impairment in a clinical environment [44,45,46]. Seven tests in the cognitive battery were conducted: verbal comprehension, visual–auditory learning, spatial relations, sound blending, concept formation, visual matching, and reversed numbers. Each subtest yields a sex- and age-standardized score, called the W score, as described previously [34]. General intellectual ability (GIA) was measured as a weighted composite of the 7 test scores. The median internal consistency reliability for all ages is 0.87 [47].

2.6. Cambridge Neuropsychological Test Automated Battery (CANTAB)

The CANTAB test is a battery of cognitive assessments given on a touchscreen device that has utility in neurotoxicology research [48,49,50]. CANTAB instructions were given to the children in Spanish by the study researcher. Because the CANTAB scales administered in this study use nonverbal, visual, and auditory stimuli, no language translation was required. Three CANTAB tests were used to assess executive functions: Stockings of Cambridge (SOC), the Intra-dimensional/Extra-dimensional shift task (IED), and Spatial Span (SSP). Among adult populations, reliability estimates are appropriate for the IED (ICC = 0.79) and moderate for the SOC and SSP tasks (ICC = 0.60 and 0.68, respectively) [51]. However, test–retest reliability assessments among children have not been conducted. In one study, intraclass correlations (ICC) for the IED were 0.78–1.0 in children and adolescents ages 11–17 with ADHD [52].

The SOC measures spatial planning capacity. In this task, children viewed 2 colored balls within 3 “stockings.” One stocking display was in the middle, and the second one was at the top of the screen. Children were asked to recreate a third stocking display at the top of the screen by putting balls in their chosen place, 1 at a time. Movement of the balls at the bottom of the stocking was not permitted before the ball on the top was moved. Four problems were displayed, and the number of moves and complexity increased progressively with trials. Spatial planning was measured as the number of problems that were correct with the minimum number of moves.

The IED task is an assessment of cognitive flexibility (rule acquisition and reversal). 2 dimensions were displayed: pink shapes and white lines. Children were instructed to click on the dimension they thought was correct. Implicit rules were applied, and the children learned these as responding proceeds. Sound and color feedback from the computer alerted the children as to whether their choice was correct. A total of 9 stages were administered. In the IED, stages 1 and 2 consisted of shapes. After a couple of trials, the rule regarding which pink shape was correct changed. Once children learned the new rule, the responding continued with more trials. In stages 3, 4, and 5, white lines were added on the shapes. Stages 6 and 7 introduced new shapes and new white line patterns, considered the intra-dimensional shift. The white lines became the important stimuli at stages 8 and 9, constituting the extra-dimensional shift. The IED outcomes of interest were estimated as the number of stages finished, the number of trials completed, the number of errors, and the pre-and post-extra-dimensional shift errors. In scoring children’s performance, an adjustment approach from the CANTAB Administration Guide was used to adjust the total number of errors and trials. Specifically, as in previous research [52,53], 25 errors were counted per failed stage because children that advanced to more stages could make more errors. With this scoring strategy, 5 children failed the IED test prior to stage 4. Two children were assigned zero errors after they failed to complete stage 9. The values for these 7 children were replaced with errors missing on the pre-and post-extra-dimensional shift. The SSP is an assessment of short-term visual memory, where 10 white squares are randomly displayed on a computer screen. The 10 squares change color 1 at a time. Children were asked to touch the boxes, remembering the sequence of color changes. As the test progressed, the number of boxes increased from 2 to 9. Thus, working memory was measured as span length (2–9 boxes). Other outcomes included errors, the number of attempts, and latency to response. In our study, 27 children (8%) failed to replicate the pattern during the 2-square trial after 3 tries and were assigned a maximum span length of 1.

2.7. Statistical Methods

Statistical analyses were conducted with SAS version 9.4 (SAS Institute Inc., Cary, NC, USA). Summary statistics for sociodemographic, biological, and cognitive measures included means (standard deviations) or medians (ranges), depending on the distribution. Biomonitoring data from the National Health and Nutrition Examination Survey (NHANES) 2007–2010 were utilized to compare exposure distributions between the study children and similarly aged children in the U.S.

Urinary pesticide biomarker values under the detection limit were assigned LOD/2. Then, values were normalized to creatinine excretion. Finally, due to non-normal distribution, these variables were natural log (ln)-transformed for further analysis. The 4 urinary pyrethroid metabolites were highly correlated (Supplemental Table S1). Therefore, we selected 3-PBA, the general biomarker of pyrethroids, for further statistical analysis. To conduct stratified models by BLL, described below, the children were split into 2 groups based on whether their BLLs fell <or were ≥3.5 µg/dL.

We compared participant characteristics between the analytical sample (those with complete cognitive performance and urinary biomarker data) and the children excluded from the study (Table 1). Multiple imputation (MI) of demographic characteristics was conducted to employ the sample of 241 that had complete data on exposure and outcome measures. Imputation was implemented in SAS across 50 cycles, and commands were selected for each variable to match its distribution (i.e., normal or Poisson). Using imputed data, separate and combined (TCPy and 3-PBA in the same model) generalized linear models were fitted to test the associations between each ln-transformed creatinine-adjusted pesticide metabolite and each cognitive endpoint. An identity log or link function was added using a generalized estimation equation to address school clusters. The reported beta coefficients in our models represent a change in a cognitive endpoint for every 1-unit change in ln-transformed pesticide urinary concentration.

Table 1.

A comparison of demographic characteristics and cognitive performance between Montevideo children aged ~7 years with urine and cognitive data (included in the study) and those without urine and cognitive data (excluded) (n = 241).

Variables N Included N Excluded
Female 101 41.9% 57 50.9%
Male 140 58.1% 55 49.1%
Age, months 241 80.6 ± 6.4 110 82.7 ± 6.2
HOME Inventory score 235 44.6 ± 8.4 97 44.6 ± 7.2
Blood lead levels, µg/dL 231 4.2 ± 2.1 84 4.0 ± 2.0
Hemoglobin, g/dL 236 13.2 ± 1.1 86 13.2 ± 1.1
General intellectual ability 241 90.2 ± 16.4 98 87.2 ± 17.6
SSP, span length 241 3.6 ± 1.0 97 3.5 ± 1.2
SSP, total errors 241 10.4 ± 4.1 97 10.7 ± 4.6
SOC, problems solved 241 5.0 ± 1.8 98 5.0 ± 1.8
IED, total trials 241 146 (64–386) 101 141 (68–425)
IED, stages completed 241 8 (2–9) 101 7 (1–9)
IED, total errors 241 54 (8–181) 101 56 (9–222)
IED, predimensional errors 241 7 (3–40) 101 7 (3–47)
IED, postdimensional errors 241 21 (0–36) 101 23 (0–35)

Categorical variables are presented as N and %, continuous variables presented as Mean ± SD, continuous variables, continuous variables that were not normally distributed presented as Median (range); Abbreviations: SSP: Spatial Span, SOC: Stockings of Cambridge, IED: Intra-Extra Dimensional Set Shift.

Intraclass correlations for the cognitive outcomes within schools ranged 0.06–0.16, and clustering by school was considered in the models. Multivariable models included age, sex, the HOME score, season of the year in which the assessment was conducted, the BLL category, and blood hemoglobin as covariates. These variables were selected based on prior literature [32,33,34], followed by covariate retention based on a > 10% difference between the crude and adjusted associations. In addition, we conducted stratified analyses to assess effect modification by BLL category (< or ≥3.5 µg/dL). We applied the Holm–Bonferroni sequential procedure [54] to all analyses to adjust for familywise error rate inflation.

3. Results

3.1. Sample Characteristics

Table 1 shows sample characteristics, including GIA and EF scores, for children with complete urine and cognitive data compared to children excluded from the analysis. The complete sample had more males (58.1% vs. 41.9%), was about 2 months younger, and completed more trials on the IED task. No other differences were noted.

3.2. Comparisons of Urinary Pesticide Metabolite Concentrations among Uruguayan and U.S. Children

Table 2 presents the distributions of all pesticides in Uruguayan children with complete urine and cognitive data (n = 241) compared to similarly aged children in the U.S. around the same period. The geometric mean urinary levels of TCPy and 3-PBA were roughly an order of magnitude higher in Uruguayan children compared to 6–11-year-old children in the U.S. At the 95th percentile, urinary concentrations of trans-DCCA were about 3-fold higher in Uruguayan children compared to U.S. children during 2007–2009 and about 1.4-fold higher in Uruguayan children compared to U.S. children during 2009–2010.

Table 2.

A comparison of urinary pesticide metabolite concentrations (ng/mg creatinine) among children with complete data (n = 241) aged ~7 years from Montevideo with children aged 6–11 years from the U.S. population.

Pesticide Metabolite N Geometric Mean P50 P75 P95 LOD #<LOD b
SAM children
TCPy 241 17.84 16.76 27.09 60.85 0.22 0
3-PBA 3.44 3.30 5.53 11.60 0.45 1
λCA 0.13 0.08 0.18 0.72 0.11 139
cis-DCCA 0.82 0.81 1.51 3.44 0.09 2
trans-DCCA 2.91 2.90 4.97 12 0.18 1
NHANES Child Biomonitoring Data (6–11 years old)
TCPy (2007–2008) 385 1.72 1.85 3.18 6.04 0.10 -
TCPy (2009–2010) 386 1.12 1.46 2.38 5.81 -
3-PBA (2007–2008) 371 0.39 0.36 1.13 9.88 0.10 -
3-PBA (2009–2010) 383 0.55 0.48 1.43 8.51 -
Trans-DCCA (2007–2009) a 384 - - - 4.01 0.60 -
Trans-DCCA (2009–2010) a 386 - - - 8.64 -
Cis-DCCA (2007–2008) a 384 - - - - 0.50 -
Cis-DCCA (2009–2010) a 386 - - - - -

a Geometric mean and median not calculated, the proportion of results below the limit of detection was too high to provide valid result. b #<LOD, and ƛCA not provided by The Fourth National Report on Human Exposure to Environmental Chemicals, 2019. Abbreviations: TCPy; 3,5,6-trichloro-2-pyridinol; 3-PBA, 3-Phenoxybenzoic Acid; cis- and trans-DCCA, cis/trans-3-(2,2-dichlorovinyl)-2,2-dimethylcyclopropane carboxylic acid; and λCA, λ cyhalothrin.

3.3. Association between Urinary TCPy, 3-PBA, and Cognitive Function Tests

Table 3 presents the associations between ln-transformed TCPy and 3-PBA and cognitive endpoints following multiple imputation of missing data, and with the Holm–Bonferroni correction. There were no statistically significant associations between GIA scores and pesticide metabolite concentrations. Regarding IED endpoints, there were statistically significant inverse associations between 3-PBA with the total number of trials completed in the IED task, 3-PBA with the total number of errors, and TCPy with the total number of extra-dimensional stage errors. When the 2 urinary biomarkers were modeled together, 3-PBA continued to be associated with total trials and total errors on the IED task. On the other hand, TCPy was no longer associated with postdimensional errors on the IED task. Although not reported, crude models were consistent with adjusted models.

Table 3.

Adjusted associations for natural log-transformed chlorpyrifos metabolite TCPy (3,5,6-trichloro-2-pyridinol) and pyrethroid metabolite 3-PBA (3-Phenoxybenzoic acid) concentrations in urine with general intellectual ability measured by the Woodcock–Muñoz Battery, and domains of executive functions measured by the CANTAB in the multiple imputed samples (n = 241).

Models of Single Metabolites, β [95% CI] 1 Models of Both Metabolites Together, β [95% CI] 1,2
Outcome TCPy 3-PBA TCPy 3-PBA
General intellectual ability −0.82 [−3.10, 1.45] −0.95 [−2.89, 0.99] −0.55 [−3.22, 2.12] −0.84 [−3.12, 1.43]
IED, stages completed 0.01 [−0.002, 0.02] 0.03 [0.01, 0.05] 0.002 [−0.02, 0.02] 0.03 [0.004, 0.05]
IED, total trials −0.04 [−0.07, −0.01] −0.07 [−0.10, −0.04] * −0.02 [−0.05, 0.02] −0.07 [−0.10, −0.03] *
IED, total errors −0.06 [−0.11, −0.007] −0.12 [−0.18, −0.07] * −0.02 [−0.08, 0.04] −0.12 [−0.18, −0.07] *
IED, predimensional errors 0.004 [−0.07, 0.08] −0.05 [−0.13, 0.03] 0.02 [−0.05, 0.10] −0.05 [−0.13, 0.02]
IED, postdimensional errors −0.11 [−0.17, −0.06] * −0.11 [−0.19, −0.02] −0.08 [−0.15, −0.02] −0.09 [−0.17, −0.01]
SSP, span length −0.02 [−0.07, 0.02] 0.01 [−0.02, 0.05] −0.03 [−0.07, 0.01] 0.02 [−0.02,0.06]
SOC, problems solved 0.01 [−0.05, 0.07] 0.01 [−0.03, 0.06] 0.01 [−0.07, 0.08] 0.01 [−0.05, 0.07]

1 Models adjusted for age at assessment (months), sex, season (fall, winter, spring, summer) blood lead (<to 3.5 μg/L and ≥3.5 μg/L), school clusters, hemoglobin, and HOME inventory scores. 2 Models include both pesticide metabolites as predictors of cognitive performance. * p < 0.05 with Holm–Bonferroni correction. Abbreviations: SSP: Spatial Span, SOC: Stockings of Cambridge, IED: Intra-Extra Dimensional Set Shift.

3.4. BLL as an Effect Modifier

Table 4 shows the results of analyses stratified by BLL< or ≥3.5 μg/dL. The association between urinary TCPy and 3-PBA and cognitive performance did not differ among children with BLLs of <3.5 μg/dL and those with ≥3.5 μg/dL as evidenced by overlapping confidence intervals.

Table 4.

Adjusted associations for natural log-transformed chlorpyrifos metabolite TCPy (3,5,6-trichloro-2-pyridinol) and pyrethroid metabolite 3-PBA (3-Phenoxybenzoic acid) concentrations in urine with general intellectual ability measured by the Woodcock–Muñoz Battery, and domains of executive functions measured by the CANTAB in the multiple imputed samples (n = 241) stratified by blood lead levels (<3.5 and ≥3.5 μg/dL).

Models of Single Metabolites, β [95% CI] 1 Models of Both Metabolites Together, β [95% CI] 1,2
TCPy 3-PBA TCPy 3-PBA
BLL < 3.5 BLL ≥ 3.5 BLL < 3.5 BLL ≥ 3.5 BLL < 3.5 BLL ≥ 3.5 BLL < 3.5 BLL ≥ 3.5
General intellectual ability −1.38 −0.31 −1.42 −0.51 −1.18 0.06 −1.34 −0.36
[3.84, 1.08] [−5.64, 5.01] [−3.52, 0.68] [−3.15, 2.12] [−3.25, 0.89] [−5.98, 6.12] [−3.67, 0.99] [−3.31, 2.59]
IED, stages completed 0.02 0.01 0.03 * 0.02 0.01 −0.003 0.04 0.02
[0.003, 0.03] [−0.06, 0.06] [0.02, 0.05] [−0.02, 0.06] [−0.01, 0.02] [−0.03, 0.02] [0.01, 0.06] [−0.01, 0.04]
IED, total trials −0.08 * −0.03 −0.08 * −0.05 −0.03 −0.01 −0.08 * −0.05
[−0.13, −0.04] [−0.07, 0.01] [−0.12, −0.04] [−0.09, −0.01] [−0.05, −0.01] [−0.06, 0.04] [−0.13, −0.04] [−0.09, −0.01]
IED, total errors −0.08 * −0.04 −0.16 * −0.09 −0.04 0.0004 −0.16 * −0.09
[−0.26, −0.04] [−0.11, 0.02] [−0.22, −0.09] [−0.21, 0.01] [−0.08, 0.003] [−0.08, 0.08] [−0.25, −0.07] [−0.16, −0.01]
IED, predimensional errors −0.001 0.01 −0.05 −0.05 0.02 0.03 −0.06 −0.05
[−0.08, 0.07] [−0.07, 0.10] [−0.15, 0.05] [−0.14, 0.04] [−0.05, 0.08] [−0.05, 0.11] [−0.15, 0.04] [−0.14, 0.03]
IED, postdimensional errors −0.21 −0.04 −0.14 −0.08 −0.08 −0.07 −0.12 −0.06
[−0.33, −0.08] [−0.11, 0.03] [−0.28, −0.0005] [−0.15, 0.002] [−0.15, −0.02] [−0.16, 0.01] [−0.26, 0.02] [−0.14, 0.02]
SSP, span length −0.02 −0.02 0.02 0.009 −0.03 −0.04 0.03 0.01
[−0.06, 0.01] [−0.07, 0.01] [−0.02, 0.06] [−0.03, 0.05] [−0.07, 0.002] [−0.09, 0.01] [−0.01, 0.06] [−0.03, 0.06]
SOC, problems solved 0.01 0.01 0.01 0.01 0.01 0.01 0.01 0.01
[−0.06, 0.08] [−0.05, 0.07] [−0.04, 0.07] [−0.04, 0.07] [−0.07, 0.08] [−0.07, 0.08] [−0.05, 0.07] [−0.06, 0.08]

1 Models adjusted for age at assessment (months), sex, season (fall, winter, spring, summer), school clusters, hemoglobin, and HOME inventory scores 2 Models include both pesticide metabolites as predictors of cognitive performance. * p < 0.05 with Holm–Bonferroni correction. Abbreviations: SSP: Spatial Span, SOC: Stockings of Cambridge, IED: Intra-Extra Dimensional.

3.5. Sensitivity Analyses

We conducted a sensitivity analysis among the nonimputed sample (n = 222). We repeated our main analysis and, for the most part, results remained unchanged (Supplemental Table S2). Upon the Holm–Bonferroni correction, statistical significance was reached for inverse associations between TCPy, the total number of postdimensional errors 3-PBA, the total number of trials completed, and errors on the IED task. As in models with imputed data, results remained unchanged when TCPy and 3-PBA were modeled together (Supplemental Table S2).

4. Discussion

In contrast to the strong evidence that prenatal pesticide exposure affects children’s cognitive development, questions remain on how childhood exposure to commonly used pesticides like chlorpyrifos and pyrethroids affect cognitive performance, including EFs. In this cross-sectional analysis, we investigated the association between 2 urinary metabolites indicative of pesticide exposure with general cognitive abilities and EFs among ~7-year-old children from Montevideo, Uruguay. Urban children exposed to pyrethroids, individually and in combination with chlorpyrifos, completed fewer trials on rule acquisition and reversal tests on the IED (cognitive flexibility) task. However, it is important to point out that the effect sizes for these associations were small. Lead continues to be a toxicant of concern globally and has similar clinical manifestations to pesticides, including neurodevelopmental deficits [55,56]. In addition to accounting for lead exposure in our main analysis, we explored effect modification of BLL (≥3.5 vs. <3.5 μg/dL) on the associations between urinary pesticide concentrations and each cognitive endpoint. We found no evidence of effect modification by BLL. Although not statistically significant, our findings of negative association between both pesticide biomarkers and general cognitive abilities are similar to previous cohort studies on prenatal exposures to OP pesticides (measured as urinary biomarkers DAP and TCPy), or pyrethroids (3-PBA) and global measures of IQ in school-age children [4,5,6,7,8,9,10,11,12].

Pesticide use in Latin America and the Caribbean accounts for 20% of global consumption [2], but these regions lack regulations or rigorous implementation compared to the U.S. This may be a reason for higher exposure in this region [57] and the differences in biomarker concentrations between U.S. and Uruguayan children of similar ages. For example, pesticide metabolites were up to 10-fold higher in Uruguayan children than in the general U.S. population of 6–11-year-olds. During the last 20 years, the growing use of pesticides in Uruguay has been linked to technological innovations and the expansion of agriculture, especially that associated with soybean cultivation [58]. In 2012, pesticide use peaked in Uruguay. Specifically, 38,000 tons of pesticides were produced [2], and more than 85% of the active ingredients were imported [2].

Several cross-sectional studies in Latin America found associations between residential distance to crops and EFs [20,21,22,23], indicating that pesticide exposure from agricultural operations is harmful to children’s neurodevelopment. However, although we did not have information regarding sources of exposure, the children in our study live in a large metropolitan area and it is likely that exposures are at least in part linked to pesticide residues on food [59,60,61]. We are also aware that residential pesticide use indoors is common in urban populations [62], and it is a possible source of exposure, although we have not quantified such pesticide use among our study participants. Although sources of exposure in our study population compared to other regions of Latin America may differ, pesticide metabolite levels measured in our study are comparable to levels reported in studies that were conducted in 2007 for children living in agricultural regions, where children experience long-term exposure. For example, median urine samples of children living in rural agricultural regions of Nicaragua had lower median levels of TCPy (3.7 vs. 16.76 ng/mg) and 3-PBA (2.8 vs. 3.30 ng/mg) metabolites than those found in our study [63]. Similarly, children from rural Costa Rica had lower median levels of TCPy (1.4 vs. 16.76 ng/mg) and 3-PBA (0.8 vs. 3.30 ng/mg) [28]. Our study was conducted in 2009–2013, when pesticide use in Uruguay had increased [2]. Thus, the comparative exposure patterns and levels in Uruguay may be due to changes in pesticide use over time and indicate that chronic pesticide exposure is a significant public health concern in cities. Furthermore, emerging research indicates that urban populations may be more likely exposed to pesticides than rural populations. Specifically, in some studies, Hispanic/Latino children living in cities had a higher prevalence of pyrethroid pesticides than those from rural towns (75.0% vs. 57.5%) [64,65], highlighting the need for research on urban populations.

Regarding the link between biomarkers of pesticide exposure and cognitive performance in children, our results are consistent with a cross-sectional study in 7-year-old children that found an inverse association between DAPs, a urinary biomarker of OP exposure, and the Wisconsin Card Sorting Test (WCST), a measure of cognitive flexibility [25]. Although we did not observe associations with other domains of EFs, other cross-sectional studies found associations between urinary biomarkers, including DAPs, TCPy, and 3-PBA and working memory in 5–6-year-old children [10,24,28]. Different domains of EFs have distinct developmental trajectories. Specifically, cognitive flexibility has a crucial period of development that ranges from 7 to 9 years of age [66]. On the other hand, attentional control emerges during infancy and develops rapidly in early childhood [66]. In the context of the timing of neurodevelopment, it is possible we saw an association specifically with cognitive flexibility because the 7–8-year-old children in our sample were experiencing growth in this domain. Because EFs continually mature over time [66], it is important to study EF domains at distinct ages.

The brain’s prefrontal cortex plays a vital role in the neural mechanisms underlying EFs [67]. The mechanistic paths by which chlorpyrifos and pyrethroids can harm children’s neurocognitive development are not understood. However, the associations described in epidemiologic studies for both pesticide metabolites may be explained by animal studies. Chlorpyrifos degrades into oxon metabolites that block acetylcholinesterase enzymes responsible for breaking down acetylcholine (AChe) [68]. AChe is a chief neurotransmitter that is essential in modulating cognitive function. Neurodevelopmental impacts of nonorganophosphate pesticides, such as pyrethroids, have been studied less in humans. Animal studies suggest that pyrethroids affect the nervous system by interrupting sodium channels in axonal membranes of neurons and inducing neuronal apoptosis [69]. Furthermore, animal studies have revealed that early life exposure to pyrethroids diminishes muscarinic cholinergic receptors, thereby hindering the dopaminergic system [70]. Another possible explanation for our findings is that among humans, hindered dopaminergic activity in the prefrontal cortex is associated with EF impairment, specifically in the domains of attention and cognitive flexibility [70]. Oxidative stress is another hypothesized mechanism for the neurotoxicity of chlorpyrifos and pyrethroids. Neurons are susceptible to oxidative stress [71] due to their higher-fat content, low antioxidative capabilities, and higher oxygen demands [72]. These critical neuronal processes underlying synaptic proliferation underpin the maturation of the prefrontal cortex [73], the regions that support the development of EFs [74,75].

We found an inverse association between TCPy and postdimensional errors, which was not in the direction that we hypothesized. The IED is a complex task that measures cognitive flexibility and entails learning when the implied rule of the test shifts (i.e., when the descriptors/characteristics of the accurate response change). Following this shift, the task is relatively more complex than before. The criterion for understanding every rule is built on successive responses, and if a child fails to learn after 50 trials, the test terminates. IED responses may depend on the development of the prefrontal cortex and EFs [76]. For example, IED performance in 7-year-old children is poorer compared to older children [76]. Therefore, optimal performance is likely to improve first in tasks assessing pre-executive shifts. We used pre-executive shift errors as a confounder in regression models to further explore the observed associations. The results remained unchanged (results not shown), suggesting that errors made prior to the extra-dimensional shift did not influence the total trials or total errors on the IED task. It is likely that due to the age of the children in this study, the task could have been challenging to most children post-shift, regardless of pesticide exposure.

The present study offered several strengths. First, pesticide exposures were estimated using biomarkers in urine. Unlike urinary biomarkers of exposure, residential distance used in previous studies [20,21,22,23] does not reflect all routes of exposure or the specific pesticides absorbed by the body. Compared to external/environmental measures of contaminants, biomarkers are believed to be more directly related to adverse health effects [59]. Biomonitoring is the gold standard for exposure assessment because it reflects all sources and routes of exposure, uptake, absorption, and metabolism [77]. Second, although there are no previous investigations to which we could compare our findings, our study sought to build a foundation for understanding if coexposure to lead may result in poorer performance than pesticide exposure alone. Associations between lead and EFs have been well established [78] and it is important to consider effect modification by lead in future studies. Third, to our knowledge, we are the first study to assess the relationship between pesticides and EFs using the CANTAB. This test battery has some advantages over standardized clinical tests, including measuring different domains of EFs and robust data relating specific performance deficits to focal brain aberrations [49]. Many tests in the CANTAB use nonverbal cues, meaning they can be given in any geographical area regardless of the language. Because of its computerized nature, the CANTAB scores are slightly affected by variations among testers, and the administration is less cumbersome than previously used measures like the Weschler Intelligence Scale (WISC-IV) and Wisconsin Sorting Tasks. Fourth, we utilized a comprehensive set of covariates that were chosen a priori [32,33,34,35] and further selected based on a >10% difference between the crude and adjusted associations. In contrast to most studies on childhood exposures to pesticides [10,22,23,25,28], we adjusted our models for iron status (blood hemoglobin) and the HOME Inventory. Iron deficiency [79,80] and the HOME Inventory [34] are salient predictors of cognitive performance in children. Fifth, we accounted for the effect of hydration status on urinary metabolite excretion by adjusting for urinary creatinine [81]. Sixth, to compensate for missing covariate data, we conducted our analysis in a sample attained using multiple imputation [82,83]. Furthermore, we conducted sensitivity analyses in a complete case sample. Sensitivity results remained the same as the main results, thereby increasing confidence in the observed inferences.

This study also had limitations. First, we had a modest sample size which limited our statistical power and resulted in wide confidence intervals. This limited our ability to make definitive conclusions on the associations between pesticide exposure and EFs in children. Second, we measured pesticide metabolites in single spot urine samples, without repeated measurements over time. Pesticide metabolites have relatively short biological half-lives, ranging 6.4–27 h [84]. While our measures indicate that the study children experienced recent exposure to both pesticides, without repeat sampling, we are unable to determine whether urinary metabolite concentrations in this study represent longer-term exposures. We assume, but cannot corroborate, that children were chronically exposed to pesticides due to the relatively high urinary concentrations as well as the extensive use of pesticides during the study years 2010–2014 [2]. In addition, prenatal exposure patterns or levels are unknown for our study sample. Another limitation is that measures of performance on neuropsychological tests alone may not detect subtle exposure-induced impacts on neural function. For example, a study did not find significant differences in cognitive flexibility test performance in adolescents with prenatal pesticides exposure but observed a concurrent decrease in prefrontal cortex activation (measured with near-infrared spectroscopy) [85]. Combining neuroimaging studies with neuropsychological assessment could be a direction for future research. Finally, this was a cross-sectional study, which limited our ability to infer causal relationships and estimate risk of poor cognitive outcomes due to pesticide exposure.

This study and its findings suggest several directions for future research, including a focus on longitudinal assessment of childhood pesticide exposures and their impact on EFs into adolescence. Future studies should identify sources of pesticide exposure in urban populations as well as assess genetic or other factors related to pesticide metabolism to further account for differences in metabolite concentrations in urine and their relationship with neurodevelopmental outcomes. It has been well established that different paraoxonase-1 (PON1) polymorphisms affect neurodevelopment in children exposed to OP pesticides [86,87]. Some studies have linked pesticide exposures with lower AChe activity [88,89] in school children from Ecuador. Thus, AChe could serve as a biomarker of neurological effects of pesticide exposure in children.

5. Conclusions

We found evidence that childhood exposure to chlorpyrifos or pyrethroids are associated with specific tasks related to executive function but not general cognitive abilities in urban children aged ~7 years. Lead did not modify these relationships. Longitudinal studies with repeated measures are necessary to understand the potential cognitive effects associated with pesticide exposure, which may motivate the phasing out of specific pesticides that are hazardous to human health and assist in the creation of new guidelines for the use of pesticides in urban areas.

Acknowledgments

We would like to acknowledge Delma Ribeiro and Graciela Y for assistance with biological sample collection; Natalia Agudelo, Karina Horta, and Alejandra Beiso for administering cognitive assessments; and Aurora Leites for conducting household visits. We would also like to thank Lai-Har Chi for the analysis of pesticide metabolites and creatinine.

Supplementary Materials

The following are available online at https://www.mdpi.com/article/10.3390/ijerph20075288/s1, Table S1: Spearman Coefficients (ρ) for creatinine-adjusted pesticide metabolites in children aged ~7 years from Montevideo; and Table S2: Adjusted associations for natural log-transformed chlorpyrifos metabolite TCPy (3,5,6-trichloro-2-pyridinol) and pyrethroids metabolite 3-PBA (3-Phenoxybenzoic acid) concentrations in urine with general intellectual ability measured by the Woodcock-Muñoz Battery, and domains of executive functions measured by the CANTAB in the non-imputed sample.

Author Contributions

Conceptualization, D.R., K.K., J.R.O.; methodology, K.K., J.R.O., E.I.Q., N.M., G.B.; formal analysis, D.R.; data curation, K.K., J.R.O., E.I.Q.; writing—original draft preparation, D.R. writing—review and editing, K.K., J.R.O., G.B., N.M., E.I.Q.; project administration, K.K., E.I.Q.; funding acquisition, K.K., E.I.Q. All authors have read and agreed to the published version of the manuscript.

Institutional Review Board Statement

The study was conducted according to the guidelines of the Declaration of Helsinki and approved by the Institutional Review Board (or Ethics Committee) of the Pennsylvania State University (#28597), Catholic University of Uruguay (B041108), and University at Buffalo (#1066).

Informed Consent Statement

Written informed consent (caregivers) or verbal assent (children) was obtained from all subjects involved in the study.

Data Availability Statement

The data are not publicly available due to sensitive participant information. The data presented in this study are available on request from the corresponding author.

Conflicts of Interest

The authors declare no conflict of interest. The funder had no role in the design of the study; in the collection, analyses, or interpretation of data, in the writing of the manuscript, or in the decision to publish the results.

Funding Statement

D.R. was supported by the T32-Initiative for Maximizing Student Development: from the National Institute of Health (NIGMS): T32 GM144920 (PI Dubocovich). The study was funded by the National Institute of Environmental Health Sciences (NIEHS)/Fogarty International Center: R21 ES16523 and R21 ES019949 (PI Kordas).

Footnotes

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

References

  • 1.Dórea J.G. Exposure to environmental neurotoxic substances and neurodevelopment in children from Latin America and the Caribbean. Environ. Res. 2021;192:110199. doi: 10.1016/j.envres.2020.110199. [DOI] [PubMed] [Google Scholar]
  • 2.Carcamo M.I. Los Plaguicidas Altamente Peligrosos (PAP) en Uruguay. 2020. [(accessed on 1 September 2022)]. Available online: https://ipen.org/documents/highly-hazardous-pesticides-uruguay.
  • 3.Grandjean P., Landrigan P.J. Developmental neurotoxicity of industrial chemicals. Lancet. 2006;368:2167–2178. doi: 10.1016/S0140-6736(06)69665-7. [DOI] [PubMed] [Google Scholar]
  • 4.Marks A.R., Harley K., Bradman A., Kogut K., Barr D.B., Johnson C., Calderon N., Eskenazi B. Organophosphate pesticide exposure and attention in young Mexican-American children: The CHAMACOS study. Environ. Health Perspect. 2010;118:1768–1774. doi: 10.1289/ehp.1002056. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5.Rauh V.A., Garfinkel R., Perera F.P., Andrews H.F., Hoepner L., Barr D., Whitehead R., Tang D., Whyatt R.W. Impact of Prenatal Chlorpyrifos Exposure on Neurodevelopment in the First 3 Years of Life Among Inner-City Children. Pediatrics. 2006;118:e1845–e1859. doi: 10.1542/peds.2006-0338. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6.Bouchard M.F., Chevrier J., Harley K.G., Kogut K., Vedar M., Calderon N., Trujillo C., Johnson C., Bradman A., Barr D.B., et al. Prenatal exposure to organophosphate pesticides and IQ in 7-year-old children. Environ. Health Perspect. 2011;119:1189–1195. doi: 10.1289/ehp.1003185. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7.Eskenazi B., Kogut K., Huen K., Harley K.G., Bouchard M., Bradman A., Boyd-Barr D., Johnson C., Holland N. Organophosphate pesticide exposure, PON1, and neurodevelopment in school-age children from the CHAMACOS study. Environ. Res. 2014;134:149–157. doi: 10.1016/j.envres.2014.07.001. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8.Grandjean P., Harari R., Barr D.B., Debes F. Pesticide Exposure and Stunting as Independent Predictors of Neurobehavioral Deficits in Ecuadorian School Children. Pediatrics. 2006;117:e546–e556. doi: 10.1542/peds.2005-1781. [DOI] [PubMed] [Google Scholar]
  • 9.Gunier R.B., Bradman A., Harley K.G., Kogut K., Eskenazi B. Prenatal residential proximity to agricultural pesticide use and IQ in 7-year-old children. Environ. Health Perspect. 2017;125:057002. doi: 10.1289/EHP504. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10.Harari R., Julvez J., Murata K., Barr D., Bellinger D.C., Debes F., Grandjean P. Neurobehavioral deficits and increased blood pressure in school-age children prenatally exposed to pesticides. Environ. Health Perspect. 2010;118:890–896. doi: 10.1289/ehp.0901582. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11.Rauh V., Arunajadai S., Horton M., Perera F., Hoepner L., Barr D.B., Whyatt R. Seven-year neurodevelopmental scores and prenatal exposure to chlorpyrifos, a common agricultural pesticide. Environ. Health Perspect. 2011;119:1196–1201. doi: 10.1289/ehp.1003160. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12.Rowe C., Gunier R., Bradman A., Harley K.G., Kogut K., Parra K., Eskenazi B. Residential proximity to organophosphate and carbamate pesticide use during pregnancy, poverty during childhood, and cognitive functioning in 10-year-old children. Environ. Res. 2016;150:128–137. doi: 10.1016/j.envres.2016.05.048. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13.Mérida-Ortega Á., Rothenberg S.J., Torres-Sánchez L., Schnaas L., Hernández-Alcaraz C., Cebrián M.E., García-Hernández R.M., Ogaz-González R., López-Carrillo L. Polyunsaturated fatty acids and child neurodevelopment among a population exposed to DDT: A cohort study. Environ. Health. 2019;18:17. doi: 10.1186/s12940-019-0456-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14.Bahena-Medina L.A., Torres-Sánchez L., Schnaas L., Cebrián M.E., Chávez C.H., Osorio-Valencia E., Hernández R.M.G., López-Carrillo L. Neonatal neurodevelopment and prenatal exposure to dichlorodiphenyldichloroethylene (DDE): A cohort study in Mexico. J. Expo. Sci. Environ. Epidemiol. 2011;21:609–614. doi: 10.1038/jes.2011.25. [DOI] [PubMed] [Google Scholar]
  • 15.Handal A.J., Harlow S.D., Breilh J., Lozoff B. Occupational exposure to pesticides during pregnancy and neurobehavioral development of infants and toddlers. Epidemiology. 2008;19:851–859. doi: 10.1097/EDE.0b013e318187cc5d. [DOI] [PubMed] [Google Scholar]
  • 16.Torres-Sánchez L.E., Rothenberg S.J., Schnaas L., Cebrián M.E., Osorio E., del Carmen Hernández M., García-Hernández R.M., del Rio-Garcia C., Wolff M.S., López-Carrillo L. In Utero p,p′-DDE Exposure and Infant Neurodevelopment: A Perinatal Cohort in Mexico. Environ. Health Perspect. 2007;115:435–439. doi: 10.1289/ehp.9566. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.Ogaz-González R., Mérida-Ortega Á., Torres-Sánchez L., Schnaas L., Hernández-Alcaraz C., Cebrián M.E., Rothenberg S.J., García-Hernández R.M., López-Carrillo L. Maternal dietary intake of polyunsaturated fatty acids modifies association between prenatal DDT exposure and child neurodevelopment: A cohort study. Environ. Pollut. 2018;238:698–705. doi: 10.1016/j.envpol.2018.03.100. [DOI] [PubMed] [Google Scholar]
  • 18.Osorio-Valencia E., Torres-Sánchez L., López-Carrillo L., Cebrián M.E., Rothenberg S.J., Chávez M.D.C.H., Schnaas L. Prenatal p,p′-DDE exposure and establishment of lateralization and spatial orientation in Mexican preschooler. NeuroToxicology. 2015;47:1–7. doi: 10.1016/j.neuro.2014.12.011. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19.Torres-Sánchez L.E., Schnaas L., Rothenberg S.J., Cebrián M.E., Osorio-Valencia E., Hernández M.D.C., García-Hernández R.M., López-Carrillo L. Prenatal p,p’-DDE Exposure and Neurodevelopment among Children 3.5–5 Years of Age. Environ. Health Perspect. 2013;121:263–268. doi: 10.1289/ehp.1205034. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20.Friedman E., Hazlehurst M.F., Loftus C., Karr C., McDonald K.N., Suarez-Lopez J.R. Residential proximity to greenhouse agriculture and neurobehavioral performance in Ecuadorian children. Int. J. Hyg. Environ. Health. 2020;223:220–227. doi: 10.1016/j.ijheh.2019.08.009. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21.Suarez-Lopez J.R., Checkoway H., Jacobs Jr D.R., Al-Delaimy W.K., Gahagan S. Potential short-term neurobehavioral alterations in children associated with a peak pesticide spray season: The Mother’s Day flower harvest in Ecuador. Neurotoxicology. 2017;60:125–133. doi: 10.1016/j.neuro.2017.02.002. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22.Eckerman D.A., Gimenes L.S., De Souza R.C., Galvão P.R.L., Sarcinelli P.N., Chrisman J.R. Age related effects of pesticide exposure on neurobehavioral performance of adolescent farm workers in Brazil. Neurotoxicol. Teratol. 2007;29:164–175. doi: 10.1016/j.ntt.2006.09.028. [DOI] [PubMed] [Google Scholar]
  • 23.Martos Mula A., Figueroa E.N., Ruggeri M.A., Giunta S.A., Wierna N.R., Bonillo M., Bovi M.G. Diferences in cognitive execution and cholinesterase activities in adolescents with environmental exposure to pesticides in Jujuy (Argentina) Rev. Toxicol. 2005;22:180–184. [Google Scholar]
  • 24.Cartier C., Warembourg C., Le Maner-Idrissi G., Lacroix A., Rouget F., Monfort C., Limon G., Durand G., Saint-Amour D., Cordier S., et al. Organophosphate insecticide metabolites in prenatal and childhood urine samples and intelligence scores at 6 years of age: Results from the mother–child PELAGIE cohort (France) Environ. Health Perspect. 2016;124:674–680. doi: 10.1289/ehp.1409472. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 25.Sánchez Lizardi P., O’Rourke M.K., Morris R.J. The effects of organophosphate pesticide exposure on Hispanic children’s cognitive and behavioral functioning. J. Pediatr. Psychol. 2008;33:91–101. doi: 10.1093/jpepsy/jsm047. [DOI] [PubMed] [Google Scholar]
  • 26.Foong S.Y., Ma N.L., Lam S.S., Peng W., Low F., Lee B.H., Alstrup A.K., Sonne C. A recent global review of hazardous chlorpyrifos pesticide in fruit and vegetables: Prevalence, remediation and actions needed. J. Hazard. Mater. 2020;400:123006. doi: 10.1016/j.jhazmat.2020.123006. [DOI] [PubMed] [Google Scholar]
  • 27.Andersen H.R., David A., Freire C., Fernández M.F., d’Cruz S.C., Reina-Pérez I., Fini J.B., Blaha L. Pyrethroids and developmental neurotoxicity—A critical review of epidemiological studies and supporting mechanistic evidence. Pt 2Environ. Res. 2022;214:113935. doi: 10.1016/j.envres.2022.113935. [DOI] [PubMed] [Google Scholar]
  • 28.van Wendel de Joode B., Mora A.M., Lindh C.H., Hernández-Bonilla D., Córdoba L., Wesseling C., Hoppin J.A., Mergler D. Pesticide exposure and neurodevelopment in children aged 6–9 years from Talamanca, Costa Rica. Cortex. 2016;85:137–150. doi: 10.1016/j.cortex.2016.09.003. [DOI] [PubMed] [Google Scholar]
  • 29.Canfield R.L., Henderson Jr C.R., Cory-Slechta D.A., Cox C., Jusko T.A., Lanphear B.P. Intellectual impairment in children with blood lead concentrations below 10 μg per deciliter. N. Engl. J. Med. 2003;348:1517–1526. doi: 10.1056/NEJMoa022848. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 30.Surkan P.J., Zhang A., Trachtenberg F., Daniel D.B., McKinlay S., Bellinger D.C. Neuropsychological function in children with blood lead levels < 10 μg/dL. Neurotoxicology. 2007;28:1170–1177. doi: 10.1016/j.neuro.2007.07.007. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 31.Barg G., Daleiro M., Queirolo E.I., Ravenscroft J., Mañay N., Peregalli F., Kordas K. Association of Low Lead Levels with Behavioral Problems and Executive Function Deficits in Schoolers from Montevideo, Uruguay. Int. J. Environ. Res. Public Health. 2018;15:2735. doi: 10.3390/ijerph15122735. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 32.Desai G., Barg G., Queirolo E.I., Vahter M., Peregalli F., Mañay N., Kordas K. A cross-sectional study of general cognitive abilities among Uruguayan school children with low-level arsenic exposure, potential effect modification by methylation capacity and dietary folate. Environ. Res. 2018;164:124–131. doi: 10.1016/j.envres.2018.02.021. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 33.Desai G., Barg G., Vahter M., Queirolo E.I., Peregalli F., Mañay N., Millen A.E., Yu J., Kordas K. Executive functions in school children from Montevideo, Uruguay and their associations with concurrent low-level arsenic exposure. Environ. Int. 2020;142:105883. doi: 10.1016/j.envint.2020.105883. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 34.Frndak S., Barg G., Canfield R.L., Quierolo E.I., Mañay N., Kordas K. Latent subgroups of cognitive performance in lead- and manganese-exposed Uruguayan children: Examining behavioral signatures. NeuroToxicology. 2019;73:188–198. doi: 10.1016/j.neuro.2019.04.004. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 35.Kordas K., Ardoino G., Coffman D.L., Queirolo E.I., Ciccariello D., Mañay N., Ettinger A.S. Patterns of Exposure to Multiple Metals and Associations with Neurodevelopment of Preschool Children from Montevideo, Uruguay. J. Environ. Public Health. 2015;2015:493471. doi: 10.1155/2015/493471. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 36.Kordas K., Burganowski R., Roy A., Peregalli F., Baccino V., Barcia E., Mangieri S., Ocampo V., Mañay N., Martínez G., et al. Nutritional status and diet as predictors of children’s lead concentrations in blood and urine. Environ. Int. 2018;111:43–51. doi: 10.1016/j.envint.2017.11.013. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 37.Roy A., Queirolo E., Peregalli F., Mañay N., Martínez G., Kordas K. Association of blood lead levels with urinary F2-8α isoprostane and 8-hydroxy-2-deoxy-guanosine concentrations in first-grade Uruguayan children. Environ. Res. 2015;140:127–135. doi: 10.1016/j.envres.2015.03.001. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 38.Bradley R.H., Caldwell B.M., Corwyn R.F. The Child Care HOME Inventories: Assessing the quality of family child care homes. Early Child. Res. Q. 2003;18:294–309. doi: 10.1016/S0885-2006(03)00041-3. [DOI] [Google Scholar]
  • 39.Farahat F.M., Ellison C.A., Bonner M.R., McGarrigle B.P., Crane A.L., Fenske R.A., Lasarev M.R., Rohlman D.S., Anger W.K., Lein P.J., et al. Biomarkers of chlorpyrifos exposure and effect in Egyptian cotton field workers. Environ. Health Perspect. 2011;119:801–806. doi: 10.1289/ehp.1002873. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 40.Singleton S.T., Lein P.J., Farahat F.M., Farahat T., Bonner M.R., Knaak J.B., Olson J.R. Characterization of α-cypermethrin exposure in Egyptian agricultural workers. Int. J. Hyg. Environ. Health. 2014;217:538–545. doi: 10.1016/j.ijheh.2013.10.003. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 41.Fabiny D.L., Ertingshausen G. Automated reaction-rate method for determination of serum creatinine with the CentrifiChem. Clin. Chem. 1971;17:696–700. doi: 10.1093/clinchem/17.8.696. [DOI] [PubMed] [Google Scholar]
  • 42.Woodcock R.W., McGrew K.S., Mather N., Schrank F.A. Woodcock-Johnson III diagnostic supplement to the tests of cognitive abilities. Riverside. 2003;10:003435520104400407. [Google Scholar]
  • 43.Muñoz-Sandoval A.F., Woodcock R.W., McGrew K.S., Mather N., Ardoino G. Batería III Woodcock-Muñoz. Cienc. Psicológicas. 2009;3:245–246. doi: 10.22235/cp.v3i2.155. [DOI] [Google Scholar]
  • 44.Taub G.E., McGrew K.S. A Confirmatory Factor Analysis of Cattell-Horn-Carroll Theory and Cross-Age Invariance of the Woodcock-Johnson Tests of Cognitive Abilities III. Sch. Psychol. Q. 2004;19:72. doi: 10.1521/scpq.19.1.72.29409. [DOI] [Google Scholar]
  • 45.Scantlebury N., Bouffet E., Laughlin S., Strother D., McConnell D., Hukin J., Fryer C., Laperriere N., Montour-Proulx I., Keene D., et al. White matter and information processing speed following treatment with cranial-spinal radiation for pediatric brain tumor. Neuropsychology. 2016;30:425. doi: 10.1037/neu0000258. [DOI] [PubMed] [Google Scholar]
  • 46.Ledbetter C., Moore A.L., Mitchell T. Cognitive effects of ThinkRx cognitive rehabilitation training for eleven soldiers with brain injury: A retrospective chart review. Front. Psychol. 2017;8:825. doi: 10.3389/fpsyg.2017.00825. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 47.Schrank F.A., McGrew K.S., Woodcock R.W. Technical Abstract (Woodcock-Johnson III Assessment Service Bulletin No. 2) Riverside Publishing; Itasca, IL, USA: 2001. [Google Scholar]
  • 48.Fray P., Robbins T. CANTAB battery: Proposed utility in neurotoxicology. Neurotoxicol. Teratol. 1996;18:499–504. doi: 10.1016/0892-0362(96)00027-X. [DOI] [PubMed] [Google Scholar]
  • 49.Luciana M., Nelson C.A. Assessment of neuropsychological function through use of the Cambridge Neuropsychological Testing Automated Battery: Performance in 4-to 12-year-old children. Dev. Neuropsychol. 2002;22:595–624. doi: 10.1207/S15326942DN2203_3. [DOI] [PubMed] [Google Scholar]
  • 50.Pase M.P., Stough C. An evidence-based method for examining and reporting cognitive processes in nutrition research. Nutr. Res. Rev. 2014;27:232–241. doi: 10.1017/S0954422414000158. [DOI] [PubMed] [Google Scholar]
  • 51.Lowe C., Rabbitt P. Test\re-test reliability of the CANTAB and ISPOCD neuropsychological batteries: Theoretical and practical issues. Neuropsychologia. 1998;36:915–923. doi: 10.1016/S0028-3932(98)00036-0. [DOI] [PubMed] [Google Scholar]
  • 52.Gau S.S.F., Shang C.Y. Executive functions as endophenotypes in ADHD: Evidence from the Cambridge Neuropsychological Test Battery (CANTAB) J. Child Psychol. Psychiatry. 2010;51:838–849. doi: 10.1111/j.1469-7610.2010.02215.x. [DOI] [PubMed] [Google Scholar]
  • 53.Potter G.G., McQuoid D.R., Payne M.E., Taylor W.D., Steffens D.C. Association of attentional shift and reversal learning to functional deficits in geriatric depression. Int. J. Geriatr. Psychiatry. 2012;27:1172–1179. doi: 10.1002/gps.3764. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 54.Holm S. A simple sequentially rejective multiple test procedure. Scand. J. Stat. 1979;6:65–70. [Google Scholar]
  • 55.Delgado C.F., Ullery M.A., Jordan M., Duclos C., Rajagopalan S., Scott K. Lead Exposure and Developmental Disabilities in Preschool-Aged Children. J. Public Health Manag. Pract. 2018;24:e10–e17. doi: 10.1097/PHH.0000000000000556. [DOI] [PubMed] [Google Scholar]
  • 56.Dórea J.G. Environmental exposure to low-level lead (Pb) co-occurring with other neurotoxicants in early life and neurodevelopment of children. Envion. Res. 2019;177:108641. doi: 10.1016/j.envres.2019.108641. [DOI] [PubMed] [Google Scholar]
  • 57.Winkler M., Atuhaire A., Fuhrimann S., Mora A., Niqagaba C., Oltramare C. Environmental Exposures, Health Effects and Institutional Determinants of Pesticide Use in Two Tropical Settings. Swiss Network for International Studies; Geneva, Switzerland: 2019. [Google Scholar]
  • 58.Soutullo A., Ríos M., Zaldúa N., Teixeira-de-Mello F. Soybean expansion and the challenge of the coexistence of agribusiness with local production and conservation initiatives: Pesticides in a Ramsar site in Uruguay. Environ. Conserv. 2020;47:97–103. doi: 10.1017/S0376892920000089. [DOI] [Google Scholar]
  • 59.Whitacre D.M. Reviews of Environmental Contamination and Toxicology. Volume 202 Springer; New York, NY, USA: 2008. [Google Scholar]
  • 60.Mert A., Qi A., Bygrave A., Stotz H.U. Trends of pesticide residues in foods imported to the United Kingdom from 2000 to 2020. Food Control. 2022;133:108616. doi: 10.1016/j.foodcont.2021.108616. [DOI] [Google Scholar]
  • 61.Hjorth K., Johansen K., Holen B., Andersson A., Christensen H.B., Siivinen K., Toome M. Pesticide residues in fruits and vegetables from South America—A Nordic project. Food Control. 2011;22:1701–1706. doi: 10.1016/j.foodcont.2010.05.017. [DOI] [Google Scholar]
  • 62.Lu C., Barr D.B., Pearson M., Bartell S., Bravo R. A longitudinal approach to assessing urban and suburban children’s exposure to pyrethroid pesticides. Environ. Health Perspect. 2006;114:1419–1423. doi: 10.1289/ehp.9043. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 63.Rodríguez T., de Joode B.V.W., Lindh C.H., Rojas M., Lundberg I., Wesseling C. Assessment of long-term and recent pesticide exposure among rural school children in Nicaragua. Occup. Environ. Med. 2012;69:119–125. doi: 10.1136/oem.2010.062539. [DOI] [PubMed] [Google Scholar]
  • 64.Trunnelle K.J., Bennett D.H., Tulve N.S., Clifton M.S., Davis M.D., Calafat A.M., Moran R., Tancredi D.J., Hertz-Picciotto I. Urinary Pyrethroid and Chlorpyrifos Metabolite Concentrations in Northern California Families and Their Relationship to Indoor Residential Insecticide Levels, Part of the Study of Use of Products and Exposure Related Behavior (SUPERB) Environ. Sci. Technol. 2014;48:1931–1939. doi: 10.1021/es403661a. [DOI] [PubMed] [Google Scholar]
  • 65.Arcury T.A., Chen H., Quandt S.A., Talton J.W., Anderson K.A., Scott R.P., Jensen A., Laurienti P.J. Pesticide exposure among Latinx children: Comparison of children in rural, farmworker and urban, non-farmworker communities. Sci. Total Environ. 2021;763:144233. doi: 10.1016/j.scitotenv.2020.144233. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 66.Anderson P. Assessment and development of executive function (EF) during childhood. Child Neuropsychol. 2002;8:71–82. doi: 10.1076/chin.8.2.71.8724. [DOI] [PubMed] [Google Scholar]
  • 67.Funahashi S., Andreau J.M. Prefrontal cortex and neural mechanisms of executive function. J. Physiol.-Paris. 2013;107:471–482. doi: 10.1016/j.jphysparis.2013.05.001. [DOI] [PubMed] [Google Scholar]
  • 68.Mansour S.A., Mossa A.-T.H. Lipid peroxidation and oxidative stress in rat erythrocytes induced by chlorpyrifos and the protective effect of zinc. Pestic. Biochem. Physiol. 2009;93:34–39. doi: 10.1016/j.pestbp.2008.09.004. [DOI] [Google Scholar]
  • 69.Narahashi T., Frey J.M., Ginsburg K.S., Roy M.L. Sodium and GABA-activated channels as the targets of pyrethroids and cyclodienes. Toxicol. Lett. 1992;64:429–436. doi: 10.1016/0378-4274(92)90216-7. [DOI] [PubMed] [Google Scholar]
  • 70.Chen K.C., Baxter M.G., Rodefer J.S. Central blockade of muscarinic cholinergic receptors disrupts affective and attentional set-shifting. Eur. J. Neurosci. 2004;20:1081–1088. doi: 10.1111/j.1460-9568.2004.03548.x. [DOI] [PubMed] [Google Scholar]
  • 71.Aloizou A.-M., Siokas V., Vogiatzi C., Peristeri E., Docea A.O., Petrakis D., Provatas A., Folia V., Chalkia C., Vinceti M., et al. Pesticides, cognitive functions and dementia: A review. Toxicol. Lett. 2020;326:31–51. doi: 10.1016/j.toxlet.2020.03.005. [DOI] [PubMed] [Google Scholar]
  • 72.Sayre L.M., Perry G., Smith M.A. Oxidative Stress and Neurotoxicity. Chem. Res. Toxicol. 2008;21:172–188. doi: 10.1021/tx700210j. [DOI] [PubMed] [Google Scholar]
  • 73.Popa-Wagner A., Mitran S., Sivanesan S., Chang E., Buga A.M. ROS and Brain Diseases: The Good, the Bad, and the Ugly. Oxid. Med. Cell Longev. 2013;2013:963520. doi: 10.1155/2013/963520. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 74.Yuan P., Raz N. Prefrontal cortex and executive functions in healthy adults: A meta-analysis of structural neuroimaging studies. Neurosci. Biobehav. Rev. 2014;42:180–192. doi: 10.1016/j.neubiorev.2014.02.005. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 75.Barch D.M., Harms M.P., Tillman R., Hawkey E., Luby J.L. Early childhood depression, emotion regulation, episodic memory, and hippocampal development. J. Abnorm. Psychol. 2019;128:81. doi: 10.1037/abn0000392. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 76.Luciana M. Practitioner Review: Computerized assessment of neuropsychological function in children: Clinical and research applications of the Cambridge Neuropsychological Testing Automated Battery (CANTAB) J. Child Psychol. Psychiatry. 2003;44:649–663. doi: 10.1111/1469-7610.00152. [DOI] [PubMed] [Google Scholar]
  • 77.Fenske R.A., Bradman A., Whyatt R.M., Wolff M.S., Barr D.B. Lessons learned for the assessment of children’s pesticide exposure: Critical sampling and analytical issues for future studies. Environ. Health Perspect. 2005;113:1455–1462. doi: 10.1289/ehp.7674. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 78.Soetrisno F.N., Delgado-Saborit J.M. Chronic exposure to heavy metals from informal e-waste recycling plants and children’s attention, executive function and academic performance. Sci. Total Environ. 2020;717:137099. doi: 10.1016/j.scitotenv.2020.137099. [DOI] [PubMed] [Google Scholar]
  • 79.Halterman J.S., Kaczorowski J.M., Aligne C.A., Auinger P., Szilagyi P.G. Iron deficiency and cognitive achievement among school-aged children and adolescents in the United States. Pediatrics. 2001;107:1381–1386. doi: 10.1542/peds.107.6.1381. [DOI] [PubMed] [Google Scholar]
  • 80.Jáuregui-Lobera I. Iron deficiency and cognitive functions. Neuropsychiatr. Dis. Treat. 2014;10:2087. doi: 10.2147/NDT.S72491. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 81.Allen R., Mage D., Gondy G., Christensen C., Barr D., Needham L. The use of a creatinine correction for reporting children’s urinary pesticide concentrations. Epidemiology. 2004;15:S69–S70. doi: 10.1097/00001648-200407000-00171. [DOI] [PubMed] [Google Scholar]
  • 82.Bartlett J.W., Seaman S.R., White I.R., Carpenter J.R., Alzheimer’s Disease Neuroimaging Initiative Multiple imputation of covariates by fully conditional specification: Accommodating the substantive model. Stat. Methods Med. Res. 2015;24:462–487. doi: 10.1177/0962280214521348. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 83.Rubin D.B. Multiple Imputation for Nonresponse in Surveys. John Wiley & Sons; Hoboken, NJ, USA: 1987. [Google Scholar]
  • 84.Baker S.E., Olsson A.O., Barr D.B. Quantification of selected pesticide metabolites in human urine using isotope dilution high-performance liquid chromatography/tandem mass spectrometry. Pt 2J. Expo. Anal. Env. Epidemiol. 2000;10:789–798. doi: 10.1038/sj.jea.7500123. [DOI] [PubMed] [Google Scholar]
  • 85.Sagiv S.K., Bruno J.L., Baker J.M., Palzes V., Kogut K., Rauch S., Gunier R., Mora A.M., Reiss A.L., Eskenazi B. Prenatal exposure to organophosphate pesticides and functional neuroimaging in adolescents living in proximity to pesticide application. Proc. Natl. Acad. Sci. USA. 2019;116:18347–18356. doi: 10.1073/pnas.1903940116. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 86.Engel S.M., Wetmur J., Chen J., Zhu C., Barr D.B., Canfield R.L., Wolff M.S. Prenatal exposure to organophosphates, paraoxonase 1, and cognitive development in childhood. Environ. Health Perspect. 2011;119:1182–1188. doi: 10.1289/ehp.1003183. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 87.Eskenazi B., Harley K., Bradman A., Weltzien E., Jewell N.P., Barr D.B., Furlong C.E., Holland N.T. Association of in utero organophosphate pesticide exposure and fetal growth and length of gestation in an agricultural population. Environ. Health Perspect. 2004;112:1116–1124. doi: 10.1289/ehp.6789. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 88.Suarez-Lopez J.R., Nazeeh N., Kayser G., Suárez-Torres J., Checkoway H., López-Paredes D., Jacobs D.R., Jr., de la Cruz F. Residential proximity to greenhouse crops and pesticide exposure (via acetylcholinesterase activity) assessed from childhood through adolescence. Environ. Res. 2020;188:109728. doi: 10.1016/j.envres.2020.109728. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 89.Suarez-Lopez J.R., Jacobs Jr D.R., Himes J.H., Alexander B.H., Lazovich D., Gunnar M. Lower acetylcholinesterase activity among children living with flower plantation workers. Environ. Res. 2012;114:53–59. doi: 10.1016/j.envres.2012.01.007. [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Data Availability Statement

The data are not publicly available due to sensitive participant information. The data presented in this study are available on request from the corresponding author.


Articles from International Journal of Environmental Research and Public Health are provided here courtesy of Multidisciplinary Digital Publishing Institute (MDPI)

RESOURCES