Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2017 Jan 1.
Published in final edited form as: Cortex. 2015 Oct 20;74:383–395. doi: 10.1016/j.cortex.2015.09.011

A 10-month prospective study of organophosphorus pesticide exposure and neurobehavioral performance among adolescents in Egypt

Diane S Rohlman a,b,*, Ahmed A Ismail c, Gaafar Abdel Rasoul c, Matthew R Bonner d, Olfat Hendy e, Kristin Mara f, Kai Wang f, James R Olson d,g
PMCID: PMC4786370  NIHMSID: NIHMS731823  PMID: 26687929

Abstract

Chlorpyrifos is an organophosphorus (OP) pesticide widely used around the world for agricultural operations. Although studies have examined exposure in children, there is limited information on adolescents who are occupationally exposed. Furthermore, there is limited research addressing the change in exposure patterns and outcomes across the application season. The goal of the current study was to examine the impact of chlorpyrifos exposure on neurobehavioral performance in adolescents before, during and after the application season. The longitudinal study was conducted in Egypt from April 2010 to January 2011, quantifying exposure and neurobehavioral performance with repeated measures prior to, during, and following the application period. At each test session, participants completed a neurobehavioral test battery and urine was collected for analysis of the chlorpyrifos metabolite 3,5,6-trichloro-2 pyridinol (TCPy) (biomarker of exposure). Cumulative urinary TCPy over the study period was used to classify participants into low (<median) and high (≥median) exposure groups. The urinary TCPy concentrations increased for both groups during the application season and decreased following the end of application. TCPy levels were significantly elevated in the high exposure group compared to the low exposure groups at all time intervals except baseline. Deficits in cumulative neurobehavioral performance were found among the high exposure group compared with the low exposure group. Additionally, changes in neurobehavioral performance across the application season indicate a pattern of impaired performance in the high exposure group compared to the low exposure group. Deficits increased during the application season and remained even months after application ceased. This study is the first to examine the impact of changes in pesticide exposure and neurobehavioral performance not only before and after the application season, but also within the application season. Furthermore, this study examines the impact of pesticide exposure on an adolescent population who may be at greater risk than adult populations.

Keywords: Pesticide, Chlorpyrifos, Adolescent, Neurobehavioral, Exposure

1. Introduction

Chlorpyrifos, an organophosphorus (OP) insecticide, is widely used around the world, and in 2007, was the primary insecticide used in agricultural applications (Grube et al., 2011). It is an acetylcholinesterase inhibitor, and at high doses is known to cause adverse health effects from occupational exposure (Meyer-Baron, Knapp, Schaper, & van Thriel, 2015; Mackenzie Ross, McManus, Harrison, & Mason, 2013). The primary metabolite of chlorpyrifos, 3,5,6-trichloro-2 pyridinol (TCPy) is often used as a urinary biomarker of chlorpyrifos exposure. Because of concern about adverse health effects, chlorpyrifos was phased out of residential use in the United States in 2001, although it is currently still used for agricultural applications in the US and is still commonly applied in other countries.

While the impact of high exposures (i.e., poisoning) to OP insecticides is well understood, the impact of low level exposures, particularly on neurobehavioral functioning, is still under debate. Despite several reviews supporting an association between low level OP exposure and deficits in neurobehavioral performance (Gonzalez-Alzaga et al., 2014; Jurewicz & Hanke, 2008; Mackenzie Ross et al., 2013; Meyer-Baron et al., 2015; Muñoz-Quezada et al., 2013; Rohlman, Anger, & Lein, 2011), other reviews have emphasized inconsistent findings and limited exposure assessments as reasons for the continued uncertainty (Burns, McIntosh, Mink, Jurek, & Li, 2013; Li, Lowe, McIntosh, & Mink, 2012). Reviews of OP exposure in adults have reported deficits in motor skills and slower reaction times, and impairments in short-term memory and executive function (Ismail, Bodner, & Rohlman, 2012; Mackenzie Ross et al., 2013; Meyer-Baron et al., 2015; Rohlman et al., 2011). Although specific outcomes across studies may vary, no study has reported an improvement in cognitive or motor function associated with exposure to OPs (Mackenzie Ross et al., 2013). Inconsistent findings across studies are often attributed to methodological issues, such as, small sample sizes, varying exposure levels across studies, and multiple ways of assessing exposure (e.g., job category vs biomarkers of exposures) (Mackenzie Ross et al., 2013; Rohlman et al., 2011). In addition, most studies have examined only a single time point; few studies have measured exposure at multiple time points (Mackenzie Ross et al., 2013; Muñoz-Quezada et al., 2013). There is a need for prospective study designs with comprehensive exposure assessment to more completely understand the impact of exposure on neurobehavioral functioning, particularly in the short-term.

Several studies have examined OP exposure in children who are primarily exposed through diet, residential exposure and para-occupational exposure (Bouchard, Bellinger, Wright, & Weisskopf, 2010; Grandjean, Harari, Barr, & Debes, 2006; Lizardi, O’Rourke, & Morris, 2008), including the longitudinal birth cohort studies in the United States (Engel et al., 2011; Eskenazi et al., 2010; Rauh et al., 2011). However, there is limited information on adolescents who are occupationally exposed. Occupational exposure levels are typically higher than residential exposure levels and may provide an opportunity to find weak associations if they exist. Studies with children have indicated that exposure to OPs is associated with deficits in neurobehavioral performance and neurodevelopment, particularly prenatal exposure (Gonzalez-Alzaga et al., 2014; Muñoz-Quezada et al., 2013). Furthermore, although the evidence linking biomarkers of exposure with neurobehavioral deficits in adults is sparse, exposure-response gradients in children from studies examining prenatal exposure have been observed (Muñoz-Quezada et al., 2013) While a few studies have examined occupational exposure in children and adolescents (Abdel Rasoul et al., 2008; Eckerman et al., 2007; Rohlman, Bodner, Arcury, Quandt, & McCauley, 2007), these studies have relied on a cross-sectional design and have limited exposure measures.

1.1. Egyptian pesticide applicators

Adolescents are hired as seasonal workers to apply pesticides to the cotton crop in Egypt. Pesticide application to the cotton crop is regulated by the Ministry of Agriculture who maintain a standardized schedule for application across regions and utilize similar equipment and application procedures across sites. Application occurs in three waves lasting a few days to two weeks. The OP pesticide chlorpyrifos is the primary insecticide applied, although the application also includes profenofos, another organophosphate compound, and other pyrethroid pesticides. Previous research has identified high pesticide exposures and decreased neurobehavioral performance in adolescents working as pesticide applicators in Egypt (Abdel Rasoul et al., 2008; Rohlman et al., 2014). However, no study has documented changes in neurobehavioral performance across the application season, to determine whether effects of exposure are cumulative across time and whether recovery occurs after cessation of applications.

The goal of the current study was to examine the impact of chlorpyrifos exposure on neurobehavioral performance in adolescent pesticide applicators and non-applicators in Egypt from April 2010 to January 2011, quantifying exposure and neurobehavioral performance with repeated measures prior to, during and following the summer application period (June–August).

2. Material and methods

A longitudinal study with repeated measures, examining pesticide exposure in adolescents, was conducted in the Menoufia Governorate, Egypt from April 2010 to January 2011. The Egyptian Ministry of Agriculture standardizes and regulates the application of pesticides to the cotton crop across districts. Seasonal workers, including adolescents, are hired to assist with the application of the pesticides. The tasks for all seasonal workers include cleaning and maintaining the equipment, mixing pesticides, holding flags in the fields to guide applicators during the application process, and applying pesticides with backpack sprayers. The pesticide application season for the cotton crop typically begins in late June and goes through mid-August (Table 1). Chlorpyrifos is the primary OP applied to the cotton crop, followed by an application of pyrethroids (lambda-cyhalothrin or alpha-cypermethrin), and benzoylurea (Diflubenzuron, Lufenuron, Chlorfluazuron) and then another application of OPs, typically profenofos or a combination of profenofos and chlorpyrifos, depending on the infestation. Although there are slight variations in the timing of the chlorpyrifos application across districts, the methods of application are consistent across field stations. Application occurs daily during the application season and typically occurred during the afternoon. Adolescents may also be engaged in pesticide application outside employment by the Ministry of Agriculture.

Table 1.

Urine biomarker and neurobehavioral data from 35 test sessions was combined into a baseline and 10 time intervals. The sample size for each test session, dates of the test session, and information about pesticide application at each field station is indicated.

Time interval Test session Days from baseline Date Number of participants in session Total enrolled TCPy values available Pesticide applied
Al-Shohada Berket El-Sabea
Pre-spray 0   1   0 April 11/12 58 58 X
  2   9 April 20/21 58 58
  3 27 May 8/9 59 66
1   4 52 June 2/3 66 76 X
  5 62 June 12/13 58 78
2   6 66 June 16/17 55 78 CPF
  7 69 June 19/20 57 78 CPF
  8 73 June 23/24 62 78 X CPF CPF
Spray 3   9 76 June 26/27 63 78 CPF CPF
10 80 June 30/31 66 78 CPF CPF
4 11 84 July 3/4 59 78 X CPF CPF
12 87 July 7/8 58 78 X CPF CPF
5 13 90 July 10/11 63 79 X CPF CPF
14 94 July 14/15 65 85 X CPF CPF
6 15 97 July 17/18 63 87 X CPF CPF
16 101   July 21/22 68 89 X PYR PYR
7 17 104   July 24/25 68 89 X PYR PYR
18 108   July 28/29 62 89 X PYR PYR
19 111   July 31/August 1 62 89 X PYR PYR
8 20 115   August 4/5 63 89 X CPF PFF
21 118   August 7/8 63 89 X CPF PFF
22 122   August 11/12 64 89 X PFF PFF
23 125   August 14/15 58 89 X PFF PFF
24 129   August 18/19 63 89 X PFF
9 25 132   August 21/22 62 89 X
26 136   August 25/26 65 89
27 139   August 28/29 60 89 X
28 146   September 4/5 63 89 X
Post-Spray 10 29 157   September 15/16 58 89 X
30 164   September 22/23 58 89 X
31 171   September 29/30 61 89
32 185   October 13/14 61 89 X
33 213   November 10/11 62 89 X
34 245   December 12/13 56 89
35 269   January 5/6 55 89 X
Open in a new tab
*

CPF = Chlorpyrifos; PYR = Pyrethroids; PFF = Profenofos.

2.1. Recruitment and data collection

Male adolescents (12–21 years old) hired by the Ministry of Agriculture were recruited from two field stations in the Menoufia governorate (i.e., Al-Shohada and Berket El-Sabea). Male adolescents from the same communities, but not working as applicators for the Ministry of Agriculture, were also recruited through convenience sampling from the same districts as the applicators (i.e., utilizing contacts through the staff from the local Ministry of Agriculture). These adolescents never worked for the Ministry of Agriculture as pesticide applicators, although they may have applied pesticides at home or as private applicators. Because many of the adolescents, both those hired by the Ministry of Agriculture and those not hired, report working as private pesticide applicators, and all participants live in an agricultural community whereby they may be exposed to pesticides through drift, all participants were considered to have the opportunity for pesticide exposure, therefore, urinary metabolite levels were used to classify participants into low and high exposure groups.

Data collection, for both applicators and non-applicators, occurred at the primary field station for each community. The pesticides and equipment are stored at the field stations and the workers and supervisors meet at the field stations prior to going out to the fields. Participants were enrolled between April 2010 and July 2010; 11% of the participants were enrolled after field spraying began. During April 2010 through early January 2011, pesticide exposure and neurobehavioral performance was evaluated at 35 time points prior to, during, and following the pesticide application season (Table 1). Each test session consisted of one day of testing at Al-Shohada, followed by a second day of testing at Berket El-Sabea. In order to examine changes across the season, these time points were collapsed into a baseline time period and 10 non-overlapping intervals lasting between one and four weeks in length (Table 1). These intervals represented time periods across the application season: pre-spray, spray, and post-spray and have been previously described (Khan et al., 2014). The baseline and two time intervals occurred prior to the insecticide application period, six intervals were during the insecticide application season and two intervals were post-season, after the OP application ended. Shorter time intervals between test sessions were selected during the application period to look for immediate changes in exposure and neurobehavioral performance associated with pesticide application.

2.2. Urine collection and analysis

Spot urine samples were collected during each test session at the beginning of the work shift, transferred on wet ice to the laboratory at Menoufia University (Shebin El-Kom, Egypt), where they were stored at −20°C until being shipped to the University at Buffalo (Buffalo, NY, USA) on dry ice for analysis. Urine samples were analyzed for 3,5,6-trichloro-2 pyridinol (TCPy), a specific metabolite of chlorpyrifos and biomarker of exposure as described earlier (Farahat et al., 2011). The method involves hydrolysis, extraction, derivatization, and analysis by negative-ion chemical ionization gas chromatographye–mass spectrometry (GC/MS) utilizing 13C-15N-3,5,6-TCPy as an internal standard. The Jaffe reaction was used for colorimetric analysis of creatinine (Fabiny & Ertingshausen, 1971) and urine TCPy concentrations are expressed as micrograms TCPy per gram creatinine. The within-run imprecision of GC/MS assay is very low (<2% coefficient of variation and an intra-class correlation coefficient of .997) and the minimum detection level was .5 ng/mL. TCPy levels from 25 of the 35 test sessions were available and used in the analysis (Table 1). Cumulative urinary TCPy for each participant was calculated using the area under the curve for the plotted values for the ten time intervals.

2.3. Neurobehavioral testing

We administered seven computer-based and six individually administered tests to assess a range of neurobehavioral functions. Tests were selected based on previous research with adolescent and adult populations occupationally exposed to OP insecticides (Rohlman et al., 2014). Computer-based tests were administered through the Behavioral Assessment and Research System (Rohlman et al., 2003), which has been used in previous studies with this population (Rohlman et al., 2014).

Prior to enrollment, written informed consent was obtained from all participants and, for those under 18, their parent or legal guardian. All participants were compensated for their time for each test session, which included questionnaires, neurobehavioral testing and collection of biological samples (approximately 1 day’s salary per visit). The study was approved by the Institutional Review Board at Oregon Health and Science University in June 2009 and by the Medical Ethics committee of the Faculty of Medicine at Menoufia University in July 2009.

2.4. Statistical methods

Because of learning effects, neurobehavioral data from the first two available sessions for each participant were excluded from the analysis. The effect of practice on completing the neurobehavioral tests was reduced by utilizing alternate forms (i.e., different sequences of numbers or stimuli) for tests as appropriate. All of the analyses were conducted in R (R Core Team, 2014) and included years of education and field station as covariates. Because age and years of education were highly correlated [r = .88, t(87) = 17.7, p < .001], only years of education was included in the model. Since the field stations had slightly different schedules of application we included field station in all analyses as a covariate. Other studies (Farahat et al., 2010) have reported differences in exposure at different field stations.

A previous analysis of a subset of this data (Crane et al., 2013) revealed that the adolescents not working for the Ministry of Agriculture were also experiencing elevated TCPy metabolite levels, indicating environmental or other exposure opportunities. Therefore, participants were divided into high and low TCPy exposure groups. This allows us to investigate, more quantitatively, exposure-response gradients across a larger range of exposure. Cumulative TCPy levels over the study period were calculated for each participant as an estimate of exposure and the median (1559) was used to assign participants into either the low exposure (<median) or high exposure (≥median) group. Urinary TCPy data from 25 of the 35 time points was available and were included in this analysis. Missing data points were imputed by drawing a line between the missing data points and using the slope of the line and the number of days between the time intervals to impute the missing value.

We first examined the impact of chlorpyrifos exposure on the overall neurobehavioral performance of the high and low exposure groups by calculating a summary score for each neurobehavioral test outcome. To find the summary score for each outcome measure, we imputed the missing values by calculating the slope between the two surrounding (non-missing) values and then multiplying the slope by the number of days between the previous time point and the time point where the missing value should be. The trapezoid method was used to find the area under the curve to determine the summary score for each neurobehavioral outcome (Atkinson, 1989). For each neurobehavioral outcome measure, multiple linear regression was used to examine its overall difference between the high and low exposure groups, controlling for years of education and field station. A p-value less than .05 was used to determine significance.

In accordance with Khan et al. (2014), data from the 35 test sessions were grouped into baseline and 10 time intervals (Table 1). Utilizing the same approach described above, the cumulative score for each neurobehavioral outcome measure during a time interval was calculated. These data were used to examine changes in performance across the application season between high and low exposure groups at baseline and each of the 10 time intervals. For each of the neurobehavioral outcome measures at each time interval, multiple linear regression was used to examine differences between the high and low exposure groups, while controlling for years of education and field station.

3. Results

A total of 59 adolescents working as applicators were recruited from the Ministry of Agriculture and 39 adolescents not working for the Ministry of Agriculture (i.e., non-applicators) were also recruited. Three adolescents were excluded from the analysis due to low participation in the study sessions, resulting in a sample size of 95 (57 applicators and 38 non-applicators). Because of learning and practice effects, neurobehavioral data from a participant’s first two test sessions were excluded, therefore participants with fewer than three test sessions were excluded from the analysis, leaving a sample size of 56 applicators and 33 non-applicators. Overall response rates across test sessions ranged from 62% to 100% with a mean response rate of 74% across all test sessions (Table 1).

A greater proportion of adolescents in the high exposure group (77%) reported working as applicators for the Ministry of Agriculture compared to adolescents in the low exposure group (48%; Table 2). However, applicators in the low exposure group reported working for the Ministry of Agriculture for more years than those in the high exposure group (3.8 years vs 2.7 years, respectively) and on average report working approximately one more day a week (5.6 days a week vs 4.4 days a week). There was a statistically significant difference between the low exposure and high exposure groups on age [t(86) = −3.07, p = .003] and years of education [t(83) = −2.62, p = .01], however, there was no significant difference between the groups on computer use. The majority of both low and high exposure participants (69% and 77%, respectively) reported a low family income, less than 500 Egyptian pounds and approximately half of the high exposure participants (47%) lived within 25 m of agricultural fields, compared to 36% of the low exposure participants. Both groups report that pesticides, primarily insecticides, were applied at home.

Table 2.

Sociodemographic characteristics of participants in the low and high exposure groups.

Variables Low exposure (n = 42) mean (SD) High exposure (n = 47) mean (SD)
Age* 16.9 (1.8) 15.7 (1.8)
Education (years)* 10.3 (1.4) 9.4 (2.0) 
Home pesticide use (years) 3.3 (1.2)  3.1 (1.6) 
Occupational application of pesticides (years)* 3.8 (1.5)  2.7 (1.4) 
Days/week of pesticide application** 5.6 (1.0)  4.4 (1.2) 
Hours/day of pesticide application*
5.4 (.6)   
5.0 (.7)   
 
% (n)
% (n)
Application status*
 Worked for Ministry of Agriculture 47.6 (20) 76.6 (36)
 Did not work for Ministry of Agriculture 52.4 (22) 23.4 (11)
Field stations
 Berket El-Sabea 52.4 (22) 48.9 (23)
 El-Shohada 47.6 (20) 51.1 (24)
Family monthly income (<500 Egyptian pound) 69.0 (29) 76.6 (36)
Work as private applicatora (yes) 69.0 (29) 68.1 (32)
Computer use (once a week or more) 71.4 (30) 78.7 (37)
Live within 25 m to agricultural field (yes) 35.7 (15) 46.8 (22)
Types of pesticides applied at home
 Herbicides 23.8 (10) 31.9 (15)
 Insecticides 69.0 (29) 70.2 (33)
 Rodenticides 16.7 (7)   8.5 (4)   
Open in a new tab
*

p < .05 for group difference.

**

p < .001 for group difference.

a

Participants reported they had mixed and applied pesticides at home during the prior year.

3.1. Pesticide exposure

Median urinary TCPy concentrations for the high and low exposure groups at baseline and each of the ten intervals are shown in Fig. 1. The urinary TCPy concentrations were significantly elevated for the high exposure group compared to the low exposure group at all time points, with the exception of baseline (day 0). Concentrations increased from baseline levels for both groups, but a greater increase was found in the high exposure group. Participants in the high exposure group showed an increase in urinary TCPy concentrations from baseline with the peak occurring near the end of the first chlorpyrifos application (day 90). Following the end of the first application period, concentrations decreased, but remained elevated compared to baseline approximately 4 months after the end of the pesticide application season. Similar to the high exposure group, participants in low exposure group also had an increase in urinary TCPy concentrations during the first application of chlorpyrifos, however, their concentrations remained elevated throughout the remainder of the application season, and did not decrease until the application season had ended (day 129). The concentrations in the low exposure group returned to baseline values once exposure ended. Among participants working for the Ministry of Agriculture, hours worked was inversely correlated with cumulative TCPy levels (−.34, p-value < .001).

Fig. 1.

Fig. 1

Open in a new tab

Median urine TCPy concentrations for the low and high exposure groups at baseline and the 10 intervals. There was no difference between the groups at baseline, all other time points showed a significant difference. The shaded area represents periods of chlorpyrifos application.

3.2. Neurobehavioral performance

Overall performance between the high and low exposure groups was examined by calculating a summary score for each neurobehavioral test outcome and then examining the differences between the two groups (Table 3). The majority of neurobehavioral summary measures, 18 out of 22, indicated worse performance for the high exposure group compared to the low exposure group (Fig. 2). These differences were statistically significant for seven measures, on four of the neurobehavioral tests: Finger Tapping (left and right hand trials), Selective Attention (number of trials), Trails (part A and part B), and Santa Ana (dominant and non-dominate hand). Only four measures (Selective Attention Latency, Reversal Learning time to learn and time to reverse, and Digit Span reverse) showed improved performance for the high exposure group compared with the low exposure group, although these differences were not significant.

Table 3.

Parameter estimates and p-values from multiple regression analyses examining the differences between the low and high exposure groups for each of the neurobehavioral outcome measures. Negative differences correspond to poorer performance of the high exposed group compared to the low exposed group, while positive differences indicate the opposite. Items in bold indicate the high exposed group performed significantly worse than the low exposed group.

Test measure Low exposure versus high exposure
Parameter estimate t-statistic DF p-value
Tapping: right hand −1564 2.47   91 .02a,b
Tapping: left hand −1506 2.46   91 .02a,b
Tapping: alternating hands −273 .50   91 .62
Reaction Time −4299 1.55   91 .12a
Trail Making: part A −1089 2.24   50 .03
Trail Making: part B −2863 3.22   50 .002
Symbol Digit −8648 −.22   90 .83
Similarities −304 1.75   50 .09b
Benton Visual Retention −27 .29   50 .77b
Serial Digit Learning: score 57 −.82   90 .42b
Serial Digit Learning: number of trials −59 −.65   90 .52b
Digit Span: forward −36 .67   91 .51b
Digit Span: reverse 0 −.004 91 .99
Selective Attention: latency 4033 .19   90 .85
Selective Attention: median inter-stimulus interval 41290 −1.23   90 .22
Selective Attention: number of trials −9712 2.42   90 .02
Block Design −62 .32   50 .75b
Visual Motor Integration −172 1.48   50 .14b
Santa Ana: dominant hand −998 2.67   50 .01b
Santa Ana: non-dominant hand −630 2.31   50 .02b
Reversal Learning: time to learn 2011 .51   90 .61
Reversal Learning: time to reverse 813 .22   90 .82
Open in a new tab
a

p < .05 for an education difference for the given test outcome.

b

p < .05 for a field station difference for the given test outcome.

Fig. 2.

Fig. 2

Open in a new tab

Standardized mean differences between high and low exposure groups on neurobehavioral outcome measures, controlled for field station and education. Negative differences correspond to poorer average performance among high exposed participants while positive differences indicate the opposite. The 95% confidence intervals for each estimated difference (solid vertical line) show the plausible range of the effect relative to zero.

To examine changes in neurobehavioral performance across time, the performance of the high and low exposure groups at baseline and each of the 10 time intervals are presented in Table 4 and Fig. 3. Although there were significant differences between the high and low exposure groups on six outcome measures at baseline, the performance of the groups were more similar (i.e., there were fewer differences between the groups) during the intervals prior to the application season (interval one) and during the early part of the application season (intervals two to five). However, as the pesticide application season continued the participants in the high exposure group performed significantly worse than those in low exposure group on a greater number of outcome measures throughout the application season and after the application season ended. These deficits in performance were associated with the end of the first round of chlorpyrifos application during interval six, improved slightly during interval seven and then got worse during interval eight, which corresponds to the last application period.

Table 4.

The number of significant differences (p-value < 0.05) between low and high exposure groups at baseline and the ten time intervals. All significant outcomes indicate that the high exposed group performed significantly worse than the low exposed group. The shaded area indicates the pesticide application period.

Time Intervals
Baseline 1 2 3 4 5 6 7 8 9 10
Tapping: right hand • • • • • • •
Tapping: left hand • • • •
Tapping: alternating hands • •
Simple Reaction Time • • • •
Trail Making: part A • •
Trail Making: part B • •
Symbol Digit • • • •
Similarities •
Benton Visual Retention • •
Serial Digit Learning: score • •
Serial Digit Learning: number of trials • • • • • • •
Digit Span: forward • • • •
Digit Span: reverse •
Selective Attention: latency •
Selective Attention: number of trials • • •
Selective Attention: ISI
Block Design •
Visual Motor Integration • •
Santa Ana: dominant hand • • • • • • • •
Santa Ana: non-dominant hand • • • •
Reversal Learning: time to learn • •
Reversal Learning: time to reverse • •
Total p-values < 0.05 8 2 2 3 4 3 9 6 11 8 9
Open in a new tab

• indicateds p-values < 0.05

Fig. 3.

Fig. 3

Fig. 3

Open in a new tab

Performance of the low and high exposed groups on neurobehavioral outcomes across the application season. Higher scores indicate better performance for the following tests: TAP (Finger Tapping), Santa Ana, SAT (Selective Attention), Benton, and DST (Digit Span). Lower scores indicate better performance for the following tests: Trail (Trail Making), SDL (Serial Digit Learning), SDT (Symbol-Digit).

4. Discussion

An examination of changes in neurobehavioral performance over time indicated a cumulative effect of pesticide exposure on neurobehavioral performance. The number of significant neurobehavioral deficits between the high and low exposed groups increased during the application season, following the first application of chlorpyrifos (Table 4; Fig. 3). Furthermore, these deficits remained for several months after the application period ended. There were consistently lower scores among the adolescents with higher levels of exposure on all of the outcome measures (Table 4), with the exception of scores on the Serial Digit Learning test at two time intervals. The tests with significant differences at multiple intervals include tests assessing psychomotor and executive function (i.e., Tapping, Symbol-Digit), fine motor (i.e., Santa Ana), and working memory and attention (i.e., Serial Digit Learning, Digit Span, Selective Attention). Furthermore, these effects persisted for five months after application had ended.

Additionally, overall performance on the neurobehavioral tests between the high and low exposure groups revealed a pattern of impaired performance on the majority of tests. These differences were significant on seven outcome measures from tests assessing psychomotor function, (i.e., Tapping, Trials A), fine motor coordination (i.e., Santa Ana), executive function (Trails B) and vigilance or attention (i.e., Selective Attention). These findings are consistent with a review of the literature which found that the majority of studies most frequently report impaired motor skills and slower reaction times, along with deficits in executive function and short-term memory (Mackenzie Ross et al., 2013).

Adolescents in the study were assigned into either the low or high exposure group based on their cumulative urinary TCPy levels over the entire 9 month study period (high exposure group ≥median; low exposure group < median). Slightly different exposure patterns were found between the high and low exposure groups. The median urinary TCPy concentrations in the high and low exposure groups both increased during the first application of chlorpyrifos. However, concentrations in the high exposure group began to decrease after the first application period, in spite of additional pesticide applications to the cotton crop. In contrast, concentrations in the low exposure group remained elevated throughout the entire application season, not decreasing until the application season had ended. Although, the concentrations in the high exposure group were higher then those in the low exposure group at all time points.

Some participants, including those who were employed by the Ministry of Agriculture and some who were not employed, were engaged in pesticide application either at home or as private pesticide applicators (Table 2). This work may have provided these adolescents with additional opportunities for exposure. Additionally, adolescents working for the Ministry of Agriculture were involved in a range of tasks associated with the application process. For example, in addition to wearing the backpack sprayers to apply the pesticides, other tasks include cleaning and maintaining the equipment, mixing pesticides, and holding flags in the fields to guide applicators during the application process. Observations during application by the research staff indicated that participants performing these tasks had less contact with the pesticides than the participants applying the pesticides. Several applicators hired by the Ministry of Agriculture reported working only a few hours during the cotton season. On average the hours worked for Ministry of Agriculture applicators ranged from 8 to 42 h per week across the application season. Interestingly, applicators working for the Ministry of Agriculture in the low exposure group reported working an average of 30 h per week, but applicators in the high exposure group reported working for the Ministry of Agriculture an average of 22 h per week (Table 2). However, in spite of these reports of increased hours worked in the low exposure group, their TCPy concentrations remained lower than those in the high exposure group. More information about the specific job tasks performed while working for the Ministry of Agriculture, the frequency of application outside of the Ministry of Agriculture, the type of pesticides that are applied, and information about the equipment and procedures being used is needed to more completely understand the exposure patterns.

Although previous studies have reported deficits associated with cumulative exposure, these have primarily focused only on pre- and post-season comparisons (Bazylewicz-Walczak, Majczakowa, & Szymczak, 1999; Daniell et al., 1992) or examined exposure across multiple years (Roldan-Tapia et al., 2006; Roldan-Tapia, Parron, & Sanchez-Santed, 2005). This is the first study to examine changes in performance at multiple time points during the application season. The current study provides evidence that deficits cumulate across the application season and continue months after the end of application. In addition, these previous studies were all conducted with adult workers and not adolescents. Adolescence is characterized as a period of rapid development. In addition to the hormonal and physiological changes associated with puberty, there are also significant developmental changes in the brain, primarily the prefrontal cortex (Crews, He, & Hodge, 2007; Spear, 2010; Steinberg, 2008). These changes are associated with behavioral changes including increases in novelty seeking and risk-taking behavior, emotional reactivity and changes in information processing speed and tasks of executive function (e.g., response inhibition, working memory and attention). Research is needed to determine if the changes occurring during this time of development make the adolescent brain more vulnerable to disruption by environmental toxins or more resilient (Kalia, 2008; Spear, 2002; Steinberg, 2008). Additionally, vulnerability is impacted by the ability to metabolize toxins, which can also vary across ages (Connors et al., 2008; Eskenazi et al., 2010; Kalia, 2008). Children have more years to live than adults, they have more time to develop diseases due to early exposures, and some effects may not appear until the child is older (Costa, Aschner, Vitalone, Syverson, & Soldin, 2004; Godfrey & Barker, 2001; Landrigan, Kimmel, Correa, & Eskenazi, 2004; Reuhl, 1991).

Several reviews have indicated an association between OP pesticide exposure and neurobehavioral deficits in spite of inconsistent findings across studies (Gonzalez-Alzaga et al., 2014; Jurewicz & Hanke, 2008; Mackenzie Ross et al., 2013; Meyer-Baron et al., 2015; Muñoz-Quezada et al., 2013; Rohlman et al., 2011), however, these reviews have also identified methodological factors or limited exposure information as reasons for inconsistent findings. Four research gaps (Muñoz-Quezada et al., 2013) have been identified as lacking in prior studies examining developmental outcomes: 1) the need to examine repeated exposures over time to understand the impact of cumulative vs short-term exposures; 2) the lack of information available on the specific OP pesticide that populations are exposed to; 3) the need for additional studies examining populations at higher risk, including those exposed to para-occupational or occupational exposures; and 4) the lack of common exposure and outcome metrics to allow comparison across studies. The current study addresses all of these concerns. The study is prospective, examining exposure and neurobehavioral performance before, during and after the pesticide application season. Pesticide application to the cotton crop is standardized across the governorate, utilizing standardized methods of application and protocol of pesticides to be applied, primarily chlorpyrifos. The focus on adolescents, who are experiencing a period of development and in addition to environmental exposures, are also occupationally exposed. Finally, we included the use of a common biomarker and standardized neurobehavioral methods, which were selected based on a review of the literature that identified these measures as demonstrating differences between exposed and non-exposed populations.

Although the study may be limited by a moderate sample size and variability in response rates across test sessions, the prospective design and repeated measures provide an opportunity to examine changes from multiple time points across the application season. Participants also demonstrated variability in performance on some tests. It is uncertain on whether this is associated with exposure or motivation of the participants. A final limitation is the focus on only a single pesticide exposure, although chlorpyrifos is widely used around the world and is the primary insecticide used in agricultural applications in Egypt and elsewhere (Grube et al., 2011). Furthermore, urinary TCPy, a sensitive and specific biomarker of exposure provided a comparatively good estimate of exposure. Although the unique characteristics of chlorpyrifos use in Egypt might limit generalizability, our study design increased internal validity which is also important. Additional information, which was not available in the current study, is needed to more closely examine the association between exposures to multiple pesticides and performance.

5. Conclusions

Biomarkers of chlorpyrifos exposure (urinary TCPy concentrations) showed an increase during the pesticide application season with recovery following the end of the application season. This pattern was found in participants from both the low exposure and high exposure groups, although participants in the high exposure group had significantly elevated metabolite levels throughout the 10-month study period. Similar to other studies, deficits in neurobehavioral performance were found between the high and low exposure groups. Changes in neurobehavioral performance across the application season indicate a pattern of impaired performance in the high exposure group compared to the low exposure group. Furthermore it was found that neurobehavioral deficits increased during the application season and remained for months after application ceased. This study is the first to examine the impact of changes in pesticide exposure and neurobehavioral performance before, during and after the application season. The findings indicate that neurobehavioral deficits increase during the application season, as exposure also increases, and remain after the application ends, even when the biomarkers of exposure are reduced. This is particularly important when considering the developmental changes that occur during adolescence. This cumulative impact of exposure is important in understanding the long-term impact of pesticide exposure on neurodevelopment.

Supplementary Material

1

Acknowledgments

We thank the Egyptian Ministry of Agriculture and the adolescents for their participation. In addition, we would like to thank Steve Hutton (Dow Agrosciences, Indianapolis, Indiana, USA) for providing 13C-15N-3,5,6-TCPy, Barbara McGarrigle for the urinary trichloro-2-pyridinol analytical work, Lizette Ortega and Megan TePoel, for their assistance with data analysis and manuscript preparation. Finally, we would like to thank Mahmoud Ismail, Tameem Abou Eleinin and Mohammed Fouaad and other members of the Research Team at Menoufia University for their assistance with data collection.

The work was supported by the Fogarty International Center and the National Institute of Environmental Health Sciences (R21 ES017223 and R01 ES022163).

Footnotes

Oregon Health and Science University and Dr. Rohlman have a significant financial interest in Northwest Education Training and Assessment, LLC, a company that may have a commercial interest in the results of this research and technology. This potential conflict of interest was reviewed by the University of Iowa and Oregon Health and Science University and an approved Conflict of Interest in Research management plan was implemented.

Supplementary data

Supplementary data related to this article can be found at http://dx.doi.org/10.1016/j.cortex.2015.09.011.

References

  1. Abdel Rasoul GM, Abou Salem ME, Mechael AA, Hendy OM, Rohlman DS, Ismail AA. Effects of occupational pesticide exposure on children applying pesticides. Neurotoxicology. 2008;29(5):833–838. doi: 10.1016/j.neuro.2008.06.009. http://dx.doi.org/10.1016/j.neuro.2008.06.009. [DOI] [PubMed] [Google Scholar]
  2. Atkinson KE. An introduction to numerical analysis. 2nd. New York: Wiley; 1989. [Google Scholar]
  3. Bazylewicz-Walczak B, Majczakowa W, Szymczak M. Behavioral effects of occupational exposure to organophosphorous pesticides in female greenhouse planting workers. Neurotoxicology. 1999;20(5):819–826. [PubMed] [Google Scholar]
  4. Bouchard MF, Bellinger DC, Wright RO, Weisskopf MG. Attention-deficit/hyperactivity disorder and urinary metabolites of organophosphate pesticides. Pediatrics. 2010;125(6):e1270–1277. doi: 10.1542/peds.2009-3058. http://dx.doi.org/10.1542/peds.2009-3058. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Burns CJ, McIntosh LJ, Mink PJ, Jurek AM, Li AA. Pesticide exposure and neurodevelopmental outcomes: review of the epidemiologic and animal studies. Journal of Toxicology and Environmental Health. Part B Critical Reviews. 2013;16(3–4):127–283. doi: 10.1080/10937404.2013.783383. http://dx.doi.org/10.1080/10937404.2013.783383. [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Connors SL, Levitt P, Matthews SG, Slotkin TA, Johnston MV, Kinney HC, et al. Fetal mechanisms in neurodevelomental disorders. Pediatric Neurology. 2008;38:163–176. doi: 10.1016/j.pediatrneurol.2007.10.009. [DOI] [PubMed] [Google Scholar]
  7. Costa LG, Aschner M, Vitalone A, Syverson T, Soldin OP. Developmental neuropathology of environmental agents. Annual Review of Pharmacology and Toxicology. 2004;44:87–110. doi: 10.1146/annurev.pharmtox.44.101802.121424. [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Crane AL, Abdel Rasoul G, Ismail AA, Hendy O, Bonner MR, Lasarev MR, et al. Longitudinal assessment of chlorpyrifos exposure and effect biomarkers in adolescent Egyptian agricultural workers. Journal of Exposure Science and Environmental Epidemiology. 2013;23(4):356–362. doi: 10.1038/jes.2012.113. http://dx.doi.org/10.1038/jes.2012.113. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Crews F, He J, Hodge C. Adolescent cortical development: a critical period of vulnerabililty for addiction. Pharmacology, Biochemistry, and Behavior. 2007;86(2):189–199. doi: 10.1016/j.pbb.2006.12.001. [DOI] [PubMed] [Google Scholar]
  10. Daniell W, Barnhart S, Demers P, Costa LG, Eaton DL, Miller M, et al. Neuropsychological performance among agricultural pesticide applicators. Environmental Research. 1992;59(1):2170228. doi: 10.1016/s0013-9351(05)80241-5. [DOI] [PubMed] [Google Scholar]
  11. Eckerman DA, Gimenes LS, de Souza RC, Galvao PR, Sarcinelli PN, Chrisman JR. Age related effects of pesticide exposure on neurobehavioral performance of adolescent farm workers in Brazil. Neurotoxicology and Teratology. 2007;29(1):164–175. doi: 10.1016/j.ntt.2006.09.028. http://dx.doi.org/10.1016/j.ntt.2006.09.028. [DOI] [PubMed] [Google Scholar]
  12. Engel SM, Wetmur J, Chen J, Zhu C, Barr DB, Canfield RL, et al. Prenatal exposure to organophosphates, paraoxonase 1, and cognitive development in childhood. Environmental Health Perspectives. 2011;119(8):1182–1188. doi: 10.1289/ehp.1003183. http://dx.doi.org/10.1289/ehp.1003183. [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Eskenazi B, Huen K, Marks A, Harley KG, Bradman A, Barr DB, et al. PON1 and neurodevelopment in children from the CHAMACOS study exposed to organophosphate pesticides in utero. Environmental Health Perspectives. 2010;118(12):1775–1781. doi: 10.1289/ehp.1002234. http://dx.doi.org/10.1289/ehp.1002234. [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. Fabiny DL, Ertingshausen G. Automated reaction-rate method for determination of serum creatinine with the CentrifiChem. Clinical Chemistry. 1971;17(8):696–700. [PubMed] [Google Scholar]
  15. Farahat FM, Ellison CA, Bonner MR, McGarrigle BP, Crane AL, Fenske RA, et al. Biomarkers of chlorpyrifos exposure and effect in Egyptian cotton field workers. Environmental Health Perspectives. 2011;119(6):801–806. doi: 10.1289/ehp.1002873. http://dx.doi.org/10.1289/ehp.1002873. [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Farahat FM, Fenske RA, Olson JR, Galvin K, Bonner MR, Rohlman DS, et al. Chlorpyrifos exposures in Egyptian cotton field workers. Neurotoxicology. 2010;31(3):297–304. doi: 10.1016/j.neuro.2010.02.005. http://dx.doi.org/10.1016/j.neuro.2010.02.005. [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Godfrey KM, Barker DJ. Fetal programming and adult health. Public Health Nutrition. 2001;4:611–624. doi: 10.1079/phn2001145. [DOI] [PubMed] [Google Scholar]
  18. Gonzalez-Alzaga B, Lacasana M, Aguilar-Garduno C, Rodriguez-Barranco M, Ballester F, Rebagliato M, et al. A systematic review of neurodevelopmental effects of prenatal and postnatal organophosphate pesticide exposure. Toxicology Letters. 2014;230(2):104–121. doi: 10.1016/j.toxlet.2013.11.019. http://dx.doi.org/10.1016/j.toxlet.2013.11.019. [DOI] [PubMed] [Google Scholar]
  19. Grandjean P, Harari R, Barr DB, Debes F. Pesticide exposure and stunting as independent predictors of neurobehavioral deficits in Ecuadorian school children. Pediatrics. 2006;117(3):e546–556. doi: 10.1542/peds.2005-1781. http://dx.doi.org/10.1542/peds.2005-1781. [DOI] [PubMed] [Google Scholar]
  20. Grube AD, David Kiely T, Wu La. Pesticide industry sales and usage report: 2006 and 2007 market estimates. U.S. EPA; 2011. [Google Scholar]
  21. Ismail AA, Bodner TE, Rohlman DS. Neurobehavioral performance among agricultural workers and pesticide applicators: a meta-analytic study. Occupational and Environmental Medicine. 2012;69(7):457–464. doi: 10.1136/oemed-2011-100204. http://dx.doi.org/10.1136/oemed-2011-100204. [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Jurewicz J, Hanke W. Prenatal and childhood exposure to pesticides and neurobehavioral development: review of epidemiological studies. International Journal of Occupational Medicine and Environmental Health. 2008;21(2):121–132. doi: 10.2478/v10001-008-0014-z. http://dx.doi.org/10.2478/v10001-008-0014-z. [DOI] [PubMed] [Google Scholar]
  23. Kalia M. Brain development: anatomy, connectivity, adaptive plasticity, and toxicity. Metabolism Clinical and Experimental. 2008;57(Suppl 2):S2–S5. doi: 10.1016/j.metabol.2008.07.009. [DOI] [PubMed] [Google Scholar]
  24. Khan K, Ismail AA, Abdel Rasoul G, Bonner MR, Lasarev MR, Hendy O, et al. Longitudinal assessment of chlorpyrifos exposure and self-reported neurological symptoms in adolescent pesticide applicators. BMJ Open. 2014;4(3):e004177. doi: 10.1136/bmjopen-2013-004177. http://dx.doi.org/10.1136/bmjopen-2013-004177. [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Landrigan PJ, Kimmel CA, Correa A, Eskenazi B. Children’s health and the environment: public health issues and challenges for risk assessment. Environmental Health Perspectives. 2004;112(2):257–265. doi: 10.1289/ehp.6115. [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Li AA, Lowe KA, McIntosh LJ, Mink PJ. Evaluation of epidemiology and animal data for risk assessment: chlorpyrifos developmental neurobehavioral outcomes. Journal of Toxicology and Environmental Health. Part B, Critical Reviews. 2012;15(2):109–184. doi: 10.1080/10937404.2012.645142. http://dx.doi.org/10.1080/10937404.2012.645142. [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. Lizardi PS, O’Rourke MK, Morris RJ. The effects of organophosphate pesticide exposure on Hispanic children’s cognitive and behavioral functioning. Journal of Pediatric Psychology. 2008;33(1):91–101. doi: 10.1093/jpepsy/jsm047. http://dx.doi.org/10.1093/jpepsy/jsm047. [DOI] [PubMed] [Google Scholar]
  28. Mackenzie Ross SM, McManus IC, Harrison V, Mason O. Neurobehavioral problems following low-level exposure to organophosphate pesticides: a systematic and meta-analytic review. Critical Reviews in Toxicology. 2013;43(1):21–44. doi: 10.3109/10408444.2012.738645. http://dx.doi.org/10.3109/10408444.2012.738645. [DOI] [PubMed] [Google Scholar]
  29. Meyer-Baron M, Knapp G, Schaper M, van Thriel C. Meta-analysis on occupational exposure to pesticidese–neurobehavioral impact and dose-response relationships. Environmental Research. 2015;136:234–245. doi: 10.1016/j.envres.2014.09.030. http://dx.doi.org/10.1016/j.envres.2014.09.030. [DOI] [PubMed] [Google Scholar]
  30. Muñoz-Quezada MT, Lucero BA, Barr DB, Steenland K, Levy K, Ryan PB, et al. Neurodevelopmental effects in children associated with exposure to organophosphate pesticides: a systematic review. Neurotoxicology. 2013;39:158–168. doi: 10.1016/j.neuro.2013.09.003. http://dx.doi.org/10.1016/j.neuro.2013.09.003. [DOI] [PMC free article] [PubMed] [Google Scholar]
  31. Rauh V, Arunajadai S, Horton M, Perera F, Hoepner L, Barr DB, et al. Seven-year neurodevelopmental scores and prenatal exposure to chlorpyrifos, a common agricultural pesticide. Environmental Health Perspectives. 2011;119(8):1196–1201. doi: 10.1289/ehp.1003160. http://dx.doi.org/10.1289/ehp.1003160. [DOI] [PMC free article] [PubMed] [Google Scholar]
  32. Reuhl KR. Delayed expression of neurotoxicity: the problem of silent damage. Neurotoxicology. 1991;12:341–346. [PubMed] [Google Scholar]
  33. Rohlman DS, Anger WK, Lein PJ. Correlating neurobehavioral performance with biomarkers of organophosphorous pesticide exposure. Neurotoxicology. 2011;32(2):268–276. doi: 10.1016/j.neuro.2010.12.008. http://dx.doi.org/10.1016/j.neuro.2010.12.008. [DOI] [PMC free article] [PubMed] [Google Scholar]
  34. Rohlman DS, Bodner T, Arcury TA, Quandt SA, McCauley L. Developing methods for assessing neurotoxic effects in Hispanic non-English speaking children. Neurotoxicology. 2007;28(2):240–244. doi: 10.1016/j.neuro.2006.03.021. http://dx.doi.org/10.1016/j.neuro.2006.03.021. [DOI] [PubMed] [Google Scholar]
  35. Rohlman DS, Gimenes LS, Eckerman DA, Kang SK, Farahat FM, Anger WK. Development of the Behavioral Assessment and Research System (BARS) to detect and characterize neurotoxicity in humans. Neurotoxicology. 2003;24(4–5):523–531. doi: 10.1016/s0161-813x(03)00023-8. [DOI] [PubMed] [Google Scholar]
  36. Rohlman DS, Ismail AA, Abdel-Rasoul G, Lasarev M, Hendy O, Olson JR. Characterizing exposures and neurobehavioral performance in Egyptian adolescent pesticide applicators. Metabolic Brain Disease. 2014;29(3):845–855. doi: 10.1007/s11011-014-9565-9. http://dx.doi.org/10.1007/s11011-014-9565-9. [DOI] [PMC free article] [PubMed] [Google Scholar]
  37. Roldan-Tapia L, Nieto-Escamez FA, del Aguila EM, Laynez F, Parron T, Sanchez-Santed F. Neuropsychological sequelae from acute poisoning and long-term exposure to carbamate and organophosphate pesticides. Neurotoxicology and Teratology. 2006;28(6):694–703. doi: 10.1016/j.ntt.2006.07.004. http://dx.doi.org/10.1016/j.ntt.2006.07.004. [DOI] [PubMed] [Google Scholar]
  38. Roldan-Tapia L, Parron T, Sanchez-Santed F. Neuropsychological effects of long-term exposure to organophosphate pesticides. Neurotoxicology and Teratology. 2005;27(2):259–266. doi: 10.1016/j.ntt.2004.12.002. http://dx.doi.org/10.1016/j.ntt.2004.12.002. [DOI] [PubMed] [Google Scholar]
  39. Spear LP. Alcohol’s effect on adolescents. Alcohol Research and Health. 2002;26:287–291. [PMC free article] [PubMed] [Google Scholar]
  40. Spear LP. The behavioral neuroscience of adolesence. New York: W.W. Norton & Company; 2010. [Google Scholar]
  41. Steinberg L. A social neuroscience perspective on adolescent risk-taking. Developmental Review. 2008;28:78–106. doi: 10.1016/j.dr.2007.08.002. [DOI] [PMC free article] [PubMed] [Google Scholar]
  42. Team, R.C. R: A language and environment for statistical computing, 2012. Vienna, Austria: R Foundation for Statistical Computing; 2014. [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

1
NIHMS731823-supplement-1.docx (77.3KB, docx)

RESOURCES