Skip to main content
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2021 Mar 1.
Published in final edited form as: Neurotoxicol Teratol. 2020 Feb 14;78:106865. doi: 10.1016/j.ntt.2020.106865

Potential Frameworks to Support Evaluation of Mechanistic Data for Developmental Neurotoxicity Outcomes: A Symposium Report

Laura Carlson 1, Deborah A Cory-Slechta 2, Frances A Champagne 3, Laura Dishaw 1, Elaine Faustman 4, William Mundy 5, Deborah Segal 6, Christina Sobin 7, Carol Starkey 8, Michele Taylor 1, Susan L Makris 6, Andrew Kraft 6
PMCID: PMC7160758  NIHMSID: NIHMS1568191  PMID: 32068112

Abstract

A key challenge in systematically incorporating mechanistic data into human health assessments is that, compared to studies of apical health endpoints, these data are both more abundant (mechanistic studies routinely outnumber other studies by several orders of magnitude) and more heterogeneous (e.g. different species, test system, tissue, cell type, exposure paradigm, or specific assays performed). A structured decision-making process for organizing, integrating, and weighing mechanistic DNT data for use in human health risk assessments will improve the consistency and efficiency of such evaluations. At the Developmental Neurotoxicology Society (DNTS) 2016 annual meeting, a symposium was held to address the application of existing organizing principles and frameworks for evaluation of mechanistic data relevant to interpreting neurotoxicology data. Speakers identified considerations with potential to advance the use of mechanistic DNT data in risk assessment, including considering the context of each exposure, since epigenetics, tissue type, sex, stress, nutrition and other factors can modify toxicity responses in organisms. It was also suggested that, because behavior is a manifestation of complex nervous system function, the presence and absence of behavioral change itself could be used to organize the interpretation of multiple complex simultaneous mechanistic changes. Several challenges were identified with frameworks and their implementation, and ongoing research further developing these approaches represents an early step towards full evaluation of mechanistic DNT data for assessments.

Keywords: Neurotoxicology, developmental neurotoxicology, mechanistic toxicology, risk assessment

Introduction

Risk assessment has typically focused on apical health outcomes identified in human and animal studies. In recent years, however, there has been a push to incorporate more fully mechanistic data, including those from a broad range of laboratory techniques that examine toxicant effects at the tissue, cellular, and molecular level. These mechanistic studies can examine precursor effects, inform the human relevance of animal data, inform susceptibility, or inform chemical mode of action and/or support biological plausibility linking a chemical exposure to an adverse outcome such as developmental neurotoxicity (DNT). Mechanistic information includes any measurements related to health outcomes that inform the biological/chemical events associated with chemical exposures but are not generally considered by themselves to be adverse outcomes.. In 2014 and 2018, the National Academy of Sciences recommended that mechanistic data be better and more efficiently incorporated into the assessments performed by the EPA’s Integrated Risk Information Systems program by “standardizing an approach for synthesizing evidence within data streams (human, animal, and mechanistic) and integrating evidence across data streams” (Council 2014, National Academies of Sciences and Medicine 2018). Specifically, these reports identify that mechanistic evidence can provide support of conclusions about chemical hazards and their related mechanisms of action and suggest methods for identifying and presenting reviewed studies in tabular and graphic forms so that study characteristics are clear (Kushman, Kraft et al. 2013). The committee expects similar evaluation methods for other types of mechanistic evidence to emerge on a case-by-case basis and to include methods for determining at what stage and how mechanistic data could be used in an IRIS assessment (National Academies of Sciences and Medicine 2018). Nonetheless, establishment of a framework for when and how mechanistic data would be identified, evaluated, and used remains challenging. The issues surrounding evaluation and use of mechanistic data in systematic review are an area of ongoing research in the broader scientific community. Addressing this recommendation presents both opportunities and challenges to the risk assessment community.

For risk assessors, the use of systematic review tools can increase transparency and scientific rigor by providing a comprehensive summary of the literature, in a manner that is both replicable and minimizes the potential for reviewer bias (Cook, Mulrow et al. 1997, Higgins and Green 2011, Rooney, Boyles et al. 2014). While this manuscript is not a systematic review, it will discuss methods, potential frameworks, and principles for organizing and evaluating mechanistic DNT data that may be useful in the context of a systematic review. These ideas were part of a symposium held at the 2016 Developmental Neurotoxicology Society annual meeting, which focused on organizing principles and frameworks for evaluation of mechanistic data relevant to interpreting developmental neurotoxicology data. The symposium speakers’ presentation titles and abstracts are presented in supplemental information (table S1).

A robust mechanistic database can also be a valuable tool for risk assessors, particularly for emerging contaminants or chemicals for which extensive human and animal toxicity data are not available. These mechanistic in vitro and in vivo data can provide critical support for the biological plausibility of adverse health outcomes. For example, they can inform structureactivity relationships and provide information on potential variability in physiological responses across species. In addition, mechanistic data can identify and quantify key events at the tissue, cellular, and molecular levels that precede adverse outcomes; this can then lead to identification of predicative biomarkers and targeted testing strategies to enhance chemical evaluation. However, for data-rich chemicals, the sheer number of mechanistic studies can also present challenges. Mechanistic studies exploit a diverse set of study designs that can include cell culture systems, computational modeling, and alternative animal models. Relationships between upstream chemical interactions at the molecular level (e.g., receptor binding) and recognized downstream adverse outcomes (such as behavioral impairment) are not necessarily well defined or understood in these systems.

Furthermore, interpreting mechanistic data requires knowledge of assay reproducibility, biological relevance, predictive validity, and the relationship of the model system to the whole animal model and/or to humans. For example, in vitro DNT assays must consider the cell and/or tissue types utilized, the physiology of these cells in vitro, potential species and sex differences, influence of endocrine/hormonal signals, as well as the impact of these chemicals on neurodevelopment at different developmental periods. Additionally, whether a chemical can partition across media (blood, cerebrospinal fluid) or cross the blood:brain or placental barriers in mammals is a consideration when evaluating the relevance of a particular in vitro assay. This manuscript discusses methods, potential frameworks, and principles for organizing and evaluating mechanistic neurotoxicity and developmental neurotoxicity data that were part of the 2016 DNTS symposium “Systematic Evaluations of Mechanistic Data for Developmental Neurotoxicity Outcomes.”.

Traditional Developmental Neurotoxicology Testing

The traditional approach for evaluating developmental neurotoxicity in regulatory guideline studies (EPA 1998, OECD 2007, OECD 2012, OECD 2018) includes dosing rats during gestation and lactation, and then evaluating neurobehavior (i.e., functional observations, auditory startle habituation, motor activity, and/or cognitive testing), brain weights, and neuropathology at juvenile and/or adult life stages in the offspring. This process can lack sensitivity for identifying neurodevelopmental outcomes and is expensive and time consuming. The guideline tests are not designed to provide an understanding of underlying mechanisms that are responsible for observed DNT effects, and the data sets collected may have high variability (Sachana, Bal-Price et al. 2018). In addition, many guideline studies generally quantify nonspecific apical endpoints (gross neurological abnormalities, behavioral changes, physical development, pathology), and many potential neurotoxicants may not be revealed by current testing strategies (Vorhees, Sprowles et al. 2018). It is estimated that there are over 65,000 substances of interest for toxicity testing, with minimal hazard assessments for less than 20% and full hazard assessments possible for less than 2% (Crofton, Mundy et al. 2012). Given the current number of chemicals with little or no DNT data, alternative testing methods are being used to supplement traditional toxicology testing so that more information can be gathered about data-poor chemicals in a rapid and efficient manner to allow for prioritization (Crofton, Mundy et al. 2011, Tsuji and Crofton 2012, Fritsche, Crofton et al. 2017, Behl, Ryan et al. 2019).

There are usually a smaller number of human and animal toxicological studies available relative to the number of in-vitro studies available for use in risk assessments. This is largely attributable to the relatively low cost and often rapid throughput of many mechanistic assays. In some instances, semi-automated techniques for data collection and analysis have been developed to screen rapidly for signals indicative of potential toxicity. In addition to cell- and molecularbased assays, non-mammalian animal models (e.g., Danio rerio, Drosophila melanogaster, Caenorhabditis elegans) have elucidated toxicity pathways in vivo, while still retaining many of the advantages of in vitro testing methods (Peterson, Nass et al. 2008). Although these alternative test models cannot definitively determine the risk of toxicity of a chemical to humans, they offer a bridge between in vitro screening batteries and mammalian toxicology studies. They also can facilitate chemical prioritization by identifying which inadequately tested or untested chemicals currently in the marketplace act via mechanisms that are frequently associated with toxic responses in humans (Bal-Price, Hogberg et al. 2010, Behl, Hsieh et al. 2015).

Mechanistic Data from High-throughput Screening Assays for Developmental Neurotoxicity

High throughput screening (HTS) and high content screening (HCS) assays are part of strategic approach to toxicity testing. These assays may include automation and robotics to facilitate screening larger numbers of compounds with reduced time. In vitro assays are being used as screening methods to identify potential neurotoxicants, and prioritize them for further in vivo testing. Brain development is a highly complex process that occurs well into early adulthood in humans, and there are many mechanisms that could lead to adverse effects on the nervous system. Thus, in vitro assays that can inform multiple mechanisms for chemical perturbation of neurodevelopment are essential. To enhance the utility of in vitro assays, many are based on suspected key events using the adverse outcome pathway (AOP) framework to contextualize generalized developmental neurotoxicity (Aschner, Ceccatelli et al. 2017). AOPs are structured representations of how a chemical exposure affects a biological system at the organ, tissue, and cellular level, that are linearly linked to an adverse effect (such as DNT). The AOP framework is considered relevant to risk assessment (Ankley, Bennett et al. 2010, Villeneuve, Crump et al. 2014, Villeneuve, Crump et al. 2014, Kleinstreuer, Sullivan et al. 2016), and notably, is chemically agnostic. AOPs begin with a chemical action, and this is defined as the molecular initiating event (MIE; often representing signaling pathways known to be involved in development and neurodevelopment) which lead to changes in key events at the cellular and organ level, ultimately resulting in (or contributing towards) an adverse outcome (e.g., reduced learning ability, cognition, and memory). An ideal DNT assay encompasses several critical factors, including representation of a dynamic stage of development (represented by neurodevelopmental processes) and an endpoint that can be measured in vivo and in vitro (Lein, Silbergeld et al. 2005, Bal-Price, Hogberg et al. 2010, Bal-Price, Crofton et al. 2015, Bal-Price, Lein et al. 2017). Key events in an AOP can theoretically be defined based on the known biological development of a specific neurodevelopmental process that is known to be part of an apical neurotoxic outcome. These key events can be used as an organizing principle for mechanistic data across levels of biological organization (Figure 1).

Figure 1:

Figure 1:

Schematic of an adverse outcome pathway from the molecular initiating event to the adverse outcome that occurs at the whole organism level (Ankley, Bennett et al. 2010).

Traditionally, HTS uses “target-based” biochemical assays due to the ability to transfer the assay methodology to automated, HTS platforms. In terms of an organizing AOP framework (Table 1), the target would refer to a MIE relevant to neurodevelopment and include known sites of action of neurotoxicants such as ion channels, enzymes, neurotransmitter binding sites, growth factor receptors, transcription factors, and kinases (Bal-Price, Hogberg et al. 2010, Bal-Price, Crofton et al. 2015, Mundy, Padilla et al. 2015, Aschner, Ceccatelli et al. 2017, Bal-Price, Lein et al. 2017). More recently, due to advances in technology, “phenotypic screening” assays (reviewed in (Fritsche 2017, Fritsche, Crofton et al. 2017, Bal-Price, Hogberg et al. 2018) have been developed that can rapidly assess chemical-induced changes at higher levels of biology including cell- and tissue-based assays of morphology (high-content imaging) (Ryan, Sirenko et al. 2016), electrophysiology (microelectrode arrays) (Wallace, Strickland et al. 2015, Brown, Hall et al. 2016, Cotterill, Hall et al. 2016, Frank, Brown et al. 2017) and growth and behavior of small alternative species such as zebrafish (Sipes, Padilla et al. 2011, Miller, Chandrasekaran et al. 2018). Depending on the assay, some of these may be “high content” assays, rather than “high throughput” assays, but there have been substantial advances to automate phenotypic screens and make them higher throughput. Phenotypic screening assays cast a wide net since they do not require knowledge of the specific chemical action, but instead look at events that represent mechanistic data at the cellular, organ and organism level. Phenotypic screening has led to the direct assessment of the neurodevelopmental processes contributing to brain development at the cellular level, including neural stem cell proliferation (Breier, Radio et al. 2008, Fritsche, Barenys et al. 2018), differentiation and migration (Visan, Hayess et al. 2012), apoptosis (Druwe, Freudenrich et al. 2015, Harrill, Freudenrich et al. 2018), neurite outgrowth (Druwe, Freudenrich et al. 2016, Ryan, Sirenko et al. 2016, Harrill, Freudenrich et al. 2018), myelination (Schmidt, Lehmann et al. 2017), synapse formation (Harrill, Robinette et al. 2011), and formation of a simple functioning network (Wallace, Strickland et al. 2015, Brown, Hall et al. 2016, Cotterill, Hall et al. 2016, Frank, Brown et al. 2017). Because the processes are measured in intact, viable cells, these assays integrate perturbations in upstream signaling with downstream effects. In addition, there are known neurotoxicants that have been shown to affect these neurodevelopmental processes both in vitro and in vivo, identifying them as potential key events in an AOP for developmental neurotoxicity (Mundy, Padilla et al. 2015, Aschner, Ceccatelli et al. 2017). Thus, neurodevelopmental processes can provide a useful organizing principle for evaluating mechanistic data in terms of the relationships between effects at different levels of biology (altered receptor binding during synaptogenesis can alter cell cycle, ultimately leading to changes in synaptogenesis and decreased neuronal network function and impaired learning/memory) including apical effects currently used as indicators of developmental neurotoxicity. However, it is recognized that cell-based assays lack the complexity of whole animal models and have some important limitations (not representative of pregnancy, lack of maternal/fetal metabolism, differences in cell types, low level of complexity, representative of limited biological stages and/or sex, and differences and/or lack of the blood and cerebrospinal fluid barriers). Overall, these strategies hold promise to enhance toxicity testing as most were designed to recapitulate aspects of in vivo neurodevelopment, but further validation studies are needed to demonstrate that they are an accurate predictor of DNT in developing mammals, and that the chemical effects on key events identified in vitro can also be recapitulated in an intact system.

Table 1:

Perturbation of key neurodevelopmental events results in developmental neurotoxicity when using model chemicals.

Key Event Consequence of Disruption Model Chemical
Proliferation Incorrect cell number Ethanol (Miller 1986)
Migration Abnormal cell position MeHg (Guo, Yan et al. 2013)
Differentiation Change in cell identity Ethanol (Tingling, Bake et al. 2013)
Neurite Growth Abnormal connections Cocaine (Jones, Fischer et al. 1996)
Synaptogenesis Abnormal connections Ethanol (Inomata, Nasu et al. 1987)
Apoptosis Incorrect cell number Ketamine (Huang, Zhang et al. 2014)

Evaluation of the utility and relevance of mechanistic data from cell-based screening for DNT can be based on two factors: technical understanding of assay characteristics and the biological context of the model system used. Since high-throughput assays are used to generate data on thousands of chemicals, they can be useful for prioritization of chemicals for further evaluation (detecting chemicals that may be neurotoxic). Ideally, these assays are designed to be highly replicable and robust measures for a specific endpoint to increase confidence in the generated data (Fritsche, Alm et al. 2015, Fritsche, Crofton et al. 2017, Fritsche, Grandjean et al. 2018). The cellular models used should exhibit the endpoint of interest (e.g., neurite outgrowth) over a finite developmental period in vitro, and the ability to quantify this endpoint using high-throughput technology should be demonstrated. This proof-of-principle testing is typically done using an “endpoint-specific” positive control chemical (previously shown to selectively perturb the endpoint), which should elicit a reproducible response at a given concentration (Mundy, Padilla et al. 2015, Aschner, Ceccatelli et al. 2017). The positive control can also be used to determine the appropriate metrics for assessing whether a chemical is “active” in the assay and provide a potency value. Evaluation of sensitivity/applicability of an assay is also based on the use of a “training set” of known positive reference chemicals that have previously been shown to alter the key event of interest in vitro (Mundy, Padilla et al. 2015, Aschner, Ceccatelli et al. 2017). Negative reference controls that have no known effects on the endpoint should also be evaluated. The training set of reference chemicals helps to validate an assay and limit the likelihood of false positives and negatives, thereby increasing confidence in assay results. This is particularly relevant because not all chemicals work well for testing in cellular models (i.e., chemicals with a high volatility, insolubility in culture media, or sorption to plastic culture dishes). Finally, the cell-based assays should include multiple, simultaneous measures of cell viability/cytotoxicity to assess whether the chemical exposure is producing a confounding observation of non-specific effects on cell health. Evaluation of mechanistic data from cell- and tissue-based assays should also consider the biological context of the model system used, including the source of the tissue or cells (e.g., CNS region, developmental stage, sex), complexity of the culture, and the neurodevelopmental processes that are manifested in vitro. Demonstration and understanding of the biological context of the in vitro system used to assess a specific neurodevelopmental process will increase confidence in the data obtained. Of note, aspects of CNS developmental maturation can differ across systems (e.g., depending on the cellular composition and other features of the model), including differences in functional capacity/capabilities (Seongeun, Andrew et al. 2007, Clarke and Barres 2013, Belle, Enright et al. 2018); this can complicate in vitro to in vivo extrapolations (Wilk-Zasadna, Bernasconi et al. 2015).

There are multiple types of cellular and tissue culture models that can be utilized to study neurodevelopment, including transformed cell lines, primary cell cultures (derived directly from nervous system tissue) and neural stem cells (neural stem cells/progenitor cells, induced pluripotent stem cells) (Druwe, Freudenrich et al. 2015, Druwe et al. 2016). Cell lines have been extensively used in screening assays and mechanistic studies due to ease of culture and availability, but they may not be physiologically relevant since they have been transformed and immortalized or are derived from diseased tissues like cancer tumors. Primary cultures derived directly from rodent brain regions of interest are widely used and may be preferred since they are “true” neural cells that maintain some regional identity (e.g., hippocampal vs neocortical). However, they must be freshly prepared from developing brain tissue. and equivalent Human primary cells are not widely available. However, there are human induced pluripotent stem cells (IPSC) that are commercially available in limited quantity (although with potential restrictions depending on ethical concerns and country specific legislation) (Tukker, de Groot et al. 2016). Yet, they may be less well characterized and more variable than rodent models. Neural stem cells are a promising model of DNT in vitro. They can be derived from rodent or human embryonic or adult-induced pluripotent stem cells, are commercially available, and proliferate readily in vitro. Human derivation increases their physiological relevance (e.g., they possess human proteins and receptors), and cells are clearly sourced from male or female tissue. They can be induced to differentiate (using variable methods) and proliferate into neurons and glia, but regional identity must be carefully characterized. Regardless of the source, the complexity of in vitro neural cultures can vary and range from cultures of a single cell type (e.g., a human neuroprogenitor cell line) to cultures containing multiple cell types (e.g., co-cultures of primary rodent neurons and astrocytes). A single neuroprogenitor cell type may be appropriate for measurement of chemical effects on proliferation, while co-cultures of neurons and glia would be required to provide the necessary conditions to accurately model synaptogenesis in vitro. The “appropriate” complexity will depend upon what is being measured. In general, more complex cultures consisting of multiple cell types growing for long periods of time will exhibit neurodevelopmental processes that are more analogous to those observed in vivo, even up to the formation of neural synapses in vitro, that display electrical activity (Cotterill, Hall et al. 2016). Neurodevelopmental assays utilizing cellular models need to characterize fully the composition of the cultures to limit the introduction of non-targeted toxicity (such as erroneous cell types being present and counted/imaged whenever an assay is targeting effects on a neuron with a specific NT phenotype). It is important to note that induced pluripotent stem cells are driven by growth factor cocktails to different phenotypes, and that the longer culture time usually requires an absence of antibiotics in the culture media which increases the risk of contamination of cultures. Furthermore, only a few neuronal cell types can be successfully grown in culture, and so there is limited ability to test the full diversity of brain cell types.

The development and use of high-throughput assays for developmental neurotoxicity was predicated on the action of a chemical at a molecular target which leads to a sequence of key events at increasing levels of biological organization, ultimately resulting in adverse effects in an individual. Thus, the standardized presentation of the relationship between key events in the form of an AOP provides a useful framework by which mechanistic data at different levels of biology can be organized. Cell-based assays for key neurodevelopmental processes have been particularly useful since they provide a test system (live cell) which can integrate molecular effects with changes in morphology and function. The key processes of neurodevelopment (proliferation, differentiation and migration, apoptosis, growth/synaptogenesis, myelination) represent dynamic stages that are altered by toxicants and can be measured in vitro and in vivo. Figure 2 illustrates an example in which specific molecular initiating events can ultimately elicit neurodevelopmental events and result in adverse effects on brain morphology, neurochemistry, electrophysiology, or behavior. In this way they can provide an organizing principle by which mechanistic data for neurotoxicants can be evaluated and can be part of a systematic approach to synthesize and classify data on molecular targets, signaling pathways and disease processes related to developmental neurotoxicity. It is important to note that there are still research opportunities and data gaps for development of models to recapitulate other aspects of brain development that are not represented in the current testing battery.

Figure 2:

Figure 2:

A potential example framework showing how relationships between mechanistic data from different sources can be organized. Mechanistic data relevant to neurodevelopment are mapped using the adverse outcome pathway construct. In this example, the adverse effects are a change in layer thickness, and the original molecular initiating event (MIE) is an effect on cyclin dependent kinase (CDK). There are many potential MIEs, pathways, events, and adverse effects that could populate this framework. (, ROS=reactive oxygen species, AChE= acetylcholinesterase, VGCC = voltage gated calcium channel,, Learning & Memory (L/M), Neurotransmitter (NT), electroencephalogram (EEG), long term potentiation (LTP,), GO =gene ontology KEEG =Kyoto Encyclopedia of Genes and Genomes).

“Key Characteristics” as an alternative organization framework for DNT mechanistic data

To develop ways to organize mechanistic data on developmental neurotoxicants for evaluation and use in human health assessments, valuable insight can be gained by exploring how mechanistic data has been categorized for carcinogens. For example, Hanahan and Weinberg (Hanahan and Weinberg 2000, Hanahan and Weinberg 2011) have proposed features of cancer that can be used as a framework for understanding the complex mechanisms involved in the development of cancer. More recently, Smith et al. developed a process for organizing mechanistic data on relevance to carcinogenesis based on key characteristics (Smith, Guyton et al. 2016). This was done as part of an effort to develop a systematic way to evaluate mechanistic data used to support human health assessments of possible carcinogens. Smith et al. (2016) noted that at least one of characteristic needs to be present for carcinogenesis to occur (carcinogens often possess many of the ten characteristics) and that the key characteristics can include different mechanistic endpoints, but that they are not themselves mechanisms or AOPs (Smith, Guyton et al. 2016).

Using the key characteristics for carcinogenesis, Smith et al. (2016) were able to identify relevant mechanistic information through a literature search and screen and organize the results of the literature search to facilitate the syntheses of mechanistic information within collections of interconnected AOPs, or Adverse-Outcomes Networks (AON). A similar process can be envisioned for mechanistic data on chemicals that are developmental neurotoxicants. Although this may not be as straight forward as it is for carcinogens, it is known that there are many characteristics that may be inherent to developmental neurotoxicants. For example, some can cause increased or decreased apoptosis, disrupt normal cell migration, produce oxidative stress, or disrupt long-term potentiation to name but a few possible characteristics. In fact, key characteristics have also been developed for endocrine disrupting chemicals (EDCs) (La Merrill, Vandenberg et al. 2019), male reproductive toxicants (Arzuaga, Smith et al. 2019) and female reproductive toxicants (Luderer, Eskenazi et al. 2019), in support of organizing mechanistic studies for evaluations of endocrine and reproductive toxicity. There is an ongoing effort for development of key characteristics of developmental neurotoxicants (Ahearn 2019).

National Research Council Report on Developmental Toxicity and Risk Assessment: Signaling Pathways as an Organizational Framework

Development is a process of complex gene regulation controlled by both time (developmental stage) and space (region of interest). From a single genome, thousands of different gene combinations must be expressed at specific times and places in the developing organism, and, from the developing egg, the information for the selective uses of combinations must be generated. A major component of this regulation is the transfer of chemical information (i.e., signals) between cells during development. These intercellular signals and their responses occur during all stages of development. Cellular responses to the signals are governed by both the genotype and by the previous history of cell responses.

By the turn of the 21st century, decades of scientific research in the fields of biology, embryology, biochemistry, and genetics led to the identification of 17 intercellular signaling pathways commonly used during development (Gerhart 1999, NRC 2000). The 17 signaling pathways identified by the 2000 NRC report (Table 2) are shared by most animals, explaining in part the array of developmental components and processes that are conserved across diverse phyla, and they are identified by their transduction intermediates. These pathways are important to a range of developmental processes, including organogenesis, cytodifferentiation, growth and tissue renewal, maintenance of homeostasis, and even response to injury. Although the 2000 report categorized these pathways into different stages of development, research since indicates that many of these pathways are active and essential throughout development, and even into adulthood (Tiemeier, Lenroot et al. 2010, Álvarez-Buylla and Ihrie 2014, O’Shaughnessy, Thomas et al. 2019, Sathyanesan, Zhou et al. 2019). Even so, research on the mechanistic basis of developmental biology and toxicology has been successful in the past through framing analyses based on disruptions in signaling pathways (Pires-daSilva and Sommer 2003). Thus, organizing systematic DNT evaluations around signaling pathways important to brain development and maturation (including the 17 pathways noted in the NRC report as well as others identified through ongoing research) can help focus analyses, although opportunities exist to fill data gaps and better refine such an approach (Antebi, Nandagopal et al. 2017).

Table 2:

List of signaling pathways that are active across development (NRC 2000).

Signaling Pathways Active Across Development (NRC 2000)
Wingless-Int
Transforming Growth Factor Beta (receptor serine and threonine kinase)
Hedgehog
Receptor tyrosine kinase (small G protein)
Notch-Delta
Cytokine receptor (cytoplasmic tyrosine kinases; STAT pathway)
Interleukin-1-Toll Nuclear Factor Kappa Beta
Nuclear hormone receptor
Apoptosis
Receptor phosphotyrosine phosphatase
Receptor guanylate cyclase
Nitric oxide receptor
G-protein couples receptor (large G proteins)
Integrin
Cadherin
Gap Junction
Ligand-gated Cation channel

These principles have also been applied to the exploration of the mechanisms of normal and disrupted neurological development. There is a plethora of studies on the role of specific developmental signaling pathways in the ontogeny of the nervous system. Some examples include Wnt (Arenas 2014), transforming growth factor β (Liu and Niswander 2005), hedgehog (Gulino, Di Marcotullio et al. 2007), notch signaling (Lasky and Wu 2005, Louvi and ArtavanisTsakonas 2006) cytokine signaling (Mousa and Bakhiet 2013), glycogen synthase kinase (Hur and Zhou 2010), and a serine/threonine kinase termed mTOR (Lee 2015). Additional signaling pathways have also been identified that are active in neurological development (Fritsche, Crofton et al. 2017). An example is the insulin/insulin-like growth factor (IGF)-signaling cascade that activates two other major signaling pathways: P13K (a lipid kinase) and RAS/Mitogen-activated protein kinase (MAPK) (reviewed by (Vogel 2013)). Characterizing the complexities of such signaling cascades will be useful in developing the key events in AOPs, which have been recognized as a critical aspect of the future of developmental toxicity testing (DeSesso 2017). However, as these pathways are redundant across development and across species, careful work needs to be completed in vivo to delineate how, and when, these pathways are perturbed by a chemical exposure and, hence, lead to DNT. These in vivo data can then be compared to HTS assays, in order to determine whether these tests recapitulate biology, and if not, how to design better testing strategies for DNT. Many of these potential targeted pathways are summarized in the OECD/EFSA workshop report that details non-animal test methods for DNT for regulatory purposes (Fritsche 2017).

Bioinformatic Pathway Analysis for DNT

Recent efforts to incorporate structural frameworks into the systematic review of neurodevelopmental endpoints have also included a focus on bioinformatic pathway analyses. Several factors need to be considered when identifying the biological pathways that are hypothesized to result in adverse neurodevelopmental endpoints. As many gene regulatory networks are evolutionarily conserved, identifying commonalities across model systems is a feasible strategy. Incorporating a combination of genetic studies derived from human (case studies, epidemiologic, familial genetic linkage) and murine studies (comparative mouse studies, knockout models, genetic linkage), Boyles and coauthors were able to identify neural tube defect (NTD) candidate genes previously proposed to be associated with increased NTD incidence (Boyles, Hammock et al. 2005). In addition, environmental toxicogenomic studies focused on assessing environmentally induced impacts (e.g., cadmium or methylmercury) during specific developmental windows (e.g., neurulation) were incorporated for an integrated systems-based approach (Robinson, Yu et al. 2009, Robinson, Yu et al. 2010). The interface between expression of NTD candidate genes and the environment can also be evaluated using toxicogenomic-based assessments which analyze thousands of genes simultaneously. Robinson et al., (2010) constructed a subset of NTD candidate genes across mice and humans ((Boyles, Hammock et al. 2005, Harris and Juriloff 2007). They integrated those findings with their previous research (Robinson, Yu et al. 2009, Robinson, Yu et al. 2010) to evaluate NTD candidate gene expressed across strain, time and dose. This effort facilitated evaluation of the complex interactions of gene and environment in the context of exposure, response and susceptibility factors, which is critically important for neurodevelopmental risk assessments. Candidate genes were further characterized by NTD phenotype and grouped according to gene ontology (GO) classification with the DAVID Bioinformatics Database 6 (Dennis, Sherman et al. 2003).

Bioinformatic pathway analyses revealed that many of the NTD genes are involved in developmental-related processes (organ, nervous system, neural tube formation), embryonic morphogenesis (organ, embryonic, tube) as well as in transcription-related processes. In addition, several candidate genes on the list represent multiple developmental signaling pathways (e.g., hedgehog and Wnt signaling), MAPK signaling and cell proliferation/apoptosis regulation.

While these data demonstrate the potential of bioinformatic tools to describe genes of interest, limitations were also apparent. For instance, while there is evidence supporting developmental roles in neural tube morphogenesis for the majority of candidate genes, only a subset were linked with the GO term “neural tube development”. This emphasizes the need for additional database-derived approaches to analyze gene subsets. One such platform that was presented at the 2016 symposium on is the Comparative Toxicogenomics Database (CTD; http://ctdbase.org/), an open-source platform that links toxicant exposures to disease endpoints (Davis, Grondin et al. 2019). Some of the example pathways common to neurodevelopmental toxicant exposure and related neurodevelopmental disease endpoints identified using CTD include Wnt and MAPK signaling pathways and apoptosis. With the endpoints identified, the potential impacts across lifestage can be assessed within a risk assessment framework. Indeed, children are known to be particularly vulnerable to toxic exposures which have the potential to adversely impact normal neurodevelopmental trajectories.

Since the 2016 symposium, more recent approaches have been initiated to optimize hazard assessment for human developmental neurotoxicity utilizing animal-free ontology driven testing strategies (Baker, Boobis et al. 2018, Hessel, Staal et al. 2018). Given the complexity of the human brain, animal studies are limited in their predictive value (construct validity) to human neurodevelopmental disorders (e.g., autism, attention-deficit hyperactivity disorder). Alternative methods are being developed to not only understand human relevance of animal toxicity data but also to support the reduction and replacement of animal testing (Russell and Burch 1959).

Indeed, the EPA announced a directive in 2019 to reduce animal testing 30% by 2025 (EPA 2019). Ontology approaches can provide a formal framework for organizing knowledge of chemical effects in a biological network that can, in turn, lead to modeled predictions of toxicological outcomes. Hessel et al., provided an overview of human brain development and overlaid proposed DNT test batteries based on biological processes and their respective developmental timeframes. They also summarized the variable degree to which each of these assays are developed, standardized and validated (Hessel, Staal et al. 2018). In addition, a transcriptome comparison across time matched neural progenitor cells of different species (human, mouse, and rat) identifies unique gene expression patterns, but with clustering in similar GO terms like cell migration, gliogenesis, and neurogenesis (Masjosthusmann, Becker et al. 2018). In many cases, an ontology would encompass quantitative AOPs.

Several examples of developmental toxicity in silico models have been developed for specific individual developmental processes (Kleinstreuer, Dix et al. 2013, Leung, Hutson et al. 2016, Hutson, Leung et al. 2017). While no particular in silico models for neurodevelopmental processes have been developed thus far, using approaches to mine available data from wide research areas of neuroscience and toxicology will help risk assessors move beyond single assays or batteries of assays to design “physiology driven software models of embryo neurodevelopment”(Hessel, Staal et al. 2018).

There is also a need for inclusion of toxicokinetic and toxicodynamic information into a broader framework for evaluating neurodevelopmental risks (Robinson, Port et al. 2010). Since both toxicokinetics and toxicodynamics play important roles in defining toxicant response in adults, it follows that these data be considered when defining developmental response. In this context, several improvements in quantitative neurodevelopmental risk assessment can occur, including biologically based extrapolations within/across species and systems as well as across compounds and databases to identify windows of susceptibility (Faustman, Gohlke et al. 2005).

In summary, conserved gene regulatory networks were identified in a case example using environmental exposures. This study illustrates that bioinformatic analyses across species could be used to identify convergent developmental pathways that are a highly correlated to DNT. Consequently, this workflow could be used to identify toxicants that impact these conserved pathways, and the affected mechanism(s). This framework would address what level of exposure and what time point(s) impacts are observed. Finally, taking advantage of bioinformatic tools to build gene by environment analyses could identify contributions within and between organisms, to define common teratogenic responses.

Behavior as a primary organizing principle for the systematic evaluation of mechanistic data for developmental neurotoxicity outcomes: A Case Example using Lead Toxicity

A central tenet of neuroscience is that, ultimately, behavior reflects nervous system function. Complex behaviors have a long and complex developmental trajectory, and the way changes in behavior can be manifest is often highly variable and adaptive, depending on the situational context. To add further complexity, behavioral development has a bidirectional relationship with underlying mechanisms in that cellular and molecular changes can alter behavior, and behavioral changes can lead to in shifts in the underlying cellular and molecular processes (Hoffman, Hornig et al. 2004, Fernald, Neufeld et al. 2006). To begin to use behavioral changes to potentially organize toxicant-induced mechanistic changes, one must identify quantified behavioral outcome(s) at key developmental stages, and then determine what developmental changes or disruptions have been associated with these phenotypic outcomes. If this is accomplished, behavior may be used to guide the model of key mechanistic changes at sequential developmental stages (reviewed in (Sobin and Golub 2018)).

As a simple case example, rearing behavior in a novel environment as a measure of memory (i.e., as compared to rearing behavior in a familiar environment) in lead-exposed mice at pre-adolescence can be used to organize understanding of possible pathways and mechanisms associated with the effects of early-life lead exposure. The primary goal for a behavioral model of early life low-level lead exposure (blood lead level 3.0 – 5.0 μg/dL) is to identify behavioral tests that are sensitive in young or pre-adolescent animals. Rearing is thought to be largely controlled via hippocampal cholinergic transmission. Elevated levels of hippocampus acetylcholine (ACh) have been associated with rearing in a novel environment, both from studies in rats exploring home versus novel environments (Thiel, Huston et al. 1998), and through studies evaluating modulation of cholinergic activity from opioids (agonists (Van Abeelen and van Nies 1983) and antagonists (Van Abeelen, Ellenbroek et al. 1975)). Memory is also an ideal behavioral endpoint to focus on as it has been shown to be disrupted in preadolescent children with early chronic low-level lead exposure (Bellinger, Leviton et al. 1991, Lanphear, Dietrich et al. 2000, Sobin, Flores-Montoya et al. 2015). Additionally, memory deficits are often associated with mechanistic changes in the hippocampus/dentate gyrus (DG), and chronic lead exposure has been shown to alter this and other regions of the brain (Gilbert, Mack et al. 1996, Ruan, Chen et al. 1998, Moreira, Vassilieff et al. 2001, Gilbert, Kelly et al. 2005). Changes in the DG could potentially lead to short- and long-term impacts since it is the center for learning and memory and contributes to neurogenesis throughout the lifespan (Jessberger, Clark et al. 2009, Aimone, Deng et al. 2011). Importantly, lead can also act on other neurological systems (stress and hypothalamic-pituitary-adrenal axis functioning, neurotransmitter release) and has varied neurotoxicological effects (Cory-Slechta 1995, Cory-Slechta, Crofton et al. 2001, White, CorySlechta et al. 2007).

Another example of a behavioral effect that can be used to organize mechanistic data, is the Object-in-Place Task which assesses spatial and object visual memory retrieval in a single paradigm. The object-in-place task consists of an acclimation period, learning trials, and pretest and test trials including tests of spatial memory and object memory (De Viti, Martino et al. 2010, Sobin, Flores-Montoya et al. 2017). Primary effects of low-level lead exposure included differences in horizontal exploration and vertical exploration. Decreases in spatial memory and greater object memory have also been observed (Sobin, Flores-Montoya et al. 2017). The same pattern of associations was observed for global rearing following lead exposure (Sobin, FloresMontoya et al. 2017). As stated above, several studies have demonstrated that lead disrupts cholinergic transmission. For example, in rat pups with lead exposure from postnatal day (PND) 7 to PND 28, there was selective reduction (35%) in cholinergic activity in septal nuclei and hippocampi (Bielarczyk, Tomsig et al. 1994). In another study, perinatal lead exposure produced loss of cholinergic projections to the hippocampus and decreased cholinergic innervation in rat neonates, and this deficit persisted into adulthood (Bourjeily and Suszkiw 1997). It has also been shown that lead exposure decreases the breakdown of acetylcholine (Ach) and increases acetylcholinesterase (AChE) in the hippocampus (Reddy, Basha et al. 2003).

Many studies have shown that disruption of the cholinergic system also impacts dopaminergic and glutamatergic systems. In the hippocampus, ACh receptors are expressed in over 90% of all GABAergic neurons (Van der Zee and Luiten 1993). Synaptic plasticity in the hippocampus is directly influenced by cholinergic effects on glutamate transmission (Hasselmo). Cholinergic interneurons have also been identified in the hippocampus (Frotscher, Vida et al. 2000). Lead alters muscarinic modulation of glutamatergic transmission (Wang, Luo et al. 2007). ACh has primary and complex effects on synaptic plasticity in the hippocampus (Drever, Riedel et al. 2011). Finally, specifically in the hippocampal glutamatergic system, ACh functions as a neuromodulator, altering change in “state” of neurons acting via “volume transmission” (Picciotto, Higley et al. 2012).

In this brief example, abnormal rearing behavior in pre-adolescent mice with chronic developmental lead exposure was used to organize complex mechanistic data (memory-dependent rearing behaviors linked to disrupted hippocampal cholinergic transmission, leading to altered GABAergic and glutamatergic neurotransmission in the hippocampus). In this example, a single behavioral observation was utilized to identify plausible underlying mechanistic changes that are known to be associated with that behavior and then led to the identification of other potential mechanistic changes hypothesized to be linked to the changes. Many tested behaviors associated with other brain mechanisms and pathways have not been shown to be disrupted in young lead exposed mice and it is important to incorporate knowledge of inconsistent or negative findings in new models..

Behavioral changes that are quantified at multiple stages of development can be used to organize the complex mechanistic effects of potential developmental neurotoxicants. The example provided was very simple, using one behavioral effect occurring at only one point in time (pre-adolescence). For the purposes of using behavior to organize understanding of toxicant effects on pathways and mechanisms, it is important to consider that behavior can be used to reveal at least four qualitatively different types of mechanistic effects. These include initial alterations in mechanistic functions in the absence of obvious damage; periods of exceptional mechanistic vulnerability in brain systems (“developmental windows”); cumulative effects within a single behavioral domain at different developmental stages; and shifting effects across behavioral domains during development from shifting disruption of underlying mechanisms.

Challenges with Neurotoxicity Evaluation Framework Implementation

Nervous system development involves the execution of very intricate, well-orchestrated events including the appropriate differentiation of glia and neurons, tangential, and radial cell migration, programmed cell death, synapse formation, myelination and neural circuit establishment. These events (amongst others) occur across defined, and potentially overlapping developmental periods which differ according to brain compartment, and many rely on cues from surrounding microenvironment (e.g., CNS region). Toxicant exposure during discrete periods of vulnerability can detrimentally shift or disrupt normal developmental trajectories, leading to life-long consequences in function. Moreover, chemical exposures occur within the context of a myriad of other modifying factors that are unique on an individual level (i.e., socioeconomic status, age, stress, nutrition, as well as health status; Figure 3), and these variables must be considered when identifying susceptible populations.

Figure 3:

Figure 3:

The blue circle indicates an individual and the arrows indicate the variety of variables and risk factors that are unique to each individual and can influence or modify an individual’s response to a developmental neurotoxicant. These many variables and factors emphasize that the environmental and biological context of an exposure is important when considering its impact. All these risk factors collectively influence an individual’s overall health status.

As discussed earlier, developmental timing is a major influential component of exposure context and is interrelated to pregnancy itself. There are complex (and often overlapping) events between the development of the brain and immune system from gestation to early postnatal life in humans. Due in part to these overlapping processes, maternal inflammation during pregnancy can contribute to neuronal dysfunction and altered behavioral phenotypes in the offspring (Hava, Vered et al. 2006, Spann, Monk et al. 2018). Emergent evidence suggests that maternal immune system activation during specific periods of neural development is a common risk factor for several CNS disorders, including schizophrenia, autism and epilepsy (Knuesel, Chicha et al. 2014, Estes and McAllister 2016). The impact of short- and long-term maternal immune activation are dependent on genetic predisposition, sex, specific window of brain development, and the specific insult (Howerton and Bale 2012, Knuesel, Chicha et al. 2014). However, immune activation during pregnancy is not always adverse (Bilbo and Schwarz 2009), as cytokines are critically involved in many important brain development processes (including neuronal/glial cell migration, differentiation, and synaptic maturation). To further underscore the role of context, levels of microglial activation (measured by the authors, as “reactive or amoeboid morphology”) during neurodevelopment are dependent not only on developmental timing but also on sex of the child. Male rats were shown to have significantly more activated microglia early in postnatal development (postnatal day 4), while females have more microglia with activated morphology later in development, as juveniles and adults (postnatal day 30–60) (Schwarz, Sholar et al. 2012).

Discussion

Frameworks for describing mechanistic DNT data present challenges. Scientists are developing AOPs to help illustrate the relationship between early life mechanistic events and later life adverse health effects. To date, the number of AOPs in the AOP wiki related specifically to DNT are limited, and mostly focused on those related to thyroid hormone disruption (https://aopwiki.org/aops). This limited number of AOPs reflects the complex nature of nervous system development. One of the key issues is a determination of when mechanistic changes are adverse, since the context and environment of an exposure can influence responses. A change in brain morphology or behavior related to exposure is not necessarily indicative of an adverse phenotype within the context of the organism. For example, a shift toward a more anxiogenic neuroendocrine state or a modulation of reproductive success may serve as an advantage for a species living within a high threat environment, promoting survival and the propagation of genes (Cameron, Champagne et al. 2005). Recent discussions regarding the role of sex differences in the brain also provide useful insights. Rather than being causal in predicting behavioral differences between males and females, these neurobiological features may function to diminish sex-differences that are attributable to system-wide organ differences between males and females (de Vries and Forger 2015).

One of the common criticisms of using DNT AOPs organized by neurodevelopmental process is that complexity increases greatly when trying to integrate modifying factors (such as age, epigenetics, and regional specificity). Individual differences in response to exposures is the rule rather than the exception and there are a vast range of modifying factors that will influence exposure-induced effects. Within an individual, there will be variability in the neurodevelopmental impact observed in response to exposure, depending on the age at which assessment occurs and the specific neurological processes being analyzed. Behavioral output can also modulate these outcomes. Physical activity, exposure to anxiogenic environments, and social encounters may be used to evaluate behavioral phenotypes resulting from exposure and can impact the function of neural systems, including neuroplasticity, serotonergic activity and the functioning of the hypothalamic-pituitary-adrenal (HPA) axis. Collectively, these factors will increase the complexity of outcome measures and create significant challenges to categorizing data from existing and future studies.

As has previously been suggested (Council 2009), one strategy to begin to discern how exposure context and modifying factors modulate the response to exposure is to consider contexts/factors that share upstream or downstream mechanisms with a neurotoxicant, without which interactions would not be expected. For example, lead exposure and stress both impact the HPA axis as well as the mesocorticolimbic regions of brain (Cory-Slechta, O’Mara et al. 1998, Cory-Slechta, O’Mara et al. 1999, Berger, Barros et al. 2002, Barros, Berger et al. 2004, Martinez-Tellez, Hernandez-Torres et al. 2009, Rossi-George, Virgolini et al. 2011, Segal, Lin et al. 2015, Sobolewski, Conrad et al. 2018), a fact that likely explains why they have common adverse outcomes as well, including deficits in attention-related behaviors and cognition (Nigg, Knottnerus et al. 2008, Li, Olsen et al. 2010, Boucher, Jacobson et al. 2012). Given their common biological targeting, it could be expected that lead and stress would interact. Indeed, studies have demonstrated such interactions (Segal, Lin et al. 2015). The fact that most neurotoxic metals appear to have effects on glucocorticoid functions (Makino, Tanaka et al. 1996, Elez, Dundjerski et al. 2001, Brkljacic, Vojnovic Milutinovic et al. 2004, Spuches and Wilcox 2008) is consistent with the observations that other metals (e.g., methylmercury and arsenic) can likewise be shown to have neurotoxic consequences that are modified by prenatal stress (Sobolewski, Conrad et al. 2018).

Though DNT risk assessments primarily focus on neurobiological and behavioral outcomes, the interactions between the brain and other biological systems needs careful consideration (Segal, Lin et al. 2015). Within the field of psychoneuroimmunology, the reciprocal impact of stress and immune function on the brain has been well documented and there has been increasing focus on how disruption to any one of these systems can impact neurodevelopment (Knuesel, Chicha et al. 2014). The gut microbiome also impacts brain development (Heijtz, Wang et al. 2011) with the potential for reciprocal gut-brain interactions driving phenotypic variation (Catron, Keely et al. 2019, Catron, Swank et al. 2019). Similar to the framework being adopted for sex-differences in the brain (de Vries and Forger 2015), a whole body perspective on brain development may create novel avenues of research within DNT risk assessment where indirect effects on brain function consequent to systemic effects of exposure on other systems are integrated and the bi-directional pathways leading to long-term outcomes are more carefully considered.

Conclusions

An alternative DNT in vitro testing battery for regulatory purposes has been identified after a series of workshops, that includes in vitro and alternative assays for application of an Integrated Approach to Testing and Assessment (IATA) (Tollefsen, Scholz et al. 2014, Fritsche 2017, Fritsche, Crofton et al. 2017, Bal-Price, Hogberg et al. 2018). Guidance on how to use/interpret this battery are in progress.(Sachana, Bal-Price et al. 2018), and include “collation of available DNT methods and their scoring for readiness, selection of methods to form a DNT testing battery, the generation of a reference set of chemical that will be tested using the battery, and case studies exemplifying data interpretation, as well as the development of an OECD guidance document” (Sachana, Bal-Price et al. 2018). The OECD IATA project (http://www.oecd.org/chemicalsafety/risk-assessment/iata-integrated-approaches-to-testing-and-assessment.htm) that engages regulators, scientists, and stakeholders to try to increase new alternative methods (NAMs) in the regulatory arena by considering the decision context needs and uncertainty. Structured frameworks, such as those discussed in this publication, are essential for weighing different types of mechanistic across different levels of biological organization and methods (Tollefsen, Scholz et al. 2014). This approach represents a first step toward the incorporation of an understanding of the mechanistic basis of DNT into the regulatory process by using phenotypic screening assays.

Using frameworks to structure mechanistic data with developmental neurotoxicity data is a challenging task. There is a range of structured frameworks that encompass mechanistic data types, each differing in size and complexity. Screening and use of mechanistic DNT AOPs are still ongoing research areas for quantitative use in risk assessment, and there are a limited number of AOPs available for developmental neurotoxicity. Importantly, the influence of stress, social environment, sex, age, and other susceptibility factors can all influence the ability to detect effects on neurological function and outcomes (e.g. by altering the effective dose of potential toxicants), creating a unique challenge to incorporate so many complex factors in a given assessment. In vitro data possess inherent limitations, and even in vivo data can be challenging to encompass the full extent of real-world exposure scenarios. DNT data are diverse and complex, and framework implementation is a work in progress. Typically, DNT endpoints that are used in risk assessment are those representing morphological, pathological or functional changes in behavior. To better incorporate emerging and non-traditional data types into DNT risk assessment, there is a need for better in vitro to in vivo extrapolation (IVIVE), well described animal and human models, and AOP models that can consider dynamic changes and qualitative dose response in those various.tissues. The variety of modifying factors that can influence neurotoxicity make this an exciting and challenging area of research.

Supplementary Material

1

Highlights.

  • Mechanistic data from developmental neurotoxicity studies are abundant and heterogeneous

  • Risk assessments can utilize mechanistic DNT data for diverse uses

  • Potential organizing frameworks for evaluation of mechanistic data are discussed

  • Nervous System is complex, leading to challenges with framework implementation

Acknowledgements

The authors are grateful to Andrew Hotchkiss, John Vandenberg, Ingrid Druwe, Katherine, O’Shaughnessy, Tim Shafer, and Jennifer Nichols for conducting technical reviews of earlier drafts of this manuscript.

Funding Sources

Funding in support of the symposium held at the June 2016 DNTS annual meeting was provided by the U.S. Environmental Protection Agency, National Center for Environmental Assessment, order# EP-16-H-000250. This work is supported in part by an appointment to the Internship/Research Participation Program at Office of Research and Development (National Center for Environmental Assessment), U.S. Environmental Protection Agency, administered by the Oak Ridge Institute for Science and Education through an interagency agreement between the U.S. Department of Energy and EPA. The research that supported this work was made possible by grants from the National Institute of Child Health and Human Development (NICHD), National Institutes of Health, (R21HD060120, CS, PI); the National Center for Research Resources, a component of the National Institutes of Health (5G12RR008124); the National Institute of General Medical Sciences (SC1GM111172, CMV PI); the National Institute on Minority Health and Health Disparities Grant (212MD007592); the Center for Clinical and Translational Science, The Rockefeller University, New York, New York; the Paso del Norte Health Foundation, El Paso, Texas; the University Research Institute, The University of Texas, El Paso; and from the J. Edward and Helen M. C. Stern Professorship in Neuroscience, The University of Texas, El Paso (CS). The funders had no role in the design, implementation, data analyses, interpretation, manuscript preparation, or in the decision to submit the paper for publication.

Footnotes

Conflict of Interest: None of the authors have any financial or other conflicts of interest to declare.

Disclaimer: The views expressed in this manuscript are those of the authors and do not necessarily reflect the views or policies of the U.S. Environmental Protection Agency.

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

References

  1. Ahearn A. (2019). “Key Characteristics: A New Approach to Identifying Potential Toxicants, with Martyn Smith.” Podcasts: The Researcher’s Perspective 2019(1). [Google Scholar]
  2. Aimone JB, Deng W and Gage FH (2011). “Resolving new memories: a critical look at the dentate gyrus, adult neurogenesis, and pattern separation.” Neuron 70(4): 589–596. [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Álvarez-Buylla A and Ihrie RA (2014). Sonic hedgehog signaling in the postnatal brain Seminars in cell & developmental biology, Elsevier. [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Ankley GT, Bennett RS, Erickson RJ, Hoff DJ, Hornung MW, Johnson RD, Mount DR, Nichols JW, Russom CL, Schmieder PK, Serrrano JA, Tietge JE and Villeneuve DL (2010). “Adverse outcome pathways: a conceptual framework to support ecotoxicology research and risk assessment.” Environ Toxicol Chem 29(3): 730–741. [DOI] [PubMed] [Google Scholar]
  5. Antebi YE, Nandagopal N and Elowitz MB (2017). “An operational view of intercellular signaling pathways.” Curr Opin Syst Biol 1: 16–24. [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Arenas E. (2014). “Wnt signaling in midbrain dopaminergic neuron development and regenerative medicine for Parkinson’s disease.” Journal of Molecular Cell Biology 6(1): 42–53. [DOI] [PubMed] [Google Scholar]
  7. Arzuaga X, Smith MT, Gibbons CF, Skakkebæk NE, Yost EE, Beverly BE, Hotchkiss AK, Hauser R, Pagani RL and Schrader SM (2019). “Proposed Key Characteristics of Male Reproductive Toxicants as an Approach for Organizing and Evaluating Mechanistic Evidence in Human Health Hazard Assessments.” Environmental health perspectives 127(6): 065001. [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Aschner M, Ceccatelli S, Daneshian M, Fritsche E, Hasiwa N, Hartung T, Hogberg HT, Leist M, Li A, Mundi WR, Padilla S, Piersma AH, Bal-Price A, Seiler A, Westerink RH, Zimmer B. and Lein PJ (2017). “Reference compounds for alternative test methods to indicate developmental neurotoxicity (DNT) potential of chemicals: example lists and criteria for their selection and use.” Altex 34(1): 49–74. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Baker N, Boobis A, Burgoon L, Carney E, Currie R, Fritsche E, Knudsen T, Laffont M, Piersma AH and Poole A. (2018). “Building a developmental toxicity ontology.” Birth defects research 110(6): 502–518. [DOI] [PubMed] [Google Scholar]
  10. Bal-Price A, Crofton KM, Sachana M, Shafer TJ, Behl M, Forsby A, Hargreaves A, Landesmann B, Lein PJ, Louisse J, Monnet-Tschudi F, Paini A, Rolaki A, Schrattenholz A, Suñol C, van Thriel C, Whelan M. and Fritsche E. (2015). “Putative adverse outcome pathways relevant to neurotoxicity.” Critical Reviews in Toxicology 45(1): 83–91. [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Bal-Price A, Hogberg HT, Crofton KM, Daneshian M, FitzGerald RE, Fritsche E, Heinonen T, Bennekou SH, Klima S. and Piersma AH (2018). “Recommendation on test readiness criteria for new approach methods (NAM) in toxicology: exemplified for developmental neurotoxicity (DNT).” Altex 35(3): 306. [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Bal-Price A, Lein PJ, Keil KP, Sethi S, Shafer T, Barenys M, Fritsche E, Sachana M. and Meek ME (2017). “Developing and applying the adverse outcome pathway concept for understanding and predicting neurotoxicity.” NeuroToxicology 59: 240–255. [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Bal-Price AK, Hogberg HT, Buzanska L. and Coecke S. (2010). “Relevance of in vitro neurotoxicity testing for regulatory requirements: challenges to be considered.” Neurotoxicol Teratol 32(1): 36–41. [DOI] [PubMed] [Google Scholar]
  14. Bal-Price AK, Hogberg HT, Buzanska L, Lenas P, van Vliet E. and Hartung T. (2010). “In vitro developmental neurotoxicity (DNT) testing: Relevant models and endpoints.” NeuroToxicology 31(5): 545–554. [DOI] [PubMed] [Google Scholar]
  15. Barros VG, Berger MA, Martijena ID, Sarchi MI, Perez AA, Molina VA, Tarazi FI and Antonelli MC (2004). “Early adoption modifies the effects of prenatal stress on dopamine and glutamate receptors in adult rat brain.” Journal of Neuroscience Research 76(4): 488–496. [DOI] [PubMed] [Google Scholar]
  16. Behl M, Hsieh JH, Shafer TJ, Mundy WR, Rice JR, Boyd WA, Freedman JH, Hunter ES 3rd, Jarema KA, Padilla S. and Tice RR (2015). “Use of alternative assays to identify and prioritize organophosphorus flame retardants for potential developmental and neurotoxicity.” Neurotoxicol Teratol 52(Pt B): 181–193. [DOI] [PubMed] [Google Scholar]
  17. Behl M, Ryan K, Hsieh J-H, Parham F, Shapiro AJ, Collins BJ, Sipes NS, Birnbaum LS, Bucher JR and Foster PM (2019). “Screening for developmental neurotoxicity at the national toxicology program: The future is here.” Toxicological Sciences 167(1): 6–14. [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Belle AM, Enright HA, Sales AP, Kulp K, Osburn J, Kuhn EA, Fischer NO and Wheeler EK (2018). “Evaluation of in vitro neuronal platforms as surrogates for in vivo whole brain systems.” Scientific Reports 8(1): 10820. [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Bellinger D, Leviton A, Sloman J, Rabinowitz M, Needleman HL and Waternaux C. (1991). “Low-level lead exposure and children’s cognitive function in the preschool years.” Pediatrics 87(2): 219–227. [PubMed] [Google Scholar]
  20. Berger MA, Barros VG, Sarchi MI, Tarazi FI and Antonelli MC (2002). “Long-term effects of prenatal stress on dopamine and glutamate receptors in adult rat brain.” Neurochemical Research 27(11): 1525–1533. [DOI] [PubMed] [Google Scholar]
  21. Bielarczyk H, Tomsig J. and Suszkiw J. (1994). “Perinatal low-level lead exposure and the septo-hippocampal cholinergic system: selective reduction of muscarinic receptors and cholineacetyltransferase in the rat septum.” Brain research 643(1–2): 211–217. [DOI] [PubMed] [Google Scholar]
  22. Bilbo S. and Schwarz J. (2009). “Early-life programming of later-life brain and behavior: a critical role for the immune system.” Frontiers in Behavioral Neuroscience 3(14). [DOI] [PMC free article] [PubMed] [Google Scholar]
  23. Boucher O, Jacobson SW, Plusquellec P, Dewailly E, Ayotte P, Forget-Dubois N, Jacobson JL and Muckle G. (2012). “Prenatal methylmercury, postnatal lead exposure, and evidence of attention deficit/hyperactivity disorder among Inuit children in Arctic Quebec.” Environmental Health Perspectives 120(10): 1456–1461. [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Bourjeily N. and Suszkiw J. (1997). “Developmental cholinotoxicity of lead: loss of septal cholinergic neurons and long-term changes in cholinergic innervation of the hippocampus in perinatally lead-exposed rats.” Brain research 771(2): 319–328. [DOI] [PubMed] [Google Scholar]
  25. Boyles AL, Hammock P. and Speer MC (2005). “Candidate gene analysis in human neural tube defects.” American Journal of Medical Genetics. Part C: Seminars in Medical Genetics 135C(1): 9–23. [DOI] [PubMed] [Google Scholar]
  26. Breier JM, Radio NM, Mundy WR and Shafer TJ (2008). “Development of a high-throughput screening assay for chemical effects on proliferation and viability of immortalized human neural progenitor cells.” Toxicol Sci 105(1): 119–133. [DOI] [PubMed] [Google Scholar]
  27. Brkljacic J, Vojnovic Milutinovic D, Dundjerski J. and Matic G. (2004). “Mercury inhibits rat liver and kidney glucocorticoid receptor hormone binding activity.” Cell Biol Toxicol 20(3): 171–182. [DOI] [PubMed] [Google Scholar]
  28. Brown JP, Hall D, Frank CL, Wallace K, Mundy WR and Shafer TJ (2016). “Editor’s Highlight: Evaluation of a Microelectrode Array-Based Assay for Neural Network Ontogeny Using Training Set Chemicals.” Toxicol Sci 154(1): 126–139. [DOI] [PubMed] [Google Scholar]
  29. Cameron NM, Champagne FA, Parent C, Fish EW, Ozaki-Kuroda K. and Meaney MJ (2005). “The programming of individual differences in defensive responses and reproductive strategies in the rat through variations in maternal care.” Neuroscience & Biobehavioral Reviews 29(4–5): 843–865. [DOI] [PubMed] [Google Scholar]
  30. Catron TR, Keely SP, Brinkman NE, Zurlinden TJ, Wood CE, Wright JR, Phelps D, Wheaton E, Kvasnicka A, Gaballah S, Lamendella R. and Tal T. (2019). “Host Developmental Toxicity of BPA and BPA Alternatives Is Inversely Related to Microbiota Disruption in Zebrafish.” Toxicol Sci 167(2): 468–483. [DOI] [PubMed] [Google Scholar]
  31. Catron TR, Swank A, Wehmas LC, Phelps D, Keely SP, Brinkman NE, McCord J, Singh R, Sobus J, Wood CE, Strynar M, Wheaton E. and Tal T. (2019). “Microbiota alter metabolism and mediate neurodevelopmental toxicity of 17beta-estradiol.” Sci Rep 9(1): 7064. [DOI] [PMC free article] [PubMed] [Google Scholar]
  32. Clarke LE and Barres BA (2013). “Emerging roles of astrocytes in neural circuit development.” Nat Rev Neurosci 14(5): 311–321. [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. Cook DJ, Mulrow CD and Haynes RB (1997). “Systematic reviews: synthesis of best evidence for clinical decisions.” Annals of internal medicine 126(5): 376–380. [DOI] [PubMed] [Google Scholar]
  34. Cory-Slechta D. (1995). “Relationships between lead-induced learning impairments and changes in dopaminergic, cholinergic, and glutamatergic neurotransmitter system functions.” Annual review of pharmacology and toxicology 35(1): 391–415. [DOI] [PubMed] [Google Scholar]
  35. Cory-Slechta DA, Crofton KM, Foran JA, Ross JF, Sheets LP, Weiss B. and Mileson B. (2001). “Methods to identify and characterize developmental neurotoxicity for human health risk assessment. I: behavioral effects.” Environmental health perspectives 109(Suppl 1): 79. [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. Cory-Slechta DA, O’Mara DJ and Brockel BJ (1998). “Nucleus accumbens dopaminergic mediation of fixed interval schedule-controlled behavior and its modulation by low-level lead exposure.” Journal of Pharmacology and Experimental Therapeutics 286(3): 794–805. [PubMed] [Google Scholar]
  37. Cory-Slechta DA, O’Mara DJ and Brockel BJ (1999). “Learning versus performance impairments following regional administration of MK-801 into nucleus accumbens and dorsomedial striatum.” Behavioural Brain Research 102: 181–194. [DOI] [PubMed] [Google Scholar]
  38. Cotterill E, Hall D, Wallace K, Mundy WR, Eglen SJ and Shafer TJ (2016). “Characterization of early cortical neural network development in multiwell microelectrode array plates.” Journal of biomolecular screening 21(5): 510–519. [DOI] [PMC free article] [PubMed] [Google Scholar]
  39. Council NR (2009). Phthalates and cumulative risk assessment: the tasks ahead, National Academies Press. [PubMed] [Google Scholar]
  40. Council NR (2014). Review of EPA's Integrated Risk Information System (IRIS) Process. Washington, DC, The National Academies Press. [PubMed] [Google Scholar]
  41. Crofton KM, Mundy WR, Lein PJ, Bal-Price A, Coecke S, Seiler AE, Knaut H, Buzanska L. and Goldberg A. (2011). “Developmental neurotoxicity testing: recommendations for developing alternative methods for the screening and prioritization of chemicals.” ALTEX-Alternatives to animal experimentation 28(1): 9–15. [PubMed] [Google Scholar]
  42. Crofton KM, Mundy WR and Shafer TJ (2012). “Developmental neurotoxicity testing: A path forward.” Congenital Anomalies 52(3): 140–146. [DOI] [PubMed] [Google Scholar]
  43. Davis AP, Grondin CJ, Johnson RJ, Sciaky D, McMorran R, Wiegers J, Wiegers TC and Mattingly CJ (2019). “The Comparative Toxicogenomics Database: update 2019.” Nucleic Acids Res 47(D1): D948–d954. [DOI] [PMC free article] [PubMed] [Google Scholar]
  44. De Viti S, Martino A, Musilli M, Fiorentini C. and Diana G. (2010). “The Rho GTPase activating CNF1 improves associative working memory for object-in-place.” Behavioural brain research 212(1): 78–83. [DOI] [PubMed] [Google Scholar]
  45. de Vries GJ and Forger NG (2015). “Sex differences in the brain: a whole body perspective.” Biology of sex differences 6(1): 15. [DOI] [PMC free article] [PubMed] [Google Scholar]
  46. Dennis G Jr., Sherman BT, Hosack DA, Yang J, Gao W, Lane HC and Lempicki RA (2003). “DAVID: Database for Annotation, Visualization, and Integrated Discovery.” Genome Biol 4(5): P3. [PubMed] [Google Scholar]
  47. DeSesso JM (2017). “Future of developmental toxicity testing.” Current Opinion in Toxicology 3: 1–5. [Google Scholar]
  48. Drever BD, Riedel G. and Platt B. (2011). “The cholinergic system and hippocampal plasticity.” Behavioural brain research 221(2): 505–514. [DOI] [PubMed] [Google Scholar]
  49. Druwe I, Freudenrich TM, Wallace K, Shafer TJ and Mundy WR (2015). “Sensitivity of neuroprogenitor cells to chemical-induced apoptosis using a multiplexed assay suitable for high-throughput screening.” Toxicology 333: 14–24. [DOI] [PubMed] [Google Scholar]
  50. Druwe I, Freudenrich TM, Wallace K, Shafer TJ and Mundy WR (2016). “Comparison of human induced pluripotent stem cell-derived neurons and rat primary cortical neurons as in vitro models of neurite outgrowth.” Applied in vitro Toxicology 2(1): 26–36. [Google Scholar]
  51. Druwe I, F. T. M., Kathleen W, S. T. J. and M. W. R. (2016). “Comparison of Human Induced Pluripotent Stem Cell-Derived Neurons and Rat Primary Cortical Neurons as In Vitro Models of Neurite Outgrowth.” Applied In Vitro Toxicology 2(1): 26–36. [Google Scholar]
  52. Elez D, Dundjerski J. and Matic G. (2001). “Cadmium affects the redox state of rat liver glucocorticoid receptor.” Cell Biol Toxicol 17(3): 169–177. [DOI] [PubMed] [Google Scholar]
  53. EPA, U. S. (1998). “Guidelines for neurotoxicity risk assessment.” Federal Register 63(93): 26926–26954. [Google Scholar]
  54. EPA, U. S. (1998). Health effects test guidelines OPPTS 870.6300 developmental neurotoxicity study. Washington, DC, U.S: Environmentall Protection Agency. [Google Scholar]
  55. EPA, U. S. (2019). Directive to Prioritize Efforts to Reduce Animal Testing, U.S: EPA. [Google Scholar]
  56. Estes ML and McAllister AK (2016). “Maternal immune activation: Implications for neuropsychiatric disorders.” Science 353(6301): 772–777. [DOI] [PMC free article] [PubMed] [Google Scholar]
  57. Faustman EM, Gohlke J, Judd NL, Lewandowski TA, Bartell SM and Griffith WC (2005). “Modeling developmental processes in animals: applications in neurodevelopmental toxicology.” Environ Toxicol Pharmacol 19(3): 615–624. [DOI] [PubMed] [Google Scholar]
  58. Fernald LC, Neufeld LM, Barton LR, Schnaas L, Rivera J. and Gertler PJ (2006). “Parallel deficits in linear growth and mental development in low-income Mexican infants in the second year of life.” Public health nutrition 9(2): 178–186. [DOI] [PubMed] [Google Scholar]
  59. Frank CL, Brown JP, Wallace K, Mundy WR and Shafer TJ (2017). “From the Cover: Developmental Neurotoxicants Disrupt Activity in Cortical Networks on Microelectrode Arrays: Results of Screening 86 Compounds During Neural Network Formation.” Toxicol Sci 160(1): 121–135. [DOI] [PubMed] [Google Scholar]
  60. Fritsche E. (2017). Report on Integrated Testing Strategies for the identification and evaluation of chemical hazards associated with the developmental neurotoxicity (DNT), to facilitate discussions at the Joint EFSA. OECD Workshop on DNT ENV/JM/MONO; (s). [Google Scholar]
  61. Fritsche E. (2017). “Report on Integrated Testing Strategies for the identification and evaluation of chemical hazards associated with the developmental neurotoxicity (DNT), to facilitate discussions at the Joint EFSA/OECD Workshop on DNT.” ENV/JM/MONO(2017)4/ANN1. [Google Scholar]
  62. Fritsche E, Alm H, Baumann J, Geerts L, Håkansson H, Masjosthusmann S. and Witters H. (2015). “Literature review on in vitro and alternative Developmental Neurotoxicity (DNT) testing methods.” EFSA Supporting Publications 12(4): 778E-n/a. [Google Scholar]
  63. Fritsche E, Barenys M, Klose J, Masjosthusmann S, Nimtz L, Schmuck M, Wuttke S. and Tigges J. (2018). “Current availability of stem cell-based in vitro methods for Developmental Neurotoxicity (DNT) testing.” Toxicological Sciences 165(1): 21–30. [DOI] [PubMed] [Google Scholar]
  64. Fritsche E, Crofton KM, Hernandez AF, Bennekou SH, Leist M, Bal-Price A, Reaves E, Wilks MF, Terron A. and Solecki R. (2017). “OECD/EFSA workshop on developmental neurotoxicity (DNT): The use of non-animal test methods for regulatory purposes.” Altex 34(1): 311–318. [DOI] [PubMed] [Google Scholar]
  65. Fritsche E, Grandjean P, Crofton KM, Aschner M, Goldberg A, Heinonen T, Hessel EV, Hogberg HT, Bennekou SH and Lein PJ (2018). “Consensus statement on the need for innovation, transition and implementation of developmental neurotoxicity (DNT) testing for regulatory purposes.” Toxicology and applied pharmacology 354: 3–6. [DOI] [PMC free article] [PubMed] [Google Scholar]
  66. Frotscher M, Vida I. and Bender R. (2000). “Evidence for the existence of non-GABAergic, cholinergic interneurons in the rodent hippocampus.” Neuroscience 96(1): 27–31. [DOI] [PubMed] [Google Scholar]
  67. Gerhart J. (1999). “1998 Warkany lecture: signaling pathways in development.” Teratology 60(4): 226–239. [DOI] [PubMed] [Google Scholar]
  68. Gilbert M, Kelly M, Samsam T. and Goodman J. (2005). “Chronic developmental lead exposure reduces neurogenesis in adult rat hippocampus but does not impair spatial learning.” Toxicological Sciences 86(2): 365–374. [DOI] [PubMed] [Google Scholar]
  69. Gilbert M, Mack C. and Lasley S. (1996). “Chronic developmental lead exposure increases the threshold for long-term potentiation in rat dentate gyrus in vivo.” Brain research 736(1–2): 118–124. [DOI] [PubMed] [Google Scholar]
  70. Gulino A, Di Marcotullio L, Ferretti E, De Smaele E. and Screpanti I. (2007). “Hedgehog signaling pathway in neural development and disease.” Psychoneuroendocrinology 32: S52–S56. [DOI] [PubMed] [Google Scholar]
  71. Guo B, Yan CH, Cai S, Yuan XB and Shen XM (2013). “Low level prenatal exposure to methylmercury disrupts neuronal migration in the developing rat cerebral cortex.” Toxicology 304: 57–68. [DOI] [PubMed] [Google Scholar]
  72. Hanahan D. and Weinberg RA (2000). “The hallmarks of cancer.” Cell 100(1): 57–70. [DOI] [PubMed] [Google Scholar]
  73. Hanahan D. and Weinberg RA (2011). “Hallmarks of cancer: The next generation.” Cell 144(5): 646–674. [DOI] [PubMed] [Google Scholar]
  74. Harrill JA, Freudenrich T, Wallace K, Ball K, Shafer TJ and Mundy WR (2018). “Testing for developmental neurotoxicity using a battery of in vitro assays for key cellular events in neurodevelopment.” Toxicology and Applied Pharmacology 354: 24–39. [DOI] [PubMed] [Google Scholar]
  75. Harrill JA, Robinette BL and Mundy WR (2011). “Use of high content image analysis to detect chemical-induced changes in synaptogenesis in vitro.” Toxicol In Vitro 25(1): 368–387. [DOI] [PubMed] [Google Scholar]
  76. Harris MJ and Juriloff DM (2007). “Mouse mutants with neural tube closure defects and their role in understanding human neural tube defects.” Birth Defects Research, Part A: Clinical and Molecular Teratology 79(3): 187–210. [DOI] [PubMed] [Google Scholar]
  77. Hasselmo ME “Neuromodulation: acetylcholine and memory consolidation.” Trends in Cognitive Sciences 3(9): 351–359. [DOI] [PubMed] [Google Scholar]
  78. Hava G, Vered L, Yael M, Mordechai H. and Mahoud H. (2006). “Alterations in behavior in adult offspring mice following maternal inflammation during pregnancy.” Developmental Psychobiology: The Journal of the International Society for Developmental Psychobiology 48(2): 162–168. [DOI] [PubMed] [Google Scholar]
  79. Heijtz RD, Wang S, Anuar F, Qian Y, Björkholm B, Samuelsson A, Hibberd ML, Forssberg H. and Pettersson S. (2011). “Normal gut microbiota modulates brain development and behavior.” Proceedings of the National Academy of Sciences 108(7): 3047–3052. [DOI] [PMC free article] [PubMed] [Google Scholar]
  80. Hessel EV, Staal YC and Piersma AH (2018). “Design and validation of an ontology-driven animal-free testing strategy for developmental neurotoxicity testing.” Toxicology and applied pharmacology 354: 136–152. [DOI] [PubMed] [Google Scholar]
  81. Higgins JP and Green S. (2011). Cochrane handbook for systematic reviews of interventions, John Wiley & Sons. [Google Scholar]
  82. Hoffman KL, Hornig M, Yaddanapudi K, Jabado O. and Lipkin WI (2004). “A murine model for neuropsychiatric disorders associated with group A β-hemolytic streptococcal infection.” Journal of Neuroscience 24(7): 1780–1791. [DOI] [PMC free article] [PubMed] [Google Scholar]
  83. Howerton CL and Bale TL (2012). “Prenatal programing: at the intersection of maternal stress and immune activation.” Hormones and behavior 62(3): 237–242. [DOI] [PMC free article] [PubMed] [Google Scholar]
  84. Huang C, Zhang X, Zheng J, Chen C, Chen Y. and Yi J. (2014). “Upregulation of miR-137 protects anesthesia-induced hippocampal neurodegeneration.” International journal of clinical and experimental pathology 7(8): 5000. [PMC free article] [PubMed] [Google Scholar]
  85. Hur EM and Zhou FQ (2010). “GSK3 signalling in neural development.” Nature Reviews. Neuroscience 11(8): 539–551. [DOI] [PMC free article] [PubMed] [Google Scholar]
  86. Hutson MS, Leung MC, Baker NC, Spencer RM and Knudsen TB (2017). “Computational model of secondary palate fusion and disruption.” Chemical research in toxicology 30(4): 965–979. [DOI] [PMC free article] [PubMed] [Google Scholar]
  87. Inomata K, Nasu F. and Tanaka H. (1987). “Decreased density of synaptic formation in the frontal cortex of neonatal rats exposed to ethanol in utero.” Int J Dev Neurosci 5(5–6): 455–460. [DOI] [PubMed] [Google Scholar]
  88. Jessberger S, Clark RE, Broadbent NJ, Clemenson GD, Consiglio A, Lie DC, Squire LR and Gage FH (2009). “Dentate gyrus-specific knockdown of adult neurogenesis impairs spatial and object recognition memory in adult rats.” Learning & memory 16(2): 147–154. [DOI] [PMC free article] [PubMed] [Google Scholar]
  89. Jones L, Fischer I. and Levitt P. (1996). “Nonuniform alteration of dendritic development in the cerebral cortex following prenatal cocaine exposure.” Cereb Cortex 6(3): 431–445. [DOI] [PubMed] [Google Scholar]
  90. Kleinstreuer N, Dix D, Rountree M, Baker N, Sipes N, Reif D, Spencer R. and Knudsen T. (2013). “A computational model predicting disruption of blood vessel development.” PLoS computational biology 9(4): e1002996. [DOI] [PMC free article] [PubMed] [Google Scholar]
  91. Kleinstreuer NC, Sullivan K, Allen D, Edwards S, Mendrick DL, Embry M, Matheson J, Rowlands JC, Munn S, Maull E. and Casey W. (2016). “Adverse outcome pathways: From research to regulation scientific workshop report.” Regulatory Toxicology and Pharmacology 76: 39–50. [DOI] [PMC free article] [PubMed] [Google Scholar]
  92. Knuesel I, Chicha L, Britschgi M, Schobel SA, Bodmer M, Hellings JA, Toovey S. and Prinssen EP (2014). “Maternal immune activation and abnormal brain development across CNS disorders.” Nature Reviews. Neurology 10(11): 643–660. [DOI] [PubMed] [Google Scholar]
  93. Kushman ME, Kraft AD, Guyton KZ, Chiu WA, Makris SL and Rusyn I. (2013). “A systematic approach for identifying and presenting mechanistic evidence in human health assessments.” Regulatory toxicology and pharmacology : RTP 67(2): 266–277. [DOI] [PMC free article] [PubMed] [Google Scholar]
  94. La Merrill MA, Vandenberg LN, Smith MT, Goodson W, Browne P, Patisaul HB, Guyton KZ, Kortenkamp A, Cogliano VJ and Woodruff TJ (2019). “Consensus on the key characteristics of endocrine-disrupting chemicals as a basis for hazard identification.” Nature Reviews Endocrinology: 1–13. [DOI] [PMC free article] [PubMed] [Google Scholar]
  95. Lanphear BP, Dietrich K, Auinger P. and Cox C. (2000). “Cognitive deficits associated with blood lead concentrations< 10 microg/dL in US children and adolescents.” Public health reports 115(6): 521. [DOI] [PMC free article] [PubMed] [Google Scholar]
  96. Lasky JL and Wu H. (2005). “Notch signaling, brain development, and human disease.” Pediatric research 57(5 Part 2): 104R. [DOI] [PubMed] [Google Scholar]
  97. Lee DY (2015). “Roles of mTOR Signaling in Brain Development.” Experimental Neurobiology 24(3): 177–185. [DOI] [PMC free article] [PubMed] [Google Scholar]
  98. Lein P, Silbergeld E, Locke P. and Goldberg AM (2005). “In vitro and other alternative approaches to developmental neurotoxicity testing (DNT).” Environmental toxicology and pharmacology 19(3): 735–744. [DOI] [PubMed] [Google Scholar]
  99. Leung MC, Hutson MS, Seifert AW, Spencer RM and Knudsen TB (2016). “Computational modeling and simulation of genital tubercle development.” Reproductive Toxicology 64: 151–161. [DOI] [PubMed] [Google Scholar]
  100. Li J, Olsen J, Vestergaard M. and Obel C. (2010). “Attention-deficit/hyperactivity disorder in the offspring following prenatal maternal bereavement: a nationwide follow-up study in Denmark.” European Child and Adolescent Psychiatry 19(10): 747–753. [DOI] [PubMed] [Google Scholar]
  101. Liu A. and Niswander LA (2005). “Bone morphogenetic protein signalling and vertebrate nervous system development.” Nature Reviews Neuroscience 6(12): 945. [DOI] [PubMed] [Google Scholar]
  102. Louvi A. and Artavanis-Tsakonas S. (2006). “Notch signalling in vertebrate neural development.” Nature Reviews. Neuroscience 7(2): 93–102. [DOI] [PubMed] [Google Scholar]
  103. Luderer U, Eskenazi B, Hauser R, Korach KS, McHale CM, Moran F, Rieswijk L, Solomon G, Udagawa O. and Zhang L. (2019). “Proposed key characteristics of female reproductive toxicants as an approach for organizing and evaluating mechanistic data in hazard assessment.” Environmental health perspectives 127(7): 075001. [DOI] [PMC free article] [PubMed] [Google Scholar]
  104. Makino Y, Tanaka H, Dahlman-Wright K. and Makino I. (1996). “Modulation of glucocorticoid-inducible gene expression by metal ions.” Mol Pharmacol 49(4): 612–620. [PubMed] [Google Scholar]
  105. Martinez-Tellez RI, Hernandez-Torres E, Gamboa C. and Flores G. (2009). “Prenatal stress alters spine density and dendritic length of nucleus accumbens and hippocampus neurons in rat offspring.” Synapse 63(9): 794–804. [DOI] [PubMed] [Google Scholar]
  106. Masjosthusmann S, Becker D, Petzuch B, Klose J, Siebert C, Deenen R, Barenys M, Baumann J, Dach K. and Tigges J. (2018). “A transcriptome comparison of time-matched developing human, mouse and rat neural progenitor cells reveals human uniqueness.” Toxicology and applied pharmacology 354: 40–55. [DOI] [PubMed] [Google Scholar]
  107. Miller GW, Chandrasekaran V, Yaghoobi B. and Lein PJ (2018). “Opportunities and challenges for using the zebrafish to study neuronal connectivity as an endpoint of developmental neurotoxicity.” NeuroToxicology 67: 102–111. [DOI] [PMC free article] [PubMed] [Google Scholar]
  108. Miller MW (1986). “Effects of alcohol on the generation and migration of cerebral cortical neurons.” Science 233(4770): 1308–1311. [DOI] [PubMed] [Google Scholar]
  109. Moreira EG, Vassilieff I. and Vassilieff V. S. l. (2001). “Developmental lead exposure: behavioral alterations in the short and long term.” Neurotoxicology and Teratology 23(5): 489–495. [DOI] [PubMed] [Google Scholar]
  110. Mousa A. and Bakhiet M. (2013). “Role of Cytokine Signaling during Nervous System Development.” International Journal of Molecular Sciences 14(7): 13931–13957. [DOI] [PMC free article] [PubMed] [Google Scholar]
  111. Mundy WR, Padilla S, Breier JM, Crofton KM, Gilbert ME, Herr DW, Jensen KF, Radio NM, Raffaele KC, Schumacher K, Shafer TJ and Cowden J. (2015). “Expanding the test set: Chemicals with potential to disrupt mammalian brain development.” Neurotoxicology and Teratology 52(Pt A): 25–35. [DOI] [PubMed] [Google Scholar]
  112. National Academies of Sciences, E. and Medicine (2018). Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation. Washington, DC, The National Academies Press. [Google Scholar]
  113. Nigg JT, Knottnerus GM, Martel MM, Nikolas M, Cavanagh K, Karmaus W. and Rappley MD (2008). “Low blood lead levels associated with clinically diagnosed attention-deficit/hyperactivity disorder and mediated by weak cognitive control.” Biol Psychiatry 63(3): 325–331. [DOI] [PMC free article] [PubMed] [Google Scholar]
  114. NRC (2000). Scientific frontiers in developmental toxicology and risk assessment. Washington, DC, National Academy Press. [PubMed] [Google Scholar]
  115. O’Shaughnessy KL, Thomas SE, Spring SR, Ford JL, Ford RL and Gilbert ME (2019). “A transient window of hypothyroidism alters neural progenitor cells and results in abnormal brain development.” Scientific reports 9(1): 4662. [DOI] [PMC free article] [PubMed] [Google Scholar]
  116. OECD (2007). OECD guidelines for the testing of chemicals, Section 4: Health effects Test no. 426: Developmental neurotoxicity study; Paris, France. [Google Scholar]
  117. OECD (2012). OECD Guidelines for the Testing of Chemicals, Section 4: Health Effects Test no. 443: Extended one-generation reproductive toxicity study - Revised; Paris, France. [Google Scholar]
  118. OECD (2018). Test No. 443: Extended One-Generation Reproductive Toxicity Study. [Google Scholar]
  119. Peterson RT, Nass R, Boyd WA, Freedman JH, Dong K. and Narahashi T. (2008). “Use of non-mammalian alternative models for neurotoxicological study.” Neurotoxicology 29(3): 546–555. [DOI] [PMC free article] [PubMed] [Google Scholar]
  120. Picciotto MR, Higley MJ and Mineur YS (2012). “Acetylcholine as a neuromodulator: cholinergic signaling shapes nervous system function and behavior.” Neuron 76(1): 116–129. [DOI] [PMC free article] [PubMed] [Google Scholar]
  121. Pires-daSilva A. and Sommer RJ (2003). “The evolution of signalling pathways in animal development.” Nature Reviews Genetics 4(1): 39. [DOI] [PubMed] [Google Scholar]
  122. Reddy GR, Basha MR, Devi CB, Suresh A, Baker JL, Shafeek A, Heinz J. and Chetty CS (2003). “Lead induced effects on acetylcholinesterase activity in cerebellum and hippocampus of developing rat.” International Journal of Developmental Neuroscience 21(6): 347–352. [DOI] [PubMed] [Google Scholar]
  123. Robinson JF, Port JA, Yu X. and Faustman EM (2010). “Integrating genetic and toxicogenomic information for determining underlying susceptibility to developmental disorders.” Birth Defects Res A Clin Mol Teratol 88(10): 920–930. [DOI] [PubMed] [Google Scholar]
  124. Robinson JF, Yu X, Hong S, Griffith WC, Beyer R, Kim E. and Faustman EM (2009). “Cadmium-induced differential toxicogenomic response in resistant and sensitive mouse strains undergoing neurulation.” Toxicol Sci 107(1): 206–219. [DOI] [PMC free article] [PubMed] [Google Scholar]
  125. Robinson JF, Yu X, Hong S, Zhou C, Kim N, DeMasi D. and Faustman EM (2010). “Embryonic toxicokinetic and dynamic differences underlying strain sensitivity to cadmium during neurulation.” Reprod Toxicol 29(3): 279–285. [DOI] [PMC free article] [PubMed] [Google Scholar]
  126. Rooney AA, Boyles AL, Wolfe MS, Bucher JR and Thayer KA (2014). “Systematic review and evidence integration for literature-based environmental health science assessments.” Environmental health perspectives 122(7): 711. [DOI] [PMC free article] [PubMed] [Google Scholar]
  127. Rossi-George A, Virgolini MB, Weston D, Thiruchelvam M. and Cory-Slechta DA (2011). “Interactions of Lifetime Lead Exposure and Stress: Behavioral; Neurochemical and HPA Axis Effects.” Neurotoxicology: 83–99. [DOI] [PMC free article] [PubMed] [Google Scholar]
  128. Ruan D-Y, Chen J-T, Zhao C, Xu Y-Z, Wang M. and Zhao W-F (1998). “Impairment of long-term potentiation and paired-pulse facilitation in rat hippocampal dentate gyrus following developmental lead exposure in vivo.” Brain research 806(2): 196–201. [DOI] [PubMed] [Google Scholar]
  129. Russell WMS and Burch RL (1959). “The principles of humane experimental technique.” The principles of humane experimental technique. [Google Scholar]
  130. Ryan KR, Sirenko O, Parham F, Hsieh J-H, Cromwell EF, Tice RR and Behl M. (2016). “Neurite outgrowth in human induced pluripotent stem cell-derived neurons as a high-throughput screen for developmental neurotoxicity or neurotoxicity.” Neurotoxicology 53: 271–281. [DOI] [PubMed] [Google Scholar]
  131. Sachana M, Bal-Price A, Crofton KM, Bennekou SH, Shafer TJ, Behl M. and Terron A. (2018). “International regulatory and scientific effort for improved developmental neurotoxicity testing.” Toxicological Sciences 167(1): 45–57. [DOI] [PubMed] [Google Scholar]
  132. Sathyanesan A, Zhou J, Scafidi J, Heck DH, Sillitoe RV and Gallo V. (2019). “Emerging connections between cerebellar development, behaviour and complex brain disorders.” Nature Reviews Neuroscience: 1. [DOI] [PMC free article] [PubMed] [Google Scholar]
  133. Schmidt BZ, Lehmann M, Gutbier S, Nembo E, Noel S, Smirnova L, Forsby A, Hescheler J, Avci HX and Hartung T. (2017). “In vitro acute and developmental neurotoxicity screening: an overview of cellular platforms and high-throughput technical possibilities.” Archives of toxicology 91(1): 1–33. [DOI] [PubMed] [Google Scholar]
  134. Schwarz JM, Sholar PW and Bilbo SD (2012). “Sex differences in microglial colonization of the developing rat brain.” J Neurochem 120(6): 948–963. [DOI] [PMC free article] [PubMed] [Google Scholar]
  135. Segal D, Lin Y-S, Ginsberg G. and Sonawane B. (2015). “A Conceptual Framework for Evaluating the Interaction of a Chemical and Nonchemical Stressor in Human Health Risk Assessments: A Case Study for Lead and Psychosocial Stress.” Human and Ecological Risk Assessment: An International Journal 21(7): 1840–1868. [Google Scholar]
  136. Seongeun C, Andrew W. and Mark RB (2007). “Brain Slices as Models for Neurodegenerative Disease and Screening Platforms to Identify Novel Therapeutics.” Current Neuropharmacology 5(1): 19–33. [DOI] [PMC free article] [PubMed] [Google Scholar]
  137. Shigemoto-Mogami Y, Hoshikawa K, Goldman JE, Sekino Y. and Sato K. (2014). “Microglia enhance neurogenesis and oligodendrogenesis in the early postnatal subventricular zone.” Journal of Neuroscience 34(6): 2231–2243. [DOI] [PMC free article] [PubMed] [Google Scholar]
  138. Sipes NS, Padilla S. and Knudsen TB (2011). “Zebrafish—As an integrative model for twenty‐first century toxicity testing.” Birth Defects Research Part C: Embryo Today: Reviews 93(3): 256–267. [DOI] [PubMed] [Google Scholar]
  139. Smirnova L, Hogberg HT, Leist M. and Hartung T. (2014). “Developmental neurotoxicity - challenges in the 21st century and in vitro opportunities.” Altex 31(2): 129–156. [DOI] [PMC free article] [PubMed] [Google Scholar]
  140. Smith MT, Guyton KZ, Gibbons CF, Fritz JM, Portier CJ, Rusyn I, Demarini DM, Caldwell JC, Kavlock RJ, Lambert P, Hecht SS, Bucher JR, Stewart BW, Baan R, Cogliano VJ and Straif K. (2016). “Key characteristics of carcinogens as a basis for organizing data on mechanisms of carcinogenesis.” Environmental Health Perspectives 124(6): 713–721. [DOI] [PMC free article] [PubMed] [Google Scholar]
  141. Sobin C, Flores-Montoya MG and Alvarez JM (2017). “Early chronic low-level Pb exposure alters global exploratory behaviors but does not impair spatial and object memory retrieval in an object-in-place task in pre-adolescent C57BL/6J mice.” Neurotoxicology and teratology 61: 104–114. [DOI] [PubMed] [Google Scholar]
  142. Sobin C, Flores-Montoya MG, Gutierrez M, Parisi N. and Schaub T. (2015). “δ-Aminolevulinic acid dehydratase single nucleotide polymorphism 2 (ALAD2) and peptide transporter 2*2 haplotype (hPEPT2*2) differently influence neurobehavior in low-level lead exposed children.” Neurotoxicology and Teratology 47(0): 137–145. [DOI] [PMC free article] [PubMed] [Google Scholar]
  143. Sobin C. and Golub M. (2018). “Behavioral Outcome as a Primary Organizing Principle for Mechanistic Data in Developmental Neurotoxicity.” Handbook of Developmental Neurotoxicology: 337–347. [Google Scholar]
  144. Sobolewski M, Conrad K, Marvin E, Allen JL and Cory-Slechta DA (2018). “Endocrine active metals, prenatal stress and enhanced neurobehavioral disruption.” Horm Behav 101: 36–49. [DOI] [PMC free article] [PubMed] [Google Scholar]
  145. Spann MN, Monk C, Scheinost D. and Peterson BS (2018). “Maternal immune activation during the third trimester is associated with neonatal functional connectivity of the salience network and fetal to toddler behavior.” Journal of Neuroscience 38(11): 2877–2886. [DOI] [PMC free article] [PubMed] [Google Scholar]
  146. Spuches AM and Wilcox DE (2008). “Monomethylarsenite competes with Zn2+ for binding sites in the glucocorticoid receptor.” J Am Chem Soc 130(26): 8148–8149. [DOI] [PubMed] [Google Scholar]
  147. Thiel C, Huston J. and Schwarting R. (1998). “Hippocampal acetylcholine and habituation learning.” Neuroscience 85(4): 1253–1262. [DOI] [PubMed] [Google Scholar]
  148. Tiemeier H, Lenroot RK, Greenstein DK, Tran L, Pierson R. and Giedd JN (2010). “Cerebellum development during childhood and adolescence: a longitudinal morphometric MRI study.” Neuroimage 49(1): 63–70. [DOI] [PMC free article] [PubMed] [Google Scholar]
  149. Tingling JD, Bake S, Holgate R, Rawlings J, Nagsuk PP, Chandrasekharan J, Schneider SL and Miranda RC (2013). “CD24 expression identifies teratogen-sensitive fetal neural stem cell subpopulations: evidence from developmental ethanol exposure and orthotopic cell transfer models.” PLoS One 8(7): e69560. [DOI] [PMC free article] [PubMed] [Google Scholar]
  150. Tollefsen KE, Scholz S, Cronin MT, Edwards SW, de Knecht J, Crofton K, Garcia-Reyero N, Hartung T, Worth A. and Patlewicz G. (2014). “Applying adverse outcome pathways (AOPs) to support integrated approaches to testing and assessment (IATA).” Regulatory Toxicology and Pharmacology 70(3): 629–640. [DOI] [PubMed] [Google Scholar]
  151. Tsuji R. and Crofton KM (2012). “Developmental neurotoxicity guideline study: issues with methodology, evaluation and regulation.” Congenit Anom (Kyoto) 52(3): 122–128. [DOI] [PubMed] [Google Scholar]
  152. Tukker AM, de Groot MW, Wijnolts FM, Kasteel EE, Hondebrink L. and Westerink R. (2016). “Is the time right for in vitro neurotoxicity testing using human iPSC-derived neurons?” ALTEX-Alternatives to animal experimentation 33(3): 261–271. [DOI] [PubMed] [Google Scholar]
  153. Van Abeelen J, Ellenbroek G. and Wigman H. (1975). “Exploratory behaviour in two selectively-bred lines of mice after intrahippocampal injection of methylscopolamine.” Psychopharmacologia 41(2): 111–112. [DOI] [PubMed] [Google Scholar]
  154. Van Abeelen J. and van Nies J. (1983). “Effects of intrahippocampally-injected naloxone and morphine upon behavioural responses to novelty in mice from two selectively-bred lines.” Psychopharmacology 81(3): 232–235. [DOI] [PubMed] [Google Scholar]
  155. Van der Zee E. and Luiten P. (1993). “GABAergic neurons of the rat dorsal hippocampus express muscarinic acetylcholine receptors.” Brain research bulletin 32(6): 601–609. [DOI] [PubMed] [Google Scholar]
  156. Villeneuve DL, Crump D, Garcia-Reyero N, Hecker M, Hutchinson TH, LaLone CA, Landesmann B, Lettieri T, Munn S. and Nepelska M. (2014). “Adverse outcome pathway (AOP) development I: strategies and principles.” Toxicological Sciences 142(2): 312–320. [DOI] [PMC free article] [PubMed] [Google Scholar]
  157. Villeneuve DL, Crump D, Garcia-Reyero N, Hecker M, Hutchinson TH, LaLone CA, Landesmann B, Lettieri T, Munn S. and Nepelska M. (2014). “Adverse outcome pathway development II: best practices.” Toxicological Sciences 142(2): 321–330. [DOI] [PMC free article] [PubMed] [Google Scholar]
  158. Visan A, Hayess K, Sittner D, Pohl EE, Riebeling C, Slawik B, Gulich K, Oelgeschläger M, Luch A. and Seiler AE (2012). “Neural differentiation of mouse embryonic stem cells as a tool to assess developmental neurotoxicity in vitro.” Neurotoxicology 33(5): 1135–1146. [DOI] [PubMed] [Google Scholar]
  159. Vogel T. (2013). Insulin/IGF-signalling in embryonic and adult neural proliferation and differentiation in the mammalian central nervous system Trends in Cell Signaling Pathways in Neuronal Fate Decision, IntechOpen. [Google Scholar]
  160. Vorhees CV, Sprowles JN, Regan SL and Williams MT (2018). “A better approach to in vivo developmental neurotoxicity assessment: alignment of rodent testing with effects seen in children after neurotoxic exposures.” Toxicology and applied pharmacology 354: 176–190. [DOI] [PubMed] [Google Scholar]
  161. Wallace K, Strickland JD, Valdivia P, Mundy WR and Shafer TJ (2015). “A multiplexed assay for determination of neurotoxicant effects on spontaneous network activity and viability from microelectrode arrays.” Neurotoxicology 49: 79–85. [DOI] [PubMed] [Google Scholar]
  162. Wang L, Luo L, Luo Y-Y, Gu Y. and Ruan D-Y (2007). “Effects of Pb2+ on muscarinic modulation of glutamatergic synaptic transmission in rat hippocampal CA1 area.” Neurotoxicology 28(3): 499–507. [DOI] [PubMed] [Google Scholar]
  163. White L, Cory-Slechta D, Gilbert M, Tiffany-Castiglioni E, Zawia N, Virgolini M, Rossi-George A, Lasley S, Qian Y. and Basha MR (2007). “New and evolving concepts in the neurotoxicology of lead.” Toxicology and applied pharmacology 225(1): 1–27. [DOI] [PubMed] [Google Scholar]
  164. Wilk-Zasadna I, Bernasconi C, Pelkonen O. and Coecke S. (2015). “Biotransformation in vitro: an essential consideration in the quantitative in vitro-to-in vivo extrapolation (QIVIVE) of toxicity data.” Toxicology 332: 8–19. [DOI] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

1

RESOURCES