A proposal to facilitate weight-of-evidence assessments: Harmonization of Neurodevelopmental Environmental Epidemiology Studies (HONEES)

Abstract

The ability to conduct weight-of-evidence assessments to inform the assessment of potential environmental neurotoxicants is limited by lack of comparability of study methods, data analysis, and reporting. There is a need to establish consensus guidelines for conducting, analyzing, and reporting neurodevelopmental environmental epidemiologic studies, while recognizing that consistency is likewise needed for epidemiology studies examining other health outcomes. This paper proposes a set of considerations to be used by the scientific community at-large as a tool for systematically evaluating the quality of proposed and/or published studies in terms of their value for weight-of-evidence assessments. Particular emphasis is placed on evaluating the sensitivity and specificity of neurodevelopmental function tests, as these characteristics directly affect the risk of false positive and false negative decisions at the level of the individual, thus influencing the risk of incorrect conclusions at the level of study findings. The proposed considerations are the first step in what must be a larger consensus-based process and can serve to catalyze such a discussion. Achieving consensus in these types of endeavors is difficult; however, opportunities exist for further interdisciplinary discussion, collaboration, and research that will help realize this goal. Broad acceptance and application of such an approach can facilitate the expanded use of environmental epidemiology studies of potential neurodevelopmental toxicants in the protection of public health, and specifically children's health.

1. Introduction

Pediatric neurodevelopmental disorders have generated substantial attention from the scientific community, the public and media. This high visibility is due in part to an observed increased prevalence of autism spectrum disorders (Rice et al., 2010) and possibly attention deficit hyperactivity disorder (ADHD) (Pastor and Reuben, 2008). Many chemicals are known to be human neurotoxicants, at least following acute exposure to adults (Grandjean and Landrigan, 2006). In addition, associations between specific environmental chemicals and neuropsychiatric disorders have been reported (e.g., depression and air pollutants (Szyszkowicz et al., 2009); anxiety and PCB 153 (Plusquellec et al., 2010); ADHD and polyfluoroalkyl chemicals (Hoffman et al., 2010); ADHD and PCBs and lead (Eubig et al., 2010); autism and pesticides (Roberts et al., 2007)).

In spite of the growing concerns about perturbations in neurodevelopment that may be associated with environmental exposures, few environmental chemicals are regulated on the basis of neurodevelopmental outcomes. A review of the U.S. Environmental Protection Agency (US EPA) Integrated Risk Information System (IRIS) database illustrated the limited impact of studies assessing neurodevelopmental endpoints in children have had on the characterization of hazards resulting from early life stage exposures to environmental toxicants. The IRIS database is a publicly available peer-reviewed source of toxicological information maintained by the US EPA for over 550 environmental chemicals (US EPA, 2010). For those chemical assessments that were issued in 1997 or later (a total of 86, listed in Table 1), detailed hazard and dose response information is provided in a “Toxicological Review,” thereby allowing an examination of the use and influence of neurobehavioral data in oral and inhalation reference value derivation. Out of the 86 assessments examined, 19 had no neurotoxicity data in either animals or humans, 25 had only animal but not human neurotoxicity data, and 33 had adult human neurotoxicity data (30 of which were supported by animal data). Only 9 of the 86 IRIS toxicological databases that were examined included any information on neurotoxicity in children, and of those only 3 reported data from actual neurodevelopmental testing (the others being 2 chemicals with reports of congenital anomalies and 4 chemicals with case reports in children that suggested effects on the nervous system). The chemicals for which neurodevelopmental testing was conducted were methyl mercury, perchloroethylene, and trichloroethylene. While these data were in all cases considered qualitatively important in the weight-of-evidence evaluation of the chemical toxicity profile, methyl mercury was the only final assessment for which neurodevelopmental testing data in children were actually used as the quantitative basis for a reference value.

Table 1.

IRIS Chemicals with Toxicological Reviews

No neurotoxicity data in animals or humans
Acrolein	Methylene Diphenyl Diisocyanate (monomeric and polymeric MDI)	Thallium chloride
Bentazon (Basagran)		Thallium oxide
Berylliuma	Mirexa	Thallium selenite
1,3-Butadiene	Naphthalenea	1,2,3-Trichloropropane
Chromium(III), insoluble salts	Phosgene	2,2,4-Trimethylpentane
Chromium(VI)	Propionaldehyde	Zinc and compounds
Ethylene glycol monobutyl ether (EGBE) (2-Butoxyethanol)a	Thallium carbonate

No neurotoxicity data in animals; some data in adult humans (mostly positive); no data in children
Chloroform	Hexachlorocyclopentadiene (HCCPD)	Thallium nitrate

Animal DNT data (all positive); no adult animal data; no adult or child human data
Beryllium and compounds	1,2-Dibromoethane	Perchlorate and Perchlorate saltsa
Chlorite (sodium salt)	Diesel engine exhaust	2,2',4,4'-Tetrabromodiphenyl ether (BDE-47)
2,2',3,3',4,4',5,5',6,6'-Decabromodiphenyl ether (BDE-209)	2,2',4,4',5,5'-Hexabromodiphenyl ether (BDE-153)

Adult animal data; some DNT data, no human data; results in animals are mixed
Acetonitrile	Cyclohexane	2,2',4,4',5-Pentabromodiphenyl ether (BDE-99)
Bromobenzene	Dibutyl phthalatea
Cerium oxide and Cerium compounds	1,1-Dichloroethylene (1,1-DCE)	Phenol
Cis- and trans-1,2-Dichloroethylenea	1,3-Dichloropropene	Platinuma
Cumene	Ethyl tert-butyl ether (ETBE)a	Quinoline
	2-Methylnaphthalene	Tributyltin oxide (TBTO)
		1,3,5-Trinitrobenzene

Neurotoxicity testing in adult animals and data in adult humans (all are positive); a few assessments of DNT in animals; no data in children
Acetone	Chlordecone (Kepone)	Nitrobenzene
Acrylamidea	n-Hexane	1,1,2,2-Tetrachloroethanea
Barium and Compounds	2-Hexanone	Tetrahydrofurana
Benzene	Hydrogen cyanidea	Thallium(I), soluble salts
Carbon tetrachloridea	Methanola	Thallium acetate
Chlordane (technical)	Methyl chloride	Thallium(I) sulfate
Chloroprene	Methyl ethyl ketone (MEK)	Toluene
Dichloroacetic acid	Methyl isobutyl ketone (MIBK)	1,1,1-Trichloroethane
Dichlorobenzenesa	Methyl methacrylate	Vinyl chloride
1,4-Dioxanea	Pentachlorophenola	Xylenes

Data in children
Neurodevelopmental outcomes observed (congenital anomalies)	Case reports in children identified effects on nervous system	Neurodevelopmental testing was conducted in children
Chlorine dioxide	Boron and compounds	Methyl mercury
Trichloroacetic acida	Bromate	Perchloroethylenea
	Chloral hydrate	Trichloroethylenea
	Hydrogen sulfide

^aExternal Review Draft

For these particular examples, the use of the data from neurobehavioral testing in children was limited for reference value derivation by a number of considerations related to study conduct or data reporting. These included issues such as adequacy of sample size, inconsistent testing methodologies, questions about the selection or implementation of testing procedures, inadequate consideration of confounding factors, uncertainties regarding exposure characterization, reproducibility of study findings, and inconsistencies due to timing of exposure or life stage of assessment. For two of the draft assessments (perchloroethylene and trichloroethylene) that included neurodevelopmental data in children, other toxicological outcomes were observed at lower exposure levels and thus were selected to derive reference values (US EPA, 2008, 2010). It is important, however, to recognize the possibility that the full range of neurobehavioral outcomes at the lower end of the dose-response array may not have been fully characterized, thus limiting the weight-of-evidence evaluation of the data, and/or selection of the point of departure.

There are several factors that limit consideration of neurodevelopment in weight-of-evidence assessments. First, neurobehavioral testing in children is not mandated by any regulatory agency. Second, studies in animals may not fully capture the range of adverse outcomes in neurological development that might be observed in children. In fact, it has been noted that developmental neurotoxicity might not be evident in routine toxicology tests (Landrigan and Grandjean, 2006; Raffaele et al., 2010). Third, use of epidemiology data in weight-of-evidence assessments has been hindered by differences in methods used in conducting neurodevelopmental epidemiologic studies as well as differences in data analysis and reporting (Goodman et al., 2010; Youngstrom et al., 2010).

To address these and related issues, a symposium on “Advancing Neurodevelopmental Evaluation in Children: An Interdisciplinary Approach” was held on June 29, 2010 at the joint 34^th annual Neurobehavioral Teratology Society, 50^th annual Teratology Society, and 23^rd annual Organization of Teratology Information Specialists meetings, in Louisville, KY. The objective of the symposium was to present recent multidisciplinary research centered on advancing epidemiological evaluation of children for the detection of neurobehavioral effects potentially associated with exposure to environmental chemicals and the application of these data to human health risk assessment. The symposium presentations included a critical review of commonly used neurodevelopmental tests in epidemiological studies of associations between environmental chemical exposure and adverse health effects, and tests that should be considered for future studies (Youngstrom et al., 2010; Kenworthy et al., 2010; Anthony et al., 2010); a critical review of tests used in environmental epidemiology studies of PCBs (Goodman, 2010; Goodman et al., 2010); discussion of a 2009 interdisciplinary Needs Assessment Workshop which resulted in innovative thinking about paths forward for neurodevelopmental function testing in environmental epidemiology (LaKind et al., 2010); and considerations for selection, administration, and interpretation of neurodevelopmental tests in studies of environmental chemicals. The symposium provided a forum for further discussion on advancing the science of environmental neurodevelopmental/ epidemiological research.

A crucial finding of the analysis of epidemiology literature on PCBs and neurodevelopment was that our ability to conduct weight-of-evidence assessments, even with several studies, is limited when study methods, data analysis, and reporting lack comparability (Goodman et al., 2010). Goodman et al. (2010) called for establishing consensus standards for the conduct, analysis, and reporting of neurodevelopmental environmental epidemiologic studies with the recognition that consistency is needed for epidemiology studies examining other outcomes, as well. Youngstrom et al. (2010), in their assessment of use of neurodevelopmental function tests used in environmental epidemiology, noted that “the field of environmental epidemiology may be nearing a stage where a formal set of reporting guidelines could be developed to help the design of future studies, as has been done with clinical trials, studies of diagnostic assessment tools, and medical epidemiological studies.”

We attempted to address these issues by proposing a set of considerations to be used by the scientific community at-large as a tool for systematically evaluating the quality of proposed and/or published studies in terms of their value for weight-of-evidence assessments. We specifically propose considerations and quality criteria for epidemiological research in the area of neurodevelopmental disorders and exposure to chemicals. Particular emphasis is placed on evaluating the sensitivity and specificity of the measures, as these characteristics directly affect the risk of false positive and false negative decisions at the level of the individual, thus influencing the risk of incorrect conclusions at the level of study findings.

We recognize that the considerations proposed here, which we refer to as HONEES (Harmonization of Neurodevelopmental Environmental Epidemiology Studies), are the first steps in what must be a larger consensus-based process. We hope that this serves to catalyze such a discussion. Ultimately, the end product – a clearly articulated set of standards and quality criteria – is intended to help inform the design of proposed studies and to assist funding agencies in deciding which proposals are most likely to advance the field. Perhaps most importantly, use of this type of guidance would enhance the ability of scientists to utilize results from such studies as the National Children's Study (http://www.nationalchildrensstudy.gov/Pages/default.aspx) for weight-of-evidence assessments in support of chemical regulation for developmental neurotoxicants.

2. Methods

The proposed considerations are modeled on the STROBE (STrengthening the Reporting of OBservational studies in Epidemiology) guidelines (von Elm et al., 2008) (http://www.strobestatement.org/). They also incorporate elements from the CONSORT (Consolidated Standards of Reporting of Trials) criteria for clinical trials (Moher et al., 2001), the STARD criteria (Standards for Reporting of Diagnostic Accuracy) for assessment studies (Bossuyt et al., 2003), the Evidence Based Medicine criteria for the validity of a study of harm (Straus et al., 2005), and prior recommendations from the fields of toxicology and psychology (Youngstrom et al., 2010). These considerations were gathered, reviewed for relevance, and then collated into a more relevant set of recommendations focused on epidemiological studies related to neurodevelopment and chemical exposures.

For the purposes of reporting and evaluating studies of the neurodevelopmental effects associated with toxicant exposure, the STROBE guidelines provide an excellent starting point. STROBE includes both general recommendations and specific criteria that are relevant to longitudinal cohort studies, case control studies, and cross-sectional studies – the three major designs used to study neurodevelopmental effects of chemical exposure. For present purposes, the major contribution of the STROBE guidelines was to increase the focus on study factors that might change assessment results, such as variables that could influence the sensitivity or specificity of the assessment to exposure effects.

After comparing the tools available in the literature for evaluating epidemiological studies (STROBE), studies of diagnostic efficiency (STARD), clinical trials (CONSORT), and recent “evidence based assessment” reviews in psychology (Hunsley and Mash, 2005; 2008), we used the Quality Assessment of Diagnostic Accuracy Studies (QUADAS) as the basis for a coding tool (Whiting et al., 2003). The original QUADAS was derived from the 25 criteria in the STARD Guidelines for assessment studies, and the content captured the major study design and reporting elements that are common to both epidemiological (e.g., STROBE) and diagnostic (e.g., STARD) criteria, while also including a stronger emphasis on exposure and outcome assessment and discriminating accurately between groups. The QUADAS has been used in multiple reviews and meta-analyses, accumulating a substantial evidence base supporting its use (Hollingworth et al., 2006; Whiting et al., 2006).

The QUADAS distilled the 25 recommendations from the STARD into 14 key criteria that could be coded from reports of studies in order to evaluate the quality of the study and reporting, and the potential for bias to have influenced findings. Two of these 14 criteria did not appear relevant to studies of environmental chemical exposure. One criterion (QUADAS #4) pertained to the length of time between the screening and reference standard test. The intention of Criterion #4 was to capture whether the screening and reference tests were given close enough together that the person's diagnostic status was unlikely to change in between test sessions. Treating “diagnostic change over time” as a source of bias does not appear relevant for studies of environmental chemical exposure both because the central issue is not agreement between two tests and because the diagnostic status may change over time as a result of the toxicant exposure. In fact, long-term outcomes may be a central question of interest, not a source of bias in the study design (e.g., a study may examine the relationship between a toxicant level in cord blood at birth and the children's later functioning at age 10). Thus we omitted this criterion from the revised coding tool. Similarly, a second criterion (QUADAS #7) focused on whether the screening and reference tests were independent of each other. For purposes of examining neurodevelopmental effects of exposure, the independence of test results does not appear to be a central design issue (e.g., blood levels and IQ tests are independent measures).

The Evidence Based Medicine (EBM) guidelines for evaluating the quality of evidence for studies of harm (Straus et al., 2005) also provided a helpful set of questions and principles for evaluating studies of effects of chemical exposure. The final augmented set (Table 2) includes several additional criteria adapted from the EBM guidelines, as denoted in the item list. These include factors such as whether the subjects were similar in all respects except for chemical exposure, or whether the neurodevelopmental outcomes were assessed using the same methods in both the exposed and non-exposed groups (avoiding potential “work-up” bias).

Table 2.

Proposed Guidance for Evaluating Neurodevelopmental Environmental Epidemiology Studies: Harmonization of Neurodevelopmental Environmental Epidemiology Studies (HONEES)

Item	Yes	No	Unclear
Sampling & Participants
1.* Were participant selection criteria clearly described?
2.# Were there clearly defined groups of participants, similar in all important ways other than exposure to the chemical? (e.g. IQ scores, SES, age)
3.* Were the participants representative of the population to whom results would be generalized in practice?
4.* Were withdrawals from the study explained? (e.g., flow diagram, or other accounting)
Assessment Procedures
5.# Were exposures and clinical outcomes measured in the same ways in both groups? (i.e., was the assessment of outcomes either objective or blinded to exposure?)
6.# Was the follow-up of study participants sufficiently long for the outcome to occur?
7.# Did the follow-up have an acceptable level of participant retention to avoid bias?
8. Did the whole sample (or a random selection of the sample) receive testing using a “gold standard” assessment tool?
9.* Did participants receive the same assessments regardless of degree of exposure to toxicant?
10.* Was the method for exposure measurement described in enough detail to permit replication or application to new cases?
11.* Is the assessment tool likely to correctly measure or classify the target construct? (adequate diagnostic sensitivity and accuracy may be particularly important for neurodevelopmental studies)
12.* Was the neurodevelopmental assessment procedure appropriate? * Were methods described in sufficient detail to permit replication? * Was the test appropriate for the ages at which it was used? * Did the protocol avoid burden or fatigue effects that might invalidate results? (e.g., two hours might be the upper limit of length for neurodevelopmental assessments of young children; babies need to be in optimal state before testing can begin) * Are there normative data for comparison, outside of the study of environmental chemicals? * Was the administration valid, or were there major departures from standardization? (e.g., poor training, used outside of age norms, translated version without supporting psychometric data. Study must demonstrate evidence of good training, administration, and scoring and checks for continued administration validity across the length of the study by the review of video tapes or other similar measures)
13.* Was exposure status determined without knowledge of the results of the neurodevelopmental assessment?
14.* Were the neurodevelopmental assessment results interpreted without knowledge of the exposure status? (e.g., blinding of assessment administration and scoring staff; computerized administration)
15.* Was the contextual and supporting information used to interpret the test similar in the research protocol versus standard clinical practice?
16.* Were uninterpretable/ intermediate test results reported? (e.g., treatment of missing data, and reporting of borderline or midrange scores versus only extreme groups)
Interpretation and Causal Inference
17.# Do the results of the study fulfill some of the methodological tests for inferring causation?
a.# Is it clear that the exposure preceded the onset of the outcome?
b.# Is there a dose-response gradient?
c.# Is there any positive evidence from other “dechallenge-rechallenge” studies?
d.# Is the association consistent from study to study?
e.# Does the association make biological sense?

^*Adapted from Whiting et al., 2003.

^#Adapted from Straus et al., 2005.

3. Results

The product of this effort is a set of proposed considerations and recommendations (HONEES) for a process of selecting a test, evaluating its validity, and presenting findings. The main areas for evaluation of the study are Sampling and Participants, Assessment Procedures, and Interpretation and Causal Inference. In the case of the proposed considerations in Table 2, coding would be accomplished by assigning a numerical value to a “yes” or “no” response (with the response of “unclear” leading to further evaluation of the study). The emphasis on criteria and process is consistent with the philosophy of Evidence Based Medicine and we selected the parameters in Table 2 because they have been refined over more than a decade of interdisciplinary review, they focus on evaluating and quantifying the potential risk of harm, and they include study characteristics that influence estimation of risks. New evidence continues to evolve, and existing tests will continually be revised or replaced. Thus a canonical list is ultimately a less useful product as it will be static and will grow ever more dated as the field progresses. Principles and processes, on the other hand, equip the audience to seek new data, evaluate it, and decide when to change assessment practices.

4. Conclusions

The ultimate goal of conducting neurodevelopmental environmental epidemiology studies is to improve public health by providing information necessary to determine whether and how to limit exposures to specific environmental chemicals. However, even multiple, well-conducted observational studies may not provide regulatory agencies with information needed for decision-making if the methods used (including methods for measuring and expressing exposure and methods for examining associations) are not sufficiently consistent across studies. In epidemiology studies using neurodevelopmental function tests; this problem is exacerbated by the large number of available test batteries, and associated differing combinations of subtests (Youngstrom et al., 2010). This has resulted in calls for establishing consensus standards for the conduct, analysis, and reporting of epidemiologic studies (Goodman et al., 2010; Goodman, 2010; Bellinger, 2009; Moher et al., 2010).

The proposed considerations in Table 2 are offered as a means of catalyzing the conversation needed to build a set of consensus standards that can be used by the scientific community to encourage the type of inter-study harmonization necessary for improved weight-of-evidence assessments. Ultimately, for such standards to be of value, acceptance by interested parties, through round-table or consensus conferences involving key scientific and policy leaders, will be necessary. We are aware of the difficulty in achieving consensus in these types of endeavors, but opportunities exist for further interdisciplinary discussion, collaboration, and research that will help realize this goal. Broad acceptance and application of such an approach can facilitate the expanded use of environmental epidemiology studies of potential neurodevelopmental toxicants in the protection of public health, and specifically children's health.

Abbreviations

ADHD

attention deficit hyperactivity disorder

CONSORT

Consolidated Standards of Reporting of Trials

HONEES

Harmonization of Neurodevelopmental Environmental Epidemiology Studies

IRIS

Integrated Risk Information System

PCBs

polychlorinated biphenyls

QUADAS

Quality Assessment of Diagnostic Accuracy Studies

US EPA

United States Environmental Protection Agency

STARD

Standards for Reporting of Diagnostic Accuracy

STROBE

STrengthening the Reporting of OBservational studies in Epidemiology