Skip to main content
PMC home page
Environmental Health Perspectives logoLink to Environmental Health Perspectives
. 2014 Jun 25;122(10):1007–1014. doi: 10.1289/ehp.1307175

The Navigation Guide Systematic Review Methodology: A Rigorous and Transparent Method for Translating Environmental Health Science into Better Health Outcomes

Tracey J Woodruff 1,✉, Patrice Sutton 1
PMCID: PMC4181919  PMID: 24968373

Abstract

Background: Synthesizing what is known about the environmental drivers of health is instrumental to taking prevention-oriented action. Methods of research synthesis commonly used in environmental health lag behind systematic review methods developed in the clinical sciences over the past 20 years.

Objectives: We sought to develop a proof of concept of the “Navigation Guide,” a systematic and transparent method of research synthesis in environmental health.

Discussion: The Navigation Guide methodology builds on best practices in research synthesis in evidence-based medicine and environmental health. Key points of departure from current methods of expert-based narrative review prevalent in environmental health include a prespecified protocol, standardized and transparent documentation including expert judgment, a comprehensive search strategy, assessment of “risk of bias,” and separation of the science from values and preferences. Key points of departure from evidence-based medicine include assigning a “moderate” quality rating to human observational studies and combining diverse evidence streams.

Conclusions: The Navigation Guide methodology is a systematic and rigorous approach to research synthesis that has been developed to reduce bias and maximize transparency in the evaluation of environmental health information. Although novel aspects of the method will require further development and validation, our findings demonstrated that improved methods of research synthesis under development at the National Toxicology Program and under consideration by the U.S. Environmental Protection Agency are fully achievable. The institutionalization of robust methods of systematic and transparent review would provide a concrete mechanism for linking science to timely action to prevent harm.

Citation: Woodruff TJ, Sutton P. 2014. The Navigation Guide systematic review methodology: a rigorous and transparent method for translating environmental health science into better health outcomes. Environ Health Perspect 122:1007–1014; http://dx.doi.org/10.1289/ehp.1307175

Introduction

There is an urgent unmet need to shorten the time between scientific discovery and improved health outcomes. Population exposure to toxic environmental chemicals is ubiquitous [Centers for Disease Control and Prevention (CDC) 2014; U.S. Environmental Protection Agency (EPA) 2013c], and adverse health outcomes associated with exposure to such chemicals are prevalent and on the rise (Newbold and Heindel 2010; Olden et al. 2011; U.S. EPA 2013c; Woodruff et al. 2010; World Health Organization and United Nations Environment Programme 2013). The health and economic benefits of translating scientific discoveries into actions to prevent harm and reap benefits have been clearly demonstrated. For example, global efforts to remove lead from gasoline have produced health and social benefits estimated at $2.4 trillion dollars annually (Tsai and Hatfield 2011); and the value of better air quality, including reductions in premature death and illness, and improved economic welfare and environmental conditions from the programs implemented pursuant to the Clean Air Act Amendments of 1990, will reach almost $2 trillion dollars in 2020 (U.S. EPA 2011). However, many potential benefits have been squandered due to delays in acting on the available science (European Environment Agency 2013). Because of deficiencies in the current regulatory structure for manufactured chemicals, a failure or delay in acting on the science means that exposure to toxic chemicals persists while evidence of harm mounts (Vogel and Roberts 2011).

Failing or delaying to take action to prevent exposure to harmful environmental chemicals is not an inconsequential or neutral policy choice. For example, the costs in 2008 to the U.S. health care system for treatment of childhood illnesses linked to toxic environmental exposures has been estimated to be > $76 billion (Trasande and Liu 2011). Failure to prevent even low-level environmental exposures can have large society-wide adverse consequences for health if exposures are ubiquitous (Bellinger 2012).

To the extent that science informs public policy to prevent harm, a robust method to synthesize what is known about the environmental drivers of health in a transparent and systematic manner is a necessary foundational step to making the science actionable. The body of science is voluminous, of variable quality, and largely unfamiliar to decision makers. Early warning signals of harm can be masked by the fragmented, complex, and at times, conflicting nature of the available information, undermining our capacity to act wisely. Yet, consistently applied and transparent rules and descriptors about how environmental health science is translated into strength of evidence conclusions have been lacking [Beronius et al. 2010; Gee 2008; National Research Council (NRC) 2009, 2011].

Today, methods of research synthesis prevalent in environmental health mirror that of clinical medicine > 40 years ago when the clinical sciences largely relied on a system of expert-based narrative reviews on which to recommend treatment decisions (Rennie and Chalmers 2009). In a landmark paper published in 1992 in the Journal of the American Medical Association, Antman et al. (1992) showed the superiority of systematic review methods by comparing expert opinion-based recommendations for treatment of myocardial infarction published in scientific reviews and clinical textbooks to statistical analyses of the combined results of randomized controlled trials. Antman et al. documented the lack of timely incorporation of experimental evidence into expert-based recommendations and showed that some expert reviews did not mention effective therapies, whereas others recommended therapies proven to be ineffective or even dangerous. From there, explicit approaches that harness expertise to a rigorous, transparent, and systematic methodology to evaluate a clearly formulated question were advanced, and are now embodied in prominent empirically demonstrated methods such as the Cochrane Collaboration (Higgins and Green 2011) and Grading of Recommendations Assessment, Development and Evaluation (GRADE) (Guyatt et al. 2008b). These methods are regularly relied on to inform decisions on billions of dollars of health care in order to achieve cost savings and better health outcomes (Fox 2010). Howells et al. (2012) estimated that utilization of systematic review and meta-analysis of the preclinical evidence (i.e., animal studies undertaken prior to human drug trials) could reduce the cost of developing drugs for treating stroke by $1.1–7.9 billion, the savings due to improving the validity of the evidence informing decisions on whether to advance drugs to clinical trials. It is anticipated that U.S. health care policy decisions will increasingly rely on systematic review methodologies; for example, health care reform legislation has allocated $1.1 billion dollars for comparative effectiveness research (CDC 2009).

The field of environmental health is now embarking on a similar journey. Reviews of the scientific evidence are as integral to decision making about exposure to environmental chemicals in national and local government agencies and industry as they are for making treatment decisions in clinical medicine. However, predominant approaches in use for evaluating the evidence in environmental health are > 30 years old, based on expert opinion, and with notable exceptions (Department of Health and Human Services 2006; National Toxicology Program 2013; U.S. EPA 2013b) generally do not provide strength of evidence summaries for outcomes other than cancer. Improved methods of risk assessment that better reflect our current understanding of the science have been articulated by the National Academy of Sciences in Phthalates and Cumulative Risk Assessment: The Task Ahead (NRC 2008) and in Science and Decisions: Advancing Risk Assessment (NRC 2009). Systematic approaches to evidence-based decision making that can improve our capacity to meet the needs of decision makers are also currently under way at the National Toxicology Program (Rooney et al. 2014) and under consideration at the U.S. EPA (NRC 2011, 2014a, 2014b). Described below are the results of the application of a novel method for systematic and transparent review in environmental health that demonstrate that such advances are not only desirable but within our grasp.

Discussion

Overview of the Navigation Guide Methodology

With the goal of expediting the development of evidence-based recommendations for preventing harmful environmental exposures, beginning in 2009 a collaboration of scientists and clinicians undertook the development of the Navigation Guide methodology for systematic review. The Navigation Guide methodology was developed by coupling the rigor of systematic review methods being used by the clinical sciences to the “bottom line” approach to research synthesis being used by the International Agency for Research on Cancer (IARC 2006). Features of systematic reviews used in clinical medicine encompass specifying an explicit study question, conducting a comprehensive search, rating the quality and strength of the evidence according to consistent criteria, and performing meta-analyses and other statistical analyses. IARC’s method allows for combining the results of human and nonhuman evidence into a single concise statement of health hazard (Woodruff et al. 2011).

As such, the Navigation Guide methodology translates the achievements of the past 20 years in evidence-based medicine into environmental health.

The Navigation Guide methodology involves four steps:

  • Specify the study question: Frame a specific question relevant to decision makers about whether human exposure to a chemical or class of chemicals or other environmental exposure is a health risk.

  • Select the evidence: Conduct and document a systematic search for published and unpublished evidence.

  • Rate the quality and strength of the evidence: Rate the quality of individual studies and the quality of the overall body of evidence based on prespecified and transparent criteria. The Navigation Guide methodology conducts this process separately for human and nonhuman systems of evidence. As a consequence, the methodology involves an additional step of integrating the quality ratings of each of these two streams of evidence. The end result is one of five possible statements about the overall strength of the evidence: “known to be toxic,” “probably toxic,” “possibly toxic,” “not classifiable,” or “probably not toxic.”

  • Grade the strength of the recommendations.

We were part of a team of scientists that developed the Navigation Guide method and applied steps 1–3 to the question “does developmental exposure to perfluorooctanoic acid (PFOA) affect fetal growth?” (Johnson et al. 2014; Koustas et al. 2014; Lam et al. 2014). Step 4 of the method, “grade the strength of the recommendations,” involves integrating the strength of the evidence on toxicity (from step 3) with information about exposure, the availability of less toxic alternatives, and patient values and preferences. This step was not addressed in the PFOA case study because of the limitations of our resources. Below we highlight the features of the method that are new to environmental health, features that differ from methods used in evidence-based medicine, a comparison of the results of the Navigation Guide method to previous reviews of PFOA exposure and toxicity, limitations of the Navigation Guide method, and future directions.

Navigation Guide Features New to Environmental Health Reviews

To initiate the development of the Navigation Guide methodology, we convened a novel interdisciplinary team of 22 individuals from governmental and nongovernmental organizations and academia (Woodruff et al. 2011). Two members of this team, Daniel Fox (President Emeritus of the Milbank Memorial Fund) and Lisa Bero (currently Co-Chair of the Cochrane Collaboration) were world-renowned experts on systematic review methodologies used in the clinical sciences. Seven members were scientists or environmental health advocates from international, national, state, and local government agencies and a nongovernmental organization directly engaged in developing and/or employing strength-of-evidence conclusions in decision making on environmental chemicals: David Gee (European Environmental Agency), Vincent James Cogliano (IARC), Kathryn Guyton (U.S. EPA), Lauren Zeise (California Environmental Protection Agency), Julia Quint (California Department of Public Health, retired), Karen Pierce (San Francisco Department of Public Health), and Heather Sarantis (Commonweal). Eleven were health professionals with expertise in women’s, reproductive, pediatric, and/or environmental health: Jeanne Conry (American Congress of Obstetricians and Gynecologists District IX and Kaiser Permanente), Mark Miller (UCSF Pediatric Environmental Health Specialty Unit), Sarah Janssen (Natural Resources Defense Council), Beth Jordon and Rivka Gordon (Association of Reproductive Health Professionals), Sandy Worthington (Planned Parenthood Federation of America), Pablo Rodriguez (Brown Medical School and Women & Infants Hospital of Rhode Island), Michelle Ondeck and Judith Balk (University of Pittsburgh), Victoria Maizes (University of Arizona), and Ted Schettler (Science and Environmental Health Network). Finally, our own expertise has involved decades of work at the interface of environmental and occupational health and public policy. At the time of publication of the method, none of the collaborators reported a competing financial interest.

To conduct the first application of the Navigation Guide method, we assembled a team of nine scientists from academia and the U.S. EPA that encompassed the multidisciplinary expertise required to apply the methodology, including in environmental health sciences, epidemiology, toxicology, risk assessment, biostatistics, and the science of systematic reviews (Johnson et al. 2014; Koustas et al. 2014; Lam et al. 2014). One team member, Karen Robinson (Director of the Evidence Based Practice Center at Johns Hopkins University), was an expert on the identification, synthesis, and presentation of evidence for informing health care decisions and research; three team members, Patrice Sutton, Erica Koustas, and Paula Johnson had formal training in Cochrane and/or GRADE methodologies. None of the review team reported a competing financial interest.

The method developed and applied through these interdisciplinary teams builds on the best practices in research synthesis in evidence-based medicine and environmental health. Key points of departure of the Navigation Guide from current methods of expert-based narrative reviews in environmental health include the following.

1. A protocol. The application of the Navigation Guide is guided by a detailed protocol developed prior to undertaking the review (Figure 1). In contrast, expert-based narrative review methods do not provide a document that predefines a specific question to be answered and sets up the “rules” of the evaluation. A predefined protocol is a staple of systematic reviews in the clinical sciences because it reduces the impact of review authors’ biases, provides for transparency of methods and processes, reduces the potential for duplication, and allows for peer review of the planned methods (Higgins and Green 2011). Notably, the protocol also provides a transparent forum to incorporate the expertise of nonscientists, including health-impacted populations and their advocates, in framing a meaningful study question. The protocol is developed around a “PECO” statement [participants, exposure, comparator, and outcome(s)], which provides the framework from which studies are identified and selected for inclusion. The PECO statement is similar to recommendations by the National Academy of Sciences for improving the design of risk assessment through planning, scoping, and problem formulation to better meet the needs of decision makers (NRC 2009).

Figure 1.

Figure 1

Open in a new tab

Steps in the Navigation Guide protocol. PECO, participants, exposure, comparator, and outcome(s).

2. Standardized and transparent documentation including expert judgment. Systematic reviews are not “automated” or “computerized” or otherwise conducted without applying judgment (Guyatt et al. 2011). The fundamental shift from existing methods of expert review in environmental health science is that each step of the Navigation Guide is conducted in a thorough, consistent, and transparent manner, and all information, including judgments, is documented and displayed in the same way. In short, the rationale for a decision is traceable, reproducible, and comprehensible.

3. Assessment of “risk of bias.” The assessment of “risk of bias,” defined as characteristics of a study that can introduce systematic errors in the magnitude or direction of the results (Higgins and Green 2011), is a new concept in environmental health. Systematic review methodologies distinguish between study-quality criteria that can introduce a systematic error in the magnitude or direction of the result (i.e., risk of bias or “internal validity”) from other methodological quality or reporting elements, which are related to important standards by which a study is conducted (e.g., adherence to human subjects and animal welfare requirements) or reported (e.g., complete information provided), but that do not systematically influence study outcomes. A study conducted to the highest methodological standards can still have important risk of bias that will affect the magnitude or direction of a study outcome.

Risk of bias domains have been well developed and empirically shown to influence study outcomes in experimental human studies (Higgins et al. 2011; Roseman et al. 2011). However, risk of bias domains that are equally agreed upon for human observational studies are lacking. In the PFOA case study, we based our risk of bias domains for observational human studies on the domains used by the Cochrane Collaboration and the Agency for Healthcare Research and Quality (Higgins and Green 2011; Viswanathan et al. 2012), including recruitment strategy, blinding, confounding, incomplete outcome data, selective reporting, and exposure assessment.

Domains for risk of bias for animal studies are also under development. Although 30 instruments have been identified in the environmental health literature for evaluating the quality of animal studies, they are mostly composed of domains related to reporting requirements, such as compliance with regulatory requirements, description of the statistical model, and test animal details; importantly, they do not include all the risk of bias domains in use in human experimental studies (Krauth et al. 2013).

To develop risk of bias domains for applying the Navigation Guide to animal studies, we adapted the risk of bias domains used in human experimental studies that have an empirical basis, including a) sequence generation, b) allocation concealment, c) blinding, d) incomplete outcome data, and e) selective reporting [see Figure 1 in Johnson et al. (2014) and Figure 1 in Koustas et al. (2014)]. According to GRADE, these five criteria address nearly all issues that bear on the quality of human experimental evidence (Balshem et al. 2011). Further, these elements have been shown in the preclinical animal literature to influence study outcomes (Vesterinen et al. 2010). Our rationale was that risk of bias in a nonhuman experiment is comparable to risk of bias in human and preclinical animal experiments.

Further, in both human and animal studies, we included a “conflict of interest” risk of bias domain. This domain has been proposed—but not yet adopted—by Cochrane and GRADE as an important risk of bias (Bero 2013). This is based on empirical data from studies of the health effects of tobacco (Barnes and Bero 1997, 1998), the safety and efficacy of pharmaceuticals (Bero et al. 2007; Lexchin et al. 2003; Lundh et al. 2012), and medical procedures (Popelut et al. 2010; Shah et al. 2005), which have all shown that, on average, source of funding influences study outcome.

The assessment of risk of bias in the PFOA case study revealed worrisome truths about the conduct and reporting of experimental animal studies in environmental health. In particular, we found that included toxicological studies uniformly did not apply methodological approaches that are empirically recognized as minimizing bias in human experimental study outcomes. In particular, none of the studies reported how or if they used adequate allocation concealment, regardless of whether the studies were conducted through Good Laboratory Practices (GLP), by industry groups, or by independent research laboratories. Suboptimal experimental animal study design and reporting is prevalent in the preclinical literature, and introduces bias into study findings (Bebarta et al. 2003; Landis et al. 2012; Macleod et al. 2004; McPartland et al. 2007; van der Worp and Macleod 2011; van der Worp et al. 2007; Vesterinen et al. 2011). For example, studies by the Collaborative Approach to Meta-Analysis and Review of Data from Experimental Studies (CAMARADES) collaboration have shown that studies using randomization and allocation concealment reported less improvement in heart response measures in animal models of focal ischemia treated with the pharmaceutical NXY059 (Macleod et al. 2008) and less improvement in neurobehavioral scores in animal models of intracerebral hemorrhage (Frantzias et al. 2011) than other studies.

In our outreach efforts related to the Navigation Guide, we found that environmental health researchers in many and varied settings reported that methodological approaches to reduce bias in toxicological studies were not widely recognized or were not customary practices.

A second challenge to conducting risk of bias assessments and quantitative analyses in the PFOA case study was that the necessary data were not all reported in the published studies. Our efforts to contact study authors to get the needed data were moderately successful [i.e., 18 of 28 (64%) authors that were contacted responded] and were critical to our ability to conduct the review. We anticipate that contacting study authors will be a necessary step for those conducting systematic reviews until such time that steps are undertaken—by journals, funding agencies, and through study registries—to standardize optimal reporting. Our findings underscore the urgency of calls for improved access to the data needed to conduct scientifically robust reviews of environmental health science (Goldman and Silbergeld 2013) and the importance to environmental health of nascent efforts in the preclinical arena to develop improved experimental animal study design and reporting (Landis et al. 2012; van der Worp and Macleod 2011; Vesterinen et al. 2011).

4. Comprehensive and efficient search strategy. The outcome of the Navigation Guide search method demonstrated the potential for systematic reviews to be more comprehensive than traditional reviews. We evaluated four more human studies than did an expert panel appointed to review the health effects of PFOA (C8 Science Panel 2011). The search strategy used to gather data for the C8 panel was not published. However, because these four papers did not present data that proved to be essential to the conclusions of the review (i.e., the data included were from small studies that did not weight heavily in the meta-analysis), they could have been identified by the C8 Panel’s search but excluded from their reference list. Our comprehensive search strategy captured studies that measured PFOA exposure and fetal growth parameters but did not necessarily draw associations between the two. The four additional studies included in our review did not have birth weight or other fetal growth measures as the primary outcome or main topic of the paper (Fromme et al. 2010; Kim S et al. 2011; Kim SK et al. 2011; Wang et al. 2011). However, because our search identified these studies, we included them, contacted the study authors, and obtained additional relevant data to support our review from authors of two of these studies (Fromme et al. 2010; Kim S et al. 2011) and were referred by one author (Wang et al. 2011) to an article under peer review at the time on the same cohort with more relevant data (Chen et al. 2012). We also identified 10 more nonhuman studies than were included in our own earlier nonsystematic literature review. Our adoption of a search filter for animal studies in use in the preclinical literature (Hooijmans et al. 2010) greatly expedited the development of a search for relevant animal studies.

We found that casting a wide net for relevant studies was feasible because of the development of a PECO statement from which we developed very explicit criteria used to efficiently screen titles and abstracts and because of the use of a software program that expedited the screening process. For this case study, our search strategy identified slightly more than 2,000 nonhuman and 3,000 human potentially relevant studies. For the human data, it took 1 person-day to screen titles and abstracts (resulting in 248 articles eligible for full text review) and 1 week to do a full text review, which identified 18 relevant studies for evaluation. The time for the evaluation of the nonhuman data was similar.

Further, by applying a method that seeks to extract the exact same information, laid out in the same transparent way, our ability to interpret and understand the results was straightforward. As the application of systematic reviews expands, we anticipate greater efficiencies will be gained, for example, through the development of improved search filters and screening and management systems.

5. Separation of the science from values and preferences. The PFOA case study demonstrated steps 1–3 of the Navigation Guide methodology, the result of which was a concise statement regarding PFOA’s toxicity. However, toxicity is just one aspect of a risk management decision in environmental health. In step 4 of the Navigation Guide, which is modeled after GRADE’s methods for rating treatment recommendations (Guyatt et al. 2008a), other important factors are brought to bear on recommendations for prevention, including values and preferences, extent of exposures, the availability of safer alternatives, and costs and benefits. Thus, the Navigation Guide transparently and explicitly delineates the science from other key considerations. Although we did not have the resources to operationalize step 4 in the PFOA case study, we hope to do so in future case studies.

Navigation Guide Features Different from Evidence-Based Medicine

Because of differences between environmental and clinical health sciences related to the evidence base and decision context, systematic review methodologies used in the clinical sciences were not seamlessly applicable to environmental exposures (Woodruff et al. 2011). Two key points of departure of the Navigation Guide methodology from evidence-based medicine are as follows.

1. The body of human observational studies is assigned a “moderate” quality rating. The Navigation Guide assigns a priori a “moderate” quality rating to the body of human observational evidence. This initial quality rating of “moderate” is independent of the specifics of the studies in the assessment. The actual quality of the body of human observational studies is then accounted for through upgrading or downgrading the “moderate” rating based on a priori criteria. In contrast, systematic reviews in the clinical sciences, which proceed from the availability of human experimental evidence, assign an a priori rating to the body of human observational studies of “low” quality. In particular, Cochrane and GRADE have been developed primarily based on evaluation of randomized controlled clinical trials (RCTs), and in this context, relative to RCTs, GRADE considers human observational studies to be “low”-quality evidence (Balshem et al. 2011).

Our rationale to assign the body of human observational studies a rating of “moderate” and not “low” quality was based on the absolute and relative merit of human observational data in evidence-based decision making in environmental and clinical health sciences. Overall, human observational studies are recognized as being a reliable source of evidence in the clinical sciences because not all health care decisions are, or can be, based on RCTs. The contribution of observational studies to certain health care decisions is underscored by the conclusion of a 2008 Institute of Medicine (IOM) panel, which found observational studies to be the preferred method for evaluating the causes of disease, which would include the contribution of environmental agents. The IOM panel noted that observational and experimental studies each can provide valid and reliable evidence, with their relative value dependent on the clinical question (IOM 2008). In this context, the IOM report (IOM 2008) stated that

Observational studies are generally the most appropriate for answering questions related to prognosis, diagnostic accuracy, incidence, prevalence, and etiology.

Moreover, recognition of the absolute value of human observational data to evidence-based clinical decision making is increasing. There are several reasons for this. For example, the speed and complexity with which new medical interventions and scientific knowledge are being created make it unlikely that the evidence base required for treatment and cost-effective health care delivery across subpopulations can be built using only RCTs (Peterson 2008). It is also expected that electronic medical records will revolutionize medical research by facilitating comprehensive, longitudinal observational data in an instant (Halvorson 2008). Finally, ethical considerations virtually preclude experimental human data from the environmental health evidence stream. Therefore, relative to the evidence available for decision making in environmental health, human observational studies are the “gold standard” of the evidence base.

2. Diverse evidence streams are combined. In vitro, in vivo, in silico, and human observational studies all inform decision making on environmental chemical exposures. However, there is currently no agreed-upon standard method in clinical medicine for evaluating evidence simultaneously across disparate evidence streams. We therefore adapted a mixture of IARC’s method for integrating human and nonhuman evidence (IARC 2006) linked to strength of evidence descriptions in use by the U.S. EPA (1991, 1996). Although this transparently produced a clear, concise, and recognizable bottom line (i.e., “known to be toxic,” “probably toxic,” “possibly toxic,” “not classifiable,” or “probably not toxic”), further development of precise criteria, definitions, and nomenclature for strength of evidence that meets the needs of a wide range of decision makers will be an important undertaking as uptake of methodology moves forward.

Comparison of the Navigation Guide Method to Previous Reviews of PFOA and Fetal Growth

The authors of the review conducted with the Navigation Guide methodology concluded that “developmental exposure to PFOA adversely affects human health based on sufficient evidence of decreased fetal growth in both human and nonhuman mammalian species” (Lam et al. 2014). To compare these results to previous reviews, we searched PubMed (http://www.ncbi.nlm.nih.gov/pubmed) without date or language restrictions for reviews of “PFOA” or “pefluorooctanoic acid.” Of the 48 papers identified, 12 included discussions of reproductive or developmental health. Two additional reviews (Butenhoff et al. 2006; Stahl et al. 2011) were identified at the time we were embarking on this project, and we also included those publications. Of 14 reviews, all but 1 (Stahl et al. 2011), which was not indexed in PubMed, were also identified by our search strategy for the PFOA case study (Johnson et al. 2014).

Table 1 compares the 14 reviews of PFOA exposure and toxicity identified by our search to seven key features of systematic and transparent review methods, that is, Cochrane and GRADE. All 14 reviews were conducted using nonsystematic, expert-based narrative methods. Of the 14 reviews, 13 defined a study question, 9 included a summary of findings table, 3 specified criteria for included studies, 2 included limited information about their search strategy, 2 conducted data analysis, and 1 assessed the quality of individual studies. None of the 14 reviews systematically or transparently assessed risk of bias for individual studies, and none integrated human and nonhuman evidence to produce an overall summary of the strength of the evidence (Butenhoff et al. 2004; DeWitt et al. 2009; Hekster et al. 2003; Jensen and Leffers 2008; Kennedy et al. 2004; Kudo and Kawashima 2003; Lau et al. 2004, 2007; Lindstrom et al. 2011; Olsen et al. 2009; Post et al. 2012; Stahl et al. 2011; Steenland et al. 2010; White et al. 2011). These 14 reviews either produced vague or indeterminate answers to the question of PFOA’s toxicity, or presented a clear answer (i.e., “PFOA is a known developmental toxicant”) (White et al. 2011) without specifying the search methods, study inclusion criteria, or statistical methods that produced the answer. Our comparison of the methods and results of these narrative reviews to the Navigation Guide method demonstrated that the application of the Navigation Guide provided more transparency about the steps taken in the review and a consistent path to a clear answer compared with the methods of expert-based narrative review that are currently employed in environmental health. Our results demonstrated that improved methods of research synthesis under development at the National Toxicology Program (Birnbaum et al. 2013; Rooney et al. 2014) and under consideration by the U.S. EPA (NRC 2011, 2014a, 2014b; U.S. EPA 2013a) are fully achievable.

Table 1.

Comparison of the PFOA reviews’ methods according to key features of Cochrane and GRADE systematic and transparent review methods.

Reference Specify study question Specify inclusion/exclusion criteria Conduct reproducible search Assess “risk of bias” Data analysis and/or meta-analyses Summary of findings table Assess quality and strength of body of evidence
Navigation Guide PFOA case study 2014a Yes Yes Yes Yes Yes Yes Yes
Post et al. 2012 Yes No No No Some data analysis (BMD, BMDL) Yes No
Lindstrom et al. 2011 Yes No No No No No No
Stahl et al. 2011 Yes No No No No Yes No
White et al. 2011 Yes No No No No Yes No
Steenland et al. 2010 Yes No No No No Yes No
DeWitt et al. 2009 Yes No No No No No No
Olsen et al. 2009 Yes Inclusion criteria No No No Yes Assess methodological weaknesses of included studies
Jensen and Leffers 2008 No No No No No No No
Lau et al. 2007 Yes No No No No Yes No
Butenhoff et al. 2004 Yes Yes No No Some data analysis (MOE, LBMIC10) Yes No
Kennedy et al. 2004 Yes No No No No Yes No
Lau et al. 2004 Yes No Limited discussion of literature search No No No No
Hekster et al. 2003 Yes Some inclusion criteria described in cited report by same authors Limited discussion of literature search No No Yes No
Kudo and Kawashima 2003 Yes No No No No No No
Abbreviations: BMD, benchmark dose; BMDL, BMD lower confidence limit; LBMIC10, lower 95% confidence limit of a modeled 10% response; MOE, margin of exposure. aData presented by Johnson et al. (2014), Koustas et al. (2014), and Lam et al. (2014).
Open in a new tab

Limitations

A limitation of the Navigation Guide systematic review method is that although its overall architecture is based on empirically proven and/or time-tested methods (i.e., methods in use by Cochrane, GRADE, IARC, and the U.S. EPA), novel aspects of the method need further development and validation, including a) rating the quality and strength of nonmammalian animal and in vitro and in silico evidence streams; b) reaching a consensus on risk of bias domains for human observational studies and nonhuman studies; c) developing well-defined, measurable evidentiary bars for the factors used to downgrade the quality of environmental health evidence (i.e., indirectness, inconsistency, imprecision, and publication bias) and for upgrading human evidence (i.e., dose response, large magnitude of effect, and confounding minimizes effect); and e) exploring whether it makes a difference to the final quality rating if we assign the entire body of human observational studies a “moderate” rating and then downgrade for lesser quality study designs, or, as proposed in the NTP’s framework, we assign different types of human observational studies different ratings from the start (i.e., cross-sectional studies, case–control studies, and case series or reports are rated as “low” quality, and cohort and nested case–control studies are rated as “moderate” quality) (Rooney et al. 2014). Improved statistical tools for data analysis and integration will also advance the application of systematic review methods in environmental health. Whether the use of our nomenclature for the final strength of evidence ratings (i.e., “known to be toxic,” “possibly toxic,” and so on) will be useful to decision makers is also untested, and consensus methods for classifying strength of evidence for noncancer health outcomes is a critical research and policy need (Gee 2008).

In addition, the application of the Navigation Guide method—just like any expert-based narrative review—can be poorly executed. For example, a systematic review can be conducted that does not specify a study question relevant to decision making, or an incomplete search strategy can fail to uncover information pertinent to the review. However, a poorly performed systematic review is more readily detected because the methods are transparently displayed.

The capacity for improved methods of research synthesis in environmental health to spur timely health protective decision making is also limited by the shortcomings of the available evidence stream that is produced by current systems of generating scientific knowledge. One key example is the need for an unconflicted underlying evidence stream. As the Deputy Editor (West) of JAMA observed in 2010, “the biggest threat to [scientific] integrity [is] financial conflicts of interest” (Rennie 2010). Moreover, risk of bias assessments leave unaddressed the inherent biases in environmental health science methodologies that generate false negatives and rely on strength of evidence criteria that are unequal to the task of addressing complex and multicausal disease etiologies (Gee 2008). Finally, there are many other formidable nonscientific, social, and political barriers to prevention-oriented action (European Environment Agency 2013; Michaels 2008).

Future Directions

Shortening the time between scientific discovery and the prevention of exposures to toxic environmental chemicals is inextricably linked to the success of private and public sector efforts to advance safer and sustainable alternatives to toxic chemicals. The assessment of toxicity is an essential underpinning of such efforts (Edwards 2009; Malloy et al. 2013; Matus et al. 2012; Park et al. 2014; U.S. EPA 2012). As such, the Navigation Guide methodology has broad applicability to support efforts by businesses, governments, and consumers to compare and choose among various chemicals using a standardized and rigorous method.

As in the clinical application of systematic reviews, development of systematic and transparent methods of research synthesis in environmental health will be an ongoing process. Some immediate methodological needs relate to how to routinely integrate critical concepts into the interpretation of data, including low-dose effects, concordance in response across species, and human variability (including age and comorbidities). These issues were considered in the PECO question and statistical analyses of the PFOA case study, but a more thorough and overarching framework for how to integrate these concepts in systematic reviews is still needed. For example, failure to use animals with clinically relevant comorbidities, such as hypertension in stroke models, has been shown to bias the assessment of drug efficacy (Macleod et al. 2008; Sena et al. 2010), and we would expect that including animals with chronic conditions may affect findings for environmental chemicals. Robust methods to assess publication bias in environmental health science are also a need, because researchers can have financial and or other conflicts that can promote bias in opposite directions.

Uptake of methods of systematic and transparent review represents a new way of doing business in environmental health sciences. A realistic starting place is to recognize the potential for many or all of the challenges related to using systematic reviews in clinical medicine (i.e., perceived threats to physician autonomy, patient choice, and so on) to become our challenges. We will need to overcome a lack of knowledge of environmental health science and research synthesis methods by every key target audience. The application of systematic reviews in environmental health is inherently an interdisciplinary “team science” undertaking, and success will require formalizing the necessary expertise and assembling and training review teams in these new methods and relevant communication skills.

Conclusion

Systematic and transparent methods of research synthesis are empirically based and can serve as a roadmap to more efficient and transparent decision making using the available data. The use of systematic review methods allows decision makers to act on any quality of evidence and in any direction. Moreover, the use of systematic reviews can prevent wasteful expenditures on studies that are duplicative or otherwise unnecessary for decision making (Chalmers and Glasziou 2009).

In his 1965 address to the Royal Society of Medicine, Sir Austin Bradford Hill, the statistician who pioneered the RCT, admonished his audience that while science is always incomplete and subject to change,

[it] does not confer upon us a freedom to ignore the knowledge we already have, or to postpone the action that it appears to demand at a given time. (Hill 1965)

Hill (1965) emphasized that “strong evidence” does not imply “crossing every ‘t’, and swords with every critic, before we act.” He proposed differential standards of evidence for different actions, a recommendation echoed by the National Academy of Sciences a half-century later in Science and Decisions: Advancing Risk Assessment (NRC 2009).

Because systematic review methods transparently distinguish between science, values, and preferences, they can help sharpen the terms of debates regarding whether we strive for more precision or more decisions about the meaning of the science to health.

This first case study of the Navigation Guide methodology demonstrated the successful application of a systematic and rigorous method for research synthesis designed to optimize transparency and reduce bias in the evaluation of environmental health information. Government agencies can use the Navigation Guide methodology to craft evidence-based statements regarding the relationship between an environmental exposure and health (steps 1–3). Government agencies called on to make risk management decisions can also apply step 4 of the Navigation Guide to grade the strength of recommendations for prevention. Professional societies, health care organizations, and other potential guideline developers working with toxicologists can use the Navigation Guide to craft consistent and timely recommendations to improve patient, and ultimately population, health outcomes (steps 1–4). The institutionalization of robust methods of systematic and transparent review would provide a concrete mechanism for linking science to timely action to prevent harm. Although simple in concept, Navigation Guide methodology will require sustained visionary leadership harnessed to substantive investment as well as the intellectual curiosity and commitment of environmental and clinical health scientists and advocates.

Acknowledgments

We are indebted to D. Atchley, D. Axelrad, L. Bero, P. Johnson, E. Koustas, and J. Lam for providing invaluable comments and suggestions on this commentary. D. Atchley also provided research assistance.

Footnotes

Funding for this commentary was provided to the University of California, San Francisco (UCSF) Program on Reproductive Health and the Environment by the Clarence Heller Foundation, the Forsythia Foundation, the Fred Gellert Family Foundation, the Passport Foundation, and the New York Community Trust. Financial support for development of the perfluorooctanoic acid (PFOA) case study was through grants from the New York Community Trust and the U.S. Environmental Protection Agency (EPA) through a contract with Abt Associates (GAIA-0-6-UCSF 17288). For 2009–2013, support for the development and dissemination of the Navigation Guide methodology was provided by the Clarence Heller Foundation, the Passport Foundation, the Forsythia Foundation, the Johnson Family Foundation, the Heinz Endowments, the Fred Gellert Foundation, the Rose Foundation, Kaiser Permanente, the New York Community Trust, the Philip R. Lee Institute for Health Policy Studies, the Planned Parenthood Federation of America, the National Institute of Environmental Health Sciences (ES018135 and ESO22841), and U.S. EPA STAR grants (RD83467801 and RD83543301).

The contents of this paper are solely the responsibility of the authors and do not necessarily represent the offi­cial views of the U.S. EPA. Further, the U.S. EPA does not endorse the purchase of any commercial products or services mentioned in the publication.

The authors declare they have no actual or potential competing financial interests.

References

  1. Antman EM, Lau J, Kupelnick B, Mosteller F, Chalmers TC. A comparison of results of meta-analyses of randomized control trials and recommendations of clinical experts. Treatments for myocardial infarction. JAMA. 1992;268:240–248. [PubMed] [Google Scholar]
  2. Balshem H, Helfand M, Schunemann HJ, Oxman AD, Kunz R, Brozek J, et al. Grade guidelines: 3. Rating the quality of evidence. J Clin Epidemiol. 2011;64:401–406. doi: 10.1016/j.jclinepi.2010.07.015. [DOI] [PubMed] [Google Scholar]
  3. Barnes DE, Bero LA. Scientific quality of original research articles on environmental tobacco smoke. Tob Control. 1997;6:19–26. doi: 10.1136/tc.6.1.19. [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Barnes DE, Bero LA. Why review articles on the health effects of passive smoking reach different conclusions. JAMA. 1998;279:1566–1570. doi: 10.1001/jama.279.19.1566. [DOI] [PubMed] [Google Scholar]
  5. Bebarta V, Luyten D, Heard K. Emergency medicine animal research: does use of randomization and blinding affect the results? Acad Emerg Med. 2003;10:684–687. doi: 10.1111/j.1553-2712.2003.tb00056.x. [DOI] [PubMed] [Google Scholar]
  6. Bellinger DC. Comparing the population neurodevelopmental burdens associated with children’s exposures to environmental chemicals and other risk factors. Neurotoxicology. 2012;33:641–643. doi: 10.1016/j.neuro.2012.04.003. [DOI] [PubMed] [Google Scholar]
  7. Bero L. Why the Cochrane risk of bias tool should include funding source as a standard item [Editorial]. Cochrane Database Syst Rev. 2013;12:ED000075. doi: 10.1002/14651858.ED000075. [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Bero L, Oostvogel F, Bacchetti P, Lee K.2007Factors associated with findings of published trials of drug–drug comparisons: why some statins appear more efficacious than others. PLoS Med 4e184; 10.1371/journal.pmed.0040184 [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Beronius A, Rudén C, Håkansson H, Hanberg A. Risk to all or none?: A comparative analysis of controversies in the health risk assessment of bisphenol A. Reprod Toxicol. 2010;29:132–146. doi: 10.1016/j.reprotox.2009.11.007. [DOI] [PubMed] [Google Scholar]
  10. Birnbaum LS, Thayer KA, Bucher JR, Wolfe MS.2013Implementing systematic review at the National Toxicology Program: status and next steps [Editorial]. Environ Health Perspect 121A108–A109.; 10.1289/ehp.1306711 [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Butenhoff JL, Gaylor DW, Moore JA, Olsen GW, Rodricks J, Mandel JH, et al. Characterization of risk for general population exposure to perfluorooctanoate. Regul Toxicol Pharmacol. 2004;39:363–380. doi: 10.1016/j.yrtph.2004.03.003. [DOI] [PubMed] [Google Scholar]
  12. Butenhoff JL, Olsen GW, Pfahles-Hutchens A.2006The applicability of biomonitoring data for perfluorooctanesulfonate to the environmental public health continuum. Environ Health Perspect 1141776–1782.; 10.1289/ehp.9060 [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. C8 Science Panel. Probable Link Evaluation of Preterm Birth and Low Birthweight. 2011. Available: http://www.c8sciencepanel.org/pdfs/Probable_Link_C8_Preterm_and_LBW_birth_5Dec2011.pdf [accessed 5 December 2011]
  14. CDC (Centers for Disease Control and Prevention). Description of Funded Activities. 2009. Available: http://www.cdc.gov/fmo/topic/recovery_act/ [accessed 6 February 2014]
  15. CDC (Centers for Disease Control and Prevention). Fourth National Report on Human Exposure to Environmental Chemicals, Updated Tables. 2014. Available: http://www.cdc.gov/exposurereport/pdf/FourthReport_UpdatedTables_Aug2014.pdf [accessed 28 August 2014]
  16. Chalmers I, Glasziou P. Avoidable waste in the production and reporting of research evidence. Lancet. 2009;374:86–89. doi: 10.1016/S0140-6736(09)60329-9. [DOI] [PubMed] [Google Scholar]
  17. Chen MH, Ha EH, Wen TW, Su YN, Lien GW, Chen CY, et al. 2012Perfluorinated compounds in umbilical cord blood and adverse birth outcomes. PLoS One 78e42474; 10.1371/journal.pone.0042474 [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Clean Air Act Amendments of 1990. 1990. Public Law 101–549.
  19. Department of Health and Human Services. The Health Consequences of Involuntary Exposure to Tobacco Smoke: A Report of the Surgeon General. Atlanta, GA:Department of Health and Human Services, Centers for Disease Control and Prevention, Coordinating Center for Health Promotion, National Center for Chronic Disease Prevention and Health Promotion, Office on Smoking and Health. 2006. Available: http://www.surgeongeneral.gov/library/reports/secondhandsmoke/fullreport.pdf [accessed 20 August 2014] [PubMed]
  20. DeWitt JC, Shnyra A, Badr MZ, Loveless SE, Hoban D, Frame SR, et al. Immunotoxicity of perfluorooctanoic acid and perfluorooctane sulfonate and the role of peroxisome proliferator-activated receptor alpha. Crit Rev Toxicol. 2009;39:76–94. doi: 10.1080/10408440802209804. [DOI] [PubMed] [Google Scholar]
  21. Edwards S. A New Way of Thinking: The Lowell Center Framework for Sustainable Products. Lowell, MA:Lowell Center for Sustainable Production, University of Massachusetts Lowell. 2009. Available: http://www.sustainableproduction.org/downloads/LowellCenterFrameworkforSustainableProducts11-09.09.pdf [accessed 20 August 2014]
  22. European Environment Agency. Late Lessons from Early Warnings: Science, Precaution, Innovation. EEA Report No. 1/2013. 2013. Available: http://www.eea.europa.eu/publications/late-lessons-2 [accessed 28 August 2014]
  23. Fox DM. Berkeley, CA: University of California Press; 2010. The Convergence of Science and Governance: Research, Health Policy, and American States. [Google Scholar]
  24. Frantzias J, Sena ES, Macleod MR, Al-Shahi Salman R. Treatment of intracerebral hemorrhage in animal models: meta-analysis. Ann Neurol. 2011;69:389–399. doi: 10.1002/ana.22243. [DOI] [PubMed] [Google Scholar]
  25. Fromme H, Mosch C, Morovitz M, Alba-Alejandre I, Boehmer S, Kiranoglu M, et al. Pre- and postnatal exposure to perfluorinated compounds (PFCs). Environ Sci Technol. 2010;44(18):7123–7129. doi: 10.1021/es101184f. [DOI] [PubMed] [Google Scholar]
  26. Gee D. Establishing evidence for early action: the prevention of reproductive and developmental harm. Basic Clin Pharmacol Toxicol. 2008;102:257–266. doi: 10.1111/j.1742-7843.2008.00207.x. [DOI] [PubMed] [Google Scholar]
  27. Goldman LR, Silbergeld EK.2013Assuring access to data for chemical evaluations. Environ Health Perspect 121149–152.; 10.1289/ehp.1206101 [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Guyatt G, Oxman AD, Akl EA, Kunz R, Vist G, Brozek J, et al. Grade guidelines: 1. Introduction—GRADE evidence profiles and summary of findings tables. J Clin Epidemiol. 2011;64:383–394. doi: 10.1016/j.jclinepi.2010.04.026. [DOI] [PubMed] [Google Scholar]
  29. Guyatt GH, Oxman AD, Kunz R, Falck-Ytter Y, Vist GE, Liberati A, et al. Going from evidence to recommendations. BMJ. 2008a;336:1049–1051. doi: 10.1136/bmj.39493.646875.AE. [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso-Coello P, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ. 2008b;336:924–926. doi: 10.1136/bmj.39489.470347.AD. [DOI] [PMC free article] [PubMed] [Google Scholar]
  31. Halvorson GC. Electronic medical records and the prospect of real time evidence development. In: Evidence-Based Medicine and the Changing Nature of Health Care. 2007 IOM Annual Meeting Summary (McClellan MB, McGinnis JM, Nabel EG, Olsen LM, eds). Washington, DC:National Academies Press, 128–132. 2008. Available: http://www.nap.edu/openbook.php?record_id=12041&page=128 [accessed 28 August 2014]
  32. Hekster FM, Laane RW, de Voogt P. Environmental and toxicity effects of perfluoroalkylated substances. Rev Environ Contam Toxicol. 2003;179:99–121. doi: 10.1007/0-387-21731-2_4. [DOI] [PubMed] [Google Scholar]
  33. Higgins JPT, Altman DG, Sterne JAC. Chapter 8: Assessing risk of bias in included studies. In: Cochrane Handbook for Systematic Reviews of Interventions, Version 5.1.0 [Updated March 2011] (Higgins JPT, Green S, eds). 2011. Available: http://handbook.cochrane.org/ [accessed 29 August 2014]
  34. Higgins JPT, Green S. Cochrane Handbook for Systematic Reviews of Interventions, Version 5.1.0 [Updated March 2011] (Higgins JPT, Green S, eds). 2011. Available: http://handbook.cochrane.org/ [accessed 21 August 2014]
  35. Hill AB. The environment and disease: association or causation? Proc R Soc Med. 1965;58:295–300. [PMC free article] [PubMed] [Google Scholar]
  36. Hooijmans CR, Tillema A, Leenaars M, Ritskes-Hoitinga M. Enhancing search efficiency by means of a search filter for finding all studies on animal experimentation in PubMed. Lab Anim. 2010;44:170–175. doi: 10.1258/la.2010.009117. [DOI] [PMC free article] [PubMed] [Google Scholar]
  37. Howells DW, Sena ES, O’Collins V, Macleod MR. Improving the efficiency of the development of drugs for stroke. Int J Stroke. 2012;7:371–377. doi: 10.1111/j.1747-4949.2012.00805.x. [DOI] [PubMed] [Google Scholar]
  38. IARC (International Agency for Research on Cancer). IARC Monographs on the Evaluation of Carcinogenic Risks to Humans: Preamble. 2006. Available: http://monographs.iarc.fr/ENG/Preamble/index.php [accessed 29 August 2014]
  39. IOM (Institute of Medicine). Knowing What Works in Health Care: A Roadmap for the Nation (Eden J, Wheatley B, McNeil B, Sox H, eds). Washington, DC:National Academies Press. 2008. Available: http://books.nap.edu/openbook.php?record_id=12038 [accessed 20 August 2014]
  40. Jensen AA, Leffers H. Emerging endocrine disrupters: perfluoroalkylated substances. Int J Androl. 2008;31:161–169. doi: 10.1111/j.1365-2605.2008.00870.x. [DOI] [PubMed] [Google Scholar]
  41. Johnson PI, Sutton P, Atchley DS, Koustas E, Lam J, Sen S, et al. 2014The Navigation Guide—evidence-based medicine meets environmental health: systematic review of human evidence for PFOA effects on fetal growth. Environ Health Perspect 1221028–1039.; 10.1289/ehp.1307893 [DOI] [PMC free article] [PubMed] [Google Scholar]
  42. Kennedy GL, Jr, Butenhoff JL, Olsen GW, O’Connor JC, Seacat AM, Perkins RG, et al. The toxicology of perfluorooctanoate. Crit Rev Toxicol. 2004;34:351–384. doi: 10.1080/10408440490464705. [DOI] [PubMed] [Google Scholar]
  43. Kim S, Choi K, Ji K, Seo J, Kho Y, Park J, et al. Trans-placental transfer of thirteen perfluorinated compounds and relations with fetal thyroid hormones. Environ Sci Technol. 2011;45(17):7465–7472. doi: 10.1021/es202408a. [DOI] [PubMed] [Google Scholar]
  44. Kim SK, Lee KT, Kang CS, Tao L, Kannan K, Kim KR, et al. Distribution of perfluorochemicals between sera and milk from the same mothers and implications for prenatal and postnatal exposures. Environ Pollut. 2011;159(1):169–174. doi: 10.1016/j.envpol.2010.09.008. [DOI] [PubMed] [Google Scholar]
  45. Koustas E, Lam J, Sutton P, Johnson PI, Atchley DS, Sen S, et al. 2014The Navigation Guide—evidence-based medicine meets environmental health: systematic review of nonhuman evidence for PFOA effects on fetal growth. Environ Health Perspect 1221015–1027.; 10.1289/ehp.1307177 [DOI] [PMC free article] [PubMed] [Google Scholar]
  46. Krauth D, Woodruff TJ, Bero L.2013Instruments for assessing risk of bias and other methodological criteria of published animal studies: a systematic review. Environ Health Perspect 121985–992.; 10.1289/ehp.1206389 [DOI] [PMC free article] [PubMed] [Google Scholar]
  47. Kudo N, Kawashima Y. Toxicity and toxicokinetics of perfluorooctanoic acid in humans and animals. J Toxicol Sci. 2003;28:49–57. doi: 10.2131/jts.28.49. [DOI] [PubMed] [Google Scholar]
  48. Lam J, Koustas E, Sutton P, Johnson PI, Atchley DS, Sen S, et al. 2014The Navigation Guide—evidence-based medicine meets environmental health: integration of animal and human evidence for PFOA effects on fetal growth. Environ Health Perspect 1221040–1051.; 10.1289/ehp.1307923 [DOI] [PMC free article] [PubMed] [Google Scholar]
  49. Landis SC, Amara SG, Asadullah K, Austin CP, Blumenstein R, Bradley EW, et al. A call for transparent reporting to optimize the predictive value of preclinical research. Nature. 2012;490:187–191. doi: 10.1038/nature11556. [DOI] [PMC free article] [PubMed] [Google Scholar]
  50. Lau C, Anitole K, Hodes C, Lai D, Pfahles-Hutchens A, Seed J. Perfluoroalkyl acids: a review of monitoring and toxicological findings. Toxicol Sci. 2007;99:366–394. doi: 10.1093/toxsci/kfm128. [DOI] [PubMed] [Google Scholar]
  51. Lau C, Butenhoff JL, Rogers JM. The developmental toxicity of perfluoroalkyl acids and their derivatives. Toxicol Appl Pharmacol. 2004;198:231–241. doi: 10.1016/j.taap.2003.11.031. [DOI] [PubMed] [Google Scholar]
  52. Lexchin J, Bero LA, Djulbegovic B, Clark O. Pharmaceutical industry sponsorship and research outcome and quality: systematic review. BMJ. 2003;326:1167–1170. doi: 10.1136/bmj.326.7400.1167. [DOI] [PMC free article] [PubMed] [Google Scholar]
  53. Lindstrom AB, Strynar MJ, Libelo EL. Polyfluorinated compounds: past, present, and future. Environ Sci Technol. 2011;45:7954–7961. doi: 10.1021/es2011622. [DOI] [PubMed] [Google Scholar]
  54. Lundh A, Sismondo S, Lexchin J, Busuioc OA, Bero L.2012Industry sponsorship and research outcome. Cochrane Database Syst Rev 12MR000033; 10.1002/14651858.MR000033.pub2 [DOI] [PubMed] [Google Scholar]
  55. Macleod MR, O’Collins T, Howells DW, Donnan GA. Pooling of animal experimental data reveals influence of study design and publication bias. Stroke. 2004;35:1203–1208. doi: 10.1161/01.STR.0000125719.25853.20. [DOI] [PubMed] [Google Scholar]
  56. Macleod MR, van der Worp HB, Sena ES, Howells DW, Dirnagl U, Donnan GA. Evidence for the efficacy of NXY-059 in experimental focal cerebral ischaemia is confounded by study quality. Stroke. 2008;39:2824–2829. doi: 10.1161/STROKEAHA.108.515957. [DOI] [PubMed] [Google Scholar]
  57. Malloy TF, Sinsheimer PJ, Blake A, Linkov I. Use of multi-criteria decision analysis in regulatory alternatives analysis: a case study of lead free solder. Integr Environ Assess Manag. 2013;9:652–664. doi: 10.1002/ieam.1449. [DOI] [PubMed] [Google Scholar]
  58. Matus KJ, Clark WC, Anastas PT, Zimmerman JB. Barriers to the implementation of green chemistry in the United States. Environ Sci Technol. 2012;46:10892–10899. doi: 10.1021/es3021777. [DOI] [PubMed] [Google Scholar]
  59. McPartland JM, Glass M, Pertwee RG. Meta-analysis of cannabinoid ligand binding affinity and receptor distribution: interspecies differences. Br J Pharmacol. 2007;152:583–593. doi: 10.1038/sj.bjp.0707399. [DOI] [PMC free article] [PubMed] [Google Scholar]
  60. Michaels D. New York:Oxford University Press. 2008. Doubt Is Their Product: How Industry’s Assault on Science Threatens Your Health. [Google Scholar]
  61. National Toxicology Program. OHAT Nominations Under Consideration and Evaluations. 2013. Available: http://ntp.niehs.nih.gov/pubhealth/hat/noms/index.html [accessed 22 August 2014]
  62. Newbold R, Heindel J. Cambridge, UK: Cambridge University Press, 92–102; 2010. Developmental exposures and implications for early and latent disease. In: Environmental Impacts on Reproductive Health and Fertility (Woodruff TJ, Janssen SJ, Guillette LJ Jr, Giudice LC, eds) [Google Scholar]
  63. NRC (National Research Council). Phthalates and Cumulative Risk Assessment: The Task Ahead. Washington, DC:National Academies Press. 2008. Available: http://www.nap.edu/openbook.php?record_id=12528 [accessed 22 August 2014] [PubMed]
  64. NRC (National Research Council). Science and Decisions: Advancing Risk Assessment. Washington, DC:National Academies Press. 2009. Available: http://www.nap.edu/catalog.php?record_id=12209 [accessed 23 August 2014] [PubMed]
  65. NRC (National Research Council). Review of the Environmental Protection Agency’s Draft IRIS Assessment of Formaldehyde. Washington, DC:National Academies Press. 2011. Available: http://www.nap.edu/catalog.php?record_id=13142 [accessed 23 August 2014] [PubMed]
  66. NRC (National Research Council). Review of the Environmental Protection Agency’s State-of-the-Science Evaluation of Nonmonotonic Dose–Response Relationships as They Apply to Endocrine Disruptors. Washington, DC:National Academies Press. 2014a. Available: http://www.nap.edu/catalog.php?record_id=18608 [accessed 29 August 2014]
  67. NRC (National Research Council). Review of EPA’s Integrated Risk Information System (IRIS) Process. Washington, DC:National Academies Press. 2014b. Available: http://www.nap.edu/catalog.php?record_id=18764 [accessed 22 August 2014] [PubMed]
  68. Olden K, Freudenberg N, Dowd J, Shields AE. Discovering how environmental exposures alter genes could lead to new treatments for chronic illnesses. Health Aff (Millwood) 2011;30:833–841. doi: 10.1377/hlthaff.2011.0078. [DOI] [PMC free article] [PubMed] [Google Scholar]
  69. Olsen GW, Butenhoff JL, Zobel LR. Perfluoroalkyl chemicals and human fetal development: an epidemiologic review with clinical and toxicological perspectives. Reprod Toxicol. 2009;27:212–230. doi: 10.1016/j.reprotox.2009.02.001. [DOI] [PubMed] [Google Scholar]
  70. Park SJ, Ogunseitan OA, Lejano RP. Dempster-Shafer theory applied to regulatory decision process for selecting safer alternatives to toxic chemicals in consumer products. Integr Environ Assess Manag. 2014;10:12–21. doi: 10.1002/ieam.1460. [DOI] [PubMed] [Google Scholar]
  71. Peterson ED. Washington, DC:National Academies Press. In: Evidence-Based Medicine and the Changing Nature of Health Care. 2007 IOM Annual Meeting Summary (McClellan MB, McGinnis JM, Nabel EG, Olsen LM, eds); 2008. Research methods to speed the development of better evidence—the registries example. pp. 132–141. [Google Scholar]
  72. Popelut A, Valet F, Fromentin O, Thomas A, Bouchard P.2010Relationship between sponsorship and failure rate of dental implants: a systematic approach. PLoS One 5e10274; 10.1371/journal.pone.0010274 [DOI] [PMC free article] [PubMed] [Google Scholar]
  73. Post GB, Cohn PD, Cooper KR. Perfluorooctanoic acid (PFOA), an emerging drinking water contaminant: a critical review of recent literature. Environ Res. 2012;116:93–117. doi: 10.1016/j.envres.2012.03.007. [DOI] [PubMed] [Google Scholar]
  74. Rennie D. Integrity in scientific publishing. Health Serv Res. 2010;45:885–896. doi: 10.1111/j.1475-6773.2010.01088.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  75. Rennie D, Chalmers I. Assessing authority. JAMA. 2009;301:1819–1821. doi: 10.1001/jama.2009.559. [DOI] [PubMed] [Google Scholar]
  76. Rooney AA, Boyles AL, Wolfe MS, Bucher JR, Thayer KA.2014Systematic review and evidence integration for literature-based environmental health science assessments. Environ Health Perspect 122711–718.; 10.1289/ehp.1307972 [DOI] [PMC free article] [PubMed] [Google Scholar]
  77. Roseman M, Milette K, Bero LA, Coyne JC, Lexchin J, Turner EH, et al. Reporting of conflicts of interest in meta-analyses of trials of pharmacological treatments. JAMA. 2011;305:1008–1017. doi: 10.1001/jama.2011.257. [DOI] [PubMed] [Google Scholar]
  78. Sena ES, Briscoe CL, Howells DW, Donnan GA, Sandercock PA, Macleod MR. Factors affecting the apparent efficacy and safety of tissue plasminogen activator in thrombotic occlusion models of stroke: systematic review and meta-analysis. J Cereb Blood Flow Metab. 2010;30:1905–1913. doi: 10.1038/jcbfm.2010.116. [DOI] [PMC free article] [PubMed] [Google Scholar]
  79. Shah RV, Albert TJ, Bruegel-Sanchez V, Vaccaro AR, Hilibrand AS, Grauer JN. Industry support and correlation to study outcome for papers published in Spine. Spine (Phila Pa 1976) 2005;30:1099–1104. doi: 10.1097/01.brs.0000161004.15308.b4. [DOI] [PubMed] [Google Scholar]
  80. Stahl T, Mattern D, Brunn H.2011Toxicology of perfluorinated compounds. Environ Sci Eur 2338; 10.1186/2190-4715-23-38 [DOI] [Google Scholar]
  81. Steenland K, Fletcher T, Savitz DA.2010Epidemiologic evidence on the health effects of perfluorooctanoic acid (PFOA). Environ Health Perspect 1181100–1108.; 10.1289/ehp.0901827 [DOI] [PMC free article] [PubMed] [Google Scholar]
  82. Trasande L, Liu Y. Reducing the staggering costs of environmental disease in children, estimated at $76.6 billion in 2008. Health Aff (Millwood) 2011;30:863–870. doi: 10.1377/hlthaff.2010.1239. [DOI] [PubMed] [Google Scholar]
  83. Tsai PL, Hatfield TH. Global benefits from the phaseout of leaded fuel. J Environ Health. 2011;74:8–14. [Google Scholar]
  84. U.S. EPA (U.S. Environmental Protection Agency). Guidelines for Developmental Toxicity Risk Assessment. 1991. Available: http://cfpub.epa.gov/ncea/cfm/recordisplay.cfm?deid=23162#Download [accessed 27 February 2014]
  85. U.S. EPA (U.S. Environmental Protection Agency). Guidelines for Reproductive Toxicity Risk Assessment. 1996. Available: http://www.epa.gov/raf/publications/pdfs/REPRO51.PDF [accessed 27 February 2014]
  86. U.S. EPA (U.S. Environmental Protection Agency). The Benefits and Costs of the Clean Air Act from 1990 to 2020. 2011. Available: http://www.epa.gov/air/sect812/feb11/fullreport_rev_a.pdf [accessed 6 February 2014]
  87. U.S. EPA (U.S. Environmental Protection Agency). Design for the Environment Program Master Criteria for Safer Ingredients. 2012. Available: http://www.epa.gov/dfe/pubs/projects/gfcp/index.htm#Master [accessed 22 August 2014]
  88. U.S. EPA (U.S. Environmental Protection Agency). Systematic Review Workshop. 2013a. Available: http://www.epa.gov/IRIS/irisworkshops/systematicreview/ [accessed 21 May 2013]
  89. U.S. EPA (U.S. Environmental Protection Agency). Air Quality: EPA’s Integrated Science Assessments (ISAs). 2013b. Available: http://www.epa.gov/ncea/isa/ [accessed 24 September 2013]
  90. U.S. EPA (U.S. Environmental Protection Agency). America’s Children and the Environment Homepage. 2013c. Available: http://www.epa.gov/ace/ [accessed 6 February 2014]
  91. van der Worp HB, Macleod MR. Preclinical studies of human disease: time to take methodological quality seriously. J Mol Cell Cardiol. 2011;51:449–450. doi: 10.1016/j.yjmcc.2011.04.008. [DOI] [PubMed] [Google Scholar]
  92. van der Worp HB, Sena ES, Donnan GA, Howells DW, Macleod MR. Hypothermia in animal models of acute ischaemic stroke: a systematic review and meta-analysis. Brain. 2007;130:3063–3074. doi: 10.1093/brain/awm083. [DOI] [PubMed] [Google Scholar]
  93. Vesterinen HM, Egan K, Deister A, Schlattmann P, Macleod MR, Dirnagl U. Systematic survey of the design, statistical analysis, and reporting of studies published in the 2008 volume of the Journal of Cerebral Blood Flow and Metabolism. J Cereb Blood Flow Metab. 2011;31:1064–1072. doi: 10.1038/jcbfm.2010.217. [DOI] [PMC free article] [PubMed] [Google Scholar]
  94. Vesterinen HM, Sena ES, ffrench-Constant C, Williams A, Chandran S, Macleod MR. Improving the translational hit of experimental treatments in multiple sclerosis. Mult Scler. 2010;16:1044–1055. doi: 10.1177/1352458510379612. [DOI] [PubMed] [Google Scholar]
  95. Viswanathan M, Ansari M, Berkman N, Chang S, Hartling L, McPheeters L, et al. Assessing the Risk of Bias of Individual Studies in Systematic Reviews of Health Care Interventions. Agency for Healthcare Research and Quality Methods Guide for Comparative Effectiveness Reviews. AHRQ Publication No. 12-EHC047-EF. 2012. Available: http://effectivehealthcare.ahrq.gov/ehc/products/322/998/MethodsGuideforCERs_Viswanathan_IndividualStudies.pdf [accessed 22 August 2014] [PubMed]
  96. Vogel SA, Roberts JA. Why the Toxic Substances Control Act needs an overhaul, and how to strengthen oversight of chemicals in the interim. Health Aff (Millwood) 2011;30:898–905. doi: 10.1377/hlthaff.2011.0211. [DOI] [PubMed] [Google Scholar]
  97. Wang IJ, Hsieh WS, Chen CY, Fletcher T, Lien GW, Chiang HL, et al. The effect of prenatal perfluorinated chemicals exposures on pediatric atopy. Environ Res. 2011;111(6):785–791. doi: 10.1016/j.envres.2011.04.006. [DOI] [PubMed] [Google Scholar]
  98. White SS, Fenton SE, Hines EP. Endocrine disrupting properties of perfluorooctanoic acid. J Steroid Biochem Mol Biol. 2011;127:16–26. doi: 10.1016/j.jsbmb.2011.03.011. [DOI] [PMC free article] [PubMed] [Google Scholar]
  99. Woodruff TJ, Janssen SJ, Guillette LJ Jr, Giudice LC. New York:Cambridge University Press. 2010. Environmental Impacts on Reproductive Health and Fertility. [Google Scholar]
  100. Woodruff TJ, Sutton P, The Navigation Guide Work Group. An evidence-based medicine methodology to bridge the gap between clinical and environmental health sciences. Health Aff (Millwood) 2011;30:931–937. doi: 10.1377/hlthaff.2010.1219. [DOI] [PMC free article] [PubMed] [Google Scholar]
  101. World Health Organization and United Nations Environment Programme. State of the Science of Endocrine Disrupting Chemicals–2012. 2013. Available: http://www.who.int/ceh/publications/endocrine/en/ [accessed 6 February 2014]

Articles from Environmental Health Perspectives are provided here courtesy of National Institute of Environmental Health Sciences

RESOURCES