Invited Perspective: Systematic Review for Environmental Pollutants—A Work in Progress

The development of environmental health policies is grounded in evidence related to agent characteristics, exposures, and associated risks to human and ecological health. Methods to assemble evidence to support decision making have evolved greatly over the last two decades, moving from reviews based on expert judgment to transparent and systematic approaches to gathering, evaluating, synthesizing, and integrating research findings (Figure 1).¹ For example, 15 years ago, the U.S. Environmental Protection Agency replaced the encyclopedic, but not integrative, listings of its criteria documents with the Integrated Science Assessment (ISA), a systematically assembled and evaluated compilation of evidence following a systematic review approach, when considering revisions of the National Ambient Air Quality Standards.² Responding to guidance from a 2011 report of the U.S. National Academies, the agency also introduced systematic review into its Integrated Risk Information System (IRIS) assessments, which provide hazard assessments and dose–response relationships for cancer and noncancer health outcomes.³ Figure 1 illustrates a framework for IRIS assessments, recommended in 2014 by a subsequent committee of the National Academies, that describes the sequence of processes for determining a hazard (including the role of systematic review) and estimating dose response.¹ While the ISA and IRIS processes serve different purposes, each has been modified over time to incorporate evolving aspects of systematic review.

Figure 1.

Systematic review in the context of the IRIS process. Reprinted with permissions from National Research Council, Division on Earth and Life Studies, Board on Environmental Studies and Toxicology, Committee to Review the IRIS Process.¹

In this issue of Environmental Health Perspectives, Boogaard et al.⁴ describe their experience in conducting a systematic review for the Health Effects Institute (HEI) of human studies on the health risks of traffic-related air pollution. They focus on their experience with synthesis of the epidemiological literature, carried out using an approach developed by the Office of Health Assessment and Translation (OHAT) of the U.S. National Toxicology Program and describe seven lessons learned.⁵ In synthesizing evidence for a group of studies (e.g., human studies on asthma outcomes) with the OHAT approach, the individual studies that meet inclusion criteria are evaluated for risk of bias, and an overall confidence rating is assigned to the body of literature reviewed. This same step is completed separately for animal studies and mechanistic research in full hazard assessments. The OHAT methods draw on those of the Cochrane Collaboration, the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) working group—initially developed for clinical research and pooling randomized clinical trials⁶—and the Navigation Guide, developed by Woodruff et al. for environmental agents.⁷

Boogaard et al. document two broad challenges in using the OHAT approach for their review: a) assessing the quality of individual studies and b) assigning the level of confidence to the human evidence stream. For the former, the review panel turned to a risk-of-bias tool modified from GRADE by the working group for the recent revision of the World Health Organization’s Air Quality Guidelines.⁴ The working group implemented modifications to address the mismatch between GRADE and the characteristics of epidemiological studies of air pollution. For the latter, the HEI panel complemented the OHAT approach with their own narrative assessment that focused on confidence with regard to the strength of evidence for an association rather than following OHAT’s directive toward confidence in the body of evidence. The addition of this review step outside the protocol by the HEI panel reflected the frustration of Boogaard et al. with the fit of the OHAT methods to the studies considered. The replicability of the narrative approach is uncertain, and it wanders partially into the realm of evidence integration.

The seven lessons learned are useful and based on a rich experience with a complex body of literature on traffic-related air pollution. In my opinion, the most critical lesson learned, however, was the mismatch between OHAT’s schema and the panel’s judgments about the studies considered. This problem is not particular to OHAT, as existing schemas for assembling, reviewing, and synthesizing evidence on environmental agents are imperfect and evolving (see, for example, the reports by committees of the National Academies on the IRIS Handbook and on the review approach used under the U.S. Toxic Substances Control Act^1,8,9). The experience of the HEI panel echoes that of the World Health Organization working group, which replaced rigid application of GRADE with a modified approach for assessing risk of bias and assigning a level of confidence, making it better attuned to epidemiological studies of air pollution. There are now multiple schemas for systematic review of studies on the risks of environmental agents—for example, the Navigation Guide and the protocols for IRIS and for the reviews carried out under the Toxic Substances Control Act.^10,11 Committees of the National Academies have provided guidance to the Environmental Protection Agency on these protocols, pointing to opportunities for continuing their refinement.^8,9 These approaches are being modified as experience is gained in using them. Although this invited perspective and the paper by Boogaard et al. focus on systematic review of human studies, there is also an effort to develop review methods for evidence-based toxicology.¹²

The experience of the HEI panel raises the question as to whether groups carrying out systematic reviews of environmental agents should depart from standardized systematic review methods and their own published protocols when methods do not fit well with their task. The narrative approach, although documented in the HEI report, returns to reliance on expert judgment to a greater degree than the OHAT approach. Experience shows that for agencies conducting ongoing reviews, adherence to standardized methods is requisite to ensure transparency, given the scrutiny that such reviews may receive.

After watching the evolving role of systematic review in assessing the risks of environmental agents, I offer the following observations. Although methods will evolve, systematic reviews of the literature on environmental agents will continue to be integral to hazard assessment, as they bring transparency and reduce subjectivity. There are existing review approaches for environmental agents, albeit with flaws, that should be utilized as the starting point for such reviews, and any departures from them should be well documented and justified. Lessons learned from reviews should be cataloged, as done by Boogaard et al., to guide refinements. With various governmental and nongovernmental groups developing methods for systematic reviews in environmental health, I call for coordination and collaboration among them. And, remember, systematic review has replaced far less transparent and objective approaches.

Refers to https://doi.org/10.1289/EHP11532