Comment on “Evidence Synthesis of Observational Studies in Environmental Health: Lessons Learned from a Systematic Review on Traffic-Related Air Pollution”

We read with interest the discussion by Boogaard et al.¹ of their experience using the OHAT systematic review framework² to assess the level of confidence in the epidemiological literature included in their systematic review on traffic-related air pollution. OHAT is the abbreviation for Office of Health Assessment and Translation, which has become the Health Assessment and Translation group in the Integrative Health Assessment Branch of the Division of Translational Toxicology at the National Institute of Environmental Health Sciences. The authors concluded that the OHAT approach, and other frameworks using the Grading of Recommendations, Assessment, Development, and Evaluations, require substantial modification to better align with environmental health questions. We appreciate their interest in both using and improving the OHAT approach. However, they did not take full advantage of the flexibility built into the approach, and their recommendations reflect several inaccuracies and misinterpretations such that most of their suggested modifications are already part of the method.

Boogaard et al. state that when translating confidence ratings to level-of-evidence conclusions, the OHAT approach does not consider four “important relevant factors” (study number and size, direction and magnitude of associations, consistency of results, generalizability) or three “additional features” (sufficient time between exposure and outcome, adequate exposure contrast, time windows for exposure and outcome). We agree that these issues are important to evidence integration, which is why each is considered in Step 5 of the OHAT methodology (except for “adequate exposure contrast,” which is considered in Step 4).² The authors suggest considering these issues a second time in Step 6; however, double counting any of these factors would bias the evidence assessment. Additionally, the OHAT approach anticipates that additional project-specific considerations may be warranted for developing confidence ratings. Throughout the OHAT handbook, an “other” category is already included to accommodate potential issues.

In setting initial confidence for bodies of evidence, the authors appear to have correctly followed the OHAT approach; however, they inaccurately stated that their method differed from the OHAT guidance. When determining final study confidence, the authors reported that they did not use two of the grading factors described in the handbook—indirectness or large magnitude of effect—thereby skipping several considerations that they later cite as missing from the OHAT approach. It is not clear why the authors skipped those factors, but it is inappropriate to omit steps in the approach and then describe considerations in those steps as missing.

We disagree with the implication that the OHAT approach does not allow for a narrative assessment of the body of literature when considering whether to upgrade or downgrade the final confidence rating. The handbook states that narrative synthesis of the evidence is the most appropriate approach and that documenting scientific judgements is critical for transparency.² The OHAT approach requires justification for how initial confidence ratings were reached, how individual upgrade and downgrade factors were considered, whether an upgrade or downgrade was warranted, and how these factors were considered collectively.² We agree with the authors that the OHAT approach does not explicitly state if/when quantitative and qualitative analyses can be used concurrently.

In Table 1, Boogaard et al. present “key elements of the OHAT approach” alongside suggested improvements. However, multiple elements are incompletely or inaccurately portrayed, leading to redundant or inapplicable recommendations. To clarify and correct the information we have reproduced this table and added our responses (Table 1). We hope our response clarifies the flexibility built into the OHAT approach and shows how the method addresses critical factors for assessing and integrating environmental health literature. This exchange presents the opportunity to further open dialog and training between users and systematic review experts in the Division of Translational Toxicology of the National Institute of Environmental Health Perspectives.

Table 1.

Response to suggested improvements presented in Table 1 of Boogaard et al.¹

Text from Table 1 of Boogaard et al.1		Response
Key elements of OHAT approach	Suggested improvement by the Traffic Review Panel
Evidence synthesis
Use a GRADE-type approach to assess confidence in the quality of the body of evidence.	Complement the GRADE-type assessment with a broader, “narrative approach” to maximize what can be learned from observational studies in environmental health.	The “key element” presented in the first column is incomplete and misrepresents the OHAT approach. As a result, the suggested improvement is misleading because, although the OHAT approach uses a modified version of GRADE, it also strongly encourages a narrative approach.2,3
Assign an initial low or moderate level of confidence to all types of observational studies.	Consider that in environmental health, where randomized controlled trials (RCTs) are generally not appropriate, some observational studies can offer high-confidence evidence.	The “key element” described in the first column is incomplete and misrepresents the OHAT approach, which already includes the suggested improvement. Furthermore, the suggestion inaccurately implies that the OHAT approach considers RCTs to be the gold standard of study design and does not consider that some observational studies can offer high-confidence evidence. In OHAT’s approach, there is no gold standard, and initial confidence in a body of evidence is not based on author-reported study design (e.g., cross-sectional, cohort, case-control). Instead, it is based on the presence or absence of four key study design features: controlled exposure, exposure prior to outcome, individual outcome data, comparison group used. In addition, the OHAT handbook specifically states, “a single well-designed and -conducted observational study and/or several well-designed and -conducted observational studies can offer high confidence.” See Figure 8 in Hazard Identification Scheme in the OHAT handbook.2
Assess the statistical heterogeneity of results and downgrade the confidence rating if substantial heterogeneity is found.	Sources of heterogeneity can strengthen or weaken the confidence in the evidence and should be carefully explored. Some heterogeneity is expected in studies due to different populations, locations, and study settings. Consider primarily the direction of the effect estimate rather than its magnitude.	Again, the authors’ suggested improvement is, in fact, part of the OHAT approach. Boogaard et al.1 are referring to a downgrade factor called “unexplained inconsistency,” which is more applicable to heterogeneous experimental animal studies or RCTs. For observational studies, heterogeneity is expected (i.e., not unexplained). In OHAT’s approach, the upgrade factor “Cross-species/population/study consistency” considers heterogeneity in both human and animal studies to strengthen confidence in a body of evidence if there is a consistent direction of association across studies. See Cross-species/population/study consistency in Step 5 of the OHAT handbook.2
Assess publication bias using Egger’s test and funnel plots and downgrade accordingly.	Publication bias is not necessarily expected when large and collaborate (multicenter) studies comprise most of the evidence and/or if evidence has accrued over several decades. Use additional approaches to explore the possibility of publication bias.	Again, the authors’ suggested improvement is already part of the OHAT approach. OHAT recommends several approaches in addition to using Egger’s test and funnel plots (e.g., trim and fill technique, “and other approaches”) to explore the possibility of publication bias and in no way indicates that these are the only approaches. See Publication Bias in Step 5 of the OHAT handbook.2
Risk of bias in individual studies
Compare study with randomized controlled trials or hypothetical target experiment as ideal study.	Do not consider RCTs as ideal study.	The OHAT approach does not compare studies to RCTs or hypothetical target experiments. We suspect the authors may be referring to the risk of bias in non-randomized studies of interventions4 method, which does use an ideal RCT at the target experiment. The OHAT approach does not consider RCTs to be ideal for evaluating environmental exposures and health outcomes. Indeed, there is no mention of an “ideal study” in the OHAT method.
Evaluate bias in different domains (e.g., confounding, selection bias, measurement error).	Focus on identifying the most likely influential sources of bias—based on methodologic and subject matter expertise—classifying each study on the basis of how effectively it has addressed each potential bias and determine whether results differ across studies in relation to each hypothesized source of bias.	Evaluating bias in each domain and focusing on the most influential sources of bias are not mutually exclusive. Boogaard et al. are ignoring the fact that the OHAT approach recommends that reviewers focus on the sources of bias most likely to be influential for environmental studies and to give more weight to key domains/key elements that have the greatest potential to impact the results (e.g., exposure, outcome, and confounding). However, the OHAT approach maintains the importance of systematically assessing all bias domains to identify whether there is additional serious risk of bias concerns (e.g., selection bias) and would not support an expert driven approach where every study was assessed for different factors.
Rate potential biases (e.g., low, moderate, high) using a risk of bias tool.	Rate biases considering the suggestions in the row above. Those ratings should not be used to dismiss studies based on bias but to conduct sensitivity analyses comparing findings from studies of high bias and low/moderate bias.	It is not clear what method the authors are referring to in the first column, as the OHAT approach does not have a three-point risk of bias scale and ratings are not “low, moderate, high” as suggested by the authors. The OHAT approach has four risk-of-bias ratings: “probably low,” “definitely low,” “probably high,” and “definitely high,” and the authors’ suggested improvement is already part of the OHAT approach.

Note: GRADE, Grading of Recommendations Assessment, Development, and Evaluation.

Refers to https://doi.org/10.1289/EHP11532