The relationship between childhood blood lead levels below 5 μg/dL and childhood intelligence quotient (IQ): Protocol for a systematic review and meta-analysis

Abstract

Background

The causal association between childhood lead (Pb) exposure and decrements in intelligence quotient (IQ) is well-established, and no safe blood lead level (BLL) in children has been identified. An international pooled analysis of seven prospective studies published by Lanphear et al. in 2005 quantified the relationship between childhood BLL and IQ. Further studies of Pb and IQ have been published more recently with mean BLLs generally lower than in the studies analyzed by Lanphear et al. In this article, we present the protocol for a systematic review to estimate an updated Pb-IQ relationship focusing on BLLs below 5 micrograms per deciliter (μg/dL).

Study question

What is the quantitative relationship between childhood BLLs and IQ at ages 4–22 years at BLLs below 5 μg/dL?

Data sources

A comprehensive search of the scientific literature will utilize citation mapping and key word searching. In the citation mapping approach, we will identify seed references that are relevant to our study question, and will then identify more recent references that have cited at least one of the seed references. The key word search will be conducted in the PubMed, Biosis Previews, Scopus, and Web of Science databases. We will also search electronic grey literature databases.

Study eligibility criteria, study screening and data extraction

We will include studies that measured BLL in children at any age, assessed full-scale IQ of the same children (concurrent with or subsequent to BLL sample collection) at ages 4–22, and estimated a continuous quantitative relationship between BLL and IQ. We will consider only studies with a central tendency BLL < 10 μg/dL.

The title and abstract of each record will be reviewed independently by two authors to determine whether the study in question satisfies the inclusion criteria. The full text of each article remaining after title-abstract screening will be reviewed independently by two authors to determine whether the study in question satisfies the inclusion criteria. Two authors will independently extract study characteristics and data from each included study.

Risk of bias assessment

Studies meeting inclusion criteria will be evaluated for risk of bias (RoB) using the Navigation Guide method applied in a previous systematic review of neurodevelopmental effects (Lam et al. 2017), with adaptation to our study question. Each study will be independently evaluated by two review authors.

Data analysis and synthesis

We intend to conduct a random-effects meta-analysis to summarize the effects of children’s exposure to Pb on IQ scores. Additionally, we plan to perform sensitivity analyses using sub-group analyses and/or meta-regression techniques to assess the impact of study design and study population characteristics to examine potential heterogeneity of results across studies.

We will assign a confidence level rating (high, moderate, low, or very low) to the effect estimate from the meta-analyses/meta-regressions.

1. INTRODUCTION

1.1. Background

The causal association between childhood lead (Pb) exposure and decrements in intelligence quotient (IQ) is well-established, and no safe blood lead level (BLL) in children has been identified (Agency for Toxic Substances and Disease Registry 2020; National Toxicology Program 2012; U.S. Environmental Protection Agency 2013). Several comprehensive assessments of the scientific evidence on the health effects of Pb exposure have been conducted. The 2013 Integrated Science Assessment (ISA) for Lead, prepared by the U.S. Environmental Protection Agency (EPA), reports clear evidence of cognitive function deficits in children from 4 to 11 years of age with mean or group BLLs between 2 and 8 micrograms of Pb per deciliter of blood (μg/dL) (U.S. Environmental Protection Agency 2013). This conclusion, which is based on well-conducted longitudinal epidemiologic studies, was supported by experimental animal studies that demonstrate analogous effects (i.e. decrements in learning, memory and executive function) in rodents and monkeys. In 2012 the National Toxicology Program (NTP) also concluded that there was sufficient evidence that children’s BLLs <5 μg/dL were associated with decreased cognitive performance as indicated by decreased IQ and decrements on other tests of cognition, as well as lower academic performance (National Toxicology Program 2012). Finally, a toxicological profile published by the Agency for Toxic Substances and Disease Registry (ATSDR) in 2020 reports similar conclusions and includes more recent studies of lead and IQ (Agency for Toxic Substances and Disease Registry 2020).

An international pooled analysis of seven prospective studies by Lanphear et al. (Lanphear et al. 2005; 2019) reported that increases in concurrent and peak BLLs were associated with decreases in Full Scale IQ (FSIQ) measured in children between ages 5 and 10 years old. These authors also reported a larger incremental effect of Pb on children’s IQ at lower compared to higher BLLs (i.e. supralinear relationship), which was substantiated in a subsequent reanalysis by Rothenberg and Rothenberg (Rothenberg and Rothenberg 2005). Findings from individual studies of BLL are generally consistent with the findings of Lanphear et al. reporting Pb-associated decreases in cognitive function in children and adolescents and a supralinear relationship between BLL and cognitive function (U.S. Environmental Protection Agency 2013).

This study is frequently cited in the literature and remains a key reference for environmental decision-making. Lanphear et al. (Lanphear et al. 2005; 2019) and reanalyses using the same international pooled data (Crump et al. 2013; Kirrane and Patel 2014) continue to be used in EPA regulatory analyses. EPA estimates the benefits of improvements in children’s cognitive function resulting from decreased Pb exposure by pairing the Pb-IQ relationship derived from the Lanphear et al. pooled international data with estimates of the gain in lifetime earnings associated with the estimated change in IQ (U.S. Environmental Protection Agency 2019).¹

1.2. Rationale for Review

The median concurrent BLL in Lanphear et al. 2005 was 9.7 μg/dL. Since Lanphear et al. (Lanphear et al. 2005) was originally conducted, a number of additional studies of Pb and IQ have been published (Bauer et al. 2020; Braun et al. 2012; Chiodo et al. 2004; Haynes et al. 2018; Hong et al. 2015; Kim et al. 2009; Kordas et al. 2011; Lucchini et al. 2012; Menezes-Filho et al. 2018; Min et al. 2009; Riojas-Rodriguez et al. 2010). The mean BLLs in these studies are generally lower than in the studies analyzed by Lanphear et al. because BLLs have decreased over time due to the phaseout of Pb in gasoline and consumer products, and remediation of Pb-based paint. The Centers for Disease Control and Prevention (CDC) defines a reference BLL in children to correspond to the 97.5^th percentile of the U.S. population (1–5 years old) using data from the National Health and Nutrition Examination Survey (NHANES) (Advisory Committee on Childhood Lead Poisoning Prevention 2012; Centers for Disease Control and Prevention 2012). The current reference value, which is based on data from NHANES 2015–2018, is 3.5 μg/dL (Ruckart et al. 2021), and many of the recent studies are conducted in populations with mean BLLs ≤3.5 μg/dL. Because Pb contamination is persistent in the environment, and there is no known safe BLL, lead exposure is still a pressing environmental concern.

Epidemiologic studies report associations of IQ with Pb exposure metrics during different childhood lifestages or time periods (i.e. prenatal period through adolescence) (U.S. Environmental Protection Agency 2013). Because exposure metrics such as concurrent, peak, and lifetime mean BLLs are generally highly correlated in epidemiologic studies, it is challenging to distinguish the relative importance of each lifestage or metric with respect to the strength of the association. The epidemiologic evidence, however, is supported by experimental studies of rodents and monkeys that indicate that Pb exposures during multiple lifestages can induce impairments in learning. These findings are supported by our understanding that the nervous system continues to develop throughout childhood and into adolescence (U.S. Environmental Protection Agency 2013).

There is a large body of prospective studies that examine the relationship between Pb exposure and cognitive function. Although repeated blood Pb measurements within children is a strength of these studies, cross-sectional analyses that examine the relationship of blood Pb that is measured concurrently with IQ in young children are particularly informative for policy setting because they address the effect of recent exposures. Although, blood Pb may reflect both recent exposure as well as past exposures because Pb is both taken up and released from the bone, blood Pb concentration generally indicates recent exposure in young children with short exposure histories and rapid bone turn-over.

This proposed review will estimate an updated Pb-IQ concentration response function with more data informing the relationship at BLLs of interest for current policy decisions – typically < 5 μg/dL. We will include studies that measured BLL at any childhood lifestage and assessed full-scale IQ concurrent with or subsequent to BLL sample collection at ages 4–22.

1.3. Study Question

What is the quantitative relationship between childhood BLLs and IQ at age 4–22 years at BLLs below 5 μg/dL?

1.4. Objectives

The objectives of this systematic review are to:

• Identify studies conducted in humans of the continuous quantitative relationship between BLLs measured during childhood and IQ scores measured at ages 4–22 years, with mean BLL<10 μg/dL. Although our study question is focused on the relationship between BLL and IQ at <5 μg/dL, studies with higher mean BLLs may provide evidence relevant to estimating the relationship in our target range.

• Assess the risk of bias in the identified studies.

• Primary analytic objective: Conduct a meta-analysis of all identified studies with mean BLL < 5 μg/dL.

• Secondary analytic objective: Conduct a meta-analysis of all identified studies with mean BLL <10 μg/dL.

• If data allows, conduct a meta-regression of effect size estimates to examine study design and study population characteristics that may affect heterogeneity of results across studies.

• Conduct meta-analyses for subgroups of identified studies and perform additional sensitivity analyses, as appropriate.

2. METHODS

Preparation of this protocol was informed by COSTER (Whaley et al. 2020) and PRISMA (Moher et al. 2015; Shamseer et al. 2015). Many elements of the methods outlined in this protocol were adapted from the Navigation Guide systematic review of polybrominated diphenyl ethers (PBDEs) and IQ by Lam and colleagues (Lam et al. 2017; Lam et al. 2015), including search terms, procedures for screening search results, data extraction, and risk of bias evaluation.

2.1. PECOS Statement

Participants:

humans of age 4–22 years at time of IQ test.

Exposures:

Any BLL measured in individual children prior to or concurrent with IQ test, with central tendency BLL in the studied group <10 μg/dL. Although our study question is focused on the relationship between BLL and IQ at <5 μg/dL, studies with higher mean BLLs may provide evidence relevant to estimating the relationship in our target range.

Comparator:

Children with lower BLL as represented in a continuous linear function (including those with log transformations), i.e. per unit increase in BLL (in μg/dL).

Outcome:

IQ measured in individual children at ages 4–22 years. IQ assessments include (but are not limited to): Wechsler Preschool and Primary Scale of Intelligence (WPPSI), Wechsler Intelligence Scale for Children (WISC), Wechsler Abbreviated Scale of Intelligence (WASI), Stanford-Binet Intelligence Scale, and the McCarthy Scales of Children’s Abilities (MSCA).

Study Design:

Observational studies meeting the above criteria including at least one measurement of blood lead taken prior to or concurrent with the IQ assessment.

2.2. Criteria for Selecting Studies

We will select studies that measured BLL in children prior to the assessment of IQ, assessed full-scale IQ of the same children (concurrent with or subsequent to BLL sample collection) at ages 4–22, and estimated a continuous quantitative relationship between BLL and IQ (i.e. change in IQ per unit increase in BLL). We will consider only studies with a central tendency (geometric mean or arithmetic mean; or median in the absence of a reported mean) BLL < 10 μg/dL.

We will search for peer-reviewed articles and studies from the grey literature, and include all studies meeting our inclusion criteria. For studies from the grey literature, we will ascertain whether sufficient information is publicly available regarding study methods and results. Grey literature studies lacking sufficient information will be excluded.

Non-English studies will be included only if the abstract is available in English and the abstract provides sufficient information to judge the study as clearly relevant, satisfying all elements of the PECOS statement.

2.3. Search Methods

A comprehensive search of the scientific literature, developed and executed in collaboration with an Information Specialist, will comprise two main approaches. The first approach will identify potentially relevant references through citation mapping, or relational reference searching. In this approach, we will develop a set of seed references that are relevant to the topic of interest. In this case, we have selected studies of childhood Pb exposure and IQ decrements from the 2006 U.S. EPA Air Quality Criteria Document (AQCD) for Lead (U.S. Environmental Protection Agency 2006), the 2012 NTP Monograph on Health Effects of Low-Level Lead (National Toxicology Program 2012), the 2013 U.S. EPA Integrated Science Assessment (ISA) for Lead (U.S. Environmental Protection Agency 2013), and the 2020 ATSDR Toxicological Profile for Lead (Agency for Toxic Substances and Disease Registry 2020), as well as additional studies known to the review authors to be relevant to the study question (see the full list of seed studies in Appendix 1). The seed references are then used to capture any more recent references that have cited at least one of the references in the seed set. The citation mapping will be conducted in Web of Science, so any of the seed references that are not in Web of Science will not contribute to the citation mapping results.

The second approach will be a key word search using search terms relevant to lead and IQ. Given the potential overlap between words and phrases such as “leadership” and “leads to” with “lead”, we will also consider exclusion terms to eliminate extraneous references. The key word search will be conducted in PubMed, BIOSIS Previews, Scopus, and Web of Science. A list of provisional search strings with exclusion terms has been provided in Appendix 1. In addition to the list of terms established by the team, we will use topic analysis, a natural language processing technique, to identify other potentially relevant terms from the references captured by the citation mapping approach.

We also plan to search the grey literature using the search string “lead” OR “Pb” in the OpenGrey (http://www.opengrey.eu/) and Grey Literature Report (http://greylit.org) databases.

Search methods and results will be reported in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses – literature search extension (PRISMA-S) checklist (Rethlefsen et al. 2021).

2.4. Study Selection Process (including reference management)

2.4.1. Title and abstract screening

Search results obtained from PubMed, BIOSIS Previews, Scopus, and Web of Science will be compiled in Health and Environmental Research Online (HERO), a database of scientific studies and other references used to develop EPA’s risk assessments. Duplicates will be removed and the identified studies will be exported to SWIFT-ActiveScreener (SWIFT-AS) (www.sciome.com). SWIFT-AS is a software application employing machine learning in real-time based on inclusion and exclusion screening decisions to predict relevant references. Using this method, references will be queued based on predicted relevance, with records screened until an estimated 95% of the relevant records have been identified (Howard et al. 2020). The title and abstract of each record will be reviewed independently by two authors (DA, EC) to determine whether the study in question satisfies the inclusion criteria. In case of any discrepancies between the authors’ determinations, a third reviewer (EK) will review the record in question and make a final determination. In addition, ten percent of records predicted by SWIFT-AS to be off-topic will undergo additional manual review by one author (EK or HK) to evaluate performance of the software algorithm.

A screening form will be used to screen references and record the independent judgment of each reviewer. References will be excluded if any of the following criteria are clearly met based solely on the title and abstract:

1. Non-English language article lacking an English language abstract

2. Review article

3. Article contains no original data (e.g., editorial)

4. Article did not involve human subjects (i.e., animal evidence only)

5. Article did not report BLL measurements

6. Central tendency BLL > 10 μg/dL (central tendency may be reported as geometric mean, arithmetic mean or median)

7. Article did not report IQ test scores

8. Study participants were not ages 4–22 years at time of IQ testing

9. Article did not quantify continuous relationship between BLL and IQ

10. IQ testing was conducted before blood samples were collected

11. Non-English language article lacking clear PECOS relevance

12. Other reason (explanation required).

The following instructions will be provided to review authors conducting the title and abstract screening:

“When excluding a reference, please select only ONE (1) exclusion reason. Please review the exclusion reasons in order and select the FIRST exclusion reason relevant to the reference being screened. Please add in any additional notes in the comment box to explain your selection if necessary.”

For citations where the database contains no abstract, authors will attempt to obtain the abstracts from an Internet search. Articles for which the abstract remains unavailable will be screened based on titles and PubMed MeSH headings. Any study not excluded based on the above criteria will be included for full-text review.

2.4.2. Full-text review

The full text of each article remaining after title-abstract screening, and the full text of each grey literature reference, will be reviewed independently by two authors (DA, EK) to determine whether the study in question satisfies the inclusion criteria. Discrepancies will be discussed and resolved between the two reviewers and if needed a third author (EC) will be consulted to resolve any discrepancies. The rationale for excluded studies will be recorded.

The review of articles and other references against the pre-specified inclusion and exclusion criteria will be performed using a structured form and database (i.e. Microsoft Excel (www.microsoft.com) in combination with DistillerSR (www.evidencepartners.com)).

Citations eligible for full text review will be excluded if one or more of the following criteria are met:

1. Review article concerning Pb and IQ relationship

2. Article/reference contains no original data (e.g., editorial, review paper not relevant to study question, etc.)

3. Article/reference did not involve human subjects (i.e., animal evidence only)

4. Article/reference did not report BLL measurements

5. Central tendency BLL > 10 μg/dL (central tendency may be reported as geometric mean, arithmetic mean or median)

6. Article/reference did not report IQ test scores

7. Study participants were not ages 4–22 years at time of IQ testing

8. Article/reference did not quantify continuous relationship between BLL and IQ

9. IQ testing was conducted before blood samples were collected

10. Other reasons (explanation required).

The following instructions will be provided to review authors conducting full text screening:

“When excluding a reference, please select only ONE (1) exclusion reason. Please review the exclusion reasons in order and select the FIRST exclusion reason relevant to the reference being screened. Please add in any additional notes in the comment box to explain your selection if necessary.”

Studies which are excluded after assessment of full text will be listed in a table of excluded studies along with the reason for their exclusion.

In instances where multiple records may report on the same study, all such records that are determined relevant according to the criteria stated above will be advanced for data extraction.

For any non-English articles advanced to full-text screening (i.e. those with an abstract in English and determined to be clearly relevant to our study question), we will obtain an English translation.

2.5. Data Collection Process and Data Elements

Two authors (EC,EK) will independently extract study characteristics and data from each included study in DistillerSR. Authors will review any relevant retraction statements and errata for the included studies and incorporate these as appropriate during data extraction. The data extracted by each author will be compared for quality assurance/quality control, and if needed a third author (HK) will be consulted to resolve any discrepancies. In addition, a quality control check of 20% of the data will be performed by a third author (HK).

The extracted data will be used to assess risk of bias and as inputs to the data analysis. Data extraction will be conducted in two phases. The initial phase will involve extraction of study characteristics. Risk of Bias assessment (see below) will then be conducted, followed by the second phase of data extraction, extraction of data (study results). Data extraction forms for study characteristics and study results are shown in Appendix 2.

The draft data extraction forms were pilot tested by two authors (EC,EK) using two studies selected by a third author (DA) known to that author to be PECOS-relevant (Braun et al. 2012; Chiodo et al. 2004). No substantial changes were identified to be necessary based on pilot testing; some adjustments were made to the formatting of the DistillerSR data extraction forms to better accommodate studies reporting multiple measures of exposure and multiple effect estimates.

Key characteristics of the included studies will be summarized in a table, using the format shown in Table 1.

Table 1.

Key characteristics of the included studies – table shell.

Study Identification Reference Location Years in which blood samples were collected	Sample Size	Blood Lead Levels (BLLs) Age at blood sample collection Central tendency BLL +SD Range (min-max or interquartile range) % <5 μg/dL (report each statistic as available for entire study population and by study groups)	BLL Methods Blood draw method (venous or capillary) LOD % <LOD	IQ Assessment IQ test name Age at IQ test	Confounders Assessed
Study 1
Study 2
Etc.

For any studies that do not report all the data needed for risk of bias assessment and/or data analysis, we will request these data from the study contact author by email. If study authors do not respond to requests after being contacted through 2 email messages over the course of 1 month, review authors will note that attempts to contact study researchers were unsuccessful. Any pertinent information obtained from personal communications with study authors will be made publicly available (e.g., as supplemental materials to the final journal publication of the systematic review).

In instances where multiple records report on the same study, information from the multiple records will be combined for data extraction, and the information will be carried forward for risk of bias (RoB) assessment and data analysis as a single study rather than multiple studies. For any papers that present results from multiple independent cohorts, each cohort will be treated as a separate study with its own data extraction form.

2.6. Risk of Bias Assessment

Studies meeting inclusion criteria will be evaluated for RoB using the approach applied in the Navigation Guide systematic review of PBDEs and neurodevelopmental effects (Lam et al. 2017), with adaptation to the context of our study question and additional clarifying edits. RoB questions address the following domains: selection bias, blinding, exposure assessment, outcome assessment, confounding, incomplete outcome data, selective outcome reporting, financial conflict of interest, and other. The specific RoB questions and instructions are provided in Appendix 3. For the confounding domain, revisions to the Navigation Guide instructions were made to the lists of “important confounders” and “other potentially important confounders” for the context of our lead systematic review based on expert opinion and knowledge gathered from the literature (U.S. Environmental Protection Agency 2013).

Each study will be independently evaluated by two review authors (EC, EK), and their judgments recorded in a structured form created in Microsoft Excel (www.microsoft.com). For each study, each domain will be rated as “high”, “probably high”, “probably low”, or “low” risk of bias depending on the standardized evaluation criteria (see Appendix 3). The rationale for each rating will be recorded. Any discrepancies in judgments between authors will be reconciled after the independent evaluations are completed. If necessary, a third author (HK) will be consulted to resolve any discrepancies. Discrepancies and their resolution will be recorded in Excel. Reviewers shall comment on the potential direction and magnitude of bias only when there is empirical evidence to support a judgment.

The risk of bias instructions were pilot tested by two authors (EC,EK) using two studies selected by a third author (DA) known to that author to be PECOS-relevant (Braun et al. 2012; Chiodo et al. 2004). Based on pilot testing, the risk of bias instructions were clarified for the selection bias, exposure assessment, and confounding domains.

2.7. Data Analysis

We intend to conduct a random-effects meta-analysis to summarize the effects of children’s exposure to lead on IQ scores. Additionally, we plan to perform sensitivity analyses using sub-group analyses and/or meta-regression techniques to assess the impact of study design and study population characteristics to examine potential heterogeneity of results across studies. All PECOS-relevant studies with complete information will be included in the analysis. Characteristics from each study will be compiled and reviewed to establish comparability between studies or to identify data transformations necessary to ensure such comparability. Key characteristics to be considered include:

• Study design

• Geographic region

• Time period of data collection

• Blood lead levels (central tendency and other percentiles)

• Age at measurement of BLL

• Age at IQ assessment

• IQ test/assessment tool used

• Type of BLL data/summary statistic available from the original study (e.g. concurrent BLL vs. average of two or more measurements)

• Child and parental characteristics

• Risk of bias considerations

• Modeling considerations (e.g., stratification, adjustment for potential intermediate variables, etc.)

Summaries of these characteristics for each included study will be assessed by three or more review authors (EK, HK, EC) to determine comparability between studies and to identify any potential sources of heterogeneity. If transformations to reported effect estimates are necessary to ensure a common scale across different tests of intelligence, these will be documented.

After applying necessary data transformations, we will extract the estimated slope of the linear model (including linear models with log transformations; e.g., linear-log models) and its associated standard error for each study. We will then combine effect estimates across comparable studies, using a random effects model to account for potential heterogeneity across studies. Given the diversity of studies that examine the relationship between blood lead levels and IQ, we do not expect that the true effect of each study population would be the same. Therefore, we plan to use a random effects model to estimate the mean of the true effects across studies (Borenstein et al. 2010). Furthermore, if there is notable heterogeneity between studies, as measured by I² and prediction intervals, we will conduct sensitivity analyses to investigate the sources of heterogeneity. Unlike Cochran’s Q, I² is not sensitive to the number of studies included in the meta-analysis. I² will provide a proportional estimate of the variance not explained by sampling error, while prediction intervals provide information on the magnitude of variation (Borenstein et al. 2017). The Cochrane Handbook guide to interpretation of I² is as follows (Higgins et al. 2020):

• 0%–40%: might not be important;

• 30%–60%: may represent moderate heterogeneity;

• 50%–90%: may represent substantial heterogeneity;

• 75%–100%: considerable heterogeneity.

Sensitivity analyses may include subgroup analyses and/or meta-regression depending on the number of studies included in the subgroups of interest. Some of the study characteristics that we plan to examine as potential sources of heterogeneity include central tendency and other percentiles of BLLs, age at measurement of BLL, age at IQ assessment, IQ assessment type, the type of BLL metric, and risk of bias judgments.

The primary quantitative result of our analysis will be the summary estimate of the slope of the linear concentration-response model and the corresponding prediction interval for studies with central tendency BLLs < 5 μg/dL. Sensitivity analysis results may include a combination of the following elements: 1) summary estimate of the slope of the linear concentration-response model and the corresponding prediction interval for studies with central tendency BLLs < 10 μg/dL; 2) subgroup estimates of the slope of the linear concentration-response model and corresponding prediction intervals; 3) the difference in subgroup estimates of the slope and corresponding confidence intervals; 4) meta-regression slopes and confidence intervals estimating the impact of study characteristics on effect size; and/or 5) meta-regression R² indices quantifying the proportion of variance explained by study characteristics. While we cannot formally assess the shape of the concentration-response relationship across studies without individual-level study data, meta-regression techniques allow us to assess whether study-level BLLs modify the relationship between BLL and IQ.

If we identify any eligible studies from the grey literature that have not been peer reviewed, we will conduct separate analyses with and without such studies. An important objective of this work is to derive an updated estimate of the relationship between BLL and IQ for use in EPA regulatory support documents. EPA’s guidance on data quality emphasizes that peer review is an important consideration in selecting studies for regulatory support analyses (U.S. Environmental Protection Agency 2003). It is therefore useful to conduct a version of the data analysis that excludes studies lacking peer review.

Finally, if there are enough studies, we will assess the potential for publication bias using funnel plots and Egger regression on the estimates of effect size (Light and Pillemer 1984), and predict the impact of hypothetical “missing” studies (Duval and Tweedie 2000).

All analyses will be conducted in R (https://www.r-project.org) or a similar statistical analysis tool. In the event that these proposed methods for data analysis are altered to tailor to the evidence base from included studies, the protocol will be amended accordingly and the rationale for any changes will be justified in the documentation.

2.8. Assessing Confidence in the Effect Estimate

We will assess confidence in the effect estimate from the meta-analyses/meta-regressions by considering, at a minimum, the following factors: risk of bias, consistency, precision, magnitude of effect, dose-response relationship, reporting and publication bias, and external validity. A narrative judgment for each characteristic will be developed by group discussion among all authors, based on the considerations described in Table 2. Sources consulted in development of the considerations in Table 2 included Grading of Recommendations Assessment, Development, and Evaluation (GRADE) guidelines (Balshem et al. 2011; Guyatt et al. 2011), the Navigation Guide PBDEs systematic review protocol, (Lam et al. 2015), the NTP Office of Health Assessment and Translation systematic review handbook (National Toxicology Program 2019), EPA’s Preamble to the Integrated Science Assessments (U.S. Environmental Protection Agency 2015), and the draft handbook for EPA’s Integrated Risk Information System (U.S. Environmental Protection Agency 2020).

Table 2.

Considerations for evaluating characteristics of the body of evidence

Characteristic	Considerations
Risk of bias	Risk of bias is considered for the set of studies underlying the effect estimate as a whole, with greatest emphasis on risk of bias in those studies with the greatest weight in the meta-analysis or meta-regression. Few or limited indications of serious risk of bias lends greater confidence to the effect estimate. Confidence in the effect estimate is reduced if there is strong reason to believe that substantial risk of bias affects the body of evidence as a whole and has a substantial impact on the effect estimate. If there is substantial risk of bias, authors should indicate anticipated direction of the bias if evidence supports such a conclusion.
Consistency	Consistency in effect estimates across different studies conducted in disparate populations (e.g., in different locations) lends greater confidence to the effect estimate. Considerations in determining inconsistency include similarity of point estimates, overlap of confidence intervals, and statistical tests of heterogeneity. Some degree of inconsistency across studies is expected due to differences in the underlying distribution of risk factors (covariates) across cohorts. Unexplained large inconsistency in effect estimates from different studies reduces confidence in the overall effect estimate; however confidence is not reduced if inconsistency can be explained by factors such as differences in study designs or differences in cohort characteristics. If inconsistency is found, several potential sources of inconsistency (see list in Data Analysis section) will be considered for evaluation by meta-regression to inform judgment of this characteristic.
Precision	Greater precision is indicated by a relatively narrow confidence interval for a meta-estimate and lends greater confidence to the effect estimate. Lower precision reduces confidence in a meta-estimate.
Magnitude of effect	A large magnitude of effect provides confidence that an observed effect estimate is not strongly influenced by chance or confounding, and thus can increase confidence in an effect estimate. However, a smaller magnitude of effect does not reduce confidence.
Dose-response relationship	A dose-response gradient (increasing effect associated with greater exposure) within individual studies or across studies in the body of evidence increases confidence in an effect estimate.
Reporting and publication bias	Studies with statistically significant results are more likely to be published in peer-reviewed journals, which can result in overestimates of effect sizes. Some degree of publication bias may be expected for any research topic; confidence in an effect estimate is reduced only when there is strong reason to believe that publication bias has strongly affected the effect estimate. Comprehensive searching, including searching of the grey literature and searching for studies in languages other than English may mitigate concerns for publication bias. Funnel plots and statistical techniques may aid in assessing the presence of publication bias and its potential impact on an effect estimate, but these approaches are less reliable with a smaller body of evidence.1
External validity	Confidence is reduced only when there is strong reason to think that the biology in the population of interest, or the nature of its exposure, is so different than the population tested that the magnitude of effect will differ substantially.

¹Some factors typically considered as likely to indicate potential for publication bias, such as findings of “early studies,” are not applicable to our study question which concerns an effect that has been studied for several decades now.

After completion of the narrative judgment for each characteristic, a Confidence Level rating, indicating confidence in the effect estimate, will be assigned to the effect estimate by group discussion among all authors. Choices for the confidence level rating are: High, Moderate, Low, or Very Low. Definitions for each of the confidence levels, which were adapted from the GRADE guidelines (Balshem et al. 2011), are shown in Table 3. Selection of the confidence level by the authors will be based on the assessment of characteristics in Table 2 and the confidence level definitions in Table 3. If a confidence level other than High is selected, authors will comment (to the extent possible) on whether the overall uncertainty in the effect estimate is expected to represent bias in a particular direction or is expected to represent random error. The format for reporting the authors’ consensus judgments is shown in Table 4. Multiple versions of Table 4, each corresponding to a different effect estimate, may be prepared depending on the results and findings of the meta-analysis and meta-regression.

Table 3.

Confidence in the effect estimate: confidence levels and their definitions.

Confidence Level	Definition
High	We are very confident that the true effect lies close to that of the estimate of the effect. Further research is very unlikely to change our confidence in the estimate of effect.
Moderate	We are moderately confident in the effect estimate: The true effect is likely to be close to the estimate of the effect, but there is a possibility that it is substantially different. Further research could have an important impact on our confidence in the estimate of effect and may change the estimate.
Low	Our confidence in the effect estimate is limited: The true effect may be substantially different from the estimate of the effect. Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate.
Very Low	We have very little confidence in the effect estimate: The true effect is likely to be substantially different from the estimate of effect. Any estimate of effect is very uncertain

Adapted from GRADE guidelines (Balshem et al. 2011).

Table 4.

Format for reporting conclusions regarding level of confidence in an effect estimate – table shell.

Characteristic	Assessment
Risk of bias	[Narrative assessment of considerations shown in Table 2]
Consistency	[Narrative assessment of considerations shown in Table 2]
Precision	[Narrative assessment of considerations shown in Table 2]
Magnitude of effect	[Narrative assessment of considerations shown in Table 2]
Dose-response relationship	[Narrative assessment of considerations shown in Table 2]
Reporting and publication bias	[Narrative assessment of considerations shown in Table 2]
External validity	[Narrative assessment of considerations shown in Table 2]
Overall Confidence Level	[High, Moderate, Low, or Very Low; based on definitions shown in Table 3]
Effect estimate: Number of studies: Number of study participants: Figure/table reference: