Abstract
Background
A critical component of systematic review methodology is the assessment of the risks of bias of studies that are included in the review. There is controversy about whether funding source should be included in a risk of bias assessment of animal toxicology studies.
Objective
To determine whether industry research sponsorship is associated with methodological biases, the results, or conclusions of animal studies examining the effect of exposure to atrazine on reproductive or developmental outcomes.
Methods
We searched multiple electronic databases and the reference lists of relevant articles to identify original research studies examining the effect of any dose of atrazine exposure at any life stage on reproduction or development in non-human animals. We compared methodological risks of bias, the conclusions of the studies, the statistical significance of the findings, and the magnitude of effect estimates between industry sponsored and non-industry sponsored studies.
Results
Fifty-one studies met the inclusion criteria. There were no differences in methodological risks of bias in industry versus non-industry sponsored studies. 39 studies tested environmentally relevant concentrations of atrazine (11 industry sponsored, 24 non-industry sponsored, 4 with no funding disclosures). Non-industry sponsored studies (12/24, 50.0%) were more likely to conclude that atrazine was harmful compared to industry sponsored studies (2/11, 18.1%) (p value = 0.07). A higher proportion of non-industry sponsored studies reported statistically significant harmful effects (8/24, 33.3%) compared to industry-sponsored studies (1/11; 9.1%) (p value = 0.13). The association of industry sponsorship with decreased effect sizes for harm outcomes was inconclusive.
Conclusion
Our findings support the inclusion of research sponsorship as a risk of bias criterion in tools used to assess risks of bias in animal studies for systematic reviews. The reporting of other empirically based risk of bias criteria for animal studies, such as blinded outcome assessment, randomization, and all animals included in analyses, needs to improve to facilitate the assessment of studies for systematic reviews.
Keywords: Bias, conflicts of interest, chemical, funding source, atrazine, systematic review, meta-analysis, research synthesis, risk assessment, toxicology, animal research
Introduction
Results from animal studies are a critical, and often the only, input to assessing potential harm from exposure to chemicals. However, the lack of reproducibility of findings from animal research has reduced public confidence in the utility of animal experiments [1] and led to claims that animal research is a waste of financial resources [2]. These problems with animal research have resulted in significant debate about how to assess biases in animal studies used in systematic reviews, risk assessments and other regulatory decisions [3-5]. A critical component of systematic review methodology is the assessment of the risks of bias of studies that are included in the review.
Risk of bias occurs when the methodological characteristics of a study produce a systematic error in the magnitude or direction of the results [6]. Bias can shift effect estimates to be larger or smaller. For example, in controlled human clinical drug trials, studies with a high risk of bias (such as those lacking randomization, allocation concealment, or blinding of participants and outcome assessors) produce larger treatment effect sizes, thus falsely inflating the efficacy of the test interventions, compared to studies that have these design features [7, 8]. However, biased human studies assessing the harms of drugs are more likely to report smaller estimates of adverse effects [9].
Less is known about methodological risks of bias in animal studies, although a systematic review of instruments for assessing risks of bias in animal studies identified criteria that have been shown empirically to bias effect estimates in animal models [13]. For example, analyses of animal studies examining interventions for stroke, multiple sclerosis and trauma have shown that lack of randomization, blinding, specification of inclusion and exclusion criteria, statistical power, and failure to use comorbid animals are associated with inflated effect estimates of pharmaceutical interventions [10-12].
Industry funding for research and industry relationships with academic researchers pose an additional risk of bias. Considerable evidence shows a strong association between industry funding, investigator financial conflicts of interest, and biased outcomes in clinical research, even when controlling for methodological characteristics of the studies [14]. There is little evidence regarding the influence of these conflicts of interest on the outcomes of animal research [15-17]. There are conflicting results concerning the association of industry funding and research outcomes among the small cohorts of animal studies that have been examined and further research on the influence of conflicts of interest on animal studies is needed [16, 17]. There is controversy about whether funding source should be included in risk of bias assessments for studies included in systematic reviews [19].
Atrazine (6-chloro-N-ethyl-N′-(1-methylethyl)-1,3,5-triazine-2,4-diamine) is used as an herbicide. Atrazine is commonly found in drinking water in the United States. The EPA has concluded that “atrazine is an endocrine disruptor”[20] but not that atrazine affects amphibian sexual development [21]. As of 2013, the EPA has not altered these conclusions [22]. Atrazine studies are a good topic for an analysis of funding bias because concerns have been raised about the influence of industry sponsorship on the design and results of studies examining the effects of atrazine on reproductive and developmental outcomes [23].
The objective of this study is to determine whether industry research sponsorship is associated with the methods, conclusions, or results of animal studies examining the effect of exposure to atrazine on reproductive or developmental outcomes. We test three specific hypotheses. First, we hypothesize that industry sponsored studies will be less likely to have conclusions indicating harm from atrazine than non-industry sponsored studies. Second, we test the hypothesis that industry sponsored studies will be less likely to report statistically significant results indicating harm from atrazine than non-industry sponsored studies. Third, we test the hypothesis that industry sponsored studies will have smaller effect estimates of harm than non-industry sponsored studies. In addition, we compare the methodological risks of bias of industry sponsored vs. non-industry sponsored studies to determine if there are differences in the methods of the studies.
Methods
We searched for studies that addressed the following question: “Does exposure to atrazine have adverse reproductive or developmental effects in non-human animals”? We searched for studies that had non-human animal subjects that were exposed to any dose of atrazine during any life stage. Exposure levels of atrazine were classified and adverse outcomes were grouped as described below.
Inclusion / Exclusion Criteria
Articles were included if they met the following criteria: (1) study conducted using whole animals; (2) original research, defined as a study that presented original data and did not specifically state that it was a review; (3) atrazine compared to no exposure or control (eg, vehicle or some other exposure); (4) contains at least one group receiving only atrazine exposure; and (5) reports results data for at least one developmental and/or reproductive health outcome.
Studies were excluded if they met any of the following criteria: (1) pharmacokinetic or pharmacodynamic studies; (2) editorials, letters to the editor, commentaries, abstracts, unpublished reports, systematic reviews, meta-analyses; (3) studies comparing only different doses of atrazine; (4) studies in which atrazine was present in all the comparison groups; (5) in vitro-analysis; (6) studies with no comparison groups.
Abstracts and article titles were first screened for inclusion. The full text of each article was then discussed by two authors who made a final decision about inclusion.
Search Strategy
There we no language restrictions for the search. We searched Medline from January 1, 1966 to June 26th, 2013 using a search term combination containing the following MeSH terms, text words and word variants:
(atrazine) AND (animal* OR preclinical OR “pre-clinical” OR mice OR rats OR rabbits OR dog OR dogs OR monkey OR monkeys OR “animal experimentation”[MeSH Terms] OR “models, animal” [MeSH Terms] OR “invertebrates” [MeSH Terms] OR “Animals” [MH] OR “animal population groups” [MeSH Terms]) NOT (humans[mh] NOT animals[mh:noexp]) AND (health effect OR health effects OR toxic OR toxicity OR toxicities OR efficacy OR efficacies OR toxicology OR safety OR harm* OR drug effects[sh] OR therapeutic use[sh:noexp] OR adverse effects[sh] OR poisoning[sh] OR pharmacology[sh:noexp] OR chemically induced[sh]) AND eng[la] NOT review[pt] NOT systematic review* NOT meta-analysis[pt]
We also searched, between May 1 and July 30, 2013 the following toxicology databases for articles that met our inclusion criteria:
EPA Science Inventory http://www.epa.gov/gateway/science/
NIOSHTIC 2 http://www2.cdc.gov/nioshtic2/Nioshtic2.htm
Toxline http://toxnet.nlm.nih.gov/cgi-bin/sis/htmlgen?TOXLINE
USEPA Health and Environmental Studies Online http://hero.epa.gov/
TSCA Test Submissions: http://www.ntis.gov/products/ots.aspx
We identified 11 additional citations that were not in Medline. Nine could not be obtained even after contacting the authors. Two went on to full text screening. We searched the reference lists of all articles that met the inclusion criteria and identified one additional reference. Of the 3 additional references identified, one did not meet the inclusion criteria after full text screening.
Data Extraction
Single-Coded Data Collection
DK collected the following characteristics from each included study:
Study citation information
Title of the study, month of publication, year of publication, and journal name.
Author affiliation
Author(s) affiliation(s) was obtained from the article and classified into (1) industry, if all authors were employed by industry (2) non-industry, if no author was employed by industry, or (3) combined, if at least one author was employed by industry and at least one author was not employed by industry. If a single author had affiliations with industry and non-industry sources, the study was coded as “combined”.
Study design criteria
The following information was extracted from each study: (1) dose(s) of atrazine tested; (2) animal species and strain; (3) number of control and treated animals at the start of the study; (4) whether or not the investigator(s) assessed the effect of environmentally relevant concentrations (i.e. less than 1.89 ppm) of atrazine. We defined “environmentally relevant” as concentrations at or below 1.89 ppm. 18.9 ppb is considered the average concentration in community water systems according to the Agency for Toxic Substances and Disease Registry (ATSDR), and we multiplied that value by 100 to account for uncertainty and variability in the estimates.
Double-Coded Data Collection
Two coders (DK and HV) independently collected data on characteristics of the studies that required judgment: methodological risks of bias, effect size and reported statistical significance of outcomes, study sponsorship source(s), investigator financial ties, and author conclusions. A third coder (LB) resolved any discrepancies.
Coding of Study Design Criteria to Assess Risk of Bias and other Methodological Criteria
We assessed risk of bias for the included studies using criteria derived from a review of tools for assessing bias in animal studies [13].
For each publication, we coded each criterion as (1) yes, if the criteria were met (low risk of bias); (2) no, if the criteria were not met (high risk of bias); and when applicable (3) partial, if the criteria were partially met. The following criteria were assessed:
Randomization
Was the treatment randomly allocated to animal subjects so that each subject has an equal likelihood of receiving the intervention? Randomization was coded as (1) yes (2) no (3) partial.
Concealment of allocation
Were processes used to protect against selection bias by concealing from the investigators how treatment was allocated at the start of the study? Concealment of allocation was coded as (1) yes (2) no (3) partial.
Blinded outcome assessment
Was the investigator(s) involved with performing the experiment, collecting data, and assessing the outcome of the experiment unaware of which subjects received the treatment and which did not? Blinding was coded as (1) yes or (2) no.
Statement of compliance with animal welfare requirements
Did the author(s) state whether or not they complied with regulatory requirements for the handling and treatment of test animals? Statement of compliance with animal welfare requirements was coded as (1) yes or (2) no.
Test animal description
Did the author(s) describe in detail the test animal characteristics including, the animal species, strain, sub-strain, genetic background, age, supplier, sex, weight. At least one of these characteristics must be reported for this criterion to be coded as “yes.” Test animal description was coded as (1) yes or (2) no.
Environmental parameters
Did the author(s) adequately describe the housing and husbandry, nutrition, water, temperature, lighting conditions? At least one of these characteristics must be present for this criterion to be met. Environmental parameters were coded as (1) yes (2) no (3) partial.
Sample size calculation
Did the authors perform a sample size calculation to justify the total number of animals used in the study? Sample size calculation was coded as (1) yes or (2) no.
Inclusion/exclusion criteria
Were criteria used for including or excluding subjects specified? Inclusion/Exclusion criteria were coded as (1) yes (2) no (3) partial.
Exposure levels justified
Did the authors explain why the atrazine exposures levels were studied and whether a dose/response model was used to select the dose(s)? Exposure levels justified was coded as (1) yes or (2) no.
Timing of exposure (whether exposure period was sufficient to address outcome of interest)
Did the authors use a sufficient exposure period to address the outcome of interest? Timing of exposure was coded as (1) yes or (2) no.
Optimal time window for outcome assessment
Did the investigator provide sufficient time to pass before assessing the outcome? The optimal time window used in animal research should reflect the time needed to see the outcome. For example, it may take longer to see tumor development than molecular changes. Optimal time window for outcome assessment was coded as (1) yes (2) no (3) partial.
All animals accounted for
Did the investigator describe if animals were not included in the analysis and why they were not included? All animals accounted for was coded as (1) yes (2) no (3) partial.
Sponsorship Source
Study sponsorship source as disclosed in each publication was coded as (1) any industry; (2) non-industry; (3) no sponsorship statement; and (4) no sponsorship.
Financial Ties of Authors
Investigator financial ties as disclosed in each publication were coded as (1) if at least one author of the study reported having a financial conflict of interest; (2) all authors reported having no conflicts of interest; (3) there was no disclosure statement. It was beyond the scope of this study to search for undisclosed author financial ties.
Role of the Financial Sponsor
For studies that disclosed a sponsor of any type, the role of the sponsor was coded as (1) not mentioned; (2) statement that the sponsor was not involved; (3) statement that the sponsor was involved. When applicable, we reported the sponsor's involvement (e.g., in the design, analysis or dissemination of the study).
Coding of Conclusions of Included Studies
To test the hypothesis that industry sponsored studies would be less likely to have conclusions indicating harm from atrazine than non-industry sponsored studies, we coded the conclusions of each paper as reported in the abstract and conclusion sections. We coded author conclusions as (1) “harmful” if the overall conclusion suggested that atrazine was not safe at any of the doses tested; (2) “harmful at environmentally relevant concentrations” if the overall conclusion suggested that atrazine was not safe at concentrations below 1.89 ppm (i.e. the upper limit of what is considered “environmentally relevant”); (3) “harmful at above environmentally relevant concentrations” if the author(s) only tested atrazine at high concentrations and concluded that atrazine was not safe at those high concentrations; (4) “neutral” if the paper did not draw a conclusion regarding the safety of atrazine or stated that the limitations of the study were so severe that the results were not valid; and (5) “no effect” if the overall conclusion suggested that atrazine was safe.
Coding of Results of Included Studies
We report 1) the number of studies that reported statistically significant results indicating harm and 2) the magnitude of the harm effect estimates.
To test the hypothesis that industry sponsored studies would be less likely to report statistically significant results indicating harm from atrazine than non-industry sponsored studies, we assigned a code to each paper as follows.
Coding of individual outcome results
Only results for developmental or reproductive outcomes were recorded. Many different outcomes were reported and they were grouped into the following categories: 1) developmental abnormalities / mortality, 2) growth, 3) time to developmental landmark, 4) organ weight / size, 5) developmental enzymes, 6) reproductive hormones and 7) reproductive ability. See Supplemental File 1 for the specific outcomes included in each category.
For the purposes of this study, we coded a developmental or reproductive health related outcome result as “harmful” if the author(s) reported a statistically significant (for example, p < 0.05) adverse reproductive or developmental health effect. We used the level of statistical significance reported in the paper. We coded that atrazine had “no effect” on reproductive or developmental health if the author(s) reported a statistically non-significant adverse effect (for example, p ≥ 0.05) or a statistically non-significant beneficial effect. We coded an effect as “beneficial” if atrazine showed a reproductive or developmental health benefit.
For studies that assessed a range of atrazine concentrations for a given outcome, we developed a decision rule for assigning a code for that outcome. If the only concentrations evaluated for an outcome were above environmentally relevant concentrations, we did not code the result as we are interested in the effect of atrazine at environmentally relevant concentrations. If an outcome was measured at concentrations both above and below 1.89 ppm, we coded only the values for the environmentally relevant concentrations. Therefore, if any of the exposure concentrations that tested harmful coincided with environmentally relevant concentrations, we coded the outcome as “harmful.”
Coding of results at the level of the study
A paper was coded as reporting “harmful” results if it reported more statistically significant outcomes demonstrating harmful effects than demonstrating no or not harmful effects; as “no effect” if it reported more outcomes that showed “no effect” compared to statistically significant harmful or not harmful outcomes; and as “neutral” if there an equal number of “no effect” and “harmful” outcomes.
Magnitude of Effect Estimates of Included Studies
We conducted meta-analyses to test the hypothesis that industry sponsored studies would have smaller effect estimates of harm than non-industry sponsored studies.
Data collection for meta-analysis
For each individual outcome, we collected the following data (often derived from tables, graphs, figures, etc.): measure of effect, confidence interval, measure of variability (eg, standard deviation (SD) or standard error (SE)), p-value, statistical test used for each outcome, and the number of treated and untreated animals. If multiple time points were reported, we included all time points in the meta-analysis as to not assume a primary endpoint or arbitrarily assign an endpoint in the analysis.
Meta-analysis
We conducted a meta-analysis of the studies that had analyzable data. For a study to have analyzable data, it needed to report both a mean value and a measure of dispersion (standard error or standard deviation) or provide adequate data so that we could calculate these measures ourselves.
We calculated the effect of atrazine using a standardized mean difference (SMD) for each outcome. Due to the lack of independence of animals between outcomes within studies, we averaged SMDs and variances across outcomes for each study, yielding k average SMDs and variances for k studies. We pooled the data across studies and estimated summary average SMDs using random-effects models [24]. Specifically, we estimated the average SMD for each included study and used the inverse variance method to calculate study weights. The inverse variance method assumes that the variance for each study is inversely proportional to its importance; therefore, more weight is given to studies with less variance than studies with greater variance. The SMD null hypothesis (Ho: estimate = 0) states that there is no difference in effect of atrazine exposure on the specific outcomes when compared to a control.
For meta-analysis including outcomes for which the harmful direction is indicated as an increase, a number greater than zero suggests that atrazine is harmful when compared to the control exposure. Similarly, for meta-analyses including outcomes for which the harmful direction is indicated as a decrease, a number less than zero suggests that atrazine is harmful when compared to the control exposure.
We grouped outcomes within each outcome category according to whether an increase or decrease in the measure indicated a harmful outcome. A detailed description of whether a unit increase and decrease was considered to be harmful by outcome can be found in Supplemental Table 2. We stratified the direction of harm by increase and decrease because some outcome categories (e.g., reproductive hormones) are represented in both meta-analyses. For example, the meta-analyses of an outcome category (e.g., reproductive hormones) indicated by a unit increase causing harm (e.g. an increase in the hormone estradiol) and of a unit decrease causing harm (e.g. an increase in the hormone testosterone) contain independent data between meta-analyses.
We examined heterogeneity among the studies using the I2 statistic. We interpreted an I2 estimate greater than 50% as indicating moderate or high levels of heterogeneity. We anticipated high levels of heterogeneity as previous meta-analyses of animal studies have found high levels of heterogeneity between studies, potentially resulting from typical, small sample sizes in animal models [25].
We evaluated differences in pooled effect estimates by sponsorship sources to test the hypothesis that published industry sponsored studies are more likely to have outcomes that favor their sponsors (that is, outcome indicating less harm) than non-industry sponsored studies. We investigated the potential causes of heterogeneity by conducting a priori subgroup analyses using the χ2 statistic with a significance level of 0.10. We conducted a subgroup analysis by type of outcome measure because we hypothesized that the effects of atrazine could vary by outcome and harm direction.
Additional Statistical Analysis
In addition to the effect estimates from the meta-analyses, we report the frequencies of each risk of bias criterion, study characteristics and the coding of the results and conclusions by sponsorship source.
Results
Our searches identified 51 studies that met the inclusion criteria (Figure 1). There were 12 industry funded studies, 29 non-industry and 10 with no disclosures. The studies were published between 1984 and 2013 with the majority (41 studies) published between 2005 and 2010. Amphibians, fish and reptiles were most commonly studied (n = 39), followed by rats (n=8), birds / fowl (n=3) and mice (n=1).
Figure 1. Flow of Included Studies.
Methodological risks of bias and study characteristics
There were no differences in methodological risks of bias in industry versus non-industry sponsored studies. The majority of studies gave a description of the test animals and the animal environment and reported that they were compliant with animal welfare requirements (Table 1). About half of the studies (26 of 51) reported that they were randomized, although none described concealment of allocation. Blinded outcome assessment, inclusion and exclusion criteria for the animals, sample size calculations, optimal time window investigated, and accounting for all animals were rarely or never reported. Industry sponsored studies were less likely to justify their choice of atrazine exposure levels (4/12 studies, 33%) compared to non-industry sponsored studies (19/29, 66%) (p value = 0.06).
Table 1. Characteristics and Risks of Bias of Included Studies by Sponsorship Source (n=51).
| Characteristic | Category | Sponsorship Source | ||
|---|---|---|---|---|
| Any Industry (N =12) | Non-Industry (N =29) | No Disclosure (N =10) | ||
| Risk of Bias3 | Randomization | 6 | 15 | 5 |
| Concealment of allocation | 0 | 0 | 0 | |
| Blinded outcome assessment | 4 | 4 | 0 | |
| Inclusion/exclusion criteria | 0 | 2 | 0 | |
| Sample size calculation | 0 | 0 | 0 | |
| Test animal description | 12 | 29 | 10 | |
| Animal environment described | 12 | 29 | 9 | |
| Exposure levels justified | 4 | 19 | 5 | |
| Optimal time window investigated | 1 | 0 | 0 | |
| All animals accounted for | 2 | 3 | 2 | |
| Compliant with animal welfare requirements | 7 | 16 | 3 | |
| Financial conflict of interest | Yes | 1 | 0 | 0 |
| No | 4 | 4 | 1 | |
| No disclosure | 7 | 25 | 9 | |
| Author affiliation | Combined | 3 | 1 | 1 |
| Non-industry employed | 9 | 28 | 8 | |
| Industry | 0 | 0 | 1 | |
Eighty percent (41/51) of studies had no disclosures about the financial conflicts of interest of the authors, although the majority of studies had authors who were not industry employees.
Conclusions of studies
Of the 51 included studies, 39 (77%) reported that they tested environmentally relevant concentrations of atrazine: 11 of 12 (92%) industry sponsored studies, 24 of 29 (83%) non-industry sponsored studies, and 4 of the 10 (40%) studies with no funding disclosures.
As shown in Table 2, among studies that tested environmentally relevant concentrations of atrazine, non-industry sponsored studies (12/24, 50%) were more likely to conclude that atrazine was harmful compared to industry sponsored studies (2/11, 18%) (p value = 0.07) (Table 2).
Table 2. Results and Conclusions of Studies that tested Environmentally Relevant Doses of Atrazine by Sponsorship Source (n=39).
| Sponsorship Source | ||||
|---|---|---|---|---|
| Direction of effect | Any industry (N=11) | Non-industry (N = 24) | No Disclosure (N = 4) | |
| Results | Harmful | 1 | 8 | 1 |
| No effect | 8 | 13 | 3 | |
| Neutral | 2 | 3 | 0 | |
| Conclusions | Harmful | 2 | 12 | 1 |
| No effect | 8 | 10 | 2 | |
| Neutral | 1 | 2 | 1 | |
Statistical significance of study results
As shown in Table 2, among studies that tested environmentally relevant concentrations of atrazine, a higher proportion of non-industry sponsored studies reported statistically significant harmful effects (8/24, 33%) compared to industry-sponsored studies (1/11; 9%) (p value = 0.13).
Magnitude of study results: Meta-analyses of effect estimates
We identified 4 outcome categories for which a higher value indicated greater harm (Figure 2). The outcome of developmental enzymes was not included in Figure 2 as there was no variability in the sources of funding (n=3). The pooled estimate for all studies examining outcomes where the direction of harm is indicated by an increase was 0.82 (95% CI 0.07, 1.57), showing a significant increase in risk of harm among atrazine exposed animals compared to control animals. Non-industry sponsored studies, in general, yielded stronger effects for outcomes in which the direction of harm was an increase than industry-sponsored studies (Figure 2). For the outcome of time to developmental landmark, the pooled effect from industry-sponsored studies was significantly less than the pooled effect from non-industry sponsored studies, indicating less harm in the industry-sponsored studies (p value = 0.04).
Figure 2. Meta-Analyses of the Effect of Atrazine on Harm Outcomes Indicated by an Increase in the Effect Size.
We identified 5 outcome categories for which a lower value indicated harm (Figure 3). The outcome of developmental enzymes was not included in Figure 3 as there was insufficient variability in sponsorship. The pooled estimate for all studies examining the effects of atrazine on outcomes where the direction of harm is indicated by a decrease was -0.05 (95% CI -0.49, 0.38) and yielded no significant effect. Industry-sponsored studies, in general, yielded stronger effects for outcomes in which the direction of harm was a decrease than non-industry sponsored studies.
Figure 3. Meta-Analyses of the Effect of Atrazine on Harm Outcomes Indicated by a Decrease in Effect Size.
Conclusions
We tested three specific hypotheses to determine whether industry sponsorship of research is associated with the results or conclusions of non-human animal studies examining the effect of atrazine exposure on reproductive or developmental outcomes. First, we tested the hypothesis that industry sponsored studies would be less likely to have conclusions indicating harm from atrazine than non-industry sponsored studies. This hypothesis was supported; 81% of industry supported studies did not conclude atrazine is harmful compared to 50% of non-industry supported studies.
Second, we tested the hypothesis that industry sponsored studies would be less likely to report statistically significant results indicating harm from atrazine than non-industry sponsored studies. This hypothesis was supported; 91% of industry sponsored studies did not report statistically significant outcomes for harm compared to 67% of non-industry sponsored studies.
However, the hypothesis that industry sponsored studies would have smaller effect estimates of harm than non-industry sponsored studies was not supported by the meta-analyses conducted. Industry-sponsored studies, in general, yielded larger effects for outcomes in which the direction of harm was a decrease and smaller effects for outcomes in which the direction of harm was an increase than non-industry sponsored studies. Thus, the association of sponsorship with size of effect estimates is inconclusive. The exclusion of studies that did not assess atrazine at environmentally relevant concentrations and the small number of homogeneous outcome measures available for each meta-analysis are possible reasons that consistent differences in sizes of effect estimates between industry and non-industry sponsored studies were not observed.
We did not identify differences in methodological risks of bias between the industry and non-industry sponsored studies. The information needed to assess risks of bias was poorly reported. Blinded outcome assessment, inclusion and exclusion criteria for the animals, sample size calculations, optimal time window investigated, and accounting for all animals were rarely or never reported. About half of the studies reported that they were randomized, although none described concealment of allocation. Risk of bias criteria such as randomization and blinding should be consistently reported as there is empirical evidence that they affect the outcomes of animal research [10, 11, 26]. In order to assess the quality of studies that will be included in systematic reviews all methodological risk of bias criteria must be fully reported. Furthermore, recent guidelines such as the Animal Research: Reporting of In Vivo Experiments (ARRIVE) guidelines recommend the reporting of selected risk of bias criteria [27]. Descriptive criteria, such as information on the test animals and the animal environment, were well reported in the included atrazine studies but are more relevant for determining compliance with regulatory requirements than assessing risks of bias [13].
Since the differences in findings between industry and non-industry sponsored studies could not be explained by differences in the methodological risks of bias in the studies, another possible explanation is publication bias [25, 26, 29] or selective outcome reporting. Publication bias is the failure to publish entire studies with statistically significant results and reporting bias is the failure to publish specific outcomes with statistically nonsignificant results. We could not assess the extent of selective outcome reporting in the studies included in our analysis because we had access only to the published reports and not the raw data.
Our findings suggest that reporting on conflicts of interest in animal toxicology studies also needs improvement. Eighty percent of the included articles failed to include any conflict of interest disclosures for the authors and twenty percent did not disclose the funding sources for the research. The reporting of funding sources is now common practice among clinical journals [28]. Given the prevalence of academic industry interactions in the field of toxicology and chemical risk assessment, similar requirements for reporting should be enforced for journals in environmental science and toxicology.
Complete and accurate information on exposure is critical for studies that will be used in systematic reviews assessing risks of chemical exposures. In this study, we classified exposures as environmentally relevant or not according to an accepted standard. We found that approximately three-quarters of the studies assessed atrazine at environmentally relevant exposures and limited our analyses to these studies. Although there were no differences in the proportions of industry sponsored and non-industry sponsored studies that tested atrazine at environmentally relevant exposures, the industry sponsored studies were less likely to justify their choice of atrazine exposure levels.
Our study has a number of limitations. Although we searched multiple databases, we may have missed studies that were unpublished. The decision to exclude studies that did not test atrazine at environmentally relevant concentrations reduced our sample size. The heterogeneity of outcome measures in the included studies, even when grouped into similar categories, also limited the number of studies available for the meta-analyses testing the association of industry sponsorship with size of effect estimates.
In summary, industry sponsored studies were more likely to report findings that favored the sponsor. The non-industry sponsored studies were more likely to report statistically significant results and have conclusions indicating that atrazine is harmful compared to industry sponsored studies. This finding supports the inclusion of research sponsorship as a risk of bias criteria in tools used to assess risks of bias in animal studies for systematic reviews. The reporting of other risk of bias criteria for animal studies, such as blinded outcome assessment, randomization, and all animals included in analyses, needs to improve to facilitate the assessment of studies for systematic reviews.
Supplementary Material
Highlights.
A critical component of systematic reviews is the assessment of risks of bias of included studies.
There is controversy about whether funding source should be included in risk of bias assessments.
This study examined the association of industry sponsorship with the methods, conclusions, or results of animal studies of atrazine.
Industry sponsored studies were more likely than studies with other funders to report results and conclusions that favoured the sponsor.
There were no differences in methodological risks of bias between industry and non-industry sponsored studies.
Research sponsorship should be included as a criterion in tools for assessing risks of bias.
Acknowledgments
Funding Source: National Institute of Environmental Health Sciences (Grant # R21ES021028) (http://www.niehs.nih.gov). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Footnotes
Conflict of Interest: The authors declare no competing conflicts of interest.
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
References
- 1.van der Worp HB, et al. Can animal models of disease reliably inform human studies? PLoS Med. 2010;7(3):e1000245. doi: 10.1371/journal.pmed.1000245. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Macleod MR, et al. Biomedical research: increasing value, reducing waste. Lancet. 2014;383(9912):101–4. doi: 10.1016/S0140-6736(13)62329-6. [DOI] [PubMed] [Google Scholar]
- 3.Woodruff TJ, Sutton P G. Navigation Guide Work. An evidence-based medicine methodology to bridge the gap between clinical and environmental health sciences. Health Aff (Millwood) 2011;30(5):931–7. doi: 10.1377/hlthaff.2010.1219. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Rooney AA, et al. Systematic review and evidence integration for literature-based environmental health science assessments. Environ Health Perspect. 2014;122(7):711–8. doi: 10.1289/ehp.1307972. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.National Academies of Science. Review of EPA's Integrated Risk Information System (IRIS) Process. Washington (DC): 2014. [PubMed] [Google Scholar]
- 6.Higgins J, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0. John Wiley and Sons; Chichester, UK: 2011. [Google Scholar]
- 7.Schulz KF, Grimes DA. Blinding in randomised trials: hiding who got what. Lancet. 2002;359(9307):696–700. doi: 10.1016/S0140-6736(02)07816-9. [DOI] [PubMed] [Google Scholar]
- 8.Schulz KF, Grimes DA. Allocation concealment in randomised trials: defending against deciphering. Lancet. 2002;359(9306):614–8. doi: 10.1016/S0140-6736(02)07750-4. [DOI] [PubMed] [Google Scholar]
- 9.Nieto A, et al. Adverse effects of inhaled corticosteroids in funded and nonfunded studies. Arch Intern Med. 2007;167(19):2047–53. doi: 10.1001/archinte.167.19.2047. [DOI] [PubMed] [Google Scholar]
- 10.Bebarta V, Luyten D, Heard K. Emergency medicine animal research: does use of randomization and blinding affect the results? Acad Emerg Med. 2003;10(6):684–7. doi: 10.1111/j.1553-2712.2003.tb00056.x. [DOI] [PubMed] [Google Scholar]
- 11.Crossley NA, et al. Empirical evidence of bias in the design of experimental stroke studies: a metaepidemiologic approach. Stroke. 2008;39(3):929–34. doi: 10.1161/STROKEAHA.107.498725. [DOI] [PubMed] [Google Scholar]
- 12.Sena ES, et al. Factors affecting the apparent efficacy and safety of tissue plasminogen activator in thrombotic occlusion models of stroke: systematic review and meta-analysis. J Cereb Blood Flow Metab. 2010;30(12):1905–13. doi: 10.1038/jcbfm.2010.116. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Krauth D, Woodruff TJ, Bero L. Instruments for assessing risk of bias and other methodological criteria of published animal studies: a systematic review. Environ Health Perspect. 2013;121(9):985–92. doi: 10.1289/ehp.1206389. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Lundh A, et al. Industry sponsorship and research outcome. Cochrane Database Syst Rev. 2012;12:MR000033. doi: 10.1002/14651858.MR000033.pub2. [DOI] [PubMed] [Google Scholar]
- 15.Krauth D, et al. Nonindustry-sponsored preclinical studies on statins yield greater efficacy estimates than industry-sponsored studies: a meta-analysis. PLoS Biol. 2014;12(1):e1001770. doi: 10.1371/journal.pbio.1001770. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Bennett CL, et al. Association between pharmaceutical support and basic science research on erythropoiesis-stimulating agents. Arch Intern Med. 2010;170(16):1490–8. doi: 10.1001/archinternmed.2010.309. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Abdel-Sattar M, et al. The Relationship between Risk of Bias Criteria, Research Outcomes, and Study Sponsorship in a Cohort of Preclinical Thiazolidinedione Animal Studies: A Meta-Analysis. Evid Based Preclin Med. 2014;1(1):11–20. doi: 10.1002/ebm2.5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Macleod M. Some salt with your statin, professor? PLoS Biol. 2014;12(1):e1001768. doi: 10.1371/journal.pbio.1001768. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Bero LA. Why the Cochrane risk of bias tool should include funding source as a standard item. Cochrane Database Syst Rev. 2013;12:ED000075. doi: 10.1002/14651858.ED000075. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Agency EP. Atrazine Chemical Summary. 2007:12. [Google Scholar]
- 21.Agency EP. Atrazine Updates: Amphibians. 2010 July 23, 2014]; Available from: http://www.epa.gov/pesticides/reregistration/atrazine/atrazine_update.htm#amphibian.
- 22.Agency EP. Atrazine Updates: Atrazine Evaluation Process. 2013 July 23, 2014]; Available from: http://www.epa.gov/pesticides/reregistration/atrazine/atrazine_update.htm#atrazine.
- 23.Hayes T. There is no denying this: Defusing the confusion about Atrazine. BioScience. 2004;54(12):1138–1149. [Google Scholar]
- 24.DerSimonian R, Laird N. Meta-analysis in clinical trials. Control Clin Trials. 1986;7(3):177–88. doi: 10.1016/0197-2456(86)90046-2. [DOI] [PubMed] [Google Scholar]
- 25.Macleod MR, et al. Pooling of animal experimental data reveals influence of study design and publication bias. Stroke. 2004;35(5):1203–8. doi: 10.1161/01.STR.0000125719.25853.20. [DOI] [PubMed] [Google Scholar]
- 26.Sena ES, et al. Publication bias in reports of animal stroke studies leads to major overstatement of efficacy. PLoS Biol. 2010;8(3):e1000344. doi: 10.1371/journal.pbio.1000344. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Kilkenny C, et al. Improving bioscience research reporting: the ARRIVE guidelines for reporting animal research. PLoS Biol. 2010;8(6):e1000412. doi: 10.1371/journal.pbio.1000412. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Drazen JM, et al. Toward more uniform conflict disclosures: the updated ICMJE conflict of interest reporting form. Natl Med J India. 2010;23(4):196–7. [PubMed] [Google Scholar]
- 29.ter Riet G, et al. Publication bias in laboratory animal research: a survey on magnitude, drivers, consequences and potential solutions. PLoS One. 2012;7(9):e43404. doi: 10.1371/journal.pone.0043404. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.