Questioning Existing Cancer Hazard Evaluation Standards in the Name of Statistics

Crump et al. (2020) argue that they present “a more comprehensive analysis of the animal carcinogenicity data” in their use of a multi-response permutation procedure, adjusting for a large number of tumors across multiple studies. In essence, they posit that because every rodent cancer bioassay examines dozens of organs, and it is not uncommon for a chemical to have multiple bioassays, multiple testing corrections should be applied across all studies and tumors ever examined, to determine whether a chemical is a carcinogen. This argument, wrapped in a seemingly innocuous statistical framework, sets a potentially far-reaching precedent to revisit decades of standard practice of cancer hazard evaluation performed by national and international agencies. They seek to erect new barriers to chemical evaluation, and propose a very high bar to classify a chemical as positive in a rodent bioassay. We strongly question whether the proposed approach will strengthen public health protections, noting that extrinsic factors may account for a majority of cancer risk (Wu et al., 2016).

Multiple guidelines have been devised for evaluation and analysis of animal carcinogenicity data, most after consultations with experts, stakeholders, and the general public. Principal examples include Organization for Economic Cooperation and Development Guidance Document 116 on the Conduct and Design of Chronic Toxicity and Carcinogenicity Studies, the International Agency for Research on Cancer (IARC) Preamble to the IARC Monographs, the European Union Guidance to Regulation (EC) No 1272/2008 on classification, labeling, and packaging (CLP) of substances and mixtures, the US EPA Guidelines for Carcinogen Risk Assessment, and the National Toxicology Program Handbook for Preparing Report on Carcinogens Monographs. These documents detail very similar statistical methods for significance and trend analysis of the animal cancer bioassays. Most importantly, all of them recommend a holistic approach to evaluating the strength of evidence from suitable long-term studies in laboratory animals, with data on tumor incidence in multiple tissues/organs assessed through both gross and microscopic veterinary pathology examination.

A large body of evidence has accrued, as a product of 40+ years of cancer hazard evaluations (Pearce et al., 2015), to examine whether rodent cancer bioassays are informative of human cancer hazards. Rodent bioassays are one of the critical data streams, remain a pillar of cancer prevention research, and have proven to be highly informative for potential human cancer hazards. For example, of the 60 agents for which at least 1 tumor site was identified in both humans and animals, 52 (87%) exhibited tumors in at least one of the same organ and tissue system in humans and animals (Krewski et al., 2019).

Where do Crump et al. (2020) stand on this? They start with the sensible premise that, for a fixed significance threshold, multiple testing will result in an increased false positive rate. The authors account for multiple comparisons by using careful permutation methods; however, their effort serves to distract from important issues in appropriately weighing evidence. Their argument, in effect, presents a caricature of decades of guidance and practice on holistic evaluation of cancer hazard as essentially a “p-hacking” exercise. Their representation does not accurately reflect existing cancer hazard evaluation guidelines, which synthesize multiple sources of evidence.

Moreover, even viewing the issue from a purely statistical standpoint, there are reasons to question the approach of Crump et al. (2020). First, it is not accurate to claim that evaluation of bioassays has amounted to sifting through hundreds of p-values and relying on only the most significant findings. For example, only the liver and thyroid tumors from (Stout, Ruecker, unpublished data. Chronic Study of Glyphosate Administered in Feed to Albino Rats) were cited in the IARC evaluation (Guyton et al., 2015), even though adrenal and neurofibroma tumors also showed statistically significant trends. According to the authors’ Table 3, only 1 of the 7 studies with p < .01 was cited by the IARC Monograph, which could be interpreted to imply that a much less stringent multiple testing penalty should be applied than by Crump et al. (2020). Second, the positive tumor association findings cited in the manuscript have low control incidences, implying a low background rate, which is very difficult to estimate with concurrent controls. If the pooled historical control rate is small, then any tumor appearance should be considered biologically significant, and would exhibit far greater statistical significance had the control rate been considered. Third, the proposed application of permutation testing to all historical studies does not accurately reflect the sequential nature of hazard evaluation, in which studies that follow demonstration of potential hazard could arguably be subjected to less stringent test thresholds. Finally, it has been recognized that a primary difficulty with animal bioassay data is not high false positive rates, but low statistical power, and that false positives are primarily an issue with common tumors (Portier, 1994).

The statistics community has been increasingly vocal (Wasserstein et al., 2019) that p-values and arbitrary significance thresholds should not be used as a sole or even primary basis for decision making. This is especially true for scientific questions of practical importance that can be quantified in terms of meaningful effect sizes, for which bodies of supporting evidence can be summoned, and where the consequences of false negatives are high. Indeed, the reasoning behind Crump et al. (2020) could easily be abused as a roadmap for negative findings, by encouraging underpowered study designs (thus less likely to be found significant) and to include large numbers of less toxicologically relevant outcomes to raise tha bar for the stringency of significance thresholds. Interestingly, examination of p-value distributions from studies of liver carcinogenicity by some of the same authors (Crump et al., 1998) offered evidence that standard significant testing thresholds, uncorrected for multiple testing, may in fact be underreporting the number of carcinogenic chemicals, with opposite implications to the argument Crump et al. (2020) make now.

In conclusion, although one cannot entirely dismiss the possibility that well-studied chemicals have a greater opportunity to appear carcinogenic, it remains unproven that current statistical testing standards lead to a major false positive problem in cancer hazard identification. A distracting focus purely on multiple testing may instead create the opposite problem, increasing the burden-of-proof for identifying human health hazards in a manner that creates numerous false negatives to the detriment of public health.  

DECLARATION OF CONFLICTING INTERESTS

I.R. and W.A.C. served on or chaired a number of World Health Organization/International Agency for Research on Cancer Monographs working groups (as an overall chair, or a chair of “Mechanistic and Other Relevant Evidence” sub-group). Most pertinent to this letter is I.R.’s service as a sub-group chair (Mechanistic and Other Relevant Data) of the Working Group for IARC Monographs on the Evaluation of Carcinogenic Risks to Humans Volume 112 “Some Organophosphate Insecticides and Herbicides” which included evaluations of malathion, parathion, diazinon, glyphosate, and tetrachlorvinphos.