The dilemmas of science for policy

Abstract

The aim of obtaining the most accurate and complete scientific information as input for regulatory decision making is not always the most suitable one, due to the consequences on the population level of alternative regulatory options.

Regulatory decision‐making often faces the question of what is an appropriate level of evidence. For instance, which approach is better suited to regulate potentially toxic chemicals: obtaining the best—most complete, most accurate, and so on—information that science is able to produce on each individual compound or relying on more basic knowledge to quickly assess a wide range of compounds? The first one would appear as the obvious answer, but, given the time and resource constraints under which regulators often operate, it may not be the most efficient one. Gathering the most complete and accurate scientific information means that regulatory agencies would be able to deal with only a limited number of substances. In contrast, by limiting the required evidence, they might be able to regulate many more substances albeit at the risk of sometimes making wrong decisions.

The key issue is how to establish the appropriate level of evidence for regulatory purposes and to choose the scientific methodologies best suited for generating this evidence.

This is a typical dilemma faced by regulatory agencies in many areas, and the example above is the cause of ongoing controversy among regulators, manufacturers, environmentalists, and public health advocacy groups. The key issue is how to establish the appropriate level of evidence for regulatory purposes and to choose the scientific methodologies best suited for generating this evidence.

Another example shows how the same dilemma befalls the field of benefit assessment. Many foods with health claims assert that certain ingredients will confer health benefits for consumers, beyond the nutritional characteristics. Again, what is the appropriate regulatory strategy: to analyze the product in question using the most accurate and complete scientific methodologies at hand, even if that means that only a small number of products will achieve authorization? Or recurring to basic or incomplete scientific information, to maximize the number of products that consumers can choose from? In the first case, consumers can be assuaged that the food in question will generate the health benefits claimed by the manufacturer, but they will have a very limited range of products to choose from. In the latter case, consumers will be able to choose from a large variety of foods with health claims, of which they can be fairly sure, but they also have to accept that some of those claims are false.

These examples demonstrate that, at least for regulatory purposes, obtaining the best scientific evidence is not always appropriate for decision‐making. We argue that the solution should be based on empirical analysis of the consequences of alternative regulatory options that inevitably affect entire sets of products and processes, as well as human populations. Consequently, one relevant factor to be taken into account when selecting the level of evidence and associated methodologies are the effects of regulatory options on the population level.

Regulatory dilemmas and methodological controversies

The decision to control a substance that may entail risks, or to authorize a substance that promises benefits, is the end product of a complex regulatory process based on scientific evidence on risks and benefits to ensure that the process is objective and efficient. Thus, the question of what constitutes sufficient evidence can have a direct impact on economic and other interests, influence the direction and pace of innovation, and can have significant consequences for public health and the environment.

… one relevant factor to be taken into account when selecting the level of evidence and associated methodologies are the effects of regulatory options on the population level.

Most regulatory agencies mainly assess risks in order to protect human health and the environment. A smaller, but growing area of regulation focuses on benefit assessment to ensure that products, which claim to offer specific benefits, such as pharmaceuticals, nutritional complements, or functional foods, do in fact produce such benefits.

Over time, the controversy about the proper level of evidence in risk assessment has led to the emergence of two positions: a defense of evidence requirements that are as strict as feasible, with the objective of protecting scientific‐technological innovation from undue restrictions, or reducing the evidence requirements, with the objective of making it easier to adopt regulations that protect health and the environment 1, 2.

In the case of benefit assessment, the controversies related to evidence requirements for pharmaceutical products and foods have also given rise to two alternative positions, which can be illustrated with the case of European health claims regulation: protecting consumers from false claims by imposing strict evidence requirements for product authorization 3, or reducing the requirements so consumers have access to a wider range of products that may offer health benefits 4, 5. The main difference between pharmaceutical products and foods is that in the latter case, the food products themselves are not subjected to regulation (only the health claims are). In the case of pharmaceuticals, the products themselves are regulated to ensure efficiency and to protect patients from any unintended negative effects.

… regulatory controversy has methodological consequences.

These controversies related to standards of evidence affect their respective methodological counterparts. In risk assessment, it concerns the type of admissible evidence, and the way it is analyzed; for instance, whether data from human trials are a necessary prerequisite, or if data from animal assays, or in vitro studies or molecular structure analysis, are sufficient. Another aspect under discussion is the extrapolation models to assess acceptable exposure in humans based on animal or in vitro tests.

Most regulators now accept mechanistic and in vitro evidence for risk assessment of chemical compounds. These evidence requirements are a product not only of debate about the best strategies to protect health and the environment, but also result from a better understanding of how various substances interact with humans and ecosystems. In addition, regulators increasingly accept data generated by “non‐standard” scientific methods such as short‐term tests that allow testing a large number of substances quickly albeit with a reduced level of confidence; the weight‐of‐evidence approach that draws from (meta)analysis of several independent studies on the same topic (each of which, taken by itself, not being considered conclusive); or structure‐activity relationships (SARs) in which structural or other similarity between different molecules is considered indicative of similarity of their toxicity or interactions with the human body.

A regulatory framework that makes use of such methodologies is the European chemicals regulation REACH, which analyzes and regulates all chemical substances that are on the market 6. It makes heavy use of non‐standard scientific methodologies, particularly SARs 7. There is still controversial debate on how to use this evidence in decision‐making so as to process a huge number of untested compounds: analyzing each one individually—which is the default strategy under REACH—or regulating an entire category of substances based on “innate” properties such as bioaccumulation or environmental persistence without detailed data on each substance 8. This example shows a regulatory controversy that is typical for risk assessment, and directly relates to the type and level of admissible evidence for decision‐making.

In benefit assessment, the methodological controversies are particularly visible in the area of health claims. In European regulatory practice, the principal source of evidence required for authorization of a health claim are human intervention studies, usually randomized controlled trials (RCTs), the same kind of evidence required for authorization of pharmaceuticals. European regulators primarily demand human intervention studies because of their capacity for identifying causal relationships between intake of an ingredient and a specific and well‐defined outcome 9.

There are a number of researchers in the nutrition sciences who argue that this level of evidence requirement is unrealistic and ineffective 4, 5, 10 for health claims authorization of food products, as well as for nutrition research. This standard of evidence is not only very difficult to attain, but would also imply a delegitimization of a lot of existing research in the nutrition field based on observational or mechanistic evidence.

Again, the regulatory dilemma has a methodological counterpart. In risk assessment, several authors have argued for more expedite tests that allow for timely generation of decision‐relevant regulatory information 1. In benefit assessment, some criticize the heavy reliance on clinical trials for generating decision‐relevant knowledge in health claims regulation and argue that there is a larger range of methodologies to generate reliable data for decision‐making 4, 5.

Thus, regulatory controversy has methodological consequences. It can favor some methodologies, but also facilitate the development of entirely new methods for generating relevant data. It can influence scientific disciplines by generating demand for certain types of studies and methods. And it can pitch different types of scientific methodologies against each other: “standard” academic methodologies against “non‐standard” but regulation‐relevant methodologies as in risk assessment; and “standard” methodologies from different scientific fields as in health claims regulation.

Population‐level dimensions of regulatory options

We therefore argue that effects on the population level ought to be taken into account when making methodological choices for both risk and benefit assessment. The idea is that lower levels of evidence will—at least in most cases—result in a regulation that overall is better suited to protect human health and the environment. The most serious drawback of lower standards would be a larger number of false positives, for instance substances that are being labeled as problematic when in fact they are not. But as this regulatory strategy would be able to analyze and thereby identify a larger number of problematic substances, it is overall better suited to protect human health and the environment. Considering the sum of all substances that are potentially detrimental to health, a more permissive regulatory strategy based on minimizing false negatives may be preferable to requiring more accurate and complete knowledge to minimize false positives.

… to select the most appropriate regulatory strategy in each particular case, it is necessary to first analyze the possible effects that result from adopting one strategy or another

It is important to understand that the argument regarding the population‐level effects of lower evidence requirements pertains to specific circumstances and cases. Any lowering of the evidence requirements in regulation could, in terms of regulatory efficiency, lead to negative effects, such as delegitimization of regulatory agencies and regulatory science; economic effects, owing to the unnecessary prohibition of products and their substitution with more expensive alternatives; and legitimization of bad science, at least in certain cases, due to the absence of universally accepted standards. Thus, to select the most appropriate regulatory strategy in each particular case, it is necessary to first analyze the possible effects that result from adopting one strategy or another. The results of this analysis, rather than any a priori considerations, should underlie the decision about the level of evidence that the regulation in question requires.

The same argument about population effects can be applied to the regulation of health claims and benefit assessment. A more stringent level of evidence would protect consumers from false or incorrect information. But some products, which do offer beneficial effects, will not be analyzed at all, given the elevated evidence requirements, and their potential health benefits will be lost to consumers. In contrast, a more permissive level of evidence would result in a wider range of products claiming health benefits even though some do not possess these. But, consumers who want to improve their eating habits will, overall, have a wider range of options. The idea that underlies this regulatory strategy is that on balance, the positive health effects for the individual as well as society at large outweigh the fact that some of the supposedly beneficial products in reality are not beneficial. The principal question is which of those two regulatory strategies will, in the end, produce more health benefits on a global level. The trade‐off between the foreseen population effects and the negative effects mentioned above will indicate which regulatory strategy is the most adequate in each case.

There is no general solution, as the population‐level effects will depend on what is being regulated. Their impact is surely different for the regulation of health claims compared to pharmaceuticals. In the case of health claims, in which regulation affects the classification and labeling of products but not their authorization, the population‐level consequences of false positives or false negatives are not likely to be severe. In the case of pharmaceuticals, the health consequences of errors could potentially be serious if patients suffer from unintended side effects or do not receive an effective treatment.

Conclusion

In risk regulation and benefit regulation, the answer to the question which standard of evidence is the “best” or “most appropriate” one, that is, the one with the best overall outcomes for public health, cannot be determined without previous empirical study. The answer will depend on the characteristics of the substances in question, their use, consumer habits, research resources, and so on, as well as on the objectives of public policy and regulation. Regulatory strategies and the concomitant methodological choices should be assessed as to their global effects on the outcomes. Taking into account the population‐level effects of different regulatory strategies could allow for more relevant methodological choices and help public policy to better protect human health and the environment.

Conflict of interest

The authors declare that they have no conflict of interest.