Policy recommendations in the discussion section of a research article

Short abstract

Should a research article in Injury Prevention make policy recommendations in the discussion section?

Imagine that the Acme Auto Company has designed Device X to prevent death in a vehicle crash. You and I have completed the first randomized controlled trial of Device X; we studied 400 drivers who were randomly assigned to X or a placebo device at the time of their crash and ascertained which drivers died.

The risk of death among the drivers with Device X was 0.075 compared with 0.15 for drivers without X (table): risk ratio 0.5, 95% confidence interval 0.28 to 0.90. Our draft manuscript concludes: “We found that drivers who crashed in vehicles with Device X had a risk of death which was half that of similar drivers in similar vehicles without X. If our findings represent the casual effects of X, this device can prevent about half the driver deaths that would otherwise occur in a crash.” After reading our draft, a colleague suggests we make a policy recommendation in the discussion section of our paper. Should we?

Trial arm and outcomes of 400 drivers who were in a crash.

	Died	Survived	Total	Risk of death
Device X	15	185	200	0.075
Placebo	30	170	200	0.15

Interpreting results or stating preventive implications does not require a statement about policy; the two sentences in quotations above provide an adequate interpretation. Calls for more data or research are often unnecessary, but they are not my topic.¹ A policy recommendation is advice that some action should be taken by someone to promote health: a behavior should be adopted, advice should be given, a public education campaign should begin, a product should be purchased, a law should be enacted, and so on.

I will offer three reasons for not giving a policy recommendation in the discussion section of our research study of Device X. Similar ideas have been expressed by others.²,³,⁴,⁵

Reason 1: The results of the study may not be correct

Any study, even a randomized trial, may produce biased estimates of casual associations. Even if our hypothetical study had no apparent limitations (I have never written or read a study without limitations), our estimate of Device X's effect is still uncertain. Given the observed data, the maximum likelihood estimate for the risk ratio is 0.5, but risk ratios from 0.26 to 0.94 all have likelihoods at least 1/8th the maximum (fig).⁶,⁷ A second study might find that Device X reduced mortality by only 10%, not 50%. There is a small possibility that X has no causal influence on the risk of death or might even be harmful. If we studied Device X using a design other than a randomized trial, we would want to be even more cautious about recommendations, as our estimate of the effect of X might be biased by differences between the drivers with X and those without X.⁸,⁹

The relative likelihood of a range of risk ratio values, from 0.1 to 2.5, for the risk ratio of death among drivers who crashed in vehicles with Device X compared with otherwise similar drivers and vehicles without Device X.

Reason 2: All the important risks and benefits have not been studied

We studied the effect of X on death only. But perhaps X greatly increases the risk of quadriplegia, traumatic brain injury, and limb amputation. Perhaps X is ineffective or harmful for subgroups of drivers or vehicles. Even if X prevents bad health outcomes in a crash, we have no information about whether it might affect the risk of a crash—if it halved the risk of death, but doubled the risk of crashing, it might offer no net benefit for driver health. Could X have detrimental effects for other occupants of the same vehicle or occupants in other vehicles? If Device X adds 281 kg of weight to each vehicle, what does that imply for fuel consumption, dependency on imported oil, and increased emissions that might contribute to global warming (which may have health effects). If X adds $9481 to the cost of each vehicle, do the benefits justify that cost?

We can never have all the evidence we might want, so lack of information alone does not prohibit policy making. But a serious discussion of policy regarding Device X should consider what evidence is needed and whether some policy is justified by what is known. It is rare that the discussion section of a research paper will have the space for this review or that the findings of a single research study will fill the gap in knowledge that tips the balance in favor of a particular policy.²

Reason 3: manuscript space rarely allows a full review of alternative policy choices

Even if we are convinced that Device X is beneficial, has no adverse consequences, and the costs are justified, what policy choice would be best? Choices include advising drivers to purchase X, offering X on some new vehicles, putting X into all new vehicles, retrofitting all vehicles with X, or requiring X in vehicles by law. Before recommending a public campaign to encourage the purchase of X, we might want evidence that a substantial part of the public would respond favorably to such a campaign. Before suggesting that health practitioners should advise patients to use X, we might want evidence that patients would follow this advice and we should consider whether the time needed to deliver this advice is justified, compared with other demands upon the time of health professionals.

Is there any harm in policy recommendations in research articles?

It would be tragic if a recommended policy was so misguided that its implementation produced harm. Ineffective advice would waste time and money. Even a useful policy may be wasteful if a superior policy is feasible. We cannot expect that policy advice will always be correct, but if research can contribute to effective policies, I think the best policy is more likely to emerge if it is based upon thorough review of the evidence and thoughtful consideration of alternatives.

I doubt that the policy suggestions given in the discussion section of some injury research papers are likely to result in serious harm; I suspect most readers recognize the offhand nature of these ideas and do not take them seriously. The harm from facile policy recommendations in research studies is probably subtle: (1) a paragraph of policy advice makes the article longer, wasting journal space and reader time; (2) policy discussion diverts attention from the strengths and limitations of the research; (3) casual policy recommendations can give the impression that injury research does not demand critical thinking. In the most egregious instances, recommendations are so loosely related to the research that the discussion becomes an editorial expressing the unsupported opinions of the authors.

Can researchers contribute to policy formation?

Research papers provide evidence that can contribute to policy. For example, early case‐control studies of injuries examined the association between alcohol use and traffic crashes.¹⁰,¹¹,¹² These studies all reported evidence of harm associated with alcohol use, although none made policy recommendations. Today we have laws, educational campaigns, and other policies regarding drinking and driving. The early case‐control studies helped provide evidence for those policies.

Researchers can contribute to policy beyond their research. Some testify before legislative panels, some provide expert advice to public information campaigns, and some write thoughtful articles about policy alternatives. A recent commentary¹³ and article¹⁴ in Injury Prevention are examples of thinking about policy, and the journal encourages articles about policies.¹⁵ I am not recommending that injury researchers avoid policy recommendations—I am suggesting that they do so in a serious manner and that policy discussions at the end of research papers are usually too short to be useful.

Should injury prevention have a policy about policy?

I am not advocating a ban on policy recommendations in research studies. I have no objection to a policy recommendation that is clearly supported by the research; I just think that situation is rare.² But I worry that Injury Prevention actually encourages authors to make policy statements. The journal's advice to authors states “Whenever possible, the Discussion should conclude with a separate section entitled Implications for Prevention.” I think some authors interpret these words as an invitation to recommend policies. I suggest these words could be cut from the instructions with no important loss. Or replaced with language that explains what “implications” are and why they are not usually policy recommendations.