Commentary on Aschmann et al: How do we incorporate harms in analyses of randomized trials?

The standard primary analysis of a randomized controlled trial is about benefits and ignores harms altogether. We have an intervention that lowers blood pressure, and hope that it reduces cardiovascular events; the primary analysis of a trial will compare event rates to give a headline result such as “the experimental intervention led to an absolute 6% reduction in the risk of a cardiovascular event (95% CI 2.5%, 9.5%; p=0.001)”. Harms enter our evaluation of the treatment informally. We describe adverse events and consider how their prevalence and severity compares to that 6% reduction in event rate. This is a process that starts during study design. The investigators might say that, taking into account current knowledge of the side-effects, risks, financial costs and inconvenience of the intervention, we wouldn’t want to treat more than 20 patients in order to prevent one event, so define the minimal, clinically significant difference as 5%, and power the trial accordingly.

Aschmann et al.¹ argue that we need to move beyond this rather wooly approach and adopt formal, quantitative inclusion of harms in the primary analysis of randomized trials. They propose that the “benefit-harm” balance should be the primary endpoint, recommend that patients be given “preference surveys” so that harms and benefits can be appropriately weighted, and call for trials to have “sufficient statistical power to explicitly measure clinically relevant differences in the benefit-harm balance”. It is certainly a beguiling proposal. But, as it turns out, it is completely misguided.

The basic problem is that although Aschmann et al. make repeated reference to decision-making, their paper does not involve any consideration of decision theory. Take the idea of a “clinically relevant difference in the benefit-harm balance”. In classical decision theory, once you have taken account of all benefits and harms, you choose the option with the best expected outcome; there is no such thing as a “clinically relevant difference”. Clinical relevance is what we think about when we do not formally incorporate harms: for instance, we might describe the effects of a drug as “clinically irrelevant” because, although it is of benefit, the degree of benefit is too small to justify its side-effects, risks, costs and inconvenience. If you take into account those harms in calculating a benefit-harm balance, then clinical relevance makes no sense: you take the drug if the benefit-harm balance is greater than zero.

There are obvious consequences for statistical power if we would still act even if differences were very small. Take the case of a pain drug, where we consider the total harms of the drug to be a perfect trade-off for a benefit of 0.75 points on the 0 – 10 pain scale. In the sample size calculation, the investigators elect to specify that the drug must reduce pain by at least 1 point. If the standard deviation of the pain scale is 2 points, this gives a sample size of 125 patients for 80% power. If the sample size is conducted on the harm-benefit scale, the difference between groups is 0.25 points, requiring a sample size of 2000. Indeed, under traditional decision theory, the clinically relevant difference on the benefit-harm scale is an infinitely small positive, so the sample size for any trial using benefit-harm as an endpoint is infinity.

The larger problem here is related to applying concepts of inference – inherent in sample size calculation and trial analysis – to decision-making. In brief, a rational decision maker takes the option with the highest expected utility, irrespective of inference. If I have a choice between investing my retirement account in fund A or B and I work out that fund A will be a better investment 53% of the time and fund B will be a better investment 47% of the time with a p-value of 0.7, I still have to invest, and so I’m left with either choosing fund A or flipping a coin. I’ll take fund A, thank you very much.

My own view is that we should conduct clinical trials pretty much as we do now. If a treatment has been demonstrated using standard inferential methods to be superior to control, we might then want to conduct a decision analysis, which is absent inference, to determine whether there is overall net benefit, and make clinical recommendations accordingly. Hard-wiring benefit-harm balance into trial design will in fact lead to more harm than benefit for the clinical trials enterprise.