Effect size in CASTLE-AF trial: the issue of ‘the tiny effect’

Tiny effect is among the issues of our time ‘that science has taken a turn towards darkness.’ ¹

In the Catheter Ablation for Atrial Fibrillation with Heart Failure (CASTLE-AF) trial,² the primary end point was death or hospitalisation for worsening heart failure. The authors concluded that: ‘after a median follow-up of 37.8 months, the primary composite end point occurred in significantly fewer patients in the ablation group than in the medical therapy group (51 patients (28.5%) vs 82 patients (44.6%); difference 16.1% (CI) 5%–25%. HR 0.62; 95% CI 0.43 to 0.87; p=0.007).’ That was statistically significant.

Although the inferential statistical analysis provides information about the reliability of the result, the p value conveys little information on the significance of the clinically observed effect. This problem is solved by the concept of effect size, ‘which was developed to allow clinically meaningful comparisons of efficacy between treatment trials. Without using this concept, comparing two treatment trials can be difficult. as the name suggests, an effect-size estimate can place an easily interpretable value on the direction and magnitude of an effect of a treatment, a difference between two treatment groups, or any other numerical comparison or contrast.’³ The effect size measurement takes two factors into account: the difference between the mean values of the measures for the two groups and the variance. For convention, this is applied by the Cohen’s d principle. This means that the small effect size <0.2, medium 0.2–0.8 and larger >0.8.

For the CASTLE-AF trial, the effect size calculation⁴ was ф=2 arcsine √p. Thus, ф_0.285=2 arcsine √0.285=1.32, ф_0.446=2 arcsine √0.446=1.60 and h=ф_0.285–ф_0.446=0.28!! A tiny effect.

What about the tiny effect? The tyranny of the statistical significance fills the literature with trivial and incorrect findings. Horton remarked recently, ‘the case against science is straightforward: much of the scientific literature, perhaps half, may simply be untrue. Afflicted by studies with small sample sizes, tiny effects, invalid exploratory analyses, and flagrant conflicts of interest, together with an obsession for pursuing fashionable trends of dubious importance.’¹ In this context, a tiny effect is an effect without a clinical impact for management that adds no value to the patient care.

In this way, although the study was able to find a more statistically significant difference between ablation group compared with the medical therapy group in the composite end point, the magnitude of the difference was tiny.⁵ We can conclude that this trial found a real clinical irrelevance with statistical significance.