In their 2021 article, among other things, Frommlet & Heinze offer recommendations to improve statistical analyses when replicating animal trials. They suggested a “Mixed-model analysis” that allows the treatment effect to vary among the replications. (The treatment effect is defined as a difference in means between actively treated animals and sham- or placebo-treated animals.) They performed a simulation to study how power is affected by (i) the variability of the treatment effect among the replications and (ii) the number of replications (while keeping total number of animals the same). They fitted the linear mixed model and used restricted maximum likelihood (REML) estimation to estimate the variance parameters. (REML is a standard and recommended method for estimating variance parameters in linear mixed models.) The power results, displayed in their Figure 2A, were impressive, finding power to increase as the number of replications increased while maintaining the total number of animals; this finding was regardless of the variability of treatment effects among the replications. However, the authors noted, “The linear mixed model is actually quite conservative, and particularly when there are only three replications, the type I error is controlled at a level way below 0.01.” To fit their linear mixed model, they used the MIXED procedure in the SAS/STAT software, version 9.4 (SAS System for Windows, SAS Institute, Inc.). And they used the MIXED’s default “containment method” for the computing the error degrees of freedom. (Contained degrees of freedom are the ones we can figure when writing down the analysis of variance table in a beginning statistics course.) Though I’ve not checked, it is likely the case the other statistical softwares that fit linear mixed models also have contained degrees of freedom as default. A small but important tweak to the model is choosing to compute the error degrees of freedom with the method proposed by Kenward and Roger (2009) or, alternatively, by Satterthwaite’s method. In so doing, the Type I error rates are more in line with nominal levels and power is substantially increased in the “fewer replicates / more animals per replicate” settings (see Figure).
Figure.
Power for detecting effect sizes of size 1 (green), 2 (blue) and Type I error (red) when estimating degrees of freedom with the containment method (solid) and the Kenward-Roger method (dashed). The standard deviation of treatment effect is 0.01 in (A), 0.5 in (B), and 1 in (C). Black horizontal line is at 0.05 – the significance level.
Finally, I want to highly commend Frommlet and Heinze in supplying such fine supplementary material with their article. Their results were all computationally reproducible, and I simply added “/ ddfm = kr2” to their original code to get the corrected results.
Funding
This work was funded by the National Institute of General Medical Sciences of the National Institutes of Health (NIH) under grants P20 GM109005 (PI: Marjan Boerma), P20 GM109096 (PI: Judith Weber), and by the National Center for Advancing Translational Science of the NIH under grant UL1 TR003107 (PI: Laura James).
Footnotes
Declaration of Conflicting Interests
The Author declares that there is no conflict of interest.
Data Availability
The data used for figures and the software code used to produce the figures are available upon request to rdlandes@uams.edu.
References
- 1.Frommlet F, Heinze G. Experimental replications in animal trials. Laboratory Animals 2021; 55(1): 65–75. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Kenward MG, Roger JH. An improved approximation to the precision of fixed effects from restricted maximum likelihood. Computational Statistics & Data Analysis 2008; 53(7): 2583–2595. [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Data Availability Statement
The data used for figures and the software code used to produce the figures are available upon request to rdlandes@uams.edu.