Statistical notes for clinical researchers: Sample size calculation 3. Comparison of several means using one-way ANOVA

In this third article about sample size determination, we will discuss sample size determination procedure for comparison of several means. Usually analysis of such data is performed using the analysis of variance (ANOVA) procedure. Because of the complex nature that more than two group means are compared, various types of effect sizes have been suggested including Cohen's f, Eta squared (η²), Partial Eta squared (${\mathit{\eta }}_{\mathit{p}}^{\mathrm{2}}$), and Omega squared (ω²). Therefore, we will discuss sample size determination procedure using Cohen's f and then will explore various types of effect sizes for ANOVA and their interchangeability.

Sample size determination using Cohen's f measure

Cohen's f measure is an extended version of Cohen's d which is defined as a standardized difference, difference divided by standard deviation ($\mathit{d}=\frac{{\mathit{\mu }}_{\mathrm{1}}-{\mathit{\mu }}_{\mathrm{2}}}{\mathit{\sigma }}$) in comparison of two sample means. Cohen's f is expressed as a square root of mean squared difference divided by variance, whose numerator represents an average difference of group means from the grand mean and denominator represents the common standard deviation.

$\text{Cohen's\hspace{0.5em}}\mathit{f}=\sqrt{\frac{{\sum }_{\mathit{j}=\mathrm{1}}^{\mathit{p}}{\left({\mathit{\mu }}_{\mathit{j}}-\mathit{\mu }\right)}^{\mathrm{2}}/\mathit{p}}{{\mathit{\sigma }}^{\mathrm{2}}}}$, where p is the number of groups. Based on the data in Table 1, Cohen's f is calculated as follows: $\sqrt{\frac{\mathrm{100.6}/\mathrm{4}}{{\mathrm{8.6}}^{\mathrm{2}}}}=\sqrt{\frac{\mathrm{25.2}}{\mathrm{73.9}}}=\mathrm{0.58}$

Table 1. Descriptive statistics of a variable from four groups.

Group	Mean	SD	(µj - µ)2
1	18.4	8.6	36.6
2	22.2	8.6	5.1
3	25.1	8.6	0.4
4	32.1	8.6	58.5
Total	24.5	8.6	${\sum }_{\mathit{j}=\mathrm{1}}^{\mathit{p}}{\left({\mathit{\mu }}_{\mathit{j}}-\mathit{\mu }\right)}^{\mathrm{2}}=\mathrm{100.6}$

Cohen (1988) suggested that interpretation of Cohen's f was such as f = 0.1, small effect, f = 0.25, medium effect, and f = 0.40, large effect for the behavioral science.1 Therefore, the calculated Cohen's f value 0.58 can be interpreted as a large effect. If we anticipate outcome values for four comparative groups as appeared in Table 1, we can calculate the required sample size to keep small Type 1 error and large power. Let's set the condition, as α error level = 0.05 and power level = 0.8. Also four group means are assumed as 18.4, 22.2, 25.1, and 32.1, respectively, and a common standard deviation 8.6 is provided. The free software G*Power gives the result of an appropriate sample size as ten per group (total sample size = 40) with the practical power value of 0.85. The steps to perform the sample size calculation are displayed as follows:

Step 1: Select statistical test types

Menu: Tests - Means - Many Groups ANOVA: One-way (one independent variable)

Step 2: Calculation of Cohen's f measure

Menu: Determine - select the procedure as Effect size from means - Number of groups: 4 - Provide means and common SD - Calculate - Calculate and transfer to main window

Step 3: Set α error probability = 0.05 and β error probability = 0.2 - Calculate (a total sample size of 40 and actual power level of 0.847 are obtained.)

Additionally, we can check the changes of power level (1 - β error probability) as the total sample size changes from 12 to 80, given the prior conditions are fixed except β error probability as shown in the figure below.

Other types of effect sizes for ANOVA

Several types of effect sizes for ANOVA are more commonly reported compared to Cohen's f because most statistical software programs provide statistics such as total sum of squares (SS_total), sum of squares of effects (SS_effect) or sum of squares of error (SS_error), which are related to them. If previous studies report only other types of effect sizes different from Cohen's f and if group means and variances are not available, researchers should convert those effect sizes into Cohen's f to calculate an adequate sample size.

1. Eta squared (η²)

Eta squared is expressed as sum of squares between groups (SS_effect) divided by the total sum of squares of the dependent variable (SS_total), ${\mathit{\eta }}^{\mathrm{2}}=\frac{{\mathit{\text{SS}}}_{\mathit{\text{effect}}}}{{\mathit{\text{SS}}}_{\mathit{\text{total}}}}$. The quantity is the same with the usual r squared (R²) which we use as a measure of degree that a model explains the data. The estimate of η² value was calculated as $\mathrm{0.341}\left(=\frac{{\mathit{\text{SS}}}_{\mathit{\text{effect}}}}{{\mathit{\text{SS}}}_{\mathit{\text{total}}}}=\frac{\mathit{1996.998}}{\mathit{5863.715}}\right)$, which means the ANOVA model using the MATERIAL independent variable explained 34.1% of variability in the dependent variable (Table 2). Eta squared can be converted into Cohen's f and vice versa as follows: $\mathit{f}=\sqrt{{\mathit{\eta }}^{\mathrm{2}}/\left(\mathrm{1}-{\mathit{\eta }}^{\mathrm{2}}\right)}$ or η² = f ² / (1 + f ²).

Table 2. An exemplary ANOVA table.

	Sum of Squares	df	Mean Square	F
Between Groups	1996.998	3	665.666	13.084
Within Groups	3866.717	76	50.878
Total	5863.715	79

2. Partial Eta squared (${\mathit{\eta }}_{\mathit{p}}^{\mathrm{2}}$)

The effect size of partial Eta square measure is preferred to Eta squared in a two-way factorial design. The main reason is that when other independent variables are included in the model, η² value becomes smaller compared to the original value, therefore it cannot represent an effect size in multivariate situation. The partial Eta squared is expressed as SS_effect divided by the sum of SS_effect and SS_error, ${\mathit{\eta }}_{\mathit{p}}^{\mathrm{2}}=\frac{{\mathit{\text{SS}}}_{\mathit{\text{effect}}}}{{\mathit{\text{SS}}}_{\mathit{\text{effect}}}+{\mathit{\text{SS}}}_{\mathit{\text{error}}}}$. Partial Eta squared measure can be obtained by selecting 'estimates of effect size' on the option window during performing two-way ANOVA by selecting successive procedures of Analysis - General Linear Model - Univariate in IBM SPSS statistical package version 23.0 (IBM Corp., Armonk, NY, USA). As appeared in Table 3, the estimates of ${\mathit{\eta }}_{\mathit{p}}^{\mathrm{2}}$ values for MATERIAL, LIGHT, and the interaction term were calculated as 0.469, 0.015, and 0.410, respectively. Partial Eta squared can be converted into Cohen's f for a specific term using the formula, $\mathit{f}=\sqrt{{\mathit{\eta }}_{\mathit{p}}^{\mathrm{2}}/\left(\mathrm{1}-{\mathit{\eta }}_{\mathit{p}}^{\mathrm{2}}\right)}$

Table 3. Partial Eta squared measures from two-way ANOVA with an interaction term.

Source	Type III sum of Squares	df	Mean Square	Partial Eta Squared
Corrected model	3602.103a	7	514.586	0.614 = 3602.1 / (3602.1 + 2261.6)
Intercept	47894.642	1	47894.642	0.955 = 47894.6 / (47894.6 + 2261.6)
LIGHT	34.716	1	34.716	0.015 = 34.7 / (34.7 + 2261.6)
MATERIAL	1996.998	3	665.666	0.469 = 1997 / (1997 + 2261.6)
LIGHT * MATERIAL	1570.389	3	523.463	0.410 = 1570.4 / (1570.4 + 2261.6)
Error	2261.612	72	31.411
Total	53758.357	80
Corrected total	5863.715	79

3. Omega squared (ω²)

The Omega squared measure was suggested to correct the biasedness of Eta squared measure. The Eta squared was slightly biased because the calculation procedure was made purely based on statistics from the sample without any adjustment considering population measure. Omega-squared is calculated as ${\mathit{\omega }}^{\mathrm{2}}=\frac{{\mathit{\text{SS}}}_{\mathit{\text{effect}}}-{\mathit{\text{df}}}_{\mathit{\text{effect}}}\mathit{*}{\mathit{\text{MS}}}_{\mathit{\text{error}}}}{{\mathit{\text{SS}}}_{\mathit{\text{total}}}-{\mathit{\text{MS}}}_{\mathit{\text{error}}}}$, where SS_total and MS_residual represent total sum of squares and mean square of error, respectively, and df_effect is degrees of freedom of the effect. An Omega square has slightly lower value and generally is considered more accurate compared to an Eta squared (Table 4). Omega squared can be approximately converted into Cohen's f using the formula, $\mathit{f}\approx \sqrt{{\mathit{\omega }}^{\mathrm{2}}/\left(\mathrm{1}-{\mathit{\omega }}^{\mathrm{2}}\right)}$.

Table 4. Omega squared and Eta squared calculated in a two way factorial design.

Effect	SSeffect*	dfeffect*	MSerror*	$$\frac{{\mathit{\text{SS}}}_{\mathit{\text{effect}}}-{\mathit{\text{df}}}_{\mathit{\text{effect}}}*{\mathit{\text{MS}}}_{\mathit{\text{error}}}}{{\mathit{\text{SS}}}_{\mathit{\text{total}}}-{\mathit{\text{MS}}}_{\mathit{\text{error}}}}$$	ω2
MATERIAL	1997	3	31.41	1902.77/5832.31	0.326
LIGHT	34.72	1	31.41	3.31/5832.31	0.001
MATERIAL * LIGHT	1570.39	3	31.41	1476.16/5832.31	0.253

^*Figures were from Table 3.

Cohen (1988) suggested interpretation of effect sizes expressed as η² or ω²: small effect, η² or ω² = 0.01; medium effect, η² or ω² = 0.06; large effect, η² or ω² = 0.14 for the behavioral science.1