Statistical power as a function of Cronbach alpha of instrument questionnaire items

Abstract

Background

In countless number of clinical trials, measurements of outcomes rely on instrument questionnaire items which however often suffer measurement error problems which in turn affect statistical power of study designs. The Cronbach alpha or coefficient alpha, here denoted by C_α, can be used as a measure of internal consistency of parallel instrument items that are developed to measure a target unidimensional outcome construct. Scale score for the target construct is often represented by the sum of the item scores. However, power functions based on C_α have been lacking for various study designs.

Methods

We formulate a statistical model for parallel items to derive power functions as a function of C_α under several study designs. To this end, we assume fixed true score variance assumption as opposed to usual fixed total variance assumption. That assumption is critical and practically relevant to show that smaller measurement errors are inversely associated with higher inter-item correlations, and thus that greater C_α is associated with greater statistical power. We compare the derived theoretical statistical power with empirical power obtained through Monte Carlo simulations for the following comparisons: one-sample comparison of pre- and post-treatment mean differences, two-sample comparison of pre-post mean differences between groups, and two-sample comparison of mean differences between groups.

Results

It is shown that C_α is the same as a test-retest correlation of the scale scores of parallel items, which enables testing significance of C_α. Closed-form power functions and samples size determination formulas are derived in terms of C_α, for all of the aforementioned comparisons. Power functions are shown to be an increasing function of C_α, regardless of comparison of interest. The derived power functions are well validated by simulation studies that show that the magnitudes of theoretical power are virtually identical to those of the empirical power.

Conclusion

Regardless of research designs or settings, in order to increase statistical power, development and use of instruments with greater C_α, or equivalently with greater inter-item correlations, is crucial for trials that intend to use questionnaire items for measuring research outcomes.

Discussion

Further development of the power functions for binary or ordinal item scores and under more general item correlation strutures reflecting more real world situations would be a valuable future study.

Background

Use of instrument questionnaire items is essential for measurement of outcome of interest in innumerable numbers of clinical trials. Many trials use well-established instruments; for example, major depressive disorders are often evaluated by scores on the Hamilton Rating Scale of Depression (HRSD) [1] in psychiatry trials. However, it is by far more often the case when instruments germane to a research outcome are not available. In such cases, of course, questionnaire items need to be developed to measure the outcome, and their psychometric properties should be evaluated for construct validity, internal consistency, and reliability among others [2, 3]. The internal consistency of instrument items quantifies how similarly in a interrelated fashion the items represent an outcome construct that the instrument is aiming to measure [4], whereas reliability is defined as the squared correlation between true score and observed score [3].

Cronbach alpha also known as coefficient alpha [5], hereafter denoted by C_α, has been very widely used to quantify the internal consistency and reliability of items in clinical research and beyond [6] although internal consistency and reliability are not exchangeable psychometric concepts in general. For this reason, some argue that C_α should not be used for quantifying either concept (e.g.,[7, 8]). One the other hand, for special cases where items under study are parallel such that items are designed as replicates to measure a unidimensional construct or attribute, C_α can quantify internal consistency and reliability as well [2] although in general C_α is not necessarily a measure of unidimensionality or homogeneity [4, 8]. In this paper, we consider parallel items; for example, items within a same factor could be considered parallel for a unidimensional construct. In this sense, items of HRSD are not parallel since it measures depression, a multidimensional construct with many factors.

The Cronbach alpha by mathematical definition is an adjusted proportion of total variance of the item scores explained by the sum of covariances between item scores, and thus ranges between 0 and 1 if all covariance elements are non-negative. Specifically, for an instrument with k items with a general covariance matrix Σ among the item scores, C_α is defined as

$${\mathsf{C}}_{\alpha }=\frac{\mathsf{k}}{\mathsf{k}-\mathsf{1}}\left(\frac{{\mathsf{1}}^{\mathsf{T}}\mathsf{\Sigma }\mathsf{1}-\mathsf{trac}e\left(\mathsf{\Sigma }\right)}{{\mathsf{1}}^{\mathsf{T}}\mathsf{\Sigma }\mathsf{1}}\right)=\frac{\mathsf{k}}{\mathsf{k}-\mathsf{1}}\left(\mathsf{1}-\frac{\mathsf{trac}e\left(\mathsf{\Sigma }\right)}{{\mathsf{1}}^{\mathsf{T}}\mathsf{\Sigma }\mathsf{1}}\right)\text{,}$$ 1

where trace(.) is the sum of the diagonal elements of a square matrix, 1 is a column vector with k unit elements, and 1^T is the transpose of 1. This quantification is therefore based on the notion that relative magnitudes of covariances between item scores compared to those of corresponding variances serves as a measure of similarities of the items. Consequently, items with higher C_α are preferred to measure the target outcome. However, C_α is a lower bound for reliability, but is not equal to reliability unless the items are parallel or essentially τ-equivalent [3, 8]. The sum of the instrument items serves as a scale for the outcome, and is used for statistical inference including testing statistical hypotheses. At the design stage of clinical trials, information about magnitude of reliability or internal consistency of developed parallel items is crucial for power analysis and sample size determinations. Nonetheless, power functions based on C_α have been lacking for various study designs.

In this paper, to derive closed-from power functions, we formulate a statistical model for parallel items that relates the item scores to a measurement error problem. Under this model, C_α (1) is explicitly expressed in terms of an inter-item correlation. We examine relationship among C_α, a test-retest correlation and reliability of scale scores that enables testing significance of C_α through Fisher z-transformation. We explicitly express statistical power as a function of C_α for the following comparisons: one-sample comparison of pre- and post-treatment mean differences, two-sample comparison of pre-post mean differences between groups, and two-sample comparison of mean differences between groups. Simulation study results compare derived theoretical power with empirical power and discussion and conclusion follow.

Methods

Statistical model

We consider the following model for item score Y_ij to the j-th parallel item for the i-th subject:

$${\mathsf{Y}}_{\mathsf{i}\mathsf{j}}={\mu }_{\mathsf{i}}+e_{\mathsf{i}\mathsf{j}}$$ 2

The parameter μ_i represents the “true score” of the target (outcome) construct for the i-th subject. At the population level, its expectation and variance are assumed to be $\mathsf{E}\left({\mu }_{\mathsf{i}}\right)=\mu$ and $\mathsf{V}\mathsf{a}\mathsf{r}\left({\mu }_{\mathsf{i}}\right)={\sigma }_{\mu }^{\mathsf{2}}$, which we call the true score variance. The error term $e_{\mathsf{i}\mathsf{j}}$ represents the deviate of the item score Y_ij from the true score μ_i, i.e., $e_{\mathsf{i}\mathsf{j}}$ is the measurement error of Y_ij. The expectation and variance of $e_{\mathsf{i}\mathsf{j}}$ for all subjects are assumed to be $\mathsf{E}\left(e_{\mathsf{i}\mathsf{j}}\right)=\mathsf{0}$, i.e., unbiasedness assumption, that is, E_j(Y_ij) = μ_i and E_iE_j(Y_ij) = E(μ_i) = μ, where E_j denotes the expectation over j. It is also assumed that $\mathsf{V}\mathsf{a}\mathsf{r}\left(e_{\mathsf{i}\mathsf{j}}\right)={\sigma }_e^{\mathsf{2}}$, which we call the measurement error variance. We further assume the following: μ_i and e_ij are mutually independent, i.e., μ_i ⊥ e_ij; and the elements of e_ij’s are independent for a given subject, i.e., conditional independence, that is, e_ij ⊥ e_ij′|μ_i for j ≠ j′. Note that this conditional independence does not imply marginal impendence between Y_ij and Y_ij′. In short, model (2) is a mixed-effects linear model for data with a two-level structure in a way that repeated item scores are nested within individuals.

Under those assumptions, we have $\mathsf{V}\mathsf{a}\mathsf{r}\left({\mathsf{Y}}_{\mathsf{i}\mathsf{j}}\right)\text{≡}{\sigma }^{\mathsf{2}}={\sigma }_{\mu }^{\mathsf{2}}+{\sigma }_e^{\mathsf{2}}$, that is, the total variance of the item scores is the sum of the true score variance and the measurement error variance. Inter-item (score) covariance can be obtained as $\mathsf{C}\mathsf{o}\mathsf{v}\left({\mathsf{Y}}_{\mathsf{i}\mathsf{j}},{\mathsf{Y}}_{\mathsf{i}{\mathsf{j}}^{\prime }}\right)={\sigma }_{\mu }^{\mathsf{2}}$ for j ≠ j′. Therefore, the diagonal elements of covariance matrix Σ under model (2) are identical and so are the off-diagonal elements. This compound symmetry covariance structure, also known as essential τ-equivalence, is the covariance matrix of parallel items each of which targets the underlying true score for a unidimensional construct. Furthermore, the compound symmetry covariance structure can be regarded as a covariance matrix of “standardized” item scores with unequal variances and covariances. Inter-item (score) correlation, denoted here by ρ, can accordingly be obtained as

$$\mathsf{Corr}\left({\mathsf{Y}}_{\mathsf{i}\mathsf{j}}{\mathsf{Y}}_{\mathsf{i}{\mathsf{j}}^{\prime }}\right)\text{≡}\rho =\frac{{\sigma }_{\mu }^{\mathsf{2}}}{{\sigma }^{\mathsf{2}}}=\frac{{\sigma }_{\mu }^{\mathsf{2}}}{{\sigma }_{\mu }^{\mathsf{2}}+{\sigma }_e^{\mathsf{2}}}\text{.}$$ 3

Although item scores are correlated within subjects, they are independent between subjects. Note that this inter-item correlation is not necessarily equal to item-score reliability that quantifies a correlation between true and observed scores.

In this paper, we assume that the true score variance ${\sigma }_{\mu }^{\mathsf{2}}$, instead of the total variance σ², is fixed at the population level, and it does not depend on the item scores of the subjects. Stated differently, the total variance σ² depends only on ${\sigma }_e^{\mathsf{2}}$ which depends on item scores and thus σ² is assumed to be an increasing function of only measurement errors of the item scores. Let us call this assumption the fixed true score variance assumption, which is crucial and reasonable from the perspective of measurement error theory in general. This assumption is crucial because it makes the total variance as a function of only measurement error variance as mentioned above, and it is reasonable because at the population level true score variance should not be varying whereas magnitudes of measurement error variance depend on reliability of items. Consequently, the true score variance ${\sigma }_{\mu }^{\mathsf{2}}$ is not a function of inter-item correlation ρ, but the measurement error variance ${\sigma }_e^{\mathsf{2}}$ is a decreasing function of ρ since from equation (3) we have

$${\sigma }_e^{\mathsf{2}}=\left(\mathsf{1}-\rho \right){\sigma }^{\mathsf{2}}=\left(\mathsf{1}/\rho -\mathsf{1}\right){\sigma }_{\mu }^{\mathsf{2}}\text{.}$$ 4

It follows that as the item scores are closer or more similar to each other within subjects, the measurement errors will be smaller, which follows that the total variance is also a decreasing function of ρ since

$${\sigma }^{\mathsf{2}}={\sigma }_{\mu }^{\mathsf{2}}+{\sigma }_e^{\mathsf{2}}={\sigma }_{\mu }^{\mathsf{2}}/\rho \text{.}$$ 5

We assume that the magnitudes of both ${\sigma }_e^{\mathsf{2}}$ and ${\sigma }_{\mu }^{\mathsf{2}}$ are known and thus that of σ² for the purpose of derivation of power functions based on normal destructions instead of t-distributions, although replacement by t-distributions should be straightforward yet with little difference in results for sizable sample sizes.

Cronbach alpha, scale score and its variance

We assume that there are k items in an instrument, i.e., j =1, 2, …, k. The C_α (1) of k items under model (2) and aforementioned assumptions can be expressed as

$${\mathsf{C}}_{\alpha }=\frac{\mathsf{k}{\sigma }_{\mu }^{\mathsf{2}}}{{\sigma }_e^{\mathsf{2}}+\mathsf{k}{\sigma }_{\mu }^{\mathsf{2}}}=\frac{\mathsf{k}\rho }{\mathsf{1}+\rho \left(\mathsf{k}-\mathsf{1}\right)}\text{.}$$ 6

It is due to the fact that $\mathsf{\Sigma }={\sigma }_e^{\mathsf{2}}\mathsf{I}+{\sigma }_{\mu }^{\mathsf{2}}\mathsf{1}{\mathsf{1}}^{\mathsf{T}}$ under model (2) where I is a k-by-k identity matrix. C_α in equation (6) is seen to be an increasing function of both ρ and k as depicted in Fig. 1. Therefore, the number of items needs to be fixed for comparison of C_α of several candidate sets of items. It follows that for a fixed number of items, higher C_α is associated with smaller measurement error of items through higher inter-item correlation ρ. From equation (6), ρ can be expressed in terms of C_α as follows:

$$\rho =\frac{{\mathsf{C}}_{\alpha }}{\mathsf{k}-{\mathsf{C}}_{\alpha }\left(\mathsf{k}-\mathsf{1}\right)}\text{.}$$ 7

Fig. 1.

Relationship between Cronbach alpha (C _α) and inter-item correlation (ρ) over varying number of items (k)

Of note, the corresponding correlation matrix is denoted by $\mathsf{{\rm P}}=\left(\mathsf{1}-\rho \right)\mathsf{I}+\rho \mathsf{1}{\mathsf{1}}^{\mathsf{T}}$, an equi-correlation matrix.

The k correlated items are often summed up to a scale that is intended to measure the target construct. The scale score is denoted here by

$${\mathsf{S}}_{\mathsf{i}}={\displaystyle \sum _{\mathsf{j}=\mathsf{1}}^{\mathsf{k}}{\mathsf{Y}}_{\mathsf{i}\mathsf{j}}}\text{,}$$

which can be viewed as an observed summary score for the i-th subject. Suppressing the subscription i in S_i, its mean and variance can be obtained as follows:

$$E_j\left(S\right)=k{\mu }_i\text{,}$$ 8

and

$$Var\left(S\right)=k{\sigma }^2\left\{1+\rho \left(k-1\right)\right\}\text{.}$$ 9

With respect to the mean (8), average scale score S_i/k when used as observed score is an unbiased estimate of true score μ_i for the i-th subject. The reliability, denoted here by R, defined as the squared correlation between true score and observed score can be obtained as follows:

$$\mathsf{R}=\mathsf{C}\mathsf{o}\mathsf{r}{\mathsf{r}}^{\mathsf{2}}\left({\mathsf{S}}_{\mathsf{i}}/\mathsf{k},{\mu }_{\mathsf{i}}\right)=\frac{\mathsf{k}\rho }{\mathsf{1}+\rho \left(\mathsf{k}-\mathsf{1}\right)}={\mathsf{C}}_{\alpha }\text{.}$$ 10

This equation supports Theorem 3.1 of Novick and Lewis [9] that R = C_α if and only if the items are parallel. Since statistical analysis results do not depend on whether S_i/k or S_i is used, we use the sum S in what follows.

With respect the total variance (9), if the total variance, instead of the true score variance, is assumed to be fixed, Var(S) is an increasing function of ρ, which conforms to an elementary statistical theory that variance of sum of correlated variables increases with increasing correlation. On the contrary, under the fixed true score variance assumption, it can be seen that Var(S) is a decreasing function of ρ since equation (9) can be re-expressed in terms of ${\sigma }_{\mu }^{\mathsf{2}}$ via equation (5) as follows:

$$\mathsf{V}\mathsf{a}\mathsf{r}\left(\mathsf{S}\right)=\mathsf{k}{\sigma }_{\mu }^{\mathsf{2}}\left(\mathsf{1}/\rho +\mathsf{k}-\mathsf{1}\right)={\mathsf{k}}^{\mathsf{2}}{\sigma }_{\mu }^{\mathsf{2}}/{\mathsf{C}}_{\alpha }\text{.}$$ 11

The last equation is due to equation (7). It follows that Var(S) is also a decreasing function of C_α. In sum, increase of ρ decreases the magnitude of σ² which in turn decreases the magnitude of Var(S); therefore such indirect decreasing effect of ρ on Var(S) is larger than direct increasing effect of ρ on Var(S) in equation (9).

Cronbach alpha and test-retest correlation

Reliability R of instruments is sometimes evaluated by test-retest correlation [3]. Based on model (2), the test and retest item scores can be specified as ${\mathsf{Y}}_{\mathsf{i}\mathsf{j}}^{\mathsf{t}e\mathsf{s}\mathsf{t}}={\mu }_{\mathsf{i}}+e_{\mathsf{i}\mathsf{j}}$ and ${\mathsf{Y}}_{\mathsf{i}\mathsf{j}}^{\mathsf{r}e\mathsf{t}e\mathsf{s}\mathsf{t}}={\mu }_{\mathsf{i}}+e_{\mathsf{i}\mathsf{j}}$, respectively with a common μ_i for both test and retest scores for each subject, i = 1, 2,…, N. The test-retest correlation can then be measured by the correlation, denoted by Corr(S_test, S_retest), between scale scores ${\mathsf{S}}_{\mathsf{t}e\mathsf{s}\mathsf{t}}={\displaystyle {\sum }_{\mathsf{j}=\mathsf{1}}^{\mathsf{k}}{\mathsf{Y}}_{\mathsf{i}\mathsf{j}}^{\mathsf{t}e\mathsf{s}\mathsf{t}}}$ and ${\mathsf{S}}_{\mathsf{r}e\mathsf{t}e\mathsf{s}\mathsf{t}}={\displaystyle {\sum }_{\mathsf{j}=\mathsf{1}}^{\mathsf{k}}{\mathsf{Y}}_{\mathsf{i}\mathsf{j}}^{\mathsf{r}e\mathsf{t}e\mathsf{s}\mathsf{t}}}$ representing the scale scores of test and retest, respectively. Under the aforementioned assumptions for model (2) it can be shown that

$$Cov\left(S_{\mathit{test}},S_{\mathit{retest}}\right)=k^2\rho {\sigma }^2\text{,}$$ 12

and from equation (10)

$$Var\left(S_{\mathit{test}}\right)=Var\left(S_{\mathit{retest}}\right)=k{\sigma }^2\left\{1+\rho \left(k-1\right)\right\}\text{.}$$ 13

It follows that:

$$\mathsf{Corr}\left({\mathsf{S}}_{\mathsf{t}e\mathsf{s}\mathsf{t}},{\mathsf{S}}_{\mathsf{r}e\mathsf{t}e\mathsf{s}\mathsf{t}}\right)=\frac{\mathsf{C}\mathsf{o}\mathsf{v}\left({\mathsf{S}}_{\mathsf{t}e\mathsf{s}\mathsf{t}}{\mathsf{S}}_{\mathsf{r}e\mathsf{t}e\mathsf{s}\mathsf{t}}\right)}{\sqrt{\mathsf{V}\mathsf{a}\mathsf{r}\left({\mathsf{S}}_{\mathsf{t}e\mathsf{s}\mathsf{t}}\right)}\sqrt{\mathsf{V}\mathsf{a}\mathsf{r}\left({\mathsf{S}}_{\mathsf{t}e\mathsf{s}\mathsf{t}}\right)}}=\frac{\mathsf{k}\rho }{\mathsf{1}+\rho \left(\mathsf{k}-\mathsf{1}\right)}=\mathsf{R}={\mathsf{C}}_{\alpha }\text{.}$$ 14

This equation shows that the test-rest correlation is the same as both C_α and R due to equations (6) and (10), which provides another interpretation of C_α. This property is especially useful when there is only one item available, in which case estimation of C_α or ρ is impossible by definition. However, the test and retest scores can be thought of as two correlated parallel item scores, and thus their correlation can serve as C_α of the single item. It is particularly fitting since ρ = C_α = R based on either equation (6), (7), or (14) when k = 1.

Taken together, the power ${\phi }_{{\mathsf{C}}_{\alpha }}$ of testing significance of C_α against any null value should be equivalent to that of testing significance of a correlation using a Fisher’s z-transformation as long as items are parallel, that is,

$${\phi }_{{\mathsf{C}}_{\alpha }}=\mathsf{1}-\Phi \left[{\Phi }^{-\mathsf{1}}\left(\mathsf{1}-\alpha /\mathsf{2}\right)-\sqrt{\mathsf{N}-\mathsf{3}}\left(\frac{\mathsf{1}}{\mathsf{2}}ln\left(\frac{\mathsf{1}+{\mathsf{C}}_{\alpha }}{\mathsf{1}-{\mathsf{C}}_{\alpha }}\right)+\frac{{\mathsf{C}}_{\alpha }}{\mathsf{2}\left(\mathsf{N}-\mathsf{1}\right)}\right)\right]$$

for a two-tailed significance level α, where Φ is the cumulative distribution function of a standardized normal distribution, and Φ⁻¹ is its inverse function, i.e., Φ(Φ⁻¹(x)) = Φ⁻¹(Φ(x)) = x. We note that although it is necessary to be added for validation of unbiasedness of the test statistics under the null hypothesis, the probability under the other rejection area will be ignored for all test statistics considered herein. For general covariance structures for non-parallel items, however, many other tests for significance of reliability and C_α have been developed [10–17].

Pre-post comparison

We consider application of a paired t-test to the case of comparison of within-group means of scale scores between pre- and post-interventions. Based on model (2), the pre- and post-intervention item scores can be specified as ${\mathsf{Y}}_{\mathsf{i}\mathsf{j}}^{\mathsf{p}\mathsf{r}e}={\mu }_{\mathsf{i}}+e_{\mathsf{i}\mathsf{j}}$ and ${\mathsf{Y}}_{\mathsf{i}\mathsf{j}}^{\mathsf{post}}={\mu }_{\mathsf{i}}+{\delta }_{\mathsf{P}\mathsf{P}}+e_{\mathsf{i}\mathsf{j}}$, respectively; the mean of the post-intervention item scores are shifted by δ_PP, an intervention effect. Consequently, we have

$$E\left(S_{\mathit{post}}\right)-E\left(S_{pre}\right)=k{\delta }_{\mathit{pp}}\text{,}$$ 15

where ${\mathsf{S}}_{\mathsf{p}\mathsf{r}e}={\displaystyle {\sum }_{\mathsf{j}=\mathsf{1}}^{\mathsf{k}}{\mathsf{Y}}_{\mathsf{i}\mathsf{j}}^{\mathsf{p}\mathsf{r}e}}$ and ${\mathsf{S}}_{\mathsf{post}}={\displaystyle {\sum }_{\mathsf{j}=\mathsf{1}}^{\mathsf{k}}{\mathsf{Y}}_{\mathsf{i}\mathsf{j}}^{\mathsf{post}}}$ are the pre- and post-intervention scale scores, respectively. A moment estimate of δ_PP from (15) can be estimated as

$${\widehat{\delta }}_{\mathsf{P}\mathsf{P}}=\left({\overline{\mathsf{S}}}_{\mathsf{post}}-{\overline{\mathsf{S}}}_{\mathsf{p}\mathsf{r}e}\right)/\mathsf{k}\text{,}$$ 16

where $\overline{\mathsf{S}}={\displaystyle {\sum }_{\mathsf{i}=\mathsf{1}}^{\mathsf{N}}{\displaystyle {\sum }_{\mathsf{j}=\mathsf{1}}^{\mathsf{k}}{\mathsf{Y}}_{\mathsf{i}\mathsf{j}}}}/\mathsf{N}$ and N is the total number of subject. Its variance can be obtained as

$$\mathsf{V}\mathsf{a}\mathsf{r}\left({\widehat{\delta }}_{\mathsf{P}\mathsf{P}}\right)=\frac{\mathsf{2}\left(\mathsf{1}-\rho \right){\sigma }^{\mathsf{2}}}{\mathsf{k}\mathsf{N}}=\frac{\mathsf{2}\left(\mathsf{1}/\rho -\mathsf{1}\right){\sigma }_{\mu }^{\mathsf{2}}}{\mathsf{k}\mathsf{N}}\text{.}$$ 17

It is because from equations (12) and (13) we have

$$\begin{array}{c}\mathsf{V}\mathsf{a}\mathsf{r}\left({\overline{\mathsf{S}}}_{\mathsf{post}}-{\overline{\mathsf{S}}}_{\mathsf{p}\mathsf{r}e}\right)=\mathsf{V}\mathsf{a}\mathsf{r}\left({\overline{\mathsf{S}}}_{\mathsf{post}}\right)+\mathsf{V}\mathsf{a}\mathsf{r}\left({\overline{\mathsf{S}}}_{\mathsf{p}\mathsf{r}e}\right)-\mathsf{2}\mathsf{C}\mathsf{o}\mathsf{v}\left({\overline{\mathsf{S}}}_{\mathsf{post}},{\overline{\mathsf{S}}}_{\mathsf{p}\mathsf{r}e}\right)\\ =\mathsf{k}{\sigma }^{\mathsf{2}}\left\{\mathsf{1}+\rho \left(\mathsf{k}-\mathsf{1}\right)\right\}/\mathsf{N}+\mathsf{k}{\sigma }^{\mathsf{2}}\left\{\mathsf{1}+\rho \left(\mathsf{k}-\mathsf{1}\right)\right\}/\mathsf{N}-\mathsf{2}{\mathsf{k}}^{\mathsf{2}}\rho {\sigma }^{\mathsf{2}}/\mathsf{N}\\ =\mathsf{2}\mathsf{k}{\sigma }^{\mathsf{2}}\left(\mathsf{1}-\rho \right)/\mathsf{N}=\mathsf{2}\mathsf{k}{\sigma }_{\mu }^{\mathsf{2}}\left(\mathsf{1}/\rho -\mathsf{1}\right)/\mathsf{N}\text{.}\end{array}$$

The following test statistic can then be used for testing H₀: δ = 0

$${\mathsf{T}}_{\mathsf{P}\mathsf{P}}=\frac{{\widehat{\delta }}_{\mathsf{P}\mathsf{P}}}{\sqrt{\mathsf{V}\mathsf{a}\mathsf{r}\left({\widehat{\delta }}_{\mathsf{P}\mathsf{P}}\right)}}=\frac{\sqrt{\mathsf{k}\mathsf{N}}{\widehat{\delta }}_{\mathsf{P}\mathsf{P}}}{{\sigma }_{\mu }\sqrt{\mathsf{2}\left(\mathsf{1}/\rho -\mathsf{1}\right)}}=\frac{\sqrt{\mathsf{N}}\left({\overline{\mathsf{S}}}_{\mathsf{post}}-{\overline{\mathsf{S}}}_{\mathsf{p}\mathsf{r}e}\right)}{{\sigma }_{\mu }\sqrt{\mathsf{2}\mathsf{k}\left(\mathsf{1}/\rho -\mathsf{1}\right)}}\text{.}$$ 18

Now, the statistical power φ_PP of T_PP for detecting non-zero δ_PP can be expressed as follows:

$${\phi }_{\mathsf{P}\mathsf{P}}=\Phi \left\{\left|{\delta }_{\mathsf{P}\mathsf{P}}/{\sigma }_{\mu }\right|\sqrt{\frac{\mathsf{k}\mathsf{N}}{\mathsf{2}\left(\mathsf{1}/\rho -\mathsf{1}\right)}}-{\Phi }^{-\mathsf{1}}\left(\mathsf{1}-\alpha /\mathsf{2}\right)\right\}\text{.}$$ 19

This statistical power is an increasing function of ρ for a fixed σ_μ, which we assume. It follows that the power is also an increasing function of C_α as seen next. When δ_PP is standardized by σ_μ and ρ is replaced by equation (7), equation (19) can further be expressed in terms of ${\Delta }_{\mathsf{P}\mathsf{P}}={\delta }_{\mathsf{P}\mathsf{P}}/{\sigma }_{\mu }$ and C_α as follows:

$${\phi }_{\mathsf{P}\mathsf{P}}=\Phi \left\{\left|{\Delta }_{\mathsf{P}\mathsf{P}}\right|\sqrt{\frac{\mathsf{N}}{\mathsf{2}\left(\mathsf{1}/{\mathsf{C}}_{\alpha }-\mathsf{1}\right)}}-{\Phi }^{-\mathsf{1}}\left(\mathsf{1}-\alpha /\mathsf{2}\right)\right\}\text{.}$$ 20

This power function is seen to be independent of k, the number of items. Stated differently, the power will be the same between two instruments with different numbers of items as long as their C_α’s are the same even if the correlation of items will be smaller for the instrument with fewer items.

When sample size determination is needed for a study using an instrument of any number of items with a known C_α for a desired statistical power φ, typically 80 %, it can be determined from equation as follows:

$$\mathsf{N}=\frac{\mathsf{2}\left(\mathsf{1}/{\mathsf{C}}_{\alpha }-\mathsf{1}\right){\mathsf{z}}_{\alpha ,\phi }^{\mathsf{2}}}{{\Delta }_{\mathsf{P}\mathsf{P}}^{\mathsf{2}}}\text{,}$$ 21

where

$${\mathsf{z}}_{\alpha ,\phi }={\Phi }^{-\mathsf{1}}\left(\mathsf{1}-\alpha /\mathsf{2}\right)+{\Phi }^{-\mathsf{1}}\left(\phi \right)\text{.}$$ 22

The sample size (21) is seen to be a decreasing function of increasing C_α and Δ. In a possibly rare case in which determination of number of items with known correlations among them is needed for development of an instrument, it has to be determined from equation (19), instead of equation (20), as follow:

$$\mathsf{k}=\frac{\mathsf{2}\left(\mathsf{1}/\rho -\mathsf{1}\right){\mathsf{z}}_{\alpha ,\phi }^{\mathsf{2}}}{\mathsf{N}{\Delta }_{\mathsf{P}\mathsf{P}}^{\mathsf{2}}}\text{.}$$ 23

Comparison of within-group effects between groups

In clinical trials, it is often of interest to compare within-group changes between groups. For instance, a clinical trial can be designed to compare of pre-post effect of an experimental treatment between treatment and control groups, that is, an interaction effect between group and time point. Based on model (2), the pre- and post-intervention item scores can be specified as ${\mathsf{Y}}_{\mathsf{i}\mathsf{j}}^{\mathsf{p}\mathsf{r}e\left(\mathsf{0}\right)}={\mu }_{\mathsf{i}}^{\left(\mathsf{0}\right)}+e_{\mathsf{i}\mathsf{j}}$ and ${\mathsf{Y}}_{\mathsf{i}\mathsf{j}}^{\mathsf{post}\left(\mathsf{0}\right)}={\mu }_{\mathsf{i}}^{\left(\mathsf{0}\right)}+{\delta }_{\mathsf{0}}+e_{\mathsf{i}\mathsf{j}}$ for the control group ${\mathsf{Y}}_{\mathsf{i}\mathsf{j}}^{\mathsf{p}\mathsf{r}e\left(\mathsf{1}\right)}={\mu }_{\mathsf{i}}^{\left(\mathsf{1}\right)}+e_{\mathsf{i}\mathsf{j}}$ and ${\mathsf{Y}}_{\mathsf{i}\mathsf{j}}^{\mathsf{post}\left(\mathsf{1}\right)}={\mu }_{\mathsf{i}}^{\left(\mathsf{1}\right)}+{\delta }_{\mathsf{1}}+e_{\mathsf{i}\mathsf{j}}$ for the treatment group. The primary interest will be testing H_o: δ_BW = δ₁ – δ₀ = 0, i.e., whether or not the difference in pre-post differences between groups will be the same. Consequently, we have

$$E\left\{D_{trt}\left(S\right)\right\}-E\left\{D_{\mathit{control}}\left(S\right)\right\}=k{\delta }_{BW}\text{,}$$ 24

where ${\mathsf{D}}_{\mathsf{t}\mathsf{r}\mathsf{t}}\left(\mathsf{S}\right)={\mathsf{S}}_{\mathsf{post}\left(\mathsf{1}\right)}-{\mathsf{S}}_{\mathsf{p}\mathsf{r}e\left(\mathsf{1}\right)}={\displaystyle {\sum }_{\mathsf{j}=\mathsf{1}}^{\mathsf{k}}{\mathsf{Y}}_{\mathsf{i}\mathsf{j}}^{\mathsf{post}\left(\mathsf{1}\right)}}-{\displaystyle {\sum }_{\mathsf{j}=\mathsf{1}}^{\mathsf{k}}{\mathsf{Y}}_{\mathsf{i}\mathsf{j}}^{\mathsf{p}\mathsf{r}e\left(\mathsf{1}\right)}}$ and ${\mathsf{D}}_{\mathsf{c}\mathsf{o}n\mathsf{trol}}\left(\mathsf{S}\right)$ can be similarly defined. A moment estimate of δ_BW from (24) can be obtained as

$${\widehat{\delta }}_{\mathsf{B}\mathsf{W}}=\left({\overline{\mathsf{D}}}_{\mathsf{t}\mathsf{r}\mathsf{t}}-{\overline{\mathsf{D}}}_{\mathsf{c}\mathsf{o}n\mathsf{trol}}\right)/\mathsf{k}\text{,}$$ 25

where N is the number of subjects per group, ${\overline{\mathsf{D}}}_{\mathsf{t}\mathsf{r}\mathsf{t}}\text{≡}{\overline{\mathsf{D}}}_{\mathsf{t}\mathsf{r}\mathsf{t}}\left(\mathsf{S}\right)={\overline{\mathsf{S}}}_{\mathsf{post}\left(\mathsf{1}\right)}-{\overline{\mathsf{S}}}_{\mathsf{p}\mathsf{r}e\left(\mathsf{1}\right)}=$${\displaystyle {\sum }_{\mathsf{i}=\mathsf{1}}^{\mathsf{N}}{\displaystyle {\sum }_{\mathsf{j}}^{\mathsf{k}}{\mathsf{Y}}_{\mathsf{i}\mathsf{j}}^{\mathsf{post}\left(\mathsf{1}\right)}}}/\mathsf{N}-{\displaystyle {\sum }_{\mathsf{i}=\mathsf{1}}^{\mathsf{N}}{\displaystyle {\sum }_{\mathsf{j}}^{\mathsf{k}}{\mathsf{Y}}_{\mathsf{i}\mathsf{j}}^{\mathsf{p}\mathsf{r}e\left(\mathsf{1}\right)}}}/\mathsf{N}$, and, ${\overline{\mathsf{D}}}_{\mathsf{c}\mathsf{o}n\mathsf{trol}}$ can similarly be defined. The variance of ${\widehat{\delta }}_{\mathsf{W}\mathsf{B}}$ is

$$\mathsf{V}\mathsf{a}\mathsf{r}\left({\widehat{\delta }}_{\mathsf{B}\mathsf{W}}\right)=\frac{\mathsf{4}\left(\mathsf{1}-\rho \right){\sigma }^{\mathsf{2}}}{\mathsf{k}\mathsf{N}}=\frac{\mathsf{4}\left(\mathsf{1}/\rho -\mathsf{1}\right){\sigma }_{\mu }^{\mathsf{2}}}{\mathsf{k}\mathsf{N}}\text{.}$$ 26

Therefore, the following test statistic can be used for testing the null hypothesis H_o: δ_BW = 0,

$${\mathsf{T}}_{\mathsf{B}\mathsf{W}}=\frac{{\widehat{\delta }}_{\mathsf{B}\mathsf{W}}}{\sqrt{\mathsf{V}\mathsf{a}\mathsf{r}\left({\widehat{\delta }}_{\mathsf{B}\mathsf{W}}\right)}}=\frac{\sqrt{\mathsf{k}\mathsf{N}}{\widehat{\delta }}_{\mathsf{B}\mathsf{W}}}{\mathsf{2}{\sigma }_{\mu }\sqrt{\left(\mathsf{1}/\rho -\mathsf{1}\right)}}=\frac{\sqrt{\mathsf{N}}\left({\overline{\mathsf{D}}}_{\mathsf{t}\mathsf{r}\mathsf{t}}-{\overline{\mathsf{D}}}_{\mathsf{c}\mathsf{o}n\mathsf{trol}}\right)}{\mathsf{2}{\sigma }_{\mu }\sqrt{\mathsf{k}\left(\mathsf{1}/\rho -\mathsf{1}\right)}}\text{.}$$ 27

The statistical power φ_BW of T_BW for detecting non-zero δ_BW can thus be expressed as follows:

$${\phi }_{\mathsf{B}\mathsf{W}}=\Phi \left\{\left|{\delta }_{\mathsf{B}\mathsf{W}}/{\sigma }_{\mu }\right|\sqrt{\frac{\mathsf{k}\mathsf{N}}{\mathsf{4}\left(\mathsf{1}/\rho -\mathsf{1}\right)}}-{\Phi }^{-\mathsf{1}}\left(\mathsf{1}-\alpha /\mathsf{2}\right)\right\}\text{.}$$ 28

Again, this statistical power is an increasing of ρ and of C_α as well as seen next. When δ_BW is standardized by σ_μ and ρ is replaced by equation (7), equation (28) can further be expressed in terms of ${\Delta }_{\mathsf{B}\mathsf{W}}={\delta }_{\mathsf{B}\mathsf{W}}/{\sigma }_{\mu }$ and C_α as follows:

$${\phi }_{\mathsf{B}\mathsf{W}}=\Phi \left\{\left|{\Delta }_{\mathsf{B}\mathsf{W}}\right|\sqrt{\frac{\mathsf{N}}{\mathsf{4}\left(\mathsf{1}/{\mathsf{C}}_{\alpha }-\mathsf{1}\right)}}-{\Phi }^{-\mathsf{1}}\left(\mathsf{1}-\alpha /\mathsf{2}\right)\right\}\text{.}$$ 29

Again, this power function is seen to be independent of k, the number of items.

Sample size for a desired statistical power φ can be determined from (27) as follows:

$$\mathsf{N}=\frac{\mathsf{4}\left(\mathsf{1}/{\mathsf{C}}_{\alpha }-\mathsf{1}\right){\mathsf{z}}_{\alpha ,\phi }^{\mathsf{2}}}{{\Delta }_{\mathsf{B}\mathsf{W}}^{\mathsf{2}}}\text{.}$$ 30

Again, this sample size (30) is seen to be a decreasing function of increasing C_α and Δ. When number of items is needed for development of an instrument, it can be determined from equation (28) as follow:

$$\mathsf{k}=\frac{\mathsf{2}\left(\mathsf{1}/\rho -\mathsf{1}\right){\mathsf{z}}_{\alpha ,\phi }^{\mathsf{2}}}{\mathsf{N}{\Delta }_{\mathsf{B}\mathsf{W}}^{\mathsf{2}}}\text{.}$$ 31

Two-sample between-group comparison

Comparison of means between groups using an instrument is widely tested in clinical trials. Based on model (2), the intervention item scores from control and treatment groups can be specified as ${\mathsf{Y}}_{\mathsf{i}\mathsf{j}}^{\left(\mathsf{0}\right)}={\mu }_{\mathsf{i}}+e_{\mathsf{i}\mathsf{j}}$ and ${\mathsf{Y}}_{\mathsf{i}\mathsf{j}}^{\left(\mathsf{1}\right)}={\mu }_{\mathsf{i}}+{\delta }_{\mathsf{T}\mathsf{S}}+e_{\mathsf{i}\mathsf{j}}$, respectively. The primary interest will be testing H_o: δ_TS = 0, i.e., whether or not the means are the same between the two groups. Under this formulation, we have

$$E\left(S_{trt}\right)=E\left(S_{\mathit{control}}\right)+k{\delta }_{TS}\text{,}$$ 32

where ${\mathsf{S}}_{\mathsf{t}\mathsf{r}\mathsf{t}}={\displaystyle {\sum }_{\mathsf{j}=\mathsf{1}}^{\mathsf{k}}{\mathsf{Y}}_{\mathsf{i}\mathsf{j}}^{\left(\mathsf{1}\right)}}$ and ${\mathsf{S}}_{\mathsf{c}\mathsf{o}n\mathsf{trol}}={\displaystyle {\sum }_{\mathsf{j}=\mathsf{1}}^{\mathsf{k}}{\mathsf{Y}}_{\mathsf{i}\mathsf{j}}^{\left(\mathsf{0}\right)}}$ represents scale scores under treatment and control groups, respectively. A moment estimate of δ_TS can be obtained from (32) as

$${\widehat{\delta }}_{\mathsf{T}\mathsf{S}}=\left({\overline{\mathsf{S}}}_{\mathsf{t}\mathsf{r}\mathsf{t}}-{\overline{\mathsf{S}}}_{\mathsf{c}\mathsf{o}n\mathsf{trol}}\right)/\mathsf{k}\text{,}$$ 33

where ${\overline{\mathsf{S}}}_{\mathsf{t}\mathsf{r}\mathsf{t}}={\displaystyle {\sum }_{\mathsf{i}=\mathsf{1}}^{\mathsf{N}}{\displaystyle {\sum }_{\mathsf{j}=\mathsf{1}}^{\mathsf{k}}{\mathsf{Y}}_{\mathsf{i}\mathsf{j}}^{\left(\mathsf{1}\right)}}}/\mathsf{N}$, ${\overline{\mathsf{S}}}_{\mathsf{c}\mathsf{o}n\mathsf{trol}}={\displaystyle {\sum }_{\mathsf{i}=\mathsf{1}}^{\mathsf{N}}{\displaystyle {\sum }_{\mathsf{j}=\mathsf{1}}^{\mathsf{k}}{\mathsf{Y}}_{\mathsf{i}\mathsf{j}}^{\left(\mathsf{0}\right)}}}/\mathsf{N}$ and N is the number of participants per group. The variance of ${\widehat{\delta }}_{\mathsf{T}\mathsf{S}}$ can be obtained as

$$\mathsf{V}\mathsf{a}\mathsf{r}\left({\widehat{\delta }}_{\mathsf{T}\mathsf{S}}\right)=\frac{\mathsf{2}\left\{\mathsf{1}+\rho \left(\mathsf{k}-\mathsf{1}\right)\right\}{\sigma }^{\mathsf{2}}}{\mathsf{k}\mathsf{N}}=\frac{\mathsf{2}\left\{\mathsf{1}/\rho +\mathsf{k}-\mathsf{1}\right\}{\sigma }_{\mu }^{\mathsf{2}}}{\mathsf{k}\mathsf{N}}\text{.}$$ 34

The corresponding test statistic T_TS can be built as

$${\mathsf{T}}_{\mathsf{T}\mathsf{S}}=\frac{{\widehat{\delta }}_{\mathsf{T}\mathsf{S}}}{\sqrt{\mathsf{V}\mathsf{a}\mathsf{r}\left({\widehat{\delta }}_{\mathsf{T}\mathsf{S}}\right)}}=\frac{\sqrt{\mathsf{k}\mathsf{N}}{\widehat{\delta }}_{\mathsf{T}\mathsf{S}}}{{\sigma }_{\mu }\sqrt{\mathsf{2}\left(\mathsf{1}/\rho +\mathsf{k}-\mathsf{1}\right)}}=\frac{\sqrt{\mathsf{N}}\left({\overline{\mathsf{S}}}_{\mathsf{t}\mathsf{r}\mathsf{t}}-{\overline{\mathsf{S}}}_{\mathsf{c}\mathsf{o}n\mathsf{trol}}\right)}{{\sigma }_{\mu }\sqrt{\mathsf{2}\mathsf{k}\left(\mathsf{1}/\rho +\mathsf{k}-\mathsf{1}\right)}}\text{.}$$ 35

And the power function φ_TS of T_TS can be expressed as

$${\phi }_{\mathsf{T}\mathsf{S}}=\Phi \left\{\left|{\delta }_{\mathsf{T}\mathsf{S}}/{\sigma }_{\mu }\right|\sqrt{\frac{\mathsf{k}\mathsf{N}}{\mathsf{2}\left(\mathsf{1}/\rho +\mathsf{k}-\mathsf{1}\right)}}-{\Phi }^{-\mathsf{1}}\left(\mathsf{1}-\alpha /\mathsf{2}\right)\right\}\text{.}$$ 36

It should be noted that this statistical power (36) is also an increasing function of ρ in contrast to a situation when a fixed total variance assumption is more reasonable in which both ${\sigma }_e^{\mathsf{2}}$ and ${\sigma }_{\mu }^{\mathsf{2}}$ are a function of ρ but σ² is not. For example, observations without measurement errors from clusters are often assumed to be correlated and power of between-group tests using such correlated observations is a decreasing function of ρ [18]. Again, when δ_TS is standardized by σ_μ and ρ is replaced by equation (7), equation (33) can further be expressed in terms can further be expressed in terms of ${\Delta }_{\mathsf{T}\mathsf{S}}={\delta }_{\mathsf{T}\mathsf{S}}/{\sigma }_{\mu }$ and C_α as follows:

$${\phi }_{\mathsf{T}\mathsf{S}}=\Phi \left\{\left|{\Delta }_{\mathsf{T}\mathsf{S}}\right|\sqrt{{\mathsf{C}}_{\alpha }\mathsf{N}/\mathsf{2}}-{\Phi }^{-\mathsf{1}}\left(\mathsf{1}-\alpha /\mathsf{2}\right)\right\}\text{.}$$ 37

Again, this power function is seen to be independent of k, the number of items.

Sample size for a desired statistical power φ can be determined from (37) as follows:

$$\mathsf{N}=\frac{\mathsf{2}{\mathsf{z}}_{\alpha ,\phi }^{\mathsf{2}}}{{\mathsf{C}}_{\alpha }{\Delta }_{\mathsf{B}\mathsf{W}}^{\mathsf{2}}}\text{.}$$ 38

Again, the sample size (38) is seen to be a decreasing function of increasing C_α and Δ. When number of items is needed for development of an instrument, it can be determined from equation (36) as follow:

$$\mathsf{k}=\frac{\mathsf{2}\left(\mathsf{1}/\rho -\mathsf{1}\right){\mathsf{z}}_{\alpha ,\phi }^{\mathsf{2}}/{\Delta }_{\mathsf{T}\mathsf{S}}^{\mathsf{2}}}{\mathsf{N}-\mathsf{2}{\mathsf{z}}_{\alpha ,\phi }^{\mathsf{2}}/{\Delta }_{\mathsf{T}\mathsf{S}}^{\mathsf{2}}}\text{.}$$ 39

Results

To validate equation (14) and the power functions (20), (29), and (37), we conduct simulation study for each test. For the simulation, the random item scores are generated based on model (2) assuming both μ_i and e_ij are normally distributed although this assumption is not required in general. Under this normal assumption, however, it can be shown that all the moment estimates herein are the maximum likelihood estimates [19]. We then compute scale scores by summing up the item scores for each individual.

We fix a two-tailed significance level of α = 0.05 and ${\sigma }_{\mu }^{\mathsf{2}}$ = 1 without loss generality for all simulations, and determine ${\sigma }_e^{\mathsf{2}}$ and σ² through ρ determined by given k and C_α. We randomly generate 1000 data sets for each combination of design parameters that include effect size Δ, number of items k, and sample size N. We then compute empirical power $\tilde{\phi }$ by counting data sets from which two-tailed p-values are smaller than 0.05; that is, $\tilde{\phi }={\displaystyle {\sum }_{\mathsf{s}}^{\mathsf{1000}}\mathsf{1}\left({\mathsf{p}}_{\mathsf{s}}<\alpha \right)}/\mathsf{1000}$ where p_s represents a two-sided p-value from the s-th simulated data set. For the testing, we applied corresponding t-tests assuming the variances of the moment estimates are unknown, which is practically reasonable. We used SAS v9.3 for the simulations.

Test-retest correlation

The results are presented in Table 1 that shows the empirically estimated test-retest correlations (i.e., average of 1000 estimated Pearson correlations for each set of design parameter specifications) are approximately the same as the pre-assigned C_α, regardless of sample size N, which is as small as 30, and number of items k. Therefore, equality between C_α and test-retest correlation (14) is well validated.

Table 1.

Empirical simulation-based estimates of test-retest correlation Corr(S _test, S _retest) in equation (14)

	Corr(S test, S retest)
	Total N = 30		Total N = 50
C α	k = 5	k = 10	k = 5	k = 10
0.1	0.10	0.10	0.10	0.10
0.2	0.20	0.20	0.20	0.20
0.3	0.30	0.29	0.30	0.30
0.4	0.39	0.39	0.40	0.39
0.5	0.49	0.50	0.49	0.50
0.6	0.59	0.59	0.60	0.60
0.7	0.69	0.69	0.70	0.70
0.8	0.79	0.80	0.80	0.79
0.9	0.90	0.90	0.90	0.90

Note: Total N: total number of subjects; C _α: Cronbach alpha; k: number of items

Pre-post intervention comparison

Table 2 shows that the theoretical power φ_PP (20) is very close to the empirical power ${\tilde{\phi }}_{\mathsf{P}\mathsf{P}}$ obtained through the simulations. The results validate that the power φ_PP increases with increasing C_α (or equivalently increasing correlation for the same k) in the “pre-post” test settings, regardless of sample size N and number of items k. Furthermore, it shows that the statistical power does not depend on k for a given C_α even if correlation ρ does.

Table 2.

Statistical power of the pre-post test T _PP (18): σ _μ = 1

			k = 5		k = 10
Total N	Δ PP	C α	φ PP	${\tilde{\phi }}_{\mathsf{P}\mathsf{P}}$	φ PP	${\tilde{\phi }}_{\mathsf{P}\mathsf{P}}$
30	0.4	0.5	0.341	0.337	0.341	0.310
		0.6	0.475	0.459	0.475	0.458
		0.7	0.658	0.626	0.658	0.649
		0.8	0.873	0.849	0.873	0.830
		0.9	0.996	0.997	0.996	0.995
50	0.3	0.5	0.323	0.309	0.323	0.296
		0.6	0.451	0.424	0.451	0.433
		0.7	0.630	0.633	0.630	0.614
		0.8	0.851	0.849	0.851	0.844
		0.9	0.994	0.995	0.994	0.992

Note: Total N: total number of subjects; k: number of items; ${\Delta }_{\mathsf{P}\mathsf{P}}={\delta }_{\mathsf{P}\mathsf{P}}/{\sigma }_{\mu }$; C _α: Cronbach alpha; φ _PP: theoretical power (20); ${\tilde{\phi }}_{\mathsf{P}\mathsf{P}}$: simulation-based empirical power

Between-group whithin-group comparison

Table 3 shows that the theoretical power φ_BW (29) is very close to the empirical power ${\tilde{\phi }}_{\mathsf{B}\mathsf{W}}$ obtained through the simulations. Therefore, the results validate that the statistical power φ_BW increases with increasing C_α for testing hypotheses concerning between-group effects on within-group changes regardless of N, sample size per group, and k. Again, it shows that the statistical power does not depend on k for a given C_α even if correlation ρ does.

Table 3.

Statistical power of the between-group within-group test T _BW (25): σ _μ = 1

			k = 5		k = 10
N per group	Δ BW	C α	φ BW	${\tilde{\phi }}_{\mathsf{B}\mathsf{W}}$	φ BW	${\tilde{\phi }}_{\mathsf{B}\mathsf{W}}$
30	0.4	0.5	0.194	0.179	0.183	0.194
		0.6	0.268	0.264	0.254	0.268
		0.7	0.387	0.375	0.359	0.387
		0.8	0.591	0.618	0.594	0.591
		0.9	0.908	0.884	0.901	0.908
50	0.3	0.5	0.164	0.184	0.214	0.184
		0.6	0.242	0.254	0.261	0.254
		0.7	0.387	0.367	0.365	0.367
		0.8	0.511	0.564	0.591	0.564
		0.9	0.893	0.889	0.893	0.889

Note: N per group: number of subjects per group; k: number of items; ${\Delta }_{\mathsf{B}\mathsf{W}}={\delta }_{\mathsf{B}\mathsf{W}}/{\sigma }_{\mu }$; C _α: Cronbach alpha; φ _BW: theoretical power (27); ${\tilde{\phi }}_{\mathsf{B}\mathsf{W}}$: simulation-based empirical power

Two-sample between-group comparison

Table 4 shows again that the theoretical power φ_TS (37) is very close to the empirical power ${\tilde{\phi }}_{\mathsf{T}\mathsf{S}}$ obtained through the simulations. The results validate that the statistical power increases with increasing Cronbach α even for two-sample testing in cross-sectional settings that does not involve within-group effects. it shows that the statistical power does not depend on k for a given C_α even if correlation ρ does. Again, it shows that the statistical power does not depend on k for a given C_α even if correlation ρ does.

Table 4.

Statistical power of the between-group within-group test T _TS (32): σ _μ = 1

			k = 5		k = 10
N per group	Δ TS	C α	φ TS	${\tilde{\phi }}_{\mathsf{T}\mathsf{S}}$	φ TS	${\tilde{\phi }}_{\mathsf{T}\mathsf{S}}$
50	0.7	0.5	0.697	0.676	0.697	0.697
		0.6	0.774	0.758	0.774	0.760
		0.7	0.834	0.812	0.834	0.813
		0.8	0.879	0.872	0.879	0.882
		0.9	0.913	0.901	0.913	0.895
100	0.5	0.5	0.705	0.682	0.705	0.679
		0.6	0.782	0.791	0.782	0.769
		0.7	0.841	0.820	0.841	0.832
		0.8	0.885	0.879	0.885	0.908
		0.9	0.918	0.929	0.918	0.912

Note: N per group: number of subjects per group; k: number of items; ${\Delta }_{\mathsf{T}\mathsf{S}}={\delta }_{\mathsf{T}\mathsf{S}}/{\sigma }_{\mu }$; C _α: Cronbach alpha; φ _TS: theoretical power (34); ${\tilde{\phi }}_{\mathsf{T}\mathsf{S}}$: simulation-based empirical power

Discussion

We demonstrate by deriving explicit power functions that higher internal consistency or reliability of unidimensional parallel instrument items measured by Cronbach alpha C_α results in greater statistical power of several tests regardless of whether comparisons are made within or between groups. In addition, the test-retest reliability correlation of such items is shown to be the same as Cronbach alpha C_α. Due to this property, testing significance of C_α can be equivalent to testing that of a correlation through the Fisher z-transformation. Furthermore, all of the power functions derived herein can even be applied to trials using single item instrument with measurement error since the power function depends only on C_α which can be estimated via test-retest correlations for single item instruments as mentioned earlier. The demonstrations are made theoretically, and validations are made through simulation studies that show that the derived test statistics and their corresponding power functions are very close to each other. Therefore, the sample size determination formulas (21), (30), and (38) are valid and so are the determinations of number of items (22), (31), and (39) in different settings.

In fact, for longitudinal studies aiming to compare within-group effects using such as T_PP (18) and T_BW (27), the fixed true score variance assumption is not critical since the true score μ_i’s in model (2) are cancelled by taking differences of Y between pre and post-interventions and thus makes the variance of the pre-post differences depend only on measurement error variance ${\sigma }_e^{\mathsf{2}}$. For example, the variance equations (17) and (26) can be expressed in term of only ${\sigma }_e^{\mathsf{2}}$, a decreasing function of ρ, through equation (4) as follows: $\mathsf{V}\mathsf{a}\mathsf{r}\left({\widehat{\delta }}_{\mathsf{P}\mathsf{P}}\right)=\mathsf{2}{\sigma }_e^{\mathsf{2}}/\left(\mathsf{k}\mathsf{N}\right)$ and $\mathsf{V}\mathsf{a}\mathsf{r}\left({\widehat{\delta }}_{\mathsf{B}\mathsf{W}}\right)=\mathsf{4}{\sigma }_e^{\mathsf{2}}/\left(\mathsf{k}\mathsf{N}\right)$. In other words, both the power functions φ_PP (20) and φ_BW (29) are increasing function of C_α or ρ regardless of whether total variance or true score variance is assumed fixed.

In contrast, however, for cross-sectional studies aiming to compare between-group effects using T_TS (35), the fixed true score variance assumption is critical since the variance equation (34) cannot be expressed only in term of only ${\sigma }_e^{\mathsf{2}}$, and furthermore it can be shown that under a fixed total variance assumption $\mathsf{V}\mathsf{a}\mathsf{r}\left({\widehat{\delta }}_{\mathsf{T}\mathsf{S}}\right)$ (34) is an increasing function of ρ (see equation (10)) and so is the power function. In sum, the fixed true score variance assumption enables all of the power functions to be an increasing function of C_α or ρ in a unified fashion. For example, Leon et al. [20] used a real data set of HRSD ratings to empirically demonstrate that the statistical power of a two-sample between-group test is increasing with increased C_α, although they increased C_α by increasing number of items k, not necessarily by increasing ρ for a fixed number of items.

In most cases, item scores are designed to be binary or ordinal scores on a likert scale. Therefore, the applicability of the derived power functions and sample size formulas to such cases could be in question since the scores are not normally distributed. Furthermore, it is not easy to build a model like (2) for non-normal scores particularly because measurement error variances depend on the true construct value. For example, variance of a binary score is a function of its mean. Perhaps, construction of marginal models in the sense of generalized estimating equations [21] can be considered for derivation of power functions assumption even if this approach is beyond the scope of the present study. After all, we believe that our study results should be able to be applied to non-normal scores by virtue of the central limit theorem. Another prominent limitation of our study is the very strong assumption of essentially τ-equivalent parallel items which may not be realistic at all [8], albeit conceivable for a unidimensional construct. Therefore, further development of power functions under relaxed conditions reflecting more real world situations should be a valuable future study.

Conclusion

Instruments with greater Cronbach alpha should be used for any type of research since they have smaller measurement error and thus have greater statistical power for any research settings, cross-sectional or longitudinal. However, when items are parallel targeting a unidimensional construct, Cronbach alpha of an instrument should be enhanced by developing a set of highly correlated items but not by unduly increasing the number of items with inadequate inter-item correlations.

Abbreviations

HRSD

Hamilton Rating Scale of Depression