Sample Size Calculations for Blinding Assessment

Abstract

Blinding is a critical component in randomized clinical trials along with treatment effect estimation and comparisons between the treatments. Various methods have been proposed for the statistical analyses of blinding-related data but there is little guidance for determining the sample size for this type of data, especially if blinding assessment is done in pilot studies. In this paper, we try to fill this gap and provide simple methods to address sample size calculations for a ‘new’ study with different research questions and scenarios. The proposed methods are framed in terms of estimation/precision or statistical testing to allow investigators to choose the best suited method for their goals. We illustrate the methods using worked examples with real data.

1. Introduction

Blinding has been widely perceived to be important in randomized controlled trials (RCT) and other comparative evaluations. Traditionally, blinding-related issues have been discussed more qualitatively or conceptually, for example, “blinding is important” and “double (or triple) blind is the best” (Hopton and Macpherson, 2011; Jadad, et al., 1996; Kolahi, et al., 2009; Wilsey, et al., 2016). In the last two decades, a growing number of studies on blinding have pursued a quantitative approach to study design, data collection, analysis and interpretation (Arandjelović, 2012; Bang, et al., 2010; Chow and Shao, 2004; James, et al., 1996; Jeong, et al., 2013; Wilsey, et al., 2016; Wright, et al., 2012). Nowadays, blinding is also emphasized in non-pharmacological trials, like trials involving devices or physical therapy, in order to demonstrate internal validity. Meta-analyses of blinding have offered some optimistic news about the feasibility of blinding for some interventions, which are traditionally believed to be hard to blind, including non-drug or non-injection (Boutron, et al., 2007; Brinjikji, et al., 2010; Freed, et al., 2014; Hopton and Macpherson, 2011; Houweling, et al., 2014; Moroz, et al., 2013; Wilsey, et al., 2016).

Despite various statistical and methodological proposals, there is still little guidance concerning the determination of minimal or adequate sample size (N) for blinding assessment in a statistically justifiable manner. In the past, it was not unusual for a statistician to advise clients seeking sample size calculations for a blinding assessment to take a sample of at least N=30, or perhaps N=100 patients, in the absence of a good reference. In some cases, statisticians may have even said, “Use any N available”. Recently, Shin et al. (Shin, et al., 2016) proposed a pilot RCT on acupuncture with 2 centers and selected a blinding index (BI) as one of the study outcomes (Bang, et al., 2004). This team recruited 40 participants which may be reasonable for a pilot study testing a clinical outcome, but it is unclear whether it is sufficient for testing a blinding outcome. In another example, Vernon and his colleagues proposed a new study for blinding assessment of real vs. control cervical manipulation procedures with multiple chiropractors following an initial study (Vernon, et al., 2013). The investigators decided to use a BI for evaluation of blinding as the primary outcome, but found a lack of statistically sound recommendations in the blinding literature to inform the choice of the sample size (Vernon, 2017) https://clinicaltrials.gov/ct2/show/NCT01772966.

The growing interest in statistical analyses of blinding outcomes in recent years (Arandjelović, 2012; Baethge, et al., 2013; Crisp, 2015; Hertzberg, et al., 2008; Houweling, et al., 2014; Wright, et al., 2012) creates the need to address sample size calculations in order to improve the quality of inference on analysis of blinding data. The key question is how to determine the sample size for a new stand-alone study, such as a pilot study focusing on masking, evaluation of short-term blinding (Walter, et al., 2005), or part of a large phase III RCT, in which blinding assessment may be defined as a secondary aim.

In this article, we propose simple and intuitive ways to address this question. Since blinding studies, unlike evaluations of clinical effectiveness, do not have clear-cut aims, we present three different research scenarios and describe methodologies to obtain sample sizes in each case. The proposed methods can be integrated into traditional frameworks (e.g., estimation or statistical testing) with clear underlying mechanisms and operational characteristics, and the calculations are straightforward. Of note, we recommend the proposed methods for power calculations while designing a ‘new’ study, and that post-hoc power calculations be avoided (CONSORT, 2010; Hoenig and Heisey, 2001).

2. Methods: background, notation, and proposals

Studies focusing on treatment comparison or survey research generally have clear and well-defined goals, such as detection of treatment effect with high power, or ensuring the margin of error not exceeding a desired bound. Conversely, blinding assessment studies tend to have somewhat subjective or varied goals. Regardless of the divergent goals, blinding studies are driven by two questions at the design and analysis stages: 1) was the blinding broken? and 2) if so, are the outcome data affected? These two questions can be converted into statistically relevant scenarios reliant on the format of blinding assessment data. Typically, blinding data are presented as a 2×3 table with two allocation arms (Treatment (T=1) vs. Control (C=2)) and three choices for guess (1(=T), 2(=C), 3(=Don’t know)) as in Table 1. Some researchers use a 2×2 format without the ‘Don’t know’ option, while some others use a 2×5 format accounting for degree of belief (e.g., ‘strongly believe’, ‘somewhat believe’), which can be reduced to a 2×3 format (Bang, et al., 2004; James, et al., 1996; Mathieu, et al., 2014; Wright, et al., 2012).

Table 1.

Notations used to summarize typical blinding data

	Guess, count			Total
Allocation	1 (=T)	2 (=C)	3 (=Don’t know)	Sample size
Treatment (T)	$n_{11}$	$n_{12}$	$n_{13}$	$n_{1\mathrm{\cdot }}$ (= $n_{T\mathrm{\cdot }}$)
Control (C)	$n_{21}$	$n_{22}$	$n_{23}$	$n_{2\mathrm{\cdot }}$ (= $n_{C\mathrm{\cdot }})$

In this view, the three statistically relevant scenarios would be: 1) testing the independence of allocation and guess; 2) estimating arm-specific trinomial proportions of guess and their contrast; and 3) testing the effect of allocation-guess interaction on the clinical outcome. In the rest of this section, we derive the formulas for the sample size required for each of the three scenarios with adoption or adaptation of existing methods or ideas, followed by the Worked examples in the next section.

Let n_ij represent a cell count in a 2×3 table with blinding data (see Table 1; i=1,2 and j=1,2,3). The row sums, n₁. and n₂., define the sample sizes for each arm, and the total sample size is equal to N=n₁.+n₂. Unless specified differently, we assume 1:1 allocation (n₁.=n₂.=n and N=2*n) and α=0.05. Also, we denote joint and conditional probabilities as p_ij and p_j|i, respectively. Parameter and estimator notation may be used interchangeably when the context is clear, as it is a common practice in research on N/power.

2.1 Scenario 1: Testing the independence of allocation and guess

The first and most natural consideration for an investigator seeing data in a 2×3 table (as in Table 1) would be a classical test of the independence of allocation (row) and guess (column) or the homogeneity of the response in multiple groups (Agresti, 2013; Mathieu, et al., 2014). The Pearson Chi-square and Likelihood Ratio (LR) tests are standard tests for the association in an unordered rxc table, with r arms and c guesses.

Under the alternative hypothesis, Chi-square and LR statistics, commonly denoted as $X^2$ and $G^2$ in the literature, have large-sample noncentral chi-squared distributions. Let $p_{ij}$ denote the joint probability in cell (i,j), where i represents the allocation (i=1,2), j represents the guess (j=1,2,3), and ${\overline{p}}_{ij}$ denote the joint probability in cell (i,j) under the null (the independence hypothesis); $\sum p_{ij}=\sum {\overline{p}}_{ij}=1.$ Using this notation, the noncentrality parameter (λ) for Pearson Chi-square equals

$$\lambda =N{\displaystyle \sum }_{i,j}{\left(p_{ij}-{\overline{p}}_{ij}\right)}^2/{\overline{p}}_{ij},$$

and the noncentrality parameter for LR statistic equals

$$\lambda =2N{\displaystyle \sum }_{i,j}{p}_{ij}\text{log}\left(\frac{p_{ij}}{{\overline{p}}_{ij}}\right).$$

The desired sample size (N) for a chi-squared test can be obtained from the power equation $P\left(X_{\upsilon ,\lambda }^2>{\chi }_{\upsilon }^2\left(\alpha \right)\right)$, where $\upsilon =\left(r-1\right)\times \left(c-1\right)$ is the degrees of freedom (df). If the data from ‘previous’ studies are available and we want to detect the similar observed differences, $p_{ij}$ and ${\overline{p}}_{ij}$ can be estimated as n_ij/N and (n_i.*n._j)/N², respectively. Approximations can be done using published or built-in table, e.g., (Agresti, 2013); see the Worked Examples below.

2.2 Scenario 2: Estimation of (a) arm-specific trinomial proportions and (b) their contrast

Blinding data for a given arm $i$ can be viewed as a sample from a trinomial distribution with probabilities $\left(p_{1|i},p_{2|i},p_{3|i}\right)$, and ${\displaystyle \sum }_{j=1}^3{p}_{j|i}$=1. In this subsection, we consider determination of the sample size required in each arm (n) to ensure the estimation of these probabilities and/or their difference (namely, BI) with a specified level of precision.

First, let us consider the estimation of the probabilities. In this case, the objective is to find the smallest sample size $n$ for arm $i$such that the estimated proportions are simultaneously within specified distances of true population proportions with a probability of at least 1-α, that is,

$$P\left(\underset{j=1}{\overset{3}{{\displaystyle \mathrm{\cap }}}}\left|{\widehat{p}}_{j|i}-p_{j|i}\right|\le d_j\right)\ge 1-\alpha \text{for}\text{arm}i.$$

The sample size determination procedures proposed by Tortora (Tortora, 1978) and Thompson (Thompson, 1987) are based on the simultaneous confidence intervals (CIs) for the multinomial model. Tortora constrained the width of the jth category interval, j=1,2,3, to be ≤2 $d_j$ and obtained the following formula for n,

$$n=\underset{j=1,2,3}{\underbrace{\max }}\left[z_{\alpha /\left(2\times 3\right)}^2{p}_{j|i}\left(1-p_{j|i}\right)/{d_j}^2\right]$$

where $z_{\alpha /\left(2\times 3\right)}$ is the upper (α/6)*100th percentile of the standard normal distribution. The total sample size N can be obtained as N=2*n, where n is the larger value between the sample sizes obtained for each arm. In the absence of prior knowledge about $p_{j|i}$, the ‘worst-case’ (in view of the maximal n required) would be to use the probabilities vector (1/2,1/2,0) for each arm. Assuming $d_j=d$for each category j, the (maximal) n for each arm can be further simplified as

$$n=\frac{z_{\alpha /6}^2}{4d^2}=\frac{1.43279}{d^2}\text{for}\alpha =0.05.$$

In contrast, Thompson described the general form of the ‘worst’ parameter vector under the constraint of the equal width, $d_j=d$. His theoretical result depends on the specified level of $\alpha$: for example, (1/3,1/3,1/3) is the ‘worst-case’ parameter vector if $\alpha$=0.05, whereas (1/2,1/2,0) is the ‘worst-case’ parameter vector if $\alpha$=0.025; see Table 1 in (Thompson, 1987). His procedure results in smaller sample sizes and hence is attractive under the condition of the equal width intervals. The conservative sample size using the Thompson’s method is given by

$$n=\frac{2\mathrm{\cdot }{z}_{\alpha /6}^2}{9d^2}=\frac{1.27359}{d^2}\text{for}\alpha =0.05$$ Eq (1)

which is lower than the Tortora’s counterpart. Both methods would result in sample size estimates that ensure a specified confidence range for the estimated probabilities.

Next, we consider the estimation of the BI. Bl is an arm-specific index for blinding assessment defined as a contrast between the probability of correct guess and the probability of incorrect guess:

$${\text{BI}}_{\mathrm{T}}={\mathrm{p}}_{1|1}-{\mathrm{p}}_{2|1}{\text{and BI}}_{\mathrm{C}}={\mathrm{p}}_{2|2}-{\mathrm{p}}_{1|2}.$$

In a 2×2 format without the ‘Don’t know’ option, the BI reduces to ${\text{BI}}_{\mathrm{i}}=2{\mathrm{p}}_{\mathrm{i}}-1$ for $i=T,C$, where $p_T=p_{1|1}$ and $p_C=p_{2|2}$. Estimators for the BIs can be obtained by replacing the arm-specific probabilities by their estimators:

$${\widehat{\text{BI}}}_{\mathrm{T}}=\left(n_{1T}-n_{2T}\right)/n_T\mathit{and}{\widehat{\text{BI}}}_{\mathrm{C}}=\left(n_{2c}-n_{1c}\right)/n_c.$$

BI quantifies the correct guess beyond chance (50%) or imbalance between correct vs. incorrect guesses in the blinding data as in Table 1. For example, BI=0 represents ‘random guess’; BI=0.3 represents a 30% imbalance in guess toward the correct guess, say, 40% participants guessed T vs. 10% guessed C in arm T; BI=−0.3 represents an imbalance of 30% toward the incorrect guess. BI is usually interpreted in terms of possible blinding scenarios, along with qualitative data (e.g., reasons for guess), whenever available. After all, there are contexts in which correct guesses are not undesirable, and may provide insight, such as when they reflect ‘wishful thinking’ (Bang, 2016; Brinjikji, et al., 2010).

A standard 2-sided CI for BI with (1-α) confidence level for the treatment arm $i$is:

$$({\widehat{p}}_{1|i}-{\widehat{p}}_{2|i})\pm z_{\alpha /2}^{}\mathrm{\cdot }\sqrt{\left[{\widehat{p}}_{1|i}\left(1-{\widehat{p}}_{1|i}\right)+{\widehat{p}}_{2|i}\left(1-{\widehat{p}}_{2|i}\right)+2{\widehat{p}}_{1|i}{\widehat{p}}_{2|i}\right]/n_i}.$$

Assuming 1:1 allocation as before, the objective is to find a sample size $n$ such that the inequality $z_{\alpha /2}^{}\mathrm{\cdot }\sqrt{{\widehat{[p}}_{1|i}\left(1-{\widehat{p}}_{1|i}\right)+{\widehat{p}}_{2|i}\left(1-{\widehat{p}}_{2|i}\right)+2{\widehat{p}}_{1|i}{\widehat{p}}_{2|i}]/n}$ $\le d$ holds for a specified threshold d. Solving this inequality in terms of n yields

$$n=z_{\alpha /2}^2\left[{\widehat{p}}_{1|i}\left(1-{\widehat{p}}_{1|i}\right)+{\widehat{p}}_{2|i}\left(1-{\widehat{p}}_{2|i}\right)+2{\widehat{p}}_{1|i}{\widehat{p}}_{2|i}\right]/d^2.$$

Using the method of Lagrange multipliers, we can show that the maximal sample size is reached when p_3|i=0 and p_1|i=p_2|i=(1-p_3|i)/2. In this case, the maximal sample size is attained at the trinomial vector (p_1|i=p_2|i=p_3|i)=(1/2, 1/2, 0) simplifying the above formula to n=z²_α/2/d². This result has two important implications: a) allowing ‘Don’t know’ as a guess category decreases the required sample size; and b) in a 2×2 format, the maximal value of Var( ${\widehat{BI}}_i$)= 4*Var( ${\widehat{p}}_i)$ is reached at the binomial vector (1/2, 1/2), resulting in the maximum sample size equal to n=z²_α/2/d². Interestingly, this sample size is 4 times of the well-known conservative sample size for estimation of the probability of event in binomial data.

To summarize, with good estimates for trinomial or binomial probabilities, the sample size for each arm is given as n=z²_α/2[p_1|i(1-p_1|i)+p_2|i(1-p_2|i)+2p_1|ip_2|i)]/d² for a 2×3 format, and n = z²_α/2[4p_i(1-p_i)]/d² for a 2×2 format. In the absence of good estimates, the conservative sample size

$$n={z^2}_{\alpha /2}/d^2$$ Eq (2)

with p=1/2 can be used for both formats. It can be seen from these formulae that a fundamental operational characteristic, common in virtually all sample size estimations, applies here as well: the more stringent the threshold (or the narrower the CI), the larger the sample size required.

2.3 Scenario 3: Testing the effect of allocation-guess interaction on the clinical outcome

Breached blinding has a potential to distort clinical findings, leading to biased estimates of treatment effects with unknown direction (Bang, 2016; Mathieu, et al., 2014). In the presence of allocation-guess interaction, the estimated average treatment effect (ATE) may depend on the guess status (1(=T), 2(=C), 3(=Don’t know)) resulting in meaningfully different ATEs in subgroups of different guess status. For instance, the ATE estimate obtained from those who guessed T can be positive, in favor of treatment, while the estimate from those who guessed C can be negative or null. In these situations, detecting the interaction between allocation and guess with a reasonable power could be of scientific interest.

Testing the effect of allocation-guess interaction on a (continuous) clinical outcome may be understood in a framework of a two-way fixed effects unbalanced ANOVA. Let $y_{\mathit{ijk}}$ be the outcome from the kth patient in arm i with guess j, where i=1,2; j=1,2,3; and k=1,…,n_ij, and the cell sizes n_ij (non-zero) are defined in Table 1. In a 2×3 case, a univariate general linear model can be parametrized as

$$y_{\mathit{ijk}}={\mu }_{ij}+{\epsilon }_{\mathit{ijk}},i=1,2;j=1,2,3$$

where ${\epsilon }_{\mathit{ijk}}$are normally distributed independent and identical errors with mean zero and variance ${\sigma }^2$. This model can be written in matrix notation as $y=XB+\epsilon$, where $X$ is a design matrix of size $N\times m$of zeros and ones (m=6 in our case), and $B=\left[\begin{array}{ccc}{\mu }_{11}& {\mu }_{12}& {\mu }_{13}\end{array}\begin{array}{ccc}{\mu }_{21}& {\mu }_{22}& {\mu }_{23}\end{array}\right]$ (Elston and Bush, 1964).

Testing the null hypothesis of no interaction between allocation and guess is equivalent to testing the equality of ATE for each category of guess. This hypothesis can be viewed as a special case of a linear hypothesis H₀: LB=0 vs. H_A: LB≠0, with L being a $q\times m$ contrast matrix of full rank, where q is the number of contrasts. For example, for an overall test of no interaction effect, $L=\left[\begin{array}{cc}1& 0\\ 0& 1\end{array}\begin{array}{cc}-1& -1\\ -1& 0\end{array}\begin{array}{cc}0& 1\\ -1& 1\end{array}\right]$. The F-statistic is given by

$$F=\frac{\left(L\widehat{\mathrm{B}}\right)\prime {(L{(X^{\prime }X)}^{-1}L\prime )}^{-1}\left(L\widehat{B}\right)/q}{e\prime e/\left(N-m\right)}$$

with $\widehat{B}={(X^{\prime }X)}^{-1}X\prime y$ and $e^{\prime }e=\left(y-X\widehat{B}\right)\prime \left(y-X\widehat{B}\right)$. Under H_A, F is distributed as F(q, N-m, λ) with a noncentrality parameter $\lambda =\left(LB\right)\prime {(L{(X\prime X)}^{-1}L\prime )}^{-1}\left(LB\right)/{\sigma }^2$. The sample size is computed by inverting the power equation P(F(q, N-m, λ)≥F_α(q, N-m)); see (Castelloe and O’Brien, 2001; Elston and Bush, 1964; Muller and Peterson, 1984; O’Brien, 1986; O’Brien and Shieh, 1992) for details and general theory.

In summary, use of this method requires outcome data for allocation by guess status from historical studies, preliminary data, or pre-specified values (Chow and Shao, 2004; Wright, et al., 2012). Thus, the utility of this method may be limited in practice when it is difficult to come up with plausible inputs, although it may be of ultimate interest related to blinding assessment. See Section 3.3 for an illustrative example from Chow and Shao, where suspected breached blinding might have been problematic and warranted further investigation.

3. Worked examples

Even with technically sound methods, a crucial issue is how to implement the methods and make sensible decisions in practice. In this section, we illustrate sample size calculations using inputs from published data to assist in designing a ‘new’ study.

If a new study is designed as a pilot study where blinding is the primary outcome, trialists may use our N methods directly (e.g., a wide margin can be predefined for a pilot, and a narrow margin for a real study). If blinding is defined a secondary or tertiary outcome in a ‘new’ study, trialists may choose N for primary clinical outcome, which is typically done in RCTs (Briggs, 2000). If N was obtained for a primary clinical outcome, we can calculate what power or precision of the estimate of the blinding parameter can be attained with this N. Toward making an overall conclusion, we adopt Cohen’s logic (Cohen, 1990): determine the sample size necessary to detect a negligible signal/indication of breached blinding with high probability. After the research is carried out using that sample size, and the result is not significant, the conclusion is justified that no nontrivial signal exists, at a given level. This does, in fact, probabilistically show the intended null hypothesis of no more than a trivial small signal (i.e., blinding is acceptable).

3.1. Scenario 1: Testing the independence of allocation and guess

The sample size under this scenario can be obtained manually or using SAS macros powerRxC or Unifypow (SAS Institute, Cary, NC), among others (Castelloe and O’Brien, 2001; O’Brien and Shieh, 1992). As an illustrative example of a manual calculation, the real (rather than hypothetical) data from a study for blinding assessment of a sham cervical manipulation procedure (Vernon, et al., 2013) is used as an input for calculation of a sample size for a new study. Rigorous blinding evaluation was chosen as a primary aim of the new study (Vernon, 2017). The self-reported guess status collected from a secondary analysis in an earlier study is presented in Table 2 and it may be used to obtain the observed joint probabilities in a 2×3 table: 0.25, 0.14, 0.11 for arm T and 0.14, 0.23, 0.125 for arm C for guess 1,2,3, respectively. The expected joint probabilities under independence are: 0.195, 0.1875, 0.115 for guess of 1,2,3 in both arms. Using this information, the noncentrality parameter for Pearson Chi-square equals $\lambda$=0.054*N. The approximate power for a number of different N values can be obtained from the table ‘Power of Chi-squared Test for $\alpha$=0.05’ in (Agresti, 2013) or various statistical software: if N=20, $\lambda =1.08$, the power is approximately 0.13 (with df=2). Similar calculations show that N=176 will be required to test the same hypothesis with 80% power. In addition, the noncentrality parameter for the LR test equals $\lambda$=0.055*N, which is very similar to a Pearson’s as expected. Sample SAS codes are provided in Appendix. Of note, resulting estimates can be unstable or unreliable with low cell counts.

Table 2.

Blinding data from Vernon et al. (2013) and Sample size/Power

a. Blinding data
	Guess			Total
Allocation	Real	Sham	Don’t know	Sample size
Real	16	9	7	32
Sham	9	15	8	32

b. Sample size and power ( $\alpha =0.05$)
N	Power of Pearson Chi-square	Power of LR
20	0.14	0.14
50	0.30	0.30
100	0.56	0.55
176	0.80	0.81
200	0.85	0.86
300	0.96	0.96

3.2. Scenario 2: Estimation of (a) arm-specific trinomial proportions and (b) their contrast

Proportions obtained from previous studies can be used to inform sample size calculations. For example, Vernon et al. (Vernon, et al., 2013) observed the trinomial proportions (0.5, 0.28, 0.22) in arm T and (0.28, 0.47, 0.25) in arm C in an immediate post-treatment evaluation of blinding. Park et al.’s acupuncture trial (Park, et al., 2005) also observed the proportions of guess near 0.5 (e.g., 26/49). In the absence of prior data on trinomial proportions or with the observed proportions close to 0.5, as in the trials above, the conservative sample size for one arm can be obtained using $n=\frac{z_{\alpha /6}^2}{4d^2}$. For example, with α=0.05, the sample size equals n=574, 144, 36 for d=0.05, 0.1, 0.2, respectively. The sample sizes corresponding to the Thompson’s method equal n=510, 128, 32. Since in the blinding context, p_1|i and p_2|i near 0.5 are quite plausible (whereas extreme scenarios such as (0;0;1) are rare), these conservative sample sizes are reasonably justified.

Next, we discuss the estimation of BI, the contrast of the arm-specific proportions. Recent systematic reviews and meta-analyses on blinding provide a new insight on the range of feasible values of the BI in different types of studies (Baethge, et al., 2013; Freed, et al., 2014; Moroz, et al., 2013). For example, Freed et al. (Freed, et al., 2014) focused on meta-analysis of BI in trials of psychiatric disorders. It is remarkable that a large number of studies included in Freed et al. produced the BI (in weighted average) close to 0 in the control arm, where BI=0 corresponds to a ‘random guess’, supposedly the most ideal blinding scenario (Bang, 2016). Therefore, in the absence of good input about a possible value of BI in a future study, BI=0 could be a reasonable starting point for sample size calculation.

For a blinding data collected in a 2×3 format, BI=0 is implied by all parameter vectors of the form of $\left(p_{1|i},p_{1|i},1-2p_{1|i}\right)$, where $0\le p_{1|i}\le 1/2$. Table 3 presents sample calculations for this case obtained for two thresholds (d=0.1 and d=0.2) for $p_{1|i}=0.1,0.2,0.3,0,4,0.5$. The results in the table clearly demonstrate that as $p_{1|i}$ values get closer to zero (implying that more people are expected to answer ‘Don’t know’), the required sample size decreases. When $p_{1|i}$ value gets closer to 0.5, a larger sample size is required. The ‘worst-case’ corresponds to $p_{1|i}=0.5$, in which case 2×2 and 2×3 formats are equivalent and the required sample size reaches its maximum value. All confirm our theoretical results in Section 2.2. Notice a 5-fold difference in sample size required for the case (0.1, 0.1, 0.8) as opposed to the case (0.5, 0.5, 0) for the same value of α and d. The tighter the desired width of the CI (=2*d), the larger the sample size required, e.g., 4 times for d=0.1 vs. 0.2.

Table 3.

Sample size (n_T.) for the estimation of BI in treatment arm ( $\alpha =0.05$)

BIT = P1|T-P2|T	d=0.2	d=0.1
BIT = 0
0.1-0.1	20	77
0.2-0.2	39	154
0.3-0.3	58	231
0.4-0.4	77	308
0.5-0.5*	97	385
BIT = 0.1
0.1-0.0	9	35
0.2-0.1	28	115
BIT = 0.1, 0.2, 0.3, 0.4
0.3-0.2/0.3-0.1	48/35	189/139
0.4-0.3/0.4-0.2/0.4-0.1	67/54/40	266/216/158
0.5-0.4/0.5-0.3/0.5-0.2/0.5-0.1	86/73/59/43	342/292/235/170

P_1|T=expected proportion of persons who guessed 1 (=T).

P_2|T=expected proportion of persons who guessed 2 (=C).

BI_T denotes blinding index and n_T. denotes sample size in Treatment arm.

^*The required sample size in this case is equivalent to a conservative sample size in a 2×2 format (without “Don’t Know” category), n=z²_α/2/d².

As an illustration, when a future study on blinding assessment of cervical manipulation is designed after a pilot study (Vernon, et al., 2013), and the team anticipates BI values to be in the range [0, 0.1], the required sample size per arm would be 86 (assuming parameter vector of (0.5, 0.4, 0.1)) or 97 (assuming (0.5, 0.5, 0)) with d=0.2. These sample sizes correspond to N=2*86=172 or N=2*97=194 which is comparable to the N=176 obtained under Scenario 1.

As noted above, our methods can also be used to assess the power or estimation precision for blinding (defined as a secondary or tertiary outcome) in the studies with the sample size obtained for clinical outcome. To exemplify the use of our methods for this common scenario, assume that a research team is planning a new trial to estimate the effect of real vs. sham acupuncture on muscle spasticity as primary outcome. The Ashworth scale for muscle spasticity is defined as a dichotomized outcome (yes/no increase in muscle tone). The team is informed by a previous trial (Park, et al., 2005) and wants to detect a clinically meaningful difference of 33% vs. 22% for real vs. sham acupuncture, respectively. The sample size n=276 for each arm is required to detect this difference with α=5% and 80% power with a 1:1 allocation via the Fisher exact test. If we decide to use the Thompson’s formula for a trinomial vector and a conservative formula for BI, Equations (1) and (2), we get d=0.07 and 0.12, respectively, with the given n. Thus, we may expect that the total sample size N=552 is sufficient to make the goal of reliable blinding assessment achievable in the planning stage.

3.3. Scenario 3: Testing the effect of allocation-guess interaction on the clinical outcome

Chow and Shao (Chow and Shao, 2004) analyzed the Brownell and Stunkard’s data (Brownell and Stunkard, 1982) focusing on breached blinding, the role of consent form, and its potential impact on the clinical outcome in the weight loss trial. The observed blinding data (Table 4) yielded substantial agreement between the allocation and guess and very high BI values (BI_T=0.67 and BI_C=0.52; markedly larger than the previously suggested threshold, 0.2 (Freed, et al., 2014; Kolahi, et al., 2009; Park, et al., 2008)), which may be indicative of possible breach in blinding. Moreover, dramatic interaction between allocation and guess on the clinical outcome of weight loss is apparent in Figure 1, created using the summary statistics provided in the original papers (Brownell and Stunkard, 1982; Chow and Shao, 2004). These summary statistics may be used to reproduce the raw outcome data closely.

Table 4.

Blinding and outcome data from Brownell and Stunkard (1982) and Sample size/Power

a. Blinding data
	Guess, count			Total
Allocation	Active drug	Placebo	Don’t know	Sample size
Active drug	19	3	2	24
Placebo	3	16	6	25

b. Mean weight loss (kg) in subgroups defined by allocation and guess
	Guess
Assignment	Active drug	Placebo	Don’t know	Overall
Active drug	9.6	3.9	12.2	9.1
Placebo	2.6	6.1	5.8	5.6

c. Sample size and power ( $\alpha =0.05$)
Source	$\sigma$	N	Power
Main effect of guess (df=2)	4	148	0.860
	4	98	0.682
	5	148	0.668
	5	98	0.481
Overall interaction (df=2)	4	148	0.994
	4	98	0.948
	5	148	0.942
	5	98	0.806
Tailored interaction (df=1) ATE among guess=T vs. ATE among guess=C	4	148	0.994
	4	98	0.954
	5	148	0.947
	5	98	0.829

df: degrees of freedom; ATE: average treatment effect.

Figure 1. Clinical outcome by allocation and guess status.

Created using data from Brownell and Stunkard (1982) and Chow and Shao (2004).

drug: Guess T; placebo: Guess C; dnk: Guess ‘Don’t know’.

In this and similar situations, we would be interested in the sample size required to test the overall interaction (i.e., the equality of ATEs for all guess categories) as well as the custom interaction (e.g., the equality of ATEs between those who guessed T and those who guessed C) with sufficient power in a new study. The method for sample size determination described in Section 2.3 can be implemented using GLMpower procedure in SAS; see Appendix for a sample code. Cell means, cell counts and error variance ${\sigma }^2$are the sample required inputs. An estimate of the error variance may be obtained by fitting a two-way ANOVA model to the re-constructed raw data or crudely approximated as $\sigma$≈ ( $y_{\mathit{max}}-y_{\mathit{min}}$)/6, where $y_{\mathit{max}}$ and $y_{\mathit{min}}$ stand for maximal and minimal values of a reasonably symmetric outcome. Using the re-constructed raw data, we obtained $\approx 5$, thus confirming the previous estimation (Chow and Shao, 2004). We ran an additional power analysis with $\sigma \approx \left(4-\left(-20\right)\right)/6=4$ using information provided in raw data plot (Brownell and Stunkard, 1982) for an illustration purpose.

Again, let us assume that we are planning to design a new trial, with the primary outcome defined as weight loss and blinding being selected as one of the secondary outcomes. We hypothesize in this case that the clinically meaningful difference in weight loss between treatment and placebo is about 3.5 kg with a conservative value σ=5. Assuming a 1:1 allocation, α=1% and power=95% (these strict conditions are assumed to minimize false positive results and ensure very high power), n=74 for each arm (total N=148) will be required to test the difference in means.

Using the results in Table 4, we can conclude that this sample size will be sufficient to detect overall and custom interactions defined above for blinding with high power (>90%) at α=5%. Let us now consider another, conservative situation with sample size N=98. This sample size is ~30% lower than N=148 but is double the original sample size (N=49) used in Brownell and Stunkard (1982). Using this sample size with the standard assumptions of α=5% and power=80%, we will be able to detect a true difference of <3 kg in treatment vs. control groups. Using the results in Table 4, we can now conclude that a sample size in the range of 100-150 would be sufficient to detect interactions of interest defined above with power >80% assuming $\sigma \approx 5$ and α=5%.

On the other side, the power to detect the difference in ATE among those who guessed T vs. those who answered ‘Don’t know’ is very low (~5%), which is expected as the associated ATE lines have nearly the same slopes in Figure 1. At the same time, we might assume that those who chose ‘Don’t know’ tend to be neutral or less biased. A similar analysis can be repeated with a different classification for guess status: guess correctly vs. guess incorrectly vs. not guess, as attempted by Chow and Shao.

This exercise demonstrates the feasibility of sample size and power calculations in the setting where we design a new study and reasonable blinding data along with the outcome data are available for inputs as educated guess. Although this type of data is rare in the present literature, we believe that growing research on blinding will yield more collection and reporting of similar data.

4. Discussion

In this paper, we consider the three qualitatively different scenarios relevant to quantitative analysis of blinding, and present methods of sample size determination for planning a future study with a blinding assessment component. The scenarios are framed in terms of estimation and precision (Scenario 2a & b)) as well as statistical testing and power (Scenario 1 and 3). We illustrated the three ─ real and hypothetical ─ scenarios with Worked examples using published data and exemplified the calculations manually or using a statistical software.

The methods and examples in this paper offer users a suite of formulae that can be used to determine sample sizes in order to conduct a spectrum of studies, from a pilot or feasibility study (Walter, et al., 2005) to a full scale RCT. Since power and sample size calculation should be tailored to a specific research question, the choice of the particular method and formula depends on the goals and input availability. The proposed methods could be particularly essential for studies testing and establishing a newly developed control, sham or placebo intervention. Blinding assessment is also desirable for studies that test if the two treatments are easily distinguishable and there are no clinical outcomes (say, masking).

The proposed methods can be implemented in a flexible manner for different purposes. For example, when designing a pilot study, researchers may decide to use the formulae in the paper with a wider margin of error than that which is suitable for a real trial: for example, 0.2 can be set as a targeted threshold for estimation of the BI in a pilot study, and 0.1 in a larger, actual trial. If a research team decides to evaluate blinding in a very large trial, the sample size formulae may be used to create a subsample from the entire sample, e.g., (COMMIT, 2005), to which a blinding questionnaire could be administered. Similarly to power/sample size analysis relevant to a clinical outcome, if the sample size used in blinding assessment is substantially lower than the ones justifiable by the methods in our paper, the designation of “pilot or feasibility” study may be reasonable.

We want to discuss some limitations. First, we considered studies with a 1:1 allocation and 2 arms since blinding data is optimally interpreted in this setting. With a 2:1 allocation, we are not fully certain if P(correct guess)=0.5 is still the most ideal value; some patients may use 50% in their guess (between T vs. C), while others may use 66.7% if they were informed about the allocation ratio and remember it (Bang, 2016; Brownell and Stunkard, 1982). N/power calculations for advanced designs (e.g., with 2:1 allocation, clustering or crossover designs) as well as accounting for other complex issues (e.g., informative drop-out) in RCT are possible topics for future research (Bang, et al., 2010; Park, et al., 2005; Roy, 2012; Zhang, et al., 2013).

Second, we did not handle multiple testing rigorously, partly because we do not pursue rigorous hypothesis testing in blinding. Of note, we framed the problem in classical superiority testing with a 2-sided CI/test, not equivalence test or a 1-sided hypothesis/CI, which may be more relevant to blinding. The main reason behind our decision was to avoid unnecessary complexity (e.g., equivalence margin), because blinding is a tool, not a goal, and any numerical analysis alone should not be used for binary designation (e.g., success or failure) (Bang and Park, 2013; Zhang, et al., 2013). Along the same line, we emphasize the importance of the estimation-based method described in Scenario 2 in the blinding context. Finally, post-hoc or retrospective power calculation should be avoided (CONSORT, 2010; Hoenig and Heisey, 2001).

In closing, we propose the methods for sample size and power calculations which can be used for exploratory or planning purposes, and can address different research questions, inputs and settings commonly encountered in blinding.

Appendix: Sample SAS codes

1. Scenario 1

data Vernon;
 do Treat=1 to 2;
  do Guess=1 to 3;
   input freq @@; output;
  end;
 end;
datalines;
16 9 7
9 15 8
;
%powerRxC(data=Vernon,row=Treat,col=Guess,count=freq,nrange=%str(20,50,100,176,200 to 500 by 100))

2. Scenario 3

data Chow;
input Treat Guess Outcome Weight;
datalines;
1 1 9.6 19
1 2 3.9 3
1 3 12.2 2
2 1 2.6 3
2 2 6.1 16
2 3 5.8 6
;
proc glmpower data=Chow;
class Treat Guess;
model Outcome = Treat | Guess;
weight Weight;
contrast ‘Inter-overall’ Treat*Guess 1 0 −1 −1 0 1, Treat*Guess 0 1 −1 0 −1 1;
contrast ‘Inter-tailored (ATE in 1(=T) vs. 2(=C))’ Treat*Guess 1 −1 0 −1 1 0;
power
stddev = 4 5
ntotal = 148 98
power = .;
run;