Optimal Design and Purposeful Sampling: Complementary Methodologies for Implementation Research

Abstract

Optimal design has been an under-utilized methodology. However, it has significant real-world applications, particularly in mixed methods implementation research. We review the concept and demonstrate how it can be used to assess the sensitivity of design decisions and balance competing needs. For observational studies, this methodology enables selection of the most informative study units. For experimental studies, it entails selecting and assigning study units to intervention conditions in the most informative manner. We blend optimal design methods with purposeful sampling to show how these two concepts balance competing needs when there are multiple study aims, a common situation in implementation research.

Introduction

The theory of optimal design is a valuable statistical methodology for designing a wide variety of field studies to optimize their statistical efficiency (see, e.g., Bellhouse 1984; Berger and Wong 2005; Bhaumik and Whittinghill 1991; Bhaumik 1993, 1995; Goos and Jones 2011; Heiberger, Bhaumik, and Holland 1993; Kiefer 1959; Liski et al. 2002; Raudenbush 1997; Raudenbush and Liu 2000; Raudenbush et al. 2011; Shah and Sinha 1989; Spybrook et al. 2011). An optimal design maximizes the information yield for the study, taking into account the budget constraint limiting the resources available for the study. For observational studies, the concept of optimal design can be applied to select study units (organizations, providers, or patients) in the most informative manner. For experimental studies, the same concept can also be applied to assign study units to intervention conditions in the most informative manner.

The application of optimal design is usually based on a pre-specified statistical model (such as a regression model) that is believed to represent the phenomenon being studied. The aims for the study are usually represented as parameters in the pre-specified model, such as the slope parameter in the regression model. The information yield is usually measured using a pre-specified target criterion that measures the level of statistical information available for a candidate design, such as the reciprocal for the variance for the estimated slope coefficient for the regression model. This article provides an overview of the potential application of this methodology to facilitate the design of implementation research studies.

There is a close similarity between optimal design and purposeful sampling, a design methodology widely used in qualitative research (Palinkas et al. 2013). There is an important opportunity in mixed methods implementation research to integrate these methodologies for broader applications.

The essence of purposeful sampling is to select information-rich cases for the most effective use of limited resources (Palinkas et al. 2013, Patton 2002). One type of example is the selection of extreme or deviant (outlier) cases for the purpose of learning from unusual manifestation of phenomenon of interest, and the selection of cases with maximum variation for the purpose of documenting unique or diverse variations that have emerged in adapting to different conditions. This approach can be used to identify important common patterns that cut across variations. This type of extreme selection will increase the between-unit variance. Another type of example is the selection of homogeneous cases for the purpose of reducing variation, simplifying analysis, and facilitating group interviewing. This homogeneity is nested within the units chosen in the first type of selection and hence it will reduce the within-unit variability. So between-unit variability is increased by diversification but within-unit variability is reduced by selection of homogeneous cases. Therefore, the reduction of within-unit variability compensates the between-unit variability. As a result, the total variability may be reduced even further after capturing the extreme units. We will discuss below scenarios in which the theory of optimal design leads to designs similar to those resulting from purposeful sampling.

We review the principles of optimal design in the next section, and provide illustrative examples of studies that aim to assess a single parameter. We then review the theory of optimal design for studies with multiple aims, and therefore multiple parameters of interest, to illustrate the important role for optimal design to balance competing study aims. The final section discusses specific issues for mixed methods implementation research studies.

Optimal Design for Single-Aimed Studies

In this section, we review the principles of optimal design, illustrated with several hypothetical implementation studies that aim to assess a single parameter.

Comparison between Two Implementation Strategies

Consider an implementation study that compares two implementation strategies applied to a group of agencies that provides mental health services to families of children with emotional or behavioral disorders. We assume that agencies are recruited and assigned randomly to the two implementations strategies. We assume further that all clinics are similar with respect to factors such as organizational structure, culture and climate, availability of funding and resources, and characteristics of clients. Therefore, we assume a simple two-sample model for the outcomes,

$${\mathrm{Y}}_{\text{ij}}={\mathrm{\mu }}_{\mathrm{i}}+{\mathrm{\varepsilon }}_{\text{ij}},{\mathrm{\varepsilon }}_{\text{ij}}\sim \mathrm{N}(0,{\mathrm{\sigma }}^2);\mathrm{i}=0,1;\mathrm{j}=1,\dots ,{\mathrm{n}}_{\mathrm{i}}$$ (1)

The index i denotes the two implementation strategies: i=0 denotes the standard implementation strategy, i=1 denotes the novel implementation strategy being evaluated. For strategy i, the index j denotes the agencies assigned to this strategy; n_i denotes the number of agencies assigned to strategy i. We do not make the usual assumption of equal sample size between the two strategies, for reasons to be discussed later. The outcome for the j-th agency assigned to strategy i is denoted as Y_ij; the expected outcome for an agency assigned to strategy i is denoted as μ_i. Furthermore, the random disturbance for the j-th agency assigned to strategy i is denoted as ε_ij; the disturbance terms are assumed to be statistically independent and follow a normal distribution with common variance σ².

The primary aim for the study is to estimate the implementation effect, i.e., the expected difference between the two implementation strategies

$$\mathrm{\delta }={\mathrm{\mu }}_1-{\mathrm{\mu }}_0.$$ (2)

We assume for now that the cost to recruit an agency into the study and obtain the outcome measure is constant across agencies and does not depend on the specific strategy the agency is assigned to. We denote the constant cost per agency as C, the total budget available as B. The total number of agencies the study can afford, n_total, is therefore

$${\mathrm{n}}_{\text{total}}=\lfloor \mathrm{B}/\mathrm{C}\rfloor ,$$ (3)

where ⌊B/C⌋ denotes the largest integer smaller than or equal to the fraction B/C, a.k.a. the floor function for B/C. For example, if B = $40,000, and C = $3,000, then B/C = 133.333…, and ⌊B/C⌋ = 133.

Under these assumptions, it is straightforward to obtain an unbiased estimate for the primary parameter of interest, the implementation effect parameter δ in Equation (2), by taking the sampling mean m₁ for the outcomes obtained from agencies assigned to strategy 1, and the sample mean m₀ for the outcomes obtained from agencies assigned to strategy 0, then taking the difference between the two sample means:

$$\mathrm{d}={\mathrm{m}}_1-{\mathrm{m}}_0.$$ (4)

With the availability of the unbiased estimate d in Equation (4), it is reasonable to specify the target criterion (TC) to be maximized for the optimal design as the reciprocal of the variance for the unbiased estimate d:

$$\text{TC}=1/\text{Var}(\mathrm{d})=1/[{\mathrm{\sigma }}^2\times ((1/{\mathrm{n}}_1)+(1/{\mathrm{n}}_0))],$$ (5)

where n₁ and n₀ denote the number of agencies assigned to strategy 1 and 0, respectively. More specifically, the optimal design task for this study is to determine n₁ and n₀ that maximize the target criterion TC in Equation (5) under the budget constraint

$${\mathrm{n}}_1+{\mathrm{n}}_0={\mathrm{n}}_{\text{total}}=\lfloor \mathrm{B}/\mathrm{C}\rfloor .$$ (6)

It can be shown that the optimal sample allocation is given as follows:

$${\mathrm{n}}_1={\mathrm{n}}_0=\lfloor \mathrm{B}/\mathrm{C}\rfloor /2\text{if}\lfloor \mathrm{B}/\mathrm{C}\rfloor \text{is an even number};$$ (7a)

$$\{{\mathrm{n}}_1,{\mathrm{n}}_0\}=\{(\lfloor \mathrm{B}/\mathrm{C}\rfloor -1)/2,(\lfloor \mathrm{B}/\mathrm{C}\rfloor +1)/2\}\text{if}\lfloor \mathrm{B}/\mathrm{C}\rfloor \text{is an odd number}.$$ (7b)

In Equation (7b), either n₁ or n₀ can take either of the two values shown on the right hand side of the equation. For example, if ⌊B/C⌋ = 133, then we can either take n₁=61 and n₀=62, or take n1=62 and n₀=61. To avoid unnecessary technical complexity, we simplify Equations (7a-7b) as follows:

$${\mathrm{n}}_1\cong (\mathrm{B}/\mathrm{C})/2,\text{and}$$ (7c)

$${\mathrm{n}}_0\cong (\mathrm{B}/\mathrm{C})/2,$$ (7d)

with the approximate equalities in Equations (7c-7d) indicating the possibility of rounding when (B/C) / 2 is not an integer.

The optimal design shown in Equations (7a-7b/7c-7d) validates the usual design of equal allocation to the two implementation strategies, with the minor deviation in (7b) when the total number of agencies allowed under the budget constraint is an odd number.

However, the assumptions that lead to the optimal design (7a-7b/7c-7d) might be inappropriate for some implementation studies. Deviations from these assumptions might lead to alternative optimal designs. We discuss several such variations in the subsections below.

Unequal cost

The cost per agency might differ between the two implementation strategies, with a higher cost C₁ required for the novel strategy than the cost C₀ for the standard strategy. Under this assumption, a smaller share of agencies should be allocated to the novel strategy due to its higher cost (and lower “bang per buck”).

For this scenario, the budget constraint is given as follows:

$${\mathrm{C}}_1\times {\mathrm{n}}_1+{\mathrm{C}}_0\times {\mathrm{n}}_0\le \mathrm{B}.$$ (8)

The optimal design task is to determine n₁ and n₀ that maximize the target criterion TC in Equation (5) under the budget constraint (8).

It can be shown that the optimal sample allocation is given as follows:

$${\mathrm{n}}_1\cong \mathrm{B}/[\text{sqrt}({\mathrm{C}}_1)\times \mathrm{\theta }],\text{and}$$ (9a)

$${\mathrm{n}}_0\cong \mathrm{B}/[\text{sqrt}({\mathrm{C}}_0)\times \mathrm{\theta }],$$ (9b)

where θ = sqrt(C₁) + sqrt(C₀). Equations (9a-9b) are approximate due to rounding when the right hand sides of these equations are not exact integers.

The optimal sample sizes in Equations (9a-9b), n₁ and n₀, are (approximately) inversely proportional to the ratio of the square root of the costs:

$${\mathrm{n}}_1/{\mathrm{n}}_0\cong \text{sqrt}({\mathrm{C}}_0)/\text{sqrt}({\mathrm{C}}_1).$$ (10)

Assume hypothetically that the cost per agency is higher for the novel strategy than the standard strategy by a factor of four,

$${\mathrm{C}}_1:{\mathrm{C}}_0=4:1.$$ (11)

Following Equation (10), the optimal sample size for the novel strategy is lower (approximately) by a factor of two (the square root of four) compared to the optimal sample size for the standard strategy:

$${\mathrm{n}}_1:{\mathrm{n}}_0\cong 1:2.$$ (12)

Unequal residual variance

The residual variance σ² might be lower for the novel strategy due to its provisions to address agency level idiosyncrasies, leading to an expanded model,

$${\mathrm{Y}}_{\text{ij}}={\mathrm{\mu }}_{\mathrm{i}}+{\varepsilon }_{\text{ij}},{\varepsilon }_{\text{ij}}\sim \mathrm{N}(0,{{\mathrm{\sigma }}_{\mathrm{i}}}^2);\mathrm{i}=0,1;\mathrm{j}=1,\dots ,{\mathrm{n}}_{\mathrm{i}};$$ (13)

where σ_i² = Var(ε_ij) denotes the residual variance for agencies assigned to the i-th strategy.

Under this scenario, the target criterion TC is given as

$$\text{TC}=1/\text{Var}(\mathrm{d})=1/[({{\mathrm{\sigma }}_1}^2/{\mathrm{n}}_1)+(({{\mathrm{\sigma }}_0}^2/{\mathrm{n}}_0))].$$ (14)

The optimal design task is to determine n₁ and n₀ that maximize the target criterion TC in Equation (14) under the budget constraint (8).

It can be shown that the optimal sample allocation is given as follows:

$${\mathrm{n}}_1\cong \mathrm{B}\times {\mathrm{\sigma }}_1/[\text{sqrt}({\mathrm{C}}_1)\times \mathrm{\theta }],\text{and}$$ (15a)

$${\mathrm{n}}_0\cong \mathrm{B}\times {\mathrm{\sigma }}_0/[\text{sqrt}({\mathrm{C}}_0)\times \mathrm{\theta }],$$ (15b)

where θ = σ₁ × sqrt(C₁) + σ₀ × sqrt(C₀). The optimal sample sizes, n₁ and n₀, are (approximately) proportional to the respective residual standard deviations, while also inversely proportional to the square root of the respective costs:

$${\mathrm{n}}_1/{\mathrm{n}}_0\cong {\mathrm{\sigma }}_1\times \text{sqrt}({\mathrm{C}}_0)/[{\mathrm{\sigma }}_0\times \text{sqrt}({\mathrm{C}}_1)].$$ (16)

Assume for now that cost is equal across agencies (C₁ = C₀), while agencies assigned to the novel strategy have a lower residual standard deviation by a factor of 1.5:

$${\mathrm{\sigma }}_1:{\mathrm{\sigma }}_0=1:1.5.$$ (17)

It follows from Equation (16) that the optimal sample size would be lower, by a factor of 1.5, for the novel strategy:

$${\mathrm{n}}_1:{\mathrm{n}}_0\cong 1:1.5.$$ (18)

For a second example, assume the cost ratio in Equation (11), and the ratio in Equation (17) for residual standard deviations. It follows from Equation (16) that the optimal sample sizes would differ by a factor of 3:

$${\mathrm{n}}_1:{\mathrm{n}}_0\cong 1:3.$$ (19)

The optimal sample size for the novel strategy is lower by a factor of two (Equation (12)) due to its higher cost (Equation (11)), and also lower by a factor of 1.5 (Equation (18)) due to its lower residual standard deviation (Equation (17)). Combining the two factors, the optimal sample size for the novel strategy is lower by a factor of 3 (= 2 × 1.5).

Non-random recruitment and assignment

The assumption of random recruitment and assignment of agencies might not be feasible for many implementation studies, resulting in a potential bias in the estimated implementation effect, d, in Equation (4). Therefore, the focus on the variance of the estimated implementation effect, Var(d), in Equation (5) is not an appropriate target criterion to evaluate the performance for candidate designs. Instead, it would be more appropriate to use the reciprocal of the mean squared error,

$$\text{TC}=1/\text{MSE}(\mathrm{d})=1/\mathrm{E}[{(\mathrm{d}-\mathrm{\delta })}^2]=1/\{\text{Var}(\mathrm{d})+{[\mathrm{E}(\mathrm{d})-\mathrm{\delta }]}^2\},$$ (20)

as the target criterion to be maximized, to take into account the possibility of having bias, E(d) − δ, in the estimated implementation effect, d, in addition to taking into account the uncertainty due to Var(d).

While the variance criterion, Var(d), is fairly straightforward to analyze for unbiased estimators, as in Equations (5), the characterization of the MSE criterion in Equation (20) is much more complicated due to the presence of the bias component, E(d) − δ. For further discussions on such a comprehensive framework for the design of survey studies, please see Groves and Lyberg (2010).

Dose-Response

As another example, consider an implementation study that aims to assess the dose-response relationship for an implementation program, such as the relationship between agency's performance improvement (the outcome response) and the intensity (the dose) of agency's participation in a training program, such as the number of staff hours devoted to the program. Assume that the intensity of participation is available from administrative records, while performance improvement needs to be assessed through on-site interviews with agency staff and clients. With limited resources, it is not feasible to assess performance improvement for all agencies. The optimal design task, therefore, is to select the most informative agencies to make the most efficient use of the available resources.

We assume that the dose-response relationship is anticipated to be linear, with homoscedastic variance,

$${\mathrm{Y}}_{\mathrm{j}}=\mathrm{\alpha }+\mathrm{\beta }\times {\mathrm{X}}_{\mathrm{j}}+{\varepsilon }_{\mathrm{j}},{\varepsilon }_{\mathrm{j}}\sim \mathrm{N}(0,{\mathrm{\sigma }}^2);\mathrm{j}=1,\dots ,\mathrm{n}.$$ (21)

The performance improvement for the j-th agency is denoted as Y_j; the intensity of participation is denoted as X_j. The intercept (the expected outcome in the absence of participation) is denoted as α. The dose-response parameter (the amount of performance improvement attributable to each unit of participation), denoted as β, is assumed to be the primary aim for the study.

We assume that intensity of participation, X, is statistically independent of residual disturbance, ε, therefore the estimated slope parameter, b, is unbiased for β. Under this assumption, it is reasonable to specify the target criterion (TC) to be maximized for the optimal design as the reciprocal of the variance for the estimated slope parameter, b:

$$\text{TC}=1/\text{Var}(\mathrm{b})=(\mathrm{n}-1)\times {\mathrm{S}}^2(\mathrm{X})/{\mathrm{\sigma }}^2,$$ (22)

where S²(X) denotes the sample variance for the X's among the agencies in the study. The optimal design task is to select the agencies to maximize the target criterion in Equation (22) under the budget constraint to be specified below.

For now we assume that the cost C is constant across agencies, so that the budget constraint is given as follows:

$$\mathrm{n}=\lfloor \mathrm{B}/\mathrm{C}\rfloor ,$$ (23)

similar to the budget constraint in Equation (3) for the first example.

Under this assumption, the sample size n is independent of the composition of the sample selection (which agencies are selected). Therefore the optimal design task (to maximize TC in Equation (22)) is equivalent to selecting the agencies to maximize the sample variance S²(X). This is achieved by selecting the most extreme agencies, i.e., those with the highest intensities of participation and those with the lowest intensities of participation, to provide the most informative contrast. This is analogous to the strategy of extreme or deviant case, and the strategy of maximum variation, for purposeful sampling.

It is important to note that the strategy of selecting extreme agencies might not be optimal under variations in the underlying assumptions. The subsection below provides an illustration for the contrary when the constant cost assumption is violated.

Unequal cost

Consider a hypothetical example consisting of 100 agencies with the distribution for the intensity of participation (X) shown in Table, with ten agencies at each level of intensity, 0, 1, 2, …, 9:

Table 1. Intensity of Participation.

Intensity (X)	0	1	2	3	4	5	6	7	8	9
Frequency	10	10	10	10	10	10	10	10	10	10

Assume further that the cost for enrolling each agency is $10,000 for each of the twenty most extreme agencies (the ten agencies with X = 0 and the ten agencies with X = 9); and $2,500 for each of the other 80 agencies with X = 1, 2, …, 8. Assume that the total budget available is B = $40,000.

Under these assumptions, the strategy of extreme or deviant cases will enroll four of the most extreme agencies, namely, two agencies with X = 0 and two agencies with X = 9, utilizing the entire budget of $40,000 ($10,000 for each agency). This design results in the maximum sample variance possible,

$${\mathrm{S}}^2(\mathrm{X})=4\times {(4.5)}^2/3=27.$$ (24)

Following Equation (22), the target criterion for this extreme-agencies design is given as follows:

$$\text{TC}=1/\text{Var}(\mathrm{b})=[3\times 27]/{\mathrm{\sigma }}^2=81/{\mathrm{\sigma }}^2.$$ (25)

Alternatively, the study can achieve a more efficient use of the resources by enrolling less extreme agencies that are less costly to enroll. In particular, the optimal design for this study is to enroll none of the most extreme (and more costly) agencies, and instead enroll eight agencies with X = 1 and eight agencies with X = 8, again utilizing the entire budget of $40,000 ($2,500 for each agency), resulting in the following sample variance:

$${\mathrm{S}}^2(\mathrm{X})=16\times {(3.5)}^2/15=13.07.$$ (26)

The sample variance in Equation (26) is much smaller than the sample variance achieved with the strategy of extreme cases (S²(X) = 27 given in Equation (24)), because the less extreme agencies provide less statistical information per agency. However, the lower statistical information per agency is more than compensated by the larger sample size available with the less extreme agencies (total n = 16 instead of 4).

Following Equation (22), the target criterion for this less-extreme-agencies design is given as follows:

$$\text{TC}=1/\text{Var}(\mathrm{b})=15\times 13.07/{\mathrm{\sigma }}^2=196/{\mathrm{\sigma }}^2,$$ (27)

which is superior to the target criterion achieved with the most extreme agencies, 81 / σ² given in Equation (25), by a factor of 2.42 (=196/81). Therefore, the selection of the most extreme agencies needs to take into consideration the validity of underlying assumptions, such as the assumption of constant cost across agencies.

Optimal Design for Multiple-Aimed Studies

Many implementation studies have multiple aims and, therefore, multiple parameters of interest. These aims may lead to different designs, making it necessary to seek a balance among the aims. Optimal design provides a useful framework for this balancing task.

As an illustrative example, we expand the first example in the previous section of an implementation study comparing two implementation strategies with the primary aim to assess the (overall) implementation effect, and incorporate an additional aim to assess the interaction between the strategies and an agency-level moderator, rurality, i.e., whether the agency is located in a rural community. This second aim is motivated by concerns about a possible disparity for the rural agencies where the novel implementation strategy might work less well due to lack of resources and peer support.

Model for Outcomes

We assume that agencies are assigned randomly to the two implementations strategies, stratified by rurality status, with the following two-way analysis of variance (ANOVA) model for the outcomes:

$${\mathrm{Y}}_{\text{ijk}}={\mathrm{\mu }}_{\text{ij}}+{\varepsilon }_{\text{ijk}},{\varepsilon }_{\text{ijk}}\sim \mathrm{N}(0,{\mathrm{\sigma }}^2);\mathrm{i}=0,1;\mathrm{j}=0,1;\mathrm{k}=0,1,2,\dots ,{\mathrm{n}}_{\text{ij}}.$$ (28)

The index i denotes the implementation strategies as before; the index j denotes the rurality status (=0 for a non-rural agency, =1 for a rural agency). For a strategy-rurality combination ij, the index k denotes the agencies with strategy i and rurality status j; n_ij denotes the number of agencies with strategy i and rurality status j. The outcome for the k-th agency with strategy-rurality combination ij is denoted as Y_ijk; the expected outcome for an agency with strategy-rurality combination ij is denoted as μ_ij. Furthermore, the random disturbance for the k-th agency with strategy-rurality combination ij is denoted as ε_ijk.

Study Aims and Parameters

The implementation effect among rural agencies is given by

$${\mathrm{\delta }}_1={\mathrm{\mu }}_{11}-{\mathrm{\mu }}_{01},$$ (29)

where μ₁₁ denotes the expected outcome for a rural agency (j=1) assigned to the novel implementation strategy (i=1), μ₀₁ denotes the expected outcome for a rural agency (j=1) assigned to the standard implementation strategy (i=0). Similarly, the implementation effect among non-rural agencies is given by

$${\mathrm{\delta }}_0={\mathrm{\mu }}_{10}-{\mathrm{\mu }}_{00}.$$ (30)

The moderating effect for rurality on the implementation effect, i.e., the interaction between the effect of rurality and the effect of implementation, is therefore given by

$$\mathrm{\gamma }={\mathrm{\delta }}_1-{\mathrm{\delta }}_0,$$ (31)

the difference between the implementation effect among rural agencies (δ₁) and the implementation effect among non-rural agencies (δ₀).

The second aim for this expanded implementation study is to assess the moderating effect parameter, γ, given in Equation (31).

To address the first aim for this study, namely, to assess the overall implementation effect, we need to specify the overall implementation effect. Consider the following thought experiment. If we assign all agencies in the study population to the novel implementation strategy, the expected outcome μ₁₊ is given as follows:

$${\mathrm{\mu }}_{1+}={\mathrm{P}}_0\times {\mathrm{\mu }}_{10}+{\mathrm{P}}_1\times {\mathrm{\mu }}_{11},$$ (32)

where P₀ denotes the proportion of agencies in the study population located in a non-rural community, and P₁ (= 1 − P₀) denotes the proportion of agencies located in a rural community. Similarly, if we assign all agencies in the study population to the standard implementation strategy, the expected outcome μ₀₊ is given as follows:

$${\mathrm{\mu }}_{0+}={\mathrm{P}}_0\times {\mathrm{\mu }}_{00}+{\mathrm{P}}_1\times {\mathrm{\mu }}_{01}.$$ (33)

The overall implementation effect δ_overall is the difference obtained in this thought experiment:

$${\mathrm{\delta }}_{\text{overall}}={\mathrm{\mu }}_{1+}-{\mathrm{\mu }}_{0+}={\mathrm{P}}_0\times {\mathrm{\delta }}_0+{\mathrm{P}}_1\times {\mathrm{\delta }}_1.$$ (34)

Our first study aim is to assess the overall implementation effect parameter, δ_overall, given in Equation (34).

Parameter Estimation

To estimate the parameters δ_overall in Equation (34) and γ in Equation (31), we first estimate the expected outcome parameters by the corresponding sample means:

$$\text{Estimate}{\mathrm{\mu }}_{\text{ij}}\text{by}{\mathrm{m}}_{\text{ij}},\mathrm{i}=0,1;\mathrm{j}=0,1;$$ (35)

where m_ij denotes the sample mean for the outcomes from agencies with rurality status j assigned to implementation strategy i. It follows from Equation (35) that the implementation effect for rural agencies, δ₁ in Equation (29), is estimated by

$${\mathrm{d}}_1={\mathrm{m}}_{11}-{\mathrm{m}}_{01}.$$ (36)

Likewise, the implementation effect for non-rural agencies, δ₀ in Equation (30), is estimated by

$${\mathrm{d}}_0={\mathrm{m}}_{10}-{\mathrm{m}}_{00}.$$ (37)

The overall implementation effect parameter, δ_overall in Equation (34), is estimated by

$${\mathrm{d}}_{\text{overall}}={\mathrm{P}}_0\times {\mathrm{d}}_0+{\mathrm{P}}_1\times {\mathrm{d}}_1.$$ (38)

The moderating effect parameter, γ in Equation (31), is estimated by

$$\mathrm{g}={\mathrm{d}}_1-{\mathrm{d}}_0.$$ (39)

All of the estimates presented in this subsection are unbiased for the respective parameters.

Budget Constraint

We assume equal residual variance σ², as shown in Model (28), and constant cost C per agency, resulting in the following budget constraint:

$${\mathrm{n}}_{00}+{\mathrm{n}}_{10}+{\mathrm{n}}_{01}+{\mathrm{n}}_{11}\le \mathrm{B}/\mathrm{C},$$ (40)

where n_ij denotes the number of agencies with rurality status j assigned to implementation strategy i, and B denotes the total budget available.

Target Criterion and Optimal Design for Overall Implementation Effect

The optimal design task is to determine the most informative sample sizes, n₀₀, n₁₀, n₀₁, and n₁₁, to maximize the target criterion (to be specified below), under the budget constraint (40). With the two study aims, we have several options to specify the target criterion for the optimal design.

If we focus entirely on the first study aim to estimate the overall implementation effect parameter δ_overall in Equation (34), it is reasonable to specify the target criterion (TC) to be maximized as the reciprocal for the variance for the estimate d_overall in Equation (38):

$$\text{TC}=1/\text{Var}({\mathrm{d}}_{\text{overall}})=1/\{{\mathrm{\sigma }}^2\times [{{\mathrm{P}}_0}^2\times ({{\mathrm{n}}_{10}}^{-1}+{{\mathrm{n}}_{00}}^{-1})+{{\mathrm{P}}_1}^2\times ({{\mathrm{n}}_{11}}^{-1}+{{\mathrm{n}}_{01}}^{-1})]\}.$$ (41)

The optimal design task for this scenario is to maximize the target criterion TC in Equation (41) under budget constraint (40). It can be shown that the optimal sample sizes are given as follows:

$${\mathrm{n}}_{00}\cong {\mathrm{P}}_0\times (\mathrm{B}/\mathrm{C})/2,$$ (42a)

$${\mathrm{n}}_{10}\cong {\mathrm{P}}_0\times (\mathrm{B}/\mathrm{C})/2,$$ (42b)

$${\mathrm{n}}_{01}\cong {\mathrm{P}}_1\times (\mathrm{B}/\mathrm{C})/2,\text{and}$$ (42c)

$${\mathrm{n}}_{11}\cong {\mathrm{P}}_1\times (\mathrm{B}/\mathrm{C})/2.$$ (42d)

The optimal design shown in Equations (42a-42d) allocates the rural and non-rural subsamples proportional to the corresponding proportions in the study population. Each subsample is then assigned randomly, 50:50, to the two implementation strategies.

Target Criterion and Optimal Design for Moderating Effect

If we focus entirely on the second study aim to estimate the moderating effect parameter γ in Equation (31), it is reasonable to specify the target criterion (TC) to be maximized as the reciprocal for the variance for the estimate g in Equation (39):

$$\text{TC}=1/\text{Var}(\mathrm{g})=1/[{\mathrm{\sigma }}^2\times ({{\mathrm{n}}_{10}}^{-1}+{{\mathrm{n}}_{00}}^{-1}+{{\mathrm{n}}_{11}}^{-1}+{{\mathrm{n}}_{01}}^{-1})].$$ (43)

The optimal design task for this scenario is to maximize the target criterion TC in Equation (43) under budget constraint (40). It can be shown that the optimal sample sizes are given as follows:

$${\mathrm{n}}_{00}\cong (1/2)\times (\mathrm{B}/\mathrm{C})/2,$$ (44a)

$${\mathrm{n}}_{10}\cong (1/2)\times (\mathrm{B}/\mathrm{C})/2,$$ (44b)

$${\mathrm{n}}_{01}\cong (1/2)\times (\mathrm{B}/\mathrm{C})/2,\text{and}$$ (44c)

$${\mathrm{n}}_{11}\cong (1/2)\times (\mathrm{B}/\mathrm{C})/2.$$ (44d)

The optimal design shown in Equations (44a-44d) allocates the rural and non-rural subsamples equally. Each subsample is then assigned randomly, 50:50, to the two implementation strategies.

If the study population happens to have the same proportion of rural and non-rural agencies, i.e., P₀ = P₁ = 50%, the optimal designs for the two study aims coincide, therefore the same design is optimal for both study aims.

Alternatively, if the rural and non-rural agencies do not have the same proportions in the study population, i.e., P₀ ≠ P₁, the two study aims lead to different optimal designs. For example, if only 20% of the agencies in the study population are located in rural communities (P₁ = 20%), the first study aim allocates 20% of the sample to rural agencies, while the second study aim allocates 50% of the sample to rural agencies. The conflict between the two study aims needs to be balanced, taking into account both target criterion (41) and target criterion (43).

Target Criterion and Optimal Design for Both Effects Considered Jointly

In order to balance multiple study aims, a composite target criterion needs to be specified for the optimal design to represent the various study aims, such as the following composite target criterion (CTC) for the current example:

$$\text{CTC}=1/[{\mathrm{w}}_1\times \text{Var}({\mathrm{d}}_{\text{overall}})+{\mathrm{w}}_2\times \text{Var}(\mathrm{g})],$$ (45)

where w₁ and w₂ (=1−w₁) denote the importance weights specified for the two respective study aims. Such a composition target criterion is known in the optimal design literature as a weighted A-optimality criterion (Shirakura and Tong 1993). The optimal design task for this scenario is to specify the most informative sample sizes n₀₀, n₁₀, n₁₀, and n₁₁, to maximize the composite target criterion (CTC) in Equation (45) under budget constraint (40).

The weights w₁ and w₂ in Equation (45) are to be specified according to the relative importance of the two study aims to the overall study. In the extreme, if the first study aim is considered overwhelmingly more important than the second study aim, we should specify w₁ as overwhelmingly larger than w₂. In this situation, the optimal design task is dominated by the component Var(d_overall) in the composition target criterion, resulting in a design that is focused on the first study aim. On the other hand, if the second study aim is considered overwhelmingly more important than the first study aim, we should specify w₂ as overwhelmingly larger than w₁, resulting in a design that is focused on the second study aim. If neither study aim is considered to be overwhelmingly more important than the other, the optimal design will accomplish an appropriate balance between the two study aims.

It can be shown that the optimal sample sizes for this study are given as follows:

$${\mathrm{n}}_{00}\cong {\mathrm{R}}_0\times (\mathrm{B}/\mathrm{C})/2,$$ (46a)

$${\mathrm{n}}_{10}\cong {\mathrm{R}}_0\times (\mathrm{B}/\mathrm{C})/2,$$ (46b)

$${\mathrm{n}}_{01}\cong {\mathrm{R}}_1\times (\mathrm{B}/\mathrm{C})/2,\text{and}$$ (46c)

$${\mathrm{n}}_{11}\cong {\mathrm{R}}_1\times (\mathrm{B}/\mathrm{C})/2,$$ (46d)

where

$${\mathrm{R}}_0={\mathrm{Q}}_0/({\mathrm{Q}}_0+{\mathrm{Q}}_1),{\mathrm{R}}_1={\mathrm{Q}}_1/({\mathrm{Q}}_0+{\mathrm{Q}}_1),\text{and}$$ (47)

$${\mathrm{Q}}_0=\text{sqrt}({\mathrm{w}}_1\times {{\mathrm{P}}_0}^2+{\mathrm{w}}_2),{\mathrm{Q}}_1=\text{sqrt}({\mathrm{w}}_1\times {{\mathrm{P}}_1}^2+{\mathrm{w}}_2).$$ (48)

Furthermore, it can be shown that R₀ (R₁) falls between P₀ (P₁) and ½, the leading terms in Equations (42a-42d) and (44a-44d):

$$\text{If}{\mathrm{P}}_0\le ½,\text{then}:{\mathrm{P}}_0\le {\mathrm{R}}_0\le ½\le {\mathrm{R}}_1\le {\mathrm{P}}_1;$$ (49a)

$$\text{If P}0>½,\text{then}:{\mathrm{P}}_1<{\mathrm{R}}_1<½<{\mathrm{R}}_0<{\mathrm{P}}_0.$$ (49b)

The format of Equations (46a-46d) is analogous to the format of Equations (42a-42d) and (44a-44d), with the leading terms P₀ in Equations (42a-42b) and P₁ in Equations (42c-42d), and the leading term ½ in Equations (44a-44d), replaced by the leading terms R₀ in Equations (42a-42b) and R₁ in Equations (42c-42d), to reflect the compromise between the two study aims. More specifically, the first study aim for the overall implementation effect results to the leading terms P₀ in Equations (42a-42b) and P₁ in Equations (42c-42d); the second study aim for the moderating effect results in the leading term ½ in Equations (44a-44d); the composite target criterion, CTC in Equation (45), results in the leading terms R₀ in Equations (46a-46b) and R₁ in Equations (46c-46d). According to Equations (49a-49b), the leading terms in Equations (46a-46d) falls between the leading terms in Equations (42a-42d) and the leading terms in Equations (44a-44d), therefore the optimal design based on the composite target criterion, CTC in Equation (45), indeed can be interpreted as a compromise between the optimal design for the first study aim (Equations (42a-42d)) and the optimal design for the second study aim (Equations (44a-44d)).

Consider a hypothetical example with 20% of the agencies in the study population located in rural communities (P₁ = 20%). The first study aim allocates 20% of the sample to rural agencies, while the second study aim allocates 50% of the sample to rural agencies. Assume that the total budget B is $40,000, the cost per agency is $400, so the budget constraint (40) allows a total sample size of 100. The optimal design (46a-46d) based on the composite target criterion (CTC) in Equation (45), and the budget constraint (40), is shown in Table 2 below:

Table 2. Optimal Design with P₁ = 20%, n_total = B / C = 100.

w1	w2	n00	n10	n01	n11
1.00	0.00	40	40	10	10
0.80	0.20	32	32	18	18
0.60	0.40	29	29	21	21
0.40	0.60	27	27	23	23
0.20	0.80	26	26	24	24
0.00	1.00	25	25	25	25

The first row in Table 2 shows the results based on the first study aim: 80% of the sample is allocated to non-rural agencies (40 assigned to each implementation strategy), 20% of the sample is allocated to rural agencies (10 assigned to each implementation strategy). The last row shows the results based on the second study aim: 50% of the sample is allocated to non-rural agencies, 50% to rural agencies. The intermediate rows show the results based on various compromises between the two study aims. As more importance is placed on the second study aim, the optimal sample allocation shifts from the skewed allocation for the first study aim towards the equal allocation for the second study aim.

The use of optimal design with a composite target criterion provides an alternative to the widely used practice of making study design decisions based entirely on the primary study aim. While it might be appropriate to focus on a single primary study aim when this study aim is overwhelmingly more important than the other study aims, many implementation studies do have multiple study aims that need to be considered jointly with no single study aim that is overwhelmingly more important than others. For those studies, it would be worthwhile taking the extra efforts and apply the optimal design approach, using an appropriately specified composite target criterion, to allow the competing study aims to be considered jointly in the design decision.

Group Randomized Trial

We assumed implicitly in the previous examples that the study is focused on the agency level, therefore the residual disturbances, ε, are statistically independent. Alternatively, many implementation studies have a multi-level structure, say, with agencies nested within networks. Agencies within the same network share common features such as network policy and culture, and thus are likely to be more similar than agencies from different networks. As a result, the residual disturbances are statistically dependent, with intra-cluster correlation, among agencies in the same network. Raudenbush (1997) and Raudenbush and Liu (2000) examined optimal allocation of resources within and between clusters as a function of variance components and costs at each level. Statistical software developed for these trials has been disseminated widely as a methodological resource through the William T. Grant Foundation (Raudenbush et al. 2011; Spybrook et al. 2011). The software enables one to calculate power analysis for individual and group randomized trials.

Discussion

There is a promising potential for synergy in the integration of optimal design and purposeful sampling in mixed methods implementation research. This conjunction can facilitate the selection of information-rich cases to maximize effective use of limited resources. This is especially important in complex implementation studies where there is a need for balance across multiple study aims.

The method of optimal design offers a model for the systematic identification of cases and conditions for qualitative or mixed method research that enable efficient use of in depth insight from limited sample sizes at each stage of implementation. Multi-stage purposeful designs also offer better fit with quantitative experimental and quasi-experimental designs frequently used in implementation research.

While optimal design provides a systematic, quantitative approach to select information-rich cases, this methodology requires a variety of assumptions. The more flexible purposeful sampling may serve as a useful complement to optimal design, to allow a qualitative approach to formulate and evaluate these assumptions. At the same time, the optimal design approach provides a useful framework to assess the sensitivity of design decisions to deviations from the usual assumptions.