Authors’ reply: Letter to the Editor: Preference option randomized design (PORD) for comparative effectiveness research: Statistical power for testing comparative effect, preference effect, selection effect, intent-to-treat effect, and overall effect (SMMR, Vol. 28, Issue 2, 2019)

We appreciate Dr. Walter and his colleagues’ pointing out a limitation concerning (A2) assumption.¹ In short, we adopted the A2 assumption because in preference option randomized design (PORD), there are no participants who prefer treatment B in A_S (even though there are participants who prefer treatment B in R_A, that is B_A), or those who prefer treatment A in B_S (even though there are participants who prefer treatment A in R_B, that is A_B). In theory, however, it might have been more reasonable to assume that P(A_A∣A_S) = P(A_B∣B_R∪A_B) = P(A_B∣R_B)/(1 − P(B_B∣R_B)) = P(A_B∣R_B)/(1 − P(B_A∣R_A)), this equation holding by a strict NSBR (no selection bias from randomization) assumption, and likewise P(B_B∣B_S) = P(B_A∣R_A)/(1 − P(A_B∣R_B)). However, these assumptions are restricted by the following conditions: P(A_B∣R_B) ≤ 1 − P(B_A∣R_A) = P(A_S∣R_A) and P(B_A∣R_A) ≤ 1 − P(A_B∣R_B) = P(B_S∣R_A) so that P(A_A∣A_S) ≤ 1 and P(B_B∣B_S) ≤ 1. Therefore, the A2 assumption, albeit limited, might be necessary so that such a restriction can be avoided.

Although Walter and his colleagues asserted in their letter that “… a valid analysis for this type of study is already available” and “… applies to the PORD design in particular,” we find that their approach² based on post hoc preference analyses of parallel randomized trials is invalid by implicitly assuming that marginal or global probabilities are the same as conditional probabilities on treatment arms. Furthermore, their approach does not have anything to do with power analysis of preference trials. Specifically, their Table 1 (Box 1) classified study subjects into five mutually exclusive and exhaustive groups, and they assigned probabilities p1 to p5 to Groups 1 to 5, respectively. The probabilities p1 to p5 are all marginal or global probabilities, but not conditional probabilities on treatment arms, so that the sum of p1 to p5 is one, that is

$$p1+p2+p3+p4+p5=1$$ (0)

Box 1. Table 1 of Walter et al.’s paper.².

Table 1.

Framework of five preference groups of patients in a randomized trial comparing treatments A and B

Group	Actual treatment if offered A	Actual treatment if offered B	Description
1	A	B	Compliers
2	A	0	A-preferers
3	0a	B	B-preferers
4	A	A	A-insisters
5	B	B	B-insisters

^a0 indicates that neither A or B treatment is received.

In their text (Box 2), however, to derive their equation (1), they define that all α’s and β’s are conditional probabilities on arms A and B and implicitly assume that the marginal probabilities p’s are equal to the conditional probabilities α’s and β’s. If we rewrite equation (1), then we have α_A + β_B = 1 + p1 implying that α_A + β_B > = 1 by definition of a probability that p1 > = 0. Conversely, in order for p1 to be positive, α_A+β_B must be greater than 1, which is noted in their paper. Nonetheless, if all p’s are marginal following the group classifications in Table 1 (Box 1), it is immediate from equation (0) that

$$p1=1-(p2+p3+p4+p5)$$ (2)

Box 2. Excerpt from the text of Walter et al.’s paper².

We denote α_A, β_B, and α₀ to be the observed proportions receiving A, B, or 0 in study arm A, and β_A, β_B, and β₀ to be the observed proportions receiving A, B, or 0 in study arm B. Note that α_A + α_B + α₀ = β_A + β_B + β₀ = 1. We also denote the proportions of patients in the preference groups shown in Table 1 by p₁, p₂, …p₅. By equating the observed proportions with their mixture expectations as implied by Figure 4, we can estimate the ps as follows:

P₂ = β₀ (A-preferers), p₃ = α₀ (B-preferers),

P₄ = β_A (A-insisters), p₅ = α_B (B-insisters) and hence,

$$p_1={\alpha }_{\mathrm{A}}+{\beta }_{\mathrm{B}}-1(\text{compliers}).$$ (1)

In equation (2), p2+p3+p4+p5 must be less than 1 so that p1 be positive. As will be shown below, however, one can find counter examples where p2+p3+p4+p5>1 under their assumption resulting in p1<0, which contradicts the definition of probabilities for mutually exclusive and exhaustive groups or events.

Under their example in Table 2 (Box 3), both equations (1) and (2) give the same answer p1 = 0.66, which is nonetheless incorrect regardless of whether or not α_A+β_B > 1 because their implicit assumption of marginal (p’s) = conditional (α’s and β’s) probabilities is invalid.

Box 3. Table 2 of Walter et al.’s paper.².

Table 2.

Example 1

a) Patient frequencies by assigned and actual treatments
		Assigned treatment
		A	B
Actual treatment	A	430	50
	B	30	400
	0	40	50
	Total	500	500
b) Estimated preference distribution
Compliers	p1 = 0.66
A-preferers	p2 = 0.10	B-preferers	p3 = 0.08
A-insisters	p4 = 0.10	B-insisters	p5 = 0.06

Subscripts refer to preference groups shown in Table 1.

Now, consider a counter example in Table 3. Following their logic in Box 2 and their computation process in Table 2 (Box 3), we have: p2 = 0.4, p3 = 0.4, p4 = 0.4, p5 = 0.5; and α_A = 0.1 and β_B = 0.2. First, here α_A + β_B = 0.3 which contradicts equation (1). Second, if we follow their equation (1), then we have: p1 = α_A + β_B −1 = 0.3 − 1 = −0.7. If we follow equation (2), then we have: p1 = 1 −(p2 + p3 + p4 + p5) = 1 − 1.7 = −0.7. It means that none of equation (0) to (2) is valid under their implicit assumption of marginal = conditional probabilities. In other words, none of the p’s estimated under this assumption are marginal probabilities correctly assigned to the groups defined in Table 1 (Box 1). Consequently, subsequent inferences in their paper might be misleading.

Table 3.

Counter example.

Actual treatment	Assigned treatment
	A	B
A	50	200
B	250	100
0	200	200
Total	500	500

In conclusion, their approach is not valid, nor is it possible to correctly estimate p1 under their post hoc settings with their marginal = conditional probabilities approach. Nevertheless, it will be interesting to further investigate how to better estimate P(A_B∣A_S) and P(B_B∣B_S) in PORD or their p1 of compliers since PORD design has many practically appealing features for comparative effectiveness research (CER) trials in real world settings, where participants have strong preferences. In such settings, the participants’ or other parties’ preferences need to be taken into consideration at the design stage rather than conducting post hoc preference analyses of traditional randomized parallel designs which may not be inherently suitable for CER trials.