Behavioral carry-over effect and power consideration in crossover trials

Abstract

A crossover trial is an efficient trial design when there is no carry-over effect. To reduce the impact of the biological carry-over effect, a washout period is often designed. However, the carry-over effect remains an outstanding concern when a washout period is unethical or cannot sufficiently diminish the impact of the carry-over effect. The latter can occur in comparative effectiveness research, where the carry-over effect is often non-biological but behavioral. In this paper, we investigate the crossover design under a potential outcomes framework with and without the carry-over effect. We find that when the carry-over effect exists and satisfies a sign condition, the basic estimator underestimates the treatment effect, which does not inflate the type I error of one-sided tests but negatively impacts the power. This leads to a power trade-off between the crossover design and the parallel-group design, and we derive the condition under which the crossover design does not lead to type I error inflation and is still more powerful than the parallel-group design. We also develop covariate adjustment methods for crossover trials. We evaluate the performance of cross-over design and covariate adjustment using data from the MTN-034/REACH study.

1. INTRODUCTION

A crossover trial is a longitudinal study in which patients are randomized to different sequences of treatments with the intention of comparing the effects of different treatments (Hills and Armitage, 1979; Senn, 1994; Kim et al., 2021, Chapter 8). In the simplest crossover design, patients are randomly allocated to 2 groups: one group receives treatment 1 at period 1 and treatment 0 at period 2, and the other group receives the treatment sequence in the reverse order. The main advantage of a crossover design is that every patient can serve as their own control, which largely eliminates the between-patient variability and makes the crossover design more efficient than the parallel-group design.

A key assumption for the crossover design is that there is no carry-over effect, that is, the treatment assigned during the first period does not interfere with the outcome in the second period. Therefore, a crossover design is most common for treatments whose effect vanishes when discontinued and for non-absorbing endpoints. Examples include early phase trials (eg, pharmacokinetic [PK] studies and dose-finding studies) and phase III studies with chronic conditions (eg, hypertension, pain, and asthma); see Jones and Lewis (1995) for a review. Meanwhile, a washout period is often designed between periods 1 and 2 to effectively reduce the impact of the biological carry-over effect engendered by treatments taken at period 1. The washout can be passive (ie, no treatment is given during the washout period) or active (ie, a treatment is given during the washout period but measurement is delayed until steady state is reached) (Senn, 2002; Araujo et al., 2016). However, the carry-over effect remains an outstanding concern when a washout period is inappropriate or cannot sufficiently diminish the impact of the first period. For example, the MTN-034/REACH study (Nair et al., 2023) is a phase 2a HIV-prevention trial evaluating the safety of and the adherence to the monthly dapivirine vaginal ring (DVR) and daily oral tenofovir disoproxil fumarate plus emtricitabine (TDF/FTC) among adolescent girls and young women aged 16–21 years. This trial uses a 2-period crossover design without a passive washout period since withholding an effective HIV preventive agent from the population at risk is unethical. If participants initially using the monthly DVR (which requires no active effort to remain adherent for a month after inserting the ring) find the transition to daily pill-taking in period 2 particularly burdensome, then the ease and habit formed during the first period with the monthly DVR might carry over into period 2 and affect their adherence behavior in period 2. This type of carry-over effect is non-biological and is hard to be eliminated by washout periods; we term it as the behavioral carry-over effect, which is the impact that a treatment has on the subsequent outcomes due to the altering of a participant’s behavior. It can be an important consideration in other HIV prevention trials, for example, the TRIO study (Minnis et al., 2018), and more broadly in comparative effectiveness research that uses the crossover design (Hemming et al., 2020).

The presence of the carry-over effect can bias the estimation of the treatment effect. This motivates Grizzle’s (1965) 2-stage procedure, which first tests whether the carry-over effect exists or not, and the testing result determines whether to use the period 2 data in the analysis. This 2-stage procedure was then criticized by Freeman (1989) because it has low power and can inflate the type I error of subsequent analysis. Another type of method is to model the carry-over effect (Brown, 1980; Laird et al., 1992; Jones and Donev, 1996; Kunert and Stufken, 2002; Bailey and Kunert, 2006), which is sensitive to model misspecification.

In this paper, we examine the carry-over effect in crossover trials using a potential outcomes framework (Neyman, 1923; Rubin, 1974) and only require minimal statistical assumptions to gain a better understanding of its impact. Under the ideal setting with no carry-over effect, we derive the asymptotic properties of the basic estimator of the treatment effect, based on which we show that the crossover design is typically more efficient than the parallel-group design. Our findings, while aligning with existing efficiency comparisons in the literature (refer to Senn 2002, Section 9.2.1, eg), are more general, because we only assume the no carry-over effect assumption and no other assumptions (eg, modeling assumptions or normality of data). Then we investigate the situation with the carry-over effect, uncovering new insights on its influence on estimation bias, as well as the type I error and power of tests. When the carry-over effect λ₁ + λ₀ is negative, the basic estimator overestimates the treatment effect, which can inflate the type I error of one-sided tests. Conversely, when the carry-over effect is positive, the basic estimator underestimates the treatment effect, which does not inflate the type I error of one-sided tests but negatively impacts the power. This leads to a power trade-off between the crossover design and the parallel-group design, and we derive the condition under which the crossover design does not lead to type I error inflation and is still more powerful than the parallel-group design.

The rest of the paper proceeds as follows. Section 2 focuses on crossover trials with no carry-over effect, where we present a potential outcomes framework for crossover trials, investigate the basic estimator, and compare the efficiencies of a crossover and a parallel-group design. Section 3 extends the results to the case with the carry-over effect. Section 4 proposes a covariate adjustment method and studies its asymptotic properties. Section 5 examines the power trade-off of the crossover and the parallel-group design. Section 6 presents the application of our methods to the REACH study, as well as power considerations in the design of a future HIV prevention trial using data from the REACH study. Section 7 concludes with a discussion.

2. CROSSOVER TRIALS WITH NO CARRY-OVER EFFECT

2.1. Setup and assumptions

Consider a crossover trial for 2 treatments in 2 periods. A sample of n subjects are randomly allocated to 2 treatment sequences, where A_i = 1 denotes that subject i first receives treatment 1 and then treatment 0, and A_i = 0 denotes the reverse order. Let be the potential outcome at time 1 had the subject been exposed to treatment j at time 1, for j = 0, 1. Let be the potential outcome at time 2 had the subject been exposed to treatment j at time 1 and treatment k at time 2, for j, k = 0, 1. The observed outcome for subject i at time t is Y_it. Throughout the article, we make the consistency assumption that links the observed outcomes to the potential outcomes: for and for and . Let be a vector of observed baseline covariates for subject i. We assume that are independent and identically distributed with finite second-order moments, and the covariance matrix is positive definite.

Simple randomization assigns subjects to the 2 treatment sequences completely at random. This is summarized in Assumption 1.

Assumption 1 (Simple randomization)

for j, k = 0, 1, and P(A_i = 1) = π₁, where 0 < π₁ < 1 is known and π₀ = 1 − π₁.

Assumption 2 is the key assumption typically imposed in crossover trials: It states that the treatment at time 1 does not have a direct effect on the outcome at time 2; see Figure 1(A) for an illustration. Many crossover trials would plan a sufficiently long washout period between the 2 time periods to make this assumption more plausible.

FIGURE 1.

(A) Six potential outcomes under Assumption 2. (B) Six potential outcomes without Assumption 2.

Assumption 2 (No carry-over effect)

For k = 0, 1, almost surely.

Under Assumption 2, we can simply let denote the potential outcome at time 2. We are interested in the average treatment effects 2⁻¹(θ₁ + θ₂), where and . Note that it is common to assume that the treatment effect is time-invariant so that θ₁ = θ₂ (Hills and Armitage, 1979). However, as shown in Theorem 1, a time-invariant treatment effect assumption is not necessary for studying treatment effect in crossover trials.

2.2. The basic estimator

Using the full crossover period, we can calculate the difference in outcomes for every subject under treatment 1 and treatment 0. The basic estimator is commonly used, which first calculates the arm-specific average outcome difference, and then averages the 2 means from the 2 arms (Hills and Armitage, 1979; Senn, 1994; Kim et al., 2021, Chapter 8):

(1)

where is the average of Δ_i’s for subjects with A_i = a, Δ_i = Y_i1 − Y_i2, and n_a is the number of subjects with A_i = a, for a = 0, 1. Theorem 1 summarizes the statistical properties of the basic estimator .

Theorem 1

Under Assumptions 1 and 2,

(a) , where for t = 1, 2.

(b) , where .

Proof of Theorem 1 and all other proofs are in the supplementary materials. In the proof of Theorem 1(a), we show that the expected change in outcome for A_i = 1 is the average treatment effect at time 1 minus the expected change in outcome in the absence of the treatment, that is, , where denotes the expected temporal change in outcome in the absence of the treatment. Similarly, the expected change in outcome for A_i = 0 is the average treatment effect plus the expected change in outcome in the absence of the treatment, that is, . At first glance, it appears that both the group-specific average changes in outcome are biased by τ. However, randomization balances out the effect of temporal trend that is not due to the treatment, and thus the overall average change in outcome remains unbiased for 2⁻¹(θ₁ + θ₂). Moreover, with a given sample size n, Theorem 1(b) guides the optimal choice of π₁ to minimize the asymptotic variance. For example, if , the optimal choice would be equal allocation with π₁ = π₀ = 1/2. For statistical inference, can be consistently estimated by

(2)

where and are, respectively, the sample variance of Δ_i = Y_i1 − Y_i2 for subjects under A_i = 1 and A_i = 0.

There is another estimator that looks similar to the basic crossover estimator : . These 2 estimators and are the same under randomization schemes that enforce n₁ = n₀, but they are not the same in other cases (eg, under the simple randomization considered in this article). Specifically, under simple randomization with equal allocation (π₁ = π₀ = 1/2), is also unbiased for 2⁻¹(θ₁ + θ₂); however, the asymptotic variance of equals , which is larger than the asymptotic variance of . The additional variance component 4⁻¹(θ₁ − θ₂ − 2τ)² is due to the random effect of the temporal trend and the treatment effect heterogeneity at the 2 time points. Under simple randomization with unequal allocation, is biased due to the time effect τ. This important point is also discussed in Hills and Armitage (1979). Therefore, we do not consider this alternative estimator in the rest of the article.

2.3. Efficiency comparison between a crossover and parallel-group design

When there is no carry-over effect, that is, when Assumption 2 holds, it is well known that a crossover design is typically more efficient than a parallel-group one. This is because each subject can serve as their own control, which cuts down half the sample size, and the within-subject comparison can further remove the inter-subject variability (Jones and Kenward, 2015). Prior work such as Senn (2002, Section 9.2.1) has provided efficiency comparisons between a crossover and parallel-group design when there is no carry-over effect and under various additional assumptions (eg, modeling assumptions, time-invariant treatment effect, and normality of data). In this section, we provide a more general efficiency comparison that assumes only Assumption 2 and no other assumptions.

Suppose a randomized controlled trial is being planned to demonstrate the superiority or non-inferiority of an investigational treatment. First, consider a crossover design with the null hypothesis H₀: 2⁻¹(θ₁ + θ₂) = θ* versus the alternative hypothesis H_A: 2⁻¹(θ₁ + θ₂) > θ* for some pre-specified θ*, where θ* = 0 for test of superiority and θ* > 0 for test of non-inferiority. The test statistic based on the basic estimator is . From Theorem 1, under H₀, and thus, we reject H₀ if and only if T_cr > z_{1 − α}, where α is the significance level and z_{1 − α} is the (1 − α)th quantile of the standard normal distribution. Under the local alternative with a constant γ_cr > 0, the power of T_cr is Power_cr ≈ Φ( − z_{1 − α} + γ_cr/σ_cr), where Φ( · ) is the cumulative distribution function of the standard normal distribution and ≈ denotes asymptotic approximation.

Now suppose we only use the data at time 1, which is effectively a parallel-group design. The parallel-group counterpart to is a simple mean difference . Under Assumption 1, we have and , where . For testing the null hypothesis H₀: θ₁ = θ* versus the alternative hypothesis H_A: θ₁ > θ*, the test statistic is , where is a consistent estimator for . Under the local alternative with a constant γ_pr > 0, the power of T_pr is

(3)

Let Power_cr = Power_pr = 1 − β, we obtain the required sample sizes to achieve power of 1 − β under the 2 designs:

These sample size formulae have been previously derived, for example, in Jones and Kenward (2015, Section 2.4) under additional assumptions such as the time-invariant treatment effect. When θ₁ = θ₂, the 2 aforementioned null hypotheses are the same, and the ratio of the sample sizes to achieve the same power using the 2 tests, which is also called the Pitman asymptotic relative efficiency, is

For illustration, consider a simple case where θ₁ = θ₂, for t = 1, 2 and j = 0, 1, and , where ρ ∈ [0, 1) is the intraclass correlation coefficient (ICC). Then and , and n_cr/n_pr = (1 − ρ)/2. This implies that the crossover design only requires (1 − ρ)/2, that is, at most half of the sample size required by the parallel-group design, which is the key advantage of the crossover design over the parallel-group design.

3. CROSSOVER TRIALS WITH CARRY-OVER EFFECT

3.1. Setup and assumptions

When there exists the carry-over effect in a crossover trial, the treatment at time 1 may interfere with the outcome at time 2; see Figure 1(B) for an illustration of the 6 potential outcomes with carry-over effects. In this case, Assumption 1 still holds by the act of randomization, while Assumption 2 is violated.

We present a way of parameterizing the expected values of the 6 potential outcomes in Table 1 that directly extends the parameterization used in Section 2. In Table 1, denotes the average treatment effect at time 2, which compares the potential outcome had one stayed on treatment 1 to the potential outcome had one stayed on treatment 0; denotes the expected temporal change in outcome had one stayed on treatment 0; and are the expected carry-over effects under the 2 treatment regimes.

TABLE 1.

Parameterization of the 6 potential outcome means with and without Assumption 2.

Potential outcome means	With Assumption 2	Without Assumption 2
	μ	μ
	μ + θ1	μ + θ1
	μ + τ
	μ + τ
	μ + τ + θ2
	μ + τ + θ2

The 2 shadowed rows correspond to the potential outcomes that are never observed.

3.2. The basic estimator

Theorem 2 derives the statistical properties of the basic estimator defined in Equation 1 without Assumption 2.

Theorem 2

Under Assumption 1,

(a) .

(b) , where .

Similar to the proof of Theorem 1(a), we show that the expected change in outcome for A_i = 1 is , that for A_i = 0 is . Hence, the overall average change in outcome is . Hence, Theorem 2(a) implies that is still an unbiased estimator of the treatment effect under a population-level no carry-over effect assumption ( for k = 0, 1), which is weaker than the individual-level no carry-over effect assumption stated in Assumption 2. It is also straightforward to verify that Theorem 1 is a special case of Theorem 2 under Assumption 2. Lastly, note that defined in Equation 2 is still a consistent estimator of without Assumption 2.

3.3. Type I error and power analysis

With possible carry-over effects, the first question one would ask is whether using T_cr to analyze crossover trials would lead to type I error inflation. To answer this question, consider the null hypothesis versus the alternative hypothesis for some pre-specified θ*. The type I error rate of is

When λ₀ + λ₁ = 0, is an unbiased estimator for the treatment effect and the type I error rate of T_cr is α. When λ₀ + λ₁ > 0, underestimates the treatment effect and the type I error rate is less than α, meaning that the test is conservative. When λ₀ + λ₁ < 0, overestimates the treatment effect and the type I error rate is larger than α, meaning that the test is invalid. Therefore, even with the carry-over effect, the crossover design can still control the type I error rate of a one-sided test when λ₀ + λ₁ ≥ 0.

However, a carry-over effect that does not inflate the type I error rate can have a negative impact on the power. Specifically, with possible carry-over effects and under the local alternative with a constant γ_cr > 0, the power of T_cr is

(4)

Senn (1997, Table 1) briefly notes the impact of carry-over on power but does not provide a power formula. Our Equation 4 provides a general power formula in the presence of carry-over effects, which produces results that are the same as those in Senn (1997, Table 1) (with n = 44, ). Equation 4 also implies that the required sample size to achieve 1 − β power is

For illustration, consider a test of superiority and suppose that and the Pitman asymptotic relative efficiency between the crossover and the parallel-group design is

When , that is, the carry-over effects are non-negative for the purpose of controlling type I error rate and are smaller than the treatment effects, the difference between n_pr and equals

where c is some positive constant. Therefore, in order for the crossover design to have higher efficiency than the parallel-group one, we need the carry-over effect to be small. Specifically, similar to our discussion at the end of Section 2: When , we need . When ρ = 0.3, 0.5, 0.7, we need to have 2⁻¹(λ₀ + λ₁) > 0 and less than 0.41θ_Alt, 0.5θ_Alt, and 0.61θ_Alt, respectively, so that the type I error is not inflated and the crossover design is more powerful than the parallel-group one. This small carry-over effect condition may be plausible in many scenarios because carry-over effects are usually relatively small compared to the treatment effects. Therefore, a crossover design can still be more powerful than a parallel-group design in many cases even when the carry-over effect exists. In Section S2 of the supplementary materials, we discuss how to control the type I error when λ₀ + λ₁ < 0 using a sensitivity analysis approach (Rosenbaum, 2020).

4. COVARIATE ADJUSTMENT FOR CROSSOVER TRIALS

Adjusting for prognostic baseline covariates in the analysis of randomized controlled trials is encouraged by regulatory agencies because it has high potential to improve efficiency under approximately the same minimal statistical assumptions that would be needed for unadjusted estimation (FDA, 2023). It often uses a working model between the outcomes and covariates, but its estimand is the same as when using the unadjusted method and its inference does not rely on the working model being correctly specified.

Covariate adjustment for parallel-group trials has been extensively studied recently. From the theory of semiparametrics (Robins et al., 1994; Tsiatis, 2006), Tsiatis et al. (2008) established a general class of consistent and asymptotically normal estimators for the average treatment effect. When linear working models are used, covariate adjustment using an analysis of heterogeneous covariance (ANHECOVA) working model that includes all treatment-by-covariate interaction terms can lead to guaranteed efficiency gain regardless of the model is misspecified or not (Yang and Tsiatis, 2001; Lin, 2013; Ye et al., 2022, 2023). These recent results have not been extended to crossover trials, although covariate adjustment is broadly recommended for crossover trials (Metcalfe, 2010; Mehrotra, 2014; Jemielita et al., 2016).

We consider adjusting for a baseline covariate vector measured before randomization (ie, pre-randomization covariates) using the following covariate-adjusted ANHECOVA estimator

where is the sample mean of all ’s, is the sample mean of ’s from subjects with A_i = a, Δ_i and are defined in Equation 1, and is the least squares estimator of from fitting the linear working model using subjects with A_i = a.

The following heuristics reveal why ANHECOVA does not change the estimand, often gains but never hurts efficiency even when the linear working model is wrong. As randomization balances the covariate distribution, both and estimate the same quantity and thus, is an “estimator” of zero. Hence, and correspond to the same estimand. In addition, as n → ∞, converges to in probability, regardless of the linear working model is correct or not. Hence, is asymptotically equivalent to , whose variance is

Consequently, the asymptotic variance of is no larger than that of . These results are formally stated in Theorem 3, which are all in the asymptotic sense.

Theorem 3

Under Assumption 1,

(a) , where , and .

(b) Moreover, .

Theorem 3 is proved by applying Theorem 1 and Corollary 1 in Ye et al. (2023) with Δ_i as the outcome. From Theorem 3(b), we see that the asymptotic variance of is no larger than that of , where the equality holds if and only if . This occurs, for example, when , that is, when the covariates are uncorrelated with the change in outcome Δ_i. In fact, Theorem 1 of Ye et al. (2023) implies a stronger result that has the smallest asymptotic variance among all linearly adjusted estimators of the form , where are any fixed or random vectors that have the same dimension as . A consistent estimator of is

where is the sample variance of based on subjects under A_i = a, for a = 0, 1, and is the sample covariance matrix of based on the entire sample. One can easily construct a Z-test based on the covariate-adjusted estimator , which from Theorem 3 is guaranteed to be more powerful than the unadjusted counterpart T_cr in the asymptotic sense.

In this article, we have focused on adjusting for pre-randomization covariates. As discussed above, if the covariates are related to the change in outcome Δ_i, which could happen, for example, when there exist treatment-by-covariates interactions, then adjusting for pre-randomization covariates can reduce the variability of Δ_i and improve asymptotic efficiency. This is also shown in our simulation study in Section 5. However, in crossover trials, this adjustment typically yields small to modest benefits unless the covariates strongly influence the change in outcome. Previous studies have explored adjusting for period-dependent baseline covariates, that is, covariates that are measured before treatment is given within each period (Kenward and Roger, 2010; Jones and Kenward, 2015). Yet, one important caveat is that the washout period needs to be sufficiently long to ensure that the period-dependent baselines are not influenced by previous treatment (ie, by carry-over). In instances where the carry-over effect is a concern, more sophisticated methods to adjust for period-dependent covariates should be used (eg, the longitudinal G methods [Hernán and Robins, 2020, Chapter 21]). Exploring these methods will be a direction for future research.

5. POWER CALCULATIONS

To compare the power of parallel-group and crossover design, we consider 2 situations. In Case I, we consider the following simple data-generating process for which we can calculate the power using both the formula and simulations:

In Case II, we consider a more complex data-generating process of the potential outcomes with treatment-by-covariate interactions:

In both cases, X_ij, ϵ_ik ∼ N(0, 1) for j = 1, 2, 3 and k = 1, 2, 3, 4. The treatment arm indicator A_i is Bernoulli with π₀ = π₁ = 1/2. The observed outcomes for subject i are if A_i = 1 and if A_i = 0. The observed data are . For the covariate-adjusted estimator, we adjust for . We set , θ* = 0, and .

In Case I, because of the simple data-generating process, we can directly calculate that , , , and thus and . In this case, is a 3 × 3 identity matrix and . Thus, . From Corollary 1 in Ye et al. (2023), it is also easy to calculate the asymptotic variance of the covariate-adjusted ANHECOVA estimator using only the parallel-group design as . Hence, we calculate the power based on both formulae and simulations. Figure 2 shows the type I error rate and power for 4 one-sided tests T_pr, T_{pr, adj}, T_cr, and T_{cr, adj} under Case I when and λ₀ = λ₁ = λ ∈ { − 0.1, 0, 0.1, 0.3} based on the formula in (3) and (4) and its covariate-adjusted counterpart. Note that T_{pr, adj} is based on the ANHECOVA estimator that takes the same form as but with Δ_i replaced by Y_i1. In this setting, the value of b only affects the power of T_cr but not the other 2 tests, so we present for T_cr only. In the supplementary materials, we present the type I error rate and power obtained by simulation, which are shown to agree with the power in Figure 2.

FIGURE 2.

Power curves for 4 tests under Case I calculated by formula when λ = −0.1, 0, 0.1, 0.3, and under Case I. Note that the power of T_pr, T_{pr, adj}, and T_{cr, adj} is unaffected by b.

In Figure 2, the power of T_pr and T_{pr, adj} is not affected by λ and can be used to benchmark the performance of the other tests. When θ = 0, that is, under the null hypothesis of no treatment effect, the type I error rates of T_pr and T_{pr, adj} are equal to α = 0.025. The type I error rates of T_cr and T_{cr, adj} are greater than α when λ < 0, equal to α when λ = 0, and smaller than α when λ > 0. In other words, T_cr and T_{cr, adj} can control the type I error rate when λ ≥ 0; meanwhile, T_cr and T_{cr, adj} become more and more conservative as λ grows.

When θ > 0, that is, under the alternative hypothesis, the power comparison of T_cr and T_pr depends on the sign of following the calculations at the end of Section 3.3. Specifically, with b = 0, T_cr is more powerful than T_pr when λ < 0.5θ. Hence, with no carry-over effect (λ = 0), we see that T_cr has substantial power gain compared to T_pr; with a small carry-over effect (λ = 0.1), T_cr is more powerful than T_pr when θ > 0.2; with a large carry-over effect (λ = 0.3), T_pr is more powerful than T_cr for 0 < θ ≤ 0.5. Furthermore, since a larger b leads to a larger ICC, it also results in a slightly higher power of T_cr. Lastly, adjusting for covariates that are related to the change in outcome can always increase power regardless of whether the model is correct or not. Hence, we see that the covariate-adjusted test T_{cr, adj} is always more powerful than the unadjusted test T_cr; in some cases, the power gain can be up to 17%.

Figure 3 shows the empirical type I error rate and power for Case II obtained by simulation from 10 000 repetitions. The results are similar to the results from Figure 3, and covariate adjustment can improve the power when there exist treatment-by-covariates interactions.

FIGURE 3.

Power curves for 4 tests under Case II calculated by simulations when λ = −0.1, 0, 0.1, 0.3.

6. REAL DATA EXAMPLE

6.1. Application to the REACH study

In this section, we revisit the REACH study (Nair et al., 2023) described in Section 1 and apply our methods to study the adherence difference between monthly DVR and daily oral TDF/FTC.

In the REACH study, a total of 247 participants were enrolled and randomized (1:1) to 2 treatment arms of product use: using the DVR for 6 months and switching to daily oral TDF/FTC for a second 6 months, or using the daily oral TDF/FTC for 6 months and then switching to DVR for 6 months. Participants’ baseline characteristics are summarized in Table 2.

TABLE 2.

Baseline characteristics by randomized crossover sequence in the REACH study.

	DVR first	TDF/FTC first	Overall
	(n = 124)	(n = 123)	(n = 247)
Age in years
Median (IQR)	18 (17-19)	18 (17-19)	18 (17-19)
Site location
South Africa—Cape Town	30 (24.2%)	30 (24.4%)	60 (24.3%)
South Africa—Johannesburg	34 (27.4%)	33 (26.8%)	67 (27.1%)
Uganda—Kampala	30 (24.2%)	30 (24.4%)	60 (24.3%)
Zimbabwe—Harare	30 (24.2%)	30 (24.4%)	60 (24.3%)
Marital status
Single	106 (85.5%)	108 (87.8%)	214 (86.6%)
Married/cohabiting	16 (12.9%)	14 (11.4%)	30 (12.1%)
Separated/divorced	2 (1.6%)	1 (0.8%)	3 (1.2%)
Highest level of education
Primary	15 (12.3%)	18 (14.6%)	33 (13.5%)
Secondary	92 (75.4%)	97 (78.9%)	189 (77.1%)
Higher	15 (12.3%)	8 (6.5%)	23 (9.4%)
Earns own income	22 (17.7%)	31 (25.2%)	53 (21.5%)
Ever pregnant	52 (41.9%)	47 (38.2%)	99 (40.1%)
Currently has a sex partner	111 (89.5%)	108 (87.8%)	219 (88.7%)
Worry about HIV infection	86 (39.4%)	82 (66.7%)	168 (68.0%)
Diagnosed with STI	40 (32.3%)	47 (38.2%)	87 (35.2%)
Syphilis	3 (2.4%)	3 (2.4%)	6 (2.4%)
Trichomoniasis	8 (6.5%)	5 (4.1%)	13 (5.3%)
Gonorrhea	13 (10.5%)	8 (6.5%)	21 (8.5%)
Chlamydia	30 (24.2%)	41 (33.3%)	71 (28.7%)
CES depression scale
≥12 (indicative of depression)	28 (22.6%)	38 (30.9%)	66 (26.7%)
<12	89 (71.8%)	78 (63.4%)	167 (67.6%)
Missing	7 (5.6%)	7 (5.7%)	14 (5.7%)
Alcohol use disorder (AUDIT-C score)
≥3 (indicative of alcohol use disorder)	50 (40.3%)	47 (38.2%)	97 (39.3%)
<3	74 (59.7%)	75 (61.0%)	149 (60.3%)
Missing	0 (0.0%)	1 (0.8%)	1 (0.4%)
Preference for products
DVR	51 (41.1%)	43 (35.0%)	94 (38.1%)
oral TDF/FTC	43 (34.7%)	57 (46.3%)	100 (40.5%)
Equal preference	29 (23.4%)	22 (17.9%)	51 (20.6%)
Did not answer	1 (0.8%)	1 (0.8%)	2 (0.8%)

Data are presented as median (IQR) or n (%).

AUDIT-C, alcohol use disorders identification test-concise; CES, Center for epidemologic studies; IQR, Interquartile range; STI, sexually transmitted infection.

We apply our method to compare adherence to monthly DVR and daily oral TDF/FTC. The binary adherence endpoint is defined as high use at the end of period 1 and period 2, with high use of daily oral TDF/FTC defined as tenofovir-diphosphate concentrations greater than or equal to 700 fmol/punch (associated with taking an average of 4 or more tablets per week in the previous month), and high use of monthly DVR defined as greater than or equal to 4 mg dapivirine released from the returned ring (continuous use for 28 days in the previous month). These adherence definitions are widely accepted in the literature (Nair et al., 2023). It is important to note, however, that the definitions require continuous daily usage of the ring, but not for oral TDF/FTC. For covariate adjustment, we consider baseline covariates that are likely to be associated with adherence, including age, site location, preference for either product, depression, alcohol use disorder, sexually transmitted infection, and participants’ level of worry about HIV infection. We use the single imputation method to handle the missing baseline covariates (Zhao and Ding, 2022).

To discuss the impact of potential carry-over effects in the analysis of this trial, we first note that for the DVR first arm, using unadjusted sample means, the estimated adherence rate to DVR at period 1 was 55.6% and to TDF/FTC was 44.4% at period 2; for the TDF/FTC first arm, the estimated adherence rate to TDF/FTC at period 1 was 48.0% and to DVR was 52.8% at period 2. Hence, the average adherence difference at period 1 is 55.6%–48.0% = 7.6%, and average adherence difference at period 2 is 52.8%–44.4% = 8.4%. This small difference between the treatment effect at periods 1 and 2 may be attributed to a larger treatment effect in period 2, and/or a negative carry-over effect (λ₁) for the reason discussed in Section 1. However, because the difference is small, even if the carry-over effect exists, it is unlikely to substantially alter the result.

Table 3 presents both the parallel and crossover estimates and their covariate-adjusted counterparts, alongside the standard errors and 95% CIs. The parallel estimates use data from period 1 only. Estimates from all methods are similar, indicating that adherence to DVR is about 7.7%–8.5% higher compared to daily oral TDF/FTC. The SEs of the crossover estimators are smaller compared to the SEs of the parallel estimators, demonstrating that the crossover design is more efficient. Adjusting for covariates leads to slightly larger estimates and smaller standard errors. All the 95% CIs (except for the covariate-adjusted crossover one) cover 0, suggesting that the difference is not statistically significant.

TABLE 3.

Covariates-adjusted and unadjusted parallel and crossover estimators for the average treatment effect of DVR on adherence compared to daily oral TDF/FTC, with standard errors and 95% CIs.

Type	Mean	SE	95% CI
Parallel, unadjusted	0.077	0.064	(−0.048, 0.202)
Parallel, adjusted	0.082	0.062	(−0.040, 0.204)
Crossover, unadjusted	0.081	0.043	(−0.003, 0.165)
Crossover, adjusted	0.085	0.042	(0.003, 0.168)

6.2. Using the REACH study to design a hypothetical trial

Despite being highly effective for HIV prevention, adherence to daily oral TDF/FTC is low among women (Celum et al., 2019). The dual prevention pill (DPP), a daily oral pill combining oral contraceptives and TDF/FTC, has the potential to increase women’s adherence to daily oral TDF/FTC (Friedland et al., 2021). In this section, using data from the REACH study, we use simulations to evaluate the power trade-off between the crossover design and parallel-group design in a hypothetical trial comparing adherence to a single daily DPP versus adherence to 2 daily pills (1 pill of TDF/FTC and 1 pill of oral contraceptive). The central hypothesis is that a DPP regime can increase women’s adherence to TDF/FTC.

Suppose adolescent girls and young women with baseline characteristics similar to those in the REACH study are recruited for this trial, and we use the same design as described in Section 6.1 to compare DPP and 2 daily pills. In this example, we anticipate a non-negative carry-over effect primarily because the DPP is expected to increase adherence in the first period, and those who adhere consistently in the first period tend to develop a routine or habit around taking daily oral pills, which may positively influence their adherence in the second period. From the results in Section 3, using a crossover design with the non-negative carry-over effect does not lead to type I error inflation.

Our simulations are based on the data of 123 participants who took TDF/FTC first in the REACH study. In the simulation study, we set π₁ = π₀ = 0.5, , where θ ∈ {0.05, 0.08, 0.10, 0.15} for different treatment effects. The ICC (ie, ρ) is estimated to be ρ = 0.39 from the 123 participants in the REACH study. Under the simple scenario described at the end of Section 3, when , the crossover design is less powerful than the parallel one. So here we set λ₁ = λ₀ = λ ∈ {0, 0.25θ, 0.45θ} for different carry-over effects. We consider 4 test statistics: T_pr, T_{pr, adj}, T_cr, T_{cr, adj}. The simulation process involves the following steps:

1. Denote the baseline covariates and the adherence variable at week 24 from the REACH study as , with N = 123. The average of is 0.48. Fit a logistic regression model for the probability of adherence to TDF/FTC using age, site location, STI status, depression, alcohol use disorder, worry about HIV infection and baseline preferences for the products. Single imputation is applied for all missing values as described in Section 6.1. Denote the fitted model as , where .

2. Generate , where satisfies .

3. Generate from and , where s₀ is chosen (as a function of ) to ensure that and ; see details in Section S3.3 of the supplementary materials.

4. Generate from and , where s₁ is chosen (as a function of ) to ensure that and . By now, we have obtained the covariates and 4 potential outcomes for 123 individuals.

5. A random sample of size n is drawn from the 123 individuals’ covariates and potential outcomes with replacement. Each of the obtained n subjects is assigned randomly to A_i = 0 or 1 with equal probability. For A_i = 1, and ; for A_i = 0, and . Hence, we obtain the observed data , and calculate the test statistics T_pr, T_{pr, adj}, T_cr, T_{cr, adj}.

6. Steps (2)-(5) are repeated 2000 times for each θ and λ, from which we obtain the empirical powers of the test statistics.

Table 4 shows the empirical powers for T_pr, T_{pr, adj}, T_cr, T_{cr, adj} for different θ and λ based on 2000 simulations. The sample sizes n = 325, 680, and 1900 are determined such that Power_cr is close to 80% when θ = 0.10. From Table 4, first, we can see that the empirical power increases when θ increases. Second, under either the parallel or crossover design, covariate adjustment always leads to a larger power. Third, when λ = 0, the crossover tests have considerably larger power than the parallel-group tests. When λ = 0.25θ, the crossover tests still have larger power, but the power difference is not as pronounced. When λ = 0.45θ, the crossover tests become slightly less powerful than the parallel-group tests.

TABLE 4.

Empirical power of T_pr, T_{pr, adj}, T_cr, and T_{cr, adj} for a hypothetical trial using data from the REACH study. For each λ, the sample size is determined such that Power_cr is close to 80% when θ = 0.10.

n	λ	θ	Powerpr	Powerpr, adj	Powercr	Powercr, adj
325	0	0.05	0.214	0.236	0.423	0.446
		0.08	0.342	0.379	0.670	0.681
		0.10	0.439	0.484	0.797	0.818
		0.15	0.738	0.770	0.973	0.978
680	0.25θ	0.05	0.324	0.354	0.452	0.463
		0.08	0.534	0.570	0.680	0.6701
		0.10	0.679	0.720	0.799	0.808
		0.15	0.905	0.923	0.959	0.961
1900	0.45θ	0.05	0.558	0.579	0.523	0.528
		0.08	0.778	0.789	0.684	0.688
		0.10	0.876	0.889	0.791	0.797
		0.15	0.988	0.989	0.929	0.935

7. DISCUSSION

The crossover is an efficient trial design that uses participants as their own controls. The carry-over effect, especially the behavioral carry-over effect, is an outstanding concern because it can bias the estimation of the treatment effect. Using a potential outcomes framework and minimal statistical assumptions, we investigate the impact of the carry-over effect in a 2-treatment 2-period crossover trial. Our results provide a clear characterization of how the carry-over effect influences estimation bias, as well as the type I error and power of tests. When the carry-over effect λ₁ + λ₀ is negative, the basic estimator overestimates the treatment effect, which can inflate the type I error of one-sided tests. Conversely, when the carry-over effect λ₁ + λ₀ is positive, the basic estimator underestimates the treatment effect, which does not inflate the type I error of one-sided tests but negatively impacts the power. Furthermore, when λ₁ + λ₀ is positive but relatively small compared to the treatment effect, the crossover design can still be more powerful than the parallel-group design. We can further apply covariate adjustment in crossover trials for guaranteed efficiency gain. All the methods in this article can be implemented using the RobinCar package in R (Ye et al., 2023).

We discuss our results in the context of 2 studies and demonstrate that the anticipated carry-over effect can be either non-positive or non-negative: one is the REACH study evaluating monthly DVR and daily oral (TDF/FTC), the other is a hypothetical future trial evaluating a single daily DPP and 2 daily pills. Through these 2 example studies, our key message is that while the crossover design is an efficient design, the carry-over effect needs to be carefully considered in trial planning and execution.

In this article, our primary focus has been on the power trade-off between the parallel-group design and the crossover design, particularly in the context of carry-over effects. However, this is just one among various practical considerations. First, a crossover trial typically requires a longer follow-up time, about twice as long as a parallel-group design, and this duration may be even longer if a washout period is incorporated to reduce carry-over effects (Senn, 2002). The longer follow-up time can increase costs and the risk of dropout. Second, the longer commitment and the need to switch treatments in a crossover design might discourage participants, potentially slowing recruitment. On the other hand, the opportunity to try multiple treatments in one study can make the crossover design more appealing to participants in certain scenarios (Diener et al., 2019). Meanwhile, in some trials, experiencing multiple treatments is essential for the trial objective. For example, in the REACH study, the participants need to experience both treatments before entering the choice period, which allows for the study of their preference between the 2 treatments.

Supplementary Material

ujae023_Supplemental_Files

Web Appendices referenced in Sections 2-6 are available with this paper at the Biometrics website on Oxford Academic. Code necessary to reproduce simulation and application results can be found in the supplementary materials.

FUNDING

This work was supported by the National Institute of Allergy and Infectious Diseases [NIAID 5 UM1 AI068617].

CONFLICT OF INTEREST

None declared.

DATA AVAILABILITY

The data that support the findings in this paper are available from the Microbicide Trials Network. Restrictions apply to the availability of these data, which were used under license in this paper. Data are available from the authors with the permission of the Microbicide Trials Network.