Defining and Identifying Per-Protocol Effects in Randomized Trials

Abstract

In trials with non-compliance to assigned treatment, researchers might be interested in estimating a per-protocol effect – a comparison of two counterfactual outcomes defined by treatment assignment and (often time-varying) compliance with a well-defined treatment protocol. Here, we provide a general counterfactual definition of a per-protocol effect and discuss examples of per-protocol effects that are of either substantive or methodologic interest. In doing so, we seek to make more concrete what per-protocol effects are and highlight that one can estimate per-protocol effects that are more than just a comparison of always taking treatment in two distinct treatment arms. We then discuss one set of identifiability conditions that allow for identification of a causal per-protocol effect, highlighting some potential violations of those conditions that might arise when estimating per-protocol effects.

Introduction

In randomized controlled trials (RCTs) with non-compliance, the intention-to-treat effect cannot be interpreted as an estimate of treatment efficacy.¹ In such cases, researchers may wish to estimate effects that would have been observed if study participants had (possibly contrary to fact) followed a pre-designated protocol. These per-protocol effects require careful consideration and handling of the potentially time-varying causes of non-compliance, as has been discussed in previous papers.^1–5

However, prior work on per-protocol effects has rarely (if ever) explicitly stated the counterfactual definition of the target parameter, which could lead to some confusion regarding the definition of a per-protocol effect. There are also different target per-protocol parameters that can be used to answer different research questions or provide insight regarding the validity of our models or causal assumptions. Thus, we here provide a general counterfactual definition and describe several example per-protocol effects. We then comment on one sufficient set of conditions needed to identify these effects, as these have also not been explicitly discussed in relation to per-protocol effects.

Defining per-protocol effects

Let R denote randomized treatment. Participants are followed until outcome Y or end of follow-up. Compliance is assessed in time intervals (J = 1,…,m) where C_j = 1 means the participant was compliant for time-point j. Looking at compliance across follow-up, we also define compliance history $\overline{C}$ for each participant. While our example is general, note that estimating per-protocol effects requires clearly defined compliance protocols that directly correlate with the research question of interest and with substantive knowledge. Other work has explored in detail best practices for defining protocols.⁵

We define a per-protocol effect as:

$$E\left[Y^{r=p_1,\overline{c}={\gamma }_1}\right]-E\left[Y^{r=p_2,\overline{c}={\gamma }_2}\right]$$

Where r = p is a given randomized treatment and $\overline{c}=\gamma$ is a given compliance history. Suppose we were interested in two trial arms, R={0,1} and $\overline{C}$ defined by always- or never-complying with the same protocol regardless of arm. We can then define six basic per-protocol effects (Table), three of which we discuss here.

Table.

Summary of six basic per-protocol parameters given randomization to a binary treatment and constant compliance or noncompliance across follow-up

Parameters	Counterfactual Notation
A	$E\left[Y^{r=1,\overline{c}=1}\right]-E\left[Y^{r=0,\overline{c}=1}\right]$
B	$E\left[Y^{r=1,\overline{c}=1}\right]-E\left[Y^{r=0,\overline{c}=0}\right]$
C	$E\left[Y^{r=0,\overline{c}=1}\right]-E\left[Y^{r=0,\overline{c}=0}\right]$
D	$E\left[Y^{r=1,\overline{c}=1}\right]-E\left[Y^{r=1,\overline{c}=0}\right]$
E	$E\left[Y^{r=0,\overline{c}=1}\right]-E\left[Y^{r=1,\overline{c}=0}\right]$
F	$E\left[Y^{r=1,\overline{c}=0}\right]-E\left[Y^{r=0,\overline{c}=0}\right]$

Under relevant identifiability conditions (listed below), contrast A is the effect of treatment relative to comparator under full compliance to the protocol and will most closely reflect the effect of the treatment’s active ingredient. If the protocol is static and requires taking treatment every day, A would be the maximum possible effect. If a dynamic protocol is specified (e.g., take treatment until some clinical event occurs), this effect will best approximate the biologic effect over the relevant time scale. A is arguably the effect of primary interest in many settings and has been the target parameter in most modern per-protocol analyses.^1,6

It is additionally possible to use the other parameters to triangulate this effect.^7,8 Subtracting C from B yields A due to cancellation of the $E\left[Y^{r=0,\overline{c}=0}\right]$ term, as such:

$$\left\{E\left[Y^{r=1,\overline{c}=1}\right]-E\left[Y^{r=0,\overline{c}=0}\right]\right\}-\left\{E\left[Y^{r=0,\overline{c}=1}\right]-E\left[Y^{r=0,\overline{c}=0}\right]\right\}=E\left[Y^{r=1,\overline{c}=1}\right]-E\left[Y^{r=0,\overline{c}=1}\right]$$

$$B-C=A$$

Similar triangulation can be obtained with D and E. The benefit of triangulating A in this way is methodologic. If the results from taking B – C and D – E were reasonably similar to A (acknowledging that random error might make them non-identical), we might feel reassurance that we had specified our statistical or causal models correctly. However, if estimates differ, this signals potential problems that should be explored further, although we cannot ascertain the exact source of error.

Contrast B is the joint effect of being assigned to treatment and always-complying with the protocol, relative to assignment to the comparator and never-complying. This effect will capture a combination of the treatment’s biologic effect and the effect of protocol compliance. B could be used to assess the maximum benefit under various protocols, perhaps by varying the protocol of the treatment from “take every day for all of follow-up” to “take every day for a specific time period” to “take x times per week.” Unlike A, this effect includes the impact of being engaged with a treatment protocol, which might be valuable to understand before applying a protocol in a real-world setting. Comparing B to A could reveal whether the act of staying compliant with a given protocol leads to additional benefit (or harm) beyond that seen from the biologic effect alone.

F is the effect of being assigned to treatment and never-complying with the treatment protocol versus being assigned to comparator and never-complying with the comparator protocol and is useful as a tool for model validation. In a study where one expects any sub-optimal compliance to mean the participant will experience no effect of treatment or comparator, F should be null (or close to null, given the possibility of random error). While previous studies have used contrast C for model validation,^9,10 a non-null F could also indicate problems with the statistical or causal models that may be undermining the validity of other effects estimates.

For information on how to estimate per-protocol effects, we point readers toward several published papers, including Lodi, Cain, Murray, and Toh.^6,9,11,12 While these papers have focused on estimating contrast A, the methods could be used to estimate any of the contrasts.

Identifying per-protocol effects

One sufficient set of conditions to identify a causal effect is exchangeability, positivity, and counterfactual consistency.

Exchangeability (specifically, conditional exchangeability) requires the potential outcomes $Y^{r,\overline{c}}$ to be independent of observed treatment and compliance at each time-point, conditional on compliance history prior to that time point and a set of covariates L representing confounder history.^13–16 Any variables that affect compliance and the outcome should be included in L. Particular consideration should be given to whether L will differ by treatment arm or protocol.

Positivity requires the probability of being exposed or unexposed, given L, be bounded away from 0 and one; we must here consider exposure defined by randomization and compliance. If the per-protocol effect of interest was the comparison of a given treatment under always-complying versus never-complying, it is possible that there would be very few individuals who were assigned to that treatment who genuinely never took the treatment according to the rule. Another challenge to positivity could arise because a large number of participants in a given arm choose to no longer comply with the treatment because of side effects (typically included in L). This is one reason to specify a protocol in which a participant is still compliant if they stopped treatment after experiencing a side effect. However, this decision should be dictated by substantive concerns – not merely because positivity is violated.

Counterfactual consistency requires that any variation in how participants did or did not comply with the protocol is irrelevant for the effect on the outcome.¹⁷ Such variations could include taking treatment 7 days instead of 5 days or taking treatment with water, fruit juice, or a caffeinated beverage. Consistency might be particularly difficult to assume when the comparator group is “standard of care,” which could be defined differently from patient to patient.

Beyond the above, we typically also assume no model misspecification. In RCTs with noncompliance, data could be complex, due to the presence of numerous time points and a large set L. In such cases, researchers often use parametric models, which require strict, often unrealistic model form assumptions. Indeed, the primary methods that have been used to estimate per-protocol effects, namely inverse probability weighting and g-computation,^{1,6,11,15,18–21} generally require the use of parametric models. Unlike the above, though, violations of correct model specification are a statistical, rather than causal, concern.²²

Any of these assumptions may not be credible for a given trial, and there are a number of practical limitations which could preclude one from estimating or identifying the described per-protocol effects.⁵ For instance, insufficient samples of continuously non-adherent participants could make estimating parameter F impractical. Small sample sizes in general could mean one would be underpowered to detect any difference between parameter A or one of its triangulation parameters – not to mention the difficulty in assuming positivity. Additionally, one might lack the data one would need to identify the effect of interest. In particular, a trial might not measure all the confounders of the relationship between adherence and the outcome, and adherence itself may be poorly measured. Such practical limitations must be considered prior to estimating any per-protocol effect.

Regardless, these conditions (or a different sufficient set)²³ are necessary to identify per-protocol effects. While most of the assumptions are not testable using observed data, sensitivity analyses can be conducted,^24,25 and there exist estimators that can relax some of these assumptions. For example, the positivity assumption is not required for incremental propensity score effects,²⁶ while machine learning can be used with double robust estimators to relax parametric modeling assumptions.^8,27–30

Discussion

When assessing the role of compliance to assigned treatment in an RCT, researchers can define a wide range of protocols and per-protocol effects. We here provided potential contrasts that might be of substantive or methodologic interest and discussed several important considerations when attempting to identify per-protocol effects. The key piece underlying per-protocol effect estimation is the specification of the protocol, which should primarily be based on the research question. All other considerations, including the contrast(s) of interest, estimation approach, and ability to meet identifiability conditions, follow from the chosen protocol.