Start with the Target Trial Protocol, Then Follow the Roadmap for Causal Inference

Before taking flight, pilots complete a preflight checklist. Before starting surgery, operating room staff complete a presurgical checklist. These checklists synthesize the knowledge of a field into a series of steps to be considered every time a job is performed. What about when researchers want to infer causality?

A century’s worth of literature regarding the design and analysis of observational studies is available, yet common guidance is not always followed. Bykov et al.¹ reviewed a random sample of 75 observational studies on cardiovascular disease, diabetes, or osteoporosis and found that 95% “had at least one avoidable methodological issue known to incur bias.” It is, thus, critical to distill this literature into concrete instructions so that best practices are consistently implemented.

Many useful frameworks operationalize aspects of the causal and statistical inference literature into steps for researchers to follow.^2–12 Target trial emulation is a popular approach that has led to high-profile studies in recent years.^13–15 The basic structure shared by target trial emulation studies follows Hernán and Robins’⁷ statement that “at the very least, we need to specify the following key components of the [target trial] protocol: eligibility criteria, start and end of follow-up, treatment strategies, outcomes of interest, causal contrast, and data analysis plan.”¹⁶ The assignment procedure of the target trial is generally included, as well.⁷

Pearce and Vandenbroucke¹⁷ raised concerns about certain studies that use target trial emulation protocol components. First, they suggest that target trial emulation seems to focus on mimicking a conditionally randomized controlled trial, as opposed to considering a variety of study designs and identification assumptions. Second, target trial emulation is sometimes paired with a matching approach for statistical estimation; matching can change the target population and is often less efficient than other methods.^18,19 Third, after emulating a target trial, one may be tempted to over-interpret the results; we still have an observational study, subject to multiple potential biases, including intractable confounding. Ultimately, Pearce and Vandenbroucke¹⁷ conclude that target trial emulation is not the optimal starting point for causal inference.

In contrast, we believe the target trial emulation protocol is a useful starting place for analyses seeking to infer causality. In other words, we agree with Hernán and Robins’⁷ statement: “specifying the protocol of the target trial is a useful device to clarify the causal question of interest.”¹⁶ However, we disagree with their assertion: “once the causal question is stated with little ambiguity, study design and data analysis flow naturally.”¹⁶ Instead, a given research question can lead us down many potential paths, including alternative study designs, identification assumptions, estimation approaches, and interpretations. What guidance should investigators follow after they have defined their causal question?

The Roadmap for Causal and Statistical Inference^8,9,20–24 (originally described in Petersen and van der Laan⁹ and hereafter, the Roadmap) provides step-by-step directions to aid in the design, analysis, and interpretation of studies—randomized or observational. The Roadmap

1. can accommodate any study design and all identification assumptions,

2. emphasizes the importance of selecting the best statistical estimator for a given problem based on theoretical and finite sample properties, and

3. includes steps to explicitly protect against over-interpretation, while providing a path forward when we cannot estimate a causal effect.

Below, we describe how the Roadmap (summarized in the Table) accomplishes these goals.

TABLE.

The Steps of the Roadmap^8,9,20-24* with Example Questions to Ask

The Roadmap for Causal and Statistical Inference
1. Specify the research question, including the target population, exposure(s), outcome(s), time period, and context of interest. ▪ What do we want to learn from data that have been collected or will be collected? To whom do we want to apply the results?
2. Specify a causal model, such as a directed acyclic graph,10 to describe relationships between variables. ▪ Are there unmeasured confounders or time-dependent confounders? Are the outcomes missing or censored?
3. Define the causal parameter of interest with counterfactual outcomes. ▪ What hypothetical change to the causal model, even if impossible, would we make to generate counterfactuals and answer our research question? How do we want to summarize the distributions of counterfactual outcomes?
4. Describe the observed data and the statistical model. ▪ What data did we or will we actually observe? Are there functional form assumptions or can the relationship between the outcome(s), exposure(s), and adjustment variables take any form?
5. Assess identifiability. ▪ What modifications can we make to reduce the causal gap?
6. Define the statistical parameter. ▪ What function of the observed data are we aiming to estimate?
7. Choose and implement a statistical estimator based on statistical properties; obtain 95% confidence intervals. ▪ What are the theoretical properties of potential estimators (e.g., robustness)? How do they perform in finite sample simulations according to objective criteria?
8. Conduct sensitivity analyses. ▪ What can existing evidence tell us about plausible magnitudes of the causal gap?
9. Interpret the results, accounting for the prior steps. ▪ Have we estimated an association or a causal effect? What are the real-world implications?

^*The order in which the steps are presented differs in different versions of the Roadmap^8,9,20–24 because these steps inform one another and are generally specified through an iterative process.

FIRST, CHOOSE YOUR IDEAL DESTINATION

Both the Roadmap (Step 1) and target trial emulation prompt us to specify the question that we would ideally like to answer. A strength of target trial emulation is its use of language familiar to researchers from a wide variety of backgrounds. This step tells us our ideal destination. Given the myriad options for study design, analysis, and interpretation, how should we plot a course from our question to an answer with real-world relevance? How will we evaluate whether we landed on the island of causality, and what do we do if we find ourselves in the sea of statistical associations? The remainder of the Roadmap’s steps help us to answer these questions.

“MIND THE [CAUSAL] GAP!”

Step 2 of the Roadmap is to specify a causal model, such as a directed acyclic graph,^10,25 to describe existing knowledge and uncertainty about the relationships between the covariates (baseline or time-varying), exposures of interest, censoring, and outcomes. By intervening on this causal model, we then generate counterfactual outcomes corresponding to our ideal, but often impossible, experiment (Step 3). After describing which variables are actually measured (Step 4), the Roadmap’s identification step requires us to transparently state and critically evaluate the assumptions needed to infer causality (Step 5).

Most commonly, these assumptions include no unmeasured confounding (i.e., conditional exchangeability), missing/censoring at random, and sufficient data support (i.e., positivity). The Roadmap also accommodates alternative approaches such as difference-in-differences, whose alternate set of identification assumptions must be explicitly discussed; see, for example, Weber et al.²⁶ Although Hernán and Robins’⁷ cover causal models and various identification assumptions at length in their book Causal Inference: What If,¹⁶ the “assignment procedures” protocol component of the target trial emulation focuses on conditional exchangeability. In contrast, the Roadmap explicitly requires researchers to delineate all necessary identification assumptions every time they aim to estimate a causal effect. Doing so can minimize the potential for severely misleading inferences arising from violations of assumptions related to missingness or data support, among others.^27–30 Additionally, by transparently stating the identification assumptions (Step 5) and including a causal graph (Step 2), the Roadmap empowers subject-matter experts to evaluate whether these assumptions are reasonable.

TRY TO BRIDGE THE GAP

Without identification, the true value of the statistical parameter that we will estimate from the observed data will not be equivalent to our wished-for causal parameter; in other words, there will be a so-called causal gap.²¹ When working with observational data, the Roadmap’s identification step generally leads us to conclude that we are only able to estimate an association, rather than the desired causal effect. What should we do when our causal gap is nonzero? Rudolph and Keyes³¹ review approaches to improve the plausibility of the identification assumptions, including changing the research question, definition of the exposure or outcome, or target population.^7,9,22,27,28 Roadmap Steps 1–5 should be viewed as an iterative process.

RED LIGHT [CHOOSE AN ESTIMATOR]; GREEN LIGHT [ESTIMATE!]

After taking these steps, we are still usually stuck with a causal gap and the need to provide timely and actionable answers to our research questions. The Roadmap’s Step 6 and the “analysis plan” protocol component of target trial emulation guide us to choose a statistical parameter that is motivated by our underlying question.^7–9,16 This parameter is often a complex function of the observed data distribution—even in trials due to censoring or noncompliance—and is rarely equal to a coefficient in a regression model.^22,32,33

An additional benefit of the Roadmap is that it guides us to choose an estimation approach based on statistical properties (Step 7). First, the estimator should have a low bias and variance relative to available choices.⁸ Not all target trial emulation studies use matching, but the Roadmap would generally lead away from matching due to inefficiency and the risk of changing the target population. Second, the estimator should not rely on unrealistic functional form assumptions; machine learning may help minimize bias due to reliance on misspecified parametric regressions, but must be incorporated in a manner that yields valid 95% confidence intervals.^8,34 For these reasons, the Roadmap often leads us to doubly robust estimators, such as the targeted maximum likelihood estimator.^8,35

Of course, there are many possible implementations of a given approach (e.g., targeted maximum likelihood estimator with different Super Learner libraries or sample-splitting schemes). Candidate algorithms should be compared using finite sample simulations reflecting the challenges in the real data. See, for example, Montoya et al.³⁶, who based their simulation on the empirical confounder distribution but with blinded outcomes. The process of using simulations to select an estimation approach and its precise implementation, according to prespecified criteria, is integral when developing the statistical analysis plan. Afterward, we have the green light to estimate!

LOOK BEFORE YOU LEAP

Once we have a point estimate and confidence intervals, how should we interpret our results? As discussed above, if the causal gap is nonzero, then we must be transparent that we have estimated a statistical association, not a causal effect. Yet just as there is a spectrum from a perfectly conducted randomized trial to a messy observational study, there is a spectrum of possible interpretations, each with value to science.⁹

In their inaugural Roadmap paper, Petersen and van der Laan⁹ provide a hierarchy of interpretations, ranging from purely statistical to replicating the results of a hypothetical trial. Importantly, they highlight that increasing the strength of our interpretations requires increasing the strength of our assumptions. Specifically, concluding that we have emulated a randomized trial requires (1) the statistical estimator has negligible bias and its variance is well-estimated, (2) the causal gap is zero, (3) the intervention is feasible and applicable to a real-world population, and (4) the intervention could have been randomized to that population.⁹ Roadmap Steps 1–7 help us to select and implement an estimator that supports valid statistical inference (#1) and to evaluate whether there is a causal gap (#2). The goal of replicating the results of a randomized trial (#3 and 4) may or may not be desirable, depending on the context.

Like a real map, the Roadmap has evolved over time and now includes sensitivity analyses (Step 8) to guide interpretation when the causal gap is not zero.²¹ We emphasize that the difference in adjusted and unadjusted estimates does not reveal whether the adjusted approach has accounted for all confounders.¹⁰ To assess potential violations of untestable causal identification assumptions (e.g., residual confounding), we can consider available evidence regarding the plausible magnitude and direction of the causal gap.^37,38 Such sensitivity analyses are also applicable in studies using matching.

Completing Roadmap Steps 1–8 helps us to interpret our results appropriately (Step 9). First, the plausible causal gap could be small enough to allow for a close approximation of a causal effect. See, for example, Skeem et al.’s³⁹ study of alternate probation protocols for people with mental illness. Second, the confidence interval bounds could be far enough from the null relative to the plausible magnitude of the causal gap that researchers may conclude with reasonable certainty that there is a causal effect. Examples include Cornfield et al.’s⁴⁰ assessment that smoking causes lung cancer and Diaz et al.’s³⁷ assessment that Nifurtimox is effective in treating Chagas disease.

In many scenarios, however, we cannot conclude that we have estimated a causal effect. Nonetheless, the estimated association may still provide useful real-world information.^31,41,42 For example, even if we cannot prove that state-level mask mandates caused lower mortality rates from COVID-19, knowledge that the mortality rate, adjusted for measured state-level confounders, was lower with early versus delayed implementation of mask mandates is still valuable.⁴² Altogether, the Roadmap steps of specifying a causal model, discussing the plausibility of identification assumptions, conducting sensitivity analyses, and ensuring that the interpretation is consistent with these prior steps help to generate estimates with real-world implications.

CONCLUSION

No single framework can substitute for the full value of the literature on causal and statistical inference. Yet just as checklists have improved the quality of services in aviation and medicine,^43,44 we believe in the power of the Roadmap to improve the design and analysis of observational studies seeking to infer causality. Of course, any tool could be misinterpreted or instill false confidence in some users. Yet, the Roadmap supports structured choices that researchers are already making, including picking a study design, selecting adjustment sets, choosing and implementing an estimator, and deciding how to interpret results. While the Roadmap cannot prevent all missteps, it makes the reasoning behind these necessary choices more transparent, empowering readers of Roadmap papers to make informed interpretations of the study results in both randomized trials and observational studies.^9,26,29,30

ABOUT THE AUTHORS

Lauren Eyler Dang, MD, PhD recently completed her PhD in Biostatistics at the University of California, Berkeley. She will soon join the National Institute of Allergy and Infectious Diseases Biostatistics Research Branch as a Mathematical Statistician. Laura B. Balzer, PhD, is an Associate Professor of Biostatistics at the University of California, Berkeley. Both Lauren and Laura love developing, evaluating, applying, and teaching methods for causal inference.