Understanding HIV Program Effects: A Structural Approach to Context using the Transportability Framework

Abstract

Background:

Implementation science focuses on evaluating strategies for delivering evidence-based interventions to improve HIV prevention and treatment. The effectiveness of these implementation strategies is often context-dependent and reconciling the desire to produce generalizable knowledge in the face of these contextual interventions is a central challenge for implementation science researchers.

Methods:

We provide an overview of causal transportability theory and conceptualize context under this framework. We review how causal graphs can be used to illustrate the assumptions necessary to apply the results of a study to a new context, and we illustrate this approach using an example of a community adherence group intervention that aims to improve retention in HIV care. Finally, we discuss several key insights highlighted by transportability theory that are relevant to implementation science researchers.

Results:

By adopting causal transportability to consider how context may affect the success of an implementation strategy, researchers can formally diagnose when the results of a study are likely to generalize to a given setting. Moreover, selection diagrams can highlight what additional measurements would be needed in a target population in order to estimate the effect of an implementation strategy in that target population without having to repeat the initial study.

Conclusion:

Transportability translates intuition about context-dependent interventions and external validity into actionable and testable insight.

Globally, health systems are investing in a range of innovations to improve HIV services, but rigorous scientific evaluation of these strategies must contend with the fact that context -- sociopolitical, organizational, and community characteristics -- often impacts the effectiveness of these strategies. HIV self-testing, for example, aims to expand testing coverage among hard-to-reach populations, but its success varies across different key populations at risk of HIV infection.¹ Similarly, the effectiveness of differentiated service delivery models, which seek to improve the accessibility and quality of HIV treatment, depends on features of the health systems, clinics, and communities in which they are implemented.^2,3 For example, community adherence groups (CAGs) are designed to reduce clinic congestion and improve retention in care by allowing members of a group to take turns visiting the clinic to collect medications for the whole group.^4,5 CAGs are effective in some settings but not others, and these differences are usually attributed to “context”.⁶ Yet, if every implementation strategy is context-dependent, is the scientific task of producing generalizable knowledge through implementation science research impossible?

The broader literature in implementation science expresses an unresolved tension between the notion of context and generalizability. In a review of definitions of implementation science in HIV, Odeny and colleagues found that 50% of definitions emphasized context, while another 50% emphasized the importance of generalizability in implementation research.⁷ A later review found 17 distinct frameworks that highlight the role of context in implementation research.⁸ For example, the Consolidated Framework for Implementation Research defines context as the “broad scope of circumstances and characteristics” that affect an implementation effort.⁹ The importance of context for implementation science is widely appreciated, but there is little agreement on what exactly comprises context or how we might produce generalizable knowledge across contexts.⁸ For implementation science research to be useful, a coherent approach that reconciles producing generalizable knowledge in the face of context-dependent effectiveness is crucial.

Recent methodological innovations in formal causal inference offer approaches that could be useful for advancing the implementation research agenda and bringing an analytical framework to context. The causal transportability theory developed by Pearl and Bareinboim¹⁰ is a mathematically-grounded framework used to consider how effects observed in one setting might be applied to another. Rather than describing context as a singular overarching challenge, the framework focuses on the specific contextual factors that are likely to impact a given intervention. In doing so, it lays the groundwork for being able to account for these contextual characteristics and apply study results across contexts. Here, we provide a brief introduction to transportability and conceptualize context under this framework. We discuss analytical and research implications of using transportability for advancing implementation science research on the response to HIV/AIDS.

TRANSPORTABILITY

Why interventions (whether “evidence-based interventions” or “implementation strategies”) might vary in effectiveness across different settings is intuitive: if there are certain characteristics that modify the effect of an intervention, and the distribution of those characteristics varies from setting to setting, then the intervention’s effectiveness would similarly vary. Further, if we can measure and account for all the characteristics that both modify the effect of the intervention and differ between two settings, then we should be able to predict how effective an intervention would be if it were to be implemented in the new setting.

Transportability formalizes this intuition by building on the theoretical foundations of observational causal inference. In doing so, simple modifications of existing tools and statistical estimators that are widely used in the causal inference literature can be applied to predict how well an intervention might work when implemented in a new setting where it was not formally tested. That is, transportability provides tools to (1) formally evaluate whether findings in one setting could be used to generate valid estimates in another, and (2) if so, estimate what the effect would have been had the study been conducted in the new setting.

Much in the same way that observational causal inference uses directed acyclic graphs (DAGs)⁹ to identify the variables needed to control for confounding, transportability employs similar causal graphs—called selection diagrams-- to assist in isolating the important characteristics that determine whether and how the effectiveness of an intervention might differ between the study population and the population to which we wish to apply the results (the target population). Selection diagrams encode formal assumptions about the underlying causal relationships and mechanisms through which an intervention is believed to operate in the study population as well as assumptions about how the target and study populations may differ from one another.

Selection diagrams begin with a traditional DAG representing the study population—paying special attention to the mechanisms through which the intervention is hypothesized to affect the outcome and to any characteristics that may affect the outcome or modify the effectiveness of the intervention (Figure 1a). Selection diagrams explicitly consider how a target population differs from the study population by including selection nodes indicating these potential differences (Figure 1b). Unlike standard random variables that are usually included in DAGs, selection nodes do not have probability distributions and cannot be influenced by other variables in the graph. Instead, they function as indicators that point to where the data generating processes may differ between the two settings. In other words, they indicate where, if we were to draw a separate DAG for the target population, we might expect the processes that gave rise to the data might differ between contexts. Importantly, the absence of a selection node on a variable indicates that we assume there are no differences in that variable’s distribution between the two populations given its parents (ie. the most proximal causes of the variable explicitly represented on the DAG).

Figure 1a.

Directed Acyclic Graph (DAG) drawn for the study population. Note: simplified for pedagogical purposes.

Figure 1b.

Selection Diagram with a selection node on baseline characteristics. This is transportable.

Selection diagrams reveal first whether the effect of the intervention which was estimated in the study population can be transported to a specific alternative target population given the data available from both the study population and the target population. One approach to this is to measure (and thereby adjust) for enough pre-intervention variables to render selection nodes independent of the outcome (in a graph in which arrows into the intervention have been removed). In this case, the observed estimate could be used to produce a valid estimate in the target population, and we would deem that the observed estimate is transportable to the target population.¹⁰ Transportability goes beyond this, however, by also helping us to understand which specific post-intervention variables can also be used to facilitate transport, and how to use them to derive a specific transport formula to predict the effect in the target population.

In short, encoding knowledge about possible differences in populations and settings through selection diagrams enables a visual assessment of whether the data currently available from both the study and target populations are sufficient to allow for a valid estimate to be generated for the target population, and if not, what, if any, additional measurements would in either the study or target population suffice. We give several simple examples illustrating basic principles here; throughout, we assume that the intervention has not yet been implemented in the target population (observationally or experimentally), and that all variables included as nodes in the diagram have been measured in the study population:

1. Selection nodes on the outcome variable itself (Figure 1c) indicate potential differences in the effect of the intervention on the outcome across populations that cannot be explained by the measured variables. Options for obtaining a valid estimate of the causal effect of the intervention or strategy in the target population, short of repeating the experimental study, include understanding and measuring additional variables that would explain the differences, and measuring the mediator, outcome, and baseline covariates in the target population.¹¹

2. If, on the other hand, no selection nodes point into the outcome or all of the selection nodes only influence the outcome through the intervention itself (Figure 1d), then the effect of the intervention on the outcome in the study population is identical to what we would expect in the target population, and no new measures or experiments are needed.¹¹ This is known as direct transportability.

3. If pre-treatment variables alone are sufficient to make all the selection nodes independent of the outcome (Figure 1b), we need only to measure these variables in the target population. Importantly, this means that we do not need to implement the intervention in the target population to estimate its effect there, and the effect in the target population can be estimated using classic approaches to standardization.¹²

4. Finally, when selection nodes point directly at a mediator (Figure 1e), classic standardization would not be sufficient to transport the effect estimate to the target population. One alternative in this case is to measure the effect of the intervention on the mediator (together with baseline covariates) in the target population. In this scenario, we may not be able to predict the effect of the intervention on the outcome in the target population before implementing the intervention, but we can do so without having to measure the outcome itself in the target population.

Figure 1c.

Selection Diagram with selection nodes on baseline characteristics and on the outcome. This is not transportable.

Figure 1d.

Selection Diagram with a selection node only on the intervention. This is directly transportable (ie. No additional measurements or adjustments needed).

Figure 1e.

Selection Diagram that includes selection on the mediator. This is transportable, but would require that the intervention is offered in the target population. The full study would not need to be repeated as the outcome would not need to be measured in the target population.

By leveraging a formal mathematical framework for causal inference, transportability ensures that as long as the knowledge about the data generating processes and any differences in these processes between the study and target population are accurately represented, the statistical target parameter provided by the transport formula will be equal to the effect of the intervention in the target population. This means that familiar statistical estimation procedures can be readily adapted and used to estimate transported effects in new populations.^13,14

CONTEXT THROUGH THE LENS OF TRANSPORTABILITY

Though there is broad appreciation for the importance of the general idea of context for implementation science research, there are few definitions of what specific factors comprise context or how to address contextual differences once we’ve identified them. By characterizing context in the transportability framework to focus on the specific contextual factors that matter for a given intervention, we can use this information to solve many previously intractable challenges arising from contextual differences.

Defining context

Proctor’s conceptual model of implementation research posits that context can influence the effects of an implementation strategy by affecting several different steps along the way from design to health outcome: implementation outcomes, service outcomes, or clinical outcomes (Figure 2a). We represent Proctor’s conceptual model as a selection diagram, where implementation outcomes and service outcomes act as mediators of the effect of the implementation strategy on the clinical outcomes. Selection nodes represent context and depict where and how the effects of an implementation strategy may vary between the two populations (Figure 2b).

Figure 2a.

Proctor’s conceptual model of implementation science outcomes

Figure 2b.

Proctor’s conceptual model of implementation science outcomes drawn as a selection diagram

Example

Consider an example of a large cluster-randomized controlled trial that found that implementing CAGs in clinics improved HIV viral suppression and mortality. We would expect that numerous factors in the outer context, inner context, health care workers, and populations might affect the success of CAG implementation if introduced in new settings. For example, health care worker training and motivation could affect whether the model is offered to patients; acceptance and participation in CAGs may be influenced by perceived HIV stigma, clinic wait times, or how far patients need to travel to the clinic (i.e., if they live nearby, then the benefits of reduced visits are less obvious); retention in CAGs might be influenced by patient demographics (such as income or education) or availability of health workers to supervise the CAGs. Finally, comorbidities like TB prevalence may affect clinical outcomes such as mortality (Figure 3a). In reality, many of these factors are likely correlated with one another, but we use this simplified example for pedagogical purposes to highlight the role of selection diagrams.

Figure 3a.

Proctor’s conceptual model of implementation science outcomes for CAGs

In a simplified example, suppose a study of CAG’s was done in an urban environment, and a more rural clinic is interested in implementing CAGs. To inform their decision, they would like to know what the results of the study would have been had it been conducted in their rural clinic. In the target population (the rural clinic), patients tend to live further away than patients attending the urban clinics that were included in the study, but clinic wait times in the rural clinic are typically not as long as in the wait times in the urban clinics. Additionally, in the rural clinic, there are fewer available community health workers, the patients often experience higher HIV stigma, and the patient population is generally lower socioeconomic status. The trial and target populations are otherwise similar.

To assess whether we could transport the results of the trial to the target clinic, we begin by drawing a selection diagram illustrating the hypothesized causal relationships between CAGs, the various types of outcomes, and the factors that may influence these outcomes (Figure 3b). The selection diagram includes selection nodes directed at distance to clinic; clinic wait times; number of community health workers; stigma; and socioeconomic status. By excluding any additional selection nodes, we are assuming that there are no other relevant differences between the two populations. Based on this selection diagram, we can estimate the anticipated effect of introducing CAGs into the target clinic just by measuring distance to the clinic; clinic wait times; number of community health workers; stigma; and socioeconomic status in the target population. Because none of these variables are mediators between CAGs and HIV mortality, we can apply standardization using these variables to estimate the effectiveness of CAGs in the target population.¹⁵ Framing the problem of context through a selection diagram allows us to identify which specific contextual factors are important for generalizing study results to the new population and how we can estimate the anticipated effect of an implementation strategy in that population without having to repeat the study.

Figure 3b.

Selection Diagram. Relevant contextual differences are: clinic wait times; distance to clinic; number of community health workers; stigma; and socioeconomic status.

IMPLICATIONS FOR IMPLEMENTATION SCIENCE

External validity is the white whale of implementation science

Whether the results of a trial are transportable to a target population depends on how that specific target population differs from the trial population; a trial may be transportable to some populations but not others. This demands a more nuanced view of the traditional idea of “external validity” in which a trial could be considered generally externally valid or not irrespective of who the target population might be. Instead, transportability does not solely focus on the trial, but rather highlights the relationship between the trial population, context, and a specific target population.

In the pursuit of maximizing external validity, implementation science researchers have championed pragmatic trials as a means of evaluating the effectiveness of implementation strategies in usual care settings.¹⁶ However, in the face of heterogeneous effects of implementation strategies, the quest for a single effect that applies universally is hopeless even if the trial design conforms to all the established features of a pragmatic trial (e.g., no recruitment restrictions, flexible interventions). So, unless the trial population is a random sample of all future target populations (an implausible concept given that even in the same location, the data generating process may evolve over time), even results of pragmatic trials will need to be formally transported to produce evidence that is relevant for different settings (or the same setting in the future). To avoid altering standard care practices, pragmatic trials often minimize the number of measurements taken over the course of the study. However, formal transport requires individual-level measurements of variables that modify the effectiveness of the implementation strategy. To the extent that pragmatic trials minimize the number of characteristics measured, they may reduce the chance that their results can be validly transported to external settings, and thus undermine the goal of generating study results that can be applied to a range of target populations. Instead, if the objective of pragmatic trials is to produce more generalizable knowledge, it is essential that these studies understand and measure variables that mediate and modify the implementation strategies being evaluated.

Context is contextual

Another key insight that emerges from transportability is that the contextual factors that need to be measured to transport a given implementation strategy will vary depending on the specific trial and target populations. For example, returning to the hypothetical example described in Figure 3, suppose that we are interested in transporting the results of the trial to a different target clinic specializing in TB care. The clinic is located in the same general region as many of the clinics that were included in the trial but has a much higher number of patients who have TB. To transport the trial results to this target population, we would only need to measure and account for TB prevalence in the trial and target populations (Figure 4).

Figure 4.

Selection Diagram. Relevant contextual differences are: TB infection

Further, there is no guarantee that the specific contextual factors that are important for transporting one implementation strategy will be pertinent for another. This suggests that thinking about context broadly is less useful than focusing on identifying and measuring all characteristics that may modify any portion of the causal path between a specific implementation strategy and the clinical outcome of interest in our trial populations and may differ across contexts; while this does not ensure our ability to accurately estimate how effects may differ in a new setting, it helps move closer to that goal. This helps narrow the scope of what variables we need to understand about a setting in order to predict how well a strategy may perform.

Learning from heterogeneity

The most straightforward application of the transportability framework helps us answer the question of external validity: “if I had conducted this study in a different population, what would my results have been?” More broadly, however, selection diagrams provide a flexible language in which assumptions about mechanisms of action and differences between populations are clearly articulated. These diagrams unlock the ability to test our hypotheses and better understand why implementation strategies vary in effectiveness between populations.

For example, transportability can be used to understand heterogeneous results within a single study.^17,18 If we observe heterogeneous effects across sites or sub-populations of a study, we can use transportability to test our understanding of the sources of discrepant results. For instance, imagine our trial of CAGs included both public and private clinics and that the intervention appeared most effective in private clinics. Suppose patients that attend private clinics are different from those patients that attend public clinics across several demographic factors. In interpreting the results of the trial, we may be unsure whether the differences in effectiveness of CAGs between private and public clinics were driven by these differences in patient characteristics or if there are other clinic-level differences that impact how well CAGs work.

To address this question through the lens of transportability, we can estimate what the effect of CAGs would have been in the public clinics had those clinics’ patient populations resembled those of the private clinics. We can then compare this transported estimate to the observed effectiveness in private clinics; if the transported estimate is similar to what was observed, then we can conclude that differences in patient characteristics alone are sufficient to explain the discrepant results. If the transported estimate is not similar to what was observed in the private clinics, then this suggests that other factors may have been responsible for the incongruous results.¹⁷

A similar approach can also be used to understand discrepant results across trials of similar delivery strategies. For example, suppose we have two studies that both evaluated the use of CAGs for improving HIV care, but the results differed between the studies. If, after drawing a selection diagram, we believe we have measured all the important contextual factors that should explain the differences in effectiveness between the two studies, we can transport the results from the first to the second study and compare the transported results to what was actually observed in the second study. If the transported estimate equals the observed, then we can conclude that our measured set of characteristics was sufficient to explain the differences in the results. Otherwise, if the transported estimates do not match the observed results, then we may have omitted an important contextual factor that impacts the effectiveness of CAGs.

The information learned from these sorts of exercises is useful for identifying which specific contextual factors are most important for the success of a given implementation strategy, and this knowledge allows future researchers or policy-makers to ensure that the relevant characteristics are measured and accounted for when planning implementation in new target populations. In doing these exercises, if we learn that we have omitted an important contextual factor that impacts the effectiveness of the intervention, researchers should prioritize identifying what these other factors might be (through qualitative or other exploratory analyses).

CONCLUSIONS

Overall, transportability theory empowers implementation science researchers to translate intuition about context into actionable and testable science. Selection diagrams are powerful tools that facilitate a deeper understanding of the mechanisms that matter most to the success of an implementation strategy. This approach is not at odds with prior work describing how and why context matters for implementation science. Rather, it allows us to move beyond describing these challenges and applying powerful and emerging insights to carry out analyses, determine what measurements to make and how to conduct science that is externally valid and maximally relevant.