Using transportability to understand differences in mediation mechanisms across trial sites of a housing voucher experiment

Abstract

Background

Randomized trials may have different effects in different settings. Moving to Opportunity (MTO), a housing experiment, is one such example. Previously, we examined the extent to which MTO’s overall effects on adolescent substance use and mental health outcomes were transportable across the sites to disentangle the contributions of differences in population composition versus differences in contextual factors to site differences. However, to further understand reasons for different site effects, it may be beneficial to examine mediation mechanisms and the degree to which they too are transportable across sites.

Methods

We used longitudinal data from MTO youth. We examined mediators summarizing aspects of the school environment over the 10–15 year follow-up. Outcomes of past-year substance use, mental health, and risk behavior were assessed at the final timepoint when participants were 10–20 years old. We used doubly robust and efficient substitution estimators to estimate 1) indirect effects by MTO site and 2) transported indirect effects from one site to another.

Results

Differences in indirect effect estimates were most pronounced between Chicago and LA. Using transport estimators to account for differences in baseline covariates, likelihood of using the voucher to move, and mediator distributions partially to fully accounted for site differences in indirect effect estimates in 10 of the 12 pathways examined.

Conclusions

Using transport estimators can provide an evidence-based approach for understanding the extent to which differences in compositional factors contribute to differences in indirect effect estimates across sites, and ultimately, to understanding why interventions may have different effects when applied to new populations.

INTRODUCTION

Randomized trials are frequently conducted across multiple sites to improve generalizability (1). However, effects may differ across sites (e.g., 2–5). Such lack of replication is not always examined (e.g., 6), and even when it is examined and found, the result may be to either ignore it in analyses (e.g., 5) and/or to offer speculative reasons for differences without quantitative evidence (e.g., 2).

Previously, we argued that transportability (7) could be a useful tool for bringing quantitative evidence to examine why experimental effects may be different across sites (8). Transportability is the ability to identify an effect of interest in a target population from the observed data in both the target and source populations, based on identification assumptions (7). Specifically, in examining lack of replication of overall effects of a housing intervention, the Moving to Opportunity (MTO) study (9), on adolescent mental health and risk behavior, we used transported estimates of the intent-to-treat average treatment effect to estimate the extent to which site differences were due to differences in the distribution of individual-level compositional factors and intervention adherence versus due to differences in more macro-level, contextual factors (10). We now extend this work using a transport estimator of the stochastic indirect effect (11) to quantitatively examine reasons underlying site differences in indirect effects.

Our objective is to examine transportability of the indirect effects of Section 8 housing voucher receipt and use on mental health, substance use, and risk behavior outcomes through aspects of the school environment across MTO sites. We take advantage of MTO’s randomized experimental design and longitudinal follow-up (12). We assess the extent to which differences in the distribution of individual-level compositional factors, differences in intervention adherence (i.e., using the voucher to move), and differences in the conditional mediator distributions contribute to site differences in indirect effect estimates. We also discuss nuances and considerations when applying this novel transport method and outline areas for future work.

METHODS

We used data from children aged 0–10 years and their families who were enrolled in MTO at baseline (1994–1998) and followed up at the final visit, which occurred 10–15 years later in 2007 when they were 10–20 years old (12). MTO has been described previously (2, 9, 12). Briefly, it was a randomized controlled trial conducted by the U.S. Department of Housing and Urban Development in five U.S. cities (Baltimore, Boston, Chicago, Los Angeles (LA), and New York (NYC)) and enrolling 4,604 families. It was designed to assess the effects of providing Section 8 housing vouchers to encourage moving to low-poverty neighborhoods on a broad array of outcomes. Low-income families living in distressed public housing with children under 18 were eligible to enroll.

Measures

MTO randomized families into one of three groups: 1) receipt of a Section 8 housing voucher to be used to rent an apartment in a low-poverty neighborhood (<10% persons in poverty) and assistance finding housing, 2) receipt of an unrestricted Section 8 housing voucher with no housing assistance, and 3) no intervention but eligible to remain in public housing (the in-place control group). We combined the two voucher groups, consistent with previous work (10, 13, 14). The randomized intervention of voucher receipt, A, is an instrumental variable (15) that may affect the school environment, M, and youth outcomes, Y through moving with the voucher out of public housing (intervention adherence), Z, and, typically, into a lower-poverty neighborhood.

We considered four binary school environment mediators measured after baseline over the duration of follow up: 1) having ever attended a school ranked above the 50th percentile, which we refer to as a high-quality school, 2) having ever attended a school that was not Title I-designated, which we refer to as a non-high poverty school, 3) over the duration of schooling, the student attended “high poverty schools” most of the time, defined as those in which ≥ 75% of students were eligible for free or reduced price lunch, and 4) having attended more than four schools. These mediators were chosen because they were objective measures of the school environment across the duration of follow-up, had low levels of missing data, and were binary or dichotomized and non-rare. Dichotomization was necessary as the estimator used currently only works with a binary mediator (11), though there is ongoing work on an extension to relax this.

We considered seven outcomes, each measured at the 10–15-year follow-up time point, and representative of “current” mental health, substance use, or risk behavior problems: 1) any past-year externalizing disorder, which we define as either conduct disorder or oppositional defiant disorder per the Diagnostic Statistical Manual, Fourth Edition (16) (DSM-IV), 2) any past-year DSM-IV disorder, including substance use disorder, 3) z-score on the Kessler six-item youth psychological distress index (17), 4) smoked cigarettes in past month, 5) proportion of 11 problem behaviors endorsed in the behavioral problem index (BPI) (18), 6) proportion of four risky behaviors endorsed in the BPI, 7) self-reporting feeling calm/peaceful most of the time in the past month.

Baseline covariates, W, included sociodemographic characteristics of the child and family (e.g., age, race/ethnicity, etc.), behavior and learning characteristics of the child (if applicable based on age), neighborhood characteristics at baseline, and reasons for participation in MTO. Most covariates are listed in Table 1, and a full list is provided in eAppendix 2.

Table 1:

Characteristics by site, Moving to Opportunity, 1994–1997. Numbers are percentages unless otherwise specified. Survey weighted and combined across 30 imputed datasets.

Variable	Boston	Chicago	LA	NYC
Sample size (rounded per Census rules)	1100	1200	850	1200
Female	50	48	47	51
Race/ethnicity
White	7	< 1	< 1	< 1
Black	36	97	50	46
Hispanic/Latino	46	2	46	52
Other race/ethnicity	11	< 1	4	2
Age at baseline (mean year, SEa)	4.5 (0.1)	4.5 (0.1)	5.0 (0.1)	5.0 (0.1)
Parent characteristics
Graduated high school	44	35	33	31
Received GEDb	22	19	11	28
Never married	62	80	60	55
Under 18 at birth of first child	22	38	28	23
Working	28	23	20	21
Welfare receipt	68	87	86	77
Family has car	22	13	39	7
Family member with disability	17	16	9	23
Household characteristics
Household size
2 people	12	6	6	10
3 people	32	21	19	23
4 or more people	26	25	26	29
Percent of persons in baseline Census Tract under poverty level (mean, SE)	39.5 (0.3)	71.0 (0.4)	55.1 (0.5)	47.6 (0.3)
Household member was victim of a crime, past 6 months	26	42	49	51
Stopped to chat with neighbor ≥ 1× /week	51	51	47	51
Would tell neighbor if neighbor’s child was in trouble	57	56	54	49
No family in neighborhood	75	45	62	74
No friends in neighborhood	47	35	34	47
Felt unsafe in neighborhood	34	48	60	56
Very dissatisfied with neighborhood	35	51	45	54
Confident that would find an apartment	37	56	54	47
Moved households ≥ 3 times	12	9	9	8
Primary reason for moving is to get away from drugs	70	75	84	72
Primary reason for moving is to go to better schools	33	62	62	58
Applied for Section 8 voucher previously	53	21	33	47
Ever encountered discrimination in housing search	7	10	12	6
Household has ever received SSIc	13	16	11	20
Received voucher	67	70	69	65
Moved with voucher	31	33	50	33
Mediators
Ever attended high-quality school	35	42	48	38
Attended high-poverty schools over duration of schooling	53	78	65	22
Ever attended a lower poverty school	85	40	79	95
> 4 schools	33	31	40	26
Outcomes
Any DSM-IV disorder	28	32	28	28
Externalizing disorder	16	21	16	16
Distress (mean, SE)	−0.03 (0.05)	0.09 (0.04)	0.03 (0.05)	−0.05 (0.04)
Calm	59	61	61	58
BPId (mean, SE)	39 (1)	41 (1)	39 (1)	39 (1)
Risky behavior (mean, SE)	49 (1)	48 (1)	44 (1)	47 (1)
Smoke cigarettes	21	23	19	19

^aStandard error (SE)

^bGeneral Education Diploma (GED)

^cSupplemental Security Income (SSI)

^dBehavioral problem index (BPI)

A directed acyclic graph (DAG) depicting the relationships between the aforementioned measures is shown in Figure 1. Whereas confounding represents a threat to internal validity, transportability is related to external validity. If a causal effect is externally valid from a source population to a target population, then it is transportable from the source to the target. Just as DAGs are helpful for assessing whether or not a causal effect can be identified in the presence of confounding, they may also be used to assess whether or not a causal effect is transportable. Pearl and Bareinboim utilize an S node (where S represents site) with an an arrow pointing to the variable whose nonparametric distribution differs by site, generally indicating a lack of tranportability for that particular effect (7). We use their same notation in Figure 1. Differences between subfigures 1a – 1c depict different points in the DAG where site, S, may be influential. For example, an arrow from S to Y as in Figure 1c indicates that the outcome mechanism differs between the sites. Further discussion of Figure 1 is included in eAppendix 1.1.

Figure 1:

Transportable (a and b) and not transportable (c) DAGs considered.

If one has data on Z, M, and Y across sites, then one can test, using observed data, whether or not there is evidence of an S arrow into each of these variables using a non-parametric test of the equality of each distribution across the sites (19). We tested the null hypothesis, E(Y|S = 0,W,A,Z,M) = E(Y|S = 1,W,A,Z,M). If we did not reject the null hypothesis, then we assume the most general of Figures 1a–1c for which transport is still possible, which is Figure 1b, and proceed with estimating the transported indirect effect. In cases where we reject the null hypothesis of a common outcome model (Figure 1c), this indicates a different outcome mechanism between the sites and, following Pearl and Bareinboim’s roadmap, renders the indirect effect not transportable (7).

Sample

We used the youth subset of the MTO final evaluation (12). 5105 youth out of a total possible of 6308 (81%) completed the final survey; the effective response rate was 89% (12) and similar across intervention groups. We excluded those in the Baltimore site (n=762) as voucher receipt was not associated with a subsequent move to a low-poverty neighborhood. This secondary data analysis was determined to be nonhuman subjects research.

Statistical Approach

Overview

We consider mediator–outcome combinations with qualitative differences in indirect effect estimates across sites. Then we employ our transport estimator to quantify the extent to which the differences may be due to differences in the distribution of compositional factors, adherence, and mediator distribution. Differences that cannot be explained suggest that macro-level, contextual factors (or unmeasured compositional factors) may be critical to predicting the indirect effect.

We first imputed missing values using multiple imputation by chained equations, which assumes the data are missing at random conditional on the variables in the imputation model (20). Covariate missingness was low—less than 5%. Treatment assignment and adherence were not missing for any individuals. Mediator and outcome missingness were higher—between 6% and 12%. We used a high-dimensional vector of variables for imputation, which included numerous variables over the course of follow-up in addition to those included in the analysis. We generated 30 imputed datasets. The analysis was completed on each imputed dataset and the results pooled using Rubin’s combining rules (21).

We then estimated stochastic indirect effects for each mediator–outcome combination by site and by gender using, as previous MTO research has shown site and gender to be important sources of effect heterogeneity for mediation effects (14) and total effects (10, 13). The stochastic indirect effect (SIE), $E(Y_{1,g_{M|1,W}})-E(Y_{1,g_{M|0,W}})$, contrasts the expected value of Y setting A = 1 (corresponding to voucher receipt) and stochastically drawing from the distribution of M conditional on W under a = 1 versus stochastically drawing from the distribution of M conditional on W under a = 0. We estimated SIEs using a doubly robust, efficient substitution estimator in the targeted minimum loss-based framework (22). An R function to implement this estimator is available at https://github.com/kararudolph/SDE-SIE/blob/master/ivmedtmle.R. We further describe this step in eAppendix 1.2.

Next, we combined sites into groups with similar SIE estimates within gender (we refer to these combined sites as “site groups” throughout) and identified site groups with different estimates, as operationalized by a two-sample t-test (α = 0.1 level, though we acknowledge that this threshold is arbitrary). Among the site groups exhibiting differences, we tested the assumption of a common outcome model across site groups using the nonparametric omnibus test for equality in distribution (23), using the same cross-validated lasso algorithm in model fitting as described above. If this test did not reject the null hypothesis of equality, then we estimated transported, data-dependent SIEs from one of the site groups (S = 1) to the other site group with different indirect effect estimates (S = 0). (Note that we transport in either direction.).

Specifically, we are concerned with estimating $\mathbb{E}\left[Y_{1,g_{M|1,W,s}^{*}}-Y_{1,g_{M|0,W,s}^{*}}|S=0\right]$. In words, this is the transported SIE of A on Y, operating through M, in the target population. We note that this estimand marginalizes over Z, so estimates transported indirect effects that operate through Z (which is indeed the only nonzero indirect effect in our data structure due to the absence of a direct effect from A to M). We provide the assumptions under which this effect is identified in eAppendix 1.3.

We estimate transported SIEs using a doubly robust and efficient substitution estimator (11). R code to implement these estimators is available at https://github.com/jlstiles/SDEtransport. There are two versions of this transport estimator—one that estimates the mediator model pooling across S and then predicting out to the target site, S = 0, denoted “transported pooled”, and another that estimates the mediator model only using data from S = 0, denoted “transported not pooled”. In large samples and under correct model specification, both approaches define and estimate very similar parameters. The targeted minimum loss-based estimator we use to estimate these transported SIEs can currently be applied considering one binary mediator at a time, so we consider our four mediators separately. No other estimator for transported SIEs exists. We estimate these transported SIEs in both directions: from site group A (S = 1) to site group B (S = 0) and vice-versa (treating group A as S = 0 and group B as S = 1).

We incorporated survey weights that account for the treatment arm random assignment ratios (which varied by site and year), sampling of children within households, and loss to follow up into all analyses (12). R version 3.5.2 was used for all analyses.

RESULTS

Table 1 shows the differing distribution of covariates across MTO sites that are depicted by the arrow from S to W in Figure 1. The racial/ethnic composition of participants differs markedly across sites with most participants in Chicago being black (97%) but the remaining sites having more equal distribution of black and Hispanic/Latino participants. The poverty rate in the baseline Census tract also differs markedly with the average neighborhood poverty rate being 40% in Boston but over 70% in Chicago.

The distribution of mediators also differs across sites. For example, attending mostly high-poverty schools over the duration of follow-up ranged from a high of 78% in Chicago to a low of 22% in NYC. Similarly, ever attending a lower poverty school also differed across sites with this being true of 40% in Chicago and 95% in NYC. These differences could have been due to differing distributions of compositional factors, as discussed above, due to differing distributions of the mediator itself conditional on compositional factors across sites (which we later assess using an extension of the test for equality in nonparametric distributions), or a combination.

In contrast, the distribution of the outcome measures did not differ as markedly across sites. For example, the presence of past-year externalizing disorder was 15–16% in Boston, LA, and NYC versus 21% in Chicago. A similar pattern was seen with having any past-year DSM-IV disorder: 27–28% in Boston, LA, and NYC versus 32% in Chicago.

eTable 1 details estimates of each total effect, A on Y, the effect of A on each M, and the effect of each M-Y relationship considered.

The first step in transporting indirect effects across sites involves first estimating these effects by site. Table 2 shows these stochastic data-dependent indirect effects by site and gender.

Table 2:

Stochastic indirect effects (risk differences (RD)) by site group and gender.

Gender	Outcome	Mediator	Sites	RD estimate	95%CI
Female	Any Disorder	Ever attended a lower poverty school	Chicago	0.0037	−0.0014, 0.0089
			LA	−0.0022	−0.0074, 0.0029
	Externalizing disorder	Ever attended a lower poverty school	Chicago	0.0039	−0.0001, 0.0079
			NYC	−0.0001	−0.0013, 0.0010
	BPI	Ever attended a high-quality school	Boston and LA	−0.0037	−0.0083, 0.0010
			Chicago and NYC	−0.0003	−0.0014, 0.0007
		Attended > 4 schools	LA	0.0073	−0.0006, 0.0151
			Boston, Chicago, and NYC	0.0018	0.0003, 0.0033
	Risky behavior	Ever attended a lower poverty school	Chicago	0.0038	0.0011, 0.0066
			NYC	−0.0004	−0.0014, 0.0006
	Calm	Ever attended a high-quality school	LA	0.0120	0.0002, 0.0238
			Chicago and NYC	−0.0010	−0.0033, 0.0012
		Ever attended a lower poverty school	LA	0.0057	0.0004, 0.0110
			NYC	−0.0008	−0.0026, 0.0010
	Distress	Attended > 4 schools	LA	0.0293	−0.0030, 0.0617
			Chicago and NYC	0.0018	−0.0092, 0.0128
Male	Any disorder	Ever attended a lower poverty school	Chicago	0.0087	−0.0007, 0.0180
			NYC	0.0001	−0.0012, 0.0014
	Externalizing disorder	Attended high-poverty schools over duration of schooling	LA	−0.0073	−0.0240, 0.0094
			Boston and Chicago	0.0007	−0.0028, 0.0042
	Risky behavior	Ever attended a lower poverty school	Chicago	0.0046	−0.0009, 0.0100
			NYC	−0.0001	−0.0011, 0.0009
		Ever attended a high-quality school	LA	−0.0095	−0.0191, 0.0001
			Boston, Chicago, and NYC	0.0004	−0.0006, 0.0014
	Calm	Attended high-poverty schools over duration of schooling	LA	−0.0160	−0.0390, 0.0070
			Boston and Chicago	0.0016	−0.0022, 0.0054
	Smoke cigarettes	Attended > 4 schools	Chicago and LA	0.0068	0.0004, 0.0131
			NYC	−0.0010	−0.0089, 0.0070

As seen in Table 2, there is evidence that ever attending a lower poverty school (not Title I designated) mediates the effect of voucher receipt on subsequent risk of any DSM-IV disorder for boys and girls in Chicago. Specifically, the indirect effect pathway A → Z → M → Y increases the probability having a prevalent disorder at follow-up by 0.87% (95% CI: −0.07%, 1.80%) for boys and 0.37% (95% CI: −0.14%, 0.89%) for girls. This is an example of an unintended harmful intervention effect. Other pathways underlying unintended harmful intervention effects also occurred in Chicago. For example, the same mediator of ever attending a lower poverty school acted to increase the probability of having a prevalent externalizing disorder at follow-up for girls (0.39%, 95% CI: −0.01%, 0.79%) and increased risky behavior engagement for both boys and girls in Chicago.

In contrast, LA youth generally saw beneficial indirect effects (Table 2). For example, the pathway from voucher receipt through ever attending a high quality school (ranked >50th percentile) resulted in fewer risky behaviors for boys, and fewer behavioral problems and greater likelihood of feeling calm most of the time for girls in LA. Voucher receipt also increased the likelihood that LA youth reported feeling calm through mediators of attending lower poverty schools over the duration of follow-up. Moreover, in LA, the anticipated potentially harmful pathway from voucher receipt through attending more schools operated as expected, contributing to more behavioral problems and psychological distress for girls. Also, in a rare instance of commonality, this anticipated potentially harmful pathway resulted in a greater likelihood of cigarette use at follow-up for boys in both Chicago and LA.

We next tested for evidence against a common outcome model across site groups for each of the gender–mediator–outcome combinations listed in Table 2 (23). One combination demonstrated evidence against a common outcome model: the outcome of risky behavior including the mediator of ever attending a high-quality school among male youth (p-value=0.03). For the remaining combinations, we failed to reject the null hypothesis of a common outcome model, so proceeded with estimating transported indirect effects.

Figures 2–4 show transported indirect effect estimates across site groups. Comparing the transported estimates (dashed lines) to observed estimates (solid lines) within site allows us to assess the extent to which the site differences in SIEs can be attributed to differences in the distribution of compositional characteristics, adherence, and the mediator distribution between the sites. If the SIEs are transportable using measured W,Z,M, then the transported SIE will coincide with the observed SIE for a given site. This would suggest that the differences in the SIEs between sites are entirely due to differences in the aforementioned factors between the sites. If, on the other hand, the transported SIE for S = 0, say, is no closer to the observed estimate for S = 0 than is the observed estimate for S = 1, then it suggests that the SIE is not transportable based on these characteristics. If the transported estimate is between the two observed estimates, it suggests that the site differences in indirect effects are partially but not entirely attributable to measured compositional, adherence, and mediator differences.

Figure 2:

Gender–mediator–outcome combinations for which transporting in either direction partially or fully accounted for site differences. Non-transported risk difference estimates from observed data (“observed”), Transported, predicted estimates estimating $g_M^{*}$ from S = 0 data (“transported, not pooled”), and Transported, predicted estimates estimating $g_M^{*}$ pooling data across sites (“transported, pooled”).

Figure 4:

Gender–mediator–outcome combinations for which transporting did not partially account for site differences in either direction. Non-transported risk difference estimates from observed data (“observed”), Transported, predicted estimates estimating $g_M^{*}$ from S = 0 data (“transported, not pooled”), and Transported, predicted estimates estimating $g_M^{*}$ pooling data across sites (“transported, pooled”).

Figure 2 shows gender–mediator–outcome combinations for which the transported estimates are closer to the observed estimates in the target population than the observed estimates in the source population. In these cases, accounting for differences in the distribution of covariates, adherence and the mediator partially explained differences in the SIE estimates between sites. As an example, Figure 2c shows the estimates for the indirect effect of being randomized to the housing voucher group on feeling mostly calm at follow-up through ever attending a lower poverty school among girls. The transported estimate from LA to NYC is closer to the observed NYC estimate than is the observed LA estimate; differences in measured W,Z,M explained about 90–94% of the difference in point estimates between the sites.

Figure 3 shows gender–mediator–outcome combinations for which accounting for differences in the distribution of covariates, adherence and the mediator partially explained differences in the SIE estimates when transporting in one direction but not the other. For these combinations, transporting from the site group with the larger (in absolute value) indirect effect estimate to the site/site group with the smaller estimate partially or fully accounts for differences in indirect effect estimates between sites.

Figure 3:

Gender–mediator–outcome combinations for which transporting in one direction—but not the other—partially or fully accounted for site differences. Non-transported risk difference estimates from observed data (“observed”), Transported, predicted estimates estimating $g_M^{*}$ from S = 0 data (“transported, not pooled”), and Transported, predicted estimates estimating $g_M^{*}$ pooling data across sites (“transported, pooled”).

Lastly, Figure 4 shows gender–mediator–outcome combinations for which accounting for differences in the distribution of covariates, adherence and the mediator did not contribute to explaining site differences when transporting in either direction.

DISCUSSION

We used a novel estimator to transport stochastic indirect effects (11) to understand reasons underlying different indirect effects linking housing voucher receipt and use through the school environment to mental health and risk behavior outcomes among youth across MTO sites. Differences in mediation pathways were seen most frequently between the Chicago and LA sites with the Chicago site frequently exhibiting pathways contributing to unintended harmful effects (Table 2). Important mediators contributing to unintended harmful effects in Chicago included ever attending a lower poverty school, which contributed to between 0.4% and 0.9% increase in risk of having any DSM-IV disorder, externalizing disorder, and risky behavior. However, we note that these estimated increases are very small in terms of attributable risk, typically resulting in just a few extra cases. However, our estimates may be biased due to measurement error induced by dichotomizing the mediator and due to including only one mediator at a time. We discuss an extension that can overcome these limitations later in the Discussion.

We limit drawing substantive conclusions to the mediation pathways that exhibited consistent evidence of transport (or not) in both directions (Figures 2 and 4). The pathway from voucher receipt and use to ever attending a lower-poverty school to having a subsequent psychiatric or substance use disorder was transportable across sites for girls and boys. In other words, attending a lower-poverty school appeared to mediate the relationship between voucher receipt and development of a DSM-IV disorder, and although the site-specific indirect effects differed substantially, those differences were accounted for by differences in the distribution of individual- and family-level demographics, adherence, and mediator distributions across sites. Poverty level of the schools attended also mediated the effect of voucher receipt on likelihood of feeling mostly calm at follow-up. Again, although site-specific indirect effects differed substantially, the differences were accounted for by differences in the demographic composition of the sites, intervention uptake, and the mediator distribution. The pathway from voucher receipt and use to attending >4 schools to behavioral problems among girls was also transportable but to a lesser extent. In contrast, pathways contributing to outcomes of cigarette use and psychological distress were not transportable across MTO sites, reflecting that other factors, including more macro-level, contextual factors like the local policies or norms, may be responsible for site differences in these pathways.

In five of the 12 pathways we examined, accounting for differences in the distribution of covariates, adherence and the mediator partially accounted for differences only when applied in the direction of transporting from the site group with the larger effect size (in terms of absolute value) to the site group with the smaller effect size. This may point to an asymmetry in how well the identification assumptions are met, for example, if extrapolating beyond the support of the data is more of a problem when transporting in one direction versus the other. Examining the efficient influence curve is one way of assessing problematic extrapolation (24). Because the variance of our transported effect estimates is estimated as the sample variance of the efficient influence curve (11), we examined the variances of our transported estimates for indications of problems with extrapolation. The distribution of variances were similar regardless of direction of transport (see eFigure 1); therefore, extrapolation cannot explain the asymmetry in transport direction we observed.

Nonetheless, the potential for extrapolation when relying on variable combinations with poor support in the source population is an important consideration when transporting effects. For a low-dimensional example of this, consider the racial/ethnic composition of participants, which differs markedly across sites with most participants in Chicago being black (97%) but the remaining sites having more equal distribution of black and Hispanic/Latino participants. The extent to which race/ethnicity modifies the effect of A on Z, Z on M, and Z and M on Y, may contribute to explaining differences in the SIEs estimated between Chicago and other sites. However, because of poor support in this variable when Chicago is the source site, extrapolation may be more of an issue. In reality, the area of support will be a high-dimensional space determined by W,A,Z,M and their interactions, assessed at each node in Figure 1 and summarized across the nodes. Due to the curse of dimensionality, strong overlap will not be possible and even “interpolation overlap” will be impractical (25). One area for future work is to extend so-called “maximal box” approaches by Eckstein et al., 2002,(26) Crump et al., 2009,(27) and Fogarty et al., 2016 (25) to accommodate the multiple models incorporated in our estimator.

Another consideration with regard to asymmetry is the contribution of regression to the mean (28, 29), whereby estimates that are farther from the true mean tend to be followed by estimates that are closer to the true mean. In terms of transportability, this means that even if the transported predictions were simply noisy re-estimations of the source observation—not accounting for any differences between the sites—when transporting from a source population with a more extreme estimate, the transported prediction would be expected to be less extreme and closer to the null if the true mean was also closer to the null. Several factors act to increase the strength of regression to the mean: smaller sample size, poor fitting models (e.g., models that are overfit, having a large number of covariates relative to the sample size), and greater variability in estimates (30). Consequently, steps can be taken to reduce the effect of this phenomenon, including transporting from the site group with the larger sample size and/or smaller variance, using an efficient estimator as we do here, and minimizing model overfitting as we do here through our use of the lasso with a high penalty parameter. Quantifying the extent to which regression to the mean may contribute to the transported effect estimate by extending previously developed equations (28, 31) for our transport parameter is an area for future work.

Our choice to assume the data structure depicted in Figure 1b that allows for S → M, may be unappealing for two reasons. First, one could argue that it is unclear what it means to transport a mediation mechanism where the mediation model itself is allowed to differ between sites. However, in this case, we believe this remains a useful endeavor for the purposes of examining reasons for site differences. Second, there may be applications where one would like to transport indirect effects to a new population where mediator and outcome data do not exist. In these cases, we would assume the data structure depicted in Figure 1a. We can test the assumption of a common model P(M|A,W) that is used in the stochastic intervention by modifying the nonparametric omnibus test of equality in distribution, marginalizing out the intermediate variable, Z. We provide this code to the interested reader at https://github.com/kararudolph/transport/blob/master/R/omnibustest_gm.R. When applied to the gender–mediator–outcome combinations examined in this study, we found evidence against a common distribution P(M|A,W) for several of the combinations, suggesting that Figure 1b may be a better assumption of the causal structure. Another limitation related to the nonparametric omnibus test of equality in distribution is that it is designed to control the Type-I error rate (23); it is not optimized for Type-II error under the alternative hypothesis of nonequality, and consequently, we may be proceeding with transporting indirect effects across sites that have different outcome distributions. Nevertheless, the test has been shown to be powered against all alternatives except for cases where the two distributions are degenerate, which is not the case in the data we consider here, and previous simulation studies found high power (> 80%) in correctly rejecting the null hypothesis in sample sizes ranging from N=250 to 2,000 (23).

One area for future work is to develop a more general estimator for transporting to S = 0 that utilizes an optimal combination of source sites, S = 1. For example, a simple candidate would be an inverse variance-weighted combination of estimates from S = 1 sites. However, that candidate may be improved upon by up-weighting S = 1 sites that are similar to S = 0 in terms of adherence and the distribution of demographic variables that are strong modifiers of the effects incorporated in the estimator.

Still another area for future work is to extend the estimator we use to allow for continuous and/or multiple or high-dimensional M and for continuous and/or multiple or high-dimensional Z. Without these extensions, it is possible that interrelated mediators could result in exposure-induced mediator–outcome confounding and that the measurement error induced by dichotomizing continuous or multi-categoried mediators may result in another significant source of bias. We are currently pursuing such an extension. This would allow us to consider combinations of school mediators (of any distribution) that may be important.

In summary, comparing transported overall and mediated effects to their observed counterparts represents one strategy for quantitatively examining reasons underlying lack of replication of intervention effects across trial sites. In doing so, there are numerous aspects to consider, including 1) the structural causal model (Figure 1 depicts several candidates; we provide an extension of a nonparametric omnibus test for the S → M portion of Figure 1b and suggest the previously published version for testing the S → Y portion of Figure 1c (23)), 2) direction of transport considering the area of common support, and 3) steps to minimize regression to the mean. Although transporting effect estimates from one population to another has benefited from recent research interest (e.g., 8, 32, 33), it remains a relatively nascent area of research with numerous open questions.