The Authors Respond

We welcome the discussion by Huitfeldt and Stensrud¹ on our recent paper on generalizing study results. One assumption we listed in the set of sufficient conditions for generalizability was exchangeability between the study sample and the target population, perhaps conditional on a set of covariates W (such that S ⊥ Y(x)|W for x = a,a′).² Huitfeldt and Stensrud state that conditional exchangeability is expected to hold only when W includes all causes of the outcome whose distribution differ between the study and target populations. They then state that this condition will often not hold in randomized trials. Depending on the setting, we concur that S ⊥ Y(x)|W for x = a,a′ may be a strong assumption. However, this assumption is analogous to the conditional exchangeability assumption between treatment arms often made in observational studies. We also note that conditional exchangeability can hold even if W does not include all causes of the outcome whose distribution differ between the study and target populations. For example, consider the single world intervention graph³ S ← W ← U → Y(x); here, conditioning on W is sufficient to ensure S ⊥ Y(x)|W yet W is not a cause of the outcome.

The first example Huitfeldt and Stensruds’ provide of a situation in which they assert the conditional exchangeability assumption may be problematic in fact is among the situations where generalizability is possible: a set of measured, pre-treatment covariates thought to be associated with the outcome serve to selectively recruit patients into a trial. As long as some “low-risk” patients are also enrolled to satisfy the positivity assumption that 0 < P(S = 1|W = w) for all w such that 0 < P(W = w) and the same set of covariates W are measured in the target population, because the set W is known (by virtue of explicitly stated recruitment criteria), either direct standardization or inverse probability weighting could be used to generalize trial results to the target population.

The other example provided by Huitfeldt and Stensrud, in which only men are recruited into a trial on the effect of homeopathy on heart disease and the target population is women, seems to conflate several issues. First, this is an example where methods for transportability rather than generalizability are required, because the study sample is not a subset of the target population. Furthermore, depending on the hypothesized causal relationship between homeopathy and heart disease, this scenario either: 1) violates the positivity assumption if sex is considered to be a covariate in W; or 2) is not problematic, if sex is not considered to be a cause of heart disease.⁴

Returning to the assumption of conditional exchangeability between the study sample and the target population, Huitfeldt and Stensrud suggest, as was noted by Cole and Stuart,⁵ that the set W could be narrowed to include only covariates that are effect modifiers on a particular scale of interest, when the investigators are willing to specify a scale. This relaxation of the exchangeability assumption is potentially useful in certain situations. However, the set of covariates sufficient to generalize one effect measure (e.g., the risk ratio) under this relaxed assumption will not necessarily be the same set of covariates sufficient to generalize a different effect measure (e.g., the risk difference). Multiple effect measures are often presented in research, making a broader exchangeability assumption more relevant. Furthermore, while relaxing the exchangeability assumption to be conditional only on effect modifiers on a particular scale may relieve the investigator of measuring some components of W in the target population, W should still be measured in the study sample to assess whether the effect measure of interest is indeed homogeneous across those components of W the investigators would like to avoid measuring in the target population.

Finally, we note two typographical issues in our paper. First, equation 5 should be

$$E\left[Y\left(a\right)\right]=\frac{E\left[YI\left(A=a,S=1\right)/P\left(S=1|W\right)\right]}{E\left[I\left(A=a,S=1\right)/P\left(S=1|W\right)\right]}$$

with the weights inside the brackets. Furthermore, in the Appendix, the IPW estimator is written as $E\left[\widehat{Y\left(a\right)}\right]$ which erroneously suggest that the expectation is being taken of $\widehat{Y\left(a\right)}$. Instead, the intent was to write the IPW estimator as $E\left[\widehat{Y\left(a\right)}\right]$, i.e., with the circumflex accent extending over the entire expectation E[Y(a)].