A hierarchical meta-analysis for settings involving multiple outcomes across multiple cohorts

Abstract

Evidence from animal models and epidemiological studies has linked prenatal alcohol exposure (PAE) to a broad range of long-term cognitive and behavioural deficits. However, there is a paucity of evidence regarding the nature and levels of PAE associated with increased risk of clinically significant cognitive deficits. To derive robust and efficient estimates of the effects of PAE on cognitive function, we have developed a hierarchical meta-analysis approach to synthesize information regarding the effects of PAE on cognition, integrating data on multiple outcomes from six U.S. Iongitudinal cohort studies. A key assumption of standard methods of meta-analysis, effect sizes are independent, is violated when multiple intercorrelated outcomes are synthesized across studies. Our approach involves estimating the dose–response coefficients for each outcome and then pooling these correlated dose–response coefficients to obtain an estimated “global” effect of exposure on cognition. In the first stage, we use individual participant data to derive estimates of the effects of PAE by fitting regression models that adjust for potential confounding variables using propensity scores. The correlation matrix characterizing the dependence between the outcome-specific dose–response coefficients estimated within each cohort is then run, while accommodating incomplete information on some outcome. We also compare inferences based on the proposed approach to inferences based on a full multivariate analysis.

1 |. INTRODUCTION

Meta-analysis is commonly used to synthesize quantitative evidence across studies to generate a summary exposure or treatment effect that is more precise than estimates obtainable from individual studies alone. Traditionally, meta-analysis is based on estimated effect sizes. Although it is cost-effective and easy to implement, this approach has been criticized on the grounds that it is prone to ecological and confounding bias (Riley & Steyerberg, 2010; Simmonds & Higgins, 2007). Individual participant data (IPD) meta-analysis can help mitigate such biases and accommodate missing data at the participant level (Riley et al., 2010). Moreover, with access to individual level data, a choice can be made between a fully specified multivariate IPD and a two-stage IPD approach. The full multivariate approach generally uses mixed-effects multilevel regressions to model between and within heterogeneity and quantify the effect of interest in a single model. Although this approach is considered flexible, it may be challenging for conducting and communicating the findings, particularly visualization using the hallmark forest plot. The alternative IPD approach involves modelling the data in two stages. In the first step, effect size estimates for each study are obtained using separate regression models. In the second step, standard methods of meta-analysis are used to obtain an overall estimate. A key assumption with standard methods of meta-analysis is that effect sizes are independent. This assumption is violated when multiple outcomes correlated are synthesized across studies. To avoid the dependence of the effect sizes, several ad-hoc methods have been proposed including averaging the effect sizes or selecting one effect size per study. A major disadvantage of these ad-hoc approaches is that they do not make use of all available data to address the relevant research questions (Cheung, 2019).

More principled approaches have been proposed to deal with correlated effects when conducting IPD meta-analysis. These advances include multivariate meta-analysis which has been used to jointly synthesize the outcomes observed across studies to estimate multiple pooled effects simultaneously (Riley et al., 2007). However, using multivariate meta-analysis is less straightforward when studies do not consistently report on the same outcomes (Van den Noortgate et al., 2014). Another approach is the three-level meta analytic model (Cheung, 2013; Konstantopoulos, 2011; Van den Noortgate et al., 2013), which has been used to adjust for dependence of effect sizes within clusters. This approach treats participants within each cluster as contributing only one effect size, so the nonindependence is handled within the nested structure of the effects (Cheung, 2019). An alternative approach is based on a two-stage meta-analysis that uses summary measures. In this approach, dependency among effect sizes is handled via robust variance estimation in which the dependence between the outcomes is not explicitly modelled, but instead the standard errors for the overall treatment effect or meta-regression coefficients are adjusted (Hedges et al., 2010). This approach may require making a reasonable guess about the between-outcome correlations to estimate the between-study variance and to approximate the optimal weights.

In this paper, we propose an innovative approach: a hierarchical meta-analysis for the settings in which each cohort study provides multiple outcomes, resulting in correlated estimated effect sizes. The work is motivated by a project that involves the integration of data from six longitudinal cohorts, each of which used multiple interrelated tests and assessment tools to measure child cognition. Cognition is not directly observable since there is no single measure that can be regarded as a highly reliable indicator of cognition. These six longitudinal cohorts were conducted independently and used different neuropsychological test batteries to assess IQ and the same domains of cognitive function, including learning and memory, executive function, and academic achievement in reading and in mathematics (Jacobson et al., 2021). All these tests provide a comprehensive assessment of the child’s cognitive function. A major strength of the proposed approach is that it facilitates the synthesis of data across diverse outcomes within each cohort, and thereby an assessment of consistency of patterns across cohorts. Furthermore, by including multiple correlated responses from each child, the analyses make full use of available data to maximize the efficiency of estimation and enhance power of associated tests for effects. Robust variance estimation ensures valid inferences at each stage of the analysis.

In the proposed approach, we first derive the estimates and the standard errors by fitting regression models for each separate outcome of interest. By contrast to existing methods of two-stage IPD analysis, we account for the correlated effect sizes within each cohort at this stage. Specifically, within-study robust covariance matrices are obtained at this stage to be combined at the second stage. Within each cohort, not all outcomes were observed for all children. This additional complexity was addressed in the estimation of robust covariance matrices. Specifically, we derived a formula for pairwise correlation between the estimated effects using an adjustment that accounts for the fact that we have partially observed outcome measures for some children. In the second stage, we combine the summary measures within each cohort using a random-effects model. In the last stage of our hierarchical meta-analytic approach, we combine the independent, cohort-specific effect size estimates in a random-effects model to obtain a global measure of the effect size across cohorts (Lin & Zeng, 2010; Whitehead, 2002). We compare and contrast the findings from our proposed approach to those obtained using a full multivariate analysis in order to determine the degree to which the results of these two models coincide.

The remainder of the article is organized as follows. In Section 2, we introduce our motivating application which is a meta-analysis of correlated outcomes used to assess the effect of prenatal alcohol exposure (PAE) on cognition in six cohort studies. In Section 3, we introduce notation and describe the two-stage analysis and modelling framework used to combine multiple correlated outcomes within a single cohort. In Section 4, we present the modelling framework used to combine pooled effect size estimates across cohorts. In Section 5, we compare and contrast the proposed approach with the corresponding one-stage approach using simulation studies. In Section 6, we illustrate our method using data from our motivating application. Finally, in Section 7, we discuss the strengths and limitations of our method.

2 |. EFFECT OF PRENATAL ALCOHOL EXPOSURE ON COGNITION

Evidence from animal models and epidemiological studies has linked PAE to a broad range of cognitive and behavioural deficits, growth restriction, and physical anomalies, which are known collectively as fetal alcohol syndrome disorders (FASD) (Carter et al., 2016; Jacobson et al., 2004, 2008; Mattson et al., 2019). Fetal alcohol syndrome (FAS), the most severe of the FASD, is characterized by distinctive cranio-facial dysmorphology (small palpebral fissures, flat philtrum, thin vermillion), small head circumference, and growth restriction (Hoyme et al., 2005; Stratton et al., 1996) while partial FAS (PFAS) is diagnosed in the presence of the characteristic alcohol-related facial dysmorphology, a history of PAE and growth restriction, small head circumference, or central nervous system (CNS) impairment. Individuals with PAE who lack the characteristic pattern of dysmorphic features but exhibit cognitive and/or behavioural impairment are diagnosed as having alcohol-related neurodevelopmental disorder (ARND), which is the most prevalent FASD. Although the diagnosis of ARND requires a confirmed history of maternal alcohol consumption during pregnancy, there is no information in the scientific literature regarding the levels of PAE associated with an increased risk of clinically significant adverse effects.

Between 1975 and 1993, the National Institutes of Health (NIH) funded six longitudinal cohort studies in four U.S. cities–Detroit (Jacobson et al., 1993), Pittsburgh (Day et al., 1991; Richardson et al., 1999) (two cohorts), Atlanta (Brown et al., 1998; Coles et al., 2006) (two cohorts), and Seattle (Streissguth et al., 1981); these are described briefly in Appendix A. To enhance efficiency when examining the effects associated with different levels and patterns of PAE, the data are synthesized across the studies. The sample sizes in the individual longitudinal cohort studies range between 138 and 720. Participant retention was good to excellent from childhood to adolescence $(\mathit{\text{median}} =90.3\%; \mathit{\text{range}} =86.4 to 96.3\% )$. Retention from adolescence to young adulthood was excellent (≥ 91.5%) in the Atlanta-1, Seattle, and 2 Pittsburgh cohorts. The Detroit young adult follow-up, which focused on neuroimaging, was funded to assess only a subsample (43.6%) of the cohort. In all but one of these studies, mothers were recruited and interviewed prospectively about their alcohol use during pregnancy, and their children were followed longitudinally from infancy through young adulthood; one of the Atlanta cohort studies (Schuetze et al., 2007) recruited the mothers shortly following delivery, interviewed them about their drinking during pregnancy, and followed the children through early childhood. The number of maternal interviews varies by cohort. In these interviews, detailed information regarding quantity and frequency of drinking during pregnancy and dose per occasion were obtained. Data on alcohol consumption during pregnancy from all six cohorts are summarized in terms of ounces of absolute alcohol averaged across pregnancy (oz AA). In all studies, investigators administered a variety of neuropsychological tests to assess IQ and four domains of cognitive function: learning and memory, executive function, and academic achievement in reading and in mathematics.

Although there was some variation in the particular auxiliary covariates collected across the different studies, data on a broad range of covariates were provided by each cohort.

3 |. NOTATION AND MODEL FORMULATION

Let $Y_{ijk}$ be the random variable representing response $k$ for individual $j$ in cohort $i,k=1,\dots ,K_i,j=1,\dots ,J_i$, where $J_i$ is the number of individuals in cohort $i,i=1,\dots ,l$. Let $A_{ij}$ be the exposure of interest (i.e., prenatal alcohol exposure) for individual $j$ in study $i$ and $S_{ij}$ be their corresponding propensity score.

$$Y_{ijk}={\alpha }_{ik}+B_{ik}{A}_{ij}+{\gamma }_{ik}{S}_{ij}+E_{ijk},$$ (1)

where $B_{ik}$ is the effect of a one-unit increase in $A_{ij}$ (alcohol volume) on the mean for response $k$ in cohort $i$ given the propensity score $S_{ij},j=1,\dots ,J,k=1,\dots ,K,i=1,\dots ,l$.

Because the sets of covariates measured differ between cohorts, the propensity score is estimated separately for each cohort using the two-part generalized propensity score (Akkaya Hocagil et al., 2021). By using the two-part generalized propensity score, we model the causal effect of a semicontinuous exposure variable $A$ on an outcome $Y$ in the presence of a set of confounding variables $Z={\left(Z_1,\dots ,Z_p\right)}^{\prime }$. Specifically, we let $A^{+}=$ $I(A>0)$ indicates a positive value for $A$, and $\pi \left(Z\right)=E\left(A^{+}\mid Z\right)$. We consider a binary regression model defined by the link function $g(\cdot )$ mapping the interval [0,1] onto the real line and setting $g\left(\pi \left(Z;{\alpha }_1\right)\right)={\bar{Z}}^{\prime }{\alpha }_1$ where $\bar{Z}={\left(1,Z^{\prime }\right)}^{\prime }$ and ${\alpha }_1={\left({\alpha }_{10},{\alpha }_{11},\dots ,{\alpha }_{1p}\right)}^{\prime }$.

We also let $P\left(A\le a\mid A^{+}=1,Z;{\alpha }_2\right)=F^{+}\left(a\mid Z;{\alpha }_2\right)$ denote the cumulative distribution function for the positive part of $A$ given $Z$, and $A^{+}=1$ is indexed by a $(p+1)\times 1$ parameter ${\alpha }_2$. The full distribution for $A$ is therefore indexed by $\alpha ={\left({\alpha }_1^{\prime },{\alpha }_2^{\prime }\right)}^{\prime }$. A key requirement of the model for $A\mid Z,A^{+}=1$ is that it involves a simple way to compute $E\left(A\mid Z,A^{+}=1;{\alpha }_2\right)=\mu \left(Z;{\alpha }_2\right)$; we adopt a generalized linear model and ultimately compute

$$S=E(A\mid Z;\alpha )=\pi \left(Z;{\alpha }_1\right)\mu \left(Z;{\alpha }_2\right)$$ (2)

as the marginal mean for $A\mid Z$ based on the two-part model formulation (Akkaya Hocagil et al., 2021).

After we estimated the propensity score, we assume here that conditioning on the propensity score renders the exposure variable independent of all confounders and so that it is sufficient to condition on $S_{ij}$ in (1) rather than the confounders themselves to mitigate the effect of confounding (Rosenbaum & Rubin, 1983).

The parameter ${\gamma }_{ik}$ characterizes the effect of the propensity score on outcome $k$ in study $i$ (for a given level of alcohol exposure) and $E_{ijk}$ is the error term which has mean zero and variance ${\sigma }_{ik}^2,k=1,\dots ,K,i=1,\dots ,l$.

We suppose that the effects $B_{ik}$, the effect of prenatal alcohol exposure on the mean for response $k$ in cohort $i$, vary about some average exposure effect in cohort $i$ with

$$B_{ik}\mid {\beta }_i~N\left({\beta }_i,{\varphi }_i\right),$$ (3)

where ${\beta }_i$ is the exposure effect for cohort $i$ and ${\varphi }_i$ represents the heterogeneity of the response-specific exposure effects within cohort $i$.

We suppose that the average cohort-specific exposure effects are independent and vary about an overall exposure effect ${\beta }_0$ with,

$${\beta }_i~N\left({\beta }_o,{\eta }^2\right);$$ (4)

here, ${\beta }_o$ represents the “average effect” of a one-unit increase in the exposure across all cohorts and is our parameter of ultimate interest. The variance ${\eta }^2$ in (4) reflects the extent of heterogeneity of the cohort-specific exposure effects.

In the next two subsections, we describe a two-stage approach to estimation and inference with data from a single cohort, and in Section 4, we show how to synthesize cohort-specific exposure effects to obtain an estimate for the average effect of a one-unit increase in the exposure across all cohorts.

3.1 |. Stage I estimation for a single cohort

In this section, we temporarily omit the subscript $i$ and describe a two-stage approach to estimate the average exposure effect for a single cohort where the effects are correlated. Before model fitting, we standardize the responses so that they have the same first two moments as the full-scale IQ variable which has a mean of 100 and a standard deviation of 15. By conducting this standardization, the exposure effects can be expressed in terms of the decrement in IQ associated with a one-unit increase in prenatal alcohol exposure (Axelrad et al., 2007).

For the first stage, we fit separate linear models for each response, assuming

$$Y_{jk}={\alpha }_k+B_k{A}_j+{\gamma }_k{S}_j+E_{jk},$$ (5)

where $B_k$ is the effect of a one-unit increase in $A_j$ (alcohol in our application) on the mean for response $k$, given the propensity score $S_j,j=1,\dots ,J,k=1,\dots ,K$. The parameter ${\gamma }_k$ characterizes the effect of the propensity score for a given level of alcohol exposure and $E_{jk}$ is the error term that has mean zero and variance ${\sigma }_k^2,k=1,\dots ,K$.

We suppose that the effects $B_k,k=1,\dots ,K$ vary about some average exposure effect, with

$$B_k~N(\beta ,\varphi ),$$ (6)

independently and identically distributed and $\beta$ is the average exposure effect. The variance $\varphi$ reflects the extent of heterogeneity of the response-specific exposure effects for a single cohort.

If we let $X_{jk}={\left(1,A_j,S_j\right)}^{\prime }$ be the covariate vector, we can write

$$Y_{jk}=X_{jk}^{\prime }{\theta }_k+E_{jk},$$ (7)

where ${\theta }_k={\left({\alpha }_k,B_k,{\gamma }_k\right)}^{\prime }$. We assume $E_{jk}⫫\left(A_j,B_k,S_j\right)$ with $E_{jk}~N\left(0,{\sigma }_k^2\right)$ are i.i.d. for $j=1,\dots ,J,k=\mathrm{1,2},\dots ,K$. Note that $X_{jk}$ does not vary by the response type because the exposure variable $A_j$ and the propensity score $S_j$ are individual level covariates, but we retain this notation for generality.

We next define $K\times 1$ vectors $Y_j={\left(Y_{j1},Y_{j2},\dots ,Y_{jK}\right)}^{\prime },\alpha ={\left({\alpha }_1,\dots ,{\alpha }_K\right)}^{\prime },B={\left(B_1,\dots ,B_K\right)}^{\prime }$ and $\gamma ={\left({\gamma }_1,\dots ,{\gamma }_K\right)}^{\prime }$ and a $K\times 3K$ covariate matrix

$$X_j^{\prime }=\left[\begin{array}{ccccc}{X}_{j1}^{\prime }& 0& 0& \dots & 0\\ 0& X_{j2}^{\prime }& 0& \dots & 0\\ ⋮& & \ddots & & 0\\ 0& \dots & 0& 0& X_{jK}^{\prime }\end{array}\right],$$ (8)

The model given by (7) can then be represented in a unifying model

$$Y_j=X_j^{\prime }\theta +E_j$$ (9)

where $\theta ={\left({\theta }_1^{\prime },\dots ,{\theta }_K^{\prime }\right)}^{\prime }$ is a $3K\times 1$ vector of parameters, $E_j={\left(E_{j1},\dots ,E_{jK}\right)}^{\prime }$ and $E_j~N(0,\mathrm{\Sigma })$, where $\mathrm{\Sigma }$ is a $K\times K$ covariance matrix with diagonal entries ${\mathrm{\Sigma }}_{kk}={\sigma }_k^2=\mathrm{v}\mathrm{a}\mathrm{r}\left(E_{jk}\right)$. The off-diagonal entries ${\mathrm{\Sigma }}_{kl}={\sigma }_{kl}=\mathrm{c}\mathrm{o}\mathrm{v}\left(E_{jk},E_{je}\right)$ accommodate a conditional dependence (given $X_{jk},X_{jj}$, $B$) between the responses from the same individual.

Following the separate fit of the $K$ linear models at Stage I, we have estimates $\left({\hat{\theta }}_k,{\hat{\sigma }}_k^2\right),k=\mathrm{1,2},\dots ,K$. The covariance term characterizing the dependence between errors are then estimated to facilitate estimation of a robust covariance matrix characterizing the dependence between the Stage I estimators $\left({\hat{\theta }}_1,\dots ,{\hat{\theta }}_K\right)$. The elements of interest in $\hat{\theta }$ are the parameter estimates ${\hat{B}}_1,\dots ,{\hat{B}}_K$, which are consistent for $B_1,\dots ,B_K$, respectively. In the second stage of estimation, these estimates are pooled over to obtain a single estimate of the global measure of the causal effect denoted by $\beta$ in (3)

We begin the second stage by estimating the covariance between the errors $\mathrm{c}\mathrm{o}\mathrm{v}\left(E_{jk},E_{jl}\right)={\sigma }_{kl},I\ne k=1,\dots ,K$, characterizing the dependence between the Stage I estimators ${\hat{\theta }}_1,\dots ,{\hat{\theta }}_K$. The challenge in estimating the covariance between the errors $\mathrm{c}\mathrm{o}\mathrm{v}\left(E_{jk},E_{jl}\right)={\sigma }_k,I\ne k=1,\dots ,K$ is that not all responses were observed for all children in the study. We assume that the responses are missing at random (MAR) (Little & Rubin, 2019). To accommodate the fact that not all individuals contribute data for all responses, we introduce the indicators $R_{jk}=I\left(Y_{jk}\right.$ is observed $),k=1,\dots ,K$.

Specifically, if $S_{jk}\left({\theta }_k\right)=X_{jk}\left(Y_{jk}-X_{jk}^{\prime }{\theta }_k\right)$ is the desired contribution from individual $j$ to the score function for ${\theta }_k$ given $B$, the observed data score equation for estimating ${\theta }_k$ at Stage I can be written as

$$S_k\left({\theta }_k\right)=\sum _{j=1}^J{R}_{jk}{S}_{jk}\left({\theta }_k\right)=0,$$ (10)

the solutions to which are

$${\hat{\theta }}_k=\sum _{j=1}^J{R}_{jk}{\left(X_{jk}{X}_{jk}^{\prime }\right)}^{-1}{X}_{jk}{Y}_{jk},k=1,\dots ,K.$$ (11)

Then we can obtain the maximum likelihood estimate of ${\sigma }_k^2$ in the presence of partially observed outcomes as

$$\widehat{\mathrm{v}\mathrm{a}\mathrm{r}}\left(E_{jk}\right)={\hat{\sigma }}_k^2=\frac{\sum _{j=1}^J{R}_{jk}{\left(Y_{jk}-X_{jk}^{\prime }{\hat{\theta }}_k\right)}^2}{n_k},$$ (12)

where $n_k={\sum }_{j=1}^J{R}_{jk}$ is the number of individuals contributing to the estimation of ${\theta }_k,k=\mathrm{1,2},\dots ,K$. Similarly, we also obtain the maximum likelihood covariance estimate as

$$\widehat{\mathrm{c}\mathrm{o}\mathrm{v}}\left(E_{jk},E_{jl}\right)={\hat{\sigma }}_{kl}=\frac{\sum _{j=1}^J{R}_{jk}{R}_{jl}\left(Y_{jk}-X_{jk}^{\prime }{\hat{\theta }}_k\right)\left(Y_{jl}-X_{jl}^{\prime }{\hat{\theta }}_l\right)}{n_{kl}},$$ (13)

where $n_{\mathrm{k}\mathrm{l}}={\sum }_{j=1}^J{R}_{\mathrm{j}\mathrm{k}}{R}_{\mathrm{j}}$, which is consistent under a missing at random assumption (Little & Rubin, 2019).

We then let $\stackrel{\mathrm{ˆ}}{\mathrm{\Sigma }}$ denote the estimated covariance matrix for the errors where ${\sigma }_{\mathrm{k}}^2$ and ${\sigma }_{\mathrm{k}\mathrm{l}}$ are replaced by (12) and (13), respectively.

3.2 |. Stage II: Synthesis across responses within a cohort

To consider the synthesis of estimators across all responses, we note that

$$E\left({\hat{B}}_k-\beta \right)=E\left\{\left({\hat{B}}_k-B_k\right)+\left(B_k-\beta \right)\right\}=0,$$

so $\hat{B}$ is composed of $K$ dependent unbiased estimators of $\beta$. Thus,

$$\hat{B}~\mathrm{M}\mathrm{V}\mathrm{N}\left(\mu \right(\beta ),\mathrm{\Psi }(\varphi \left)\right)$$ (14)

asymptotically, where $\mu \left(\beta \right)$ is a $K\times 1$ vector with each element equal to $\beta$ and $\stackrel{\mathrm{ˆ}}{\mathrm{\Psi }}\left(\varphi \right)=J^{-1}\stackrel{\mathrm{ˆ}}{\mathrm{\Gamma }}+\mathrm{\Delta }\varphi$ denotes the unconditional covariance matrix for $\hat{B}$ where ${\mathrm{\Psi }}_{kk}\left(\varphi \right)=\mathrm{v}\mathrm{a}\mathrm{r}\left({\hat{B}}_k\right)=J^{-1}{\mathrm{\Gamma }}_{kk}+\varphi ,k=1,\dots ,K,{\mathrm{\Psi }}_{kl}\left(\varphi \right)=\mathrm{c}\mathrm{o}\mathrm{v}\left({\hat{B}}_k,{\hat{B}}_l\right)=J^{-1}{\mathrm{\Gamma }}_{kl}k\ne I=1,\dots ,K,\mathrm{\Delta }$ is a $K\times K$ identity matrix and $\varphi$ reflects the extent of heterogeneity of the response-specific exposure effects for a single cohort. We provided detailed derivation of the covariance matrix for $\hat{B}$ in Appendix B.

Then, we specify a pseudo-likelihood $\mathit{P}\mathit{L}(\beta ,\varphi )$ for $(\beta ,\varphi )$ given by

$$PL\left(\beta ,\varphi \right)\propto \frac{1}{(2\pi {)}^{\frac{K}{2}}\sqrt{\stackrel{\mathrm{ˆ}}{\mathrm{\Psi }}\left(\varphi \right)\mid }}\text{exp}\left(-\frac{1}{2}(\hat{B}-\mu (\beta ){)}^{\prime }{\stackrel{\mathrm{ˆ}}{\mathrm{\Psi }}}^{-1}(\varphi )(\hat{B}-\mu (\beta ))\right).$$ (15)

Note that (15) could be maximized with respect to $(\beta ,\varphi )$, but we proceed in a computationally convenient iterative approach based on (15). Given an estimate ${\varphi }^{\left(r\right)}$, we compute an estimate ${\beta }^{\left(r\right)}$ based on a linear combination of ${\hat{B}}_1,\dots ,{\hat{B}}_K$. The most efficient linear estimator of $\beta$ has the form

$$\hat{\beta }=\left[{\mathrm{𝟙}}^{\prime }\left[\mathrm{\Psi }\right(\varphi ){]}^{-1}\hat{B}\right]/\left[{\mathrm{𝟙}}^{\prime }\left[\mathrm{\Psi }\right(\varphi ){]}^{-1}\mathrm{𝟙}\right].$$ (16)

Here, we replace $\mathrm{\Psi }\left(\varphi \right)$, the covariance matrix of $\hat{B}$, with an estimator $\stackrel{\mathrm{ˆ}}{\mathrm{\Psi }}\left({\varphi }^{\left(r\right)}\right)=J^{-1}\stackrel{\mathrm{ˆ}}{\mathrm{\Gamma }}+\mathrm{\Delta }{\varphi }^{\left(r\right)}$. We could invert $\stackrel{\mathrm{ˆ}}{\mathrm{\Psi }}\left({\varphi }^{\left(r\right)}\right)$, but in practice, it may be difficult and while a generalized inverse could be used, the weights resulting from this approach were often found to vary greatly in magnitude and even in sign. A more stable linear estimate using inverse variance weights, whereby we replace $\mathrm{\Psi }\left(\varphi \right)$ with $\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}\left(J^{-1}{\stackrel{\mathrm{ˆ}}{\mathrm{\Gamma }}}_{kk}+{\varphi }^{\left(r\right)},k=1,\dots ,K\right)$ in (16) to obtain ${\hat{\beta }}^{\left(r\right)}$. We then maximize $\mathit{P}\mathit{L}\left({\hat{\beta }}^r,\varphi \right)$ with respect to $\varphi$ to obtain ${\varphi }^{(r+1)}$, with which we recompute ${\hat{\beta }}^{(r+1)}$ and repeat iteratively until convergence; we let $(\hat{\beta },\hat{\varphi })$ denote the estimates upon convergence. A robust variance estimate is then obtained for $\hat{\beta }$ based on $\stackrel{\mathrm{ˆ}}{\mathrm{\Psi }}\left(\hat{\varphi }\right)$, which is given by $\widehat{\mathrm{v}\mathrm{a}\mathrm{r}}\left(\hat{\beta }\right)={\left[{\mathbb{I}}^{\prime }\stackrel{\mathrm{ˆ}}{\mathrm{\Psi }}\left(\hat{\varphi }\right)\mathbb{I}\right]}^{-1}$.

3.3 |. An alternative one-stage (fully specified multivariate) approach

The parameters estimated in Sections 3.1 and 3.2 can alternatively be fitted in one step via software for fitting hierarchical mixed effect linear models. To do so, we define $K-1$ covariates $T_{jkr}=I(k=r),r=2,\dots ,K$, which indicates the response $k,k=2,\dots ,K$. We may put these $K-1$ indicators in vector format and define the $(K-1)\times 1$ vector $T_{jk}={\left(T_{jk2},\dots ,T_{jkk}\right)}^{\prime }$. We consider the first outcome as the reference type and let $T_{ik2}=1$ for the second outcome, $T_{ik3}=1$ for the third endpoint, and so on. Then we fit the model

$$Y_{jk}=\alpha +B_k{A}_j+\gamma S_j+{\tau }^{\prime }{T}_{jk}+E_{jk},k=1,\dots ,K,$$

where $\tau ={\left({\tau }_2,\dots ,{\tau }_K\right)}^{\prime }$ is $(K-1)\times 1$ vector with ${\tau }_r={\alpha }_r-{\alpha }_1$ and $r=2,\dots ,K$. We assume $B_k~N(\beta ,\varphi )$ as specified in (3) where $B_k$ is the effect of a one-unit increase in $A_j$ on the mean of response $k$ given the propensity score $S_j,\beta$ is the parameter of ultimate interest representing the “average causal effect” of a one-unit increase in the exposure across all responses within the cohort, and $\varphi$ characterizes the degree of heterogeneity in the effect across responses. We also assume $E_j={\left(E_{j1},\dots ,E_{jK}\right)}^{\prime }$ is a $K\times 1$ error term with $E_j~\text{MVN}(0,\mathrm{\Sigma })$ with $\mathrm{\Sigma }$ a $K\times K$ covariance matrix as in Section 3.1. The one-step approach involves simultaneous estimation of all fixed effects, $\beta ,\varphi$, and $\mathrm{\Sigma }$ at once. This can be fitted using software for fitting hierarchical linear mixed effects models.

After fitting hierarchical linear mixed effects model for each cohort separately, we let ${\tilde{\beta }}_i$ denote the estimate of ${\beta }_i$ obtained from fitting the hierarchical model to the data from cohort $i$ and ${\tilde{V}}_i\left({\tilde{\beta }}_i\right)$ denote the corresponding variance estimate based on the observed information matrix, $i=1,\dots ,l$.

4 |. SYNTHESIS ACROSS COHORTS

In the previous section, we described methods for synthesizing data across multiple outcomes to obtain estimates of the global causal effect using a two-stage approach based on fitting a hierarchical mixed effect model. These methods were based on analysing data from a single cohort. Here, we describe how to combine cohort-specific estimates to obtain an overall estimate of a causal effect while accommodating possible heterogeneity. The approach described in Section 3.2 is an extension of the approach described by Viechtbauer and implemented in the metafor package (Viechtbauer, 2010) which deals with independent estimates; Section 3.2 adapted the methods to deal with dependent effect estimates so what follows is a simplification of the approach for the last stage of the data synthesis. We describe it briefly as follows:

We consider ${\beta }_i$ as the global causal effect of exposure in cohort $i$ reflecting the impact of an increment in the volume of prenatal alcohol exposure on the common underlying construct; we let ${\hat{\beta }}_i$ be the corresponding estimate. Note that the studies draw individuals from different populations and so the composition of the samples varies across cohorts. Moreover, the methods used to measure exposure and the specific outcome measures differ between studies, even though they were measuring the same latent attributes regarding cognition. We therefore wish to accommodate a component of variation between studies (heterogeneity) for the true effects which we accomplished by use of a random effects model of the form

$${\hat{\beta }}_i={\beta }_i+{ϵ}_i,$$ (17)

$${\beta }_i={\beta }_{\circ }+u_i,$$ (18)

where we let ${ϵ}_i~N\left(0,{\hat{V}}_i\left({\hat{\beta }}_i\right)\right)$ reflect the sampling variation of the estimator from cohort $i$ about the true effect ${\beta }_i$, and $u_i~N\left(0,{\eta }^2\right)$ reflects the heterogeneity of the global cohort-specific causal effects across studies. The parameter ${\beta }_{\mathrm{o}}$ represents the overall global effect, which is the parameter of ultimate interest. Through this variance decomposition then upon introducing the heterogeneity between studies, we have $\widehat{\mathrm{v}\mathrm{a}\mathrm{r}}\left({\hat{\beta }}_i\right)={\hat{V}}_i\left({\hat{\beta }}_i\right)+{\eta }^2$. The synthesis is achieved in a similar spirit to Section 3.2 whereby we consider a pseudo-likelihood of the form

$$PL\left({\beta }_{\circ },\eta \right)\propto \prod _{i=1}^I\left\{\frac{1}{(2\pi {)}^{I/2}\sqrt{\left({\hat{V}}_i\left({\hat{\beta }}_i\right)+{\eta }^2\right)}}\mathrm{e}\mathrm{x}\mathrm{p}\left(-\frac{{\left({\hat{\beta }}_i-{\beta }_{\circ }\right)}^2}{2\left({\hat{V}}_i\left({\hat{\beta }}_i\right)+{\eta }^2\right)}\right)\right\}.$$ (19)

The pooled exposure effect estimate $\hat{\beta }$ 。 is obtained as a weighted average of the ${\hat{\beta }}_i$ terms with cohort weights equal to the inverse of $\hat{V}\left({\hat{\beta }}_i\right)+$ ${\hat{\eta }}^2$ where ${\hat{\eta }}^2$ is obtained as the solution to iteratively maximizing (19). The R package “metafor” can be used to carry out this final stage of the data synthesis. If the linear model of Section 3.3 is used for simultaneous estimation of the overall causal effect, then ${\tilde{\beta }}_i$ and ${\tilde{V}}_i\left({\tilde{\beta }}_i\right)$ can be used in a similar fashion to obtain the estimator ${\tilde{\beta }}_0$.

5 |. SIMULATION STUDIES

For the simulation studies, we consider $k$ correlated continuous outcomes from a single study. We generated outcomes from the following linear regression model:

$$Y_{jk}={\alpha }_k+{\beta }_k{X}_j+{\gamma }_k{Z}_j+E_{jk},$$ (20)

where $Y_{jk}$ is the random variable representing response $k$ for individual $j,k=1,\dots ,K,j=1,\dots ,J$, and ${\beta }_k$ is the effect of a one-unit increase in $X_j$ on the mean for response $k$ given the covariate $Z_j$. We let ${\beta }_k$ vary about some average exposure effect within a study, with ${\beta }_k~N\left(\beta ,{\tau }^2\right)$. The parameter ${\gamma }_k$ characterizes the effect of the covariate for a given level of exposure, $X_j$. We also assume $E_j=\left(E_{j1},\dots ,E_{jK}\right)$ is a $K\times 1$ error term with $E_j~MVN(0,\mathrm{\Sigma })$ with $\mathrm{\Sigma }$ a $K\times K$ covariance matrix. The simulations were performed in different scenarios. We generated the effect size for the exposure, ${\beta }_k$, from the normal distribution with mean 3 and variance ${\tau }^2$. Scenarios were created by manipulating the number of outcomes $\left(k\right)$ and varying between-study heterogeneity $\left({\tau }^2\right)$. We consider the scenarios in which the number of outcomes is equal to 3, 5, and 10 and ${\tau }^2$ takes the values 0.10, 0.25, and 0.50. For each combination of the simulation parameters, we generated 1000 datasets with the sample size of 500 for each outcome. For each dataset, we performed the two types of meta-analysis, that is, the one based on the proposed approach versus the full multivariate analysis.

We evaluated the performance of the proposed approach in simulation settings previously described, over 1000 iterations. The estimates of interest were the average exposure effect. To allow for a comprehensive comparison, performance was assessed on a range of metrics: empirical mean bias (EBIAS), average model based standard error (ASE), empirical standard error (ESE), and coverage probability (CP). The results are summarized in Table 1 and some interesting patterns emerge.

TABLE 1.

Results of a simulation study assessing the performance of our hierarchical meta-analysis and a full multivariate analysis in estimating the effect size for the exposure in a variety of settings

		Hierarchical meta-analysis				One-stage method
${\tau }^2$	$k$	EBIAS	ASE	ESE	ECP (%)	EBIAS	ASE	ESE	ECP (%)
0.10
	10	0.007	0.16	0.14	96.0	<0.001	0.10	0.14	92.0
	5	0.006	0.20	0.20	96.0	<0.001	0.13	0.14	99.0
	3	0.009	0.23	0.25	95.0	<0.001	0.14	0.14	93.0
0.25
	10	<0.000	0.11	0.10	95.0	0.020	0.18	0.24	88.0
	5	<0.000	0.13	0.13	95.0	0.030	0.19	0.21	91.0
	3	0.010	0.15	0.15	95.0	0.020	0.21	0.23	94.0
0.50
	10	0.004	0.16	0.15	95.0	0.040	0.26	0.29	96.0
	5	0.009	0.21	0.20	94.0	0.060	0.27	0.30	97.0
	3	0.010	0.23	0.24	94.0	0.120	0.31	0.34	92.0

Overall, both methods performed well in terms of empirical mean bias. There was a suggestion that one-stage method had lower bias when ${\tau }^2$ was very low, though both methods had low bias in this case as well. As ${\tau }^2$ increased, the bias seems to increase for the one-stage method. The similarity between ASE and ESE suggests that inference is working well for the hierarchical meta analysis method, over a wide range of scenarios (including numbers of outcomes and the value of $\tau$). Coverage probability for the proposed method was about the nominal 0.95 for all scenarios considered in this paper. However, the coverage probabilities were less reliable for the one-stage method. There was no particularly clear pattern, though a sense that things became a bit more unstable as ${\tau }^2$ increased. Empirical standard errors tend to be slightly larger than the average model-based standard errors, especially as ${\tau }^2$ becomes larger. This most likely reflects the limitations in standard mixed model software.

Overall, the patterns seen in Table 1 reflect the classic variance/bias tradeoff. Our proposed approach has an appealing robustness; however, the cost of this robustness is a slight increase in standard errors in settings (small ${\tau }^2$) when the one-stage method is working well.

6 |. PRENATAL ALCOHOL EXPOSURE AND COGNITIVE FUNCTION IN CHILDREN

We now come back to our motivating application that involves data from six longitudinal cohort studies to assess the effects of PAE on intelligence quotient (IQ), which is a measure of cognitive function. The proposed hierarchical meta-analytic approach is well suited to assess the effect of PAE on IQ measure because it enables us to pool data from diverse, correlated outcomes across cohorts. Table 2 lists the tests used to measure IQ that are considered in this paper along with the summary statistics. As it has been shown in Table 2, these six cohorts used different IQ tests including the Wechsler Intelligence Scale for Children (WISC), Stanford–Binet Intelligence Scales, Kaufman Assessment Battery for Children, and Differential Ability Scales (DAS). Together, all these subtests provide a comprehensive assessment of child’s IQ.

TABLE 2.

IQ related outcomes assessed at age 7 in the six cohorts

Cohort	Endpoints	Mean (SD)
Detroit (n = 336
	WISC Verbal IQ	87.4 (12.4)
	WISC Performance IQ	83.6 (13.0)
	WISC Freedom from distractibility	93.6 (14.9)
Pittsburgh 1 (n = 720)
	Stanford-Binet Verbal reasoning	100.0 (11.8)
	Stanford-Binet Abstract reasoning	85.0 (13.9)
	Stanford-Binet Quantitative reasoning	98.0 (18.0)
	Stanford-Binet Short-term memory	92.6 (15.2)
Pittsburgh 2 (n = 268)
	Stanford-Binet Verbal reasoning	96.3 (12.2)
	Stanford-Binet Abstract reasoning	88.0 (16.2)
	Stanford-Binet Quantitative reasoning	94.4 (18.4)
	Stanford-Binet Short-term memory	91.4 (15.6)
Atlanta 1 (n = 223)
	Kaufman ABC Simultaneous processing	88.6 (14.1)
	Kaufman ABC Sequential processing	89.1 (14.1)
Atlanta 2 (n = 138)
	DAS Verbal standard score	79.6 (15.2)
	DAS Nonverbal standard score	87.7 (14.9)
	DAS Spatial standard score	81.7 (14.2)
Seattle (n = 510)
	WISC Verbal IQ	106.3 (15.5)
	WISC Performance IQ	107.8 (13.9)

To yield sufficiently precise estimates of effect sizes, we considered a broad set of potential confounders when fitting separate linear models for each outcome. Because each cohort provided a somewhat different set of control variables, we employed a propensity score approach to adjust for potential confounders (Akkaya Hocagil et al., 2021). We estimated the propensity score for each cohort separately and included the propensity score in the linear model as an additional covariate as in model (5).

For each outcome in each study, the effect of alcohol was estimated from model (5). Table 3 lists the estimated effect size and standard errors from the first stage of the hierarchical meta-analytic approach. With the exception of WISC Freedom from Distractibility, and Kaufman ABC simultaneous processing in Detroit and Atlanta Cohort 1, respectively, none of the effects of PAE on IQ were statistically significant. The aim of the second stage of the proposed methods was to pool the estimates of the PAE and estimate the cohort specific overall true mean effect ${\beta }_i$ while adjusting for the fact that outcomes are correlated within a cohort and accommodating incomplete information on some outcomes. Table 4 shows the estimated effect sizes and standard errors for each cohort. Table 4 also shows the estimated effect sizes for each cohort obtained from the fully specified multivariate model that was constructed using SAS procedure “proc mixed.” The two methods provided impressively similar estimates for effect sizes and the standard errors. Although the difference was not substantial, these two methods did provide slightly different estimates for the between outcomes heterogeneity $\left({\tau }^2\right)$ and there was no observed heterogeneity between outcomes in Seattle and the two Pittsburgh cohorts.

TABLE 3.

Estimated outcome-specific IQ-related effects at age 7 from Stage I estimation

Cohort	Response type	Effect size	SE
Detroit	WISC Verbal IQ	−4.2	3.2
Detroit	WISC Performance IQ	−3.7	3.2
Detroit	WISC Freedom from distractibility	−10.3	3.1
Pittsburgh Cohort 1	Stanford Binet Verbal reasoning	−5.8	3.0
Pittsburgh Cohort 1	Stanford Binet Abstract reasoning	−5.0	3.0
Pittsburgh Cohort 1	Stanford Binet Quantitative reasoning	−1.9	3.0
Pittsburgh Cohort 1	Stanford Binet Short term memory	−5.3	3.0
Pittsburgh Cohort 2	Stanford Binet Verbal reasoning	−0.3	3.1
Pittsburgh Cohort 2	Stanford Binet Abstract reasoning	−1.8	3.0
Pittsburgh Cohort 2	Stanford Binet Quantitative reasoning	−1.1	3.1
Pittsburgh Cohort 2	Stanford Binet Short term memory	−3.5	3.1
Atlanta Cohort 1	Kaufman ABC Simultaneous processing	−6.9	2.9
Atlanta Cohort 1	Kaufman ABC Sequential processing	−1.9	2.9
Atlanta Cohort 2	DAS Verbal standard score	−5.9	3.2
Atlanta Cohort 2	DAS Nonverbal standard score	1.7	3.3
Atlanta Cohort 2	DAS Spatial standard score	−0.9	3.3
Seattle	WISC Verbal IQ	−0.5	2.6
Seattle	WISC Performance IQ	−1.9	2.6

TABLE 4.

Pooled effect size estimates of prenatal alcohol exposure for each cohort at Stage II

	Hierarchical Approach			Multivariate Approach
Cohort	Effect Size	SE	${\widehat{\tau }}^2$	Effect Size	SE	${\widehat{\tau }}^2$
Detroit	−6.1	3.2	7.8	−6.1	3.1	6.2
Pittsburgh Cohort 1	−4.3	2.4	0.0	−4.0	2.6	0.0
Pittsburgh Cohort 2	−1.6	2.5	0.0	−1.6	2.5	0.0
Atlanta Cohort 1	−4.4	3.0	5.1	−4.4	3.4	6.2
Atlanta Cohort 2	−1.9	3.0	7.8	−2.0	3.2	8.2
Seattle	−1.2	2.3	0.0	−1.2	2.3	0.0

To combine the independent effect size estimates across cohorts and obtain a global effect size estimate of PAE on IQ at age 7 years, we used the R package “metafor” to pool the estimates resulted from our hierarchical meta-analysis and from the fully specified multivariate model (Table 5). For the completeness, we also conducted conventional meta-analysis, which ignores the correlation among outcomes and synthesizes information across cohorts in one step. The resulting global effect sizes from our hierarchical meta-analysis and the full multivariate model were almost identical with similar estimates of the heterogeneity parameter ${\tau }^2$. Use of the conventional meta-analytic approach (ignoring the dependence across the outcomes within cohorts) led to a larger effect size estimate and a more conservative standard error. Thus, ignoring the dependence across outcomes alters the final pooled estimate and the associated standard error and provides a very different impression of the extent of the between cohort heterogeneity.

TABLE 5.

Stage III estimated effect sizes of prenatal alcohol exposure on IQ at age 7

Method	Global effect size	SE	${\widehat{\tau }}^2\left(\text{se}\right)$
Hierarchical meta-analytic approach	−3.2	0.8	1.0 (2.3)
One-stage (full multivariate approach)	−3.1	0.8	0.9 (2.3)
Conventional meta-analysis (ignoring the correlation)	−3.5	0.6	4.7 (2.7)

7 |. DISCUSSION

In this paper, we propose the use of a hierarchical meta-analysis to synthesize data on multiple outcomes from multiple studies. The studies we included were conducted independently and used different neuropsychological test batteries to assess IQ and the same domains of cognitive function. The approach was particularly helpful in terms of synthesizing data across diverse outcomes within each cohort. Furthermore, by including multiple correlated responses from each child, the analyses make full use of available data to maximize the efficiency of estimation and enhance power of associated tests for effects. Robust variance estimation ensures valid inferences at each stage of the analysis.

Our hierarchical meta-analysis consists of three stages. In the first stage, we obtain effect size estimates for each outcome separately, while adjusting for control variables via a propensity score approach. In the second stage, we obtain cohort-specific pooled effect size estimates while adjusting for between-outcome correlation and incomplete data. In the last stage, we combine effect sizes across the cohorts employing a random-effects model. Our procedure follows the same steps as conventional methods for two-stage IPD analysis for making inferences about the effect size but extends these analyses by accounting for the correlation between outcomes and by accommodating incomplete data on some outcomes.

Our approach has several advantages over the one-stage IPD meta-analysis. First, it builds upon the two-stage IPD meta-analysis that practitioners are already familiar with. Second, with our approach, one can create forest plots to visualize the estimated effect sizes for each outcome. Third, our approach is less likely to encounter convergence problems compared with the one-stage IPD meta-analysis. Finally, our approach uses the known within study variances, which helps to provide more precise estimates.

We evaluated and compared our approach with a fully specified multivariate analysis. Previous studies evaluated this question in different settings. Olkin and Sampson (1998) showed that in the case of comparing multiple treatments and a control with respect to a continuous outcome, the traditional meta-analysis based on estimated treatment contrasts is equivalent to the least squares regression analysis of individual patient data if there are no study-by-treatment interactions and the error variances are constant across trials. Mathew and Nordstrom (1999) claimed that the equivalence holds even if the error variances are different across trials. Empirically, meta-analysis using original data has been found to be generally similar but not identical to meta-analysis using summary statistics. Whitehead (2002) and Lin and Zeng (2010) showed that for all commonly used parametric and semiparametric models, there is no asymptotic efficiency gain by analysing original data if the parameter of main interest has a common value across studies, the nuisance parameters have distinct values among studies, and the summary statistics are based on maximum likelihood. More recently, Kontopantelis (2018) conducted a comprehensive simulation study to compare one-stage and two-stage IPD analysis and concluded that a fully specified one-stage model is preferable especially when investigating interactions. We extend the results from these existing studies to the setting in which there are correlated outcomes within multiple cohorts. In simulation scenarios considered in this paper, we observed that the proposed approach can successfully reduce bias relative to the fully specified multivariate approach. Our simulation results suggest that, when the number of outcomes is small and the between outcomes variance is large, our proposed approach outperforms the multivariate analysis.

We illustrated our approach using data on childhood IQ from six cohorts. We included 18 outcomes from these cohorts. We showed how to extend two-stage IPD meta-analysis in the presence of correlated effect size estimates and how to address missing data on outcomes by providing an adjustment formula for the pairwise correlation. With this new approach, one can conduct two-stage IPD meta-analysis in the presence of correlated effect size estimates while taking advantage of visualization via forest plots. Our hierarchical meta-analysis consists of three stages. In the first stage, we obtained effect size estimates for each outcome separately while adjusting for control variables via propensity score approach. In the second stage, we employed our proposed approach to obtain cohort-specific pooled effect size estimates while adjusting for between-outcome correlation and incomplete data. In the last stage, we combined effect sizes across the cohorts employing a random-effects model. We compared the results from our approach with the results from the fully specified multivariate approach and the conventional meta-analysis that ignores the fact that the effect sizes are being combined are dependent. In this comparison, we showed that ignoring within-cohort correlation can markedly alter meta-analysis results in important ways. When we compare our method with the full multivariate approach, our method performed well and thus provides a useful innovative tool for performing and interpreting meta-analyses with the correlated effect sizes. While the proposed approach empirically performs very well in the scenarios we considered in the simulation studies, there may be situations that we did not consider where the performance does not share the simplicity and efficiency as was seen here.

APPENDIX A: DETAILED DESCRIPTION OF THE SIX COHORT STUDIES

In this appendix, we provide a detailed description of the six cohort studies that we use data from in our application. We specifically provide information on study design, sampling selection, and sample size for each cohort.

Detroit Cohort birth years: 1986–1989

All women $(N>6400)$ enrolling in the antenatal maternity clinic at a large, inner-city hospital were interviewed regarding their alcohol use at their first prenatal visit ($M=23.4$-week gestation; $SD=7.9$), using a timeline follow-back interview (Jacobson et al., 2002). Moderate and heavy drinking women were overrepresented in the sample by including all women reporting at least 0.5 oz AA at conception and a random sample of approximately 5% of the lower level drinkers and abstainers. The timeline follow-back interview was repeated at each prenatal clinic visit $(M=5.4$ visits). To reduce the risk that alcohol might be confounded with cocaine exposure, 78 heavy cocaine (<2 days/week), light alcohol (<7 drinks/week) users were also included in the final sample, which consisted of 480 pregnant women and their children. Participants were followed up at 6.5, 12, and 13 months and 7, 14, and 19 years.

Pittsburgh Cohort 1 birth years: 1983–1986

Participants were recruited from the prenatal clinic at a maternity hospital if they were English speaking, age 18 or older, and in their fourth or fifth gestational month.

The birth sample consisted of 763 live singleton infants. The alcohol, tobacco, and drug use interview was repeated in the seventh gestational month and at delivery, when second and third trimester substance use information was obtained. The cohort consisted of women who were pre-dominantly low income and of fairly equal numbers of Caucasian and African American women. Participants were followed up at 8 and 18 months, and 3, 6, 10, 14, 16, and 22 years.

Pittsburgh Cohort 2 birth years: 1988–1993

English-speaking women in their fourth or fifth month of pregnancy attending the prenatal clinic at a large inner-city hospital who were 18 years old or older were interviewed regarding their usual, maximum, and minimum consumption of cocaine, alcohol, marijuana, tobacco, and other drugs prior to pregnancy and during the first trimester. Every woman who reported any cocaine/crack use during the first trimester was enrolled in the study cohort, as was the next woman interviewed who reported no cocaine or crack use during both the year prior to pregnancy and the first trimester. Although crack/cocaine use was the criterion for recruitment, a large proportion of these women also drank moderate-to-heavy levels of alcohol. The alcohol and drug use interview was repeated at the end of the second and third trimesters, and offsprings were assessed at 1, 3, 7, 10, 15, and 21 years. The birth cohort consisted of 295 women and infants; the women were predominantly of low socio-economic status and were roughly equally divided by Caucasian and African American race.

Atlanta Cohort 1 birth years: 1980–1986

Five hundred twenty-seven low socioeconomic status (SES), pregnant women were recruited at their first prenatal visit at an urban Atlanta hospital serving a primarily African American, low income population. Women who reported drinking at least 1 oz AA/week during pregnancy were recruited. Nondrinkers, who were similar in demographic background, were recruited at the same time to serve as controls. Women were interviewed at recruitment about their alcohol and drug use; the majority reported drinking on weekends in a “binge” pattern. Infants were evaluated following birth. Subsamples were followed up at 6 and 12 months and 7, 14, and 22 years.

Atlanta Cohort 2 birth years: 1992–1994

Three hundred six mothers and their infants were recruited shortly after delivery at an urban Atlanta hospital; 111 reported having drunk alcohol during pregnancy, 71 of whom also had used cocaine (based on self-report or urine screen); 44 used cocaine but no alcohol; 151 did not drink alcohol or use cocaine. All participants were English speaking, 19 years or older, and had singleton births; most were African American and low SES. The infants were assessed at 2 and 8 years.

Seattle Cohort birth years: 1975–1976

All women who were enrolled in prenatal care by the fifth month of pregnancy at two large Seattle hospitals were eligible to participate. To ascertain PAE, participating mothers $(N=1529)$ were administered a Quantity-Frequency-Variability interview (Cahalan & Cisin, 1968) regarding alcohol, tobacco, and drug use for two time periods: during pregnancy and just prior to pregnancy recognition; 462 newborns were selected based on an algorithm derived from maternal absolute alcohol (AA)/day, alcohol use/occasion, volume variability, and frequency of intoxication constructed to overrepresent infants born to heavier drinkers. Controls included both abstainers and light drinkers. Infants were followed up at 8 and 18 months and 4, 7, 11, 14, 21, 25, and 30 years. Although cohort retention was high (e.g., 82% at 14 years), other children not initially selected whose mothers had been interviewed during pregnancy were added at follow- up assessments to keep the sample size close to 500 at each examination.

APPENDIX B: DERIVATION OF THE COVARIANCE MATRIX FOR $\stackrel{\mathit{ˆ}}{\mathit{B}}$

In this appendix, we provided detailed derivation a robust covariance matrix characterizing the dependence between Stage I estimators (i.e., $\left.{\hat{\theta }}_1,{\hat{\theta }}_2,\dots ,{\hat{\theta }}_K\right)$. This derivation prove the results in Section 3.2 of the main text.

The expression for the covariance between ${\hat{\theta }}_k$ and ${\hat{\theta }}_l$ is obtained based on a general formula for robust variance estimation. If $S_{jk}\left({\theta }_k\right)=$ $X_{jk}\left(Y_{jk}-X_{jk}^{\prime }{\theta }_k\right)$ is the desired contribution from individual $j$ to the score function for ${\theta }_k$ given $B$, the observed data score equation for estimating ${\theta }_k$ at Stage I can be written as

$$S_k\left({\theta }_k\right)=\sum _{j=1}^J{R}_{jk}{S}_{jk}\left({\theta }_k\right)=0,$$ (B1)

the solutions to which are

$${\hat{\theta }}_k=\sum _{j=1}^J{R}_{jk}{\left(X_{jk}{X}_{jk}^{\prime }\right)}^{-1}{X}_{jk}{Y}_{jk},k=1,\dots ,K.$$ (B2)

If we stack the score function in (B1), we obtain $S\left(\theta \right)={\left(S_1^{\prime }\left({\theta }_1\right),\dots ,S_K^{\prime }\left({\theta }_K\right)\right)}^{\prime }$.

Then given $B={\left(B_1,\dots ,B_K\right)}^{\prime }$, we note that

$$\sqrt{J}(\hat{\theta }-\theta )\stackrel{d}{~}\mathrm{M}\mathrm{V}\mathrm{N}\left(0,{𝒜}^{-1}\left(\theta \right)ℬ\left(\theta \right){𝒜}^{-1}\left(\theta \right)\right)$$ (B3)

as $J\to \mathrm{\infty }$, where $𝒜\left(\theta \right)=E\left\{-\partial S\left(\theta \right)/\partial {\theta }^{\prime }\right\}$ is a block diagonal $3K\times 3K$ matrix of the form

$$𝒜\left(\theta \right)=\left[\begin{array}{cccc}{𝒜}_{11}\left({\theta }_1\right)& 0& \dots & 0\\ 0& {𝒜}_{22}\left({\theta }_2\right)& \dots & 0\\ ⋮& & \ddots & ⋮\\ 0& & & {𝒜}_{KK}\left({\theta }_K\right)\end{array}\right],$$

where the $k\mathrm{t}\mathrm{h}$ 3 × 3 diagonal submatrix is given by

$${𝒜}_{kk}\left({\theta }_k\right)=E\left\{-\partial S_k\left({\theta }_k\right)/\partial {\theta }_k^{\prime }\right\}=E\left\{\sum _{j=1}^J{R}_{jk}{X}_{jk}{X}_{jk}^{\prime }\right\}=JE\left\{X_{jk}{X}_{jk}^{\prime }\mid R_{jk}=1\right\}P\left(R_{jk}=1\right),$$

$k=1,\dots ,K$. If we let ${\mathrm{\Omega }}_{kk}=P\left(X_{jk}{X}_{jk}^{\prime }\mid R_{jk}=1\right)$ be a 3 × 3 matrix, we can then write

$${𝒜}_{kk}\left({\theta }_k\right)=J{\mathrm{\Omega }}_{kk}P\left(R_{jk}=1\right).$$ (B4)

Note that $ℬ\left(\theta \right)=E\left\{S\left(\theta \right)S^{\prime }\left(\theta \right)\right\}$ is also a $3K\times 3K$ matrix. Under the assumption that the response data are missing at random (i.e., $R_{jk}\perp Y_{jk}\mid X_{jk}$), the diagonal elements of $ℬ\left(\theta \right)$ are the covariance matrices of the score functions for ${\theta }_k,{ℬ}_{kk}\left(\theta \right)=\mathrm{c}\mathrm{o}\mathrm{v}\left(S_k\left({\theta }_k\right)\mid B\right),k=1,\dots ,K$, where

$${ℬ}_{kk}\left(\theta \right)=E\left\{\sum _{j=1}^J{R}_{jk}{S}_{jk}\left({\theta }_k\right)S_{jk}^{\prime }\left({\theta }_k\right)\right\}=E\left\{\sum _{j=1}^J{R}_{jk}{X}_{jk}{X}_{jk}^{\prime }\mathrm{v}\mathrm{a}\mathrm{r}\left(E_{jk}\right)\right\},$$

because the error terms are assumed independent of the covariates. This can then be written as

$${ℬ}_{kk}\left(\theta \right)=J{\mathrm{\Omega }}_{kk}P\left(R_{jk}=1\right){\sigma }_k^2,k=1,\dots ,K.$$ (B5)

In a similar fashion, we note that

$$\begin{gathered} {ℬ}_{kl}\left(\theta \right)=\mathrm{c}\mathrm{o}\mathrm{v}\left(S_k\left({\theta }_k\right),S_l\left({\theta }_l\right)\mid B\right)=E\left\{\sum _{j=1}^J{R}_{jk}{R}_{jl}E\left\{S_{jk}\left({\theta }_k\right)S_{jl}^{\prime }\left({\theta }_l\right)\mid X_{jk},X_{jl},R_{jk}=R_{jl}=1\right\}\right\} \\ =E\left\{\sum _{j=1}^J{R}_{jk}{R}_{jl}{X}_{jk}{X}_{jl}^{\prime }\mathrm{c}\mathrm{o}\mathrm{v}\left(E_{jk},E_{jl}\right)\right\} \\ =JE\left\{X_{jk}{X}_{jl}^{\prime }\mid R_{jk}=R_{jl}=1\right\}P\left(R_{jk}=R_{jl}=1\right){\sigma }_{kl} \\ =J{\mathrm{\Omega }}_{kl}P\left(R_{jk}=R_{jl}=1\right){\sigma }_{kl}, \end{gathered}$$

where ${\mathrm{\Omega }}_{\mathrm{k}\mathrm{l}}=E\left\{X_{j\mathrm{k}}{X}_{jl}^{\prime }\mid R_{j\mathrm{k}}=R_{\mathrm{j}\mathrm{l}}=1\right\}$ is a 3 × 3 matrix. If $X_{j\mathrm{k}}=X_{jl}$ as in this setting, this becomes

$${ℬ}_{kl}\left(\theta \right)=\mathrm{c}\mathrm{o}\mathrm{v}\left(S_k\left({\theta }_k\right),S_l\left({\theta }_l\right)\mid B\right)=J{\mathrm{\Omega }}_{kk}P\left(R_{jk}=R_{jl}=1\right){\sigma }_{kl}$$ (B6)

because ${\mathrm{\Omega }}_{kk}={\mathrm{\Omega }}_{kl}=\mathrm{\Omega }$ for all $k\ne l$.

If we wish to estimate the covariance of $\sqrt{J}\left({\hat{\theta }}_k-{\theta }_k\right)$ and $\sqrt{J}\left({\hat{\theta }}_l-{\theta }_l\right)$ given $B$, we note that this has the general form

$$\text{cov}\left(\sqrt{J}\left({\hat{\theta }}_k-{\theta }_k\right),\sqrt{J}\left({\hat{\theta }}_l-{\theta }_l\right)\mid B\right)={𝒜}_{kk}^{-1}\left(\theta \right)B_{kl}\left(\theta \right){𝒜}_{ll}^{-1}\left(\theta \right).$$

Inserting the derived expressions gives the $(k,l),$ 3 × 3 submatrix of the full covariance matrix in (B3) as

$$\mathrm{c}\mathrm{o}\mathrm{v}\left(\sqrt{J}\left({\hat{\theta }}_k-{\theta }_k\right),\sqrt{\mathrm{J}}\left({\hat{\theta }}_l-{\theta }_l\right)\mid B\right)=\frac{{\sigma }_{kl}{\mathrm{\Omega }}^{-1}P\left(R_{jk}=R_{jl}=1\right)}{P\left(R_{jk}=1\right)P\left(R_{jl}=1\right)}.$$ (B7)

We estimate (B7) as follows. Because $X_{jk}=X_j$ is available for all individuals, we estimate $\mathrm{\Omega }={\mathrm{\Omega }}_{kk}={\mathrm{\Omega }}_{kl}\mathrm{s}\mathrm{i}\mathrm{m}\mathrm{p}\mathrm{l}\mathrm{y}$ as $\stackrel{\mathrm{ˆ}}{\mathrm{\Omega }}={\sum }_{j=1}^J\left(X_{jk}{X}_{jk}^{\prime }\right)/J$. Moreover, we estimate $P\left(R_{jk}=R_{jl}=1\right)$ empirically as $\hat{P}\left(R_{jk}=R_{jl}=1\right)=n_{kl}/J$ where $n_{kl}={\sum }_{j=1}^J{R}_{jk}{R}_{jl}$, and likewise let $\hat{P}\left(R_{jk}=1\right)=n_k/J$ where $n_k={\sum }_{j=1}^J{R}_{jk},k=1,\dots ,K$. Replacing unknown quantities with their estimates gives

$$\widehat{\mathrm{c}\mathrm{o}\mathrm{v}}\left(\sqrt{J}\left({\hat{\theta }}_k-{\theta }_k\right),\sqrt{J}\left({\hat{\theta }}_l-{\theta }_l\right)\mid B\right)=\frac{{\hat{\sigma }}_{kl}}{J^{-1}\sum _{j=1}^J\left(X_{jk}{X}_{jl}^{\prime }\right)}\frac{J{n}_{kl}}{n_k{n}_l},$$ (B8)

where ${\hat{\sigma }}_{\mathrm{k}\mathrm{l}}$ is given by (13).

Let $\mu \left(\beta \right)=\mathbb{I}\beta$ where $\mathbb{I}$ is a $K\times 1$ vector of ones and $\beta$ is a scalar. We then let $\mathrm{c}\mathrm{o}\mathrm{v}\left\{\sqrt{J}\right(\hat{B}-\mu \left(\beta \right))\mid B\}=\mathrm{\Gamma }$, where $\mathrm{\Gamma }$ is the covariance matrix for $\hat{B}$ obtained by selecting the corresponding elements of (B7) related to the coefficients of the exposure variable in the $K$ marginal least squares estimates. We aim to use $\mathrm{c}\mathrm{o}\mathrm{v}\left(\sqrt{J}\right(\hat{B}-\mu \left(\beta \right))\mid B)$ to combine the estimates across all responses, but we note there is an additional component of variation in the estimators of the exposure effects because the $B_k$ terms are themselves independent and identically distributed (i.e., $B_{ik}~N\left({\beta }_i,{\varphi }_i\right)$).

Thus, $\mathrm{c}\mathrm{o}\mathrm{v}(\hat{B}\mid B)=J^{-1}\mathrm{\Gamma }$, where $\mathrm{\Gamma }$ is a $K\times K$ matrix with diagonal elements ${\mathrm{\Gamma }}_{kk}$ and off-diagonal elements ${\mathrm{\Gamma }}_{kl},I=1,\dots ,K,k=1,\dots ,K$,

$$\text{var}\left({\hat{B}}_k\right)=J^{-1}{\mathrm{\Gamma }}_{kk}+\varphi ,k=1,\dots ,K,$$ (B9)

and because $B_k\perp B_1$,

$$\mathrm{c}\mathrm{o}\mathrm{v}\left({\hat{B}}_k,{\hat{B}}_l\right)=J^{-1}{\mathrm{\Gamma }}_{kl},k\ne I=1,\dots ,K.$$ (B10)

We denote the unconditional covariance matrix for $\hat{B}$ as $\mathrm{c}\mathrm{o}\mathrm{v}\left(\hat{B}\right)=\mathrm{\Psi }\left(\varphi \right)=J^{-1}\mathrm{\Gamma }+\mathrm{\Delta }\varphi$ where ${\mathrm{\Psi }}_{kk}\left(\varphi \right)$ is given by (B9), ${\mathrm{\Psi }}_{kl}\left(\varphi \right)$ is given by (B10), and $\mathrm{\Delta }$ is a $K\times K$ identity matrix. Given an estimate of $\varphi$, we estimate this covariance matrix by

$$\widehat{\mathrm{C}\mathrm{o}\mathrm{v}}\left(\hat{B}\right)=J^{-1}\stackrel{\mathrm{ˆ}}{\mathrm{\Gamma }}+\mathrm{\Delta }\hat{\varphi }=\stackrel{\mathrm{ˆ}}{\mathrm{\Psi }}\left(\hat{\varphi }\right).$$