A Bayesian multivariate factor analysis model for causal inference using time-series observational data on mixed outcomes

Summary

Assessing the impact of an intervention by using time-series observational data on multiple units and outcomes is a frequent problem in many fields of scientific research. Here, we propose a novel Bayesian multivariate factor analysis model for estimating intervention effects in such settings and develop an efficient Markov chain Monte Carlo algorithm to sample from the high-dimensional and nontractable posterior of interest. The proposed method is one of the few that can simultaneously deal with outcomes of mixed type (continuous, binomial, count), increase efficiency in the estimates of the causal effects by jointly modeling multiple outcomes affected by the intervention, and easily provide uncertainty quantification for all causal estimands of interest. Using the proposed approach, we evaluate the impact that Local Tracing Partnerships had on the effectiveness of England’s Test and Trace programme for COVID-19.

1 Introduction

This article considers the problem of estimating from observational data the causal effect of an intervention (or treatment/policy) on an outcome of interest in applications where: (i) there are multiple sample units, some of which receive the intervention; (ii) for each unit, the outcome of interest is measured at multiple time points; and (iii) the units receiving the intervention and the times at which the intervention is implemented are not randomized. This problem arises often in epidemiology (Barnard et al., 2022), public health (Bruhn et al., 2017), economics (Hsiao et al., 2012), marketing (Brodersen et al., 2015), political science (Xu, 2017), and social studies (Ben-Michael et al., 2023). Often the sample units in such applications represent groups of individuals; for example, the units might be hospitals or administrative regions. See Ferman et al. (2020) and Samartsidis et al. (2019) for a list of real-world examples.

Methods for causal evaluation in settings where (i)–(iii) apply can be broadly classified as causal factor analysis (or “matrix completion”) (Kim and Oka, 2014; Gobillon and Magnac, 2016; Xu, 2017; Athey et al., 2021; Nethery et al., 2023; Samartsidis et al., 2020; Pang et al., 2021, among others) or synthetic control (Abadie et al., 2010; Hsiao et al., 2012; Brodersen et al., 2015; Robbins et al., 2017; Ben-Michael et al., 2021, among others) approaches. For an overview of these methods and an explanation of how they account for potential confounding, see Clarke et al. (2023), Xu (2023), and Samartsidis et al. (2019).

Existing approaches have limitations. First, for some of them, it is hard to obtain uncertainty intervals for the causal effects of interest. Second, most of them are designed to deal with continuous outcomes. An exception is Nethery et al. (2023), who develop a factor model for a single count outcome. However, these authors do not consider binomial outcomes, which arise frequently in applications; nor do they account for uncertainty in the number of latent factors. Third, few of these methods allow joint modeling of multiple outcomes affected by the intervention. Such joint modeling may improve statistical efficiency, which is particularly valuable when evaluating policies in contexts where data are sparse, i.e. only available annually or quarterly over a limited number of years. In recent work (Samartsidis et al., 2020), we used the multivariate factor analysis model of De Vito et al. (2021) to jointly model multiple normally distributed outcomes. However, this model cannot be applied to binomial/count data. It also assumes that variability shared across any of the multiple outcomes is shared across all the multiple outcomes.

Here, we develop a general approach to causal inference from time-series observational data that tackles these limitations. This method uses a multivariate factor analysis model that can be applied to multiple outcomes of mixed type (continuous/binomial/count) and allows for sharing of the loadings parameters across these outcomes. The model is Bayesian and thus automatically provides uncertainty quantification for all quantities of interest. Our model accounts for uncertainty in the number of latent factors and allows sharing of loadings across any subsets of the multiple outcomes. The Markov chain Monte Carlo (MCMC) algorithm we propose for fitting the model combines modern data augmentation and sampling techniques to overcome slow mixing issues stemming from the well-known identifiability problems of factor analysis models. The potential benefits of joint outcome modeling are demonstrated via a simulation study. We apply the proposed method to real data on a new policy for COVID-19 contact tracing in UK, providing valuable insights on the effectiveness of this policy. Finally, although our model is developed for intervention evaluation problems, it can be of use in other fields where multivariate factor analysis models are of interest, such as cancer biology (Avalos-Pacheco et al., 2022).

The remainder is structured as follows. In Section 2, we introduce the motivating COVID-19 contact tracing policy evaluation problem. Section 3 presents our Bayesian model. In Section 4, we discuss model fitting. Section 5 presents a simulation study that demonstrates the benefit of jointly modeling multiple outcomes. Section 6 contains results from our motivating data set. Finally, Section 7 summarizes the article’s findings and lists some directions for future research.

2 COVID-19 contact tracing policy evaluation problem

The NHS Test & Trace (TT) programme was launched in May 2020 as part of the containment strategy for the COVID-19 pandemic in UK. One of its main functions was to ensure that individuals who tested positive for COVID-19 (the cases) were aware¹ of their infection, and to inform them of their legal duty to self-isolate to prevent onward transmission. When TT communicated with a case (henceforth this communication will be referred to as tracing), the case was asked to list the individuals (contacts) with whom he/she had been in close physical proximity within 48 h of his/her symptom onset (for asymptomatic cases, date of positive test is used). TT would then attempt to reach these contacts to provide advice and tell them to self-isolate. Multiple studies have provided evidence in favor of TT (Fetzer and Graeber, 2021).

Starting in July 2020, local authorities gradually introduced local tracing partnerships (LTPs). These LTPs involved the local authority working with national TT to improve tracing of cases and contacts. An LTP team was composed of staff based locally, whose job was to trace cases resident in that local authority who had not been able to be traced by TT. This is done mainly by telephone, often using the team’s local knowledge and, in several authorities, by visiting a case’s home if telephone communication had not been successful. The question we address here is whether LTPs (the intervention) have had an impact on the effectiveness of TT. Which local authorities introduced LTPs, and when, depended mainly on the interest expressed by local authorities, rather than being randomized, which complicates estimation of the causal effects of LTPs.

We judge the effectiveness of TT in terms of four outcomes measured daily: (i) the proportion of cases on that day that were ultimately completed, where a case is considered complete if TT manage to contact him/her; (ii) the proportion of completed cases whose completion was timely, i.e., occurs within 48 h of the case being registered with TT; (iii) the total number of contacts elicited from cases completed on that day; and (iv) the proportion of these elicited contacts who were subsequently completed. We shall refer to these as case completion, timely case completion, number of contacts and contact completion, respectively. We have daily measurements of these four outcomes on 181 spatial units during the study period July 01, 2020 to November 15, 2020 (i.e., 138 days). Each of these units is either an upper-tier local authority (UTLA) or part of a UTLA.

Figure 1 provides some graphical summaries of the data. The roll-out of LTPs is summarized in Fig. 1(a). The first LTP was formed on July 15, 2020 and by November 15, 2020, 118 (66%) of the units were covered by an LTP. Figure 1(b) shows the number of cases for each day and unit; during July–September the number of cases is low for most of the units, but it increases thereafter. The data on the four outcomes (Fig. D.1 of the supplementary material available at Biostatistics online) show considerable variability across units for the same day and across time within the same unit. The existing methods for policy evaluation listed in Section 1 cannot be used to investigate whether LTPs improved some of these outcomes because they are not applicable to binomial data.

Fig. 1.

Graphical summaries of the data. (a) Proportion of units (out of 181) that had already formed an LTP by each day. (b) Total number of new cases for each day and unit, where units have been ordered by average number of new cases.

3 Methods for estimation of causal effects

3.1 Latent factor model and causal effects

Let $i=1,\dots ,N$ index the sample unit and $t=1,\dots T$ denote time (e.g., in days) relative to a fixed calendar time (e.g., July 1st 2020 in our application). Let N₁ and $N_2=N-N_1$ be the numbers of control units and treated units, respectively (i.e., units that do not and units that do experience the intervention). Without loss of generality, assume units are ordered so that $1,\dots ,N_1$ are control units. Let T_i denote the last time at which unit i has not experienced the intervention (for control units, T_i = T). Let $T_{\text{min}}=\min \{T_1,\dots T_N\}$ and ${\tilde{N}}_t=\sum _{i=N_1+1}^N\mathbb{I}(T_i<t)$.

Suppose there are D₁ continuous, D₂ binomial and D₃ count outcomes. Let y_itd be the observed value of the dth continuous outcome ($d=1,\dots ,D_1$). For the dth binomial outcome ($d=1,\dots ,D_2$), let k_itd be the observed number of successes in a known number, n_itd, of independent Bernoulli trials. Let z_itd be the observed value of the dth count outcome ($d=1,\dots ,D_3$).

We shall define the causal effect of the intervention on any one of the $D=D_1+D_2+D_3$ outcomes in unit i at time t in terms of the difference between its observed value and the outcome that unit i would have if it were not treated at time t. The latter value is called a potential untreated outcome (Holland, 1986). We denote the potential untreated continuous, binomial, and count outcomes as $y_{\mathit{itd}}^{(0)},\hspace{0.17em}{k}_{\mathit{itd}}^{(0)}$, and $z_{\mathit{itd}}^{(0)}$, respectively. We also define the potential treated outcomes $y_{\mathit{itd}}^{(s)},\hspace{0.17em}{k}_{\mathit{itd}}^{(s)}$, and $z_{\mathit{itd}}^{(s)}$ ($s=1,\dots T_i$) as the outcomes that unit i would have at time t if time s had been the last time at which unit i had not experienced the intervention. We make the stable unit treatment value assumption (SUTVA), which implies that the potential outcomes of one unit do not depend on whether, or at what time, the intervention is applied to other units. We further make the no-treatment-anticipation assumption (Athey and Imbens, 2021) which means that $y_{\mathit{itd}}^{(s)}=y_{\mathit{itd}}^{(0)},\hspace{0.17em}{k}_{\mathit{itd}}^{(s)}=k_{\mathit{itd}}^{(0)}$, and $z_{\mathit{itd}}^{(s)}=z_{\mathit{itd}}^{(0)}$ for all $t\le s$. With these assumptions, the observed outcomes are related to the potential outcomes by $y_{\mathit{itd}}=y_{\mathit{itd}}^{(0)},\hspace{0.17em}{k}_{\mathit{itd}}=k_{\mathit{itd}}^{(0)}$, and $z_{\mathit{itd}}=z_{\mathit{itd}}^{(0)}$ if $t\le T_i$; and $y_{\mathit{itd}}=y_{\mathit{itd}}^{(T_i)},\hspace{0.17em}{k}_{\mathit{itd}}=k_{\mathit{itd}}^{(T_i)}$, and $z_{\mathit{itd}}=z_{\mathit{itd}}^{(T_i)}$ if $t>T_i$.

There is a growing interest in using factor analysis (FA) to model the potential untreated outcomes (see Section 1 for references), or as the data generating mechanism to study the properties of synthetic control-type approaches (Abadie et al., 2010; Hsiao et al., 2012; Ben-Michael et al., 2021; Ferman, 2021). This is due to the FA model’s ability to adjust for observed covariates and to allow for the presence of unmeasured confounding with a particular structure. Here, we propose a multivariate, mixed-outcome FA model for the potential untreated outcomes. Specifically, for each $i=1,\dots ,N$ and $t=1,\dots ,T$, we assume

$$\begin{array}{c}{y}_{\mathit{itd}}^{(0)}\sim \mathrm{N}\left({\mu }_{\mathit{itd}},{\sigma }_{id}^2\right),{\mu }_{\mathit{itd}}={\mathit{\lambda }}_i^{\top }{\mathit{f}}_{td}+{\mathit{\eta }}_{1,d}^{\top }{\mathit{x}}_{it},(d=1,\dots ,D_1)\\ k_{\mathit{itd}}^{(0)}\sim \text{Bin}\left(n_{\mathit{itd}},p_{\mathit{itd}}\right),\text{logit}(p_{\mathit{itd}})={\mathit{\lambda }}_i^{\top }{\mathit{g}}_{td}+{\mathit{\eta }}_{2,d}^{\top }{\mathit{x}}_{it},(d=1,\dots ,D_2)\\ z_{\mathit{itd}}^{(0)}\sim \text{NegBin}\left(w_{\mathit{itd}}{q}_{\mathit{itd}}{\xi }_d^{-1},{(1+{\xi }_d)}^{-1}\right),\hspace{0.17em}\text{log}(q_{\mathit{itd}})={\mathit{\lambda }}_i^{\top }{\mathit{h}}_{td}+{\mathit{\eta }}_{3,d}^{\top }{\mathit{x}}_{it}.(d=1,\dots ,D_3).\end{array}$$ (3.1)

Here, ${\mathit{f}}_{td},{\mathit{g}}_{td},{\mathit{h}}_{td}$ are J-vectors of unobserved factors at time t; ${\mathit{\lambda }}_i$ is an unobserved J-vector of loadings for unit i; ${\mathit{x}}_{it}$ is a P-vector of (possibly time-dependent) exogenous (not affected by the intervention) covariates; and ${\mathit{\eta }}_{\ell ,d}$ are its regression coefficients ($\ell =1,2,3$; $d=1,\dots ,D_{\ell }$). $\text{NegBin}(a,b)$ denotes the negative binomial distribution with mean $a(1-b)/b$ and variance $a(1-b)/b^2$. It follows from (3.1) that $\mathbb{E}(z_{\mathit{itd}})=w_{\mathit{itd}}{q}_{\mathit{itd}}$ and $\mathbb{V}\text{ar}(z_{\mathit{itd}})=w_{\mathit{itd}}{q}_{\mathit{itd}}(1+{\xi }_d)$. So, ${\xi }_d>0$ is the degree of overdispersion relative to the Poisson distribution. Here, w_itd is a known offset; if there is no offset, w_itd = 1. In some applications, there might be enough information to estimate a separate dispersion for each unit; in such cases, we can replace ξ_d by ξ_id.

We assume that. for all $i=1,\dots ,N$,

$$T_i\perp \perp \{(y_{it1}^{(0)},\dots ,y_{it{D}_1}^{(0)},k_{it1}^{(0)},\dots ,k_{it{D}_2}^{(0)},z_{it1}^{(0)},\dots ,z_{it{D}_3}^{(0)}):\hspace{0.17em}t=1,\dots ,T\}|{\mathit{x}}_{i1},\dots ,{\mathit{x}}_{iT},{\mathit{\lambda }}_i.$$

This implies that after controlling for observed potential confounders ${\mathit{x}}_{i1},\dots ,{\mathit{x}}_{iT}$ and unobserved potential confounders ${\mathit{\lambda }}_i$, there is no confounding of the causal effects of the intervention on the dth continuous, dth binary and dth count outcome of unit $i>N_1$ at time $t>T_i$, defined as

$$\begin{array}{c}{\alpha }_{\mathit{itd}}=y_{\mathit{itd}}^{(T_i)}-y_{\mathit{itd}}^{(0)}=y_{\mathit{itd}}-y_{\mathit{itd}}^{(0)},\hspace{1em}{\gamma }_{\mathit{itd}}=k_{\mathit{itd}}^{(T_i)}-k_{\mathit{itd}}^{(0)}=k_{\mathit{itd}}-k_{\mathit{itd}}^{(0)},\hspace{1em}{\delta }_{\mathit{itd}}=z_{\mathit{itd}}^{(T_i)}\\ \hspace{1em}-z_{\mathit{itd}}^{(0)}=z_{\mathit{itd}}-z_{\mathit{itd}}^{(0)},\end{array}$$ (3.2)

respectively. An alternative to γ_itd would be to define the causal effect on the dth binomial outcome of unit i at time t in terms of the success probability of the Bernoulli trials. To do this, we would assume $k_{\mathit{itd}}^{(T_i)}=k_{\mathit{itd}}\sim \text{Bin}\left(n_{\mathit{itd}},\hspace{0.17em}{p}_{\mathit{itd}}^{(T_i)}\right)$ (for $i>N_1,\hspace{0.17em}t>T_i$) and define the causal effect as

$${\beta }_{\mathit{itd}}=p_{\mathit{itd}}^{(T_i)}-p_{\mathit{itd}}.$$ (3.3)

For the dth continuous outcome, we define (with slight abuse of notation) the average (over time) causal effect in unit i as ${\alpha }_{id}=\sum _{t=T_i+1}^T{\alpha }_{\mathit{itd}}/(T-T_i)$ and the overall (over both time and units) causal effect as ${\alpha }_d=(\sum _{i=N_1+1}^N\sum _{t=T_i+1}^T{\alpha }_{\mathit{itd}})/(\sum _{t=1}^T{\tilde{N}}_t)$. Corresponding average causal effects, β_id, β_d, γ_id, γ_d, δ_id, and δ_d, for binary and count outcomes are defined analogously.

Policy makers are also interested in identifying treated units whose response to the intervention (i.e., causal effect) is extreme compared to other treated units. This could be useful, for example, to identify units to which alternative interventions should be applied. For the dth continuous outcome, unit i and time $t>T_i$, define $r_{\mathit{itd}}^{(\alpha )}$ as the (scaled) rank of the α_itd values of the ${\tilde{N}}_t$ units treated by time t, i.e.,

$$r_{\mathit{itd}}^{(\alpha )}=\frac{\sum _{j:T_j<t}\mathbb{I}({\alpha }_{\mathit{jtd}}\le {\alpha }_{\mathit{itd}})}{{\tilde{N}}_t+1}.$$ (3.4)

In (3.4), we have scaled by ${\left({\tilde{N}}_t+1\right)}^{-1}$ to ensure that ranks are comparable between any times t and s (both $>T_{\text{min}}$) for which ${\tilde{N}}_t\ne {\tilde{N}}_s$. Further, let $r_{id}^{(\alpha )}=\sum _{t=T_i+1}^T{r}_{\mathit{itd}}^{(\alpha )}/(T-T_i)$ be the average (over time) rank of unit i. Ranks for β, γ, and δ are defined analogously.

3.2 LTP application

In our motivating problem, where intervention is the introduction of an LTP, N = 181, T = 138, $D_1=0$ (no continuous outcomes), $D_2=3$ (case completion, timely case completion and contact completion), $D_3=1$ (number of contacts), and $N_1=63$ units have no LTP during study period.

For the first binary outcome (case completion), $n_{it1}$ is the number of cases registered on day t for unit i, and $k_{it1}$ is the number of these cases that were ultimately completed. For the second binary outcome (timely case completion), $n_{it2}$ is the number of cases completed on day t for unit i, and $k_{it2}$ is the number of these cases that were completed within 48 h. For the count outcome (number of contacts), $w_{it1}$ is the number of cases that were completed on day t for unit i, and $z_{it1}$ is the number of contacts elicited from these $w_{it1}$ cases. For the third binary outcome (contact completion), $k_{it3}$ is the number of contacts that were ultimately completed from the $n_{it3}=z_{it1}$ contacts. No covariates ${\mathit{x}}_{it}$ are considered.

In Section 3.1, we assumed n_itd and w_itd are not affected by the intervention. In the LTP application, however, $n_{it2}=k_{it1}$ and $n_{it3}=z_{it1}$, which may both be affected. Also, $w_{it1}$ is related to $k_{ij1}$ and $k_{ij2}$ for $j\le t$, and so may be affected. It is also possible that $n_{it1}$ is affected (successful tracing can prevent new cases). This changes the interpretation of the causal effects ${\gamma }_{it1},\hspace{0.17em}{\gamma }_{it2},\hspace{0.17em}{\gamma }_{it3}$, and ${\delta }_{it1}$. As shown in Section A of the supplementary material available at Biostatistics online, they can be interpreted (under some additional assumptions) as separable direct effects on $k_{it1},\hspace{0.17em}{k}_{it2},\hspace{0.17em}{k}_{it3}$, and $z_{it1}$, respectively. These are different from the corresponding total effects, defined as ${\tilde{\gamma }}_{\mathit{itd}}=k_{\mathit{itd}}-{\tilde{k}}_{\mathit{itd}}^{(0)}$ and ${\tilde{\delta }}_{\mathit{itd}}=z_{\mathit{itd}}-{\tilde{z}}_{\mathit{itd}}^{(0)}$, respectively, where ${\tilde{k}}_{\mathit{itd}}^{(0)}$ and ${\tilde{z}}_{\mathit{itd}}^{(0)}$ are obtained by replacing n_itd and w_itd with their potential untreated outcomes in (3.1). We do not consider the total effects (even though they can be estimated), because we believe they can be misleading. For example, suppose that for some $i>N_1$ and $t>T_i,\hspace{0.17em}{p}_{it3}^{(T_i)}=p_{it3}$ and $n_{it3}>{\tilde{n}}_{it3}$, that is, LTP has no impact on contact completion probability but increases the number of contacts elicited. Then $k_{it3}>{\tilde{k}}_{it3}^{(0)}$ (because larger number of trials), and hence ${\tilde{\gamma }}_{it3}>0$, suggesting that the LTP increased the number of contacts completed. In contrast, ${\gamma }_{it3}$, interpreted as the total number of additional contacts completed thanks to the LTP, would be zero.

3.3 Outline of inference

In order to estimate the causal effects defined in Section 3.1, we need to estimate the potential untreated outcomes of the treated units $i>N_1$ in their postintervention periods $t>T_i$. We do this under the Bayesian paradigm as follows.

First, we set prior distributions for the parameters of the multivariate FA model (3.1). These are detailed in Section 3.4. We then fit, using the MCMC algorithm in Section 4, the FA model to pre-intervention data only (i.e., for each i, we only use data up to time T_i). The posterior samples that we obtain allow us to draw samples from the posterior predictive distribution of the potential untreated outcomes conditional on ${\mathit{x}}_{it}$, n_itd and w_itd. We transform them using (3.2), to obtain samples from the posterior distribution of causal effects α_itd, γ_itd, and δ_itd. For β_itd, we further need to account for uncertainty in $p_{\mathit{itd}}^{(T_i)}$. For each $i>N_1,\hspace{0.17em}t>T_i$, and $d=1,\dots ,D_2$, we a priori assume that $p_{\mathit{itd}}^{(T_i)}\sim \text{Beta}(a_p,b_p)$. Thus, conditional on postintervention data, $p_{\mathit{itd}}^{(T_i)}\sim \text{Beta}(a_p+k_{\mathit{itd}},b_p+n_{\mathit{itd}}-k_{\mathit{itd}})$. We obtain samples from the posterior of β_itd using (3.3), i.e., by subtracting the draws from the posterior of p_itd from the samples obtained from the beta posterior of $p_{\mathit{itd}}^{(T_i)}$.

Let ${\alpha }_{\mathit{itd}}^{(\ell )}$ denote the $\ell$th sample ($\ell =1,\dots ,L$) from the posterior distribution of α_itd. We estimate α_itd as $\sum _{\ell =1}^L{\alpha }_{\mathit{itd}}^{(\ell )}/L$ (the posterior mean) and calculate 95% posterior credible intervals (CIs) using the 2.5% and 97.5% percentiles of the ${\alpha }_{\mathit{itd}}^{(\ell )}$. Samples from the posterior (and hence point estimates and 95% CIs) of α_id, α_td, α_d, and $r_{\mathit{itd}}^{(\alpha )}$ can be obtained from the ${\alpha }_{\mathit{itd}}^{(\ell )}$ using the expressions provided in Section 3.1. For binomial and count outcomes, point estimates and 95% CIs are obtained analogously.

3.4 Prior distributions

One of the main challenges in fitting the FA model (3.1) is that the number of latent loadings J is not known. To account for uncertainty in J, we start by setting it to $J^{\ast }$, an upper limit for the number of factors we expect. Following Gao et al. (2016), we assign a three-level three-parameter beta prior to the loadings. In particular, for $j=1,\dots ,J^{\ast }$ we let

$$\begin{array}{c}\hspace{1em}\hspace{1em}{\lambda }_{ij}\sim \mathrm{N}(0,\frac{1}{{\varphi }_{ij}}-1),\hspace{1em}{\varphi }_{ij}\sim \text{TPB}(a_{\lambda },b_{\lambda },\frac{1}{{\zeta }_j}-1),\hfill \\ {\zeta }_j\sim \text{TPB}(c_{\lambda },d_{\lambda },\frac{1}{\rho }-1),\hspace{1em}\rho \sim \text{TPB}(e_{\lambda },f_{\lambda },\nu ),\hfill \end{array}$$ (3.5)

where $x\sim \text{TPB}(a,b,c)$ means that $x\in (0,1)$ has density $\pi (x|a,b,c)=\frac{\Gamma (a+b)}{\Gamma (a)\Gamma (b)}{c}^b{x}^{b-1}{(1-x)}^{a-1}{(1+(c-1)x)}^{-a-b}$. Following Zhao et al. (2016), we set $a_{\lambda },\dots ,f_{\lambda }=0.5$ and $\nu =0.1$.

Let $\mathbf{\Lambda }$ be the $N\times J^{\ast }$ matrix with rows ${\mathit{\lambda }}_i$. The prior (3.5) induces three levels of regularization in $\mathbf{\Lambda }$ (Zhao et al., 2016). The overall matrix shrinkage is controlled by ρ. The ζ_j induce loading-specific shrinkage. When ζ_s is estimated as $\approx 1$ for some $1\le s\le J^{\ast }$, we will also have that ${\lambda }_{is}\approx 0$ for all i, thus effectively removing the sth column of $\mathbf{\Lambda }$. This allows us to recover the effective number of factors supported by the data as the number of nonzero² columns in $\mathbf{\Lambda }$. Finally, the ${\varphi }_{ij}$ allow the scale of the jth loading to differ across the N units. This is useful because given the interpretation of λ_ij as potential unobserved confounders, one may expect them not to be similar across units, especially between control and treated units.

It is possible that each element λ_ij of ${\mathit{\lambda }}_i$ affects only some of the outcomes. To address this issue, one potential solution would be to assume a different loadings vector for each outcome. However, this is inefficient, especially when T_i’s are small (see simulation study of Section 5). Another solution would be to assume that there are loadings that are specific to each outcome and loadings that are shared by all outcomes (De Vito et al., 2021). However, such an approach does not allow for loadings that are shared by subsets of the outcomes. Grabski et al. (2023) propose a model for normal outcomes that allows for shared loadings among a subset of outcomes. However, $v_{j\ell }$ are effectively either zero or one, thus not allowing loadings to affect the outcomes to different extents (which is the case when $0<v_{j\ell }<1$). Here, we allow for this via the prior on the factor parameters.

Let f_tdj, g_tdj, and h_tdj denote the elements of ${\mathit{f}}_{td},\hspace{0.17em}{\mathit{g}}_{td}$, and ${\mathit{h}}_{td}$, respectively ($j=1,\dots ,J^{\ast }$). We assume that $f_{\mathit{tdj}}\sim \mathrm{N}(0,s_{1,dj}),\hspace{0.17em}{g}_{\mathit{tdj}}\sim \mathrm{N}(0,s_{2,dj})$ and $h_{\mathit{tdj}}\sim \mathrm{N}(0,s_{3,dj})$. For each j, let ${\mathit{v}}_j$ be a vector of size D with elements $v_{j\ell }$ defined as

$$v_{j\ell }=\{\begin{array}{cc}{s}_{1,\ell ,j},\hfill & \ell \le D_1\hfill \\ s_{2,\ell -D_1,j},\hfill & D_1<\ell \le D_1+D_2\hfill \\ s_{3,\ell -D_1-D_2,j},\hfill & \ell >D_1+D_2\hfill \end{array}.$$

We introduce variables $M_j\in \left\{1,\dots ,D\right\}$ which, for each j, indicate the outcome “most affected” by loading j. Conditional on M_j = l, we set v_jl = 1 and assume $v_{j\ell }\sim \text{Uni}[0,1]$ for each $\ell \ne l$. When $v_{j\ell }$ is close to zero for some $\ell$, the values of the corresponding factor will be close to zero, and hence the effect of loading j on outcome $\ell$ will be small. An alternative (which we test on our real data) is to let $v_{j\ell }\sim \text{Uni}[0,1]$ for all j and $\ell$. This introduces more free parameters than needed but has worked well in previous studies (Roy et al., 2021).

For fixed d and j, we have chosen not to impose any temporal structure on ${\left\{f_{\mathit{tdj}}\right\}}_{t=1}^T$ (or ${\left\{g_{\mathit{tdj}}\right\}}_{t=1}^T,\hspace{0.17em}{\left\{h_{\mathit{tdj}}\right\}}_{t=1}^T$) to simplify posterior computations. We expect the loss in efficiency to be small. The reason is that in policy evaluation problems, the number of control units at each time point is typically sufficiently high to allow accurate estimation of factor parameters without the need to impose further structure, see e.g., simulation study in Samartsidis et al. (2020).

For the remaining model parameters, we specify weakly informative prior distributions. For all i and $d=1,\dots ,D_1$, we assume ${\sigma }_{id}^2\sim \text{Uni}(0,{10}^2)$. We prefer this prior to the conjugate $\text{IG}\left(ϵ,ϵ\right)$ with small ϵ, because the latter encourages values of ${\sigma }_{id}^2$ near zero. This can lead to over-fitting as non-systematic fluctuations are encouraged to be absorbed by the loadings/factors term. For all $\ell =1,2,3$ and d, we assume that regression coefficients ${\mathit{\eta }}_{\ell ,d}\sim \mathrm{N}(0,{10}^2\mathit{I})$. For $d=1,\dots ,D_3$, we let ${\xi }_d\sim \text{Uni}(0,20),$ i.e., we assume that the variance of the counts is at most 20 times higher compared to a Poisson model. For each loading j, we set $P(M_j=\ell )=1/D$ for $\ell =1,\dots ,D$.

Finally, we set $a_p=b_p=1$ i.e., let $p_{\mathit{itd}}^{(T_i)}\sim \text{Uni}(0,1)$. The choice of prior on $p_{\mathit{itd}}^{(T_i)}$ is not crucial for the LTP data as n_itd ($t>T_i$) are large, allowing us to estimate these parameters accurately. However, when n_itd are low, the posterior uncertainty in $p_{\mathit{itd}}^{(T_i)}$ under independent uniform priors will be high, leading to high uncertainty in β_itd (see (3.3)). In such cases, efficiency can be gained by imposing structure on $p_{\mathit{itd}}^{(T_i)}$. For example, one can assume that for fixed i and d, $\text{logit}(p_{\mathit{itd}}^{(T_i)})$ arise from an AR(1) process.

4 Posterior computations

We are interested in drawing samples from the high-dimensional posterior with density

$$\pi \left({\left\{{\mathit{\lambda }}_i,{\left\{{\xi }_{id}\right\}}_{d=1}^{D_3}\right\}}_{i=1}^N,{\left\{{\left\{{\mathit{g}}_{td}\right\}}_{t=1}^T,{\mathit{\eta }}_{2,d}\right\}}_{d=1}^{D_2},{\left\{{\left\{{\mathit{h}}_{td}\right\}}_{t=1}^T,{\mathit{\eta }}_{3,d}\right\}}_{d=1}^{D_3},\mathit{\theta }|\text{data}\right),$$ (4.6)

where

$$\mathit{\theta }=\left\{{\left\{{\left\{{\mathit{f}}_{td}\right\}}_{t=1}^T,{\left\{{\sigma }_{id}^2\right\}}_{i=1}^N\right\}}_{d=1}^{D_1},{\left\{{\xi }_d\right\}}_{d=1}^{D_3},{\left\{{\left\{{\varphi }_{ij}\right\}}_{i=1}^N,{\zeta }_j,{\left\{v_{j,l}\right\}}_{\ell =1}^D,M_j\right\}}_{j=1}^{J^{\ast }},\rho \right\}$$

and

$$\text{data}={\left\{{\left\{{\left\{y_{\mathit{itd}}\right\}}_{d=1}^{D_1},{\left\{k_{\mathit{itd}},n_{\mathit{itd}}\right\}}_{d=1}^{D_2},{\left\{z_{\mathit{itd}},w_{\mathit{itd}}\right\}}_{d=1}^{D_3},{\mathit{x}}_{it}\right\}}_{t=1}^{T_i}\right\}}_{i=1}^N.$$

The reason for grouping some parameters in $\mathit{\theta }$ is that they are easy to sample. Posterior (4.6) is analytically intractable; we thus use MCMC to sample from it. We propose a Metropolis-within-Gibbs sampler in which subsets of parameters are drawn (updated) in blocks from their full conditionals. Here, we provide an outline of the algorithm; for further details see Section B of the supplementary material available at Biostatistics online.

The parameters in $\mathit{\theta }$ are straightforward to update. We update each ${\sigma }_{id}^2$ ($i=1,\dots ,N$ and $d=1,\dots ,D_1$) using a Metropolis–Hastings step. The ${\mathit{f}}_{td}$ ($t=1,\dots ,T$ and $d=1,\dots ,D_1$) and ${\mathit{\eta }}_{1,d}$ ($d=1,\dots ,D_1$) are drawn from their multivariate normal full conditional distributions. For the loadings shrinkage parameters, we follow Gao et al. (2016). They show that by using an equivalent representation of the prior (3.5), one can update all ${\varphi }_{ij}$ ($i=1,\dots ,N$ and $j=1,\dots ,J^{\ast }$), ζ_j ($j=1,\dots ,J^{\ast }$) and ρ using Gibbs steps. We draw each M_j ($j=1,\dots ,J^{\ast }$) from its full conditional (these parameters can only take D values), after integrating out v_dj ($d=1,\dots ,D$). Conditional on M_j, the v_dj ($d=1,\dots ,D$ and $j=1,\dots ,J^{\ast }$) are either set to one or drawn from their truncated inverse-gamma full conditionals (see Section B of the supplementary material available at Biostatistics online).

The most challenging step in the proposed MCMC algorithm is to update ${\mathit{\lambda }}_i,\hspace{0.17em}{\mathit{g}}_{td}$, and ${\mathit{h}}_{td}$. Their full conditionals are not available in closed form and hence Gibbs updates are not possible. Moreover, standard algorithms which employ a constant variance–covariance matrix in their proposal (e.g., MALA, preconditioned MALA, HMC) can be inefficient in FA models. This due to the label-switching, rotation, and scale ambiguity problems (Zhao et al., 2016, among others). Consider, for example, the choice of a proposal variance for an element h_tdj of ${\mathit{h}}_{td}$. Tuning this value is nontrivial as h_tdj may represent a different factor (due to label-switching) or change its scale (due to scale ambiguity) at each iteration of MCMC. To overcome these problems, we make use of data augmentation techniques. In particular, we employ methods developed in the case of univariate models (Polson et al., 2013; Zhou and Carin, 2015). As explained below, introducing auxiliary variables enables efficient sampling of ${\mathit{\lambda }}_i,\hspace{0.17em}{\mathit{g}}_{td}$, and ${\mathit{h}}_{td}$. Finally, we note that under the proposed data augmentation scheme, drawing the remaining parameters ${\mathit{\eta }}_{2,d},\hspace{0.17em}{\mathit{\eta }}_{3,d}$, and ξ_d is also conducted efficiently, see the remaining of this section.

Let $\text{PG}(a,b)$ denote the Pólya–Gamma distribution with parameters a and b. Following Polson et al. (2013), for each $i=1,\dots ,N,\hspace{0.17em}t\le T_i$ and $d=1,\dots ,D_2$, we introduce latent variables ω_itd which are a priori $\text{PG}(n_{\mathit{itd}},0)$ distributed. Using results from Polson et al. (2013), we can show that the likelihood contribution of each binomial data point conditional on ω_itd is

$$\pi (k_{\mathit{itd}}|n_{\mathit{itd}},{\mathit{\lambda }}_i,{\mathit{g}}_{td},{\mathit{\eta }}_{2,d},{\mathit{x}}_{it},{\omega }_{\mathit{itd}})\propto \hspace{0.17em}\text{exp}\hspace{0.17em}\left\{-\frac{{\omega }_{\mathit{itd}}}{2}{\left(\frac{{\kappa }_{\mathit{itd}}}{{\omega }_{\mathit{itd}}}-{\mathit{\lambda }}_i^{\top }{\mathit{g}}_{td}-{\mathit{\eta }}_{2,d}^{\top }{\mathit{x}}_{it}\right)}^2\right\},$$ (4.7)

where ${\kappa }_{\mathit{itd}}=k_{\mathit{itd}}-n_{\mathit{itd}}/2$. The likelihood (4.7) is a quadratic form of both ${\mathit{g}}_{td}$ and ${\mathit{\eta }}_{2,d}$, and these parameters have multivariate normal priors. Therefore, introducing ω_itd allows us to draw each ${\mathit{g}}_{td}$ ($d=1,\dots D_2$ and $t=1,\dots ,T$) and ${\mathit{\eta }}_{2,d}$ ($d=1,\dots ,D_2$) from their multivariate normal full conditionals (see SM Section B). Further, the full conditional of ω_itd is a $\text{PG}(n_{\mathit{itd}},{\mathit{\lambda }}_i^{\top }{\mathit{g}}_{td}+{\mathit{\eta }}_{2,d}{\mathit{x}}_{it})$ (Polson et al., 2013), a result which allows us to update these parameters easily.

Let $\text{CRT}(a,b)$ denote the Chinese restaurant table distribution with parameters a and b. Following Zhou and Carin (2015), for each $i=1,\dots ,N,\hspace{0.17em}t\le T_i$ and $d=1,\dots ,D_3$, we introduce latent variables $L_{\mathit{itd}}\sim \text{CRT}(z_{\mathit{itd}},w_{\mathit{itd}}{q}_{\mathit{itd}}/{\xi }_d)$. These can be drawn as $L_{\mathit{itd}}=\sum _{l=1}^{z_{\mathit{itd}}}{b}_l$, where $b_l\sim \text{Bernoulli}\left(\frac{w_{\mathit{itd}}{q}_{\mathit{itd}}/{\xi }_d}{w_{\mathit{itd}}{q}_{\mathit{itd}}/{\xi }_d+l-1}\right)$. Recall that $q_{\mathit{itd}}=\text{exp}\hspace{0.17em}\left({\mathit{\lambda }}_i^{\top }{\mathit{h}}_{td}+{\mathit{\eta }}_{3,d}^{\top }{\mathit{x}}_{it}\right)$. From Section 3.1 of Zhou and Carin (2015), we have that

$$\begin{array}{c}\pi \left(z_{\mathit{itd}},L_{\mathit{itd}}|w_{\mathit{itd}},q_{\mathit{itd}},{\mathit{x}}_{it},{\xi }_d\right)=\text{CRT}\left(L_{\mathit{itd}};z_{\mathit{itd}},\frac{w_{\mathit{itd}}{q}_{\mathit{itd}}}{{\xi }_d}\right)\text{NegBin}\left(z_{\mathit{itd}};\frac{w_{\mathit{itd}}{q}_{\mathit{itd}}}{{\xi }_d},\frac{1}{1+{\xi }_d}\right)\hfill \\ =\frac{{\xi }_d^{z_{\mathit{itd}}}{L}_{\mathit{itd}}!|S(z_{\mathit{itd}},L_{\mathit{itd}})|}{{(1+{\xi }_d)}^{z_{\mathit{itd}}}{(\text{log}(1+{\xi }_d))}^{L_{\mathit{itd}}}}\text{Pois}\left(L_{\mathit{itd}};\frac{w_{\mathit{itd}}{q}_{\mathit{itd}}}{{\xi }_d}\text{log}(1+{\xi }_d)\right),\hfill \end{array}$$ (4.8)

where S(a, b) are Stirling numbers of the first kind. In (4.8), q_itd only appear in the second term (Poisson probability mass function). As shown in Section A of the supplementary material available at Biostatistics online, this enables us to update ${\mathit{h}}_{td}$ ($d=1,\dots ,D_3$, and $t=1,\dots ,T$), ${\mathit{\eta }}_{3,d}$ ($d=1,\dots ,D_3$) and ${\mathit{\lambda }}_i$ ($i=1,\dots ,N$) using the simplified manifold Metropolis adjusted Langevin (SMMALA) algorithm (Girolami and Calderhead, 2011), which would be very hard unless introducing the latent variables L_itd and ω_itd. The SMMALA algorithm exploits the geometry of the parameter space to adapt the variance–covariance matrix of its normal proposal at each iteration of MCMC, thus mitigating the problems caused by label-switching and scale ambiguity explained above. In our simulation studies, we found that SMMALA massively outperformed MALA and HMC, for which convergence was extremely slow.

Finally, we update each negative binomial dispersion parameter ξ_d ($d=1,\dots ,D_3$) using the Barker method (Livingstone and Zanella, 2022). To reduce dependence on the L_itd ($t=1,\dots ,T_i$), we integrate these parameters out when updating ξ_d. This is straightforward using (4.8), see Section A of the supplementary material available at Biostatistics online. The Barker method requires the specification of a stepsize parameter; following Livingstone and Zanella (2022), we tune this for each i during the burn-in phase of the MCMC to achieve an acceptance rate near 40%.

5 Simulation studies

We perform a simulation study to demonstrate benefits, in terms of efficiency of causal effect estimates, that can be obtained by modeling multiple outcomes jointly (i.e., loadings are shared across outcomes) rather than individually (i.e., each outcome has its own loadings matrix). See Section C of the supplementary material available at Biostatistics online for additional simulations in settings that differ from the one considered here.

5.1 Data generating mechanism

We simulate B = 2, 500 synthetic data sets. For each one, we generate the untreated outcomes from the FA model of (3.1) with $D_1=D_2=D_3=1$, T = 24 (2 years of monthly data), N = 80 and ${\mathit{\lambda }}_i,{\mathit{f}}_t,{\mathit{g}}_t,{\mathit{h}}_t\in {\mathbb{R}}^7$. Since we consider only one outcome of each type, we omit the outcome index d for the remainder of this section to ease notation.

For all i, we set ${\lambda }_{i1}=1$ (to control the mean), and draw ${\lambda }_{ij}\sim \mathrm{N}(0,1)$ for $j=2,\dots ,7$. Let ${\tilde{\mathit{f}}}_j={\left(f_{1j},\dots ,f_{Tj}\right)}^{\top },\hspace{0.17em}{\tilde{\mathit{g}}}_j={\left(g_{1j},\dots ,g_{Tj}\right)}^{\top }$, and ${\tilde{\mathit{h}}}_j={\left(h_{1j},\dots ,h_{Tj}\right)}^{\top }$. For each j, we generate ${\tilde{\mathit{f}}}_j,\hspace{0.17em}{\tilde{\mathit{g}}}_j$ and ${\tilde{\mathit{h}}}_j$ from a $\mathrm{N}(m_{1j}1,s_{1j}^2\mathit{R}),\hspace{0.17em}\mathrm{N}(m_{2j}1,s_{2j}^2\mathit{R})$ and $\mathrm{N}(m_{3j}1,s_{3j}^2\mathit{R})$, respectively, where $1$ is a T-vector of ones and R is a T × T correlation matrix. The values of m_dj and $s_{\ell j}$ ($\ell =1,2,3$) are shown in Table 1. These specifications imply that $\mathbb{E}({\mu }_{it})=5,\hspace{0.17em}\mathbb{E}(p_{it})=0.6$, and $\mathbb{E}(q_{it})=4$. Some of the $s_{\ell j}$ are set to zero so that each loading j affects only a subset of the outcomes. For fixed outcome, the nonzero $s_{\ell j}$ are all equal for simplicity. The values of $s_{\ell }$ are chosen such that the 97.5% quantiles of μ_it, p_it, and z_it (over units, times, and simulations) are roughly 7.5, 0.85, and 10, respectively. R has elements $R_{ts}=\text{exp}(\text{log}(0.8)|t-s|)$. This nondiagonal R introduces temporal correlations within the data on each unit.

Table 1.

Mean and standard deviation used to simulate the seven factors of Section 5.

	Mean			Standard deviation
j	Normal	Binomial	Count	Normal	Binomial	Count
1	$m_{11}=5$	$m_{21}=\text{logit}(0.6)$	$m_{31}=\text{log}(4)$	0	0	0
2	0	0	0	s 1	s 2	0
3	0	0	0	s 1	0	s 3
4	0	0	0	0	s 2	s 3
5	0	0	0	s 1	0	0
6	0	0	0	0	s 2	0
7	0	0	0	0	0	s 3

The value of σ is chosen such that ${\mathit{\lambda }}_i^{\top }{\mathit{f}}_t$ accounts for 80% of the variability in the y_it. We draw the n_it from a $\text{Pois}({\tilde{n}}_{it})$ distribution. For each i, ${\tilde{n}}_{i1}=5,\hspace{0.17em}{\tilde{n}}_{iT}$ is drawn from a $\text{Uni}(50,200)$ distribution and ${\tilde{n}}_{it}={\tilde{n}}_{i1}+\frac{t-1}{T-1}({\tilde{n}}_{iT}-{\tilde{n}}_{i1})$ for $1<t<T$. Similarly, we draw the w_it from a $\text{Pois}({\tilde{w}}_{it})$ distribution where for each i, ${\tilde{w}}_{i1}=5,\hspace{0.17em}{\tilde{w}}_{iT}$ is drawn from a $\text{Uni}(25,75)$ distribution and ${\tilde{w}}_{it}={\tilde{w}}_{i1}+\frac{t-1}{T-1}({\tilde{w}}_{iT}-{\tilde{w}}_{i1})$ for $1<t<T$. We have made ${\tilde{n}}_{it}$ and ${\tilde{w}}_{it}$ increasing to mimic the LTP data of Section 5. Finally, we set ξ = 2.

In each data set, T_i are chosen as follows. For every i and t, we draw $u_{it}\sim \text{Uni}(0,1)$ and let

$${\varpi }_{it}=\{\begin{array}{cc}\text{expit}\left({\kappa }_0+{\kappa }_1{p}_{it}+{\kappa }_2{q}_{it}\right)\hfill & t>t_{\text{min}}\hfill \\ 0\hfill & t\le t_{\text{min}}\hfill \end{array},$$ (5.9)

where $\text{expit}(·)=\text{exp}(·)/(1+\text{exp}(·))$. Then, for each i we set $T_i=\text{min}\left\{t:u_{it}<{\varpi }_{it}\right\}$. We set the minimum number of pre-intervention time points to $t_{\text{min}}=8$. The value of κ₀ is chosen such that the average over simulated data sets N1 is 40. The values of κ₁ and κ₂ control the degree of unobserved confounding of the effects β_it and δ_it, respectively. We set κ₁ such that the average (over simulated data sets) value of the empirical completion probability $k_{it}/n_{it}$ is roughly 10% higher in controls units for all $t>t_{\text{min}}$. The value of κ₂ is chosen such that on average (over simulated data sets), $z_{it}/w_{it}$ is roughly 10% higher in controls units for every $t>t_{\text{min}}$.

For each simulated data set b, we perform a multivariate (MV) and a univariate (UV) analysis. MV analysis is carried out by fitting the FA model of Section 3 to all three outcomes jointly. UV analysis is carried out by fitting the FA model of Section 3 to each one of the outcomes individually. In both cases, the models that we fit are correctly specified. For both MV and UV analyses, we run MCMC for 100,000 iterations, saving posterior draws every 50 iterations to obtain 2,000 posterior draws. Of these, 500 are discarded as burn-in. The maximum number of factors $J^{\ast }$ in MV and UV analyses is set to 25 and 15, respectively.

For each b, we generate the data of treated units in their post-intervention period under L = 50 different scenarios, corresponding to causal effects of increasing magnitude. Thus, we obtain L different estimates (and CIs) for all causal effects defined in Section 3. For each $\ell =1,\dots ,L$, let ${\tilde{\alpha }}_{it}^{(\ell )},\hspace{0.17em}{\tilde{\beta }}_{it}^{(\ell )}$, and ${\tilde{\delta }}_{it}^{(\ell )}$ be the causal effect of intervention on μ_it, $\text{logit}(p_{it})$ and $\text{log}\hspace{0.17em}\left(q_{it}\right)$, respectively. We use the tilde ($\widetilde{·}$) notation to indicate that these are the effects on the mean function rather than on the outcome as in (3.2). For the binomial and count outcomes, the effects are applied on the logit-scale and log-scale, respectively, to avoid negative values. For each $\ell$, we assume that for all $i:T_i<T$ and $t>T_i,\hspace{0.17em}{\tilde{\alpha }}_{it}^{(\ell )}={\tilde{\alpha }}^{(\ell )},\hspace{0.17em}{\tilde{\beta }}_{it}^{(\ell )}={\tilde{\beta }}^{(\ell )}$, and ${\tilde{\delta }}_{it}^{(\ell )}={\tilde{\delta }}^{(\ell )}$. The magnitude of ${\tilde{\alpha }}^{(\ell )},\hspace{0.17em}{\tilde{\beta }}^{(\ell )}$, and ${\tilde{\delta }}^{(\ell )}$ increases with $\ell$ and ${\tilde{\alpha }}^{(1)}={\tilde{\beta }}^{(1)}={\tilde{\delta }}^{(1)}=0$. Despite considering L scenarios for the intervention effect, we only need to run the MV and UV analyses once for each b, as the draws from the posterior of the potential untreated outcomes do not depend on postintervention data.

5.2 Results

We report results only for ${\mathcal{C}}_i=\left\{{\alpha }_i,{\beta }_i,{\gamma }_i,{\delta }_i\right\}$, representing the average response of each unit to the intervention. We compare the performance of the MV and UV approaches in terms of (i) the bias of the point estimates of the ${\mathcal{C}}_i$; (ii) the standard error of the point estimates of the ${\mathcal{C}}_i$; (iii) the width of the 95% CIs of ${\mathcal{C}}_i$; (iv) the probability of detecting an intervention effect (power). For scenario $\ell =1$ (i.e., no intervention effect), we define a detection as a CI that does not include zero to obtain the false positive rate. For any other scenario (i.e., positive intervention effect), we define a detection as a CI whose lower bound is larger than zero. For each estimand in ${\mathcal{C}}_i$, we summarize each performance measure by taking the weighted average over simulated data sets and treated units. The weights are introduced to account for the varying number of treated units. More specifically, we assign a weight of $1/(B{N}_2^{(b)})$ to each treated unit in simulated data set b, where $N_2^{(b)}$ is the total number of treated units in that data set. To investigate the impact of T_i, we further calculate the measures over simulations and units with T_i = 8, T_i = 16, and T_i = 23.

The results for scenario $\ell =1$ (no intervention effect) are presented in Table B.1 of the supplementary material available at Biostatistics online. Overall, there are no major problems with the estimates provided by UV and MV approaches: the bias of the point estimates of ${\mathcal{C}}_i$ is negligible (compared to the standard deviation), and the false positive rates are close to the nominal 5%. The exceptions are the estimates of β_i, γ_i, and δ_i provided by UV approach for T_i = 8 i.e., for units with limited data in the preintervention period. For these, we find considerable bias (compared to the standard error) and inflated false positive rates. The estimates of α_i are not affected because there is no confounding of these effects: the ${\varpi }_{it}$ in (5.9) are not a function of μ_it.

In terms of efficiency, we see that the MV approach outperforms the UV approach. In particular, the standard errors of the point estimates of ${\mathcal{C}}_i$, as well as the width of CIs, are on average smaller for the MV approach, as can be seen in Table 1 of Appendix B of the supplementary material available at Biostatistics online. The gains in efficiency due to joint outcome modeling are higher when T_i is small. This is expected, because when T_i is large there is sufficient data per treated unit on each outcome to estimate the loadings and so there is less benefit from using data on the other outcomes. Finally, it is worth noting that for all estimands and both approaches, the measures of efficiency (standard error, CI width) are better for T_i = 16 than they are for T_i = 8 and T_i = 23. The reason is that the efficiency in the estimates of ${\mathcal{C}}_i$ improves with both T_i and $T-T_i$ (total number of postintervention time points). When T_i increases, there is more data to estimate the loadings and thus the potential untreated outcomes. When $T-T_i$ increases, there is more data for each one of the ${\mathcal{C}}_i$. In our simulation with fixed T, units of moderate T_i achieve the better balance between T_i and $T-T_i$.

The bias of point estimates, standard error of point estimates and width of CIs in scenarios $\ell >1$ are very similar to scenario $\ell =1$ and therefore not discussed³. Figure 2 shows the power achieved by the UV and MV approaches across all the different scenarios. We see that for all four estimands, the gains in efficiency due to joint outcome modeling substantially improve the probability of detecting a nonzero intervention effect. For example, ${\tilde{\beta }}_i=0.4$ is detected with probability 44% using the UV approach, whereas it is detected with probability 65% when using the MV approach. Figure 2 in Appendix B of the supplementary material available at Biostatistics online shows the power for different values of T_i. For reasons explained above, we see that the improvements in power achieved by the MV approach compared to the UV are greater when T_i is either 8 or 16.

Fig. 2.

Power of detecting an intervention effect for a randomly chosen treated unit (y axis) as a function of the magnitude of the intervention effect (x axis). (a)–(d) correspond to causal effects α_i, β_i, γ_i, and δ_i, respectively. The results are based on 2,500 simulated data sets.

For binomial outcomes, the uncertainty in the estimates of a treated unit’s potential untreated outcomes p_it and $k_{it}^{(0)}$, and thus the accuracy of the estimates of β_it and γ_it, depends on both T_i and the values of n_it in the pre-intervention period. More specifically, the lower the counts ${\left\{n_{it}\right\}}_{t=1}^{T_i}$ are, the less information to estimate the loadings there is. Similarly, for count outcomes, the uncertainty in the estimates of a treated unit’s $z_{it}^{(0)}$ ($t>T_i$) depends on ${\left\{w_{it}\right\}}_{t=1}^{T_i}$. Figure 3(a) presents a heatmap of the CI width for ${\beta }_i$ obtained by the MV approach for different combinations of T_i and ${\overline{n}}_i=\frac{1}{T_i}\sum _{i=1}^{T_i}{n}_{it}$. For any fixed T_i, the width of CIs decreases with ${\overline{n}}_i$. Figure 3(b) shows the percentage decrease in CI width achieved by the MV approach compared to the UV approach for different combinations of T_i and ${\overline{n}}_i$. We see that for fixed T_i, the gains in efficiency due to joint outcome modeling are similar for the different values of ${\overline{n}}_i$. For the count outcome, we present the power achieved for moderate ${\tilde{\delta }}_i$ ($\approx 1.25$) as a measure of efficiency; see Fig. 3(c). We find that power increases with ${\overline{w}}_i$ for fixed T_i. Figure 3(d) shows the percentage increase in power for moderate $\tilde{\delta }$ achieved by the MV approach compared to the UV. Again, we see that for fixed T_i, the gains do not differ much over across the ${\overline{w}}_i$.

Fig. 3.

Effect of n_it and w_it on the efficiency of the causal estimates. (a) average (over simulated data sets and treated units) CI width of β_i achieved by the MV model, as a function of ${\tilde{n}}_i=\sum _{t=1}^{T_i}{n}_{it}/T_i$ and T_i. (b) average (over data sets and units) % reduction in CI width of β_i as a function of ${\tilde{n}}_i$ and T_i achieved by the MV model compared to the UV. (c) power of detecting a moderate intervention effect from δ_i achieve by the MV model, for different values of ${\tilde{w}}_i=\sum _{t=1}^{T_i}{w}_{it}/T_i$. (d) % increase in power of detecting a moderate effect on δ_i achieved by the MV model compared to the UV. In all panels, entries that were obtained as the average of less than 50 simulated data sets were discarded.

6 Evaluation of LTPs

In this section, we apply our multivariate FA model to the LTP data of Section 2. We set $J^{\ast }=25$ and ran three MCMC chains starting from different initial values. For each chain, we ran for 250,000 iterations, discarding the first 50,000 as burn-in, and thinning the remaining draws at every 100 iterations to obtain 2,000 draws from the posterior. Convergence was assessed visually by comparing the posterior densities and trace plots obtained from the three chains for multiple randomly selected causal estimands; these showed that all three chains have converged to the same stationary distribution and that mixing was good. Results for the case completion outcome are shown in Fig. 4. Figures D.2, D.3, and D.4 of the supplementary material available at Biostatistics online show results for the remaining outcomes.

Fig. 4.

Results for outcome completion. (a) 95% CIs for $p_{\iota t1}$ and $p_{\iota t1}^{(T_{\iota })}$, where ι is a unit chosen for illustration. (b) scatterplot of the posterior mean of ${\beta }_{it1}$. Filled yellow dots represent ${\beta }_{\iota t1}$. Panels (c) and (d) show posterior summaries of ${\beta }_{i1}$ and ${\gamma }_{i1}$, respectively, where units have been ordered by increasing mean posterior ${\beta }_{i1}$ in both plots. The vertical yellow lines indicate unit ι. Boxes and whiskers represent the 75% and 95% CIs, respectively.

Potential outcomes. Figure 4(a) refers to a unit, say ι, chosen for illustration. The plot shows 95% CIs for $p_{\iota t1}$ and $p_{\iota t1}^{(T_{\iota })}$ in yellow and blue, respectively. For most days, the bands for $p_{\iota t1}^{(T_{\iota })}$ are above the bands for $p_{\iota t1}$, indicating that the LTP had a positive effect on completion probability on these days. We also see that the 95% CIs for $p_{\iota t1}$ are substantially wider than the CIs for $p_{\iota t1}^{(T_{\iota })}$; this is due to the large number of cases in the post-LTP period, allowing $p_{\iota t1}^{(T_{\iota })}$ to be estimated with high precision. Hence, uncertainty in ${\beta }_{\iota t1}$ is mainly due to the uncertainty in $p_{\iota t1}$.

Effects by unit and day. Scatterplots of the posterior means of ${\beta }_{it1}$ and ${\gamma }_{it1}$ are shown in Fig. 4(b) and Figure D.5(a), respectively, where the ${\beta }_{\iota t1}$ and ${\gamma }_{\iota t1}$ are shown in yellow. There is considerable heterogeneity in the estimated effects, both in terms of magnitude and sign. We observe an adverse effect during mid-September, a period during which several of the UTLAs operating LTPs reported that the number of people employed locally for contact tracing was insufficient to deal with the increasing number of cases. However, there are overall more positive than negative effects (especially toward the end of the study), suggesting that LTPs improved case completion TT on average.

There is also considerable heterogeneity of estimated effects for the other outcomes, see Section D of the supplementary material available at Biostatistics online. For timely case completion, there are both periods with positive and periods with negative effects (Fig. D.2(a) and (b) of the supplementary material available at Biostatistics online). We note that when we considered timely case completion out of $n_{it1}$ (rather than $k_{it1}$), the results (not shown) suggested that timeliness improves. This is expected, because LTPs lead to additional cases being completed, some of which are completed within 48 hours. For contact completion (Fig. D.3 of the supplementary material available at Biostatistics online) and number of contacts (Fig. D.4 of the supplementary material available at Biostatistics online), the estimated effects are centered around zero indicating that on average, LTPs did not affect the performance of TT for these outcomes.

A possible explanation as to why LTPs appear to improve TT performance in some units/days while negatively affecting it in some others is that the model of LTP employed varies between units (e.g., 5-day vs. 7-day working pattern or larger teams vs. smaller teams). Population factors, such as demographic characteristics of the cases, may also play a role. Finally, it is possible that staff delivering LTPs were becoming more effective as they acquired more experience.

Average unit effects. The posterior distribution of the average (over time) unit effects ${\beta }_{i1}$ is summarized in Fig. 4(c), where we have sorted units by increasing mean posterior ${\beta }_{i1}$. There are units that benefited from the LTPs on average ($\text{Pr}\left({\beta }_{i1}<0\right)$ small), units that showed an adverse effect on average ($\text{Pr}\left({\beta }_{i1}>0\right)$ small), and units that seemed unaffected (95% CIs centered at zero). This is also the case for the other three outcomes (see Section D of the supplementary material available at Biostatistics online). Figure 4(d) summarizes the posterior distribution of ${\gamma }_{i1}$, where units are sorted as in Fig. 4(c). Note that LTPs can have a strong effect (positive or negative) on the ${\gamma }_{i1}$ (or ${\gamma }_{i2},\hspace{0.17em}{\gamma }_{i3}$) of a unit even if ${\beta }_{i1}$ (or ${\beta }_{i2},\hspace{0.17em}{\beta }_{i3}$) on the same unit is close to zero. This is because the latter measure depends on the total number of cases on each day: a 2% improvement in completion probability, will result in tens of additional completed cases in units with thousands of new cases, but possibly none in units with tens of new cases.

Overall effects. The estimated average effects over all units and post-intervention time points, β_d, γ_d (d = 1, 2, 3), and δ₁, are presented in Table 2 of the supplementary material available at Biostatistics online. These results suggest that case completion benefited from the LTPs as we estimate an additional 3.61 (95% CI [3.02–4.26]) completed cases per unit per day). Further, it appears that LTPs led to a drop in the number of contacts (we estimate that on average, 1.99 less contacts per unit per day were obtained due to LTPs).

Table 2.

Posterior mean along with 95% CIs for the “overall” effects β_d (d = 1, 2, 3), γ_d (d = 1, 2, 3) and δ₁. We present overall effects for the entire study period, as well as the period October 1, 2020–November 15, 2020.

		Whole period		10/2020 onward
Outcome	Effect	Estimate	95% CI	Estimate	95% CI
Completion	β 1	0.013	0.006–0.021	0.044	0.038–0.051
	γ 1	3.612	3.015–4.262	5.734	4.885–6.618
Timeliness	β 2	–0.074	–0.080––0.068	0.014	0.007–0.020
	γ 2	0.218	–0.221–0.660	1.973	1.367–2.602
Contact completion	β 3	-0.014	–0.022––0.007	-0.007	–0.015–0.001
	γ 3	-1.992	–3.743––0.439	-2.416	–4.746––0.411
Number of contacts	δ 1	-0.552	–7.852–4.893	-0.994	–10.870–6.480

Posterior ranks. Identifying the units that benefit the least from their LTPs is important to improve the way of implementing LTPs in these units (e.g., by increasing the size of the teams). We do this using the posterior ranks. In Fig. D.6 of the supplementary material available at Biostatistics online, we summaries the posterior distribution of $r_{\iota t1}^{(\beta )},\hspace{0.17em}{r}_{\iota t2}^{(\beta )},\hspace{0.17em}{r}_{\iota t3}^{(\beta )}$, and $r_{\iota t1}^{(\delta )}$. For most of the days in unit ι’s postintervention period, the mean posterior $r_{\iota t1}^{(\beta )}$ is high ($\approx 0.85$), suggesting that unit ι was one of the units who benefited most from the LTPs in terms of case completion. The ranks for the remaining estimands as not as high.

Sensitivity analyses. In Section D of the supplementary material available at Biostatistics online, we perform a series of sensitivity to study the robustness of our findings to some hyper-parameter values and modeling choices. In particular, we repeat the analysis setting ν = 1, setting $J^{\ast }=15$ and $J^{\ast }=35$, as well as allowing the $v_{jl}\sim \text{Uni}[0,1]$ for all j and $\ell$. The results suggest that our findings are not sensitive to these specifications.

7 Discussion

Motivated by an application concerning COVID-19, we have proposed a novel methodology for evaluating the impact of an intervention using observational time-series data. This methodology generalizes an existing causal multivariate FA method for normally distributed outcomes (Samartsidis et al., 2020) to the mixed outcome setting. This is an important addition to the literature in the field because, to our knowledge, it is the only method that can simultaneously (i) deal with outcomes of mixed type, (ii) make use of shared variability between subsets of multiple outcomes to improve the statistical efficiency of causal estimates, and (iii) provide uncertainty quantification for all causal estimands of interest.

We used the proposed methodology to estimate the impact of LTPs on the effectiveness of England’s NHS TT. The results suggest that on average, LTPs did not have a strong effect on contact completion or number of contacts elicited but improved case completion and timely case completion. However, there is big heterogeneity in the estimates of the causal effects both between and within (over time) units. It is likely that the heterogeneity is partly driven by differences in the model of LTP employed in each unit and day. Unfortunately, further investigation is not possible because this information has not been routinely collected. Nonetheless, our analyses highlight the importance of recording such data and we hope that they will encourage future collection.

Our approach leans itself to a number of extensions. First, it is likely that the potential untreated outcomes of units in spatial proximity are correlated, but our FA model does not make use of the geographical location of units. This could be addressed by assuming that for any j, $\mathbb{C}\text{or}({\lambda }_{ij},{\lambda }_{\iota j})$ is a function of the distance between units i and ι. Second, we have assumed that continuous outcomes are normally distributed. This assumption might not hold in some data sets, even after transformations are applied. Hence, it is worth considering more flexible continuous distributions, e.g., the Student t and Gamma. Third, we will impose temporal structure on the factor parameters (e.g., with splines) parameters, to improve efficiency further.

The computation time required to apply our method to a typically sized policy evaluation data set is reasonable. For example, MCMC took approximately 90 min and 8 h to run on the simulated and real data, respectively, both on a 3.6 GHz Intel i7 machine with 16 GB of RAM. The main computational complexity arises from the need to invert a $J^{\ast }\times J^{\ast }$ matrix to update each of the N vectors of loadings and TD vectors of factors and is thus $O\left((N+TD)J^{{\ast }^3}\right)$. This can be prohibitive when applying to data from other fields such as biology. To overcome this issue, we will consider alternative approaches for estimation, e.g., using variational methods.

8 Software

Implementation codes are available at https://github.com/psamartsidis/cbfa.git.