SUMMARY
Nursing home bed-hold policies provide continuity of care for Medicaid beneficiaries by paying nursing homes to reserve beds so residents can return to their facility of occupancy following an acute hospitalization. In 2001, Michigan implemented bed-hold policies in nursing homes. We investigated the impact of these policies on mortality and hospitalizations using 1999–2004 quarterly data from nursing homes in Michigan and nursing homes in 11 states that did not implement such policies. Synthetic Control has been used to estimate the effects of policies by accounting for changes over time unrelated to the intervention. Synthetic Control is intended for scalar continuous outcome at each period, and assumes a single treated unit and multiple control units. We propose a Bayesian procedure to overcome these limitations. It imputes the outcomes of nursing homes in Michigan if they were not exposed to the policy by matching to non-exposed nursing homes that are associated with the exposed ones in the pre-policy period. Because sampling from a Bayesian model is computationally challenging, we describe an approximation procedure that can be implemented using existing software. Our approach can be applied to other studies that examine the impact of policies.
Keywords: Bayesian analysis, Causal inference, Difference-in-differences, Observational studies, Synthetic control
1. Introduction
1.1. Overview
Almost all nursing homes in the United States are certified to participate in the Medicare and Medicaid programs. To participate, they must adhere to more than 170 quality and health standards set by the Centers for Medicare and Medicaid Services (Mukamel and others, 2014). Each state sets its Medicaid nursing home reimbursement policies as well as other policies that impose additional standards (Wiener and Stevenson, 1998). Many states enacted nursing home bed-hold policies, under which nursing homes are paid to reserve beds so that Medicaid residents may return to their facility of occupancy following an acute hospitalization. The main goal of these policies was to encourage continuity of care for Medicaid beneficiaries (Intrator and others, 2009). Intrator and others (2007) showed that nursing home residents in states that enacted bed-hold policies had higher odds of hospitalization, and Unruh and others (2013) showed that presence of bed-hold policy was associated with an increase in the likelihood of hospitalizations of long-stay nursing home residents. Unruh and others (2013) posited that bed-hold policies incentivize hospitalization of Medicaid nursing home residents when the marginal profit associated with bed-hold payment surpasses the marginal profit associated with caring for an individual in the nursing home. In addition, relocation to a new facility may lead to morbidity for many reasons including the duplication of tests and increased likelihood of medical errors (Intrator and others, 2009). Thus, two important outcomes in assessing the effects of these policies on nursing home quality of care are the nursing home’s rates of acute hospitalization and mortality.
Evaluation of policy implications in the absence of randomized experiments has been an important research question in health services research and economics. Comparison of pre-policy to post-policy outcomes only for units that are exposed to the policy may be contaminated by temporal trends or by effects of other events that occurred between both periods (Abadie, 2005). However, when only a fraction of the units had been exposed to the policy, the units that were not exposed to the policy could be used to identify temporal variations in the outcomes that are not due to policy exposure (Abadie, 2005).
In order to estimate the effects of a policy from panel data, the economics and statistical literature describe procedures that rely on various identifying assumptions. One commonly used assumption is the unconfoundedness assumption, which asserts that given observed pre-policy covariates the probability that a unit was exposed to the policy did not depend on post-policy outcomes (Imbens and Rubin, 2015). When this assumption is met, comparing the observed outcomes of exposed units to unexposed units with similar probability of being exposed yields a valid estimate of the policy effect. When the unconfoundedness assumption is violated, other assumptions are required. One possible assumption is that treatment assignment is independent of the outcomes given the covariates and lagged outcomes (Angrist and Pischke, 2008b). Another possible assumption is the “common trend” assumption, which asserts that if exposed units had not been subjected to the policy, they would have experienced the same time-related variations as the unexposed units (Lechner, 2011). These two assumptions are not necessarily nested and choosing between them must be rationalized based on substantive knowledge of the assignment process (Angrist and Pischke, 2008b).
The most compelling studies that rely on time trends based assumptions report outcomes for exposed and unexposed units for a period that is long enough to show these underlying trends. Attention is given to the manner by which deviations from the trends relate to changes in policy (Angrist and Pischke, 2008a). With scalar continuous outcomes, it is easier to summarize trends, but with multiple discrete outcomes, this is not a trivial task. We propose a method that relies on likelihood functions to match exposed and unexposed units on pre-policy trajectories. Using the trajectories of the matched unexposed units it imputes the missing potential outcomes for the units exposed to the policy. The method is based on a Bayesian formulation that can incorporate multiple time points, multiple exposed and unexposed units, and multiple outcomes. Because Bayesian estimation procedures can be computationally intensive, we also propose an efficient approximation procedure that results in statistically valid interval estimates.
1.2. Notations and framework
The causal effect of a policy 
on an outcome 
, where 
may be a vector of outcomes, for nursing home 
(
) at time 
(
) is the comparison of two “potential” outcomes, 
and 
, corresponding to the two possible levels of 
: 
indicates the exposure to the policy at time 
, and 
indicates non-exposure to the policy. Only one value of 
can be observed: the potential outcome corresponding to the action actually taken at that time for that unit (Rubin, 1978). We assume the stable unit treatment value assumption (SUTVA) (Imbens and Rubin, 2015), so that this notation is functionally well defined.
We also assume that a policy can only change once within the study period, because units that have been exposed to a policy may behave differently from units that were never exposed when a policy is repealed. Formally, 
. Let 
if 
and 
otherwise, and let 
be the number of pre-policy periods for unit 
with 
. For notational simplicity, we assume that the number of pre-policy periods is equal for all units, 
. 
is generally not of interest, because unit 
is not exposed to the policy before time 
, and researchers are commonly interested in the effects of a policy from time 
onward. In addition to the potential outcomes and treatment assignment, we record an array of covariates 
, which are, by definition, unaffected by the treatment. We distinguish between two sets of covariates. The first set includes baseline covariates that do not change over time (
). The second are outcomes variables that are recorded before the policy was changed, 
, and are therefore unaffected by it.
Causal effects are summarized by estimands, 
, which are functions of the unit-level potential outcomes on a common set of units (Rubin, 1978) depicted in the 
-row array 
, where 
, 
, 
, 
and 
.
A common estimand of interest when examining a policy intervention is the super-population average treatment effect on the units that were exposed to the policy at time 
(Lechner, 2011), 
. Another estimand of interest is the finite population analog of 
, 
, where 
. This estimand estimates the average effect for the set of treated units in the sample. These estimands can also be summarized over the entire post-policy period with either 
or 
.
Generally, the labeling of the 
units in 
is assumed to be a random permutation of the 
rows, or in other words, the 
-row array is row exchangeable (Rubin, 1978, 2007). However, because the outcome vectors are recorded over time, units may be influenced by unobserved events such that 
is exchangeable in batches.
A key component of causal effect estimation is the assignment mechanism, 
, which is the process that determines which units were exposed to the policy (Imbens and Rubin, 2015). Note, that by changing 
for each unit we only change which values in 
are observed, but not the values themselves. We denote the observed values as 
, where 
. The combination of 
with the assignment mechanism specifies a joint distribution for all observables:
 |
(1.1) |
A class of assignment mechanisms that is important to Bayesian inference is the ignorable assignment mechanism. Ignorable assignment mechanisms are defined by their freedom from dependence on any missing potential outcomes, 
. Unconfounded assignment mechanism is a special type of ignorable assignment mechanism that does not depend on the potential outcome (Imbens and Rubin, 2015), 
. An assignment mechanism is super-population probabilistic when 
. Many randomized and non-randomized studies assume that the assignment mechanism is probabilistic and unconfounded, because it allows for causal interpretation of the observed outcomes for exposed and unexposed units within sub-populations defined by 
.
2. Current procedures
2.1. Propensity score based methods
When the assignment mechanism is unconfounded and individualistic, propensity score based methods were shown to yield valid treatment effect estimates (Rosenbaum and Rubin, 1983). The propensity score is the conditional probability of unit 
to receive treatment based on pre-policy characteristics, 
, where 
are the parameters governing this distribution.
Generally, the propensity scores are unknown and only estimates of them are available. The statistical literature proposes several procedures for estimating intervention effects in unconfounded studies using the estimated propensity scores (see Gutman and Rubin, 2015, for references). Full matching (Rosenbaum, 1991), which is a sophisticated form of sub-classification has been used in longitudinal studies (Stuart and Green, 2008). It was shown to be a generally valid statistical method for estimating intervention effects with scalar outcomes and a single-time point, but it was not always the most efficient method (Gutman and Rubin, 2015).
2.2. Group-based trajectory modeling
In many applications of propensity score methods, pre-exposure lagged outcomes are considered as independent covariates and their relationship and trends are not included in the propensity score model. Although, one can use a linear trend of these lagged outcomes as another covariate, this is rarely implemented in practice. Group based trajectory modeling was proposed to estimate causal effects with pre-exposure lagged outcomes (Haviland and Nagin, 2005). Trajectory groups are latent equivalency classes for the history of the outcome variable, and are based on pre-exposure values of the variable that, after exposure, is the outcome variable. Let 
be the trajectory group for unit 
, and let 
be the population probability of 
. Trajectory modeling partitions units into 
trajectory groups based on 
using a finite mixture model
 |
(2.1) |
where 
denoted the multivariate distribution of 
. Assuming that the exposure is unconfounded given 
, 
, then 
can be estimated by
 |
(2.2) |
where 
is the estimated probability within the exposed group of being in trajectory 
, and 
is individual 
’s estimated indicator of being in trajectory 
. For possible estimation procedures see Haviland and Nagin (2005).
Group-based trajectory modeling can be combined with propensity score matching to balance covariates other than the pre-exposure lagged outcome that may be related to future outcomes. Haviland and others (2007) proposed a procedure that identifies the trajectory groups, and within each group, full matching based on the propensity score is used to match a treated unit to a variable number of controls. They used this procedure to estimate the effects of boys joining gangs at age 14 on subsequent violence while controlling for measured characteristics and trajectories of boys prior to age 14. Commonly, group-based trajectory matching has been restricted to a small number of trajectory groups, and it did not adjust for post-exposure unexpected events that were unrelated to the policy but may have affected the system.
2.3. Difference-in-differences (DiD) and its extensions
When the assignment mechanism is confounded, different identifying assumptions are needed to estimate the exposure effect. Difference-in-differences (DiD) is a quasi-experimental method that relies on the common trend assumption to estimate the effects of a policy with a confounded assignment mechanism (Lechner, 2011). The common trend assumption states that conditional on 
, the differences in the expected potential non-exposed outcomes over time are unrelated to the exposure group assignment (Lechner, 2011).
The conventional DiD models assume that in the absence of the policy, the average outcomes for units that are exposed to the policy and units that are not would have followed parallel paths over time (Abadie, 2005). Formally, with multiple periods the conventional DiD model is
 |
(2.3) |
where 
is the intercept, 
’s are periods coefficients, 
is the exposed group pre-policy coefficient, 
is a constant additive policy effect, and 
is an unknown vector parameter. The appeal of DiD is from its simplicity, and the ability to estimate 
using the linear regression model:
 |
(2.4) |
The least squares standard errors may be inaccurate in the presence of correlations between 
within groups and between periods. The bootstrap (Efron and Tibshirani, 1986) and the robust covariance matrix estimators (White, 1980) were proposed as possible adjustments (see Bertrand and others, 2004, for more details).
The conventional DiD was designed for scalar outcome variable, and was not extended to multiple outcomes. In addition, the common trend assumption implemented in the conventional DiD is scale dependent and it may be implausible if the effect of the policy is not additive, or when there are pre-policy characteristics that are associated with the outcome variables and are unbalanced between the exposed and the unexposed groups (Athey and Imbens, 2005; Abadie, 2005). Possible extensions that address some of these limitations, such as the “Changes-in-Changes” method (Athey and Imbens, 2005) and the semi-parametric estimator proposed by Abadie (2005) were only developed for two time points and a scalar outcome.
2.4. Synthetic control
Synthetic control method extends the DiD framework by allowing the effects of a policy to vary with time (Abadie and others, 2010). Instead of making the common trend assumption, this method assumes that the assignment mechanism is ignorable and relies on lagged outcome to estimate the effects of a policy. Synthetic control was intended for cases with a single unit that is exposed to the policy, multiple units that are unexposed to the policy and multiple periods (Abadie and others, 2010). It constructs an artificial control group that is more similar to the unit exposed to the policy in the pre-policy period than any of the control units on their own. This artificial group consists of weighted paths of the unexposed units, and it is used to predict the outcome of the exposed unit in the absence of a policy. Synthetic Control suffers from a few limitations. It was designed for a scalar continuous outcome at each period, and it was not extended to multiple discrete outcomes. Inference with Synthetic Control is limited to “placebo test” that do not have a corresponding interval estimate, and it is computationally intensive with large number of units. Lastly, when the number of potential controls is large Synthetic Control may suffer from interpolation biases and over-fitting (Abadie and others, 2015).
3. Proposed method
We propose a new method that assumes that the assignment mechanism is ignorable. This method addresses some of the limitation of the methods in Section 2. It supports many trajectory groups with multiple outcome vectors. It allows for non-parallel effect over time, and it supports large numbers of exposed and unexposed units. The proposed procedure treats the estimation of policy effects as a missing data problem (Rubin, 1978), and it imputes the missing potential control outcomes, 
.
A Bayesian inference for the effects of an exposure considers the observed values 
, 
, 
, 
to be a realization of random variables and 
, and 
to be unobserved random variables (Rubin, 1978). This perspective explicitly confronts the missing potential outcomes by taking the specification of (1.1), conditioning on the observed values and deriving the posterior distribution of 
:
 |
(3.1) |
Equation (3.1) shows that to estimate 
we only need to estimate the posterior predictive probability of the missing potential outcomes, 
, because 
is a degenerate distribution for finite population estimands and a known function for super-population estimands. This posterior predictive distribution is represented as:
 |
(3.2) |
Assuming that the assignment mechanism is ignorable, (3.2) becomes:
 |
(3.3) |
Based on (3.3), only the identification of 
, is required to estimate 
. Let 
be a set of parameters that follows prior distribution 
we define 
, where 
is the density function of 
. While the form of 
is specific to the application, it will often be convenient to express it as a product of conditional distributions, for example,
 |
(3.4) |
where 
, and 
are density functions and 
is a vector parameter.
3.1. Modeling assumptions
SUTVA and ignorability are commonly made and explicitly declared assumptions. The following assumptions are often made implicitly to simplify the estimation problem and may be inappropriate in some applications. Because Bayesian formulation requires the identification of the prior and joint distributions, we explicitly specify these assumptions. The first assumption is
 |
(3.5) |
This assumption is commonly made in prediction-related applications. Notice that it does not state that 
and 
are independent, because these parameters are used to describe the potential outcomes and the lagged outcomes that may be a priori correlated (Rubin, 2007).
Using factorization (3.4) and assuming (3.5), 
can be factored into
 |
(3.6) |
A second simplifying assumption is the no contamination of imputation across treatments (Rubin, 2007; Gutman and Rubin, 2015). Formally, this is accomplished by two assumptions:
 |
(3.7) |
where 
. Assumptions (3.7) entail that given 
,
two separate processes, 
and 
, define the behavior of units in the exposed and unexposed groups. In addition, the parameters governing these processes are a priori independent.
Because 
and 
are only averaging over differences in potential outcomes of units that are exposed to the policy, only 
is necessary in order to estimate them. Moreover, both estimands do not depend on 
and 
. Formally, 
. For simplicity we will use 
from this point forward, but all of the derivations hold for 
. Replacing 
with 
in (3.1) and assuming (3.5) and (3.7) we have that (derivation in Supplementary materials available at Biostatistics online)
 |
(3.8) |
Equation (3.8) shows that to estimate 
we only need to define 
, 
, and 
. Suppose that for each exposed unit 
, there is an unexposed unit 
that is highly correlated with unit 
if it had not been exposed to the policy. Thus, unit 
’s observed outcomes can predict unit 
’s post-policy outcomes if unit 
was not exposed. Formally,
 |
(3.9) |
where 
and 
are the parameters governing these distributions. This formulation assumes that given 
, 
and the observed 
, 
’s are independent. The first product of (3.9) encompasses the outcomes of exposed units post-policy if they would have not been exposed to the policy, and the second product encompasses the outcomes of the exposed units pre-policy.
Because the pairing of unexposed units to exposed units is unknown, we integrate over all possible pairs. Let 
, where 
represent the matched unexposed unit of exposed unit 
, then
 |
(3.10) |
where 
and 
. Assuming a uniform prior distribution over 
and that 
we can derive 
as
 |
(3.11) |
where 
represents post-policy exposed units and 
represents pre-policy exposed units. Equation (3.11) is proportional to
 |
(3.12) |
Equation (3.12) expresses the main theme of this method. For each exposed unit, the method identifies an unexposed unit that is associated with it in the pre-policy period. The associations between two units are determined using the likelihood of the unexposed unit to predict the exposed unit in the pre-policy period. Based on these associations the method imputes the missing potential outcomes of the exposed unit.
3.2. Computation
We treat the unknown matching permutation, 
, as missing data, and use a data augmentation procedure (Tanner and Wong, 1987, DA) to sample from 
. These samples are then used to generate 
. We summarize the procedure to estimate 
:
- For a given
, let 
, sample permutation 
, from a multinomial distribution with probabilities 
. Calculating 
across all possible permutations can be computationally demanding. To address it we use a version of the Metropolis–Hasting algorithm proposed by Gutman and others (2013) for sampling permutations. Each iteration of this algorithm comprises two sub-steps:
- (a) Randomly choose two observations
such that 
and propose a new permutation 
by replacing 
and if 
assign 
. - (b) Accept the new permutation with probability
Sample
from 
given 
.Sample
from 
given 
and 
.
This algorithm can become computationally intensive for large data sets. A possible approximation procedure that reduces the computations replaces Step 1 with pair matching on the value of the 
, where 
is the maximum likelihood estimator of 
. Step 2 is replaced by sampling from the asymptotic Normal approximation of the posterior distribution 
, where 
is the inverse of Fisher’s Information matrix at 
. Lastly, instead of repeating Steps 2 and 3 for many iterations, which increases computational time and storage, one can generate a few data sets where 
is imputed, analyze them separately and combine the analyses using standard multiple imputation rules (Little and Rubin, 2002). These approximations may result in smaller interval estimates, but in many cases because the variability associated with Steps 2 and 3 is significantly larger than the variability of Step 1, this procedure is generally statistically valid. We use simulations to compare the approximation procedure to the methods in Section 4, and to the full sampling procedure with our dataset in Section 5.
4. Simulations
We compared the performance of the approximation method in Section 3.2 with current methods that account for pre-intervention trends using simulations. All of the simulation configurations included a scalar 
, and assumed that there was no treatment effect.
The values of 
for the 
exposed units were generated from the normal distribution, 
. The values of 
for the 
unexposed units were generated from the normal distribution, 
.
Because the policy had no effect on the outcome, the values of 
and the potential outcomes, 
and 
for the exposed and unexposed units were generated from the linear model 
, where 
, 
, 
, and 
.
Table 1 describes the three levels of each of three of the factors 
, and 
. The factors 
, 
, 
, 
and 
also vary in each of the 
settings, yielding a 
factorial design. For each configuration of the factors, 
replications were produced.
Table 1.
Factors and corresponding levels used in the simulation analysis
| Factor | Levels | Description |
|---|---|---|
|
|
Ratio of sample sizes |
|
|
Treatment group population size |
|
|
Bias of the covariate in the treatment group |
|
|
Bias in time trend in the treatment group |
|
|
Bias in interaction of time trend and covariate |
|
|
Auto-correlation over time |
|
|
Conditional standard deviation |
|
|
Number of time points pre-policy |
The simulations were executed using R 2.15.0 (R Core Team, 2015). Full matching was implemented using the optmatch package (Hansen and Klopfer, 2006). The flexmix package (Bettina and Friedrich, 2008) was used to estimate the trajectory classes. To address possible serial correlation with the sampling variance of the DiD estimator we used the multiwayvcov (Graham and others, 2015) and assumed an unrestricted covariance structure over time within units. This method was found to perform well with more than 50 units (Bertrand and others, 2004). The effect over the whole intervention period was estimated using a hierarchical model with random intercept for each unit, implemented with the lme4 package (Bates and others, 2015).
4.1. Results
We compared propensity score matching, trajectory matching with propensity score and the newly proposed method to estimate 
. In addition, we compared the three methods to commonly used DID with and without covariate adjustment to estimate 
. For trajectory matching, the trajectory groups were estimated using a finite mixture of linear regression models with either 3, 6, or 12 components. The number of mixture components was selected based on the BIC. Within trajectory groups we used full matching on the propensity score. Pair matching was used for propensity score matching and for the newly proposed method. For the new method, we assumed that 
, thus 
. To complete the Bayesian model we assumed that 
, and 
are independent and that 
, 
, and 
.
For each configuration, and for each of the 100 replications, we calculated the estimated treatment effect, the estimated sampling variance, the corresponding 95% interval width, and determined whether the interval covered or did not cover the null value. Then, we calculated for each procedure and each configuration, the mean coverage rate, the bias, the mean estimated sampling variance, the mean squared error (MSE), and the mean interval length.
Figure 1a displays the 95% coverage across configurations for the overall estimates of 
for all of the methods. The newly proposed method is the only method that is generally statistically valid, with propensity score matching second with a median coverage of 0.78. Figures 1b–d display the coverage for the estimated treatment effect, 
, at 
, respectively. DiD based methods did not provide estimates for individual time points and were excluded from the figures. The coverages of all the methods decrease over time, but the proposed method was still generally valid after 16 time points. To identify the chief determinants of the rejection rate, we performed ANOVA on the logit-transformed values as in Gutman and Rubin (2013). We identified that P, 
, and their combination were the most important factors. With larger 
and/or smaller 
the 95% coverage is higher. The new procedure has mean coverage of 0.77 for large 
and small 
. With large 
, time is less correlated with the outcome, and combined with small 
the new method detects spurious trends. When 
increases this problem is mitigated.
Fig. 1.
95% coverage for all of the configurations across 100 replications. DiD based methods can only estimate the overall effect, and do not provide a time point effect. Thus, they were not included in the plots for a single time point post-policy. (a) 95% coverage across all time points. (b) 95% coverage first time point post-policy. (c) 95% coverage ninth time point post-policy. (d) 95% coverage last time point post-policy.
Table 2 compares the mean across configurations of the absolute bias, interval size, and RMSE for all of the methods. The new method had better coverage because it generally had the smallest bias. It also has the smallest interval estimate for the overall estimate and for each time point.
Table 2.
Mean across 1296 configurations of coverage, absolute bias, variance, and RMSE for all of the procedures over the whole post-policy period and for the first and last periods
| 95% Coverage | Bias | Variance | MSE | ||
|---|---|---|---|---|---|
| Overall | DiD with X | 0.52 | 0.32 | 0.16 | 0.19 |
| DiD no X | 0.27 | 0.32 | 0.07 | 0.19 | |
| PS matching | 0.68 | 0.25 | 0.20 | 0.15 | |
| Trajectory group matching | 0.41 | 0.51 | 0.23 | 0.34 | |
| New method | 0.91 | 0.03 | 0.04 | 0.08 | |
| First time point post-policy | PS matching | 0.67 | 0.19 | 0.14 | 0.08 |
| Trajectory group matching | 0.59 | 0.21 | 0.13 | 0.06 | |
| New method | 0.95 | 0.01 | 0.03 | 0.00 | |
| Time point 16 post-policy | PS matching | 0.68 | 0.33 | 0.26 | 0.26 |
| Trajectory group matching | 0.38 | 0.80 | 0.33 | 0.84 | |
| New method | 0.91 | 0.05 | 0.07 | 0.01 |
5. Application: nursing homes bed-hold policy
5.1. Data
The data source was composed of quarterly aggregated data for 18,078 Medicare/Medicaid certified nursing homes across the United States for the period beginning January 1, 1999 through December 31, 2004. The data were derived from the Minimum Data Set (MDS) nursing home resident assessments, Medicare inpatient claims, and the On-line Survey of Certification and Reporting (OSCAR). The MDS includes over 400 items relating to demographics, diagnoses, treatments, and measures of both physical and cognitive function. Medicare inpatient claims provided information on hospital discharges. OSCAR supplied nursing homes characteristics, such as occupancy rates, staffing, case-mix, and number of beds. Data on states’ policies was collected by the Center for Gerontology and Healthcare Research at Brown University. Our focus was on state specific Medicaid policy called bed-hold which required the states Medicaid program to reimburse nursing homes when Medicaid clients residing in nursing homes were hospitalized.
Among the 18,078 nursing homes, 3629 were not observed for all 24 quarters. For simplicity, we dropped those nursing homes from our analysis, and restricted it to the remaining 14,449 with full data for the 24 quarters. This restriction is reasonable because we do not expect the policy to influence the closing or opening of nursing homes. In the examined time period, one state (MI) introduced a bed-hold policy, one state repealed the policy (IL), one state repealed the policy and then reintroduced it (MA), eleven states never had it (CO,ID,NC,NH,NV,OR,RI,TX,UT,VA,WA), and the rest had the policy for all 24 quarters. We concentrated on the effect of bed-hold policy on nursing homes in MI in comparison to nursing homes in the eleven states that never had any bed-hold policy during the 24 quarters. MI adopted the bed-hold policy on January 1, 2001, resulting in 8 pre-policy quarters and 16 post-policy quarters. There were 413 nursing homes in MI and 2433 nursing homes in the eleven states that never implemented the policy.
The policy effects were examined on three quarterly outcomes for nursing home 
in quarter 
, 
: the number of hospitalizations from a nursing home, the number of deaths among nursing home residents, and the total number of months occupied by all nursing home residents. The number of residents’ month was included in the analysis to examine the potential impact of bed-hold policies on nursing homes marginal income. For descriptive purposes we reported the number of hospitalizations and deaths per resident month. Figure 2 presents the average hospitalization rate, mortality rate, and the number of residents’ months for MI and the 11 states that never implemented the bed-hold policy. Over the 24 quarters, the hospitalization rate in MI increased from 0.06 to 0.08 hospitalizations per resident month, the mortality rate decreased from 0.18 to 0.14 deaths per resident month and the average number of residents’ months increased from 265 to 284. Similarly, over this period, most of the states that did not implement the bed-hold policy experienced an increase in the average number of nursing homes residents, an increase or a stable rate of hospitalization, and a increase or a stable mortality rate.
Fig. 2.
Summary of the hospitalization rates, mortality rates, and average residents’ months for MI and the 11 states that did not enact the bed-hold policy. MI is the bold line in all of the figures. (a) Quarterly hospitalization rate by state (1999–2004). (b) Quarterly mortality rate by state (1999–2004). (c) Average number of persons’ month by state (1999–2004).
5.2. Models
Let 
, where 
are the number of hospitalizations, deaths, and residents’ months in facility 
at quarter 
under policy 
, respectively. In addition, let 
, where 
, be the probability function of a Poisson log linear model. We define 
using a series of conditional models
 |
(5.1) |
where 
is the Normal density function with mean 
and variance 
. This model assumes that given 
, 
, and 
only depend on 
and 
, respectively. To complete the model we define the prior distribution of the parameters in likelihood (5.1). We assume independent prior distributions for the parameters of the 3 outcomes: 
, and that 
were independent across pairs. Using an improper uniform prior distribution from 
for 
results in an improper posterior distribution, because we assumed that only one unexposed nursing home could be matched to a nursing home in MI. Thus, only 
pairs would appear in the posterior distribution, and the distributions of 
would be improper. Therefore, the following proper prior distributions were used
 |
where 
is the average of the observed information matrices across all possible pairs 
, and 
is an indicator function that is equal to 1 when the condition is satisfied. This is a relatively diffused proper prior distribution that ensures that the predicted 
will not be larger by three times the observed 
and 
.
Given samples from the posterior distribution we created 40 datasets with imputed 
for 
. Within each imputed dataset, we estimated the effect of the policy at time 
using the average differences of the observed and imputed outcomes across facilities. In addition, we used hierarchical models to estimate the average difference across all 16 quarters. Let 
and 
, 
, be the difference in hospitalization and mortality rate for facility 
at quarter 
. We modeled 
and 
. Using this hierarchical model the average change in hospitalization rate due to the policy is 
. A similar model can be used for the death rate, which is summarized by 
. We estimated 
, 
, the average effects in each quarter and their corresponding standard errors at each of the 40 complete datasets and used Rubin’s combination rule to estimate the effects of the bed-hold policy.
5.3. Results
The average change in hospitalization rate per person month over the 16 quarters was 0.01 (SE = 0.003). The average number of person months per facility quarter was 275, thus in each quarter for each nursing home we expect the bed-hold policy to increase the number of hospitalization by 2.75. The average change in death rate per person month over the 16 quarters was 
0.0002 (SE = 0.003). The average difference in the number of residents’ months is 0.25 (SE = 2.6). Figure 3 displays the observed average hospitalization rate, mortality rate, and number of resident months for MI, and the average predicted values based on the matched nursing homes. Pre-policy, the predicted values for all three outcomes were relatively similar to the observed values in MI. Post-policy, the predicted death rates and the number of resident months were similar to the estimated ones mimicking the peaks and valleys. The observed hospitalization rate increased significantly in MI, while the predicted hospitalization rate only increased slightly over the 16 quarters. The difference in rates increased from 0.008 in the first quarter post-policy to 0.014 in the last quarter. Similar results were observed for the approximation algorithm (data not shown).
Fig. 3.
Summary of the hospitalization rates, mortality rates, and average residents’ months for MI. The black solid line represents the observed value. The horizontal line represents the change of policy period (P). The gray dashed line from 1999 to 2001 represents the predictive ability of the matching, and the dotted lines is 95% interval estimates. The black dashed line from 2001 to 2004 represents the predictive values that would have been expected in the absence of bed-hold policy. The gray area from 2001 to 2004 represent the 95% interval estimate if bed-hold policy was not enacted in MI. (a) Quarterly hospitalization rate for MI (1999–2004). (b) Quarterly mortality rate for MI (1999–2004). (c) Average number of persons’ month for MI (1999–2004).
To further examine the effect of the bed-hold policy on nursing home characteristics, we examined the change in hospitalization and mortality rates for nursing homes in large and small nursing homes. Large and small nursing homes were defined as those with bed numbers in the upper and lower quartiles of nursing homes, respectively. The average change in hospitalization rate for large nursing homes was 0.015 (SE = 0.003) and for small nursing homes it was 0.003 (SE = 0.01). The average mortality rate difference for large nursing homes was 
0.007 (SE = 0.003) and 0.006 (SE = 0.008) for small nursing homes. Overall, the effect of the policy on mortality is not significantly different for small and large nursing homes; however, the effect on hospitalization is higher for large nursing homes in comparison to small nursing homes.
We compared our results for hospitalization and mortality rates with a version of the Synthetic Control method proposed by Acemoglu and others (2016). Similar increasing trends in hospitalization rates were observed in MI. The overall effect on hospitalization rate was 0.007 (p-value 
0) and the overall effect on mortality rate was 0.002 (p-value = 0.36). The newly proposed method had a larger point estimate for the effects of the policy on hospitalization, but its 95% interval estimate included the Synthetic Controls’ estimate (see Supplementary materials available at Biostatistics online).
6. Conclusion
We developed an efficient method to estimate effects of policy using lagged outcomes. We provided a Bayesian justification and explicitly described all of the assumptions the method relies on. Our proposed method is not restricted to a single trend throughout the study period, but only to how well an unexposed unit “predicts” an exposed unit in the pre-policy period. The method can handle multiple outcomes, multiple treated units, and multiple periods. We also described an efficient computational approximation that can be implemented with available software.
Because we rely on a parametric model, the proposed method may suffer from model mis-specification. In a simulation analysis, we have shown that this does not influence the validity of the model completely, because imputations using mis-specified model tend to increase estimated variance, yielding valid coverage intervals (Little and Rubin, 2002, Section 10.2.4). In cases where the exposed and unexposed units share similar trajectories, our method is superior to other methods.
Our analysis shows that nursing home bed-hold policy significantly increases hospitalization, and this increase persisted 4 years after the implementation. We did not identify significant policy effects on mortality or on the number of resident months. These findings indicate that bed-hold policy may result in major financial burdens, but without significant increase in overall patient health.
In conclusion, we described a new method to estimate causal effects when the outcomes of interest are observed for all units prior to the intervention. The method relies on matching using likelihood functions, and it allows for valid and efficient estimation of heterogeneous effects.
Supplementary Material
Funding
This work was supported by the National Institute on Aging [P01AG027296], Robert Wood Johnson Foundation Program [64435], and the National Science Foundation [Grant DMS-1557466].
References
- Abadie A. (2005). Semiparametric difference-in-differences estimators. The Review of Economic Studies 72, 1–19. [Google Scholar]
- Abadie A., Diamond A. and Hainmueller J. (2010). Synthetic control methods for comparative case studies: estimating the effect of California’s tobacco control program. Journal of the American Statistical Association 105, 493–505. [Google Scholar]
- Abadie A., Diamond A. and Hainmueller J. (2015). Comparative politics and the synthetic control method. American Journal of Political Science 59, 495–510. [Google Scholar]
- Acemoglu D., Johnson S., Kermani A., Kwak J. and Mitton T. (2016). The value of connections in turbulent times: evidence from the United States. Journal of Financial Economics 121, 368–391. [Google Scholar]
- Angrist J. D. and Pischke J. (2008a). The credibility revolution in empirical economics: how better research design is taking the con out of econometrics. Journal of Economic Perspectives 24, 3–30. [Google Scholar]
- Angrist J. D and Pischke J.-S. (2008b). Mostly Harmless Econometrics: An Empiricist’s Companion. Princeton, NJ: Princeton University Press. [Google Scholar]
- Athey S. and Imbens G. W. (2005). Identification and inference in nonlinear difference-in-difference models. Econometrica 74, 431–497. [Google Scholar]
- Bates D., Mächler M., Bolker B. and Walker S. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software 67, 1–48. [Google Scholar]
- Bertrand M., Duflo E. and Mullainathan S. (2004). How much should we trust differences-in-differences estimates? Quarterly Journal of Economics 119, 249–275. [Google Scholar]
- Bettina G. and Friedrich L. (2008). FlexMix version 2: finite mixtures with concomitant variables and varying and constant parameters. Journal of Statistical Software 28, 1–35.27774042 [Google Scholar]
- Efron B. and Tibshirani R. (1986). Bootstrap methods for standard errors, confidence intervals, and other measures of statistical accuracy. Statistical Science 1, 54–75. [Google Scholar]
- Graham N., Arai M. and Hagstromer B. (2015). multiwayvcov: Multi-Way Standard Error Clustering. Vienna, Austria: R Foundation for Statistical Computing. [Google Scholar]
- Gutman R., Afendulis C. C. and Zaslavsky A. M. (2013). A bayesian procedure for file linking to analyze end-of-life medical costs. Journal of the American Statistical Association 108, 34–47. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Gutman R. and Rubin D. B. (2013). Robust estimation of causal effects of binary treatments in unconfounded studies with dichotomous outcomes. Statistics in Medicine 32, 1795–1814. [DOI] [PubMed] [Google Scholar]
- Gutman R. and Rubin D. B. (2015). Estimation of causal effects of binary treatments in unconfounded studies. Statistics in Medicine 34, 3381–3398. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Hansen B. B. and Klopfer S. O. (2006). Optimal full matching and related designs via network flows. Journal of Computational and Graphical Statistics 15, 609–627. [Google Scholar]
- Haviland A. M. and Nagin D. S. (2005). Causal inferences with group based trajectory models. Psychometrika 70, 557–578. [Google Scholar]
- Haviland A., Nagin D. S. and Rosenbaum P. R. (2007). Combining propensity score matching and group-based trajectory analysis in an observational study. Psychological Methods 12, 247. [DOI] [PubMed] [Google Scholar]
- Imbens G. and Rubin D. B. (2015). Causal Inference in Statistics, and in the Social and Biomedical Sciences. Cambridge University Press. [Google Scholar]
- Intrator O., Grabowski D. C., Zinn J., Schleinitz M., Feng Z., Miller S. and Mor V. (2007). Hospitalization of nursing home residents: the effects of states’ medicaid payment and bed-hold policies. Health Services Research 42, 1651–1671. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Intrator O., Schleinitz M., Grabowski D. C., Zinn J. and Mor V. (2009). Maintaining continuity of care for nursing home residents: effect of states’ medicaid bed-hold policies and reimbursement rates. Health Services Research 44, 33–55. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Lechner M. (2011). The estimation of causal effects by difference-in-difference methods. Foundations and Trends in Econometrics 4, 165–224. [Google Scholar]
- Little R. J. A. and Rubin D. B. (2002). Statistical Analysis with Missing Data, Second Edition. Hoboken: Wiley-Interscience. [Google Scholar]
- Mukamel D. B., Haeder S. F. and Weimer D. L. (2014). Top-down and bottom-up approaches to health care quality: the impacts of regulation and report cards. Annual Review of Public Health 35, 477–497. [DOI] [PubMed] [Google Scholar]
- R Core Team. (2015). R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing. [Google Scholar]
- Rosenbaum P. R. (1991). A characterization of optimal designs for observational studies. Journal of the Royal Statistical Society. Series B (Methodological) 53, 597–610. [Google Scholar]
- Rosenbaum P. R. and Rubin D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika 70, 41–55. [Google Scholar]
- Rubin D. B. (1978). Bayesian inference for causal effects: the role of randomization. Annals of Statistics 6, 34–58. [Google Scholar]
- Rubin D. B. (2007). Statistical inference for causal effects, with emphasis on applications in epidemiology and medical statistics. In: Rao C. R., Miller J. P. and Rao D. C. (editors), Handbook of Statistics, Volume 27, Chapter 2 The Netherlands: Elsevier, pp. 28–63. [Google Scholar]
- Stuart E. A. and Green K. M. (2008). Using full matching to estimate causal effects in nonexperimental studies: examining the relationship between adolescent marijuana use and adult outcomes. Developmental Psychology 44, 395–406. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Tanner M. A. and Wong W. H. (1987). The calculation of posterior distributions by data augmentation. Journal of the American Statistical Association 82, 528–540. [Google Scholar]
- Unruh M. A., Grabowski D. C., Trivedi A. N. and Mor V. (2013). Medicaid bed-hold policies and hospitalization of long-stay nursing home residents. Health Services Research 48, 1617–1633. [DOI] [PMC free article] [PubMed] [Google Scholar]
- White H. (1980). A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity. Econometrica: Journal of the Econometric Society 48, 817–838. [Google Scholar]
- Wiener J. M. and Stevenson D. G. (1998). State policy on long-term care for the elderly. Health Affairs 17, 81–100. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.