Evaluating the Impact of Parent‐Reported Medical Home Status on Children's Health Care Utilization, Expenditures, and Quality: A Difference‐in‐Differences Analysis with Causal Inference Methods

Abstract

Objective

To evaluate the effects of the parent‐reported medical home status on health care utilization, expenditures, and quality for children.

Data Sources

Medical Expenditure Panel Survey (MEPS) during 2004–2012, including a total of 9,153 children who were followed up for 2 years in the survey.

Study Design

We took a causal difference‐in‐differences approach using inverse probability weighting and doubly robust estimators to study how changes in medical home status over a 2‐year period affected children's health care outcomes. Our analysis adjusted for children's sociodemographic, health, and insurance statuses. We conducted sensitivity analyses using alternative statistical methods, different approaches to outliers and missing data, and accounting for possible common‐method biases.

Principal Findings

Compared with children whose parents reported having medical homes in both years 1 and 2, those who had medical homes in year 1 but lost them in year 2 had significantly lower parent‐reported ratings of health care quality and higher utilization of emergency care. Compared with children whose parents reported having no medical homes in both years, those who did not have medical homes in year 1 but gained them in year 2 had significantly higher ratings of health care quality, but no significant differences in health care expenditures and utilization.

Conclusions

Having a medical home may help improve health care quality for children; losing a medical home may lead to higher utilization of emergency care.

The notion of a patient‐centered medical home (“medical home” hereafter) was originally proposed in 1967 by the American Academy of Pediatrics (AAP) for children with special health care needs (CSHCN) and redefined by AAP in 2004 as an ideal model of care for all children (American Academy of Pediatrics Medical Home Initiatives for Children with Special Needs Project Advisory Committee 2004; Sia et al. 2004). As an enhanced model for primary care practice, a medical home represents “a clinical setting that serves as a central resource for a patient's ongoing medical care” (Dickens et al. 1992; Stange et al. 2010). There are substantial efforts across the nation to promote the medical home model with financial support provided by the 2010 Patient Protection and Affordable Care Act.

Generally, the concept of a medical home can be defined in three alternative ways: (1) an exposure to an intervention (Friedberg et al. 2014, 2015), (2) a particular type of primary care practice model (Hasselt et al. 2015), or (3) be based on patients’ experiences of care (Strickland et al. 2004; Romaire and Bell 2010). Studies based on each of these general definitions have reached mixed conclusions about the effects of medical homes on health care utilization, expenditures, and quality. Defining medical homes based on patients’ experiences, Romaire, Bell, and Grossman (2012b) found few significant differences in annual utilization and expenditures, among children with special health care needs, between those with and without medical homes. In a separate article, the same authors reported that access to medical homes among children and youth was associated with lower odds of incurring emergency room expenditures but higher odds of incurring expenditures in outpatient and prescription services (Romaire, Bell, and Grossman 2012a).

One national analysis found that children with special health care needs without medical homes were two to three times more likely to report an unmet need for health services than those with medical homes (Miller et al. 2013). Another recent study focused on medical homes among adults, finding that adults with medical homes had higher rates of preventive care and positive patient experiences than those without (Beal, Hernandez, and Doty 2009). However, the above studies were based on cross‐sectional analyses, which did not consider the changes in medical home status over time and did not differentiate between the effects of gaining and losing medical homes. More studies are needed to strengthen the evidence of the link between medical homes and health care utilization, expenditures, and quality. Advanced causal inference methods are an appealing approach to meeting this need because such methods can reduce biases stemming from imbalances in observed characteristics (when assignment is nonrandom) in patients with and without medical homes.

In this article, we applied the causal difference‐in‐differences estimators to a nationally representative panel dataset to examine how changes in parent‐reported medical home status over a 2‐year period affect children's health care utilization, expenditures, and quality. To our knowledge, this is the first study to use national, longitudinal data to draw causal inferences in patient‐reported medical home effects. Our study also explicitly addresses the differential effects of gaining and losing medical homes.

Methods

Data Source

We used the longitudinal component of the Medical Expenditure Panel Survey (MEPS), which the Agency for Healthcare Research and Quality (AHRQ) conducts on a large‐scale, nationally representative basis to collect data on health care utilization, expenditures, and quality. We examined how changes in medical home status affect children's health care during a 2‐year period, using the eight panels from 2004 to 2012. Each panel has data in two consecutive years for all respondents in the panel. Children younger than 18 in the first year of a panel were included in the analysis sample.

Dependent Variables

Our dependent variables were health care utilizations, expenditures, and parent‐reported rating of children's health care quality. MEPS data on health care utilization are based on health care encounters in physician offices, emergency rooms (ERs), hospitals, and other settings. In our study, we examined utilization by type of service, specifically by the annual numbers of doctor visits, ER visits, and hospital discharges. In the MEPS files, expenditures “refer to what is paid for health care services” (Agency for Healthcare Research and Quality 2003). More specifically, expenditures in MEPS are defined as the sum of direct payments for care provided during the year, including out‐of‐pocket payments and payments by private insurance, Medicaid, Medicare, and other sources. Payments for over‐the‐counter drugs are not included in MEPS total expenditures. Indirect payments not related to specific medical events, such as Medicaid Disproportionate Share and Medicare Direct Medical Education subsidies, are also not included (Agency for Healthcare Research and Quality 2003). Our study used total annual expenditures and three specific types of annual expenditures: inpatient care, ER visits, and doctor office visits, all of which are readily available from the MEPS. We converted expenditures to 2012 U.S. dollars using 2004–2012 medical consumer price index information from the U.S. Bureau of Labor Statistics.

The MEPS collected information about health care quality using Consumer Assessment of Healthcare Providers and Systems (CAHPS), an AHRQ‐sponsored family of survey instruments designed to measure quality of care from the consumer's perspective (Agency for Healthcare Research and Quality 2003). In our study, we based the dependent variable of children's health care quality on one CAHPS survey item, which asked parents to rate their children's health care from 1 (worst) to 10 (best). Our dependent variables did not include other CAHPS measures because the measures were based on 22 questions to define the medical home.

Independent Variables

Prior studies have confirmed the feasibility of using MEPS data to define medical home status (Bethell, Read, and Brockwood 2004; Romaire and Bell 2010). In our study, we determined a child's medical home status by using the procedure described by Romaire and Bell (2010), who used 22 MEPS questions, including asking whether a child has a usual source of care, and many CAHPS survey items (e.g., “In the last 12 months, when you needed care right away how often did you get care as soon as you thought you needed?” and “In the last 12 months, how often did doctors or other health providers explain things in a way that was easy to understand?”). To qualify as having a medical home, a child must have a usual source of care and score no less than 75 points in four medical home domains (accessible care, comprehensive care, family‐centered care, and compassionate care) or qualify as a legitimate skip. The scoring details in each domain are described in Romaire and Bell (2010). If a child does not have a usual source of care or if one or more domain scores are less than 75 points, the child is deemed as not having a medical home. A child is deemed to have a missing medical home status if there are missing data in the survey items, which cannot qualify as a legitimate skip.

The MEPS dataset included 37,835 children younger than 18 years. Of these, 9,153 (24 percent) had a nonmissing medical home status in the 2‐year period, including 2,979 with a medical home in both years 1 and 2; 1,479 without a medical home in year 1 but who gained a medical home in year 2; 1,330 with a medical home in year 1 but who lost the home in year 2; and 3,365 without a medical home in either year.

Based on Andersen's model of health care–seeking behavior (Andersen 1995), which has been used by prior research on medical homes and children's health care (Diedhiou et al. 2010), we classified the independent variables into three categories: (1) predisposing characteristics (age, gender, and race/ethnicity), (2) enabling factors (family income, insurance, and urban residence), and (3) need factors (parent‐perceived physical and mental health statuses on whether their child has a special health care need). While most of the variable construction was straightforward (e.g., a dichotomous variable of gender), it is worth noting that the MEPS first started to identify children with special health care needs in 2000 using an instrument called the CSHCN Screener, which was based on the Maternal and Child Health Bureau CSHCN definition (Bethell et al. 2002). The MEPS also provided detailed information about health insurance status. In our study, we used MEPS variables that summarized a person's health insurance coverage for 1 year, specifically focusing on the following insurance statuses: covered by private insurance, covered by public insurance (Medicaid, CHIP, or Medicare), uninsured, and unknown insurance status. We did not distinguish among types of public insurance because (a) the MEPS did not distinguish between children enrolled in Medicaid and those enrolled in CHIP and (b) few children were covered by Medicare. Our analyses included seven dummy indicators for survey panels to adjust for any secular trends or for between‐panel differences unrelated to the research question. Table 1 summarizes both variable construction and descriptive statistics of all covariates.

Table 1.

Baseline Sample Characteristics (Mean and SD or Percentage) by Arms

	“0‐1” Arm (n = 1,479)a	“0‐0” Arm (n = 3,365)b		“1‐0” Arm (n = 1,330)a	“1‐1” Arm (n = 2,979)b
		Unadjusted	PS‐Weighted		Unadjusted	PS‐Weighted
Age	7.7 (4.9)	7.6 (4.9)	7.8 (5.0)	7.2 (4.9)	7.3 (5.0)	7.3 (5.1)
Sex: female	47.9	49.7	47.2	48.9	48.5	49.7
Race/ethnicity
Hispanic	15.3	24.6***	17.2	16.4	13.4**	17.9
Non‐Hispanic black	15.4	14.1***	15.5	12.1	12.2**	12.3
Non‐Hispanic Asian	1.9	4.1***	2.1	4.9	3.4**	4.8
Non‐Hispanic other	67.4	57.2***	65.2	66.6	71.0**	65.0
Poverty level
Poor	19.6	23.1***	19.5	16.4	13.0*	17.0
Near poor	5.6	6.6***	5.9	5.5	4.3*	5.8
Low income	13.8	16.6***	14.9	11.9	13.1*	13.1
Middle income	32.1	30.9***	31.6	32.8	33.8*	33.2
High income	29.0	22.7***	28.1	33.4	35.9*	31.0
Residence: urban	85.6	83.1	85.1	83.2	89.6***	84.3
Insurance status
Private	60.9	49.3***	59.6	66.9	71.9***	64.1
Medicare/Medicaid	30.8	38.7***	31.6	25.9	21.4***	28.8
Uninsured	5.6	6.2***	5.2	5.8	4.2***	5.6
Unknown	2.7	5.8***	3.6	1.5	2.4***	1.6
Special health care need
Yes	33.4	31.1	31.6	25.6	25.5	25.7
No	65.4	68.2	67.3	73.7	73.9	73.4
Unknown	1.3	0.8	1.1	0.6	0.6	0.9
Perceived health
Excellent	49.6	43.3**	51.0	50.5	57.8***	51.4
Very good	29.8	31.7**	28.9	29.4	28.3***	30.4
Good	16.8	19.6**	16.4	16.9	12.0***	15.3
Fair	3.4	4.4**	3.2	3.0	1.7***	2.7
Poor	0.5	1.0**	0.5	0.3	0.2***	0.2
Perceived mental health
Excellent	56.8	50.5*	58.6	57.5	63.8***	57.8
Very good	25.1	28.4*	24.6	25.4	23.9***	25.2
Good	13.9	17.2*	13.8	13.8	11.0***	13.8
Fair	3.1	3.2*	2.5	2.7	1.1***	2.9
Poor	1.0	0.8*	0.4	0.3	0.3***	0.4

^aIn estimating the average effect of the treated, the propensity score weighting does not change the sample mean of the treatment arm.

^bThe unadjusted column is the regular sample means or proportions of survey data, and the PS‐weighted column is weighted mean or proportion where each subject is weighted by propensity score/(1 − propensity score). Differences between the control and treatment arms are tested by design‐based t‐tests for means and chi‐square tests for categorical variables.

Significance levels are denoted by ***p < .001, **p < .01, and *p < .05.

Statistical Methods

We used the difference‐in‐differences (DID) approach. A DID analysis usually has two arms (control and treatment) and two stages (pre and post). Subjects in the control arm do not change their treatment status. Subjects in the treatment arm share the same treatment status as the control arm in the prestage but change to a different treatment status in the poststage. The DID approach can remove the biases due to inherent differences between the two arms at the prestage, as well as avoid a secular trend within each arm between the two stages unrelated to the treatment effect (Ashenfelter and Card 1984; Meyer 1995).

In our data, many subjects’ medical home status changed in the 2 years of a panel. Since gaining and losing medical homes are two different processes that may not share the same effects on health care, we conducted two sets of DID analyses. One set compared the children without a medical home in both years (the “0‐0” arm) with those without a medical home in year 1 but who gained a home in year 2 (the “0‐1” arm). The other set compared the children with a medical home in both years (the “1‐1” arm) with those who had a medical home in year 1 but lost a home in year 2 (the “1‐0” arm). We estimated the average treatment effect on the treated. To differentiate the two sets of comparisons, we denoted the effects as the average treatment effect on those children gaining a medical home (ATGT, between the “0‐0” and “0‐1” arms) and the average treatment effect on those children losing a medical home (ATLT, between the “1‐0” and “1‐1” arms), respectively. By definition, for the same medical home effect, the ATGT and ATLT are in opposite directions. However, we cannot safely assume that they are equal in magnitude; therefore, we estimated these two effects separately.

Traditionally, DID analyses are implemented by a linear model, such as regression or ANCOVA (Ashenfelter and Card 1984). This classic approach is still subject to biases due to correlations between the prestage characteristics and the secular trends (Heckman, Ichimura, and Todd 1997; Abadie 2005). We used the semiparametric DID method based on inverse propensity score weighting (IPW), which can mitigate the above biases (Abadie 2005). We also applied a doubly robust (DR) estimator (Lunceford and Davidian 2004), which is a variant of the IPW estimator but offers additional robustness to misspecification of the model estimating the propensity score. We briefly review the two methods below and refer the readers to the references above for more technical details.

In a DID analysis with two repeated measures on each subject, let π _i, i = 1…n, denote the propensity score of the ith subject, that is, the conditional probability that the ith subject is in the treatment arm given the observed baseline covariates. Let T _i be the binary indicator for the treatment status of the ith subject, with T _i = 1 for treatment and 0 for control. Then, the propensity score is π _i = P(T _i = 1|X _i), where X _i is the observed baseline characteristics of subject i. Let Y _i,1 and Y _i,2 denote the two repeated outcome observations for subject i, and ΔY _i = Y _i,2 − Y _i,1. Let ρ = P(T _i = 1), that is, the marginal probability of a subject receiving the treatment status. The IPW estimator for average treatment effect on the treated is given by the following:

$${\widehat{\mathrm{ATT}}}_{\mathrm{IPW}}=\frac{{\sum }_i{T}_i\mathrm{\Delta }{Y}_i}{{\sum }_i{T}_i{w}_i}-\frac{{\sum }_i{w}_i(1-T_i)\mathrm{\Delta }{Y}_i}{{\sum }_i{w}_i(1-T_i)},\text{where}{w}_i=\left\{\begin{array}{cc}\frac{(1-\mathit{\rho }){\mathit{\pi }}_i}{\mathit{\rho }(1-{\mathit{\pi }}_i)},& T_i=0\\ 1,& T_i=1\end{array}\right.$$ (1)

The DR estimator further employs a classic DID model on the outcome with the following mean function:

$$E(\mathrm{\Delta }{Y}_i)=X_i\mathit{\beta }+\mathit{\lambda }{T}_i$$ (2)

Let $m_i(1)=X_i\widehat{\mathit{\beta }}+\widehat{\mathit{\lambda }}$ and $m_i(0)=X_i\widehat{\mathit{\beta }}$, where $\widehat{\mathit{\beta }}$ and $\widehat{\mathit{\lambda }}$ are the regular ordinary least squares (OLS) estimators. Instead of the classic DID effect λ, the DR estimator is given by (Lunceford and Davidian 2004):

$$\begin{array}{cc}\hfill {\widehat{\mathrm{ATT}}}_{\mathrm{DR}}& =\frac{1}{n}\sum {\mathit{\pi }}_i^{-1}\left[T_i\mathrm{\Delta }{Y}_i-(T_i-{\mathit{\pi }}_i)m_i(1)\right]\hfill \\ & -\frac{1}{n}\sum {(1-{\mathit{\pi }}_i)}^{-1}\left[(1-T_i)\mathrm{\Delta }{Y}_i-({\mathit{\pi }}_i-T_i)m_i(0)\right]\hfill \end{array}$$ (3)

Before applying the IPW and DR estimators, a logistic regression needs to be fitted to estimate all propensity scores π _i. Next, substitute the estimated propensity scores in (1) and (3) to calculate the estimates. Both the IPW and DR estimators have closed forms as weighted averages; hence, they are easy to implement. The accurate standard errors of these two estimators are complicated, and simple and conservative standard errors are often applied. See Lunceford and Davidian (2004) for more details. We used the conservative simple standard errors in our study.

Survey weights were adjusted in all statistical analyses. We followed the approach by Pfeffermann and Landsman (2011) to adjusting survey weights in IPW and DR. As suggested by Potter (1990) and Yu (1994), we truncated extremely small and large weights at the 5th and 95th percentiles of the analysis sample, respectively. We also evaluated the design effects of the MEPS (i.e., inflation in variance estimates) when accounting for survey design in the final ATGT and ATLT estimates. We found a mild design effect varying between 0.9 and 2.5. To avoid false significance in DID analyses for the health care outcomes, we controlled the false discovery rate at 5 percent using the Benjamini–Hochberg step‐up method to adjust p‐values for multiple comparisons (Benjamini and Yekutieli 2001).

Sensitivity Analysis

We conducted a series of sensitivity analyses. Fu, Dow, and Liu (2007) suggested that DID estimates may be sensitive to the choice of estimation methods. We compared our estimates with two classic DID models: the OLS estimator and the generalized linear model (GLM) approach. The OLS estimator is an ANCOVA model taking the same form as equation (2). To account for heteroskedasticity in cost and utilization outcomes, we used the robust standard error instead of the regular OLS standard error. In the GLM approach, we used negative binomial regression for utilization variables to account for variance inflation, a log linear model for cost outcomes, and a logistic binomial model for quality ratings. We converted the GLM estimates to additive effects using recycled prediction. Both the OLS and GLM models included variables for medical home status, time, other covariates, and interactions between medical home status and time, whose coefficients represent the DID effects.

The second sensitivity check was on the influence of outliers. We repeated the analyses above for truncated outcomes, except for the ratings of health care quality, which is a bounded measure. Except for ER visits and hospital discharges, we truncated the outcomes at their 95th percentiles of nonzero values. We dichotomized ER visits and hospital discharges (no use vs. any use) because many children had no ER visits and most had no hospital discharges in a 2‐year period.

The third sensitivity check accounted for possible common‐method bias (Podsakoff et al. 2003). While we collected measures of utilization and expenditures through the MEPS Household Component and the MEPS Medical Provider Component, both the ratings of a child's health care quality and a child's medical home status relied solely on parents’ responses to the MEPS Household Component questionnaires, thus subjecting the outcomes to a possible common‐method bias. To strengthen the sensitivity analysis, we considered four variables related to parents’ attitudes toward health insurance and health care in the MEPS survey: (1) whether health insurance is needed, (2) whether health insurance is worth its cost, (3) whether the reporting parent is more likely to take risks than the average person, and (4) whether the reporting parent's child can overcome an illness without the help of a medically trained person. These attitude variables may serve as markers to reduce the common‐method biases. Propensity score–based estimates were redone by including these four marker variables.

The fourth sensitivity check was on the role of missing medical home statuses. Our main analysis was based on children with a medical home status over 2 years. If one of the 22 questions defining medical home status was answered as “don't know,” the medical home status for that year would be set as missing. This led to roughly three quarters of the children in the pooled MEPS data having a missing medical home status in at least 1 year. To examine the impact of a missing medical home status on the analysis results, we adopted the imputation rule in Romaire and Bell (2010), who imputed the missing survey response in the MEPS data as a “No” to questions on the existence of an event and as a “No difficulty” to questions on the experiences. Their rationale was that parents were more likely to recall a negative experience and the existence of an event than a positive experience and no event. Hence, the imputed values were the answers that were harder to recall. We applied the IPW estimator to the imputed dataset for a sensitivity check.

Results

Table 1 summarizes the comparisons of the baseline characteristics between the treatment and control arms. Within each pair of treatment and control arms, there were significant differences in all aspects except for age, gender, and need for special health care. Compared with the “0‐1” arm, the “0‐0” arm was significantly different in the following aspects: race/ethnicity distribution (more Hispanic and fewer non‐Hispanic other), level of poverty (higher), insurance status (fewer children with private insurance and more with Medicare/Medicaid), and perceived general and mental health status (worse). Compared with the “1‐0” arm, the “1‐1” arm was significantly different in race/ethnicity distribution (fewer Hispanic and more non‐Hispanic other), poverty level (lower), urban residence (more), insurance status (more privately insured and fewer with Medicare/Medicaid), and perceived general and mental health status (better). This imbalance in observable covariates, a common phenomenon in nonexperimental studies lacking randomization, would jeopardize the validity of the classic DID approach if the year 1 characteristics were related to the changes in year 2. However, after our propensity score weighting, there was no significant difference, suggesting that the propensity score adjustment had balanced the observed characteristics.

Table 2 presents the sample descriptive statistics for the study outcomes at year 1 and the raw change in year 2 by study arms without propensity score adjustments. Quality ratings were significantly different at year 1 within each set of comparisons. Furthermore, all study arms had at least one outcome with a significant change in year 2. Together, the results of significant between‐arm differences in year 1 and significant within‐arm changes across years 1 and 2 suggest that secular trends existed in outcomes unrelated to the changes in medical home statuses. The DID approach is particularly suitable for mitigating the bias related to such trends.

Table 2.

Unadjusted Means and Standard Errors of All Outcomes by Arms and Years

Outcomes	“0‐1” Arm (n = 1,479)		“0‐0” Arm (n = 3,365)		“1‐0” Arm (n = 1,330)		“1‐1” Arm (n = 2,979)
	Year 1	Changes in year 2a	Year 1b	Changes in year 2a	Year 1	Changes in year 2a	Year 1b	Changes in year 2a
Utilization
No. of doctor visits	5.0 (0.3)	−0.3 (0.2)	4.7 (0.1)	−0.4 (0.1)***	4.5 (0.2)	−0.2 (0.2)	4.4 (0.1)	−0.3 (0.1)**
No. of ER visits (per 100 children)	26 (2)	−4 (2)	25 (1)	−4 (1)**	19 (1)	2 (2)	20 (1)	−4 (1)***
No. of hospital discharges (per 100 children)	5 (1)	−2 (1)**	5 (1)	−2 (1)*	6 (1)	−3 (1)**	5 (1)	−2 (1)***
Expenditures
Total	2,743 (164)	−367 (202)	2,587 (127)	−670 (151)***	2,697 (344)	−605 (332)	2,634 (158)	−654 (136)***
ER	147 (15)	4 (22)	143 (11)	−45 (14)***	124 (16)	14 (22)	144 (23)	−48 (24)*
Inpatient	539 (108)	−152 (162)	482 (115)	−227 (122)	778 (233)	−466 (247)	498 (101)	−261 (108)*
Doctor visits	754 (45)	−134 (49)**	671 (27)	−80 (32)*	707 (27)	−102 (40)*	670 (40)	−121 (28)***
Ratings of health care quality	8.9 (0.04)	0.4 (0.04)***	8.5 (0.03)***	0.0 (0.00)	9.3 (0.03)	−0.3 (0.04)***	9.4 (0.02)***	0.0 (0.02)

^aWithin each arm, changes in year 2 were tested against 0, that is, no changes.

^bOutcomes in year 1 were compared within two pairs (“0‐0” vs. “0‐1” and “1‐1” vs. “1‐0”).

Significance levels are denoted by ***p < .001, **p < .01, and *p < .05.

Table 3 summarizes the ATGT results of the first set of DID analysis, comparing the “0‐0” and “0‐1” arms. There were no significant effects on cost or utilization outcomes, but the ratings of health care quality were significantly improved by roughly 0.3 points (95 percent CI: 0.2–0.4, p < .0001, based on the IPW method). The results were consistent between the IPW and DR methods.

Table 3.

ATGT Estimates and Standard Errors (Comparisons between the “0‐0” and “0‐1” Arms)

Outcomes	Estimation Methods
	IPW	DR	OLS	GLM
Utilization
No. of doctor visits	0.2 (0.2)	0.2 (0.2)	0.1 (0.3)	−0.1 (0.3)
No. of ER visits (per 100 children)	1.0 (2.4)	−0.2 (2.4)	0.3 (3.0)	0.4 (3.3)
No. of hospital discharges (per 100 children)	−1.0 (1.0)	−0.5 (0.8)	0.0 (1.1)	−0.2 (1.3)
Expenditures
Total	262 (234)	314 (233)	303 (292)	−185 (151)
ER	54 (28)	41 (27)	52 (29)	9 (13)
Inpatient	−90 (191)	−18 (190)	75 (218)	8 (30)
Doctor visits	3 (57)	23 (56)	−53 (67)	−20 (50)
Ratings of health care quality	0.32 (0.05)†	0.34 (0.05)†	0.33 (0.06)†	0.41 (0.06)†

p‐values were adjusted using the Benjamini–Hochberg step‐up procedure to account for multiple comparison. The false discovery rate level was controlled at 0.05.

DR, doubly robust method; GLM, generalized linear models fitted by generalized estimation equation; IPW, inverse propensity score weighting method; OLS, classic ordinary least squares regression and robust standard error.

Significant results are denoted by ^† after multiple‐comparison adjustment.

The results of sensitivity analyses were similar to those in the main analyses. Specifically, the OLS estimates were also similar to the ones based on the main causal inference models, and estimates from the GLM approach were slightly higher. All methods had the same statistically significant findings for the ratings of health care quality.

Table 4 compares the estimated effects of medical home status in the second set of DID analysis between the “1‐0” and “1‐1” arms. It shows that medical home status had significant effects on the utilization of emergency rooms and on the ratings of health care quality. According to the IPW estimates, for every 100 children there would be 7.6 fewer ER visits (95 percent confidence interval: 2.8–13.8) had they not lost their medical status in year 2. The rating of health care quality dropped by 0.3 points (95 percent CI: 0.2–0.4, p < .0001) on average.

Table 4.

ATLT Estimates and Standard Errors (Comparisons between the “1‐0” and “1‐1” Arms)

Outcomes	Estimation Methods
	IPW	DR	OLS	GLM
Utilization
No. of doctor visits	0.1 (0.2)	0.0 (0.2)	0.0 (0.3)	0.2 (0.2)
No. of ER visits (per 100 children)	7.6 (2.8)†	6.3 (2.7)†	6.1 (2.8)†	7.7 (3.8)†
No. of hospital discharges (per 100 children)	−0.2 (1.3)	0.0 (1.3)	−0.4 (1.3)	0.7 (1.8)
Expenditures
Total	247 (323)	8 (301)	48 (508)	177 (191)
ER	92 (42)†	88 (41)†	61 (31)	40 (20)
Inpatient	−13 (253)	−183 (238)	−204 (374)	−10 (31)
Doctor visits	−37 (54)	−39 (53)	19 (57)	23 (56)
Ratings of health care quality	−0.31 (0.05)†	−0.30 (0.05)†	−0.31 (0.05)†	−0.38 (0.06)†

All abbreviations and effect scales are the same as in Table 3.

Significant results are denoted by ^† after multiple‐comparison adjustment.

All analyses were repeated on the truncated outcomes to check the robustness of the results to outliers. This sensitivity check yielded similar results as in Tables 3 and 4, although the estimated effect sizes are smaller due to truncation in the outcome. Three methods, GLM, IPW, and DR, found significant ATLT effects on ER utilization. The OLS method found significant ATLT effects on ER expenditures only, but not on ER utilization. Including marker variables for parent attitudes (to account for possible common‐method bias) in our propensity score–based approaches did not lead to notable differences in estimates. Using the IPW estimator on the imputed dataset, we found almost identical results for health care ratings as in Tables 3 and 4. Using ATLT, the effect on ER utilization was still significant but smaller, and the effect on ER expenditure was not significant.

Conclusion and Discussion

Using rigorous causal inference methods and a unique dataset with two types of DID estimates, this study found two significant effects of parent‐reported medical home status for their children. First, both the ATGT and ATLT effects suggest that having a medical home may lead to higher perceived quality of care for children. Second, the ATLT effects indicate that losing medical home status may cause children to have more ER visits (seven more visits per year per 100 children), compared with children who had medical homes in both years. Similar results have been reported in prior research on medical homes. For example, one study that analyzed observation data found that quality was improved and ER visits declined for children with asthma (six fewer visits per year per 100 patients) who were enrolled in a Medicaid medical home program (Domino et al. 2009). One cross‐sectional analysis of the MEPS data found that having a medical home reduces ER visits by roughly 13 percent (reported as an incidence rate ratio in table 3 of Romaire, Bell, and Grossman 2012a). Two studies that used national survey data found that having a medical home was associated with a lower likelihood of ER visits among children (Diedhiou et al. 2010; Long et al. 2011), although the two studies did not report the exact magnitude of this association (i.e., change in number of ER visits).

Our study did not find significant results of total expenditures, a finding that is consistent with the literature on medical homes defined as interventions. For example, one recent review assessed 19 studies of medical homes and found that, with the exception of one subgroup analysis, none of the reviewed studies reported statistically significant cost savings among patients with medical homes (Jackson et al. 2013). The subgroup analysis indicated that cost savings may occur among patients who have medical homes for more than 1 year (Maeng et al. 2012). In comparison, our study evaluated changes in medical home status within a relatively short period of 2 years, which may explain the insignificant result of total expenditures.

Our study was strengthened by using nationally representative data, applying causal inference methods, and distinguishing the effects of gaining and losing medical homes. It was further bolstered by a series of sensitivity checks. For example, we found similar results between the complete‐case analysis and the imputed dataset analysis (which accommodated missing data). Unlike Romaire and Bell (2010), who used the imputed data analysis as the main results and the complete‐case analysis as a sensitivity check, we used the complete‐case analysis as our main results and the imputed data analysis as a sensitivity check.

Despite its strengths, our study had several limitations. First, our measure of medical home status relied on parent‐reported survey information, which is subject to recall bias. Because of such a reliance, our study also differed from medical home studies that are based on practice models, which typically use facility‐based measures (e.g., to identify whether a doctor's office qualifies as a medical home) and from medical home studies of interventions on primary care practices. Missing information in the self‐reported data also reduced the sample size and may have led to selection biases. However, our sensitivity analysis using imputed data showed similar results to the complete dataset. Second, the medical home status was defined as a dichotomous variable in this study using a cutoff from prior research (Romaire and Bell 2010). While the dichotomous variable simplifies comparison, it may miss important detail in the extent of “medical homeness.” For example, an increase from 25 to 70 in the score for measuring a medical home from years 1 to 2 means that the child was still considered as not having a medical home in year 2 (because the score was less than 75)—despite the considerable increase in medical home score over the 2‐year period. A continuous variable to measure medical home status might be able to capture the improvement in health care, but it remains a topic for future methodological studies to determine how to construct unbiased estimates for continuous treatment variables in DID analyses. Third, like prior research (Romaire and Bell 2010), the binary definition of medical home status allowed us to examine the effects of gaining or losing medical homes, but it did not allow us to examine the relationships between specific questions used in the medical home definition and specific health care outcomes. Such relationships remain an important topic for future studies as recently noted by some experts (Shi et al. 2015). Fourth, while our analysis of the effects of losing a medical home indicated significant results in ER visits and expenditures, it is not clear why children lost their medical homes, an issue that has not been examined by the published studies. Fifth, future studies are needed to examine why ER visits increase for children who lose their medical homes. Sixth, our sample size did not allow us to focus on specific groups of children, such as children with a specific special health care need or in a specific age group. For example, our study sample included 113 children who were 17 years old in year 1 and 18 years old in year 2. Given the challenges of transition to adulthood, future studies should focus on the effect of medical homes on this specific age group. Seventh, the causal inference methods we applied require a number of technical assumptions. This study satisfied the testable assumptions to establish a causal relationship, including the longitudinal difference‐in‐differences design and balancing observable confounders by propensity score weighting. We also conducted extensive sensitivity checks to examine the robustness of the empirical results to the choice of estimators. However, all causal inference methods require untestable assumptions to establish causality. The most important but untestable one is perhaps the unconfoundedness of potential outcomes given observed covariates. Despite our extensive effort, our estimates were still based on an observation study, not a controlled experiment. Caution must be taken in interpreting the potential causal link between self‐reported medical home status and the resulting outcomes. See Abadie (2005) for a careful discussion of technical assumptions used by the causal difference‐in‐differences estimators.

However, the superiority of causal inference methods (specifically, robustness and unbiasedness) to classic regression estimators and of difference‐in‐differences design to cross‐sectional design grounded in statistical theories. In brief, due to the lack of randomization, the classic regression estimator requires a correctly specified functional form of the regression equation, but the causal inference method used in this study only requires a rich set of covariates but no parametric form in the mean function. Similarly, cross‐sectional analysis requires that no unobservable or omitted variable associated with the main predictor be allowed to affect the outcome (i.e., observables are sufficient), whereas the DID analysis acknowledges a baseline difference not explained by any observables. That being said, when the extra assumptions required by classic regression and cross‐sectional analyses are satisfied, those designs are more efficient than the causal inference method and DID analysis, respectively.

The study findings have potential policy implications. First, our findings suggest that we need policies to help children avoid losing their medical homes. Since the reasons for such loss are unclear, future studies may help identify opportunities for retaining or gaining medical home status and target policy interventions toward them. At a minimum, it would be advisable to take steps to stabilize children's health coverage (reducing the likelihood of financially driven interruptions in their access to medical homes) and to strengthen pediatric practices in which large shares of their patients have medical homes. Second, the finding that quality improvements are concurrent with gaining a parent‐reported medical home suggests that policies that encourage children to have medical homes (through either patient‐facing strategies such as value‐based insurance design or provider‐facing strategies such as incentives to provide medical home services) may be valuable. However, discrepancies exist in the definition of a medical home, depending on individual‐level and practice‐level assessments of this status (Long and Garg 2015). Finding the optimal mix of patient‐ and provider‐facing strategies is a high priority for our health system and a fertile area for future research.

Supporting information

Appendix SA1: Author Matrix.