Impact of a mental health based primary care program on emergency department visits and inpatient stays

Abstract

Objective

Integrating primary care services into specialty mental health clinics has been proposed as a method for improving health care utilization for medical conditions by adults with serious mental illness. This paper examines the impact of a mental health based primary care program on emergency department (ED) visits and hospitalizations.

Method

The program was implemented in seven New York City outpatient mental health clinics in two waves. Medicaid claims were used to identify patients treated in intervention clinics and a control group of patients treated in otherwise similar clinics in New York City. Impacts of the program were estimated using propensity score adjusted difference-in-differences models on a longitudinally followed cohort.

Results

Hospital stays for medical conditions increased significantly in intervention clinics relative to control clinics in both waves (ORs=1.21 (Wave 1) and 1.33 (Wave 2)). ED visits for behavioral health conditions decreased significantly relative to controls in Wave 1 (OR=0.89), but not in Wave 2. No other significant differences in utilization trends between the intervention and control clinics were found.

Conclusion

Introducing primary care services into mental health clinics may increase utilization of inpatient services, perhaps due to newly identified unmet medical need in this population.

1. INTRODUCTION

Serious mental illness (SMI) is associated with a reduction in life expectancy of about 8 years[1], relative to the general population, with the excess mortality driven primarily by physical health conditions, such as cardiovascular disease and cancer[2–4]. The causes of poor health in this population are complex. They include common social determinants of health, such as poverty, poor health behaviors, such as smoking, disparities in medical care[5, 6], and adverse side-effects of medications used to treat mental illnesses[7]. Treatment of physical health conditions among adults with serious mental illness has historically been a challenge for the health care system due to fragmentation between the specialty mental health sector, where mental illnesses are treated, and general medical care, where physical health conditions are treated. Fragmentation, it is thought, constitutes a barrier to preventive care and management of physical health conditions, and contributes to inappropriate utilization of health care services, including high use of emergency department care and inpatient stays. Mental illness is associated with more frequent emergency department visits [8, 9], high risk for ambulatory care sensitive hospital admissions [10–13] and re-hospitalization [14] for physical health conditions.

One strategy for improving care for physical health conditions among adults with serious mental illness is to reduce fragmentation by integrating care for physical health conditions into specialty mental health clinics[15]. This model of integrated care has several potential advantages. First, it locates physical health care within the clinics that already serve as the primary point of contact with the health care system for adults with serious mental illness, greatly reducing the burden of obtaining care. Patients may be more comfortable receiving care in these settings, given the high rates of discrimination they report in general medical settings [16] . Second, co-location of physical and mental health care has the potential to improve integration between treatments for diverse medical conditions. Providers in specialty mental health clinics are likely to have broader understanding of their patients’ lives that they can bring to bear on providing care that addresses the full scope of their needs. However, there is also evidence that simple co-location, without policies to actively promote integration, do not positively affect care[17]. Mental health based primary care services have been supported by the Substance Abuse and Mental Health Services Administration through the Primary Behavioral Health Care Integration (PBHCI) program, which has provided grants to over 150 mental health clinics since 2009 [18–20]. PBHCI grantee clinics are funded to provide screening and monitoring of common chronic physical health conditions along with wellness service, such as smoking cessation or physical activity groups, to their patients.

Despite the potential advantages and the ongoing policy efforts, the potential impact of providing access to primary care services in specialty mental health clinics on utilization of emergency departments and inpatient services by adults with SMI is unclear. On the one hand, the expectation is that engagement in primary care will reduce emergency room visits, and have a preventive impact on serious medical events that require hospitalization. Given that these events reflect poor health outcomes for patients and are costly to the health care system, reducing their frequency is a highly desirable outcome. On the other hand, if patients have not been receiving adequate primary care services, they are likely to have unmet needs for medical care. If these patients gain access to primary care and have their needs identified, their utilization of intensive medical services may increase, rather than decrease. Providing needed care to an underserved population is also a desirable outcome.

In fact, evidence to date on the impact of improving access to primary care on use of emergency departments and inpatient stays is mixed. A primary care based medical home program was found to reduce emergency department visits in North Carolina[21]. However, a Medicaid experiment in Oregon found that increasing insurance coverage resulted in an increase in emergency department visits [22]. In an RCT that tested integration of primary care services into a specialty mental health clinic for patients with SMI and a comorbid physical condition, the integrated care program did not impact either emergency room visits or inpatient stays, although the follow-up period for that study was only 12 months [23]. A study of a PBHCI clinics in Oregon found that the integrated care program reduced inpatient stays but did not impact emergency department visits [24]. No studies have had examined the impact of this model using claims data, which include information on care utilization of ED and inpatient services regardless of where those services occurred.

This study examines the impact of PBHCI, an intensive, grant-funded mental health based primary care program on utilization of emergency departments and inpatient services in the state of New York, where 7 PBHCI programs were implemented in two waves. The study adds to the literature by examining this model of care using Medicaid claims data, which capture the vast majority of care received by Medicaid enrollees. It also provides an example of the effects that the program can have on when implemented in a range of ‘real world’ clinical settings.

2. METHODS

2.1. Data Source

Data come from a Medicaid claims data warehouse maintained by the New York State Office of Mental Health (OMH). The database includes all Medicaid enrolled individuals who received a behavioral health service in the past five years, where behavioral health service is defined broadly to include 1) visits that occurred in a behavioral health clinic setting, 2) visits in any setting with a psychiatric diagnosis, or 3) prescriptions for a psychiatric drug. For these individuals, the database includes all Medicaid claims and managed care encounter data, including client demographic, enrollment, prescription drug and service utilization, including all general medical and behavioral health inpatient, outpatient, and emergency services. All study procedures were approved by the IRBs of the RAND Corporation and the New York State Office of Mental Health

2.2. Intervention and Control Clinics

The first two waves of PBHCI grants awarded to clinics in NYS were included in this study (later waves were not included due to limited time of service provision after receiving the grants). Four clinics received grants in 2010 and began providing services in February 2011, and another three clinics were awarded grants in 2012 and began providing services in February 2013. These PBHCI grantees are all specialty mental health clinics licensed by OMH and located in New York City. The 40 community based OMH licensed clinics located in New York City which did not have a co-license or operating certificate to provide primary care services were used as controls.

2.3. Study Period

Analyses were conducted separately for each of the two waves of PBHCI grants due to their different start dates, as shown in Figure 1. The pre-PBHCI baseline period for each wave was defined as the two years prior to initiation of PBHCI services, February 2009 through January 2011 for wave 1 and February 2011 through January 2013 for wave 2. PBHCI grants provided funds for the program for a 4-year period. The PBHCI intervention period included the period from the initiation of PBHCI services though the most recent date for which complete claims data are available, February, 2015.

Figure 1.

Intervention Timeline

2.4. Study Sample

The sample includes enrollees, age 18 through 64, who were continuously enrolled in Medicaid, and received treatment in a study clinic (either PBHCI or control), during both the baseline and intervention periods. Continuous enrollment was defined, following prior studies[25], as having at least nine months of enrollment during a year with no more than 2 continuous months without enrollment. Treatment at a PBHCI or control study clinic was defined as one or more visits at a study clinic at any time during the two year baseline period, and one or more visits during the four year (wave 1) or two year (wave two) follow-up period. Individuals with dual Medicaid-Medicare eligibility were excluded because we are unable to observe Medicare-only covered utilization. For the wave 1 analysis, the sample was comprised of 6,716 PBHCI patients and 13,039 control clinic patients. For the wave 2 analysis, the sample was comprised of 1,887 PBHCI patients and 11,542 control clinic patients.

2.5. Utilization Outcomes

2.5.1. Inpatient Stays

Data on inpatient stays and associated diagnoses were used to identify patients with 1 or more inpatient stays for a behavioral health (defined using state specific psychiatric inpatient rate codes and/or a primary diagnosis between ICD-9 290.00 to 319.00) or a medical condition (defined as any other inpatient episode that was not for a behavioral health cause) and patients with frequent inpatient stays, defined as 3 or more during a 12-month period, for behavioral health or medical conditions.

2.5.2. Emergency Department (ED) Visits

Data on ED visits and associated diagnoses were used to identify patients with 1 or more ED visits for a behavioral health (defined as a primary diagnosis between ICD-9 290 to 319) or a medical condition (defined as not having a primary diagnosis between ICD-9 290 to 319) and patients with frequent ED visits, defined as 4 or more during a 12-month period, for behavioral health or medical conditions.

2.6. Patient Characteristics

Information on patient demographic characteristics, diagnoses, service utilization and costs during the baseline period were used to adjust for differences between patients seen in PBHCI and control clinics for each wave. Demographic characteristic were age, sex, race (White, Black, Asian, Hispanic, Other/Unknown) and eligibility category (disability vs other). Mental health diagnoses were classified as SMI vs other, where SMI includes diagnostic codes for schizophrenia, other psychoses, or bipolar disorder. Physical diagnoses were categorized according to the Chronic Illness and Disability Payment System (CDPS), which classifies diagnoses by cost based on propensity to utilize care [26]. Beneficiaries were classified as CDPS High Cost Physical Diagnosis vs Other. Service utilization during the two year baseline period was characterized according to the following five indicators: 1) having 12 or more mental health clinic visits, 2) having an inpatient stay for a mental health diagnosis, 3) having an inpatient stay for a medical diagnosis, 4) any service with a diagnosis of substance use, and 5) any service with a diagnosis of developmental disability.

2.7. Statistical Analysis

Analyses were conducted in parallel for the two PBHCI waves. The waves were analyzed separately because they had different starting times, different periods of follow-up, and slightly different grant requirements. The request for proposals for wave 2 had more detailed requirements for on-site wellness programming than that for wave 1. Each wave was analyzed using a difference-in-differences (DD) approach with propensity score weighting. Propensity scores were estimated in a logistic regression model predicting treatment in a PBHCI clinic (vs comparison clinic) using information on demographic (age, sex, race/ethnicity and aid category), diagnosis (SMI and high cost CDPS category) and service use (number of clinic visits, inpatient stay for medical or psychiatric diagnosis, receipt of services for substance use or developmental disability) during the pre-PBHCI period. The scores were used to implement a doubly robust estimate of the impact of PBHCI[27]; logistic regression models were specified by wave for each of the service use outcomes using both inverse probability of treatment weights and statistical control for patient characteristics, the individual propensity score, binary indicators for the time period and treatment in a PBHCI clinic, and the statistical interaction between time period and treatment in a PBHCI clinic. Generalized estimating equations were used to account for multiple observations per person. A statistically significant interaction between time period and treatment in a PBHCI clinic is interpreted as an impact of the PBHCI program.

3. RESULTS

In both waves, there were statistically significant differences in patient demographic characteristics, diagnoses and utilization at baseline between the patients seen in PBHCI clinics and those seen in control clinics, though these differences are generally small in magnitude (Table 1). PBHCI clinic patients were significantly more likely to be male, Black or Hispanic, and to have a severe mental illness. They were more likely to have a psychotic disorder or bipolar disorder, more likely to have a mental health hospitalization, and less likely to be engaged in outpatient mental health treatment (12 or more outpatient mental health visits) during the two-year baseline period. In addition, they were significantly more likely to have an inpatient medical stay, despite having a smaller proportion of clients with a high cost general medical diagnosis (ns trend for wave 1, p=.07, wave 2 p<.0001).

Table 1.

Baseline Characteristics of PBHCI and Control group patients for Wave 1 and Wave 2

	Wave 1					Wave 2
Variable	PBHCI Group (N=6712) (n=6,712)		No IC Group (N=13012) (n=13,012)		p Value	PBHCI Group (N=1881) (n=1,881)		No IC Group (N=11514) n=11,514)		p Value
Demographic	n	%	n	%		n	%	n	%
Age (SD)	43.41	(10.94)	43.28	(11.09)	0.4513	45.93	(10.91)	44.81	(11.14)	<.0001
Male	2385	35.53%	4120	31.66%	<.0001	707	37.59%	3847	33.41%	0.0004
Race/Ethnicity					<.0001					<.0001
White	1965	29.28%	4241	32.59%		484	25.73%	3637	31.59%
Black	2272	33.85%	4214	32.69%		664	35.30%	3888	33.77%
Asian	61	0.91%	278	2.14%		26	1.38%	250	2.17%
Hispanic	1733	25.82%	2725	20.94%		410	21.80%	2357	20.47%
Other/Unknown	681	10.15%	1554	11.94%		297	15.79%	1382	12.00%
Aid Category-Disability	4902	73.03%	8903	68.42%	<.0001	1540	81.87%	8614	74.81%	<.0001
Diagnosis
Serious Mental Illness	2197	32.73%	3794	29.16%	<.0001	895	47.58%	3691	32.06%	<.0001
High Cost Physical Diagnosis	729	10.86%	1525	11.72%	0.0725	146	7.76%	1436	12.47%	<.0001
Service Use
12+ Mental Health Clinic Visits	4053	60.38%	10200	78.39%	<.0001	1436	76.34%	9402	81.66%	<.0001
Inpatient –Mental Health	1066	15.88%	1895	14.56%	0.014	335	17.81%	1767	15.35%	0.0065
Inpatient - Medical	2106	31.38%	3586	27.56%	<.0001	564	29.98%	3189	27.70%	0.0406
Substance Use Treatment	1355	20.19%	1848	14.20%	<.0001	184	9.78%	1748	15.18%	<.0001
Developmental Disability	481	7.17%	521	4.00%	<.0001	18	0.96%	421	3.66%	<.0001

The utilization outcomes were generally similar in prevalence in the two waves (Table 2). Between 44.62% and 50.40% of patients had at least one ED visit during both the pre- and post- periods, with ED visits for behavioral conditions much less common than ED visits for physical health conditions. Between 7.38% and 8.41% of patients had 4 or more ED visits within a year in either the pre- or post-intervention period. Between 16.86% and 24.53% of patients had at least one inpatient stay, with stays for medical conditions about 1.5 to 2 times as common as stays for behavioral health conditions. Between 2.18% and 3.24% of patients had 3 or more inpatient stays. Table 2 also shows the unadjusted differences in the time trends between the PBHCI and control clinics, i.e. the unadjusted difference-in-differences comparisons. The largest difference-in-differences is for inpatient stays for medical conditions in Wave 2, where there was a relative increase in PBHCI clinics of 2.10%.

Table 2.

Comparison of unadjusted trends in ED Visits and IP Stays between PBHCI and Control Clinics, Waves 1 and 2.

Utilization Measures	WAVE 1
	PBHCI Clinics (N=6,712)			Control Clinics (N=13,012)			Difference in Differences (PBHCI-Control)
	Pre (%) (10,745 PY)	Post (%) (20,378 PY)	Post-Pre	Pre (%) (20,805 PY)	Post (%) (39.033 PY)	Post-Pre
Emergency Department
Any ED Visit	50.40%	46.95%	−3.45%	46.44%	44.62%	−1.82%	−1.63%*
BH ED Visit	12.63%	10.18%	−2.45%	10.23%	8.76%	−1.47%	−0.98%*
Med ED Visit	45.51%	43.36%	−2.15%	42.51%	41.36%	−1.15%	−1.00%
4 or More ED Visits	8.37%	8.41%	0.04%	7.38%	7.39%	0.01%	0.03%
4 or More BH ED Visits	0.93%	1.19%	0.26%	0.87%	1.01%	0.14%	0.12%
4 or More Med ED Visits	7.74%	7.72%	−0.02%	6.79%	6.74%	−0.05%	0.03%
Inpatient
Any IP Stay	24.53%	21.64%	−2.89%	20.36%	17.58%	−2.78%	−0.11%
BH IP Stay	10.96%	8.56%	−2.40%	7.83%	6.43%	−1.40%	−1.00%*
Med IP Stay	15.30%	14.50%	−0.80%	13.66%	12.07%	−1.59%	0.79%*
3 or More IP Stays	3.24%	3.14%	−0.10%	2.18%	2.32%	0.14%	−0.24%
3 or More BH IP Stays	1.80%	1.63%	−0.17%	1.01%	1.07%	0.06%	−0.23%
3 or More Med IP Stays	1.54%	1.54%	0.00%	1.20%	1.29%	0.09%	−0.09%
	WAVE 2
Utilization Measures	PBHCI Clinics (N=1,881)			Control Clinics (N=11,514)			Difference in Differences (PBHCI Control)
	Pre (%) (2,507 PY)	Post (%) (2,464 PY)	Post-Pre	Pre (%) (17,524 PY)	Post (%) (17,246 PY)	Post-Pre
Emergency Department
Any ED Visit	48.50%	47.44%	−1.06%	46.85%	44.96%	−1.89%	0.83%
BH ED Visit	12.33%	9.62%	−2.71%	10.28%	8.97%	−1.31%	−1.40%
Med ED Visit	43.88%	43.83%	−0.05%	42.82%	41.47%	−1.35%	1.30%
4 or More ED Visits	7.82%	8.20%	0.38%	7.60%	7.66%	0.06%	0.32%
4 or More BH ED Visits	0.88%	0.97%	0.09%	0.91%	1.17%	0.26%	−0.17%
4 or More Med ED Visits	7.42%	7.67%	0.25%	7.03%	6.92%	−0.11%	0.36%
Inpatient
Any IP Stay	20.38%	19.24%	−1.14%	19.36%	16.86%	−2.50%	1.36%
BH IP Stay	8.78%	7.02%	−1.76%	7.59%	6.22%	−1.37%	−0.39%
Med IP Stay	12.84%	13.64%	0.80%	12.73%	11.43%	−1.30%	2.10%*
3 or More IP Stays	3.15%	3.17%	0.02%	2.28%	2.30%	0.02%	0.00%
3 or More BH IP Stays	1.28%	1.70%	0.42%	1.16%	1.11%	−0.05%	0.47%
3 or More Med IP Stays	1.91%	1.50%	−0.41%	1.16%	1.24%	0.08%	−0.49%

ED=Emergency Department; IP=Inpatient; BH=Behavioral Health; Med=Medical; PY=Person-Years. Percents represent events per person year.

^*indicates that the DiD estimate is statistically significant at the p=.05 level accounting for within-person correlations and without covariate adjustment.

Propensity scores were estimated in logistic regression models in which the baseline characteristics reported in Table 1 were specified as predictors of being a PBHCI vs. control clinic patient. Inverse probability of treatment weights applied to the sample reduced variation between groups on all characteristics to within 0.1 standard errors for both waves (results available on request), suggesting a good match between the PBHCI and the control patients.

Propensity score adjusted estimates of the impact of PBHCI on emergency room visits and hospital stays are shown in Table 3. Wave 1 of PBHCI was associated with a significant reduction in the likelihood that an individual has a visit to the ED (OR=0.94, 95% CI 0.88-0.99), and this reduction is specific to visits that have a behavioral health diagnosis (OR=0.89, 95% CI 0.81-0.98). There is no significant association between PBHCI and the likelihood of having an ED visit with a medical diagnosis. We also find no association between PBHCI and the likelihood of having frequent ED visits (4 or more). There were no associations between PBHCI and ED visits for wave 2.

Table 3.

Propensity Score adjusted difference-in-differences estimates of PBHCI impacts on ED Visits and IP Stays, wave 1 and wave 2

	Wave 1				Wave 2
Utilization Measure	Odds Ratio	95% Confidence Interval		P-value	Odds Ratio	95% Confidence Interval		P-value
Emergency Department Visits
Any ED Visit	0.94	0.88	0.99	0.021	1.06	0.94	1.20	0.318
BH ED Visit	0.89	0.81	0.98	0.020	0.87	0.71	1.07	0.194
Medical ED Visit	0.97	0.91	1.02	0.244	1.08	0.96	1.21	0.220
4 or More ED Visits	0.99	0.89	1.10	0.904	1.06	0.86	1.31	0.584
4 or More BH ED Visits	1.15	0.84	1.57	0.377	0.86	0.43	1.76	0.686
4 or More Med ED Visits	0.99	0.89	1.11	0.918	1.08	0.87	1.33	0.505
Inpatient Stays
Any IP Stay	1.09	1.01	1.18	0.034	1.23	1.00	1.50	0.049
BH IP Stay	0.90	0.79	1.02	0.087	1.04	0.75	1.42	0.824
Med IP Stay	1.21	1.10	1.32	<.0001	1.33	1.07	1.65	0.010
3 or More IP Stays	0.96	0.80	1.16	0.700	0.92	0.62	1.37	0.688
3 or More BH IP Stays	0.91	0.70	1.18	0.482	1.52	0.83	2.77	0.172
3 or More Med IP Stays	0.97	0.76	1.25	0.810	0.65	0.41	1.05	0.078

ED=Emergency Department; IP=Inpatient; BH=Behavioral Health; Med=Medical.

PBHCI was associated with an increase in the likelihood of having an inpatient stay in both wave 1 (OR=1.09, 95% CI 1.01-1.18) and wave 2 (OR=1.23, 95% CI 1.00-1.50). When stays for behavioral and general medical diagnoses are distinguished, PBHCI was associated with increased stays for general medical diagnoses but not behavioral diagnoses in both waves. The relative odds of hospitalization for a medical diagnosis in PBHCI vs control clinics is 1.21 (95% CI: 1.10-1.32) in wave 1 and 1.33 (95% CI: 1.07-1.65) in wave 2. PBHCI was not significantly associated with the likelihood of having frequent hospitalizations for either psychiatric or general medical diagnoses in either wave.

Two alternative model specifications were examined to test the robustness of the findings (findings available from authors). First, to account for clustering of patients within clinics, we estimated a series of models with a fixed effect for the clinic in which patients were treated. Several of these models failed to converge, but the statistical significance of the main findings regarding inpatient stays for medical conditions was sustained. Second, to test the dependence of the findings on the scale of measurement, odds ratio vs prevalence, we estimated a series of linear regression models corresponding to the logistic regression models reported in Table 3. Results from the linear regression models differed slightly in statistical significance for some outcomes, but did not differ for the findings regarding inpatient stays for medical conditions.

3. DISCUSSION

People with SMI have high rates of chronic physical health conditions that add to their health burden, complicate their MH treatment, and shorten their lives. The historical separation between the specialty MH and physical health systems has contributed to this health disparity because the specialty MH system is by and large the health home of this population[28]. Although acute care (ED and inpatient) utilization may be appropriate, it oftentimes signals inadequate access to high-quality physical health care. Hence, delivery system reform efforts that like PBHCI seek to integrate physical and mental health care for this population are expected to eventually lead to reductions in ED and inpatient utilization. However, the impact of integrated care on those patterns is unclear. Due to under-treatment in the status quo, integrated care might lead to increases in utilization of both outpatient and acute care in the short run. After the unmet need for physical health care has been reasonably met, there may be a drop in avoidable emergency room visits and hospital stays.

This study examined the impact of PBHCI, a grant-funded program that facilitated access to key components of primary care in mental health clinics. Our most important finding is that PBHCI was associated with more individuals using inpatient physical health care. The program was not associated with inpatient utilization for behavioral health conditions, suggesting the effect was specific to physical health, the intended target of the program. In addition, the results are consistent with an effect of PBHCI on the likelihood of having an inpatient stay and no effect of PBHCI on the likelihood of having frequent inpatient stays. Together, these findings suggest that the program addressed unmet needs for medical care without contributing to high utilization of inpatient services. Although this was not a controlled trial and thus, this study is subjected to selection bias, our use of a state of the art quasi-experimental design and the fact that both waves yielded the same finding strengthen our confidence that PBHCI was causally associated with the observed expansion in physical health inpatient utilization.

There is also some evidence of an impact of PBHCI on reducing ED utilization for behavioral health reasons, but the evidence is not as strong as for inpatient services because results were statistically significant only for Wave 1. Although our study was not designed to assess for drivers of the observed utilization, there are at least two plausible explanations for this finding. One is that regardless of their chief complaint, when people with SMI use the ED, even if solely for physical health complaints, ED clinicians may assign a larger weight to their behavioral health diagnosis and denote it as the primary diagnosis for the visit [29]. It is possible that ED visits for minor physical complaints, which are most likely to have been coded with behavioral health diagnoses, were reduced because patients had access to physical health services in the clinic where they receive mental health services. The other is that access to integrated physical health services may have beneficial effects on these patients’ mental health, perhaps through an increased focus on overall health including access to wellness services, reduced stress, increased support and care coordination, or by enabling greater use of MH services.

These findings contribute to a decidedly mixed body of evidence regarding the impact of mental health based primary care services on utilization of care. Well-designed RCTs have produced inconsistent results, with one study conducted in the VA health system reporting a reduction in ED visits for physical health conditions (but no other effects on ED or inpatient utilization)[30] and one study conducted in a community mental health center reporting no effects on either ED or inpatient utilization[31]. The one prior study to examine the impact of PBHCI found a reduction in inpatient hospitalization and no impact on ED visits[24]. One potential explanation for this mixed body of results is that the impact of introducing primary care services into mental health clinics is dependent on prior patterns of utilization which vary across clinics. Studies which can help identify the settings in which an intervention such as PBHCI is likely to have the greatest impact should be a priority for future research. The length of follow-up may also lead to differences across studies, although the study by Krupski et al (2016) found PBHCI associated with lower likelihood of inpatient hospitalization with a follow up period similar in length to this study. That study did not differentiate between hospitalizations for medical vs. behavioral health conditions, limiting direct comparability of the findings.

A strength of this study is the use of a differences-in-differences approach to analyze data gathered from a cohort of patients seen both before and during PBHCI implementation in both the PBHCI and control clinics. That our data is on the same group of individuals for both time periods adds to the internal validity of the design but it also introduces a potential limitation with respect to generalizability, due to the fact that we do not capture the effects of turnover in patient caseloads. Patients who are seen within the same clinic during both a baseline and follow-up period, which is a condition of inclusion in the sample analyzed here, may represent a more stable treatment population compared to new clinic clients, or those who presented for treatment but did not remain engaged. Arguably, clients who are engaged in mental health treatment are the prime target of programs like PBHCI that are located within these clinics. Further work is needed to examine the impact of PBHCI on new patients with SMI, or those who are more loosely connected to care.

A potential limitation of the study is our lack of ability to examine heterogeneity of effects across the clinics. It is likely that implementation of the PBHCI program varied across the clinics based on specific circumstances or decisions made by clinic directors, and these differences may have moderated the impact of the program. Without a larger sample of clinics and more detailed implementation information, examination of heterogeneity of effects is not possible.