The effect of CMS’ Inpatient Psychiatric Facility Quality Reporting program on the use of restraint and seclusion

Abstract

Background:

Patients in inpatient psychiatry settings are uniquely vulnerable to harm. As sources of harm, research and policy efforts have specifically focused on minimizing and eliminating restraint and seclusion. The Centers for Medicare and Medicaid’s Inpatient Psychiatric Facility Quality Reporting (IPFQR) program attempts to systematically measure and reduce restraint and seclusion. We evaluated facilities’ response to the IFPQR program and differences by ownership, hypothesizing that facilities reporting these measures for the first time will show a greater reduction and that ownership will moderate this effect.

Methods:

Using a difference-in-differences design and exploiting variation among facilities that previously reported on these measures to The Joint Commission, we examined the effect of the IPFQR public reporting program on the use and duration of restraint and seclusion from the end of 2012 through 2017.

Results:

There were a total of 9,705 observations of facilities among 1,841 unique facilities. Results suggest the IPFQR program reduced duration of restraint by 48.96% (95% CI, 16.69%–68.73%) and seclusion by 53.54% (95% CI, 19.71%–73.12%). There was no change in odds of zero restraint and, among for-profits only, a decrease of 36.89% (95% CI, 9.32%–56.07%) in the odds of zero seclusion.

Conclusions:

This is the first examination of the effect of the IFPQR program on restraint and seclusion, suggesting the program was successful in reducing their use. We did not find support for ownership moderating this effect. Additional research is needed to understand mechanisms of response and the impact of the program on non-targeted aspects of quality.

Introduction

Since the publication of the Institute of Medicine’s Report To Err Is Human in 1999, experts and policy makers have recognized the enormous toll of iatrogenic illness and injury in the U.S. health care system.¹ Patients of inpatient psychiatric care are uniquely vulnerable to harm. Such harms arise most proximally from errors, commission of abuse, negligence, and use of containment measures.^2–5 These vulnerabilities are exacerbated by symptoms of psychiatric illness, such as difficulties with cognition and anergia, which can compromise self-advocacy. Much research and policy attention focused on patient safety in inpatient psychiatry has focused specifically on restraint and seclusion, describing the physical and psychological consequences associated with their use.^6–10

Seclusion (when a patient is involuntarily confined to an area) and restraint (a method that restricts a patient’s freedom or ability to move) are controversial methods of containment. They are controversial because while they have a safety rationale (i.e., emergency measures to protect patients and staff if a patient is at risk of hurting themselves or others), they can be traumatizing to patients and can lead to patient and staff injury (or death).^6–10 Several prominent professional and patient advocacy organizations (such as the National Association of State Mental Health Directors, National Association of Psychiatric Nurses, and the World Health Organization) have called for the reduction and eventual elimination of seclusion and restraint in inpatient psychiatry.^9,11,12

Seclusion and restraint use can vary in part due to various inner-organizational factors, such as staffing level, culture, and incentive structure.^13,14 These inner factors, in turn, are likely influenced by outer-level characteristics and contexts, including extreme market failures (e.g., lack of choice, families often cannot “be at the bedside” to observe the treatment environment and act as advocates).² In the empirical literature, interventions to reduce restraint and seclusion have primarily focused at the inner-organizational level rather than addressing outer-level factors.¹⁵

While inpatient psychiatric care has been excluded from many market-based accountability initiatives, the Centers for Medicare and Medicaid (CMS)’s Inpatient Psychiatric Facility Quality Reporting (IPFQR) program is a relatively recent attempt to systematically measure and incentivize inpatient psychiatric care quality nationally, including reducing restraint and seclusion. The IPFQR program is a public reporting program established by the Patient Protection and Affordable Care Act (ACA) and implemented in 2012 (for fiscal year 2014 payment determination). Through the IPFQR program, psychiatric facilities (both units of general hospitals and psychiatric freestanding facilities) are compelled to report on a suite of quality measures or face a 2% payment reduction in their Medicare annual payment update (i.e., pay for reporting). Prior to the IPFQR program, The Joint Commission (TJC) had a reporting program that required accredited freestanding psychiatric facilities to participate (psychiatric units of general hospitals had the option), although facilities’ performance was not easily accessible to the public. There has not yet been a study to examine the effects of the IPFQR program on quality performance or even broadly the effectiveness of public reporting interventions on the quality of inpatient psychiatric care.

Ownership

Efforts to improve quality and transparency in inpatient psychiatry are particularly salient in light of investigations into unsafe care. For-profit, nonprofit, and government-owned facilities have all come under scrutiny by federal accountability entities and journalists for issues such as patient death and neglect, unqualified or unsupervised staff, and inappropriate admission.¹⁰ News articles and investigations, however, do not provide unbiased and systematic data on safety; therefore, their ability to inform our understanding of disparities in quality among ownership types is limited.

Variation in quality by ownership is important to understand especially given substantial shifts in ownership of psychiatric beds in the past decade.² Most psychiatric facilities and beds are privately owned, and the share of for-profit beds has been increasing.² These shifts are coupled with policy changes that aim to increase Medicaid coverage for inpatient psychiatric care.² Research examining differences in quality by ownership (in general) finds that on average nonprofits do not necessarily behave or perform differently than for- profit hospitals.^16,17 However, this could be different among inpatient psychiatric facilities, where market failures are more pronounced than other hospital settings. Prior research has found disparities in quality of inpatient psychiatric care among different ownership types on certain measures (eg, postdischarge readmission) suggesting nonprofits provide higher quality of care than for-profits.¹⁸ However, the literature is both sparse and dated. A relatively recent analysis of TJC’s precursor program to the IPFQR program found no meaningful difference in performance between for-profits and nonprofits on several measures, but government-owned facilities were poor-performing outliers.¹⁹ Little is known about whether these quality differences translate into differential responsiveness to public reporting.

Theory of public reporting in inpatient psychiatry

While public reporting programs are usually the first step by payers before financial risk is attached to performance, evidence suggests they can motivate quality improvement on their own.²⁰ From a payer or population perspective, public reporting is expected to improve quality through either selection, where patients or health plans use the information to shop for providers, or through provider behavior change, where providers respond to the information out of internal motivation to maintain or improve their reputation or to increase market share.²¹ These channels are not necessarily mutually exclusive, as facilities’ intrinsic response to the incentive to manage reputation can stimulate selection among patients or health plans and thus influence market share.

Theoretically, for-profit organizations may be more likely to exploit the market failures and vulnerability of psychiatric patients in order to maximize profits (to act opportunistically even if it causes harm to patient welfare), whereas nonprofit organizations’ prohibition on distribution of profits to organizational leadership and orientation toward service to the community might better support their propensity to invest in quality, even when there may be no financial benefit to doing so. Therefore, we might expect difference in response to the IPFQR program to be contingent on the degree to which the program depends on motivation that is intrinsic (not related to potential financial gain or risk) versus extrinsic (related to financial incentives).

When it comes to the effect of public reporting in the context of inpatient psychiatric care, the selection theory (where patients and/or their family members select an inpatient provider based on quality) seems unlikely. In order for selection to work, there would need to be a demand response to publicly reported quality information, whereby patients or their agents (e.g., health plans, providers) preferentially select a hospital based on its reputation. Patients of inpatient psychiatric care generally lack choice and often are hospitalized during a time of crisis. Likewise, concerns over bed scarcity could limit the degree to which health plans and providers are able to select on quality or steer patients,²² though it is possible that this is occurring or that facilities fear that this could occur. Regardless of a demand response, psychiatric facilities could still be motivated to improve performance through intrinsic motivation to provide high quality care or manage reputation (which has been observed as a major driver in response to public reporting in other contexts, such as among cardiac surgeons²³), or out of anticipation of financial risk being attached to performance in the future.

Current study

Restraint and seclusion measures in the IPFQR program target aspects of safety where there exists a robust evidence-base in their potential to cause harm. While the suite of measures in the IPFQR program has fluctuated over time, restraint and seclusion have remained since the onset of the program. Further, good performance on these measures cannot be easily obtained though checking a process-based box like some of the other measures (e.g., screening for alcohol use) and therefore improvement might necessitate meaningful investment in structures and processes that support quality more broadly. Such investments might include increasing staffing, training staff in crisis de-escalation techniques, and providing patient-centered care more generally.^24–27

The IPFQR restraint and seclusion measures were adopted by CMS from TJC’s pre-cursor program, through which some facilities were already reporting performance on these measures prior to implementation of the IPFQR. Our analytic strategy exploits this natural variation in when facilities first started reporting performance. We examine whether the IPFQR program is associated with incremental changes in the use of restraint and seclusion, as well as in achieving the more salient and ultimate goal of zero use of these containment measures.^28,29 We hypothesize that facilities reporting on these measures for the first time will demonstrate greater improvement from pre to post-implementation of the IPFQR program relative to those who had already been reporting on these measures to TJC.

We also examine the moderating role of ownership among for-profits, nonprofits, and government-owned facilities. We hypothesize that nonprofits will be more responsive to the IPFQR program than for-profits given the lack of apparent financial benefit associated with performance. To our knowledge, this is the first study to evaluate the effect of a public reporting program on rates of restraint and seclusion among United States inpatient psychiatric facilities as well as differences in response by ownership.

Methods

Study population and data

The study population included all inpatient psychiatric facilities that participated in CMS’ IPFQR program from the last quarter of 2012 through 2017 reporting periods. Restraint and seclusion data from the IPFQR program for these years were linked to the 2013, 2015, and 2017 American Hospital Association’s (AHA) Annual survey to obtain information on facility type (general or psychiatric freestanding) and ownership (for-profit, nonprofit, and government). The AHA Annual Survey has been used extensively in health services research to describe hospital characteristics.³⁰ Information on prior participation in TJC’s pre-cursor program was obtained from TJC. Those reporting performance to TJC prior to the last quarter of 2012 (when reporting to the IPFQR program began) comprised the comparison group and those newly exposed to reporting through the IFPQR program constituted the intervention group.

Variables

Outcomes

Our primary outcomes of interest were restraint and seclusion. Restraint and seclusion are both National Quality Forum-endorsed outcome measures that capture the number of total patient hours in restraint/seclusion (the numerator) out of total patient days divided by 1,000 (the denominator).³¹

Predictors

The primary predictors were an indicator for if the facility was in the intervention group (newly exposed to reporting on restraint and seclusion.) versus comparison, an indicator for if the observation was from the pre-period (before 2014) versus the post-period (2014 and later), and ownership (for-profit, nonprofit, and government). Because relative performance on the restraint and seclusion measures were only knowable among the intervention group after initial reporting to CMS, the first two periods of data are being treated as the “pre” period; having two pre periods allows for a basic assessment of the parallel trends assumption required of a difference-in-differences analysis. These two pre-periods are 10/1/2012–3/31/2013 and 4/1/2013–12/31/2013. All other periods in the IPFQR program are demarcated by full calendar years (i.e., 2014, 2015, 2016, and 2017). A continuous variable for the year was also included to account for year-level differences.

Volume was included as a control in the analysis given its established relationship with quality in general healthcare and in the context of inpatient psychiatric care as well.^32,33 Volume was derived from the denominators of the restraint and seclusion measures, which captures total patient hours in a given reporting period divided by 1,000. Additional characteristics included facility type (general hospital versus psychiatric freestanding). We manually searched for ownership and facility type for 178 (9.67%) facilities that were in the IPFQR data but not in the AHA Annual Survey.

Analyses

For all models, we used a difference-in-differences estimator to estimate the average effect of the IPFQR public reporting program among facilities in the intervention group. Both measures of restraint and seclusion have clumping at zero, which is both a statistical and conceptual issue. Conceptually, a change in duration of restraint and seclusion versus a change in achieving “zero” rates are different ways to measure “success” with potentially distinct policy implications. The former approach to measurement captures incremental change without regard to a benchmark of performance. The latter approach would not capture incremental change but is instead an “all or nothing measure.” Therefore, for restraint and seclusion we fit two separate models each looking at the change in total hours in a given facility among those facilities with any hours in the pre-period, as well as the change in probability of achieving zero hours among all facilities.

For all models, we used generalized estimating equations (GEE) to account for correlation of observations within facilities across the six time-periods. The GEE specification produces estimates representing the population average effect. For the continuous outcomes, we fit a gamma distribution with a log link. For the binary outcomes, we fit a binomial distribution with a logit link. We used an exchangeable correlation structure following the quasilikelihood under the independence model criterion. While GEE is robust to misspecification of the error structure, we also clustered standard errors on the facility. For all models, an interaction between a dummy for the post period and a dummy for the intervention group was fit and represents the difference-in-differences estimate. Additional three-way interactions were fit between post, intervention and a categorical variable for ownership (for-profit, nonprofit, and government). The unit of analysis was the hospital.

Results

There was a total of 9,705 observations of facilities among 1,841 unique facilities. Of these, 7,286 observations were in the intervention group and 2,419 were in the comparison group. In each reporting period, a negligible number of observations (approximately 2%) had missing data for measures of restraint (N = 192 total observations) or seclusion (N = 217 total observations). Overall, facilities in the comparison group were more likely to be for-profit and government-owned, less likely to be psychiatric units in general hospitals, and more likely to be larger in size relative to the intervention group. Facilities in the comparison group were less likely to have zero restraint or seclusion but had lower duration of each in both the pre and post periods compared to the intervention group. See Table 1.

Table 1:

Characteristics of facility-observations by intervention versus comparison

	Comparison (N = 2,419)		Intervention (N = 7,286)
	N	%	N	%	χ2
For-profit	1,016	42.00	1,900	26.08	219.08	p≤0.001
Nonprofit	563	23.27	4,129	56.67	811.07	p≤0.001
Government	840	34.73	1,257	17.25	327.32	p≤0.001
General (versus freestanding)	332	13.72	6,106	83.80	4000.00	p≤0.001
Volume					2100.00	p≤0.001
<20 percentile	408	16.92	1,495	21.05
20–40 percentile	99	4.11	1,804	25.40
40–60 percentile	204	8.46	1,698	23.91
60–80 percentile	498	20.66	1,405	19.78
80–100 percentile	1,202	49.85	700	9.86
Zero restraint in pre-perioda	107	13.34	866	37.80	164.81	p≤0.001
Zero restraint in post-period	148	9.20	1,541	32.03	324.23	p≤0.001
Zero seclusion in pre-period	203	25.37	1,209	52.93	181.28	p≤0.001
Zero seclusion in post-period	441	27.43	2,283	47.60	200.57	p≤0.001
		Median		Median	WMWc
Median restraint in pre periodb	729	0.09	1,620	0.14	−4.10	p≤0.001
Median restraint in post period	1,447	0.09	3,171	0.13	−3.10	p≤0.001
Median seclusion in pre period	649	0.07	1,256	0.14	−4.82	p≤0.001
Median seclusion in post period	1,293	0.07	2,517	0.10	−3.98	p≤0.001

Notes:

^aPercentages of zero restraint and seclusion are calculated using denominators that are restricted to facilities reporting in the given reporting period. For the pre-period, total observations were 802 for the comparison group and 2,291 for the intervention group. For the post-period, total observations were 1,609 and 4,811, respectively.

^aMedians are conditional on having any in the pre period.

^cWilcoxon-Mann-Whitney test.

Change in duration (lower is better)

Among facilities starting off with rates of restraint greater than zero in the pre-period, the difference-in-differences estimate shows that the IPFQR program was associated with a significant reduction in the duration of both restraint and seclusion (48.96% [95% CI, 16.69%–68.73%)] and 53.54% [95% CI, 19.71%–73.12%)], respectively; see Table 2). There was no evidence of ownership being associated with changes in either restraint or seclusion rates (see Table, Supplemental Digital Content 1 for full models).

Table 2:

Model results of the effect of the IPFQR intervention on total duration and odds of zero restraint and seclusion

	Effect of IPFQR Intervention
Models	DD estimate (%)	p value
Total Duration of Restraint	−48.96%	0.03
Total Duration of Seclusion	−53.54%	0.01
Odds of zero restraint	2.89%	0.82
Odds of zero seclusion	−23.52%	0.07
Government (ref = nonprofit)	28.76%	0.22
For-profit (ref = nonprofit)	−36.89%	0.04

Notes:

Assuming that the comparison group is appropriately identified, the difference-in-differences (DD) estimate represents the effect of the IFPQR intervention. For the first three models, estimates from models with only a two-way interaction between intervention and post are shown. For the last model looking at the change in odds of zero seclusion, estimates from the model with a three-way interaction between intervention, post, and ownership are shown. The government and for-profit estimates show the additional effect of the associated ownership type relative to nonprofits.

Change in odds of zero (higher is better)

Among all facilities, the difference-in-differences estimates indicate there was no statistically significant change in the odds of having zero restraint associated with the IPFQR program; there was also no evidence for a moderating effect by ownership (see Table 2). However, analyses of seclusion show evidence for a three-way interaction with ownership, demonstrating a reduction of 36.89% (95% CI, 9.32%–56.07%) in odds of zero seclusion among for-profits (see tables, Supplemental Digital Content 2 & 3 for full models and predicted probabilities).

Discussion

The findings provide support for our hypothesis that the IPFQR program would reduce restraint and seclusion. Specifically, newly-reporting facilities had a greater reduction in total duration of both restraint and seclusion relative to facilities previously reporting on these measures to TJC. However, while a reduction in the amount of both restraint and seclusion was detected, we did not find that the reporting program led to an increase in achieving rates of zero. In fact, odds of zero seclusion actually decreased among the intervention group relative to the comparison group, with this effect being driven by for-profits. We found no support for the hypothesis that nonprofits would be more responsive to the reporting program. While there has been much interest in reducing restraint and seclusion in inpatient psychiatry, most interventions empirically studied have examined organizational-level interventions (e.g., changes in staffing levels, staff-patient relationships, or hospital polices) and have been primarily restricted to single sites.^13,15,34,35 To our knowledge, this is the first study to examine a national policy intervention to reduce rates of restraint and seclusion among the entire population of psychiatric facilities in the United States, although smaller-scale studies have been conducted on the effect of policies such as national accreditation standards.³⁶ Indeed, until the IPFQR program, most psychiatric facilities were only required to report deaths related to restraint and seclusion as part of CMS’ Conditions of Participation and, unlike the IPFQR measures; even these data are not publicly reported.

The IPFQR program’s effect on restraint and seclusion reduction could have been influenced by various motivations of these facilities, including a desire to maintain or increase market share or reputational assets, an intrinsic desire to provide quality care, or a mixture of these. While the IPFQR program is not a pay-for-performance program and there is little evidence that patients shop for inpatient psychiatric care, facilities may have implemented internal interventions to reduce restraint and seclusion anticipating that it will evolve into a pay-for-performance program, or that health plans might provide a demand response through use of provider networks or explicit financial incentives. Possibly, the IPFQR program intervention supported facilities in tracking their performance and in understanding how they compare with their peers, thus motivating performance through an intrinsic desire to improve standard care or manage reputation.

The decrease in odds of achieving zero seclusion among the for-profits could be because in some situations using seclusion as a containment measure is a substitute for restraint.¹³ Another hypothesis is that this reduction is an artifact of measurement error in the pre period, such that some facilities who were not already tracking hours of seclusion erroneously reported a rate of zero. Thus, the change in odds of zero seclusion might reflect better tracking and more accurate reporting among facilities. Indeed, for-profit facilities had notably higher rates of zero seclusion in the pre-period compared to both nonprofits and government-owned facilities. Moreover, even though the comparison group had lower duration in both periods, the intervention group had higher rates of zero. Overall, these divergent patterns suggest that performance on restraint and seclusion depends on how the measures are operationalized by accountability programs and what the policy objectives are.

Further adding to the difficulty in interpreting the observed changes is the fact that we cannot distinguish if a value reflects the amount of time in restraint or seclusion for fewer patients or shorter periods of time for more patients, as measures are at the facility-level. The relative value of these scenarios (i.e., more people with lower duration versus fewer people with greater duration) and its meaning in interpreting responsiveness to the IPFQR program cannot be clearly delineated. Indeed, research investigating changes in restraint and seclusion specifically in response to injurious assaults found average patient-level restraint and seclusion durations declined from 2007 to 2013 but not the number of episodes.³⁷ Measures that capture patient-level information regarding the number and duration of restraint and seclusion episodes would better allow for these nuances to be described and could inform researchers and policymakers on the nature of the underlying quality problems and relevant remedies.

There are several limitations of this study. First, the comparison group previously had been exposed to reporting on these measures to TJC in the pre-period and was additionally exposed to the IPFQR program along with the intervention group, rather than ideally being fully unexposed in both pre and post periods. Further, there could be other unobserved differences that might be associated with response to the IPFQR program. Second, the pre-period is composed of two consecutive time periods during the last quarter of 2012 and most of 2013, which technically was the first year of reporting. Because reporting these performance metrics was the intervention itself, true pre-intervention performance was unavailable. Both of these limitations, however, are expected to reduce the observed difference between the comparison and intervention group. While inspection of trends in the pre-period suggest parallel trends between the intervention and comparison groups in terms of direction, there appears to be a steeper increase from the first to second period among the intervention group in the pre-period, which threatens our parallel-trends requirements of a difference-in-differences analysis (see graphs, Supplemental Digital Content 4 & 5); nevertheless, this increase is contrary to the post-period trend, differences in trends are not statistically significant, and our results are robust to the exclusion of this second period. Finally, we did not have information on case-mix. However, we have no reason to expect that there were meaningful changes in case-mix between treatment and comparison group across time in such a way that would bias our estimates.

The IPFQR program is relatively nascent compared to other national quality reporting programs. There is still debate over the utility of the restraint and seclusion measures in capturing meaningful variation in quality. This is the first study to examine the effect of the IFPQR program on rates of restraint and seclusion, suggesting that the public reporting stimulated a response to these measures. Our study does not support calls to eliminate these measures from the IPFQR program, though policy efforts should consider complementary ways of operationalizing measures of restraint and seclusion in order to better capture quality. Future research should more deeply explore the mechanisms underlying response to the IFPQR program and the extent to which response comes from intrinsic versus extrinsic motivation.³⁸ Important next areas of research would be to examine differences by ownership in the effect of the IFPQR program on non-targeted aspects of quality that are both proximal and less proximal to the targeted measures.^6,7,39