Targeting Nursing Homes Under the Quality Improvement Organization Program’s 9th Statement of Work

Abstract

In the Quality Improvement Organization (QIO) program’s latest Statement of Work, the Centers for Medicare and Medicaid Services (CMS) is targeting its nursing home activities toward facilities that perform poorly on two quality measures—pressure ulcers and restraint use. The designation of target facilities is a shift in strategy for CMS and a direct response to criticism that QIO program resources were not being targeted effectively to facilities or clinical areas that most needed improvement. Using administrative data, this article analyzes implications of using narrowly defined criteria to identify facilities that need improvement, particularly in light of considerable evidence showing that nursing home quality is multidimensional and may change over time. The analyses show that one in four facilities is targeted for improvement nationally but that approximately half of some states’ facilities are targeted while other states have almost none targeted. The analyses also convey deeper limitations to using threshold values on individual measures to identify poorly performing homes. Target facilities can be among the top performers on a range of other quality measures, and their performance on targeted measures themselves may change over time. The implication of these features is that a very different group of facilities would have been chosen had the QIO program targeted other measures or examined performance at a different point in time. Ultimately, CMS has chosen a blunt instrument to identify poorly performing nursing homes, and supplemental strat-egies—such as soliciting input from state survey agencies and more closely aligning quality improvement and quality assurance efforts—should be considered to address potential limitations.

On February 5, 2008, the U.S. Centers for Medicare and Medicaid Services (CMS) released the names of almost 4,000 nursing homes to be targeted under the Quality Improvement Organization (QIO) program’s 9th Statement of Work (SOW). Although the CMS announcement emphasized that facilities on the list should not be perceived as the worst-quality homes overall,¹ each performed poorly enough on the pressure ulcer or restraint use quality measure from Nursing Home Compare to merit being targeted. With the caveat that facility participation in the QIO program is voluntary, CMS has instructed QIOs that at least 85% of the nursing homes they assist in quality improvement be drawn from this “target facilities” list.

The designation of QIO target facilities is a shift in strategy for CMS and a direct response to criticism from the Institute of Medicine (IOM), the Government Accountability Office (GAO), and the U.S. Congress.^2–4 CMS previously gave QIOs broad guidance on the number of participants to enroll in each state, but it did not direct facility selection or define the practice areas on which QIOs should offer assistance. QIOs had the discretion and, because of the way their performance was evaluated, the incentive to work with facilities that were already on the path to quality improvement instead of those that were struggling. In outlining the program’s new direction, CMS highlighted recommendations made by the IOM and the GAO and emphasized its commitment for QIOs to work with nursing homes and clinical areas most in need of improvement.

This article examines CMS’s new approach to targeting nursing homes in the 9th SOW. More specifically, the analyses assess the implications of using narrowly defined, point-in-time criteria to identify facilities in need of improvement, not only in the context of trying to focus resources on poorly performing homes, but also in light of the considerable evidence showing that nursing home quality is multidimensional (i.e., a facility may perform well on some dimensions of quality and not on others) and may change over time. The article analyzes facility performance on a range of quality indicators and examines the stability of pressure ulcer and restraint performance within homes over time. The results are used to convey broader insights about targeting nursing home quality improvement efforts and as a springboard for discussion of the role of the QIO in improving nursing home care.

The Nursing Home QIO Program

Government-directed quality improvement in nursing homes is a relatively new endeavor, begun nationally in 2002. Initially oriented to the Federal Nursing Home Quality Initiative and its emphasis on public reporting, QIOs were directed to help consumers understand and use available quality information, respond to complaints by and on behalf of Medicare beneficiaries, and work with nursing homes directly to improve performance on clinical measures, many of which were beginning to be publicly reported at the time.⁵ Because QIOs do not function in a regulatory capacity, they are able to work with nursing homes in a consultative way. QIOs typically focus on clinical improvement and work with a nursing home’s clinical staff, including physicians, offering a range of assistance in group and individual settings (e.g., disseminating educational materials that highlight best practices and assisting individual homes through on-site visits). Provider involvement in the QIO program is voluntary.

Given the significant investment in the QIO program (the 9th SOW is budgeted at $1.128 billion over the 3-year contract period), increased attention has been given to whether the program is an effective use of resources. Extant studies generally present a mixed picture of the program’s effect but have also been limited in their ability to distinguish QIO effects from secular trends, unrelated quality improvement initiatives, and other changes in the policy environment, details of which are described in a recent IOM report.³ The literature assessing QIO performance regarding nursing home quality improvement is limited, in part because the program is relatively new. Some articles have focused on the potential of the QIO program broadly^6–8 and others on nursing home performance in particular areas.^9–11 The most detailed assessment of the nursing home QIO program is an article in the Annals of Internal Medicine by CMS staff attributing improvements in pressure ulcer and physical restraint rates to QIO efforts.¹² Still, an editorial accompanying the article raises methodological concerns, some of which the CMS authors detail themselves.¹³

The most recent SOW of the QIO program centers on four themes—protecting beneficiaries, care transitions, patient safety, and prevention. CMS provides the most detail for QIOs’ nursing home work in the area of patient safety, identifying a list of facilities to be targeted for improvement and setting an expectation that participants’ pressure ulcer and physical restraint use rates will decline by 8% and 20%, respectively. Beyond evaluating the QIO program’s overall effectiveness, it is important to examine CMS’s new targeting strategy for nursing homes and its implications. By identifying target facilities and directing QIOs to focus on two quality measures, CMS has taken important steps to focus resources on struggling nursing homes in specific areas of need. The analyses and discussion below focus on the nature and potential effectiveness of this targeting, comparing targeted and nontargeted facilities along several dimensions and assessing the potential strengths and limitations of the approach.

METHODS

Objectives

Three objectives guided these analyses. The first was to describe the distribution of QIO target facilities across states. Given the wide interstate variation in nursing homes’ performance in various quality measures,¹⁴ it was hypothesized that there would be variation in the number of target facilities according to state, something that might be problematic in the context of a national quality improvement program. The second was to compare characteristics of targeted and nontargeted nursing homes nationwide, including performance on nontargeted dimensions of quality. Given previous research showing that nursing home quality is multidimensional, with facilities performing well on some quality measures and less well on others,^14,15 it hypothesized that a sizeable minority of targeted facilities would perform highly on other dimensions of quality, potentially undercutting the notion of QIO resources being focused on the poorest-quality homes. The third was to examine past facility performance on targeted quality measures. Given previous research showing the changeable nature of nursing home quality performance over time,¹⁴ it was hypothesized that a sizable minority of targeted and nontargeted facilities would have been categorized differently with respect to their target status in previous years, something that could raise questions about using a static facility list in the context of a 3-year SOW.

Data

To identify facilities that will be targeted for improvement on pressure ulcers, restraints, or both measures in the QIO 9th SOW, the analyses use the national list of these facilities released by CMS in February 2008 (list available upon request).

To examine research questions of interest, two primary sources of nursing home data are used. The first source is the On-line Survey, Certification, and Reporting (OSCAR) system, which contains survey and certification data for all Medicaid- and Medicare-certified facilities in the United States.¹⁶ Collected and maintained by CMS, the OSCAR data include information about whether homes are in compliance with federal regulatory requirements. Nursing homes submit facility, resident, and staffing information. Survey agencies enter deficiencies into OSCAR when facilities are found to be out of compliance with federal regulatory standards. OSCAR data have important limitations that should be noted, including a lack of explicit auditing procedures of facility-reported information, potential variation across states, and possible underreporting of serious quality problems.^17,18

The second source are data from the Minimum Data Set (MDS) quality indicator/quality measure (QI/QM) system. The MDS is an instrument that assesses residents’ functional, cognitive, and affective levels upon admission and at least quarterly thereafter and has demonstrated good reliability and validity in many resident characteristics. At the same time, researchers have noted interfacility and interstate variation in ascertaining certain conditions of interest.¹⁴ The QI/QM data system constitutes facility aggregates of resident-level MDS data and are reported monthly. Although some problems have been identified regarding data accuracy, QI/QM data have shown good reliability in identifying potential quality problems.^19,20 Because residents are surveyed every quarter, monthly QI/QM data were aggregated to the quarter level. CMS used QI/QM data to identify target facilities. In particular, using the first three quarters of 2007, facilities were designated as target facilities if, for at least two of these quarters, pressure ulcer rates were greater than 20% or restraint rates were greater than 11%.

Analyses

First, the analyses describe the number of target facilities and their distribution across states according to target category. Second, using the most recent OSCAR observation for each facility, the analyses compare characteristics of target and nontarget facilities (using t-tests to compare means). These traits include size (number of beds), chain and ownership (for-profit; nonprofit; and government) status, occupancy rate (residents/beds), staffing (aide and nurse staffing per resident), payer mix (percentage of residents primarily relying on Medicaid, Medicare, and other sources), and case mix (average facility activity of daily living (ADL) score). Third, to describe the range of performance by target facilities on other publicly reported QI/QMs and on health-related survey deficiencies, information is presented on the performance quintiles into which facilities fall across measures (from 1 (best) to 5 (worst)). The QI/QMs reviewed include percentage of residents with catheter use, urinary tract infections, and excessive weight loss; spending most of their time in bed or in a chair (bed-fast); losing ability in ADLs; physical restraint use; pain; and pressure sores for low- and high-risk residents. If performance of target facilities is the same as for all nursing homes, 20% of target facilities would be in each of the performance quintiles. The analyses also assess overlap between target facilities and facilities identified by CMS as chronically poor performers (Special Focus Facilities). Finally, to examine the stability of facilities’ performance on the target measures over time, the analyses examine what facilities’ target status would have been going back to 2004 using QI/QM data and applying the same methodology used by CMS with 2007 data.

RESULTS

Variation Across States

CMS designated 4,037 facilities—almost one in four facilities nationwide—to be targeted for improvement under the 9th SOW for pressure ulcers (2,475 facilities), restraint use (1,942 facilities), or both (380). The proportion of targeted facilities varies widely across states (Figure 1), with 11 states having fewer than 10% of facilities identified for improvement on either measure (Alaska, Nebraska, Vermont, Minnesota, Iowa, Hawaii, New Hampshire, Delaware, North Dakota, Maine, and Wisconsin) and four states having at least half of facilities so designated (Arkansas, California, Oklahoma, and Louisiana). The proportion of facilities targeted for each measure also varies widely according to state, and states do not necessarily have similar proportions of facilities targeted for each measure.

Figure 1.

Percentage of nursing homes targeted in the Quality Improvement Organization (QIO) program’s 9th Statement of Work (SOW) according to state. Nursing homes are being targeted for improvement in the QIO program’s 9th SOW based on their 2007 performance on two quality measures: pressure ulcers for high-risk residents and restraint use. The state-level data presented above are based on the list of target facilities published by the Centers for Medicare and Medicaid Services in February 2008.

Characteristics of Target Facilities

Table 1 summarizes select nursing home traits according to target status. Comparing target and nontarget facilities using data from the most recent survey, target facilities are larger than nontargeted facilities and much more likely to be for profit than nonprofit. Target facilities have lower nurse and nurse aide staffing ratios, a greater proportion of residents relying on Medicaid financing than on Medicare or private payment, and higher acuity in the form of resident ADL.

Table 1.

Summary of Facility Characteristics According to Quality Improvement Organization Program Target Status

Online Survey, Certification, and Reporting Measures†	Not Targeted	Targeted for Either Measure	Targeted for Restraints	Targeted for Pressure Ulcers	Targeted for Both Measures
Certified beds, n	102.4	112.0**	109.2**	115.7**	112.2**
Chain, %	53.0	52.0	51.9	51.3	47.4*
For profit, %	62.5	77.8**	78.7**	77.1**	80.8**
Nonprofit, %	30.8	18.3**	17.5**	18.8**	15.8**
Government, %	6.7	4.0**	3.8**	4.1**	3.4*
Occupancy rate‡	1.00	1.00	1.01	0.98**	0.97*
Aide staff, FTE per resident	0.46	0.43**	0.45	0.42**	0.45
Nurse staff, FTE per resident (registered nurse and licensed practical nurse)	0.19	0.16**	0.16**	0.17*	0.17
Medicaid, %	58.1	66.0**	67.2**	64.7**	67.2**
Medicare, %	16.9	13.5**	12.2**	15.2**	13.3**
Other, %	24.9	20.5**	20.6**	20.1**	19.5**
Mean limitations in activities of daily living, n	3.91	3.99**	4.05**	3.91	4.00**

^†Based on most recent survey observation;

^‡Relative to state mean.

Significantly different from non-targeted facilities at the P < *.05 and ** .01 (t-test).

FTE = full-time equivalent.

Performance on Other Quality Measures

Figure 2 shows the distribution of target facilities’ performance across a range of quality measures and survey deficiencies. Although a higher proportion of target facilities are generally in the lower performance quintiles (i.e., > 20% of target facilities are generally in quintiles 4 and 5), a sizable minority of target facilities are in the highest or second-highest performance quintiles on most measures. For instance, 17.9% and 16.9% of target facilities were in the two best performance quintiles on the pain QI/QM, respectively. Using a recent CMS barometer of chronic poor performance on state survey inspections, 64% of Special Focus Facilities are not being targeted for improvement on pressure ulcers or restraints. In addition, 194 target facilities were deficiency free on their last inspection (another 191 had one deficiency), and 52% had fewer survey deficiencies than the state average.

Figure 2.

All target facilities: performance on other quality indicators and deficiencies according to quintile. To describe the range of performance by target facilities on various Minimum Data Set quality indicators and quality measures and on health-related survey deficiencies, Figure 2 presents information on the performance quintiles into which these facilities fall across measures (from 1 (best) to 5 (worst)). If the performance of target facilities is the same as for all nursing homes, 20% of target facilities would be in each of the performance quintiles. Deficiency data from Online Survey, Certification, and Reporting system and performance is relative to state average. ADL = activity of daily living; UTI = urinary tract infection.

Stability of Target Status over Time

Figure 3 shows facilities’ hypothetical target status back over 3 years (i.e., the duration of QIO contract cycles). Using the same methodology that CMS employed with 2007 data, 26% of target facilities would not have been labeled as such in 2006; these portions increase to 29% and 33% based on 2005 and 2004 performance, respectively. Although some of the change in target facility past performance reflects broader trends (e.g., reductions in pressure ulcer and restraint rates nationwide), a similar picture of change emerges for nontarget facilities. Twenty percent of nontarget facilities would have been targeted for improvement based on their 2006 performance, with these proportions increasing to 31% and 33% in 2005 and 2004, respectively. In other words, one-third of target and nontarget facilities would have been designated differently based on their performance 3 years ago.

Figure 3.

Facilities’ hypothetical past Quality Improvement Organization target status, 2004–2007. The top panel begins with target facilities and looks back in time to assign what their target status would have been in previous years. The bottom panel begins with nontarget facilities and looks back over the same time period. Target status in previous years is defined based on the 2007 criteria (≥20% for pressure ulcers; ≥11% for restraints). Circles are target facilities, and squares are nontarget facilities. The nursing homes included are those that have all years of data from 2004 to 2007 (3,924 facilities are included in all analyses for target facility analyses, and 11,014 facilities are included in the nontarget facility analyses). The percentages for each row (year) may not add to 100% because of rounding.

DISCUSSION

CMS is taking a new approach for the QIO program’s 9th SOW for nursing homes. Responding to criticism that QIOs previously could “cherry pick” facilities or focus on low-priority clinical areas, the agency has specified two measures of importance—pressure ulcers and restraints—and identified the facilities most in need of improvement. By directing QIO nursing home efforts more closely, CMS hopes to improve the program’s effectiveness and produce greater value for Medicare beneficiaries nationwide. However, the first thing one notices about the list of 4,000 facilities targeted for improvement is how widely the numbers vary according to state. One in four facilities is targeted for improvement nationally, but upward of half of some states’ facilities were targeted, whereas other states have almost none. Whether differences in case mix, ascertainment biases in identifying the target conditions, or the quality of care that is delivered are leading to these discrepancies, the implication is that the work and, presumably, the budgets in front of state QIOs are highly variable. By extension, nursing homes and clinicians working in them will have varying levels of QIO support available depending on their geographic location, a feature that seems problematic in the context of a national program that should benefit all Medicare beneficiaries.

More important, the analyses convey deeper limitations to using threshold values on individual measures to identify poorly performing homes. Target facilities are often among the top performers on a range of other quality measures, implying that a very different group of facilities would have been chosen had the QIO program targeted other measures. At the root of this problematic directive is the fundamental finding, previously noted, that nursing home care quality is multidimensional.^14,15 Facilities that excel in one area will not necessarily excel in another, something conveyed by the poor within-facility correlation across quality measures. Thus, targeting facilities on the basis of only two measures will inevitably capture facilities that perform well on other dimensions. Indeed, even the two parameters chosen by CMS, restraints and pressure ulcers for high-risk residents, are not highly correlated (based on our analyses, the correlation coefficient between the two measures was 0.11, meaning that one measure explains a little more than 1% of the variation in the other).

Identifying a static list of facilities to be targeted for improvement during a 3-year contract cycle also fails to account for the potential volatility of nursing home performance. The results showed that facility performance on the two targeted measures changes over time; in other words, next year’s target (or nontarget) facilities are not necessarily this year’s, let alone 3 years ago, when as many as one-third of facilities would have had a different target status. This feature is not unique to the selected measures; indeed, these analyses show similar trends on any number of the different parameters by which researchers and the public might judge the quality of care (available on request).

Another critical point, also identified by others, is that resident acuity can heavily influence quality measures, including deficiencies.^14,21 If a facility specializes in wound care and attracts difficult-to-treat residents, for instance, its pressure ulcer rate—which does not distinguish between prevalent and incident cases—will appear problematically higher than other facilities. Target facilities ironically could include some homes that are providing exemplary care for a challenging resident population.

Ultimately, using threshold values on two quality measures, especially at a given point in time, is a blunt instrument to identify poorly performing nursing homes. QIOs, policymakers, and other stakeholders, therefore, should not assume that the current approach identifies the type of struggling homes that the GAO and others have called on the QIOs to engage. Likewise, clinicians who work in nursing homes and those who refer patients to them should not assume that poor performance on individual measures necessarily implies poor quality performance overall or that a snapshot of a nursing home at a point in time is necessarily representative of facility care at a later date. Clinicians should also be cognizant of the potential influence of resident case mix in assessing nursing homes’ quality-related performance.

Perhaps CMS’s identification strategy could be bolstered with ongoing input from key stakeholders such as state survey agencies (SSAs). In the example above of a facility specializing in wound care, surveyors might be able to distinguish these facilities from those where high pressure ulcer rates are more indicative of care problems. CMS seems to recognize the importance of this supplemental input in its requirement that each QIO work with a small number of facilities that CMS and the SSA have identified as chronically poor performers, but this directive (to work with one such facility per contract year) is limited in the context of the broader SOW.

More broadly, quality improvement and quality assurance should be viewed as complementary tools that government can use to improve nursing home quality. The possibility of sanctions or closure for poor performance are important deterrents, yet using an approach that is exclusively punitive will be constrained in its ability to help struggling providers identify root causes of problems and work effectively in addressing them. Regulatory efforts also do little to engage clinicians in understanding and addressing potential care problems at nursing homes.

If SSAs were to become more engaged in the nursing home QIO program, an area where this might occur is in priority setting. Although pressure ulcer and restraint reduction are worthy goals, the wide variation in these measures across states raises the possibility that there may be other, more-pressing concerns to be addressed, at least in some states. Giving QIOs discretion to identify one such area in consultation with the SSA, for example, could help focus program efforts more closely on areas of need. These collaborations could raise some questions about the input of survey agencies into a consultative endeavor, but it seems possible to establish an iterative process by which inspectors identify performance problems and QIOs help providers design quality improvement interventions to ameliorate them.²²

LIMITATIONS

The analytical insights presented above are limited by the administrative data on which they are based. Although the MDS QI/QMs have reasonable reliability across a large number of facilities, there can be substantial interfacility variation in reliability that has direct implications for estimates derived for the quality measures.^23,24 The factors that contribute to some of the discrepancies identifed are unclear (e.g., why pressure ulcer and restraint rates vary widely across states). Moreover, it is likely that multiple factors contribute to the lack of correlation across quality measures and to the volatile nature of quality-related performance over time, including technical reasons such as small sample size or measurement error, turnover in assessment staff, or even systematic changes in how items are interpreted. Turnover in clinical leadership, the ebb and flow of financial pressures faced by facilities, and even changing competition for nursing labor from other healthcare providers additionally complicate these factors. Unfortunately, little is known about these phenomena, their distribution across nursing homes, or their relative influence on true quality performance.

CONCLUSION

In responding to reports from the IOM and the GAO, CMS has taken important steps to improve the effectiveness of the nursing home QIO program. Replacing the more laissez-faire approach of the last contract cycle, CMS is providing substantially more direction to QIOs on measures to improve and facilities to work with, but as outlined above, important limitations to this approach need to be recognized and addressed to ensure that program resources are targeted effectively. If they are not, the overarching vision of the QIO program to deliver “the right care for every person every time” will be difficult to achieve.