Medicare's Quality Improvement Organization Program Value in Nursing Homes

Abstract

CMS operates the quality improvement organization (QIO) program to improve the quality of care delivered to Medicare beneficiaries. Although there have been several studies regarding the effectiveness of this program, there have not been studies regarding this program's value. This article seeks to answer the value question using cost-utility analysis. Although additional research is warranted, the results suggest that CMS' investment in the QIO program, estimated at $2,063 to $7,667 per quality-adjusted life year (QALY) gained for nursing home quality improvement (QI) work, represents a good value for health care dollars.

Introduction

The Medicare Program currently covers an estimated 42.1 million elderly and disabled beneficiaries (Centers for Medicare & Medicaid Services, 2005). To help improve the quality of care delivered under Medicare, CMS (2005) operates a national network of 53 QIOs responsible for each U.S. State, Territory, and the District of Columbia. One key aspect of the QIOs' responsibilities is to “… improve quality of care for beneficiaries by ensuring that beneficiary care meets professionally recognized standards of health care …” (Centers for Medicare & Medicaid Services, 2005). Since the early 1990s, the QIOs' primary approach has been to achieve this goal through the provision of QI technical assistance to health care providers (e.g., hospitals, nursing homes, physician practices, home health agencies, and managed care plans) in a collaborative environment (Jencks and Wilensky, 1992; Jencks, 1995).

Although there has been increasing literature examining the effectiveness of QIO work, there have been no studies on the value of their work. That is, if the program is effective in improving care, are the benefits worth the investment in the program by CMS? This article seeks to answer that question using cost-utility analysis, where the benefits are measured in QALYs gained. QALYs are not restricted to changes in the quantity of life, and allow us to measure changes in the quality of life. The QALY measure on values or weights that reflect the desirability of health States emphasizes the critical role of consumer preferences in valuing outcomes (Drummond et al., 2005). Translating gains in health care quality potentially attributable to QIO work into QALYs facilitates comparison to other health care interventions that are currently promoted in the U.S. health care system whose benefits are also measured in QALYs.¹ This article does not seek to answer the effectiveness question, but rather presents results based on a range of reasonable assumptions regarding QIO effectiveness.

QIOs work on multiple QI projects in different health care settings. They were asked to assist nursing homes across the country to improve quality of care as measured by CMS' publicly reported nursing home quality measures, which were launched in October 2002. The scope of activities included educational conferences, collaborative learning sessions, distribution of QI materials, and individual technical assistance provided to nursing homes. The question of the value of QIO work in this setting is particularly important because this was a new project for the QIO program, and a priority area for CMS.

Methods

Quality Measures

QIO nursing home QI activities began in July 2002. To track changes in quality-of-care, we utilized the CMS nursing home quality measures for national public reporting from 2002-2005. The measures are derived from the resident-level minimum data set (MDS) that are submitted by nursing facilities to their respective State health departments. These measures have been validated (Morris et al., 2003) and endorsed by the National Quality Forum, and facility-specific results are reported publicly on CMS' Nursing Home Compare Web site at: http://www.medicare.gov/NHCompare/home.asp).

Although QIOs were asked to support improvement in all of the quality measures, each QIO was asked to choose three to five measures to focus on in their respective States. To be more conservative in our assumptions about improvements due to QIO activity, we included only the top five measures across the country as areas of focus. These measures are summarized in Table 1. The baseline measurement period was the 2^nd quarter 2002. The remeasurement period for this study was the period used by CMS to evaluate QIOs on their performance-based contract, the 2^nd quarter 2004. The national performance data during the baseline and remeasurement periods were provided by CMS.

Table 1. Nursing Home Quality Measure Summary Descriptions of Medicare's Quality Improvement Organization Program: 2002-2003.

Measure	Summary Description
Short-Stay Residents With Moderate/Severe Pain	The percent of post-acute residents who reported experiencing moderate pain at least daily or horrible/excruciating pain at any frequency during the assessment period.
Residents Who Had Loss of Function	The percent of chronic residents with worsening ability to perform activities of daily living when compared to the prior assessment.
Long-Stay Residents With Moderate/Severe Pain	The percent of chronic residents who reported experiencing moderate pain at least daily or horrible and/or excruciating pain at any frequency during the assessment period.
Residents With Physical Restraints	The percent of chronic residents who were physically restrained daily on the target assessment.
Residents With Pressure Sores	The percent of chronic residents with pressure ulcers on the target assessment.

NOTES: Short-stay residents are patients who are admitted to a facility and stay less than 30 days. The remainder of the measures pertains to traditional nursing home residents, who tend to remain in a facility from several months to several years.

SOURCE: Abt Associates: Quality Measures for National Public Reporting: User's Manual, Version 1.2. January 2003.

Estimating QIO Impact

For measures that improved during the study period, we estimated the number of residents affected by the improved rate using the national denominator for each measure at the time of remeasurement. That is, the estimated number of residents whose status improved was (Baseline Rate—Remeasurement Rate)* (Remeasurement Denominator). This product represents the number of residents that avoided the negative outcome assessed by the quality measure because of the improvement in the measure when compared to baseline. Because the CMS/QIO initiative was the only national initiative at the time focusing on the study topics, it is reasonable to assume that most of the improvement can be attributed to QIO activities. We estimated the QIO contribution to improvement to be 75 percent of the observed improvement. For the purposes of sensitivity analysis, we performed the calculation using a range of 50 to 100 percent attributable improvement.

QALY Estimate

To estimate QALYs gained for improvement in the quality measures, we used the Health Utility Index Mark 2 (HUI2) developed by Torrance et al. (1996). Although there are many available multiattribute utility scales, a review performed by the Health Technology Assessment Program of the National Health Service of the United Kingdom found that based on practicality, reliability, and validity, the HUI2 and the EQ-5D developed by the EuroQol Group were the best preference-based measures available (Brazier et al., 1999). We chose the HUI2 because the health States of the instrument more closely matched the States described by the quality measures. The HUI2 is a widely cited generic multi-attribute preference-based system for assessing health-related quality of life. It is an empirically derived formula conceived from interviews with a general population sample. To satisfy the QALY concept, quality weights must be: based on preferences; anchored on perfect health and death; and measured in an interval scale (Torrance, Thomas, and Sackett, 1972). The more preferred health States receive a higher weight. Health States used in developing the quality of life scores were derived from the multi-attribute utility function based on the visual analog scale (VAS), which is a standard zero to one interval health preference scale (Torrance and Feeny, 1972).

Specifically, we utilized the HUI2 multi-attribute utility function on dead-healthy scale using the mobility and pain attributes to reflect changes in the quality measures. Utility scores, with 95 percent confidence intervals (CIs), were calculated for the level 5 mobility attribute, described as unable to control or use arms and legs, and for the level 4 pain attribute, described as frequent pain; frequent disruption of normal activities; discomfort requires prescription narcotics for relief. The scores from the HUI2 do not differentiate between short-stay and long-stay residents, so one value is used for both subgroups.

While analyzing each attribute, all other (e.g., cognition, sensation) were set at level 1—normal healthy functioning. Our calculations for QALYs assume no discounting; therefore, the QALY gained for a health State avoided for 1-year is: 1-(UTILITY). We did not assume any change in life expectancy, nor did we assume effects lasting longer than a year. To determine the number of QALYs gained for the entire population, this value is multiplied by the number of net residents positively affected for the measures considered. Sensitivity analysis is performed based on both the 95 percent CI for the (1-UTILITY) values, as well as the reasonable range for QIO attribution of observed QI.

Cost estimate

The cost of Medicare's investment in the QIO nursing home project was estimated using a QIO cumulative expenditures report during the period 2002-2005. QIOs report actual expenditures for each task. The report utilized for this analysis was 88.4 percent complete—the total estimated cost was derived using linear assumptions about expenditures. The cost per QALY gained was calculated by dividing the total estimated cost by the total QALYs gained that were attributable to QIOs.

Results

Of the five measures most commonly chosen by QIOs during the study period to focus their improvement efforts on, three of the measures—(1) percent of short-stay residents with moderate to severe pain; (2) percent of long-stay residents with moderate to severe pain; and (3) percent of residents with physical restraints—had absolute improvement of 2.22 to 4.0 percentage points, all of which were statistically significant (p<0.0001). This represented a relative improvement of 12.9 to 37.4 percent. Of the two remaining measures—percent of residents who had loss of function and percent of residents with pressure sores—the absolute worsening was only 0.04 and 0.19 percentage points, representing a relative change of 0.3 percent and 2.2 percent. The specific results are noted in Table 2.

Table 2. Baseline and Remeasurement Performance Rates of Medicare's Quality Improvement Organization Program: 2002-2004.

Measure	Baseline Rate (2002Q2)	Remeasurement Rate (2004Q2)	Rate Difference
	Percent
Short-Stay Residents With Moderate/Severe Pain	25.42	22.14	3.28
Residents Who Had Loss of Function	15.42	15.44	(0.04)
Long-Stay Residents With Moderate/Severe Pain	10.67	6.67	4.00
Residents With Physical Restraints	9.72	7.50	2.22
Residents With Pressure Sores	8.49	8.68	(0.19)

NOTE: Q² is 2^nd quarter.

SOURCE: Centers for Medicare & Medicaid Services: Data from the quality improvement organization program, 2002-2004.

The improvement in the three measures translates into 20,288 short-stay residents who avoided moderate to severe pain, 46,966 long-stay residents who avoided moderate to severe pain, and 26,832 residents who avoided physical restraints during the remeasurement period when compared to performance levels at the baseline measurement period. The number of these residents who avoided the negative outcome due to QIO activities varies according to assumptions regarding the level of QIO impact. Table 3 depicts these values depending on 50, 75, or 100 percent attribution of improvement to QIO activities.

Table 3. Estimated Number of Residents Who Avoided a Negative Outcome During the Remeasurement Period When Compared to the Baseline Period Because of Medicare's Quality Improvement Organization Program Activities: 2002-2004.

Measure	Attribution
	50 Percent	75 Percent	100 Percent
	Number of Residents
Short-Stay Residents With Moderate/Severe Pain	10,144	15,216	20,288
Long-Stay Residents With Moderate/Severe Pain	23,483	35,225	46,966
Residents With Physical Restraints	13,416	20,124	26,832

SOURCE: Shih, A., The Commonwealth Fund, Dewar, D.M., University of Albany, and Hartman, T., IPRO, 2006.

Using the HUI2 multiattribute utility function, the QALY gain for improvements in the moderate/severe pain quality measure was 0.3816 per resident, with a 95 percent CI of 0.2616 to 0.5016. The QALY gain for avoiding restraints was 0.4452, with a 95 percent CI of 0.3252 to 0.5652. Using these QALY estimates, the total number of QALYs gained due to QIO activities is noted on Table 4 using 50, 75, and 100 percent QIO attribution.

Table 4. Total Number of QALYs Gained Due to the Medicare's Quality Improvement Organization Program Activities, by QALY Estimate and Attribution: 2002-2004.

QALY Estimate Measure	Attribution
	50 Percent	75 Percent	100 Percent
	QALYs Gained
Low (Lower Bound of 95 Percent CI)	13,160	19,740	26,320
Mean	18,805	28,208	37,610
High (Upper Bound of 95 Percent CI)	24,450	36,675	48,900

NOTES: QALY is quality-adjusted life years. CI is confidence interval.

SOURCE: Shih, A., The Commonwealth Fund, Dewar, D.M., University of Albany, and Hartman, T., IPRO, 2006.

With a total estimated program cost of $100,895,928—the costs per QALY gained using the mean QALY estimates for the three measures and 75 percent QIO attribution is $3,577. The sensitivity analysis using the 95 percent CIs for the QALYs and 50, 75, and 100 percent QIO attribution is presented in Table 5.

Table 5. Sensitivity Analysis for Cost Per QALY Gained Through Medicare's Quality Improvement Organization Program: 2002-2004.

Measure	Attribution
	50 Percent	75 Percent	100 Percent
Low (Lower Bound of 95 Percent CI)	$7,667	$5,111	$3,833
Mean	5,365	3,577	2,683
High (Upper Bound of 95 Percent CI)	4,127	2,751	2,063

NOTES: QIO is quality improvement organization. QALY is quality-adjusted life years. CI is confidence interval.

SOURCE: Shih, A., The Commonwealth Fund, Dewar, D.M., University of Albany, and Hartman, T., Health Quality Improvement, IPRO, 2006.

Discussion

The results of this study suggest that CMS is paying $2,063 to $7,667 per QALY gained through the QIO nursing home QI program. Although there is no defined threshold to what constitutes a good value, a common range used in the U.S. is $50,000 to $100,000 per QALY (Neumann, 2004), though some have argued that a QALY may be worth $200,000 or more (Hirth et al., 2000). In this context, the QIO program appears to be a very sound investment for CMS, and by extension, taxpayers.

A significant limitation of this study is its reliance on assumptions for the proportion of improvement that could be attributed to QIO activities. Although there have been previous studies of QIO activities, these studies have primarily been of QIO interventions in the hospital setting, and have not tried to quantify QIO effect on statewide performance. Most of the studies have shown QIO effectiveness (Chu et al., 2003; Marciniak et al., 1998), though one recent study showed little or no effect of QIO activities (Snyder and Anderson, 2005). This last study, however, had serious limitations that highlight some of the challenges associated with research on QIO effectiveness. The authors attempted to evaluate QIO performance by comparing intervention and non-intervention hospitals. However, by contract design, QIOs devoted a large part of their activities to statewide activities, and therefore there were no true nonintervention hospitals (Jencks, 2005). The key challenge becomes then, when observing statewide clinical QI (Jencks, Huff, and Cuerdon, 2003), how much of it can be attributed to QIO activities? This is particularly difficult in the hospital setting because there are multiple other national programs to improve hospital care in the same clinical topics as the QIOs. For instance, the American College of Cardiology (2004), and the American Heart Association (Labresh et al., 2003) both have significant overlap with the QIOs work in the inpatient treatment of acute myocardial infarction and heart failure. By focusing on the nursing home QI project, this study ameliorates some of the problems with studying QIO impact.

Unlike the hospital setting, there were limited national programs (although there existed scattered State-based programs) to facilitate improvement in nursing home care in the clinical topics that QIOs were working on during the period 2002-2005, allowing us to reasonably make assumptions about QIO contribution to improvements in the quality measures, even without a control group. In addition, by performing sensitivity analysis, it allows policymakers to make decisions based on more conservative or liberal assumptions.

In the discussion of attributable performance improvement, it is important to note that two of the five measures that QIOs focused on did not improve, but slightly worsened. This negative change was not accounted for in the final QALY analysis, as it is not plausible that the QIOs could have worsened care. In addition, when we performed preliminary analysis, the change was minimal compared to the improvement in the other measures and the net impact on the final calculations was negligible. Nevertheless, those two measures (pressure sores and functional loss) are important clinical areas, and it is unfortunate that there was no observed improvement. It is possible that QIO work has a greater impact on certain care processes, accounting for the differential changes among measures, but this needs to be further investigated.

Other limitations relate to the quality measures themselves. Although they have been validated by Morris et al. (2003), there remains some concern about whether or not the elements from which the measures are derived, the MDS, are accurately and reliably captured. For instance, it may be that changes in the measures reflect a change in how the MDS elements are captured over time (e.g., new information systems), rather than true changes in health status of the patients. Although estimating the impact of this is beyond the scope of this article, it may warrant further exploration, as it would also impact the validity of the Medicare nursing home compare public reporting initiative.

Another significant limitation of this study relates to the application of the multiattribute utility function. The HUI2, whose weights are based on the general population, has not been similarly applied, to our knowledge, to our research population in this context. Nursing home residents tend to have multiple ailments. Therefore, their functioning on attributes assessed in the HUI2 is likely less than the healthy State. Because we utilized a healthy State to calculate the disutility associated with restraints and pain, we likely overestimated the gains from avoiding those negative outcomes. In addition, the attributes from the preference scale do not match exactly the descriptions of the nursing home quality measures. For instance, restraints in the nursing home setting do not necessarily mean loss of control of arms and legs as described in the HUI2 scale, and the HUI2 pain attribute utilized in the analysis is less severe than the pain described in the nursing home quality measure.

Although one cannot quantify the net effect of these issues in utilizing the HUI2, it would likely slightly reduce the estimate of QALYs gained. Although beyond the scope of this article, there is a need for other researchers to test other instruments for estimating QALYs in this context. Indeed, one might also use other measures of utility than QALYs, though that limits the ability to compare the results to other health care interventions.

The quality measures assess care for a given period of time (1 quarter, except for short-stay measures, which span 2 quarters). The QALY estimate does not assume that the conditions assessed (e.g., experiencing pain or being held in restraints), last the entire year, but that the prevalence of the condition remains constant over that year. Beyond 1 year, we do not assume that the improved performance levels are sustained. If the QIO activities lead to sustained improvement once the QIOs stop providing assistance, then the number of QALYs gained from CMS' investment is much larger than the estimates provided here, and the cost per QALY much lower.

Finally, the estimated costs are likely overstated. We utilized total QIO contract expenditures for the nursing home QI work, but the remeasurement period for this study ended in 2005. In addition, not all QIO contract expenditures were applied towards improving the five measures analyzed. Although these overstatements of cost are partially offset by other QIO program expenditures not incurred by QIOs (e.g., CMS program costs) the net effect of this is to understate the value of the QIO program.

It is difficult to estimate the sum impact of the limitations of this study. However, even if the estimated cost per QALY gained was actually one order of magnitude larger (i.e., 10 times the reported cost) due to overestimates of QALYs gained due to QIO activities, it would still be well within the range of what is considered a good value for health care dollars in the U.S.

Conclusions

This study provides estimates of QIO program value that allow easy comparison to other health care intervention activities. Even with conservative assumptions about QIO program impact, investment in QIO nursing home QI activities appears to be a good value for health care dollars. These results should be confirmed with additional research, and value-oriented studies should be performed of other QIO and large-scale QI activities to help inform health care policymakers.