Improving the Quality of Long-Term Care with Better Information

Abstract

Publicly reporting information stimulates providers’ efforts to improve the quality of health care. The availability of mandated, uniform clinical data in all nursing homes and home health agencies has facilitated the public reporting of comparative quality data. This article reviews the conceptual and technical challenges of applying information about the quality of long-term care providers and the evidence for the impact of information-based quality improvement. Quality “tools” have been used despite questions about the validity of the measures and their use in selecting providers or offering them bonus payments. Although the industry now realizes the importance of quality, research still is needed on how consumers use this information to select providers and monitor their performance and whether these efforts actually improve the outcomes of care.

Improving the quality of health care using clinical information is achieved either by identifying targets for quality improvement (QI) efforts or by reporting intra- or interprovider performance differences to consumers, regulators, or purchasers using accepted indicators of quality of care. QI is a means of improving clinical care in specific areas, with comparative reporting, particularly public reporting, acting as a stimulant for improvement. The rationale is that providers will be stimulated to invest in internal quality improvement efforts if they believe that consumers will choose providers based on public reports of provider quality or if they will be otherwise rewarded or penalized because of these comparisons. These two strategies can operate synergistically or be implemented independently. QI uses clinical information to gauge changes in a provider's own performance after changing some existing practices or procedures. Reports comparing providers’ performance are predicated on the assumption that the underlying comparisons are valid. Both approaches have advocates, and numerous companies, ranging from software vendors to specialized consulting groups, have emerged to support providers’ QI efforts.

The long-term care service sector is a diverse group of institutional and community-based providers but only Medicare- or Medicaid-certified nursing homes (NH) and home health agencies (HHA) are subject to uniform data-reporting requirements. In some states, however, assisted-living facilities and state and privately funded home care agencies serve many frail elderly individuals. Among nursing home and home health agency providers, both the QI and the comparative performance reporting traditions have strong advocates and are being supported both intellectually and financially by federal and state quality initiatives. Indeed, the existence of universal, mandated clinical data sets has facilitated the implementation of both internally motivated QI efforts and public reporting. In the case of home health agencies, the uniform clinical assessment tool mandated by the government grew out of an impetus to create case-specific internal and external performance measures to facilitate this integrated application of quality measures. In contrast, the uniform assessment mandated for nursing homes in 1991 was designed primarily to plan care (Morris et al. 1990; Shaughnessy et al. 2002).

In 1998 the Institute of Medicine (IOM) began a follow-up study of the progress, or lack thereof, in improving the quality of care in nursing homes (IOM 2001). A central issue in that report dealt with the adequacy of data regarding long-term care quality on which to make policy, specifically how to evaluate the relative merits of a regulatory approach to quality assurance versus an information-based approach designed to stimulate quality improvement. The IOM report recommended promoting the public reporting of information about the quality of long-term care providers but cautioned that there still were many unanswered questions about the adequacy of the data on which to base such comparisons.

This article examines the conceptual and empirical validity of the data underlying the quality measures now in use in long-term care and highlights the principal assumptions underlying the current and proposed uses. Then the article looks at the impact of quality information on the introduction of CQI efforts, including how the information is presented and used. This is followed by a review of how the public reporting of quality information has influenced long-term care consumers, their advocates, and long-term care provider organizations. Finally, the article recommends further methodological and applied research in this area.

The questions relevant to long-term care providers and policymakers that this article addresses are

• How reliable and valid are the data used to construct quality measures on which public reporting is based? Do the current measures reflect the quality of the provider or the impact of case-mix differences?

• If providers improve their care, will the outcomes actually improve?

• Are the current measures of quality consistent with consumers’ interests?

• How can we determine the “overall” best providers, and how should we establish benchmarks of quality?

Background

Assessing Nursing Home Residents

In 1984, a committee of the Institute of Medicine (IOM) began studying the quality of care in nursing homes. Led by Sidney Katz, the committee's recommendations (IOM 1986) led to the 1987 Nursing Home Reform Act (OBRA). One of these recommendations was mandating a comprehensive assessment that would provide a uniform basis for establishing a nursing home resident's care plan, or minimum data set (MDS). The rationale was the perceived inability of staff to identify patients’ needs because of inadequate training and education. The MDS was a product of the recommendations of hundreds of experts representing the academic disciplines and the professional organizations serving geriatrics, psychiatry, nursing, physical and occupational therapies, nutrition, social work, and resident rights advocates (Morris et al. 1990). The goal was an instrument to capture the basic information needed to develop a care plan that considered individuals’ comorbidities, strengths, and residual capacities. An initial version was nationally implemented in 1991, followed by a revised and larger version introduced in 1996 (Morris et al. 1997).

After universally available patient information was assembled in computerized form in 1998, it was used for policy applications and not just to drive clinical care planning (Mor 2004). Nursing home case-mix reimbursement systems, initially developed for certain states’ Medicaid programs, were refined using the more detailed data in the MDS. The resulting resource utilization groups (RUGs-III) system became the basis for Medicare's prospective payment system for skilled nursing facilities (Fries et al. 1994). The availability of clinically relevant, universal, uniform, and computerized information about all nursing home residents raised the possibility of using this information to improve the quality of the nursing homes’ care. The Centers for Medicare and Medicaid Services’ (CMS) Nursing Home Case-Mix and Quality Demonstration, which had refined the RUGs case-mix classification system, thereupon created readily usable quality indicators based on computerized data from the resident assessment instrument (Zimmerman et al. 1995). These indicators were refined, and MDS-based quality measures accounting for short-stay, postacute patients as well as the long-stay residents were created. In November 2002, the CMS mandated and began publicly reporting them, first in a six-state pilot and then nationally (Harris and Clauser 2002). The revamped quality improvement organizations (QIOs) funded by the Centers for Medicare and Medicaid Services then were assigned to work with nursing home providers to improve their quality of care (Baier et al. 2003, 2004).

Home Health Agency Outcomes

Throughout the 1990s, researchers at the University of Colorado worked with home health agencies to establish a system to monitor the quality of care for HHA patients (Shaughnessy et al. 1994). Based on the Outcome and Assessment Information Set (OASIS), both the state of New York and the Robert Wood Johnson Foundation supported a pilot test of a quality assurance system: Outcome-Based Quality Improvement (OBQI). The OASIS data describe patients’ diagnoses, medical condition, treatments, and functional and cognitive status. The participating home health agencies reviewed reports of the proportion of patients who improved or deteriorated in selected domains between their admission to the service and subsequent discharge. Data on the change in patients’ status were constructed by comparing their condition at the two points in time. In 1999, the Centers for Medicare and Medicaid Services required the OASIS as a means of uniformly recording information about all Medicare beneficiaries using a home health service. With the adoption of OASIS, the entire Medicare-certified home health care industry began to submit the required data to the CMS for the Prospective Payment System (PPS) implemented in October 2000, as well as data for monitoring quality and improvement (Sangl et al. 2005; Stoker 1998). In 2004 this system was extended to the entire nation, and now consumers can compare agencies’ QIs in local spapers, at the CMS website (http://www.cms.hhs.gov/quality/hhqi/), or by telephone.

Conceptual Issues in Quality Measurement

Quality is measured using information about individual patients’ experience (e.g., pressure ulcers) and aggregating it to determine the “rate” among all patients of a given type served by the provider. The individual data come from clinical assessments of patients that are recorded and then computerized. Measures designed to reflect the “quality” of the provider are constructed after considering a number of technical, sampling, and statistical stability and adjustment issues, as well as the conceptual issues inherent in measuring quality in nursing homes and home health agencies. Next we address several of these issues, using examples from both types of long-term care providers.

Which Aspects of Quality Are Important?

Publicly reported measures of provider quality should reflect the value that society in general, and consumers (and their advocates) in particular, attribute to various aspects of quality. When we could report only hospitals’ mortality rates or countries’ number of live births, clinicians and policymakers were disappointed that the more refined and desirable aspects of health care were ignored. Although Mukamel (1997) suggested criteria for selecting quality measures according to their utility and meaningfulness to designated audiences, the existing data tend to emphasize clinical rather than psychosocial issues. But the quality of long-term care is fundamentally multidimensional and encompasses clinical care issues, functional independence, quality of life, and patients’ and families’ satisfaction with care (Mor et al. 2003c). In the case of NHs and HHAs, despite the availability of much information about patients, consumer advocates and many clinicians do not feel that the data on the Nursing Home Compare website, which is maintained by the CMS, capture important aspects of quality. For example, it does not mention quality of life (Kane et al. 2003). In addition, although patients’ and families’ satisfaction is widely used, particularly in the nursing home industry, it has not been incorporated into a national reporting system (Castle 2004; Kane et al. 2003; Simmons et al. 1997). Finally, some critics of the OASIS data for home health do not believe that the outcome data reported capture the content of nurses’ education of families (Fortinsky et al. 2003).

Aggregated Quality Measures

Standardized, mandatory, patient assessment systems are computerized in all U.S. nursing homes (NH) and in all home health agencies (HHA) serving Medicare beneficiaries. These assessments are made by the nursing staff when the patient is admitted into the service and periodically thereafter (for HHAs, upon discharge). Only those patients cared for long enough to have two assessments are included in the calculation of an aggregated measure of provider quality. Patients who cannot change (i.e., who already have a pressure ulcer or whose functioning will not improve) are excluded (Sangl et al. 2005). Furthermore, because these aggregate measures of provider quality are based on clinical assessments made by different kinds of nurses in different facilities and agencies, the resulting quality measures may reflect differences in clinical assessment practices, such as directly asking patients about their pain (Wu et al. 2003).

Comparing Quality

Consumers using publicly reported data to compare providers are essentially asking whether their experience will be better with one versus another. As noted, basing aggregated quality measures on clinical assessments means that patients and their advocates who are comparing the performance of providers may not be able to differentiate between “real” differences between two providers and those that merely reflect differences in how the nurses in the two agencies conducted their assessments. Thus, differences in how the data are collected may undermine the validity of interfacility comparisons, which is at the heart of efforts to report providers’ performance publicly (Sangl et al. 2005).

Care versus Outcomes

Quality measures reflect providers’ performance in their administration of treatments as well as the outcomes of those treatments (Mor et al. 2003c; Sangl et al. 2005). The proportion of restrained NH residents indicates the kind of care given in the home, whereas the proportion of HHA beneficiaries who become better able to move by themselves from bed to chair is also an outcome, presumably of the patients’ natural recovery rate and the HHAs’ treatment, support, and family education. Establishing benchmarks to compare providers assumes agreement on appropriate and inappropriate care and could reveal poor quality of care. Conversely, the quality of the outcome is a measure of the clinically desirable result of the nursing home or home health care.

The kinds of treatments that may be provided vary substantially. For example, treatments of postsurgical patients pertain to wound care and recovery and differ from those for patients admitted with terminal prognoses. In any case, the universal applicability of indicators of care may be limited to evidence of effectiveness (e.g., flu shots) or consensus about inappropriateness (e.g., physical restraints). Currently, the CMS's publicly reported NH quality measures are a mixture of process and outcomes, whereas the HHA data are almost exclusively based on outcomes. Some critics have argued that both the NH's and the HHAs’ approaches overemphasize outcomes, since they fail to address important processes of care (Fahey et al. 2003; Fortinsky et al. 2003; Sangl et al. 2005). A comparison of providers according to the rate at which their patients’ function changes is intended to show the impact of rehabilitative, nursing, and medical treatments, whereas the rates of pressure ulcer incidence are presumed to reflect inadequate skin care. Whether in nursing homes or HHAs, nurses are particularly interested in indicators of performance that can be specifically associated with the interventions they provide (Rantz et al. 1996).

Validity of Quality Measures

Establishing measures of performance and interpreting their meaning to various constituencies require a shared understanding of quality. This is why so many quality standard–setting organizations have broadly representative groups reviewing performance measures of quality and why the CMS asked the National Quality Forum to recommend the final indicators of quality that would be posted on the CMS's websites for both nursing homes and home health agencies (Kizer 2001; Kurtzman and Kizer 2005; Sangl et al. 2005). Assessing provider performance, particularly that based on patients’ outcomes, implies that providers are accountable for the observed score and that the quality measure resonates with our understanding of what true quality is. The Donabedian model of good structure facilitating excellent care processes, which, in turn, produce the desired outcomes, explicitly or implicitly, informs much of the literature on quality measurement (Donabedian 1980). Many studies have examined the relationship between staffing levels (structure) and various indicators of quality (process and outcome). Harrington and colleagues reported that the performance of nursing homes with more staff is superior, but others have not found such consistent results (Harrington et al. 2000; Rantz et al. 2004a; Schnelle et al. 2004c). Most recently, Rantz and her colleagues identified those nursing homes that performed best on the CMS's publicly reported quality measures, but the medical records reviewed by her researcher were found to be unrelated (Rantz et al. 2004b). However, detailed care processes are difficult to document based only on records. Just as important, Schnelle and his colleagues repeatedly found in the facilities they studied that information in the records did not necessarily match the actual care observed by the research staff (Schnelle et al. 2004a; Simmons et al. 2002).

It is important to differentiate the validity of the aggregated providers’ measures from that of the patients’ data in the MDS or OASIS assessments. Much research points to the construct and predictive validity of the MDS data, ranging from cognition, diagnoses, ADLs, and the like (Mor 2004; Sangl et al. 2005). Similarly, several studies of the OASIS refer to the validity of the data, both in the correlation of pertinent items and the prediction of events such as hospitalization (Fortinsky et al. 2003; Fortinsky and Madigan 2004). There is far less information about the validity of the provider measures now being used, both in their relationship to other structural, process, and regulatory indicators of quality (e.g., deficiency citations from inspectors) and whether they capture the impact of real changes in patient care thought to be associated with good quality (Bates-Jensen et al. 2003; Madigan 2002; Mor et al. 2003a; Zimmerman 2003). Research on the CMS's nursing home quality measures now being publicly reported found that they were not significantly correlated and were poorly correlated to the number, or severity, of regulatory deficiencies, even when controlling for the interstate variation in regulatory “severity” (Mor et al. 2003c; Sangl et al. 2005). Furthermore, Schnelle and his colleagues observed little relationship between the indicator of MDS-based restraint quality and care processes in nursing homes, even though the high-restraint facilities revealed other kinds of poor care (Schnelle et al. 2004b).

Establishing Benchmarks or Comparison Groups

Almost all providers are compared as a group or, in some cases, against a specific standard of care. Among the issues in establishing benchmarks are whether to use different benchmarks for different types of providers (peer based), whether benchmarks should be “targets” for improvement that may change as providers improve, or whether benchmarks should be based on the observed quality distribution across providers. There is not necessarily a “right” answer to these questions. For example, establishing minimums as measured by particular quality measures may not be appropriate in all cases, since many areas of performance have no evidence-based standards that could determine a minimum (Mor et al. 2003a; Shaughnessy and Richard 2002). Conversely, relying on only empirically based benchmarks (e.g., below the median) may “institutionalize” the poor performance of providers operating at the median. Furthermore, while national benchmarks might make sense in the long run, large geographic differences in medical practice may mean that patients entering long-term care from acute care may have had different treatments in different regions of the country. For example, the large interstate variation in the use of feeding tubes among cognitively impaired residents of nursing homes is likely to affect the homes’ performance on quality indicators, ranging from weight loss to drug use (Mitchell et al. 2005).

A related issue is whether to consider regional variations in care patterns at all. For example, in markets offering alternative long-term care options, such as home health, inpatient rehabilitation, and even assisted living, a different mix of patients are admitted to and reside in nursing homes. Recent research on the prevalence of long-stay nursing home residents assessed as requiring little functional or medical support services, as well as the mix of cases, revealed substantial interstate variation between 1999 and 2002 (Grabowski and Angelelli 2004; Grabowski et al. 2004). This research confirms that observed differences in hospitalization rates are strongly related to Medicaid payment rates (Intrator et al. 2005; Intrator and Mor 2004). Thus, states and facilities with higher hospitalization rates of long-stay residents may, paradoxically, appear to be better because their patients are discharged when they become sick, whereas in other states they may remain in the nursing home (Grabowski and Angelelli 2004).

Technical Issues in Quality Measurement

Just because it is possible to construct aggregated measures that reflect providers’ performance does not mean the measures are technically sound or valid. Constructing valid measures of provider quality requires addressing issues such as small sample sizes, low prevalence, and therefore instability, as well as knowing how much difference between providers is reflected in differences in the actual care provided.

Variation in Reliability of Measurement

The reliability of the MDS and the OASIS was extensively tested in their development and implementation in the 1990s and more recently (Hittle et al. 2003; Mor 2004; Mor et al. 2003b; Morris et al. 1997). The two instruments’ items achieve reasonable to excellent levels of interrater reliability as measured by the Kappa statistic (Sangl et al. 2005).1 However, most interrater reliability tests are made under optimal conditions and may not reflect “real-world” conditions, since providers participating in such intrusive field studies tend to differ from the average provider (Mor et al. 2003b). The largest multifacility reliability study undertaken to date asked research nurses with established high levels of interrater reliability to independently assess more than 5,000 nursing home residents in 209 facilities (approximately 28 per facility). Despite the high average rates of interrater reliability recorded, substantial interfacility variation in observed reliability levels was found (Mor et al. 2003b). Just as important, the direction of disagreement was examined and found to vary both between and among the facilities in the six states that the study examined. Thus, the facilities’ Kappas were systematically lower in some states; disagreements between the raters were nonrandom; and in some facilities the raters were less likely to detect a problem like pressure ulcers or pain, whereas in others they were more likely than the research nurses to rate residents as having the clinical problem. Recent statistical analyses of these data reveal that directional bias in the data can result in significant differences in the relative quality ranking of facilities (Roy and Mor 2005).

This literature suggests that in both NHs and HHAs, more attention must be directed to training the staff in making the MDS and OASIS assessments, since the interfacility variation in reliability can undermine the validity of the aggregated quality measures. Similarly, the variation among the staff of a HHA or NH can undermine efforts to measure the results of quality improvement initiatives. Consequently, some in the home health and nursing home industry have called for more consistent training practices and commitment to high-quality data (Fortinsky and Madigan 2004; Pentz and Wilson 2001).

Risk Adjustment

Comparing providers on the basis of quality measures assumes comparable patients and similarly reliable data. Risk adjustment seeks to equilibrate the patients that the providers are serving. In addition to specifying which types of patients are included in a given quality measure, statistical regression–based approaches, or stratification, can be used to adjust risk. Stratification promotes transparency, since providers can readily identify which patients are in which stratum (Arling et al. 1997; Berg et al. 2002; Zimmerman 2003). The regression-based approach, used in all OBQI measures for HHAs, essentially compares the observed and the expected rate of the clinical event (e.g., an incident pressure ulcer), where the expected rate is predicated on what would occur were the mix of patients served by one provider like that served by the average provider (Hittle et al. 2003; Mukamel and Spector 2000). Both approaches have advocates and detractors. Stratification may result in small numbers of patients per stratum, making the resulting estimate unstable. But regression-based approaches can be very sensitive to the statistical model used and its stability (Mukamel et al. 2003).

Even when using regression-based risk adjustment techniques, the CMS's publicly reported nursing home quality measures include fewer adjusters than do home health agencies’ regression-adjusted models (Sangl et al. 2005). HHA quality measures tend to examine change from the start of service to discharge, whereas many NH measures are based on prevalence, because their residents are served for extended periods. Therefore, it is hard to identify a “baseline” status for nursing home patients, which has not already been influenced by the quality of the nursing home. For example, being bedridden is predictive of acquiring a pressure ulcer (Berlowitz et al. 2001; Mukamel and Spector 2000). However, patients may have become bedridden because of inadequate mobility care earlier. Statistically controlling for this “effect” could adjust away earlier poor care (Zimmerman 2003).

Home health agencies face a different type of risk adjustment issue, since it is well known that social support and family members’ help influence patients’ outcome or improvement. However, although current HHA outcome measures include many adjusters, they do not adjust for the adequacy of patients’ informal support. This could be relevant, as it is reasonable to assume that not all HHA patients have similar family and social support.

The inadequacy of current risk adjustment models is exemplified in research examining correlates of the CMS's NH quality measures. Using annual survey data and the quarterly quality measures, Baier and colleagues found that aggregated measures of case mix (e.g., ADL, high acuity levels) were lower among the facilities with high quality measures (Baier, Gifford, and Mor 2005). Furthermore, the study found that facilities serving predominantly Medicaid patients also were ranked high, even though numerous studies found that poor quality, lower staffing levels, more regulatory deficiencies, and a greater risk of termination from the Medicare/Medicaid programs were associated with high concentrations of Medicaid patients (Castle 2002; Grabowski and Castle 2004; Mor 2004).

Composite Quality Measures

Consumers, regulators and even payers would prefer having a single metric to measure the quality of providers (Fortinsky et al. 2000; Mukamel and Spector 2003). Nonetheless, several studies have found very little correlation among the various provider quality measures used in nursing homes (Baier, Gifford, and Mor 2005; Mor et al. 2003c; Sangl et al. 2005; Stevenson and Studdert 2005). A recent report commissioned by the Medicare Payment Commission to study the consequences of offering prospective payments for HHAs acknowledged similarly low correlations among HHA measures but nonetheless created a single quality summary score for the existing HHA measures (Outcome Concepts Systems 2004). When analyzing the data, the authors observed offsetting effects on the composite measure; that is, providers performed very well on one measure but poorly on another, resulting in a finding of no effect, which is one of the dangers of combining uncorrelated measures.

Selection and Provider Specialization

One difficulty of comparing providers is that some types of providers offer a different mix of specialty services and therefore attract different patients. Much of the literature documents how hospital-based HHA or NH providers differ from those without a hospital affiliation (Fortinsky et al. 2003; Mor 2004; Zinn, Aaronson, and Rosko 1994), and the influence of specialty care units in nursing homes also has been well documented (Banaszak-Holl et al. 1997; Zinn and Mor 1994). Analyses of the characteristics of nursing home patients at the time of their admission reveal substantial interfacility variation in the proportion of patients with a preexisting pressure ulcer, lending credence to the notion that facilities may have a reputation for special competence in this area (Mor et al. 2003a). Obviously, geographic proximity has an enormous influence on the facility chosen, but the provider's specialization is important as well.

Experience with Long-Term Care Quality Improvement Efforts

Almost from the beginning of the design and testing of the OASIS and the MDS, investigators and providers tried to use the information to influence practice for both individual patients and organizations. The MDS was designed to facilitate care planning with “resident assessment protocols” (RAPs) to identify clinical areas of care possibly requiring extra attention (Hawes et al. 1997; Morris et al. 1990). The aggregation of some RAPs to the level of the NH could be used to identify the most common clinical problems. In the case of home health agencies, the outcome-based quality improvement approach was built into the patient documentation process (Kramer et al. 1990; Shaughnessy et al. 1994; Shaughnessy, Crisler, and Bennett 2000). Nurses record patients’ functioning and clinical condition at their admission and then again at their discharge. Any changes in condition could be attributed to the care provided, up to and above the natural rate of improvement expected for HHA patients.

Since OASIS was designed explicitly with outcome measurement and agency feedback in mind, early evaluations of the introduction of OASIS focused on responses to the reports summarizing each agency's outcome performance relative to the group averages. Shaughnessy and colleagues undertook a series of interrelated demonstration and evaluation projects as they continued to refine the conceptualization and measurement of home health care outcome–based quality (Shaughnessy et al. 1995, 2002). Their evaluation revealed a significant reduction in the rate of hospitalization and in the risk-adjusted rates of improvement in the OBQI target outcome measures of health status in both demonstration trials (p < .05) when compared with similar HHAs (Shaughnessy et al. 2002).

Beginning in 1990, the first set of quality indicators derived from the MDS began to be developed and tested under the six-state Nursing Home Case-Mix and Quality Demonstration (Zimmerman et al. 1995). Building on the MDS's universal implementation and computerization, government regulators anticipated that creating indicators of nursing homes’ performance would guide and enable more systematic regulatory oversight. The more enlightened administrators felt that such information could improve their own facility's quality, and advocates thought that making this information available would create greater “transparency” to guide consumers’ choices of a long-term care facility (Mor et al. 2003c).

In the late 1990s, the Center for Medicare and Medicaid Services expanded its commitment to using quality indicators to improve the quality of nursing homes. First, the CMS tried to improve and expand the existing quality indicators (Berg et al. 2002). The CMS also devised measures to respond to the quality-of-life concerns of long-term care facility residents regarding the quality of food and their preferences, autonomy, and perception of treatment with respect, but the CMS soon recognized that these measures were still in the early stages of development.

In November 2002 the CMS applied a set of indicators to the entire country. A new set of chronic, long-stay, as well as postacute, short-stay, quality measures were promulgated in January 2004. Some of the existing measures were dropped while new measures were added based on a review by the National Quality Forum (NQF) (Kizer 2001). As part of this rollout, CMS reinforced its efforts to involve the quality improvement organizations (QIOs) in stimulating providers to improve their performance. Almost all the states’ QIOs have now created or adapted quality improvement training materials for the nursing home industry (Kissam et al. 2003).

Unfortunately, despite the many studies describing the scope of quality improvement activities in nursing homes, there have been few systematic evaluations of their impact (Bates-Jensen et al. 2003; Berlowitz et al. 2003; Lee and Wendling 2004). Several surveys of facilities’ QI programs revealed them to be limited to nonexistent (Lee and Wendling 2004). Saliba found relatively low adherence to pressure ulcer prevention guidelines in a sample of Veterans Administration facilities (Saliba et al. 2003), and Berlowitz and his colleagues documented considerable variation in the extent of QI implementation in the prevention of pressure ulcers, with greater efforts noted in those nursing homes emphasizing innovation and teamwork (Berlowitz et al. 2003).

In a series of applied studies to train nursing homes to use quality indicators as the stimulus for improvement, Rantz and her colleagues observed similar results in facilities in Missouri (Rantz et al. 2001, 2003; Wipke-Tevis et al. 2004). Their efforts began with a randomized trial of more than 100 facilities exposed to either training or quality measure feedback and consultation. They found no significant improvement, which resulted in their efforts to strengthen the intervention and to identify predictors of successful implementation (Rantz et al. 2001). While several studies have documented improvement following the introduction of specific QI interventions, these studies have generally used highly selective facilities (Baier et al. 2003, 2004). Given the difficulty of implementing and sustaining improvement, some have concluded that the success of the quality improvement movement in nursing homes is predicated on leadership that is ill prepared to implement these innovations (Schnelle, Ouslander, and Cruise 1997). Indeed, one of the main recommendations of the Institute of Medicine's report on long-term care quality was to enhance managerial capacity in nursing homes in order to improve quality (IOM 2001).

The Impact of Public Reporting

As noted, in 2002 CMS released Nursing Home Compare as a national resource for consumers, their advocates, and providers to compare, with state and national averages, facilities’ most recent survey and certification inspection reports as well as their MDS-derived quality measures. In 2004 the CMS released a national version of Home Health Compare, which performed a similar function. Both Nursing Home and Home Health Compare report only a subset of all the measures developed and tested over the years (Berg et al. 2002; Shaughnessy et al. 2002; Zimmerman 2003). In addition, numerous states have assembled their own Web-based “report cards” summarizing the quality of nursing homes using different ways of presenting the information (Castle and Lowe 2005; Harrington et al. 2003; Mattke et al. 2004).

Although there is evidence that interest in this kind of quality information is substantial, according to the number of Internet site “hits” and the attention of several states, we do not know who uses this information and whether, or how, it informs or influences consumer decision-making. Indeed, it is not even clear who is looking at the websites. Several reports suggest that in regard to acute care, the public reporting has attracted the attention of more providers than consumers, although large employers have been somewhat more sensitive to using the health plans’ reports of quality (Chernew et al. 2004; Hibbard and Pawlson 2004; Hibbard, Stockard, and Tusler 2003). In addition, there is evidence that reports of the quality of hospital and health plans have only slightly altered practice patterns, choice, and perhaps even the quality of care provided (Mukamel and Mushlin 2001; Mukamel et al. 2000; Romano and Zhou 2004).

The audiences for public reports of long-term care providers’ performance include elderly consumers and their family members, but hospital discharge planners might be the most important audience (Potthoff, Kane, and Franco 1997; Sangl et al. 2005). Most patients are admitted to HHAs or NHs directly from a hospital (Intrator and Berg 2002). Hospital stays are short, focused almost exclusively on medical or surgical treatments; discharge planning is often just an afterthought. Decisions about the postacute setting or provider are characteristically made hastily with insufficient knowledge about the patients’ prognosis and the anticipated duration of care needed, and virtually no knowledge about the quality of available alternative providers. Bowles and colleagues recently reported that shorter hospital stays have affected nursing activities associated with discharge planning and postacute care for older adults (Bowles, Naylor, and Foust 2002). Indeed, one review found that predischarge assessment, education, and appropriate follow-up reduced readmission by 12 to 75 percent (Benbassat and Taragin 2000). Furthermore, a systematic meta-analysis found that organized discharge planning that included specific mechanisms to effect the transfer of the treatment plan was associated with a variety of positive patient outcomes (Richards, Coast, and Peters 2003). However, a recent survey of discharge planners in California hospitals revealed that they rarely considered data on the quality of nursing homes (Collier and Harrington 2005). Since part of discharge planning is finding an appropriate postacute discharge venue, having information about the relative quality of long-term care organizations could reduce rehospitalizations.

The efforts made by QIOs around the country to direct hospital discharge planners to the Compare websites have apparently been only somewhat successful. A project in Rhode Island designed to examine hospital discharge planners’ interaction with patients and families when considering postcancer surgery placement options discovered that discharge planners did not know about, and did not feel that they had time to explain, the various options to patients and their families (Bourbonniere, Mor, and Allen 2003). Furthermore, anecdotal evidence from the results of QIO efforts in various areas around the country reveals that discharge planners and their hospital employers have little incentive to make selecting the discharge setting easier, since their primary goal is to discharge patients quickly.

Although there is little information about the response to public reports of nursing home quality, there is even less information about home health care agencies’ response to the public reports of their performance. Many of the same issues are pertinent to both nursing homes and home health agencies, particularly discharge planning, since most markets include multiple HHAs from which discharge planners and patients must choose.

Information about the quality of nursing homes and home health agencies has been reported publicly for only a few years. Since the public continues to trust the opinions of friends and family about the choice of their physician and hospital more than most other sources, perhaps as families begin to accumulate experience with long-term care decision-making, they will become increasingly aware of the availability of public reports (Kaiser Family Foundation and the Agency for Health Research and Quality 2000). This relatively inefficient approach parallels how consumers choose their health insurance plans, their hospitals, and their physicians, so why should it be different for long-term care providers? Because most Americans try not to think about requiring long-term care, it is unlikely that they would browse websites linked to the CMS Compare sites. Rather, most Americans will encounter long-term care services following a hospitalization or similar medical encounter, either for themselves or their parents. This means that consumers must rely on professionals to find out about the alternatives and to help them choose. Even in a planned “elective” admission for a hip or knee replacement, patients and families are likely to assume that the admitting physician directs the hospital admission and the postacute recovery program. Consequently, since only a third of admissions to NHs or HHAs are directly from home, publicly reported quality information may have only a limited impact on consumers’ choice of provider unless hospitals become more proactive (Decker 2005).

Gaps in Research Knowledge

Although long-term care has, in many ways, leaped over the public reports of hospital and physician quality by having adopted uniform clinical measurements, substantial gaps remain in our knowledge about the quality of existing measures, how they are reported, how to get the designated audiences to use the information, and whether and how providers can institute quality improvement programs. Improving the quality of information about providers is one area of research with both technical and conceptual gaps. Conceptually, we need to know what consumers value and what kinds of information about providers they want. Technically, we need workable models for systematically handling measurement errors that may be confounded with true quality differences and better ways of handling small samples, rare events, and instability. Operationally, we need to know who uses and would use quality performance data and whether the mode of presenting the information and the context in which it is placed would enhance its utility to consumers and their advocates. Finally, we need to understand better the implications of establishing clinically relevant performance benchmarks—not relative to statistical averages or rankings of providers—for consumers’ and providers’ understanding of the information.

Valuing Quality of Life versus Areas of Quality of Care

Monitoring the quality of long-term care using OASIS- or MDS-derived performance measures necessarily limits the areas of quality reported to the public. Information about quality of life, autonomy, and residents’ satisfaction is not currently available from either universally available instrument (Mor 2004; Sangl et al. 2005). However, some argue that it is precisely these dimensions that are of greatest concern to consumers and their advocates (Kane et al. 2003; Kane et al. 2004). Measures of quality derived from a clinical tool are necessarily based on values different from those of the consumer. An updated version of the MDS is now being designed for nursing homes that is supposed to reflect recent research on residents’ quality of life (Kane et al. 2003). Future testing of a revised MDS that includes the residents’ “voice” should address the fundamental issue of how to obtain unbiased information about residents’ views about staff, food, and autonomy, particularly if staff members are asking the questions. These issues are equally important to home care, and the complications of obtaining the information are at least as great, because home care workers cannot ask recipients of HHA services about their “satisfaction” with the care they receive or whether they have unmet needs. While much research has been on nursing home populations, almost none has focused on these issues in home health agencies. In sum, we may end up having to obtain the family members’ perspective, as is done for hospice care (Teno et al. 2001a, 2001b).

Another area of quality that is often mentioned but little studied as an indicator of quality is consumers’ satisfaction with their experience as a recipient of care (Kane et al. 1997, 2003). Numerous resident satisfaction instruments have been developed and are being routinely fielded by chains as well as states to assess the preferences of their “customers” (Lowe et al. 2003). The CMS has been pushing for the development of a modified consumer assessment measure that can be applied to nursing homes that is based on the one used for health plans (Carman et al. 1999). The quality-of-life research by Kane and colleagues also addressed consumer satisfaction, and other investigators have developed and tested their own consumer satisfaction surveys (Castle 2004; Kane et al. 2003).

In addition to the possible mismatch between clinical performance measures and those that might interest consumers, consumers (and purchasers) would like to know which the “best” overall provider is. The recent emphasis on “pay for performance” requires that several metrics of quality be reduced to a single dimension on which to base the financial incentive (Goldfield et al. 2005). However, existing performance measures are clearly multidimensional (Mor et al. 2003c). We know that NHs performing best on one measure might be performing poorly on another and suspect that this is likely the case among HHAs (Rantz et al. 2004b). Indeed, in a recent study comparing the quality performance of Veterans Administration and community nursing homes, Berlowitz and his colleagues concluded that since nursing homes’ performance was not correlated across multiple quality measures, purchasers would not be able to use the data to make decisions (Berlowitz et al. 2005). Whether consumers and their advocates are able to understand this and to identify those measures of greatest interest to them in choosing a provider is a very important research question that must be addressed. Similar problems face those people educating consumers to properly interpret information about the quality of health plans and hospitals (Shaller et al. 2003; Sofaer et al. 2000).

Coping with Measurement and Statistical Complexity

While both the MDS and the OASIS have been subjected to a great deal of reliability testing, and both instruments, under volunteer “test” conditions, perform reasonably well in the items’ interrater reliability, recent research reveals that even acceptable levels of reliability still allow for systematic bias in the direction of the errors (Roy and Mor 2005). This is consistent with evidence from analyses suggesting a consistent underassessment of pain and depression (Miller et al. 2002; Wu et al. 2003, 2005). Because this is likely a universal issue associated with clinical administrative data on which measures of provider quality are based, generalized strategies are needed to audit the reliability and directionality of “disagreements.” Statistical models also are needed to use the results of these audits to adjust quality measures for biased measurement error, since it would be highly counterproductive to penalize providers who conduct more thorough assessments. Using statistical analyses of large-scale reliability data, Roy and Mor (2005) proposed a statistical model that could address this problem in conjunction with an audit, but more work is required to generalize this approach.

The Impact of the Public Reporting Format

The format in which information about provider quality is presented has become a lively area of research over the last several years (Hibbard and Peters 2003; Shaller et al. 2003). Hibbard and Peters tested formats for information about quality and found that they dramatically changed consumers’ perceptions of the importance of the information (Hibbard and Peters 2003). In both Nursing Home and Home Health Compare, the CMS presents the actual rates of the performance measures. Consumers can compare the rates of a particular provider with all others in the state and with the national averages. However, there is limited guidance regarding the meaning of the differences in rates between a provider and either state or national averages. Acceptable performance measure rates are not defined, and how much departure from the average, or the top, is meaningful is not explained. Furthermore, the stability of a measure is not indicated, particularly for small facilities with relatively few patients contributing to the performance measure. Even though both NH and HHA Compare have minimal sample sizes, the stability of a measure based on only 20 observations is questionable (Mor et al. 2003a).

Several states that have invested in nursing home reporting systems of their own have adopted a different perspective, which was summarized by Mattke and his colleagues (Mattke et al. 2003). They identified numerous deficits in these sites related to the ease of understanding the content and the ease of navigating the website and accordingly tried to avoid these pitfalls in designing and testing a site for the state of Maryland. Rather than using the actual rates for each quality measure, they divided facilities into the top 20th percentile, the bottom 10th percentile, and the remainder. They also chose to use more quality measures but then grouped them into clinical care domains, with a count of the number of measures in each domain that fell into each of the three classes. While giving consumers and purchasers the actual rate may be desirable, we do not know whether this approach is the best for this target audience or whether a simpler format that identifies facilities that perform better or worse than expected would be better (Marshall, Romano, and Davies 2004). But this approach would require that experts and advocates agree on the approach to determining “better” or “worse,” since the Maryland model uses an empirical distribution to identify good and poor facilities, an identification that can be problematic if most providers do not do well in some areas.

Summary

The adoption of uniform, clinically relevant patient information systems for both nursing homes and home health agencies has already begun to transform these industries. Not only do they provide the basis for a common clinical language, they also form the groundwork for two interrelated initiatives designed to improve the care of long-term patients. By feeding back quality performance data to provider organizations, leaders at all levels can begin examining and changing their current practices to reduce the occurrence of undesirable clinical events and to increase the rate of functional improvement. This impetus, which may be willingly adopted by only a minority of providers in each industry, is reinforced by reporting the same information to the public and the providers’ local competition (Castle 2001; Crisler and Richard 2002; Lucas et al. 2005; Zinn, Weech, and Brannon 1998). Spurred by either competition or fear of what consumers might find out about them on public websites, providers have signed up for their state's quality improvement initiatives (Lee and Wendling 2004). Nursing home chains also are using some of these quality improvement approaches internally and are using competition among their different subunits or facilities to stimulate action (Mukamel and Spector 2003). This is not to say that all this will necessarily improve the care offered by the average NH or HHA, nor will it necessarily affect the bottom tier of facilities, since they are unlikely to be able to make the needed organizational changes (Mor 2004). But the providers, though worried, appear to be more energized and are beginning to feel that they have the tools to make the changes needed to improve the quality of their care.

The research community and the government have a responsibility to make sure that the technical aspects of the quality measures being used to compare NH and HHA providers are up to the challenge of being used both to stimulate the organizational changes needed to redesign care processes and to allow for legitimate and valid comparisons across providers. The current crop of measures, albeit a great improvement over the limited validity of the admittedly idiosyncratic survey and certification process, continue to leave much to be desired (Sangl et al. 2005). While they appear to be reliably measuring quality in certain areas, the measures cannot capture a global notion of quality. Furthermore, problems with the consistency of measurement across providers may undermine the legitimacy of the comparisons for which these measures were created. There is evidence that this is the case in nursing homes, but the research on home health care has not even begun. Nonetheless, we should not stop the public reporting or other uses of these quality measures simply because they continue to have significant deficits; rather, we should treat them as merely one other product that should be continuously improved.