Developing Dialysis Facility-Specific Performance Measures for Public Reporting

Abstract

The Balanced Budget Act (BBA) of 1997 directed CMS to implement a system to measure and report the quality of dialysis services under Medicare by 2000. Because of this tight timeframe, a rapid-cycle measurement development process was initiated to develop dialysis facility-specific measures that could be released to the public. The result was “Dialysis Facility Compare” which has served as a template for the development of public reporting initiatives for other providers in the Medicare Program. This article describes the process used for developing and reporting these performance measures and the lessons learned for future work in this area.

Introduction and Background

Public reporting of provider-specific performance with respect to quality indicators has an important potential role in promoting quality in health care. In recent years, CMS has initiated public reporting efforts related to a number of health care settings in order to assist consumers in making health care decisions as well as to drive population-based quality improvement efforts. These initiatives, stimulated in part by the BBA, have focused on practice settings such as nursing homes and on health care delivery mechanisms such as Medicare managed care plans. In addition, a number of channels, including print materials, telephone helplines, an interactive Internet information database called “Medicare Compare,” and State and community-based outreach and education programs, have been utilized by CMS in disseminating health care quality information to Medicare beneficiaries in particular (McCormack et al., 2001).

CMS, through the Medicare Program, is the predominant payer for renal dialysis services, which were utilized by more than 270,000 Americans in 2000 (Centers for Medicare & Medicaid Services, 2001a). Medicare expenditures in 1999 for this patient population were more than $10 billion (U.S. Renal Data System, 2001). Also, as the Federal agency that operates the Medicare Program, CMS is responsible to the public for ensuring that an acceptable quality of care is delivered to Medicare beneficiaries.

The BBA required CMS to develop, by not later than January 1, 1999, and implement, by not later than January 1, 2000, a method to measure and report the quality of renal dialysis services provided under the Medicare Program. In order to meet this ambitious timeframe, CMS implemented a two-stage plan to respond to this requirement. First, it enhanced an existing system, known as the End Stage Renal Disease (ESRD) Core Indicators (CI) Project that focused on reporting renal dialysis quality performance on a regional and national level using data collected by dialysis facilities through medical record abstraction. Second, it began a new effort to provide dialysis facility-specific information to the public using existing CMS administrative data. These two initiatives were developed independent of each other. In this article, we briefly summarize the enhancement of the ESRD CI Project, and describe in detail the process leading to the development of facility-specific public reports that culminated in “Dialysis Facility Compare” on the www.medicare.gov Web site hosted by CMS.

Enhancing Existing ESRD Reporting System

As a first step in responding to the BBA mandate, CMS funded the development of renal dialysis clinical performance measures (CPMs) based on the National Kidney Foundation's Dialysis Outcome Quality Initiative (NKF-DOQI) clinical practice guidelines (PRO-West, 1999). Since 1994, CMS has had in place a national renal dialysis surveillance system that has annually documented sustained and important improvements in dialysis care (Centers for Medicare & Medicaid Services, 2001b). However, the indicators used in that system, the ESRD CI Project, had been based generally on expert opinion rather than scientific evidence (Health Care Financing Administration, 1996; McClellan, Frederick, Helgerson, et al., 1995). The publication of the NKF-DOQI Clinical Practice Guidelines in 1997 was the result of an intensive 2-year effort by renal care experts to review the literature related to four key areas of importance to dialysis care and to establish clinical practice guidelines based on the best available scientific evidence. The four areas of dialysis care included (1) adequacy of hemodialysis (HD), (2) adequacy of peritoneal dialysis (PD), (3) vascular access, and (4) management of anemia (National Kidney Foundation Dialysis Outcomes Quality Initiative, 1997).

In April 1998, CMS contracted with PRO-West, a Seattle-based private, non-profit health care quality improvement organization, to develop renal dialysis CPMs based on the NKF-DOQI clinical practice guidelines that could be used for quality improvement efforts. The work was done with the participation of a broad range of stakeholders from the renal community. By the end of 1998, 16 CPMs were developed to assess performance in the four areas previously listed. Many of the newly developed CPMs were similar, if not the same, as the indicators used in the ESRD CI Project. The collection of data to calculate the CPMs was pilot-tested in 1999 using similar sampling and collection methodologies as used in the ESRD CI Project: requesting dialysis facility staff to abstract information from selected patients' medical records on a national and regional random sample of adult in-center hemodialysis and a national random sample of adult peritoneal dialysis patients. Also in 1999, CMS's national renal dialysis surveillance system (ESRD CI Project) was merged with the ESRD CPM Project and the system is now called the ESRD CPM Project (Health Care Financing Administration, 1999).

The ESRD CPM Project allows CMS to measure and report the quality of dialysis services in the area of adequacy of dialysis (HD and PD), anemia management, and vascular access on a population basis, nationally and regionally. Each year, CMS reports the project's findings to the public in its Annual ESRD CPM Project Report. Included in each year's annual report is a detailed list and description of all the CPMs. The latest report (December 2001), as well as prior year reports, can be found on CMS's Web site.

Although the ESRD CPM Project, and the ESRD CI Project prior to 1999, has allowed CMS to report the quality of dialysis services on a national and regional basis, the system was not designed for reporting at the dialysis facility-specific level. On average only three patients per dialysis facility are selected for inclusion in the annual data collection effort, a sample size far too low for reporting at the facility-specific level. A major reason for this circumstance is that the data are collected retrospectively through intensive medical record review and recorded on hard copy data collection forms. While it is possible to conduct an intensive review on a sample of dialysis patients, the cost and resources to complete this type of review on every dialysis patient would be prohibitive. Although the national and regional findings have been associated with sustained, significant improvements in the quality of dialysis care across the country, they have not provided information that assists consumers in evaluating care at a facility-specific level.

Since 1994, renal dialysis providers have been reporting clinical information on their patients to their ESRD Network via the ESRD CPM Project. ESRD Networks are regional organizations contracted by CMS to perform quality oversight activities to assure the appropriateness of services and protection for dialysis patients. The dialysis facilities, with the help of their network, use the CPM project's findings to benchmark their own performance and to identify opportunities for improvement.

Also, since Medicare is the largest payer of dialysis services, CMS maintains administrative data (claims as well as other descriptive information) on these patients and on all Medicare-approved dialysis facilities in its ESRD Program Medical Management Information System (PMMIS) in order to operate the Federal ESRD Program. In addition, this administrative database includes patient level data for two clinical areas, adequacy of hemodialysis and anemia management that can be used to calculate facility-specific rates. The data collected for the CPM Project are too sparse to permit facility-level reporting. Thus, other than instituting a new data collection burden, the only viable source for facility-level information is the use of existing Medicare claims or administrative data.

Facility-Specific Measures

Development Effort

In 1999, CMS implemented a new project independent of the ESRD CPM Project discussed previously, to address the public reporting of facility-specific measures in response to the BBA directive. It again contracted with PRO-West to facilitate the development of dialysis facility-specific measures that could be released in reports to the public for their use in making dialysis treatment choices. Measure selection was to include descriptive as well as quality information related to dialysis patient care processes and/or outcomes as directed by the BBA. Because this activity was largely driven by timeframes established by the BBA, CMS decided that the initial set of measures to release to the public would be based on dialysis facility-specific data already captured by CMS or available to CMS primarily from its administrative databases. The data collected for the CPM project could not be used as the source for any facility-specific quality measures because individual patient data are not collected in sufficient numbers per facility to permit valid facility-level reporting. In the following sections, we describe the process that was used to develop and to report the selected renal dialysis facility-specific performance measures.

Process

Establishment of an Expert Panel

In order to meet the timeframe required by the BBA, it was necessary to complete the developmental process for selecting the measures to be publicly reported in a 12-month period. It was clear to CMS that buy-in from the renal community would be essential to the success of this process. Therefore, in January 1999, prior to beginning the development work, CMS sponsored an Information Needs for Accountability and Consumer Information in Dialysis: Reconciling the Challenges conference in Baltimore, Maryland. Representatives from the major renal organizations, the ESRD Networks, and other interested parties were in attendance. The primary objectives of this conference were to engage the renal community, to provide a forum for discussion and exchange of information, and to develop a partnership to plan and implement the measures development process. The challenge was to balance the need for deliberative, technical input to the process, with the need for communication and feedback from stakeholder groups.

As a result of this meeting, two key groups were established by PRO-West to assure appropriate input by the renal community into the development process. The first group, known as the Stakeholders Council, was formed to create a mechanism by which representatives from a broad spectrum of interests could provide comment and feedback regarding the identification and selection of measures to report. Members of the Council were identified through invitations to professional societies, advocacy organizations, dialysis chains, health maintenance organizations, trade groups, members of other government agencies, and key individuals with an interest in ESRD. Throughout the course of the project, approximately 39 people representing 30 organizations participated on the council.

The second group, known as the Consumer Information Workgroup (subsequently referred to as the workgroup), was convened by PRO-West to serve as an expert panel to identify specific measures to include in the public reports that could be recommended to CMS. Nominations for the workgroup were solicited from the organizations represented on the council and through a systematic process intended to identify the needed expertise for the project. Workgroup members were asked to serve not as formal representatives of organizations with which they were associated, but as technical experts responsible for balancing a complex set of potentially competing priorities necessary in creating a measurement set. The workgroup, which ultimately comprised 11 members, provided the perspectives of nephrologists, nephrology nurses, facility administrators, social workers, ESRD Network representatives, researchers, patients, and patient advocates. From time to time, the workgroup received consultation from experts in other disciplines who were experienced with issues related to public reporting of health care information.

Measures Development and Selection

In June 1999, the workgroup met to begin developing the facility-specific measures. The first task of the workgroup was to identify the desired attributes of performance measures and to determine which of these attributes should be given priority for public reporting. After conducting a review of the literature describing other projects focused on reporting information to consumers, and considering presentations by various experts on topics ranging from the reliability of existing dialysis facility-specific data to lessons learned from other health care reporting projects, the workgroup agreed on a set of principles that would guide their work. Based on the constraint that the measures to be reported needed to be mined from existing CMS data, the workgroup understood that the data might not be available for selecting measures that are ideal from both a scientific and a consumer needs perspective.

The workgroup decided that each reported measure must be relevant, practical, and comprehensible to consumers. In creating the measures, the workgroup considered how to balance validity, reliability, accuracy, timeliness, conciseness, and usefulness of measures to consumers. In addition, the workgroup agreed to consider issues such as potential unintended consequences of reporting specific information, periodicity of reporting based on data availability, and methods which might increase the meaningfulness of data to consumers beyond simply providing rates or percentages.

In preparing to identify potential measures, the workgroup considered published literature related to the consumer perspective on information about dialysis facilities and listened to reports from several workgroup members who had conducted their own limited patient focus groups in an attempt to identify what dialysis patients want to know about dialysis facilities. The workgroup concluded that these sources suggested that, absent information, patients assumed that high quality technical care was being provided at facilities. Dialysis patients also placed a high priority on descriptive information about facility characteristics, amenities, policies, and staff qualifications and numbers (Rubin et al., 1997). Based on these considerations, the workgroup developed a list of approximately 90 topics from which candidate measures could be selected, although initially little emphasis was placed on the quality of existing data and technical feasibility. The list was reduced to 60 topic areas, as 30 topics were consigned to a list for future consideration after followup work showed that existing data to create the measures were not available.

A key element of the process was to reach out broadly to the renal community to get input on potential measures. In June 1999, simultaneously with the workgroup's internal development of measures, a Call for Measures was issued to 250 organizations and individuals comprising a broad cross-section of the renal community. Approximately 100 forms were received with recommendations ranging from requests to report facility addresses to recommendations for publishing patient-generated quality of life information for each dialysis facility. While the vast majority of the responses proposed measures consistent with those independently developed by the workgroup, each response was reviewed by the workgroup in light of the previously described target attributes for the measures.

At this stage, the list was narrowed to 40 topic areas by consensus. A nominal voting process was then used to refine this list to 15 candidate measures (Table 1). The list of potential measures included facility descriptive information, staffing information, percent of patients awaiting transplant, and three quality measures related to adequacy of hemodialysis, anemia management, and patient survival (mortality).

Table 1. List of 15 Candidate Measures¹ as Refined by the Consumer Information Workgroup.

Candidate Measure

Facility name and address. Facility telephone number. Name of facility manager. Facility ownership or management type. Is the facility owned or managed by a chain organization? Name and address of chain home office. Facility's Medicare certification date. Does facility offer shifts starting on or after 5 p.m.? Number of dialysis stations. Types of treatment offered. Patient to staff ratio. Percentage of patients awaiting transplant. Percent of the facility's patients who received adequate dialysis. Percent of the facility's patients treated for anemia (low blood count) whose anemia was adequately managed. Actual compared with expected patient survival.

¹These measures were identified by a Consumer Workgroup lead by PRO-West as recommendations to the Centers for Medicare & Medicaid Services for inclusion in dialysis facility-specific public reports.

SOURCE: (PRO-West, 1999.)

Because the list of 15 candidate measures was derived primarily by a nominal process, the workgroup agreed to majority vote by written ballot in recommending items on which there was not unanimity among members. During the workgroup's deliberations, the members debated the suitability of the three proposed quality measures for public reporting (percent of the facility's patients receiving adequate hemodialysis as measured by a urea reduction ratio [URR] of 65 percent or greater, percent of the facility's patients on erythropoietin whose anemia was adequately managed as measured by a hematocrit of 33 percent or greater, and the facility's patient survival rate based on its standardized mortality ratio). The ballot also included questions asking the respondent to recommend whether the quality measure, if reported, should be risk adjusted for any patient characteristics, or for any other characteristics.

The workgroup did not endorse risk adjustment for the adequacy of dialysis (URR) and the anemia management (hematocrit) quality measures. The workgroup also voted to recommend that CMS not report survival (mortality) rate in a continuous manner, but to report a facility's patient survival rate as “lower than expected,” “same as expected,” or “higher than expected” based on specific statistical cut points.

The deliberations of the workgroup were often difficult. There is no gold standard for determining how to balance concerns of validity, reliability, timeliness, and value to the public in making these choices. As a result, the workgroup did not reach unanimity on the initial quality measures proposed. CMS was also advised by PRO-West to carefully consider whether to report publicly these measures that had not been universally endorsed by the workgroup. However, a majority of the members did vote to proceed with the three measures in the initial publicly reported data, acknowledging that this is only a first step, with the need for continued improvement and evolution of measures and measurement science in the ESRD field.

The draft list of 15 proposed dialysis facility-specific measures developed by the workgroup was submitted to the council for review and comment. In November 1999, PRO-West submitted the draft list of proposed measures with comments to CMS for consideration. Additional comments from the council were also submitted to CMS. After a review of the comments and the proposed facility-specific measures, CMS accepted 14 of the 15 measures. The name of the facility manager was not accepted due to privacy concerns. CMS then worked with PRO-West and the University of Michigan Kidney Epidemiology and Cost Center (UMKECC) to create the database that would be used to populate the first facility-specific reports and to prepare technical specifications or descriptions for each measure.

Dialysis Facility-Specific Measures

Developing Dialysis Facility Compare Web Site

As the workgroup was deliberating on what facility-specific measures to recommend to CMS for public reporting, CMS made the decision that the channel for reporting the dialysis facility-specific information to the public would be the Internet. Dialysis Facility Compare (DFC) was proposed as the name for a new feature on the www.medicare.gov Web site that would be modeled after Nursing Home Compare.

Following CMS's approval of the proposed dialysis facility-specific descriptive and quality measures to be included on DFC, CMS asked the workgroup to provide assistance in preparing the technical specifications and descriptions of each measure and in developing instructional information for inclusion in DFC. Web site screens displaying the facility-specific information were designed and a mocked-up computer version of DFC was constructed. CMS contracted with Seniors Research Group and American Management Systems Center for Advanced Technologies to evaluate the language that was proposed for the actual Web site and to conduct usability testing of DFC, respectively. Cognitive testing methods via one-on-one interviews were used to determine if the individuals tested would understand the dialysis facility comparative information to be communicated in the DFC Web site.

A total of 51 one-on-one interviews were conducted in two cities during the summer of 2000 (Indianapolis, Indiana and Alexandria, Virginia). Individuals who were interviewed consisted of pre-ESRD patients, new dialysis patients, patients that had been on dialysis for 1 year or more, family members of dialysis patients, and renal care professionals. The majority of the individuals interviewed found all of the measures to be important in helping to evaluate or choose a dialysis facility (Seniors Research Group, 2000). As expected, the descriptions of the measures that were the most confusing for the patients to understand were those for the three quality measures (adequacy of dialysis, anemia management, and patient survival) and the patient-to-staff ratios.

Using the information and recommendations from the work conducted by Seniors Research Group, CMS revised each measure's descriptive language as needed, made revisions to instructional information also included on the Web site, and made changes to the format of the actual screens or pages of DFC. In August 2000, five users participated in the DFC usability testing to determine if the information was clearly presented and understandable. This usability testing also validated the Web site design before the site went live by identifying missing functionality or information, assessing navigation, and assessing error prevention and recovery (American Management Systems Center for Advanced Technologies, 2000). Results from the usability testing were used by CMS to refine DFC, including textual changes as well as navigation changes.

Database Creation

As the revisions to DFC were being conducted, PRO-West worked with the UMKECC to construct the database that would be used to populate the first dialysis facility-specific reports on DFC. A variety of CMS data sources were used to populate the facility descriptive information, the 1998 Independent Renal Facility Cost Reports were used to calculate the patient-to-staff ratios and the percentage of patients awaiting transplant, the 1998 Medicare claims data were used to calculate the adequacy of dialysis and anemia management quality measures, and the 1996-1998 data from the ESRD PMMIS were used to calculate the patient survival information. (Since 1998, dialysis facilities have been required to report each patient's most recent URR [reported as a range rather than the actual URR value] when submitting a claim to Medicare for that patient's hemodialysis treatment and since 1989, they have been required to report each patient's last monthly hematocrit value when submitting a claim for erythropoetin.) All of the calculated measures were produced by the UMKECC for CMS. A detailed description of the methodology used to calculate the three quality measures can be found at the UMKECC Web site at Internet address: www.med.umich.edu/kidney.

Advanced Data Preview

By fall 2000, the proposed DFC Web site was completed and the database containing the 14 selected facility-specific measures for all of the verified Medicare-approved dialysis facilities was prepared. However, before CMS released the dialysis facility-specific reports to the public, the facilities were given the opportunity to preview their data and to submit comments about their data to CMS. Dialysis facilities' administrators were provided with a hidden Web site address where they could review their facility's data and submit comments to CMS.

More than 50 percent of the dialysis facilities included in the preview period submitted comments to CMS on one or more of their measures. In a subanalysis of the comments received, a high percentage (20 percent or more) of facilities commented that the data for some of the facility descriptive information, patient-to-staff ratio and the anemia management measure were incorrect, and/or misleading as reported or described on the preview Web site.

CMS considered all comments submitted by the facilities and made decisions to not report some of the recommended measures, such as the patient-to-staff ratio and percentage of patients awaiting transplant. These measures were not reported because of legitimate concerns about the quality of their data source—Independent Renal Facility Cost Reports. Also, the name and address of the chain home office measure was revised to report only the chain affiliation.

Because many of the dialysis providers raised valid concerns about the accuracy of the data in the source(s) used for the descriptive measures, CMS decided that it needed to find a better source for this information. CMS had been working with the ESRD Networks to implement the Network Standard Information Management System (SIMS) and by the end of 2000, this new system was fully functional in all of the 18 networks. SIMS is an electronic information system that supports the business functions of the ESRD networks and allows consistent communication between CMS and the networks. SIMS maintains current facility descriptive data, which can be quickly updated when networks are notified of changes by their dialysis facilities. Since CMS can access these data, CMS decided that SIMS would be the source for all of the facility descriptive information reported on DFC.

On January 19, 2001, DFC debuted on the www.medicare.gov Web site. Seven of the 12 facility-specific measures, including the 3 quality measures, were released for over 3,500 dialysis facilities. In April 2001, the remaining five facility-specific measures were added to DFC (Table 2). One month after DFC debuted, CMS was notified of a possible discrepancy in the anemia management measure. On further investigation, it was determined that there was a programming error in the calculation of the measure, therefore this measure was disabled on the Web site and not available until July 2001 when the three quality measures were updated using 1999 Medicare claims data.

Table 2. Twelve Dialysis Facility-Specific Measures Released by CMS to the Public on Dialysis Facility Compare (DFC).

Facility Measure		Released	Source	Description or Comment
1.	Name and address	1/19/2001	ESRD Network SIMS	Address for the physical location of the facility.
2.	Telephone number	1/19/2001	SIMS
3.	Initial Medicare Certification Date	1/19/2001	CMS's OSCAR	This date is the date associated with the facility's current Medicare provider number.
4.	Shifts starting or continuing after 5 p.m.	1/19/2001	ESRD Networks Annual Reports. SIMS for the 4/2001 DFC release.	This item was renamed “shifts starting at 5 p.m. or later” for the 4/2001 DFC release.
5.	Percent of the patients who received adequate HD (defined as a URR ≥ 65 percent)	1/19/2001	CMS's REBUS/PMMIS; includes information from Medicare claims data.	For the 01/19/2001 DFC release measures were calculated using the 1998 Medicare claims. For the 7/2001 release 1999 Medicare claims were used. The measures are prepared by the University of Michigan Kidney Epidemiology and Cost Center (UMKECC).
6.	Percent of the patients on Epoetin whose anemia was adequately managed (defined as hematocrit ≥ 33 percent).	1/19/2001	REBUS/PMMIS	For the 01/19/2001 DFC release measures were calculated using 1998 Medicare claims. For the 7/2001 release 1999 billing data were used. The rates are prepared by UMKECC.
7.	Patient survival categories: reported as expected, better than expected1, or worse than expected1	1/19/2001	REBUS/PMMIS	For the 01/19/2001 DFC release, the survival categories were a 3-year average (1996-1998). For the 7/2001 release the survival categories were a 3-year average (1997-1999). The rates are prepared by UMKECC.
8.	Ownership type	4/2001	SIMS	Reported as profit or non-profit
9.	Owned by a chain organization	4/2001	SIMS	Reported as yes or no
10.	Chain affiliation	4/2001	SIMS	Reported as the name of the national or local corporation/chain.
11.	Number of dialysis stations	4/2001	SIMS	Reported as the total number of stations. Self-reported by the dialysis facility.
12.	Modalities offered	4/2001	SIMS	Reported as HD, home HD training, and peritoneal dialysis. Self reported by the dialysis facility.

¹At least 20 percent better or worse than the “as expected” survival group.

NOTES: CMS is Centers for Medicare & Medicaid Services. ESRD is end stage renal disease. SIMS is Standard Information Management System. OSCAR is Online Survey Certification and Reporting System. REBUS is Renal Beneficiary and Utilization System. DFC is dialysis facility compare. HD is hemodialysis. URR is urea reduction ratio. PMMIS is Program Medical Management Information System.

SOURCE: Frederick, P.R., Centers for Medicare & Medicaid Services, 2002.

Discussion

The development of DFC represented an important milestone in CMS's efforts to increase public reporting regarding the quality of health care delivered to Medicare beneficiaries. This effort, which was conducted in close consultation with many segments of the renal community, can provide lessons for future efforts regarding other settings of care. The challenges encountered during this effort that are described in this article are not unique to dialysis, but can and should be expected in similar measurement and reporting efforts for other health care providers. This is particularly important, as CMS is now in the midst of enhancing its current efforts to report on quality of care in nursing homes, and is considering providing quality information for settings such as hospitals and home health care.

Several unique factors regarding dialysis care were significant in this project. First, as the predominant payer for dialysis care, CMS was able to efficiently engage stakeholders from across the spectrum of the renal community in a way that would have been far more difficult if a large number of other purchasers had significant investments in seeking quality information on behalf of consumers. Second, the successful precedent of national and regional level surveillance related to quality of dialysis care represented by the ESRD CPM Project facilitated acceptance of the notion of public reporting about the quality of dialysis care in a way that is duplicated in few, if any, other care settings. Third, the BBA directive for CMS to report on the quality of dialysis care substantially diminished discussion about whether the project should take place, and allowed stakeholders to focus on how public reporting should be implemented.

Nevertheless, many potential challenges confront efforts to create valid, reliable, and useful public reports regarding the quality of health care, and this project was not immune to such challenges. The aggressive timeframe mandated by the BBA precluded a more rigorous, academic approach to the development of dialysis facility-specific public reports, and constrained the type of information that could be reported. Because of the potential expense and intrusiveness associated with collection of new data, CMS was compelled to rely on existing data sources. Few, if any, of the data sources available were designed for the purpose of collecting clinical, or even administrative, data for public reporting. Indeed, most of the data systems were intended for financial, regulatory, and other administrative functions. As with nearly all such systems, it is particularly difficult to identify patient-level quality or clinical process and outcome information relevant to consumers.

Despite the data limitations, and because of the BBA directive, CMS was intent on presenting some data regarding dialysis health care processes and outcomes. There was little disagreement among stakeholders that, if ideally measured and reported, adequacy of dialysis and management of anemia are important characteristics of patient care that could reasonably be expected to represent important aspects of dialysis facility quality. However, the data sources available for these measures suffer from limitations that rendered public reporting somewhat controversial. First, the data available for public presentation described care delivered nearly 2 years before the period when the data are publicly reported. Second, it is not possible to adjust the data for individual patient factors that can influence clinical results. While the workgroup was aware of the potential benefit of risk adjustment, the majority of members felt that the value of presenting data even if imperfect, more than balanced the liabilities associated with reporting. Third, although Medicare is the primary payer for over 80 percent of dialysis patients (Centers for Medicare & Medicaid Services, 2001a), the existing administrative data sources do not capture clinical information for patients in Medicare managed care plans, non-Medicare patients, or those for whom Medicare is a secondary payer. Therefore, the data available for public reporting do not necessarily describe the universe of patients in many facilities.

In addition, because many dialysis facilities care for a relatively small number of patients, overall facility data can be unduly influenced (in either direction) by a relatively small number of individual patients. For this reason, data describing facilities with a very small number of patients (10 or fewer) were suppressed in DFC. However, this potential still exists for facilities just above the threshold size.

Some stakeholders were also concerned that, because there was no ongoing method to validate the adequacy (URR) and anemia (hematocrit) clinical data that were reported by facilities on Medicare claims forms, the accuracy of the data was highly suspect. CMS conducted a study comparing the URR ranges and the hematocrit values from dialysis patients' Medicare claims for October, November, and December 1998 to URR ranges and hematocrit values obtained on the same patients in the ESRD CPM Project from the same timeframe. The study found general, although not universal, agreement between the claims data and the CPM Project data. However, both data sources were found to yield approximately the same results when classifying patients as either above or below a threshold value (Centers for Medicare & Medicaid Services, 2002).

The survival data presented on DFC were among the most troubling to some stakeholders in the renal community. The survival measure is based on longstanding work conducted by the UMKECC. The survival or mortality methodology was developed by the UMKECC in the early 1990s while under contract with the National Institutes of Health as the U.S. Renal Data System (Wolfe et al., 1992). The methodology was developed with input from the renal scientific community and has undergone considerable development to more accurately portray mortality in dialysis facilities, including averaging the survival data over 3 years and adjusting for several patient characteristics (e.g., age, sex, race, and diabetes status) (Wolfe, 1994). The inclusion of this measure in the public reports was especially important to the consumer representatives on the workgroup.

Despite the great lengths that were taken by the workgroup to improve the suitability of this measure for DFC (i.e., presenting the measure as either better or worse than expected only if a stringent statistical test for outlier status was met), some observers strongly opposed inclusion of the survival data. Conflicting opinions about the relation of the survival data to quality of care surfaced in the academic literature after CMS made the decision to report the information (Lacson et al., 2001; Wish, 2001; Wolfe, Held and Port, 2001). Even though the current (January 2001) update of DFC lists only 2 percent (66 of 3,247) of U.S. dialysis facilities as having worse than expected patient survival, a high level of discomfort persists among some who do not believe that these data should be presented as reflecting quality of care.

CMS, the workgroup, and the contractors involved in developing the measures were well aware of the imperfection of the quality measures. In order to address these issues, DFC includes a variety of explanations, descriptions, and resources intended to assist users in interpreting the information. Indeed, a consistent message in the supporting information on DFC is that patients should discuss their own care with their physicians, and that patients, physicians, and other providers can act as a team to assure excellent clinical care for an individual patient regardless of the experience or characteristics of other patients treated at the same unit.

Some stakeholders were concerned that public reporting of dialysis facility quality data could have adverse consequences. These consequences included fears that facilities would selectively enroll patients with characteristics that would assist the facilities in scoring well on the public reports. Other observers were worried that patients would be upset if the facilities in which they were treated were not listed as high performers, particularly if theirs was the only unit in their geographic area. Other concerns expressed were that the reputations of fine facilities would be tarnished by inaccurate data, that the time lag between care delivery and data validity rendered useless any value associated with the data, and that patients would not be interested in technical measures related to dialysis adequacy and anemia management.

Despite these concerns, there is little evidence that DFC has had an adverse influence on the quality of care, or access to dialysis care, in the U.S. Indeed, during 2001, the DFC Web site experienced an average of over 30,000 page views per month. Although Federal privacy policies have precluded ongoing tracking that could be used to characterize users, each year approximately 90,000 individuals begin dialysis (Centers for Medicare & Medicaid Services, 2001b), so it is reasonable to assume that many patients and family members are also among those viewing the site. Anecdotal evidence, gained by the authors in conversations with members of the renal community, suggests that many facilities have carefully evaluated their own data on the Web site, and that some have been stimulated to enhance their internal quality improvement activities based on the reports.

The database populating DFC has not remained static. Descriptive data regarding dialysis facility characteristics have been updated monthly, and new dialysis facilities are added to the DFC database as their Medicare certification is verified. The three quality measures are updated annually with the next scheduled update planned for fall 2002. That update will reflect Medicare claims data from 2000 for the URR and hematocrit measures and 1998-June 2001 ESRD PMMIS data for the patient survival measure.

Efforts are underway to further evaluate and modify DFC. CMS is actively engaged in pilot-testing a system that will allow dialysis providers to submit data to the networks and CMS electronically, which will allow a more current collection and reporting of clinical data on all dialysis patients. It is anticipated that this system will be available for use by all dialysis facilities in 2003, and that this system will then become the source of the data that is used for calculating the quality measures that are reported on DFC.

Further evaluation, now underway, will be used to evaluate consumer response to the DFC site and its content. Based on these evaluations, future enhancements can be made to DFC to improve its usefulness to consumers.

The practice of public reporting of health care performance data is in its infancy, and many challenges remain in order to make such reports useful in assisting consumer choice as well as driving quality improvement. However, if done properly, public reporting of quality performance information has the potential of changing the dialogue on quality in Medicare. Unlike survey and certification data, performance data provides information across the entire spectrum of performance and enables beneficiaries not only to avoid poor facilities, but also to identify and select excellent facilities as well. Supplemented with a sound quality improvement program, it also provides the information, tools, and motivation for facilities to adopt best practices to improve their performance as well.

The development of DFC demonstrates that CMS working in partnership with the key stakeholders affected by the reporting activity can take important steps towards public reporting and successfully work through difficult and sometimes contentious measurement and reporting issues. The lessons learned from this experience can be used to inform future facility-specific performance measurement reporting efforts undertaken by CMS or others. Continued buy-in and collaboration will be essential to the continuing efforts of CMS to advance quality and consumer choice through public reporting of health care facility performance.