Abstract
BACKGROUND
Public reporting of provider performance can assist consumers in their choice of providers and stimulate providers to improve quality. Reporting of quality measures is supported by advocates of health care reform across the political spectrum.
OBJECTIVE
To assess the availability, credibility and applicability of existing public reports of hospital and physician quality, with comparisons across geographic areas.
APPROACH
Information pertaining to 263 public reports in 21 geographic areas was collected through reviews of websites and telephone and in-person interviews, and used to construct indicators of public reporting status. Interview data collected in 14 of these areas were used to assess recent changes in reporting and their implications.
PARTICIPANTS
Interviewees included staff of state and local associations, health plan representatives and leaders of local health care alliances.
RESULTS
There were more reports of hospital performance (161) than of physician performance (103) in the study areas. More reports included measures derived from claims data (mean, 7.2 hospital reports and 3.3 physician reports per area) than from medical records data. Typically, reports on physician performance contained measures of chronic illness treatment constructed at the medical group level, with diabetes measures the most common (mean number per non-health plan report, 2.3). Patient experience measures were available in more hospital reports (mean number of reports, 1.2) than physician reports (mean, 0.7). Despite the availability of national hospital reports and reports sponsored by national health plans, from a consumer standpoint the status of public reporting depended greatly on where one lived and health plan membership.
CONCLUSIONS
Current public reports, and especially reports of physician quality of care, have significant limitations from both consumer and provider perspectives. The present approach to reporting is being challenged by the development of new information sources for consumers, and consumer and provider demands for more current information.
Electronic supplementary material
The online version of this article (doi:10.1007/s11606-010-1438-2) contains supplementary material, which is available to authorized users.
KEY WORDS: public reporting, provider performance, quality
For over a decade, providing the public with comparative information on provider quality has been a key element of proposals to reform America’s health care system.1 Public reports comparing provider quality measures can have a number of possible consequences,2 but advocates stress their potential to support consumers in choosing among providers3 and to motivate providers to improve the quality of their care.4,5 Private employers were early supporters of this strategy.6 In the public sector, the National Health Quality Forum (NQF) was formed as a public-private partnership with one of its objectives being to “…create a foundation for consistent data reporting and collection,”7 the Bush administration endorsed Chartered Value Exchanges to support public reporting of provider performance at the community level,8 and the recently passed Patient Protection and Affordable Care Act includes provisions to support performance reporting. The Robert Wood Johnson Foundation has a major programmatic effort to improve quality that includes promoting reporting of provider performance by community level organizations.9 Despite the considerable support that exists for public reporting of provider quality, results from studies of the impact of reports on consumer and provider decisions have been equivocal.10,11 Nevertheless, the availability of public reports continues to grow, suggesting that measurement and public reporting of provider quality is likely to be an important part of the US health care system in the future.
In this article, we assess the present status of public reporting of provider quality in 21 geographic areas, focusing on the following questions: To what extent are current public reports likely to be effective in supporting consumer choice of providers and in motivating providers to improve quality of care? Does the status of public reporting vary across geographic areas? How are recent developments likely to influence the effectiveness of reports going forward?
METHODS
Organizing Framework We define public reports as reports that compare measures of provider performance and are accessible to consumers. We include reports that health plans make available to their members, but exclude reports produced by organizations only for internal purposes. We do not include reports that consumers must pay to access. We organize our assessment using three dimensions of public reporting—availability of reports, their credibility and their applicability to their audiences—that have been identified in the literature as related to potential effectiveness. Consumers and providers may view these dimensions differently,12 and the importance of different dimensions may vary with the focus of the report (hospital or physician).
We address availability under the assumption that, if reports are widely available and relatively easy to access, consumers and providers are more likely to be aware of them and the information they contain, a first step to being influenced by the reports. One measure of availability is the number of reports (directed at hospital or physician performance) that exist in any given area. (We do not normalize by population, as we are interested in the availability of information to a given consumer in a study area; use of a report by one person does not preclude use by another.) The larger the number of reports, the more likely that consumers have encountered at least one report and that providers are aware that their performance is being measured and reported. Because most health plan reports are available only to plan members, we also estimate the percent of the population in a given area that has access to plan reports. Finally, we document when reports contain linkages to websites containing similar reports, as this makes additional measures of performance more readily available. In our analysis, we do not assess the efforts of report sponsors to “market” their reports, although this clearly could affect consumer and provider awareness of reports.
We assume that reports with greater credibility are more likely to influence consumer or provider decisions. For consumers, we assume that the primary indicator of credibility is the source of the report, consistent with findings in the literature,11 and that consumers consider health plans to be less trustworthy sources of information, especially when that information is provided online.13 For providers, we assume that credibility is influenced by whether measures in reports are endorsed by respected national bodies or, alternatively, developed locally through a collaborative process involving providers; reports are likely to be viewed as less credible if their measures lack those imprimaturs. Whether reports provide information on patient outcomes, processes of care, or both could influence their credibility with providers, although this could vary with context or provider. For instance, some physicians might regard patient outcome measures for diabetes, such as blood sugar level, as not credible because patient health habits can affect these measures, while others might argue that measures of patient health outcomes increase report credibility. Therefore, in our assessment, we document the presence of each type of measure in public reports.
Reports may be available and credible, but may not contain information that users (providers or consumers) consider applicable to their individual situations. We assume the applicability of reports will affect whether consumers find reports useful in comparing providers. For example, the literature suggests that consumers view patient experience as an important component of quality. 11 Quality of care measures linked to a specific clinical condition are less widely applicable, as they are of value primarily to consumers being treated for that condition. Reports containing multiple measures pertaining to a single condition could be more valuable to consumers with that condition, because they provide more information about different aspects of treatment. However, the literature is not clear on this point, an alternative view being that multiple measures may confuse consumers.11 We document the number of measures in reports that address specific conditions. We also assume that reports will be less applicable if they do not contain information on the type of physician used by consumers.14 Reports that contain only information on primary care physicians, for example, may have more limited applicability than reports combining measures for both primary care physicians and specialists. In addition, the literature suggests that information reported at the physician level is valued more highly by consumers than similar measures constructed at the medical group level;11 we note the number of reports available to consumers that include measures constructed at these different levels.
We assume that the applicability of the contents of public reports to a provider’s specific situation will be important in stimulating provider responses to reports as well. For example, measures of quality that apply only to primary care physicians are not likely to stimulate quality improvement activities by specialists. For hospitals, it is not clear if broadly based measures of hospital quality are more likely to stimulate improvement efforts than measures pertaining to the treatment of specific diagnoses. With regard to the latter, hospital responses could depend on the number of market competitors providing the treatment to which the measure applies. Therefore, we document whether hospital reports contain general versus specific measures.
Study Areas Our assessment is based on data collected as part of the evaluation of The Robert Wood Johnson Foundation’s Aligning Forces for Quality Initiative (AF4Q).9 Fourteen non-profit organizations (alliances) received initial grants in 2006–2007; three organizations chose to define their entire state as the relevant area for their work, with others focusing on counties or metropolitan areas. Grantees attempt to coordinate and accelerate improvements in quality of care by aligning public reporting, consumer engagement and quality improvement activities. Our analysis utilizes data collected from 7 additional areas (2 states and 5 communities) as well, resulting in 21 study areas in total. The additional areas provide information about the status of public reporting in places without AF4Q grantees and were chosen because of their similarity in location, population size and demographics to one or more AF4Q areas, but are not used for comparison purposes in this analysis.
Tracking Activities We reviewed websites of health plan, hospital and medical associations; quality improvement organizations; state Departments of Health; and AF4Q grantees. In 14 areas, we reviewed websites for the 5 largest health plans, including all national plans. In 7 areas where there were fewer than 5 significant plans, we reviewed the websites for plans whose members together constituted approximately 75 percent or more of the total private sector health plan enrollment. In all 21 areas, we conducted telephone interviews with staff of organizations sponsoring public reports to verify our search findings, gather further details regarding measure sources and their construction, and identify any reports that had not emerged through our search process. We repeated these interviews annually to update our findings, resulting in data that are current through 2009, which we summarize in Tables 1, 2, 3 and 4. (Tables containing detailed information on public reports in specific areas are available online.) Table 1 contains information on report availability across areas, while Table 2 presents indicators of report credibility. The remaining tables address various aspects related to applicability of reports.
Table 1.
Availability of Quality Reports
| Reports with physician quality measures | Reports with hospital quality measures | |||
|---|---|---|---|---|
| Mean | Range | Mean | Range | |
| Total reports• | 4.9 | 1–9 | 7.7 | 5–12 |
| Reports available to all * | 1.9 | 0–7 | 3.7 | 1–7 |
| % of Population with access to health plan sponsored reports† | 21.36% | 1.13%–54.31% | 32.76% | 1.13%–61.09% |
| % of Health plan enrollees with access to health plan sponsored reports† | 28.29% | 1.30%–65.1% | 43.84% | 8.6%–78.3% |
•Table entries summarize reporting in 21 study areas
*Reports available to the public (without a secure log-in for web reports). Almost all health plan reports require a secure log-in and therefore are not available to everyone in an area. They are not included in the calculation unless accessible to everyone in a study area
†Proportion of population or private health plan enrollees in MSA who are enrolled in health plans producing reports with physician or hospital quality measures. Enrollment data are from 2006 Interstudy health plan survey and population estimates include private health plan, Medicaid and Medicare enrollments and estimated uninsured individuals
Table 2.
Credibility of Quality Reports
| Reports with physician quality measures• | Reports with hospital quality measures• | ||||
|---|---|---|---|---|---|
| Mean | Range | Mean | Range | ||
| Sponsor* | Local | 1.0 | 0–4 | 1.8 | 1–4 |
| State | 0.5 | 0–2 | 1.6 | 0–5 | |
| Health plan | 3.4 | 1–5 | 4.3 | 1–9 | |
| Source of measures† | National | 2.0 (0.7) | 0–5 (0–3) | 2.3 (1.8) | 1–6 (1–6) |
| Local/unique | 4.1 (1.0) | 1–9 (0–4) | 6.1 (2.1) | 4–11 (0–4) | |
| Type of data‡ | Claims | 3.3 (1.0) | 1–6 (0–3) | 7.2 (3.2) | 4–10 (0–6) |
| Patient record | 2.6 (0.9) | 0–5 (0–3) | 2.4 (1.6) | 1–6 (0–4) | |
| Patient survey | 1.1 (0.5) | 0–5 (0–3) | 1.2 (0.9) | 0–3 (0–2) | |
| Provider supplied (physician)/safety survey (hospital) | 0.2 (0.1) | 0–1 (0–1) | 4.3 (0.7) | 1–8 (0–2) | |
•Table entries summarize reporting in 21 study areas. Results are presented for reports by all sponsors and, in parentheses, reports by state and local sponsors only
*Type of organization acting as primary sponsor of provider quality report. Local sponsors include business coalitions, hospital associations and local health care quality organizations. State reports include Departments of Health or other State funded organizations, such as PHC4 in Pennsylvania and the Office of Statewide Health Planning and Development in California. Health plan reports include local, regional and national health plans
†National measure source includes all reports utilizing nationally endorsed measure sets, such as CMS Hospital Compare measures. Local/unique measures include measures developed or modified by the reporting entity or a local health care quality organization in consultation with local physician and hospital leaders. Unique measures include those developed by health plans for their members. Reports may use a combination of national and local/unique measures
‡Type of data used in quality measure construction. Claims data include health plan billing data and other administrative data such as hospital discharge data provided to CMS. Patient record data include data obtained directly from medical records; claims data may be used to identify patients for medical record review. Patient survey data are data obtained from a survey provided to patients such as CAHPS or locally/nationally sponsored surveys. Provider supplied/safety survey data are data obtained from surveys completed voluntarily by providers such as Leapfrog or health information technology readiness surveys
Table 3.
Applicability of Physician Quality Reports
| Mean | Range | ||
|---|---|---|---|
| Types of measures*• (no. of reports) | Care process/outcome | 4.5 (1.1) | 1–8 (0–3) |
| Patient experience | 0.7 (0.5) | 0–4 (0–3) | |
| Cost | 1.4 (0.1) | 0–3 (0–1) | |
| Efficiency | 1.7 (0) | 1–2 (0–0) | |
| Level of measure construction† (no. of reports) | Practice/group | 3.0 (0.9) | 0–8 (0–4) |
| Individual | 2.0 (0.7) | 1–3 (0–2) | |
| Type of physician included in report‡ (no. of reports) | Primary care | 4.4 (1.0) | 1–9 (0–4) |
| Specialist | 3.0 (0.7) | 1–5 (0–3) | |
| Chronic illness §, ‖ | |||
| Diabetes | Prevalence | 8.1% | 5.3%–11.7% |
| Average no. of measures in reports | 1 (2.3) | 0–2.5 (0–9) | |
| Asthma | Prevalence | 13.25% | 9.6%–16.4% |
| Average no. of measures in reports | 0.2 (0.3) | 0–0.75 (0–2) | |
| Depression¶ | Prevalence | 7.6% | 6.1%–8.8% |
| Average no. of measures in reports | 0.1 (0.1) | 0–0.9 (0–2) | |
| Hypertension | Prevalence | 27% | 21.1%–35.2% |
| Average no. of measures in reports | 0 (0.1) | 0–0.3 (0–1) | |
| Heart disease | Prevalence | 3.8% | 2.0%–5.4% |
| Average no. of measures in reports | 0.7 (1.4) | 0–2 (0–3) | |
•Table entries summarize reporting in 21 study areas. Results are presented for reports by all sponsors and, in parentheses, reports by state and local sponsors only
*Types of measures included in provider quality report. Process and outcome measures include care process or outcomes (i.e., laboratory results) related to inpatient or ambulatory care whether constructed with claims or medical record data. Patient experience measures include measures related to a patient’s satisfaction with care. Cost measures include measures related to the cost of a procedure. Efficiency measures include measures that take into consideration a weighting of cost based on risk or are constructed based on quality and cost measures. Efficiency measures are primarily provided by national health plans and combine cost and quality measures
†Quality scores for physicians are provided to consumers at the individual physician or group/clinic level. The total can add to more than total number of reports
‡Quality measures in physician quality reports for primary care physicians, specialist, or both. Specialist reports primarily include national health plan reports, reports focusing on cardiac surgery and the Consumer Checkbook report in Memphis
§Disease prevalence source (excluding depression) is the 2007 Behavioral Risk Factor Surveillance System (BRFSS), average for each study area
‖Average number of quality measures in physician reports addressing treatment for the condition (the denominator for the average is the total number of reports in each community that contain care process/outcome measures). Where a quality report provides consumers with one composite score(s) and a consumer can access the component measures of the composite, all measures included in the composite are counted. For organizations that report only an overall composite designation that covers a variety of conditions, the measure is not counted under the average number of measures for a specific chronic condition. For organizations that report a composite at a disease-specific level, that measure is counted as “1” under the specific condition and is used in the average measure calculation
¶Substance Abuse and Mental Health Services Administration, Office of Applied Statistics, National Survey on Drug Use and Health, 2005–2006 (state-level data)
Table 4.
Applicability of Hospital Quality Reports
| Mean• | Range | ||
|---|---|---|---|
| Types of measures*• (no. of reports) | Care process/outcome | 7.3 (3.0) | 5–11 (1–6) |
| Patient experience | 1.2 (0.9) | 0–3 (0–2) | |
| Cost | 2.5 (0.7) | 0–5 (0–3) | |
| Efficiency | 1.0 (0.0) | 1–1 (0–0) | |
| Number of reports with general quality measures | Patient safety | 4.3 (0.7) | 1–8 (0–2) |
| Patient experience | 1.2 (0.9) | 0–3 (0–2) | |
| Length of stay | 3.2 (0.4) | 0–6 (0–1) | |
| Volume | 4.7 (1.0) | 1–8 (0–4) | |
| Complications | 1.6 (0.8) | 0–6 (0–2) | |
| Mortality | 4.8 (0.6) | 1–8 (0–2) | |
| Surgical infection | 2.5 (1.5) | 1–4 (0–3) | |
| Number of reports with specific quality measures | Obstetrics | 2.1 (0.4) | 1–5 (0–3) |
| Knee/hip replacement | 0.3 (0.3) | 0–2 (0–2) | |
| Pneumonia | 2.6 (1.6) | 1–5 (0–4) | |
| Heart failure | 2.6 (1.6) | 1–5 (0–4) | |
| Heart attack | 2.7 (1.6) | 1–5 (0–4) | |
| Coronary artery bypass graft | 0.7 (0.7) | 0–2 (0–2) | |
| Other cardiac | 0.7 (0.6) | 0–2 (0–2) | |
•Table entries summarize reporting in 21 study areas. Results are presented for reports by all sponsors and, in parentheses, reports by state and local sponsors only
*Types of measures included in provider quality report. Process and outcome measures include care process or outcomes (i.e., laboratory results) related to inpatient or ambulatory care whether constructed with claims or medical record data. Patient experience measures include measures related to a patient’s satisfaction with care. Cost measures include measures related to the cost of a procedure. Efficiency measures include measures that take into consideration a weighting of cost based on risk or are constructed based on quality and cost measures. Efficiency measures are primarily provided by national health plans and combine cost and quality measures.
Interviews with AF4Q Participants In the 14 AF4Q areas, we conducted 275 on-site interviews with community stakeholders (employers, providers, health plan representatives, government officials, AF4Q staff) in 2006–2007.15 Subsequently, at 6-month intervals from 2007–2009, we conducted follow-up interviews with AF4Q project directors. Each interview lasted approximately 1 h, was guided by a structured interview protocol and was recorded, transcribed, coded and entered into a database, resulting in approximately 5,500 pages searchable using ATLAS.ti software. We used these interview data primarily in our assessment of the future prospects for public reporting.
RESULTS
Availability In comparing hospital performance, consumers in all areas had access to the CMS Hospital Compare16 and Leapfrog reports.17 However, participation in the Leapfrog survey is voluntary, so comparison of specific hospitals in any given area may not be possible. We do not include either of these national sources in our analysis, but do include regional, local and health plan reports that draw some of their measures from CMS Hospital Compare or Leapfrog.
Measures of hospital quality were widely available, with the smallest number of reports (five) in Memphis and Nashville (Table 1). However, when health plan reports accessible only by members were excluded, consumers in Kansas City, Memphis and Nashville had only one report on hospital quality available. The largest number of hospital reports from all sources (12 and 11 reports, respectively) were available in Minnesota and Cincinnati. Consumers in Minnesota and one community in California had the largest number of reports of physician quality (nine reports) available to them. In contrast, consumers in the Willamette Valley of Oregon had only one report with physician quality measures and, when considering only reports available to everyone in an area, consumers in the Willamette Valley and Nashville had access to no reports addressing physician quality. Much of the difference in hospital and physician report availability is related to reporting efforts of provider associations; state hospital associations in ten study areas produced hospital reports, but no state medical associations in any area produced physician reports.
While health plan reports with physician and hospital quality measures were available in all areas, the degree to which they were accessible to consumers in a specific area depended on patterns of health plan enrollment (Table 1). Among study areas, Nashville had the largest percent of the population (54%) with access to a health plan report on physician quality, while Western New York had the smallest (1%). There was similar variation in access to reports on hospital quality produced by plans, ranging from 61% in Lehigh County, PA, to 1% in Western New York.
All reports were available on websites of sponsoring organizations, with only eight sponsoring organizations also making their reports available in paper form. Hospital reports were twice as likely, compared to physician reports, to provide linkages to other reports, with linkages to CMS Hospital Compare and Leapfrog the most common. Reports sponsored by health plans were the least likely to contain linkages to other reports.
Credibility Reporting by health plans accounted for 1–9 hospital reports and 1–5 physician reports per area, with health plans in total providing 71 of 103 physician reports and 91 of 161 hospital reports across all study areas (Table 2). The number of reports using measures developed by national organizations was similar for hospital reports and physician reports. Nearly half of hospital reports provided Hospital Quality Alliance (HQA) measures available through the CMS Hospital Compare website. Nineteen of the reports with HQA measures also provided consumers with additional quality information, most commonly Leapfrog patient safety measures. Hospital reports that did not include HQA measures most commonly provided cost information or information on patient volumes and never events. Physician reports most commonly provided measures derived from NCQA’s HEDIS measures.
Hospital reports typically contained measures constructed at least in part from claims data. A greater proportion of physician (53% versus 32% for hospital) reports contained measures constructed using data drawn from medical records. The relatively large number of hospital reports with provider-supplied measures (Table 2) reflects, primarily, inclusion of measures taken from the Leapfrog survey. Data for the Leapfrog measures were voluntarily supplied by hospitals.
Applicability Relatively few public reports of physician performance included patient experience, cost or efficiency measures, along with their clinical quality measures (Table 3). On average, treatment of diabetes had the largest average number of measures in physician reports, but diabetes has a lower prevalence than several other diseases typically addressed with fewer measures (Table 3). Physician public reports in general had very little information applicable to individuals who did not have diabetes or heart disease (where heart disease measures were directed at cholesterol management.) In addition, excluding health plan reports, consumers in Kansas City, Nashville, New Hampshire and the Willamette Valley had no access to quality measures relating to the common chronic conditions in 2009 (Table 3).
About 60 percent of reports contained measures of physician quality of care constructed at the medical group level (Table 3). A larger percentage of reports provided information on primary care physicians (90%) than specialists (61%), with reports from three national plans (UnitedHealthcare, Cigna and Aetna) containing measures of performance for both. Relatively few reports not sponsored by health plans contained measures relating to the performance of specialists (Table 3).
Public reports of hospital performance provided measures of hospital-wide quality (e.g., mortality, complications from treatment), potentially of interest to all consumers and information specific to a small number of conditions (Table 4). Fifty-three percent of reports with hospital measures contained data taken from the Leapfrog survey, and 16 percent contained H-CAHPS data on patient experience collected by CMS. In hospital reports, measures that pertained to specific procedures could be accessed through searchable databases, where consumers could choose a disease category and/or a specific procedure. The most common information of this type pertained to heart attack treatment, heart failure, pneumonia and surgical infection prevention, areas addressed in the CMS Hospital Compare website. Data on CABG mortality rates for hospitals and cardiac surgeons were available in reports by state governments of California, Pennsylvania and New York, while reports produced by other sponsors in Wisconsin and Ohio contained similar measures. Consumers in relatively few communities could access data in other clinical areas, such as obstetrical care and knee and hip surgery (Table 4).
DISCUSSION
As described above, we conducted interviews with community stakeholders to obtain richer data on the evolution of existing public reports, as well as plans for future public reporting efforts. Our interviews documented several recent developments that, if sustained, could enhance the credibility and applicability of reports, although perhaps not their numbers. They also highlighted limits to the effectiveness of current public reporting efforts.
Availability We found that health plans sponsored the majority of public reports, but virtually all of these reports are available only to plan members through a secure log in. We also found considerable overlap in the contents of different hospital reports available in any area, so that the presence of multiple reports does not necessarily mean that consumers have access to more information. The availability of public reports is also limited somewhat by the almost exclusive use of the Internet as the distribution channel.18 However, recent growth in the percent of the population seeking information using the Internet suggests that this barrier may decline in importance in the future.19
Credibility While health plans are relatively common, as noted previously these reports are likely to be less credible to consumers than reports produced by other sponsors.13 Also, the limited amount of data on individual physicians possessed by any single health plan can raise physician concerns about the reliability of physician-level measures in reports produced by plans. Responding to this limitation, some local organizations now are producing reports by combining data from multiple health plans (e.g., Puget Sound, Detroit) or bypassing plans to construct measures using data from medical records (e.g., Minnesota, Cleveland). The financial support being provided by the Obama administration for the introduction of medical records systems in physician offices20 should make this latter approach increasingly feasible. Interview respondents in some of our study areas suggested that the use of medical records data was a potentially significant step for enhancing report credibility with providers, and therefore in motivating quality improvement efforts. If it becomes the norm, the number of health plan-sponsored reports present in any given area could decline, but the credibility and usefulness of public reporting overall could improve. (Similarly, health plan efforts to create “tiered” provider networks or institute pay for performance might have greater credibility with providers if not based solely on the claims data possessed by individual plans.)
Applicability The most serious challenge to achieving the goals of public reporting would appear to be, as in the past, developing measures that are widely applicable and useful for consumers and providers. Our data suggest that there is considerable variation both across reports and geographic areas in this respect. While some interview respondents believed that the inclusion of measures of treatment of chronic illness in reports can stimulate providers to improve their performance, and provide useful information for consumers with these conditions, these measures may be of limited general interest. Furthermore, the reports we reviewed took very different approaches to presenting this information; measures ranged from complex to easily understood, varied widely in the number pertaining to any specific condition and suggested little consensus regarding the desirability of aggregating information into composite measures. Reports also did not contain information about variation in treatment by race, ethnicity or preferred language. These observations suggest that there is not yet a strong consensus on how to best present this information and, possibly, on its ultimate value to consumers.The incorporation of patient experience measures has been encouraged by the Robert Wood Johnson Foundation for participants in its AF4Q Initiative. These measures could attract more consumers to public reports, exposing them to the range of clinical quality information also available. Tempering this prospect, however, is the growing amount of less formal data on patient satisfaction available to consumers. For instance, “Angies list” now provides a forum for consumers to discuss their health care experiences,21 while Zagat ratings of providers are available in some areas.22,23 These sources are familiar to Internet users, and consumers may perceive their contents to be more applicable and compelling than patient experience scores constructed from survey responses. This new type of “public reporting” is likely to provide increased competition for the reports reviewed in our analysis.
The Future for Public Reporting While public reporting of provider performance appears to be firmly entrenched in our health care system, the present approach to public reporting now faces transformative challenges. Consumer demands for “real time” information of all types seem likely to pressure sponsors to make the contents of their reports more current. Most content now is collected and reported annually, with measures based on data that are 1–2 years old. Some of our interview respondents suggested that reports soon will be based on data “pushed” from electronic medical records through sponsor portals and into standardized public reporting formats on a monthly basis.A second challenge is the need to address the “low numbers” problem in current physician reporting.24 Purchasers and consumers value data pertaining to individual physicians, but many report sponsors may not possess sufficient data to construct reliable, representative measures (especially relating to treatment of specific diseases) at this level.25 Current approaches to addressing this problem need further development. The feasibility of combining health plan data for public reporting purposes has been demonstrated (e.g., Puget Sound Health Alliance), but data on public program members may not be reflected in measures constructed using this approach. Requiring that physicians provide data on all (or a sample) of their patients to an independent entity for auditing and measure construction also is feasible (e.g., Minnesota Community Measurement), results in measures representative of a physician’s entire practice and permits the construction of a wider range of outcome measures. Our interview respondents noted that EMRs in physician practices are not required for this approach to be successful and, in practice, current EMRs often do not support the collection and validation of data needed to construct performance measures.Finally, there appears to be building momentum for a national physician report based on Medicare data, with the foundation being laid by the CMS Physician Quality Reporting Initiative (PQRI; www.cms.hhs.gov/pqri). As this effort moves forward, the current public reporting landscape may change; health plans and physicians may see participation in other reporting efforts as an unnecessary expense and, as many sponsors now do with respect to hospital quality measures, simply link to the national report. Then, the survival of local reporting efforts could rest on their ability to develop innovative, more timely, content that consumers and providers value and trust.
Limitations There are several limitations to our analysis. Although we included non-AF4Q areas in our study, our findings may over-represent the availability of physician public reports prepared by community organizations, given that the development of such reports was a stated goal for AF4Q grantees, and grantees were selected in part based on their perceived ability to do so. Also, because the AF4Q program initially targeted the development of physician reports containing measures on the treatment of chronic conditions and, more recently, encouraged inclusion of patient experience measures, our findings may overestimate the presence of these measures in reports.26 Third, there are limits to the conclusions that can be reached based on our general indicators of availability, credibility and applicability of public reports. For example, we do not attempt to access the relative quality of the measures used in different public reports, nor do we address issues such as whether multiple measures relating to a specific condition are of greater value to consumers than “composite” measures of care.27 While questions such as these are important, different research strategies are required to address them.Despite these considerations, some general conclusions regarding the status of public reporting are warranted. It seems clear that current reports on physician performance, in particular, have significant limitations with respect to their availability, credibility and applicability. Therefore, in most geographic areas, currently available public reports are not likely to provide the support for consumer decision making, or the motivation for physician quality improvement, envisioned by health care reformers. Their shortcomings, however, do not appear insurmountable. Recent developments in EMR technology, continuing efforts to refine quality measurement, and building payer pressures to improve the value received for dollars spent on health care suggest that the relatively gradual evolution of public reporting to date does not necessarily reflect its future trajectory. Our report card on the current status of public reporting of provider performance by necessity offers only a snapshot of an evolving facet of the US health care delivery system that seems likely to have an increasing influence on physician and hospital practice in the future.
Electronic Supplementary Materials
Below is the link to the electronic supplementary material.
(DOC 247 kb)
Acknowledgements
We thank Jessica Mittler, Patricia Keenan and two anonymous reviewers for their comments on various versions of the manuscript, as well as Bethany Shaw for her research assistance. Funding for this work was provided through a grant from The Robert Wood Johnson Foundation to The Pennsylvania State University for evaluation of the Foundation’s Aligning Forces for Quality initiative.
Conflict of Interest None disclosed.
References
- 1.Bernstein AB, Gauthier AK. Choices in health care: what are they and what are they worth? Med Care Res Rev. 1999;56(Supplemental: Power of Choice):5–23. [DOI] [PubMed]
- 2.Werner RM, Asch DA. The unintended consequences of publicly reporting quality information. J Am Med Assoc. 2005;293(10):1239–44. doi: 10.1001/jama.293.10.1239. [DOI] [PubMed] [Google Scholar]
- 3.Rothberg MB, Morsi E, Benjamin EM, Pekow PS, Lindenauer PK. Choosing the best hospital: the limitations of public quality reporting. Health Aff. 2008;27(6):1680–7. doi: 10.1377/hlthaff.27.6.1680. [DOI] [PubMed] [Google Scholar]
- 4.Berwick DM, James B, Coye MJ. Connections between quality measurement and improvement. Med Care. 2003;3(41):I30–8. doi: 10.1097/00005650-200301001-00004. [DOI] [PubMed] [Google Scholar]
- 5.Hibbard JH, Stockard J, Tusler M. Does publicizing hospital performance stimulate quality improvement efforts? Health Aff. 2003;22(2):84–94. doi: 10.1377/hlthaff.22.2.84. [DOI] [PubMed] [Google Scholar]
- 6.Galvin R, Milstein A. Large employers’ new strategies in health care. N Engl J Med. 2002;347:939–42. doi: 10.1056/NEJMsb012850. [DOI] [PubMed] [Google Scholar]
- 7.Kizer KW. Establishing health care performance standards in an era of consumerism. J Am Med Assoc. 2001;286(10):1213–7. doi: 10.1001/jama.286.10.1213. [DOI] [PubMed] [Google Scholar]
- 8.Department of Health and Human Services. Press Release. HHS secretary awards health leaders with special distinction for improving quality and value of health care. http://www.hhs.gov/news/press/2008pres/02/20080201a.html. Accessed June 15, 2010. 2008.
- 9.Painter, Lavizzo-Mourey Grantwatch commentary: aligning forces for quality, a program to improve health and health care in communities across the United States. Health Aff. 2008;27:1461–4. doi: 10.1377/hlthaff.27.5.1461. [DOI] [PubMed] [Google Scholar]
- 10.Fung CH, Lim Y-W, Mattke S, Damberg C, Shekelle PG. Systematic review: the evidence that publishing patient care performance data improves quality of care. Ann Intern Med. 2008;148:111–23. doi: 10.7326/0003-4819-148-2-200801150-00006. [DOI] [PubMed] [Google Scholar]
- 11.Harris KM, Buntin MB. Choosing a health care provider: The role of quality information. Research Synthesis Report No. 14. Princeton, NJ: The Synthesis Project. The Robert Wood Johnson Foundation, 2008. [PubMed]
- 12.Miller TP, Brennan TA, Milstein A. How can we make more progress in measuring physicians’ performance to improve the value of care? Health Aff. 28(5):1429–1437. [DOI] [PubMed]
- 13.Dutta-Bergman M. Trusted online sources of health information: differences in demographics, health beliefs, and health-information orientation. J Med Internet Res. 2003;5(3):e21. doi: 10.2196/jmir.5.3.e21. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Lubalin JS, Harris-Kojetin L. What do consumers want and need to know in marking health care choices? Med Care Res Rev. 1999;56(Supplemental: Power Choice):67–102. doi: 10.1177/107755899773743873. [DOI] [PubMed] [Google Scholar]
- 15.Alexander, Christianson, Hearld, Hurley, and Scanlon. Challenges of Capacity Building in Multi-sector Community Health Alliances. Health Education and Behavior. In press. [DOI] [PubMed]
- 16.Department of Health and Human Services. About hospital compare. http://www.hospitalcompare.hhs.gov/Hospital/Search/Welcome.asp?version=default&browser=IE%7C7%7CWindows+Vista&language=English&defaultstatus=0&pagelist=Home. Accessed June 15, 2010.
- 17.The Leapfrog Group. The leapfrog group fact sheet. http://www.leapfroggroup.org/about_us/leapfrog-factsheet. Accessed June 15, 2010; 2009.
- 18.Tu HT, Hargraves JL. Seeking health care information: Most consumers still on the sidelines. Issue Brief No. 61. Washington, DC: Center for Studying Health System Change; 2003. [PubMed]
- 19.Fox S. Online health search 2006. (Pew Internet Project: October 29, 2006). http://www.pewinternet.org/Reports/2006/Online-Health-Search-2006.aspx. Accessed June 15, 2010; 2006.
- 20.Department of Health and Human Services, Health Information Technology For the Future of Health and Care. Regulations and Guidance. Meaningful Use. http://healthit.hhs.gov/portal/server.pt?open=512&objID=1496&parentname=CommunityPage&parentid=1&mode=2&in_hi_userid=10741&cached=true. Accessed June 15, 2010.
- 21.Kowalczyk L. Looking for Dr. Right; With more and more websites rating physicians, the question is: Can you trust them? Boston Globe. 2009;(June 8): G6.
- 22.Freudenheim M. Noted rater or restaurants brings its touch to medicine. The New York Times. February 16; 2009.
- 23.Freudenheim M. Zagat turns focus to medicine. New York Times, February 16, 2009. http://www.nytimes.com/2009/02/16/business/worldbusiness/16iht-16zagat.20207252.html?_r=1&scp=6&sq=zagat&st=cse. Accessed June 15, 2010
- 24.Scholle SH, Roski J, Dunn DL, Adams JL, Dugan DP, Pawlson LG, Kerr EA. Availability of data measuring physician quality performance. Am J Manage Care. 2009;15(1):67–72. [PMC free article] [PubMed] [Google Scholar]
- 25.Hofer TP, Hayard RA, Greenfield S, Wagner EH, Kaplan SH, Manning WG. The unreliability of individual physician ‘report cards’ for assessing the costs and quality of care of a chronic disease. J Am Med Assoc. 1999;281(22):2098–105. doi: 10.1001/jama.281.22.2098. [DOI] [PubMed] [Google Scholar]
- 26.Choe HM, Bernstein SJ, Standiford CJ, Hayward RA. New diabetes HEDIS blood pressure quality measure: potential for overtreatment. Am J Manage Care. 2010;16(1):19–24. [PubMed] [Google Scholar]
- 27.Shubrook JH, Snow RJ, McGill SL, Brannan GD. “All-or-none” (bundled) process and outcome indicators of diabetes care. Am J Manage Care. 2010;16(1):25–a33. [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Below is the link to the electronic supplementary material.
(DOC 247 kb)