Limitations of and Barriers to Using Performance Measurement: Purchasers' Perspectives

Abstract

Although health plan performance data are becoming increasingly more available, many purchasers are still not using these data to make their purchasing decisions. In this article, we review barriers that private purchasers face to using performance data. In addition, we consider the effects of the larger health care purchasing environment and employers' quality improvement activities on their use of the data. We conclude that a variety of factors, including trends, the health care purchasing environment, characteristics of firms, and problems with performance data and their presentation to users create barriers to incorporating this information into health care decisionmaking.

Background

Despite the increasing availability of plan performance data, health care purchasers still face barriers and limitations to their use. Several different measurement tools exist that provide a variety of data on clinical and service quality indicators of health care. However, these tools and data, while important to some purchasers, are either not used or underused by many of them. This article reviews published literature on the limitations and barriers that health care purchasers, both employers and purchasing coalitions, face when incorporating performance data into their purchasing decisions. We examine whether employers are using data and then explore the reasons that plan performance data are not as widely used as they could be and the reasons for purchasers' underuse of the data. We focus primarily on the experience of non-governmental purchasers. In addition, we consider how the larger health care purchasing environment and employers' quality improvement activities affect their use of data. We conclude that a variety of factors ranging from the larger health care system trends to specific characteristics of purchasers to problems within the data and its presentation to users create barriers to incorporating performance data into health care decisionmaking. Finally, we formulate recommendations for future research to more fully explore some of the questions raised by this review.

Are Purchasers Using Performance Data?

Employers and other purchasers are buying health care in an era that emphasizes the principles of value-based purchasing. These principles hold health care providers accountable for both the cost and quality of health care (Meyer, Rybowski, and Eichler, 1997). In value-based purchasing, purchasers ideally are using information on both cost and quality to make decisions. Purchasers evaluate equivalent information for competing health plans and providers, and their purchasing decisions are based on demonstrated performance or proposed approaches to improvement. Quality information therefore plays a critical role in the contracting process. Performance measurement data should become increasingly more useful and used, as purchasers continue to incorporate measures of accountability, quality, and cost into their purchasing decisions.

Some employers are incorporating the principles of value-based purchasing in their health care purchasing decisions. These employers are using cost and quality information to select primarily health plans but also institutional providers as well. Those purchasers who are using performance data are employing a variety of different strategies of data review. Employers can evaluate either health plan and/or provider information. Others concentrate on consumer satisfaction measures. Another approach that employers might use is to focus their attention on expensive conditions or procedures or on those that affect workplace productivity. Finally, some employers are developing their own quality improvement initiatives which involve measuring quality and comparing it across providers (Meyer, Rybowski, and Eichler, 1997; Darby, 1998).

The benefits of a health care purchasing environment in which a formal comparison of performance information should hold providers accountable, improve quality, and reduce costs are apparent. However, not all employers purchase health care after systematically comparing performance and quality of competing plans. In addition, although many purchasers collect data on quality, not all employers actually use data to make decisions. Finally, while some employers are heavily engaged in quality improvement activities and effective incorporation of performance data into their purchasing decisions, many more appear to be influenced by factors other than quality.

A study by Lo Sasso et al. (1999) describes results from two independent employer surveys: the 1997 Mercer/Foster Higgins National Survey of Employer-Sponsored Health Plans and their own 1999 survey of members of two business coalitions on the use of performance data in health care purchasing. The survey by Lo Sasso et al, based on a purposive sample, was designed to complement the Mercer Survey. Sixty-eight percent of respondents from the Mercer survey said that employers bear some responsibility for assessing the quality of health plans that they offer. However, less than one-half took action in managing their health plans, and only 19 percent of respondents negotiated performance guarantees. Results from the Lo Sasso et al. survey suggested that 72 percent of respondents adopted performance standards for health plans and providers with which they contract. In addition, 53 percent report purchasing only from plans and/or providers that meet or exceed these performance standards. Finally, their survey results assigned greater importance to responsible purchasing information when making purchasing decisions, compared with the findings of the Mercer study (Lo Sasso et al., 1999).

A National Business Coalition on Health (NCBH) survey also found relatively high levels of performance data use. Of NCBH's 96 members (representing 90 percent of all United States business coalitions), 90 percent of the 75 respondents reported involvement with data collection or analysis. Two-thirds reported their involvement as extensive (Fraser et al., 1999).

Other studies concluded that employers had more limited use of performance standards. One study found that fewer than one-half of employers included any performance standards in their health plan contracts or monitored quality of care (Deloitte, Touche, LLP, 1998). Another study found that 42 percent and 74 percent of all firms and large firms, respectively, used employer-specific standards (Washington Business Group on Health, 1998; Merrick et al., 1999). A review of purchasing initiatives in 15 communities found that very few purchasers are using quality-related information to select health plans. Those purchasers using quality information in their health plan selection were the larger and more prominent employers and purchasing coalitions in the 15 communities (Lipson and De Sa, 1996).

Are Purchasers Initiating Quality Improvement Activities?

Despite the push toward more value-based purchasing of health care, debate exists about whether employers actually are devoting more or less time to improving quality. Some suggest that the stabilization in health care premiums seen in the mid-1990s brought purchasers' attention away from costs and more towards quality and access (Lo Sasso et al., 1999). According to Castles, Milstein, and Damberg (1999), employers' interest and activity in this area stems from “…the increasing recognition of how thinly quality management has been supported by the health care industry.” The inability of health plans to assess the quality of care their members receive is a consequence of health plans' failures to develop appropriate clinical information systems and comparable performance data on providers, hospitals, and health plans. As a result of the inability of health plans to report quality, employers have demanded more evidence of quality and health plan performance.

Others disagree with the viewpoint that more employers are actively engaged in quality measurement activities. Instead, health care cost control has decreased the attention employers paid to measuring quality. In some large companies, employers have actually devoted fewer resources to controlling costs and measuring quality as health care costs have been controlled. One author suggests that while purchasers request information, they rarely use it in purchasing decisions. Most purchasers do not build any performance-based quality initiatives into their purchasing decisions. They “…look to carriers and plans to clamp down on providers' costs and are largely indifferent to how that is done” (Meyer, Rybowski, and Eichler, 1997).

Employers' Role in Purchasing Health Care

Some employers are moving away entirely from imposing their own decisions about selecting health care plans. Instead, employers are placing the burden of selection on employees by providing them with the widest possible array of options for health care plan arrangements and provider networks. A survey of 14 large employers demonstrated that, within the context of value purchasing, employees are being encouraged to assume more financial and personal responsibility for their choices. Employers are providing financial incentives for employees to use the lowest cost plan meeting the company's required minimum standards but also are providing employees with a choice of plans. To facilitate these plan comparisons, employers also are providing comparative information on health plan quality and patient satisfaction (Maxwell et al., 1998). These authors claim that the individual employee's responsibility is an important component of value purchasing. However, despite employers' provision of comparative information to employees, their findings suggest a decreased role for employers in quality improvement activities, since the burden for health care decisionmaking increasingly falls on the employee.

Plans in which value-based purchasing based on quality measurement is less likely, such as point-of-service plans or preferred provider organizations are becoming more popular, as purchasers try to accommodate the strong preferences for wider access to providers. The increasing popularity of these types of plans would decrease the opportunity for employers to negotiate over value-based purchasing characteristics such as Health Plan Employer Data and Information Set (HEDIS^®) measures and National Committee for Quality Assurance (NCQA) accreditation (Lo Sasso et al., 1999).

Finally, employers are downsizing their human resource staffs. Such downsizing severely compromises employers' abilities to initiate quality improvement initiatives. Such inadequacy of resources is surprising, since there are potential savings to be realized from health care quality improvement and cost management (Meyer, Rybowski, and Eichler, 1997).

Barriers/Limitations To Using Performance Data

A limited number of employers are using plan performance data. In addition, there appears a lack of consensus about whether employers are increasing their involvement in quality-related activities. In some cases, employers are backing off entirely from directly managing certain aspects of the health benefits they offer their employees. These trends provide a backdrop for a discussion of the specific barriers to and limitations of using plan performance data.

Data Factors

Too Much Information

Purchasers are facing significant barriers to using performance data. In general, the most significant barriers focus on the issues of having too much information or not the right kind of information to make a decision. Hibbard et al. (1997) suggest that the amount of information and options available to purchasers potentially is so overwhelming that it acts as a deterrent to using performance data. Purchasers contend with as many as three different categories of performance indicators (e.g., service quality, consumer satisfaction, and HEDIS^®), each with multiple measures. In addition, each State may have multiple markets, and each market may have three to four health plans. “The more States they purchase in and the more plans they consider in each market, the less likely they are to use HEDIS^® or consumer satisfaction data when making their decision…. Purchasers who did not use either consumer or HEDIS^® data purchased services in more than twice as many States as did those who used the data” (Hibbard et al., 1997).

The increased interest in NCQA accreditation speaks to the information overload issue. Not only is the amount of information prohibitive to making decisions, but purchasers are finding it difficult to assimilate all of the variables into some measure from which they can make a decision. According to Hibbard et al. (1997), “NCQA accreditation is likely an attractive selection criterion because it integrates and summarized several characteristics in one easy-to-understand measure. It also reduces the information-processing burden by allowing decisionmakers to rely on expert assessment rather than their own assessment of the data.” In addition, these authors found that 12 percent reported making their decision based on only a single dimension such as cost or geographic access. More than one-half of the respondents found it difficult to incorporate all of the variables into their decision.

Adding to the data burden are the plethora of other measurement systems that present difficulties in comparing information across plans, including some developed by some companies reflecting their own quality standards. The information requested by these systems might differ by definitions, time periods, sampling methods, or adjustment factors. This lack of standardization adds to the cost of providers to meet these needs of these different requirements, and limits the ability to compare measures across measurement systems (Eddy, 1998; Maxwell et al., 1998).

The complexity of contracting networks also contributes to the information overload (Eddy, 1998). Performance measurement becomes problematic in markets in which plans contract with a wide network of providers who, in turn, contract with many plans. Administrators must determine how they will control the quality of care being delivered by providers. Providers, in turn, must determine how they will respond to various guidelines and quality management programs from each plan. Health plans must report on hundreds of measures to comply with Federal and State governments, private employers, and regulatory and oversight organizations. Complicating the picture is the fact that purchasers might require different benefit packages, so standardized measures might ask about the about the performance of an intervention that is not covered by all of the plans (Eddy, 1998; Schauffler, Brown, and Milstein, 1999; Miller and Leatherman, 1999).

Wrong or Inaccurate Information

The other most commonly cited limitation of plan performance data is its inability to measure information that users really want. In general, cost is among the most important pieces of information in health plan selection. Employers have demonstrated that they will switch plans for small changes in the premiums they pay (Darby, 1998). According to the Mercer survey, employers relied more on cost, premiums, and financial strength when selecting a plan than they relied on other responsible purchasing information. The other responsible purchasing factors that were rated very highly by more than one-half of the firms included in the Mercer study were geographic coverage and member access, particularly by the self-insured employers. To large firms, NCQA accreditation or other accreditation and the ability to provide HEDIS^® information was more important in plan selection, particularly those that are self-insured, and among members of employee coalitions. Member satisfaction surveys were less important to the larger firms (Lo Sasso, 1999).

Respondents from a survey of 33 large employers representing 1.8 million covered lives indicated that quality and outcomes are important to purchasers. However, when purchasers were asked about which performance information most influenced their decision, they listed consumer satisfaction and NCQA accreditation. Purchasers reported using three types of performance measures fairly consistently: (1) HEDIS^®, (2) consumer satisfaction data, and (3) NCQA accreditation. Purchasers used hospital outcomes data much less consistently. In summary, researchers suggest purchaser preference for financial information, geographic, network, and consumer satisfaction information (Hibbard et al., 1997).

Purchasers are basing their decisions on a limited array of information. Studies suggest that purchasers do not use HEDIS^® because HEDIS^® measures do not address their decision criteria which include the plan's financial stability, cost, and geographic access, outcomes, service quality (Hibbard et al., 1997). Also, HEDIS^® aggregates information at the health plan level while purchasers want information about specific providers (Fraser et al., 1999). In addition, the authors state that “HEDIS^® is viewed as not providing information they need on outcomes, cost effectiveness, and service quality….” Some employers blame HEDIS^® itself for its underuse in selecting plans. HEDIS^® is not seen as an accurate measure of quality and does not meet the needs of the customer (Eddy, 1998).

Outcome measurement is also suspect. Hibbard et al. (1997) found that purchasers questioned the validity of outcomes measurement methodology in hospital data. In addition, the authors reported that hospital outcomes data were not packaged to meet employers' needs.

Finally, the difficulty in quantifying savings generated by quality improvement activities acts as a deterrent to implementing initiatives and measuring their effect (McNeill, 1999). McNeill cites Sean Sullivan, president and chief executive officer of the Institute for Health and Productivity Management, who says that “Quality improvement is not dead, but it is comatose. We need to wake it up with evidence.”

Employer Factors

Ignorance of Plan Performance Information

Researchers found other barriers to using plan performance data. Some employers did not use data because they didn't know it existed. Research findings indicate that the percentages of purchasers who knew that hospital outcomes data were available to them ranged from 25-71 percent across different regions in the country. An average of 75 percent were aware that consumer satisfaction data were available to them and 78 percent reported that HEDIS^® data were available. The results regarding the availability of HEDIS^® data might reflect either perceived or actual availability of the data. However, the hospital outcomes data were available to all purchasers in the survey (Hibbard et al., 1997).

Lack of Direct Involvement with Data

As a result of downsizing human resource staffs, purchasers often delegate the role of monitoring quality to others, thereby removing themselves from using data directly to evaluate performance. Some employers use consultants to make purchasing decisions for them: 12 percent of purchasers rely on consultants to recommend plans and do not make choices themselves although 48 percent used consultants for recommending or obtaining data. Twenty-one percent did not select plans; rather, they maintained existing relationships with long-term plans. In addition, some purchasers expected managed care plans to monitor hospital quality (Hibbard et al., 1997). Hibbard et al. see this as a troubling arrangement. They believe that, if purchasers hold plans accountable for cost and at the same time ask plans to monitor and select hospitals on quality, then health plans might not have sufficient incentives to selectively contract with high quality hospitals.

Limited Employer Resources

Although there are some data to the contrary, those businesses that are the most involved with data seem to be the larger firms and the purchasing coalitions. In addition, those larger companies with sufficient market clout and the ability to take a long-term view of health care can internally support quality improvement activities. There are many more employers who do not have the same market clout nor do they have internal resources available to devote to health care quality monitoring and improvement. The overriding concern of these employers is cost and this factor dominates their health care decisionmaking (Darby, 1998).

Health Care System Factors

Financial Disincentives

According to Berenson (1998), quality improvement initiatives are facing a number of obstacles, including the lack of “…a market imperative to achieve NCQA accreditation or do well on HEDIS^® quality measures.” By not risk adjusting premiums, health plans are penalized for attracting patients with significant health problems who require more complex and more costly care. The lack of risk adjustment methods for health plan premiums, therefore, acts as a disincentive for health plans to compete on quality. If health plans consistently score higher than average quality ratings, they run the risk of attracting disproportionately more enrollees with high cost conditions. Consequently, the absence of risk adjustment methods serves as a financial disincentive for quality measurement (Dudley et al., 1998; Fraser et al., 1999; Lipson and De Sa, 1996).

Discussion

Literature suggests that the health care purchasing is plagued by an overabundance of information that is not measuring critical issues of concern to many purchasers. The overwhelming amount of data available to purchasers, and the fact that the data often are not comparable between plans or in a form that is useful to purchasers, are factors that inhibit use entirely or cause underuse of performance data. Factors contributing to this information overload include the number of plans employers offer employees, often within several different market areas; performance data which aren't necessarily comparable; and perhaps lack of knowledge about the existence of plan performance data.

The data-related factors which might deter employers from using health plan performance data are complicated by larger health care system issues. While some purchasers are devoting significant attention to quality improvement, others are less interested in quality measurement and are investing fewer resources and staffing in this area. In addition, some employers, in an effort to provide employees with more choice, are placing the burden of evaluating quality information on their employees. In fact, there is some debate as to whether cost control has generated, overall, more or less attention to quality on the part of the purchasers.

The findings presented here suggest some measures that might encourage employers and other purchasers to use performance data more consistently. Schaller, Sharpe, and Rubin (1998) have recommended that performance data should be comparable across plans. Such efforts are underway. The mission of the Quality Forum, for example, includes developing a framework for measurement and reporting and standardizing measurement and reporting to address the inconsistencies of measures, and lack of comparative data (Miller and Leatherman, 1999). Such standardized measures should be relevant to the concerns of purchasers and presented in a manner that meets their needs. Hibbard et al. (1997) suggest that summary measures or ways to integrate large amounts of information are useful. However, the validity of such summary indicators would need empiric assessment.

Technical assistance and other tools are vital to help assimilate large amounts of information for purchasers and promote interest and awareness of the quality measurement process. Such technical assistance might try to alert those employers who are unaware of the existence of performance to their use. However, if employers are backing away from direct involvement with data in general and are shrinking their human resource staffs, then this task becomes even more difficult.

Finally, employers are relying heavily on a few variables such as cost, geographic access, and a wide provider network. If these features continue to remain the most important considerations in purchasing health care, then few incentives exist to encourage employers to use other performance measures.

Recommendations for Further Research

This discussion of barriers and limitations to using plan performance data suggests directions for future research. As a starting point, researchers might want to recognize and explore the difference between the larger health system trends and factors within performance data themselves that might encourage or discourage data use. For example, some literature suggests that employer-interest in this issue is waning as a result of cost control in health care, while other information suggests that quality improvement activities have increased. A fuller understanding of this relationship could help direct efforts to generate increased employer interest in this area. In particular, the relationship between more recent cost increases and purchaser interest in quality initiatives might be explored.

A minority of employers and other purchasers are using value-based purchasing methods. Research exists to document the activities of these purchasers. A study of the characteristics and motivations of those employers who are conducting minimal or no value purchasing would be elucidating. Such research might examine their current practices in health care purchasing, the roles of their human resource staffs devoted to health care, and the criteria by which they select health plans. Such research might also question employee satisfaction with their health plans and the factors they consider most important in selecting plans. Finally, research might examine communication strategies and/or measures that would effectively demonstrate the cost effectiveness to quality improvement in health care.

Finally, the finding that some employers might not know of the availability of outcome data, and possibly also HEDIS^® and consumer satisfaction data, is troubling and should be further investigated. Such research might identify other sources of health care information that could be used by employers. In addition, such research might investigate the outsourcing of human resource activities in health care purchasing and its effects on quality improvement activities.