Performance Measurement in Healthcare: Part II – State of the Science Findings by Stage of the Performance Measurement Process

Abstract

Objective:

This paper summarizes findings of a comprehensive, systematic review of the peer-reviewed and grey literature on performance measurement according to each stage of the performance measurement process – conceptualization, selection and development, data collection, and reporting and use. It also outlines implications for practice.

Methods:

Six hundred sixty-four articles about organizational performance measurement from the health and business literature were reviewed after systematic searches of the literature, multi-rater relevancy ratings, citation checks and expert author nominations. Key themes were extracted and summarized from the most highly rated papers for each performance measurement stage.

Results:

Despite a virtually universal consensus on the potential benefits of performance measurement, little evidence currently exists to guide practice in healthcare. Issues in conceptualizing systems include strategic alignment and scope. There are debates on the criteria for selecting measures and on the types and quality of measures. Implementation of data collection and analysis systems is complex and costly, and challenges persist in reporting results, preventing unintended effects and putting findings for improvement into action.

Conclusion:

There is a need for further development and refinement of performance measures and measurement systems, with a particular focus on strategies to ensure that performance measurement leads to healthcare improvement.

The purpose of our review was to summarize the current business and healthcare literature on performance measurement (PM) systems and to make recommendations for research and practice. Details of methods are provided in Part I (Healthcare Policy, 1.4). This second paper reports in greater depth on themes and issues extracted from the peer-reviewed and grey literature in relation to stages of the PM process.

The PM Process

The PM literature lacks consensus on concepts and definitions. However, the PM process is typically described as having approximately four stages (Nadzam and Nelson 1997; Nutley and Smith 1998; Bourne et al. 2000; Ibrahim 2001; Smith and Goddard 2002), although many authors caution that the process is more dynamic and less linear than a simple set of stages implies. The stages are (a) conceptualization, (b) selection and/or development of measures, (c) data collection and processing and (d) reporting and using results.

Conceptualization

Two major issues on conceptualization of PM systems are prominent in the literature: aligning with organizational strategic direction and determining the appropriate scope for the system.

Strategy

There is increasing emphasis on aligning PM activities with the strategic direction of the organization, and a general sentiment in both business and health that such alignment is rare in practice. However, maintaining a strategic focus is acknowledged to be more difficult in healthcare than in business for several reasons.

First, organizational goals are often difficult to operationalize in healthcare because of the complexity of treatments, settings and patient groups (Baker and Pink 1995). Public service organizations have broader goals (including societal goals) and “a more complex pattern of accountability than the corporate financial statement” (Smith 1993: 137). The dual management model (professional and administrative) and the interrelationships among multiple internal and external stakeholders (Kleinpell 1997; Lemieux-Charles et al. 2002), each with its particular interest in setting the PM agenda (Nadzam and Nelson 1997; Collopy 1998), create greater complexity. In health services the policy environment is very fluid (Smith and Goddard 2002), perhaps more so than in business environments.

Second, causal links between service and health outcomes are very difficult to specify for both medical and public health interventions, owing to the limits of evidence in medicine and the reality that healthcare is only one of several predictors of health status (Williams et al. 1992; Handler et al. 2001; Leggat et al. 1998).

Third, “customer” dynamics are less straightforward in healthcare than in the purchase of a commercial product or service (Newhouse 2002). People seek care out of necessity, not desire. The provider often has a local monopoly on a given service, limiting both comparators for judgments about performance and opportunities to seek alternatives (Smith 1993). An important commercial goal is repeat business, while in healthcare it is often viewed as an unfortunate necessity because a definitive cure is unattainable. The consumer is also typically less knowledgeable about the service content than in commercial transactions (Jennings and Staggers 1999) and is often vulnerable by virtue of being ill and possibly afraid when seeking care. These realities complicate the patient satisfaction and perceived care quality domains of PM (Jennings and Staggers 1999). The message about the task of strategic conceptualization of a PM system is clear in both sets of literature: “what gets measured gets delivered,” and there are undesirable consequences for organizations, from a strategic point of view, that collect the wrong measures (Voelker et al. 2001).

Scope

The second major issue in conceptualization of PM systems in both literatures is determining the appropriate system scope. Scope decisions apply to three dimensions: vertical (level of the healthcare organization or system), horizontal (breadth across the continuum of care or business units) and longitudinal (temporal) (Collopy 1998). In business there is a trend towards involving all levels of the organization in a common vision that can be reinforced by the PM system itself (Neely et al. 1995; Epstein and Manzoni 1998; Lockamy 1998; Legnini et al. 2000). “One of the major problems with conventional PM is the ease with which organizational wholes are carved up, and their interactions with their environments cease to be of interest as management functions devise measures (and associated targets) for their own territory. This reductionism is associated with some of the problems identified by managers when they seek to improve performance” (Holloway 2001: 173).

Healthcare PM activities are also highly fragmented, verified by the sheer number of single-level or single-service systems described in the literature. Single-level focus creates debates about the value of one over the other: some charge that the patient level is often not addressed in system-level approaches (e.g., Greenhalgh et al. 1996), while others express the opposite concern (e.g., Barrell 2000). Many call for greater consolidation through overarching goals and greater consensus and coordination (Eddy 1998; Kizer 2001), and increasingly multi-level systems are being conceptualized (e.g., Moscovice et al. 1995; Luttman 1998; Evans et al. 2001; Handler et al. 2001). Even so, Nutley and Smith (1998: 53) contend that “calls for a top to bottom PM architecture have largely been ignored.” Others caution that the PM for high-level management and accountability differs from that needed for daily operations (McLoughlin et al. 2001; Voelker et al. 2001).

The horizontal scope of systems is also debated. The business literature reports a few companies attempting to establish measures that capture relevant information across company boundaries (such as with supplier networks), but acknowledges this to be very difficult (Fawcett and Cooper 1998). The roots of healthcare PM are clearly in acute care, and hospital-bounded approaches dominate. Separate PM systems are under development and are testing for other components such as public health (Corso et al. 2000; Handler et al. 2001; Kates et al. 2001), but our review found no systems spanning acute and community care. DeRosario (1999: 38) notes that “to catch the next wave of performance change, we need to begin measuring activities that occur between healthcare sectors,” and others concur (Hall 1996; Kizer 2001). A PM system should match the service delivery model, and it is likely that broader PM systems will emerge with the trend towards regionalized, integrated health services in many jurisdictions. With respect to the temporal dimension, a few authors suggest that PM systems need to address and measure the process of care over time for an individual (Bishop and Pelletier 2001).

Measures selection or development

Many authors stress that, according to measurement theory, measures themselves are just a reflection of reality. In addition, the choice of what to measure among the many options is an imprecise process (van Peursem et al. 1995), reflecting a system of values and social goals (Sheldon 1998). Ibrahim (2001: 431) writes that “performance indicators are inherently controversial” because they require a judgment about what constitutes quality.

Frameworks

After general conceptualization, the next task in PM is to select or develop measures. Optimally, a framework ensures balance across strategic improvement areas and guides the measurement process. An ideal framework describes domains (measure groupings) and dimensions (e.g., organizational levels), but most frameworks reviewed are simply a list of indicators and/or domains (e.g., Lied 1999). More complex frameworks also include one or more dimensions such as level of the healthcare system (McEwan and Goldner 2000) or stakeholder perspective (Nadzam and Nelson 1997; Kizer 2001; McIntyre et al. 2001). We found little consistency in the combinations of 21 domains used in 17 major health PM frameworks reviewed (Adair et al. 2003).

We identified eight business frameworks that included both non-financial and financial measures (Lebas 1995; Neely et al. 1995; Kaplan and Norton 1996, 2001; Epstein and Manzoni 1998; Kueng and Krahn 1999; Kueng 2000; Kanji and Moura 2002) – called multi-dimensional or portfolio approaches – that are tabulated in the full report (Adair et al. 2003). Neely et al. (2000) and Kueng (2000) provide noteworthy reviews of business approaches. The most popular framework in business is the Balanced Scorecard (BSC), which has also been applied in healthcare. Some other approaches to the management of quality in the business literature are noteworthy because of their recent diffusion into healthcare and their close relationship with PM. First are the quality award programs, including the Malcolm Baldrige National Quality Award, the European Foundation for Quality Management’s Business Excellence Model (Neely et al. 1995; Kueng and Krahn 1999; DeBaylo 1999) and many spin-off quality award programs. One widely adopted program, Hoshin Kanri, that developed in Japan in the 1960s and has been disseminated widely is noteworthy for having extensive coverage in the popular press worldwide but virtually none in the western research literature (Tennant and Roberts 2000). The BSC and other portfolio approaches have evolved towards the selection of more forward-looking, strategy-focused measures, but many criticisms of these early-stage approaches persist (Kueng and Krahn 1999; Mooraj et al. 1999; Kueng 2000; Baughan et al. 2002; Brignall 2002; Morgan and Braganza 2002) that parallel the healthcare PM literature.

Issues In Choosing Measures

Several predominant themes relate to measures selection, including the sheer growth in numbers of measures and systems, as well as issues related to the types of measures and their limitations.

In recent years, measures (both indicators and comprehensive instruments) have become so numerous that it would be nearly impossible to catalogue them completely (Nutley and Smith 1998; Sheldon 1998). The national indicator library of the Joint Commission on Accreditation of Healthcare Organizations (JCAHO) is believed to have more than 1,000 measures, and the database of the Agency for Healthcare Research and Quality (AHRQ) contained more than 1,197 in 53 sets by 1995 (AHRQ 2002). Unless indicators are commonly defined, comparative reporting is difficult, if not impossible. The development of measures databases is a welcome sign that this duplication of effort may be waning (e.g., Jennings and Staggers 1999; Hermann et al. 2000). Collaborative efforts to standardize measures are another promising development (Braun and Zibrat 1996; Leggat et al. 1998).

Guidelines or criteria for indicator selection are numerous in both literatures and, again, there is little consistency across sets. Table 1 lists criteria catalogued and synthesized conceptually from health literature papers that are cited in the full report but are too numerous to cite here (Adair et al. 2003). They represent suggested, rather than tested, criteria. The more recent literature puts greater emphasis on the importance of choosing indicators that are meaningful, strategic and evidence-based.

TABLE 1.

Criteria for performance measures selection

CRITERION	DESCRIPTION
Evidence-based	There are valid and reliable operational definitions for the measure that have been demonstrated through rigorous research
Strategic	The measure directs attention towards the ultimate change desired
Important	The measure addresses an important or serious health or health services problem (usually defined as health burden or cost) such that there will be sufficient impact from collection and service improvement initiatives
Attributable	Causal links between the measure, service improvements and health outcomes are known
Actionable	The measure addresses a service area that can benefit from improvement
Feasible	Data collection, reporting and follow-through are cost-effective (potential benefits outweigh costs) and there is reasonable technical capacity for collection and analysis, including risk adjustment of compared measures
Relevant and meaningful	The measure is relevant to most stakeholders, including policy makers, managers, clinicians and the public
Understandable	The measure is understandable to a non-technical audience (often just a communication issue)
Balanced	The set of measures is balanced across types of treatments, treatment settings, major health problems, age groups, special populations and levels of the healthcare system. The set is balanced across short- and long-term measures, and balance and appropriateness are considered across process-and outcome-type measures
Responsive	The measure is sensitive to change over time
Robustness	Potential adverse effects of the measure can be mitigated, and vulnerability to gaming is minimal
Non-ambiguous	The measure is clear in terms of which direction for service change is desirable

Financial indicators are still used as part of health PM systems (e.g., cost per weighted case), but as in business, non-financial indicators have taken centre stage. In discussing BSC applications in health, Voelker et al. (2001) claim that a primary focus on financial measures may actually hinder organizational growth and success. In healthcare, financial measures are notoriously difficult to action because most costs are not variable and there is little flexibility in hiring and firing staff (Brookfield 1992). Because of the complex and multifaceted purposes of healthcare, focusing too heavily on financial measures may diminish prospects for overall improvement. Most PM systems in health continue to collect traditional input/output measures such as service utilization (e.g., bed occupancy, surgery facility use, length of stay and numbers of discharges and admissions), despite repeated commentary that they are poor indicators of performance (Mark et al. 1997; Nutley and Smith 1998). Mortality remains the predominant traditional outcome measure, with the distinct disadvantage that it reflects a rare and end-stage event relative to the total volume of healthcare provided. In a Canadian study of existing indicators reported in 2000, Lemieux-Charles et al. (2000: 52) observed that “indicators measuring integration, coordination and continuity of care, as well as responding to population health needs, were rarely used. These types of measures are critical as we redesign our service delivery systems to address population needs.” Klazinga et al. (2001) consider the ultimate performance measures to be those reflecting overall population health.

Similarly, others express concern about “opportunistic systems” that emphasize readily available measures at the expense of newer, more important and meaningful measures (West 1996; Elkan and Robinson 1998; Nutley and Smith 1998; Smith and Goddard 2002). Shaw (1997: 217) characterizes this as the “spectre of convenience” and asks, “should measures be based on existing available data as ad hoc criteria for achievement, or should health service policy targets first be identified and data then captured specifically to measure their achievement?” A dynamic tension exists between the need for locally meaningful and strategic measures and the benefits of selecting and using standardized measures that enable meaningful comparison.

The business literature also underscores the point that the choice about what not to measure is as important as what to measure, since “things that are measured are considered important while the things not measured are generally considered of less importance” (Waggoner et al. 1999: 54). This literature also notes that once collected, measures are rarely deleted, even if they are obsolete (Neely et al. 2000). Given limited resources, each measure chosen represents an opportunity cost.

The component literatures reveal an important parallel debate about process versus outcomes measures (e.g., Evans et al. 2001; Rubin et al. 2001; Mannion and Davies 2002). The business literature uses other terms, e.g., “a debate on whether performance indicators should be focused on procedures (activities) or on results (output)” (Kueng 2000: 77), but the concepts are identical. Despite some arguments that process measures are more practical, most writers consider them complementary to outcomes or results (e.g., Baker 1995), and all should be chosen to fulfill the specific measurement objective (Wynia et al. 1996).

There are widespread concerns about the paucity of validation work. Eddy (1998: 7) describes current measures as “blunt, expensive, incomplete, and distorting.” There is strong consensus that measures must be evidence-based. Gross et al. (2000) evaluated coronary bypass mortality-related indicators across 24 hospitals and concluded that indicator definitions significantly affected computed rates and changed relative standings. “There are no generally agreed-on external criteria for validity of indicators” (Gross et al. 2000: 210).

Data collection and analysis

Both component literatures strongly note the unanticipated cost and complexity of PM systems. The business literature describes data collection and analysis as “complex, frustrating, difficult, challenging, important, abused and misused” (Lebas 1995: 23). Costs rise because of the high level of technical and managerial expertise required, new information technology and ongoing maintenance. Some also attribute costs (monetary and strategic) to measuring too many different things. “Measuring something makes it important and therefore motivates people. Measuring everything means nothing is important and therefore de-motivates” (Johnston and Fitzgerald 2001: 183). Kueng (2000) identifies success factors in the data collection stage as a parsimonious set of generally accepted indicators, automation and personal involvement of staff and management.

In healthcare, many organizations have lacked the capacity to implement effective systems, and failed attempts are abundant. They generally underestimate the scope and complexity of the infrastructure required to manage healthcare adequately and, by implication, the measurement of its performance (McIntyre et al. 2001). Voelker et al. (2001) and Braun and Zibrat (1996) attribute system failures at this stage to staff and management turnover, technical problems with information systems, budget constraints and competing priorities. Kates et al. (2001) express concern about mandating PM systems in public service organizations without guidance in their implementation and use. Both literatures express concerns about the cost–benefit relation of PM initiatives.

Other issues related to data collection include data sources and quality. Administrative data have long been considered a rich source for PM if properly “mined,” and researchers in particular have produced notable examples of their creative and rigorous use (e.g., Brownell et al. 2001). But many now suggest that the value of secondary data has been overstated, at least as typically formatted (Bishop and Pelletier 2001; McLoughlin et al. 2001). Problems cited include poor reflection of performance, lack of data elements for sensitive diagnosis and risk adjustment, lack of availability and stability of data at smaller levels of aggregation and generally poor quality (Kelman and Smith 2000; Brown 2002). Many writers bemoan the effort devoted to the analysis of retrospective or secondary data at the expense of the collection of more relevant data (Sheldon 1994; Stryer et al. 2000; Voelker et al. 2001). In the more general context of effectiveness research, after 10 years of experience with secondary data, AHRQ’s Patient Outcome Research Team (PORT) investigators are also calling for more prospective and real-time data (Stryer et al. 2000).

Many advocate for routine prospective data collection, fully integrated with clinical practice, that can be used for the delivery of care as well as rolled up for management use (McLoughlin et al. 2001). Concerns remain about the diversion of clinician time from patient care to data recording tasks (Naylor 1999). Ullman et al. (1996: 361) suggest that research-based, standardized measures are “too unwieldy and time consuming to mesh well with the practice ecology.” Several hybrid approaches are proposed (e.g., Schneider et al. 1999; Brook et al. 2000; Hoelzer et al. 2001), and many commentators still consider the electronic health record, with the appropriate data for PM thoughtfully built in and integrated with more general operational data, to be the best solution in the long run (Aller 1996; Slater 1997).

The literature is replete with concerns about PM data quality. These include issues of missing data, reliability, validity, accuracy, precision, statistical and clinical significance and timeliness (Kleinpell 1997; Mark et al. 1997; Shaw 1997; Collopy 1998; Jencks 2000; Roper and Mays 2000; Pink et al. 2001). McKee and James (1997) provide an excellent review of data quality issues that arise when comparing outcomes data across systems that use different diagnostic and severity adjustment schemes, and report error rates as high as 20% to 40%. Many cite the need for consistent definitions and processes and data quality checks (Shaw 1997; Nutley and Smith 1998) and for the transparent reporting of data collection issues that underlie the reported measures (Pink et al. 2001). Pink et al. (2001) consider expert involvement of both researchers and management as essential.

With respect to methods for analysis, sound statistical methods have long been available but many authors suggest that they usually fall by the wayside in practice (Leggat et al. 1998; Nutley and Smith 1998; Roper and Mays 2000; Smith and Goddard 2002). Adjustment methods are many and varied, and consensus is lacking about the best methods for a given analytic problem (Mant and Hicks 1996; Iezzoni 1997; Shahian et al. 2001; Schneider 2002; Smith and Goddard 2002). Several authors stress that the problem is not so much the methods’ mechanics but the lack of understanding of their limitations and inconsistency in application (Ibrahim 2001; Zaslavsky 2001). An obvious solution is to ensure that adequate analytic expertise is brought to the PM task. Organizational comparisons should disclose all analytic methods and reveal potential sources of bias. As well, a “healthy skepticism about ratings or ranking [should] be maintained” (Schneider 2002: 3). Smith and Goddard (2002) suggest that devising better ways to communicate complex results to non-experts could strengthen the link between research and strategic policy.

Reporting and use

A first general theme on the topic of reporting PM information is practical advice on effective presentation for various audiences, with the emphasis on evidence-based communications. A more prominent and controversial topic is the growing practice of reporting performance information to external stakeholders via report cards. Several authors provide excellent reviews of the issues and evidence related to public release of performance data (Leatherman and McCarthy 1999; Marshall et al. 2000; Hoey et al. 2002). Barrell (2000: 15) expresses the general sentiment on this matter: “There seem to be basically two schools of thought: those who believe we can’t afford to do it, and those who believe we can’t afford not to.” In a rare and interesting empirical study that examined organizational response to public disclosure of quality data in the United States, McCormick et al. (2002) demonstrated that in a voluntary system, providers with lower-quality scores were four to six times more likely to withdraw from future disclosure than those with higher scores.

We also found a large literature on the issue of using PM to produce improvement. The business literature clearly advocates a strong link between performance measurement and performance management (Lebas 1995), including the development of causal models between measures, actions taken and subsequent improvement (Lebas 1995; Neely et al. 1995; Neely 1999) through an organizational change process (Kueng 2000). With respect to alignment of incentives for change, Epstein and Manzoni (1998) cite Kerr’s folly (rewarding A while hoping for B) as a common practice in many companies, due to an inability to break out of old patterns of reward and recognition, the lack of an overall system view and focusing on the short term.

The health literature addresses three themes on the application of PM information: its use by organizations as a whole, by individual service providers and, externally, by consumers to make care choices. A second theme is how PM is used for both positive change as well as its unintended or adverse effects. A third is the organizational culture in which PM is embedded.

First, on the issue of “actioning” results, Goddard et al. (2000: 99) observe that “most schemes appear to rely on a vague hope that providers will ‘do something’ in response to the data.” The importance of organizations learning how to link the PM results to actions, rather than having the PM system simply keep records, is restated in many ways (Camp and Tweet 1994; Baker and Pink 1995; Collopy 1998; Voelker et al. 2001). The few studies on organizational (Turpin et al. 1996; Leggat et al. 1998; Lemieux-Charles et al. 2000) or individual provider behavioural change (Jencks 2000; Marshall et al. 2000) in response to organizationwide PM suggest that impact is minimal (Barrell 2000; Legnini et al. 2000; Marshall et al. 2000; Schneider 2002). It is likely that in some settings individual managers and clinical leaders have found effective ways to use and apply performance measurement information, just as in some settings quality improvement has been applied effectively – many examples are provided by the Institute for Healthcare Improvement (2002) – but virtually no rigorous studies have described effective broader-level PM practice and elucidated its features.

The more recent healthcare literature includes descriptions of new mechanisms involving financial incentives for performance at the organizational or individual level. These mechanisms go by a variety of labels, including value-based purchasing, quality- based purchasing, performance-based contracting and pay-for-performance. With respect to alignment of financial incentives at the organizational level, there were many reported instances in US healthcare and some in the United Kingdom. A straight forward incentive system that simply provides high performers with extra funds and penalizes low performers is criticized as having the potential to flow funds to services serving regions with less health need, if the contributors to poorer performance are environmental and socio-economic rather than actual differences in care (Elkan and Robinson 1998). In a fairly innovative concept for incentive alignment, Ward (2000) describes a scheme for improving performance in NHS trusts. In this scheme, funding is not allocated according to performance ranking; instead, greater autonomy and spending latitude are given to higher-ranking organizations (Ward 2000). While financial incentives may seem like common sense, they continue to be controversial and are largely unproven to date (e.g., Giuffrida et al. 2000).

With respect to the potential for adverse effects, the literature contains many examples of (mostly theoretical) adverse effects, which are summarized in Table 2. Goddard et al. (1998, 2000), Smith (2002) and Smith and Goddard (2002) have drawn from the management control literature and written extensively on unintended effects in the public sector and healthcare. They consider that “some of these dysfunctional consequences are the result of the imperfect or incomplete data on which indicators are based, some are due to how the data are used and interpreted, and some are simply intrinsic to any system of PM” (Goddard et al. 1998: 26).

TABLE 2.

Unintended or adverse effects of performance measurement

Attention can be focused narrowly on improvement of the measure itself, rather than the underlying process Measures can be selected that divert attention and effort away from more important problems, or measures can be focused on the short term at the expense of longer-term issues Measurement may encourage an attitude of seeking simplistic solutions to complex problems Individual managers can use measurement to serve their own agendas rather than the needs or priorities of the whole organization Measures can be “gamed” or distorted Average performance may be considered sufficient, encouraging complacency and discouraging risk-taking Measures can be used to lay blame rather than find solutions Good results are disseminated while poorer results are suppressed Broader performance expectations or standards can dominate local priorities Unrealistic performance targets can lower morale and engender defeatism

^Sources:Smith 2002; Smith and Goddard 2002; Goddard et al. 1998, 1999, 2000; van Peursem et al. 1995; Collopy 1998; Elkan and Robinson 1998; Leggat et al. 1998; Proctor and Campbell 1999; McLoughlin et al. 2001.

A third theme in the health literature is the relatively recent acknowledgment that organizational contextual issues are paramount to effective PM use because of the invariably complex health system environments. Smith (1993: 150) suggests that while PM systems are assumed to be neutral reporting devices, in reality they are “operating in a far messier and less well understood organizational context.” Barnsley et al. (1996), Leggat et al. (1998) and others outline the organizational culture issues in PM. Legnini et al. (2000) provide a very detailed set of recommendations for realigning incentives to encourage positive use of PM information, according to organizational context and stakeholder perspective. Table 3 lists other suggestions. A more comprehensive and holistic approach to PM is being promoted (McKee and Sheldon 1998; Smith 2002), and the emergence of new models may be imminent (Viccars 1998; Campbell et al. 2001).

TABLE 3.

Suggested solutions

Leadership and commitment of senior managers/decision-makers is essential Take a systems approach, including consideration of organizational, contextual issues Focus on positive personal development, including education, supports for role change and realignment of incentives Maintain a positive, constructive, solution-focused orientation, not a blaming approach Consider performance measures as flags for identification of areas for improvement, rather than absolute measures of performance Commit to PM as a long-term endeavour Resource PM appropriately; ensure that the appropriate technical and managerial expertise and adequate funds are available Foster continuous, open communication with emphasis on interpretation of findings, avoiding simplistic explanations Encourage ownership of PM through collaborative, participatory approaches Consider all stakeholders’ perspectives Plan for performance management, not just measurement, i.e., ensure that mechanisms are in place to use results

^Sources:Greenhalgh et al. 1996; Mant and Hicks 1996; Turpin et al. 1996; Ford et al. 1997; Collopy 1998; Goddard et al. 1998; Leggat et al. 1998; Nutley and Smith 1998; Bodenheimer 1999; Proctor and Campbell 1999; Gross et al. 2000; Voelker et al. 2001; Weinberg 2001; Zairi and Jarrar 2001; Inamdar et al. 2002; Jarvi et al. 2002; Mannion and Davies 2002.

Summary and Implications for Practice

The literature reviewed on PM reveals several points of consensus as well as divergence, as summarized in Table 4. Overall, no author advocated abandonment of PM, but most recommended moving forward with more awareness of the pitfalls and making informed choices (Smith 1993; van Peursem et al. 1995; Shaw 1997; Eddy 1998; Sennett 1998). Epstein (1995: 4) urges realistic expectations, reminding us not to “let the perfect be the enemy of the good.” Many recommend using PM to create a shift towards a culture of improvement (Proctor and Campbell 1999; Bishop and Pelletier 2001; McLoughlin et al. 2001). In the United States, Braun et al. (1999) and others suggest a national, staged approach including standardized core measures. Berwick (1998) presents an insightful review that challenges current assumptions about healthcare performance. Finally, Lied and Sheingold (2001: 394) summarize the current state of practice on PM as follows: “There are real concerns that the act of measurement itself has taken on such a symbolic significance over and above the power of such information to promote beneficial and worthwhile change. We do not yet know how to make such systems deliver on the promises made for them.”

TABLE 4.

Points of consensus and divergence in the PM literature

Consensus

Performance can be measured and improved, and performance measurement can be beneficial Performance measures should include non-financial measures with a focus on quality, customer needs and, more broadly, stakeholder needs There is a need to move towards more meaningful and strategic measures There is a need to dedicate sufficient effort at the conceptualization stage, including consideration of the relevance of proposed measures to system change as well as their potential adverse effects PM is a complex and technically challenging exercise that needs appropriate expertise, resource allocation, an evidence base and awareness of the pitfalls PM system implementation represents significant organizational change, not just the collection and reporting of data More emphasis and effort are needed on “actioning” results for improvement

Divergence

The extent to which PM systems should be integrated across all levels of an organization and, specifically, whether measurement of management performance and measurement of clinical performance should be integrated processes The degree to which measures should change over time or remain static for historical comparison The optimal horizontal scope of measures The relative emphasis on process vs. outcome measures In the health literature, whether or not patient-level outcomes should be measured routinely by clinicians for all patients vs. using sampling or case-based approaches The extent to which performance results should be reported publicly The extent to which measures have specific utility for consumers and the general public The utility and relevance of administrative data for, in particular, outcomes measurement The extent of customization vs. standardization of measures

Finally, there are some key structural aspects of healthcare that challenge actionability. The long and strong tradition of professional autonomy, particularly among physicians, focuses philosophically on individuals, not systems. In many jurisdictions, healthcare professionals have contractual (not employee) relationships with service organizations. There are ethical obligations, real or perceived, to provide often heroic and expensive care even where the likelihood of a successful outcome is small. Optimizing performance in such an environment is different from eliminating inefficiencies in a manufacturing process. Clinical care frequently involves trial and error, particularly where cases are intractably difficult or where the science is imprecise, and what one observer would describe as wasteful, another might view as creative and responsive. These caveats suggest that we pay particular attention to the literature that counsels a balanced, nuanced and comprehensive approach to PM and its uses.

Conclusion

The research literature on PM is expanding daily and the ideas are advancing, but our team has read nothing since completion of the major report that stands out in contradiction with the overall findings presented here. A number of encouraging developments are noted on the policy front in Canada since the review: a recognition of the need for leadership in the federal/provincial/territorial accords on indicator reporting and subsequent comparative national reports, the addition (to Saskatchewan’s Health Quality Council) of three more provincial HQCs (Ontario, Quebec and Alberta) and the establishment of the Canadian Patient Safety Institute. At the same time, the controversial Maclean’s Health Report has come and gone. Much of the current energy is focused on wait times and patient safety. We need to address PM more comprehensively, and work remains as well at the service level – in regions and on the front line. Just as it is no longer acceptable to disseminate clinical treatment without evidence, the stakes are too high to implement healthcare PM without developing the evidence base.