Making Health System Performance Measurement Useful to Policy Makers: Aligning Strategies, Measurement and Local Health System Accountability in Ontario

Abstract

This study examined the experience of the Ontario Ministry of Health and Long-Term Care in enhancing its stewardship and performance management role by developing a health system strategy map and a strategy-based scorecard through a process of policy reviews and expert consultations, and linking them to accountability agreements. An evaluation of the implementation and of the effects of the policy intervention has been carried out through direct policy observation over three years, document analysis, interviews with decision-makers and systematic discussion of findings with other authors and external reviewers. Cascading strategies at health and local health system levels were identified, and a core set of health system and local health system performance indicators was selected and incorporated into accountability agreements with the Local Health Integration Networks. despite the persistence of such challenges as measurement limitations and lack of systematic linkage to decision-making processes, these activities helped to strengthen substantially the ministry's performance management function.

Canada's provinces face challenges with regard to costs, efficiency, access and the quality and safety of their healthcare services (Health Council of Canada 2007). In Ontario, healthcare expenditures have been steadily growing, accounting for 46% of the total provincial budget and reaching CAD$40.4 billion in 2008/09 (Ontario Ministry of Finance 2008). At the same time, the expectations of Ontarians have risen with respect to timely access to high-priority services such as cancer treatments and primary healthcare services. Policy makers in Ontario are seeking to balance these expectations with available resources while promoting performance, and have attempted various strategies over recent years such as strengthening primary healthcare and setting up Local Health Integration Networks (LHINs) to enhance health system performance and improve system integration (MoHLTC 2004). Ontario's 14 LHINs oversee over 150 hospital corporations operating on over 200 sites, over 600 long-term care homes, home care programs and thousands of community agencies, while management of drug programs and funding for the province's 23,000 physicians remain provincial responsibilities.

Over the past decade, a growing body of research has suggested that the use of strategy-based performance management tools in the public sector can result in substantial improvements in both health outcomes and cost-effectiveness gains (Jha et al. 2003; Kaplan and Norton 2005, 2006; Porter and Teisberg 2004). for example, the Veterans Health Administration in the United States achieved significant improvements in targeted health outcomes between 1995 and 2000 by linking its strategy, a core set of performance indicators, and a vigorous performance management system built on strengthened information management (Kizer 1999; Asch et al. 2004; Jha et al. 2003; Perlin et al. 2004; Perlin 2006). In the meantime, the strategic use of performance measurement through implementation of the balanced scorecard model (Kaplan and Norton 1992, 1996) at the Cleveland Clinic, the Mayo Clinic and Duke University Children's Hospital (Inamdar et al. 2002) has been associated with improvements in clinical and financial performance. Other examples demonstrate that focusing on clear linkages among strategy, performance measurement and accountability through performance measurement is a key feature in performance improvement efforts and can lead to increased value for health systems (Bevan and Hood 2006; Lomas 2003; Porter and Teisberg 2004).

Before 2004, Ontario had experimented with a limited number of strategy-based performance management elements. The province had developed extensive analytic capabilities through research institutes focused on the health system; developed a substantial hospital report card process that linked into common health system strategies (Brown et al. 2005, 2006); and had created a strategy-based scorecard for its cancer system that was used for both public reporting and performance management of regional cancer systems (Greenberg et al. 2005). starting in 2004, the focus on the performance of the healthcare system changed in a number of ways: the government implemented a core strategy that focused on targeted initiatives to reduce wait times and strengthen primary healthcare; it created Local Health Integration Networks (LHINs) to devolve large amounts of managerial authority to the local level in order to improve integration and efficiency across the health system; it created a vehicle for public reporting of performance by establishing the Ontario Health Quality Council; and finally, it created several reform teams called Health Results Teams to drive performance improvement (MoHLTC 2004).

As planning and funding for the system shifted to the local level, these reforms had major implications for the Ministry of Health and Long-Term Care (MoHLTC). The ministry had to move away from hands-on management of the health system towards a stewardship model (Pfeffer and Sutton 2006) through which it established and communicated goals, held partners accountable for these goals and established policies for the health system.

The size, complexity and lack of clearly articulated strategies for the health system as a whole meant that the existing reform strategies had to be reviewed and synthesized to build a strategy map covering the health system in its entirety. This paper explores the Ontario experience since 2005 in developing and using a health system strategy map and strategy-based scorecard to enhance the stewardship role of government, and discusses how the information garnered from this approach has been used to strengthen the health system by aligning strategies, performance measurement and accountability.

The paper attempts to answer two key questions: How can strategy-based performance measurement be developed in the absence of an explicit and comprehensive strategy for the whole health system? And more broadly, how can this information be used by health ministries to target specific performance improvements for its agents? A retrospective evaluation of the implementation and effects of the policy intervention (Contandriopoulos et al. 2000) has been carried out through direct policy observation over three years, document analysis, interviews of decision-makers and systematic discussion of findings with other authors and external reviewers. four of the authors of this paper (JV, TH, SA, AB) were privileged observers when the other two authors (SK, NK) were not involved in the policy intervention. The authors documented the findings with data as much as possible, grounded them into theory and referenced them with the relevant scientific literature. furthermore, all policy papers and documents quoted are referenced and publicly available. Overall, this paper presents the methods and results of the policy intervention and discusses the valuable lessons and the applicability of this approach to other contexts.

Methods: Linking Strategy-Based Performance Measurement and Accountability

Developing the health system strategy map

The MoHLTC applied the strategy mapping approach to healthcare (Greenberg et al. 2005; Persaud and Nestman 2006), in which (a) government priorities and objectives are identified, (b) these aims are grouped into a logical set of strategic goals and (c) the goals are mapped in relation to one another. The methodological steps in formulating strategy maps have been previously defined by Kaplan and Norton (2004). However, as there was no single overarching strategic document in the system, the strategy would have to be derived from public statements of government intent and existing documents that detailed often silo-based planning efforts.

The Ontario health system strategy map was developed through six consecutive steps, which took place between November 2004 and March 2005:

• Step 1: Common themes in health system policies, investments and public statements were identified, grouped by strategic themes and mapped in a logical sequence through a comprehensive policy review of different information sources, such as the MoHLTC budget submission and policies.

• Step 2: Intersectoral cabinet submissions and related policies affecting health system strategies were reviewed and mapped within the health system strategic themes.

• Step 3: External experts were consulted to validate the groupings of strategic themes and develop a draft health system strategy map outlining cause-and-effect relationships.

• Step 4: The draft health system strategy map was shared with other ministries and stakeholder groups.

• Step 5: Relevant inputs from the consultation process were incorporated in the draft health system strategy map, which was then internally validated by different committees and ultimately by the Executive Management Committee of the MoHLTC.

• Step 6: The first iteration of the scorecard was distributed to 13 international experts in health system performance assessment, who appraised the model.

Once the framework was established, the next task was to identify how performance in Ontario could be measured against this foundation.

Developing a strategy-based health system performance scorecard

Despite the absence of a single performance measurement framework for the health system, there was an abundance of available performance information on health system performance (Hamilton 2006; Health Canada 2002, 2004, 2006; Health Council of Canada 2007). Although the review team identified over 2,000 performance and volume measures, few indicators were aligned to the strategic themes identified. A pre-screen of this inventory of measures was carried out by the MoHLTC in order to pre-select a set of indicators for review by a technical expert panel. The criteria used for pre-selection were validity and alignment with strategic themes; data quality issues; calculation feasibility and timeliness; and possibly the feasibility of reporting at multiple levels of the health system.

The review of inventoried indicators revealed that only 156 initially met these criteria and of these, only 54 could be cascaded to reflect different levels within the healthcare system. These indicators were submitted to an 18-member health system performance expert panel chosen based on technical knowledge of health system performance indicators. members evaluated each indicator based on available evidence summarized in descriptive sheets detailing the rationale and the supporting evidence for inclusion in the scorecard. different selection criteria were reviewed to identify those most appropriate to this exercise, detailed in Table 1.

TABLE 1.

Criteria used for indicator selection by expert panel

Criteria	Description
Importance	Reflects critical aspects of health system functioning and the strategic dimension
Relevance	Provides information that can be used to monitor and measure health system performance over an extended period of time
Feasibility	The needed data required are readily available or obtainable with reasonable effort
Reliability	The indicator produces consistent results
Validity	The indicator is an accurate reflection of the dimension it is supposed to assess

Members used the descriptive sheets, gathering related evidence to rate the indicators against each criterion using a five-point Likert scale (a psychometric scale in which respondents specify their level of agreement with a statement). The results were collated in a report and used as the basis for a modified Delphi workshop (a systematic forecasting process that utilizes independent experts). Although an attempt was made to distribute indicators across the system goals articulated in the strategy map, a small number of performance dimensions ended up without accepted indicators. Consequently, several additional indicators were proposed by the expert panel, and descriptive sheets were commissioned for use in a second meeting in which panelists made their final recommendations for a set of 26 system performance measures. In accordance with the feasibility criteria, all 26 measures selected by the experts could be calculated by ministry analysts using readily available administrative and survey data.

Linking strategy, measurement and accountability

Performance improvement requires aligning strategy, performance information, resource allocation, incentives and accountability (Jha et al. 2003). The MoHLTC conceptualized its performance management cycle, in which the ministry (a) sets its strategic priorities, (b) selects key performance indicators related to strategy to gauge progress, (c) uses these indicators to support resource allocation processes, (d) holds those receiving resources accountable for results and (d) assesses whether performance improvements have the desired impact on the performance of the health system in order to adjust strategies accordingly (Figure 1).

FIGURE 1.

A strategy-based performance management framework for the Ontario MoHLTC

It is important to emphasize that the development of this process did not start at the top with the intentional development of a single strategy. Rather, the process of developing the performance management structure led to a reverse engineering of the strategy and stimulated the development of other tools to help align activities to it.

In order to link strategy, performance measurement and accountability, the MoHLTC undertook with its LHINs the joint exercise of incorporating performance indicators aligned with the health system strategy map and scorecard into accountability agreements, following a process similar to the one used for the development of the health system scorecard. The ability of LHINs to influence specific performance indicators was an additional criterion that was included in the selection of performance indicators.

Results

The 2005 Ontario health system strategy map

All health system–related strategies identified within the MoHLTC and across the government of Ontario were mapped, grouped within strategic themes and broken down into subdimensions of performance (Table 2).

TABLE 2.

Ontario health system strategy map performance dimensions and subdimensions

Performance dimensions	Performance subdimensions
Increase availability of high-quality, relevant evidence	Appropriateness of allocation of resources Availability of evidence High-quality evidence
Increase access to and uptake of evidence for decision-making and accountability	Increased access to evidence Increased uptake of evidence
Increase productive use and appropriate allocation of resources across the system	Appropriateness of resource allocation to achieve health system outcomes Productive use of resources to achieve financial efficiency
Increase access to key healthcare services	Availability of programs and services
Improve patient-centredness, integration and quality of health services	Appropriateness Acceptability Responsiveness Competence Safety Continuity of care
Improve healthy behaviours through health promotion and disease prevention	Health promotion Disease prevention
Improve clinical outcomes	Clinical effectiveness
Improve health status	Health conditions Human function Well-being Mortality
Increase sustainability and equity of the health system	Financing Technology/capital infrastructure Human resources Confidence

These strategic themes were articulated in a way that linked the intermediate objectives and ultimate goals of the Ontario health system (WHO 2000). The strategy map shows that the value of other intermediate objectives and their associated actions is measured in terms of their effect on health system ultimate goals, such as improving health status, equity and health system sustainability (Figure 2).

FIGURE 2.

The Ontario health system strategy map, 2005

Core set of performance indicators

The 26 performance indicators selected covered all dimensions of the framework except for the first dimension, which lacked a quantitative measure meeting the predefined criteria. Therefore, a qualitative assessment was undertaken by experts and reviewed by the initial panel. Table 3 outlines the final set of indicators selected for inclusion. Each indicator is linked to a strategy map goal and subdimension of performance.

TABLE 3.

Core set of performance indicators for the Ontario health system scorecard

Performance dimension	Performance indicators
Increase availability of high-quality, relevant evidence	Qualitative assessment of the availability of high-quality, relevant evidence for decision-making
Increase access to and uptake of evidence for decision-making and accountability	Percentage of clinical cases being treated according to clinical practice guidelines
Increase productive use and appropriate allocation of resources across the system	Percentage of alternate level of care (ALC) days
	Emergency department visits that could be managed elsewhere
	Hospitalization rate for ambulatory care–sensitive conditions
Increase access to key healthcare services	Median wait times in priority areas: cancer surgery, cardiac procedures, cataract surgery, joint replacement, MRI/CT scan, long-term care placement
	Population aged 12 and over who report having a regular medical doctor
	Regular provider of diabetes care
	Percentage of population who report unmet need
Improve patient-centredness, integration and quality of health services	Percentage of patients with cancer who died in acute care beds
	Perceptions of availability and quality of healthcare servicesy
	Inpatient readmission rates for acute myocardial infarction, psychiatric conditions, neonatal care
	Percentage of people accommodated in their first choice of long-term care home
	Percentage of adverse events (in-hospital fractures, new stage 2+ skin ulcers in chronic-stay patients)
Improve healthy behaviours through health promotion and disease prevention	Risk factors for chronic disease
	Flu vaccination
	Preventive screening
Improve clinical outcomes	30-day post-hospital acute myocardiai infarction (AMI) survival rate
	5-year survival rate for prostate, breast, colorectal and lung cancer
	Measure of functional improvement for rehabilitation patients
Improve health status	Teenage pregnancy rates
	Sexually transmitted disease (STD) rates
	Potential years of life lost (PYLL)
	Health-adjusted life expectancy (HALE) for overall population
Increase sustainability and equity of the health system	Healthcare spending
	Change in productivity
	Change in health human resources supply

Finally, all 26 performance indicators were calculated using historical data with five-year trends where available by the MoLTC and a number of partner organizations including the Canadian Institute for Health Information, the Institute for Clinical Evaluative Sciences and Cancer Care Ontario. Results were interpreted in continuous collaboration with the members of the technical expert panel involved in the selection of the set of performance indicators and those organizations involved in data calculation.

Local health system accountability agreements

Prior to the creation of the LHINs, performance management between the payer (MoHLTC) and the providers was largely focused on financial sustainability and in some cases, volumetric measures. The set of performance indicators included in the health system scorecard was a relevant performance management tool to ensure strategic alignment between the health system and local health system strategies.

In 2007, the first generation of accountability agreements were developed between the MoHLTC and the newly created LHINs, which receive about half of the MoHLTC's budget (Bhasin and Williams 2007). Accountability agreements comprising 10 performance indicators aligned with the health system scorecard were developed (Table 4). Five developmental (pilot) performance indicators covering additional performance dimensions were included in the agreement for monitoring by the LHINs and inclusion in accountability agreements in the mid term. These agreements were in turn cascaded to other levels of the health system, such as hospitals or longterm care facilities. A ministry team calculates quarterly results for all performance indicators and posts them in a dashboard that flags performance occurring outside negotiated corridors. LHINs must report quarterly on action plans related to their performance and are accountable for attaining specific annual performance goals. It is anticipated that LHINs will be accountable for an integrated and balanced set of out-come-based performance indicators, with consequences for low-performing LHINs.

TABLE 4.

Core set of performance indicators for accountability agreements between the MoHLTC and the LHINs

Performance dimension	Accountability indicators	Developmental indicators
Improve coordination and integration of services within local health system	Percentage of alternate level of care (ALC) days	N/A
	Rate of emergency department visits that could be managed elsewhere
	Hospitalization rate for ambulatory care–sensitive conditions (ACSC)
	Median wait time to long-term care home placement
Increase access to key healthcare services	90th percentile wait times for cancer surgery	N/A
	90th percentile wait times for diagnostic (MRI/CT) scan
	90th percentile wait times for cardiac bypass procedures
	90th percentile wait times for cataract surgery
	90th percentile wait times for hip and knee replacement
Improve patient-centredness, patient safety and quality of healthcare services	Readmission rates for acute myocardial infarction (AMI)	Perception of change in quality of care
		Percentage of chronic-stay patients in complex continuing care with new stage 2 or greater skin ulcers
		Percentage of in-hospital cancer deaths as a proportion of all cancer deaths
		Psychiatric readmission rates in hospitals
Increase sustainability and equity of the health system	N/A	Change in hospital productivity

Discussion

Lessons from the Ontario experience: Achievements and areas for improvement

While many of the principles articulated by Kaplan and Norton (1992, 1996, 2004) in their work on balanced scorecards and strategy maps have been adopted through this experience, significant modifications were made to the way in which these tools were applied for the following reasons:

• Context of strategy. While the strategies related to specific reform, efforts were clearly defined by the government of Ontario (e.g., wait time reduction strategy), the absence of an overarching strategy for the health system as a whole meant that the strategy map had to be built based on a review and synthesis of the existing collection of disparate strategies. Therefore, the resulting strategy map can be interpreted only as an articulation of the system's emergent, rather than deliberate, strategy (Mintzberg 1994). Owing to the dynamic nature of the policy environment, constructing a strategy map for the health system on the basis of explicit intent was not possible. The resultant scorecard should therefore be viewed only as a snapshot of system performance in areas of strategic importance, which may change from year to year, rather than as a true evaluation of the effectiveness of a particular strategy.

• Unit of analysis. As opposed to the traditional application of the balanced scorecard, the unit of analysis is the entire health system. Therefore, measures chosen are focused largely on system outcomes.

• Role of the ministry. While there is a parallel between the stewardship role of the MoHLTC and that of a traditional corporate headquarters, the former does not have the same degree of control over its “business units” (e.g., provider organizations) as do typical corporations. This makes the job of creating strategic alignment across the system much more difficult.

Several lessons can be drawn from this policy intervention.

First, it illustrated the value of using strategy-based performance information for decision-making. The regular monitoring of a core set of strategy-based performance indicators by the ministry's decision-makers helped refocus the role of the MoHLTC on health system outcomes and its overall stewardship function (WHO Regional Office for Europe 2008).

Second, the process for developing the health system scorecard was important in building credibility for health system performance assessment and improvement. separating the process of strategy mapping (by policy makers), selecting performance indicators (by experts) and negotiating local health system performance improvement targets (by MoHLTC and LHIN executives) are strong assets in building a culture of trust, accountability and performance improvement.

Finally, if the development of accountability agreements between the MoHLTC and the LHINs served as a powerful tool to steer local health system performance, it is important to ensure that local management can respond to local needs while still meeting system-level priorities. If the scorecard provides any support for ensuring this balance, it is by articulating a set of goals – the desired outcomes and their indicators for the system – that can help shift attention away from more restrictive process requirements that would reduce responsiveness to local needs. As the performance measurement system develops, it will be important to ensure that the overall number of indicators is limited so that local management has the ability to introduce additional, local indicators.

However, several challenges arose from this experience.

First, there were difficulties in defining, measuring and regularly monitoring the performance of the health system. In 2006, the strategy map was redeveloped to broaden the scope of health system performance dimensions covered, and a few initial indicators were excluded from the core set and replaced owing to reliability and validity issues.

The second set of challenges relates to the need to provide concise and synthetic information to policy makers about complex systems such as a health system. These efforts have been frustrated by a dearth of effective tools such as visually representative aids (Spiegelhalter 2005). The work has also been limited by difficulties in summarizing overall performance into composite measures.

Third, the strategy map was set through consultative techniques, which require different contributors to take on the interests of the health system. However, performance management in the Canadian context is typically implemented through a negotiated process whereby groups represent their own interests. This approach can lead to a gap between intended strategy and negotiated levels of performance. This challenge may be addressed by allowing LHINs and other partners to determine how best to maximize overall performance in a way that adjusts for local needs.

Finally, the process of systematically linking performance information to the decision-making cycle of the MoHLTC has also proven to be challenging. Even if the links among strategy, performance measurement and accountability have been clearly established, the link to the resource allocation process has to be further developed (Sharpe and Keelin 1998). In addition, the performance improvement phase of the cycle has to be further strengthened through benchmarking and continuous performance improvement activities (Schoen et al. 2006; Kaplan and Norton 2005).

Conclusion

A strategy-based approach offers an innovative way to make health system performance information measurement relevant to the policy environment: establishing the strategic context, selecting indicators to setting performance targets at the local health system level and incorporating them into performance agreements. The Ontario experience suggests that the development and use of strategy-based scorecards can be useful to policy makers if clear principles are respected: performance information has to be relevant, credible and intuitive. However, the main challenge remains in systematically embedding performance information in the decision-making processes of health ministries, and implies a strong corporate discipline as well as investing in priority-setting capacities in order to allocate resources more strategically. Overall, aligning strategies, measurement and local health system accountability proved feasible in Ontario and is, in our opinion, a promising approach to be taken up by other constituencies to improve health system performance.