Health system frameworks and performance indicators in eight countries: A comparative international analysis

Abstract

Objectives:

Performance indicators are a popular mechanism for measuring the quality of healthcare to facilitate both quality improvement and systems management. Few studies make comparative assessments of different countries’ performance indicator frameworks. This study identifies and compares frameworks and performance indicators used in selected Organisation for Economic Co-operation and Development health systems to measure and report on the performance of healthcare organisations and local health systems. Countries involved are Australia, Canada, Denmark, England, the Netherlands, New Zealand, Scotland and the United States.

Methods:

Identification of comparable international indicators and analyses of their characteristics and of their broader national frameworks and contexts were undertaken. Two dimensions of indicators – that they are nationally consistent (used across the country rather than just regionally) and locally relevant (measured and reported publicly at a local level, for example, a health service) – were deemed important.

Results:

The most commonly used domains in performance frameworks were safety, effectiveness and access. The search found 401 indicators that fulfilled the ‘nationally consistent and locally relevant’ criteria. Of these, 45 indicators are reported in more than one country. Cardiovascular, surgery and mental health were the most frequently reported disease groups.

Conclusion:

These comparative data inform researchers and policymakers internationally when designing health performance frameworks and indicator sets.

Introduction

For more than two decades, regulators, policymakers, researchers and clinicians have endeavoured to improve the quality of healthcare by designing and applying indicators of performance. There are national and international incentives for rating the performance of health systems. The World Health Organisation (WHO)¹ and others^2,3 have attempted to rank health systems for the insights gained from global comparisons, while consumers have an interest in selecting the best provider for treatment for their particular condition and knowing that their taxes are being spent wisely.⁴ To meet these multiple demands, performance indicators (‘measurable elements of practice performance for which there is evidence or consensus that they can be used to assess the quality, and hence change of quality, of care provided’) and performance frameworks (‘conceptual frameworks that set out the rationale and design principles for an indicator set’)^5,6 are typically designed to routinely monitor aspects of healthcare performance such as effectiveness, efficiency, safety and quality.⁷ The quest for a single composite indicator of quality, prevalent in the early days of indicator development, has largely been abandoned in favour of multidimensional frameworks.¹ Indicator sets commonly contain a combination of structure, process and outcome assessments.⁸

The Organisation for Economic Co-operation and Development (OECD) publishes 60 internationally comparable indicators of healthcare quality.⁹ These are useful and influential. However, many countries, even those with advanced data systems, have difficulty linking practice performance to outcomes because of limitations in data availability and poor capabilities to link data. Notwithstanding these kinds of shortcomings, it is useful to assess the frameworks and performance indicators in a sample of countries for the insights this provides.

Some health systems have moved faster than others in adopting performance indicators as tools for quality improvement and have made details of their indicators and systems for applying them publicly available at national, regional or institutional levels. We selected eight prominent health systems for review and assessment based on purposively selecting a sample that were exemplars in using indicators and making their data and performance systems available: Australia, Canada, Denmark, England, the Netherlands, New Zealand, Scotland and the United States. At the time of our review, all had made progress in selecting or applying indicators to measure or stimulate improved performance and most had developed a framework for conceptualising performance improvement or indicator use.

This research aims to identify and analyse indicators and their frameworks which report on the performance of healthcare organisations and local health areas. This will provide comparative cases and information on progress for the benefit of regulators, policymakers and researchers within those countries and elsewhere, but is of particular use to policymakers interested in constructing future frameworks.

Methods

We searched for relevant performance indicators and their domains across the eight countries. Following this, we analysed performance indicators that were nationally consistent (used across the country rather than just regionally), locally relevant (measured and reported publicly at a local level, for example, a health service) and measured patient-level metrics. We conducted our study in four stages.

Stage 1: identify comparable nations using performance indicators to monitor and report on healthcare

To make comparison relevant, all selected countries are OECD members and have been classified by the World Bank as high income.¹⁰ Data on the rates of health expenditure and life expectancy for 19 countries were obtained from Australia’s Health 2010¹¹ and from OECD reports⁹ (including Australia, Austria, Canada, Denmark, Finland, France, Germany, Greece, Ireland, Italy, Japan, the Netherlands, New Zealand, Norway, Portugal, Spain, Sweden, the United Kingdom and the United States).¹² After screening by the research team, the eight countries we noted above were selected for detailed review on the basis that each had made substantial progress in using indicators and developing performance frameworks and had made their indicators and performance frameworks widely available.

Stage 2: finding performance indicators

We conducted our Internet search of performance indicator systems in the eight comparator countries in May 2013. The OECD and Departments or Ministries of Health and associated government health organisations in each country were searched. A scoping table detailing the indicators by country was developed. Indicators that were collected consistently on a national scale were included, but could be relevant and useful to local quality improvement efforts. The purpose of the table was to compile an initial ‘long-list’ of available indicators and then to identify a ‘short-list’ of those reported in multiple countries.

Stage 3: detailed review of selected performance indicators

The performance indicators were subjected to a detailed assessment and were classified according to whether they applied to community/hospital/population, country of origin and clinical specialty.

Stage 4: country-specific frameworks

The health system performance frameworks for each country were reviewed together with their accompanying online and published documentation. Domains within the performance frameworks were compared.

Results

Performance indicator frameworks

A summary of each country’s approach to performance indicator use is shown in Tables 1 and 2. Most of the eight countries have an overarching framework for the selection and reporting of indicators which establish the broader aims for their implementation activity and play a large role in indicator selection and use. The number and focus of frameworks varied greatly between the eight countries, but typically included reference to both monitoring and improving quality and efficiency of the healthcare system. There appears to be considerable overlap between the definitions of many of the domains such as effectiveness and appropriateness. Indicators are sometimes also used to promote consumer choice at a regional or local level.

Table 1.

Summary of the eight countries: demographic characteristics and health performance frameworks.

	Australia	Canada	Denmark	England	The Netherlands	New Zealand	Scotland	United States
Estimated population (rank)a,13	22,262,501 (55)	34,568,211 (37)	5,556,452 (11)	53,900,00014 (22)b	16,807,037 (64)	4,365,113 (125)	5,300,00014 (22)b	316,668,567 (3)
Life expectancy at birth: overall years (rank)	82 (10)	82 (13)	79 (48)	80b (30)	81 (21)	81 (25)	80b (30)b	79 (51)
Infant mortality: deaths per 1000 live births (rank)	4.49 (190)	4.78 (182)	4.14 (197)	4.5b (189)b	3.69 (205)	4.65 (145)	4.5b (189)	5.9 (174)
GDP ($US) (rank)	986.7 billion (19)	1.513 trillion (14)	213.6 billion (55)	2.375 trillionb (9)b	718.6 billion (24)	134.2 billion (64)	2.375 trillionb (9)b	15.94 trillion (2)
GDP per capita ($US) (rank)	42,400 (94)	43,400 (142)	38,300 (32)	37,500b (34)b	42,900 (23)	30,200 (50)	37,500b (34)b	50,700 (14)
Healthcare expenditure (%GDP) (rank)	8.7 (2010) (48)	11.3 (15)	11.4 (14)	9.6b (32)b	11.9 (7)	10.1 (30)	9.6b (32)b	17.9 (2)
Type of health system	Universal coverage – Medicare Voluntary private insurance available	Publicly funded – Medicare provides universal coverage for all hospital and physician services out-of-pocket expenses dental, optometry and pharmaceuticals Voluntary private insurance available	Publicly funded – out-of-pocket expenses dental, optometry and pharmaceuticals Voluntary private insurance available	Publicly funded – NHS Voluntary private insurance available	Universal coverage ensured – mix of public and private insurance	Publicly funded Voluntary private insurance available	Publicly funded – NHS Voluntary private insurance available	Public and private insurance – majority private insurance
Health system performance frameworks	PAF and ROGS provide key conceptual principles	Framework is conceptualised across four dimensions: (1) health status, (2) non-medical determinants of health, (3) health system performance and (4) community and health system characteristics	No framework as yet	NHS Outcomes Framework CCG Outcomes Indicator Set QOF	Overarching framework to meet four needs: (1) staying healthy, (2) getting better, (3) living independently with a chronic illness and (4) end-of-life care	Six health targets, three focus on patient access and three on prevention. Primary Health Organisation targets Quality and Safety markers Atlas of Healthcare Variation	12 Quality Outcome Indicators; HEAT targets and other measurements at local and national levels	Two locally reported The Commonwealth Fund – no set framework reports across health dimensions (see below) Hospital Compare – no framework reports on seven dimensions (see below)
Dimensions/domains reported	PAF – safety, effectiveness, appropriateness, quality, access, efficiency, equity, competence, capability, continuity, responsiveness, sustainability ROGS – effectiveness, appropriateness, quality, access, efficiency, equity	Eight domains: (a) acceptability, (b) accessibility, (c) appropriateness, (d) competence (e) continuity, (f) effectiveness, (g) efficiency and (h) safety	Under development	NHS Outcomes – five domains: premature death, quality of life, recovery, positive experience and care/safety CCG – adds to the overarching NHS Outcomes QAO Framework – four domains – clinical, organisational, patient care experiences and additional services	Three overarching themes: (1) quality of care, (2) access to care and (3) healthcare expenditure	Diverse themes. Atlas domains: maternity, gout, demography, cardiovascular disease, polypharmacy and surgical procedures	Described as Quality Ambitions: safe, person-centred and effective	The Commonwealth Fund – four domains: access, prevention and treatment, costs and potentially avoidable hospital use and health outcomes Hospital Compare – seven dimensions – general information, timely and effective care, readmissions, complications and death, use of medical imaging, survey of patients’ experiences, Medicare payment and number of Medicare patients
Framework purpose	ROGS, PAF: - to support improved local-level performance assessment - to support a safe, high-quality Australian health system, through improved transparency and accountability	To determine (1) the health of Canadians and (2) how well the health system performs and operates on the principles of providing report that is secure, that respects Canadians’ privacy and is also consistent, relevant, flexible, integrated, user-friendly and accessible	N/A	NHS Outcomes Framework and CCG Outcomes Indicator Set: - to provide a national-level overview of how well the NHS is performing - to provide an accountability mechanism between the Secretary of State for Health and the NHS Commissioning Board for the effective spend of some £95 billion of public money - to act as a catalyst for driving up quality throughout the NHS by encouraging a change in culture and behaviour QOF is not about performance management per se, but incentivising and rewarding good practice	Used to compare healthcare system performance in other years and countries, with policy and procedure and where possible between healthcare providers	Health targets are a set of national performance measures designed to improve the performance of health services that reflect significant public and government priorities. They provide a focus for action for DHBs and are focussed on accountability not quality improvement. Primary Health Organisation targets to improve the health of enrolled populations and reduce inequalities in health outcomes through supporting clinical governance and rewarding quality improvement within PHOs. Improvements in performance against a range of nationally consistent indicators result in incentive payments to PHOs. QSMs will be used to evaluate the success of the national safety campaign and determine whether the desired changes in practice and reductions in harm and cost have occurred. Atlas of Healthcare Variation: aims to stimulate debate by highlighting variation, rather than making judgements about why variation exists or whether it is appropriate, leading to improvements in healthcare services.	To structure and coordinate the range of measurements that are taken across the NHS Scotland. - 12 QOIs are used for national reporting on longer term progress towards the Quality Ambitions and the Quality Outcomes. These are intended as indicators of quality and do not have associated targets. - HEAT targets describe the specific and short-term priority areas for focussed action in support of the Quality Outcomes.	Commonwealth Fund: uses comparative data to assess the performance of their healthcare systems, establishes priorities for improvement and sets achievement targets Hospital Compare: to help stimulate and support improvements in the quality of care delivered by Medicare hospitals, with the intention of improving hospitals’ quality of care through the distribution of objective, easy to understand data on hospital performance and quality information from consumer perspectives
Data sources	Multiple data sources as identified in the data plan 2-13-2016 Australian Institute of Health and Welfare national data holdings National Partnership Agreement Data submissions Australian Bureau of Statistics data Other collections	Statistics Canada CIHI Canadian Hospital Reporting Project	Clinical Quality Development Programme (RKKP), individual registries and databases, Sundhed.dk	Health and Social Care Information Centre; Royal College of Physicians	Dutch Hospital Databank	Health Quality and Safety Commission/Atlas of healthcare variation Primary Health Organisation Performance Programme	Information Services Division Scotland Scottish Government	Main sources of data include Centre for Medicare and Medicaid, The Joint Commission, Centers for Disease Control and Prevention and other Medicare data and data from within Hospital Referral Regions

DBH: District Health Boards; GDP: gross domestic product; NA: Nott applicabel; NHS: National Health Service; QAO; Quality and Outcomes; PAF: Performance and Accountability Framework; PHO: Primary Health Organisations; ROGS: Report on Government Services; QOF: Quality and Outcomes Framework; CCG: Clinical Commissioning Group; QOI: Quality Outcome Indicator; QSM: quality and safety marker; CIHI: Canadian Institute for Health Information.

^aRank refers to CIA World Factbook,¹⁵ country compared to the rest of the world.

^bFigures for UK in total, not England or Scotland specifically.

Table 2.

Domains of performance indicators by country.

	Australia – PAF	Australia – ROGS	England – High Quality Care for All	England – NHS Outcomes Framework	Canada – Canadian Health Indicator Framework	The Netherlands – dimensions of healthcare performance	The Netherlands – healthcare needs	Scotland – Quality Measurement Framework	United States – Agency for Healthcare Research and Quality	United States – Commonwealth Fund	OECD	Total
Effectiveness	X	X	X	X	X	X			X		X	8
Access	X	X			X	X		X		X	X	7
Safety	X		X	X	X	X			X		X	7
Efficient	X	X			X			X (‘Efficiency and governance’)			X	5
Quality	X	X				X		X				4
Appropriateness	X	X			X			X				4
Outcomes of care/health improvement				Three domains relate to outcomes			Four domains relate to outcomes	X		X		4
Patient-centred/experience			X	X					X		X	4
Cost						X				X	X	3
Equity	X	X									X	3
Responsiveness	X					X					X	3
Competence/capability	X				X							2
Continuity	X				X							2
Timely									X			1
Acceptability					X							1
Sustainability	X											1
Avoidable hospital use										X		1

PAF: Performance and Accountability Framework; ROGS: Report on Government Services; NHS: National Health Service; OECD: Organisation for Economic Co-operation and Development.

In Australia, the National Health Performance Authority (NHPA)¹⁶ was established under the Australian National Health Reform Act 2011¹⁷ as an independent portfolio agency to monitor and report on healthcare system performance; it has since been merged with the Australian Institute of Health and Welfare. NHPA commenced operations in 2012. As part of its Strategic Plan 2012–2015,¹⁶ NHPA is required to regularly review its Performance and Accountability Framework (PAF) to ensure it remains relevant and continues to address the needs of the Australian public for high-quality healthcare. The PAF consists of 48 national indicators: 31 indicators for Medicare Locals now called Primary Health Care Networks; (geographically based primary care co-ordinating agencies) and 17 indicators for performance of Local Hospital Networks and hospitals¹⁸ (see Table 1).

The Canadian framework has two main goals: to determine (1) the health of Canadians and (2) how well the health system performs and operates according to the published principles of providing reports that respect Canadians’ privacy and are also consistent, relevant, flexible, integrated, user-friendly and accessible.^19,20 The indicator framework is conceptualised in terms of the provision of high-quality comparative information across four dimensions. Within these, eight domains of health system performance are defined.^19,21

The Danish do not have a formal framework, but one is currently being developed. Instead, as a proxy framework, the Danish Institute for Quality and Accreditation in Healthcare (IKAS) manages the Danish Healthcare Quality Program (DDKM) as a national accreditation and standard–based programme. At the time of the study, this provides advanced indicators and applies them throughout the country. These standards are overseen by the International Society for Quality in Healthcare (ISQua).²² The Danish National Indicator Project (DNIP) merged with the Clinical Quality Development Programme (RKKP) in 2010.⁸ Although Denmark lacks a formal framework, the DNIP manual outlines the thinking behind its clinical indicators and planned future indicators (1) to improve prevention, diagnostics, treatment and rehabilitation; (2) to provide documentation for setting priorities and (3) to create an information resource for patients and consumers.

An example of a framework which operates at multiple geographical levels is that used in England’s National Health Service (NHS). This comprises three performance frameworks: the NHS Outcomes Framework, which focuses on performance and accountability; the Clinical Commissioning Group (CCG) Outcomes Indicator Set aimed at helping the CCG in planning and benchmarking and providing information to consumers and the Quality and Outcomes Framework (QOF) which is a voluntary pay for performance programme for general practice in England.²³

The Dutch framework, by comparison, is relatively streamlined and more consumer-focussed. Representatives of the Netherlands’ Ministry of Health, Welfare and Sport collaborated with academic researchers to develop the conceptual framework after reviewing the strategic goals of the healthcare system, information needs of policymakers and studying existing theory and international experiences.²⁴ The resulting framework divides healthcare into four specific community needs: (1) staying healthy, (2) getting better, (3) living independently with a chronic illness and (4) end-of-life care.²⁵

New Zealand has included an atlas of healthcare variation as one of its four health system performance monitoring mechanisms,^26–29 and other countries such as the United States and the United Kingdom utilise an atlas, too. The atlas is organised according to clinical areas: maternity, demography, cardiovascular disease, gout, polypharmacy and surgical procedures.²⁹ In 2013, the NZ Health Quality and Safety Commission’s campaign Open for Better Care was commenced to measure whether planned changes in practice occur and whether they have resulted in reduced costs and harms.²⁸

Scotland conceptualised its Quality Measurement Framework on three levels to structure and coordinate the range of measurements that are taken across NHS Scotland. For monitoring long-term progress, there are 12 Quality Outcome Indicators (QOIs) which do not have specific targets; short-term priority areas are focussed on by the ‘HEAT’ targets: Health improvement for the people of Scotland (H), Efficiency and governance improvements (E), Access to services (A) and Treatment appropriate to individuals (T); and the third category includes all other national and local reporting requirements.

In the United States, three identifiable entities report on healthcare performance. One reports nationally (the US Department of Health and Human Services’ Agency for Healthcare Research and Quality (AHRQ)), one internationally (The Commonwealth Fund) and one locally (Hospital Compare). While there is no single integrated framework, the AHRQ measures health system performance across four dimensions¹³ and the Commonwealth Fund aims to be a catalyst for change by identifying promising practices to help the United States create a high-performing healthcare system.¹⁴ The Commonwealth Fund spans four dimensions of health system performance: access, including insurance and affordability of care; prevention and treatment, including quality of ambulatory, acute, post-acute and palliative care; avoidable hospital use and cost, such as care that could have been avoided if the patient received appropriate care initially; and indicators assessing the extent to which people can enjoy long and healthy lives.¹⁵

Of the 11 frameworks published in five countries and the OECD, the most frequently used (self-reported) domains were effectiveness (eight), access and safety (seven each) and efficiency (five; Table 2). There are likely to be considerable overlap between the definitions of some of the domains such as effectiveness and appropriateness. For example, the OECD considers these two domains as separate while the Australian framework considers appropriateness as a subset of effectiveness.^6,18 Because of this, and hierarchical relationships between domains within some frameworks, it is difficult to report the number of indicators used against each domain for each country.

Indicator choice

The search in eight countries found 401 indicators that fulfilled the ‘nationally consistent and locally relevant’ criteria we applied. Of these, 45 indicators are reported in more than one country. Table 3 contains a breakdown of indicators by country.

Table 3.

Nationally consistent and locally relevant indicators by country.

Country – primary source for an indicator	N
England	111
Canada	86
United States	94
Denmark	68
Australia	56
New Zealand	33
Scotland	24
The Netherlands	15

The search yielded 219 community-level, 231 hospital-level and 37 population-level indicators. Some indicators were classified into more than one category (Table 4).

Table 4.

Number of international indicators by domain and community, hospital and population.

Domain	Community	Hospital	Population	Total
Access	41	45	0	86
Patient experience	25	21	1	47
Safety and quality	146	145	2	293
Efficiency	2	11	0	13
Population health outcomes	5	9	34	48
Total	219	231	37	N/A

We classified the indicators, where possible, into major disease groups (Table 5). Cardiovascular, surgery and mental health were the most frequently reported disease groups. Indicators tend to be more specifically linked to a clinical condition or disease group in some countries such as Denmark.³⁰

Table 5.

Nationally consistent and locally relevant indicators by disease group.

Major clinical grouping	N	%a
Cardiovascular disease	62	15
Surgery	45	11
Mental health	42	10
Cancer	26	6
Endocrine disease	21	5
Respiratory disease	20	5
Musculoskeletal	17	4
Maternal and child health	17	4
Emergency	11	3
Radiology	6	1
Chronic kidney disease	5	1
Neurological disease	4	1

^aDenominator = 401.

Review processes

Regular review of the performance framework and indicators is conducted in most of the eight countries by government or government-funded, arm’s-length bodies. For example, the Canadian framework has continually developed and evolved since its inception, as a result of collaboration from a dedicated group including the Canadian Institute for Health Information (CIHI), Statistics Canada (SC), Health Canada (HC) and representatives from other stakeholder groups.¹⁹ Similarly, the National Institute for Health and Clinical Excellence (NICE) has a key role in indicator development in England. NICE is responsible for managing the development process of clinical and public health indicators for the QOF and the CCG indicator sets.³¹ NICE also recommends whether the existing indicators should continue to be part of the frameworks. NICE has developed guides,^32,33 which set out in detail the processes involved in managing the development of indicators. Thus, indicators tend to be developed in a relatively open and transparent way, with input from individuals and stakeholder organisations. Of course, this statement masks the contested and political components of indicator development and use, which does not figure explicitly in policy documentation, academic articles or this review and is mostly country-specific.³⁴

Reporting

The timing and mechanism of reporting on indicators were not consistent between countries, nor were they always internally consistent. This can be seen in the reporting on Canada’s health system performance, where various indicators are reported via multiple channels. There were 101 performance indicators listed on the SC website.³⁵ The CIHI also has indicators listed under the Canadian Hospital Reporting Project (CHRP). While some of the indicators are the same as those listed by SC, there are some additional hospital performance indicators (21 in total).³⁶ Additionally, the Government of Canada³⁷ has a candidate list of 70 indicators that were approved for use by Federal, Provincial and Territorial Ministers of Health in 2004. However, it is difficult to gauge how many indicators are in use, because only certain indicators are selected for inclusion in the annual reports and there appear to be various degrees of overlap. For example, the Health Indicators 2013 report²⁰ provides results for 88 indicators, 13 of which were additional indicators chosen to be included to measure performance by socioeconomic status at provincial and national levels.²⁰ Although this appears confusing from an external perspective, variable reporting may be more effective in some instances, as CIHI addresses reporting needs by acknowledging different audiences and tailoring reporting for their requirements.

Meanwhile, in the Netherlands, a report detailing results for all 125 indicators is published every 2 years.³⁸ From 2011, the information was updated via a website twice a year and from 2014, the report will be published every 4 years.³⁸ The indicators are reported at the national level, not locally. Indicators reported locally (at regional levels) occur via the Dutch Hospital Database. There are two dedicated websites that provide consumers of healthcare with information about the quality of a service and provide ratings for their service.³⁹ These are Independer – www.independer.nl – and Kiesbeter – www.kiesbeter.nl or ‘Choose Better’.

Similarly, in Denmark, clinical indicators are reported, and a structured audit process is initiated every 6 months by audit groups of clinicians at national, regional and local levels to explain the risk-adjusted results and to plan improvements. After the audit process is complete, the data are released publicly, including comments on the results from the audit groups.⁴⁰ Reports on many of the indicators are available on the www.sundhed.dk website.

Discussion

In this study, the eight countries selected for review were using indicators and had implemented a performance framework, several over more than a decade. The progress they have made, and choices taken in selecting and using indicators, might be of value for other health systems in contemplating the development of their indicators or frameworks, or modifying their performance mechanisms. A key finding was the widespread support for implementing a healthcare system performance framework. The importance of a logical, universally acceptable and viable conceptual framework to underpin development of a national performance indicator set is also emphasised in the literature.^41,42 A conceptual framework sets out the rationale and design principles for the indicator set and links it to the wider health system context. It seeks to answer the question ‘performance of what – and to what ends?’⁶ Reasons given for developing such a framework are as follows: (1) to define the scope and dimensions of measurement;^24,6 (2) to help align the performance measurement system with other policy priorities and financial exigencies;⁴³ (3) to provide a clearly defined vision to encourage buy-in by clinicians and consumers^24,43 and (4) by involving potential end-users of the data in the design of the framework, to ensure its future usefulness.⁴⁴ A conceptual framework encompassing multiple domains and with balanced representation across structure, process and outcome indicators is considered to be a key element of health reform over time.⁴⁵

Although we presented the self-reported domains by country, consistency of definitions between countries and the level of semantic overlap was not tested; however, these are likely to be substantial. For example in the Australian PAF, the domain appropriateness is subordinate to, or sub-class of, the domain effectiveness.¹⁸ In Canada, these two domains are not grouped, but are classified as separate concepts. Definitions for these domains are often not explicit in the policy documents.²⁰ Definitional consistency between countries should be the subject of more research or efforts to internationally standardise.

Although there is a substantial literature dealing with the design, properties and scientific soundness of individual indicators, there is considerably less attention given to how indicators are used in practice and the impact they may have on the behaviour of health professionals, or on the quality of care. While there is no answer to questions such as how many indicators, which domains should be targeted or what should be the right mix of indicators, there is a fundamental debate centred on whether the purpose of performance indicators is accountability or quality improvement.^41–43,46 Internationally, there is a split between those countries which emphasise public reporting and accountability (e.g. the UK NHS’s ‘star-ratings’ system of 2001–2005)⁴⁷ and those that use results for non-publicised feedback to organisations to stimulate improvement.

It is broadly agreed that monitoring performance imposes an inherent pressure on healthcare organisations or services to improve practice.⁴⁸ However, the extent to which this is accomplished is disputed and under-researched. The paucity of research examining the links between indicators and improvement may be due to the difficulty in attributing change to any particular policy initiative or intervention.⁴⁹

The literature supports the use of performance indicators, suggesting that their impact is more likely to be on provider rather than consumer behaviour.^43,50 However, there is a general call for more good quality studies on impact.^50–52

England and Canada do the most extensive research and development work to select indicators. The role of NICE in England exemplifies a thorough and considered approach to continued indicator development. The process of review, seen especially in England, Canada and Australia, is critical to the continued development of performance indicators and their use.

Conclusion

A large amount of comparative information about international performance indicators is now available.⁵³ We examined the systems in use in eight countries. Assessing commonalities and differences between indicator specification and application in comparable health systems may be of value to regulators, policymakers, researchers and clinicians and forms a foundation for further research into the practical impact of indicators on the quality of healthcare.