Abstract
The objective of this paper is to apply a framework for country‐level performance assessment to the cities of Montreal, Canada, and Barcelona, Spain, and to use this framework to explore and understand the differences in their health systems. The UK National Health Service Performance Assessment Framework was chosen. Its indicators went through a process of selection, adaptation and prioritisation. Most of them were calculated for the period 2001–3, with data obtained from epidemiological, activity and economic registries. Montreal has a higher number of old people living alone and with limitations on performing one or more activities of daily life, as well as longer hospital stays for several conditions, especially in the case of elderly patients. This highlights a lack of mid‐term, long‐term and home care services. Diabetes‐avoidable hospitalisation rates are also significant in Montreal, and are likely to improve following reforms in primary care. Efficient health policies such as generic drug prescription and major ambulatory surgery are lower in Barcelona. Rates of caesarean deliveries are higher in Barcelona, owing to demographics and clinical practice. Waiting times for knee arthroplasty are longer in Barcelona, which has triggered a plan to reduce them. In both cities, avoidable mortality and the prevalence of smoking have been identified as areas for improvement through preventive services. In conclusion, performance assessment fits perfectly in an urban context, as it has been shown to be a useful tool in designing and monitoring the accomplishment of programmes in both cities, to assess the performance of the services delivered, and for use in policy development.
The objective of performance assessment is to provide governments, health authorities and populations with appropriate information about the state of their healthcare systems.1 Some of the aims of any relevant performance assessment are to build an evidence base on the relationship between a health‐system design and its performance, providing policy makers with crucial tools to develop effective, efficient and equitable systems, and to help determine priorities for healthcare interventions, contributing to their design and management.2
In recent years, an increasing amount of work and attention has been devoted to the measurement of performance within health systems at the country level, with an aim to improve them. The drivers of this trend are rooted in the current characteristics of the environment where healthcare is provided, and the concerns of governments and populations.3 Resource spending constraints, growing public expectations and concerns about safety, quality and equity are increasing the pressure on accountable healthcare systems.
Urban contexts have the elements to develop such an initiative. Big cities have several social problems, inherent to their development: low‐income families, unemployment, immigration, inequalities, an ageing population, small social networks and so on. In particular, the existing health problems in Montreal, Canada and Barcelona, Spain are among the consequences of these particular social conditions: higher HIV/AIDS rates, higher illegal drug consumption, lower birth rates, stronger social class inequalities in health status, higher prevalence of mental health problems and so on.4 Moreover, in many big cities, the available public health and healthcare services try to address these issues with specific policies and programmes, as local governments tend to have an active role in policy development and service delivery assurance as well as public health functions.5 As a result, urban contexts could be seen as a microenvironment in which to develop a performance assessment initiative, much more focused than a country‐level initiative, making it possible to measure the achievement of health goals and the effectiveness of the programmes in place.6
The objective of this paper was to apply a framework for country‐level performance assessment to the cities of Montreal and Barcelona, and to use this framework to explore and understand differences in their health systems. The present initiative is the first one at the city level, which is an indication of the uniqueness of this project, and could be an incentive for other cities interested in this kind of framework.
Methods
The project on performance assessment began in 2002, as one of the agreements between the Montreal Health Authority (Régie Régionale) and the Health Consortium of Barcelona, under the auspices of the declaration of cooperation between the governments of Quebec and Catalunya. Its objective was to develop a set of indicators that would allow Barcelona and Montreal to assess the performance of their healthcare services.
First, a systematic search in PubMed was carried out, looking for performance assessment initiatives that have actually been put in place in different countries. 7 Such initiatives were identified in the UK, Canada, Australia, New Zealand and Sweden.8,9,10,11,12,13 Afterwards, the applicability of each of the five initiatives to an urban context was assessed, valuing their capability to reflect the nuances of that setting, assessing the health status of the inhabitants of big cities, and the public health and healthcare services available. From the revised initiatives, the indicators contained in the UK initiative —UK National Health Service (NHS) Performance Assessment Framework—were seen as the most suitable for Montreal and Barcelona, as they take into account general dimensions of health, while also focusing on areas more specific to those cities (ie, HIV/AIDS, health of immigrants, concerns about community care, particular types of cancer and geographical inequities).8 Additionally, most of the indicators could be obtained from epidemiological, activity and economic information regularly available. Finally, the UK healthcare system is quite similar to the Canadian and Spanish systems, and hence one could expect that the UK NHS Performance Assessment Framework would be applicable.
The UK NHS Performance Assessment Framework, in its 2002 version, defines a set of 51 indicators (table 1) grouped in dimensions.8 The application of the framework to an urban context would be as follows:
Table 1 UK National Health Service performance indicators, 2002*.
| Dimension | Indicator |
|---|---|
| Health improvement | Deaths from all causes (15–64 years) |
| Deaths from all causes (64–74 years) | |
| Deaths from cancer | |
| Deaths from all circulatory diseases | |
| Suicide rates | |
| Deaths from accidents | |
| Serious injury from accidents | |
| Fair access | Access to elective surgery (surgery rates for hip, knee, cataract replacements and coronary heart disease) |
| Access to family planning and services (conception rates for teenage girls) | |
| Access to dentists (number of people registered with a dentist) | |
| Access to health promotion (early detection of cancer) | |
| Access to community services (number of general practitioners, practice availability) | |
| Effective delivery of appropriate healthcare | Health promotion/disease prevention (childhood immunisations, early detection of cancer) Appropriateness of surgery (surgery rates, inappropriate surgery) Primary care management (acute care, chronic care, mental healthcare, cost‐effective prescribing) Compliance with standards of care (returning home after a stroke, returning home after a hip fracture) |
| Efficiency | Maximising the use of resources (day case rate, length of stay, unit costs, generic prescribing) |
| Patient—care giver experience | Accessibility (patients who wait <2 h for emergency admissions, cancellations of operations for non‐medical reasons) |
| Cooperation/communication (delayed discharge, first outpatient appointment that the patient did not attend) | |
| Waiting times (outpatients seen within 13 weeks of referral, number of people on the waiting list for ⩾18 months) | |
| Satisfaction (patients' complaints) | |
| Health outcomes | Reducing the level of risk (conception rates for teenage girls) Reducing the level of disease or impairment (adverse events or complications of treatment, decayed, missing or filled teeth for 5‐year‐old children) Improving the quality of life for users and care givers (hospital admissions for older people, psychiatric readmissions) Reducing premature deaths (infant deaths, survival rates for cancer, avoidable deaths, hospital premature deaths) |
* Indicators are grouped in the table, to avoid overwhelming the reader with too much detailed information.
Health improvement: To reflect the overarching aims of improving the general health of urban populations and reducing health inequalities within the cities.
Fair access: To assert that the city must offer fair access to health services in relation to people's needs, irrespective of neighbourhood or district of residence, socio‐economic group, ethnicity, age or sex.
Effective delivery of appropriate healthcare: To acknowledge that there must be effective, appropriate and timely services that comply with agreed‐upon standards.
Efficiency: To ensure that healthcare services are delivered with the minimum of waste, and that the city uses its resources to achieve value for money.
Patient/care giver experience: To assess the way in which patients and their care givers experience and view the quality of the care they receive.
Health outcomes: To assess the direct contribution of the city healthcare services to improvements in overall health.
The Barcelona city officials invited a multidisciplinary group of experts (10 people) to join the project. These experts were outstanding professionals working in the Barcelona healthcare system in the areas of primary care, long‐term care, mental health, public health, health policy and planning, hospital management and healthcare services research. The multidisciplinary group of experts held three meetings to review the UK NHS Performance Assessment Framework indicators. During these meetings, indicators were selected and adapted and new indicators were added, finally arriving at a set of 107 indicators. To prioritise the indicators obtained, the multidisciplinary group of experts, together with Montreal and Barcelona officials, then assessed four dimensions for each indicator—importance, reliability, information readily available and comparability—on a scale from 1 to 5. Indicators with worse scores, and those that could not be obtained at least in the short term, were dropped, leaving a final set of 76 indicators. Afterwards, three video conferences were carried out in which the Montreal and Barcelona officials discussed the specifications of the indicators, and developed and agreed upon specific technical definitions for each of them. Information for the last year available was collected.
This was an observational cross‐sectional study. For each indicator, counts, percentages or rates were calculated, in accordance with the definitions made. When comparing percentages and rates, they were age standardised using the world standard population 2000–25, so that the values of both cities would be directly comparable. Additionally, information was collected on a set of descriptive indicators related to basic demographic information of both cities and the main characteristics of their healthcare systems.
Results
Populations in both cities are comparable in terms of size, demographic structure and literacy level (table 2). In Barcelona, there is a bigger proportion of people aged >64 years, especially women (24.9% vs 17.9%); however, in Montreal there is a higher number of people aged >64 years living alone (35.6% vs 25.8%). Montreal has a larger foreign‐born population, as massive immigration is a recent phenomenon in Barcelona; in 2003, foreigners represented 12.8% of the population of the city, while in 1991 it was 1.4%. Regarding resources, Montreal and Barcelona are quite similar in terms of the number of inhabitants per primary care doctor, per nurse and the number of public acute beds. It was not possible to compare long‐term care resources because the organisation and type of services available are different. Per capita public healthcare spending is higher in Montreal.
Table 2 Description of the population and healthcare services.
| Indicator | Sex | Barcelona | Montreal |
|---|---|---|---|
| Population | |||
| Inhabitants (n) | — | 1503884 (2001) | 1812740 (2001) |
| Population aged >64 years (%) | Men | 15.1 (2001) | 12.4 (2001) |
| Women | 24.9 (2001) | 17.9 (2001) | |
| Population aged >74 years (%) | Men | 4.5 (2001) | 4.9 (2001) |
| Women | 12.5 (2001) | 8.9 (2001) | |
| Population aged >64 years living alone (%) | — | 25.8 (2001) | 35.6 (2001) |
| Population with university degree (%) | — | 21.2 (2001) | 23.1 (2001) |
| Foreign population (%) | — | 12.8 (2003) | 27.6 (2003) |
| Structure | |||
| Inhabitants per public primary care doctor (n) | — | 1829 (2003) | 1657 (2002) |
| Inhabitants per nurse (n) | — | 1885 (2003) | 1733 (2002) |
| Public acute beds per 1000 inhabitants (n) | — | 3.1 (2003) | 2.8 (2002) |
| Public long‐term care beds per 1000 inhabitants (n) | — | 3.7 (2003) | 51.9 (2002) |
| Costs | |||
| Per capita healthcare spending (€)* | — | 1035.5 (2001) | 1584.3 (2001) |
Sources: Census 2001, Departament d'Estadística, Ajuntament de Barcelona; Human Resources Registry, Consorci Sanitari de Barcelona; Recensement 2001, Statistique Canada; Éco‐santé Québec 2002, Ministère de la Santé et des Services Sociaux du Québec.
*1Can$ = €0.6667 using 2002 Purchasing Power Parity conversion factors.
In the health improvement dimension, Montreal has a larger proportion of people reporting very good health status (91.6% vs 79.5% in men and 84.7% vs 70.9% in women; table 3). At the same time, Montreal has a larger proportion of men aged >65 years with limitations to perform one or more activities of daily living (35.3% vs 24.0%), whereas Barcelona has a larger proportion of women (42.3% vs 39.6%) in this situation. With regard to mortality, Montreal has higher rates of coronary diseases and suicide, especially for men. Barcelona, by contrast, has higher rates of cancer, cerebrovascular diseases and respiratory diseases (table 3). Traffic injuries, mortality and prevalence of smoking are similar in both cities. The percentage of smoking population is high in both cities.
Table 3 Performance indicator values for Barcelona and Montreal.
| Indicator | Sex | Barcelona | Montreal |
|---|---|---|---|
| Health improvement | |||
| Population aged >65 years with limitations to perform one or more activities of daily living (%) | Men | 24.0 (2000) | 35.3 (2001) |
| Women | 42.3 (2000) | 39.6 (2001) | |
| Population with good or very good health status (%) | Men | 79.5 (2000) | 91.6 (2001) |
| Women | 70.9 (2000) | 84.7 (2001) | |
| Mortality from cancer per 100 000 inhabitants aged <75 years | Men | 198.8 (2002) | 128.6 (1997–9) |
| Women | 100.6 (2002) | 90.7 (1997–99) | |
| Mortality from coronary diseases per 100 000 inhabitants aged <75 years | Men | 67.4 (2002) | 77.4 (1997–9) |
| Women | 23.5 (2002) | 25.9 (1997–9) | |
| Mortality from cerebrovascular diseases per 100 000 inhabitants aged <75 years | Men | 40.6 (2002) | 13.5 (1997–9) |
| Women | 29.0 (2002) | 8.3 (1997–9) | |
| Mortality from respiratory diseases per 100 000 inhabitants aged <75 years | Men | 65.9 (2002) | 20.6 (1997–9) |
| Women | 23.7 (2002) | 11.6 (1997–9) | |
| Mortality from suicide per 100 000 inhabitants aged 15–49 years | Men | 15.6 (2002) | 40.7 (1997–9) |
| Women | 3.9 (2002) | 8.2 (1997–9) | |
| Mortality from traffic injuries per 100 000 inhabitants aged 15–44 years | Men | 9.9 (2002) | 11.7 (1997–9) |
| Women | 3.3 (2002) | 1.5 (1997–9) | |
| Smoking population (%) | Men | 35.3 (2000) | 30.6 (2001) |
| Women | 22.9 (2000) | 24.6 (2001) | |
| Access | |||
| Hospitalisation rate for femur fracture surgery per 1000 inhabitants | — | 0.436 (2003) | 0.449 (2001–3) |
| Hospitalisation rate for stroke per 1000 inhabitants | — | 0.822 (2003) | 0.740 (2001–3) |
| Hospitalisation rate for myocardial infarction per 1000 inhabitants | — | 0.749 (2003) | 1.107 (2001–3) |
| Hospitalisation rate for angioplasty per 1000 inhabitants | — | 0.527 (2003) | 0.941 (2001–3) |
| Hospitalisation rate for bypass per 1000 inhabitants | — | 0.192 (2003) | 0.425 (2001–3) |
| hospitalisation rate for knee replacement per 1000 inhabitants | — | 0.491 (2003) | 0.228 (2001–3) |
| Emergencies per 1000 inhabitants (n) | — | 485.8 (2003) | 463.2 (2002–3) |
| Effective delivery of appropriate services | |||
| Diabetes‐avoidable hospitalisation rate per 1000 inhabitants | — | 0.012 (2003) | 0.026 (2001–3) |
| Duodenal and gastric ulcer‐avoidable hospitalisation rate per 1000 inhabitants | — | 0.185 (2003) | 0.126 (2001–3) |
| Cerebrovascular accident discharges of patients aged >65 years with >20 days of length of stay (%) | — | 14.2 (2003) | 35.2 (2001–3) |
| Discharges of patients with femur fracture aged >65 years with >20 days of length of stay (%) | — | 14.9 (2003) | 20.5 (2001–3) |
| Caesarean deliveries (%) | Women | 22.2 (2003) | 20.4 (2001–3) |
| Emergency admissions (%) | — | 41.2% (2003) | 58.1% (2002–3) |
| Efficiency | |||
| Substitution for major ambulatory surgery (%) | — | 57.9 (2003) | 91.9 (2001–3) |
| Gross average length of stay for ischaemic diseases | — | 8.04 (2003) | 7.84 (2001–3) |
| Gross average length of stay for cerebrovascular diseases | — | 11.06 (2003) | 15.74 (2001–3) |
| Generic drug prescription (%) | — | 14.5 (2003) | 36.5 (2001–3) |
| User experience | |||
| Waiting list for knee arthroplasty (months) | — | 22.3 (2003) | 7.8 (2004) |
| Waiting list for cataract surgery (months) | — | 2.2 (2003) | 2.3 (2004) |
| Health outcomes | |||
| Inhospital mortality from bypass per 100 bypass discharges | — | 5.17 (2003) | 2.79 (2001–3) |
| Inhospital mortality from femur fractures per 100 femur fracture discharges | — | 4.08 (2003) | 7.71 (2001–3) |
| Inhospital mortality from myocardial infarction per 100 myocardial infarction discharges | — | 12.87 (2003) | 12.09 (2001–3) |
| Inhospital mortality from stroke per 100 000 inhabitants | — | 18.54 (2003) | 17.51 (2001–3) |
| Cerebrovascular disease‐avoidable mortality per 100 000 inhabitants aged 35–64 years | Men | 21.2 (2002) | 15.4 (1997–9) |
| Women | 10.2 (2002) | 9.9 (1997–9) | |
| Lung cancer‐avoidable mortality per 100 000 inhabitants aged 5–64 years | Men | 26.8 (2002) | 31.5 (1997–9) |
| Women | 6.6 (2002) | 22.5 (1997–9) | |
| Cirrhosis‐avoidable mortality per 100 000 inhabitants aged 15–74 years | Men | 12.1 (2002) | 13.8 (1997–9) |
| Women | 3.6 (2002) | 4.8 (1997–99) | |
Sources: Barcelona Health Survey 2000, Agència de Salut Pública de Barcelona; Mortality Registry, Agència de Salut Pública de Barcelona; Hospital Discharge Registry, Consorci Sanitari de Barcelona; Provider Contracts, Consorci Sanitari de Barcelona; Recensement 2001, Statistique Canada; Fichier des décès, Ministère de la Santé et des Services Sociaux du Québec; Med‐Écho, Ministère de la Santé et des Services Sociaux du Québec; Éco‐santé Québec 2002, Ministère de la Santé et des Services Sociaux du Québec.
Rates have been standardised using the world standard population 2000–25, hence they are directly comparable.
With regard to access, Montreal has higher rates of myocardial infarction, angioplasty and hospitalisation due to bypass; hospitalisation rates due to knee replacement are higher in Barcelona; the number of emergencies per inhabitant is similar in both cities (table 3). In the effective delivery of appropriate services dimension, the number of patients aged >65 years with cerebrovascular accident or with femur fracture, with a hospital stay of >20 days, is higher in Montreal, as well as diabetes‐avoidable hospitalisation rates. The percentage of emergency admissions is also higher in Montreal (58.1% vs 41.2%). Barcelona has a higher percentage of caesarean deliveries (22.2% vs 20.4%).
In the efficiency dimension, Barcelona has a lower substitution rate for major ambulatory surgery (57.9% vs 91.9%), and lower generic drug prescription (14.5% vs 36.5%; table 3). Gross length of stay of patients with cerebrovascular diseases is higher in Montreal. Regarding user experience, waiting times for knee arthroplasty are much longer in Barcelona (22.3 vs 7.8 months; table 3).
Finally, in the health outcomes dimension, Montreal has a higher inhospital mortality from femur fractures, whereas Barcelona has a higher inhospital mortality from bypass (table 3). Regarding avoidable mortality, Montreal has higher lung cancer rates for both sexes; Barcelona has higher cerebrovascular disease rates, especially for men.
Discussion
The results of this project are an example of the applicability and usefulness of performance assessment initiatives to an urban context. This is the first effort in applying this kind of framework at the city level. The results made it possible to identify a few issues related to demographics, acute and preventive services available, clinical practice and management techniques in Montreal and Barcelona.
In terms of demographics, Montreal has a higher number of old people living alone and with limitations to perform one or more activities of daily living (table 2). Related to this is the fact that Montreal has a longer length of stay for several conditions, especially in the case of elderly patients (table 3). This highlights a lack of mid‐term and long‐term healthcare services, and a greater need for home‐care services in Montreal. Taking action, in early 2005, a plan to improve post‐acute care. Diabetes‐avoidable hospitalisation was initiated in Montreal rates are also significant in Montreal (table 3). This is something that the Montreal local health authorities hope to see an improvement in for the coming years, following an important primary care reform that was started in mid‐2004.
What this paper adds
This paper presents the application of a framework for country‐level performance assessment to the cities of Montreal and Barcelona, and explains how this framework was used to explore and understand differences in their health systems.
The results show that performance assessment is applicable in an urban context and can be used as a tool for policy development, as it has allowed both cities to identify a few issues related to demographics, clinical practice and management techniques.
The application of a performance framework to two cities, the collaboration between both cities and the extensions of the project make this a unique and successful project in the international arena.
Policy implications
Performance assessment is a useful tool to design and monitor the accomplishment of policies and to assess the services delivered.
This project is allowing decision makers in Montreal and Barcelona to identify the strengths and weaknesses of the healthcare services of both cities, in terms of available resources and quality of care.
Generic drug prescription and major ambulatory surgery are lower in Barcelona, indicating the low expansion of efficient health policies. Rates of caesarean deliveries are higher in Barcelona, which is probably related to demographics and clinical practice. Waiting times for knee arthroplasty are longer and hospitalisation rates for this condition are higher in Barcelona, mainly because of the slow diffusion of this technique in the city hospitals. This fact led to the development of a plan to reduce them in 1999, mainly guaranteeing a certain annual number of interventions, and the development of clinical units for knee arthroplasty, among other measures. Both indicators should decrease in the future as the number of people waiting for this operation diminishes.
In both cities, avoidable mortality indicates an area for improvement: lung cancer and suicide in Montreal and cerebrovascular diseases in Barcelona give plenty of action to preventive services (table 3). This is also the case for the prevalence of smoking.
All the performance assessment initiatives at the country level previously mentioned have actually been put in place, and are still being used and continuously improved, a testimony to the importance and usefulness of such initiatives for policy makers and populations. By drawing on other experiences, countries can adapt their frameworks to their own particular conditions, and learn from examples of good practice, thus avoiding errors made elsewhere.14,15,16,17 In fact, some specific studies have been carried out comparing the performance of healthcare services among countries, and within countries, looking for the relationship between health results, health determinants and resources.18,19,20,21,22,23,24 Additionally, this kind of approach could be used to assess health sector reforms and compare healthcare providers.25,26 However, the present initiative is the first at the city level, which confirms the uniqueness of this project, and could be an incentive for other cities interested in this kind of framework.
The comparison of two cities made it possible to confirm previously identified problem areas in each city (eg high mortality from suicide in Montreal, long waiting times for knee artroplasty in Barcelona), and, more interestingly, made it possible to identify other areas in which there is room for improvement (eg, caesarean deliveries and generic drug prescription in Barcelona, emergency admissions in Montreal). In some sense, instead of looking for “benchmarks” for each indicator, each city acts as a benchmark for the other. Continuing this project by collecting annual data would make it possible to monitor any improvements. The international comparison of cities makes the project especially attractive, but even without an intercity comparison, this project is an example of how the application of a performance assessment framework to an urban context could be useful in any city setting, where there is an interest in health system assessment and improvement.
As in any initiative, some limitations could be found: in this study, these are mainly related to the process followed and the availability of information. First, the selection of indicators was based on a pre‐existing set of indicators and carried out by a group of experts. The selection of the UK NHS Performance Assessment Framework was based on a careful assessment of the applicability of all the initiatives found to an urban context, and the similarities of the healthcare systems. The use of a group of experts to select, add and prioritise indicators has the limitations intrinsic to the participation of a particular group of people. Finally, some indicators that either the Barcelona or the Montreal officials thought would be interesting to include were dropped in the prioritisation process because they were not available in both cities. Although this is a limitation, the comparison of available data sources was useful to identify information gaps, and says a lot about the strengths and weaknesses of a system. Improving information systems on drug consumption, vaccines and primary care in Montreal, and on hospital activity information at the Barcelona district level would be a way to advance.
At present, we are working on the development of a strategy to disseminate the results of this project. It will include the presentation of the project in local, national, and international forums through scientific meetings and papers. We have also developed a web page for the project, based on geographical information systems, so that people can interactively see the performance of each indicator, together with its technical definition, and the comparison between cities (http://www.cmis.mtl.rtss.qc.ca/fr/performance/coop_mtl_barcelone/barcelone_presentation.html). We are already working on the continuation of this project, collecting annual data and extending the project to the metropolitan areas of both cities.
This project allows decision makers in Montreal and Barcelona to identify the strengths and weaknesses of the healthcare services of both cities, in terms of available resources and quality of care. The application of a performance assessment framework to an urban context, the collaboration between two cities and the extensions of the project make this a unique and successful project in the international arena.
Acknowledgements
We thank Agència de Salut Pública de Barcelona, Consorci Sanitari de Barcelona, Corporació Sanitària de Barcelona and Agence de la Santé et des Services Sociaux de Montréal, our partners in this project; and the Montreal and Barcelona group on performance of healthcare services: Josep Benet, Mike Benigeri, Carme Borrell, Marc Bourguignon, Xavier Castells, Louis Côté, Danièle Dorval, Jaume Estany, Gonçal Foz, Lluís Franch, Anna García‐Altés, Josep Ma Giralt, Àlex Guarga, José Manuel Pérez‐Castejón, Antoni Plasència, Alfonso Pozuelo, Aina Plaza, Elvira Torné, Corinne Zara and Lauriane Zonco.
Abbreviations
NHS - National Health Service
Footnotes
Funding: No extra funding apart from the authors' institutions.
Competing interests: None.
References
- 1.Murray C J L, Frenk J. A framework for assessing the performance of health systems. Bull World Health Organ 200078811–820. [PMC free article] [PubMed] [Google Scholar]
- 2.Pan American Health Organization Health system performance assessment and improvement in the region of the Americas. Washington, DC: PAHO, 2001
- 3.Hurst J. Performance measurement and performance management in OECD health systems: an overview of issues and challenges. “Measuring up” improving health systems performance in OECD countries. OECD Health Conference on Performance Measurement and Reporting. Ottawa, November 2001
- 4.Borrell C, Pasarín M. Inequalities in health and urban areas. Gac Sanit 200418(Suppl 2)1–4. [DOI] [PubMed] [Google Scholar]
- 5.Líndez P, Villalbí J R, Vaqué J. Public health functions, activities and structures: the role of large and middle‐sized municipalities. Gac Sanit 200115164–171. [DOI] [PubMed] [Google Scholar]
- 6.Derose S F, Schuster M A, Fielding J E.et al Public health quality measurement: concepts and challenges. Annu Rev Public Health 2002231–21. [DOI] [PubMed] [Google Scholar]
- 7.García‐Altés A, Zonco L, Borrell C.et al Measuring the performance of health care services: a review of international experiences and their application to an urban context. Gac Sanit 200620316–324. [DOI] [PubMed] [Google Scholar]
- 8.National Health Service NHS performance indicators. National figures: February 2002. London: NHS, 2003
- 9.Canadian Institute for Health Information Health indicators. Vol 2003, Number 2. Ottawa: Canadian Institute for Health Information, 2003 (Catalogue Number 82‐221‐XIE),
- 10.National Health Performance Committee Fourth national report on health sector performance indicators. A report to the Australian Health Ministers' conference. Sydney: NSW Health Department, 2000
- 11.Or Z.Improving the performance of health care systems: from measures to action (a review of experiences in four OECD countries). Paris: OECD, 2002 (Labour Market and Social Policy, Occasional Papers Number 57),
- 12.Hurst J, Jee‐Hugues M.Performance measurement and performance management in OECD Health Systems. Paris: OECD, 2000 (Labour Market and Social Policy, Occasional Paper Number 47),
- 13.McIntyre D, Rogers L, Heier E J. Overview, history and objectives of performance measurement. Health Care Financ Rev 2001227–21. [PMC free article] [PubMed] [Google Scholar]
- 14.Evans D B, Edejer T T, Lauer J.et al Measuring quality: from the system to the provider. Int J Qual Health Care 200113439–446. [DOI] [PubMed] [Google Scholar]
- 15.Upshaw V. The National Public Health Performance Standards Program: would it strengthen governance of local public health? J Public Health Pract 200061–18. [DOI] [PubMed] [Google Scholar]
- 16.McPake B, Mills A. What can we learn from international comparisons of health systems and health system reform? Bull World Health Organ 200078811–820. [PMC free article] [PubMed] [Google Scholar]
- 17.Mannion R, Goddard M. Performance measurement and improvement in health care. Appl Health Econ Health Policy 2002113–23. [PubMed] [Google Scholar]
- 18.Feachem R G A, Sekhri N K, White K L. Getting more for their dollar: a comparison of the NHS with California's Kaiser Permanente. BMJ 2002324135–141. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Evans D B, Tandon A, Murray C J L.et al Comparative efficiency of national health systems: cross national econometric analysis. BMJ 2001323307–310. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.The Conference Board of Canada Challenging health care system sustainability: understanding health system performance of leading countries. Ottawa: The Conference Board of Canada, 2004
- 21.Hussey P S, Anderson G F, Osborn R.et al How does the quality of care compare in five countries? Health Aff (Millwood) 20042389–99. [DOI] [PubMed] [Google Scholar]
- 22.Organisation for Economic Cooperation and Development Towards high‐performing health systems. Paris: OECD, 2004
- 23.Arah O A, Westert G P. Correlates of health and healthcare performance: applying the Canadian health indicators framework at the provincial‐territorial level. BMC Health Serv Res. 20051 576. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Schoen C, Davis K, How S K.et al U.S. health system performance: a national scorecard, Health Aff (Millwood) 200625w457–w475. [DOI] [PubMed] [Google Scholar]
- 25.Knowles J C, Leighton C, Stinson W.Measuring results of health sector reform for system performance: a handbook of indicators. Special Initiatives Report Number 1. Bethesda, MD: Partnerships for Health Reform, Abt Associates, 1997
- 26.Abdad P, Rouse A, Mohamemed A.et al Performance league tables: the NHS deserves better. BMJ 200232495–98. [DOI] [PMC free article] [PubMed] [Google Scholar]