Measuring the efficiency of health systems

In June 2000 the World Health Organization provided a long awaited answer to the question beloved of politicians and journalists: “How does the health system in country X compare with that in country Y?” The results, published in the World Health Report 2000,¹ delighted some governments, such as that of France, which came first, but infuriated others, such as Brazil, at 125. The rankings are based on measures of achievement of five health system goals. The achievement of health is seen as a core objective of a health system, so goals are a high level of health and a fair distribution. A health system should also be responsive to popular expectations. This includes respect for individuals (autonomy and confidentiality) and client orientation (prompt service and quality of facilities). As with health, the resulting goals relate to the absolute level of responsiveness and its distribution. The fifth goal is fair financing, with expenditure reflecting ability to pay rather than risk of illness.

In this week's BMJ some of the authors of the WHO report describe the methods they used to assess one of these goals, the attainment of health (p 307).² They relate expenditure on health, adjusted for local prices, to attainment of health. After adjusting for the level of education in the population, itself an important determinant of health, they rank the health systems of the world according to their efficiency in turning expenditure into health.

Inevitably, a few problems exist in an undertaking of this magnitude. The first is how one defines the health system. As set out in the World Health Report this encompasses “all the activities whose primary purpose is to promote, restore, or maintain health.” This is welcome as it emphasises the importance of intersectoral action in promoting health, but unfortunately it also provides a problem since a figure for “all the activities” is nowhere to be found in any national health accounts. Instead, the report argues, as the health care system accounts for most of what is incorporated in the broader health system, “little is lost in concentrating on a narrower definition that fits existing data.”¹ Consequently, we must compare inputs to the health care system with outcomes of the wider health system.

A subsidiary question is whether health outcomes can even be attributed to the activities included within the broader definition of a health system. As the report notes, there is growing evidence of the health gains that can be achieved both from health care interventions and from policies in other sectors, such as vehicle safety. But there are many other determinants of health. For example, in industrialised countries the health of populations reflects long established dietary patterns that owe more to climate, and thus the nature of agricultural produce, than to any contemporary policy. Thus, it is unsurprising that many of the countries performing best are characterised by “Mediterranean” diets. The growing evidence of how events throughout life influence health creates a further difficulty.³ Health system inputs that affect infant and child health may have consequences many years later.

A second problem is the availability of data.⁴ Many governments have only the vaguest idea of how many people live in their territory. Some have not undertaken censuses for many years,⁵ in some cases because large areas are outside their effective control. In many parts of the world population registration systems are fragmentary, and even in some industrialised countries significant gaps exist in coverage of some groups—for example, native Americans.⁶ Equally, there are substantial problems with comparability of data on the other measures used, health expenditure and education. The authors recognise this problem and have constructed an elaborate set of procedures to address it, so generating figures for disability adjusted life expectancy⁷—itself a highly controversial measure.⁸ Fundamentally, however, one cannot create data where none exist, so each step requires a series of often heroic assumptions and extrapolations.⁹ Unfortunately, though the World Health Report and its associated working papers note that many figures are estimates, it is not easy to discover just how extensive this process has been. Using complex models to generate estimates of uncertainty fails to tackle the underlying problem.

Other criticisms of this exercise have been aired elsewhere and include concern about the ideological values underpinning it and the intrinsic limitations of performance ranking.¹⁰ But some of these difficulties are insuperable, and a fairer question to ask is whether the report has achieved anything.

Despite its many limitations, arguably it has. Firstly, the WHO has stated clearly that governments have a responsibility for their health systems. It has invoked the concept of stewardship,¹¹ which implies a much more active involvement in promoting health than most governments have previously assumed.¹² Secondly, it has provided a useful conceptual framework that begins to tease out the goals of health systems. Thirdly, it has emphasised the need for a much better understanding of the undoubted impact that health systems have on health.¹³ It has not, however, provided a valid answer the question of whether one system is better than another.

Papers p 307