Strengthening accountability of the global health metrics enterprise

Introduction

The development and use of indicators to reveal population health conditions and hold policy makers and the organisations they lead to account—what might be termed the global health metrics enterprise—has surged over the past two decades. The enterprise's proponents point to its emancipatory effects, arguing that global health metrics uncover health problems and bring greater objectivity and accountability to policy making.1, 2, 3 Critics challenge the enterprise's merits and are worried, among other concerns, that the enterprise transfers power from institutions in low-income countries to ones based in high-income countries, hampers the development of national health information systems, and privileges certain forms of knowledge over others.4, 5, 6

We contend that there are strong reasons to accept global health metrics as a public good. Credible epidemiological data are vital for addressing the coronavirus disease 2019 (COVID-19) pandemic, for instance. However, many valid concerns of critics are being overlooked because metrics proponents—including the Bill & Melinda Gates Foundation, the Institute for Health Metrics and Evaluation (IHME), and The Lancet—wield far more power than critics. In this Viewpoint, we discuss these criticisms and offer ideas for strengthening accountability of the enterprise. We argue that metrics leadership is more than a technical matter: it is a political and normative concern that bears on the legitimacy of the distribution of power in global health governance. Therefore, the global health metrics enterprise deserves the same kind of critical scrutiny that its backers seek of the policy makers whose decisions they hope to shape.

Rationales for the global health metrics enterprise

Currently, the global health metrics enterprise involves many actors that produce multiple sets of indicators. Among these actors are UNICEF, supporting multi-indicator clustor surveys; WHO, responsible for world health statistics reports; and the United States Agency for International Development, supporting demographic and health surveys. However, it is the IHME-led Global Burden of Disease (GBD) studies that have attracted the greatest attention in recent years. In this Viewpoint we particularly highlight these studies although many of the arguments pertaining to the merits and critiques of the enterprise apply well beyond the IHME and GBD studies.

IHME leaders describe the GBD study as “a systematic scientific effort to quantify the comparative magnitude of health loss from diseases, injuries, and risks by age, sex, and population over time”.¹ The GBD study of 2017—the latest version published in The Lancet in November 2018⁷—is vast in scope, covering 359 diseases and injuries and 195 countries and territories.⁸ IHME founders are former WHO employees. In the 2000s, tensions emerged between IHME and WHO over leadership in the global health metrics field.⁹ In 2018, however, the two organisations signed a memorandum of understanding to collaborate to produce a single set of GBD studies.¹⁰

IHME leaders and researchers specify several aims for GBD studies.1, 11, 12, 13 By providing policymakers with up-to-date information on disease trends and drivers, they seek to help to improve population health through the facilitation of evidence-based decision making. They also hope to spur policy maker accountability for the achievement of national and global goals, including improved health system performance and the Sustainable Development Goals (SDGs). IHME leaders and researchers seek to bring to light problems that might otherwise be neglected. They aim to counter politicisation of data by advocacy groups that inflate numbers to secure greater resources for the conditions that concern them. Furthermore, they seek to advance the science of population health estimation.

GBD studies have shaped policy making in England, China, Rwanda, and Botswana, among other countries. These studies have also shaped the priorities of global organisations, including WHO, the World Bank, and the Bill & Melinda Gates Foundation.¹ In addition, GBD research has led to more than 16 000 peer-reviewed publications and reports.¹ Reflecting on this influence, IHME leaders argue that “the GBD Study…is now arguably the de-facto source for global health accounting”.¹

GBD studies have provided valuable data on many trends in global health, such as the rising burden of non-communicable diseases in sub-Saharan Africa¹⁴ and the stagnation in development assistance for HIV/AIDS.¹⁵ Like many researchers, we have found GBD studies to be useful, enabling us to make better judgments on trends in disease burden and on the level of attention particular global health issues receive.16, 17

Critical perspectives

As the global health metrics enterprise has expanded, several researchers have raised concerns about its effects on the production of knowledge and the distribution of power in global and national health governance. Several critiques pertain to the accuracy, frequency, and use of GBD data.4, 5, 6, 9, 18, 19 One concern is the lack of transparency on GBD imputation methods. Critics argue that the estimates are generated by opaque techniques and are potentially flawed and misleading. Over the long term, they want the world's population to be counted so that “complex estimation techniques are not needed”.¹⁸ Another concern is that GBD data are updated so frequently that they constitute an overwhelming flood of information for the field of global health. Additionally, observers point out that GBD figures often diverge considerably from country statistics and little effort is made to adjudicate between different sets of estimates. Beyond this, critics raise concerns that GBD estimates are used inappropriately to evaluate progress on goals, such as the SDGs.

Another critique is that the enterprise hampers the development of national health information systems. One way this happens is that resource-poor ministries in low-income countries observe global organisations using massive computer power to produce estimates in the absence of data, and thereby conclude that it is not worth investing in their own health information systems.⁵ A consequence of the neglect of national health information system development is an emphasis on the production of information useful for monitoring progress on objectives that global rather than local organisations deem valuable. Mahajan⁴ identifies an additional concern: as IHME and other institutions vacuum data¹⁹ gathered by organisations in low-income countries to produce and disseminate estimates globally, national governments must rely on these global institutions rather than information they control locally to know their very own populations.

Critics also worry that metrics oversimplify complex realities and displace other forms of knowledge, particularly those which cannot be quantified.20, 21, 22, 23 For instance, Merry and colleagues²⁴ note that certain concepts, such as a child's right to play, are difficult to translate into numbers and are therefore overlooked by governments in favour of issues more readily quantified, such as breastfeeding rates. Storeng and Béhague⁶ argue that maternal health advocates have neglected equity and rights concerns as they “play the numbers game” and advance a technocratic, medicalised agenda. Tichenor and Sridhar⁹ express concern that efforts through metrics to make “local contexts readable from a satellite's view” generalise experiences of suffering that cannot be universalised. Farmer²³ makes a similar point, noting that, “the experience of suffering…is not effectively conveyed by statistics or graphs”. Like many medical anthropologists, Adams²⁵ calls for more ethnographic research that gets at everyday realities and that “unseat tendencies toward the tyranny of numbers” in the field of global health. Critics also express concern about global health metrics crowding out other forms of quantitative research, including longitudinal studies that go beyond a few individual indicators and more holistically convey the realities of population health in individual settings.

A related critique is that metrics present a scientific veneer to a contingent undertaking, and thus acquire an authoritativeness they do not deserve. Creators and promoters of metrics understand them to be—or portray them as—objective markers of features of the world that exist beyond human interpretation. Critics question this assumption.⁴ They ask how the indicators and categories they purportedly illuminate came to exist in the first place, viewing these as products of social processes heavily reliant on interpretation. One of the most dramatic effects of metrics, they argue, is to create or alter the very phenomena their developers claim to measure.26, 27, 28 Hacking,²⁹ for instance, speaks of the “looping effects” of classifications, including categories such as autism and schizophrenia. Individuals who know that they have been designated as autistic or schizophrenic might alter their behaviour, and thereby the very meaning of the classifications (sometimes with adverse, at other times with emancipatory effects).

Arguably, the critics' deepest concern is that the enterprise transfers power from actors in low-income countries and from international organisations whose legitimacy derives from the state system to private and public sector institutions in high-income countries. Institutions in high-income countries gather data on indicators they have developed themselves, and disseminate these data as a means of pressing governments of low-income countries and UN agencies to achieve objectives that these institutions have decided are worthy, rather than the ones low-income country governments prioritise. The power of organisations in high-income countries might come less from direct control—such as the provision of financial and other resources—than through the cultivation of self-regulation: countries pursue these goals to avoid international shame.19, 21, 26

An uneven playing field

If the playing field were level, there would be less reason to worry about the potentially adverse effects of the enterprise. Proponents could develop and disseminate numbers. Critics could raise concerns. Corrections could be made. And ultimately, the rise or fall of global health metrics could be based, much like new products introduced into the marketplace, on their utility.

However, evidence indicates that the playing field is not level and that the rapid spread of metrics might have to do with more than their scientific merit. One reason is the alignment of organisations, the resources at their disposal, and their incentive structures. Global health metrics proponents are among the richest and most powerful organisations in the global health field. The IHME has been able to expand rapidly since its establishment in 2007 due to the availability of resources to produce metrics. Recently, the IHME has moved into areas even beyond health, producing population estimates, traditionally the work of the United Nations Population Division.³⁰ The primary financier is the Bill & Melinda Gates Foundation, which, over time, has provided the IHME with several hundred million dollars of support, and which uses GBD data for its own priority-setting exercises.³¹ The Lancet, the world's most widely read journal publishing on global health, has forged a close relationship with IHME, making a commitment to the regular publication of GBD data. Although driven primarily by the advancement of science, The Lancet, operating in a competitive medical journal market and owned by a large for-profit publishing company, might have incentives that extend beyond knowledge production. These incentives include the maintenance of a high journal impact factor, an objective facilitated by high citation rates for GBD studies. Meanwhile, critics operate with little funding and make their arguments in specialised outlets—especially anthropology, sociology, and development journals—with limited circulation and reaching largely only academic audiences.

Aside from the alignment of organisations, another reason that the playing field is not balanced is that global health metrics hold a particular allure that other forms of information, not easily crystallised into numbers, lack. As IHME founders Chris Murray and Allan Lopez say, “there is an inherent fascination with quantification of levels and patterns of disease”.¹ Hacking,³² who writes on a sudden enthusiasm that emerged in the first half of the 19th century for producing numbers on disease, portrays the allure more brusquely: “Disease, madness, and the state of the threatening underworld, les misèrables, created a morbid and fearful fascination for numbers upon which the bureaucracies fed.”

The spread of global health metrics taps into deeper debates on why scientific research programmes advance.29, 33, 34, 35 One question concerns stability: why do claims about the world come to be widely accepted? Is this a function of compelling evidence? Or alternatively, is this due to factors external to that evidence, such as the strength of the scientific networks that back the claims? Another question concerns the basic structure of the world. Does the world have an inherent configuration that scientists and social scientists discover? Or are the classifications we use our own convenient representations of the world? Put differently, as philosophers have asked, in our inquiries do we uncover the world carved at its joints, or are we the joints' creators?29, 36

If one holds a particular stance on stability and structure, one is likely to believe that global health metrics spread because they uncover compelling, objective features of the world that exist beyond human interpretation. Historically, we have been unable to detect the comparative magnitude of health loss human beings suffer from various diseases, injuries, and risk factors. GBD provides us with the most objective evidence available on precisely this subject. As evidence accumulates and techniques are refined, we will move toward a master map—akin to an exhaustive, geographic plot of the entirety of the earth's terrain—that accurately conveys the empirical reality of human health loss across conditions, time, and space.

If one adopts an alternative stance on stability and structure, one might understand the advance of the metrics enterprise differently. The categories that IHME and other organisations use to describe disease burden are products of human interpretation. The world of human suffering might be—and indeed is—conceptualised and carved up in many ways. The reason the GBD map—or any other map that global health metrics producers might offer of the world—receives greater and greater acceptance has not so much to do with its scientific superiority (however that term is understood) but with the resources, social connections, and power of its primary backers.

Dozens of works in the philosophy of science and the sociology of knowledge have grappled with these questions concerning why scientific research programmes advance, and they will not be resolved in this Viewpoint. The point is that these are ongoing debates that readily apply to the global health metrics enterprise and should be aired on a level playing field.

Promoting accountability

There are strong reasons to embrace proponent arguments that global health metrics are a public good. The issue is not whether metrics have merit—all but their most vehement critics accept that they do—but whether on an uneven playing field, the concerns of critics pertaining to potentially detrimental governance and knowledge effects receive enough hearing. The evidence indicates that these concerns too frequently remain on the periphery of global health debates.

A pressing concern is to ensure that organisations working on global health metrics are better held to account. Noting that private sector institutions are now involved heavily in global governance, Woods³⁷ states that, “accountability in global governance is about information, monitoring, and the enforcement of limits and rules in the use of power”. She and other scholars identify several conditions crucial to ensuring the accountability of global governance institutions: transparency (open reporting on processes, decisions, and outcomes), controllability and responsiveness (checks on the behaviour of the institution by those affected by its actions and by those entrusting it with power), and liability (facing consequences for producing negative effects).37, 38, 39, 40 A strong case can be made that several major organisations involved in global health metrics fall short on all of these dimensions.

In this Viewpoint, we do not offer a comprehensive set of actions for ensuring global health metrics accountability. We do, however, suggest the need for civil society organisations, metrics experts, national governments, and international organisations to pay considerably greater attention to this issue in order to develop more effective accountability mechanisms than currently exist. A start would be for actors concerned with global health metrics to attend to three strategic considerations that follow from the concerns of critics. First is to ensure that the global health metrics and the national health information systems agendas advance synergistically. Both of these agendas contribute to the public good. Although not necessarily incompatible, at present the global health metrics agenda, undoubtedly unintentionally on the part of its proponents, might be having adverse effects on the national health information systems agenda. The leaders of the enterprise, national policy makers, and researchers need to find ways to ensure the two agendas strengthen one another. One step would be for organisations involved in global health metrics to design and disseminate their work cautiously and with potential adverse effects in mind—a metrics Hippocratic Oath—rather than to claim either that these effects do not exist or are not their responsibility if they do.

Second is to foster pluralism in the global health metrics enterprise. A positive development is the growing cross-national network of researchers working on GBD: collaborators from 146 countries provided input in the latest iteration.¹ Still, the enterprise might be developing in ways that concentrate power rather than foster pluralism.⁹ At the core of the enterprise are several institutions, the first three of which are based in high-income countries, and the fourth of which derives considerable power from the support of high-income countries: the Institute for Health Metrics and Evaluation (the primary producer of metrics), the Bill & Melinda Gates Foundation (the primary financier), The Lancet (the primary disseminator), and WHO (formerly in competition with but now in alliance with IHME in the production of metrics). As The Lancet editor Richard Horton has argued, “Without a diverse and vigorously contested field, the discipline of global metrics for global health will likely drift and lose focus on its true purpose—serving communities in countries”.⁴¹ These comments were made in 2013, defending the rise of IHME and GBD studies against critics. His point still holds, only now for a different set of actors.

Third is to ensure diversity in the kind of knowledge that is valued and used in global health. The inherent allure of metrics—their ease of use and their alleged capacity to render legible complex social worlds—create strong potential for their dominance and for the sidelining of knowledge produced by studies employing ethnographic, comparative historical, and other qualitative methods that take time to absorb and that reveals the nuances and complexities of matters concerning human wellbeing. Just as plurality is to be valued within the metrics enterprise itself, it should also be valued with respect to the kind of knowledge that counts in global health.20, 25 Relatedly, one must keep in mind the contingent nature of the knowledge produced by metrics,²⁶ questioning about what kind of interpretive work goes into the creation of categories and numbers, and whether, once created, these acquire a taken-for-granted legitimacy that might not be warranted.

Speaking about metrics in general, Power²⁷ issues a challenge to social scientists to “open up the black box of performance measurement systems, to de-naturalise them and to recover the social and political work that has gone into their construction as instruments of control.” Undoubtedly, social progress requires strong measurement systems, a need especially pressing as the world faces a pandemic. However, it is worth asking what reforms are needed to ensure the global health metrics enterprise works more effectively to advance human wellbeing, and what benefits might accrue if funders and publishers offered equally robust support to national health information systems and to ethnographic research.