Health research: measuring the social, health and economic benefits

Despite intense interest in defining the social, health and economic impacts of health research investments globally^1–5 and in Canada^6–10 as proof of value-for-money, no validated method for measuring return on investments yet exists. Until now, issues of complexity combined with major gaps in methodology have limited the ability to link health research products to outcomes at a relevant level (e.g., to be useful to stakeholders: individual funders, decision-makers, institutions, researchers or clinicians). In this article, we discuss current approaches to measuring returns on investment, analyze key issues and gaps that need to be bridged to improve returns on investment, and present a new method that may help overcome them.

Two approaches used to define return on investment for health research

As recently reviewed by Shiel and DeRuggerio¹¹ for the Canadian Academy of Health Sciences, 2 main approaches have been used over the past 20 years to measure return on investments. The first uses what can be called “top-down” econometric calculation. The Lasker Foundation commissioned health economists to use this approach to estimate return on investments from health research in the United States.^12,13 They monetized improvements in life expectancy and quality of life between 1970 and 1990 by ascribing to them a value of roughly $1.5 trillion/year. They then attributed roughly one-third of those gains (about $500 billion) to health research,^14,15 and concluded that these economic returns far exceed the costs of the health research that contributed to them (> 20-fold). Adaptations of this method were also used more recently to estimate the returns on investment of research in Australia in 2003¹⁶ and 2008.¹⁷ In those assessments, every dollar invested in research yielded net benefits to society (benefits over costs) of $1.17 and perhaps as much as $1.40. Both reviews suggested that more than double any investment in research is being returned as a benefit to Australian society.

The second approach, a “bottom-up” approach, to measuring return on investments has been evolving for over 2 decades through the efforts of Buxton and Hanney and coworkers.^18–22 Their “payback model,” recently adopted and modified by the Canadian Institutes of Health Research in Canada,²³ has involved logic-model tracking of new knowledge in phases from knowledge production at the researcher level adding to a knowledge pool, and then from there to secondary outputs and adoption to final outcomes. They classified impacts into 5 categories: knowledge production; research targeting, capacity and absorption; informing policies and product development; health and health sector benefits; and broader economic benefits. These categories move “downstream” from research through translation to society and can capture outputs of interest for different audiences. For example, demonstrating research-related changes in the health and health sector benefits category would be of interest to clinicians. As one example of its use, Wooding and colleagues²² evaluated the payback profiles of various programs for the Arthritis Research Campaign in the United Kingdom using quantitative scales. They concluded that there is variability in outputs of research projects, short-term projects with flexible funding (e.g., investigator-initiated research projects) provide excellent value for money invested, individuals translate research findings, and, although their outputs are also highly variable (some produce very little), funding some research institutes can potentially produce the greatest impact in all 5 domains per pound invested.²²

A hybrid version of the payback approach that uses some econometric assumptions combined with more detailed data validation and linkage to research outputs has been used to estimate the recent economic benefits and health gains from investments in cardiovascular and mental research in the United Kingdom between 1975 and 1992.²⁴ They have calculated ranges that estimate that for cardiovascular disease, 10%–25% of health care benefit is attributable to United Kingdom research, with an average time lag of 17 years. By monetizing these health gains and combining them with estimated gross domestic product gains of 30% that include “spillover” effects to other sectors, they cautiously estimate a total annual internal rate of return of £0.39 for every £1 of public or charitable money invested.

Issues to resolve in determining return on investments from health research

For both the econometric and payback methods of estimating return on investment in health research suffer from some common issues that need to be resolved in order for their results to be validated and for their results to be useful to each stakeholder audience. Table 1 shows 4 evaluation examples with some of their strengths and weaknesses.

Table 1

The first major hurdle for both methods is how to deal with the so-called “attribution issue” that plagues all research.²⁵ That is, the inability to determine the exact contributions of health research (versus other factors) in achieving its end goals: positive changes in health, health care or adding substantially to social and economic prosperity. The econometric method attempts to factor the other contributors out through assumptions. The payback approach can potentially isolate health research impacts more effectively depending on what metrics it uses and how specific its output assessments are, but it has also relied on some educated assumptions about contributions of research to the changes observed.

The second issue, which is related to the first, is known as the “counterfactual.” What would have happened if the research had not been conducted and how can this be determined? What controls for the lack of health research exist? Have the best controls been used for each research finding? Measuring impacts assumes that research has helped create them without necessarily linking them. That is clearly an assumption of the econometric method in that it had to ascribe an estimated and clearly debatable percentage of impact to research (33%). The payback method could suffer from the same issues, depending on what denominators and controls are selected, but, as it is measuring from the bottom up, it is likely better able to track impacts as they occur.

The third major issue is the well-known time lag for knowledge translation and how to account for it.²⁴ From basic discovery to an effective therapy can take anywhere from 2 years to 30 years, adding to the difficulty of assessing returns at any single point in time. Even short-term longitudinal studies can thus also miss important impacts. The econometric method actually pools impacts across a large enough timeframe (e.g., over 25 years) that it presumably averages impact across enough time to capture a spectrum of individual research impacts at various points in their evolution. The payback method can use timescales, and it could use longitudinal, as opposed to cross-sectional, analysis so it could potentially determine time lags. The UK cardiovascular payback example,²⁴ however, used a retrospective review approach with some assumptions and estimated an average time lag of 17 years, but the authors were quick to qualify that as an estimate only.

The fourth issue is the need for clear definition of what to measure and how to really capture impacts in meaningful terms. Economic measures are obviously numerical and easier to defend as metrics, whereas social and health improvements are more qualitative. The latter problem has been overcome to some extent by the use of measures that attempt to quantify improvements objectively (e.g., quality-adjusted life years) and potentially monetize them. There are many other, equally important nonnumerical “indicators of impact” that have required definition and validation, along with the metrics of impact, using criteria of attractiveness and feasibility.²⁶

Unique strengths and weaknesses of both approaches can easily be summarized. Gross econometric methods are clearly useful for advocacy and decisions at policy levels, and they are certainly more cost-effective, as they use only high-level administrative data to draw conclusions. They are less helpful at improving individual program planning or delivery at either institutional or funder-specific levels as they pool research impacts rather than teasing them apart. The payback method, on the other hand, while having the advantage of being a bottom-up approach, uses information from the researcher level downstream, which makes it less prone to assumptions about linkages to outputs and confounding effects, can suffer from being too detailed, slow and costly. It can certainly trace impacts across specific domains and time scales right down to an individual project level, but this is also much more labour intensive.

What would define the best method of measuring return on investments in health research?

To be practical and useful, the best method for measuring return on investment in health research must address all the issues noted above. It must also be useful to a full range of funders and research types, compatible with existing systems, suitable for international comparisons, and able to identify the full spectrum of potential impacts.²⁶ The best method must also be feasible, not too labour intensive, and economically viable. It should be as accurate and responsive as possible within a reasonable evaluation budget that should represent a small percentage of the money invested in the research being assessed.

A new approach for addressing the gaps and needs

In early 2007, as part of its mission to provide unbiased analysis of important topics of international interest,²⁷ the Canadian Academy of Health Sciences established a blue-ribbon panel of experts to create a simple but robust composite method that might address all of the challenges noted above.²⁸ After considering the world literature and the options, including a review by RAND Europe²⁹ of the strengths and weaknesses of many existing evaluation frameworks, they recommend a method that builds on the advantages of the “payback model”^18–22 but adapts it to target-specific impacts in multiple domains at multiple levels (Figure 1). It is more of a bottom-up approach than the econometric approach, and, as in the payback model, it combines an impact category approach with logic model. This adds the advantage of specific program and project comparisons to impact framework models, such as the balanced scorecard, which can evaluate progress against targets for an organization but cannot explain how impacts occurred or how to improve them (Table 2).

Figure 1: Simplified schematic (top) of the health research impact framework developed by the Canadian Academy of Health Sciences,²⁶ showing specific areas and target audiences where health research impacts can be found. The payback model,¹⁸ which identifies the stages of knowledge production from secondary outputs through adoption to final outcomes, is shown on the bottom. Reproduced with permission from the Canadian Academy of Health Sciences.²⁶

Table 2

The new model is specifically designed to track impacts in 5 categories: advancing knowledge, capacity building, informing decision-making, health benefits, and broad economic and social benefits (that can potentially include cultural outcomes). It was designed as a “roadmap of impacts version of the payback model” to help identify where proximal impacts can occur — the health industry, other industries, government, research decision-makers, or the public or public groups — and follows them distally through stages of adoption to final outcomes in health, well-being and social or economic prosperity. For the sponsors, the framework was also designed to trace research impacts in any or all impact categories for any of the 4 “pillars of health research” in Canada (basic biomedical, applied clinical, health services and systems, and population health) or at any level (from individual projects to the whole country).

How to use the framework

Several steps are required to use the new framework appropriately. The first is defining evaluation questions. The framework is then used to determine which categories of impact are expected and where they might occur. One example of how to use the framework and where to look for the various categories of impact is provided for clinical research (Figure 2). The next key step is the strategic selection of sets of impact indicators, chosen from a starting menu of 66 preferred indicators designed to answer a number of potential impact questions. These indicators (some of which are quantitative metrics) have all been validated and meet internationally accepted standards of attractiveness and feasibility.²⁶ An example showing the evaluation of a research program aimed at developing high-quality research capacity for a provincial funder is shown in Table 3. This example uses the model to evaluate the effectiveness of research capacity development by 1 open and 2 different strategic research programs. These indicators could be used to compare capacity development per dollar but are not prescribed to do so.

Figure 2: Example of where the various impacts of clinical research can occur using the Canadian Academy of Health Sciences model.²⁶ Many potential indicators of impact can be selected for each of the 5 categories depending on the evaluation questions. Reproduced with permission from the Canadian Academy of Health Sciences.²⁶

Table 3

Some aspirational indicators were also suggested by the expert panel (e.g., areas where data are not yet being collected or where other barriers to measurement currently exist). The panel also provided a template for designing new indicators as a step toward building and expanding a library of validated health research impact indicators, with appropriate references. This will provide maximum flexibility and allow users to devise customized indicators for their specific needs while still meeting the required standards. This recommended-indicator resource²⁶ should also help users progressively resolve complexities in assessing health research impacts.

What the framework can and cannot do

The framework can be used to do some things that were previously not possible. The 66 indicators in 5 domains of evaluation allows stakeholders to answer a wide variety of questions that address the value of research and research funding. Evaluation questions about the impact of research can range in scope from individual projects to the national level. The evaluation framework can also be adapted to various evaluation perspectives, from the individual to the international perspective. The framework can also be used as a roadmap to help identify specific impacts and, perhaps, even examine the impact of a specific piece of research, research program, body of work or funding entity as its relevant products are translated downstream.

The multidimensional nature of the framework should enhance the comprehensiveness of any evaluation undertaken as well as enable its users to capture the various strengthens and weaknesses of the entity under evaluation.

Finally, the framework can describe feedback loops that may stimulate new research. It can assist in understanding and distinguishing the separate research effects from the influences of confounding factors, such as changes in the environment (e.g., a recession).

In general, tailored evaluations such as the modified payback framework enhance one's ability to discriminate between 2 or more programs. However, the enhanced flexibility comes at a cost. The multitude of indicators may make it more difficult to synthesize results from different studies if very different indicators are used. Multiple indicators may also provide confusing results if some show improvement and others worsen. Resolving discordant results within the same evaluation or between evaluations can certainly be an important limitation.

Any evaluation will also limited by the domains and indicators included in it. The 66 validated indicators currently listed do not cover the full spectrum of possibilities. New indicators will need to be developed and validated to further enhance the overall usefulness of the method and its comprehensiveness. Clearly, the payback framework cannot track impacts in domains outside the boundaries that it has established, nor can it identify impacts in audiences that have not been identified.

Challenges to be overcome

The designers of the framework recognize several important implementation challenges. First, funders will have to become familiar with the framework and indicators and how to apply them. This will take time and require dedicated people in various organizations to make an effort to implement the framework for their own purposes. Collaboration among organizations to achieve all of the practical prerequisites for any meaningful evaluation of impact on a national scale (including reliable and standardized data sources and data collection methods) will also pose some major challenges. Expanding the library of validated indicators to address all of the potential evaluation questions will be a challenge.

An even bigger challenge for all users of the new framework will be defining impact questions more explicitly, prioritizing them, and choosing sets of indicators and metrics that address the questions in a practical way. This sounds much easier than it is in practice, as experts note that there are almost an infinite number of potential health research impact questions (e.g., “Does our health services research program on decreasing wait times have an important impact on policy makers and health care?”). Currently, 2 indicators from the decision-making category relate to policy-maker impacts and a selection from the determinants of health subcategory on accessibility could be used to track these impacts.

Given the scarcity of funding for health research in Canada, strategic selection of indicators that are sensitive and specific enough to address evaluation questions, while not being too expensive or too time-consuming collect, will be a major challenge. Only the most important return-on-investment questions can realistically be considered.

Conclusions and the way forward

Both econometric and payback methods have demonstrated that economic returns from health research have been sizeable and are ongoing. Although these assessments support the notion that some health research investment is worthwhile, the latest comprehensive assessment of this topic in the United Kingdom has just concluded that a standardized and mapped way of classifying research funding with a deeper understanding of international flows of knowledge and influence is still required.²⁴

The Canadian Academy of Health Sciences has developed a new method to meet these needs.²⁶ It proposes a new framework and a library of preferred indicators (the individual strength and specificity of which are critical) that are to be selected in customized sets to address specific impact evaluation questions from a variety of health research funders and stakeholders.

This approach will not be fast, easy or inexpensive to apply, and substantial collaboration is recommended in implementing it. However, the approach should help optimize limited resources for the evaluation of impact, eliminate duplication and double-counting of impacts, and help offset the costs of evaluating impacts this well. The use of this new model and its approach to understand and progressively enhance the impact of health research investments will indirectly contribute to improved health, economic and social outcomes for all Canadians.

Key points.

• Health research is expensive, and its explicit social, health and economic impacts are hard to define.

• There are many challenges and assumptions in defining specific returns on investment in health research.

• There is no common approach to tracking health research impacts.

• The payback model can be populated with validated indicators to track overall outcomes or outcomes in a specific health research area.

• Collaboration among funders will ensure cost-effective implementation of the new framework to quantify return on investment.

• The framework can be fine-tuned as necessary to improve indicator sets, overcome gaps and progressively define returns on investment in health research.