Project Retrosight

Abstract

This project explores the impacts arising from cardiovascular and stroke research funded 15–20 years ago and attempts to draw out aspects of the research, researcher or environment that are associated with high or low impact. The project is a case study-based review of 29 cardiovascular and stroke research grants, funded in Australia, Canada and UK between 1989 and 1993. The case studies focused on the individual grants but considered the development of the investigators and ideas involved in the research projects from initiation to the present day. Grants were selected through a stratified random selection approach that aimed to include both high- and low-impact grants. The key messages are as follows: 1) The cases reveal that a large and diverse range of impacts arose from the 29 grants studied. 2) There are variations between the impacts derived from basic biomedical and clinical research. 3) There is no correlation between knowledge production and wider impacts 4) The majority of economic impacts identified come from a minority of projects. 5) We identified factors that appear to be associated with high and low impact. This article presents the key observations of the study and an overview of the methods involved. It has been written for funders of biomedical and health research and health services, health researchers, and policy makers in those fields. It will also be of interest to those involved in research and impact evaluation.

All funders have more opportunities for investment in research than they can support, many of which relate to areas of science of high potential interest and/or impact. How best to choose between these is a key issue for funders, the scientific community, governments and society. The “science of science” is a growing field that aims to understand what works in research funding (Marburger, 2005; Grant and Wooding, 2010). This requires a better understanding of research performance, and more importantly the drivers of improved performance. At a conceptual level we need to understand what factors lead to research impact. For example, what kinds of science, what kinds of scientists, and what settings are most conducive to ensuring the scientific success of research and its translation into societal benefits?

Project Retrosight

Project Retrosight was a multinational study that investigated the translation of, and payback from, basic biomedical and clinical cardiovascular and stroke research projects. The main project aims were to:

• examine the variety of payback produced by basic biomedical and clinical cardiovascular and stroke research;

• identify factors associated with high (and low) levels of payback, in particular factors relating to the characteristics of the research, how it was supported or the context in which it was carried out.

The name Project Retrosight is derived from two landmark studies in science policy. The first—Project Hindsight (1967)—was a study sponsored by US Department of Defense that examined the incremental advances of various technologies (Sherwin and Isenson, 1967). The second was Julius Comroe's book, Retrospectroscope: Insights into Medical Discovery (Comroe, 1977). Comroe examined new life-saving advances in medicine and how they had come about. At the same time, in a more or less direct response to Project Hindsight, he worked with Robert Dripps to trace the research antecedents of clinical advances in cardiovascular medicine. This study was described in an article in Science (Comroe and Dripps, 1977). The idea of Project Retrosight was to develop these ideas by tracing prospectively, with the benefit of hindsight, the payback and translation of funded research projects.

Project Retrosight builds on successful methodologies used to evaluate diabetes and arthritis research funding (Hanney et al., 2006; Wooding et al., 2005).

Our approach involved identifying the principal investigators (PIs) of all grants awarded in the early 1990s for basic biomedical and clinical cardiovascular and stroke research by specific funders in Australia, Canada and the United Kingdom. These PIs were sent simple questionnaires that were used to estimate the impact of the work funded by the grant. A random sample of grants was then selected policy makers, changes in policy or practice, etc. The case studies were then coded qualitatively and analysed to identify factors that appeared to be associated with high and low impact in both academic and wider categories. The associations that emerged were then tested and refined in discussion with researchers, funders and policy makers, and developed into a series of policy implications for research funders. There are, of course, a number of limitations to the analysis. The number of case studies used in Project Retrosight can be viewed as both a limitation and a strength. Whilst the number of case studies is greater than in most other studies of a similar nature, the sample size may still not be using a sampling framework stratified according to location of research (Canada, Australia and UK), type of research (basic biomedical or clinical), size of research grant (large or small) and perception of impact by the PI (high or low). Detailed case studies were then developed for 29 grants (in total we approached 38 cases: 6 PIs declined to participate, 3 case studies were not completed/not included for other reasons). We chose a case study approach since, in general, case studies provide a rich source of material when “how” or “why” questions are proposed (Yin, 2003). In the context of Project Retrosight, the case studies provide a detailed picture of what led to establishing the grant, how the research progressed and how it subsequently developed.

A number of approaches have been developed to describe and capture the impacts of research (Brutscher, Wooding, and Grant, 2008). We used the Payback Framework (Buxton and Hanney, 1996), which has two elements: five payback categories, which we collapsed into two impact groups for some of our analysis (summarised in Figure 1); and the payback model. The Payback Framework provides a common structure for examining why the PI applied for the research grant and what he or she hoped to achieve with the funding; the responses of the funding committees; the research process, including collaborations, use of shared resources, etc.; research outputs (e.g. publications); how those outputs influenced subsequent research topics and careers; how the research was subsequently translated into “secondary outputs”, through influencing clinical policies or product development; and how the research then translated into improvements in health and broader economic benefits.

Figure 1.

Payback Categories and Impact Groups

Examples of academic impacts included publication of papers, supervising a PhD, developing scientific methods subsequently used by other researchers, etc. Examples of wider impacts included citation in policy documents or guidelines, licensing intellectual property, briefing senior policy makers, changes in policy or practice, etc. The case studies were then coded qualitatively and analysed to identify factors that appeared to be associated with high and low impact in both academic and wider categories. The associations that emerged were then tested and refined in discussion with researchers, funders and policy makers, and developed into a series of policy implications for research funders.

There are, of course, a number of limitations to the analysis. The number of case studies used in Project Retrosight can be viewed as both a limitation and a strength. Whilst the number of case studies is greater than in most other studies of a similar nature (Wooding et al., 2005; Nason et al., 2008), the sample size may still not be large enough to rule out outcomes or differences that could have arisen by chance. Because of this, we have been deliberately cautious in interpreting our data and have tested the strength of any associations leading to policy observations. Other limitations include potential inconsistencies in case study reporting and possible confounders; for example, the definitions of basic biomedical and clinical research used, the scope of the case studies and the effects of negative findings. Equally there are significant strengths in the study method chosen, particularly in comparison to other sources of information on research funding policy. These strengths include the use of the Payback Framework to encourage consistency across cases and facilitate comparative analysis; quality assurance checks to ensure consistency across an international team; and consideration of both quantitative and qualitative case study material.

Key Findings and Policy Implications

Five Key Findings

1. The cases reveal that a large and diverse range of impacts arose from the 29 grants studied. As illustrated in Figure 1, there is a considerable range of research paybacks associated with the grants studied, and many of these would not have been identified without the structured, case study approach used in this study. This resonates with the diversity of payback identified in an earlier study on arthritis research (Wooding et al., 2004).

2. There are variations between the impacts derived from basic biomedical and clinical research. In the cases studied, basic biomedical research has a greater academic impact and clinical research a greater wider impact over the timescales investigated. All the grants studied had academic impact, but the average rating was higher in basic biomedical research than in clinical research. For the combined wider impact categories all clinical studies had some impact, compared to only six out of 15 basic biomedical case studies. This finding should be treated with caution as it may be confounded by longer time lags for basic biomedical research.

3. There is no correlation between knowledge production and wider impacts. There is no correlation between the payback category, “knowledge production”, and the three wider categories, “informing policy and product development”, “health and health sector benefits” and “broader economic benefits”. From a policy perspective this would suggest that the level of knowledge production is not a predictor of wider impacts.

4. The majority of economic impacts identified come from a minority of projects. Only four of the 29 case studies reported substantial broader economic benefits and 19 grants had no impact in this payback category. It is important that these distributional effects are understood in any assessment of research impact. Although the majority of economic impacts come from a small proportion of projects, we previously found that the value of the impact achieved from a programme of research overall can significantly outweigh the costs of doing the research (HERG, 2008).

5. We can identify factors that appear to be associated with high and low impact. We have identified a number of factors in cardiovascular and stroke research that are associated with higher and lower academic and wider impacts. These are captured in Table 2, each with an associated policy implication for research funders and policy makers to consider.

Table 2.

Factors Associated with High- and Low-Impact Research

Factor	Policy Implication
Basic biomedical research with a clear clinical motivation is associated with high academic and wider impacts.	When seeking to achieve high academic and wider impacts, encourage and support clinically motivated basic biomedical research.
Co-location of basic biomedical research in a clinical setting is associated with high wider impact.	When seeking to achieve high wider impacts from basic biomedical research, encourage and support the co-location of basic biomedical researchers with clinicians in a clinical setting (e.g. a teaching hospital or health organisation).
Strategic thinking by clinical researchers is associated with high wider impact.	When seeking to achieve high wider impacts from clinical research, focus clinical research funding on PIs or teams who think strategically about translation into clinical practice.
Research collaboration is associated with high academic and wider impact.	When seeking to achieve high academic and wider impacts, encourage and support research collaboration for both basic biomedical and clinical research.
International collaboration is associated with high academic impact.	When seeking to achieve high academic impact, encourage and support international collaboration for both basic biomedical and clinical research.
Engagement with practitioners and patients is associated with high academic and wider impacts.	When seeking to achieve high academic and wider impacts, encourage and support clinical researchers who have a record of engaging with practitioners and patients.
Basic biomedical research collaboration with industry is associated with high academic and wider impacts.	When seeking to achieve high academic and wider impacts from basic biomedical research, encourage and support collaboration with industry.
Negative or null findings are associated with low academic and wider impacts.	Research funders should acknowledge the importance and potential significance of negative or null findings when assessing the impact of research.
Initial rejection of a subsequently accepted basic biomedical research grant may be associated with low academic and wider impacts.	Further research is needed to confirm whether initial rejection of a research proposal is associated with low impact. Until this finding can be confirmed or refuted, funders may want to carefully consider such proposals.

Table 1.

Summary of the Impacts Arising from Case Study Grants

Australia	Canada	UK
All Australian projects produced peer-reviewed publications, e.g. a project on immunoelectron microscopy of amine and peptide synapses on sympathetic preganglionic neurons resulted in 18 articles that have received a total of 780 citations.	All Canadian projects produced peer-reviewed publications, e.g. a project on the determinants of increased growth of vascular smooth muscle in spontaneously hypertensive rats produced a series of journal articles. Sixteen articles produced 849 citations and included a paper in the highly prestigious Journal of Clinical Investigation.	All UK projects produced peer-reviewed publications, e.g. a project on the role of coagulation and fibrinolysis in the pathogenesis of recurrent stroke led to a series of articles, seven of which have been cited 393 times in total.
All Australian projects led to research capacity building and/or targeting, e.g. a project on high density lipoprotein (HDL) led to collaborations for the PI and advanced the career of the postdoc; it also resulted in new research techniques, further research funding for the group and better targeting of other groups through increased understanding of HDL.	All Canadian projects led to research capacity building and/or targeting, e.g. a project on the effects of simulated stroke on developing astrocytes led to two PhDs; techniques were taught.	All UK projects led to research capacity building and/or targeting, e.g. the project (above) led to two PhDs, an MD and development of a patient cohort and control group that formed the basis of a stream of work. It helped the PI establish his research group.
All Australian projects contributed to informing policy and/or product development, e.g. a project that created animal models for myocardial dysfunction contributed to the decision to create a transgenic facility at the research institute, and eventually a commercial facility.	Eleven of the 12 Canadian projects contributed to informing policy and/or product development, e.g. guidelines recommend a treatment pathway for antiphospholipid antibodies (APLA) based on the original warfarin-based project.	Four of the nine UK projects contributed to informing policy and/or product development, e.g. a project on stroke prevention in the elderly in primary care informed guidelines in a working group statement and protocols of local units in the health service.
All Australian projects contributed to health gains, e.g. a project studying the follow-up to heart attacks contributed to a major international project on health promotion, which in turn contributed to a decline in coronary heart disease in the Hunter region.	Seven of the 12 Canadian projects contributed to health gains, e.g. the treatment path for APLA patients is much improved, leading to some health gain.	Four of the nine UK projects contributed to health gains, e.g. a project analysing the automated defibrillators in Scotland's ambulances is widely cited in policies and made an important contribution to the increased survival rate following out-of-hospital cardiac arrest.
Five of the eight Australian projects contributed to economic benefits, e.g. the commercial transgenic facility developed as a result of the animal models for myocardial dysfunction is now a multimillion-dollar business that exports 80% of its services.	Two of the 12 Canadian projects contributed to economic benefits, e.g. a project used a radioimmunoassay the PI had created previously: later sold by a commercial company.	Three of the nine UK projects contributed to economic benefits, e.g. the increased life expectancy of patients with Marfan syndrome has mostly been among people of working age; therefore a number of people have been able to remain active in the workforce.

Just as science is the effort to discover and increase human understanding of how the world works and how we can influence it, science policy should be about understanding how the world of science works and how we can influence it to maximise benefits for society. Studies like Project Retrosight contribute to the growing field of the “science of science”, providing an evidence base to inform research funders in their decision making.