Table 2.

Examples of primary studies assessing the impact of cancer research

Author(s)	Title	Year	Location	Description	Unit of analysis	Main reason for assessment	Categories assessed	Methods used	Framework	Main findings	Strengths	Limitations
Brown ML, Nayfield SG, Shibley LM	Adjuvant therapy for stage III colon cancer: economic returns to research and cost-effectiveness of treatment	1994	USA	Evaluation of cost-effectiveness of adjuvant treatment for colorectal cancer and assessment of the return on investment to conduct clinical trial research	Cancer clinical trial	Accountability	Based on principles of economic evaluation. Economy, health	Cost effectiveness analysis (CEA) followed by evaluation of the social return on research investment. Human capital approach used to value return on investment	Nil	Cost effectiveness of treatment estimated at $2094 US dollars per year of life saved. The net present value of the return on the $10.84 million investment in the trial estimated at $1.66 billion dollars	The cost of conducting research incorporated into analysis of the value of the research investment. Time-period for assessment of the costs and benefits extended to 2020 to capture downstream effects	Assumptions made about adoption of trial results into practice rather than an assessment of actual practice change
Ugolini D, Bogliolo A, Parodi S, Casilli C, Santi L	Assessing research productivity in an oncology research institute: the role of the documentation centre	1997	Italy	Estimation of the academic impact of departments within a cancer research institution	Cancer research centre	Allocation	Academic impact	Bibliometric assessment using an journal impact factor based metric	Nil	Most publications from the research centre that were identified and analysed scored highly (8–10/10) on the normalised journal impact factor score	An early (1997) attempt to evaluate the academic impact of a cancer institute	Metric, quantitative based approach only looking at a narrow interpretation of impact. Using the impact factor of a journal to assess the quality of individual research articles and for allocation of resources
Ugolini D, Casilli C, Mela GS	Assessing oncological productivity: Is one method sufficient?	2002	Italy	Estimation of the contribution to academic cancer publications from different European countries using bibliometric methods	European countries	Analysis	Academic impact	Bibliometric method using the number of occurrences of an oncological publication in a journal (by author country of origin) compared to country population/GDP and the "mean impact factor" of the occurrences of the publication by author	Nil	The UK made the highest contribution to European cancer publication output (21.12%), whereas Sweden performed best in the metric of number of publication occurrences versus country population and the Netherlands was ranked first for the mean impact factor of the occurrences	Comparison of two methods to understand how to assess the impact of cancer researchers' work	Quantitative metric used that does not assess the quality of the individual articles or the contribution of the authors to the work. The authors acknowledge the limitations of using bibliometrics alone
Coyle D, Grunfeld E, Wells G	The assessment of the economic return from controlled clinical trials	2003	Canada	Evaluation of the potential payback of conducting a clinical trial to test the optimal follow up for patients with colorectal cancer. Two trial designs were compared: an equivalence trial and an effectiveness trial (looking for a survival benefit from more intensive follow up)	Cancer clinical trial	Allocation	Economic and health impact (potential)	Assessment of “time to payback” for two alternative hypothetical cancer trials. This is the number of years until the returns from conducting a clinical trial outweigh the costs	Nil	An effectiveness clinical trial would be worthwhile to look for a 5% improvement in survival from more intensive follow up (if a 5% improvement is considered likely). An equivalence trial would not be a worthwhile investment	Ex-ante evaluation shows how impact assessment can prevent investment in cancer trials that are unlikely to be worthwhile	An evaluation of potential rather than realised impact. Any error in the assumptions made and results obtained in these types of analysis means that some potentially impactful trials will not be performed
Lewison G, Sullivan R	The impact of cancer research: how publications influence UK cancer clinical guidelines	2008	UK	Evaluation of the cancer research that are cited in UK clinical guidelines	Any type of cancer research as cited in guideline	Analysis	Policy/guidelines	Identification of 43 UK guidelines from three guideline series (NICE, SIGN and Clinical Evidence) and bibliometric software used to analyse the guideline citations. Comparison of the citations versus those in the world oncology literature over 3-year period	Nil	UK papers were cited more frequently in cancer clinical guidelines than expected from their presence in the world oncology literature. The publications were generally more clinical than basic	Outlines and executes a method for evaluating the impact of cancer research on guidelines	The authors highlight that small clinical trials with negative outcomes are unlikely to be cited in guidelines, and discuss the limitations of using clinical guideline impact as a surrogate for practice change
Lewison G, Tootell S, Roe P, Sullivan R	How do the media report cancer research? A study of the UK's BBC website	2008	UK	Evaluation of the impact of cancer research on the BBC news website (accessed by 13.2 million people annually in the UK (2008 figures))	Any cancer research as cited in the media	Analysis	Media impact (dissemination)	Search of BBC archive health Section 1998–2006. Percentage of BBC stories focusing on different cancer sites were compared with the UK's burden of disease (WHO 2002). Research level (basic versus clinical) of cited papers was determined. The potential citation rate (PCI), actual citation rate (ACI) and funding of any cited paper compared to the global oncology research papers	Nil	Research on breast, cervical and skin cancer are over-reported in the media compared to their burden of disease whereas lung cancer is under-reported. New and improved drugs are the research topic most cited. UK research was over-cited in the UK media compared to its place in world oncology research	Novel attempt to investigate the impact of research in the media. Methodology clearly explained	Only one media website archive used for analysis
Saad et al.	The geography of clinical cancer research: analysis of abstracts presented at the American Society of Clinical Oncology Annual Meetings	2009	Brazil	Evaluation of cancer research disseminated at an international conference	Individual research projects presented in abstract form at a conference	Analysis	Academic impact and dissemination	Bibliometric analysis of a sample of abstracts		Over 50% of abstracts were from the USA. Clinical trials were more likely than "other" types of research to be presented in poster or oral form (vs publication only)	Large, international cancer conference and several years of submissions chosen for the analysis	Analysed only a sample (10%) of all abstracts, no comparison between the two time periods sampled performed and no information on the abstracts that were submitted but rejected
Lewison G, Markusova V	The evaluation of Russian cancer research	2010	UK	Evaluation of the extent of cancer research in Russia, if research meets the needs of the country, and how the impact of research compares on a world scale	Cancer research performed by one country	Analysis	Academic/scholarly impact	Quantification of number of cancer publications compared to the national wealth and disease burden of cancer in Russia compared to other countries. Citation scores for Russian cancer publications compared with the citations to cancer papers worldwide in the same years. Analysis of collaboration on Russian cancer research papers	Nil	Russia publishes one sixth as many cancer papers as its wealth and disease burden would suggest. Clinical cancer research papers receive more citation than basic ones (the reverse of what is seen elsewhere). Russian cancer research scores well for the percentage of reviews and collaboration in cancer research is dominated by former socialist states. Russian cancer research is incorporated into UK clinical guidelines, but rarely into UK media	Multiple indicators used to map some important impacts of cancer research	Mainly quantitative indicators used. No assessment of broader aspects of societal impact. Only looked at the impact on UK clinical guidelines and media rather that in Russia (likely to be much higher)
Lakdawalla DN, Sun EC, Anupam BJ, Reyes CM, Goldman DP, Philipson TJ	An economic evaluation of the war on cancer	2010	USA	Evaluation of the returns on investment from the USA government into cancer research from 1971	National cancer research	Accountability	Economic impact from a societal perspective	Quantification of gains in cancer survival using willingness to pay estimates derived from the literature and a comparison with the cost in cancer spending	Nil	Cancer survival increased by 3.9 years from 1988–2000 which equates to 23 million additional life years. The authors estimate that an average life year is worth 82,000 dollars and therefore the value of this survival is monetised at 1.9 trillion dollars in social value. Uses an 18-year time lag from research investment to survival outcome attributable to that research	Attempts to monetise a complex concept of willingness to pay for cancer survival. Detailed description of analysis	Complex methods, willingness to pay values from previous literature. The actual investment costs in different cancer types and different research types (basic versus trials) not outlined
Montague S, Valentim R	Evaluation of RT&D: from "prescriptions for justifying" to "user-orientated guidance for learning"	2010	Canada	Evaluation of the impact of a cancer RCT (MA17), which reported that an aromatase inhibitor called letrozole reduced rates of disease recurrence when given to patients after radical treatment for breast cancer	One cancer trial	Analysis	Multiple types of impact (based on CAHS framework)	Canadian Academy of Health Sciences Framework (CAHS) used to guide impact evaluation. Indicators of impact from CAHS framework combined with Bennett’s hierarchy	CAHS model, Bennett hierarchy	Impacts in all categories of the CAHS can be identified for this trial. The authors produce an impact timeline to show the pathway to impact in chronological order	Use of a conceptual framework means that the impact of this cancer trial is communicated in a transparent and organised fashion. The use of the hierarchy of events helps to demonstrate the processes through which impact has occurred	The theory of action hierarchy makes assumptions that prior events influence events higher up the chain
Sullivan R, Lewison G, Purushotham AD	An analysis of research activity in major UK cancer centres	2011	UK	Evaluation of the impact of research from UK major cancer centres on knowledge production (citation impact), clinical management, and the general-public (via media citation)	Research from cancer centres in the UK	Analysis	Academic/scholarly, clinical management, general public/media	Academic papers linked with cancer centres in the UK (via location or author) identified via database review. The journals of the papers were categorised by research level (basic versus clinical). Potential citation impact (PCI) and actual citation impact (ACI) calculated for the included papers and any international collaboration on the papers documented. Calculation of the number of papers from UK cancer centres that are reported in UK guidelines and in the media	Nil	UK cancer centres are heterogeneous in terms of their overall research output and the type of research performed. Overall, there was more focus on basic/fundamental research being published by UK cancer centres compared to applied research. There is heterogeneity in the proportion of papers from UK cancer centres that are cited in guidelines and the media that does not correlate with the size of the centre or the conventional citation impact of the papers	The authors combine publication citation impact with other impacts (on guidelines and media) to give a broader overview of cancer centre research impact	Mainly quantitative measures of impact. The authors mention the potential future use of the Research Impact Framework but they do not use this or any other framework in their current analysis
Donovan C, Butler L, Butt AJ, Jones TH, Hanney SR	Evaluation of the impact of National Breast Cancer Foundation-funded research	2014	UK	Evaluation of the academic and wider impacts resulting from funding by a specific cancer charity in Australia (National Breast Cancer Foundation NBCF)	Programme of cancer research projects	Accountability	Multiple (based on Payback framework)	Desk analysis of data held by the charity, survey of chief investigators, 16 case studies, bibliometrics and international benchmarking using bibliometrics. The case studies used document and archival analysis, citation analysis, searching for citations in guidelines and interviews with principal investigators of projects	Payback	153 responses to a survey sent to 242 NBCF-funded researchers showed that the research they performed had impacts on drug development, higher degree attainment, capacity building for future research, policy, and health gain. Findings showed differences in impact between basic and applied cancer research, for example, basic research was more likely to lead to product development whereas applied research was more likely to impact on policy, behaviour, and practice	Transparent methodology using a recognised framework and a mixture of quantitative and qualitative methods to gain in depth insight into impact of the research	Impact from the perspective of the researchers, no other stakeholders approached to comment on perceived impact of the funded research
Glover M, Buxton M, Guthrie S, Hanney S, Pollitt A, Grant J	Estimating the returns to UK publicly funded cancer-related research in terms of the net value of improved health outcomes	2014	UK	Evaluation of the economic value of cancer research in the UK	National cancer research	Accountability	Economic	Economic analysis (bottom up approach used). Five main steps: Estimated spending on research between 1970–2009 (£15 billion), estimated the net monetary benefit using a monetized value for a QALY, estimated the cost of delivering the benefit by identifying a list of the most important interventions that had contributed to this benefit (via discussions with experts), estimated the proportion of NMB attributable to UK research and the time lapse between funding and health gain (using bibliometric analysis of clinical guidelines) and finally, the internal rate of return from cancer research on health benefits	Nil	Time lag between research spending and impact on health gain estimated as 15 years. Overall return on public spending on cancer research estimated as 10.1%	Considered the time lag between research spending and impact on health gain. Sensitivity analysis performed where possible and acknowledged areas of uncertainty. Acknowledged the purely quantitative nature of this assessment and accompanying case studies were performed (Guthrie et al.)	Multiple assumptions made to perform this economic analysis (recognised and outlined by the authors). The authors outline the difficulty in differentiating the impact of smoking cessation in their calculation of the impact of cancer research overall
Guthrie S, Pollitt A, Hanney S, Grant J	Investigating time lags and attribution in the translation of cancer research. A case study approach	2015	UK	Evaluation of the attribution of health gains due to investment in cancer research and the time lags between research investment and health gain being recognised using a case study approach	Six examples of cancer research topics which included clinical trials	Accountability	Health gain	Case study approach using mainly desk/documentary analysis	Nil	Individual narratives for each case study	The use of case studies enables the reader to understand the process through which impact has occurred and the time lines involved	Requires in depth documentary analysis to contextualise and explain case study specific impacts