Tracking the impact of research on policy and practice: investigating the feasibility of using citations in clinical guidelines for research evaluation

Abstract

Objectives

To investigate the feasibility of using research papers cited in clinical guidelines as a way to track the impact of particular funding streams or sources.

Setting

In recent years, medical research funders have made efforts to enhance the understanding of the impact of their funded research and to provide evidence of the ‘value’ of investments in particular areas of research. One of the most challenging areas of research evaluation is around impact on policy and practice. In the UK, the National Institute of Health and Clinical Excellence (NICE) provide clinical guidelines, which bring together current high-quality evidence on the diagnosis and treatment of clinical problems. Research referenced in these guidelines is an indication of its potential to have real impact on health policy and practice.

Design

This study is based on analysis of the authorship and funding attribution of research cited in two NICE clinical guidelines: dementia and chronic obstructive pulmonary disease.

Results

Analysis identified that around a third of papers cited in the two NICE guidelines had at least one author based in the UK. In both cases, about half of these UK attributed papers contained acknowledgements which allowed the source of funding for the research to be identified. The research cited in these guidelines was found to have been supported by a diverse set of funders from different sectors. The study also investigated the contribution of research groups based in universities, industry and the public sector.

Conclusions

The study found that there is great potential for guidelines to be used as sources of information on the quality of the research used in their development and that it is possible to track the source of the funding of the research. The challenge is in harnessing the relevant information to track this in an efficient way.

Article summary

Article focus

• Explore the feasibility of extracting the funding source of research papers cited in clinical guidelines.

• Identify who funded the research which supports the guideline development.

• Investigate the shared characteristics of the publications cited in the two guidelines chosen for the study (dementia and chronic obstructive pulmonary disease).

Key messages

• Looking at citation in clinical guidelines could potentially be used as one of the tools in the impact evaluators toolkit.

• For this methodology to be fully exploited, the accessibility of data and acknowledgement of funders in articles need to be improved.

• This study helps to investigate the connection between funding inputs with changes in medical practice and the pathways for these to arise.

Strengths and limitations of this study

• An assumption is made that the fact that a research article is cited in a clinical guideline is a proxy for potential impact; however, this impact is not proven in this study.

• The methodology relies on the correct and full attribution of funders in research articles; however, evidence shows that this information can be incomplete.

• NICE guidelines are very closely linked with the National Health Service in the UK, as such this limits actual results of the study, but the methodology could be applied to other similar documents.

Introduction

Medical research has advanced rapidly in recent times in all areas from basic sciences (ie, decoding the human genome) to the development of more precise diagnostic tools and novel treatments. At the same time, public interest in biomedical advances and the appetite for more effective treatments¹ are increasing in parallel. The demand for the results of biomedical research to lead to improvement in healthcare has never been higher.^2–4

Across the research world, and particularly in the biosciences, there has been a drive to better demonstrate and understand the impacts of research, essentially so that funds can be allocated to maximum effect.¹ There remains a concern that the research community as a whole could be better at translating the findings of medical research into tangible health and healthcare benefits.^5–7 Thus, the need to better understand research impact and in particular the pathways to that impact is a key priority for research funders.^8–11 However, determining the impact of research is challenging, particularly in basic and fundamental research where the time lag between original research and subsequent impacts on health can be long and the attribution difficult to track. In addition, perhaps one of the most challenging areas has been trying to understand the nature of the pathways to and subsequent impact of research on policy and practice.

National and international clinical guidelines are intended to bring together the best and most current evidence about the prevention, diagnosis, prognosis and therapy of clinical problems. Clinical guidelines are a form of systematic review and, in the UK, focus on the defined medical needs of the National Health Service (NHS). It should be noted that clinical guidelines are not standards of care but are recommendations to the non-specialist or general practitioner. In the UK, clinical guidelines are provided by the National Institute for Health and Clinical Excellence (NICE),¹² and since 2005, these have had legal standing in the NHS in England and Wales. As such, the results of this particular study are potentially limited to the NHS; however, the methodology could possibly be applied to guidelines governed by other bodies. The guidelines exist to help standardise and improve patient care and can help to introduce cost-efficiencies to the delivery of healthcare. The guidelines are evidence based and their formulation brings pieces of important and influential research together. For a funder, if research it has supported is referenced as part of the evidence supporting a national and/or international clinical guideline, then it is an indication that this research is likely to be influencing policy and practice. Hence, clinical guidelines are potentially an attractive resource to support impact tracking and assessment.^{13 14}

For those engaged in evaluation, historically it has been difficult to extract information from a guideline in a way that helps support analysis of the references and funding sources: simply put, UK clinical guidelines are not designed to support the requirements of funders trying to track the impact of their support. However, work is underway at the US National Center for Biotechnology Information (part of the National Library of Medicine, National Institutes of Health) to digitise the content of major international clinical guidelines to encourage wider access to their content and enable greater ability to mine their content and allow automated links to individual cited research papers via databases such as PubMed.

In 2009, the UK Medical Research Council, the Wellcome Trust and the National Institute for Health Research, who among them commit nearly £2bn annually to support biomedical and applied health research, commissioned a detailed analysis of the research cited on a small number of UK clinical guidelines to explore the potential of the information in broader research impact tracking. The objectives of this research were threefold. First, this study explored the feasibility of extracting the funding source of the research papers cited on a guideline. Second, it identified who funded the research cited in the selected clinical guidelines. Third, it explored the extent to which there are shared characteristics of the publications cited in these guidelines.

The then-current NICE guidelines for the management and treatment of two disease areas were selected: dementia (2006) and chronic obstructive pulmonary disease (COPD) (2004).^{15 16} These guidelines were of interest for the purposes of this analysis since (1) they had been available unchanged for several years and (2) there was a likelihood that all three project sponsor funders would have funded some of the underlying research evidenced in the guidelines. The two guidelines were also in quite different clinical areas, so the authors wanted to see if there were differences in the process and/or adoption of research into practice. For each guideline, all cited research was examined to pick out its characteristics (eg, age, bibliometric indicators) and identify any funding attributions.

Methods

Data extraction

The first step was to extract a list of publications from each of the guidelines and export them into a Microsoft Excel spreadsheet. This was performed automatically using bespoke RAND Europe computer scripts, based on the PERL scripting language. Here, we briefly describe the methodology since a full description is available elsewhere.¹⁷ A total of 744 references were extracted from the dementia guideline and 446 from the COPD guideline.

Data cleaning

The extracted bibliographic references were cleaned and structured to permit analyses of funding source and paper performance indicators. Any references identified as non-academic or peer-reviewed publications (eg, references to a website, grey literature) and all publications before 1980 were removed since these could not be investigated using the Web of Science.

After extraction and initial cleaning, a total of 616 references were found for the dementia guideline (79.4% of the original 776 references) and 412 references for the COPD guideline (83.9% of the original 491 references).

Data processing

For the funding analyses, the extracted publications were searched for in Web of Science¹⁸ to find the institution and country affiliations of the authors listed. One aim was to identify the publications with at least one UK author on the assumption that this would facilitate further funding analysis. Another aim was to use all the extracted publications for further bibliometric analysis (see Grant et al¹⁹ for a similar methodology).

From the dementia guideline, 494 of the 616 extracted publications (80.1%) were found in the Web of Science. While from the COPD guideline, 335 of 412 publications were available (81.3%). Any publications not found in the Web of Science were processed individually through a search methodology utilising the publication libraries of RAND Europe and Cambridge University. All search processes were duplicated by a second researcher to eliminate errors.

This methodology used a simple search of both the RAND library by article title, using Google Scholar and the Cambridge University online library and free access journals, through the Google search engine. If the author affiliation and country remained unidentified, then the RAND library was searched by journal, followed by browsing for the article using the reference data available. In addition, the title could be searched for by keyword within the journal. Finally, where possible, Cambridge University print holdings were searched to find any articles that were not accessible online.

Funder acknowledgement

Where publications had at least partial UK attribution, the funding source was searched for in the Research Outputs Database.^{20 21} This is a database housed at the Wellcome Trust recording the funding sources for UK and Irish publications in the biomedical sciences for the period 1988–2001. Funding acknowledgements were found for around one-third of publications using this database. For the remaining publications, which fell outside of the appropriate date range or were not found in the Research Outputs Database, the full text of the publication was found and funding acknowledgements recorded directly, where available.

Funding source references were then standardised and categorised by broad sector. UK funders acknowledged on cited papers were categorised into the following categories: industry, not-for-profit, hospital trust, government department, government agency (not controlled by ministries), local or regional authority, foundation, none given and unknown.

Bibliometric and paper characteristic analysis

Author affiliations and country locations were identified for 595 of the 616 (97%) dementia guideline publications and 402 of the 412 (98%) COPD guideline publications. These affiliated papers form the basis of the following descriptive and bibliometric analysis.

The citation impact of the publications referenced in both clinical guidelines was analysed by sector using the concept of citation profiling. This is based on a normalising technique called ReBased Impact (RBI), which takes account of the field in which a paper appears and the date since its publication to effectively provide a proxy measure for the ‘quality’ of each paper. The world average RBI is 1; the most highly cited articles have an RBI >8.²²

Whole counts were used throughout this analysis; if more than one funding source is cited in a publication, this was recorded as one publication for each funding source.

Results

Attribution by funding organisation

In a large proportion of cited papers, no funding acknowledgement was listed.

Nearly half (104 of the 228; 46%) of publications in the dementia guideline with a UK author did not acknowledge any funder. Funding information was available for 117 papers, with the full text of seven papers inaccessible and, hence, missing the funding information. For the 148 publications in the COPD guideline with at least one UK author, 60 included no funding acknowledgement (41%). Funding information was available for 81 publications. The full text of seven publications was not accessible and therefore no funding data were available.

Examination of the funding acknowledgements for the Medical Research Council, the Department of Health (England), the NHS and the Wellcome Trust revealed that these funders were overtly linked to only a small proportion of papers cited in the guidelines (see figure 1—numbers of publications by funding source).

Figure 1.

Numbers of publications by funding source.

Attributions by funding organisation over time

To determine whether the practice of acknowledgement of funding has improved over time, funding acknowledgement was analysed by year of publication. Although it appears that more recent publications have more complete funding acknowledgements than older ones, over the whole period, and for both guidelines, there was no clear statistical relationship between the age of publication and the presence or absence of a funding acknowledgement (see figure 2—funding acknowledgement by year).

Figure 2.

Funding acknowledgement by year.

The clinical guidelines, on the whole, cited recent research; the majority of research papers cited in these two UK clinical guidelines were published after 2000, that is, within 5 years of the release of these guidelines. The average duration between the publication date of papers cited and the publication date of the citing guideline was 5 years for dementia and 3 years for COPD.

Attribution by funding sector

Industry was not as prominent a funding source for publications cited in the dementia guideline, where acknowledgements were distributed across a range of funding sources across sectors. In the COPD guideline, industry was the most frequently linked funder after ‘none given’.

Attribution by country

Of the 616 publications extracted from the dementia guideline, 228 (37.2%) had at least one UK-based author, while from the COPD guideline, 148 publications (35.9%) had at least one UK-based author. Researchers based in the UK and USA combined were linked to the majority of papers cited in both guidelines.

Despite dominance of the UK and USA-based researchers in the cited papers, many other countries contributed to these publications. Papers cited in the dementia and COPD guidelines were linked to authors from 37 and 36 countries, respectively (see figure 3—contribution of the most active countries).

Figure 3.

Contribution of the most active countries.

Attribution by research sector

The three research funders use different means of disbursing their money. Funding includes grants to universities and hospitals, alongside direct support for intramural research. Publication analysis by associated institutions revealed that researchers with university addresses, followed by those with hospital addresses, were linked to the bulk of papers cited on both guidelines. More than 80% of publications cited in the two guidelines involved authors based at universities. The scientific contribution from other types of publicly funded institutions, as well as from non-profit institutions, was low.

Nearly 20% of the publications cited in the COPD guideline involved authors from industry—a slightly higher proportion than in the Dementia guideline.

Citation quality

Clinical guideline drafting committees are obliged to base their recommendations on the ‘best’ research available. The citation impact of the publications referenced in both clinical guidelines was analysed by sector using the concept of citation profiling.

Overall, papers cited across both guidelines had high RBIs. Papers linked to universities, companies (industry) and publicly funded research institutions were particularly highly cited. At the time of the analysis, for the COPD guideline, cited papers linked to publicly funded research institutions had RBI>8; for the dementia guideline, papers linked to universities, companies and publicly funded research institutions all had particularly impressive RBIs (see figure 4—citation score by institutional sector).

Figure 4.

Citation score by institutional sector.

Discussion

Funding attribution

Perhaps the greatest challenge in research impact assessment is dealing with attribution. Attribution of research outcomes and impacts to a specific funder is complex and, in many cases, improbable. This arises from most medical research receiving funding from multiple sources, involving a host of researchers (often across institutions) and being incremental such that considerable time elapses between original research and impact on health. The tide has turned on this issue and funding bodies are increasingly working together to identify where their funding has made a difference and contributed to an outcome or impact. Exclusively ‘claiming’ impact ignores the complexities and reality of scientific research and we are more interested in noting our contribution alongside others and learning from this.

Nevertheless, there is much we can do to help us better understand the connection between funding inputs and changes in medical practice. This research project was intended to flesh out some of the issues that we face in trying to link research funding to research output (ie, research papers) and specific outcomes (clinical guidelines).

As described, there was some variability in the quality of acknowledgement information provided on papers. In our analysis, we did not explore whether the differences in acknowledgement quality and completeness varied according to the nature of the paper (ie, underpinning vs more applied research). A study of 43 UK clinical guidelines (and associated Health Technology Assessments) related to cancer demonstrated that the number of funding sources acknowledged in papers varied with the ‘basicness’ of the publications: the more clinical papers have “fewer (funding) sources and the more basic papers have more.”¹³

An interesting direction for future research work on clinical guidelines would be to investigate whether the high proportion of cited publications with at least one UK author in the UK clinical guidelines could be the result of a specific funding strategy. That is, some research could be being funded to help create evidence for the development of clinical guidelines. This specific funding strategy might explain partially why UK-authored publications are over-cited in the UK guidelines, relative to the share of publications.

The advice given to funding recipients by UK funders regarding how they should be acknowledged in publications, and the extent to which these requirements have been enforced, has varied significantly over the last two decades. While most funders have included a requirement for acknowledgement in their terms and conditions of award, it has only been since 2008 that there has been published guidance about a standard format.²³ This may explain why industry is fairly highly cited across both guidelines as part of a researcher's more stringent contractual obligations. However, it is worth noting that the extent to which researchers are following the standard format is unknown. Furthermore, the reality remains that, given the incremental nature of much research, it is not easy to precisely attribute a publication to its source of funding.

One broader question is how to ensure that research information is accessible in order to avoid spending funds on research that either cannot be used or may be duplication.²⁴

Temporal issues

We did find a correlation between the publication of the clinical guidelines and the dates of the papers cited within it. These results corroborate earlier research.¹³ Other bibliometric studies on UK clinical guidelines also found that a significant share of publications cited are published within the 10 years prior to the release of these guidelines.^{19 25}

Therefore, this finding was not unexpected, since clinical guidelines should be based on new evidence, although on occasion the newest evidence may not be the best. However, it would be interesting to investigate whether the most recent publications cited are the outcome of research specifically funded to support the development of a clinical guideline, thus potentially explaining the peak in publications cited in the two guidelines a few years before their release.

National contributions

The UK's contribution is high in both guidelines, since its world share in medical research measured by its share in the total number of publications published in the field amounted to approximately 8.6% in 2006.²⁶ This high proportion of papers linked to UK-based authors echoes the finding of previous studies. Interestingly, our analysis—where around a third of papers are linked to UK-based authors—reveals a higher proportion than we have seen in other analyses.

Conclusions and recommendations

Having greater access to the work cited on clinical guidelines would present new opportunities for funders and the research community alike to better understand some of the mechanisms that take research from the laboratory to the bedside. While this would be but one tool in the research impact evaluator's toolkit, it would be one that could be relatively easy to harness if both access and acknowledgements were improved. As described, work is underway between the UK NICE and the National Center for Biotechnology Information, and in the future, it is envisioned that other guideline providers will make their content available in much more structured and accessible formats to permit analyses of this nature.

However, to bring clarity to the tracking of research inputs through to published output, the age-old problem of ensuring accurate and complete acknowledgement information on peer-reviewed published work needs to be addressed. This requires a change in the culture of how researchers use acknowledgements and greater liaison with publishers and information providers who could do much to enhance the quality and completeness of funding data as a paper is submitted for publication. Clarity and perhaps demarcation between the requirements of an ‘acknowledgement’ section on a paper and description of the ‘funding’ that has supported the work would seem an easy step to help remedy this.²⁷ This is particularly pertinent as information providers such as Thomson Reuters and Elsevier are now developing complex reporting and analytical tools that provide an ability to scrutinise the characteristics of published work—including who is described as funding the work—in new and exciting ways.

Furthermore, if the methodology of this paper is to be generalised both beyond formal guidelines and to other healthcare delivery and research output systems and metrics, then novel methods of identifying and tracking researchers and their outputs, via global identifier systems such as that proposed by the ORCID²⁸ (Open Researcher and Contributor ID) initiative, will be important.

Moves to address the definition of an ‘acknowledgement’ may also help to address the issue of ‘over-authorship’ and author inflation that has been seen in recent years. This is thought to be fuelled in part by the drive towards impact assessment through national research allocation formulae, such as the UK Research Excellence Framework.²⁹ Some contributors listed as ‘authors’ might be more appropriately ‘thanked’. Many journals now contain a section that asks for a description of ‘contributions’—these often do not contain all those listed in the author list so there may be a place for a more defined ‘acknowledgement as thanks’ section.

Funders should also ensure clarity around their requirements for a ‘funding acknowledgement’. Complexities arise, for example, when research takes place in buildings funded by a specific donor or when a specific piece of research uses a piece of equipment funded by a specific donor. And for how long after funding has been received by a researcher should they continue to provide a funding acknowledgement?

We found that there is great potential for national and international guidelines to be used as sources of information to help further our understanding on the impact of research on practice; the challenge is to be able to harness that information in an efficient way—so that we are able to use this information to feed into future research strategy and thereby make the research cycle more effective.