Short abstract
This study presents 100 ideas for metrics that can be used to assess and communicate the value of biomedical research to wider audiences including patients, providers, administrators, and legislators.
Keywords: Bibliometrics, Biomedical Research, Science of Science
Abstract
Biomedical research affects society in many ways. It has been shown to improve health, create jobs, add to our knowledge, and foster new collaborations. Despite the complexity of modern research, many of the metrics used to evaluate the impacts of research still focus on the traditional, often academic, part of the research pathway, covering areas such as the amount of grant funding received and the number of peer-reviewed publications. In response to increasing expectations of accountability and transparency, the Association of American Medical Colleges (AAMC), in collaboration with RAND Europe, undertook a project to help communicate the wider value of biomedical research. The initiative developed resources to support academic medical centers in evaluating the outcomes and impacts of their research using approaches relevant to various stakeholders, including patients, providers, administrators, and legislators. This study presents 100 ideas for metrics that can be used to assess and communicate the value of biomedical research. The list is not comprehensive, and the metrics are not fully developed, but they should serve to stimulate and broaden thinking about how academic medical centers can communicate the value of their research to a broad range of stakeholders.
Biomedical research affects society in many ways. It has been shown to improve health, create jobs, add to our knowledge, and foster new collaborations. Despite the complexity of modern research, many of the metrics used to evaluate the impacts of research still focus on the traditional, often academic, part of the research pathway, covering areas such as the amount of grant funding received and the number of peer-reviewed publications.
In response to increasing expectations of accountability and transparency, the Association of American Medical Colleges (AAMC), in collaboration with RAND Europe, undertook a project to help communicate the wider value of biomedical research. The initiative developed resources to support academic medical centers in evaluating the outcomes and impacts of their research using approaches relevant to various stakeholders, including patients, providers, administrators, and legislators.
This article presents 100 ideas for metrics that can be used assess and communicate the value of biomedical research. The list is not comprehensive, and the metrics are not fully developed, but they should serve to stimulate and broaden thinking about how academic medical centers can communicate the value of their research to a broad range of stakeholders. Some metrics, such publication numbers, are measurable in the short and medium terms. Others, such as community-level health and economic outcomes, are longer-term and appeal to audiences other than academic medical centers.
The metrics included in this study do not represent a systematic set. Rather, the metrics were identi- fied and selected in the course of several steps as part of a larger research project. First, RAND Europe reviewed research evaluation frameworks used internationally, along with the tools/methods these frameworks included.1 Second, workshops were held with medical college faculty and research leaders to identify appropriate approaches for American medical colleges alongside identifying the key stake- holder audiences for the evaluation. In a third phase, three key external stakeholder groups—community members, research administrators, and state legislators—confirmed the salience of the identified metrics.2 Metrics identified through these three steps, combined with the knowledge of the research team of existing metrics, formed the foundation for the current list of metrics.
Not all the metrics and areas will be of interest or applicable to all biomedical research institutions and organizations. This article, rather, provides a long list of the metrics from which a relevant subset could be selected, depending on the specific institutional context.
To help identify relevant metrics for a particular application, we have classified the metrics in three ways:
-
Stakeholders. Each metric is likely to be of interest to one or more stakeholder groups. Table 1 maps each metric against the relevant stakeholders:
In the table, “state and local legislators” refers to policymakers. In this study, the focus is on their interests as policymakers and representatives of the region and the local population. Any role that these individuals may have as funders in this context is not considered—those interests are all captured under the “funders and donors” category.
“Funders and donors” includes any external group or individual providing funding for research, researchers, or research equipment. This includes the National Institutes of Health (NIH) as well as smaller funders and donors, both public and private.
“External academic stakeholders” are any academics based at another institution. These academics can be potential collaborators or people that the institution might want to recruit.
“Board management” (of the research institution) refers to the person or group of people responsible for oversight of all the functions of the institution, not just research.
“Community” refers to the local population served by the institution (which will include but is not limited to patients). Of particular interest are those who might be engaged in the research process in some way (community partners, trial participants).
“Research management” (within the research institution) refers to the person or group of people responsible for oversight of the research function at the institution specifically.
“Patients” refers to people who are being treated or have recently been treated at the institution.
Table 1.
Metrics for Assessing and Communicating the Value of Biomedical Research, by Stakeholder Group
| State and local legislators | Funders and donors | External academic stakeholders | Board management | Community | Research management | Patients | |||
|---|---|---|---|---|---|---|---|---|---|
| Research impacts | 1 | Number of journal articles published | y | y | y | y | |||
| 2 | Number of citations | y | y | y | y | y | |||
| 3 | Number of research output downloads | y | y | y | y | ||||
| 4 | Mentions in social media | y | y | y | y | ||||
| 5 | Number and size of grant awards | y | y | y | y | y | |||
| 6 | Number and size of awards from major funders | y | y | y | y | y | |||
| 7 | Number of different research funders supporting research | y | y | y | y | y | |||
| 8 | Success rate of applications | y | y | y | |||||
| 9 | Catalogue of infrastructure | y | y | y | y | y | |||
| 10 | Use of infrastructure by other researchers | y | y | y | y | y | |||
| Measures of prestige | 11 | Number of editorships of high profile journals | y | y | y | y | |||
| 12 | Number of staff on relevant boards and committees | y | y | y | y | y | y | y | |
| 13 | Number of academy members | y | y | y | y | y | |||
| 14 | Number and type of prizes | y | y | y | y | y | |||
| 15 | Number of (international) speaker invitations/conference invitations | y | y | y | y | y | |||
| 16 | Number of media mentions | y | y | y | y | y | |||
| 17 | Number of applications per open post | y | y | y | |||||
| 18 | Percentage of out-of-state and international applications per research job/PhD post | y | y | y | |||||
| 19 | Track record of new hires | y | y | y | y | ||||
| 20 | Undergraduate applications | y | y | y | y | ||||
| 21 | Grade Point Average of incoming students | y | |||||||
| Teaching and career development impacts | 22 | Grade Point Average of graduates | y | y | |||||
| 23 | Longitudinal data on career progression of students | y | y | y | y | y | |||
| 24 | Number of PhD graduates | y | y | y | |||||
| 25 | Completion rate of PhD graduates | y | y | y | |||||
| 26 | Number of publications per PhD graduate | y | y | y | |||||
| 27 | 5/10/15-year career outcomes for PhD graduates | y | y | y | y | ||||
| 28 | K to R conversion rate | y | y | y | |||||
| 29 | Career outcomes for researchers | y | y | y | y | ||||
| 30 | Subject coverage of the professional development programme | y | y | y | |||||
| 31 | Uptake of the professional development programme | y | y | y | |||||
| 32 | Feedback on the professional development programme | y | y | y | |||||
| 33 | Improved educational attainment/reduced drop out rate | y | y | y | y | y | |||
| Research and institutional processes | 34 | Start-up time for research projects | y | y | y | y | y | y | |
| 35 | Start-up time for clinical trials | y | y | y | y | y | y | y | |
| 36 | Average time from funding to publication | y | y | y | y | ||||
| 37 | Proportion of funds spent on administration | y | y | y | y | ||||
| 38 | Support staff to researcher ratio | y | y | y | y | y | |||
| 39 | Prompt payment of community partners | y | y | y | y | y | y | ||
| 40 | How hiring decisions are made | y | y | y | |||||
| 41 | How decisions are made to apply for grants | y | y | y | y | y | y | ||
| 42 | How publications decisions are made | y | y | y | y | y | |||
| 43 | Proportion of publications that are open access | y | y | y | y | y | y | y | |
| 44 | Proportion of trials where protocol and findings are published | y | y | y | y | y | y | y | |
| 45 | Description of institution's policy on health equity in research | y | y | y | y | y | |||
| 46 | Proportion of projects which consider health equity in their design and conduct | y | y | y | y | y | y | ||
| Networks and dissemination | 47 | Number of collaborations on grant applications and projects | y | y | y | y | |||
| 48 | Levels of co-authorship | y | y | y | y | ||||
| 49 | Bilbiometric networks | y | y | y | |||||
| 50 | Total number of different collaborators across all projects | y | y | y | y | ||||
| 51 | Description of range of collaborations | y | y | y | y | ||||
| 52 | Number of research projects engaging community partners | y | y | y | y | y | y | y | |
| 53 | Number of research projects engaging community partners for the entire duration of the project | y | y | y | y | y | |||
| 54 | Number of articles co-authored with community partner | y | y | y | |||||
| 55 | Existence of specifically tailored material for different community groups | y | y | y | y | y | y | y | |
| 56 | Size of communications office | y | y | ||||||
| 57 | Number of staff engaged in outreach | y | y | y | y | ||||
| 58 | Number of people attending outreach events and their perceptions | y | y | y | y | y | |||
| 59 | Level of participation in clinical trials | y | y | y | y | y | y | y | |
| 60 | Number of projects with an industry partner | y | y | y | y | y | |||
| 61 | Industrial research funding for PhD/secondment positions in industry and PhD scholarships | y | y | y | y | y | |||
| 62 | Number of policy secondments | y | y | y | |||||
| Policy impacts | 63 | Number of invitations from policy makers | y | y | y | ||||
| 64 | Number of citations on clinical guidelines | y | y | y | y | y | |||
| 65 | Number of citations in policy documents | y | y | y | y | y | y | ||
| Health impacts | 66 | Improved health of patients | y | y | y | y | y | y | |
| 67 | Improved quality of care metrics | y | y | y | y | y | |||
| 68 | Number of lives touched | y | y | y | y | ||||
| 69 | Narrowing of health/health-care disparities | y | y | y | y | y | |||
| 70 | Improved awareness of preventative measures in the community | y | y | y | y | y | y | ||
| 71 | Number of treatment developed in house | y | y | y | y | y | y | y | |
| 72 | Number of new treatments available (adopted from elsewhere) | y | y | y | y | ||||
| 73 | Percentage, number, and range of types of clinicians on research projects | y | y | y | y | ||||
| 74 | Number of uses of research infrastructure in clinical practice | y | y | y | y | y | y | ||
| Economic impacts and commercialisation | 75 | Level of (local) spending | y | y | y | ||||
| 76 | Amount of direct employment | y | y | y | |||||
| 77 | Number and type of new offices (including subsidiaries) in the area | y | y | y | y | y | y | y | |
| 78 | Size of tech transfer office | y | y | y | y | ||||
| 79 | Existence of Intellectual Property policy | y | y | y | y | y | |||
| 80 | Number of patent applications | y | y | y | y | y | |||
| 81 | Number of patents awarded | y | y | y | y | y | |||
| 82 | Number of patent citations | y | y | y | y | y | |||
| 83 | Number of licensing agreements and licensing revenue | y | y | y | y | y | |||
| 84 | List/examples of know-how taken up by industry | y | y | y | y | y | |||
| 85 | Number of private sector innovations/products/devices brought to market | y | y | y | y | y | |||
| 86 | Number of spin outs | y | y | y | y | y | |||
| 87 | VC money invested in startups | y | y | y | y | y | |||
| 88 | Number and size of consultancy agreements | y | y | y | y | y | |||
| 89 | Contract funding from industry | y | y | y | y | ||||
| 90 | Number of and list of new treatments | y | y | y | y | ||||
| 91 | Fraction of indirect costs covered | y | y | y | |||||
| 92 | Cost/benefit calculations | y | y | y | y | y | y | ||
| Broader metrics | 93 | Perceptions of equity, quality, access | y | y | y | y | y | ||
| 94 | Perceptions of staff | y | y | y | y | ||||
| 95 | Perceptions of community partners | y | y | y | y | y | y | ||
| 96 | Perceptions of external experts | y | y | y | y | y | y | ||
| 97 | Perceptions of people participating in research | y | y | y | y | ||||
| 98 | Attitudes of participants towards science and towards research | y | y | y | y | y | y | ||
| 99 | Narratives of success | y | y | y | y | y | y | y | |
| 100 | Narratives of performance | y | y | y | y |
NOTE: Any metric could potentially be of interest to some members of any stakeholder group. This table gives an indication of the most likely interests and focus of each group.
Sources. Each metric draws on data collected through one or more sources. Table 2 maps the metrics against the relevant sources that could be used to collect them.
Table 2.
Metrics for Assessing and Communicating the Value of Biomedical Research, by Potential Data Sources
| Bibliometric analysis | Analysis of research administrative documents | Analysis of wider institutional records | Internal policy documents | Funder records | Records of external academic bodies | Educational outcomes and workforce tracking data | Quality improvement data | Public health data | Electronic health records data | StarMetrics data | Patent database | Review of key policy documents | Clippings service | Social media analysis | Feedback forms from events and courses | PI survey | Departmental secretary survey | Wider staff survey | Alumni survey | Survey or interviews with companies | Survey or interviews with research participants | Survey or interviews with community partners | PI interviews | Staff interviews | External researcher site visits or peer review | Representative case studies | High impact case studies | Labor market analysis | |||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Research impacts | 1 | Number of journal articles published | y | ||||||||||||||||||||||||||||
| 2 | Number of citations | y | |||||||||||||||||||||||||||||
| 3 | Number of research output downloads | y | |||||||||||||||||||||||||||||
| 4 | Mentions in social media | y | |||||||||||||||||||||||||||||
| 5 | Number and size of grant awards | y | y | ||||||||||||||||||||||||||||
| 6 | Number and size of awards from major funders | y | y | y | |||||||||||||||||||||||||||
| 7 | Number of different research funders supporting research | y | y | y | |||||||||||||||||||||||||||
| 8 | Success rate of applications | y | |||||||||||||||||||||||||||||
| 9 | Catalogue of infrastructure | y | y | ||||||||||||||||||||||||||||
| 10 | Use of infrastructure by other researchers | y | y | ||||||||||||||||||||||||||||
| Measures of prestige | 11 | Number of editorships of high profile journals | y | y | |||||||||||||||||||||||||||
| 12 | Number of staff on relevant boards and committees | y | y | ||||||||||||||||||||||||||||
| 13 | Number of academy members | y | |||||||||||||||||||||||||||||
| 14 | Number and type of prizes | y | y | ||||||||||||||||||||||||||||
| 15 | Number of (international) speaker invitations/conference invitations | y | |||||||||||||||||||||||||||||
| 16 | Number of media mentions | y | y | ||||||||||||||||||||||||||||
| 17 | Number of applications per open post | y | |||||||||||||||||||||||||||||
| 18 | Percentage of out-of-state and international applications per research job/PhD post | y | |||||||||||||||||||||||||||||
| 19 | Track record of new hires | y | y | y | |||||||||||||||||||||||||||
| 20 | Undergraduate applications | y | |||||||||||||||||||||||||||||
| 21 | Grade Point Average of incoming students | y | |||||||||||||||||||||||||||||
| Teaching and career development impacts | 22 | Grade Point Average of graduates | y | y | |||||||||||||||||||||||||||
| 23 | Longitudinal data on career progression of students | y | y | ||||||||||||||||||||||||||||
| 24 | Number of PhD graduates | y | y | ||||||||||||||||||||||||||||
| 25 | Completion rate of PhD graduates | y | y | ||||||||||||||||||||||||||||
| 26 | Number of publications per PhD graduate | y | y | ||||||||||||||||||||||||||||
| 27 | 5/10/15-year career outcomes for PhD graduates | y | y | ||||||||||||||||||||||||||||
| 28 | K to R conversion rate | y | y | ||||||||||||||||||||||||||||
| 29 | Career outcomes for researchers | y | y | ||||||||||||||||||||||||||||
| 30 | Subject coverage of the professional development programme | y | |||||||||||||||||||||||||||||
| 31 | Uptake of the professional development programme | y | |||||||||||||||||||||||||||||
| 32 | Feedback on the professional development programme | y | |||||||||||||||||||||||||||||
| 33 | Improved educational attainment/reduced drop out rate | y | |||||||||||||||||||||||||||||
| Research and institutional processes | 34 | Start-up time for research projects | y | ||||||||||||||||||||||||||||
| 35 | Start-up time for clinical trials | y | |||||||||||||||||||||||||||||
| 36 | Average time from funding to publication | y | |||||||||||||||||||||||||||||
| 37 | Proportion of funds spent on administration | y | |||||||||||||||||||||||||||||
| 38 | Support staff to researcher ratio | y | |||||||||||||||||||||||||||||
| 39 | Prompt payment of community partners | y | y | ||||||||||||||||||||||||||||
| 40 | How hiring decisions are made | y | y | ||||||||||||||||||||||||||||
| 41 | How decisions are made to apply for grants | y | y | ||||||||||||||||||||||||||||
| 42 | How publications decisions are made | y | y | ||||||||||||||||||||||||||||
| 43 | Proportion of publications that are open access | y | y | ||||||||||||||||||||||||||||
| 44 | Proportion of trials where protocol and findings are published | y | y | ||||||||||||||||||||||||||||
| 45 | Description of institution's policy on health equity in research | y | |||||||||||||||||||||||||||||
| 46 | Proportion of projects which consider health equity in their design and conduct | y | y | y | |||||||||||||||||||||||||||
| Networks and dissemination | 47 | Number of collaborations on grant applications and projects | y | y | y | ||||||||||||||||||||||||||
| 48 | Levels of co-authorship | y | |||||||||||||||||||||||||||||
| 49 | Bilbiometric networks | y | |||||||||||||||||||||||||||||
| 50 | Total number of different collaborators across all projects | y | y | y | |||||||||||||||||||||||||||
| 51 | Description of range of collaborations | y | y | ||||||||||||||||||||||||||||
| 52 | Number of research projects engaging community partners | y | y | ||||||||||||||||||||||||||||
| 53 | Number of research projects engaging community partners for the entire duration of the project | y | y | ||||||||||||||||||||||||||||
| 54 | Number of articles co-authored with community partner | y | |||||||||||||||||||||||||||||
| 55 | Existence of specifically tailored material for different community groups | y | y | y | |||||||||||||||||||||||||||
| 56 | Size of communications office | y | |||||||||||||||||||||||||||||
| 57 | Number of staff engaged in outreach | y | y | ||||||||||||||||||||||||||||
| 58 | Number of people attending outreach events and their perceptions | y | y | ||||||||||||||||||||||||||||
| 59 | Level of participation in clinical trials | y | |||||||||||||||||||||||||||||
| 60 | Number of projects with an industry partner | y | y | ||||||||||||||||||||||||||||
| 61 | Industrial research funding for PhD/secondment positions in industry and PhD scholarships | y | y | ||||||||||||||||||||||||||||
| 62 | Number of policy secondments | y | y | ||||||||||||||||||||||||||||
| Policy impacts | 63 | Number of invitations from policy makers | y | ||||||||||||||||||||||||||||
| 64 | Number of citations on clinical guidelines | y | y | ||||||||||||||||||||||||||||
| 65 | Number of citations in policy documents | y | y | ||||||||||||||||||||||||||||
| Health impacts | 66 | Improved health of patients | y | y | y | ||||||||||||||||||||||||||
| 67 | Improved quality of care metrics | y | y | y | |||||||||||||||||||||||||||
| 68 | Number of lives touched | y | y | ||||||||||||||||||||||||||||
| 69 | Narrowing of health/health-care disparities | y | y | ||||||||||||||||||||||||||||
| 70 | Improved awareness of preventative measures in the community | y | |||||||||||||||||||||||||||||
| 71 | Number of treatment developed in house | y | |||||||||||||||||||||||||||||
| 72 | Number of new treatments available (adopted from elsewhere) | y | y | ||||||||||||||||||||||||||||
| 73 | Percentage, number, and range of types of clinicians on research projects | y | y | y | |||||||||||||||||||||||||||
| 74 | Number of uses of research infrastructure in clinical practice | y | y | ||||||||||||||||||||||||||||
| Economic impacts and commercialisation | 75 | Level of (local) spending | y | y | |||||||||||||||||||||||||||
| 76 | Amount of direct employment | y | y | ||||||||||||||||||||||||||||
| 77 | Number and type of new offices (including subsidiaries) in the area | y | y | ||||||||||||||||||||||||||||
| 78 | Size of tech transfer office | y | |||||||||||||||||||||||||||||
| 79 | Existence of Intellectual Property policy | y | |||||||||||||||||||||||||||||
| 80 | Number of patent applications | y | y | y | y | ||||||||||||||||||||||||||
| 81 | Number of patents awarded | y | y | y | |||||||||||||||||||||||||||
| 82 | Number of patent citations | y | |||||||||||||||||||||||||||||
| 83 | Number of licensing agreements and licensing revenue | y | |||||||||||||||||||||||||||||
| 84 | List/examples of know-how taken up by industry | y | y | ||||||||||||||||||||||||||||
| 85 | Number of private sector innovations/products/devices brought to market | y | |||||||||||||||||||||||||||||
| 86 | Number of spin outs | y | y | ||||||||||||||||||||||||||||
| 87 | VC money invested in startups | y | |||||||||||||||||||||||||||||
| 88 | Number and size of consultancy agreements | y | |||||||||||||||||||||||||||||
| 89 | Contract funding from industry | y | |||||||||||||||||||||||||||||
| 90 | Number of and list of new treatments | y | y | ||||||||||||||||||||||||||||
| 91 | Fraction of indirect costs covered | y | |||||||||||||||||||||||||||||
| 92 | Cost/benefit calculations | y | y | y | y | y | y | ||||||||||||||||||||||||
| Broader metrics | 93 | Perceptions of equity, quality, access | y | y | |||||||||||||||||||||||||||
| 94 | Perceptions of staff | y | y | y | y | ||||||||||||||||||||||||||
| 95 | Perceptions of community partners | y | |||||||||||||||||||||||||||||
| 96 | Perceptions of external experts | y | |||||||||||||||||||||||||||||
| 97 | Perceptions of people participating in research | y | |||||||||||||||||||||||||||||
| 98 | Attitudes of participants towards science and towards research | y | |||||||||||||||||||||||||||||
| 99 | Narratives of success | y | |||||||||||||||||||||||||||||
| 100 | Narratives of performance | y | |||||||||||||||||||||||||||||
| NOTE: This table does not represent an exhaustive list, but rather highlights the potential data sources that stand out as most appropriate for each metric. | |||||||||||||||||||||||||||||||
-
Type. Each metric has been classified into a particular type. The nine types identified and captured in Figure 1 cover the pathway from research production to research impact. Each type represents a different aspect of this pathway. The nine types are as follows:
Research impacts (Research)—metrics that capture the direct development and outputs arising from conducted research
Measures of prestige (Prestige)—metrics that reflect external recognition of quality of conducted research
Teaching and career development impacts (Career)—metrics that capture the teaching and career development of researchers
Research and institutional processes (Process)—metrics that capture the effectiveness and efficiency of the administrative and insti- tutional processes underlying research
Networks and dissemination (Network)—metrics that capture the interactions of researchers and the academic institution with external stakeholders
Policy impacts (Policy)—metrics that capture the changes in policy to which research conducted at the institution has contributed
Health impacts (Health)—metrics that capture the changes in health outcomes to which research conducted at the institution has contributed
Economic impacts and commercialization (Economics)—metrics that capture the changes in the (local) economy to which research conducted at the institution has contributed
Broader metrics (Broader)—approaches that can capture information across a range of these categories.
Figure 1.
Diverse Research Outcomes Appeal to Stakeholders Internal and External to an Institution
Bonham, A. and P.A. Alberti (under review) From Inputs to Impacts: Assessing and Communicating the Full Value of Biomedical Research. Academic Medicine.
Considerations for Good Use of the Metrics
The classifications provided can be used to guide institutions as they select metrics and evaluation approaches for measuring their performance. To provide a balanced evaluation, an academic medical center is likely to want to select a range of metrics and types that cover various stakeholder groups, while also drawing on the data sources already available. The two classification tables aim to enable institutions to match metrics to their interests and available data.
It is important to note that not all metrics are appropriate to all fields of medical research. For example, some fields are more likely to produce patents than others. In addition, metrics cannot always be compared across fields. For example, what counts as a large number of publications in one field might seem a small amount to another. Fields and their differences should therefore be given substantial con- sideration in the selection of metrics.
Furthermore, the maturity and stage of development of a project or institution will determine whether or not some metrics are appropriate. Newer projects or institutions may not yet have produced long-term societal impacts. Metrics capturing such impacts would be inappropriate to the early stages of development, but could be introduced later in time.
Notes
S. Guthrie, W. Wamae, S. Diepeveen, S. Wooding, and J. Grant, Measuring Research: A Guide to Research Evaluation Frameworks and Tools, Santa Monica, Calif.: RAND Corporation, MG-1217-AAMC, 2013.
A. Bonham and P. M. Alberti, “From Inputs to Impacts: Assessing and Communicating the Full Value of Biomedical Research,” Academic Medicine, forthcoming.
The research described in this study was conducted by RAND Europe in collaboration with the Association of American Medical Colleges.