Enhancing social value considerations in prioritising publicly funded biomedical research: the vital role of peer review

Abstract

The main goal of publicly funded biomedical research is to generate social value through the creation and application of knowledge that can improve the well-being of current and future people. Prioritizing research with the greatest potential social value is crucial for good stewardship of limited public resources and ensuring ethical involvement of research participants. At the National Institutes of Health (NIH), peer reviewers hold the expertise and responsibility for social value assessment and resulting prioritization at the project level. However, previous research has shown that peer reviewers place more emphasis on a study’s methods (“Approach”) than on its potential social value (best approximated by the criterion of “Significance”). Lower weighting of Significance may be due to reviewers’ views on the relative importance of social value, their belief that social value is evaluated at other stages of the research priority-setting process, or the lack of guidance on how to approach the challenging task of assessing expected social value. The NIH is currently revising its review criteria and how these criteria contribute to overall scores. To elevate the role of social value in priority setting, the agency should support empirical research on how peer reviewers approach the assessment of social value, provide more specific guidance for reviewing social value, and experiment with alternative reviewer assignment strategies. These recommendations would help ensure that funding priorities align with the NIH’s mission and the obligation of taxpayer-funded research to contribute to the public good.

INTRODUCTION

With an annual budget of nearly $50 billion, the National Institutes of Health (NIH) is the world’s largest noncommercial funder of biomedical research [1,2]. The size of NIH’s budget and scope of its mission give it authority over a critical component of the research landscape as well as substantial influence on the commercial and non-commercial investments that capitalize on the research it funds. The central purpose of publicly funded biomedical research is to generate social value through the creation of data, information, or technology that can be used to improve the well-being of current and future populations. Rid and Roestenberg state that “clinical research generates socially valuable information either by testing new clinical, public health or other inventions or by gaining insights for developing or implementing new interventions” [3]. Social value is at the heart of the NIH’s mission – to “seek fundamental knowledge about the nature and behavior of living systems and the application of that knowledge to enhance health, lengthen life, and reduce illness and disability” [4]. The potential for social value also justifies public investment in biomedical research and the exposure of study participants to research risks and burdens [5–7].

The NIH receives five times as many applications for research projects as it can fund [8], creating an opportunity to prioritize research with the greatest potential social value. Yet, insofar as social value maps best to the NIH grant review criterion of Significance, it plays a secondary role to the criterion of Approach in NIH funding decisions. Empirical studies of NIH reviewer scores show that the Approach scores make the greatest contribution to Overall Impact scores (and subsequent likelihood of funding), while Significance scores make an important but smaller contribution [9,10].

Despite the centrality of social value to the NIH mission, the role that social value assessment plays in research priority setting by funders, peer reviewers, or research institutions has received limited attention. Some scholars have promoted the involvement of various decision-makers in assessing social value (e.g., funders, peer reviewers, or institutional entities), but the question of who bears primary responsibility for assessing social value in funding decisions is unresolved [11,12]. There is also little empirical evidence of how decision-makers evaluate potential social value [13]. Determining the appropriate locus for this evaluation and providing sufficient tools for evaluation are essential to elevating the importance of potential social value in setting priorities for research funding.

In this paper, we defend the centrality of social value, identify the key role of peer reviewers in its assessment, explore challenges to the prospective evaluation of potential social value, and recommend steps NIH (and other funders) can take to strengthen social value assessment in prioritizing research funding.

THE IMPORTANCE OF SOCIAL VALUE IN RESEARCH PRIORITY SETTING

The protection of human subjects underpins a long-standing argument for ensuring that projects have sufficient potential social value. According to the Nuremberg Code (1947), the “good of society” is a prerequisite for ethical human experiments [14]. Social value is also embedded in the Belmont Report’s description of “beneficence,” which asserts a societal obligation to “recognize the longer-term benefits and risks that may result from the improvement of knowledge and from the development of novel medical, psychotherapeutic, and social procedures” [5]. Permitting research only if it can benefit society protects participants from exploitation and counterbalances risks to participants [6,13]. An analogous social value requirement justifies non-therapeutic harm to research animals [15]. Thus, both participant protection and animal welfare require that each project meet a minimum threshold of potential social value to offset risks and burdens.

A more encompassing view of why social value matters is that its production is essential to good stewardship of scarce research resources [16]. Stewardship of resources applies to all publicly funded (and perhaps some privately funded) research, not restricted to research with human or animal subjects. As stewards of taxpayer or donor money, public and philanthropic research funders have an obligation to allocate scarce research resources toward activities that have the greatest potential to translate into benefits to society. In contrast to yes-no decisions on ethical inclusion of human subjects or animals, prioritization of research projects for funding requires comparative judgments where studies are ranked from most to least meritorious. Because not all proposals can be funded, responsible stewardship demands assessing potential social value along a continuum and funding the most promising projects. Therefore, social value assessment requires decision-makers with the expertise, information, and mandate to prioritize individual projects according to their relative potential.

WHO SHOULD EVALUATE SOCIAL VALUE?

Prioritizing support for research projects happens at multiple levels (Figure 1). At the highest level, Congress sets annual budget allocations for each NIH Institute or Center and funds programs reflecting national priorities, such as the Cancer Moonshot and the Helping to End Addiction Long-term^® (HEAL) initiative on opioid addiction research [17–19]. At the next level, NIH Institutes and Centers establish strategic plans, set funding priorities, and develop special initiatives with dedicated funds [17]. At the third level, NIH Institutes and Centers select individual research proposals to fund through the grant review process, which encompasses peer review and subsequent high-level institute review [10]. At the final level of priority setting, research institutions may prioritize projects within specific funding streams or disease areas to allocate scarce research resources and manage demands on potential participants [20,21].

Figure 1.

Decision levels for research priority setting. NIH, National Institutes of Health.

Promoting socially valuable research ought to play an important role in each stage of this pathway. Not all stages or decision-makers, however, are equally suited to assessing the potential social value of specific research endeavors. Prospective social value assessment requires two key factors: the responsibility to assess and compare a set of individual proposals and the expertise to judge potential social value. To prioritize research with maximum potential social value, decision-makers must have the authority to make comparative evaluations that enable project-level decisions. Such evaluations require an understanding of the factors that make a project likely to produce social value, such as the importance of the problem that the proposal addresses, the magnitude of potential social benefit if the study achieves its aims, the state of the scientific field as it relates to the likelihood that the study will achieve those aims (components of which reasonably fall under the auspices of Approach), and the timeline for realizing benefits.

Upstream decision-makers in Congress and at the NIH Institutes and Centers have high-level responsibility for the research agenda, but they lack specific responsibility or expertise for project-level decisions (beyond NIH Advisory Council confirmation of project funding decisions). NIH staff in charge of strategic planning have a broad view of the scientific enterprise and research trends within general scientific areas, but they do not have specialized expertise in every area within their portfolios. Downstream, institutional review boards (IRBs) and institutional animal care and use committees (IACUCs) have a responsibility to ensure studies have sufficient potential social value to balance net risks, but their determinations are not comparative across projects, they lack specific expertise to make scientific judgments, and they typically request modifications to study designs rather than rejecting studies outright. Institutional scientific review committees do have the responsibility and scientific expertise to evaluate social value, but they have limitations related to their scope of responsibility: only a small subset of studies undergoes institutional priority setting and the committees’ reach is limited to studies at a single institution.

Given the limitations of other decision-makers, the locus of social value assessment properly belongs at the proposal review level. Peer reviewers are directly responsible for project-level assessments, they score projects along a continuum instead of against a minimum threshold, and they have the expertise to evaluate potential social value. Despite NIH instructions to reviewers to evaluate each proposal independently, scores are unavoidably comparative and the grant review process results in an ordered list of proposals. Peer reviewers ideally have a deep understanding of the scientific enterprise and expert knowledge of important scientific or clinical needs within relatively narrow subfields, equipping them to evaluate the potential social value of proposed studies. While individual reviewers may not have expertise on all aspects of social value, as a committee, they are still in the best position to make social value judgments based on the information available to them. Peer reviewers serve at the direction of the NIH, so how NIH instructs reviewers is critical to establishing the role that social value plays in study assessments and rankings.

REVIEW CRITERIA: SIGNIFICANCE AS SOCIAL VALUE

NIH requires peer reviewers to assess proposals on five scored criteria: Significance, Approach, Innovation, Environment, and Investigators [22,23]. Reviewers combine the five criterion scores into a single Overall Impact score according to implicit weights they deem appropriate—a process called commensuration. Individual reviewer discretion in commensuration creates inconsistency and the potential for bias that risk undermining research’s knowledge production goals [24]. To address this issue, in November 2022 the NIH Center for Scientific Review announced proposed changes, which would combine the criteria into three factors that contribute to Overall Impact: the “Importance of the Research” encompassing Significance and Innovation, “Rigor and Feasibility” encompassing Approach, Rigor and Feasibility, and “Expertise and Resources” encompassing Investigators and Environment scores [25]. Under the proposed framework, Expertise and Resources would be judged on whether they are sufficient for the proposed research rather than being scored. The proposed changes are still under development, presenting an opportunity for NIH to enhance the evaluation of social value.

Under the existing system, social value is best approximated by Significance, which the parent announcement for R01 grant proposals describes as follows:

“Does the project address an important problem or a critical barrier to progress in the field? Is the prior research that serves as the key support for the proposed project rigorous? If the aims of the project are achieved, how will scientific knowledge, technical capability, and/or clinical practice be improved? How will successful completion of the aims change the concepts, methods, technologies, treatments, services, or preventative interventions that drive this field?” [26]

Significance thus encompasses the importance of the problem being addressed, the scientific foundation of the proposal (an aspect of feasibility or likelihood), the potential influence on fundamental knowledge or methods, and the potential impact on the field beyond the current study. The NIH working group on review criteria recommended conceptual definitions of criteria to enable high-level scientific judgements [27]. However, broad descriptions can lead to wide variation in reviewer interpretations, overlooking important elements, and under-weighting of vague criteria in Overall Impact scores.

The current description of Significance can be mapped to Rid and Roestenberg’s framework for assessing potential social value, providing additional granularity. Rid and Roestenberg break down the elements of social value assessment into two categories: the expected magnitude of social value and the likelihood of realizing that value [3]. Studies with greater magnitude of benefits (e.g., size of the affected population, degree of unmet need, potential gains in quality or quantity of life, potential impact on scientific technologies or paradigms, or potential reach of a research database) should be considered to have high social value. Considering these elements would allow for more consistent assessments of the importance of the problem, the influence on fundamental knowledge, and the influence on the field.

The likelihood that findings will influence future research or influence clinical or public health practice (including scientific premise, implementation feasibility, costs, and acceptability) are all key considerations for social value. Though Rid and Roestenberg situate the likelihood of study success within their social value framework, NIH review guidelines assign the assessment of methodological appropriateness, alternatives, scientific rigor, feasibility, and the quality of reporting and dissemination to the Approach criterion. In this way, NIH reviewers must distinguish the likelihood of success (Approach) from the magnitude of impact if successful (Significance). Nevertheless, some aspects of likelihood of success – specifically, the assessment of the scientific foundation of the proposal – fall within the NIH description of Significance.

Additional components of social value that Rid and Roestenberg’s framework and the NIH description of Significance may not sufficiently capture include the timeliness of the research, how long it may take for benefits to be realized, and the potential effects of the research on health equity. The description of Significance also does not capture whether proposed projects have the potential for negative social value—for example, research with pathogens that may increase risks of bioterrorism or pandemics and sociogenomics research that risks feeding racist or homophobic agendas [28,29].

WHY SIGNIFICANCE IS UNDERVALUED RELATIVE TO APPROACH

Currently, reviewers give lower weight to Significance than to Approach when determining Overall Impact scores, the primary determinants of which studies are ultimately prioritized for funding. For example, in a study by Lindner et al, of about 18,000 R01 applications scored in summer 2013, Approach score had the highest correlation with Overall Impact (0.85), followed by Significance (0.67) [10]. Similarly, Eblen et al showed that, of over 123,000 NIH applications reviewed in fiscal years 2010–2013, the change in Overall Impact score associated with a one-point increase in Approach score was more than double that associated with an equivalent increase in Significance score (controlling for factors like career stage, institution type, and investigator demographics had minimal effects on the results) [9].

Multiple factors might explain the secondary importance of Significance in determining overall Impact Scores. First, peer reviewers may believe that Approach is a more important factor in deciding research priorities compared to Significance. For example, reviewers may believe that social value will result only from the most well-designed studies, making Approach a prerequisite for social value and therefore a dominating criterion.

Second, it has been suggested that reviewers are reluctant to give low scores on Significance even if they actually believe lower scores are merited [27]. If Significance scores seem more subjective than Approach scores, reviewers may not feel comfortable providing honest but critical opinions despite their tacit contribution to overall assessments and recommendations.

Third, if Significance scores were roughly the same across proposals, Significance would contribute little variation to Overall Impact scores. Criteria with greater influence on Overall Impact scores have more score dispersion and higher (worse) mean scores [9,10]. Nevertheless, Significance scores are well distributed across the range of possible scores, indicating that reviewers distinguish gradations of Significance that they could factor into Overall Impact. The degree that Overall Impact scores reflect this variation is a matter of how much weight reviewers choose to give Significance relative to other factors.

Fourth, reviewers might believe that, while Significance or social value is an important part of priority setting, other decision-makers bear responsibility for evaluating it. For example, they may believe that value decisions about Significance are better suited to funding agencies, which set high-level priorities and make final decisions about funding individual projects.

Finally, evaluating potential impact is challenging, so reviewers may lack confidence in their ability to score Significance. Compared to systematic assessment of the appropriateness and rigor of research methods, reviewers may enter the review process without well-defined rubrics or procedures for assessing potential impact. Uncertainty about the magnitude and the likelihood of impact on scientific or clinical practice makes Significance difficult to judge. This uncertainty may be greatest for basic science proposals. Considering tradeoffs across different types of research poses additional challenges— comparing, for example, rare vs. common disease applications, basic vs. applied research with vastly different time horizons, or high-risk/high-reward studies vs. those offering incremental advances. Current NIH instructions may be insufficient to assist reviewers in making Significance assessments.

MOVING FORWARD: ENABLING COMPARATIVE SOCIAL VALUE JUDGMENTS

As the primary purpose of taxpayer-funded research is to produce generalizable knowledge that aligns with the interests of the public, prioritizing studies according to their relative potential for social value is crucial. There are several steps NIH should take to ensure social value is given sufficient influence in funding allocation.

First, NIH should support empirical research on how peer reviewers assess Significance, the process they go through in evaluating it, and the challenges they face. How NIH peer reviewers operationalize the concept of Significance is important to understanding the processes that drive scores. Additionally, understanding why reviewers give less weight to Significance compared to Approach under the current process is essential to shaping guidance on commensuration.

Second, NIH should more precisely specify the review criteria relevant to social value and should provide more detailed instructions on considerations of magnitude and likelihood of impact for different types of research. Evaluating potential social value is a challenging task, and reviewers need guidance that will help them make rigorous, well-justified assessments. Developing guidance to facilitate these processes can help reviewers grapple with challenging tradeoffs and consider uncertain outcomes. Reviewers need a conception of social value that fits basic and applied research, allows for a balanced portfolio of higher-risk and lower-risk studies, does not penalize studies that are likely to provide (useful) negative findings, and can be assessed prospectively. Greater specificity in instructions would reduce bias, improve inter-rater reliability, and target assessments toward what funders believe a prospective assessment of social value ought to measure.

Third, NIH should emphasize the essential role peer reviewers play in evaluating social value by elevating the status of social value in the review process. One strategy to ensure that social value receives dedicated focus is to explore alternative strategies for assigning reviewers. For example, each reviewer might be given primary responsibility for evaluating either (1) Significance and Innovation, or (2) Approach, Investigators (Expertise), and Environment (Resources). NIH ought to pilot-test criteria-specific reviewer roles as part of the current revisions to the review process.

CONCLUSION

Because peer reviewers bring relevant expertise, evaluate the merits of individual projects for the purpose of funding allocation at the individual-project level, and are in a position to make comparative judgments, they are the only decision-makers properly situated to evaluate potential social value. Yet, insofar as the NIH review criterion of Significance captures potential social value, its influence on overall priority is secondary to the evaluation of methodological and analytical strategies. Research funding decisions influence the direction of federally funded scientific knowledge generation, ultimately shaping whether and how society benefits from our collective research investments. It is therefore an ethical obligation for all publicly funded research to place greater emphasis on social value when allocating funds. The NIH has the opportunity and responsibility to improve the evaluation of potential social value and increase its importance in determining how we use our limited research funds to improve population health and advance biomedical and behavioral science.