A Challenge for Clinical Pharmacologists: How Can We Measure Scientific Impact of Publications in Drug Development and in Regulatory Decision Making?

Many articles in clinical pharmacology may highly influence drug development and regulatory decision making, far beyond the measure conveyed by the journal impact factor. Quantifying impact from the grey content (e.g., nonpublished information) remains an evolving dilemma. A collaboration is proposed with library scientists and bioinformaticians to develop methods to measure the impact of grey content on drug development and regulatory decision making. Mechanisms to reward this impact are discussed.

Methods for evaluating the impact of journal articles have been considered since the late 1950s, with the use of the Science Citation Index beginning in 1961¹. The Journal Impact Factor (JIF), which is the average number of citations per article in a given period (2 to 3 years), does not provide information about individual articles¹ and was not intended to reflect the scientific merit, quality, or relevance of the articles in a journal. Nonetheless, publication of articles in journals with a high JIF is deemed (implicitly and explicitly) to indicate the value of scholarly work and contributes to tenure and funding decisions. However, the JIF does not account for journals not included in the Journal Citation Reports, nor does it account for “grey” content (i.e., evidence not appearing in commercial publications)², including: legal briefs, theses and dissertations, government reports, patent applications, expert testimony, drug development, and regulatory decision-making documents. In response to an evolving scientific and professional communications landscape, there are new fields focused on investigating and developing non-traditional, objective measures of scientific impact including the Relative Citation Ratio by the National Institutes of Health, Cite Score by Elsevier, and Impact Quotient³, none of which have been embraced by the scientific community. Altmetrics, an evolving area, create alternative metrics to measure the impact of scholarly content based on how far and wide it travels through social media^4–6. At present, neither the JIF nor altmetrics properly evaluate the impact of an article on any real-world application to drug development or regulatory decision making. This impact is important because it can lead to societal improvements (e.g., availability of new drugs to treat diseases with unmet medical needs).

Publications in clinical pharmacology, taken together with a broad range of scientific information from journals and grey content, may redirect scientists in industry and regulatory sectors to change the design (e.g., use of adaptive design), conduct (e.g., use of itraconazole rather than ketoconazole), analysis (e.g., use of new exposure-response models), and/or interpretation of a study, thus impacting drug development and making the process more efficient. Drug development may benefit from publications that clearly explain clinical pharmacology concepts, such as dose selection for drugs developed under the animal rule⁷ or evaluation of the potential for drug interactions^8,9. However, this type of impact is difficult to quantify. Given that it now costs more than 2.6 billion dollars to get a drug to the market¹⁰, articles describing innovative approaches have the potential to improve the efficiency of drug development. For example, articles that enhance our understanding of what in vitro and in vivo studies may be appropriate, how modeling and simulation can be used to optimize study design, which design factors can reduce placebo response, and what methodology can be applied to better assess dose response, could have a major impact on the success and efficiency of drug development.

The applications of impactful articles to clinical drug development extend to preclinical drug development as well. Published findings from innovative preclinical studies may lead to changes in targets and/or screening of drug candidates in the academic and industrial setting. An important topic that has had major impact is the role of membrane transporters in the development of safe and efficacious drugs. For instance, as new transporters and their substrates and inhibitors are identified, changes in drug development are engendered that are difficult to quantify. As new information about genetic polymorphisms is discovered, articles may include important information used in drug development and regulatory decision making.

Importantly, clinical pharmacology information gained from publications may contribute to regulatory documents that may or may not be publicly available. Public documents include the product label, which contains information relevant to clinical pharmacology in ≥7 of 17 total product label sections (e.g., dosage and administration, precautions, warnings). Public documents also include the summary basis of approval, slide packets for advisory committees, and regulatory guidances. Of the 14 US Food and Drug Administration (FDA) Clinical Pharmacology guidances currently available, eight include ≥10 literature references and two include ≥50 references. The literature references were published 0.5 to 10 years prior to the respective guidance. Nine of the guidances reference ASCPT journals (Clinical Pharmacology and Therapeutics and CPT: Pharmacometrics and Systems Pharmacology).

Briefing documents are key regulatory documents that influence drug development decisions. In these documents, companies state positions that are supported by data (both confidential and data from the literature) and ask regulatory agencies if they support the positions. Combined with other sources of information, literature citations in such documents may support the company’s decision-making process, educate regulatory reviewers, or both. No metric captures these confidential data. Other regulatory documents (e.g., new drug applications, investigational drug applications) are confidential, whereas protocols and clinical study reports are published in registries and as articles.

Another limitation of the JIF, as it relates to the impact of literature on drug development decisions, is that it only assesses impact over a 2–3-year period. Drug discovery and development timelines tend to be much longer, with clinical studies taking 1–2 years to complete and a full drug development program taking > 6 years to complete. Thus, the true impact of a publication on drug development cannot be accurately assessed over a 2–3- year period.

Without question, measuring the impact of a article on drug development and regulatory decisions is not a simple task. Both the JIF and altmetrics have limitations. For example, using the number of downloads as a metric is imperfect because downloading the article does not imply that the information was used to impact drug development decisions. The appropriateness of including non-traditional sources also depends on the vetting process of the information. Clearly, a report where a scientific panel was used to vet all data may be as important as a published article; unpublished negative data may also provide critical information². One potential metric is the number of citations in regulatory documents (e.g., briefing documents, new drug applications, investigational drug applications). However, because these are confidential documents, the citations are not currently available for inspection by the scientific community. A review of the number of citations of articles appearing in key clinical pharmacology journals in regulatory submission documents over a 5-year period may be eye opening to measure the real-world impact of our integrated discipline. This task would be difficult, if not impossible, without the help of colleagues in informatics at pharmaceutical companies or at regulatory agencies. At a minimum, a dialogue should be initiated between clinical pharmacology, regulatory and industry scientists and bioinformaticians to assess the impact of scientific reports on drug development and regulatory decisions. Such an exercise could be broadened beyond clinical pharmacology, providing a snapshot of the impact of multiple scientific disciplines on the discovery and development of new medicines.

Explicitly considering impact of articles on drug development and regulatory review has important implications for the journal review process. Many journals have “Study Highlights” sections, in which authors should consider including statements about how the study could influence drug development decisions. Secondly, journal reviewers are often asked to address scientific impact, but the societal impact of the work is usually not specifically called out. Some editorial teams have broad representation from academic, industrial and regulatory sectors, whereas others draw their representation mostly from academic ranks. Scientists and thought leaders who have experience in the drug development and regulatory process may be in a better position to judge this type of impact of a article. In the past, JIF was a significant consideration to decide where to submit manuscripts for publication, but in the digital age, many authors are more focused on rapid, open access, rather than JIF. The new generation of scientists also support the quick dissemination of scientific information through non-traditional methods.

Once the impact of conventional and unconventional scientific information on drug development and regulatory decision making is qualitatively or quantitatively measured, there will be implications for academia, industry and regulatory scientists. Authors of the three to five most impactful clinical pharmacology articles in drug development and regulatory science over a specified period of time should receive recognition (e.g., awards/rewards) for their articles. These articles with impact on drug development and regulatory decision making need to be recognized as important in academia and considered when preparing class materials to train the next generation of industrial and regulatory scientists.

The authors suggest that clinical pharmacologists evaluate whether to join the San Francisco Declaration on Research Assessment (DORA, https://sfdora.org/), an interdisciplinary organization that supports the assessment of research publications on the individual merit of each article, not by the JIF, and that the JIF does not have a place in funding, appointment, promotion, and tenure considerations. The DORA has support from 500 organizations and over 12,000 individuals. The authors also believe that appropriately vetted grey content should be considered valuable and impactful. In the case of impact on drug development and regulatory decision making, increased discussion, collaborations, and improved communication among clinical pharmacologists, library scientists, and informaticians is important to increase our understanding of the impact of articles in improving drug development and regulatory decisions. The authors of these articles, which ultimately lead to societal improvements, and the journals in which they are published, deserve a new level of recognition for their contributions.