Abstract
Background
More than 90% of clinical-trial compounds fail to demonstrate sufficient efficacy and safety. To help alleviate this issue, systematic literature review and meta-analysis (SLR), which synthesize current evidence for a research question, can be applied to preclinical evidence to identify the most promising therapeutics. However, these methods remain time-consuming and labor-intensive. Here, we introduce an economic formula to estimate the expense of SLR for academic institutions and pharmaceutical companies.
Methods
We estimate the manual effort involved in SLR by quantifying the amount of labor required and the total associated labor cost. We begin with an empirical estimation and derive a formula that quantifies and describes the cost.
Results
The formula estimated that each SLR costs approximately $141,194.80. We found that on average, the ten largest pharmaceutical companies publish 118.71 and the ten major academic institutions publish 132.16 SLRs per year. On average, the total cost of all SLRs per year to each academic institution amounts to $18,660,304.77 and for each pharmaceutical company is $16,761,234.71.
Discussion
It appears that SLR is an important, but costly mechanisms to assess the totality of evidence.
Conclusions
With the increase in the number of publications, the significant time and cost of SLR may pose a barrier to their consistent application to assess the promise of clinical trials thoroughly. We call on investigators and developers to develop automated solutions to help with the assessment of preclinical evidence particularly. The formula we introduce provides a cost baseline against which the efficiency of automation can be measured.
Keywords: Automation, Artificial intelligence, Clinical trial, Clinical research, Dollar cost, Labor costs, Machine learning, Meta-analysis, Systematic review
1. Introduction
Testing preclinical observations in humans poses a critical stumbling block in developing new clinical interventions that benefit patients. More than 90% of the compounds that enter clinical trials fail to demonstrate sufficient efficacy and safety to gain regulatory approval [[1], [2], [3], [4]]. Recently, this issue has raised calls for thorough evaluations of “whether an experimental treatment is promising enough to warrant testing on people” [5]. Systematic literature review and meta-analysis (SLR) methods are one mechanism to synthesize the totality of the current evidence and assess the promise of clinical trials [[6], [7], [8], [9], [10]].
Although not without flaws [11], SLR methods can capture relevant data to summarize different studies and evaluate their efficacy [12,13]. These methods also apply to preclinical evidence identifying weaknesses in animal studies to propose better mechanisms to determine the most promising therapeutic targets and drugs [14,15]. However, despite the usefulness of SLR, these research methods remain time-consuming and labor-intensive [16,17]. In this communication, we quantify their significant expense of SLR to both academic institutions and pharmaceutical companies by applying a new economic formula. We argue for better solutions to reduce the cost and conclude with suggestions of how machine learning might provide the necessary infrastructure and resources to expedite SLR methods.
The purpose of this paper is to introduce a formula for calculating the hidden cost of performing systematic reviews and meta-analyses and to highlight what the cost can amount to for both research institutions and the pharmaceutical industry.
2. Methods
2.1. An economic formula for quantifying the cost of SLR methods
Previous work quantified the labor effort involved in SLRs, but less attention has been paid to the actual dollar costs involved. The estimated labor effort to produce a single SLR ranges from a minimum of 6 months when an investigator devotes 10–20 h per week [18] to, on average, a total of 16 months involving five co-authors [19], or 1–2 years for completing a Cochrane review [20].
The formula we introduce here consists of four main input parameters (Equation (1)).
(1) |
The formula estimates the total number of SLRs, both published and unpublished via two parameters, (Npub) and (Pr(unpub)) respectively, and quantifies how much it costs to produce each review via the time (Ehrs) and cost-per-person (Cperson). The inclusion of unpublished SLRs in this calculation is critical. Recent work analyzed this issue as it relates to “network meta-analyses,” one of the most rigorous and advanced forms of this type of analysis [21]. The authors found that 44% (76/174) of network meta-analyses done by (or on behalf of) pharmaceutical companies were never published. While this may not hold for academically focused research institutions, it imposes a substantial cost for pharmaceutical companies. Combining the terms of the formula allows us to estimate the annual costs of producing all of the SLRs at an institution.
Taking an average of the labor effort values above, we estimate it takes one scientist 1.72 years (see Appendix) with an average labor cost of $82,090 per year for a scientist [22]. To calculate the expense for a years' worth of SLRs, we estimate the total number of SLRs published at major academically-focused research institutions [23] and the largest pharmaceutical companies by revenue [24]. To do this, we measured the number of SLRs published in the past five years according to PubMed queries for the main three types of comparative studies (“meta-analysis,” “systematic review” and “comparative effectiveness”) for each of the institutions and companies (Table 1). The limitation of this methodology is that if only one of the authors is company-affiliated, the cost per company should be scaled accordingly. We limited our search to the last five years to account for the temporal effects associated with the more recent popularity in these types of publications.
Table 1.
Average | NIH-funded research institution | Number of articles in the last 5 years | Company | Number of articles in the last 5 years |
---|---|---|---|---|
Johns Hopkins University | 1362 | J&J | 90 | |
University of Michigan | 751 | Roche | 421 | |
University of Pittsburgh | 568 | Pfizer | 638 | |
Washington University in St. Louis | 540 | Novartis | 575 | |
Stanford University | 655 | Sanofi | 263 | |
University of California, San Francisco | 217 | GSK | 364 | |
University of Pennsylvania | 752 | Merck | 392 | |
Massachusetts General Hospital | 752 | AbbVie | 146 | |
Brigham and Women's Hospital | 836 | Bayer | 232 | |
University of California San Diego | 175 | Abbot | 203 | |
5 year average | 660.80 | 332.40 | ||
1 year average | 132.16 | 66.48 |
3. Results
We found that on average, each major academic institution will publish 132.16 and each pharmaceutical company will publish 66.48 SLRs per year (Table 1). According to our formula, each single SLR costs $141,194.80 ($82,090 X 1.72). Given that 44% of meta-analyses in the pharmaceutical industry go unpublished (i.e., Pr(unbpub) is 0.44), we estimate the pharmaceutical industry publishes 118.71 studies, on average, per year. Therefore, the total cost of all SLRs per year for each pharmaceutical company averages $16,761,234.71, and each academic institution averages $18,660,304.77 ($141,194.80 X 132.16), as we assume all studies are published in academia (i.e., Pr(unpub) is 0)).
4. Discussion
The purpose of this paper is to introduce a formula for calculating the hidden cost of performing SLRs and to highlight what the cost can amount to for both research institutions and the pharmaceutical industry. SLRs are important but costly mechanisms to develop research hypotheses and answer research questions. We argue that automation (e.g., machine learning, artificial intelligence) could significantly lower the cost of SLRs, ensuring these important efforts will not be abandoned due to their expense. The formula we present here provides a cost baseline against which the efficiency of machine learning can be measured. Current efforts to scale the SLR review process manually include the Cochrane group [10] that addresses the scalability of effort by leveraging tens of thousands of volunteers. While estimable, essentially this distributes the workload and associated cost across a large number of volunteers, spreading it out to become more manageable. We argue that, as the number of questions to answer and the size of the literature both increase, this leaves room for automated methods to fill the need.
Considering our formula, we can explicitly tie machine-learning to specific parameters in the formula, which allows us to understand their effect on the overall effort (and cost) of SLRs. Machine learning will affect how people are paid, as tasks become automated away, but that is challenging to forecast – therefore, we will focus our analysis on the hours it saves (Ehrs) rather than the cost-per-person (Cperson). In a simple argument, one can imagine researchers leveraging tools to automatically screen papers [25] for quality, doing in seconds what used to take hours. As another example, following an approach similar to Ref. [26], authors replicated a systematic review in days, when the original work took months [27] - a significant time savings. If these tools are repurposed for preclinical evidence, and not just clinical, the costs associated with Ehrs would be dramatically reduced, allowing SLR to scale with the number of questions researchers could ask.
4.1. Limitations
There are limitations to our approach. The cost estimates we present here are based on data from the top 10 NIH-funded research institutions and the 10 largest pharmaceutical companies by revenue. As a result, the cost of systematic reviews and meta-analyses for institutions with less NIH funding or revenue would be less. However, the main purpose of the paper is to introduce a formula for calculating the hidden cost of performing SLRs and to highlight what the cost can amount to. Additionally, automation such as machine learning methods are not perfect – but they improve over time, and we are optimistic they will eventually reach human performance. Automation also cannot yet address more profound issues, such as identifying preclinical studies that are “plagued by poor design, implementation and reporting” [[28], [29], [30]]. But this is not a detriment. Instead, this is an opportunity for researchers and developers to make significant advances in artificial intelligence research. The rise in automation for SLR-related tasks is the only scalable way to understand the increasing volumes of literature [31]. The human capacity, on the other hand, to read and understand the growing body of preclinical evidence is largely set.
5. Conclusion
The cost of SLRs are significant and may pose a barrier to their consistent application to thoroughly assess the promise of clinical trials. We need better approaches and tools that enable more researchers with limited budgets and time constraints to take advantage of SLR methods. The goal is to make the available evidence more accessible and to assist in the detection of insufficient evidence before the approval and activation of clinical trials. Therefore, we call on investigators and developers to contribute to the development of automated solutions to particularly help with the assessment of preclinical evidence. Such tools could provide investigators, ethical review boards (IRBs), and efforts such as the national SMART IRB Reliance Initiative [31] with an efficient technical solution to better harness the totality of evidence and to focus the investment in clinical trials on those with sufficient evidence.
Funding
This work was supported by Evid Science, a for-profit corporation.
Conflicts of interest
Co-author Matthew Michelson is the CEO of Evid Science, which funded this research. Co-author Katja Reuter does not report any financial or other conflict of interest.
Contributor Information
Matthew Michelson, Email: mmichelson@evidscience.com.
Katja Reuter, Email: katja.reuter@med.usc.edu.
Appendix.
We estimate that a researcher might work 15 h per week on such a task (using the mid-point between 10 and 20 h). Since working at such a pace could produce a review in 6 months, at a minimum, we take an average of this six-month estimate and the 16-month estimate, yielding an estimate of 11 months per review, at roughly 15 h per week. So, working 15 h per week (37.5% of the time) for 11 months translates into 4.125 months per year. Put another way, five co-authors working 4.125 months per year is the same as 1.72 scientists working for an entire year.
References
- 1.Van Norman G.A. Drugs, devices, and the FDA: Part 1: an overview of approval processes for drugs. JACC Basic Transl Sci. 2016;1:170–179. doi: 10.1016/j.jacbts.2016.03.002. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Plenge R.M., Scolnick E.M., Altshuler D. Validating therapeutic targets through human genetics. Nat. Rev. Drug Discov. 2013;12:581–594. doi: 10.1038/nrd4051. [DOI] [PubMed] [Google Scholar]
- 3.Yang Y.T., Chen B., Bennett C. “Right-to-Try” legislation: progress or peril? J. Clin. Orthod. 2015;33:2597–2599. doi: 10.1200/JCO.2015.62.8057. [DOI] [PubMed] [Google Scholar]
- 4.Van Norman G.A. Drugs, devices, and the FDA: Part 2: an overview of approval processes: FDA approval of medical devices. JACC Basic Transl Sci. 2016;1:277–287. doi: 10.1016/j.jacbts.2016.03.009. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Kimmelman J., Federico C. Consider drug efficacy before first-in-human trials. Nature. 2017;542:25–27. doi: 10.1038/542025a. [DOI] [PubMed] [Google Scholar]
- 6.Haidich A.B. Meta-analysis in medical research. Hippokratia. 2010;14:29–37. [PMC free article] [PubMed] [Google Scholar]
- 7.Schulz K.F., Altman D.G., Moher D., CONSORT Group CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. PLoS Med. 2010;7 doi: 10.1371/journal.pmed.1000251. e1000251. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Gopalakrishnan S., Ganeshkumar P. Systematic reviews and meta-analysis: understanding the best evidence in primary healthcare. J. Fam. Med. Prim. Care. 2013;2:9–14. doi: 10.4103/2249-4863.109934. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.DerSimonian R., Laird N. Meta-analysis in clinical trials revisited. Contemp. Clin. Trials. 2015;45:139–145. doi: 10.1016/j.cct.2015.09.002. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Welcome. https://www.cochrane.org/ (n.d.)
- 11.Meta-analyses Were Supposed to End Scientific Debates. Often, They Only Cause More Controversy. Science | AAAS; 2018. https://www.sciencemag.org/news/2018/09/meta-analyses-were-supposed-end-scientific-debates-often-they-only-cause-more (accessed January 18, 2019) [Google Scholar]
- 12.Sena E., van der Worp H.B., Howells D., Macleod M. How can we improve the pre-clinical development of drugs for stroke? Trends Neurosci. 2007;30:433–439. doi: 10.1016/j.tins.2007.06.009. [DOI] [PubMed] [Google Scholar]
- 13.Gurevitch J., Koricheva J., Nakagawa S., Stewart G. Meta-analysis and the science of research synthesis. Nature. 2018;555:175–182. doi: 10.1038/nature25753. [DOI] [PubMed] [Google Scholar]
- 14.Doi S.A.R., Barendregt J.J., Mozurkewich E.L. Meta-analysis of heterogeneous clinical trials: an empirical example. Contemp. Clin. Trials. 2011;32:288–298. doi: 10.1016/j.cct.2010.12.006. [DOI] [PubMed] [Google Scholar]
- 15.McCann S.K., Sena E.S., Currie G.L., Macleod M.R., Howells D.W. Systematic review and meta-analysis: important tools in understanding drug development for stroke. In: Lapchak P.A., Zhang J.H., editors. Neuroprotective Therapy for Stroke and Ischemic Disease. Springer International Publishing; Cham: 2017. pp. 73–93. [Google Scholar]
- 16.Ip S., Hadar N., Keefe S., Parkin C., Iovin R., Balk E.M., Lau J. A Web-based archive of systematic review data. Syst. Rev. 2012;1:15. doi: 10.1186/2046-4053-1-15. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.of Medicine I. The National Academies Press; Washington, DC: 2011. Finding what Works in Health Care: Standards for Systematic Reviews. [PubMed] [Google Scholar]
- 18.Whitaker E. 2015. UCSF Guides: Systematic Review: when Will I Be Finished?https://guides.ucsf.edu/c.php?g=375744&p=3041343 [Google Scholar]
- 19.Borah R., Brown A.W., Capers P.L., Kaiser K.A. Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry. BMJ Open. 2017;7 doi: 10.1136/bmjopen-2016-012545. e012545. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Proposing and registering new Cochrane reviews. https://community.cochrane.org/review-production/production-resources/proposing-and-registering-new-cochrane-reviews (n.d.)
- 21.Schuit E., Ioannidis J.P. Network meta-analyses performed by contracting companies and commissioned by industry. Syst. Rev. 2016;5:198. doi: 10.1186/s13643-016-0377-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Medical Scientists: Occupational Outlook Handbook. U.S. Bureau of Labor Statistics; 2018. https://www.bls.gov/OOH/life-physical-and-social-science/medical-scientists.htm [Google Scholar]
- 23.Top 50 NIH-Funded Institutions of 2018. GEN - Genetic Engineering and Biotechnology News; 2018. https://www.genengnews.com/a-lists/top-50-nih-funded-institutions-of-2018/ [Google Scholar]
- 24.The Free Encyclopedia. 2019. Wikipedia contributors, List of largest pharmaceutical companies by revenue, Wikipedia.https://en.wikipedia.org/w/index.php?title=List_of_largest_pharmaceutical_companies_by_revenue&oldid=877244851 [Google Scholar]
- 25.Del Fiol G., Michelson M., Iorio A., Cotoi C., Haynes R.B. A deep learning method to automatically identify reports of scientifically rigorous clinical research from the biomedical literature: comparative analytic study. J. Med. Internet Res. 2018;20 doi: 10.2196/10281. e10281. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Michelson M. 2014 AAAI Fall Symposium Series. 2014. Automating meta-analyses of randomized clinical trials: a first look.http://www.aaai.org/ocs/index.php/FSS/FSS14/paper/download/9100/9112 [Internet], aaai.org. [Google Scholar]
- 27.Michelson M., Ross M., Minton S. AI2: leveraging machine-assistance to replicate a systematic review. Value Health. 2019;22:S34. [Google Scholar]
- 28.Mak I.W., Evaniew N., Ghert M. Lost in translation: animal models and clinical trials in cancer treatment. Am. J. Transl. Res. 2014;6:114–118. [PMC free article] [PubMed] [Google Scholar]
- 29.Rice A.S.C., Cimino-Brown D., Eisenach J.C., Kontinen V.K., Lacroix-Fralish M.L., Machin I., Preclinical Pain Consortium. Mogil J.S., Stöhr T. Animal models and the prediction of efficacy in clinical trials of analgesic drugs: a critical appraisal and call for uniform reporting standards. Pain. 2008;139:243–247. doi: 10.1016/j.pain.2008.08.017. [DOI] [PubMed] [Google Scholar]
- 30.Ghinea N., Lipworth W., Kerridge I., Day R. No evidence or no alternative? Taking responsibility for off-label prescribing. Intern. Med. J. 2012;42:247–251. doi: 10.1111/j.1445-5994.2012.02713.x. [DOI] [PubMed] [Google Scholar]
- 31.Rawat S., Meena S. Publish or perish: where are we heading? J. Res. Med. Sci. 2014;19:87–89. [PMC free article] [PubMed] [Google Scholar]