Opening a debate on open‐source modeling tools: Pouring fuel on fire versus extinguishing the flare of a healthy debate

Abstract

As model‐informed drug development becomes an integral part of modern approaches to the discovery of new therapeutic entities and showing their safety and effectiveness, modalities of incorporating the paradigm into widespread practice require a revisit. Traditionally, modeling and simulation (M&S) have been performed by specialized teams who create bespoke models for each case and have reservations about letting modeling be done by the greater mass of scientists engaged in various stages of drug development. An analogy can be drawn between M&S and automobiles: typical drivers of ordinary cars use them for daily tasks, such as going from point A to B whereas specialized Formula 1 drivers using bespoke individually made cars to test the latest technologies. The reliability and robustness of ordinary cars for the first group requires elements related to quality and endurance that are very different from those applicable to any Formula 1 car supported by a large team of engineers. In this commentary, we frame and analyze the problems concerning the structure and setup of various M&S tools, and their pros and cons. We demonstrate that many misconceptions have precluded having an open discussion on what each modality of M&S tools strives to achieve, and we provide data and evidence that support the move of M&S to main stream use by many, as opposed to specialized usage by few. Parallels are drawn in many other areas involving laboratory instrumentation, statistical analyses, and so on.

INTRODUCTION

As the concept of model‐informed drug development (MIDD) is gaining higher recognition on the back of several regulatory ¹ , ² , ³ , ⁴ and industrial initiatives, ⁵ , ⁶ the debate on the nature of the modeling platforms and associated databases that constitute a “model” is heating up.

Many players hold strong opinions particularly on the “open source” models (see the definitions we use in Table 1). People with backgrounds in computational chemistry/chemical physics are aware of parallels in that field and the strength of the open‐source community, whereas being conscious of the role that commercial players (e.g., Schrodinger; https://www.schrodinger.com) play in the space. Therefore, the subject is not new in itself, although the intensity and inclusiveness of the debate in the area of pharmacometrics and quantitative pharmacology has not been widespread. The debate mainly involved the proponents of the open‐source approaches, without detailed and evidence‐based arguments or the declaration of implicit or explicit conflicts of interest. On a large scale, these follow arguments similar to those used to criticize Innovative Medicine Initiative (IMI) projects and their use of the public purse to subsidize the activities of fully fledged commercial entities, such as big pharmaceutical companies, consultancy services, and contract research organizations, or database and tool providers. ⁷

TABLE 1.

Definitions used throughout this commentary in relation to modeling tools

Software/platform

Executable computer codes (with or without data) enabling the user to perform modeling (involving data analysis or simulations). Examples vary from common (general use) platforms, such as Microsoft Excel to specialized systems, such as NonMEM.

Computing/programming language

The ways models are built within a platform vary and sometimes this requires writing (coding) model equations in a language that might be a general programming language like FORTAN, SBML, LUA, C+, or one specific to the chosen platform, like PML for Phoenix, the R scripting language, or SimBiology for MATLAB.

Model

Set of equations and algorithms put together within a platform (using a modeling language or a graphical selection of options, etc.) for the purpose of analyzing sets of data or simulating certain scenarios.

Data

Model parameter values specified by the user (abbreviated with names or coded numerically in the given model) or sets of observations that models attempt to reproduce. If the latter type of data were used in model building, for example, by fitting or Bayesian calibration, then the outcome should not be considered as “true ab initio predictions,” but rather as “posterior predictions” (post‐diction). 22

Open source

This means that major parts of the computer code are available under a license in which the copyright holder grants every user the rights to look at, use, change, and redistribute them for any purpose to anyone. In practice, there are elements that users cannot reasonably change without rewriting an entire computer system; these mainly relate to fundamental mathematical functions, core elements of the user interface and so on. Therefore, the definitions of open source cover a spectrum from completely open to completely closed, not including the latter.

Commercial software

Open source can be replicated and distributed at will; in many cases, commercial exploitation cannot risk pirating and cannot expose the source code. That does not prevent the fundamentals of the code to be made public and does not prevent verification of the code accuracy, for example. However, these might be disclosed only to paying customers or regulatory agencies who are interested in qualification and verifications rather than competitors.

Open science

If parts of a model code are not made public (and considered to be trade secrets), then the model can be considered to be a “Black Box”; however, the fact that users cannot access or change part of the code does not necessarily mean the algorithms are not publicly known. We designate by “Glass Box” the case where the algorithms are published and transparent but the users do not have the freedom of modifying them.

Sponsors and beneficiaries

Sponsors of development of any model platform are the group responsible for creation, maintenance, improvement, and continuous development of the platform and associated databases. Unless these are supported by governments and via public tax, or via crowd funding, sponsors will be private companies or corporations who invest in the development for financial benefit. There are other beneficiaries though who use the platform for the purpose of drug development (which, if successful, brings monitory rewards), or providing consultancy for such activities through commercial services.

Some perspectives, with disclosure of conflicts, submit to the notion of influence that commercial software platforms have had in advancing certain areas of MIDD. ⁸ However, on many other occasions, there is a tendency to dismiss such notions and point out the possibility of adopting other general open source programming languages, by highlighting some perceived advantages as more desirable attributes, such as flexibility and transparency. ⁹ , ¹⁰ , ¹¹ The default assumptions of the latter are based on restrictions to freedom of sharing models made by commercial platforms, or inability for the joint development, exchange, peer‐review, and qualification of models. These claims are rarely substantiated with any evidence and dismiss the peer review by advisors, such commercial platforms use or their links to various academic centres. ⁹ For instance, despite well‐known multi‐party efforts in the development of some of the commercial software, for instance, in the area of modeling drug‐drug interactions, ¹² there are claims of missed opportunity for collaborative work that hampers a sustainable and synergetic advancement of model development and application. ⁹ Lack of sustainable modeling and simulation platforms and tools from many IMI projects, despite tens of millions of Euros spent, is an indication of the lack of return on investment for the public in such diffuse models of development, ⁷ as opposed to cases where a single entity is in charge with a clear indication of reward and penalties (monetary or otherwise).

It has been against such a backdrop that Lippert et al. ¹³ recently put out a call to the modeling community to put more effort into open‐source models in the drug development space. They perceived this to be highly beneficial regarding widespread use as well as allowing the ability to receive input and feedback from users and developers. ¹³ These attributes have all been the essential features of the development for some commercial platforms (via associated consortia) ¹⁴ and yet they were all ignored. Similarly, recent in‐depth analysis of literature data in the area of physiologically‐based pharmacokinetics (PBPKs) reveals much wider spread of usage for so‐called commercial software not only in the industry but also in academia and by regulatory agencies (Figure 1). ¹⁵

FIGURE 1.

Proportional use of various software platforms in the area of physiologically‐based pharmacokinetic over the last 20 years stratified based on the affiliations of the authorship of the report to industry, academia, or regulatory agencies. Simcyp and GastroPlus are commonly considered as commercial/non‐open‐source platforms (see the main text for definition); PK‐Sim was a commercial entity from 2003 until recently when it became freeware with many aspects of the code open. “Other” category includes freeware (e.g., R software) or commercial systems (e.g., Matlab), which are considered open‐source due to access of every user to model code. The data demonstrates the larger proportional use of open‐source systems within academia than industry (over twofold; 28% vs. 11%). However, open‐source platforms are used far much less than so‐called commercial platforms (<1/3) even within academia. See Elkhateeb et al. ¹⁵ for details of the survey

There are several interwoven threads of discussions related to the pros and cons of free‐software, open‐source, or commercial modeling platforms, their transparency, sustainability, and availability for conduct of nonprofit research. ¹⁶ Those issues are often mixed and this creates a perspective in which commercial platform providers are vilified as greedy organizations (without considering that in a free market any entity will cease to exist without providing value for what they offer). Conversely, open‐source solutions are deemed as the only recommended ones for all aspects and needs of the modeling community in the MIDD space ¹³ without analyzing the existing field data, such as those provided by Elkhateeb et al. (Figure 1) on reality of situation. ¹⁵ These often point to elements in the selection of a platform by modelers (commercial vs. noncommercial; open‐source vs. non‐open‐source) that relates to ease of use (i.e., plug and play) and robust pre‐existing model libraries instead of developing model code.

Many have witnessed presentations in various scientific meetings when the issues related to the cost (free vs. commercial) are too easily used as a substitute for the ability to modify the underlying codes (open vs. closed source code), or the transparency of algorithms (black box vs. documented equations and algorithms and code that can be seen under certain privileges but without an ability to modify them [glass box]). It is worth noting that there is not a unique and exclusive answer to the question of open source versus closed/proprietary software, there are pros and cons to both. It may sound a cliché but the answer could be “it depends!” What it depends upon are: (a) the intended purpose and the organization (Regulated Environment vs. Academic Research), and (b) the process of developing the software (single entity vs. consortia of multiple parties [whether gathered under a commercial initiative or societal activity]). This article aims to initiate a critical and systematic analysis of “what is this whole debate about.”

Many of the views that circulate in the public stem from players who may not adequately indicate their personal conflicts (whether inadvertently or intentionally). This commentary does not want to pour fuel over a heated debate about commercial versus free software; nor is it an attempt to extinguish the flare of much needed healthy debate over the transparency of models or their affordability for nonprofit research. However, it tries to make novices to the debate familiar with the angles that are often ignored or, even worse, intentionally suppressed. The authors are both associated with commercial software development while having established track records that demonstrate a firm foothold in, and commitment to, the camp of nonprofit research, via contributions made through academia and research institutions. We hope this helps with avoiding the pitfalls of a narrow and one‐sided perspective and brings to the surface the value that each system offers with documented evidence.

THE DEFINITION OF TERMS USED IN THIS ARTICLE

For the purpose of clarity in the discussions, it is best to define some of the terms in advance. The following definitions are by no means exclusive and various fields might use them differently. We focus on their use in the field of pharmacokinetics, pharmacodynamics, and quantitative systems pharmacology. Hence, in Table 1, are the definitions we have considered for the purpose of this particular commentary.

THE FRAME OF THE DEBATE

Free versus commercial

As argued by Vicini et al. ¹⁷ for the pharmacometrician/modeler software is, quite simply, the equivalent of the laboratory for the molecular biologist with all “necessary equipment,” or of the research clinic for the practicing clinician with all its “diagnostic kits” or “surgical tools.” Although no one expects free provision of equipment and diagnostic tools for all biologists or clinicians, somehow a large group of individuals believe that all modeling tools should be free of any charges. Possibly, part of that reasoning is related to the lack of “physical” existence of models, or to the belief that “back of the envelope” or “cherry on the cake” calculations and modeling should cost nothing. They also ignore the cost associated with the update and maintenance of such free tools. In reality, raw data typically mean nothing without analysis and are not usable without immersion in predictive models. Model building in computational platforms is most valuable but not valued by some as much as it is common in other areas of intellectual property. However, if we take the model as a piece of song with its music, the same attitude does not apply and the piece is not considered as just an alphabetical soup of letters and musical notes, it has intellectual value.

There are also people who do attribute a value to models but wish them to be freely available, not because they are worthless but because paying for them is inconvenient due to the added cost. We should separate these into two groups based on the intentions:

1. Part of these views are altruistic and mainly stem from academic environments who have no personal financial gain in pushing for freeware platforms. These groups are genuinely interested in verification and application of models for the benefit of science community. Some of them even have the necessary resources and capabilities provided by the public purse via government, research councils, or not‐for‐profit foundations to develop and freely distribute small models or contribute to larger further development of open‐source software in their research environment. Although the arguments by such groups are valid, it is strange that the same argument is not applied in many laboratory‐based areas. For example, if a research laboratory cannot afford to own a gene‐sequencer, they cannot enter to the arena of research involving gene sequencing. Similarly, a clinic with no access to magnetic resonance imaging (MRI) machines will not be able to focus on use of imaging for diagnosis of various disease. None of the above has halted progress of research in gene sequencing or use of MRI imaging! One may argue that it has probably slowed its progress and wider adoption. Therefore, when there is a commercial price associated with modeling platforms, it is essential for research funding bodies to consider that cost when allocating funding. Recognition of the needs of the research community might be one of the reasons for commercial platforms to provide reduced license fees or even free systems for nonprofit activities.

2. The motivation of some other players might be questionable even when disguised under the above item (apparent altruism). Underlying intentions in these cases are often hidden from the naïve observers and the conflicts are hardly disclosed. We refer to a subgroup of individuals who push free availability of modeling tools so that they can be used effectively in full‐fledged commercial activities of these individuals and associated organizations. Some of these are under consultation for drug companies and some others are related to direct support of commercial drug development activities. It would be ironic to consider that the platforms should be free for consultancy service providers while they assign a cost for their services! Obviously, these groups should add the cost of necessary platforms (when it applies) to their own intellectual activities, which are as valuable as the time and effort spent on creation of platforms, rather than promoting “open source only” solutions as a means of cost cutting to increase profit.

As stated above, many providers of commercial platforms for modeling have created different structures of pricing that distinguish between nonprofit organizations and commercial entities. In some cases, the academic licenses are completely free with restrictions on commercial consultancies; in other cases, they are associated with lower charges but the platform provides reduced functionalities or databases. Published data on applications support a wider use of the so‐called “commercial” platforms in PBPK (even by academic centers) as opposed to open source platforms. ¹⁵ This goes against the notion that the commercial nature of the platforms has a negative impact on academic research, a notion that simply ignores the various setups in place for nonprofit use of such commercial software.

The opposite case involves many of the open‐source systems (e.g., R software) where some proponents of its use have built lucrative commercial businesses around their usage via consultancy. Contrary to the no‐cost benefit of free software, the consulting fees render an overall cost comparable to, if not more than, verified commercial software!

Regulatory usage is also pointing to the direction of re‐usable robust systems rather than bespoke models, at least in areas, such as PBPK, where consensus on the basic models and input data and verification strategies plays a bigger role than the creation of a bespoke model (see Figure 1).

Indeed, there are numerous examples that demonstrate commercialization is one of the best ways to ensure sustainability, keep up with development, and provide professional and dedicated support and training on any new venture. Other advantages are improved usability via feedback from numerous end‐users who expect/demand, unlike from academic sources, ease‐of‐use and focus on functionality especially for regulatory purposes; dedicated maintenance, which is critical to industry and regulators in relation to upgrades and version control; and customized support tailored to clients’ demands.

When it comes to specialized software in scientific niche areas (PK/PD software etc.), crowdfunding is an idealistic wish rather than a reality that matches evidence. A case in point is the sustainability of software developed in the course of technology and science‐funded projects (like by the European Commission) where the rate of attrition, or frankly failures, is well known to be high. ⁷

What happens in most of these cases where there is not adequate continuous support, is that software‐enabled consulting becomes an alternative solution to a widespread use of the platform. The latter involves intermediary modelers who have been close to the development and, for a fee, do the awkward navigation through the modeling with an unpopular platform (which may not even have appropriate user guides). Many of the reports on open‐source platforms have indicated bugs in the system, which in many cases remain un‐noticed for a long time due to the narrow bandwidth of users. The path above has created its own commercial operations via consultancy firms and contract research organizations and it is not difficult to see why the stakeholders would be in favor of open‐source freeware.

Another point relates to the maintenance of platforms. For all general‐purpose software (text editors etc.), this can be done by professional programmers alone. However, maintaining and improving a scientific platform, particularly when it involves databases, including the latest findings, requires far more competence in the subject matter, which very few programmers will have. The volunteers’ basis for maintaining such software shrinks dramatically, and many of them are in fact one‐ or two‐person operations. It is difficult to imagine a pharmaceutical company, and regulatory agencies, betting on the sustainability of such open‐source software for the support of their multi‐year, multi‐million currency investment projects.

REPRODUCIBILITY AND CONFIDENCE

The issue of reproducibility is not something that is specific to modeling and it is of concern to scientists, as reported by Baker, ¹⁸ where it was reported that more than 70% of studies failed to be reproduced when tried by others. The possibility of replicating a scientific work requires access to all the materials and the details of the methods and assumptions that constituted the research; modeling is not an exception. Therefore, wider accessibility can help with independent tests of repeatability. However, the argument for “open source” should not be equated with wider access, or even with quality. A recent report by Kirouac et al. ¹⁹ examined some published quantitative systems pharmacology (QSP) open source models (R, PK‐Sim/MoBi, and MATLAB) for functionality; only 33% executable reported results could be re‐generated via a “run” script. Part of the issue stems from the fact that such models are hardly checked rigorously by “several” people and they rarely go through a “systemic” quality control process. In fact, their open‐source nature makes them more vulnerable to propagation of mistakes or unintended changes that cannot be readily identified by superficial and nonsystematic testing. This is particularly relevant in the case of multilayer hybrid models. A recent systematic attempt by Tiwari et al. ²⁰ regarding reproducing 455 kinetic models of biological processes published in peer‐reviewed research articles revealed that about half (49%) of the models could not be reproduced using the information provided in the published manuscripts. With further effort, an additional 12% of the models could be reproduced, either by empirical correction or support from authors. The other 37% remained nonreproducible models due to missing parameter values, missing initial concentration, inconsistent model structure, or a combination of these factors. The authors contacted the researchers who had published the nonreproducible models for information but less than 30% responded (everyone can imagine what will happen to a commercial setup with such a rate of response to clients!).

It should be noted that quality assurance for the platform is also different from the quality assurance for the model, and no less important. Whereas the former deals with robustness of general algorithms and the sensitivity of the outcome to given changes, the latter is concerned with performance against set observations (verification sets). Interested readers are referred to Shebley et al. ¹⁴ who provided examples of this distinction. The recent European Medicines Agency (EMA) guidance also communicates expectations regarding verification of each element. ²¹ The ultimate goal is that, given the same set of data, an operator should be able to get the same results as reported in the first place by following the methods used and specifications mentioned for the platform version. Modern platforms typically comprise hundreds of thousands of lines of code, and you cannot expect thorough checking of integrity, except through a well‐controlled quality assurance process.

Readers should also make a distinction between using models as exploratory tools—for instance, in bioinformatics applied to drug discovery and with computational biology models applied to QSP for research—versus models aimed at regulatory submissions in pharmacometrics, PBPK, or some other mature area of QSP, such as cardiac safety. For the latter, availability in a format such that they can be retested and examined within a practical timeframe and by regulators is much more important than flexibility to modify every single model code of the platform. Ability to evaluate a piece of modeling properly requires replication by other researchers using files and data made available to wider community through the original research documentation. The specification of the tools (in this case, the modeling platform, which is quality‐controlled and unadulterated) is something that Vicini et al. ¹⁷ compares with an analytical method to assay chemicals or genes, where the ability to repeat the study is still impinging on access to an liquid chromatography‐mass spectrometry machine or gene sequencer. If the settings of the machine cannot be assured to be the same, we should not expect to get exactly the same results.

To address the issue of qualification for open‐source models and platforms, some have suggested a community approach ¹³ where the models/algorithms/elements of the platform are verified by a selected group of users, who do have access to a different level of the codes than the common user. This two‐tier approach is similar to that under which commercial vendors operate. However, the cost of setting up the user groups and their time in the case of open source is paid by the public purse for all participants who come from research organizations and academia. As mentioned above, some of the beneficiaries then provide nonprofit research that gives back to the society but in many occasions these also subsidize consultancy firms, contract research organizations, and pharmaceutical companies whose activities are benefiting their owners and shareholder rather than the public.

SUSTAINING RESEARCH AND DEVELOPMENT IN MODELING

Research and development (R&D) activities are costly and never really done for free. Some aspects of R&D are always paid from the public purse (government or charities) and some by commercial organizations with a view to gain exclusive rights to exploit commercially what they have invested in. Recent analysis by Elkahteeb et al. ¹⁵ in the area of PBPK modeling has provided new insight to the fact that, at least in this particular area, the most commonly used platforms are commercial rather than open source. This might be related to the fact that the application of modeling and simulation is starting to differentiate itself from building new algorithms and new models. The former may embrace a larger group than the narrow community who is engaged with building models and algorithms. Hence, whereas the first group prefer well‐characterized and tested models with predefined libraries, the latter group insist on the ability to revise and manipulate every single element of platforms and models. An analogy can be drawn between the above and automobiles: typical drivers of ordinary cars use them for daily tasks, such as going from point A to B, whereas specialized Formula 1 drivers using bespoke individually made cars test the latest technologies. The reliability and robustness of ordinary cars for the first group requires elements related to quality and endurance that are very different from those applicable to any Formula 1 car supported by a large team of engineers and specialized drivers.

It should be stated that none of the platforms are completely binary when it comes to their ability to accommodate changes that users wish to introduce (similar to the ability of various automobiles even outside the bespoke Formula 1 family of cars). Even the most open‐source systems do not allow the user to redefine the fundamental algorithms used by their language; on the other hand, all the pre‐set closed source platforms have some level of flexibility to enable selection of models from a battery of options, or introduce tailored equations by the user. Perhaps that is why the issues of open‐source code often gets confused with those of commercial activities as the latter are easier to define.

PERSPECTIVE

The debate on open‐source software and platforms has been muddied by many factors. The conflict on the part of those who have been associated with commercial platforms (such as that by the current authors) should always be transparent and in the open. On the other side, some individuals are promoting open‐source systems without declaring their vested interest (and hence conflict) associated with their paid services using the very same open‐source tools. Public funding of such tools in the latter case subsidizes this group more than the academic establishments who do have access to subsidized commercial platforms. The assumption that a one‐size‐fits‐all open‐source modality will satisfy all the needs of the modeling community ignores the fact that, within the regulatory environment, quality assurance dictates certain settings and does not allow “everyone” to have access to “every” piece of the code (as it makes the assessment impossible). Moreover, many have assumed that shifting from open‐source strategies to commercial settings necessarily leads to reduced access for academic research, ignoring that commercial platforms generally offer other options for operation in nonprofit settings.

As modeling platforms get more complex and contain predefined sets of data and qualified submodels, it is hard to imagine how the community or organizations that put such effort into R&D could thrive and develop further without an investment that comes either from the public purse or commercial founders. The issue is whether, unlike other areas of R&D, all such activities are to be invested in open source, even though many of the users are full‐fledged commercial operations, or whether we can meet on fair ground where anyone benefiting commercially from modeling tools also pays toward the development and maintenance of such tools.

Currently, there are many nonprofit consortia building community‐wide platforms as well as commercial organizations investing in the expansion of modeling and simulation tools and growing their wider applications. The debate on open‐source platforms and their place in R&D versus routine (quality controlled) usage has just started. The current authors wish for better platforms, better accessibility, better funding for developing models of all platforms, and wider applications. Having experience with each side of the camp (academia and commercial tool provider) and working with many pharmaceutical companies as well as consultancy firms providing services in modeling and simulation, they find it unrealistic to assume a one‐size‐fits‐all solution will be agreed on or would be beneficial to all. The debate on pros and cons of each end of the spectrum should continue in an open environment with a view to providing a fair representation of what each system offers, rather than vilifying the other side of the debate or hiding facts that do not match up to reality.

DISCLAIMER

The views expressed in this perspective are the authors’ personal opinions and do not reflect the outlook from affiliated organizations.

CONFLICT OF INTEREST

A.R.‐H. is a part‐time employee of Certara, a company providing commercial software for Pharmacometrics and Systems Pharmacology space, and holds shares in the company. F.Y.B. is currently a full‐time employee of Certara and a maintainer of GNU MCSim, an open‐source software.

AUTHOR CONTRIBUTIONS

Both authors participated equally to the writing of this article.