Moving Towards Model Reproducibility and Reuseability

Abstract

This commentary provides a brief history of the U.S. funding initiatives associated with promoting multiscale modeling of the physiome since 2003. An effort led in the United States is the Interagency Modeling and Analysis Group (IMAG) Multiscale Modeling Consortium (MSM). Though IMAG and the MSM have generated much interest in developing MSM models of the physiome, challenges associated with model and data sharing in biomedical, biological and behavioral systems still exist. Since 2013, the IEEE EMBS Technical Committee on Computational Biology and the Physiome (CBaP TC) has supported discussions on promoting model reproducibility through publication. This Special Issue on Model Sharing and Reproducibility is a realization of the CBaP TC discussions. Though open questions remain on how we can further facilitate model reproducibility, accessibility and reuse by the worldwide community for different biomedical domain applications, this special issue provides a unique demonstration of both the challenges and opportunities for publishing reproducible computational models.

I. Introduction

Over the last decade the number and types of computational models being developed for biomedical research has experienced a healthy increase. The biomedical community is beginning to recognize not only the usefulness of models, but the essential role models play to integrate disparate fields of knowledge, identify gaps and present testable hypothesis to drive experiments, particularly to understand complex system across scales. Modeling of the temporal and spatial scales of the physiome [1] and Multiscale Modeling (MSM) to couple scales and physiological domains, therefore, are at the forefront of making a significant impact in biomedical discoveries, applied science and medicine.

II. Funding Initiatives

Since 2003, the United States, Europe and Japan have promoted several government funding initiatives for modeling the physiome [2]. In the U.S., a confluence of events resulted in the formation of the Interagency Modeling and Analysis Group (IMAG) in 2003. IMAG is now comprised of over 80 program directors managing programs for modeling and analysis and biomedical, biological and behavioral systems across more than ten government agencies. The formation of IMAG resulted in the subsequent release of the first interagency solicitation in 2004 to promote the development of MSM methodologies for biomedical, biological, and behavioral systems [3]. That solicitation funded 24 projects, creating the MSM Consortium in 2006. The IMAG-MSM Consortium, now in its 10th year, has over 100 MSM related projects. During this time, several other related MSM initiatives have emerged from the government agencies of the IMAG. In particular, IMAG released a three-year MSM initiative focused on clinical applications in 2007 [4], which funded 20 projects. In 2011, IMAG released a second three-year initiative focused on MSM methodologies [5] to address the challenges identified in the IMAG Futures Report [6]. This initiative funded 30 projects, and was reissued in 2015 [7] to additionally promote the development of credible multiscale models of biomedical, biological and behavioral systems.

III. Model Reproducility

One of the main activities of IMAG is to coordinate the MSM Consortium. Its mission is to grow the field of MSM in biomedical, biological, and behavioral systems, by 1) promoting multidisciplinary scientific collaboration among MSM modelers; 2) training future generations of MSM modelers; 3) developing accurate methods and algorithms to cross multiple spatiotemporal scales; 4) promoting model reproducibility, and reusability; and 5) disseminating models and insights arrived from the models to the larger research community. Since the inception of the MSM Consortium, model sharing discussions centered around developing standards and interoperable models. The MSM Consortium soon realized that infrastructures can be built to facilitate sharing, but motivation was needed for the modeling community to use these resources. The MSM Consortium recognized a strong need to change the culture of journal publications to encourage reproducible models through publication requirements.

In 2013, the IEEE EMBS Technical Committee for Computational Biology and the Physiome (CBaP TC) took on the goal of serving as an interface between government (IMAG), academia (MSM Consortium) and publications (journals with a focus on modeling). Editors-in-Chiefs of over a dozen journals for bioengineering, systems biology and biomechanics were recruited to participate on this TC. Through CBaP TC efforts, consensus was achieved to uniformly promote model reproducibility through publications. In 2015, the Biophysics Journal was one of the first journals to update their guidelines to include specific instructions for reproducible models [8]. This special issue takes the next step in exploring the process for submission and review of reproducible models for publication, from a few different fields of modeling - biomechanics, systems biology and neurophysiology. Earlier CBaP TC discussions revealed differences in the modeling cultures of different fields, therefore, this special issue tackles the differences head on. In this special issue, current state of the art efforts to prime models for reproducibility and reuseability are presented. In another category of papers, the authors were asked to share their models for the reviewers to access, simulate and reproduce the simulation results, specifically some figures in each article. The guest editors, Ahmet Erdemir and Herbert Sauro, then worked with the reviewers and authors to iterate and comment on the entire review process.

Though open questions remain on how we can further facilitate model reproducibility, accessibility and reuse by the worldwide community for different biomedical domain applications, this special issue provides a unique demonstration of both the challenges and opportunities for publishing reproducible computational models.

IV. Conclusion

For over a decade, U.S. government funding initiatives have laid the groundwork for supporting modeling and analysis of biomedical, biological and behavioral systems. In particular, the IMAG Multiscale Modeling (MSM) Consortium has worked to develop useable and reuseable computation models and infrastructure tools for model sharing. The IEEE EMBS CBaP TC discussions inspired this Special Issue on Model Sharing and Reproducibility- to show exemplars for model reproducibility, and how the publication process can indeed promote model reproducibility, accessibility and reuse by the worldwide biomedical engineering community.