Workflows and Provenance: Toward Information Science Solutions for the Natural Sciences

Abstract

The era of big data and ubiquitous computation has brought with it concerns about ensuring reproducibility in this new research environment. It is easy to assume computational methods self-document by their very nature of being exact, deterministic processes. However, similar to laboratory experiments, ensuring reproducibility in the computational realm requires the documentation of both the protocols used (workflows) as well as a detailed description of the computational environment: algorithms, implementations, software environments as well as the data ingested and execution logs of the computation. These two aspects of computational reproducibility (workflows and execution details) are discussed in the context of biomolecular Nuclear Magnetic Resonance spectroscopy (bioNMR) as well as the PRIMAD model for computational reproducibility.

Introduction

The era of big data is upon us. Along with it, computers and computation have become ubiquitous in almost every human endeavor. It should come as no surprise that concerns have been raised about the reproducibility of computational methods in research and science (Stodden et al., 2016). Reproducibility is a cornerstone of the scientific method – addressing both the universality of the reported scientific claims and providing transparency such that the scientific results can be trusted. In general terms, a process can be reproduced if both what was done and how it was done are sufficiently documented. It is often beneficial to record who conducted the process as well as when it was done, but a truly reproducible process is independent of either of them. What then are the requirements for sufficient documentation and how do those requirements translate to computation?

Method sections in the natural sciences typically have two components: the protocol used and a detailed description of the reagents, equipment and calibrations. It is easy to assume computational methods self-document by their very nature of being exact, deterministic processes, whose outcomes are dictated by “the program”. However, leaving aside non-deterministic processes for the moment, by burying the computation within a software tool, it can be very difficult to reproduce the exact process without a detailed record of the software tools used, their configuration and their execution. This problem is amplified with each software tool added to the process stream. As emphasized by Stodden et al. (2016, page 1240), “We thus focus on the ability to rerun the same computational steps on the same data the original authors used as a minimum dissemination standard, which includes workflow information that explains what raw data and intermediate results are input to which computations.”

Mirroring the situation with laboratory experiments, to ensure reproducibility in the computational realm requires the documentation of both the protocols used (workflows) as well as a precise description of the computational environment: algorithms, implementations, software environments as well as the data ingested and execution logs of the computation. These two aspects of computational reproducibility (workflows and execution details) are discussed in the context of biomolecular Nuclear Magnetic Resonance spectroscopy (bioNMR) as well as the PRIMAD model for computational reproducibility (Rauber et al., 2016).

Workflows and Provenance

A workflow is a model for complex processes in which the process is decomposed into discrete, sequential operations. Many formalized procedures fit this definition, culinary recipes for instance, as well as administrative workflows such as those exhibited by “the culture of the checklist” in the field of archival science (Bak, 2016). The rationales for representing processes as workflows can be as varied as the processes themselves. The goal of recording a cooking recipe may be to assist in the organization and timing of the interdependent tasks, as well as to increase the reproducibility of the end product by applying uniform measurements of ingredients, cooking duration, and temperature. The goal of a TRAC checklist is to improve and document the quality of an archive, increasing its trustworthiness to the community to which it serves.

Scientific workflows are useful for a similarly diverse array of purposes. However, scientific workflows have an important distinguishing characteristic from business workflows. Typical business workflows focus on sub-processes and their validity within the context of an organizational infrastructure. (For instance, documenting proper oversight and approval for a requisition request). As pointed out by Bowers and Ludäscher (2005), scientific computational workflows are distinguished from administrative workflows in their focus on data and dataflow (Ludäscher et al., 2009). The emphasis on data has two important aspects: one, managing the timing of a workflow (one step in a workflow may not be able to begin until a preceding step has completed), and two, managing the semantic data types used within the workflow (a process which requires oranges as input should not receive apples).

Apart from this simplistic description of the design and operation of scientific workflow systems, there are two significantly different perspectives from which one can consider the workflow itself. At a detailed or instance level, and in retrospective, it can be viewed as an execution log, capturing the exact sequence of events which occurred and their relationships with respect to each other (both in order and in data typing). Yet at a more abstract level, the workflow is disjoint from any particular execution event.

There are many such flavors of such an abstract workflow. It might still be detailed as in the former case but neglecting specifics of the precise execution, as in, computational step A of type α followed by computational step B of type β, without recording timestamps or execution details of the individual computations. In a more abstract case, it might be a broad sketch of general processing chunks: ingestion, cleaning, transformation, visualization, and result reporting, with very little detail on the underlying computation at all. Finally, the workflow may represent the idea for a future process or protocol which has not yet been executed – or it may even be the case that applications for conducting the individual steps in the putative workflow do not yet exist. At this abstract level, the workflow is more similar to a cooking recipe and less like a stack trace.

An important distinction between the former and latter workflow types is not just the level of abstraction, but this consequence that an abstract workflow is capable of describing events which have not yet occurred. This is the inherent distinction between retrospective provenance (a representation of prior workflow execution) and prospective provenance (a description of how to execute a future workflow) (Lim et al., 2010). It can also be thought of as the distinction between what was done and what is intended to be done.

Despite many similarities, these two different workflow or provenance “worlds” are vastly different, both in their conceptualization of a workflow as well as with the underlying tools and approaches for managing workflows. Workflow management systems designed to operate at the execution level concentrate on the details of tool operation and interoperability. Systems such as Kepler (Altintas et al., 2004) or HTCondor (Thain et al., 2005) must ensure that data of the correct type is being shuttled between individual actors (Kepler) or jobs (HTCondor). These tools may also manage the invocation and resource allocation of the individual jobs and check for completion and/or any errors.

Another approach to capture retrospective provenance is that of noWorkflow (Murta, et al., 2014). In this approach, the provenance is recorded from the execution of a standard processing script (e.g. Python), avoiding the learning curve and overhead of a specialized workflow system such as HTCondor. Prior to execution, noWorkflow parses the script and maps the dependencies between code blocks. Upon execution, noWorkflow relies on built-in Python utilities to extract the provenance and map the dataflow.

These former methods all have as common an interest in the actual execution – describing an event which occurred. The notion of “workflow thinking” is to pattern a process as a workflow regardless of the manner of its execution or whether it has been executed at all. Workflow thinking is more about conceptualizing processes as recipes and protocols, structured as dataflow graphs with computational steps, and subsequently developing tools and approaches for formalizing, analyzing and communicating these process descriptions. An important example of one approach is the YesWorkflow annotation and query system (McPhillips et al., 2015). YesWorkflow provides a few simple syntactical annotations which can be embedded within code – or within a stand-alone file. These annotations describe the flow of data through the various processes such that many of the aspects of a workflow can be visualized, queried and understood, in the absence of any execution events.

Reproducibility in biomolecular NMR spectroscopy: NMRbox

The National Center for NMR Data Processing and Analysis is a recent initiative to help foster reproducibility in the field of biomolecular NMR spectroscopy. The Center has three overlapping research directions. First, the Center is provisioning virtual machines (VMs) with most of the common software tools used by bio-NMR (Maciejewski, et al. 2017). Provisioning VMs with the software helps ensure that both the software and underlying computing environment will persist into the future. Second, the Center is modeling and capturing the metadata required to replicate the computational workflow of a bio-NMR study. Third, the Center is providing Bayesian inference modules for consistent analysis of bio-NMR data. The research developments and directions can be examined within the context of the PRIMAD model for computational reproducibility.

PRIMAD is an acronym for six key variables of a computational system which must be controlled for reproducibility. Platform (P) refers to the entire computational environment of the underlying software tool. This contains the computer hardware as well as the operating system and any ancillary software components such as shared libraries. Research objective (R) refers to the scientific goal of the research – what hypothesis is being tested or what claim is being supported or refuted. Implementation (I) refers to the actual software code by which a particular Method (M) is being invoked. Method refers to the computational approach taken, e.g., for ordering a list or pruning outliers. Actors (A) refers to the human agents who conduct the experiment. Data (D) refers to the datasets under analysis during the computational study. The report from the working group outline this model in the context of a few examples of computation (bubble sorts and statistical analysis). The research endeavors of the Center for NMR Data Processing and Analysis will be examined in the context of this model for computational reproducibility.

Platform (P)

As discussed by Rauber et al., computational results which are independent on platform are considered to be portable as well as reproducible. The field of bioNMR relies on dozens of software tools, most of which were developed in academic labs, and rely on antiquated operating systems, compilers and code libraries. A consequence of this is that most bioNMR studies are not portable. To address this issue, the Center for NMR Data Processing and Analysis is (a) provisioning VM’s with all available bioNMR software, (b) maintaining a cloud-based Platform as a Service model of accessing these VM’s, and (c) is in the process of establishing an archive of the various versions of the NMRbox VMs.

Research Objective (R)

Following the PRIMAD model, for a process to be considered reproducible, the research objective of the replicate process must be the same. This can be the most complicated barrier to reproducibility, for instance if two research groups do not agree on the overall purpose of the research, or if a subtle difference in objective is not fully explained. While this is difficult to address computationally, this is being addressed by the administrative structure of the Center. The research developments are driven by a so-called “push-pull” relationship with external investigators conducting research on “Driving Biological Projects”. By focusing technological developments on established external research projects, these external DBPs will assist ensuring that the research objects are agreed upon by the various biomedical communities.

Methods / Algorithms (M)

NMRbox aims to include two hundred software packages used by the bioNMR community. There is a great deal of overlap in the functionality of this software smorgasbord. For instance, there are perhaps a dozen software tools capable of spectral reconstruction – the process of converting time domain data to frequency domain. While there is a great deal of overlap between the various packages, there are methods and algorithms which are unique to a given tool: for instance, the maximum entropy reconstruction algorithm within the Rowland NMR toolkit. Maintaining all of the various software packages within one common VM aids in evaluating reproducibility, as the platform dependence inherent to any computational tool is eliminated.

Implementation / Source Code (I)

In some sense, it can be difficult to draw a decisive line between methods, algorithms, implementation and source code. For the purposes of this case study, we will assume that methods / algorithms are in an abstract sense (as in the Dagstuhl report by Rauber et al. of “bubble sort” vs “quick sort”) while implementation and source code contains the possibility of performance tweaks and or bugs / side effects. As such, the implementation would contain the various versions of an implementation. This is also being addressed by NMRbox in that the various VM versions will also maintain a registry of the various software tool versions contained within each. Therefore, questions of reproducibility regarding a particular implementation of a method can be explored within the NMRbox VMs.

Actors (A)

In the context of the PRIMAD model, actors refer to human agents. Computational agents are considered to be a combination of methods and implementation (in the overall context of a platform). The role of actors in reproducibility is addressed in part by capturing annotations of human agents when performing manual analysis. An example of such an annotation strategy is that of the reproducibility extensions to the program Sparky (Fenwick et al., 2015b). In this example, the Sparky program was augmented with a few routines which (a) assist in version control of the assignment process using GIT, and (b) provide a helpful conceptual model for NMR peak assignment to assist in providing meaningful snapshots along the assignment process. Ongoing research of the Center is to continue to expand the set of captured metadata along a bioNMR study to further foster reproducibility.

Data (D)

The final variable for computational reproducibility is the data sets used in the study. In the context of NMRbox, this is being addressed through the partnership with the BioMagResBank (BMRB) (Ulrich, et al., 2008) hosted by the University of Wisconsin, Madison. The BMRB has been the national repository for bioNMR data for the past several decades. One additional goal of the Center is to assist in research reproducibility by tracking additional data/metadata within the virtual machine and providing additional software tools to assist the researcher in reporting this data/metadata to the BMRB. Thus, the BMRB will have richer data depositions ensuring that all of the data required for reproducing a study are made available to the community at large.

Workflows and Provenance within NMRbox

Along with provisioning the standard bio-NMR software, NMRbox will also include utilities and resources to manage workflows and provenance. A workflow management system for bioNMR spectral reconstruction has already been developed (Fenwick et al., 2015a). Termed CONNJUR Workflow Builder (CWB), the tool allows the NMR spectroscopist to craft a spectral reconstruction process as a workflow utilizing any of three software tools: NMRPipe, the Rowland NMR Toolkit, and CONNJUR Spectrum Translator utilities. CWB stores the spectral metadata along with the reconstructions (workflow executions) within a MySQL database. Other workflow management systems such as HT Condor and YesWorkflow are also supported within NMRbox.

Conclusions

“Workflow thinking” can be a beneficial way of conceptualizing a computational process. By documenting the computational process as a workflow the computation is more transparent and more easily reproduced. When combined with retrospective provenance information, additional value can be derived from a workflow (Pimentel et al. 2016). The PRIMAD model describes additional variables which can be controlled to investigate the universality of the computational process. The new Center for NMR Data Processing and Analysis, while predating the Dagstuhl working group, provides a good case study for how these variable can be documented and controlled in the laboratories of natural scientists.

Biographies

Dr. Michael R. Gryk is currently Associate Professor of Molecular Biology and Biophysics at UCONN Health, the medical school at the University of Connecticut. He is also a doctoral student of Library and Information Science at the School of Information Sciences, University of Illinois at Urbana – Champaign. Dr. Gryk has worked in the field of structural biology, concentrating on bioNMR, since 1990. His recent research interests are in the computational and informational science aspects of biomedical research. He received his M.S. in chemistry from the University of Connecticut and his Ph.D. in biophysics from Stanford University, Stanford, California.

Dr. Bertram Ludäscher is a professor at the School of Information Sciences at the University of Illinois, Urbana-Champaign. He directs the Center for Informatics Research in Science and Scholarship (CIRSS) and is a faculty affiliate with both the National Center for Supercomputing Applications (NCSA) and the Department of Computer Science. From 2004 to 2014 he was a tenured professor at the Department of Computer Science at the University of California, Davis. His research interests range from scientific data and workflow management, to knowledge representation and reasoning. Until 2004 Ludäscher was a research scientist at the San Diego Supercomputer Center (SDSC) and an adjunct faculty at the CSE Department at UCSD. He received his M.S. (Dipl.-Inform.) in computer science from the University of Karlsruhe (K.I.T.) and his PhD from the University of Freiburg, Germany.