Evidence-based Modeling of Critical Illness: An Initial Consensus from the Society for Complexity in Acute Illness

Abstract

Introduction:

Given the complexity of biological systems, understanding their dynamic behaviors, such as the Acute Inflammatory Response (AIR), requires a formal synthetic process. Dynamic Mathematical Modeling (DMM) represents a suite of methods intended for inclusion within the required synthetic framework. DMM, however, is a relatively novel approach in the practice of biomedical research. The Society for Complexity in Acute Illness (SCAI) was formed in 2004 from the leading research groups utilizing DMM in the study of acute inflammation. This Society believes that it is important to offer guidelines for the design, development and utilization of DMM in the setting of AIR research to avoid the “garbage-in garbage-out” problem. Accordingly, SCAI identified a need for and carried out a critical appraisal of DMM as currently used in the setting of acute illness.

Methods:

The SCAI annual meeting in 2005, the 4th International Conference on Complexity in Acute Illness (ICCAI; Cologne, Germany), was structured with the intent of developing a consensus statement on the methods and execution of DMM in AIR research. The conference was organized to include a series of interactive breakout sessions that included thought leaders from both the DMM and acute illness fields, the results of which were then presented in summary form to the entire group for discussion and consensus. The information in this manuscript represents the concatenation of those presentations.

Results:

The output from the 4th ICCAI involved consensus statements for the following topics: 1) the need for DMM, 2) a suggested approach for the process of establishing a modeling project, 3) the type of “wet” lab experiments and data needed to establish a modeling project, 4) general quality measures for data to be input to a modeling project, and 5) a descriptive list of several types of DMM to provide guidance in selection of a method for a project.

Conclusion:

We believe that the complexity of biological systems requires that DMM needs to be among the methods used to improve understanding and make progress with attempts to characterize and manipulate the AIR. We believe that this consensus statement will help guide the integration, rational implementation, and standardization of DMM into general biomedical research.

INTRODUCTION: MOTIVATION AND GOALS

The acute inflammatory response (AIR) is generally recognized as a complexsystem, both in structure and behavior. Understanding and potentially manipulating the AIR requires an extension beyond the traditional scientific paradigm of analysis via sequential reductionist experimentation. Accomplishing this task requires a formal, explicit means of synthesis, heretofore an intuitive process carried out in the mind of the researcher. Dynamic mathematical modeling (DMM) has been used extensively by researchers dealing with such complex dynamic systems as studied in physics,¹ economics,² ecology,³ cognition,⁴ linguistics,⁵ chemistry,⁶ and social science,⁷ but only recently in biological and biomedical contexts.^8-13 The Society for Complexity in Acute Illness (SCAI) was established in 2004 to provide an organizational structure and a forum to facilitate the integration of DMM into general biomedical research, recognizing the surge of interests in the translational applications of DMM (e.g. the NIH Roadmap).¹⁴ The annual SCAI meeting in Cologne, Germany, 2005 (the 4^th International Conference on Complexity in Acute Illness [ICCAI]), was structured with the goal of developing a road map for the implementation of DMM as an adjunct to traditional research. Thought leaders from the critical care, DMM, and emerging intersection of DMM in critical illness were involved in hands-on fashion in these workshops. The primary areas of emphasis included:

• Identifying the purpose of a particular modeling project (i.e. what is the question being asked that DMM can help answer?).

• Delineating the relationship between the biomedical researcher and the modeler. Inherent to this was identifying the areas of commonality/divergence with respect to the terminology used and expectations of the modeling process.

• Use of the current evidence-based paradigm to provide objective parameters as to the “quality” of the reference data input in the creation of a model. This was intended to address the classic “garbage in/garbage out” dilemma associated with computational analysis.

• Listing and providing a systematic categorization of the more commonly used types of DMM, focusing on the type of data needed to develop the model (input), the type of results expected from the model (output), and the pros and cons of each technique.

This manuscript is the result of these discussions. We include recommendations as to the type and limitations of data included in the development of and generated from DMM, and suggest the need for standardization of terminology. We note that these recommendations are flexible and should evolve with further discussion.

ADDRESSING THE CHALLENGES OF MODELING: A SUGGESTED APPROACH

The following consensus points were reached by the SCAI workshop panels, which were focused on the interactions within teams that involved clinician/biologists (“Bioscientist”) and scientists involved in modeling/simulation (“Modeler):

• Definition of a Model

• Need for iterative process between Bioscientist and Modeler

  • Identification of the question to be answered by the model (Bioscientist)
  • Identification of appropriate modeling platforms for problem (Modeler)
  • Identification of data input for model construction (Bioscientist and Modeler)
  • Identification of data output for model calibration (Bioscientist and Modeler)
  • Determination of data quality measure for both model input and output (Bioscientist)

Below, we discuss these consensus points in greater detail.

We start with a formal characterization of the term “model.” We chose to focus on two primary characteristics of the term (also see Glossary at <http://biosystems.ucsf.edu/Researc/dictionary.html>).

• Models are expressions of hypotheses. Models are statements that describe the relationship of one set of data to another, usually involving some inferred causal mechanism that can be delineated either mathematically or via simulation. It is important to note that all model construction (with one exception, that of data mining, which will be expanded upon below) is an a priori process. The consequences of the initial modeling assumptions (the modeling ontology) are necessarily embedded in any subsequent analysis of the model and its output. Explicit models (i.e. not just mental models) are instantiations of hypotheses, and therefore the results from their execution may provide a “check” on the validity of a particular hypothesis.

• All models are abstractions. A model with the same degree of detail as its reference system provides no advantage with respect to facilitating understanding of the original system. Understanding only arrives with some level of reduction of detail that is determined by the purpose of the model. The degree to which a model should be abstracted is extremely difficult to determine beforehand. Often the development of a series of models is needed.

The process of building a model is iterative, involving the Biologist and the Modeler as described below. The Biologist must, at the outset, frame and clarify the question as precisely as possible, with particular focus on the appropriate scale at which the question or hypothesis is best framed (i.e. sub-cellular, cellular, tissue, organism, population). The scale may be based on the proposed level of a potential intervention or experimental perturbation. We suggest that at the initiation of a modeling project the focus should be on answering specified questions and directed at particular, concrete goals. We note that DMM can be particularly helpful in situations of controversy (i.e. where experimental evidence exists in favor of two or more alternative interpretations of a given phenomenon).¹⁵

Next, the Modeler needs to advise the Biologist as to the most appropriate modeling platform/technique for addressing the question. Active discussion is required between the Modeler and the Biologist with respect to the type and frequency of measurement of data needed to build and validate the model. The Modeler needs to relate the data that are technically necessary to develop, calibrate, and validate the potential model. The Biologist should identify the appropriate “wet lab” and/or clinical models from which the data will be generated, and must account for the pragmatic logistical limitations of each.

The Biologist must be able to assign some “quality measure” to the data. In this way, the Modeler weights the contribution of particular data to the modeling ontology. One possible approach is to use the data classification system (I-III) currently employed in the evidence-based assessment of clinical practice guidelines¹⁶ or the modified Delphi grading system used in the Surviving Sepsis Campaign guidelines for the treatment of sepsis/septic shock.¹⁷ At this point, the model is designed, constructed, and verified. Simulation results are then adjusted to match existing data, and then validated, ideally against a prospectively generated data set. The failures of the model are identified, and the whole process repeats in order to increase the fidelity of the model . The following sections contain more detailed discussion on the topics of data and the formats needed for achieving useful models, determination of quality measure for the data input for models, and a critical appraisal of the various types of modeling approaches.

DATA AND FORMAT NEEDED FOR MODELING: EXPECTATIONS FROM MODELERS AND BIOLOGISTS

Consensus Points:

• Identification of “Fundamental Laws of Biology”

• Acknowledge hierarchical structure of biology and its relationship to modular DMM design

• Identify hierarchical/organizational levels and the following issues related to modeling at each level:

  • Central characteristics of the specific organizational level
  • Types of analytes to be measured
  • Frequency of measuring analytes

The Biologist should determine the focus of the model, and thus identify the key and relevant data for examination. Data Mining is an exception. In Data Mining, essentially all analyzable data are collected and subjected to search algorithms that identify patterns and relationships within in the data. These patterns are then examined and hypotheses about mechanistic relationship (if any) are developed. Data Mining can be used in conjunction with DMM, for example to sift through simulated data generated in population simulations representing clinical trials.^{18, 19}

One means of focusing development of a specific modeling ontology is to identify “fundamental laws of biology” along with key operating principles. These include, but are not limited to the following:

• Biology is subject to the laws of chemistry and physics, and thus subject to fixed constraints on the behavior of biochemical systems. These realities determine when certain types of data can be collected (e.g. analytes derived from the activity of a particular enzyme) and how many data points may be needed at each sampling time.

• Biology has a multi-hierarchical organizational structure. The non-linearities among these hierarchies generate the “complexity” that makes biology so challenging. However, the behavior and data required to analyze one particular hierarchy are generally well established, and can form reference points in the development of DMMs intended to bridge two or more hierarchies. For the modeling process, it is important to specify subsystems and to identify those factors that can represent key features. Input and output of each subsystem must be clearly identified, including the dimension and units of the variables.

Below, we discuss central issues, analytes of relevance, and measurement frequencies relevant to particular scales of modeling.

Modeling at the Sub-cellular and Cellular Levels

Consensus Points:

• Research should initially focus on a single cell type and a specific molecular intervention

• Input and output should be specifically delineated and, where appropriate, model output should map to accepted biochemical flow diagrams

• Wet-lab time course data should be collected at intervals over 2-3 half-lives of the process being studied and contain 5-10 observations, dependent upon the kinetics being modeled

Both inflammation and associated organ damage/dysfunction have been studied at the molecular and cellular levels,^{20, 21} and DMM carried out at this level has yielded many insights.^22-27 Modeling research should ideally focus on a single cell type, and the wet lab referent should be a specifically defined molecular intervention or series thereof (Janes et al²⁵). DMM studies of this type have yielded insights into several processes, for example suggesting that the regulation of NF-κB involves oscillations of IκB expression,²³ that TNF is relatively insensitive to TNF concentrations,²⁶ that signaling for apoptosis can be reduced to two principal components,²⁵ and that bistability in apoptosis is derived from cooperativity in apoptosome formation.²⁷ The input/output levels of precursor and products should be explicitly defined based on biochemical flow diagrams. Time course data should be collected over 2-3 half-lives of the process under examination, and contain 5-10 observations, dependent upon the kinetics of the analyte.

Modeling at the Tissue/Organ Level

Consensus Points:

• Construct and refine a DMM based on a fixed intervention or perturbation

• Define objective determination of organ function

• Develop finely grained measures of organ function

• Attempt continuous data collection

• Prefer data collection methods that are non-destructive

Given the central role of organ damage/dysfunction in acute illness, ²⁸ modeling at the tissue and organ level has played a central role, especially examining the issue of physiologic variability.^{8, 9, 29} However, in the clinical setting, there is often a great deal of heterogeneity with respect the insult and limitations with respect to the objective measures of organ function and response, and inter-individual variation.³⁰ A strategy for minimizing this impact is to construct the DMM based on a fixed intervention or perturbation, to develop and use finely grained measures of organ function, and to sub-stratify the patient population as finely as possible. There needs to be a clear, objective means of determining organ function linked to some quantifiable mediator or chemical produced by the organ, or calculated from chemical measures of function. Examples include the oxygenation index for pulmonary function, cardiac output for the heart, transaminase levels for the liver, and GFR measures for the kidney. For physiologic data, continuous measurements are preferred; for chemical markers, the time course must consider the half-lives of the markers.

Modeling at the Organism Level

Consensus Points:

• Architectural and organizational issues of model construction are of the utmost importance

• Construct influence diagrams to explicitly represent relationships –

• Avoid embedding forward and negative feedback loops

• Strive for continuous physiologic data collection

• Initial model construction can rely on heuristics and qualitative rules, with refinements to follow

A central goal of DMM in acute illness is to enable one to predict the overall health of an experimental animal undergoing some form of shock based on easily-measurable parameters, with the eventual application to patient care;^{9, 10, 12} this concept might be termed “translational systems biology”, and this type of DMM has been successful in yielding basic insights into acute inflammation,^{18, 31-33} quantitative insights into the biology underlying experimental paradigms of acute inflammation in animals,^34-36 streamlining animal usage in such experimental paradigms,³⁷ and carrying out simulated clinical trials in sepsis and trauma.^{19, 37, 38} Influence diagrams should be created with careful choice of inclusion and exclusion criteria. Identification of positive- and negative-feedback loops is important. The choice of appropriate scale is important to not build pre-conceived/a priori nonlinear assumptions into the modeling ontology. An example of this would be creating an organism-level model and linking the organism's response to a sub-cellular intervention based on an intuitively assumed effect (i.e. not justified by known mechanisms or data) on the entire organism. Data collection should ideally be as continuous as possible in order to capture important bifurcations (e.g. compensated vs. non-compensated shock, primary vs. secondary organ failure, life vs. death).³⁹ Parameter values should be reconciled with literature observations.

Modeling at the Population Level

Consensus Points:

• Acknowledge difficulties associated with population heterogeneity

• Fine-grained intra-population and inter-individual data are desirable to facilitate the sub-stratification of populations

• When applying equilibrium dynamics to populations, data should be collected until equilibrium is reached

The threats of emerging infectious diseases, bioterror agents, and drug-resistant pathogens are among the circumstances providing impetus for extensive work utilizing DMM in these settings.^40-51 To facilitate sub-stratification and reduce sample heterogeneity, intra-population and inter-individual data are desirable. Black-box models (see below) should not include data transfer functions based on unsubstantiated assumptions (i.e. incorporating the planned hypothesis into the modeling ontology, thereby creating an a priori loop). At a population level assuming equilibrium dynamics may be appropriate. When doing so, data should be collected until the dynamics of the process being studied have reached equilibrium. Thereafter, data may be collected every 24 hours.

EVIDENCE-BASED MODELING: DETERMINATION OF DATA QUALITY

Consensus Points:

• Specify objective criteria for the evaluation of data quality

• Formalize the process of information integration into DMM construction

  • Conceptual models should be explicit and transparent
  • State the research questions being addressed
  • Identify the benefit of a modeling project
  • List modeling assumptions and ontologies
  • Develop flow and influence diagrams of the process to be modeled
  • Identify input and output data
  • Consider inclusion of database specialist in the modeling team
  • Reify data and metadata

• Utilize communication facilitating tools (such as modeling glossary)

• Facilitate transition between the basis of model construction and its real-world utility

• SCAI supports the creation of multiple models in multiple platforms at multiple organizational hierarchies and their placement in a freely-accessible repository.

We have arrived at a consensus summary of the DMM process, though we recognize that these issues are controversial and changeable. Such controversy can be minimized by adhering to principles of evidence-based medicine and causality analyses.⁵² Doing so includes evaluations of the data with respect to the experimental paradigm and methodology used, as well as context data/metadata. To foster transdisciplinary communication we strongly suggest the use of modeling glossaries (cited above) to reduce ambiguity and facilitate discussion (An et al, submitted).

SCAI supports the creation of a knowledge repository to aid research team members in modeling critical illness at all levels. This repository should be established to enable its use by the broadest set of end-users. SCAI also supports the use of multiple models and implementations in addressing relevant clinical or physiologic questions.

MATHEMATICAL MODELING METHODS: DESCRIPTION AND CRITICAL APPRAISAL

Consensus Point:

• Categorization of various methods of DMM being used to model the AIR (see Table 1)

Table 1.

Summary of modeling DMM methods discussed at the 2005 ICCAI.

Modeling framework	Principle	Input	Output	Advantages	Limitations	References
Ordinary Differential Equations	To model changes and interaction of different factors over time	Selected factors, system of differential equations (law knowledge), starting conditions (parameters)	Trajectory of the system (time courses of various processes)	Simulation of a multifactor system over time; insights to systems behavior; identification of bifurcations; can predict beyond training data; can encompass spatial and stochastic components	Validity of equations based on extent of knowledge; calibration required; robustness; applicable for populations of sufficiently large size	18, 32-36, 38, 55
Partial Differential Equations	To model changes and interactions where spatial organization is important (Lymph node). Have close connection to stochastic models by describing the evolution of distribution functions.	Selected factors, system of differential equations.	Trajectory of system including information about spatial arrangements.	insights to systems behavior; generalize Ordinary differential equations, by adding space.	calibration required;	Gardiner; Handbook of Stochastic Methods, Springer.
Neural Networks	Connection of input information by automatic weighting, in order to generate a prediction	Input and outcome information from real observation	Prediction of outcome	Ability to adapt/learn with new data; no expert knowledge necessary	Black-box approach, over- fitting possible	56, 57
Logistic Regression Models	Selection and combination of factors associated with a binary outcome (event); based on observations	multiple observations	Prediction of outcome	Qualitative and quantitative; does not require knowledge of underlying process	Not dynamic model; no account for mechanisms; existing sample of sufficient size required; cannot predict outside of training data	58, 59
Agent- based Models	Independent agents acting and interacting based on rules	Definition of agents, starting conditions, environment	Results on a higher level: aggregate behavior of the agents: whole system over time	Effect of micro-rule on macro- behavior; intuitive	Black-box effect, calibration difficult; computationally intensive	19, 31
Stochastic Models	To model changes and interaction of different factors over time including noisy behavior	Selected factors, system of stochastic differential equations. (law knowledge), starting conditions (parameters)	Trajectory of individual ‘objects’ and of the system by averaging or by solving appropriate partial differential equations (time courses of various processes)	Simulation of a multifactor system over time; insights to systems behavior; fluctuations in the system; applicable to small and moderate size of populations	calibration required;	Gardiner; Handbook of Stochastic Methods, Springer.

Table 1 summarizes the various methods, includes a brief text description of the method, the type of input required for its construction, the type of model output one can expect, the potential advantages of using a method, and method limitations. This list is not intended to be comprehensive in either scope or depth; interested parties are encouraged to examine the published proceedings from the 4^th ICCAI in the Journal of Critical Care 2005, 20(4).

CONCLUSION

“GIGO = Garbage in, Garbage out”

- Cartoon from the 1960s

The application of DMM has long histories in the physical sciences, yet has been slow to diffuse into general biomedical research. However, the awareness that “systems-based” diseases such as cancer, autoimmune disorders, AIDS, and sepsis must be attacked using computational approaches has been at the forefront of both the U.S. National Institutes of Health¹⁴ and Food and Drug Administrationt⁵³.

The behavior of the AIR, and its disordered state of sepsis/MOF, has attracted researchers interested in the application of DMM to biomedical systems. We have made significant strides in the relevant applications of DMM to both basic and clinical research (An et al, submitted). We also realize that biomedical application of DMM is at a critical point of its development. As with all computational approaches, DMM is susceptible to the age-old bane of computer science: “GIGO,” or “garbage in, garbage out.” Rigorous, professional best practices and review of modeling assumptions and ontologies are critical in avoiding GIGO.⁵⁴ We at SCAI have recognized this issue and have striven to avoid and forestall situations in which an inappropriate application or less-than-rigorous execution of DMM could stain the perceived validity of DMM at a time when it is coming under greater scrutiny. These considerations have motivated us to offer this manuscript: a series of consensus statements from researchers familiar with the application of DMM to the AIR. The intent is to provide a reference and guidelines for those interested in applying DMM or in evaluating the quality of a DMM application to biomedical research related to the AIR. As with all consensus statements, this one is incomplete, and the concepts it addresses are evolving. Thus, SCAI will focus on the continual refinement of these guidelines, utilizing the annual meeting to update these statements, and provide feedback to the community. We intend this manuscript to serve as a resource for those interested in initiating DMM research programs, and to assist in evaluating the efficacy of DMM as an analytical and synthetic tool.

ABBREVIATIONS

AIR

acute inflammatory response

DMM

dynamic mathematical modeling

ICCAI

International Conference on Complexity in Acute Illness

SCAI

Society for Complexity in Acute Illness