The Need for a Glycemia Modeling Comparison Workshop to Facilitate Development of an Artificial Pancreas

At the Glycemia Modeling Workshop meetings in November 2008 in Bethesda, Maryland, and in November 2009 in San Francisco, California, world experts in glycemia modeling concluded that existing simulation models should be compared and contrasted. The consensus was that the same evaluation and validation criteria should be used to better assess the state of the art, understand any inherent limitations in the models, and identify gaps in data and/or model capability. This type of Glycemia Modeling Comparison Workshop (GMCW) would exhibit elements of a contest as well as an open-ended discussion.

A GMCW would be very useful to compare and contrast various models for glycemia if they can be applied to predict future glucose levels from a common set of scenarios. These models are used to determine the dosing and timing of insulin delivery needed for a closed-loop system. If a model is going to useful, then it must conform to reality. At this point in time, various models have been proposed by investigators. These models may incorporate the effects of numerous quantifiable factors, including insulin pharmacokinetics, timing of meal carbohydrate appearance, meal size, amount of exercise, presence of stress, day-to-day variations in insulin sensitivity, insulin time–activity profiles, accuracy of glucose monitor calibration, metabolic profiles of both adults and neonates, and risks of hypoglycemia/hyperglycemia.¹ Nevertheless, it is unknown which mode is most realistic. This is because each model has different goals, inputs, and outputs.

At a GMCW, each participant would be an investigator using their own unique model. Every participant's model would be presented by its developer and then discussed by another investigator. The purpose of this workshop would be to enable workshop participants to learn about where differences in modeling approaches can lead to better or worse predictions of glycemia. This information would accelerate the development of a closed-loop algorithm or set of situational algorithms to control insulin delivery in a closed-loop artificial pancreas to produce physiologic glucose levels. The assumption is that models that can accurately predict glycemic levels in a contest setting are well suited for predicting how various other perturbations in lifestyle can be overcome with adjustments in insulin delivery to maintain target glucose levels in diabetes or trauma-induced acute hyperglycemia.²

As more models are being developed with focused capabilities, there is increasing interest in understanding the performance and limitations of these tools. Potential users of these models are interested in understanding the patterns of glycemia to develop an artificial pancreas, as well as to develop tools for delivering intensive insulin therapy in the hospital. Potential users of glycemia models for developing an artificial pancreas are interested in understanding where these models do and do not accurately predict glycemic levels in response to perturbations of the blood glucose level. Gaps in model performance need to be identified so that problems can be avoided or fixed if an artificial pancreas with autonomous control is ever to become a reality. A contest will stimulate each developer of a glycemia model to improve their model in anticipation of a public discussion of the performance of their model.

Lessons learned from analyzing and modeling time series glucose data have proven to be useful for other types of physiological analyses with military implications. Sleep history, duration of wakefulness, work patterns, rest periods, and circadian phases have been analyzed to predict performance degradation and sequential body temperature measurements have been analyzed to predict debilitating hyperthermia.

For this project, first a GMCW planning committee would be convened. This group would consist of leaders in the field of glycemia algorithm development. They would define a metric for assessing the performance of a glycemia algorithm that compares a time series of glucose levels with an empirically collected series of glucose data. Any principal investigator (PI) who would want to present their algorithm and participate in the actual GMCW would not be a candidate for this committee. A list of approximately 16 international PIs who are the most active in developing glycemia models would be assembled by the GMCW planning committee of the meeting and be invited to participate in the GMCW. A survey would be developed by the planning committee and would be distributed to these PIs, who would be participating in this event. The survey would cover capabilities, inputs, and outputs of their biomathematical models. Features of the models would be collected, summarized, analyzed, and juxtaposed by GMCW planning committee members prior to the workshop event to provide a framework for understanding the features of the models.

This contest would also be known as a bakeoff because, like the annual Pillsbury bakeoff, each participant would use the same ingredients to create the best final product.³ Four glycemia scenarios would be provided to each PI or PI's team prior to the workshop. These four data sets would each consist of a string of continuously monitored glucose levels collected from deidentified patients with type 1 diabetes. Information about calibration values, meals, and exercise from the data sets would be included. These scenarios would challenge the models of the PIs to predict glycemia levels for a variety of physiologic states, including eating, exercise, recovery from hypoglycemia, and sleep.

During the 2-day GMCW, participating PIs would each present their model and discuss the inputs, outputs, goals, and capabilities of their model. Each model would also be discussed by another modeler to provide perspective. Results of predictions based on common scenarios would be discussed. A scenario consisting of a truncated skein of continuous glucose values would be presented to each participant at the end of the first day. Predicted glucose levels based on this fifth scenario would then be calculated overnight to test the capability of each algorithm to generate output in near real time. The workshop would conclude with a summary of the features and capabilities of the world's leading glycemia models. This type of comparison of model predictions with actual empiric data would enable an examination of similarities and differences between models, as well as performance gaps in various situations.

This process of comparing model predictions with actual empiric data has never before been employed with continuous glucose data. A similar contest, the Fatigue and Performance Modeling Workshop, was convened successfully in Seattle, Washington, on June 13–14, 2002.⁴ That meeting focused on time series data related to sleep. At that event, seven models of various types of physiologic data related to sleep and performance degradation were analyzed and compared. The workshop was sponsored jointly by the National Aeronautics and Space Administration, U.S. Department of Defense, U.S. Army Medical Research and Materiel Command, Office of Naval Research, Air Force Office of Scientific Research, and U.S. Department of Transportation.

For that meeting, an invitation was sent to developers of seven biomathematical models that were cited commonly in scientific literature and/or supported by government funding. Developers who accepted this invitation to attend the workshop were asked to complete a survey about features of their biomathematical models of alertness and performance. Data from completed surveys were summarized to provide a framework for comparing features of the seven models. In that workshop, the survey and oral presentations revealed that sleep models varied greatly relative to their inputs, outputs, goals, and capabilities.⁵

The GMCW workshop deliverables would include a set of original articles suitable for publication, one per participant, about the performance of their model. A summary article would also be written to summarize the contest results, as well as describe the key features of all the biomathematical models of glycemia. The GMCW would be a valuable milestone in advancing the field of glycemic modeling for an artificial pancreas.

Abbreviations

GMCW

Glycemia Modeling Comparison Workshop

PI

principal investigator