The Accuracy of a New Real-Time Continuous Glucose Monitoring Algorithm: An Analysis

Abstract

In this issue of Journal of Diabetes Science and Technology, Keenan and colleagues used archival data from the STAR 1 clinical trial (Medtronic Diabetes) to support the claim that the new Veo™ calibration algorithm improves the accuracy of continuous glucose monitoring, particularly in the critical hypoglycemic range. Extensive data analyses are presented to support this claim; the results are convincing, and the estimated improvement in hypoglycemic detection from 55% for the standard calibration to 82% for the Veo is particularly impressive. We can therefore conclude that the Veo algorithm has the potential to improve the accuracy of hypoglycemia alarms and ultimately contribute to closed-loop control. However, the presented results should be interpreted cautiously because they are based on retrospective analysis and are heavily dependent on the distribution of blood glucose levels observed in a particular data set.

Studies have documented the benefits of continuous glucose monitoring (CGM)^1–3 and charted guidelines for CGM clinical use^4,5 and its future as a precursor to closed-loop control.⁶ However, while CGM has the potential to revolutionize the control of diabetes, it also generates data streams that are both voluminous and complex. The utilization of such data requires an understanding of the physical, biochemical, and mathematical principles and properties involved in this technology. It is important to know that CGM devices measure glucose concentration in a different compartment: the interstitium. Interstitial glucose (IG) fluctuations are related to blood glucose (BG) presumably via diffusion process.^7,8 To account for the gradient between BG and IG, CGM devices are calibrated with capillary glucose, which brings the typically lower IG concentration to BG levels. Successful calibration would adjust the amplitude of IG fluctuations with respect to BG. Calibration quality, however, is influenced by the possible time lag due to BG-to-IG transport and the sensor lag time (instrument delay, primarily due to data smoothing). Because such a time lag could greatly influence the accuracy of CGM, a number of studies were dedicated to its investigation, yielding various results.^9–12 In most studies, IG lagged behind BG (most of the time) by 4–10 min, regardless of the direction of BG change; however, negative time lag was reported as well. The formulation of the push–pull phenomenon offered reconciliation of these results and provided arguments for a more complex BG–IG relationship than a simple constant or directional time lag.^11,12 In addition, errors from loss of sensitivity and random noise confound CGM data.¹³ Thus, while the accuracy of CGM is increasing, it is still below the accuracy of direct BG measurement.^14–18 Thus calibration algorithms capable of reducing CGM error are important additions to the arsenal of data processing techniques that help transfer raw current into BG readings. Algorithms reducing the time lag are critically important as well.

In an article entitled “The Accuracy of a New Real-Time Continuous Glucose Monitoring Algorithm” in this issue of Journal of Diabetes Science and Technology, Keenan and colleagues¹⁹ evaluate the accuracy of a new calibration algorithm used in the Paradigm® Veo™ insulin pump (Medtronic Diabetes) and compare the results to the standard Paradigm REAL-Time (PRT) calibration method. Archival data from the STAR 1 clinical trial comparing insulin pump therapy augmented with real-time CGM versus standard self-monitoring of blood glucose (SMBG) were used to support the claim that the Veo algorithm improves CGM accuracy, particularly in the critical hypoglycemic range. In order to ensure a fair comparison, the authors go back to raw current and perform both the Veo and the standard PRT calibrations using the same SMBG data points. The data set is reasonably large, containing data for 72 subjects in the active CGM study arm for a total of 90,472 CGM–SMBG data pairs.

None of the calibration algorithms is described in the article by Keenan and colleagues, and it is therefore unclear whether the Veo algorithm has any analytical advantages over the standard PRT calibration. From Figure 3, we can speculate that the Veo algorithm uses a calibration function with a steeper slope at low BG levels than the standard calibration; for example, using exponential versus linear type of calibration would have that effect. In the absence of a formula, however, we can only rely on the data to compare the performance of one “black box” versus another. This being the case, the performance evaluation becomes heavily dependent on the distribution of the data used for comparison. One example would clarify this statement:

Overall, the Veo algorithm had a mean absolute relative difference (MARD) of 15.89% versus 16.14% for the standard PRT calibration, i.e., Veo outperformed PRT by 0.25%. This overall accuracy was achieved by data that had distribution presented in Table 1 (line 1).¹⁹

Table 1.

Dependence of Algorithm Accuracy on the Distribution of the Self-Monitoring of Blood Glucose Data

BG region (mg/dl)	40–80	81–120	121–240	241–400	Veo MARD	PRT MARD
Line 1: observed SMBG data distribution	11.8%	20.4%	50.4%	17.4%	15.89%	16.14%
					Difference (PRT–Veo) +0.25%
Line 2: hypothetical SMBG data distribution	5.0%	11.0%	50.0%	34.0%	15.33%	15.08%
					Difference (PRT–Veo) −0.25%

If we assume the hypothetical distribution presented in Table 1 (line 2), which is weighted toward hyperglycemia, then the MARD results become exactly the opposite: the standard PRT algorithm outperforms Veo by 0.25%. Thus, without analytical knowledge of the calibration formulas, no statements about overall accuracy could be made, because the overall accuracy can be easily biased by the distribution of the data in hand. The Clinical and Laboratory Standards Institute (CLSI) Guidelines for evaluating the accuracy of CGMs emphasize the importance of the distribution of the test data and suggest desirable distribution characteristics.²⁰This is, of course, not the case with the algorithm accuracy within each of the BG regions in Table 1: the performance of the Veo algorithm in the hypoglycemic region appears better than the performance of the standard PRT calibration. Statistical significance data are not presented; thus it remains unclear whether this result is not due to chance. A small penalty in accuracy is paid at the high BG range, which is to be expected, because calibration typically uses parameterized curves prone to certain inertia, i.e., if one end of the curve is overfitted, the other would naturally become suboptimal.

Extensive analyses aim to confirm the higher accuracy of the Veo algorithm in the hypoglycemic range. Particularly impressive is the increase in hypoglycemia sensitivity from 54.9% in the PRT to 82.3% in the Veo. Consensus and Clarke error grid analyses are used to additionally highlight the clinical importance of the observed accuracy differences. Unfortunately, because the reference SMBG readings are not frequent enough and not equally spaced in time, these data do not allow the use of the continuous glucose error grid analysis (CGEGA)¹⁷ or any other analysis of trend accuracy. Perhaps future studies could remedy this deficiency and would take full advantage of the CGEGA, which is designed on the premise that CGM accuracy should be evaluated differently in different BG regions. An expected result from the CGEGA would be a clear superiority of the Veo algorithm in the CGEGA hypoglycemic table. An additional benefit would be the better correspondence between the results and the CLSI guidelines,²⁰ which recommend the use of trend accuracy analyses, including CGEGA.

Finally, an intriguing element of the paper is the mentioning of predictive alarms based on a Savitzky– Golay filter based on a moving polynomial fit, which theoretically should be more responsive to rapid time series changes and should result in shorter instrument delays than the commonly used moving average. The combination of this type of polynomial filtering and the improved hypoglycemia accuracy of the Veo algorithm would then result in more accurate hypoglycemia alarms. Indeed, it was estimated that predictive alerts with a 30 min predictive horizon would detect over 90% of hypoglycemic events.

In conclusion, the new Veo calibration algorithm appears superior to the standard PRT calibration, particularly in the critical hypoglycemic range. Clinically, this is an important achievement that has the potential to advance the quality of hypoglycemia alarms and, ultimately, of closed-loop control. This result, however, is based on a purely retrospective analysis and should be interpreted cautiously until confirmed by simulation studies testing the capabilities of the Veo algorithm in silico or by prospective data collection.

Abbreviations

BG

blood glucose

CGEGA

continuous glucose error grid analysis

CGM

continuous glucose monitoring

CLSI

Clinical and Laboratory Standards Institute

IG

interstitial glucose

MARD

mean absolute relative difference

PRT

Paradigm REAL-Time

SMBG

self-monitoring of blood glucose