Minimizing the Impact of Time Lag Variability on Accuracy Evaluation of Continuous Glucose Monitoring Systems

Abstract

Background

Despite all commercially available continuous glucose monitoring (CGM) systems being designed to operate in the extracellular interstitial fluid, and even though there is a well-recognized time lag between the interstitial and the venous compartments, the accuracy of the CGM device readings is still evaluated against the glucose concentration in venous blood (VB) samples, thus resulting in a perceived decrease in accuracy. This article explains how different time lag compensation methods (no compensation, compensation with a fixed delay, compensation with a variable delay based on an intercompartmental diffusional model) have an impact on how CGM accuracy is evaluated.

Methods

The data set used consisted of 210 CGM/blood glucose data pairs from 18 diabetes subjects (15 type 1 and 3 type 2) selected from a data base collected during two independent clinical trials. All CGM measurements were performed using the GlucoMen ®Day CGM system (A. Menarini Diagnostics, Italy), and the reference VB glucose measurements by means of a standard laboratory instrument. For each applied time lag compensation method, the CGM accuracy evaluation was performed as recommended by the POCT05-A consensus guideline.

Results

The perceived accuracy of the CGM device significantly improved when applying both the fixed or the variable delay compensation method. However, it is worth noting how the variable delay method, which relies on a closer description of the intercompartmental diffusion processes, provided the best perception of the clinical accuracy of the device.

Conclusions

When assessing the accuracy of a CGM system, a crucial step in data analysis is to account for time lag, which enables minimization of the apparent decline in system accuracy.

Introduction

Continuous glucose monitoring (CGM) has proven to be a valuable tool for the management of diabetes in insulin-treated patients, and is expected to represent the natural midterm evolution of self-monitoring of blood glucose (BG).

Currently, all commercially available CGM systems are designed for measuring glucose concentration in the extra-cellular interstitial fluid (ISF) space, which is safer and more easily accessible than vascular space. Interestingly, CGM systems are normally calibrated using capillary BG concentrations (measured by means of conventional BG meters) as reference values. Similarly, the accuracy of theCGM system readings is evaluated against the concentra-tion of glucose in capillary or venous blood (VB) samples, despite the well-recognized existence of a time lag between the interstitial and the venous compartments.

Notably, different studies have reported conflicting average time lag values, which may range from 0 to 40 min.^1,2 A summary of the estimated average time lag values reported by various authors, evaluated by means of different methods, is shown in Table 1.

Table 1.

Summary of Interstitial Fluid Glucose versus Blood Glucose Average Time Lag Values Published since 1986

First author	Reference	Year of publication	Estimated time lag (min)
Shichiri	3	1986	5
Matthews	4	1988	0
Jansson	5	1988	2 ÷ 8
Bolinder	6	1989	10
Pickup	1	1989	0 ÷ 40
Aalders	7	1991	4 ÷ 8
Meyerhoff	8	1992	0 ÷ 18
Tamada	9	1995	20
Sternberg	10	1996	2 ÷ 12
Bantle	11	1997	10
Roe	12	1998	8 ÷ 10
Petersen	13	1999	15 ÷ 20
Tamada	14	1999	18 ± 10
Smith	15	1999	2 ÷ 4
Rebrin	16	2000	5 ÷ 12
Gross	17	2000	10
Stout	18	2001	10 ÷ 20
Feldman	19	2003	5
Boyne	20	2003	4 ÷ 10
Kulcu	21	2003	5
Weinstein	22	2007	13
Groenendaal	23	2008	1 ÷ 3
Kamath	24	2009	6 ± 1
Bailey	25	2009	8
Garg	26	2009	5 ± 3; 10 ± 3
Kovatchev	27	2009	13
Valgimigli	28	2010	11
McGarraugh	29	2011	9.6

Indeed, time lag depends on a number of factors that include the effect of insulin and other drugs that may be administered to the patient.³⁰ Interstitial fluid glucose (IFG) is highly correlated to VB glucose only when the glucose concentration in the body is relatively stable. In case of rapid changes, the time lag between the two compartments significantly reduces this correlation,³¹ thus resulting in an apparent decrease in the accuracy of CGM readings.

Kovatchev and coauthors²⁷ introduced the Poincarè plot method for retrospectively assessing the average time lag between ISF and VB compartments, given a set of CGM/BG data pairs.²⁷ This method entails the progressive sliding in time of the CGM data versus the corresponding BG reference values. The delay that provides the maximum statistical agreement between CGM and BG data is considered as the average time lag. The calculated average time lag value can then be used to rigidly shift in time the whole CGM profile with respect to reference data points prior to calculating the accuracy parameters of the CGM device.

The Poincarè plot method has undoubtedly improved the way the accuracy performance of the CGM system is evaluated because it takes into account the physiological difference between the two compartments in which the CGM device and reference method, respectively, quantify the concentration of glucose. Nonetheless, it must also be considered that the application of a fixed delay is a strong simplification of the actual phenomena occurring at the vascular/ISF interface.

Herein, we present an adaptive method based on the Rebrin and Steil’s¹⁶ two-compartment model. Based on an improved description of the intercompartmental equilibrium, this method allows us to take the variability of the time lag into account. Functioning and benefits of using this adaptive time lag compensation method for highlighting the “true” accuracy performance of the CGM devices are discussed here.

Methods

Data Base

The set of CGM/BG data pairs used in the present study was selected from a data base collected during two independent clinical trials: the first study (protocol number GMD_03) was performed on 20 type 1 diabetes subjects at the Center for Clinical Research, Medical University of Graz, Austria, and was concluded in December 2010; the second study (protocol number GMDCP06) was performed on 10 subjects (8 type 2 and 2 type 1 diabetes) at the Santa Maria della Stella Hospital in Orvieto, Italy, and was concluded in February 2011.

In both clinical trials, continuous glucose measurements were performed for 100 h periods with the GlucoMen®Day (GMD) CGM system (A. Menarini Diagnostics, Italy), while the corresponding venous BG reference measure-ments were obtained by means of a standard laboratory instrument (COBAS analyzer, Roche Diagnostics, France; method, hexokinase).

The CGM/BG data pairs employed in the present study were selected from those in the data base that were collected during meal tolerance tests and thus during rapid glycemic excursions. The final data set included 210 CGM/BG data pairs from 18 different subjects (15 type 1 and 3 type 2 diabetes).

The Gluco Men Day Continuous Glucose Monitoring System

The GMD is a microdialysis-based wearable device that is intended for 100 h of CGM in patients with diabetes.^32,33 Both the clinical accuracy of the device (which operates in the ISF compartment) and its resistance to enzymatic and electrochemical interferents have been previously assessed and the corresponding results published.^28,34

Data Analysis

The evaluation of CGM accuracy was performed as recommended by the POCT05-A consensus guideline.³⁵ For each time lag compensation method, clinical accuracy was quantified through Kovatchev’s continuous glucose error grid analysis.³⁶ Furthermore, the mean and medianvalues of the absolute relative error (MARE and MedARE), mean and median of the absolute error (MAE and MedAE), mean and median absolute rate deviation (MARD and MedARD), as well as the Pearson’s correlation coefficient (R²) were calculated.

Methods for Compensating the Time Lag

Even though all CGM systems quantify the glucose concentration in the IF, the state-of-the-art guideline for evaluating their clinical accuracy suggests to use either the capillary or the VB glucose concentration as the reference data point. From the analytical point of view, this approach is obviously incorrect and somehow unfair to CGM systems; however, this choice was guided by two important reasons. First, CGM is still perceived as a new measuring technique, and therefore, there is the need to compare it with other well-established methods.³⁷ Second, extracting sufficient amounts of ISF to be analyzed with a reference method is feasible but extremely impractical. Because the CGM system is calibrated versus reference BG concentrations, it is therefore very important and recommended to take into account the time lag between IFG and BG prior to proceeding with the evaluation of the accuracy.³⁵ For our set of CGM/BG data pairs, accuracy parameters were calculated by applying two different time lag compensation methods (method A and method B); for comparison, we also show the results from the omission of all methods of compensation.

Noncompensated Data (Interstitial Fluid versus Blood Approach)

The accuracy of the CGM system is evaluated by directly comparing given continuous glucose data (recorded at a certain time “t”) with the corresponding BG reference data (also recorded at time “t”). Hence, glucose concentration values measured in two different body compartments are directly compared. This implies that the differences existing between ISF and BG concentrations that originate from the kinetics of the diffusional equilibria will be improperly regarded as “measurement errors.”¹⁶ This type of analysis clearly shows how the IFG/BG time lag affects the assessment of CGM device accuracy.

Method A: Application of a Fixed Delay (Blood versus Blood Approach)

In an approach that considers a form of time lag, the CGM data series (which clearly reflect the concentration of glucose in the IF) is “converted” in silico into the corresponding “blood-like CGM data series.” As the first step of this method, the average time lag for the CGM/BG data pairs is evaluated using Kovatchev’s Poincarè plot method.²⁷ The whole CGM profile is then rigidly shifted in time by the calculated time lag. The accuracy is finally evaluated by comparing the shifted CGM data series with the corresponding BG reference values.

Application of a constant delay δ to compensate for the time lag implies consideration that the glucose concentrationin the ISF at given time (t corresponds to that previously present in the blood at the time (t-δ(Figure 1A). Even though this approach for taking the time lag into account represents a substantial approximation of the intercompartmental physiology, it combines a satisfactory efficacy with a remarkable simplicity of application. Furthermore, this is likely to be the only time lag compen-sation approach potentially applicable in real time.

Figure 1.

Qualitative representation of the time relation between CGM and BG data when considering (A) a fixed time lag and (B) a variable time lag (diffusional model)

The average time lag for the GMD system has been previously estimated to be 11 min.²⁸

Method B: Application of a Variable Delay (Glucose Peak Broadening Model, Interstitial Fluid versus Interstitial Fluid Approach)

The method that takes into account and compensates for time lag variability is based on the so-called glucose peak broadening (GPB) model. According to this model, the BG reference data set is used to calculate in silico the corresponding IFG values, which then become the new set of reference data. The accuracy of the CGM device is thus calculated by comparison with such IFG reference data (hereafter simply referred to as IFG), rather than the original BG values.

The GPB mathematical model considers the concentration of glucose in the interstitium at a certain time t, IFG(t), and the corresponding concentration of glucose in the VB, BG(t), as two time-dependent variables of a dynamic system, where IFG(t) is the dependent variable and BG(t) is the independent one.

Since the BG(t) data series consists of consecutive but discrete glucose measurements spaced by 10–15 min (corresponding to the reference BG values measurement frequency), the system is assumed to evolve through discrete states. IFG(t) is also considered as composed of two subvariables [Equation (1)]: a state variable, S(t), which describes the initial state of the main variable and thus corresponds to the value assumed by the main variable IFG at the preceding time (t– 1 [i.e., S(t) = IFG((t– 1)], and a transition variable, T(t), which describes how the value of the main variable changes between consecutive states accounting for the physiological phenomena that relate IFG(t) to BG(t):

$$IFG(t)=IFG(t-1)+T(t)$$ (1)

In other words, T(t) represents the change in glucose concentration that makes IFG evolve from the state “(t– 1” to the subsequent state “t.” The dynamic relation between IFG(t) and BG(t) was described using the two-compartment model proposed by the Rebrin and Steil.¹⁶ This model takes into account both the diffusion of glucose across the vascular/ISF interface and the rate of glucose clearance from the ISF compartment.

The mass balance equation of the two-compartment model reported in Figure 2 can also be rearranged as follows:³⁸

Figure 2.

Rebrin and Steil’s¹⁶ two-compartment model and corresponding mass balance equation

$$\frac{diFG(t)}{dt}=k_{1}BG(t)-k_{2}IFG(t)$$ (2)

In order to introduce the discrete states approximation, Equation (2), which considers IFG(t) and BG(t) as continuous variables, can be further modified by replacing the time derivative of IFG with the corresponding first backward difference quotient:

$$\frac{IFG(t)-IFG(t-1)}{\Delta t}=k_{1}BG(t-1)-k_{2}IFG(t-1)$$ (3)

where Δt indicates the time elapsed between states t and (t– 1. Equation (3) can then be used for expressing the transition variable T(t):

$$T(t)=[k_{1}BG(t-1)-k_{2}IFG(t-1)]*\Delta t$$ (4)

In order to minimize error that results from introducing the discrete states approximation (error that may be particularly significant when the rate of change for the BG variable is high), a further term, which accounts for the BG rate of change from the states (t- 1 to t, needs to be introduced in the expression of T(t), resulting in

$$T(t)=[k_{1}BG(t-1)-k_{2}IFG(t-1)]*\Delta t+k3[BG(t-1)]*\Delta t$$ (5)

The parameters reported in Equation (5), where k¹ = k² = 0.120 and k³ = 0.027, were identified by means of a linear least squares analysis of an independent subset of raw CGM sensor current data versus the corresponding BG reference data.

In summary, the GPB-model allows in silico calculation of a set of IFG data starting from the set of measured BG values; this new reference data set is then used for evaluating the accuracy of the CGM device.

Results and Discussions

Case study 1 (Figure 3) clearly demonstrates how the use of different time lag compensation methods affects evaluations of accuracy. Figure 3 shows a selected interval of CGM and BG data acquired during a meal tolerance test. In similar conditions (i.e., when the glycemic excursions are particularly rapid), the kinetics of the intercompartmental diffusion processes makes the time lag extremely variable, and this, in turn, negatively impacts on the bias existing between continuous and reference data pairs.

Figure 3.

Case study 1: meal tolerance test followed by using the GMD CGM system (blue dots). The green dots represent the measured reference data (VB glucose concentrations), either reported (A) according to their original timing or (B) after a rigid time shift by 11 min. The IFG values calculated in silico by means of the GPB model are shown as red dots

Despite an evident time lag, the CGM signal shown by Figure 3A is undoubtedly coherent in its trend with the reference BG data. However, without compensation for time lag, the relative bias can be as high as 94%. In Figure 3B, the characteristic average time lag estimated for the GMD system (11 min) was used to rigidly shift the set of reference BG data against the CGM profile (method A). What is clear from Figure 3B is that application of a fixed delay certainly mitigates the problem but does not provide a complete solution for it. Indeed, while the time lag on the left-hand side of the peak is substantially eliminated, relative bias is still non-negligible (up to 33%) while glycemia is decreasing (right-hand side of the peak).

Such an asymmetric result can essentially be attributed to the fact that the CGM profile (which reflects a glycemic excursion occurring within the ISF compartment) is intrinsically broader than the glucose peak obtained by interpolating the corresponding BG reference values, as anticipated by the two-compartment model.

According to the GPB model, the IFG values estimated in silico from the measured BG reference data intrinsically take into account the broadening of the glucose peak induced by the intercompartmental diffusion and result, therefore, in being significantly more correlated with the CGM signal.

Interestingly, when the glucose fluctuations are smoother (Figure 4A) and time lag is not likely to have as big an impact as in case study 1, the results obtained by applying the fixed delay method are as good as those obtained by applying the GPB model (Figure 4B).

Figure 4.

Case study 2: meal tolerance test followed by using the GMD CGM system (blue dots). The green dots represent the measured reference data (VB glucose concentrations), either reported (A) according to their original timing or (B) after a rigid time shift by 11 min. The IFG values calculated in silico by means of the GPB model are shown as red dots.

Tables 2 and 3 provide a summary of the accuracy evaluation results for the set of CGM/reference data pairs that did not have time lag compensation and for the same pairs after compensating with either the fixed delay method (method A) or the GPB model (method B).

Table 2.

Accuracy Evaluation Parameters as a Function of Different Time Lag Compensation Strategies

Parameter (measurement unit)	Time lag compensation method
	Uncompensated data	Method A: application of a fixed delay	Method B: application of a variable delay
MARE (%)	9.8	7.3	6.4
MedARE (%)	7.5	5.2	3.7
MAE (mg/dl)	14.6	10.7	9.0
MedAE (mg/dl)	11.6	8.6	6.4
MARD (mg/dl/min)	0.98	0.90	0.66
MedARD (mg/dl/min)	0.75	0.66	0.43
Pearson’s correlation coefficient (R2)	0.904	0.942	0.956

Table 3.

Continuous Glucose Error Grid Analysis (CG-EGA) Results as Obtained by Considering Different Time Lag Compensation Methods

Glycemic range	CG-EGA summary output	Time Lag compensation method
		Uncompensated data	Method A: application of a fixed delay	Method B: application of a variable delay
Hypoglycemia (<70 mg/dl; 1.4% of the data)	Accurate readings	100%	100%	100%
	Benign errors	0.0%	0.0%	0.0%
	Significant errors	0.0%	0.0%	0.0%
Euglycemia (70–180 mg/dl; 61.1% of the data)	Accurate readings	94.5%	97.2%	98.3%
	Benign errors	5.5%	2.8%	1.7%
	Significant errors	0.0%	0.0%	0.0%
Hyperglycemia(>180 mg/dl; 37.5% of the data)	Accurate readings	80.8%	87.2%	95.8%
	Benign errors	17.9%	11.5%	2.8%
	Significant errors	1.3%	1.3%	1.4%

As clearly shown by the accuracy evaluation results, the accuracy of the CGM device was perceived to improve significantly when applying either one or the other form of time lag compensation. However, it is worth noting how the method based on the GPB model, relying on a closer description of the intercompartmental diffusion processes, provided the best perception of the clinical accuracy of the device. The application of the GPB model also leads to a relevant improvement in the parameters that describe the rate accuracy, such as MARD and MedARD.

Conclusions

The use of BG values as the reference concentration data against which to evaluate the accuracy performance of subcutaneous CGM systems leads to an inherent underestimation of the “true” accuracy of continuous glucose monitors. Indeed, the physiological differences that exist between glucose concentration in the ISF and the corresponding value in the blood sample may be misinterpreted as a measurement error. When assessing the accuracy of a CGM system, a crucial step in data analysis is to account for time lag, which would enable minimization of the apparent decline in system accuracy that is particularly relevant during rapid glucose excursions. A retrospective compensation for time lag through application of a fixed delay represents a straightforward method for reducing errors in the accuracy evaluation process. Taking advantage of a closer description of the diffusion physiology involved in the mutual exchange of glucose between ISF and blood compartments, the proposed GPB method leads to a further reduction in the errors that are commonly made when assessing the accuracy of a CGM device.

However, assessment of CGM system accuracy should be performed both without any compensation for time lag, which provides an overall evaluation of the accuracy, and then with correction for the time lag in order to highlight other sources of error for the system.

Despite the general advantages provided by the use of time lag compensation methods in combination with subcutaneous CGM devices, such an approach may not be suitable for specific classes of subjects. Indeed, under particular physiological conditions (such as hypotension, shock, and insulin-induced hypoglycemia), which may be encountered in critically ill patients, the correlation between the ISF and the BG concentration^16,39 can be significantly reduced. In such cases, time lag compensation methods would provide limited improve-ments to data analysis, with a consequent decline in the accuracy of subcutaneous CGM profiles. While the worsening in the accuracy caused by these physiological alterations may be acceptable for the retrospective use of CGM data (e.g., for therapy adjustments in patients with diabetes), it may represent a significant issue for the real time CGM applications and particularly for the use in critical settings (such as in intensive care units). In such cases, the only effective way for overcoming the problem would be to drastically change the measuring compartment, switching from subcutaneous CGM systems to intravascular CGM devices.

Glossary

(BG)

blood glucose

(CGM)

continuous glucose monitoring

(GMD)

GlucoMen Day

(GPB)

glucose peak broadening

(IFG)

interstitial fluid glucose

(ISF)

interstitial fluid

(MAE)

mean absolute error

(MARD)

mean absolute rate deviation

(MARE)

mean absolute relative error

(MedAE)

median of the absolute error

(MedARD)

median absolute rate deviation

(MedARE)

median absolute relative error

(VB)

venous blood

Funding

This work was funded by A. Menarini Diagnostics, Florence, Italy.

Disclosures

Cosimo Scuffi, Fausto Lucarelli, and Francesco Valgimigli are full-time employees of A. Menarini Diagnostics.