Comparator Data Characteristics and Testing Procedures for the Clinical Performance Evaluation of Continuous Glucose Monitoring Systems

Abstract

Comparing the performance of different continuous glucose monitoring (CGM) systems is challenging due to the lack of comprehensive guidelines for clinical study design. In particular, the absence of concise requirements for the distribution of comparator (reference) blood glucose (BG) concentrations and their rate of change (RoC) that are used to evaluate CGM performance, impairs comparability. For this article, several experts in the field of CGM performance testing have collaborated to propose characteristics of the distribution of comparator measurements that should be collected during CGM performance testing. Specifically, it is proposed that at least 7.5% of comparator BG concentrations are <70 mg/dL (3.9 mmol/L) and >300 mg/dL (16.7 mmol/L), respectively, and that at least 7.5% of BG-RoC combinations indicate fast BG changes with impending hypo- or hyperglycemia, respectively. These proposed characteristics of the comparator data can facilitate the harmonization of testing conditions across different studies and CGM systems and ensure that the most relevant scenarios representing real-life situations are established during performance testing. In addition, a study protocol and testing procedure for the manipulation of glucose levels are suggested that enable the collection of comparator data with these characteristics. This work is an important step toward establishing a future standard for the performance evaluation of CGM systems.

Introduction

In the recent past, the performance, in particular the accuracy, of continuous glucose monitoring (CGM) systems has improved sufficiently to enable nonadjunctive use for clinical decision-making.¹ Together with the introduction of factory-calibration, that is, the redundancy of manual finger-stick calibration, in most systems, CGM is now an integral part of routine clinical diabetes management. Furthermore, targets for CGM-derived therapy metrics such as the “time in range” have been incorporated into national and international diabetes management guidelines,^2,3 and have been used to determine the effectiveness of new diabetes therapies.⁴ Therefore, a rigorous and standardized evaluation of CGM system performance is crucial to ensure the safety and efficacy of CGM use in people with diabetes and the validity of CGM-based outcomes in clinical trials.

This article reviews the relevant literature and presents our expert proposal for characteristics of comparator data and the testing procedures during the clinical performance evaluation of CGM systems.

Problem statement

Despite the widespread adoption of CGM system use, there are no comprehensive guidelines, regulatory or otherwise, for the study design of clinical CGM performance evaluations.⁵ Some, but not all, crucial elements of study design are covered in the POCT05 guideline put forward by the Clinical and Laboratory Standards Institute,^6,7 and the requirements for “integrated” CGM (iCGM) systems set forth by the U.S. Food and Drug Administration (FDA).⁸ This has led to a wide range of study designs that can account for the discrepancies observed in the performance levels of the same CGM system.^9,10 As a result, it is challenging to obtain a clear picture of the performance of any individual system and make meaningful comparisons between CGM systems.^10,11

A crucial element of the study design is the testing procedures, especially for the deliberate manipulation of glucose levels. These procedures can create low and high blood glucose (BG) concentrations as well as fast and slow BG concentration changes during comparator (reference) data collection. These data are subsequently used to determine CGM system performance. This performance encompasses aspects such as analytical and clinical accuracy as well as the changes in accuracy over the sensor lifetime (stability) and the reliability of CGM system alerts.¹⁰ Therefore, the testing procedures need to be designed to enable the comprehensive assessment of these different aspects of performance. Furthermore, it has been shown that the characteristics of the comparator BG measurements, and therefore also the testing procedures for the manipulation of glucose levels, have a major influence on the observed performance of a CGM system. In particular, a potential deterioration of accuracy during hypoglycemia or periods of fast BG concentration changes has been observed.^12–16 A testing procedure in which these conditions are underrepresented could lead to biased performance results, highlighting the need for standardized performance testing, especially with regard to the characteristics of the comparator data.

The aim of this article is to propose specific characteristics of the comparator measurements collected during CGM performance studies that include both the BG concentrations and their rate of change (RoC). These characteristics were selected based on an analysis of real-world CGM data and can ensure that all relevant glycemic situations are covered during performance testing. In addition, we suggest a study protocol and testing procedure capable of producing comparator data with the proposed characteristics, thus enabling a thorough investigation of CGM system performance. This article is a proposal from the Working Group on Continuous Glucose Monitoring established by the Scientific Division of the International Federation of Clinical Chemistry and Laboratory Medicine (IFCC)^11,17 and other experts in the field of CGM performance testing.

Literature background

Study protocols of CGM performance studies typically include the collection of comparator data in both in-clinic and free-living settings. In the free-living setting, subjects are instructed to measure their capillary BG levels multiple times per day with a home-use blood glucose monitoring (BGM) system. The advantage of this setting is that it represents CGM use under real-life conditions and that it can produce comparator data on every day of the sensor lifetime, which is important to assess the continued accuracy of the CGM system from insertion to removal. The disadvantage is that the distribution of comparator data is difficult to control and that the frequent measurements required for the determination of BG level RoCs are difficult to attain. This means that a robust and comprehensive estimation of accuracy and alert reliability from free-living data alone is not feasible. Furthermore, the measurement accuracy of home-use BGM systems is often inferior to laboratory-grade glucose analyzers. For these reasons, studies are designed to also include in-clinic sessions, where participants may spend several hours to multiple days at the investigational sites. Here, comparator measurements can be carried out every 5–15 min, typically for up to 12 h. Moreover, BG level dynamics can be manipulated through adaptation of insulin dosing/timing and consumption of specific meals.¹⁰

Our recent review of 129 clinical CGM performance studies published between 2002 and 2022 found that 73% of studies including in-clinic sessions reported a deliberate manipulation of glucose levels, although the glucose manipulation procedures were often described with insufficient detail.¹⁰ From the limited available information, there appear to be two main testing procedures targeting different glucose profiles during the in-clinic sessions.

First, there are studies that aim to produce either hypo- or hyperglycemic BG concentrations within a single session lasting between 8 and 12 h in total, but the actual time spent in hypo- or hyperglycemia is not reported.^18–22 This approach appears to be recommended in the POCT05 guideline,⁷ although it likely limits RoCs because of a lack of change between low and high BG concentrations. Second, there exist studies that aim to induce both hypo- and hyperglycemic BG concentrations within a single in-clinic session, with some studies opting to start with hyperglycemia followed by hypoglycemia,^{13,14,23–32} while other studies use the reverse order,^33–35 or choose the order based on the BG concentration at the start of the session.¹⁵ Inducing hypo- and hyperglycemic BG concentrations within the same session typically leads to higher RoCs.

The goal of the glucose manipulation, and thus the target characteristics of the comparator data distribution, is often not defined. Only a few articles state the vague rationale to produce comparator data covering a wide range of BG levels.^{18–20,33,36–38} In this context, the POCT05 guideline is more specific, recommending 8% of comparator values to be <80 mg/dL (4.4 mmol/L) and 5% to be >300 mg/dL (16.7 mmol/L).⁷ This recommendation was not found in the first edition of the guideline published in 2008,⁶ but was added to the second edition of the POCT05 guideline published in 2020.⁷ Its adoption in current studies is thus, so far, difficult to assess. However, a recent review, found that only 50% of the studies utilized for Conformité Européenne marking of CGM systems reported data on adults that satisfied both POCT05 distribution criteria.⁵ In contrast, the FDA iCGM requirements are more vague, only stating that “[…] clinical data must be obtained […] throughout the measuring range of the device.”⁸

The importance of producing rapidly changing BG concentrations is not mentioned in the current guidelines. Interestingly, the first edition of the POCT05 guideline recommended that “a sufficient number of points should be obtained at the extreme rates of change (< −1.5 mg/dL/min and >1.5 mg/dL/min),”⁶ without providing a precise percentage. This statement has been removed in the second edition.⁷ The majority of published articles also make no statement about the targeted BG concentration changes, with only some reports published more than 10 years ago explicitly mentioning the goal to assess CGM performance during rapidly falling and rising BG concentrations.^21,22,39

General Study Design

Study population

The elements of study design proposed in this article concern the performance evaluation of CGM systems in adults, intended to be used for therapeutic decision-making and integration into systems for automated insulin delivery (AID). The study population should thus reflect the intended use population of the CGM systems, that is, mainly people with type 1 diabetes, but also people with type 2 diabetes. To facilitate the collection of comparator data with the characteristics proposed in this article, we suggest that at least 75% of participants have type 1 diabetes. This suggestion agrees with the results of a recent literature review, where we found that on average 73.5% of participants had type 1 diabetes in studies including participants with both diabetes types.¹⁰

For CGM systems with other intentions of use, especially other intended use populations, this proposal may not be suitable, and different approaches may be needed.

Study protocol

We propose a study protocol in which the subjects spend the majority of the sensor lifetime in the free-living setting, interrupted by several in-clinic sessions lasting 8–12 h. In-clinic sessions should be scheduled so that their overall distribution equally reflects the beginning, middle, and end of the sensor lifetime, in accordance with both the POCT05 guideline and FDA iCGM requirements.^7,8 This approach minimizes the burden on the subjects associated with the in-clinic session procedures. The exact distribution, as well as the number of sessions per subject, is dependent on the sensor lifetime. A possible study protocol for a CGM system with a 14-day lifetime is provided in Figure 1. It can easily be adapted for CGM systems with different sensor lifetimes.

FIG. 1.

Schematic depiction of an example study protocol for a CGM system with a sensor lifetime of 14 days and three in-clinic sessions per participant. The dashed orange squares indicate other possible study days for in-clinic sessions, thus requiring suitable scheduling to cover study days 1, 2, 3, 7, 8, 9, 13, 14, and 15 equally often. CGM, continuous glucose monitoring.

CGM system use

Regarding the CGM sensor insertion and calibration, we also refer to the POCT05 guideline.⁷ There it is recommended that, if applicable, the sensors should be inserted by the participants themselves under the supervision of the study staff and that all intended sensor locations, for example, arm and abdomen, are examined.

To assess between-sensor precision, the CGM data from two sensors worn simultaneously by the same participant should be compared. For that, it is necessary to insert two sensors at adjacent sites, at least for a subgroup of participants. However, we recommend including only one sensor per application site and participant (determined before insertion) in the main accuracy analysis because it likely affects the statistical independence of results. Here, a study with a factory-calibrated CGM system, where two sensors were worn simultaneously, estimated that the between-participant variance in sensor-specific mean absolute relative differences (MARDs) contributed 44% to the total variation in sensor-specific MARDs.⁴⁰ This shows that the variation in characteristics between participants significantly contributes to the overall variance in accuracy. It is therefore possible that the inclusion of multiple sensors from the same participant would lead to an underestimation of variance in accuracy. Moreover, the CGM systems should be calibrated according to the minimally required schedule and we advise that calibrations should not be carried out during or immediately before the frequent sampling period (FSP) (see section “Sampling protocol and comparator measurements”).

Free-Living Testing Procedures

In the free-living setting, we propose to ask participants to follow their regular daily routine and carry out at least seven capillary BG measurements per day with a BGM system fulfilling the accuracy requirements of the current ISO15197 standard,⁴¹ or the FDA over-the-counter guidance.⁴² The timing of these measurements should follow the established seven-point profile (immediately before and 2 h after breakfast, lunch, and dinner, and before bed).⁴³

In addition, we recommend that the participants should perform the measurements in duplicate from the same finger prick. This allows the retrospective identification of outliers, for example, due to handling errors, and reduces imprecision as the results from duplicate measurements can be averaged. If considered feasible, participants may also be asked to perform a third measurement if the relative difference (with respect to the first measurement) of the duplicate exceeds ±10% if the first measurement is ≥100 mg/dL (5.6 mmol/L), or ±10 mg/dL (0.56 mmol/L) if the first measurement is <100 mg/dL (5.6 mmol/L).

In-Clinic Testing Procedures and Comparator Data Characterization

Sampling protocol and comparator measurements

We propose that the in-clinic sessions should include an FSP, where comparator measurements are carried out every 15 min over a period between 6 and 8 h. This duration agrees with the median FSP duration of 7.5 h, estimated in our recent literature review of CGM performance studies,¹⁰ and is adequate to produce comparator data with the recommended characteristics, as explained below. The 15-min sampling interval is in accordance with the POCT05 guideline,⁷ and is deemed sufficiently narrow to capture the variability of short-term BG concentration fluctuations and to estimate their RoC.⁴⁴ In a number of recent studies,^{19,20,30,45,46} this sampling interval was shortened during phases of hypo- and hyperglycemia, presumably to increase safety and boost the number of comparator data points in these glycemic regions. However, as we have mentioned in a previous article,¹⁰ this approach has led to concerns about the statistical interdependency of the measurements. Furthermore, a constant sampling interval greatly facilitates the standardization of data analysis and statistical performance evaluation. We therefore recommend a protocol with a constant sampling interval of 15 min.

In our review on CGM performance, we found that comparator BG concentrations are measured in capillary, venous, and arterialized-venous blood.¹⁰ These approaches can yield different results due to physiologically different BG concentrations in these blood samples and thus affect the observed CGM performance. It is therefore crucial to standardize the comparator measurement procedure. However, a detailed discussion of this topic lies beyond the scope of this article and will be discussed in a future article. In principle, the in-clinic testing procedures suggested in this article are independent from the chosen approach. However, it should be mentioned that a safe and effective manipulation of BG concentrations, as described below, requires that measurement results are available in near real-time to decide on food intake and/or insulin dosing.

Irrespective of the chosen comparator measurement approach, the schedule for capillary measurements with the BGM system followed during free-living days should also be implemented on days with in-clinic sessions to provide capillary comparator measurements on every study day.

The dynamic glucose region plot

Users of CGM systems routinely combine the displayed information on current sensor glucose levels and their accompanied RoC (in the form of trend arrows) to make therapeutic decisions. We therefore propose to evaluate the CGM performance under varying combinations of BG concentration and RoC, which can be illustrated as follows. The same RoC of −3 mg/(dL·min) [−0.17 mmol/(L·min), ↓↓] has a different clinical interpretation depending on whether the BG level is at 90 mg/dL (5 mmol/L) or 230 mg/dL (12.8 mmol/L). Likewise, the same BG level of 72 mg/dL (4 mmol/L) would lead to different therapeutic decisions depending on whether the RoC is +2 mg/(dL·min) [+0.11 mmol/(L·min), ↑] or −2 mg/(dL·min) [−0.11 mmol/(L·min), ↓].

We have therefore developed a graphical representation for the combination of BG concentrations and RoC, referred to as the dynamic glucose region (DGR) plot, based on similar representation called dynamic risk space.⁴⁷ Figure 2 shows how a glucose profile obtained during an in-clinic session translates to the DGR plot.

FIG. 2.

(a) Target profile of BG concentrations over time during an in-clinic session. (b) DGR plot showing the associated profile of combined RoC and BG. The gray arrows indicate the direction of the curve with respect to its evolution over time. Note that there is no RoC-BG pair for time zero, because each BG value is paired with an RoC calculated with respect to the preceding BG value. The colored background indicates the critical regions in which adequate CGM accuracy is of particular importance. The panel on the right shows the percentages of RoC-BG pairs in the respective regions, the proposed minimum required percentage for the critical regions (7.5%), the MARoC in mg/(dL·min), and the number of RoC-BG pairs (n = 28). BG, blood glucose; DGR, dynamic glucose region; MARoC, mean absolute RoC; RoC, rate of change.

As shown in Figure 2b, the area of the DGR plot is divided into five regions comprising four critical regions (red and orange) and one neutral region (green). The critical regions were chosen to reflect situations in which adequate CGM accuracy is of particular importance and are based on commonly used BG concentration thresholds.

The orange regions termed “BG high” and “BG low” acknowledge the need to test the CGM systems across their measurement range, consistent with existing guidelines.^7,8 We chose BG concentrations <70 mg/dL (3.9 mmol/L) for the “BG low” region and >300 mg/dL (16.7 mmol/L) for the “BG high” region, independent of the associated RoCs. The lower threshold was chosen in deviation from the POCT05 guideline (<80 mg/dL [4.4 mmol/L]⁷) to align with the consensus threshold for level 1 hypoglycemia,^48,49 and to be closer to the lower limit of the measurement range of most CGM systems (40 or 50 mg/dL [2.2 or 2.8 mmol/L]).

The upper threshold of 300 mg/dL (16.7 mmol/L) was chosen in accordance with POCT05 guideline but differing from the threshold for level 2 hyperglycemia (250 mg/dL [13.9 mmol/L])⁵⁰ to be closer to the upper limit of the measurement range of most CGM systems of 400 mg/dL (22.2 mmol/L), giving this region a more technical, rather than a clinical, justification.

The two red regions in Figure 2b labeled “Alert low” and Alert high” represent situations with moderate-to-fast RoCs and impending hypo- or hyperglycemia. The “Alert low” region with rapidly falling BG concentrations in the normal range has the highest clinical risk of an adverse event, for example, severe hypoglycemia, and is therefore of particular importance. It is also the region in which AID systems typically suspend insulin infusion. This region represents BG levels ≥70 mg/dL (3.9 mmol/L) and RoCs < −1 mg/(dL·min) [−0.06 mmol/(L·min)]. Furthermore, the region includes RoC-BG combinations, that, given the current RoC, would lead to BG concentrations below 70 mg/dL (3.9 mmol/L) within 30 min, which causes the upper border of this region to be tilted. The time interval of 30 min was chosen as it is gives diabetes patients sufficient time to take action and it is used as the prediction horizon in some current-generation AID systems.^51,52 RoCs between −1 mg/(dL·min) [−0.06 mmol/(L·min)] and 0 mg/(dL·min) were not included in this region to ensure that the CGM systems are tested at enough rapidly falling RoCs.

Accordingly, the “Alert high” region is defined by BG levels ≤300 mg/dL (16.7 mmol/L), RoCs > +1.5 mg/(dL·min) [+0.08 mmol/(L·min)], and current BG-RoC combinations leading to a BG level increase above 250 mg/dL (13.9 mmol/L, level 2 hyperglycemia) within the following 30 min.

The remaining RoC-BG combinations in the green region termed “Neutral” are less clinically relevant. Nevertheless, the majority of data points will be found in this region due to the practical nature of CGM performance studies. The question of how the collected data points should be distributed across the regions is addressed in the section “Recommendations for the comparator data distribution.” A summary of all region definitions is provided in Table 1.

Table 1.

Definition of the Regions in the Dynamic Glucose Region Plot and Recommended Percentages of Rate of Change—Blood Glucose Data Points in Each Region

Region	Color	Definition	Recommended percentage, %
BG low	Orange	BG <70 mg/dL (3.9 mmol/L)	≥7.5
		Any RoC
BG high	Orange	BG >300 mg/dL (16.7 mmol/L)	≥7.5
		Any RoC
Alert low	Red	BG ≥70 mg/dL (3.9 mmol/L)	≥7.5
		RoC < −1 mg/(dL·min) [−0.06 mmol/(L·min)]
		BG <70 mg/dL (3.9 mmol/L) within 30 min at current RoC
Alert high	Red	BG ≤300 mg/dL (16.7 mmol/L)	≥7.5
		RoC > +1.5 mg/(dL·min) [+0.08 mmol/(L·min)]
		BG >250 mg/dL (13.9 mmol/L) within 30 min at current RoC
Neutral	Green	All other RoC-BG pairsa	≤70

^a BG levels should be paired with RoC values calculated from the current and preceding BG level.

BG, blood glucose; RoC, rate of change.

Analysis of real-world CGM data

To demonstrate the need to create a wide range of BG concentrations and RoCs during the in-clinic sessions as well as to legitimize the critical regions of the DGR plot, real-world CGM data collected in five different studies,^4,53–56 made publicly available,⁵⁷ were pooled and analyzed. The pooled data set contains more than 4.3 million hours of CGM data from 897 individuals with type 1 diabetes between the ages of 2 and 86 years. Additional details on the data sets and data analysis are provided in the Supplementary Data.

To validate the boundaries of the “Alert low” region, CGM traces recorded in the 60 min before hypoglycemic episodes were analyzed (Fig. 3a, b). A hypoglycemic episode was included in the analysis if consecutive CGM glucose levels were <70 mg/dL (3.9 mmol/L) for at least 15 min,⁵⁸ and up to 120 min. In addition, for any episode to be included in the analysis, CGM glucose levels had to be ≥70 mg/dL (3.9 mmol/L) and monotonically decreasing (RoC ≤0) in the 60 min before the episode of interest. This led to the selection of more than 38,000 CGM traces containing a hypoglycemic episode. Considering only the steeper descents into hypoglycemia because they carry a higher risk for an adverse event, shown in Figure 3b, it is demonstrated that ∼50% of prehypoglycemia CGM traces traverse the “Alert low” region. This justifies the need to expose the CGM systems to those combinations of BG concentrations and RoCs during performance testing.

FIG. 3.

Results from the analysis of real-world CGM data concerning the “Alert low” (a, b) and “Alert high” (c, d) region. (a, c) Show the summary statistics of CGM traces from −60 to +60 min after the start of a hypo-/hyperglycemic episode according to the recommendations of the Ambulatory Glucose Profile.⁵⁹ (b, d) Show the interpolated and smoothed CGM RoC—glucose-level races of selected summary statistics from −60 to the start of the hypo-/hyperglycemic episodes. The arrows indicate the direction of the curves over time.

Similar relevance can be assigned to the “Alert high” region (Fig. 3c, d) with a corresponding analysis. A hyperglycemic episode was defined as CGM glucose levels >250 mg/dL (13.9 mmol/L) for at least 15 min,⁵⁸ and for any episode to be included in the analysis, CGM glucose levels had to be ≤250 mg/dL (13.9 mmol/L) and monotonically increasing (RoC ≥0) in the 60 min before any episode. This led to the selection of more than 42,000 CGM traces. As shown in Figure 3c, ∼25% of selected CGM traces eventually crossed the 300 mg/dL (16.7 mmol/L) threshold (dotted line in Fig. 3c), giving strong justification to the “BG high” region.

Glucose manipulation procedures

The following section introduces a possible glucose manipulation procedure designed to generate comparator data in the proposed regions of the DGR plot in a safe, ethical, and consistent manner. We want to emphasize that other testing procedures adequately covering the critical regions in the DGR plot are possible. The description of the procedure provided below focuses on the most important aspects and is not intended to be an exhaustive protocol to be used as a template.

The basis of our suggestion for the in-clinic testing procedures will form the target glucose profile (TGP), shown in Figure 2a. The TGP was designed to traverse all regions of the DGR plot within a single in-clinic session and a 7-h FSP. The TGP includes distinct and transient hyper- and hypoglycemic episodes similar to the ones observed in the real-life data set (Fig. 3). This allows a more realistic assessment of alert reliability in comparison with testing procedures that induce hypo- and hyperglycemic episodes lasting 60 min or longer. The general approach suggested by the TGP, that is, inducing hyperglycemia followed by hypoglycemia, is similar to a number of approaches reported in the literature,^{13,14,23–32} as discussed above. Using the reverse order, that is, to start with hypoglycemia followed by hyperglycemia, might be a suitable approach to populate the “BG low,” “Alert high,” and “BG high” regions. However, when the subjects' BG levels are lowered toward hypoglycemia at the start of an in-clinic session from fasting conditions, it might be difficult to produce the rapidly falling BG concentrations that are required to populate the “Alert low” region, whose clinical relevance has been shown in the analysis of real-world data.

The procedure described below is most suitable for participants with type 1 diabetes because it relies on the adjustment of timing and dosing of rapidly acting insulin. In addition, participants should have well-established insulin correction and carbohydrate-to-insulin factors and no relevant comorbidities (in particular gastroparesis). BG concentrations should be closely monitored throughout the in-clinic session and additional measurements outside the previously recommended 15-min interval may be carried out to ensure participant safety but should be excluded from data analysis. In addition, we recommend that this procedure is only carried out by study sites experienced in the management of people with diabetes.

Before the start of the FSP, the participant's BG concentrations should be stable within the target range (70–180 mg/dL [3.9–10 mmol/L]) and any AID systems should be switched to manual mode, that is, no automated adaptation of insulin delivery, until the end of the FSP. At the start of the FSP, the participants are served a meal specifically selected for this procedure. We suggest a meal containing a maximum of ∼25% of the participants' daily caloric demand consisting of 65% carbohydrates (both slow and fast absorbed), 20% fat, and 15% protein. The bolus of short-acting insulin that would normally be delivered at or around mealtime is then withheld until BG levels have reached values between 250 and 300 mg/dL (13.9 and 16.7 mmol/L). This should induce the glycemic conditions indicative of the “Alert high” and “BG high” regions, that is, a rapid BG concentration rise as well as sustained BG levels >300 mg/dL (16.7 mmol/L) for ∼30 min. The size of the bolus should be calculated based on the participant-specific therapy parameters, the carbohydrate content of the meal, and the BG level at the start of FSP with a target BG level of 100 mg/dL (5.6 mmol/L). A general increase in the bolus size to induce the subsequent hypoglycemia is typically not necessary.

Our experience indicates that, after reaching their peak, BG levels drop rapidly toward hypoglycemia traversing the “Alert low” and “BG low” regions. Should this not happen, additional insulin boluses can be delivered. Once BG concentrations below 70 mg/dL (3.9 mmol/L) have been reached or are imminent, appropriate hypoglycemia interventions are implemented. The specific amount, composition, and timing of food intake for this intervention should be decided at the discretion of study staff in agreement with the participant. However, we suggest to choose the hypoglycemia intervention strategy such that BG concentrations are maintained <70 mg/dL (3.9 mmol/L) for ∼30 min, while avoiding rebound hyperglycemia through excessive carbohydrate intake. For the remainder of the FSP, the BG levels should be kept stable in the range 100–180 mg/dL (5.5–10 mmol/L), allowing a safe release of the participants at the end of the FSP.

We suggest that any further meals or snacks are low in carbohydrate content and that the remaining insulin action of the first bolus is considered in any decisions on insulin dosing/timing. If the participants' BG levels should not respond as expected, and not all critical regions of the DGR plot are reached, corrective actions can be taken as long as they are deemed safe by the study staff. These actions can include additional insulin boluses, the adaptation of basal insulin infusion for participants using insulin pumps, food intake, or mild exercise.

The characteristics of the comparator data collected in a study with 50 participants and 195 in-clinic sessions implementing a similar testing procedure are shown in Figure 4. The main differences between the protocol followed in this study and the procedure described in this article are the use of a slightly smaller meal at the beginning of the FSP and the consumption of a freely chosen second meal after 5.5 h. Furthermore, 11 (22%) participants underwent no manipulation of insulin dosing and followed their routine therapy after consuming the test meal. This explains the lower initial BG peak and the sharp rise after 5.5 h in comparison with the TGP (Fig. 4a). This figure also demonstrates the variability in profiles regarding the time of hypoglycemia, despite applying the same procedure in every participant and in-clinic session.

FIG. 4.

Characteristics of comparator data collected in a study with 50 participants during 194 in-clinic sessions. (a) Summary statistics of the time profile of BG concentrations. (b) DGR plot showing the individual RoC-BG pairs. The area on the right shows the percentage of RoC-BG pairs in the respective regions, the proposed minimum required percentage for the critical regions (7.5%), the total number of RoC-BG pairs (n = 5285), and the MARoC in mg/(dL·min) at the bottom.

Figure 4b shows that ∼31% of all RoC-BG pairs were in the critical regions and that the “Alert low” and “BG high” regions were slightly less represented than the other critical regions.

Recommendations for comparator data distribution

After defining the critical regions in the DGR plot and describing a possible glucose manipulation procedure, we can introduce our recommendation for the characteristics of the comparator data distribution; similar but less stringent in comparison with the ISO 15197:2015 standard for BGM systems.⁴¹ Defining the characteristics of the comparator data is more practicable than mandating a particular testing procedure in a standardized study design.

The TGP (Fig. 2) and associated procedure were designed so that ideally, ∼11% of data points would be found in each critical region, leaving around 57% in the neutral region. However, due to the expected natural variability in the comparator data, exemplified in Figure 4, it is unlikely that this ideal distribution is achieved. Furthermore, it should not be necessary to apply the glucose manipulation procedure to all participants included in the study, thus allowing the possibility to include subjects with type 2 diabetes as mentioned above. We propose that the minimal percentage of data points lying in each critical region should be 7.5%, leaving at most 70% of data points in the neutral region. Note that these recommendations should be fulfilled for only the comparator data points that are used for the CGM accuracy evaluation, that is, after they have been paired with the CGM data. A proposal for the absolute number of data points to be found in each region is beyond the scope of this article as these numbers depend on the chosen performance parameters and possible minimum acceptance criteria for those parameters.

Should it be anticipated that the proposed recommendation of 7.5% of data points in each critical region is not fulfilled, it is possible for some participants to follow a glucose manipulation procedure that fills specific regions of the DGR plot. For example, if additional hypoglycemic BG concentrations are required, it is possible to implement the previously mentioned approach of starting with hypoglycemia followed by hyperglycemia. Alternatively, we propose the following procedure for discarding the least critical RoC-BG pairs from the neutral region until the recommended percentages are achieved. This procedure is exemplified on the data set shown in Figure 4, where the “BG high” and “Alert low” regions have an insufficient number of data points.

For each RoC-BG pair, a Euclidian distance measure D with respect to the center of the “neutral” region, defined at a BG level of 185 mg/dL (10.3 mmol/L) and an RoC of 0, is calculated as follows:

$$D=\sqrt{{\left(\frac{RoC}{a}\right)}^2+{\left(\frac{BG-B{G}_c}{b}\right)}^2},$$

where the value for a is set to 1 mg/(dL·min) [0.06 mmol/(L·min)], the value for b is set to 115 mg/dL (6.4 mmol/L), and BG_c represents the center of the neutral region set to 185 mg/dL (10.3 mmol/L). Subsequently, the RoC-BG pairs with the lowest values of D are successively excluded until the recommended minimal percentages for all critical regions are met.

The geometric interpretation of this procedure is exemplified in Figure 5 and shows that the elimination region takes the form of an ellipse with a constant height-to-width ratio. The shape of the ellipse, determined by the values for a and b, was chosen to stretch between −1 and +1 mg/(dL·min) [−0.06 and +0.06 mmol/(L·min)] at the horizontal edges of the neutral region (70 mg/dL [3.9 mmol/L] and 300 mg/dL [16.7 mmol/L]), in which case D has a value of 1. The size of the ellipse is roughly determined by the number of data points that need to be excluded but it should not stretch into the “BG high” and “BG low” region. This means that, if any RoC-BG pairs with a D > 1 have to be excluded, the underlying data set may be inadequate to comply with the recommended distribution of comparator data.

FIG. 5.

DGR plot exemplifying the procedure for eliminating data from the neutral region using the data presented in Figure 4. The defined center of the neutral region is marked with a red cross, and the minimal elliptic exclusion region to achieve the desired distribution of at least 7.5% of comparator data in each critical region is outlined in red. The gray concentric ellipses have the same height-to-width ratio and indicate other possible regions for elimination of data. The shape of the ellipse was chosen to stretch between −1 and +1 mg/(dL·min) [−0.06 and +0.06 mmol/(L·min)] at the horizontal edges of the neutral region (70 mg/dL [3.9 mmol/L] and 300 mg/dL [16.7 mmol/L]).

This procedure for eliminating data was designed to be easily adopted, independent from the corresponding CGM data and without the need for random sampling, therefore allowing reproducibility and facilitating standardization. For the example data set in Figure 5, a total of 165 data points, corresponding to 3.1% of all data pairs, had to be excluded for the percentage of data points in all the critical regions to be at least 7.5%. This demonstrates that a study implementing a testing procedure similar to the one described in this article can fulfill our recommendation with a justifiable number of excluded data points, even if only 78% of participants undergo the full glucose manipulation procedure.

Conclusion

This article, for the first time, provides a comprehensive proposal and justification for the distribution of comparator BG concentrations and RoCs to be produced during CGM performance testing. Our proposal was built upon results from a previous study and utilizes the newly developed DGR plot identifying clinically relevant combinations of glucose levels and their RoC. We encourage researchers to use this tool when reporting results of CGM performance studies, thus increasing transparency and facilitating the comparison of results between studies.¹⁰ A free and open-source software package to create the DGR plot is published with this article (https://github.com/IfDTUlm/CGM_Performance_Assessment).

We also suggest the basic elements for the development of a study protocol and in-clinic testing procedures capable of fulfilling the proposed characteristics of comparator data. Implementing a similar procedure in future studies would harmonize the comparator data characteristics across studies. As a result, the observed performance of CGM systems is expected to be independent from the study design and becomes comparable, even when tested in separate studies. We expect that this work is an important element of a future standard for the performance evaluation of CGM systems.

A pilot verification study is planned based on our proposal, and we would like to invite other research centers and manufacturers of CGM systems to test the feasibility of the proposed comparator data distribution and testing procedures to establish their suitability for incorporation in an internationally recognized standard for the clinical performance evaluation of CGM systems.

Authors' Contributions

M.E.: data curation, formal analysis, investigation, methodology, software, visualization, and writing—original draft; S.P.: validation and writing—review and editing; M.R.: writing—review and editing; T.S.B.: writing—review and editing; L.B.: writing—review and editing; R.B.: writing—review and editing; D.B.: writing—review and editing; P.D.: writing—review and editing; E.E.B.: writing—review and editing; M.F.: writing—review and editing; C.H.: writing—review and editing; R.H.: writing—review and editing; J.J.: writing—review and editing; D.C.K.: writing—review and editing; J.K.M.: writing—review and editing; K.M.: writing—review and editing; O.M.: writing—review and editing; J.H.N.: writing—review and editing; K.N.: writing—review and editing; J.P.: writing—review and editing; E.S.: writing—review and editing; L.S.: writing—review and editing; A.T.: writing—review and editing; N.K.T.: writing—review and editing; L.W.: writing—review and editing; R.J.S.: writing—review and editing.

G.F.: conceptualization, funding acquisition, project administration, resources, supervision, and writing—review and editing.

Author Disclosure Statement

M.E., S.P., and C.H. are employees of Institut für Diabetes-Technologie, Forschungs- und Entwicklungsgesellschaft mbH an der Universität Ulm (IfDT), which carries out clinical studies, for example, with medical devices for diabetes therapy on its own initiative and on behalf of various companies. IfDT and/or its employees have received speakers' honoraria or consulting fees in the last 3 years from Abbott, Ascensia, Berlin Chemie, BOYDSense, Dexcom, Lilly, Metronom, MySugr, Novo Nordisk, PharmaSens, Roche, Sanofi, and Terumo. M.R. is an employee of Diabetes Center Berne. T.S.B. has received research support from Abbott Diabetes, Abbott Rapid Diagnostics, Biolinq, Dexcom, Eli Lilly, Kowa, LifePlus, Medtronic, Novo Nordisk, Sanofi, and Senseonics, as well as consulting honoraria from Abbott Diabetes, Abbott Rapid Diagnostics, CeQur, LifeScan, MannKind, Medtronic, Novo, and Sanofi. L.B. declares no conflict of interest related to this work.

R.B. is a full-time employee of Rainier Clinical Research Center, which receives study grants from multiple CGM device companies for conducting their clinical trials. D.B. received lecture or consultancy fees from Abbott, Eli Lilly, Menarini, Novo Nordisk, Roche, Sanofi, and Theras. P.D. is a board member of PharmaSens. E.E.B. has no disclosures. M.F. received lecture fees from Menarini. R.H. is an employee of Roche Diabetes Care GmbH. J.J. has for the last 3 years been a lecturer/member of the scientific advisory board for Abbott, AstraZeneca, Boehringer Ingelheim, Eli Lilly, Medtronic, Nordic InfuCare, Novo Nordisk A/S, and Sanofi. D.C.K. is a consultant for Atropos Health, Better Therapeutics, EOFlow, Integrity, Lifecare, Nevro, Novo, Sanofi, and Thirdway.

J.K.M. is a member of the advisory boards of Abbott Diabetes Care, Becton-Dickinson, Boehringer Ingelheim, Eli Lilly, Embecta, Medtronic, Novo Nordisk A/S, Roche Diabetes Care, Sanofi-Aventis, and Viatris, and received speaker honoraria from A. Menarini Diagnostics, Abbott Diabetes Care, AstraZeneca, Boehringer Ingelheim, Dexcom, Eli Lilly, MedTrust, MSD, Novo Nordisk A/S, Roche Diabetes Care, Sanofi, Servier, Viatris, and Ypsomed. She is shareholder of decide Clinical Software GmbH and elyte Diagnostics and serves as CMO of elyte Diagnostics. K.M. has no disclosures.

O.M. has received clinical trials support from Sêr Cymru II COFUND Fellowship/European Union, Novo Nordisk A/S, Novo Nordisk AT, Abbott Diabetes Care, Sanofi, Dexcom, Team Novo Nordisk, SAIL, Maisels Brauerei, Medtronic AT, EFSD/EASD, Falke, BISp, and MyLife CampAPS; speaker fees from Medtronic AT, Medtronic Int., Eli Lilly, Novo Nordisk, Sanofi, TAD Pharma, and ADA.; Diatec, Berufsverband deutscher Internist*innen, Astra Zeneca; travel support from Novo Nordisk A/S, Novo Nordisk AT, Novo Nordisk UK, Medtronic AT, Sanofi, EASD, OEDG, and DDG; and is an advisor for Sanofi and TAD Pharma. J.H.N. has received research support from Abbott. K.N. holds shares in Novo Nordisk; has been a paid consultant for Novo Nordisk and Medtronic; has received speaker honorarium and honorarium for Advisory Board to her institution from Medtronic, Novo Nordisk, and Convatec, her institution has received research funding from Zealand Pharma, Novo Nordisk, Medtronic, and Dexcom.

J.P. has worked for Medtronic from 2011 to 2016. He currently sits on the Advisory Board for Roche Diabetes and has received speaker fees from Dexcom in the last 2 years. E.S. is supported by grants from the U.S. National Institutes of Health (NIH) and has received donated materials related to NIH-supported research from Abbott Diabetes Care, Roche Diagnostics, Siemens Diagnostics, Ortho Clinical Diagnostics, Abbott Diagnostics, Asahi Kasei Pharma Corp, and GlycoMark Corp. L.S. declares no conflict of interest related to this work.

A.T. is a freelance consultant. He has received fees for lectures or consultancy fees from Abbott, Berlin Chemie, Dexcom, EvivaMed, Menarini, Novo Nordisk, Roche and Sanofi in the last 3 years. N.K.T. is a consultant for Roche Diagnostics and Roche Molecular Systems, received honoraria from Nova Biomedical and Thermo Fisher, and Chair-Elect (2023–2026) of the American Association for Clinical Chemistry, Critical and Point-of-Care Testing Division. L.W. has received funding from the Diabetes Center Berne. R.S. is chair of the Clinical Chemistry Department of Isala, which carries out clinical studies, for example, with medical devices for diabetes therapy on its own initiative and on behalf of various companies. R.S. has received speaker's honoraria or consulting fees in the last 3 years from Roche and Menarini. G.F. is the General Manager and Medical Director of the Institut für Diabetes-Technologie, Forschungs- und Entwicklungsgesellschaft mbH an der Universität Ulm.

Funding Information

This work was funded by the Diabetes Center Berne, Switzerland.

Supplementary Material

Supplementary Data