Clinical Performance Evaluation of Continuous Glucose Monitoring Systems: A Scoping Review and Recommendations for Reporting

Abstract

The use of different approaches for design and results presentation of studies for the clinical performance evaluation of continuous glucose monitoring (CGM) systems has long been recognized as a major challenge in comparing their results. However, a comprehensive characterization of the variability in study designs is currently unavailable. This article presents a scoping review of clinical CGM performance evaluations published between 2002 and 2022. Specifically, this review quantifies the prevalence of numerous options associated with various aspects of study design, including subject population, comparator (reference) method selection, testing procedures, and statistical accuracy evaluation. We found that there is a large variability in nearly all of those aspects and, in particular, in the characteristics of the comparator measurements. Furthermore, these characteristics as well as other crucial aspects of study design are often not reported in sufficient detail to allow an informed interpretation of study results. We therefore provide recommendations for reporting the general study design, CGM system use, comparator measurement approach, testing procedures, and data analysis/statistical performance evaluation. Additionally, this review aims to serve as a foundation for the development of a standardized CGM performance evaluation procedure, thereby supporting the goals and objectives of the Working Group on CGM established by the Scientific Division of the International Federation of Clinical Chemistry and Laboratory Medicine.

Introduction

Systems for continuous glucose monitoring (CGM) have been available for more than 20 years, but significant advances have been made in the recent past and CGM performance has reached a level that allows non-adjunctive use for clinical decision-making. ¹ There are now several CGM systems available for routine clinical use, and CGM-derived therapy parameters are part of national and international guidelines for the therapy of diabetes mellitus.^2,3 Furthermore, CGM data are routinely used to assess outcomes in clinical trials of diabetes treatments. As a result, the performance of CGM systems, in particular accuracy, has gained renewed importance, resulting in a surge of published studies examining CGM performance in the recent past.

The performance of a CGM system can include a wide range of aspects that characterize the properties of the CGM system. In this article, we will focus on the process of a clinical performance evaluation of CGM systems and its endpoints, described in Table 1. Note that these endpoints go beyond the investigation of accuracy and encompass other aspects such as alarm and technical reliability; however, a detailed description of how to quantify these different endpoints lies beyond the scope of this article. Aspects determined by pre-clinical testing are also not covered in this article. These include biocompatibility, electrical safety, and cyber security as well as the effect of interfering substances (both pre-clinical and clinical). Additionally, the evaluation of the clinical benefit and usability is not covered.

Table 1.

Description of the Endpoints of a Clinical Performance Evaluation of CGM Systems Discussed in the Context of This Article, Based on the POCT05 Guideline. ⁴

Endpoints of a clinical CGM performance evaluation
Accuracy • Analytical point accuracy, ie, the characterization of deviations between CGM values and blood glucose concentrations measured with a suitable comparator at a single point in time, including bias, precision, and sensor-to-sensor variability • Clinical point accuracy, ie, the clinical interpretation of the impact of the analytical point accuracy • Trend accuracy, ie, the characterization of deviations between the glucose concentration rate of change (RoC) indicated by the CGM system and blood glucose concentrations measured with a suitable comparator, including a clinical interpretation of those deviations
Stability • Sensor stability, ie, the accuracy with respect to the sensor lifetime • Calibration stability, ie, the accuracy with respect to the time after calibration
Alert reliability • Threshold alert reliability, ie, the ability to correctly alert the user when glucose concentrations cross predefined hypo- and hyperglycemic thresholds, including the measurement range limits of the CGM system • Predictive alert reliability, ie, the ability to correctly predict the crossing of predefined hypo- and hyperglycemic thresholds
Technical reliability • Sensor survival, ie, the ability of a CGM sensor to correctly function until the end of its specified use lifetime • Data availability, ie, the ability of a CGM system to provide the expected number of glucose measurements without interruptions • Description of device deficiencies, ie, malfunctions and use errors and their potential to cause an adverse event • Lag time between CGM and comparator data
Adverse device effects • Occurrence of adverse events related to or caused by the examined CGM systems
User satisfaction • Experience of the users with the CGM system, eg, in terms of ease-of-use, functionality, reliability, and wear comfort

Abbreviation: CGM, continuous glucose monitoring.

When designing a study for clinical CGM performance evaluation, there exist various elements, such as the subject population, the testing protocol, the selection of comparator samples and methods, and the definition of statistical analysis, that all provide several options that must be carefully considered. Here, it has been established that the choice of study design and evaluation procedures affect the observed accuracy leading to a range of varying accuracy results reported for each CGM system. ⁵ This indicates that there is no single level of performance for any given CGM system, but potentially as many levels as there are performance studies with any specific CGM system. Consequently, it is only possible to compare different systems when they are tested in the same study or if they are tested according to a standardized protocol.^6,7

This issue of limited comparability persists until today, despite early calls for standardized clinical performance evaluations in 2007, ⁸ which resulted in the Clinical and Laboratory Standards Institute guideline POCT05, providing specifications for study and evaluation procedures. This guideline was introduced in 2008 ⁹ and revised in 2020. ⁴ While the guidance document provides a comprehensive overview of the statistical evaluation of the data, it does not provide specific guidance regarding some important details of study design, in particular the testing procedures. Therefore, it is rarely used as a complete evaluation protocol, but rather individual elements of the guideline can be found in different study protocols. In this review, we found that only 11 studies published since 2010 cite the POCT05 guideline (first or second edition), suggesting a limited adoption.

A more concise step toward standardization was made by the U.S. Food and Drug Administration (FDA) in 2018 with the definition of acceptance criteria for “integrated” CGM (iCGM) systems.^10,11 However, the FDA requirements are also vague regarding the design of the clinical studies and only state that “clinical data must be obtained from a clinical study designed to fully represent the performance of the device throughout the intended use population and throughout the measuring range of the device” (special control 1 [iii]). ¹⁰ To obtain a Conformité Européenne (CE) mark, which is necessary for product launch in most European countries, no dedicated requirements for the clinical CGM performance evaluations and its outcomes have been published. An excellent and detailed review on the regulatory approval of CGM systems in Europe, the United States, and Australia has recently been published by Pemberton et al. ¹²

Existing review articles on CGM accuracy found that the comparability of results is limited because of different procedures. However, these reviews have not characterized the different aspects of study design and performance presentation in detail.^5,12-15 This scoping literature review therefore offers a comprehensive overview of the relevant elements of clinical CGM performance evaluations published between 2002 and 2022. In addition, we provide recommendations on how clinical CGM performance studies should be presented in original articles, beyond general reporting recommendations such as CONSORT or STROBE. ¹⁶ This article is intended to support the efforts of the Working Group on CGM (WG-CGM) of the Scientific Division of the International Federation of Clinical Chemistry and Laboratory Medicine (IFCC) in their aim to define a standardized procedure for the clinical performance evaluation of CGM systems.^7,17 A further goal of this review is to support journal editors, reviewers, and regulators who are judging the quality of reports of clinical CGM performance evaluations.

The article is structured as follows. First, the methodology of the literature search and selection process are presented. Next, the review examines several aspects of clinical CGM performance studies, including general study design, use of CGM systems, comparator measurements, testing protocol, and data analysis and statistical performance evaluation. Each aspect is thoroughly reviewed and discussed, and recommendations for reporting are provided. A table with all studies included in this review together with their most relevant features and results is provided in the Supplemental Material.

Methodology of Literature Search

This scoping review is based on a systematic literature search conducted on the MEDLINE database. Therefore, this review only includes study reports published in scientific journals and excludes reports published outside of these sources, such as those from regulatory agencies or manufacturer white papers. Additionally, we should explicitly state that, because of the abundance of studies, the authors of the study were not contacted for additional data. Therefore, only information published in the main text or supplemental material of the peer-reviewed manuscripts was considered.

The search term (“CGM” OR “continuous glucose sensor” OR “continuous glucose monitoring” OR “continuous subcutaneous glucose monitoring” OR [“flash” AND “glucose”]) AND (“accuracy”[Title] OR “performance”[Title] OR “evaluation”[Title] OR “comparison”[Title]) NOT “review”[Title] NOT review[Filter] NOT (Animal[Filter]) was defined, using a period of search from 2002 to 2022. Titles and abstracts of all retrieved articles were screened. From the first stage of the selection process, we excluded manuscripts not available in the English language or if they did not contain a CGM performance evaluation. In the second selection stage, the remaining studies were inspected in more detail and excluded if one or more of the following criteria were met:

1. The manuscript was in a short format, such as a brief report or letter to the editor.

2. The manuscript described a study investigating CGM accuracy in a specific subject population aside from people with diabetes under routine care. We thus excluded studies conducted exclusively on infants, subjects with comorbidities, pregnant women, and subjects without diabetes. Furthermore, we excluded studies with patients in intensive care and patients undergoing surgery.

3. The manuscript described a study evaluating a CGM system not designed to measure glucose in the interstitial fluid (ISF).

4. The manuscript described a feasibility study intended for proof-of-concept testing including studies conducted for calibration algorithm development.

5. The manuscript described a study with observation periods for comparator measurement shorter than two hours.

6. The manuscript described a study investigating CGM accuracy under specific accompanying activities, eg, during ethanol consumption or glucagon injection.

Suitable manuscripts not found in the systematic search were also identified using the “similar articles” function and screening the references of included manuscripts (non-systematic search).

Studies included in the final review were systematically summarized by at least two researchers. If a single study was presented in multiple manuscripts, then these manuscripts were grouped for evaluation.

The process of article selection is depicted in Figure 1. The initial search yielded 603 results, of which 377 manuscripts were excluded in the first selection stage. The remaining 226 manuscripts were assessed and 112 additional manuscripts were excluded in the second selection stage, yielding 114 manuscripts. Another 30 manuscripts identified from the non-systematic search were included, yielding a total of 144 manuscripts.^18-161 This high number of manuscripts identified in the non-systematic search is the result of the inconsistent terminology for “continuous glucose monitoring system” in the titles, where various slightly varying synonyms, not covered by the systematic search term, were used. Additionally, the titles of several manuscripts only included the brand name of the tested devices. After summarizing manuscripts using data from the same study, a total of 129 studies were included in this scoping review.

Figure 1.

Schematic depiction of the study selection process.

The selected articles showed a trend toward an increasing number of publications over the years with a peak between 2017 and 2019 (Figure 2).

Figure 2.

Distribution of the years of publication of the 129 performance studies included in this review.

General Study Design

Subject Population

The number of subjects included in a clinical CGM performance study is one of the most crucial aspects of subject population selection. Unlike for blood glucose monitoring systems (BGMS),^162,163 no requirements on sample size for market approval of CGM systems have been put forward by regulatory agencies. In this context, the review by Pemberton et al ¹² found that the number of adult subjects enrolled in studies for CE marking of CGM systems prescribed in UK primary care ranged from 57 to 316.

The appropriate sample size depends on the specific purpose of the study, the expected accuracy of the system, and potential minimum accuracy requirements. The studies included in this review had population sizes ranging from 6 to 318 subjects with a median of 30 and an interquartile interval between 16 and 63 subjects. A more detailed distribution of population sizes of the studies included in this review is shown in Figure 3, which demonstrates a large variability in sample size although feasibility studies, which typically have comparatively small sample sizes, were excluded. Despite being a crucial aspect of study design that should be reported, a sample size calculation or at least a justification of the chosen sample size was only provided in 14.7% of the articles.

Figure 3.

Distribution of the total number of subjects in the 129 studies included in this review.

Another important aspect of the selected population is the diabetes type of the subjects which depends on the intended use of the examined CGM system and affects the characteristics of the comparator data. Most of the studies (65.9%) only included subjects with type 1 diabetes, which can be explained by the fact that this facilitates the collection of comparator data containing a wide range of blood glucose concentrations. A total of 30.2% of studies included people with both type 1 and type 2 diabetes, where the percentage of people with type 1 diabetes from the entire population had a large variability ranging from 8% to 99%, with a median of 73.5%. The populations of the remaining 3.9% of studies consisted of only people with type 2 diabetes or a mixture of people with diabetes (type 1 and type 2) and without diabetes.

The next aspect to be reviewed is the subjects’ age. Here, most studies (76.0%) included adults only, and the remaining studies included both adults and minors or minors only. This can be explained by the increased complexity and ethical challenges when performing clinical studies with minors. The influence of subject age on the testing protocol will be discussed in a later section. In total, 23.3%, 17.8%, and 7.8% of the reviewed studies included adolescents (age 12-17 years), children (age 6-11 years), and/or young children (age <6 years), respectively.

Finally, the median percentage of male subjects out of the total population was 54.5% with values ranging from 20.0% to 91.7%. However, 10.9% of studies did not report the subject sex, at all. In terms of ethnicity, only 24.8% reported the ethnicity of the subjects.

Number of Study Centers

Less than half (41.1%) of the studies included in this review had a multicenter design. As these studies allowed the recruitment and data collection at multiple study centers under the same protocol, the median population size of multicenter studies was 71 (interquartile interval 38-90), compared with 20 (interquartile interval 14-30) in monocenter studies. Because of potential systematic differences between glucose analyzers for comparator measurement, recently discussed by Pleus et al, ¹⁶⁴ an important aspect to consider in multicenter studies is the harmonization of devices for comparator measurement. However, only 9.4% of multicenter studies using laboratory analyzers described any measures or procedures to ensure harmonization.^{19-22,29,33,38,42,43} These studies were almost conducted by the Diabetes Research in Children Network, where blood samples from all study centers were transferred and analyzed in a central laboratory. In multicenter studies, we recommend reporting the approach for harmonization of comparator data as unharmonized comparator data could affect the observed performance.

Funding Source

A review from 2017 on the association between industry sponsorship and research outcome concluded that “sponsorship of drug and device studies by the manufacturing company leads to more favorable efficacy results and conclusions than sponsorship by other sources.” ¹⁶⁵ The role of sponsoring by CGM system manufacturers was therefore examined in this review, where it was found that 58.1% of studies were at least partially sponsored by a CGM system manufacturer. A study was considered manufacturer-sponsored if it was explicitly declared as fully or partially funded by a manufacturer of the tested CGM systems, or if the study devices of at least one CGM system were provided by the manufacturer free of charge. Furthermore, if studies were fully or partially funded by a manufacturer, it was often not clear to what extent study design, procedures, and data evaluation were influenced by the manufacturer. This emphasizes the need for detailed and transparent reporting of all study aspects so that potential biases in the study design can be identified. Furthermore, we recommend describing the influence of the funding source on the study planning and implementation process, as well as on the evaluation and interpretation of data. Examples of such a description were found in the articles by Kropff et al, ⁹² Ólafsdóttir et al, ¹¹¹ and Piona et al. ¹²²

Use of CGM Systems

The total number of tested CGM systems, calculated as the sum of the number of different systems tested in each study, was 178. However, of all reviewed studies, only 29.5% tested more than one, and up to four CGM systems in parallel (see Figure 4). These studies, often referred to as head-to-head studies, are of particularly high value because they are currently the only approach to genuinely evaluate and compare the performance of two or more different CGM systems, given that all factors influencing their performance are equal. Such studies have been performed with earlier-generation CGM systems, eg, Kovatchev et al, ⁴⁴ Damiano et al,^69,81 and Freckmann et al.^70,117,132 However, to date, no head-to-head studies comparing the performance of most current CGM systems, including Dexcom G7, FreeStyle Libre 2 and 3, and Medtronic Guardian Sensor 4, have been published. Dexcom G6 was only examined in one study with other CGM systems (FreeStyle Libre and Medtronic Enlite 2 in combination with MiniMed 640G, systems not worn simultaneously) during a summer camp for children and adolescents with type 1 diabetes. ¹⁵⁰

Figure 4.

Distribution of the number of CGM systems that were tested simultaneously in the 129 studies included in this review.

In comparison with head-to-head designs, inserting multiple sensors of the same CGM system to the same subject is more common, because it is one of the few approaches to examine sensor precision. Furthermore, this approach can increase the number of data points for analysis without requiring more subjects. Here, 39.5% of studies used more than one sensor of at least one of the tested systems simultaneously. Depending on the study duration, multiple sensors of the same CGM system can also be worn consecutively, which is often the case in studies mainly conducted in a free-living setting.

In this context, the use of sensors from different manufacturing lots should be addressed. Here, only six studies^{71,91,103,115,130,132} provided information on the number of sensor lots used, although the POCT05 guideline recommends testing on at least three lots. ⁴ This is important because it might be possible that sensor performance might change over time due to varying manufacturing conditions. We therefore also recommend that investigators provide information on the used manufacturing lots.

Out of the 178 different CGM systems tested, the principal manufacturers Abbott Diabetes Care, Dexcom, and Medtronic had similar percentages of 26.4%, 25.8%, and 29.8%, respectively, with the remaining 18.0% of tested CGM systems manufactured by other companies, eg, Senseonics or A. Menarini Diagnostics. In this context, we recommend reporting the source of the examined devices and sensors. In particular, in manufacturer-independent studies, it should be stated where the sensors and devices were purchased from (ie, directly from the manufacturer or through regular channels from the market).

The sensors of the studies included in this review were most commonly inserted on the abdomen, followed by the upper arm, and we recommend that studies clearly state whether the sensors were inserted by the subjects themselves, under the supervision of study personnel, or directly by the study personnel.

Calibration is a major topic concerning CGM accuracy even though factory-calibrated systems are becoming more prevalent. In most studies, the calibration was performed according to the instructions of the manufacturers, although in some cases this was not clearly reported. A few studies, eg, by Zueger et al ⁶⁴ and Bailey et al, ⁸⁰ examined the effect of additional and/or timing of calibrations on CGM accuracy.

Comparator Measurement Approach

The topic of comparator method selection and establishment of traceability in clinical CGM performance evaluations has been discussed in a previous article by the IFCC Working Group on CGM. ⁷

CGM sensors are typically placed in the ISF and use algorithms to calculate blood glucose concentrations; however, the compartment (typically capillary or venous blood) for which CGM systems display glucose concentrations is often not specified by the manufacturer and not standardized. Because of this and the impossibility to perform comparator measurements in the ISF, three possible compartments for comparator glucose measurements were used in the reviewed studies: capillary, venous, and arterialized-venous, where venous blood is sampled from a specifically heated arm or hand. Here it should be emphasized that there are physiological differences between these compartments that can influence glucose concentrations and thus the observed performance of the CGM systems. The total number of used comparators, calculated as the sum of the number of different comparators used in each study, was 167. The percentages of comparators using capillary, venous, and arterialized-venous blood are shown in Figure 5.

Figure 5.

Relative frequencies of capillary, venous, and arterialized-venous sampling of the 167 comparators found in this review.

Capillary blood is typically sampled via finger-prick, which requires no preprocessing when glucose concentrations are measured with a BGMS and the measurements can be carried out in a free-living setting by the subjects themselves (the study setting is discussed in the next section). In this setting, capillary blood glucose concentrations have been used for therapy decisions before the advent of CGM and are also used for CGM system calibration. Even in non-adjunctive CGM systems, capillary measurements are still recommended when CGM values are implausible or temporarily unavailable. A disadvantage of using capillary comparator measurements is the repeated finger-pricks during phases of frequent sampling. Furthermore, a BGMS usually has a reduced measurement accuracy in comparison to laboratory-grade glucose analyzers, partially caused by pre-analytic errors, keeping in mind that any inaccuracy in the comparator measurements is fully attributed to the CGM system. In most studies where capillary comparator measurements were obtained, a BGMS was used (85.9%). Only few studies (14.1%) used laboratory-grade glucose analyzers with capillary blood, which can be more accurate than BGMS but also require a larger sample volume, thus impeding frequent sampling.

In almost all reviewed studies where venous and/or arterialized-venous blood was collected, sampling was limited to in-clinic visits, because obtaining these specimens requires venous access as well as pre-analytical processing, such as centrifugation to separate plasma (except the study by Francescato et al, ⁵⁹ in which venous samples were collected at the subjects’ homes). One advantage of venous sampling is that samples can be collected frequently without repeated skin penetration. Glucose concentrations are then measured with a laboratory analyzer, of which a Yellow Springs Instruments (YSI) analyzer based on the glucose-oxidase technology was used most often (65.9%). Laboratory analyzers from other manufacturers were, for example, based on the hexokinase enzyme. Here it should be mentioned that it has been shown that there are systematic differences between laboratory analyzers, which can affect the observed CGM accuracy.^164,166 We therefore recommend examining and reporting the compliance of all comparators with the recently proposed analytical performance specifications of comparators used in clinical CGM performance evaluations. ¹⁶⁴ An example of such an examination can be found in a recent study by Kölle et al, ¹⁶⁷ which was excluded from this review because it was published in 2023. We further recommend describing the sample compartment, sample processing, and comparator glucose analyzers in sufficient detail (eg, Guerci et al, ²⁶ Francescato et al, ⁵⁹ and Yan et al ¹⁴⁴ ).

Testing Procedures

In principle, comparator data can be collected either in an in-clinic or free-living setting, keeping in mind that study endpoints such as accuracy and alert reliability can only be evaluated during times of comparator data collection. The choice of setting affects the method for comparator measurement, discussed in the previous section, as well as the amount, frequency, and characteristics of comparator data. The free-living setting represents real-life use of the CGM systems and can provide comparator data (using a BGMS) on every day of the sensor lifetime because subjects are typically instructed to measure multiple times per day. In contrast, in-clinic visits are typically sporadic and allow testing under controlled conditions with frequent collection of comparator samples, use of laboratory analyzers for comparator measurement, and deliberate manipulation of glucose concentration dynamics under the supervision of a study physician. Considering only the setting for which data were analyzed and presented, this review found that most studies (55.0%) only used data collected in an in-clinic setting, whereas 21.7% of studies only evaluated data collected in a free-living setting. The remaining 23.3% of studies used data collected in both settings and the data were either pooled or analyzed separately.

In-Clinic Setting Protocol

The in-clinic setting protocol for the collection of comparator measurements can involve multiple separate visits of varying durations. Of the 101 studies that included in-clinic data collection, 46.5% only involved a single visit. Two, three, and four or more visits occurred in 23.8%, 15.8%, and 13.9% of studies, respectively. Naturally, the number of visits depends on the sensor lifetime. While early-generation sensors had shorter maximum lifetimes of two to three days that necessitated only one or two visits, lifetimes of more than a week found in newer-generation sensors require several visits or a stationary admission of subjects to examine accuracy over the complete sensor lifetime. In this context, it should be mentioned that CGM performance can be different depending on the sensor wear time, with performance typically being reduced at the beginning and end of the sensor lifetime. The chosen distribution of in-clinic visits with respect to the day of sensor wear can thus have an impact on the observed performance.

Examining the duration of visits, it was found that 26.7% of studies with in-clinic settings included visits lasting for more than 14 hours, which typically requires stationary admission of the subjects and allows data collection during the night. Considering studies with in-clinic visits of 14 hours or less, the median duration was 7.5 hours with an interquartile interval between 5 and 12 hours. A detailed overview on the durations of in-clinic visits is provided in Figure 6.

Figure 6.

Duration of in-clinic visits of individual studies separated by visits shorter or equal to 14 hours in hours (left) or longer than 14 hours in days (right).

Another important aspect of the in-clinic protocol is the schedule of comparator sample collection. Here it was found that 82.2% of studies with in-clinic visits had at least one period where comparator measurements were carried out every 20 minutes or more frequently. This threshold was chosen as it allows the estimation of blood glucose rate of change (RoC) and allowed the inclusion of the numerous studies that reported the sampling interval as 15±5 minutes. Furthermore, it was common to have a varying sampling schedule or to adapt the sampling interval depending on the glucose concentration. In particular, some of the sampling intervals were reported to be as short as five minutes. Here, some authors voiced their concern about the possible statistical interdependency of samples collected more frequently than every 15 minutes.^73,92,112 However, to the best of the authors’ knowledge, a more thorough statistical examination of those concerns is currently missing. Another concern is the total blood volume extracted from the subjects, for which the POCT05 guideline provides recommendations. ⁴

The last aspect to consider in this section is the deliberate manipulation of glucose concentrations. This was the case in 73.3% of studies that included in-clinic sessions, whereas 18.8% explicitly stated that no deliberate manipulation took place and 7.9% provided no information. The predominant methods to manipulate glucose concentrations were the adaptation of insulin dosing and/or timing as well as providing specific meals, at least one of which occurred in over 90% of studies with in-clinic sessions and glucose manipulations. Less common was the use of exercise, possibly in combination with meal and insulin manipulation, occurring in 21.6% of those studies. The least common was the use of glucose clamping techniques, which was reported in only five studies. Here glucose concentrations were precisely regulated by external, intravenous infusion of glucose and insulin.^{31,44,51,57,110} More details are provided in Figure 7. When CGM systems are tested with minors, it is common to reduce the number of visits and/or visit duration and to refrain from deliberate glucose manipulations.

Figure 7.

Number of studies employing various approaches for deliberate glucose manipulations.

In general, the level of reported details on the applied manipulation procedures varied greatly between studies. However, in the vast majority of the studies, the procedures were not described in sufficient detail to allow the independent development of a similar procedure. We thus encourage authors to provide sufficiently detailed descriptions of these procedures.

Data Analysis and Statistical Performance Evaluation

Comparator Data Characterization

Considering the previously described incomplete reporting of glucose manipulation procedures and the well-known impact of glucose concentrations and their dynamics on CGM accuracy, it is of utmost importance that the distribution of comparator data is sufficiently characterized. Such a characterization is the most suitable way to judge whether the performance results of studies, in particular accuracy, are comparable. Furthermore, this characterization should be considered the result of the general study design, comparator method selection, and testing procedures and should thus be reported and discussed independently from the endpoints of the study (Table 1).

There are two main facets of the comparator data characteristics: the distribution of glucose concentrations themselves and the distribution of glucose concentration RoCs. When it comes to the distribution of glucose concentrations, this review found that only 55.0% of the included studies provided a suitable description. This was defined as reporting the percentages of comparator data points in at least three different glucose ranges. Here it should be pointed out that several recent studies reported the distribution of CGM values instead of comparator values.^{90,96,123,127,131,138,140,155,158,160}

However, this cannot be recommended as the CGM value distribution is dependent on CGM accuracy and thus impairs the comparison of testing procedures among studies. ¹⁶⁸ The most common choice for the glucose ranges were the established time in range (TIR) bins <70 mg/dL (<3.9 mmol/L), 70 to 180 mg/dL (3.9-10.0 mmol/L), and >180 mg/dL (>10 mmol/L), used in 62.0% of studies that reported a distribution. The remaining studies used various bins, thus making comparisons of different studies very difficult. In particular, it was not sensible to systematically assess whether recommendations of the POCT05 of having >8% of comparator values <80 mg/dL (4.4 mmol/L) and >5% of comparator values >300 mg/dL (>16.7 mmol/L) ⁴ were followed.

An in-depth examination of the 44 studies reporting comparator values in the TIR bins found a great variability among studies with a median percentage of comparator values of 7.3% <70 mg/dL (<3.9 mmol/L), 61.4% between 70 and 180 mg/dL (3.9-10.0 mol/L), and 31.4% >180 mg/dL (>10 mmol/L). More details are displayed in Figure 8a.

Figure 8.

(a) Boxplots of percentages of comparator values in the TIR bins from 44 studies. The whiskers indicate minimum/maximum percentages. If multiple comparators were used, then only the distribution of venous measurements during in-clinic visits was analyzed. If the distribution was specified for each CGM system separately, the data were pooled. (b) Estimated mean absolute rate of change (MARoC) of comparator glucose concentrations from results reported in 21 studies.

The comparator glucose concentration RoCs have an equally important influence on the observed accuracy of a CGM system. However, a meaningful calculation of RoCs is only possible if comparator measurements were performed with a sufficiently high frequency. As mentioned before, this review chose a threshold of 20 minutes. Here, it was found that out of the 83 studies with a sampling interval of 20 minutes or less, only 21 (25.3%) reported a distribution of comparator RoCs, with some studies only reporting the distribution of CGM RoCs which cannot be recommended as explained before. Among the 21 studies that reported a comparator RoC distribution, the bins used were highly inconsistent, which compromises the comparability of the results. To overcome this, the reported comparator RoC distributions were used to estimate the mean absolute rate of change (MARoC) in mg/dL/min (mmol/L/min) of comparator glucose concentrations (the methodology is described in the Supplemental Material). This is a single number that can characterize the extent of fast glucose changes, both positive and negative, in the studies. We therefore propose that future studies report this parameter alongside a suitable description of the comparator RoC distribution.

The results are depicted in Figure 8b and show that most MARoCs can be found between 0.7 and 1.0 mg/dL/min (0.039 and 0.056 mmol/L/min), with some studies reaching considerably higher MARoCs above 1.3 mg/dL/min (0.072 mg/dl/min).

When reporting the distribution of comparator glucose concentrations and their RoCs, it should be noted that these two quantities have a temporal association and it is quite difficult to visualize the actual glucose profiles that occurred during in-clinic sessions by just inspecting the distributions. In this context, we found that 37.6% of studies with in-clinic sessions reported the time course of glucose concentrations of at least one subject to illustrate the glycemic conditions during the in-clinic sessions.

Data Exclusion

There may be various reasons to exclude data from the evaluation, both recorded with the CGM systems and the comparators. These include deviations from the testing protocol and technical malfunctions. Furthermore, data exclusion can have a considerable impact on the resulting performance and should therefore be described in detail.

CGM-Comparator Pairing

Typically, CGM systems do not automatically store glucose concentrations every minute but use fixed recording intervals of 5 to 15 minutes, depending on the system used. Additionally, there are intermittently scanned CGM systems, for which measurements can be recorded at any time. This creates multiple approaches for pairing the recorded CGM values with comparator measurements, the choice of which has been shown to affect performance results. ¹¹⁷ We therefore recommend describing the method for pairing in sufficient detail, also because most studies (51.9%) did not provide such a description. If the pairing method was described, then the following options and their prevalence were observed (considering that more than one option could be used, depending on the CGM systems):

• Pairing the CGM value that was recorded closest in time, ie, either before or after, to the comparator value. This approach was the most common and made up 45.5% of all pairing methods used.

• Pairing the CGM value that was recorded simultaneously or after the comparator values. This approach was used in 27.3% of all pairing methods.

• Using linear interpolation to calculate and pair a CGM or comparator value that is concomitant with the recorded CGM or comparator value. This approach was used in 21.2% of all pairing methods.

• Pairing the CGM value that was recorded simultaneously or before the comparator values. This was the least common approach and was used in 6.1% of all pairing methods.

Statistical Accuracy Evaluation

The most important endpoint of a clinical CGM performance evaluation is accuracy, where it is common to distinguish between point accuracy, ie, the extent of agreement between CGM and comparator values at single points in time, and trend accuracy, ie, the ability of the CGM system to correctly indicate the direction and magnitude of glucose concentration changes. Furthermore, a distinction between analytical and clinical accuracy can be made (Table 1).

Numerous articles have already discussed the various parameters and approaches that can be used to characterize point accuracy.^{5,12,168,169-171} Most of the approaches describe analytical accuracy, and we refer to those articles for detailed discussions of various approaches. This review identified the following nine approaches, the prevalence of which is summarized in Figure 9:

Figure 9.

Parameters and methods to characterize point accuracy and their prevalence identified in the 129 studies included in this review.

• Mean and/or median absolute relative difference (ARD) (94.6%) between CGM and comparator measurements. Despite some criticism,^5,172 the mean ARD (often abbreviated as MARD) remains one of the most frequently used parameters to assess point accuracy. Here it should be mentioned that mean and median ARDs have a different statistical interpretation and cannot be compared with each other. Studies where both mean and median ARDs have been reported show that median ARDs can be between 2.4 ¹⁵⁶ and 7 ¹⁹ percentage points lower than mean ARDs. Another aspect of mean/median ARD calculation is whether all CGM-comparator data pairs are used at once (often referred to as aggregate mean/median ARD), or whether the mean/median ARDs of individual sensors are calculated first, and then a single number is calculated as mean/median of these sensor-specific mean/median ARDs. However, while the differences in mean/median results between these two approaches are typically small, the associated measure of variability (standard deviation or interquartile range) for the aggregate mean/median ARD is typically much larger. The evolution and in particular a trend toward a decrease of MARDs over time are shown in Figure 10. A table of individual MARDs and the associated CGM systems is provided in the Supplemental Material.

• Error grid analysis (84.5%). It is the only established approach to characterize clinical accuracy, indicating the share of deviations between CGM and comparator values in predefined zones of clinical impact, ranging from none to severe. To this date, four different error grid analyses have been proposed and their usage over time is demonstrated in Figure 11a. Clarke et al, ¹⁷³ consensus, ¹⁷⁴ and surveillance ¹⁷⁵ error grids focus on clinical point accuracy, whereas the continuous glucose error grid analysis (CG-EGA) ¹⁷⁶ was specifically developed for CGM systems and incorporates both point and trend accuracy. An increase in the use of the consensus error grid can be observed after 2015, which is most likely due to its incorporation in the standard for BGMS published by the International Organization for Standardization (ISO). ¹⁶² Conversely, the use of the CG-EGA has declined over time. Here it should be mentioned that the extent to which CG-EGA results were reported varied greatly between articles. In general, the fact that four similar methods to characterize clinical accuracy are used to this day exemplifies the problem of comparing study results with each other.

• Agreement rates (55.8%). They indicate the share of CGM measurements within certain limits, eg, ±15 mg/dL (±0.83 mmol/L) or %, of comparator measurements. If a single agreement rate is calculated from all CGM-comparator pairs over the entire glucose range, then it is very common to switch from relative to absolute difference limits at certain thresholds of lower glucose concentrations to account for the glucose concentration dependency of CGM-comparator deviations. The use of these different thresholds has changed over time which is demonstrated in Figure 11b. The figure demonstrates that the use of 100 mg/dL (5.55 mmol/l) as a threshold has become dominant from 2015 onward which coincides with the update of the ISO 15197 standard for BGMS using this threshold. ¹⁶² The use of 80 mg/dL (4.44 mmol/L) is recommended in the POCT05 guideline, ⁴ and the FDA iCGM requirements only use relative differences (“none” in Figure 11b). ¹⁰

• Mean and/or median bias (both absolute and/or relative) (45.0%) between CGM and comparator measurements, often combined with a Bland-Altman analysis (22.5%). These parameters can indicate whether CGM measurements systematically over- or underestimate comparator blood glucose concentrations, but bias alone can also mask a large amount of imprecision.

• Mean and/or median absolute difference (37.2%) between CGM and comparator measurements. These parameters are similar to mean/median ARDs and are commonly used to characterize the deviations in low glucose ranges.

• Analysis of association between CGM and comparator measurements using correlation and/or regression (31.8%). This approach was very common for early-generation CGM systems to demonstrate their general ability to provide information on glucose concentrations. However, as systems have improved, this analysis has mostly become obsolete.

• Variability in accuracy from sensor to sensor (26.4%). This is a very important aspect of CGM point accuracy and can be characterized, eg, by the distribution of individual sensor MARDs.

• Precision (24.8%) in the form of the mean or median paired ARD between two sensors and/or other approaches. Note that this parameter can only be calculated if at least two sensors from the same system are worn in parallel.

• Concurrence tables (1.6%). They divide the CGM and comparator data in various bins across the glucose range and indicate the number of CGM-comparator pairs lying within each possible combination of CGM and comparator bins. Although this approach was suggested in the first and second editions of the POCT05 guideline,^9,4 it was only reported in the manuscripts of two studies.^139,158

Figure 10.

MARDs reported in the 129 studies included in this review with respect to their year of publication (n=189).

The data points were spread out within each year to better reflect the number of reported MARDs. Included were only MARD results reported over the full glucose range. If MARD values in specific studies were reported for different comparator compartments, calibration algorithms, or insertion sites, then they were included separately.

Figure 11.

Usage of different error grids (a) and thresholds for agreement rates (b) expressed as share of all error grids/thresholds in specific time periods.

In both panels, the time intervals were chosen so that an approximately equal number of error grid/threshold usages occur in every time interval. In panel (a), the four types of error grids are the Clarke error grid, introduced by Clarke et al, ¹⁷² the consensus error grid, introduced by Parkes et al, ¹⁷³ the CG-EGA, introduced by Kovatchev et al, ¹⁷⁵ and the surveillance error grid, introduced by Klonoff et al. ¹⁷⁴ In panel (b), the thresholds for switching between absolute (below threshold) and relative (above threshold) differences are characterized. “None” means that only relative differences were used.

Only 27 studies reported any trend accuracy results, the vast majority were published in 2015 or earlier. The rate error grid, a part of the previously mentioned CG-EGA, ¹⁷⁶ was most common and used to generate 51.9% of trend accuracy results. Two other approaches for trend accuracy assessment could be identified: the calculation of rate deviations (29.6%) analogous to deviations in the context of point accuracy and concurrence tables (18.5%), both of which are recommended in the POCT05 guideline. ⁴

Another important aspect of the statistical accuracy evaluation is the reporting of confidence intervals of the previously mentioned parameters, which has gained relevance since the FDA iCGM requirements use confidence intervals to define their minimum acceptance criteria. ¹⁰ This review found that less than 25% of studies reported confidence intervals on any of the previously mentioned accuracy parameters. We recommend that all point estimates of accuracy parameters are reported with a suitably chosen and calculated confidence interval. For that, the specific properties of the data from CGM performance studies have to be taken into account, such as the previously mentioned interdependency of closely sampled data and clustering of data within CGM sensors. ¹⁷⁷

Other Aspects of CGM System Performance

Closely related to point accuracy is the stability of the sensors. In particular, the stability of accuracy over the sensor lifetime was reported in 38.8% of studies, often by stratifying one specific accuracy parameter, eg, MARD or agreement rate by the sensor wear day. The calibration stability, ie, an analysis of the accuracy with respect to the time after calibration, was only reported in two studies.^108,156

Another aspect closely related to accuracy is the reliability of hypo- and hyperglycemic threshold alerts which has been reported in 24.8% of studies, respectively. Here, it was common to calculate the percentage of correct alarms also known as sensitivity (CGM values below/above a certain threshold concurrent with the comparator measurement below/above the same threshold) and correct detections also known as specificity (comparator measurement below/above a certain threshold concurrent with a CGM value below/above the same threshold), recommended by the POCT05 guideline. The reliability of predictive threshold alerts was assessed in 4.7% of studies. An analysis of the correlation between accuracy parameters and subject characteristics, such as body mass index, age, or duration of diabetes, was carried out in 10.1% of studies.

The technical reliability of the CGM system in terms of sensor survival was reported in 28.7% of studies. This can be characterized by several approaches, such as the percentage of sensors that reached the end of their lifetime or the survival probability calculated with the Kaplan-Meier method. The availability of data, ie, the ability of the CGM system to provide the expected number of glucose data without interruptions was reported in only 6.2% of studies. Both sensor survival and data availability were often accompanied by a description of device deficiencies that occurred during the study. Finally, the lag time between CGM and comparator data was reported in 17.8% of studies. However, a variety of different methods to determine the lag time was used.

To provide an indication of CGM system safety, any adverse events and adverse device effects that were related to the investigated CGM systems during the study were reported in 46.5% of studies. A related aspect is the satisfaction of the users with the CGM systems, which is typically assessed through user questionnaires inquiring on features such as wear comfort, pain of sensor insertion, and user-friendliness. This was reported in 24.0% of studies.

Finally, a particular strength of studies examining multiple CGM systems worn by the same subjects is the ability to compare various therapy parameters such as the TIR or glucose management indicator among CGM systems. Examples of this analysis can be found in the studies by Bonora et al, ⁹⁵ Boscari et al, ¹⁵⁴ and Yeoh et al, ¹⁶¹ concluding that, in particular, the time below range can be considerably different between CGM systems.

Recommendations for Reporting

A summary of our recommendations for reporting the previously discussed aspects of general study design, CGM system use, comparator measurement, testing procedures, and data analysis/statistical performance evaluation is provided in Table 2.

Table 2.

Recommendations for Reporting the Design and Results of a Clinical CGM Performance Evaluation by the WG-CGM Established by the IFCC.

Recommendations for reporting a clinical CGM performance evaluation
General study design • Ethical approval • Inclusion/exclusion criteria including possible interfering medications • Subject population characteristics  - Demographic (age, sex, BMI, HbA1c, and ethnicity)  - Diabetes type  - Diabetes treatment regime • Sample size justification • Number of investigational sites (mono- or multicenter) • Role of funding source
CGM system use • Number of systems • Number of sensors used simultaneously • Number of sensors used consecutively • Number of planned sensor wear days and intended sensor lifetime • Manufacturing lot(s) of sensors • Source of CGM devices and sensors • Protocol for insertion and possible replacement of sensors in case of failure • Sensor insertion location, within or outside intended use • Calibration protocol, including the source of calibrators used and concentration of the calibrators
Comparator measurement approach • Sampling compartment (capillary, venous, and arterialized-venous) • Details of arterialization protocol (if applicable) • Glucose measurement method and device for comparator data • Procedures for harmonization of comparator measurements (if applicable) • Compliance with analytical performance specifications proposed by Pleus et al 163
Testing procedures • Study setting (in-clinic/free-living) • Number of in-clinic sessions/days in relation to sensor lifetime • Duration of in-clinic sessions • Sampling schedule of comparator measurements • Number of planned comparator measurements per subject and in total • Detailed description of glucose manipulation protocol
Data analysis and statistical performance evaluation • Comparator data characteristics a  - Glucose concentrations  - RoC  - Time course of comparator glucose concentrations • Exclusion of data • Pairing of CGM and comparator data • Point accuracy a • Trend accuracy a • Threshold alert reliability a • Predictive alert reliability a • Technical reliability a • Occurrence of adverse device effects a • User satisfaction (if applicable) a

Abbreviations: CGM, continuous glucose monitoring; BMI, body mass index.

^aThe technical details on how to report these aspects lie beyond the scope of this review and will be the subject of a future publication.

Limitations of This Review

A limitation of this review is that a more concise, quantitative analysis of the impact of various study design options on the observed performance was not carried out, because the number of studies examining the exact same CGM system was limited. Furthermore, the inclusion criteria of selected studies were chosen fairly broad, which might have contributed to the large observed variability. If the review had been limited to studies, eg, published within the last ten years or with a large subject population, more homogenous study designs would probably have been found.

Conclusion

This scoping review analyzed reports of 129 clinical studies published between 2002 and 2022 evaluating the performance of CGM systems. In particular, various aspects of study design were analyzed. We determined that, as a group, these studies are characterized by two important factors that interfere with attaining a clear idea of individual system performance as well as comparing the performance between systems: (1) the various options for protocol design have the potential to significantly impact study outcomes and cause inconsistency of study design and (2) crucial information about the study design is frequently not reported, jeopardizing the integrity and validity of the study findings.

The two goals of this review were first to assess the clarity with which clinical evaluations of CGM performance have been reported and second to stimulate interest in developing a future standard for reporting studies of CGM performance. Regarding the first goal, we believe that this analysis explains what information journal editors, reviewers, and regulators currently lack but need to intelligently assess the quality of a CGM performance evaluation. By advocating for adherence to our reporting recommendations outlined in this article (Table 2), we hope to see all relevant information about the study design (including comparator data characteristics) reported in a transparent and consistent manner. Regarding the second goal, we believe that this review can encourage the development of a standard for designing CGM performance studies to improve the clarity of their results. The result of both goals will be a greater understanding of the performance of CGMs, which are becoming an increasingly indispensable tool for managing diabetes.