Assessing the Predictive Accuracy of Oral Glucose Effectiveness Index using a Calibration Model

Abstract

Purpose:

Current reference methods for measuring glucose effectiveness (GE) are the somatostatin pancreatic glucose clamp and minimal model analysis of frequently sampled intravenous glucose tolerance test (FSIVGTT), both of which are laborious and not feasible in large epidemiological studies. Consequently, surrogate indices derived from an oral glucose tolerance test (OGTT) to measure GE (oGE) have been proposed and used in many studies. However, the predictive accuracy of these surrogates has not been formally validated. In this study, we used a calibration model analysis to evaluate the accuracy of surrogate indices to predict GE from the reference FSIVGTT (Sg_MM).

Methods:

Subjects (n=123, mean age 48 ±11 years; BMI 35.9±7.3 kg/m²) with varying glucose tolerance (NGT, n=37; IFG/IGT, n=78; and T2DM, n=8) underwent FSIVGTT and OGTT on two separate days. Predictive accuracy was assessed by both root mean squared error (RMSE) of prediction and leave-one-out cross-validation-type RMSE of prediction (CVPE).

Results:

As expected, insulin sensitivity, Sg_MM, and oGE were reduced in subjects with T2DM and IFG/IGT when compared with NGT. Simple linear regression analyses revealed a modest but significant relationship between oGE and Sg_MM (r=0.25, p<0.001). However, using calibration model, measured Sg_MM and predicted Sg_MM derived from oGE were modestly correlated (r = 0.21, p<0.05) with the best fit line suggesting poor predictive accuracy. There were no significant differences in CVPE and RMSE among the surrogates, suggesting similar predictive ability.

Conclusions:

Although OGTT-derived surrogate indices of GE are convenient and feasible, they have limited ability to robustly predict GE.

Introduction

Glucose itself has the capability to regulate blood glucose levels by decreasing hepatic glucose production and augmenting peripheral glucose disposal [1]. The ability of glucose to facilitate these actions and lower plasma glucose levels at basal insulin concentrations is referred to as glucose effectiveness (abbreviated either as Sg or GE) [1–3]. Both the hepatic suppression and the peripheral component contribute to overall Sg. In humans, the peripheral glucose disposal component contributes to two-thirds of Sg [2]. Although underappreciated, Sg accounts for nearly 50% of the glucose disposal following a glucose load in normal individuals. In fact, in insulin-resistant individuals, insulin-mediated glucose disposal accounts for less than 30%, and Sg accounts for well over 70% of the glucose disposal [2]. Consequently, Sg plays a major role in the development of glucose intolerance and is reduced in type 2 diabetes mellitus (T2DM) [4–7]. Disposal Sg, but not glucose-stimulated suppression of hepatic glucose production is impaired in patients with T2DM [6,7]. Recent studies have measured glucose effectiveness to assess the risk of developing T2DM and the benefits of treatment [8,9].

The two standard methods for measuring Sg are the somatostatin pancreatic-glucose clamp and minimal model analysis (MM) of frequently sampled intravenous glucose tolerance test (FSIVGTT) [1,10,2–4,6]. In the glucose clamp approach, islet hormone secretion is inhibited with concomitant somatostatin infusion thus permiTTing evaluation of glucose disposal under graded glucose and replacement insulin infusions. The FSIVGTT glucose dynamics, as modeled by the MM, is a function of the effects of glucose per se at basal insulin levels (Sg) and of incremental insulin action that is dependent on beta-cell function and insulin sensitivity. Sg derived from the FSIVGTT, Sg_MM, and the clamp-derived measure of Sg, Sg_Clamp are concordant [10]. However, both tests are time-consuming and laborious. Therefore, a surrogate measurement that is more feasible and convenient to determine Sg is necessary.

Recently, Nagaska et al. proposed a surrogate index derived from glucose and insulin concentrations during oral glucose tolerance test (OGTT) to measure glucose effectiveness (oGE) [11]. oGE albeit modest, significantly correlates with Sg_MM derived from a FSIVGTT [11,12]. In fact, this novel surrogate of Sg independently predicts glucose tolerance in children [12]. However, the predictive accuracy of this surrogate has not been formally validated. Having a reliable measure of GE is important for clinical and epidemiological studies. To that end, in the present study, calibration model analysis was used to evaluate the accuracy of these surrogates to predict results from the reference FSIVGTT, Sg_MM in a cohort with a wide range of insulin sensitivity and glucose tolerance.

Subjects and Methods

Subjects

In this study, we included data from 123 subjects who underwent FSIVGTT and OGTT procedures at one of two institutions: The Clinical Research Center (CRC)/Center for Clinical and Translational Science, Columbus, Ohio and the National Institutes of Health (NIH) Clinical Center, Bethesda, Maryland. These clinical studies were approved by the Institutional Review Boards of the above-mentioned institutions. All procedures followed were in accordance with each institution’s guidelines, and all participants gave wriTTen, informed consent prior to their participation. Subjects were considered to be diabetic or have impaired glucose tolerance if they met the American Diabetes Association (ADA) criteria for type 2 diabetes [13]. Healthy participants were in good health and were not taking any medications. Patients with IGT and T2DM were allowed to take multivitamins and anti-lipid and anti-hypertensive medications. There was one patient with T2DM who was on glipizide and metformin therapy; both of which were held on the morning of the study day. Exclusion criteria were: medical conditions including ischemic coronary heart disease, heart failure, prior cardiac surgery, chronic blood disease, severe respiratory insufficiency requiring oxygen therapy, and/or psychiatric conditions precluding participation in the study. Data from some of these subjects have been reported previously [14].

Oral glucose tolerance test

Before participating in the OGTT, each subject was provided with dietary guidelines. Each subject consumed a high carbohydrate diet (minimum of 250 g/day) for at least 3 days prior to the test. Following a 12-hour overnight fast, each subject consumed a 75g glucose solution (Fisherbrand, UN-DEX; Fisher Diagnostics, Middletown VA) over a two-minute interval. Blood samples were drawn at times 0, 30, 60, 90, and 120 min for measurement of glucose and serum insulin levels.

Frequently sampled intravenous glucose tolerance test:

Each subject underwent a 12-hour overnight fast followed by an intravenous bolus of glucose (0.3g/kg) infused over a minute at time 0 min. At time 19 min, subjects were administered an intravenous injection of insulin (0.05 U/kg; Humulin; Eli Lilly, Indianapolis, IN). Blood samples were drawn at times −10, −1, 0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 12, 14, 16, 19, 22, 23, 24, 25, 27, 30, 40, 50, 60, 70, 80, 90, 100, 120, 150, and 180 minutes for measurement of glucose and serum insulin levels. Minimal Model analysis of the FSIVGTT was utilized to estimate Sg and insulin sensitivity index (SI) as previously described using MINMOD software (version 6.02; MinMOD Millenium, Los Angeles, CA) [15].

Laboratory assays

Routine assays for serum lipids, plasma glucose and insulin, and HbA1C (A1C) were performed in the Department of Laboratory Medicine at the Clinical Center, NIH or Clinical Research Center, Columbus, Ohio. OGTT-derived surrogate indices of Sg:

oGE was calculated from the OGTT, the details of which has been published in detail previously [11,12]. Briefly, glucose effectiveness is the ability of glucose to facilitate its own disposal at basal insulin concentrations. The authors first estimated 2-h post-load plasma glucose concentrations assuming absence of insulin action and glucose effectiveness [16,11]. Next, they estimated 2-h post-load plasma glucose concentrations in the seTTing of no insulin action but active glucose effectiveness by using linear regression equations predicting 2-h plasma glucose levels from oral disposition index (oDI) [17]. An oDI of (near) zero was used to incorporate no insulin action in the predictions. oGE was derived from the equation: oGE= {[PPG_without insulin action and GE] - [PPG_with GE but without insulin action] *[2hPG/2hPG_E]}/120, where PPG = post-loading plasma glucose, GE = glucose effectiveness, 2hPG = 2-h post-glucose PG, and 2hPG_E = expected 2hPG. Additional OGTT-derived surrogate indices for glucose effectiveness (SgOGTT30 and SgOGTT60, respectively) were also obtained as previously described [18,19].

Statistical Analysis:

Frequency distributions of all outcome variables were analyzed and log transformed where appropriate. Simple linear regression was used to evaluate the correlations between log-transformed Sg_MM and the log-transformed surrogate indices of GE (oGE, Sg_OGTT30, and Sg_OGTT60). A standard calibration model was used to evaluate the validity of the surrogate indices. As described previously, this analysis differs from a typical regression [20,21]. Calibration involves inverse regression, such that the surrogate measurement is regressed on the reference measurement. Each surrogate GE index was regressed with the observed SgMM in the entire cohort to calculate surrogate-index predicted Sg_MM that could be compared to the observed Sg_MM to assess the predictive ability of that surrogate index. Comparing surrogate-index-predicted Sg_MM to observed Sg_MM assesses the predictive ability of the surrogate indices. Thus, by fiTTing a calibration model [x_i = α + Ɓy_i + ε_i, or Sg_MM = α + Ɓ (OGTT-derived surrogate index) + (random error for the ith subject)] we evaluated the absolute accuracy of the OGTT-derived surrogate indices in predicting Sg from the reference FSIVGTT. For the random error, a mean value of 0, constant variance, and a Gaussian distribution were assumed.

Utilizing the predicted Sg_MM from the calibration model, predicted residual values (e_i = x_i − x°_i) were calculated by determining the difference between the measured Sg_MM (x_i for the ith subject) and the predicted Sg_MM (x°_i = α + Ɓy°_i) for each subject. Leave-one-out cross validation was used to determine a second predicted residual by excluding the ith subject. That is, e_(i) = x_i − x°_(i) where x_i continues to denote measured Sg_MM, but x°_(i) now excludes ith subject in the calibration model predicted Sg_MM. Predictive accuracy of the OGTT-derived surrogate indices was evaluated by using these two predicted residuals to calculate square root of the mean squared error of prediction (RMSE) and leave-one-out cross-validation-type root mean squared error of prediction (CVPE). P < 0.05 was considered statistically significant.

Statistical analyses were performed using GraphPad Prism version 6.01 (GraphPad) and SAS Enterprise Guide version 5.1 (SAS Institute).

Results

Baseline characteristics

123 subjects (male, n=31; female, n=92) of mean age 48 ± 11 yr participated in our study. The clinical parameters for our subjects are displayed in Table 1. Subjects were either Caucasian or African American (Caucasian, n=55; African American, n=68) and predominantly overweight/obese (35.9 ± 7.3 kg/m²), with a mean body fat of 43.1% ± 7.3%. Subjects had a wide range of insulin sensitivity (0.53 to 14.26 [(μu/l)⁻¹ min⁻¹]) and a mean SI of 3.59 ± 2.70 [(μU/L)⁻¹ min⁻¹] in our cohort that included individuals with type 2 diabetes mellitus (T2DM, n=8), impaired fasting glucose/impaired glucose tolerance (IFG/IGT, n=78), and normal glucose tolerance (NGT, n=37). Hemoglobin A1C levels ranged from 4.8 to 7.1% with a mean of 5.8% ± 0.5%. As expected, age- and sex-adjusted SI [(μu/l)⁻¹ min⁻¹], was lower in individuals with diabetes and IFG/IGT when compared with normal (DM: 2.24 ± 0.94; IFG/IGT: 3.36 ± 0.32; and NGT: 5.31 ± 0.41, p<0.0001). Our cohort had a mean Sg_MM of 0.024 ± 0.017 min⁻¹ and a range of 0.007 to 0.099 min⁻¹. Sg_MM and oGE (adjusted for age and BMI) were higher in women than in men (Sg_MM: 0.026 ± 0.001 vs. 0.019 ± 0.003 min⁻¹; oGE: 1.64 ± 0.06 vs. 1.31 ± 0.09 mg/dL/min, p<0.001). Ageand sex-adjusted Sg_MM (min⁻¹) was significantly lower in subjects with diabetes (DM: 0.014 ± .006; IFG/IGT: 0.022 ± 0.0.002; and NGT: 0.025 ± 0.0.002, p<0.05). Likewise, the oral surrogate index of GE (oGE) also was significantly lower in subjects with diabetes and IFG/IGT (DM: 0.89 ± 0.24; IFG/IGT: 1.39 ± 0.08; and NGT: 1.88 ± 0.12 mg/dL/min, p<0.001).

Table 1.

Subject Characteristics

Clinical Parameters
Age (yrs)	48.5 ± 11.6
Female (%, n)	74.8 (n=92)
Race (W: Caucasian, AA: African American)	55 W, 68 AA
BMI (kg/m2)	34.7 (30.8 – 39.9)
Body fat (%)	45.5 (39.5 – 48.0)
Systolic blood pressure (mmHg)	127 ± 14
Diastolic blood pressure (mmHg)	79 ± 10
Fasting plasma glucose (mg/dL)	96 ± 12
2-hour glucose (mg/dL)	125 ± 41
Total cholesterol (mg/dL)	188 ± 40
Low-density lipoprotein cholesterol (mg/dL)	116 ± 32
High-density lipoprotein cholesterol (mg/dL)	53 ± 15
Triglycerides (mg/dL)	107 ± 63
Fasting plasma insulin (μU/ml)	11.8 (6.1 – 18.6)
A1C (%)	5.70 (5.50 – 6.10)
QUICKI	0.329 (0.307 – 0.360)
SI [(mu/1)−1 min−1]	2.66 (1.54 – 4.49)
AIRg(μU·ml−1·min−1)	323 (186 – 590)
DI (SI x AIR)	9.41 (5.02 – 15.7)
SgMM (min−1)	0.021 (0.014 – 0.027)
oGE (mg/dL/min)	1.51 (1.06–1.98)

Normalized parameters are mean ± standard deviation, while other values are median (25th percentile −75th percentile). n=123 subjects; QUICKI, quantitative insulin sensitivity check index; SI, insulin sensitivity index; AIRg, acute insulin response to glucose; DI, disposition index; Sg_MM, glucose effectiveness at zero insulin derived via minimal model analysis of FSIVGTT.

oGE increases with age (r=0.24, p<0.001), but is negatively associated with BMI (r=−0.64, p<0.001), total body fat percent (r= −0.26, p<0.001), 2h PPG (r=−0.50, p<0.001), and A1C levels (r=−0.31, p<0.001). There was no sexual dimorphism in these relationships. Linear correlations between log transformed surrogate indices of glucose effectiveness and log transformed reference Sg_MM are shown in Figure 1. Simple linear regression analyses revealed modest but significant relationships for each of the surrogate indices (oGE, Sg_OGTT30, and Sg_OGTT60) and Sg_MM (r = 0.25, 0.25, and 0.26 respectively, p < 0.05).

Fig. 1.

Linear correlations between surrogate indices of glucose effectiveness and the reference glucose effectiveness (SgMM) at zero insulin in 123 study participants. Pearson correlation coefficients (r) and the corresponding P values are shown for each simple linear regression analysis. Each surrogate index and the reference glucose effectiveness are log transformed. P<0.05 was considered to represent a statistically significant relationship. A: Relationship between oGE and Sg_MM. B: Relationship between Sg_OGTT30 and Sg_MM. C: Relationship between Sg_OGTT60 and Sg_MM.

We used a calibration model analysis to robustly evaluate the absolute accuracy of OGTT-derived surrogate indices of GE to predict Sg from the reference FSIVGTT. The correlations used for calibration model analysis between the Sg_MM estimated based on each GE index and the actual Sg_MM are shown in figure 2. A best-fit-line was created to indicate perfect predictive capability. Linear least-squares fit between the measured Sg_MM and predicted Sg_MM from each of the surrogate indices were rather weakly correlated (r = 0.21, 0.19, and 0.23, respectively, p < 0.01). All three of the surrogate indices demonstrated poor predictive accuracy, with virtually none of the predicted values coinciding with the best-fit-lines. CVPE and RMSE were calculated for each surrogate index to comparatively evaluate their predictive capabilities (Table 2). There were no significant differences in CVPE and RMSE among oGE, Sg_OGTT30, and Sg_OGTT60, suggesting similar predictive ability in these surrogates.

Fig. 2.

Comparison between the predicted Sg_MM from the derived glucose effectiveness surrogate indices and the measured Sg_MM. The solid line represents the best-fit-line, perfect predictive accuracy. Pearson correlation coefficients (r) and the corresponding P values are shown for each model. Predicted Sg_MM for each of the surrogate indices and the measured Sg_MM are log transformed. P<0.05 was considered to represent a statistically significant relationship. A: Predictive capability of oGE. B: Predictive capability of Sg_OGTT30. C: Predictive capability of Sg_OGTT60.

Table 2.

CVPE and RMSE estimates of error calculated from calibration analysis of surrogate indices of glucose effectiveness

	CVPE	P value	RMSE	P value
log oGE	0.226	-	0.224	-
log SgOGTT30	0.227	0.523	0.225	0.457
logSgOGTT30	0.225	0.452	0.223	0.527

RMSE, root mean squared error; CVPE, cross -validation-type root mean squared error of prediction; P values correspond to comparisons between OGTT-derived surrogate indices. P < 0.05 was considered to represent a statistically significant relationship. CVPE and RMSE values were calculated from calibration model analysis of OGTT-derived surrogate indices.

Discussion

In this study, we examined the predictive accuracy of GE surrogate index values derived from the OGTT (oGE) by comparing it to the GE parameter (Sg_MM) derived from the reference method, FSIVGTT. Our study findings suggest that although oGE has a modest relationship with Sg_MM, the predictive accuracy of this surrogate index is limited. Furthermore, we demonstrate that other simple measures of GE derived from the OGTT (Sg_OGTT30, and Sg_OGTT60) also have similar predictive ability as oGE.

Plasma glucose mediates glucose homeostasis through the suppression of endogenous glucose production (EGP) and augmentation of peripheral glucose uptake [1]. Glucose effectiveness represents the ability of glucose to successfully facilitate these actions at basal insulin concentrations [1–3]. Indeed, in healthy, nondiabetic adults, acute hyperglycemia resulted in a suppression of EGP, and increase in peripheral glucose disposal at basal plasma insulin and glucagon levels [22]. We have previously reported that Sg_MM compensates for decreasing insulin sensitivity in nondiabetic individuals to maintain glucose homeostasis [23–25] and thus low Sg_MM predicts incident diabetes [4]. Accentuated EGP is a primary determinant of fasting hyperglycemia in T2DM, a physiological process that Sg partially inhibits [26,27]. Accordingly, developing a convenient yet reliable method for assessing Sg is essential to understanding the pathogenesis of T2DM and the effect of various interventions.

Nagasaka et al. developed a novel surrogate index of GE using data from an OGTT in a large Chikuma cohort (n=502) [11] based on the assumption that 2-hour post-prandial glucose levels are determined by only GE and insulin-mediated glucose disposal. As observed in our study, oGE was lower in patients with T2DM and higher in women compared with men [11]. Similarly, in concordance with the Chikuma cohort [11] and other studies [12,28], oGE was significantly correlated with BMI and 2h PPG in our study. The presence of these relationships is not entirely surprising since the independent variables (BMI, 2h PPG) are mathematically related to oGE and its derivation. However, it is worth noting that oGE is negatively related, albeit modestly, to total body fat in our study. Thus, oGE, as a measure of glucose effectiveness has some theoretical basis in its formulation and appears to reflect reduced GE observed in impaired glucose tolerance and T2DM.

Assessing the predictive accuracy is an integral part of validating any surrogate index. Two studies, Nagasaka et al. [11] initially and Weiss et al. [12] subsequently validated the oGE in adults and adolescents, respectively. Both these validation studies as well as our own were based on examining correlations with the Sg_MM derived from the FSIVGTT method. The methodology of our study was almost identical to the studies by Nagasaka and Weiss, in which we also used 75-gram oral load during the OGTT and comparable insulin and glucose infusions during the FSIVGTT. oGE significantly correlated with Sg_MM in the study by Nagasaka et al (n=205, r = 0.322 and p < 0.001) [11] and Weiss et. al. (n=98, r = 0.35 and p < 0.001) [12]. The magnitude of the correlation coefficients is very similar and not significantly different from our study findings (r=0.25). However, correlation coefficient is a measure of linear agreement but not of predictive ability. It is possible to have an excellent correlation when the study population has a wider range or larger variability of the measurement in consideration and yet have a poor predictive ability. Therefore, for the first time, in this cross-sectional study we examined 123 adults with varying degrees of glucose tolerance to evaluate the absolute accuracy of OGTT-derived surrogate indices to predict Sg from minimal model analysis of FSIVGTT using calibration model analyses. In the calibration model, oGE (surrogate) is used to estimate the Sg_MM from the FSIVGTT (reference method). Though the OGTT-derived surrogate indices showed modest linear correlation with Sg_MM derived from FSIVGTT, calibration analysis demonstrated poor ability of the surrogate indices to predict the reference Sg_MM. Furthermore, residual analysis indicates the surrogate indices are all similar in their absolute accuracy of predicting Sg_MM.

Various factors may be responsible for the discrepancy between our study and the findings of others [11,10,12,18]. Differences may be aTTributed to the size and heterogeneity of the study populations. In comparison to Nagaska et al. [11], our participants represented an older (48 vs. 33, mean age) and a more overweight/obese (BMI: 34.7 vs. 22.3 kg/m²) cohort, as well as having a higher composition of females. However, both fasting plasma glucose (96 vs. 97 mg/dL) and 2-hour plasma glucose (125 vs. 130 mg/dL) remain very similar. Both studies utilized the same insulin sensitivity index, minimal model analysis of FSIVGTT, and included individuals with NGT, IFG/IGT, and T2DM. Hence, it seems unlikely that the discrepancies between the previous studies and our present study are resultant of differences in the participants’ levels of insulin sensitivity/resistance. There are many limitations to our study. Our study findings may suffer from the possibility that GE as measured by IVGTT and an OGTT may not be similar considering the route of administration and the associated incretin effect with OGTT. Second, the inherent variability in OGTT and in the estimation of DI may introduce significant error thus reducing the strength of the association. Thirdly, in calibration model, it is assumed that the reference method, in this case IVGTT has a very small measurement error, which is not entirely accurate. These factors may have played a role in affecting the limited predictive accuracy observed. Notwithstanding these limitations, the robust evaluation of the predictive ability and accuracy of oGE is the major strength of our study.

In conclusion, in contrast with previous studies [11,10,12,18,28], we found that despite the convenience and feasibility of these OGTT-derived surrogate indices of Sg, oGE, Sg_OGTT30, and Sg_OGTT60 have limited ability to robustly predict glucose effectiveness. Therefore, our findings suggest that these surrogate indices should be used cautiously in the design and interpretation of clinical studies. Future studies should aim to identify surrogate indices to measure Sg that are both less laborious than the two present gold standard methods, and equally as reliable.