Abstract
Background
There is an urgent need to update diabetes prediction, which has relied on the United Kingdom Prospective Diabetes Study (UKPDS) that dates back to 1970 s’ European populations.
Objective
The objective of this study was to develop a risk engine with multiple risk equations using a recent patient cohort with type 2 diabetes mellitus reflective of the US population.
Methods
A total of 17 risk equations for predicting diabetes-related microvascular and macrovascular events, hypoglycemia, mortality, and progression of diabetes risk factors were estimated using the data from the Action to Control Cardiovascular Risk in Diabetes (ACCORD) trial (n = 10,251). Internal and external validation processes were used to assess performance of the Building, Relating, Assessing, and Validating Outcomes (BRAVO) risk engine. One-way sensitivity analysis was conducted to examine the impact of risk factors on mortality at the population level.
Results
The BRAVO risk engine added several risk factors including severe hypoglycemia and common US racial/ethnicity categories compared with the UKPDS risk engine. The BRAVO risk engine also modeled mortality escalation associated with intensive glycemic control (i.e., glycosylated hemoglobin < 6.5%). External validation showed a good prediction power on 28 endpoints observed from other clinical trials (slope = 1.071, R2 = 0.86).
Conclusion
The BRAVO risk engine for the US diabetes cohort provides an alternative to the UKPDS risk engine. It can be applied to assist clinical and policy decision making such as cost-effective resource allocation in USA.
1. Introduction
The prevalence of type 2 diabetes mellitus (T2DM) in USA has risen from 4.21% (12.1 million) in 2002 [1] to 9.1% in 2012 [2]. The overall prevalence of diabetes in USA is projected to reach to 21% in 2050 [3]. This increase in the T2DM population has led to dramatically increased costs in managing diabetes. The total costs of diabetes increased approximately 41% from US$174 billion in 2007 [1] to US$245 billion in 2012. In 2012, patients with T2DM incurred 20% of total healthcare expenditures in USA, more than half of which was attributable to treating diabetes and its complications [4]. A majority of the diabetes-related costs were the result of micro/macrovascular complication events [5]. The most common diabetes-related macrovascular events include myocardial infarction (MI), congestive heart failure (CHF), and stroke. The most frequent diabetes-related microvascular events include retinopathy (e.g., edema, blindness), nephropathy [e.g., end-stage renal disease (ESRD)], and neuropathy [e.g., severe pressure sensation loss (SPSL), amputation] [6].
To better manage the growing T2DM population in an environment of constrained healthcare resources, systemwide improvement and redesign are necessary. In the ‘bigdata’ era, this is possible using outcome-driven and evidence-based diabetes management. Prediction models can help to develop sophisticated and well-designed diabetes management strategies. Models also better profile the risk of patients so that more healthcare resources can be effectively allocated to those with more health needs. Several diabetes models in USA have been used to describe disease progression and compare the cost effectiveness of different therapeutic strategies: the CORE diabetes model, the University of Michigan model for diabetes, the Swedish Institute of Health Economics model otherwise known as the Economics and Health Outcomes in T2DM Model, the United Kingdom Prospective Diabetes Study (UKPDS) outcomes model, the Centers for Disease Control-Research Triangle Institute diabetes cost-effectiveness model, the Cardiff Research Consortium model [7], and several others [8, 9]. These models have been used to support the outcome-driven evidence-based diabetes management in several areas such as comparisons between therapeutic plans [10–16], evaluating potential benefits of achieving treatment goals, [14] and policy impact on T2DM [17–19].
However, these diabetes models rely heavily on the UKPDS risk engine that was developed using data from a UK diabetes cohort collected from the 1970s. The UKPDS population differs significantly from the current US population in terms of race/ethnicity, definition of diabetes, treatment algorithm, and screening methods to assess complications and comorbidities. Further, the baseline hazard of diabetes-related events may vary over time [20] and may differ between the UKPDS population and current US population [21]. Using a UK-based risk engine to predict US diabetes management raised significant concerns on the prediction validity. There is an urgent need for a new risk engine to be developed based on a US population to support decision making in clinical practice in USA. Therefore, our goal in this study was to develop the Building, Relating, Assessing, and Validating Outcomes (BRAVO) Diabetes Risk Engine based on the US diabetes population, which may provide an alternative risk engine for US researchers and policy makers.
2. Research Design and Methods
2.1. Data Source
The Action to Control Cardiovascular Risk in Diabetes (ACCORD) trial is one of the largest studies ever conducted in adults with T2DM mainly in USA. Three potential strategies were tested through this multicenter clinical trial to lower the risk of major cardiovascular events: intensive blood sugar control, intensive blood pressure control, and treatment to lower blood lipid levels. There were 10,251 participants enrolled in this trial, with an average follow-up time of approximately 4 years. All diabetes-related events were recorded periodically during the study. The ACCORD trial started its enrollment in January 2001 [22], and the baseline demographic characteristics were reported in the Buse et al. ACCORD study [23, 24]. For the convenience of our readers, we also include the baseline information in eTable 1 of the Electronic Supplementary Material (ESM).
2.2. Risk Equations
Compared with the commonly used Markov approach in several current models [18, 25, 26], the BRAVO risk engine was developed based on a series of discrete intercorrelated risk equations. This discrete-equation approach accounted for risk escalation as diabetes progressed and for interactions between complications. It also allows for adjusting a large number of demographic and biological characteristics [27].
The BRAVO risk engine contains three separate modules (events module, risk factors module, and mortality module), each of which contains a series of regression equations to predict the occurrence of events, progression in risk factors, and mortality. In the events module, a series of risk equations were fitted to predict diabetes-related macrovascular events (stroke, MI, CHF, angina, and revascularization surgery), microvascular events (ESRD, blindness, and SPSL), and adverse events (severe hypoglycemia and symptomatic hypoglycemia). In the risk factors module, each risk factor [glycosylated hemoglobin (HbA1c), systolic blood pressure (SBP), weight, and low-density lipoprotein (LDL)] in the current cycle was predicted jointly by its value from the last cycle and other risk factors. Last, in the mortality module, an equation was fitted to predict patient death, and a second equation was developed to explore the cause of death [i.e., cardiovascular disease (CVD) or other death causes]. All the clinical outcomes have been defined in an article published by the ACCORD trial group [23].
A literature review has been conducted to identify the initial list of explanatory variables (eTable 2 of the ESM) and a backward selection process was conducted to remove those with no improvement for model fitting. Explanatory variables can be categorized into three groups: biomedical factors, demographic characteristics, and complications. The definitions of these explanatory variables are provided in eTable 2 of the ESM. The main modeling strategy we used in this study was a left-censored, time-dependent, parametric proportional hazard model, in which diabetes duration was used as the time index, instead of real time in the clinical trial. To smooth measurement fluctuation for biomarkers, we applied a moving average technique: as the models were developed based on annual cycles, all the parameters values should be aggregated annually. We aggregated the parameters values for each year by averaging all the measurements conducted within the previous 2 years. Parameters values from the current year were used to predict the probability of encountering an outcome event in the next year to account for potential bias caused by reverse causality. The history of events was also included in the initial list of explanatory variables. Having had an event at baseline or in the study periods before the current year would both be identified as a history of that event for the current year. A mixed-method algorithm including a cross-validation-based, backward model selection process, literature review, and consultation from endocrinologists was used to support the model fitting process. For binary outcomes, the c-statistic has been applied to measure the discrimination power of the model.
However, in the BRAVO risk engine, we were not only interested in the discrimination power of the model, but also the prediction accuracy. Thus, both the c-statistic and Brier score [28] were calculated to support the model selection process. For continuous outcomes [HbA1c, SBP, LDL, and body mass index (BMI)], the mean square prediction error was used to select the models. A ten-fold cross-validation framework was applied to adjust the c-statistic, Brier score, and mean square prediction error for possible over-fitting in low-dimension regressions [29]. All risk factors that improved model performance were included into the final model. For those risk factors that did not have a significant impact on model performance, inclusion and exclusion were judged by clinical endocrinology knowledge and evidence found from the current literature. Risk factors that were not statistically significant in the BRAVO model selection processes, but that were supported as risk factors by existing clinical evidence were included in the risk equations. Details regarding the functional form of each risk equation and model selection process are provided in Online Appendix 1.
2.3. Internal and External Validation Process
The internal validation process was conducted by plotting the predicted cumulative hazard against the Kaplan–Meier cumulative hazard for all outcome measures. We also calculated the log–log 95% confidence interval (CI) for cumulative incidence rate across a diabetes duration of 0–40 years in the ACCORD trial using the left-truncated method, and the predicted curve was examined if it fell within the 95% CI of the Kaplan–Meier curve.
We included all the clinical trials from the fourth and fifth Mount Hood Challenges to conduct the external validation process [7, 30], which were the Atorvastatin Study for Prevention of Coronary Heart Disease Endpoints in Non-Insulin-Dependent Diabetes Mellitus (ASPEN) trial, the Action in Diabetes and Vascular Disease: Preterax and Diamicron Modified Release Controlled Evaluation (ADVANCE) trial, and the Collaborative Atorvastatin Diabetes Study (CARDS) trial.
A total of 28 endpoints were predicted using the BRAVO risk engine through a discrete-time event microsimulation process (Online Appendix 2), under the corresponding time horizon of each trial [31]. The advantage of this modeling strategy over the traditional Markov cohort model has been discussed previously [32]. The baseline characteristics of each trial have been reported [7] and applied directly as the characteristic of simulation samples. Normal distribution was assumed for all input variables and the standard error of each variable was extracted from the corresponding literature. The values of key risk factors, including HbA1c, LDL, SBP, and BMI in each validation trial were assumed to reach the corresponding treatment target at the first year and remain constant in the following years. Then, 10,000 simulation runs were used to reach convergence in outcomes. An ordinary least-square model was used to fit the BRAVO-predicted incidence rates to observed incidence rates, and slope, intercept, and R2 were used to show prediction accuracy. The microsimulation was conducted using a joint program written through Visual Basic and C++ language.
In addition, we conducted one-way sensitivity analyses to explore the impact of six risk factors on the life expectancy in the ACCORD trial population. The values of continuous risk factors (HbA1c, SBP, LDL, and BMI) were set from one standard deviation below the mean to one standard deviation above, while categorical risk factors (smoking and severe hypoglycemia) ranged from 50 to 200% of the population average. Results from this analysis were reported in Online Appendix 4.
3. Results
The median follow-up time for patients in the ACCORD trial was 3.7 years, with a total of 39,043 person-years of data to support the model fitting process for developing the BRAVO risk engine. Hazard ratios of risk factors for non-fatal MI, stroke, CHF, angina, revascularization surgery, ESRD, blindness, and SPSL are presented in Table 1. In addition, Table 1 also presents hazard/odds ratios of prediction equations for all-cause mortality and CVD death. The inter-correlations between different co-morbidities were captured by using other co-morbidities as risk factors to predict the occurrence of the target event. For example, in the CHF equation, having a history of MI increased the risk of CHF by 82.2%. In addition, repeat events were also taken into consideration. Having a history of CHF was associated with a 247.6% increase in the risk of a second CHF event.
Table 1.
Abbreviations | Strokea | CHFa | MIa | Anginaa | Revasca |
| |||||
HbA1c | 1.353 (1.165–1.570) | 1.108 (1.001–1.227) | 1.179 (1.086–1.281) | 1.208 (1.093–1.335) | 1.070 (1.011–1.133) |
HbA1c*HbA1c | |||||
SBP | 1.029 (1.019–1.040) | 1.017 (1.011–1.023) | 1.006 (1.002–1.010) | ||
LDL | 1.009 (1.005–1.013) | 1.007 (1.005–1.009) | 1.004 (1.000–1.008) | ||
BMI | 1.060 (1.041–1.079) | 1.027 (1.007–1.048) | 1.182 (1.057–1.321) | ||
BMI*BMI | 0.998 (0.996–1.000) | ||||
Age at diagnosis | 1.047 (1.025–1.070) | 1.058 (1.043–1.072) | 1.031 (1.019–1.044) | ||
Severe hypoglycemia | 1.980 (1.259–3.114) | 2.286 (1.569–3.331) | 1.885 (1.102–3.225) | ||
Female | 0.747 (0.620–0.900) | 0.731 (0.639–0.837) | |||
Education | 0.579 (0.440–00.761) | 0.681 (0.553–0.838) | |||
Smoking | 1.195 (0.924–1.545) | ||||
Race (ref = black) | |||||
White | 1.687 (1.310–2.172) | 1.464 (1.227–1.746) | |||
Hispanic | 1.267 (0.840–1.913) | 1.110 (0.830–1.483) | |||
Others | 1.279 (0.893–1.831) | 1.181 (0.922–1.511) | |||
MI_history | 1.846 (1.302–2.617) | 1.822 (1.440–2.305) | 1.627 (1.325–1.999) | 1.461 (1.134–1.881) | |
CHF_history | 3.476 (2.689–4.494) | 1.242 (0.997–1.547) | |||
Stroke_history | 3.080 (2.053–4.622) | 1.603 (1.318–1.950) | 1.377 (1.114–1.702) | ||
Angina_history | 1.674 (1.297–2.159) | 1.408 (1.208–1.640) | |||
Revasc_history | 2.212 (1.759–2.782) | 3.290 (2.570–4.212) | 2.328 (2.050–2.644) | ||
Blindness_history | |||||
Stroke_event | |||||
CHF_event | |||||
Log(duration) | |||||
Log(scale) | 10.028 | 8.029 | 6.311 | 7.986 | 7.093 |
Log(shape) | 0.586 | 0.728 | 0.538 | 0.065 | 0.264 |
Brier score | 0.003 (0.003–0.004) | 0.008 (0.007–0.009) | 0.012 (0.011–0.013) | 0.007 (0.007–0.008) | 0.025 (0.024–0.027) |
C-statistics | 0.750 (0.711–0.784) | 0.785 (0.766–0.809) | 0.689 (0.662–0.704) | 0.727 (0.705–0.757) | 0.670 (0.654–0.687) |
Function | Weibull | Weibull | Weibull | Weibull | Weibull |
| |||||
Abbreviations | ESRDa | Blindnessa | SPSLa | All-cause mortalitya | CVD deathb |
| |||||
HbA1c | 1.129 (1.003–1.269) | 1.190 (1.105–1.282) | 1.328 (1.253–1.409) | 0.512 (0.186–1.408) | 1.297 (1.069–1.574) |
HbA1c*HbA1c | 1.048 (0.982–1.118) | ||||
SBP | 1.014 (1.006–1.022) | 1.015 (1.009–1.021) | 1.010 (1.006–1.014) | 1.014 (1.001–1.028) | |
LDL | 1.004 (1.002–1.006) | 1.002 (1.000–1.004) | |||
BMI | 1.017 (0.999–1.035) | 1.035 (1.000–1.070) | |||
BMI*BMI | |||||
Age at diagnosis | 1.021 (1.011–1.031) | 1.024 (1.014–1.034) | |||
Severe hypoglycemia | 1.517 (1.015–2.268) | ||||
Female | 0.725 (0.637–0.826) | 0.576 (0.459–0.724) | |||
Education | 0.741 (0.615–0.892) | 0.725 (0.567–0.929) | |||
Smoking | 1.765 (1.305–2.387) | ||||
Race (ref = black) | |||||
White | 1.081 (0.885–1.320) | 1.224 (1.044–1.434) | |||
Hispanic | 1.701 (1.282–2.255) | 0.847 (0.634–1.132) | |||
Others | 1.179 (0.895–1.555) | 0.552 (0.420–0.726) | |||
MI_history | 1.224 (0.971–1.542) | 2.153 (1.382–3.356) | |||
CHF_history | 1.672 (1.097–2.548) | 2.175 (1.656–2.856) | 2.145 (1.234–3.730) | ||
Stroke_history | 1.388 (1.005–1.918) | 2.073 (1.045–4.111) | |||
Angina_history | 1.517 (1.181–1.950) | ||||
Revasc_history | |||||
Blindness_history | 1.992 (1.257–3.157) | 1.366 (1.020–1.829) | |||
Stroke_event | 3.380 (1.640–6.967) | 7.456 (0.859–64.731) | |||
CHF_event | 5.686 (3.957–8.171) | 5.948 (2.352–15.045) | |||
Log(duration) | 1.250 (0.918–1.703) | ||||
Log(scale) | 6.945 | 6.431 | 6.399 | 2.452 | −5.959 |
Log(shape) | 0.259 | 0.592 | 0.501 | −6.336 | |
Brier score | 0.005 (0.005–0.006) | 0.016 (0.015–0.017) | 0.027 (0.025–0.028) | 0.010 (0.009–0.011) | 0.212 (0.191–0.222) |
C-statistics | 0.592 (0.569–0.619) | 0.632 (0.612–0.657) | 0.661 (0.648–0.681) | 0.617 (0.585–0.653) | 0.731 (0.693–0.780) |
Function | Weibull | Weibull | Weibull | Gompertz | Logistic |
95% confidence intervals are reported in parentheses
BMI body mass index, CHF congestive heart failure, ESRD end-stage renal disease, HbA1c glycosylated hemoglobin, LDL low-density lipoprotein, MI myocardial infarction, ref reference, Revasc revascularization surgery, SBP systolic blood pressure, SPSL severe pressure sensation loss
Parameters are reported as hazard ratios
Parameters are reported as odds ratios
Race was identified as a significant risk factor for predicting MI, revascularization surgery, blindness, and nephropathy. Compared with African Americans, Caucasian, Hispanic, and other race/ethnicities were associated with 68.7% (95% CI 31.0–117.2), 26.7% (95% CI – 16.0 to 91.3), and 27.9% (95% CI – 10.7 to 83.1) higher risks for MI, respectively. In addition, Caucasian, Hispanic, and other race/ethnicities were associated with a 46.4% (95% CI 22.7–74.6), 11.0% (−17.0 to 48.3), and 18.1% (95% CI −7.8 to 51.1) higher likelihood of receiving revascularization surgery, compared with African Americans. Caucasian, Hispanic, and other race/ethnicities were correlated with a risk escalation of 8.1% (95% CI – 11.5 to 32.0), 70.1% (95% CI 28.2–125.5), and 17.9% (95% CI – 10.5 to 55.5), respectively, compared with African Americans for blindness. Last, Hispanic individuals and others were associated with risk reductions of SPSL for 15.3% (95% CI −13.2 to 36.6) and 44.8% (95% CI 27.4–58.0), respectively, while Caucasians were associated with a 22.4% (95% CI 4.4–43.4) higher risk for SPSL compared with African Americans.
The BRAVO risk engine included severe hypoglycemia in multiple risk equations. Encountering one more episode of severe hypoglycemia in the current year was associated with increased risks for CHF [hazard ratio (HR) = 198%], MI (HR = 228.6%), angina (HR = 188.5%), and blindness (HR = 151.7%). Although not statistically significant, quadratic polynomials of HbA1c levels were also found to be an important predictor for predicting all-cause mortality as indicated by Bier scores and c-statistics. An HbA1c level of 7.12% was calculated to be associated with the lowest mortality risks. The equations to model time-varying risk factors including HbA1c, SBP, LDL, body weights, smoking status, and occurrence of severe hypoglycemia and symptomatic hypoglycemia are presented in eTable 3 of the ESM.
The results of our internal validation process are presented in Fig. 1. The black dashed line denotes the Kaplan–Meier curve of the observed cumulative incidence for each event type using the left-truncated method. The red solid line indicates the predicted cumulative incidence of a given event from diabetes onset until 40 years after onset. The gray area denotes the log–log 95% CI for the observed cumulative incidence at each time point. All predicted incidence curves fit close to the observed curves and all predicted curves were within the 95% CI.
The results of external validation are presented in Fig. 2. Predicted incidence rates were plotted against observed incidence rates, with a dashed line indicating 100% prediction accuracy. The incidence rates of 28 endpoints, which were predicted from the BRAVO risk engine, were clustered around the prediction accuracy line. The slope coefficient and intercept of the ordinary least-square regression on the predicted incidence rates were 1.071 and 0.001, respectively, and the R2 was estimated to be 0.86. An F-test was applied to examine the difference between the fitted line and the line of 100% prediction accuracy, and no statistical difference was identified (p = 0.82).
4. Discussion
The BRAVO risk engine with good internal and external validity was developed to offer an alternative to the established UKPDS risk engine that was based on the US population, to support decision making in US clinical practices. Health outcome predictions from the BRAVO risk engine were consistent with the previous findings of the ACCORD trial on non-fatal MI, angina, and revascularization surgery [33]. Although not statistically significant, a previous study has found that intensive glycemic control was associated with lower HRs of ESRD, blindness, and SPSL [34]. The BRAVO risk engine included HbA1c as an important risk factor for predicting these microvascular events. More interestingly, even when including the extensive covariate list from our BRAVO risk engine, these associations between HbA1c and microvascular events were still statistically significant in the BRAVO risk equations.
The impact of hypoglycemia on diabetes outcomes and mortality has been studied extensively in recent years. The occurrence of hypoglycemia was found to be associated with major microvascular and microvascular events, death, and other nonvascular outcomes [35]. Our previous study also found that in addition to the direct impact of hypoglycemia on vascular risk [36], the fear for hypoglycemia was also associated with an additional quality-adjusted life-year decrement [37]. Our risk engine is the first to fully incorporate hypoglycemia’s impact on disease course. Our engine provides a critical predictive tool to evaluate new T2DM drugs, which usually have lower hypoglycemic incidents than the older class of antidiabetic drugs such as sulfonylureas. One of the major limitations of T2DM models based on the UKPDS risk engine is that they did not model hypoglycemia as a risk factor for diabetes complications. To capture the impact of hypoglycemia, the BRAVO risk engine included severe hypoglycemia as a risk factor to predict CHF, MI, angina, and blindness. Encountering hypoglycemia was also found to be associated with higher CVD-related mortality rates. This feature of the BRAVO risk engine can directly capture the benefits of hypoglycemia prevention on cardiovascular outcomes and mortality for future diabetes models, which is a substantial innovation, compared with previously developed diabetes-related risk engines.
The existence of racial disparities in outcomes among a wide range of diabetic complications [38] made it essential for a diabetes model to include race segmentation relevant to the target population for clinical intervention. The BRAVO risk engine categorizes race into Caucasian, African American, Hispanic, and Asian individuals in accordance with their representation in the US population. The BRAVO risk engine predicted disparity patterns close to the finding from the Karter et al. study [38]. Among all four race groups, being white was associated with the highest risk for MI and nephropathy, while being African American was found to be a protecting factor with the lowest risk for MI, blindness, and the need for revascularization surgery.
One of the important findings in the ACCORD trial was a higher mortality rate in the intensive glycemic control group (HbA1c < 6%) compared with the standard glycemic control group (HbA1c 7.0–7.9%) [39]. The association between HbA1c and mortality rate was found to be ‘U’ shaped in the standard control group, with an optimal HbA1c level between 7.0 and 7.5% [39]. The BRAVO risk engine included a second-degree polynomial in the HbA1c level that fits the data better than a linear relationship between the HbA1c level and all-cause mortality. The BRAVO risk engine estimated the optimal glycemic control level for the ACCORD population was 7.12%, and any deviation from this point was associated with an increased risk of mortality.
Our modeling approach has also demonstrated a novel approach to using clinical trials with a limited length of follow-up time. While the ACCORD trial only ran for 7 years, the ACCORD cohort covered a wide range of diabetes durations. Therefore, the BRAVO risk engine used diabetes duration as a time index to simulate diabetes progression and mortality over 40 years [40]. Further, indexing time by diabetes duration allowed us to estimate the time dependency of diabetes on events and mortality.
A recently published RECODe risk engine has also used the ACCORD trial data to develop a set of risk equations for modeling the risk of diabetes complications [41]. Besides the methodological differences in the modeling strategy, outcomes inclusion, and variable definition, the BRAVO risk engine and RECODe risk engine have very different purposes of risk predictions. The RECODe risk engine is intended for use in clinical settings to assist the initial treatment decision because the models only use baseline characteristics to predict the incidence rates of diabetes outcomes in a specified period (5 or 10 years). The BRAVO risk engine aims to use an agent-based microsimulation modeling algorithm and ultimately to develop a diabetes model in predicting life-time disease progression. Because the BRAVO risk engine uses all time-dependent biomarkers, disease history, and other risk factors, the BRAVO risk engine will be more appropriate than the RECODe risk engine to support a cost-effectiveness analysis such as prioritizing therapeutic strategies or treatment targets of HbA1c, LDL, and blood pressure. A table has been developed to briefly summarize the key differences between the RECODe risk engine and BRAVO risk engine (eTable 4 of the ESM).
The BRAVO risk engine is based on the ACCORD trial population and this study population, especially for the intensive treatment group, may be very different from the general diabetes population. Risk factors progress very differently under different circumstances, and largely depend on factors such as medication adherence, lifestyle modification, and therapy escalation. Further adaptation of the BRAVO risk engine could be conducted to better reflect the natural progression of risk factors in a real-world population using electronic medical records. In addition, a few potential co-morbidities such as ulceration and amputation were not explicitly included in the BRAVO risk engine. These co-morbidities were not included as endpoints in the ACCORD trial, thus cannot be incorporated into the BRAVO risk engine.
In the future, other datasets with relevant outcome measures could be used to either further refine prediction equations or to add supplementary risk equations to the original BRAVO risk engine. Furthermore, all the risk equations were estimated separately and a microsimulation algorithm was used to combine them into one risk engine. Considering the mutually exclusive nature of some complications, this approach might still have competing risk bias. Moreover, the type of antidiabetic drugs was not included in the risk equations because the underlying assumption of the BRAVO risk engine is that all types of treatments, including lifestyle modification, impact the risk of events only through key risk factors (e.g., HbA1c, BMI). We suggest researchers choose the RECODe risk equation to directly explore the impact of different medications. Finally, ESRD was found to be associated with a lower mortality rate in our study, owing to a low sample size. Thus, we decided to exclude ESRD from the all-cause mortality equation. We will revisit this issue when we have the long-term follow-up data from the ACCORD trial.
5. Conclusions
All equations from the BRAVO risk engine have been validated internally, and the external validation also showed a good prediction accuracy for the BRAVO risk engine. A simulation disease model based on the BRAVO risk engine can be developed to predict a range of long-term diabetes-related outcomes to assist clinical and policy decision making.
Supplementary Material
Key Points.
The Building, Relating, Assessing, and Validating Outcomes (BRAVO) risk engine has been developed to predict a series of diabetes complications and mortality
The BRAVO risk engine has found a glycosylated hemoglobin level slightly above 7.0% is associated with the lowest risk for all-cause mortality
With good internal and external validation, the BRAVO risk engine can be applied to develop a diabetes prediction model and to assist decision making in clinical practice and health policy.
Footnotes
Compliance with Ethical Standards
Conflict of interest Hui Shao, Vivian Fonseca, Charles Stoecker, Shuqian Liu, and Lizheng Shi have no conflicts of interest directly relevant to the content of this article.
Electronic supplementary material The online version of this article (https://doi.org/10.1007/s40273-018-0662-1) contains supplementary material, which is available to authorized users.
Data availability
The datasets used for this study are publicly available and can be requested through the National Heart, Lung, and Blood Institute [42].
References
- 1.American Diabetes Association. Economic costs of diabetes in the U.S. in 2007. Diabetes Care. 2008;31(3):596–615. [DOI] [PubMed] [Google Scholar]
- 2.Menke A, Casagrande S, Geiss L, et al. Prevalence of and trends in diabetes among adults in the United States, 1988–2012. JAMA. 2015;314(10):1021–9. [DOI] [PubMed] [Google Scholar]
- 3.Boyle JP, Thompson TJ, Gregg EW, et al. Projection of the year 2050 burden of diabetes in the US adult population: dynamic modeling of incidence, mortality, and prediabetes prevalence. Popul Health Metr. 2010;8(1):29. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.American Diabetes Association. Economic costs of diabetes in the US in 2012. Diabetes Care. 2013;36(4):1033–46. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Uusitupa M, Siitonen O, Aro A, et al. Prevalence of coronary heart disease, left ventricular failure and hypertension in middleaged, newly diagnosed type 2 (non-insulin-dependent) diabetic subjects. Diabetologia. 1985;28(1):22–7. [DOI] [PubMed] [Google Scholar]
- 6.Fowler MJ. Microvascular and macrovascular complications of diabetes. Clin Diabetes. 2008;26(2):77–82. [Google Scholar]
- 7.Palmer AJ, M.H.M. Group. Computer modeling of diabetes and its complications: a report on the Fifth Mount Hood challenge meeting. Value Health. 2013;16(4):670–85. [DOI] [PubMed] [Google Scholar]
- 8.McEwen LN, Karter AJ, Waitzfelder BE, et al. Predictors of mortality over 8 years in type 2 diabetic patients: Translating Research Into Action for Diabetes (TRIAD). Diabetes Care. 2012;35(6):1301–9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.De Cosmo S, Copetti M, Lamacchia O, et al. Development and validation of a predicting model of all-cause mortality in patients with type 2 diabetes. Diabetes Care. 2013;36(9):2830–5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.McEwan P, Bennett H, Ward T, et al. Refitting of the UKPDS 68 risk equations to contemporary routine clinical practice data in the UK. Pharmacoeconomics. 2015;33(2):149–61. [DOI] [PubMed] [Google Scholar]
- 11.Erhardt W, Bergenheim K, Duprat-Lomon I, et al. Cost effectiveness of saxagliptin and metformin versus sulfonylurea and metformin in the treatment of type 2 diabetes mellitus in Germany: a Cardiff diabetes model analysis. Clin Drug Investig. 2012;32(3):189–202. [DOI] [PubMed] [Google Scholar]
- 12.Schwarz B, Gouveia M, Chen J, et al. Cost-effectiveness of sitagliptin-based treatment regimens in European patients with type 2 diabetes and haemoglobin A1c above target on metformin monotherapy. Diabetes Obes Metab. 2008;10(Suppl. 1):43–55. [DOI] [PubMed] [Google Scholar]
- 13.Shafie AA, Gupta V, Baabbad R, et al. An analysis of the shortand long-term cost-effectiveness of starting biphasic insulin aspart 30 in insulin-naive people with poorly controlled type 2 diabetes. Diabetes Res Clin Pract. 2014;106(2):319–27. [DOI] [PubMed] [Google Scholar]
- 14.Palmer AJ, Roze S, Valentine WJ, et al. Impact of changes in HbA1c, lipids and blood pressure on long-term outcomes in type 2 diabetes patients: an analysis using the CORE Diabetes Model. Curr Med Res Opin. 2004;20(S1):S53–8. [DOI] [PubMed] [Google Scholar]
- 15.Charokopou M, McEwan P, Lister S, et al. The cost-effectiveness of dapagliflozin versus sulfonylurea as an add-on to metformin in the treatment of type 2 diabetes mellitus. Diabet Med. 2015;32(7):890–8. [DOI] [PubMed] [Google Scholar]
- 16.Elgart JF, Caporale JE, Gonzalez L, et al. Treatment of type 2 diabetes with saxagliptin: a pharmacoeconomic evaluation in Argentina. Health Econ Rev. 2013;3(1):11. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Hoerger TJ, Zhang P, Segel JE, et al. Cost-effectiveness of bariatric surgery for severely obese adults with diabetes. Diabetes Care. 2010;33(9):1933–9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.CDC, Diabetes Cost-Effectiveness Group. Cost-effectiveness of intensive glycemic control, intensified hypertension control, and serum cholesterol level reduction for type 2 diabetes. JAMA. 2002;287(19):2542–51. [DOI] [PubMed] [Google Scholar]
- 19.Hoerger TJ, Zhang P, Segel JE, et al. Improvements in risk factor control among persons with diabetes in the United States: evidence and implications for remaining life expectancy. Diabetes Res Clin Pract. 2009;86(3):225–32. [DOI] [PubMed] [Google Scholar]
- 20.Clarke P, Gray A, Briggs A, et al. A model to estimate the lifetime health outcomes of patients with type 2 diabetes: the United Kingdom Prospective Diabetes Study (UKPDS) Outcomes Model (UKPDS no. 68). Diabetologia. 2004;47(10):1747–59. [DOI] [PubMed] [Google Scholar]
- 21.Gerstein HC, Riddle MC, Kendall DM, et al. Glycemia treatment strategies in the Action to Control Cardiovascular Risk in Diabetes (ACCORD) trial. Am J Cardiol. 2007;99(12A):34i–43i. [DOI] [PubMed] [Google Scholar]
- 22.Baser OHA, Li L, Wang L. Obese patients in the veteran population in the united states: a health care cost and utilization analysis. Value Health. 2013;16(3):A110. [Google Scholar]
- 23.Buse JB. Action to Control Cardiovascular Risk in Diabetes (ACCORD) trial: design and methods. Am J Cardiol. 2007;99(12):S21–33. [DOI] [PubMed] [Google Scholar]
- 24.Buse JB, A.S. Group. Action to Control Cardiovascular Risk in Diabetes (ACCORD) trial: design and methods. Am J Cardiol. 2007;99(12):S21–33. [DOI] [PubMed] [Google Scholar]
- 25.Barhak J, Isaman DJ, Ye W, et al. Chronic disease modeling and simulation software. J Biomed Inform. 2010;43(5):791–9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Zhou H, Isaman DJM, Messinger S, et al. A computer simulation model of diabetes progression, quality of life, and cost. Diabetes Care. 2005;28(12):2856–63. [DOI] [PubMed] [Google Scholar]
- 27.Fishman G Discrete-event simulation: modeling, programming, and analysis. Berlin: Springer Science & Business Media; 2013. [Google Scholar]
- 28.Gerds TA, Schumacher M. Consistent estimation of the expected Brier score in general survival models with right-censored event times. Biom J. 2006;48(6):1029–10. [DOI] [PubMed] [Google Scholar]
- 29.Subramanian J, Simon R. Overfitting in prediction models: is it a problem only in high dimensions? Contemp Clin Trials. 2013;36(2):636–41. [DOI] [PubMed] [Google Scholar]
- 30.Mount H Computer modeling of diabetes and its complications: a report on the Fourth Mount Hood Challenge Meeting. Diabetes Care. 2007;30(6):1638. [DOI] [PubMed] [Google Scholar]
- 31.Hayes A, Leal J, Gray A, et al. UKPDS outcomes model 2: a new version of a model to simulate lifetime health outcomes of patients with type 2 diabetes mellitus using data from the 30 year United Kingdom Prospective Diabetes Study: UKPDS 82. Diabetologia. 2013;56(9):1925–33. [DOI] [PubMed] [Google Scholar]
- 32.Caro JJ, Moller J, Getsios D. Discrete event simulation: the preferred technique for health economic evaluations? Value Health. 2010;13(8):1056–60. [DOI] [PubMed] [Google Scholar]
- 33.Gerstein HC, Miller ME, Ismail-Beigi F, et al. Effects of intensive glycaemic control on ischaemic heart disease: analysis of data from the randomised, controlled ACCORD trial. Lancet. 2014;384(9958):1936–41. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Ismail-Beigi F, Craven T, Banerji MA, et al. Effect of intensive treatment of hyperglycaemia on microvascular outcomes in type 2 diabetes: an analysis of the ACCORD randomised trial. Lancet. 2010;376(9739):419–30. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35.Zoungas S, Patel A, Chalmers J, et al. Severe hypoglycemia and risks of vascular events and death. N Engl J Med. 2010;363(15):1410–8. [DOI] [PubMed] [Google Scholar]
- 36.Zhao Y, Campbell CR, Fonseca V, et al. Impact of hypoglycemia associated with antihyperglycemic medications on vascular risks in veterans with type 2 diabetes. Diabetes Care. 2012;35(5):1126–32. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37.Shi L, Shao H, Zhao Y, et al. Is hypoglycemia fear independently associated with health-related quality of life? Health Qual Life Outcomes. 2014;12(1):167. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 38.Karter AJ, Ferrara A, Liu JY, et al. Ethnic disparities in diabetic complications in an insured population. JAMA. 2002;287(19):2519–27. [DOI] [PubMed] [Google Scholar]
- 39.Riddle MC, Ambrosius WT, Brillon DJ, et al. Epidemiologic relationships between A1C and all-cause mortality during a median 3.4-year follow-up of glycemic treatment in the ACCORD trial. Diabetes Care. 2010;33(5):983–90. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40.Jackson CH. flexsurv: a platform for parametric survival modelling in R. J Stat Softw. 2016;70(8):1–33. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 41.Basu S, Sussman JB, Berkowitz SA, et al. Development and validation of Risk Equations for Complications Of type 2 Diabetes (RECODe) using individual participant data from randomised trials. Lancet Diabetes Endocrinol. 2017;5(10):788–98. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 42.National Heart Lung and Blood Institute. Action to Control Cardiovascular Risk in Diabetes (ACCORD) data. Available from: https://biolincc.nhlbi.nih.gov/studies/accord/. Accessed 9 Apr 2018. [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Data Availability Statement
The datasets used for this study are publicly available and can be requested through the National Heart, Lung, and Blood Institute [42].