Novel Risk Engine for Diabetes Progression and Mortality in USA: Building, Relating, Assessing, and Validating Outcomes (BRAVO)

Abstract

Background

There is an urgent need to update diabetes prediction, which has relied on the United Kingdom Prospective Diabetes Study (UKPDS) that dates back to 1970 s’ European populations.

Objective

The objective of this study was to develop a risk engine with multiple risk equations using a recent patient cohort with type 2 diabetes mellitus reflective of the US population.

Methods

A total of 17 risk equations for predicting diabetes-related microvascular and macrovascular events, hypoglycemia, mortality, and progression of diabetes risk factors were estimated using the data from the Action to Control Cardiovascular Risk in Diabetes (ACCORD) trial (n = 10,251). Internal and external validation processes were used to assess performance of the Building, Relating, Assessing, and Validating Outcomes (BRAVO) risk engine. One-way sensitivity analysis was conducted to examine the impact of risk factors on mortality at the population level.

Results

The BRAVO risk engine added several risk factors including severe hypoglycemia and common US racial/ethnicity categories compared with the UKPDS risk engine. The BRAVO risk engine also modeled mortality escalation associated with intensive glycemic control (i.e., glycosylated hemoglobin < 6.5%). External validation showed a good prediction power on 28 endpoints observed from other clinical trials (slope = 1.071, R² = 0.86).

Conclusion

The BRAVO risk engine for the US diabetes cohort provides an alternative to the UKPDS risk engine. It can be applied to assist clinical and policy decision making such as cost-effective resource allocation in USA.

1. Introduction

The prevalence of type 2 diabetes mellitus (T2DM) in USA has risen from 4.21% (12.1 million) in 2002 [1] to 9.1% in 2012 [2]. The overall prevalence of diabetes in USA is projected to reach to 21% in 2050 [3]. This increase in the T2DM population has led to dramatically increased costs in managing diabetes. The total costs of diabetes increased approximately 41% from US$174 billion in 2007 [1] to US$245 billion in 2012. In 2012, patients with T2DM incurred 20% of total healthcare expenditures in USA, more than half of which was attributable to treating diabetes and its complications [4]. A majority of the diabetes-related costs were the result of micro/macrovascular complication events [5]. The most common diabetes-related macrovascular events include myocardial infarction (MI), congestive heart failure (CHF), and stroke. The most frequent diabetes-related microvascular events include retinopathy (e.g., edema, blindness), nephropathy [e.g., end-stage renal disease (ESRD)], and neuropathy [e.g., severe pressure sensation loss (SPSL), amputation] [6].

To better manage the growing T2DM population in an environment of constrained healthcare resources, systemwide improvement and redesign are necessary. In the ‘bigdata’ era, this is possible using outcome-driven and evidence-based diabetes management. Prediction models can help to develop sophisticated and well-designed diabetes management strategies. Models also better profile the risk of patients so that more healthcare resources can be effectively allocated to those with more health needs. Several diabetes models in USA have been used to describe disease progression and compare the cost effectiveness of different therapeutic strategies: the CORE diabetes model, the University of Michigan model for diabetes, the Swedish Institute of Health Economics model otherwise known as the Economics and Health Outcomes in T2DM Model, the United Kingdom Prospective Diabetes Study (UKPDS) outcomes model, the Centers for Disease Control-Research Triangle Institute diabetes cost-effectiveness model, the Cardiff Research Consortium model [7], and several others [8, 9]. These models have been used to support the outcome-driven evidence-based diabetes management in several areas such as comparisons between therapeutic plans [10–16], evaluating potential benefits of achieving treatment goals, [14] and policy impact on T2DM [17–19].

However, these diabetes models rely heavily on the UKPDS risk engine that was developed using data from a UK diabetes cohort collected from the 1970s. The UKPDS population differs significantly from the current US population in terms of race/ethnicity, definition of diabetes, treatment algorithm, and screening methods to assess complications and comorbidities. Further, the baseline hazard of diabetes-related events may vary over time [20] and may differ between the UKPDS population and current US population [21]. Using a UK-based risk engine to predict US diabetes management raised significant concerns on the prediction validity. There is an urgent need for a new risk engine to be developed based on a US population to support decision making in clinical practice in USA. Therefore, our goal in this study was to develop the Building, Relating, Assessing, and Validating Outcomes (BRAVO) Diabetes Risk Engine based on the US diabetes population, which may provide an alternative risk engine for US researchers and policy makers.

2. Research Design and Methods

2.1. Data Source

The Action to Control Cardiovascular Risk in Diabetes (ACCORD) trial is one of the largest studies ever conducted in adults with T2DM mainly in USA. Three potential strategies were tested through this multicenter clinical trial to lower the risk of major cardiovascular events: intensive blood sugar control, intensive blood pressure control, and treatment to lower blood lipid levels. There were 10,251 participants enrolled in this trial, with an average follow-up time of approximately 4 years. All diabetes-related events were recorded periodically during the study. The ACCORD trial started its enrollment in January 2001 [22], and the baseline demographic characteristics were reported in the Buse et al. ACCORD study [23, 24]. For the convenience of our readers, we also include the baseline information in eTable 1 of the Electronic Supplementary Material (ESM).

2.2. Risk Equations

Compared with the commonly used Markov approach in several current models [18, 25, 26], the BRAVO risk engine was developed based on a series of discrete intercorrelated risk equations. This discrete-equation approach accounted for risk escalation as diabetes progressed and for interactions between complications. It also allows for adjusting a large number of demographic and biological characteristics [27].

The BRAVO risk engine contains three separate modules (events module, risk factors module, and mortality module), each of which contains a series of regression equations to predict the occurrence of events, progression in risk factors, and mortality. In the events module, a series of risk equations were fitted to predict diabetes-related macrovascular events (stroke, MI, CHF, angina, and revascularization surgery), microvascular events (ESRD, blindness, and SPSL), and adverse events (severe hypoglycemia and symptomatic hypoglycemia). In the risk factors module, each risk factor [glycosylated hemoglobin (HbA_1c), systolic blood pressure (SBP), weight, and low-density lipoprotein (LDL)] in the current cycle was predicted jointly by its value from the last cycle and other risk factors. Last, in the mortality module, an equation was fitted to predict patient death, and a second equation was developed to explore the cause of death [i.e., cardiovascular disease (CVD) or other death causes]. All the clinical outcomes have been defined in an article published by the ACCORD trial group [23].

A literature review has been conducted to identify the initial list of explanatory variables (eTable 2 of the ESM) and a backward selection process was conducted to remove those with no improvement for model fitting. Explanatory variables can be categorized into three groups: biomedical factors, demographic characteristics, and complications. The definitions of these explanatory variables are provided in eTable 2 of the ESM. The main modeling strategy we used in this study was a left-censored, time-dependent, parametric proportional hazard model, in which diabetes duration was used as the time index, instead of real time in the clinical trial. To smooth measurement fluctuation for biomarkers, we applied a moving average technique: as the models were developed based on annual cycles, all the parameters values should be aggregated annually. We aggregated the parameters values for each year by averaging all the measurements conducted within the previous 2 years. Parameters values from the current year were used to predict the probability of encountering an outcome event in the next year to account for potential bias caused by reverse causality. The history of events was also included in the initial list of explanatory variables. Having had an event at baseline or in the study periods before the current year would both be identified as a history of that event for the current year. A mixed-method algorithm including a cross-validation-based, backward model selection process, literature review, and consultation from endocrinologists was used to support the model fitting process. For binary outcomes, the c-statistic has been applied to measure the discrimination power of the model.

However, in the BRAVO risk engine, we were not only interested in the discrimination power of the model, but also the prediction accuracy. Thus, both the c-statistic and Brier score [28] were calculated to support the model selection process. For continuous outcomes [HbA_1c, SBP, LDL, and body mass index (BMI)], the mean square prediction error was used to select the models. A ten-fold cross-validation framework was applied to adjust the c-statistic, Brier score, and mean square prediction error for possible over-fitting in low-dimension regressions [29]. All risk factors that improved model performance were included into the final model. For those risk factors that did not have a significant impact on model performance, inclusion and exclusion were judged by clinical endocrinology knowledge and evidence found from the current literature. Risk factors that were not statistically significant in the BRAVO model selection processes, but that were supported as risk factors by existing clinical evidence were included in the risk equations. Details regarding the functional form of each risk equation and model selection process are provided in Online Appendix 1.

2.3. Internal and External Validation Process

The internal validation process was conducted by plotting the predicted cumulative hazard against the Kaplan–Meier cumulative hazard for all outcome measures. We also calculated the log–log 95% confidence interval (CI) for cumulative incidence rate across a diabetes duration of 0–40 years in the ACCORD trial using the left-truncated method, and the predicted curve was examined if it fell within the 95% CI of the Kaplan–Meier curve.

We included all the clinical trials from the fourth and fifth Mount Hood Challenges to conduct the external validation process [7, 30], which were the Atorvastatin Study for Prevention of Coronary Heart Disease Endpoints in Non-Insulin-Dependent Diabetes Mellitus (ASPEN) trial, the Action in Diabetes and Vascular Disease: Preterax and Diamicron Modified Release Controlled Evaluation (ADVANCE) trial, and the Collaborative Atorvastatin Diabetes Study (CARDS) trial.

A total of 28 endpoints were predicted using the BRAVO risk engine through a discrete-time event microsimulation process (Online Appendix 2), under the corresponding time horizon of each trial [31]. The advantage of this modeling strategy over the traditional Markov cohort model has been discussed previously [32]. The baseline characteristics of each trial have been reported [7] and applied directly as the characteristic of simulation samples. Normal distribution was assumed for all input variables and the standard error of each variable was extracted from the corresponding literature. The values of key risk factors, including HbA_1c, LDL, SBP, and BMI in each validation trial were assumed to reach the corresponding treatment target at the first year and remain constant in the following years. Then, 10,000 simulation runs were used to reach convergence in outcomes. An ordinary least-square model was used to fit the BRAVO-predicted incidence rates to observed incidence rates, and slope, intercept, and R² were used to show prediction accuracy. The microsimulation was conducted using a joint program written through Visual Basic and C++ language.

In addition, we conducted one-way sensitivity analyses to explore the impact of six risk factors on the life expectancy in the ACCORD trial population. The values of continuous risk factors (HbA_1c, SBP, LDL, and BMI) were set from one standard deviation below the mean to one standard deviation above, while categorical risk factors (smoking and severe hypoglycemia) ranged from 50 to 200% of the population average. Results from this analysis were reported in Online Appendix 4.

3. Results

The median follow-up time for patients in the ACCORD trial was 3.7 years, with a total of 39,043 person-years of data to support the model fitting process for developing the BRAVO risk engine. Hazard ratios of risk factors for non-fatal MI, stroke, CHF, angina, revascularization surgery, ESRD, blindness, and SPSL are presented in Table 1. In addition, Table 1 also presents hazard/odds ratios of prediction equations for all-cause mortality and CVD death. The inter-correlations between different co-morbidities were captured by using other co-morbidities as risk factors to predict the occurrence of the target event. For example, in the CHF equation, having a history of MI increased the risk of CHF by 82.2%. In addition, repeat events were also taken into consideration. Having a history of CHF was associated with a 247.6% increase in the risk of a second CHF event.

Table 1.

Hazard/odds ratio estimates of risk equations in the Building, Relating, Assessing, and ValidatingOutcomes (BRAVO) risk engine

Abbreviations	Strokea	CHFa	MIa	Anginaa	Revasca
HbA1c	1.353 (1.165–1.570)	1.108 (1.001–1.227)	1.179 (1.086–1.281)	1.208 (1.093–1.335)	1.070 (1.011–1.133)
HbA1c*HbA1c
SBP	1.029 (1.019–1.040)	1.017 (1.011–1.023)			1.006 (1.002–1.010)
LDL	1.009 (1.005–1.013)		1.007 (1.005–1.009)	1.004 (1.000–1.008)
BMI		1.060 (1.041–1.079)		1.027 (1.007–1.048)	1.182 (1.057–1.321)
BMI*BMI					0.998 (0.996–1.000)
Age at diagnosis	1.047 (1.025–1.070)	1.058 (1.043–1.072)	1.031 (1.019–1.044)
Severe hypoglycemia		1.980 (1.259–3.114)	2.286 (1.569–3.331)	1.885 (1.102–3.225)
Female			0.747 (0.620–0.900)		0.731 (0.639–0.837)
Education		0.579 (0.440–00.761)	0.681 (0.553–0.838)
Smoking			1.195 (0.924–1.545)
Race (ref = black)
White			1.687 (1.310–2.172)		1.464 (1.227–1.746)
Hispanic			1.267 (0.840–1.913)		1.110 (0.830–1.483)
Others			1.279 (0.893–1.831)		1.181 (0.922–1.511)
MI_history	1.846 (1.302–2.617)	1.822 (1.440–2.305)	1.627 (1.325–1.999)	1.461 (1.134–1.881)
CHF_history		3.476 (2.689–4.494)	1.242 (0.997–1.547)
Stroke_history	3.080 (2.053–4.622)		1.603 (1.318–1.950)		1.377 (1.114–1.702)
Angina_history				1.674 (1.297–2.159)	1.408 (1.208–1.640)
Revasc_history		2.212 (1.759–2.782)		3.290 (2.570–4.212)	2.328 (2.050–2.644)
Blindness_history
Stroke_event
CHF_event
Log(duration)
Log(scale)	10.028	8.029	6.311	7.986	7.093
Log(shape)	0.586	0.728	0.538	0.065	0.264
Brier score	0.003 (0.003–0.004)	0.008 (0.007–0.009)	0.012 (0.011–0.013)	0.007 (0.007–0.008)	0.025 (0.024–0.027)
C-statistics	0.750 (0.711–0.784)	0.785 (0.766–0.809)	0.689 (0.662–0.704)	0.727 (0.705–0.757)	0.670 (0.654–0.687)
Function	Weibull	Weibull	Weibull	Weibull	Weibull
Abbreviations	ESRDa	Blindnessa	SPSLa	All-cause mortalitya	CVD deathb
HbA1c	1.129 (1.003–1.269)	1.190 (1.105–1.282)	1.328 (1.253–1.409)	0.512 (0.186–1.408)	1.297 (1.069–1.574)
HbA1c*HbA1c				1.048 (0.982–1.118)
SBP	1.014 (1.006–1.022)	1.015 (1.009–1.021)	1.010 (1.006–1.014)		1.014 (1.001–1.028)
LDL		1.004 (1.002–1.006)	1.002 (1.000–1.004)
BMI				1.017 (0.999–1.035)	1.035 (1.000–1.070)
BMI*BMI
Age at diagnosis		1.021 (1.011–1.031)	1.024 (1.014–1.034)
Severe hypoglycemia		1.517 (1.015–2.268)
Female			0.725 (0.637–0.826)	0.576 (0.459–0.724)
Education		0.741 (0.615–0.892)		0.725 (0.567–0.929)
Smoking				1.765 (1.305–2.387)
Race (ref = black)
White		1.081 (0.885–1.320)	1.224 (1.044–1.434)
Hispanic		1.701 (1.282–2.255)	0.847 (0.634–1.132)
Others		1.179 (0.895–1.555)	0.552 (0.420–0.726)
MI_history				1.224 (0.971–1.542)	2.153 (1.382–3.356)
CHF_history	1.672 (1.097–2.548)			2.175 (1.656–2.856)	2.145 (1.234–3.730)
Stroke_history				1.388 (1.005–1.918)	2.073 (1.045–4.111)
Angina_history				1.517 (1.181–1.950)
Revasc_history
Blindness_history	1.992 (1.257–3.157)		1.366 (1.020–1.829)
Stroke_event				3.380 (1.640–6.967)	7.456 (0.859–64.731)
CHF_event				5.686 (3.957–8.171)	5.948 (2.352–15.045)
Log(duration)					1.250 (0.918–1.703)
Log(scale)	6.945	6.431	6.399	2.452	−5.959
Log(shape)	0.259	0.592	0.501	−6.336
Brier score	0.005 (0.005–0.006)	0.016 (0.015–0.017)	0.027 (0.025–0.028)	0.010 (0.009–0.011)	0.212 (0.191–0.222)
C-statistics	0.592 (0.569–0.619)	0.632 (0.612–0.657)	0.661 (0.648–0.681)	0.617 (0.585–0.653)	0.731 (0.693–0.780)
Function	Weibull	Weibull	Weibull	Gompertz	Logistic

95% confidence intervals are reported in parentheses

BMI body mass index, CHF congestive heart failure, ESRD end-stage renal disease, HbA_1c glycosylated hemoglobin, LDL low-density lipoprotein, MI myocardial infarction, ref reference, Revasc revascularization surgery, SBP systolic blood pressure, SPSL severe pressure sensation loss

^aParameters are reported as hazard ratios

^bParameters are reported as odds ratios

Race was identified as a significant risk factor for predicting MI, revascularization surgery, blindness, and nephropathy. Compared with African Americans, Caucasian, Hispanic, and other race/ethnicities were associated with 68.7% (95% CI 31.0–117.2), 26.7% (95% CI – 16.0 to 91.3), and 27.9% (95% CI – 10.7 to 83.1) higher risks for MI, respectively. In addition, Caucasian, Hispanic, and other race/ethnicities were associated with a 46.4% (95% CI 22.7–74.6), 11.0% (−17.0 to 48.3), and 18.1% (95% CI −7.8 to 51.1) higher likelihood of receiving revascularization surgery, compared with African Americans. Caucasian, Hispanic, and other race/ethnicities were correlated with a risk escalation of 8.1% (95% CI – 11.5 to 32.0), 70.1% (95% CI 28.2–125.5), and 17.9% (95% CI – 10.5 to 55.5), respectively, compared with African Americans for blindness. Last, Hispanic individuals and others were associated with risk reductions of SPSL for 15.3% (95% CI −13.2 to 36.6) and 44.8% (95% CI 27.4–58.0), respectively, while Caucasians were associated with a 22.4% (95% CI 4.4–43.4) higher risk for SPSL compared with African Americans.

The BRAVO risk engine included severe hypoglycemia in multiple risk equations. Encountering one more episode of severe hypoglycemia in the current year was associated with increased risks for CHF [hazard ratio (HR) = 198%], MI (HR = 228.6%), angina (HR = 188.5%), and blindness (HR = 151.7%). Although not statistically significant, quadratic polynomials of HbA_1c levels were also found to be an important predictor for predicting all-cause mortality as indicated by Bier scores and c-statistics. An HbA_1c level of 7.12% was calculated to be associated with the lowest mortality risks. The equations to model time-varying risk factors including HbA_1c, SBP, LDL, body weights, smoking status, and occurrence of severe hypoglycemia and symptomatic hypoglycemia are presented in eTable 3 of the ESM.

The results of our internal validation process are presented in Fig. 1. The black dashed line denotes the Kaplan–Meier curve of the observed cumulative incidence for each event type using the left-truncated method. The red solid line indicates the predicted cumulative incidence of a given event from diabetes onset until 40 years after onset. The gray area denotes the log–log 95% CI for the observed cumulative incidence at each time point. All predicted incidence curves fit close to the observed curves and all predicted curves were within the 95% CI.

Fig. 1.

Comparison between predicted cumulative incidence and Kaplan–Meier observed cumulative incidence. CHF congestive heart failure, CI confidence interval, CVD cardiovascular disease, ESRD end-stage renal disease, MI myocardial infarction, SPSL severe pressure sensation loss

The results of external validation are presented in Fig. 2. Predicted incidence rates were plotted against observed incidence rates, with a dashed line indicating 100% prediction accuracy. The incidence rates of 28 endpoints, which were predicted from the BRAVO risk engine, were clustered around the prediction accuracy line. The slope coefficient and intercept of the ordinary least-square regression on the predicted incidence rates were 1.071 and 0.001, respectively, and the R² was estimated to be 0.86. An F-test was applied to examine the difference between the fitted line and the line of 100% prediction accuracy, and no statistical difference was identified (p = 0.82).

Fig. 2.

External validation results plotting predicted incidence rates against observed incidence rates. CHF congestive heart failure, CVD cardiovascular disease, MI myocardial infarction

4. Discussion

The BRAVO risk engine with good internal and external validity was developed to offer an alternative to the established UKPDS risk engine that was based on the US population, to support decision making in US clinical practices. Health outcome predictions from the BRAVO risk engine were consistent with the previous findings of the ACCORD trial on non-fatal MI, angina, and revascularization surgery [33]. Although not statistically significant, a previous study has found that intensive glycemic control was associated with lower HRs of ESRD, blindness, and SPSL [34]. The BRAVO risk engine included HbA_1c as an important risk factor for predicting these microvascular events. More interestingly, even when including the extensive covariate list from our BRAVO risk engine, these associations between HbA_1c and microvascular events were still statistically significant in the BRAVO risk equations.

The impact of hypoglycemia on diabetes outcomes and mortality has been studied extensively in recent years. The occurrence of hypoglycemia was found to be associated with major microvascular and microvascular events, death, and other nonvascular outcomes [35]. Our previous study also found that in addition to the direct impact of hypoglycemia on vascular risk [36], the fear for hypoglycemia was also associated with an additional quality-adjusted life-year decrement [37]. Our risk engine is the first to fully incorporate hypoglycemia’s impact on disease course. Our engine provides a critical predictive tool to evaluate new T2DM drugs, which usually have lower hypoglycemic incidents than the older class of antidiabetic drugs such as sulfonylureas. One of the major limitations of T2DM models based on the UKPDS risk engine is that they did not model hypoglycemia as a risk factor for diabetes complications. To capture the impact of hypoglycemia, the BRAVO risk engine included severe hypoglycemia as a risk factor to predict CHF, MI, angina, and blindness. Encountering hypoglycemia was also found to be associated with higher CVD-related mortality rates. This feature of the BRAVO risk engine can directly capture the benefits of hypoglycemia prevention on cardiovascular outcomes and mortality for future diabetes models, which is a substantial innovation, compared with previously developed diabetes-related risk engines.

The existence of racial disparities in outcomes among a wide range of diabetic complications [38] made it essential for a diabetes model to include race segmentation relevant to the target population for clinical intervention. The BRAVO risk engine categorizes race into Caucasian, African American, Hispanic, and Asian individuals in accordance with their representation in the US population. The BRAVO risk engine predicted disparity patterns close to the finding from the Karter et al. study [38]. Among all four race groups, being white was associated with the highest risk for MI and nephropathy, while being African American was found to be a protecting factor with the lowest risk for MI, blindness, and the need for revascularization surgery.

One of the important findings in the ACCORD trial was a higher mortality rate in the intensive glycemic control group (HbA_1c < 6%) compared with the standard glycemic control group (HbA_1c 7.0–7.9%) [39]. The association between HbA_1c and mortality rate was found to be ‘U’ shaped in the standard control group, with an optimal HbA_1c level between 7.0 and 7.5% [39]. The BRAVO risk engine included a second-degree polynomial in the HbA_1c level that fits the data better than a linear relationship between the HbA_1c level and all-cause mortality. The BRAVO risk engine estimated the optimal glycemic control level for the ACCORD population was 7.12%, and any deviation from this point was associated with an increased risk of mortality.

Our modeling approach has also demonstrated a novel approach to using clinical trials with a limited length of follow-up time. While the ACCORD trial only ran for 7 years, the ACCORD cohort covered a wide range of diabetes durations. Therefore, the BRAVO risk engine used diabetes duration as a time index to simulate diabetes progression and mortality over 40 years [40]. Further, indexing time by diabetes duration allowed us to estimate the time dependency of diabetes on events and mortality.

A recently published RECODe risk engine has also used the ACCORD trial data to develop a set of risk equations for modeling the risk of diabetes complications [41]. Besides the methodological differences in the modeling strategy, outcomes inclusion, and variable definition, the BRAVO risk engine and RECODe risk engine have very different purposes of risk predictions. The RECODe risk engine is intended for use in clinical settings to assist the initial treatment decision because the models only use baseline characteristics to predict the incidence rates of diabetes outcomes in a specified period (5 or 10 years). The BRAVO risk engine aims to use an agent-based microsimulation modeling algorithm and ultimately to develop a diabetes model in predicting life-time disease progression. Because the BRAVO risk engine uses all time-dependent biomarkers, disease history, and other risk factors, the BRAVO risk engine will be more appropriate than the RECODe risk engine to support a cost-effectiveness analysis such as prioritizing therapeutic strategies or treatment targets of HbA_1c, LDL, and blood pressure. A table has been developed to briefly summarize the key differences between the RECODe risk engine and BRAVO risk engine (eTable 4 of the ESM).

The BRAVO risk engine is based on the ACCORD trial population and this study population, especially for the intensive treatment group, may be very different from the general diabetes population. Risk factors progress very differently under different circumstances, and largely depend on factors such as medication adherence, lifestyle modification, and therapy escalation. Further adaptation of the BRAVO risk engine could be conducted to better reflect the natural progression of risk factors in a real-world population using electronic medical records. In addition, a few potential co-morbidities such as ulceration and amputation were not explicitly included in the BRAVO risk engine. These co-morbidities were not included as endpoints in the ACCORD trial, thus cannot be incorporated into the BRAVO risk engine.

In the future, other datasets with relevant outcome measures could be used to either further refine prediction equations or to add supplementary risk equations to the original BRAVO risk engine. Furthermore, all the risk equations were estimated separately and a microsimulation algorithm was used to combine them into one risk engine. Considering the mutually exclusive nature of some complications, this approach might still have competing risk bias. Moreover, the type of antidiabetic drugs was not included in the risk equations because the underlying assumption of the BRAVO risk engine is that all types of treatments, including lifestyle modification, impact the risk of events only through key risk factors (e.g., HbA_1c, BMI). We suggest researchers choose the RECODe risk equation to directly explore the impact of different medications. Finally, ESRD was found to be associated with a lower mortality rate in our study, owing to a low sample size. Thus, we decided to exclude ESRD from the all-cause mortality equation. We will revisit this issue when we have the long-term follow-up data from the ACCORD trial.

5. Conclusions

All equations from the BRAVO risk engine have been validated internally, and the external validation also showed a good prediction accuracy for the BRAVO risk engine. A simulation disease model based on the BRAVO risk engine can be developed to predict a range of long-term diabetes-related outcomes to assist clinical and policy decision making.

Key Points.

The Building, Relating, Assessing, and Validating Outcomes (BRAVO) risk engine has been developed to predict a series of diabetes complications and mortality

The BRAVO risk engine has found a glycosylated hemoglobin level slightly above 7.0% is associated with the lowest risk for all-cause mortality

With good internal and external validation, the BRAVO risk engine can be applied to develop a diabetes prediction model and to assist decision making in clinical practice and health policy.

Data availability

The datasets used for this study are publicly available and can be requested through the National Heart, Lung, and Blood Institute [42].