Abstract
Objective To determine whether some participants in the Diabetes Prevention Program were more or less likely to benefit from metformin or a structured lifestyle modification program.
Design Post hoc analysis of the Diabetes Prevention Program, a randomized controlled trial.
Setting Ambulatory care patients.
Participants 3060 people without diabetes but with evidence of impaired glucose metabolism.
Intervention Intervention groups received metformin or a lifestyle modification program with the goals of weight loss and physical activity.
Main outcome measure Development of diabetes, stratified by the risk of developing diabetes according to a diabetes risk prediction model.
Results Of the 3081 participants with impaired glucose metabolism at baseline, 655 (21%) progressed to diabetes over a median 2.8 years’ follow-up. The diabetes risk model had good discrimination (C statistic=0.73) and calibration. Although the lifestyle intervention provided a sixfold greater absolute risk reduction in the highest risk quarter than in the lowest risk quarter, patients in the lowest risk quarter still received substantial benefit (three year absolute risk reduction 4.9% v 28.3% in highest risk quarter; numbers needed to treat of 20.4 and 3.5, respectively). The benefit of metformin, however, was seen almost entirely in patients in the top quarter of risk of diabetes. No benefit was seen in the lowest risk quarter. Participants in the highest risk quarter averaged a 21.4% three year absolute risk reduction (number needed to treat 4.6).
Conclusions Patients at high risk of diabetes have substantial variation in their likelihood of receiving benefit from diabetes prevention treatments. Using this knowledge could decrease overtreatment and make prevention of diabetes far more efficient, effective, and patient centered, provided that decision making is based on an accurate risk prediction tool.
Introduction
The Diabetes Prevention Program was a groundbreaking randomized controlled trial in which the incidence of diabetes was dramatically reduced with a structured lifestyle intervention and significantly reduced with prophylactic metformin in patients at high risk of developing diabetes.1 This study and others like it introduced the prevention of diabetes with metformin and structured lifestyle programs to clinical practice.2 3 The American Diabetes Association now recommends intensive lifestyle interventions or metformin to prevent diabetes for people at high risk.2 However, very few Americans with pre-diabetes being treated with metformin or receiving a structured lifestyle intervention remain.3 Among the factors limiting implementation of diabetes prevention interventions have been the structural difficulties of organizing and financing large scale lifestyle improvement programs and concern that the benefits of preventive metformin treatment might not outweigh its side effects. Understanding which patients receive more or less benefits than average could make treatment both more effective and more efficient.
In many clinical trials, the average benefit reported may be concentrated in a small subset of patients, usually those with the greatest risk of developing the study outcome.4 Because risk of the outcome in untreated patients is a key determinant of the potential for benefit, large clinical trials should routinely conduct a risk stratified assessment of heterogeneity in treatment effect by using a multivariable risk prediction tool to facilitate intelligent tailoring of treatment to individual patients.5 Such information can help us to better personalize the results of clinical trials. By estimating the individual patient’s risk if untreated through use of a risk prediction tool and applying the treatment’s relative risk reduction for people at a similar risk level from a risk stratified trial analysis, we can better estimate a person’s likelihood of benefit in clinical practice.4 5 This is a form of personalized medicine often referred to as benefit based tailored treatment.6
We hypothesized that a risk stratified analysis of the Diabetes Prevention Program could identify people with much higher or much lower than average benefit from interventions to prevent diabetes and that this information could inform better clinical decisions using the principles of benefit based tailored treatment.5 6 7
Methods
The Diabetes Prevention Program
The design, rationale, outcomes, and loss to follow-up of the Diabetes Prevention Program have been described in detail elsewhere.1 8 In brief, the final Diabetes Prevention Program consisted of 3234 patients. Data from the Diabetes Prevention Program were made available through a National Institutes of Diabetes and Digestive and Kidney Diseases repository. Owing to some local institutional review boards’ decisions not to distribute data, our dataset had 3081 participants (95% of full population). All participants had a body mass index of 24 or higher (22 or higher in Asians) and a fasting plasma glucose concentration of 95 to 125 mg/dL (impaired fasting glucose) and a concentration of 140 to 199 mg/dL two hours after a 75 g oral glucose load (impaired glucose tolerance). These criteria were designed to identify patients who were at high risk for developing diabetes, but they are not the same as the American Diabetes Association’s diagnostic criteria for pre-diabetes, which use a cut-off point of 100-125 mg/dL for impaired fasting glucose and do not have a body mass index requirement.2 Eligible patients gave consent and were randomized to standard lifestyle recommendations plus 850 mg of metformin twice daily, an intensive program of lifestyle modification that included 16 lessons with a case manager, or standard lifestyle recommendations plus placebo twice daily. After a median follow-up period of 2.8 (range 1.8-4.6) years, progression to diabetes was reduced by 58% (95% confidence interval 47% to 66%) in the lifestyle modification arm and 31% (17% to 43%) in the metformin arm, both compared with the placebo arm.1
Diabetes prediction model development
Following an established methodology and a pre-specified analytic plan,5 we developed a diabetes risk prediction model based on well established risk factors for diabetes. Following this approach, we developed an internal model using proportional hazards regression to predict risk for progression to diabetes in the Diabetes Prevention Program population. Other research has shown that development of internal models directly on a complete trial population can be done without creating any bias in the estimation of heterogeneity of treatment effect as long as standard processes are used to prevent “over-fitting.” In this technique, the risk model is developed on the entire population of the study. The effect of the treatment arm is added in a second step, along with an interaction term between estimated risk and treatment arm.9
Our model started with 17 potential baseline risk factors that we chose a priori on the basis of their having been found predictive in at least three previous diabetes risk models assessed in a high quality review article10: fasting blood sugar, hemoglobin A1c, age, body mass index (in kilograms per meter of height squared), waist:hip ratio, waist circumference, height, triglycerides, high density lipoprotein cholesterol, systolic blood pressure, and physical activity metabolic equivalent as linear variables; and sex, race (white, black, non-black Latino, and other), family history of diabetes, self reported history of hypertension, self reported history of high blood glucose (including “borderline” high glucose or only during pregnancy), smoking status, and assigned study arm (table 1). (Repeat analyses considering high glucose during pregnancy as not being a high risk condition did not change the results.) We used continuous variables in their original scale. We determined selection of risk factors for inclusion in the final multivariable prediction model by backwards selection, excluding variables that had P>0.1 in the multivariate model. The model had more than 30 outcomes per independent variable considered, making over-fitting very unlikely.11 We used SAS version 9.3 for data management and regression model building. We used the “rms” package in R software version 3.0.17 to perform bootstrapped internal validation. Once the model was developed, we excluded the 17 (0.6%) patients with missing data in the selected predictors from the analysis.
Table 1.
Characteristics of participants. Values are numbers (percentages) unless stated otherwise
| Characteristics | Overall (n=3081) | Placebo (n=1030) | Metformin (n=1027) | Lifestyle (n=1024) |
|---|---|---|---|---|
| Overall diabetes cases | 655 (21.3) | 292 (28.4) | 215 (20.9) | 148 (14.5) |
| Mean (SD) age, years | 50.6 (9.0) | 50.4 (8.8) | 50.8 (8.9) | 50.6 (9.3) |
| Female sex | 2053 (66.6) | 699 (67.9) | 669 (65.1) | 685 (66.9) |
| Race/ethnicity: | ||||
| White | 1768 (57.4) | 586 (56.9) | 602 (58.6) | 580 (56.6) |
| Black | 644 (20.9) | 219 (21.3) | 221 (21.5) | 204 (19.9) |
| Hispanic | 508 (16.5) | 168 (16.3) | 162 (15.8) | 178 (17.4) |
| Other non-white | 161 (5.2) | 57 (5.5) | 42 (4.1) | 62 (6.1) |
| Family history of diabetes | 2127 (69.0) | 715 (69.4) | 699 (68.1) | 713 (69.6) |
| Current smoker | 216 (7) | 80 (7.8) | 69 (6.7) | 67 (6.5) |
| Diagnosis of hypertension | 835 (27.1) | 282 (27.4) | 267 (26.0) | 286 (27.9) |
| History of hyperglycemia | 614 (19.9) | 216 (21.0) | 192 (18.7) | 206 (20.1) |
| Mean (SD) fasting plasma glucose, mg/dL | 107.2 (7.7) | 107.4 (7.8) | 107.3 (7.9) | 107 (7.5) |
| Mean (SD) hemoglobin A1c, % | 5.9 (0.5) | 5.9 (0.5) | 5.9 (0.5) | 5.9 (0.5) |
| Mean (SD) body mass index | 33.5 (5.8) | 33.7 (5.9) | 33.5 (5.8) | 33.4 (5.7) |
| Mean (SD) systolic blood pressure, mm Hg | 124.2 (14.7) | 123.8 (14.4) | 124.5 (14.9) | 124.2 (14.8) |
| Mean (SD) triglycerides, mg/dL | 162.9 (93.5) | 167.2 (92.6) | 158.4 (90.3) | 163 (97.2) |
| Mean (SD) high density lipoprotein cholesterol, mg/dL | 45.6 (11.8) | 44.7 (11.4) | 46 (11.5) | 46.1 (12.5) |
| Mean (SD) leisure physical activity, MET-hour/week of exercise* | 15.9 (25.4) | 16.5 (28.9) | 16.1 (25.2) | 15.2 (21.6) |
| Mean (SD) height, cm | 166.8 (9.2) | 166.5 (9.2) | 167 (9.2) | 166.8 (9.2) |
| Mean (SD) waist circumference, cm | 105.0 (14.6) | 105.1 (14.4) | 104.8 (14.4) | 105.2 (14.9) |
| Mean (SD) waist:hip ratio | 0.9 (0.1) | 0.9 (0.1) | 0.9 (0.1) | 0.9 (0.1) |
*MET denotes metabolic equivalent; MET-hours represent average amount of time engaged in specified physical activities multiplied by MET value of each activity.
We divided the trial populations into quarters of pre-intervention risk on the basis of model predictions and assessed them for discrimination, calibration, Pearson’s median skewness coefficient, and the median:mean risk ratio. The last two of these both assess the amount of skew in a risk score (that is, the amount of “tail”). Pearson’s median skewness coefficient uses a normalized value calculated as 3×(mean−median)/standard deviation. The median:mean risk ratio is a simpler measure—the median risk divided by the mean. It is meant to capture in an intuitive way the risk of the typical patient relative to the average risk in the population. We also used the extreme quarter risk ratio to assess the ability of the risk prediction tool to predict a wide variation of risks. This value has particular benefit in this analysis because patients at extreme predicted probabilities of developing diabetes should have a more straightforward decision about the benefit of treatment.
Assessing heterogeneity of treatment effect
We then analyzed the absolute risk reductions for the interventions after stratification by risk of diabetes.12 We divided all three trial groups into quarters by pre-treatment risk and analyzed them independently. As no pre-specified factors strongly predict treatment-by-risk factor interactions in diabetes prevention, we did not test additional interaction effects as per recommended standards.5
We also did a net benefit assessment on these data.13 A net benefit assessment compares the effects on overall treatment benefit of treating all patients with those of treating smaller, targeted groups of patients at high risk. In the net benefit assessment, we assume that patients have a “cut-off point” below which mildly effective treatments are not worthwhile. The analysis then adjusts for minimum necessary benefit to understand the value of a treatment across the population. This cut-off point represents an estimate of the totality of benefits and harms from treatment. Benefits above the cut-off point represent net benefit. Because of its inherent subjectivity, we assessed a wide range of possible cut-off points (see web appendix for details).
External validation
As an internally developed model will generally have excellent calibration for the population on which it was developed and may also have overly optimistic discrimination, we also repeated the analysis by using the Framingham diabetes risk model.14 The Framingham model was created on a very different population (most notably without the requirement that all patients be at high risk of developing diabetes) and was intended to predict diabetes over seven years. To recognize this, we first evaluated the Framingham score as published with an adjustment for length of follow-up. Because the Framingham results were poorly calibrated to our population (as has often been seen for diabetes prediction models),15 we re-calibrated the score to the trial population by adjustment of both the intercept and the overall slope. The details of this analysis are in the web appendix.
Results
As previously described, the patient population had a high risk of progression to diabetes (table 1).1 In our sample, the study was 21% African-American and 16.5% Latino. The average fasting plasma glucose was 107 mg/dL, and 69.1% of participants had a family history of diabetes.
Assessment of risk models
The internal risk model had seven predictor variables (table 2). When we used this model, a patient’s baseline fasting plasma glucose was by far the most important predictor of development of diabetes. A self reported history of high blood glucose was also quite influential, and the patient’s hemoglobin A1c was predictive independently of the other two factors.
Table 2.
Comparison of prediction models
| Variable | Internal model (hazard ratio) | Framingham model (odds ratio)* |
|---|---|---|
| Age, years | — | 0.99 |
| Sex, male | — | 0.65 |
| Fasting plasma glucose, per 10 mg/dL | 1.98 | 1.15 |
| History of high blood glucose | 1.67 | — |
| Hemoglobin A1c, % | 1.93 | |
| Parental history of diabetes | — | 1.55 |
| Height, cm | 0.985 | — |
| Waist:hip ratio, per 0.1 unit | 1.173 | — |
| Waist circumference, per 10 cm | 1.11 | 1.05 |
| Body mass index | — | 1.04 |
| Systolic blood pressure, mm Hg | — | 1.01 |
| Triglycerides, per 10 mg/dL | 1.02 | 1.00 |
| High density lipoprotein cholesterol, mg/dL | — | 0.96 |
| Intercept | — | −18.607 |
| 3 year baseline survival† | 0.999998 | — |
| C statistic | 0.73 | 0.69 |
| Mean:median risk ratio | 0.72 | 1.09 |
| Pearson’s median skewness coefficient | 1.05 | −0.94 |
| Extreme quarter risk ratio | 6.48 | 2.18 |
*Odds ratios for Framingham model continuous variables are per 1 unit increase.
†Can be used to calculate person’s risk predictions on basis of hazard ratios; practitioners can use nomogram in web appendix.
The internally developed model had a C statistic of 0.73 and excellent calibration (fig 1). People classified in the lowest quarter of risk had between 1.1% and 9.5% three year predicted probability of developing diabetes (mean 6.9%), and those classified in the highest quarter had between 27% and 99% predicted probability (mean 45%) (table 3). This means that the highest risk quarter had a risk that was 6.5 times greater than the risk in the lowest quarter (extreme quarter risk ratio=6.5) (table 2). The risk predictions were strongly skewed to the right, with a mean three year progression rate to diabetes of 21%, but the median patient had only a 15% probability (median:mean risk ratio=0.72) (table 2).
Fig 1 Calibration plot: black dots represent deciles of risk
Table 3.
Three year Diabetes Prevention Program outcome events and Kaplan-Meier event rates by treatment and risk quarter
| Characteristics | Quarter 1 (n=766) | Quarter 2 (n=766) | Quarter 3 (n=766) | Quarter 4 (n=766) | All (n=3064) |
|---|---|---|---|---|---|
| Predicted probability, range | 1.1-9.5% | 9.5-15.1% | 15.1-27.0% | 27.0-99.8% | — |
| Predicted probability, mean | 6.9% | 12.0% | 20.1% | 44.6% | 20.9% |
| Observed rate, mean (events) | 7.1% (46) | 13.7% (88) | 23.4% (155) | 42.9% (297) | 21.9% (586) |
| Placebo rate, mean (events) | 8.3% (19) | 17.8% (37) | 29.1% (73) | 59.6% (140) | 29.1% (269) |
| Metformin rate, mean (events) | 9.6% (21) | 14.9% (32) | 24.4% (53) | 38.2% (86) | 21.9% (192) |
| Lifestyle rate, mean (events) | 3.4% (6) | 8.6% (19) | 15.5% (29) | 31.3% (71) | 14.8% (125) |
| Hazard ratio, metformin (95% CI) (versus placebo) | 1.07 (0.57 to 2.01) | 0.79 (0.49 to 1.28) | 0.82 (0.57 to 1.18) | 0.44 (0.33 to 0.59) | 0.67 (0.55 to 0.81) |
| Hazard ratio, lifestyle intervention (95% CI) (versus placebo) | 0.30 (0.12 to 0.75) | 0.45 (0.26 to 0.79) | 0.43 (0.28 to 0.67) | 0.34 (0.25 to 0.46) | 0.42 (0.34 to 0.52) |
Heterogeneity of treatment benefit
On average, patients in the metformin arm had a substantial average overall reduction in incidence of diabetes, but this benefit was almost entirely a result of those at the highest risk experiencing a dramatic relative risk reduction (as measured by the hazard ratio) and absolute risk reduction (table 3; fig 2). For the lowest quarter of risk of progression to diabetes, the metformin arm had a slightly higher risk of developing diabetes (9.6%) than did the control arm (8.3%). The hazard ratio was non-significant (1.07, 95% confidence interval 0.57 to 2.01). Even in the third quarter (second to highest risk of developing diabetes), patients in the metformin arm only had a 4.7% observed absolute risk reduction (number needed to treat (NNT)=22; hazard ratio 0.82, 0.57 to 1.18). However, in the quarter with the highest risk, patients in the control group had a 59.6% observed rate of developing diabetes and the metformin group had an absolute risk reduction of 21.4% (NNT=4.6; hazard ratio 0.44, 0.33 to 0.59). A significant interaction existed, in which patients with greater risk of diabetes incidence had a greater relative risk reduction from metformin use (P<0.001).
Fig 2 Efficacy plots
Substantial variation in absolute risk reduction also occurred in the lifestyle arm (fig 2). Patients in the lowest quarter of predicted risk had a 4.9% absolute risk reduction (NNT=20.4]), and the absolute risk reduction in the highest risk quarter was 28.3% (NNT=3.5). These findings were entirely due to variation in participants’ baseline risks, as no significant variation was seen in the hazard ratio in the lifestyle arm. The overall population had a hazard ratio of 0.42 compared with placebo. Patients in the lowest risk quarter had an average hazard ratio of 0.30 (0.12 to 0.75), and those in the highest risk had a hazard ratio of 0.34 (0.25 to 0.46). We found no interaction between risk of diabetes incidence and relative risk reduction due to the lifestyle intervention (P=0.16).
Results of sensitivity analysis using Framingham risk score
We examined our results by using an externally developed diabetes prediction model from the Framingham cohort and a recalibrated version of the same model.14 This model had similar discrimination to the internal model, with a C statistic of 0.69. However, after adjustment for length of follow-up, the Framingham model would have predicted that 33.3% of patients in the placebo group would develop diabetes; only 26.2% did. Recalibration (re-estimation of the intercept and slope of the model) resolved this problem.16 Our results were not substantively changed by use of the externally developed Framingham risk tool (details are in the web appendix).
The net benefit assessment found that a tailored approach using risk prediction was substantially more effective than treating everyone in the trial, particularly when the doctor’s or patient’s decision threshold was in the range of 0.05 to 0.2 for metformin and 0.1 to 0.2 for the lifestyle intervention. These values mean that below these thresholds all patients benefit from treatment. For example, if a patient sees no net negatives to taking metformin (including cost or side effects), then the treatment would be worthwhile for the entire population. Above these cut-off points, nobody benefits. For example, if a patient were to want metformin only if the absolute probability that it will prevent diabetes is greater than 20%, then nobody should take it. (Further details are in the web appendix.)
Discussion
In this study, we used well established clinical risk factors to identify which patients at elevated risk of diabetes are the most likely to have incident diabetes prevented by lifestyle modification and metformin. We found that average reported benefit for metformin was distributed very unevenly across the study population, with the quarter of patients at the highest risk for developing diabetes receiving a dramatic benefit (21.5% absolute reduction in diabetes over three years of treatment) but the remainder of the study population receiving modest or no benefit. Although patients at lower risk also received a much smaller absolute reduction from the lifestyle intervention than did higher risk patients, the relative effects of the lifestyle intervention were fairly constant across risk groups (hazard ratio 0.42). These results could decrease drug overuse, help to prioritize lifestyle programs, and be a model for the secondary analysis of randomized trials.
Comparison with other studies
Our risk based analysis is more effective than the conventional “one variable at a time” subgroup analysis, which is plagued by false negative results from poor statistical power and by false positives from multiple comparisons.12 17 In contrast, recent research has shown how risk based examination of treatment heterogeneity examines a critically important clinical question in a single comparison with much better statistical power.5 12 The Diabetes Prevention Program has previously been shown to have subgroup-by-treatment interactions. However, even when a treatment has a consistent relative risk reduction across risk levels, as we found for the Diabetes Prevention Program’s lifestyle intervention, the technique used in this study allows clinicians to recognize large variation in a treatment’s absolute risk reduction between patients on the basis of multivariate risk prediction. The presence (or absence) of a treatment interaction on a proportional scale does not give any direct information about the clinically most important measure of benefit and which patients to treat or not treat.
Our study differs from other analyses of the Diabetes Prevention Program by the method, which we believe should be routinely applied to large randomized trials, and the implications of our findings for the prevention of diabetes. We and others have proposed that a risk stratified analysis such as this one should be a critically important component of the primary analysis of all large clinical trials for three reasons: it provides clinicians and patients with important information on the distribution of the study population’s risk of the outcome; combining risk factors into a single risk score greatly improves statistical power for detecting whether the treatment’s relative benefit (such as hazard ratio or relative risk reduction) is similar in study participants at lower versus higher risk; and it provides clinicians and patients with the ability to better estimate the individual patient’s chance of benefiting from a treatment. This benefit can be estimated by multiplying the relative benefit by the best available estimate of the patient’s risk for the outcome if not treated.
Clinical and research implications
Each of the benefits of a risk stratified analysis is apparent from our analysis of the Diabetes Prevention Program. Firstly, our results show a wide and highly skewed distribution of risk for diabetes within the Diabetes Prevention Program study population, ranging from participants with a 1-2% risk of developing diabetes in the next three years to those with a greater than 90% probability of developing diabetes. Secondly, the risk stratified results showed that the relative benefit (hazard ratio) of the lifestyle intervention was consistent across the population but that the relative benefit for metformin was significantly greater for the highest risk quarter (hazard ratio 0.44) than it was for the remaining 75% of the study population (hazard ratio ranged from 0.79 to 1.07). Most importantly, these results allow for much better estimates of an individual patient’s likelihood of benefiting from intervention (fig 2).
Implementation of these results could make prevention of diabetes much more efficient while lowering rates of adverse events. In fact, large risk based variation in treatment benefit with metformin suggests that outcomes nearly equivalent to treating the entire Diabetes Prevention Program cohort can be achieved by treating only the quarter of the patients at highest risk. As metformin has substantial gastrointestinal side effects and may increase rates of lactic acidosis,18 justifying its use in people who are at lower risk might be difficult.
Our study’s relevance for lifestyle interventions is different from that for metformin. Unlike metformin, most of the lifestyle intervention’s “side effects” are likely to be beneficial, including improved mental health, reduced cardiac risk, enhanced quality of life, and the cosmetic effects of weight loss and improved fitness. Unfortunately, few large scale programs for providing the lifestyle support used in the Diabetes Prevention Program exist, and they are expensive.18 19 Our results suggest that these programs’ cost effectiveness will vary dramatically on the basis of a patient’s risk of developing diabetes, but it remains possible that a lifestyle intervention is cost effective for other reasons.19
This study also showed that progression to diabetes could be effectively predicted with multivariable models. We primarily based our analysis on a diabetes prediction model that was developed using Diabetes Prevention Program data and established risk factors, using an approach that minimizes over-fitting.11 The risk score itself had novel findings. We found that fasting plasma glucose was the strongest predictor for developing diabetes, but that hemoglobin A1c and self reported history of hyperglycemia were also independent risk factors. Waist circumference, height, and waist:hip ratio were additional independent predictors of diabetes. We hypothesize that the differences between these variables and other existing risk prediction models may be due to the very high risk of the Diabetes Prevention Program cohort, the short term follow-up, and the close measurement of more variables than other studies examined. Our risk prediction tool could be used by programming it into an electronic health record, by developing an app, or by using the nomogram provided in the web appendix, similar to what has been done with existing risk tools.14
Our results also highlight the importance of having an accurate risk prediction tool in implementing benefit based tailored treatment. We found that the Framingham score had good discrimination in the Diabetes Prevention Program population but suboptimal calibration (that is, it over-estimated study participants’ risk by about 25%). It is very common for a prediction model to have good discrimination but poor calibration in cohorts that are dissimilar to the population in which they were developed.15 20
A reliable estimate of a patient’s risk if not treated, however, is needed for all good clinical decision making, not just for benefit based tailored treatment. Regardless, following the benefit based tailored treatment strategy will still outperform the “treat all” strategy even using the miscalibrated Framingham model. Appropriate selection of patients would improve with a better risk tool.
Strengths and limitations of study
The primary strength of this study is the high quality, randomized evidence of the Diabetes Prevention Program. Another strength is the analytic technique used. Our primary technique to assess the heterogeneity of treatment effect provides substantial new insight in addition to the analyses used in most clinical trials. This analysis does not have the statistical biases that are found in observational studies, and it minimizes the risks of spurious false positives in approaches based on serial “one variable at a time” subgroup analysis.5 6 12 21
The inconsistent calibration of risk scores is a limitation of this study. The internally derived tool we used was pre-specified but not externally validated. The poor calibration of most diabetes risk tools implies that clinicians should use a risk prediction tool developed in or recalibrated on a patient population similar to their own.15 Ideally, local health systems will test and recalibrate established risk tools in their populations.22 The rise of electronic health records and accountable care organizations should make such internally developed risk tools more practical. Existing examples include risk tools developed in the Kaiser system and those being developed by the US Department of Veterans Affairs.22 23 Furthermore, clinicians often have additional information not accounted for by the risk tool, such as the patient’s personal dislike of drugs or the components of a lifestyle intervention, the severity of a patient’s family history, or the patient’s frailty and comorbidities. The results of the benefit based tailored treatment analysis can help to guide decisions, but good clinicians should also consider other factors.24
Furthermore, although we believe that this type of analysis should be performed regularly, this particular study was not planned until well after the Diabetes Prevention Program completed. Ideally, the relations between risk and estimated benefit would be verified in future research.
This study also has all the limitations of the Diabetes Prevention Program randomized trial from which we obtained the data. Among these limitations are the short term follow-up of the study and that the study showed a reduction in diagnosis of diabetes but was not powered to see a difference in complications of diabetes or the long term stability of the reduction in diabetes. Also, these effects may merely be short term benefits of being on treatment, although the benefit of metformin was seen to continue at least through a two week washout period.25 Finally, our study examines the role of metformin and a lifestyle intervention in prevention of diabetes, but both of these interventions have broader clinical benefits.
Conclusions and policy implications
This study has implications for the clinical prevention of diabetes, for diabetes prevention policy, and for the standard interpretation of randomized trials. We have shown that among patients with a high risk for diabetes, those at the highest risk should be treated most aggressively. Even among patients in the Diabetes Prevention Program, however, most were unlikely to develop diabetes over 2.8 years. Through better risk targeting, lifestyle interventions could become dramatically more efficient and the side effects of metformin could be limited while retaining the effectiveness of the programs.
In conclusion, the Diabetes Prevention Program proved that a lifestyle intervention and metformin can prevent diabetes in some patients with pre-diabetes. Our results show how widely that benefit varies between individual patients with pre-diabetes as a function of their risk of developing diabetes in the near future. Understanding these dramatic variations can enhance our ability to use these interventions more effectively and efficiently by tailoring our decisions about treatment to individual patients’ circumstances and preferences.
What is already known on this topic
The Diabetes Prevention Program randomized controlled trial showed that metformin or a structured lifestyle intervention could reduce transition to diabetes in a population of patients at high risk of diabetes
Most patients in the Diabetes Prevention Program did not develop diabetes
Progression to diabetes can be predicted with risk prediction scores
What this study adds
The benefit of metformin in preventing diabetes is focused almost entirely in people with an extremely high risk of developing diabetes
Patients at high risk of developing diabetes had substantially greater benefit from preventive treatment than did those at lower risk
Preventive treatments could be used far more efficiently with risk stratified analyses of randomized trials
The Diabetes Prevention Program study was conducted by the Diabetes Prevention Program Investigators and supported by the National Institute of Diabetes and Digestive and Kidney Diseases (NIDDK). The data from Diabetes Prevention Program reported here were supplied by the NIDDK Central Repositories. This manuscript was not prepared in collaboration with investigators of the Diabetes Prevention Program study and does not necessarily reflect the opinions or views of the Diabetes Prevention Program Investigators, the NIDDK Central Repositories, or the NIDDK.
Contributors: DMK initially proposed the study. All authors contributed to the study design, planning the data analysis, and interpreting the data. JBS wrote the initial manuscript, and all authors contributed to improving the manuscript. JBS is the guarantor.
Funding: Support for this study was provided by the Patient-Centered Outcomes Research Institute (PCORI: 1IP2PI000722). Additional support was provided by the National Institute of Neurological Disorders and Stroke (U01 AA022802), the Department of Veterans Affairs Quality Enhancement Research Initiative (QUERI DIB 98-001), and the Michigan Center for Diabetes Translational Research (National Institute of Diabetes and Digestive and Kidney Diseases of the National Institutes of Health (P60 DK-20572)).
Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare: no support from any organization for the submitted work; no financial relationships with any organizations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work.
Ethical approval: This study was approved by the institutional review board at the Tufts Medical Center.
Transparency declaration: The lead author (the manuscript’s guarantor) affirms that the manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned have been explained.
Data sharing: Information on the process of obtaining the study dataset is available at the NIDDK Repository website (www.niddkrepository.org/search/study/). The dataset and technical appendix can be obtained by submitting of a formal request to the NIDDK Repository. Requests for the statistical code can be submitted to jnelson2@tuftsmedicalcenter.org.
Cite this as: BMJ 2015;350:h454
Web Extra. Extra material supplied by the author
References
- 1.Knowler WC, Barrett-Connor E, Fowler SE, Hamman RF, Lachin JM,Walker EA, et al. Reduction in the incidence of type 2 diabetes with lifestyle intervention or metformin. N Engl J Med 2002;346:393-403. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.American Diabetes Association. Standards of medical care in diabetes—2013. Diabetes Care 2013;36(suppl 1):S11-66. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Karve A, Hayward RA. Prevalence, diagnosis, and treatment of impaired fasting glucose and impaired glucose tolerance in nondiabetic U.S. adults. Diabetes Care 2010;33:2355-9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Ioannidis JP, Lau J. The impact of high-risk patients on the results of clinical trials. J Clin Epidemiol 1997;50:1089-98 [DOI] [PubMed] [Google Scholar]
- 5.Kent DM, Rothwell PM, Ioannidis JP, Altman DG, Hayward RA. Assessing and reporting heterogeneity in treatment effects in clinical trials: a proposal. Trials 2010;11:85. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Kent DM, Hayward RA. Limitations of applying summary results of clinical trials to individual patients: the need for risk stratification. JAMA 2007;298:1209-12. [DOI] [PubMed] [Google Scholar]
- 7.Sussman JB, Vijan S, Choi H, Hayward RA. Individual and population benefits of daily aspirin therapy: a proposal for personalizing national guidelines. Circ Cardiovasc Qual Outcomes 2011;4:268-75. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Diabetes Prevention Program. Design and methods for a clinical trial in the prevention of type 2 diabetes. Diabetes Care 1999;22:623-34 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Burke JF, Hayward RA, Nelson JP, Kent DM. Using internally developed risk models to assess heterogeneity in treatment effects in clinical trials. Circ Cardiovasc Qual Outcomes 2014;7:163-9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Collins GS, Mallett S, Omar O, Yu LM. Developing risk prediction models for type 2 diabetes: a systematic review of methodology and reporting. BMC Med 2011;9:103. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Peduzzi P, Concato J, Feinstein AR, Holford TR. Importance of events per independent variable in proportional hazards regression analysis: II. Accuracy and precision of regression estimates. J Clin Epidemiol 1995;48:1503-10 [DOI] [PubMed] [Google Scholar]
- 12.Hayward RA, Kent DM, Vijan S, Hofer TP. Multivariable risk prediction can greatly enhance the statistical power of clinical trial subgroup analysis. BMC Med Res Methodol 2006;6:18. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Dorresteijn JA, Visseren FL, Ridker PM, Wassink AMJ, Paynter NP, Steyerberg EW, et al. Estimating treatment effects for individual patients based on the results of randomised clinical trials. BMJ 2011;343:d5888. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Wilson PW, Meigs JB, Sullivan L, Fox CS, Nathan DM, D’Agostino RB, et al. Prediction of incident diabetes mellitus in middle-aged adults: the Framingham Offspring Study. Arch Intern Med 2007;167:1068-74. [DOI] [PubMed] [Google Scholar]
- 15.Mann DM, Bertoni AG, Shimbo D, Carnethon MR, Chen H, Jenny NS, et al. Comparative validity of 3 diabetes mellitus risk prediction scoring models in a multiethnic US cohort: the Multi-Ethnic Study of Atherosclerosis. Am J Epidemiol 2010;171:980-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Steyerberg EW, Borsboom GJ, van Houwelingen HC, Eijkemans MJ, Habbema JD. Validation and updating of predictive logistic regression models: a study on sample size and shrinkage. Stat Med 2004;23:2567-86. [DOI] [PubMed] [Google Scholar]
- 17.Rothwell PM. Treating individuals: 2. Subgroup analysis in randomised controlled trials: importance, indications, and interpretation. Lancet 2005;365:176-86. [DOI] [PubMed] [Google Scholar]
- 18.Inzucchi SE, Bergenstal RM, Buse JB, Diamant M, Ferrannini E, Nauck M, et al. Management of hyperglycemia in type 2 diabetes: a patient-centered approach: position statement of the American Diabetes Association (ADA) and the European Association for the Study of Diabetes (EASD). Diabetes Care 2012;35:1364-79. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.The 10-year cost-effectiveness of lifestyle intervention or metformin for diabetes prevention: an intent-to-treat analysis of the DPP/DPPOS. Diabetes Care 2012;35:723-30. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Ridker PM, Cook NR. Statins: new American guidelines for prevention of cardiovascular disease. Lancet 2013;382:1762-5. [DOI] [PubMed] [Google Scholar]
- 21.Hayward RA, Kent DM, Vijan S, Hofer TP. Reporting clinical trial results to inform providers, payers, and consumers. Health Aff (Millwood) 2005;24:1571-81. [DOI] [PubMed] [Google Scholar]
- 22.Kennedy EH, Wiitala WL, Hayward RA, Sussman JB. Improved cardiovascular risk prediction using nonparametric regression and electronic health record data. Med Care 2013;51:251-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Bellows J, Patel S, Young SS. Use of IndiGO individualized clinical guidelines in primary care. J Am Med Inform Assoc 2014;21:432-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Hayward RA. Moneyball, gambling, and the new cholesterol guidelines. Circ Cardiovasc Qual Outcomes 2014;7:311-4 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Diabetes Prevention Program Research Group. Effects of withdrawal from metformin on the development of diabetes in the diabetes prevention program. Diabetes Care 2003;26:977-80. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.