Observational and Clinical Trial Findings on the Comparative Effectiveness of Diabetes Drugs Showed Agreement

Abstract

Objective

This study compares an observational study of diabetes treatment effectiveness to randomized controlled trials to assess their convergent validity.

Study Design and Setting

Multivariable models were developed using observational data to describe change in HbA1c (% unit) and weight (kilograms) after addition of a second-line oral diabetes drug to metformin monotherapy. Randomized trials of these scenarios were systematically identified. The models were used to simulate each trial, and simulated and actual results were compared by linear regression and meta-analysis.

Results

32 randomized trials of second-line diabetes oral therapy were identified. For all outcomes and drugs studied, simulation and actual results correlated (p < 0.001). There were no statistically significant differences between meta-analyzed randomized and simulated results for effect on HbA1c. For effect on weight, results were qualitatively comparable, but for sulfonylureas the simulated weight gain was nominally greater than seen in the randomized controlled trials.

Conclusions

An observational study of diabetes drug effectiveness showed convergent validity with randomized data. This supports cautious use of the observational research to draw conclusions about drug effectiveness in populations not studied in clinical trials. This approach may be useful in other situations where observational and randomized data need integration.

Introduction

Randomized controlled trial data are generally felt to be the most reliable form of evidence about a drug’s efficacy and safety. But, randomized data are unavailable for many drugs and devices and fail to examine many clinical situations and questions of relevance to practitioners. Analysis of observational data derived from registries, insurance claims, or electronic records is an alternative, but these studies are often subject to bias, particularly confounding by indication. Claims that particular observational studies are valid are difficult to support except through theoretical argument [1]. As a result, observational studies have limited credibility for comparative effectiveness research compared to randomized clinical trials [RCTs]. Efforts to empirically validate observational study results have had inherent limitations.

The most rigorous efforts at validation have been large systematic reviews comparing RCT and observational findings in a wide range of clinical areas [2–5]. The largest of these concluded that observational studies generally agree with RCTs, although discordant results still occur more often than would be expected by chance [6]. Any conclusion from this that observational research is usually reliable was subsequently undermined by a high-profile case in which observational studies of female hormone replacement therapy documented cardiovascular and mortality benefits but a later large scale RCT showed harm [7]. While observational methods are still considered informative, particularly for the study of rare adverse events [5], it remains controversial that their findings are valid enough to be used for clinical decision making.

An alternative approach to comparing RCTs and observational studies has been to conduct single observational studies to retrospectively simulate a particular RCT [8–11]. While these papers have been informative, they have only provided detailed case studies showing that such concordance is achievable in principle. It is difficult to be certain that the findings from such studies are not due to chance or influenced by publication bias (in which studies showing concordance might be more likely to be published).

We attempted a more systematic validation by comparing a previously conducted observational study investigating the effect of second-line oral diabetes medications on hemoglobin A1c (HbA1c) and bodymass index (BMI) against RCT’s addressing the same question. The goal was two-fold.

First, we sought to determine whether the estimates of drug effects from the observational and experimental data would be the same. Such a convergence would strengthen the observational findings, suggesting that they were unbiased. It would also support the utility of the RCT results. There is a common concern that RCTs’ estimates of drug efficacy, because such studies are conducted under artificial conditions and are subject to publication bias, may over-estimate the effectiveness of drugs in routine clinical practice [12; 13]. The concern about publication bias is particularly relevant when most studies are industry-sponsored. Convergence with observational findings would indicate that efficacy and effectiveness are comparable.

Second, we sought support for the use of these observational data to estimate drug effectiveness in specific populations where RCT data are lacking. Validation of the observational data cannot directly address these scenarios, because they are the exact situations where RCT data are not available to serve as a reference. But, a validated observational study can be used to reduce the number of assumptions needed to extend the RCT findings to new populations. To illustrate, we use the example of generalizing findings from RCT’s done in patients with mild diabetes (baseline HbA1c < 8.5%) to more severe diabetics (baseline >= 8.5%).

Methods

A multivariable linear model was developed using observational data as described in a previous publication [14]. Briefly, that model was based on a retrospective observational cohort study conducted in the United Kingdom’s THIN (The Health Information Network) database. It estimated the change in HbA1c (measured in absolute % units) that would occur after addition of a thiazolidinedione, sulfonylurea, or DPP-4 inhibitor to stable metformin monotherapy in patients with type 2 diabetes. Eligible patients were those with 180 days of metformin monotherapy prior to addition of one of those second-line therapies. The cohort was restricted to those with a baseline HbA1c measured between 3 months prior and 7 days after cohort entry. Subsequently measured HbA1c values were treated as outcome data; multivariable hierarchical linear spline models were built to describe the change in HbA1c over time after cohort entry. The class of second-line drug used was included as a categorical exposure variable. Age, sex, baseline hba1c, and baseline body-mass index (BMI) were covariates in the model both alone and as interaction terms with the medication category. The result was a model to estimate the expected follow-up HbA1c as a function of the second-line medication used, the time elapsed since initiation of a second-line drug, and the baseline patient characteristics listed above. The same structure and covariates were used to build a model to describe change in weight (kg), using data from patients with both baseline and follow-up weight measurements. The models differ from the previous publication only in that baseline BMI was treated as a continuous variable and year of treatment initiation was not included in the models (as this variable was not available for patients in the RCTs). These changes were made prior to any review of the findings based on the randomized trial results, and the models were not changed from that point on.

After the models above were developed, from observational data, published randomized controlled trials were collected and their results were compared to the output of the models. The effective comparison was between randomized trial results and causal inferences made from observational data. To minimize any possibility of selection bias, the randomized controlled trials used were identified from meta-analyses that addressed the same clinical question as the observational study did. To that end, the randomized trial literature in Medline was searched using terms developed by another research group - “(metformin) AND (type 2 diabetes mellitus OR T2DM OR noninsulin dependent diabetes OR NIDDM OR glycosylated hemoglobin OR hemoglobin A1c OR HbA1c or A1C)” [15]. Meta-analyses were only included if they had the explicit aim of evaluating the glycemic effect of adding a second diabetes drug to stable ongoing metformin monotherapy. They had to have explicit exclusion criteria including limiting studies to those that established a stable dose of metformin prior to the start of therapy and had a duration of followup between 3 and 12 months.

The publication lists in each meta-analysis did not fully overlap due to differences in publication date and inclusion criteria; for the primary analysis all publications that appeared in any of the eligible meta-analyses were included. In most cases baseline patient characteristics and study outcomes were summarized in the published meta-analyses; in a minority of cases it was necessary to return to the original studies to extract parameters such as the number of patients achieving goal HbA1c. A small number of trials were excluded because they did not provide baseline or outcome covariates. The results of the meta-analyses themselves were not used in this protocol; rather, the meta-analyses were used only to identify relevant trials in an unbiased manner.

Independent review of recent publications on add-on therapy to metformin, outside of previously published meta-analyses, was not attempted. The rationale was that exclusion criteria using lists of studies that were derived by independent investigators prior to formation of our own research question minimized the possibility of a selection bias in which trials were preferentially included that gave results consistent with the observational models.

For each arm of each clinical trial, baseline data were used to create one thousand simulated populations with the same number of individuals and distribution of baseline covariates. Because individual level data (and thereby the joint distributions of baseline covariates) were not available, the amount and direction of interaction between these characteristics were not captured by the simulation. Within each population, the models, based on the findings from the observational study, were used to predict outcomes for each individual at the end of trial followup, and the mean outcome was then calculated. For every study this resulted in 1000 predicted mean outcomes, which were used to provide both an average simulated mean outcome (analogous to the mean outcome from the actual study) and the standard error of the simulated mean (analogous to the standard error of the actual study). (Figure 2)

Figure 2.

Simulation procedure.

The same covariates and model structure were also used to predict the probability of a patient achieving an HbA1c of less than 7.0, except that a non-hierarchical logit linking function was used to model the dichotomous dependent variable. This model, and this model only, was further adjusted post hoc. Specifically, where the other models were designed to function best with a zero intercept, the model to predict probability of reaching goal was not. The fixed components of all three models are included in supplementary data (eTable 1). Results of all three models for a reference patient are shown in Figure 1.

Figure 1.

Modelled outcomes over time for a typical patient (in this case, female, age 60, BMI at baseline 32, HbA1c at baseline 8.5%).

For illustrative purposes, the probability of achieving a goal HbA1c < 7% was calculated for a range of baseline HbA1c’s from 7 to 10 for DPP-4 inhibitors. For this simplified illustration, the simulation assumed a population that was 54% male, had a mean age of 57 years, and mean baseline BMI of 31.4, at 10 months of followup. These were the mean values for the entire pooled population from all the RCT’s. The same graphic was not generated for thiazolidinediones and sulfonylureas, because for those drugs it would be less readily interpretable due to a more complex relationship between effectiveness and duration of followup, sex, and BMI [14].

The real and simulated study results were compared first through weighted linear regression, and then through random-effects meta-analysis of both the real and simulated studies. Weighted linear regression was performed using R’s lm function, with weights calculated as the inverse of the standard error, normalized so that the sum of weights in the dataset was 1. Meta-analyses were performed using R’s metafor function, with a random effects component. All statistical analysis was done using R.

Results

The search identified 4739 publications, 71 of which were meta-analyses potentially meeting the study criteria based upon title. On review of these manuscripts, only two meta-analyses met the inclusion criteria [15, 16]. Most other meta-analyses were excluded because they included studies of monotherapy. Four were excluded because they did not have sufficiently clear inclusion criteria, and in particular appeared to include studies in which metformin therapy was still being actively titrated when the second agent was added [17,18,19,20]. These four excluded meta-analyses identified only two studies that would have met inclusion criteria but were not included in the eligible meta-analyses [21,22]. Per protocol, these studies were not included in the primary analysis but were used in sensitivity analysis.

The relevant RCT’s identified from the two eligible meta-analyses consisted of 32 published trials comprising 45 active treatment arms that studied sulfonylurea, thiazolidinedione, or DPP-4 inhibitor (Table 1).(23–55) Several studies identified in the meta-analyses were not usable due to missing data on baseline variables.(56–58) A more detailed summary is provided in supplementary data (eTable 2). Not all studies provided all outcomes of interest. Change in HbA1c from baseline was reported in 12 sulfonylurea arms, 8 thiazolidinedione arms, and 19 DPP-4 arms. Change in weight from baseline was reported in 8 sulfonylurea arms, 6 thiazolidinedione arms, and 13 DPP-4 arms. Three additional studies with adequate baseline information reported on HbA1c change and on weight change, but did not provide standard errors for these outcomes; these studies were included only in sensitivity analyses, in which standard errors were imputed [24,27,29]. Percentage of patients achieving HbA1c < 7% was reported in 8 sulfonylurea arms, 3 thiazolidinedione arms, and 12 DPP-4 arms. Baseline HbA1c in the trials ranged from 6.4 to 9.9%, with a median of 8.0%. Duration of followup ranged from 3 to 12 months, with a median of 6 months.

Table 1.

Randomized controlled trial arms included in the simulation study. SU = sulfonylurea, DPP4 = DPP-4 inhibitor, and TZD = thiazolidinedione.

First Author	Year of Publication	Drug Class	N	Followup (Months)	% Male	Mean Age	Baseline BMI	Baseline HbA1c	Change in HbA1c	Change in Weight	% Achieving HbA1c < 7%
Einhorn (23)	2000	TZD	168	4	55%	56	32.1	9.86	−0.64	0.95
Fonseca(24)	2000	TZD	110	6	68%	58	29.8	8.9		1.9
Ahren(25)	2004	DPP4	56	3	70%	58	29.4	7.7	−0.6	−0.4	42%
Feinglos(26)	2005	SU	61	4	46%	58	31.7	7.45	−0.66	0.4	69%
Matthews(27)	2005	SU	313	12	49%	57	32.6	8.53		1.4
		TZD	317	12	51%	56	32.6	8.71	−0.99	1.5
Charbonnel(28)	2006	DPP4	464	6	56%	54	30.9	7.96	−0.67	−0.65	47%
Kvapil(29)	2006	SU	114	4	46%	58	30.5	9.4	−1.7	0.1
Ristic(30,31)	2006	SU	129	6	50%	62	29.5	7.6	−0.57	0.42
		SU	99	12	51%	62	30.0	7.55		0.91	47%
Umpierez(32)	2006	SU	96	6	55%	52	34.5	8.4	−1.3	1.74	56%
		TZD	107	6	52%	56	33.8	8.31	−1.23	1.85	55%
Ahren(33)	2007	DPP4	42	12	62%	58	29.6	7.6		−0.2	43%
Bosi (34)	2007	DPP4	143	6	62%	54	33.2	8.4	−0.9	0.2
Nauck(35)	2007	DPP4	588	12	57%	57	31.3	7.7	−0.51	−1.5	63%
		SU	584	12	61%	57	31.2	7.6	−0.56	1.1	59%
Bolli(36)	2008	DPP4	295	6	62%	56	32.2	8.4	−0.9	0.3
		TZD	281	6	64%	57	32.1	8.4	−1	1.9	36%
Hamann(37)	2008	SU	288	12	52%	59	32.2	8	−0.86	1.6
		TZD	285	12	53%	59	33.0	8	−0.78	2.7
Khanolkar(38)	2008	SU	25	6	60%	56	33.7	7.08	−1
		TZD	25	6	56%	59	34.6	7.33	−1.19
Raz(39)	2008	DPP4	96	7	51%	54	30.1	8.4	−0.9	0.13	22%
Scott(40)	2008	DPP4	91	4	55%	55	30.3	7.8	−0.73	−0.4	55%
		TZD	87	4	63%	55	30.4	7.7	−0.79	1.5	63%
Defronzo(41)	2009	DPP4	191	6	54%	55	31.2	8.1	−0.69	−0.87	44%
Ferrannini(42)	2009	DPP4	1396	12	53%	58	31.8	7.31	−0.44	−0.23	54%
		SU	1393	12	54%	58	31.7	7.3	−0.53	1.56	56%
Goodman(43)	2009	DPP4	125	6	53%	55	31.7	8.5	−0.66	−0.19
Kaku(44)	2009	TZD	83	7	66%	52	25.6	7.58	−0.67	1.68
Nauck(45)	2009	DPP4	213	6	47%	55	32.0	7.9	−0.6		44%
Nauck(46)	2009	SU	242	6	57%	57	31.2	8.4	−1	1	36%
Bergenstal(47)	2010	DPP4	166	6	52%	52	32.0	8.5	−0.9	−0.8
		TZD	165	6	48%	53	32.0	8.5	−1.2	2.8
Filozof(48)	2010	DPP4	519	12	52%	59	31.2	8.5	−0.81	0.08	30%
		SU	494	12	52%	60	30.8	8.5	−0.85	1.36	32%
Forst(49)	2010	DPP4	66	3	56%	62	31.7	8.5	−0.5	−0.57	15%
Goke(50)	2010	DPP4	428	12	50%	58	31.5	7.7	−0.74	−1.1
		SU	430	12	54%	58	31.3	7.7	−0.8	1.1
Pratley(51)	2010	DPP4	219	6	55%	55	32.6	8.5	−0.9	−0.96
Arechavaleta(52)	2011	DPP4	516	7	50%	56	31.5	7.5	−0.46	−0.8
		SU	519	7	54%	56	31.3	7.5	−0.52	1.2
Taskinen(53)	2011	DPP4	523	6	53%	57	29.9	8.09	−0.49	−0.4
Yang(54)	2011	DPP4	278	6	48%	54	26.3	7.9	−0.78	−1.05	47%
Yang(55)	2011	SU	215	4	58%	54	25.3	8.5	−1.39	0.1	44%

For change in HbA1c, the predictions of the observational model were correlated with the clinical trial results (p < 0.001), with r² of 0.50. Meta-analysis of the randomized trial results for sulfonylureas yielded an HbA1c change of −0.83% absolute units (95% CI −1.00 to −0.66). Meta-analysis of simulated trial results gave a change of −0.87% (95% CI −1.02 to −0.71). For thiazolidinediones, the clinical trial results were −0.93% (95% CI −1.11 to −0.77), compared to a simulated outcome of −0.98% (95% CI −1.05 to −0.90). For DPP-4 inhibitors, the clinical trial results were −0.67% (95% CI −0.75 to −0.60) compared to a simulated outcome of −0.75% (95% CI −0.81 to −0.69). (Figure 3)

Figure 3.

Comparison of real and simulated trial results for mean change in HbA1c. Upper left panel is real versus expected outcome for all studies, weighted by the standard error of the effect estimate. More heavily weighted studies having larger bubbles. Red bubbles are sulfonylurea results, green bubbles are thiazolidinedione results, and blue bubbles are DPP-4 inhibitor results. Upper right panel is real (black) and simulated (red) individual study results for sulfonylureas, with random-effects metaanalysis. Bottom left panel is real and simulated individual study results for thiazolidinediones, with random-effects meta-analysis. Bottom right panel is real and simulated individual study results for DPP-4 inhibitors, with random-effects meta-analysis.

For change in weight, the predictions of the observational model were correlated with the clinical trial results (p < 0.001), and with an r² of 0.77. Meta-analysis of the randomized trial results for sulfonylureas yielded a weight gain of 1.16 kg (95% CI 0.83 to 1.49). Meta-analysis of simulated trial results gave a gain of 1.51 kg (95% CI 1.43 to 1.59). For thiazolidinediones, the clinical trial results were 1.93 kg (95% CI 1.37 to 2.50), compared to a simulated outcome of 1.45 kg (95% CI 0.87 to 2.04). For DPP-4 inhibitors, the clinical trial results were weight loss, −0.54 kg (95% CI −0.83 to −0.24) compared to a simulated weight loss of −0.50 kg (95% CI −0.58 to −0.42). (Figure 4)

Figure 4.

Comparison of real and simulated trial results for mean change in weight. Upper left panel is real versus expected outcome for all studies, weighted by the standard error of the effect estimate. More heavily weighted studies having larger bubbles. Red bubbles are sulfonylurea results, green bubbles are thiazolidinedione results, and blue bubbles are DPP-4 inhibitor results. Upper right panel is real (black) and simulated (red) individual study results for sulfonylureas, with random-effects metaanalysis. Bottom left panel is real and simulated individual study results for thiazolidinediones, with random-effects meta-analysis. Bottom right panel is real and simulated individual study results for DPP-4 inhibitors, with random-effects meta-analysis.

For probability of achieving HbA1c < 7%, the predictions of the observational model were correlated with the clinical trial results (p < 0.02), with an r2 of 0.40. Meta-analysis of the randomized trial results for sulfonylureas yielded a probability of 45% (95% CI 38% to 53%) Meta-analysis of simulated trial results gave a probability of 41% (95% CI 35% to 47%). For thiazolidinediones, the clinical trial results were 51% (95% CI 35% to 67%), compared to a simulated outcome of 34% (95% CI 30% to 39%). For DPP-4 inhibitors, the clinical trial results were 39% (95% CI 33% to 45%) compared to a simulated outcome of 41% (95% CI 37% to 45%). (Figure 5)

Figure 5.

Comparison of real and simulated trial results for probability of achieving HbA1c < 7%. Upper left panel is real versus expected outcome for all studies, weighted by the standard error of the effect estimate. More heavily weighted studies having larger bubbles. Red bubbles are sulfonylurea results, green bubbles are thiazolidinedione results, and blue bubbles are DPP-4 inhibitor results. Upper right panel is real (black) and simulated (red) individual study results for sulfonylureas, with random-effects meta-analysis. Bottom left panel is real and simulated individual study results for thiazolidinediones, with random-effects meta-analysis. Bottom right panel is real and simulated individual study results for DPP-4 inhibitors, with random-effects meta-analysis.

For illustration, the probability of achieving goal HbA1c < 7% in an average patient was calculated for a range of baseline HbA1c’s from 7 to 10 for DPP-4 inhibitors. The simulation results were plotted compared to the corresponding RCT results. Results for DPP-4 inhibitors are shown in Figure 6. For DPP- 4 inhibitors all RCT’s were conducted in populations with average baseline HbA1c between 7% and 8.5% while the simulated results allow for extrapolation beyond this range.

Figure 6.

Probability of achieving HbA1c < 7% as a function of baseline HbA1c for DPP-4 inhibitor use. Blue circles represent RCT’s of DPP-4 inhibitor use, with size proportional to inverse of standard error. The red line represents estimated response based on the observational model.

In sensitivity analysis, the four meta-analyses that did not have sufficiently clear inclusion/exclusion criteria were examined for any additional RCTs that would have otherwise met criteria for this analysis. When those studies were included in the analysis, results were not substantively changed. Similarly when the studies that did not provide information on the standard errors of outcomes were included, results were not substantively changed.

Discussion

This analysis shows close agreement and thereby supports the convergent validity between observational research and randomized trial data on the effectiveness of oral medications for diabetes. The clinical trial data strengthen the effectiveness estimates from the observational studies, indicating that they are not severely biased or confounded. Similarly, the observational data strengthen the results from the clinical trials by showing that the efficacy demonstrated in RCT’s is similar to the drugs’ effectiveness in routine clinical practice.

The validity of the observational research is demonstrated by two different measures. More statistically robust, but less intuitive, is the highly significant correlation between predicted and actual clinical trial results (Figures 3a, 4a, and 5a), which is accompanied by r² values ranging from 0.40 to 0.76, indicating that the model based on the observational data captures a large portion of the variation between the different clinical trial results. This supports a conclusion that the observational data reveals true and causal associations.

The direct comparison of the actual magnitude of the changes in outcomes is more clinically meaningful and is highly consistent between the observational models versus the randomized trials. This is true both for HbA1c lowering, where a similar modest decrease is seen in all drug classes, and in weight change, where the drug classes have distinctly different effects that the observational data accurately detect. The overall conclusion is that any systematic bias making the observational data give different results from the randomized data is small, and clinically negligible.

However, attempts to validate observational findings beyond the limits of the randomized trials suffer from a catch-22. The observational data are most needed to describe drug effects in scenarios not studied with RCT’s; however, without RCT’s the observational data cannot be directly validated. For example, efficacy of DPP-4 inhibitors in patients with baseline HbA1c < 8.5 is well studied in clinical trials (Figure 6), so observational findings can be validated for this population. But the observational data are most needed to estimate drug effectiveness in patients with baseline HbA1c > 8.5, a population that accounts for over half of the patients receiving DPP-4 inhibitors in clinical practice [14] but is not well-represented in clinical trials.

If the RCT data alone are used, doing so requires major assumptions – for example, that the linear association between drug efficacy and baseline HbA1c does not change at higher HbA1c levels. With observational data alone, the untestable assumption of no unmeasured confounding has to be made. However, when the observational and RCT data are considered together, such large assumptions are not necessary. Instead, a more modest assumption is needed: that the observational data, which appear to be unbiased for patients with low baseline HbA1c, will continue to be unbiased at higher baseline HbA1c. This assumption is supported by the lack of any suggestion in our results that the RCT and observational estimates diverge as baseline HbA1c rises (Figure 6). Thus, demonstration of convergent validity between observational and RCT data in one population helps to support appropriately cautious conclusions about additional categories of patients. In the absence of an ambitious expanded pragmatic clinical trials agenda, this may be the most practical way to extend RCT results to unstudied populations.

In this instance, the resulting conclusion – that above an HbA1c of 8.5 the chance of achieving goal HbA1c with the addition of a single oral agent to metformin is less than 1/3 – has clinical implications. It suggests that proceeding directly to triple drug or injection therapy should be more strongly considered in such patients.

Our study has limitations. The comparison of observational results to RCT results is retrospective rather than prospective. Furthermore, the observational model necessarily omitted variables that might help predict treatment response but were not readily available in the observational study or consistently reported by the RCTs – for example, duration of diabetes prior to addition of the new drug, ethnicity, concomitant lifestyle changes, and a host of other factors.

In conclusion, systematic comparison of results from an observational study with findings from multiple RCT’s yields useful information. First, it tests the hypothesis that observational and RCT results converge. If so, this provides evidence that effectiveness and efficacy are similar, and that the observational study is relatively unbiased. Second, the validated observational study can then be used to extend the RCT results to new, unstudied clinical scenarios with more confidence than would have been possible with either data source alone. Because the robustness of this method relies on the availability of multiple comparable RCTs addressing the same clinical question, it is of greatest value for clinical issues where a large volume of clinical trial data is available but important comparative effectiveness questions remain. Such possible areas, in addition to the treatment of diabetes, include the study of weight loss drugs, cholesterol-lowering drugs, antihypertensive therapy, and antidepressants.