Exploration of Residual Confounding in Analyses of Associations of Metformin Use and Outcomes in Adults With Type 2 Diabetes

This cohort study examines residual confounding in observational studies of off-label metformin in adults with type 2 diabetes.

Key Points

Question

Do common designs for observational studies of metformin for off-label outcomes sufficiently address concerns about residual confounding?

Findings

In this cohort study of 404 458 individuals with type 2 diabetes, metformin was associated with reduced hospitalization and lower annual medical expenditures. These findings were challenged with 50 negative control outcomes and a complementary cohort of 81 791 individuals with prediabetes; these 2 validation measures exposed the residual confounding driving the primary results.

Meaning

These findings suggest that commonly used observational study designs for drug-repurposing studies of metformin are inadequate without a more thorough investigation of potential residual confounding.

Abstract

Importance

Metformin is often used as a first-line therapy for type 2 diabetes; however, frequent discontinuation with reduced kidney function and increased disease severity indicates that a comparison with any other group (eg, nonusers or insulin users) must address significant residual confounding concerns.

Objectives

To examine the potential for residual confounding in a commonly used observational study design applied to metformin and to propose a more robust study design for future observational studies of metformin.

Design, Setting, and Participants

This retrospective cohort study with a prevalent user design was conducted using an administrative claims database for Medicare Advantage beneficiaries in the US. Participants were categorized into 2 distinct cohorts: 404 458 individuals with type 2 diabetes and 81 791 individuals with prediabetes. Clinical history was observed in 2018, and end points were observed in 2019. Statistical analyses were conducted between May and December 2021.

Exposures

Prevalent use (recent prescription and history of use on at least 90 of the preceding 365 days) of metformin or insulin but not both at the start of the observation period.

Main Outcomes and Measures

Total inpatient admission days in 2019 and total medical spending (excluding prescription drugs) in 2019. Each of these measures was treated as a binary outcome (0 vs >0 inpatient days and top 10% vs bottom 90% of medical spending).

Results

The study included 404 458 adults with type 2 diabetes (mean [SD] age, 74.5 [7.5] years; 52.7% female). A strong metformin effect estimate was associated with reduced inpatient admissions (odds ratio, 0.60; 95% CI, 0.58-0.62) and reduced medical expenditures (odds ratio, 0.57; 95% CI, 0.55-0.60). However, implementation of additional robust design features (negative control outcomes and a complementary cohort) revealed that the estimated beneficial effect was attributable to residual confounding associated with individuals’ overall health, not metformin itself.

Conclusions and Relevance

These findings suggest that common observational study designs for studies of metformin in a type 2 diabetes population are at risk for consequential residual confounding. By performing 2 additional validation checks, the study design proposed here exposes residual confounding that nullifies the initially favorable claim derived from a common study design.

Introduction

Observational study designs commonly attribute benefits to metformin across a host of end points and diseases. Metformin is a generic, glucose-lowering medication that is frequently considered for drug repurposing for indications such as postoperative mortality,¹ asthma,^2,3 chronic obstructive pulmonary disease,⁴ acute kidney injury,⁵ heart failure,⁶ and age-related macular degeneration.⁷ These observational study designs commonly compare users of metformin for type 2 diabetes (T2D) with nonusers^1,3,4,5,6 or insulin users^2,8; others pool heterogeneous groups of people with or without T2D exposed to metformin vs no metformin.^2,7 Observational studies of metformin, with varying levels of preclinical or prospective support, have suggested a host of pleiotropic benefits associated with metformin, including cancer treatment and prevention,^9,10 antiaging,¹¹ neurodegenerative disease prevention,¹² and sepsis mortality mitigation.¹³

A principal challenge when studying metformin using a retrospective cohort design is metformin’s lack of an active comparator. When metformin is used to treat T2D, its ubiquitous use as a first-line therapy and frequent discontinuation with reduced kidney function and increased disease severity means that a comparison with any other group (eg, nonusers or insulin users) must address significant residual confounding concerns. The progression toward T2D can take years, and the current care path recognizes 2 distinct states along this trajectory: prediabetes and T2D. For individuals with prediabetes who are at high risk for progressing to T2D, treatment with metformin is strongly recommended to prevent or delay progression.¹⁴ For those diagnosed with T2D, the recommended pharmaceutical treatment plan typically begins with metformin, may involve adding other drugs, and may transition to insulin as diabetes progresses.^15,16,17 This treatment paradigm implies that metformin users generally represent the portion of the prediabetes population with the most severe disease and the portion of the T2D population with the least severe disease. Thus, in 2 populations in which evidence-based metformin use is expected, known differences exist in disease severity between metformin users and any potential comparison group.

To reveal why one drug may be attributed with such a wide range of benefits, we constructed an observational study with elements reflective of what is frequently seen in the recent metformin literature. There is considerable diversity in this literature with regard to overall design (eg, incident use and prevalent use), comparison group selection (eg, metformin users vs nonusers and metformin users vs insulin users), confounder selection (eg, general health indicators, diabetes severity indicators, comorbidity indices, laboratory values, and demographic characteristics), and statistical methods (eg, univariate or multivariate regression, matching, and various propensity score models) (the eAppendix in the Supplement contains a literature review of 20 metformin-repurposing studies published in the JAMA, BMJ, and The Lancet families of journals between 2015 and 2022). We explored the role residual confounding plays in these designs, making no effort to fix the limitations we believe exist, such as rigorous adjustment for diabetes duration, glycemic control, number and type of background diabetes treatments, or social determinants of health. We then expanded our study design to include negative control outcomes (ie, outcomes with no direct, mechanistic connection to metformin) and a complementary cohort (ie, prediabetes) for which we expected any bias observed in the T2D population to be reversed. Of importance, this study does not claim to have crafted the perfect metformin study design; instead, we aimed to assess whether common designs for observational studies of metformin are subject to substantial residual confounding, hypothesizing that the inclusion of a panel of negative control outcomes and a complementary cohort could reveal residual confounding.

Methods

Study Design

We used a retrospective cohort study design to compare metformin users with insulin users from a T2D population (see the comparison of metformin users with nonusers in the eAppendix in the Supplement). We expand this basic design to include a second cohort in which we compare metformin users with nonusers from a prediabetes population, an approach we call the complementary cohort design. The UnitedHealth Group Office of Human Research Affairs approved this project as negligible risk based on the secondary use of deidentified claims data, with a waiver of informed consent. This study followed the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guideline.

Data and Study Population

Data

This study used Optum Labs’ deidentified administrative claims data for individuals with Medicare Advantage or commercial insurance. The index date for the study was January 1, 2019. Calendar year 2018 represents the baseline period for all individuals in the study (ie, pharmacy claims indicating treatment, medical claims documenting comorbidities and health care use, and other covariates are all observed in 2018). Calendar year 2019 was the observation period for selected outcomes.

Study Population

Eligible individuals were aged 18 to 89 years, had both medical and pharmacy coverage with 24 months of continuous enrollment spanning calendar years 2018 and 2019, and had at least 1 medical claim with a primary, secondary, or tertiary diagnosis of prediabetes or T2D in 2018. Self-reported race was not systematically available for the study population and thus was not reported. The T2D cohort served as the primary analysis cohort because it represents most metformin users and conforms to the standard practice in the literature. Table 1 summarizes the available sample sizes, and Figure 1 presents the preadjustment and postadjustment covariate balance. Secondary analyses of different cohorts, listings of qualifying generic drug names, International Statistical Classification of Diseases and Related Health Problems, Tenth Revision (ICD-10) codes, and other claims-based logic are presented in the eAppendix in the Supplement.

Table 1. Sample Sizes for the Various Cohorts Under Study^a.

Attribute	Commercially insured		Medicare Advantage
	Prediabetes	Type 2 diabetes	Prediabetes	Type 2 diabetes
Cohort size	17 256	101 826	83 901	926 671
Metformin user	1176	36 998	3086	331 085
Insulin user	0	6203	0	73 373
Nonuser	14 762	22 447	78 705	228 934
Excluded	1318	36 178	2110	293 279

^a Metformin and insulin users met days’ supply and recency requirements for 2018. Nonusers had no pharmacy claims for any diabetes drug in 2018. Excluded individuals failed to qualify for the metformin, insulin, or nonuser groups as defined in the exposure subsection (insufficient days’ supply, insufficient recency, and/or prescribed both metformin and insulin at some point in 2018). Preadjustment and postadjustment covariate balance for these cohorts appears in Figure 1 and in the eAppendix in the Supplement.

Figure 1. Covariate Balance Plots.

A, Two overlaid balance plots (one for each outcome) for the type 2 diabetes cohort, each depicting both the unadjusted covariate balance and the balance achieved after adjustment using inverse propensity weighting (IPW). Negative values indicate that the metformin group has lower means and prevalence for the indicated covariates (in the case of sex, a higher percentage of males). The unadjusted balance markers indicate that the metformin users have spent less time in the hospital, have fewer or less severe comorbidities (lower Elixhauser Comorbidity Index in-hospital mortality scores), and appear healthier than insulin users across all Diabetes Complications Severity Index (DCSI) components. Across both outcomes, the IPW approach achieved satisfactory balance after adjusting for every observed covariate of interest. B, Two overlaid balance plots (one for each primary outcome) for the prediabetes cohort. Across both outcomes, the IPW approach achieved satisfactory balance after adjusting for every observed covariate of interest. The observed bias related to overall health and disease severity in the unadjusted analysis is mostly nullified in comparison with the type 2 diabetes cohort depicted in panel A.

Outcome, Exposure, and Covariate Definitions

We show results for 2 outcomes: inpatient admission days and total annual medical spending (excluding prescription drugs). With these outcomes, we intentionally avoid targeting specific diseases or body systems, allowing our models to rely on a set of baseline covariates applicable to any T2D study. The primary outcomes use thresholds for inpatient admission days (binary, >0 days) and total medical spending (binary, ≥90th percentile for the respective cohort). The cost outcome includes only medical claims and therefore excludes pharmaceutical claims due to the varying diabetes drug costs between exposure groups (ie, insulin cost > metformin cost > cost of no diabetes drugs). For each of 50 negative control outcomes (selected based on the absence of a well-established direct, mechanistic connection to metformin, eg, toenail fungus and low back pain), the outcome of interest is the presence or absence of at least 1 claim for that outcome in 2019.

Three prevalent use exposure groups are used. Nonusers are individuals who have no pharmacy claim for diabetes drugs in 2018. Metformin users are individuals with a cumulative 90-day or greater supply of metformin in 2018, a daily supply of metformin lasting to at least the final 30 days of 2018, and no supply of any insulin drug in 2018. Insulin users are individuals with a cumulative 90-day or greater supply of insulin in 2018, a daily supply of insulin lasting to at least the final 30 days of 2018, and no supply of metformin in 2018. These exposure groups replicate those commonly seen in published study designs. Any individual who did not meet any of these 3 exposure group definitions was excluded to ensure mutually exclusive cohorts. These exposure group definitions do not account for the presence or absence of additional diabetes drugs.

All adjusted models include covariates obtained from 2018 enrollment and administrative data documenting individuals’ sex, age (and age squared and age cubed), urban or suburban vs rural residence (determined by individuals’ postal codes), number of inpatient admission days, record of at least 1 wellness visit, presence of a medical or pharmacy claim for an influenza vaccine, Elixhauser Comorbidity Index (in-hospital mortality) score,¹⁸ history of the outcome (ie, ≥1 instance of a qualifying ICD-10 code for the outcome in 2018), and the 7 Diabetes Complications Severity Index¹⁹ component scores: retinopathy (0, 1, or 2), nephropathy (0, 1, or 2), neuropathy (0 or 1), cerebrovascular (0, 1, or 2), cardiovascular (0, 1, or 2), peripheral vascular disease (0, 1, or 2), and metabolic (0, 1, or 2). Inpatient stays and Elixhauser Comorbidity Index scores are log transformed (x′ = ln[x + 1]).

Statistical Analysis

Statistical analyses were conducted between May and December 2021. The following procedure explains how we obtain odds ratios (ORs) and 95% CIs for each of the primary and negative control outcomes in each cohort: (1) assess a subset of individuals meeting only 1 of the 2 exposure definitions (eg, metformin users and insulin users in T2D), (2) conduct an unadjusted analysis using the Fisher exact test to obtain an OR and corresponding CI, (3) estimate the propensity for metformin treatment using logistic regression and all observed covariates listed previously, (4) trim the sample to include only individuals who have a propensity for metformin treatment greater than or equal to the larger of the first percentiles of the 2 exposure groups’ propensity score distributions, and (5) fit an inverse propensity-weighted logistic regression to estimate the treatment effect of metformin using all observed covariates listed previously. A 2-sided P < .05 was considered to be statistically significant. We conducted the statistical analysis using R, version 4.1 (R Foundation).

Results

Primary Results

The study included 404 458 adults with T2D (mean [SD] age, 74.5 [7.5] years; 52.7% female). Replicating a common observational study design of individuals with T2D, we observed a strong metformin effect estimate associated with reduced inpatient admissions (OR, 0.60; 95% CI, 0.58-0.62) and reduced medical expenditures (OR, 0.57; 95% CI, 0.55-0.60) in the Medicare Advantage T2D population (Table 2). Figure 1 shows the preadjustment and postadjustment covariate balance achieved in models for the 2 example outcomes (see the eAppendix in the Supplement for propensity score distributions). Although metformin users appear healthier across numerous health indicators, inverse propensity weighting yields acceptable covariate balance in both the T2D and prediabetes cohorts, evidenced by the standardized mean differences for all covariates with absolute values less than 0.1. These strong effect sizes, extremely small P values, large E-values,²⁰ and covariate balance plots could suggest that metformin is associated with these outcomes.

Table 2. Primary Outcome Results^a.

Outcome	Odds ratio (95% CI)	P value	E-value
Inpatient days
Unadjusted	0.37 (0.36-0.38)	<10−323	4.84
IPW logistic	0.60 (0.58-0.62)	<10−231	2.73
Medical spending
Unadjusted	0.29 (0.29-0.30)	<10−323	6.29
IPW logistic	0.57 (0.55-0.60)	<10−201	2.87

Abbreviation: IPW, inverse propensity weighting.

^a Treatment effect estimates for metformin: this study was conducted in a Medicare Advantage type 2 diabetes population comparing metformin users with a control group of insulin users. The outcomes represent more than 0 inpatient admission days in 2019 and total medical spending exceeding the 90th percentile of all type 2 diabetes beneficiary expenditures (>$25 793). With such strong estimated effect sizes and small P values (no corrections applied for multiple comparisons), metformin appears strongly associated with fewer inpatient admission days and lower health care costs, even after adjustment for a range of relevant covariates.

Results in Context

We next considered the same results as part of a more robust study design that involved negative control outcomes and a complementary prediabetes cohort (see “Validation Tools” in the eAppendix in the Supplement for expanded discussion). Of importance, the negative control outcomes expose the biases in the primary and complementary cohorts. Covariate balance plots for all 50 negative control outcomes in both cohorts appear in the eAppendix in the Supplement.

To best illustrate the symmetry of the expected biases between cohorts, ORs were converted to log ORs in Figure 2 and Figure 3. We expected negative experiments to lead to log ORs of 0 on average. In the T2D cohort, metformin users were biased toward lower event rates (log OR <0) across a vast number of outcomes spanning many body systems. This finding suggests that, although the observed covariates in Figure 1A are well balanced after adjustment, substantial residual confounding associated with overall health may influence our primary result. In the prediabetes cohort, the metformin users and nonusers were similar before balancing (Figure 1B), making the groups appear comparable. Nevertheless, this same collection of negative control outcomes exhibited the opposite bias for metformin users (ie, a shift toward higher event rates as seen in Figure 2). Despite efforts to control for potential confounding, this result highlights the inadequacy of our study design to sufficiently address residual confounding associated with overall health and disease severity and confirms that the prediabetes cohort is biased in the opposite direction of the T2D cohort, making it an ideal complementary cohort.

Figure 2. Residual Confounding Plot.

Forest plots depict log odds ratios (ORs) for metformin treatment effects for 50 negative control outcomes in 2 Medicare Advantage cohorts: prediabetes (nonuser control group) and type 2 diabetes (insulin control group). Mean log ORs for the unadjusted and inverse propensity-weighted logistic regression models are denoted by vertical orange and blue lines, respectively. In prediabetes, positive bias makes metformin appear harmful (ie, mean log OR >0). In type 2 diabetes, the same modeling approach yields a multitude of significant, left-biased results suggesting treatment benefits (ie, mean log OR <0). Outcomes span many body systems and a wide range of conditions.

Figure 3. Results in Context.

Colored vertical lines indicating the treatment effect estimates for unadjusted and adjusted models are overlaid on distributions of treatment effect estimates obtained for negative control outcomes using identical modeling approaches. For both outcomes (inpatient admission days and total medical spend), the estimated treatment effect of metformin in the type 2 diabetes population appears strong, exceeding the benefit observed for most negative control outcomes. Although this finding suggests the effect might be real, the effect estimates for inpatient admission days and total medical spend are substantially reversed in the complementary prediabetes cohort. Of note, primary results are converted to log₂ – odds ratios (ORs) for visual comparison to the distribution of negative controls (ie, for type 2 diabetes inpatient admission days, OR of 0.60 = log₂OR of −0.74; for type 2 diabetes total medical spending, OR of 0.57 = log₂OR of −0.80).

Finally, our conclusions using the combined insights from the negative control outcomes and complementary cohort design differ from what typical study designs would conclude (eAppendix in the Supplement). For both primary outcomes (inpatient admission days and total medical spending), we originally observed acceptable covariate balance and obtained P values no larger than 10⁻²⁰¹ for beneficial treatment effects, which is where many published studies stop. If we go a step further and consider a few or even 50 negative control outcomes, the observed effect might appear to be stronger than nearly all the negative control effects (Figure 3), indicating that although bias exists, there is still hope of a real treatment effect. However, in the complementary cohort, the results strongly contradict those in the primary cohort. If metformin is believed to have an effect irrespective of a person’s T2D severity, the T2D and prediabetes cohort results should be similar. However, in this case, a consideration of the treatment in the complementary cohort helps reveal the health-related residual confounding affecting the primary results.

Discussion

Principal Findings

In this cohort study, we replicated a common observational study design to highlight how it may lead to strong results that favor metformin’s association with a meaningful reduction in inpatient admissions and health care expenditures for people with T2D. We then expanded the study design to include numerous negative control outcomes to expose potential bias and replicated the primary cohort analysis in a complementary cohort carefully selected to remove or reverse the bias identified in the primary cohort. This more comprehensive study design discredits the primary results, highlighting how frequently published metformin study designs may be insufficient to identify and address residual confounding. These findings are in line with those of the umbrella review of systematic reviews of metformin by Li et al,²¹ in which 167 meta-analyses of observational studies of metformin revealed an overall lack of reproducibility in randomized clinical trials and in other observational studies, which may be attributable to poorly addressed confounding.

Strengths and Limitations

A notable strength of this study is the deliberate inclusion of a complementary cohort (prediabetes) in which the exposed group of metformin users is expected to be less healthy than its comparison group. Instead of relying on covariate adjustments to balance the comparison groups from a position that favors the exposure, we test the ability of these same covariate adjustments to offset an expected health disadvantage. This complementary cohort approach serves a fundamentally different purpose than simply replicating the same design in different data sets wherein a flawed design might lead to the same residual confounding issues. Others have described similar approaches, most notably in the description of triangulation for epidemiology provided by Lawlor et al.²² We incorporate many of the same principles by generating multiple lines of evidence to evaluate a hypothesis using different populations and different control groups (eg, multiple data sets: commercially insured vs Medicare Advantage; multiple control groups: insulin users or nonusers), conducting numerous negative control experiments, and examining the treatment effect in a complementary cohort with different biases or confounding structures.

Despite the advantages of this study design, it has limitations common to claims-based studies, such as including only individuals with qualifying health insurance, selective capture of variables associated with or incentivized by financial reimbursement, and missingness of key elements of diabetes severity, such as diabetes duration.^23,24 The prevalent user design is used to replicate commonly published literature, but it is a weaker strategy than the incident user design for identifying associated treatment outcomes. In addition, the proposed design relies on a qualitative assessment of diagnostic plots presenting distributions of potentially dependent effect estimates (eg, the negative control outcomes include related pulmonary conditions and related musculoskeletal conditions). There is increasing interest in empirical calibration techniques that consider evidence against a backdrop of negative controls exhibiting bias, although these approaches have yet to be widely adopted.²⁵ In addition, we recognize that by further controlling type 1 error (false positive), we are necessarily increasing the likelihood of a type 2 error (false negative).

Although negative control experiments are recommended as best practice by leading collaboratives, such as Observational Health Data Sciences and Informatics, they are not commonly reported in observational drug-repurposing studies.²⁴ Numerous negative control experiments are even rarer. Challenges to a primary result by triangulation with a complementary cohort are virtually nonexistent. The importance of these extra validation steps stems from the inadequacy of the observable health status variables readily available in claims data to produce an unbiased estimate of the treatment effect.

Some claims-based study limitations are addressed in other types of study designs, such as prospective observational studies, registry-based studies, and randomized clinical trials. In addition, deriving baseline comorbidity profiles, medication exposure histories, or other key variables of interest can be accomplished using alternative strategies, such as joining claims data to electronic health records data, clinical notes, or self-reported data to obtain a more accurate view of an individual’s overall health. However, such strategies often come at the expense of sample size and even when executed well may still exhibit similar biases.

A challenge associated with the complementary cohort design is the identification of a suitable complementary cohort. Applying our proposed framework to other diseases with a progressive trajectory or predisease state (eg, kidney disease) would be a natural extension; however, settings that involve primary and complementary cohorts from different diseases would introduce additional challenges. Ultimately, residual confounding could emerge from any insufficiently understood or observed prescribing pattern, leading to comparison groups with unmeasured differences. A key ingredient in the complementary cohort approach is that we expect the bias to be eliminated or possibly reversed between the 2 cohorts. Complementary cohorts could even result from differences in prescribers who are early vs late adopters, possibly because of expanded access as safety concerns decrease or as changes in third-party payer approvals increase over time. Substantial background research related to differential and potentially evolving prescribing patterns could provide the necessary foundation for a complementary cohort analysis.

Conclusions

By replicating a common observational study design for metformin, we found that standard efforts to eliminate residual confounding fall short. We believe such confounding always exists to some degree, so when there is any doubt about the comparability of various exposure groups, additional steps like those proposed by Observational Health Data Sciences and Informatics and those shown here must be performed. Observational studies can motivate randomized clinical trials, influence clinical practice in settings where gold standard, prospective evidence is sparse, and get amplified by media outlets to the public; the cost of overvaluing a poorly designed observational study can lead to wasted research funds and even put participants’ and patients’ health at unnecessary risk.

Supplement.

eAppendix. Supplemental Literature Review