Predictors of Metformin Failure: Repurposing Electronic Health Record Data to Identify High-Risk Patients

Abstract

Context

Metformin is the first-line drug for treating diabetes but has a high failure rate.

Objective

To identify demographic and clinical factors available in the electronic health record (EHR) that predict metformin failure.

Methods

A cohort of patients with at least 1 abnormal diabetes screening test that initiated metformin was identified at 3 sites (Arizona, Mississippi, and Minnesota). We identified 22 047 metformin initiators (48% female, mean age of 57 ± 14 years) including 2141 African Americans, 440 Asians, 962 Other/Multiracial, 1539 Hispanics, and 16 764 non-Hispanic White people. We defined metformin failure as either the lack of a target glycated hemoglobin (HbA1c) (<7%) within 18 months of index or the start of dual therapy. We used tree-based extreme gradient boosting (XGBoost) models to assess overall risk prediction performance and relative contribution of individual factors when using EHR data for risk of metformin failure.

Results

In this large diverse population, we observed a high rate of metformin failure (43%). The XGBoost model that included baseline HbA1c, age, sex, and race/ethnicity corresponded to high discrimination performance (C-index of 0.731; 95% CI 0.722, 0.740) for risk of metformin failure. Baseline HbA1c corresponded to the largest feature performance with higher levels associated with metformin failure. The addition of other clinical factors improved model performance (0.745; 95% CI 0.737, 0.754, P < .0001).

Conclusion

Baseline HbA1c was the strongest predictor of metformin failure and additional factors substantially improved performance suggesting that routinely available clinical data could be used to identify patients at high risk of metformin failure who might benefit from closer monitoring and earlier treatment intensification.

Diabetes is a highly prevalent condition and a major risk factor for cardiovascular disease. It is estimated that 26 million adults in the United States have been diagnosed with type 2 diabetes mellitus (henceforth referred to as diabetes), 9.4 million have undiagnosed disease, and 91.8 million have elevated glucose (>100 mg/dL) without meeting the criteria for diabetes (ie, prediabetes) (1). Racial and ethnic minorities are disproportionally affected. Compared with White people, the prevalence of diabetes in the United States is twice as high in African Americans and 35% higher in Latino Americans (2). The enormous burden of this disease translates into an estimated annual cost of $327 billion accounting for 1 in 4 health care dollars (3).

For reasons of efficacy, safety, and cost, metformin is the first-line drug for treating diabetes (4). However, it is estimated that there is a 35% failure rate for metformin monotherapy and a further 5% experience an adverse event (5–8). Prior studies investigating baseline factors with drug response have had mixed results. Metformin response has been associated with baseline glycated hemoglobin (HbA1c) (8–10), body mass index (BMI) (11, 12), age (10), fasting glucose (10, 11), retinopathy (9), liver enzymes (alanine aminotransferase and aspartate aminotransferase) (13), fasting insulin (10), and estimated glomerular filtration rate (10), albeit not consistently (14, 15). Likewise, duration of diabetes has been associated with metformin response, but not in all studies (9, 15). Furthermore, these studies have been conducted in predominantly White populations and it is unclear whether factors previously associated with metformin response differ across populations.

Epidemiological studies leveraging large electronic health record (EHR) datasets have demonstrated utility in facilitating the discovery of risk biomarkers. However, EHR data present a number of challenges when considering more agnostic scans for risk factors of adverse clinical outcomes, including but not limited to the high-dimensional feature space, heterogeneous data formats, hierarchical feature representations, and complex missingness patterns. Combining EHR data sources with machine learning–based strategies can yield discovery opportunities in personalized medicine (16, 17) via sophisticated predictive modeling and feature importance interpretation.

To study these questions, we identified persons initiating metformin therapy in 3 sites across the United States located in Arizona, Mississippi, and Minnesota. We analyzed clinical data available in the EHR systems with a machine learning approach to identify demographic and clinical predictors of metformin failure.

Materials and Methods

Data Sources

The University of Mississippi Medical Center (UMMC) is an academic medical center located in Jackson, Mississippi, and includes 6 hospitals and 23 community clinics. UMMC has both internal medicine and family medicine clinics that serve as the primary care home for patients in the city of Jackson, with over 10 000 visits per month conducted by approximately 300 physicians. The majority of the residents of Jackson are African American.

Mountain Park Health Center (MPHC) is a Federally Qualified Community Health Center with 7 clinics that provide integrated primary care to more than 90 000 patients strategically located across the greater Phoenix area. Many of these patients are Latino and fall below the federal poverty level. MPHC's team of approximately 1000 staff members includes over 175 licensed health care providers (family physicians, internists, obstetrician-gynecologists, pediatricians, nurse practitioners, dentists, clinical psychologists, clinical pharmacists, and licensed clinical social workers).

The Rochester Epidemiology Project (REP) is a medical records linkage system that includes EHR data from local health care providers for 1.7 million persons who have lived in the 27-county Midwest region after January 1, 2010 (18, 19). The REP captures approximately 61% of the entire population residing in this region (18). The REP includes EHR data from many sources of care across the region including the largest providers of care in these areas (ie, Mayo Clinic, Mayo Clinic Health System clinics and hospitals, and Olmsted Medical Center and its affiliated clinics). Thus, the REP captures and updates comprehensive EHR-derived phenotypic data within this population and is uniquely positioned to characterize outcomes in communities. The REP region is predominately White and has similar age, sex, and ethnic characteristics as the entire Upper Midwest region of the United States (18, 19).

Cohort Description

Patients with at least 1 abnormal diabetes screening test that initiated metformin were identified at each site and followed through their EHR for 18 months. UMMC and MPHC completed their EHR implementations in 2012 and patients were identified starting on January 1, 2014 to allow prevalent metformin users to be excluded. In the REP, metformin initiators were identified starting on January 1, 2010. Regardless of start date, patients meeting inclusion criteria at all sites were identified through June 30, 2017. Qualifying abnormal tests included a fasting glucose ≥100 mg/dL, random glucose ≥140 mg/dL, or HbA1c ≥5.7%. We excluded all inpatient point of care glucose tests, glucose measured from tissue or other nonblood fluid, glucose tolerance tests, and all glucose or HbA1c tests measured within 10 months of a pregnancy. Pregnancy was determined by diagnoses and procedure codes (Table S1 (20)). Index date was defined as the date of first metformin prescription.

Metformin Failure Definition

We considered all HbA1c measured 45 days postmetformin initiation to allow for changes due to therapy to be measured. We defined metformin failure as either the failure to achieve or to maintain a target HbA1c (<7%) within 18 months of index date or the addition of other pharmacotherapeutic agents (Table S2 (20)).

Clinical Predictors

Baseline data for each patient were collected up to 5 years prior to the index date to allow the capture of prevalent conditions and other clinical measurements that may be measured over multiyear intervals (eg, lipid panels). Demographic variables included age at index date, sex, and race/ethnicity. The closest height and weight within ±5 years of the index date were extracted, and BMI was calculated [weight (kg)/height (m)²]. Smoking and tobacco use were dichotomously categorized as current user or not. All diagnosis codes (International Classification of Diseases, ninth revision and tenth revision) during the baseline period were classified according to the Clinical Classifications Software (CCS), developed at the Agency for Healthcare Research and Quality (21). CCS is a tool for clustering patient diagnoses and procedures into a manageable number of clinically meaningful categories. The closest heart rate and systolic and diastolic blood pressure measured prior to metformin initiation was used. Common screening laboratory tests were extracted (eg, lipids, liver function, creatinine, complete blood count, etc.) as well as vitamin B12. Baseline HbA1c was defined as the closest measurement to metformin initiation including up to 14 days postmetformin initiation.

Data Analyses

Subjects were classified as African American, Asian, Other/Multiracial, Hispanic (any race), or non-Hispanic White. Time to metformin failure was calculated from metformin initiation to HbA1c >7 or initiation of dual therapy. For time to metformin failure, follow-up was right censored at 18 months; subjects who did not fail or switch treatments were censored at 18 months. Cumulative survival probabilities were estimated using the Kaplan–Meier method. Kaplan–Meier survival curves were calculated for each race/ethnicity and comparisons across groups were made using the log rank test. Given the large number of candidate predictors and their potentially complex relationships with risk of metformin failure, we used a machine learning–based approach to address potential prediction performance of metformin failure. We trained tree-based extreme gradient boosting (XGBoost) models based on the Cox proportional hazards extension for time to metformin failure using 2 feature sets, permitting missing values to be explicitly handled by the XGBoost algorithm. Specifically, when a variable is selected and a corresponding threshold is defined for branch split directions (ie, left vs right), if missing values are encountered, the algorithm additionally learns the optimal direction of the split for observations with missing values. Based on previous findings (22), feature set 1 included baseline HbA1c and demographic variables (ie, age, sex, and race/ethnicity) and feature set 2 expanded to include all 150 clinical and demographic variables. To provide a measure of internal validation, the model training was performed on a randomly sampled training subset (70%) using the XGBoost R package in conjunction with Bayesian-based hyperparameter tuning using mlrMBO (23). Risk prediction performance in the leave out test set was evaluated based on Harrell's C-index. Feature importance for risk prediction and model interpretation was performed using SHapley Additive exPlanations (SHAP) analysis (24) and corresponding dependence plots. Feature importance summarized by mean absolute SHAP value for a given feature can be interpreted as the average magnitude of effect of that feature on the log-relative-hazard component in the underlying Cox model, summarized across all subjects. This study was approved by the Mayo Clinic and Olmsted Medical Center Institutional Review Boards. The study was considered minimal risk by both Institutional Review Boards, thus the requirement for informed consent was waived. However, patients who did not provide authorization to use their medicals records for research were excluded.

Results

Across the 3 sites, we identified 22 047 metformin initiators of whom 48% were female with a mean age of 57 ± 14 years and BMI of 35 ± 8 kg/m² (Table 1). We observed 9407 metformin failures (43%) by 18 months of follow-up (Fig. S1 (20)). Crude metformin failure rates differed by race/ethnicity with African Americans, Other/Multiracial, and Hispanic people experiencing higher failure rates than Asians and non-Hispanic White people (Fig. S2 (20)). In those with an event, median time to metformin failure was 3.9 months and ranged from 3.2 months in African Americans to 4.4 months in Asians (Table S3 (20)).

Table 1.

Study cohort characteristics by race/ethnicity, count, mean (SD), or percentage

	Full cohorta	African American	Asian	Other/Multiracial	Hispanic	Non-Hispanic White
n	22 047	2141	440	962	1539	16 764
Sex, % female	48	58	54	54	56	46
Age, years	57 (14)	52 (14)	53 (15)	50 (14)	50 (13)	58 (14)
Age range, years	18-99	18-97	20-89	18-92	18-91	18-99
BMI, kg/m2	35 (8.0)	35 (9.3)	30 (5.9)	35 (7.7)	35 (7.8)	36 (7.9)
Missing, %	22	25	18	20	10	23
Current smoker, % yes	16	21	12	16	13	16
Missing, %	12	4	13	14	8	13
Current tobacco user, % yes	19	22	15	23	16	19
Missing, %	16	12	17	43	63	10
Site, %
Mountain Park Health Center	8	10	7	35	64	2
University of Mississippi Medical Center	11	70	3	6	<1	5
Rochester Epidemiology Project	81	20	90	59	36	93

Abbreviation: BMI, body mass index.

There were 201 subjects with race/ethnicity missing who were included in the analyses.

All the variables assessed (n = 150) were grouped into demographic/lifestyle (Table 1), vital stats/laboratory (Table S4 (20)), and clinical conditions (Table S5 (20)). Distributions of most of the variables studied differed by race/ethnic groups. For example, African Americans consistently had worse vital status and laboratory measures than other populations (Table S4 (20)). However, African Americans tended to be less likely to be diagnosed with many of the clinical conditions that were considered (Table S5 (20)).

The XGBoost model for risk of metformin failure with feature set 1 corresponded to a C-index of 0.731 (95% CI 0.722, 0.740) in our leave out test set. Baseline HbA1c corresponded to the largest feature performance with higher levels associated with metformin failure. Other variables were positively associated with metformin failure, including increasing age, male sex, and race/ethnicity.

The 15 most influential variables predicting metformin failure based on the trained XGBoost model from feature set 2 are shown in Table 2 and Fig. 1A. The addition of these 15 factors led to a significant improvement in the model performance (C-index of 0.745; 95% CI 0.737, 0.754, P < .0001). Baseline HbA1c still had the highest feature performance with higher levels associated with metformin failure (Fig. 1B). All remaining variables corresponding to feature importance had values <0.055 (Table 2; Figs. S3-13 (20)). Of these, older age and higher levels of potassium, triglycerides, and mean cell hemoglobin were predictive of metformin failure. Likewise, the presence of “diabetes with complications” (CCS 50) and a higher heart rate were predictive of metformin failure. By contrast, the presence of “other screening for suspected conditions” (CCS 258) and “medical examination/evaluation” (CCS 256), and lower levels of sodium, albumin, and HDL were all inversely related to metformin failure. Three variables demonstrated a U-shape relationship with metformin failure: BMI, LDL, and creatinine. Sex and race/ethnicity were not among the top 15 predictors.

Table 2.

The top 15 most influential variables predicting metformin failure in the complete feature set ranked by variable importance in the combined population

Variable	Mean absolute SHAP value	Direction
Baseline hemoglobin A1c	0.762	Positive
Other screening for suspected conditions (CCS 258)	0.055	Inverse
Diabetes with complications (CCS 50)	0.044	Positive
Body mass index	0.041	U-shape
Potassium	0.041	Positive
Medical examination/evaluation (CCS 256)	0.030	Inverse
Sodium	0.027	Inverse
Albumin	0.025	Inverse
Triglycerides	0.022	Positive
Heart Rate	0.021	Positive
Low-density lipoprotein	0.020	U-shape
Mean cell hemoglobin	0.020	Positive
Creatinine	0.020	U-shape
High-density lipoprotein	0.018	Inverse
Age	0.016	Positive

Abbreviation: CCS, Clinical Classifications Software; SHAP, SHapley Additive exPlanations.

Figure 1.

(A) Results from the complete feature set for the top 15 variables. (B) The distribution of SHAP values for all subjects by their baseline HbA1c. SHAP values less than 0 are tied to a decrease in probability; values greater than 0 are tied to an increased probability of metformin failure for that subject.

Discussion

We performed an agnostic scan to identify risk factors for metformin failure in a diverse population. We observed a high rate of metformin failure (33%) with higher failure rates in African Americans and Hispanics compared with non-Hispanic White people and Asians. However, these racial/ethnic differences disappeared once other clinical factors were considered, suggesting that the biological response to metformin is similar regardless of race/ethnicity. Of these clinical factors, baseline HbA1c was the strongest positive predictor of metformin failure. In addition, other routinely collected clinical data (eg, diagnostic codes, labs, vitals, etc.) were predictive of metformin failure. These results suggest that routinely available EHR data could be used to identify patients at high risk of metformin failure who might benefit from closer monitoring and earlier treatment intensification.

Metformin is a biguanide and antihyperglycemic medication approved for the management of diabetes when glycemic control cannot be accomplished by lifestyle modification alone. Metformin is also recommended for diabetes prevention in patients age <60 years and/or BMI ≥35 kg/m², or HbA1c of 5.7% to 6.4%, in whom lifestyle modifications failed to reduce hyperglycemia (4). Metformin is the initial therapy of choice due to its efficacy, weight neutral effect, general tolerability, favorable cost, and protection from cardiovascular events (25). Traditional recommendations have been to use stepwise addition of medications to metformin to maintain HbA1c at target. Our study shows that baseline HbA1c corresponded to the largest feature performance with higher levels associated with metformin failure. This result is largely confirmatory with a recent study showing that when HbA1c is ≥1.5% above the glycemic target, many patients will require dual combination therapy to achieve their target levels (22). In our study, a HbA1c above 7% was already associated with a high rate of failure, suggesting that early combination therapy might benefit these high-risk patients.

Of the more novel findings, 3 variables demonstrated a U-shape relationship with metformin failure: creatinine, BMI, and low-density lipoprotein. Skeletal muscle accounts for approximately 80% of overall glucose disposal after glucose challenge (26). A main source of plasma creatinine is the skeletal muscle and low skeletal mass is associated with insulin resistance (27). Low serum creatinine is a predictor of diabetes in Caucasian morbidly obese patients (28). While high levels of creatinine are always noticeable and in general lead to actions, there is some clinical inertia toward low levels of creatinine. According to our study, low levels of creatinine could signal a high potential for metformin failure and combination therapy could be recommended in selected patients. In observational studies, renal disease was associated with metformin failure (29). Additionally, metformin should not be used in patients with an estimated glomerular filtration rate <30 mL/min/1.73 m² because of an increased risk of lactic acidosis (30). Our study suggests that in the presence of high levels of creatinine the chances of metformin failure are increased as well; thereby, metformin should not be prescribed due to its lack of effectiveness and potential risk to induce lactic acidosis.

We observed a U-shaped relationship with metformin failure such that those with a BMI <25 and over 40 were most at risk for failure. The underlying pathology of normal weight/lean diabetes is incompletely understood (31). However, 1 feature appears to be decreased beta cell function as opposed to the peripheral insulin resistance that is a hallmark in overweight and obese diabetes. As metformin functions in part by increasing insulin sensitivity in the body tissues, it is less effective therapy in those whose diabetes is characterized more by decreased insulin secretory capacity. Furthermore, insulin resistance increases incrementally with increasing BMI and body fat (32). While it is known that BMI is an imperfect surrogate for body fat, the correlation between BMI and body fat was highest in the leanest and most obese groups (32). Thus, our results suggest that metformin in morbidly obese people (BMI >40) may be insufficient to address the extent of insulin resistance in these individuals.

We observed a relationship between metformin failure and abnormal laboratory results, which likely represent biomarkers for chronic illnesses. However, the effect size for laboratory abnormalities was small compared with that of baseline HbA1c, which is consistent with the associations previously observed between high HbA1c and metformin failure (29). We further discovered that metformin failure was associated with higher potassium levels. The higher potassium could indicate either kidney disease or use of an angiotensin receptor blocker and an angiotensin-converting enzyme inhibitor (33). Both of those medication classes would be used for kidney disease, hypertension, or congestive heart failure.

Several laboratory tests were inversely associated with metformin failure including sodium, hematocrit, hemoglobin, red blood cell, and albumin with more of a U-shaped association with low-density lipoprotein. Hyponatremia is a common in patients with severe chronic illness such as congestive heart failure, chronic renal disease, or hepatic failure. There has been association with metformin failure and congestive heart failure (7). Likewise, lower low-density lipoprotein cholesterol and albumin could be signs of frailty (34). The lower hemoglobin hematocrit and red blood cell count may indicate anemia. There is a potential relationship between metformin and anemia as seen in the UK Prospective Diabetes Study and Genetics of Diabetes Audit and Research in Tayside Scotland (GoDARTS) trial (35); thus, anemia may be a direct potential cause of metformin failure and discontinuation.

We also observed an unexpected inverse association between metformin failure and having screening and medical evaluations prior to baseline. Possible explanations could be that these types of interactions with the medical system are surrogates for underlying characteristics such as patient compliance, health care access, health care literacy, patient engagement with health care system, or continuity of care (ie, an established relationship with the provider). Persons with diabetes need to be able to implement a robust management plan to achieve optimal glycemic control. Previous research has reported that missed appointment rate, independent of other demographic and clinical factors, was predictive of poor diabetes control (36). Thus, diabetes patients with an established relationship with medical providers as evidenced by undergoing screening and medical evaluation likely have better glucose control.

The strengths of this study include the use of a large, diverse population from 3 geographic regions in the United States. In addition, we agnostically assessed the predictive value of numerous clinical factors commonly available in the EHR to study an important clinical issue. This agnostic approach is significant given the inconsistencies of prior study results in predominantly White populations, and suggests that racial and ethnic differences in metformin response can be explained by underlying clinical characteristics. Although, the translation of this model to a clinical setting would be premature, the model itself demonstrates proof of concept as to the potential performance of predicting metformin failure using real-world data. Limitations of the study include the observational design and the potential for bias due to missing data (ie, predictors and outcomes). In addition, we used data on prescribed metformin and thus compliance with metformin therapy could not be assessed.

We observed a high rate of metformin failure and identified numerous clinical factors predictive of metformin failure that are easily accessible and available in EHR systems in 3 racially and ethnically diverse populations. Our study results suggest increased monitoring with potentially earlier treatment intensification to achieve glycemic control may be appropriate in patients with clinical parameters described in this paper. Further, these results call into question the ubiquitous use of metformin as the first line therapy and suggest a more individualized approach may be needed to optimize therapy.

Abbreviations

BMI

body mass index

CCS

Clinical Classifications Software

EHR

electronic health record

HbA1c

glycated hemoglobin

MPHC

Mountain Park Health Center

REP

Rochester Epidemiology Project

SHAP

SHapley Additive exPlanations

UMMC

University of Mississippi Medical Center

XGBoost

extreme gradient boosting

Funding

This study used the resources of the Rochester Epidemiology Project (REP) medical records-linkage system, which is supported by the National Institute on Aging (NIA; AG 058738), by the Mayo Clinic Research Committee, and by fees paid annually by REP users. The study was funded by a grant from the Mayo Clinic Center for Health Disparity and Community Engagement Research. The study sponsor/funder was not involved in the design of the study; the collection, analysis, and interpretation of data; writing the report; and did not impose any restrictions regarding the publication of the report.

Author Contributions

S.J.B., P.Y.T, L.L.Y.C, N.B.L., and P.A.D. contributed to study concept and design, analysis and interpretation of results, drafting manuscript, and provided critical feedback and helped shape the research, analysis, and manuscript. S., A.C. and V.H. contributed to data collection and provided critical feedback and helped shape the research, analysis and manuscript. A.P., E.D.F., G.J.M., C.G., J.E.O., J.L.S., P.S., A.V., E.R., H.L., G.D.M., J.R.C., D.S., and R.L.S. contributed to study concept and design and provided critical feedback and helped shape the research, analysis and manuscript. J.M.K. contributed to data collection, analysis and interpretation of results, drafting manuscript, and provided critical feedback and helped shape the research, analysis and manuscript. All authors contributed to final approval for publication. S.J.B. is the guarantor of this work and, as such, had full access to all the data in the study, takes responsibility for the integrity of the data, accuracy of the data analysis and controlled the decision to publish.

Data Availability

The data from this study are available from the corresponding author on reasonable request.

Current Affiliation

Alexandra Castillo, Vice President, Data & Analytics at the University of Cincinnati Health System, Cincinnati, OH 45267.